Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster
Subseries of Lecture Notes in Computer Science
5632
Petra Perner (Ed.)
Machine Learning and Data Mining in Pattern Recognition 6th International Conference, MLDM 2009 Leipzig, Germany, July 23-25, 2009 Proceedings
13
Series Editors Randy Goebel, University of Alberta, Edmonton, Canada Jörg Siekmann, University of Saarland, Saarbrücken, Germany Wolfgang Wahlster, DFKI and University of Saarland, Saarbrücken, Germany Volume Editor Petra Perner Institute of Computer Vision and Applied Computer Sciences (IBal) Kohlenstr. 2, 04107 Leipzig, Germany E-mail:
[email protected]
Library of Congress Control Number: 2009930146
CR Subject Classification (1998): I.2, F.4, I.4, I.5, H.3 LNCS Sublibrary: SL 7 – Artificial Intelligence ISSN ISBN-10 ISBN-13
0302-9743 3-642-03069-6 Springer Berlin Heidelberg New York 978-3-642-03069-7 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12714995 06/3180 543210
Preface
There is no royal road to science, and only those who do not dread the fatiguing climb of its steep paths have a chance of gaining its luminous summits. Karl Marx A Universial Genius of the 19th Century
Many scientists from all over the world during the past two years since the MLDM 2007 have come along on the stony way to the sunny summit of science and have worked hard on new ideas and applications in the area of data mining in pattern recognition. Our thanks go to all those who took part in this year's MLDM. We appreciate their submissions and the ideas shared with the Program Committee. We received over 205 submissions from all over the world to the International Conference on Machine Learning and Data Mining, MLDM 2009. The Program Committee carefully selected the best papers for this year’s program and gave detailed comments on each submitted paper. There were 63 papers selected for oral presentation and 17 papers for poster presentation. The topics range from theoretical topics for classification, clustering, association rule and pattern mining to specific data-mining methods for the different multimedia data types such as image mining, text mining, video mining and Web mining. Among these topics this year were special contributions to subtopics such as attribute discretization and data preparation, novelty and outlier detection, and distances and similarities. The section “Aspects of Data Mining” presented work that considers automating the whole data mining process. The sections also cover data mining in medicine. Four papers were selected for the Best Paper Award. The final decision on the Best Paper was made during the conference and was based on the review results, the presentation style and the discussion. Please check www.mldm.de for information about the nominated papers and the final decision. We thank all members of the Institute of Applied Computer Sciences, Leipzig, Germany (www.ibai-institut.de) who handled the conference as secretariat. We appreciate the help and understanding of the editorial staff at Springer, and in particular Alfred Hofmann, who supported the publication of these proceedings in the LNAI series. Last, but not least, we wish to thank all the speakers and participants who contributed to the success of the conference. We would also like to encourage those who could not get their paper accepted to MLDM 2009 to work out the comments of the reviewers and resubmit their paper to our next conference. See you at MLDM 2011 (www.mldm.de).
July 2009
Petra Perner
International Conference on Machine Learning and Data Mining in Pattern Recognition MLDM 2009
Chair Petra Perner
Institute of Computer Vision and Applied Computer Sciences IBaI Leipzig, Germany
Program Committee Agnar Aamodt Jacky Baltes Max Bramer Horst Bunke Krzysztof Cios Christoph F. Eick Ana Fred Giorgio Giacinto Makato Haraguchi Robert J. Hilderman Tin Kam Ho Atsushi Imiya Horace Ip Abraham Kandel Dimitrios A. Karras Adam Krzyzak Longin Jan Latecki Tao Li Brian Lovell Mariofanna Milanova Thang V. Pham Maria da Graca Pimentel Petia Radeva Michael Richter Fabio Roli Linda Shapiro Sameer Singh
NTNU, Norway University of Manitob, Canada University of Portsmouth, UK University of Bern, Switzerland University of Colorado, USA University of Houston, USA Technical University of Lisbon, Portugal University of Cagliari, Italy Hokkaido University Sapporo, Japan University of Regina, Canada Bell Laboratories, USA Chiba University, Japan City University, Hong Kong University of South Florida , USA Chalkis Institute of Technology, Greece Concordia University, Montreal, Canada Temple University Philadelphia, USA Florida International University, USA University of Queensland, Australia University of Arkansas at Little Rock, USA University of Amsterdam, The Netherlands Universidade de São Paulo, Brazil Universitat Autonoma de Barcelona, Spain University of Calgary, Canada University of Cagliari, Italy University of Washington, USA Loughborough University, UK
VIII
Organization
David Steinberg Francesco Tortorella Patrick Wang
Tel Aviv University, Israel Universita' degli Studi di Cassino, Italy Northeastern University, USA
Additional Reviewers Indriyati Atmosukarto Jiun-Hung Chen Natalia Larios Dingding Liu Marta Penas Centeno Jia Wu Sara Rolfe Shulin Yang Mabel Raza Rosalia Tungaraza Katarzyna Wilamowska Tore Amble Richard Blake Tore Bruland Tor Gunnar Houeland Dung Manh Chu Francesco Fontanella Claudio Marrocco Mario Molinara Maria Teresa Ricamato Alessandra Scotto di Freca Paolo Simeone Davide Ariu Battista Biggio Luca Didaci Giorgio Fumera Gian Luca Marcialis Daniele Muntoni Ignazio Pillai Luca Piras Ajita Rattani Roberto Tronci
University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA University of Washington, USA NTNU, Norway NTNU, Norway NTNU, Norway NTNU, Norway University of Amsterdam, The Netherlands Universita' degli Studi di Cassino, Italy Universita' degli Studi di Cassino, Italy Universita' degli Studi di Cassino, Italy Universita' degli Studi di Cassino, Italy Universita' degli Studi di Cassino, Italy Universita' degli Studi di Cassino, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy University of Cagliari, Italy
Table of Contents
Attribute Discretization and Data Preparation Improved Comprehensibility and Reliability of Explanations via Restricted Halfspace Discretization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Klaus Truemper
1
Selection of Subsets of Ordered Features in Machine Learning . . . . . . . . . O. Seredin, A. Kopylov, and V. Mottl
16
Combination of Vector Quantization and Visualization . . . . . . . . . . . . . . . . Olga Kurasova and Alma Molyt˙e
29
Discretization of Target Attributes for Subgroup Discovery . . . . . . . . . . . . Katherine Moreland and Klaus Truemper
44
Preserving Privacy in Time Series Data Classification by Discretization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ye Zhu, Yongjian Fu, and Huirong Fu Using Resampling Techniques for Better Quality Discretization . . . . . . . . Taimur Qureshi and Djamel A. Zighed
53 68
Classification A Large Margin Classifier with Additional Features . . . . . . . . . . . . . . . . . . Xinwang Liu, Jianping Yin, En Zhu, Guomin Zhang, Yubin Zhan, and Miaomiao Li Sequential EM for Unsupervised Adaptive Gaussian Mixture Model Based Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bashar Awwad Shiekh Hasan and John Q. Gan
82
96
Optimal Double-Kernel Combination for Classification . . . . . . . . . . . . . . . . Feng Wang and Hongbin Zhang
107
Efficient AdaBoost Region Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Moed and E.N. Smirnov
123
A Linear Classification Method in a Very High Dimensional Space Using Distributed Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takao Kobayashi and Ikuko Shimizu
137
PMCRI: A Parallel Modular Classification Rule Induction Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Frederic Stahl, Max Bramer, and Mo Adda
148
X
Table of Contents
Dynamic Score Combination: A Supervised and Unsupervised Score Combination Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Roberto Tronci, Giorgio Giacinto, and Fabio Roli ODDboost: Incorporating Posterior Estimates into AdaBoost . . . . . . . . . . Olga Barinova and Dmitry Vetrov
163 178
Ensemble Classifier Learning Ensemble Learning: A Study on Different Variants of the Dynamic Selection Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jo˜ ao Mendes-Moreira, Alipio Mario Jorge, Carlos Soares, and Jorge Freire de Sousa
191
Relevance and Redundancy Analysis for Ensemble Classifiers . . . . . . . . . . Rakkrit Duangsoithong and Terry Windeatt
206
Drift-Aware Ensemble Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Frank Rosenthal, Peter Benjamin Volk, Martin Hahmann, Dirk Habich, and Wolfgang Lehner
221
Concept Drifting Detection on Noisy Streaming Data in Random Ensemble Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peipei Li, Xuegang Hu, Qianhui Liang, and Yunjun Gao
236
Association Rules and Pattern Mining Mining Multiple Level Non-redundant Association Rules through Two-Fold Pruning of Redundancies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Corrado Loglisci and Donato Malerba
251
Pattern Mining with Natural Language Processing: An Exploratory Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ana Cristina Mendes and Cl´ audia Antunes
266
Is the Distance Compression Effect Overstated? Some Theory and Experimentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephen France and Douglas Carroll
280
Support Vector Machines Fast Local Support Vector Machines for Large Datasets . . . . . . . . . . . . . . . Nicola Segata and Enrico Blanzieri The Effect of Domain Knowledge on Rule Extraction from Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nahla Barakat and Andrew P. Bradley
295
311
Table of Contents
Towards B-Coloring of SOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Haytham Elghazel and Khalid Benabdeslem
XI
322
Clustering CSBIterKmeans: A New Clustering Algorithm Based on Quantitative Assessment of the Clustering Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tarek Smaoui, Sascha M¨ uller, and Christian M¨ uller-Schloer Agent-Based Non-distributed and Distributed Clustering . . . . . . . . . . . . . . Ireneusz Czarnowski and Piotr J¸edrzejowicz An Evidence Accumulation Approach to Constrained Clustering Combination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Duarte Abdala and Xiaoyi Jiang Fast Spectral Clustering with Random Projection and Sampling . . . . . . . Tomoya Sakai and Atsushi Imiya How Much True Structure Has Been Discovered?: Validating Explorative Clustering on a Hold-Out Test Set . . . . . . . . . . . . . . . . . . . . . . F. H¨ oppner Efficient Clustering of Web-Derived Data Sets . . . . . . . . . . . . . . . . . . . . . . . Lu´ıs Sarmento, Alexander Kehlenbeck, Eug´enio Oliveira, and Lyle Ungar A Probabilistic Approach for Constrained Clustering with Topological Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khalid Benabdeslem and Jihene Snoussi
337 347
361 372
385 398
413
Novelty and Outlier Detection Relational Frequent Patterns Mining for Novelty Detection from Data Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michelangelo Ceci, Annalisa Appice, Corrado Loglisci, Costantina Caruso, Fabio Fumarola, Carmine Valente, and Donato Malerba
427
A Comparative Study of Outlier Detection Algorithms . . . . . . . . . . . . . . . . Charlie Isaksson and Margaret H. Dunham
440
Outlier Detection with Explanation Facility . . . . . . . . . . . . . . . . . . . . . . . . . Manuel Mej´ıa-Lavalle and Atl´ antida S´ anchez Vivar
454
XII
Table of Contents
Learning Concept Learning from (Very) Ambiguous Examples . . . . . . . . . . . . . . . . . Dominique Bouthinon, Henry Soldano, and V´eronique Ventos
465
Finding Top-N Pseudo Formal Concepts with Core Intents . . . . . . . . . . . . Yoshiaki Okubo and Makoto Haraguchi
479
On Fixed Convex Combinations of No-Regret Learners . . . . . . . . . . . . . . . Jan-P. Calliess
494
An Improved Tabu Search (ITS) Algorithm Based on Open Cover Theory for Global Extremums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kemal Y¨ uksek and Serhat Cakaloglu The Needles-in-Haystack Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Katherine Moreland and Klaus Truemper
505 516
Data Mining on Multimedia Data An Evidence-Driven Probabilistic Inference Framework for Semantic Image Understanding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Spiros Nikolopoulos, Georgios Th. Papadopoulos, Ioannis Kompatsiaris, and Ioannis Patras Detection of Masses in Mammographic Images Using Simpson’s Diversity Index in Circular Regions and SVM . . . . . . . . . . . . . . . . . . . . . . . Andr´e Pereira Nunes, Arist´ ofanes Corrˆea Silva, and Anselmo Cardoso de Paiva Mining Lung Shape from X-Ray Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vassili Kovalev, Aliaksandr Prus, and Pavel Vankevich A Wavelet-Based Method for Detecting Seismic Anomalies in Remote Sensing Satellite Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pan Xiong, Yaxin Bi, and Xuhui Shen Spectrum Steganalysis of WAV Audio Streams . . . . . . . . . . . . . . . . . . . . . . . Qingzhong Liu, Andrew H. Sung, and Mengyu Qiao Audio-Based Emotion Recognition in Judicial Domain: A Multilayer Support Vector Machines Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E. Fersini, E. Messina, G. Arosio, and F. Archetti Learning with a Quadruped Chopstick Robot . . . . . . . . . . . . . . . . . . . . . . . . Wei-Chung Lee, Jong-Chen Chen, Shou-zhe Wu, and Kuo-Ming Lin Dissimilarity Based Vector Space Embedding of Graphs Using Prototype Reduction Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kaspar Riesen and Horst Bunke
525
540
554
569 582
594 603
617
Table of Contents
XIII
Text Mining Using Graph-Kernels to Represent Semantic Information in Text Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Teresa Gon¸calves and Paulo Quaresma A General Framework of Feature Selection for Text Categorization . . . . . Hongfang Jing, Bin Wang, Yahui Yang, and Yan Xu New Semantic Similarity Based Model for Text Clustering Using Extended Gloss Overlaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Walaa K. Gad and Mohamed S. Kamel
632 647
663
Aspects of Data Mining Learning Betting Tips from Users’ Bet Selections . . . . . . . . . . . . . . . . . . . . ˇ ˇ Erik Strumbelj, Marko Robnik Sikonja, and Igor Kononenko
678
An Approach to Web-Scale Named-Entity Disambiguation . . . . . . . . . . . . Lu´ıs Sarmento, Alexander Kehlenbeck, Eug´enio Oliveira, and Lyle Ungar
689
A General Learning Method for Automatic Title Extraction from HTML Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sahar Changuel, Nicolas Labroche, and Bernadette Bouchon-Meunier Regional Pattern Discovery in Geo-referenced Datasets Using PCA . . . . . Oner Ulvi Celepcikay, Christoph F. Eick, and Carlos Ordonez Memory-Based Modeling of Seasonality for Prediction of Climatic Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Nikovski and Ganesan Ramachandran
704 719
734
A Neural Approach for SME’s Credit Risk Analysis in Turkey . . . . . . . . . G¨ ulnur Derelio˘glu, Fikret G¨ urgen, and Nesrin Okay
749
Assisting Data Mining through Automated Planning . . . . . . . . . . . . . . . . . Fernando Fern´ andez, Daniel Borrajo, Susana Fern´ andez, and David Manzano
760
Predictions with Confidence in Applications . . . . . . . . . . . . . . . . . . . . . . . . . Mikhail Dashevskiy and Zhiyuan Luo
775
Data Mining in Medicine Aligning Bayesian Network Classifiers with Medical Contexts . . . . . . . . . . Linda C. van der Gaag, Silja Renooij, Ad Feelders, Arend de Groote, Marinus J.C. Eijkemans, Frank J. Broekmans, and Bart C.J.M. Fauser
787
XIV
Table of Contents
Assessing the Eligibility of Kidney Transplant Donors . . . . . . . . . . . . . . . . Francisco Reinaldo, Carlos Fernandes, Md. Anishur Rahman, Andreia Malucelli, and Rui Camacho Lung Nodules Classification in CT Images Using Simpson’s Index, Geometrical Measures and One-Class SVM . . . . . . . . . . . . . . . . . . . . . . . . . . Cleriston Araujo da Silva, Arist´ ofanes Corrˆea Silva, Stelmo Magalh˜ aes Barros Netto, Anselmo Cardoso de Paiva, Geraldo Braz Junior, and Rodolfo Acatauass´ u Nunes Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
802
810
823
Improved Comprehensibility and Reliability of Explanations via Restricted Halfspace Discretization Klaus Truemper Department of Computer Science, University of Texas at Dallas, Richardson, TX 75083, U.S.A. Abstract. A number of two-class classification methods first discretize each attribute of two given training sets and then construct a propositional DNF formula that evaluates to True for one of the two discretized training sets and to False for the other one. The formula is not just a classification tool but constitutes a useful explanation for the differences between the two underlying populations if it can be comprehended by humans and is reliable. This paper shows that comprehensibility as well as reliability of the formulas can sometimes be improved using a discretization scheme where linear combinations of a small number of attributes are discretized. Keywords: Discretization, Logic, Explanation, Comprehensibility, Reliability.
1
Introduction
Suppose we are to explain the differences between two populations A and B. In the well-known approach assumed here, we take two training sets A and B from the populations A and B, discretize the attributes of the training sets, and construct a propositional disjunctive normal form (DNF) formula that evaluates to True for one of the two discretized training sets and to False for the other one. Example methods carrying out these steps in various ways are described in [1,2,4,8,10,11,17,18,19,24,25,47]. The formula can be viewed as a useful explanation of the differences between the populations if (1) the formula can be comprehended and interpreted by humans, and (2) it reliably predicts membership in the two populations. This paper shows that a certain discretization where linear combinations of a small number of attributes are discretized, may help in the construction of comprehensible and reliable explanations. We first discuss the size and comprehensibility of formulas.
2
Size and Comprehensibility of Formulas
Human comprehension of data or statements is an extensively covered topic of Neurology and Psychology. One of the key concepts is chunk, defined in [20] as P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 1–15, 2009. c Springer-Verlag Berlin Heidelberg 2009
2
K. Truemper
a collection of concepts that are closely related and that have much weaker connections with other concurrently used concepts. The seminal paper [40] defines a “magical number seven, plus or minus two” of chunks as limit of short-term memory storage capacity. Subsequent work refined the main claim of [40]. For a detailed review, see [20], which argues for a “magical number 4” of chunks. In related research, [31] establishes a limit of 4 for the number of variables humans can process. In [32] an integrated treatment of working memory capacity and relational capacity is proposed that generalizes the above results. The reference concludes that working memory is limited to approximately 3-4 chunks, and that the number of variables involved in reasoning is limited to 4. We rely on these bounds in our quest for comprehensible explanations. 2.1
Formula Size
Let us define the formula size FS of the logic formulas of interest here. The formulas are always in disjunctive normal form (DNF). Such a formula is a disjunction of DNF clauses, which in turn are conjunctions of literals. In this paper, each literal is a linear inequality of the form (bt ·z > α) or (bt ·z < α), where z is a vector of attribute variables, b is a vector of constants, and α is a scalar. An example of a DNF formula is [(x < 5)∧(y > 7)]∨(z > 1)∨[(y < 1)∧(z < 0)], with DNF clauses [(x < 5) ∧ (y > 7)], (z > 1), and [(y < 1) ∧ (z < 0)]. Consider a literal (bt · z > α) or (bt · z < α) with k ≤ 4 nonzero entries bj in the vector b. In agreement with the conclusions of [32], we assume that humans can readily understand the interaction of the items represented by the k terms bj · xj with nonzero bj and convert that information into one chunk that allows further reasoning. For the case k ≤ 3, graphical representation of the corresponding halfspace in k-dimensional Euclidean space can assist in that task. We call the translation of (bt · z > α) or (bt · z < α) with k ≤ 4 nonzero bj into just one chunk elementary chunking. Any other chunking is considered non-elementary. The formula size FS of a DNF formula where each literal is of the form (bt · z > α) or (bt · z < α) with k ≤ 4 nonzero bj is defined to be equal to the number of literals in the formula. Thus, it is equal to the number of chunks that can be derived by elementary chunking. We handle Boolean variables w in the above format by encoding the literals w and ¬w by the literals (x > 0.5) and (x < 0.5), respectively, using a {0, 1} variable x. Each such term contributes 1 to the FS value of the formula. There are other ways to define formula size, using different notions of chunking. We have chosen the above definition mainly because elementary chunking is consistent with [32]. In the next section, we encounter example logic formulas that only contain literals of Boolean variables. As a matter of brevity and convenience, we skip translation to the format where literals are inequalities and use the formula directly. FS of such a formula is defined to be equal to the number of literals of the formula, which would be the FS value of the appropriately translated formula version.
Improved Comprehensibility and Reliability of Explanations
2.2
3
Comprehension Difficulty
The approach of this section is motivated by the experiments reported in [31]. We want a measure of difficulty of comprehension of DNF formulas when such formulas serve as explanations. We must first agree on what comprehension means. For our purposes, a reasonable definition is that the person who supposedly has comprehended a formula has indeed integrated the logic information implied by the formula into his/her relevant knowledge. For example, suppose a formula has the value True if a patient benefits from a certain medication, and has the value False otherwise. A physician has comprehended the formula if he/she has integrated the logic information into the treatment decision process. We determine the comprehension difficulty CD of a formula S experimentally follows. Suppose n persons are proficient in a knowledge domain X. Let S be a DNF formula that applies to X. We give each person the formula S and ask whether S is compatible with X. Mathematically, answering the question is equivalent to deciding if S ∧ X has a satisfying solution. Suppose that k persons answer the question correctly. If k ≤ n/2, then human decision making essentially is not better than a random choice of answers. Accordingly, we estimate that comprehension of S by humans is not possible, and define CD to be ∞. Assume that k > n/2, and let t be the average time required to find an answer to the question. Reasonably, we want CD to be proportional to t and inversely proportional to the fraction k/n − 0.5, which measures the improvement of accuracy over random choice. We also want CD to have the value t if the n answers are 100% accurate. The following definition of CD meets these goals. t/(2k/n − 1) if k/n > 0.5 CD = (1) ∞ otherwise 2.3
Prediction of Comprehension Difficulty
We may be tempted to predict CD using FS. The main result of this section is that such a prediction generally is not possible. Two examples below prove that claim. But later we also show that, under certain conditions, a reasonable prediction of likely comprehensibility can be made. In the first example, a formula S with large FS applies to a knowledge domain X such that CD is small. This result is not unexpected, since non-elementary chunking can have that effect. In the second example, a formula S with small FS applies to a knowledge domain X such that CD is large. This result can be expected for certain pathological situations. But the example shows that this situation can also occur in rather straightforward situations of S and X. Example 1. The domain X covers grading of one exam. For j = 1, 2, . . . , n, let the Boolean variable xj have the value True if question j of the exam is answered correctly, and the value False otherwise. The formula S is to have the value True if the entire exam has at least two questions answered incorrectly. Regardless of the specific encoding of this condition, the size FS of S must
4
K. Truemper
grow at least linearly in n. But it is easy to check if S ∧ X is satisfiable, since non-elementary chunking reduces that check to trivial counting. Thus, CD is small even when F S is large. Example 2. In [31] it is shown that understanding food preference questions can be very difficult for humans. The following rephrasing of the results of [31] produces an example of low FS and high CD. The questioning process of [31] concerns cakes in various forms and the related preferences by humans. We sketch the setting. There are four variables: process, flavor, type, and richness. Each variable can take on two values, as follows: process = fresh or frozen, flavor = chocolate or carrot, type = iced or plain, and richness = rich or lowfat. The knowledge domain X consists of bar graphs depicting the desirability of cakes with various features. For example, a value may be given for fresh cakes containing chocolate and covered with icing. In each experiment, a person must evaluate a set of statements. The person is told that all statements of the set are correct except for the last one, which allows for two choices. The person must decide which of the two choices is the correct one, using the bar graphs of the knowledge domain X. There are four types of sets of statements, called 2×2-way, 3-way, 2×3-way, and 4-way. The simplest case is of type 2×2-way. The most difficult one is of type 4-way. For the sake of brevity, we discuss only those two cases and omit the intermediate 3-way and 2×3-way cases. Here is an example set of statements for the 2×2-way case. “People prefer fresh to frozen cakes. The difference [in the rating] depends on the flavor (chocolate vs carrot ). The difference between fresh and frozen is (greater/smaller) for chocolate cakes than for carrot cakes.” The person must decide which of the two options “greater” and “smaller” in the last statement is consistent with the knowledge domain X. In each set of statements of the 4-way cases, differences of rating differences are mentioned and must be evaluated. An example set of statements is as follows. “People prefer fresh to frozen cakes. The difference [in the rating] depends on the flavor (chocolate vs carrot ), the type (iced vs plain), and the richness (rich vs lowfat ). The difference between fresh and frozen increases from chocolate cakes to carrot cakes. This increase is greater for iced cakes than for plain cakes. There is a (greater/smaller) change in the size of the increase for rich cakes than for lowfat cakes.” The number of cakes with various features listed in the bar graphs depends on the case. Specifically, for the 2×2-way case, each instance of the knowledge domain X lists 4 different cakes and their ratings. For the 4-way case, that number is 16. Let us rephrase the setting and task using formulas that represent the statements in a naive way. For example, let (difference 1(fresh, frozen) > 0) be the single literal of a formula that evaluates to True if for all cakes with identical features except for process, the rating for fresh is higher than for frozen. This formula encodes the first statement of both examples for the 2×2-way and 4-way
Improved Comprehensibility and Reliability of Explanations
5
cases. In a similar fashion, each remaining statement can be encoded by just one literal that uses a new difference term. An exception are statements such as “The difference [in the rating] depends on the flavor (chocolate vs carrot )”, which state a restriction that is implied by the bar graphs of the knowledge domain X and thus need not be translated. For the encoding of the last statement, one case of the two options is arbitrarily selected. An entire set of statements can thus be represented by a DNF formula S that consists of just one DNF clause whose literals encode the statements. FS is equal to 2 for the 2×2-way case and equal to 4 for the 4-way case. Thus, FS is small in both cases. Consistency of S with the knowledge domain X holds if and only if the encoding of the last statement uses the correct option. Reference [31] includes results for a number of experiments involving various situations. The average solution times are displayed in a graph. Visual inspection determines the average solution time for the 2×2-way case to be 25 sec and for the 4-way case to be 74 sec. Correctness counts of answers are provided for pairs of experiments. The average correctness rate implied by the counts is 1.00 for the 2×2-way case and 0.65 for the 4-way case. Using (1), the corresponding CD values are 25 and 247, respectively. The table below shows FS and CD for the two cases. Formula Size and Comprehension Difficulty Case FS 2×2-way 2 4-way 4
CD 25 247
Evidently, the increase of FS from 2 to 4 causes an almost 10-fold increase of CD from 25 to 247. Reference [31] sketches results of experiments for a 5-way case built from two 4-way cases. Since [31] does not provide a detailed description of the set of statements of the case, FS of the corresponding DNF formula cannot be determined. But extrapolating from the structure of the previous cases, we conclude that FS must still be small. Unfortunately, the average solution time is not provided, so CD cannot be computed. But an indication of the comprehension difficulty is the fact that the correctness rate k/n is 0.55, which is barely above the average performance of random choice. Thus, it seems fair to conclude that the 5-way case is close to or at the limit of human comprehension. A potential criticism of the above encoding of sets of statements in formulas S is that it only implicitly accounts for the crucial concept of rating differences. But that objection would not be raised if the knowledge domain X could be easily restated using the same literals. Indeed, the main point of the example is the following. If the knowledge domain X isn’t already expressed or cannot be easily restated in the terminology of the sets of statements or, equivalently, in terms of the literals of S, then chunking is difficult or impossible. Furthermore, in such cases the human brain has great difficulty processing more than just a few statements or formulas.
6
K. Truemper
Despite the above examples, we would like to make some use of FS when estimating CD. We do this next. 2.4
Comprehensibility Condition
We say that a formula directly uses concepts of knowledge domain X if each literal directly corresponds to some concept of X. Suppose two formulas directly use concepts of X. In light of the preceding discussion and results, it is reasonable to assume that, in the absence of non-elementary chunking, the formula with smaller size FS is easier to understand. We summarize that assumption. Monotonicity Assumption. Let knowledge domain X and formulas that directly use concepts of X be given, and suppose that non-elementary chunking is not possible. Then for any two formulas, the one with smaller FS value is easier to understand. During the construction of a formula S, we do not know whether non-elementary chunking is possible when S is evaluated in light of a knowledge domain X. It seems prudent that we conservatively assume that such chunking is not possible. Under the Monotonicity Assumption, we then should strive for a formula S that directly uses concepts of X and has smallest size among the formulas with that feature. Indeed, in agreement with [32], we postulate the following condition for comprehensibility. Comprehensibility Condition. Let knowledge domain X and a formula S that directly uses concepts of X be given. If FS is at most 4, then the formula S likely is comprehensible. On the other hand, if FS is larger than 4 and nonelementary chunking is not possible, then comprehensibility of the formula S is doubtful.
3
Construction of Comprehensible Formulas
This section summarizes a method for the construction of formulas that likely are comprehensible according to Section 2.4. The method uses SV (single-variable) and RHS (restricted-half-space) discretization. SV discretization has the customary form; one or more cutpoints are defined for each variable, and logic variables represent the occurrence of values in the various intervals. Specifically, for each cutpoint α of a variable xj , we define a logic variable wj,α that takes on the value True if xj > α, and the value False if xj < α. RHS discretization is an extension of SV discretization. Here, for a vector z of original variables, we select a cutpoint α and a vector b containing a few nonzero entries—shortly we make this notion precise—, and define a logic variable wb,α to take on the value True if bt · z > α, and the value False if bt · z < α. We assume that a learning machine M is given that derives classifying formulas from discretized data. The construction process uses SV and RHS discretization and learning machine M in a stagewise process. In stage 0, the learning
Improved Comprehensibility and Reliability of Explanations
7
machine M receives training data discretized by SV discretization and computes a formula as output. Stages 1, 2, . . . are like stage 0, except that SV discretization is replaced by a RHS discretization procedure that increases in complexity with the stage index. Thus, each stage produces an output formula. For stage p, p = 0, 1, . . . , denote by Mp the machine consisting of the relevant discretization procedure and the subsequently executed learning machine M. As we shall see in Section 5, Mp may produce formulas whose literals (bt · z > α) or (bt · z < α) have up to 2p nonzero bj . Since we want comprehensible formulas, we limit 2p to 4 in agreement with Section 2.1, and thus enforce p ≤ 2. From the formulas obtained by the machines Mp , p ≤ 2, we want to select one that likely is best according to comprehensibility and reliability. In the long version of this paper, we describe how this can be done using the Monotonicity Assumption of Section 2.4 and the theory of VC dimension [48]. Here, we can only list the main conclusion: The selected formula should have high accuracy on the training data and small formula size. The next section reviews SV discretization.
4
Review of SV Discretization
In the early developments of discretization methods, entropy combined with the minimum description length principle ([22,23,45]) and other schemes using information gain were the most widely used methods, with strong performance with regard to prediction accuracy; see for example [3,4,21,35]. Recent research has produced a number of new discretization algorithms ([12,13,37,50,44]). Using a generalized entropy function, [33] unifies and generalizes results. The above references cover research on univariate discretization, where each variable is treated by itself. In multivariate discretization, several variables are considered simultaneously, and a discretization of each variable is determined that takes into account the interaction of the variable with other variables. This more elaborate effort is undertaken with the expectation that the subsequently applied learning method will discover classifiers that in some sense are more meaningful. Representative references are [9,14,15,26,41,43]. Formally, the output of multivariate discretization has the same structure as that of univariate discretization. That is, the output consists of cutpoints for the variables, and the variables are discretized according to these cutpoints. A fundamental problem induced by discretization is the fact that near any cutpoint of a variable xj a small random change of the xj value may change the encoding. There are several ways to deal with this difficulty. In the first approach, the discretization assigns several cutpoints to a given variable for a fine-grained representation. This solution has a hidden cost, in the form of more complex and thus less comprehensible formulas. In a second, fuzzy approach, membership functions are introduced that permit membership in several intervals; see for example [6]. Here, too, formulas can become complex and incomprehensible. A third approach introduces an uncertainty interval around each cutpoint; see for example [11], where the intervals are dynamically decided
8
K. Truemper
during the learning process. The disadvantage of the third approach is that the learning method must be able to process discretized data where some entries are marked “unknown.” RHS discretization relies on the third approach. Specifically, the width of the uncertainty interval enclosing a given cutpoint for a variable xj should be large enough that random changes of the xj value turn with low probability a value below the uncertainty interval into one above that interval, and vice versa. We define the average uncertainty width uj of variable xj to be the average of the widths of the uncertainty intervals enclosing the cutpoints of xj .
5
RHS Discretization
We assume to have (1) an SV discretization method that outputs for each variable xj besides the cutpoints an average uncertainty width uj , and (2) a feature selection method (see [30,36,39]) that outputs the value of a reasonable importance measure. The RHS discretization process is as follows. First, the scheme applies the assumed SV discretization to get importance values for the variables. Second, the scheme creates new variables that are linear combinations of variables. Third, the method uses SV discretization to obtain cutpoints for the new variables. Effectively, the cutpoints constitute thresholds of linear combinations of original variables. We pause for a moment to point out that the use of linear combinations of variables followed by application of thresholds is central to well-known techniques of Machine Learning. For example, artificial neural nets (ANNs) use the two steps as fundamental building block. As a second example, support vector machines (SVMs) use the two steps in a typically high dimensional space into which the training data have been mapped. While ANNs and SVMs in general allow linear combinations of any number of variables, here we use just pairs of variables. The rule for selecting the pairs depends on the stage of the overall process. For the moment, we skip discussion of that aspect and focus on the definition of new variables from one pair of variables, say involving xk and xl . For j = k, l, suppose SV discretization has created just one cutpoint cj for xj , and let uj denote the associated uncertainty width. Let R be the rectangle in the Euclidean plane that has the four points given by (ck ± uk /2, cl ± ul /2) as corner points. The discretization rules implied by the cutpoints and their uncertainty intervals assign for any point in R the value “unknown” to both coordinates. By the derivation of the cutpoints and uncertainty intervals, that assignment is based on the behavior of xk by itself and of xl by itself. Maybe analysis of the interaction of xk and xl would support a more precise discretization where R is replaced by a smaller region. There are several ways to achieve this goal. Here, we opt for the subdivision of the rectangle R induced by its two diagonals. We explore the effect of that subdivision rather indirectly, as follows.
Improved Comprehensibility and Reliability of Explanations
9
For some constants α1 and α2 , the lines passing through the two diagonals of R are given by xk /uk + xl /ul = α1 xk /uk − xl /ul = α2
(2)
We could revise the discretization of xk and xl directly using these two lines. But more effective is the following approach, which momentarily considers α1 and α2 to be undecided, and which defines two variables y + and y − by y + = xk /uk + xl /ul y − = xk /uk − xl /ul
(3)
We add these two variables as new attributes to the training data and compute training values for them by inserting the training values for xk and xl into (3). Due to this expansion of the space of attributes, we call the variables y + and y − expansion variables and refer to the enlarged training data as expanded training data. The subsequent use of the expansion variables is decided by SV discretization applied to the expanded training data. That step decides cutpoints for the expansion variables that generally may make better use, so to speak, of the expansion variables than selection of α1 and α2 of (2) as cutpoints. In the general case of several cutpoints for each of the variables xj , we use the average uncertainty width as uj in the definition (3) of y + and y − . In the construction process of Section 3, all variables on hand at the end of a stage, including all expansion variables created so far, are candidates for pairs producing additional expansion variables for the next stage. Since stage 0 carries out SV discretization, the variables available at the end of stage 0 are just the original variables. Thus, all expansion variables created in stage 1 are derived from original variables and, by (3), are defined by equations with two nonzero coefficients. Inductively, if the defining equations of the expansion variables created in stage p ≥ 1 are rewritten in terms of the original variables, then the resulting equations have at most 2p nonzero coefficients. We restate this as follows. Let y be a new expansion variable of stage p, p ≥ 1. Suppose the literal (y < α) or (y > α) occurs in a formula learned from the training data on hand in stage p. In terms of the vector x of original variables, the variable y is defined by y = bt · x, where b has at most 2p nonzero entries, and any literal (y > α) or (y < α) can be rewritten as (bt · x > α) or (bt · x < α), respectively. In Section 2.1, we constrained the literals (bt · z > α) or (bt · z < α) to b vectors with k ≤ 4 nonzero bj so that elementary chunking of the literal is possible. In the present setting, literals produced via RHS discretization observe that constraint if the number p of stages satisfies 2p ≤ 4 and thus p ≤ 2. From now on, we impose that limit on p. Finally, we note that nominal attributes, where values are members of a nonnumerical set, do not participate in RHS discretization. The next section discusses an implementation.
10
6
K. Truemper
Implementation and Computational Results
We have added RHS discretization to an extension of the method of [46], which in turn is based on the discretization method of [7,42] and the Lsquare algorithm of [24,25]. We call the resulting method EXRHS (= EXplanations via RHS discretization). For present purposes, the details of EXRHS and the prior scheme of [46] it is based on, are not important. Suffice it to say that the prior scheme computes uncertainty intervals as part of SV discretization and establishes importance values of variables in a feature selection subroutine. As a result, the addition of RHS discretization is rather straightforward. We use data sets of UC Irvine Machine Learning Repository for tests of EXRHS. The repository has a list of 11 most popular data set. Of these, 6 data sets are well suited for tests of RHS discretization since they involve no or almost no nominal attributes and allow easy formulation of two-class classification problems. In order of popularity, the selected 6 data sets are Iris, Wine, Breast Cancer Wisconsin, Abalone, Yeast, and SPECT Heart. For each test, we split the given data set in a 50/50 ratio into training and testing data and then apply EXRHS. The 50/50 choice provides in all cases reasonable training and testing sets for evaluation of the effect of RHS discretization. The Iris data set has a total of 150 records of three classes represented by 50 records each. Goal is to explain, for each class, the difference between that class and the remaining two classes. Thus, there are three cases: Iris-1 of class 1 versus classes 2, 3; Iris-2 of class 2 versus classes 1, 3; and Iris-3 of class 3 versus classes 1, 2. The Wine data set has a total of 178 records covering three classes of wines with 59, 71, and 48 records, respectively. The goal is to explain, for each class, the difference between that class and the remaining two classes. Thus, there are three cases Wine-1, Wine-2, and Wine-3 analogously to the Iris data set. The Breast Cancer Wisconsin data set has 569 records covering benign and malignant breast tumor cases. Goal is to explain the difference between the two diagnoses. The Abalone data set has a total of 4,177 records. Each record represents an abalone instance. Goal is to decide the number of rings of the instance using various measurements. The number of rings ranges from 1 to 29. In [16], a 3-class classification problem is defined for the Abalone data set analogously to the Iris data set, except that here class 1 has the records with 1-8 rings, class 2 has those with 9 or 10 rings, and class 3 contains those with 11-29 rings. The three tests cases are called Abalone-1, Abalone-2, and Abalone-3. The Yeast data set contains 1,484 records. The cellular localization sites of proteins are to be determined. There are ten classes. For the tests, we select the three most frequently occurring classes, called CYT, NUC, and MIT. For each case, the class is to be separated from the remaining cases. This produces three test cases Yeast-1, Yeast-2, and Yeast-3. The SPECT Heart data set has 267 records. In contrast to the above data sets, all attributes are binary. The goal is prediction of a {0, 1} DIAGNOSIS variable. The original data set is given by training and testing sets of somewhat odd proportions. That is, the training data have 40 records each for the two DIAGNOSIS cases, but the testing data have just 15 cases for DIAGNOSIS = 0 and 172 cases for DIAGNOSIS = 1. For consistency
Improved Comprehensibility and Reliability of Explanations
11
with the above tests, we concatenate the original training and testing sets and then split the resulting set 50/50 into training and testing sets for use by EXRHS. Looking over the data sets, it seems reasonable to claim that the attributes directly represent concepts of the underlying knowledge domain. Thus, it is also reasonable to invoke the Monotonicity Assumption and Comprehensibility Condition. Thus, formulas with size FS ≤ 4 likely are comprehensible. On the other hand, if non-elementary chunking cannot be done, then comprehensibility of formulas with size FS > 4 is doubtful. In all tests, EXRHS runs with the default rules and settings, and no selection or tuning of parameters is used. Suppose the output formulas of stages 0, 1, and 2 have been computed. We do not use application-dependent rules to select the final formula from the output formulas, since we have no guidance for selection of such rules. In agreement with the earlier discussion, we aim for a final formula that likely is comprehensible by eliminating all output formulas whose size exceeds 4, then select the formula with highest training accuracy from the remaining formulas. If case of a tie, the formula with lower stage index is chosen. In Table 1, the results for the final formulas are summarized in the three columns under the heading Best of Size ≤ 4; the column FS has the formula size, and the remaining two columns have the training and testing accuracy. We contrast the final formulas with the output formulas of Stage 0 in three analogous columns under the heading Stage 0 Output. These output formulas are constructed using SV discretization, and no restriction is imposed on their size. The columns under Best of Size ≤ 4 in Table 1 show that EXRHS finds likely-to-be-comprehensible explanations in 12 out of 14 (= 86%) of the cases. Table 1. Summary for Formulas Best of Size ≤ 4 Stage 0 Output Accuracy (%) Accuracy (%) Case FS Train Test FS Train Test Iris-1 1 98 97 1 98 97 Iris-2 3 95 91 3 90 85 Iris-3 1 95 98 1 92 98 Wine-1 1 97 95 1 93 88 Wine-2 2 94 92 1 92 89 Wine-3 2 100 98 2 100 98 Breast 2 95 93 10 97 92 Abal.-1 2 80 79 5 81 79 Abal.-2 4 64 63 13 64 63 Abal.-3 1 70 71 6 72 72 Yeast-1 no size ≤ 4 17 61 60 Yeast-2 no size ≤ 4 10 68 64 Yeast-3 1 73 75 12 76 78 SPECT 2 81 72 2 81 72 Average 1.8 86.8 85.3 4.8 86.3 84.3 The results for Yeast-1 and Yeast-2 are not included in the averages
12
K. Truemper
The formula size FS is equal to 1 or 2 with two exceptions, where the size is 3 or 4. The average formula size is 1.8. It turns out that RHS discretization is used for 9 out of the 12 (= 75%) likely-to-be-comprehensible explanations. The 9 cases are split into 3 (= 33%) cases computed in stage 1, while the remaining 6 cases (= 67%) are determined in stage 2. Of the 14 formulas listed under Stage 0 Output, only 7 cases (= 50%) have formula size ≤ 4 and are deemed likely comprehensible. Unless non-elementary chunking is possible, comprehensibility of the remaining 7 formulas is doubtful. The formula size FS ranges from 1 to 17. The average formula size is 4.8 if the formulas produced by Yeast-1 and Yeast-2, for which Best of Size ≤ 4 has no counterpart, are ignored. The accuracy averages at the bottom of Table 1 show that the improvement in comprehensibility due to RHS discretization is accompanied by a small average training accuracy gain of 86.8−86.3 = 0.5% and a small average testing accuracy gain of 85.3 − 84.3 = 1.0%. Thus, RHS discretization produces not only likelyto-be-comprehensible formulas for 86% of the cases, but does so with a small training and testing accuracy gain rather than a loss. Test results reported for prior schemes that compute logic formulas often cite the number of computed rules. Since the formula size of a rule is at least 1, the number of rules of an explanation is a lower bound on the formula size of the explanation. That lower bound can be quite large. For example, some papers list 12, 26, 19, 42, 77, 27-109, average 18, average 6, average 16, ≥ 18 rules for two-class classification problems handled by various methods. Maybe RHS discretization can help to reduce those numbers. Table 2 provides total execution times for EXRHS in min:sec, based on runs on a 3.0GHz PC. When three cases are listed together on a line, the stated execution time is the total run time for those cases. In addition, column Attr. gives the number of attributes, not counting the class attribute, and column Train Recds has the number of training records. Table 2. Execution Times of EXRHS Case Iris(1-3) Wine(1-3) Breast Abal.(1-3) Yeast(1-3) SPECT
7
Attr.
Train Recds
4 13 30 8 8 22
75 89 285 2,089 742 134
Exec. Time (min:sec) 0:01 0:02 0:03 1:57 0:18 0:01
Extension
EXRHS has been extended to carry out Subgroup Discovery. The scheme is somewhat different from typical current Subgroup Discovery approaches [34,49],
Improved Comprehensibility and Reliability of Explanations
13
which often use an iterative investigation involving the steps of data mining, interpretation of results, and evaluation by experts; see for example [5,27,28,29,38]. In contrast, EXRHS typically determines a few highly significant subgroups without manual guidance. Each subgroup is characterized by a convex polyhedron that generally isn’t just an axis-parallel rectangle.
8
Summary
Using prior studies in Neurology and Psychology, the paper links comprehensibility of explanations given by logic formulas to the size of the formulas. That is, if a formula is not too large in a certain sense, then under certain assumptions we can expect it to be understood by humans. On the other hand, if formula size exceeds a certain bound, then under certain assumptions it may well be the case that the formula cannot be comprehended by humans. The paper introduces a restricted form of discretization called RHS discretization that can be computed with the aid of any number of prior discretization and feature selection methods. In computational tests using well-known data sets, it is shown that RHS discretization produces likely-to-be-comprehensible explanations for 86% of the cases. In contrast, when the traditional discretization approach via cutpoints is used, only 50% of the explanations have that feature. The improvement of comprehensibility is accompanied by a small average gain in accuracy instead of a loss. The methodology has been extended to cover Subgroup Discovery.
References 1. Abidi, S., Hoe, K.: Symbolic exposition of medical data-sets: A data mining workbench to inductively derive data-defining symbolic rules. In: Proceedings of the 15th IEEE Symposium on Computer-based Medical Systems (CBMS 2002) (2002) 2. Agrawal, R., Imielinski, T., Swami, A.N.: Mining association rules between sets of items in large databases. In: Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data (1993) 3. An, A.: Learning classification rules from data. Computers and Mathematics with Applications 45, 737–748 (2003) 4. An, A., Cercone, N.: Discretization of continuous attributes for learning classification rules. In: Zhong, N., Zhou, L. (eds.) PAKDD 1999. LNCS, vol. 1574, pp. 509–514. Springer, Heidelberg (1999) 5. Atzmueller, M., Puppe, F., Buscher, H.-P.: Subgroup mining for interactive knowledge refinement. In: Miksch, S., Hunter, J., Keravnou, E.T. (eds.) AIME 2005. LNCS, vol. 3581, pp. 453–462. Springer, Heidelberg (2005) 6. Au, W.-H., Chan, K.C.C., Wong, A.K.C.: A fuzzy approach to partitioning continuous attributes for classification. IEEE Transactions on Knowledge and Data Engineering 18, 715–719 (2006) 7. Bartnikowski, S., Granberry, M., Mugan, J., Truemper, K.: Transformation of rational and set data to logic data. In: Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Springer, Heidelberg (2006)
14
K. Truemper
8. Bay, S., Pazzani, M.: Detecting group differences: Mining contrast sets. Data Mining and Knowledge Discovery 5, 213–246 (2001) 9. Bay, S.D.: Multivariate discretization of continuous variables for set mining. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining (2000) 10. Boros, E., Hammer, P., Ibaraki, T., Kogan, A.: A logical analysis of numerical data. Mathematical Programming 79, 163–190 (1997) 11. Boros, E., Hammer, P., Ibaraki, T., Kogan, A., Mayoraz, E., Muchnik, I.: An implementation of logical analysis of data. IEEE Transactions on Knowledge and Data Engineering 12, 292–306 (2000) 12. Boull´e, M.: Khiops: A statistical discretization method of continuous attributes. Machine Learning 55, 53–69 (2004) 13. Boull´e, M.: MODL: A Bayes optimal discretization method for continuous attributes. Machine Learning 65, 131–165 (2006) 14. Chao, S., Li, Y.: Multivariate interdependent discretization for continuous attribute. In: Proceedings of the Third International Conference on Information Technology and Applications (ICITA 2005)(2005) 15. Chmielewski, M.R., Grzymala-Busse, J.W.: Global discretization of continuous attributes as preprocessing for machine learning. International Journal of Approximate Reasoning 15, 319–331 (1996) 16. Clark, D., Schreter, Z., Adams, A.: A quantitative comparison of dystal and backpropagation. In: Proceedings of Seventh Australian Conference on Neural Networks (ACNN 1996) (1996) 17. Clark, P., Boswell, R.: Rule induction with CN2: Some recent improvements. In: Proceedings Fifth European Working Session on Learning (1991) 18. Cohen, W.W.: Fast effective rule induction. In: Machine Learning: Proceedings of the Twelfth International Conference (1995) 19. Cohen, W.W., Singer, Y.: A simple, fast, and effective rule learner. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence (1999) 20. Cowan, N.: The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences 24, 87–185 (2001) 21. Dougherty, J., Kohavi, R., Sahami, M.: Supervised and unsupervised discretization of continuous features. In: Machine Learning: Proceedings of the Twelfth International Conference (1995) 22. Fayyad, U., Irani, K.: On the handling of continuous-valued attributes in decision tree generation. Machine Learning 8, 87–102 (1992) 23. Fayyad, U., Irani, K.: Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence (1993) 24. Felici, G., Sun, F., Truemper, K.: Learning logic formulas and related error distributions. In: Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Springer, Heidelberg (2006) 25. Felici, G., Truemper, K.: A MINSAT approach for learning in logic domain. INFORMS Journal of Computing 14, 20–36 (2002) 26. Friedman, N., Goldszmidt, M.: Discretizing continuous attributes while learning Bayesian networks. In: International Conference on Machine Learning (1996) 27. Gamberger, D., Lavraˇc, N.: Expert-guided subgroup discovery: Methodology and application. Journal of Artificial Intelligence Research 17, 501–527 (2002) 28. Gamberger, D., Lavraˇc, N., Krstaˇcic, G.: Active subgroup mining: a case study in coronary heart disease risk group detection. Artificial Intelligence in Medicine 28 (2003)
Improved Comprehensibility and Reliability of Explanations
15
ˇ 29. Gamberger, D., Lavraˇc, N., Zelezn´ y, F., Tolar, J.: Induction of comprehensible models for gene expression datasets by subgroup discovery methodology. Journal of Biomedical Informatics 37 (2004) 30. Guyon, I., Elisseef, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, 1157–1182 (2003) 31. Halford, G.S., Baker, R., McCredden, J.E., Bain, J.D.: How many variables can humans process? Psychological Science 16, 70–76 (2005) 32. Halford, G.S., Cowan, N., Andrews, G.: Separating cognitive capacity from knowledge: a new hypothesis. Trends in Cognitive Sciences 11, 236–242 (2007) 33. Jin, R., Breitbart, Y., Muoh, C.: Data discretization unification. In: Proceedings of the IEEE International Conference on Data Mining (ICDM 2007) (2007) 34. Kl¨ osgen, W.: EXPLORA: A multipattern and multistrategy discovery assistant. In: Advances in Knowledge Discovery and Data Mining. AAAI Press, Menlo Park (1996) 35. Kohavi, R., Sahami, M.: Error-based and entropy-based discretization of continuous features. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (1996) 36. Koller, D., Sahami, M.: Toward optimal feature selection. In: International Conference on Machine Learning (1996) 37. Kurgan, L.A., Cios, K.J.: CAIM discretization algorithm. IEEE Transactions on Knowledge and Data Engineering 16, 145–153 (2004) 38. Lavraˇc, N., Cestnik, B., Gamberger, D., Flach, P.: Decision support through subgroup discovery: Three case studies and the lessons learned. Machine Learning 57, 115–143 (2004) 39. Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17, 491– 502 (2005) 40. Miller, G.A.: The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 63, 81–97 (1956) 41. Monti, S., Cooper, G.F.: A multivariate discretization method for learning Bayesian networks from mixed data. In: Proceedings of the Fourteenth Conference of Uncertainty in AI (1998) 42. Mugan, J., Truemper, K.: Discretization of rational data. In: Proceedings of MML 2004 (Mathematical Methods for Learning). IGI Publishing Group (2007) 43. Muhlenbach, F., Rakotomalala, R.: Multivariate supervised discretization, a neighborhood graph approach. In: Proceedings of the IEEE International Conference on Data Mining (ICDM 2002) (2002) 44. Perner, P., Trautzsch, S.: Multi-interval discretization for decision tree learning. In: Advances in Pattern Recognition. Springer, Heidelberg (2004) 45. Quinlan, J.: Induction of decision trees. Machine Learning 1, 81–106 (1986) 46. Riehl, K.: Data Mining Logic Explanations from Numerical Data. PhD thesis, Department of Computer Science, University of Texas at Dallas (2006) 47. Triantaphyllou, E.: Data Mining and Knowledge Discovery via a Novel Logic-based Approach. Springer, Heidelberg (2008) 48. Vapnik, V., Levin, E., Cun, Y.L.: Measuring the VC-dimension of a learning machine. International Journal of Human Computer Systems 6, 851–876 (2008) 49. Wrobel, S.: An algorithm for multi-relational discovery of subgroups. In: Proceedings of First European Conference on Principles of Data Mining and Knowledge Discovery (1997) 50. Yang, Y., Webb, G.I.: Weighted proportional k-interval discretization for NaiveBayes classifiers. In: Whang, K.-Y., Jeon, J., Shim, K., Srivastava, J. (eds.) PAKDD 2003. LNCS, vol. 2637. Springer, Heidelberg (2003)
Selection of Subsets of Ordered Features in Machine Learning O. Seredin1, A. Kopylov1, and V. Mottl2 1
Tula State University, 300600, Tula, pr. Lenina, 92, Russia
[email protected],
[email protected] 2 Computing Centre of the Russian Academy of Science, 117967, Moscow, Vavilova str., 40, Russia
[email protected]
Abstract. The new approach of relevant feature selection in machine learning is proposed for the case of ordered features. Feature selection and regularization of decision rule are combined in a single procedure. The selection of features is realized by introducing weight coefficients, characterizing degree of relevance of respective feature. A priori information about feature ordering is taken into account in the form of quadratic penalty or in the form of absolute value penalty on the difference of weight coefficients of neighboring features. Study of a penalty function in the form of absolute value shows computational complexity of such formulation. The effective method of solution is proposed. The brief survey of author’s early papers, the mathematical frameworks, and experimental results are provided. Keywords: machine learning, feature selection, ordered features, regularization of training, support vector machines, parametric dynamic programming.
1 Introduction The pattern recognition problem in the presence of a large amount of features (in comparing with training set size) known as the “curse of dimensionality”. There are two standard approaches to tackle the case, namely, by a priori restrictions impositions (decision rule regularization) or dimensionality reduction by most informative features selection. The approach of joining these two techniques is proposed in the paper. The selection of informative features in pattern recognition problem in the case of their ordering is considered. Feature ordering is typical for tasks of signal and image learning. Only one-dimensional ordering is accented in this work. Indeed, most of techniques for feature selection consider feature vector as non-ordered set of numbers, moreover a lot of methods accept hypothesis that features are independent. However, there is exists a number of tasks where features are consecutive measurements along the axis of some argument, for example, observation of some signal along time axis, components of a spectrum, etc. In previous articles the authors already proposed methods of decision rule regularization and methods of feature selection. In the early papers [1,2] the method of P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 16–28, 2009. © Springer-Verlag Berlin Heidelberg 2009
Selection of Subsets of Ordered Features in Machine Learning
17
regularization which takes into account a priori information about feature interrelation was described. At the same time the research of modality combination in data mining was developed, and actually the effective technique for informative feature selection was suggested [3-5]. Taking into account a priori information about one-dimensional ordering of features directly for a selection method requires development of modified procedures. Such attempt was done in [6] where the model of feature interrelation was represented as quadratic penalty on difference between the informative weights of neighbor features. In this paper we will investigate the new penalty criterion in the form of the absolute value function. It should be noted, that the method of potential functions is chosen as a theoretical background for suggested algorithms. The reason for such selection is high popularity of the method as the basis for support vector machine learning [7]. The paper has the following structure – in the second section the idea of the support vector machine learning will be briefly reminded. In the third section the effective feature selection procedure regardless to their relationship will be described. The fourth section focuses on the idea of the learning regularization for the case of ordered features. The next three sections are devoted to algorithms of regularized order-based feature selection. Experimental results are presented in the eighth section.
2 SVM – The Basis for Modifications We will develop methods of learning with respect to structural relations between features by insertion of additional penalties into existing well known criteria. The incorporated into the model coefficient of regularization will define the balance between “classical” approach and regularization based on a priory information. Let (x j , g j ), j = 1,..., N – will be a training set, where x = ( xi , i = 1,..., n) ∈ R n –
real-valued feature vector of recognition object, g = {±1} – index of classification; a = {ai , i = 1,..., n} ∈ R n – directional vector of the optimal separable hyperplane and b ∈ R is its shift defined as decision of well-known criterion [7]: N ⎧ n 2 a + C δ j → min(a1 ,..., an , b, δ1 ,..., δ N ), ∑ ∑ i ⎪ j =1 ⎪ i =1 ⎨ n ⎪ g ⎛ a x + b ⎞ ≥ 1 − δ , δ ≥ 0, j = 1,..., N . j ⎜ ∑ i ij j j ⎟ ⎪⎩ ⎝ i =1 ⎠
(1)
Here the non-negative parameter C and auxiliary variables δ j , j = 1,..., N , introduced for the case of linear non-separability of objects of two classes. Usually the task (1) is solved in the dual form: n ⎧N 1 N N ⎛ ⎞ ⎪∑ λ i − ∑∑ ⎜ g j g k ∑ xij xik ⎟ λ j λ k → max(λ1 ,..., λ N ), 2 ⎪ i =1 ⎠ j =1 k =1 ⎝ i =1 ⎨N ⎪ λ g = 0, 0 ≤ λ ≤ C 2 , j = 1, … , N , j j j ⎪⎩∑ j =1
(2)
18
O. Seredin, A. Kopylov, and V. Mottl
as the task of quadratic programming related to the non-negative Lagrange multipliers λ j , j = 1,..., N . The relation of coefficients of the directional vector of the optimal separable hyperplane and Lagrange multipliers is defined as follows: ai =
∑
j: λ j > 0
g j λ j xij , i = 1,..., n.
(3)
The simplicity of algorithmic and numerical realization of this method, the evident reference to the so-called support elements in the training set (only they, in fact, form the separable hyperplane), and good experimental results have made this formulation of the pattern recognition problem the most popular in recent times. These are the reasons for such criterion to be the basis for constructing our method of feature selection in the case of feature ordering or interrelation. It is necessary to note that formulation (1) is most simple one, so to speak the academic one from the number of criteria, joined by the common title of support vector machines. In this form the solution based on inner products between feature vectors of objects. The decision rule is linear in the initial feature space. There are exist formulations with another type of kernels, another kind of penalties for non-separable cases. There are discussions in literature about relationship of SVM and method of potential functions [3], methods of featureless pattern recognition [5]. For clarity of our reasoning we will rely on the canonical formulation of the problem (1).
3 Feature Selection Based on Potential Functions Combining The method of the potential functions (or kernels) combining in featureless pattern recognition and regression estimation was published in [3,4]. It has been shown that this technique can be transferred on the case where objects are presented by their features and can be efficiently applied as a non-iterative informative feature selection. The non-negative weights ri ≥ 0, i = 1,..., n , each of which is corresponds to component of the directional vector of the sought for separable hyperplane, are incorporated into the “classical” Vapnik’s (1) SVM criterion as it was proposed in [8]: n
∑ i =1
ai2 + 1/ μ n ⎛ 1 ⎞ + ∑ ⎜ + 1 + μ ⎟ ln ri + ri μ i =1 ⎝ ⎠ N
+C ∑ δ j → min(r1 ,..., rn , a1 ,..., an , b, δ1 ,..., δ N ),
(4)
j =1
here μ is non-negative parameter of selectivity [9]. We propose to solve the problem of minimizing criterion (4) using the method of Gauss-Seidel by separating the variables into two groups: first – ai , i = 1,..., n, b, δ j ≥ 0, j = 1,..., N , and second ri ≥ 0, i = 1,..., n , and implement step-by-step minimization criteria for one group of variables, with a fixed second one. While coefficients ri , i = 1,..., n are fixed the initial problem is actually reduced to the classical training problem by support vectors. The dual form of criterion related to
Selection of Subsets of Ordered Features in Machine Learning
19
non-negative Lagrange multipliers λ j , j = 1,..., N almost coincide with criterion (2) of SVM: N
∑λ i =1
i
−
n 1 N N ⎛ ⎞ g j g k ∑ ri xij xik ⎟ λ j λ k → max(λ1 ,..., λ N ). ∑∑ ⎜ 2 j =1 k =1 ⎝ i =1 ⎠
(5)
The difference between the last criterion and “classical” formulation is the presence of additional coefficients ri , i = 1,..., n in the matrix of quadratic form. The restrictions remain unchanged, while coefficients of the directional vector of the separable hyperplane are calculated using the rule: ai = ri ∑ j =1 g j λ j xij , i = 1,..., n . N
While parameters ai , i = 1,..., n, b, δ j ≥ 0, j = 1,..., N are fixed the calculation of weighted coefficients is utterly simple: ri =
ai2 + (1 μ) , i = 1,..., n. (1 μ) + 1 + μ
(6)
The stopping rule of the iterative process of learning can be defined, for example, on 1 n the condition of convergence of sequences ri , i = 1,..., n : ∑ ri step +1 − ri step < ε , ε > 0 . n i =1 It is necessary to make a reservation that introducing the notion of “informative feature” we do not have in mind the actual informational characteristic of feature like it was introduced for Akaike informational criterion or Shannon entropy criterion. We only suggest that for the whole set of measurable features there are exist subsets of features which adequate to either data analysis task. As a synonym of “informative feature” it is possible to consider term of “adequate feature” or “relevant feature”.
4 The Regularization of Signal Recognition: The Principle of Decision Rule Smoothness In early works [1] the approach to decision rule regularization was proposed by taking into account a priori information about the features ordering. To make the learning process prefer decision rules with smooth changing of coefficients of the directional vector of the separable hyperplane we propose to incorporate the additional quadratic penalty on difference of neighboring component to the criterion (1): n
∑a i =1
2 i
n
N
i=2
j =1
+α∑ (ai − ai −1 ) 2 + C ∑ δ j → min(a1 ,..., an , b, δ1 ,..., δ N ).
(7)
Here and further coefficient α ≥ 0 define the ratio of penalty on unsmoothness of ordering coefficients of the sought for optimal separable hyperplane. From the computational point of view both primal and dual tasks remain quadratic. The difference from the classical criterion is in incorporating the additional component
20
O. Seredin, A. Kopylov, and V. Mottl
J ′(a) = ∑ i = 2 (ai − ai −1 ) 2 . It is clear that such quadratic function can be written as n
J ′(a) = aT B a , where B (n × n) has the following form: 0 " 0 0 0⎞ ⎛ 1 −1 ⎜ ⎟ 1 2 1 0 0 0⎟ − − " ⎜ ⎜ 0 −1 2 " 0 0 0⎟ ⎜ ⎟ B( n × n) = ⎜ # # # % # # #⎟ ⎜ 0 0 0 " 2 −1 0⎟ ⎜ ⎟ 0 0 " −1 2 −1 ⎟ ⎜ 0 ⎜ 0 0 0 " 0 −1 1 ⎟⎠ ⎝
.
Therefore, it is more convenient to write the objective function in the problem of finding parameters of the optimal separable hyperplane (7) in the vector form: N
aT (I + αB)a + C ∑ δ j → min(a, b, δ1 ,..., δ N ) , j =1
under the same restrictions. It is obvious that matrix B is positive defined. The criterion in dual form also does not undergo changes, but the matrix of quadratic form will be slightly corrected: 1 N N ⎧N λ − ∑ ∑∑ ( g j gk xTj (I + αB)−1 xk ) λ j λ k → max(λ1 ,..., λ N ), j ⎪ 2 j =1 k =1 ⎪ j =1 ⎨N ⎪ λ g = 0, 0 ≤ 0 ≤ λ j ≤ C 2, j = 1,..., N . j j ⎪⎩∑ j =1 The directional vector of the optimal separable hyperplane will be calculated by following formula: a = (I + αB) −1 ∑ λ j g j x j . j :λ j > 0
5 Selection of Subsets of Ordering Features The feature selection procedure described in third section does not take into account the specificity of tasks of signal and image analysis. The classical formulation of the pattern recognition problem suppose that objects of recognition are represented by their features and it the order they where recorded does not matter. Roughly speaking, if somebody reorders components of feature vectors then the result of building of decision rule or the result of feature selection will not change. But we also draw attention to following fact: for some specific objects like signals and images peculiarity of their registration, namely, neighborhood of observations (samples/pixels) can be taken into account. Imposing such restrictions is called regularization (sometimes stabilizing) of decision rules of recognition. How to take into account such structural restrictions on a directional vector is shown in Section 4. In this section we will demonstrate how it is
Selection of Subsets of Ordered Features in Machine Learning
21
possible to join these two techniques: feature selection and assumption that on the set of features there are exist more or less informative groups. Let modify criterion (4), namely, we will add extra penalty on difference of neighboring weighted coefficients ri ≥ 0, i = 1,..., n under the previous constraints: n ⎡ ai2 + 1/ μ ⎛ ⎤ 1⎞ + ⎜ μ + 1 + ⎟ ln ri ⎥ +α∑ f (ri , ri −1 ) + μ⎠ ri i =1 ⎣ ⎝ ⎦ i=2 n
∑⎢
(8)
N
+C ∑ δ j → min(r1 ,..., rn , a1 ,..., an , b, δ1 ,..., δ N ). j =1
We propose to solve the problem of minimizing criterion (8) using the method of Gauss-Seidel by separating the variables into two groups: first – ai , i = 1,..., n, b, δ j ≥ 0 , j = 1,..., N , and second ri ≥ 0, i = 1,..., n , and implement step-by-step minimization criteria for one group of variables, with a fixed second one. There is no difficulty to certain that if coefficients ri , i = 1,..., n are fixed than solution in dual form is coincide with task (5). But finding just informative weights would not be so simple as (6). Therefore, for the search of weight coefficients at the each step of coordinate-wise optimization it is necessary to find the minimum of following criterion (here, for short, we introduce new notions ci = ai2 + 1/ μ , i = 1,..., n and d = μ + 1 + 1/ μ , remind that on this substep of iterative procedure values of ai , i = 1,..., n already found and fixed): n
⎡ ci
∑⎢ r i =1
⎣
i
n ⎤ + d ln ri ⎥ +α∑ f (ri , ri −1 ) → min(r1 ,..., rn ). ⎦ i=2
(9)
In the two next sections we will consider different ways of penalties on differences between weighted coefficients associated with neighboring ordered features of recognition object (for example, signal), namely we will consider penalties in the form of quadratic function and in the form of absolute value function. It is necessary to note, that in this approach the a priori information about feature ordering possess restrictions on weighted coefficients of feature informativeness, but not on components of directional vector of the separable hyperplane as, for example, in [2].
6 Feature Subset Selection with Taking into Account Quadratic Difference between Neighboring Weight Coefficients In this section we will consider situation where the penalty function is quadratic: (r − r ) 2 f (ri , ri −1 ) = i i −1 , i = 1,..., n . In this case the criterion (9) will turn into: ri ri −1 n
⎡ ci
∑⎢ r i =1
⎣
i
n ⎤ (r − r ) 2 + d ln ri ⎥ +α∑ i i −1 → min( r1 ,..., rn ). ⎦ i = 2 ri ri −1
(10)
22
O. Seredin, A. Kopylov, and V. Mottl
The search of minimum of (10) reduced to the solution of system of nonlinear equations for the parameters ri , i = 1,..., n :
⎧ c1 ⎛ r1 r2 ⎞ ⎪ − + d + α ⎜ − ⎟ = 0, r ⎝ r2 r1 ⎠ ⎪ 1 ⎪ ⎛ ri −1 + ri +1 ri r ⎞ ⎪ ci + + i ⎟ = 0, i = 2,..., n − 1, ⎨− + d + α ⎜ − r r r r i i −1 i +1 ⎠ ⎝ ⎪ i ⎪ ⎪ − cn + d + α ⎛⎜ − rn −1 + rn ⎞⎟ = 0. ⎪⎩ rn ⎝ rn rn −1 ⎠
(11)
Each equation in this system includes only 2-3 unknown variables. The method of simple iterations can be used to solve the problem.
7 Feature Subset Selection with Taking into Account Absolute Value of Difference between Neighboring Weight Coefficients Numerous experiments have shown that taking into account the interrelation between features in the form of a quadratic penalty «dilutes» an informative subarea in the space of the ordered features. To avoid this disadvantage, it was decided to use the absolute value function as the penalty on difference of weight coefficients. n ⎡ ci ⎤ ln d r + +α ∑ ∑ ln ri − ln ri −1 → min(r1 ,..., rn ). ⎢ i⎥ i =1 ⎣ ri ⎦ i=2 n
(12)
The search algorithm for optimum values for the coefficients of the direction vector remains the same, but the minimization of criterion concerning weight factors ri ≥ 0, i = 1,..., n , represents a new problem. Let us substitute variables: ui = ln ri , i = 1,… n then the criterion (12) can be rewritten in the following form: n
∑ ⎡⎣c e i =1
i
− ui
n
+ dui ⎤⎦ + α∑ ui − ui −1 → min(u1 , … un ).
(13)
i=2
The objective function in (13) is pair-wise separable, that is represented by the sum of functions of not more than two variables. Let us denote the functions of one variable in criterion (13) as ψ i (ui ) = ci e −ui + dui , and functions of two variables as
γ i (ui , ui −1 ) = α ui − ui −1 . Then the objective function in criterion (13) takes the following form: n
n
i =1
i=2
J (u1 , u2 , … , un ) = ∑ ψ i (ui ) + ∑ γ i (ui −1 , ui )
(14)
Selection of Subsets of Ordered Features in Machine Learning
23
For minimization of the objective function (14) we will take advantage of the procedure based on a principle of Dynamic Programming [11]. The procedure in this case is based on a recurrent decomposition of the initial problem of optimization of a function of n variables into a succession of n elementary problems, each of which consists in optimization of a function of only one variable. The elementary functions of one variable Ji (ui ) , to be minimized at each step of minimization of separable function are called here Bellman functions, as well as in the classical dynamic programming procedure. The procedure of dynamic programming finds a global minimum of pair-wise separable function in two passes, at first in forward direction, and then in the backward direction. On forward pass i = 1,..., n the Bellman functions are determined in accordance with forward recurrent relation Ji (ui ) = ψ i (ui ) + min ⎡⎣ γ i (ui −1 , ui ) + Ji −1 (ui −1 ) ⎤⎦ , i = 1,..., n . ui −1
(15)
Last Bellman function Jn (un ) directly shows, how the minimum value of criterion, as a whole, depends on value of the variable un and therefore, its optimum value can be found as uˆn = arg min Jn (un ) . Other elements of the sought for decision un
uˆi , i = n − 1,… ,1 , can be found by means of backward recurrent relation ui −1 (ut ) = arg min ⎣⎡ γ i (ui −1 , ui ) + Ji −1 (ui −1 ) ⎦⎤ , ui −1
(16)
which is the inverted form of forward recurrent relation (15). Application of this relation on the backward move is obvious: (17)
uˆi −1 = ui −1 (ut ) , i = n − 1,… ,1
Thus, regardless of the form of functions ψ i (ui ) and γ i (ui , ui −1 ) in pair-wise separable objective function, the algorithm of the dynamic programming finds the point of its global minimum, if, of course, such combination of values of variables exists within the area of their variation, executing the known number of operations, proportional to the number of variables. In the case of continuous variables, e.g. if ui ∈ R , a numerical realization of the dynamic programming procedure, is possible only if there exists a finitely parameterized function family J (u, q) concordant with node functions ψ i (ui ) and edge functions γ ( u , u ) in the sense that Bellman functions J (u ) belong to this family at i
i −1
i
i
i
each step. In this case, the forward pass of the procedure consists in a recurrent re-evaluating of parameters q i that completely represent the Bellman functions J (u ) = J (u , q ) . In particular, as is shown in [11], if the node and edge functions are i
i
quadratic, the Bellman functions will be quadratic too. The parametric representation is also possible in the case of using absolute value of difference of adjacent variables instead of quadratic node functions [12].
24
O. Seredin, A. Kopylov, and V. Mottl
It can be proven, in such a case, that if the node functions ψ i (ui ) and edge functions γ i (ui −1 , ui ) are convex, all the Bellman functions are also convex. As it is shown in [12], if the function γ i (ui −1 , ui ) in the objective function (14) has the form γ (u , u ) = α u − u , α > 0 , and the Bellman function J (u ) is convex funci
i
i −1
i
i −1
i −1
i −1
tion and is everywhere differentiable in the range of definition, the backward recurrent relation ui −1 (ui ) has the following form: ⎧ui−α −α −1 , ui ≤ ui −1 , ⎪ α ui −1 (ui ) = ⎨ui , ui−α −1 < ui < ui −1 , ⎪ α α ⎩ui −1 , ui ≥ ui −1 .
(18)
α where ui−α −1 and ui −1 can be obtained as the solution of equations d d ′ α Ji′−1 (ui−α [ J i −1 (ui−α [ J i −1 (uiα−1 )] = α respectively. −1 ) = −1 )] = −α and J i −1 (ui −1 ) = −α dui −1 duiα−1 One can easily see that the evaluated value of ui −1 is completely defined by the value α of adjacent variable ui in the range of ui−α −1 < ui < ui −1 , and is independent of it at the rest of the value area of the variable ui . It is just the fact that gives such a procedure the ability to preserve abrupt changes of the parameters, and accordingly not "dilute" an informative subarea in the space of the ordered features. When parameters ci and d are nonnegative, the functions ψ i (ui ) = ci e − ui + dui are convex, and there exists a parameterized Bellman functions family, concordant with node functions ψ i (ui ) and edge functions γ i ( ui −1 , ui ) . This fact gives us the possibility to apply described above non-iterative procedure. 1.
J1′(u1 ) = −c1 exp(−u1 ) + d , u1−α = − ln[(d + α) / c1 ] , u1α = − ln[(d − α) / c1 ] .
2.
For i = 2, … n , ⎧ −α, ui ≤ ui−α −1 ⎪ α < Ji′(ui ) = −ci exp(−ui ) + d + ⎨ Ji′−1 (ui ), ui−α u −1 i < ui −1 ; ⎪ α, ui ≥ uiα−1 ⎩ uiα : Ji′(uiα ) = α , ui−α : Ji′(ui−α ) = −α .
3.
4.
un : Jn′ (un ) = 0 . For i = n − 1,…1 , ui −1
−α ⎧ui−α −1 , ui ≤ ui −1 , ⎪ α = ⎨ui , ui−α −1 < ui < ui −1 , ⎪ α α ⎩ui −1 , ui ≥ ui −1 .
In accordance with the step 2 of the above algorithm, the derivatives of the Bellman functions are composed from the fragments of functions in the form of
Selection of Subsets of Ordered Features in Machine Learning
25
qk exp(−u ) + pk , where k is the number of the fragment. Therefore the boundaries of the fragments, as well as parameters qk and pk for each fragment k , constitute parameters of the Bellman function derivatives. The leftmost boundary of the fragments α coincides with ui−α −1 and the rightmost boundary will coincides with ui −1 . For the parametric representation of such functions it is necessary to keep in memory the boundary points of all intervals of the derivatives of Bellman functions.
8 Experimental Results For the experimental research of proposed algorithms test data were generated as described below. Two classes of recognition data were distributed near two centers. The centre of the first class is 100 artificial observations with values equal to zero. The centre of second class differs from the first one on interval from 70-th to 80-th samples. Second signal has values of 0.4 instead of 0 (Fig.1).
Fig. 1. Centers of first and second classes (left) and examples of recognition objects (right)
Objects of “entire assembly” were generated from two centers by adding uniform noise (zero mean value, –0.8, +0.8) (Fig. 1). Whole number of objects in entire assembly was equal to 6000 (up to 3000 in each class). Hundred of training sets with number of objects equal to 20, 40, 60, 80, 100 and 200 were picked out randomly. Remain objects were used as the test set. The examples of weight coefficients values ri , i = 1,...,100 for the SVM, “pure” feature selection, and selection of feature subsets with penalties in form of quadratic function and absolute value function are shown in Fig. 2. Experimental results in the form of average error rate on test sets for different sizes of training sets (20-200 objects) are shown in Fig.3. It is clear that adding of the regularization, based on the search of informative subarea in feature space (criterion (10) – dotted line, criterion(12) – dashed line) lead to better predicted properties of decision rule as standard SVM (solid line).
26
O. Seredin, A. Kopylov, and V. Mottl
Fig. 2. Examples of weight coefficients values (from the top): SVM (1), SVM with feature selection (4), SVM and feature subset selection taking into account quadratic difference of neighbor weight coefficients (10), SVM and feature subset selection taking into account absolute value difference of neighbor weight coefficients (12)
Fig. 3. Experimental results (error rate on test set vs. number of objects in training set)
The choice of optimal value for the depth of regularization α remains an open question. We used the procedure of cross validation to find the best value.
9 Conclusion Article shows a way to combine relevant feature selection and restrictions on such selection, reasonable for the solved task, in one criterion. The summarizing review of
Selection of Subsets of Ordered Features in Machine Learning
27
our early publications, which actually lead to the proposed idea, has done. The basic idea of the proposed approach is to formalize idea of taking into account the one-dimensional ordering of features, which is typical for the tasks of signals analysis. The criterion is constructed and the scheme of its numerical optimization is offered. It is necessary to provide comprehensive analysis of the behavior of the proposed algorithm of selection of subset of relevant features in both modeling and real data experiments. It also seems reasonable to extend the methodology to the case of the two dimensional ordering, that is especially important for image analysis tasks. Work is supported by grants of Russian Foundation for Basic Research 08-01-00695, 08-01-99003, 08-07-90700, 09-07-00394.
References 1. Seredin, O.S., Dvoenko, S.D., Krasotkina, O.V., Mottl, V.V.: Machine Learning for Signal Recognition by the Criterion of Decision Rule Smoothness. Pattern Recognition and Image Analysis 11(1), 87–90 (2001) 2. Seredin, O., Mottl, V.: Regularization in Image Recognition: the Principle of Decision Rule Smoothing. In: Proceedings of the Ninth International Conference Pattern Recognition and Information Processing, Minsk, Belarus, vol. II, pp. 151–155 (2007) 3. Mottl, V.V., Seredin, O.S., Krasotkina, O.V., Muchnik, I.B.: Fusing of potential functions in reconstructing dependences from empirical data. Doklady Mathematics 71(2), 315–319 (2005); From Doklady Akademii Nauk 401(5), 607–612 (2005) 4. Mottl, V.V., Seredin, O.S., Krasotkina, O.V., Muchnik, I.B.: Principles of multi-kernel data mining. In: Perner, P., Imiya, A. (eds.) MLDM 2005. LNCS (LNAI), vol. 3587, pp. 52–61. Springer, Heidelberg (2005) 5. Mottl, V.V., Seredin, O.S., Dvoenko, S.D., Kulikowski, C.A., Muchnik, I.B.: Featureless pattern recognition in an imaginary Hilbert space. In: Proceedings of 16th International Conference Pattern Recognition, ICPR 2002, Quebec City, Canada, August 2002, vol. II, pp. 88–91 (2002) 6. Seredin, O., Mottl, V.: The selection of informative interrelated features in pattern recognition. Tavricheskiy Vestnik Informatiki i Matematiki 2, 180–185 (2008) (in Russian) 7. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998) 8. Mottl, V., Tatarchuk, A., Sulimova, V., Krasotkina, O., Seredin, O.: Combining Pattern Recognition Modalities at the Sensor Level Via Kernel Fusion. In: Proceedings of 7th International Workshop Multiple Classifiers Systems, Prague, Czech Republic, pp. 1–12 (2007) 9. Tatarchuk, A., Mottl, V., Eliseyev, A., Windridge, D.: Selectivity Supervision in Combining Pattern-Recognition Modalities by Feature- and Kernel-Selective Support Vector Machines. In: Proceedings of the 19th International Conference on Pattern Recognition, Tampa, Florida, USA, December 7-11 (2008) 10. Seredin, O., Kopylov, A., Mottl, V., Pryimak, A.: Selection of subsets of interrelated features in pattern recognition problem. In: Proceedings of 9th International Conference on Pattern Recognition and Image Analysis: New Information Technologies, Nizhni Novgorod, vol. 2, pp. 151–154 (2008) 11. Mottl, V., Kopylov, A., Blinov, A., Kostin, A.: Optimization techniques on pixel neighborhood graphs for image processing. In: Graph-Based Representations in Pattern Recognition Computing, Supplement 12, pp. 135–145. Springer, Wien (1998)
28
O. Seredin, A. Kopylov, and V. Mottl
12. Kopylov, A.V.: Parametric dynamic programming procedures for edge preserving in signal and image smoothing. In: Proceedings of the 7th International Conference on Pattern Recognition and Image Analysis, St.Petersburg, October 18-23, vol. I, pp. 281–284 (2004) 13. Kopylov, A.V.: Dynamic programming for edge-preserving smoothing in signal and image analysis and pattern recognition with interrelated features. In: Proceedings of 9th International Conference on Pattern Recognition and Image Analysis: New Information Technologies, Nizhni Novgorod, vol. 1, pp. 325–328 (2008)
Combination of Vector Quantization and Visualization Olga Kurasova1,2 and Alma Molyt˙e1 1
Institute of Mathematics and Informatics, Akademijos str., 08663 Vilnius, Lithuania
[email protected],
[email protected] 2 Vilnius Pedagogical University, Studentu str., 08106 Vilnius, Lithuania
Abstract. In this paper, we present a comparative analysis of a combination of two vector quantization methods (self-organizing map and neural gas), based on a neural network and multidimensional scaling that is used for visualization of codebook vectors obtained by vector quantization methods. The dependence of computing time on the number of neurons, the ratio between the number of neuron-winners and that of all neurons, quantization and mapping qualities, and preserving of a data structure in the mapping image are investigated.
1
Introduction
Any set of objects may often be characterized by some features x1 , x2 , . . . , xn . A combination of values of all features characterizes a particular object Xi = (xi1 , xi2 , . . . , xin ) from the whole set X = {X1 , X2 , . . . , Xm }, where n is the number of features and m is the number of analyzed objects. As the number of features is often more than two, we deal with multidimensional data. So X1 , X2 , . . . , Xm are n-dimensional vectors. Often they are interpreted as points in the n-dimensional space Rn , where n defines the dimensionality of space. In fact, we have a matrix of numerical data X = {X1 , X2 , . . . , Xm } = {xij , i = 1, . . . , m, j = 1, . . . , n}. The rows of this matrix are vectors Xi = (xi1 , xi2 , . . . , xin ), i = 1, . . . , m, where xij is the jth component of the ith vector. The data matrix can be analyzed by various statistical methods. Especially when the amount of data is huge, the statistical methods are often not sufficient. In order to get more knowledge from the analyzed data, it is necessary to use data mining methods. A lot of data mining methods are developed for multidimensional data analysis: classification, clustering, visualization, etc. In this paper, we analyze clustering and visualization of multidimensional data. Clustering is useful for reducing the amount of data. Vector quantization is performed in clustering methods. Visual data mining aims at integrating a human in the data analysis process, applying human perceptual capabilities to the analysis of large data sets available in todays computer systems. When combining clustering and visualization it is possible to get more knowledge than by the methods used individually. P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 29–43, 2009. c Springer-Verlag Berlin Heidelberg 2009
30
2
O. Kurasova and A. Molyt˙e
Vector Quantization and Visualization
Vector quantization is a classical signal-approximation method that usually forms a quantized approximation to the distribution of the input data vectors Xl ∈ Rn , l = 1, . . . , m, using a finite number of so-called codebook vectors Mi ∈ Rn , i = 1, . . . , N . Once the codebook is chosen, the approximation of Xl , l = 1, . . . , m, means finding the codebook vector Mi closest to Xl , usually in the Euclidean metric [1]. Vector quantization is used for data compression, missing data correction, etc. It can be used for data clustering, too. In that case, the codebook vectors are representatives of clusters. Some methods for vector quantization are based on neural networks: self-organizing map (SOM) [1], neural gas (NG) [2], learning vector quantization [1]. Here the neurons correspond to the codebook vectors. The self-organizing map (SOM) is a class of neural networks that are trained in an unsupervised manner using a competitive learning [1]. It is a well-known method for vector quantization. Moreover, the SOM is used for mapping a highdimensional space onto a low-dimensional one. The neural gas is a biologically inspired adaptive algorithm, proposed in [2]. It sorts the input signals according to how far away they are. The learning vector quantization is a supervised method for data classification. It is not analyzed in this paper. It is purposeful to visualize the codebook vectors obtained after quantization in order to get more knowledge from the analyzed data set. A large class of methods has been developed for multidimensional data visualization [3], [4]. The visual presentation of the data allowed us to see the data structure, clusters, outliers, and other properties of multidimensional data. In this paper, one of the most popular methods of visualization, i.e., multidimensional scaling [5] is used for visualization of codebook vectors. 2.1
Self-Organizing Map and Neural Gas
An array of vectors (codebook) M is formed in both the neural gas network and the self-organizing map. Here the codebook vectors are often called neurons. The array M is one-dimensional in the neural gas, M = {M1 , M2 , . . . , MN }, where Mi = (mi1 , mi2 , . . . , min ), i = 1, . . . , N , N is the number of codebook vectors. A self-organizing map is a two-dimensional grid. Usually, the neurons are connected to each other via a rectangular or hexagonal topology. The rectangular SOM is a two-dimensional array of neurons M = {Mij , i = 1, . . . , kx , j = 1, . . . , ky }, where ij ij Mij = (mij 1 , m2 , . . . , mn ), kx is the number of rows, ky is the number of columns, and the total number of neurons is N = kx × ky . The goal of these quantization methods is to change the values of codebook vectors (neurons) so that they would represent the properties of the vectors Xl , l = 1, . . . , m analyzed. At the end of learning, the codebook vectors become quantized vectors of the vectors Xl . The neural gas algorithm is as follows: 1. Selecting initial values: – the number of codebook vectors (neurons) N ; – values of the parameters λi , λf , Ei , Ef used in the learning rule;
Combination of Vector Quantization and Visualization
31
– the number of training epochs eˆ (each input vector Xl is passed to the network eˆ times, the number of training steps tmax = eˆ × m); – initial values of codebook vectors Mi = (mi1 , mi2 , . . . , min ), i = 1, . . . , N ; as usual they are initialized at random and mik ∈ (−0.5 · 10−5 , 0.5 · 10−5 ), k = 1, . . . , n. 2. Network learning: FOR t = 0 TO tmax FOR l = 1 TO m // at each learning step, the input vector Xl is passed to the neural network The Euclidean distances from the input vector Xl to each codebook vector Mi , i = 1, . . . , N , are calculated. The obtained distances M1 + Xl , M2 + Xl ,. . . , MN + Xl are sorted in an ascending order. A neuron set W1 , W2 , . . . , WN is obtained, Wk ∈ {M1 , M2 , . . . , MN }, k = 1, . . . , N : W1 + Xl ≥ W2 + Xl ≥, . . . , ≥ WN + Xl . The distance between Xl and W1 is minimal. The neuron W1 is called a winner. The neuron (vector) Wk , k = 1, . . . , N is adapted according to the learning rule: Wk (t + 1) = Wk (t) + E(t) · hλ · (Xl − Wk (t)), where E(t) = Ei (Ef /Ei )(t/tmax ) , hλ = e−(k−1)/λ(t) , λ(t) = λi (λf /λi )(t/tmax ) . END // end of passing of the input vectors END // end of learning The self-organizing map algorithm is as follows: 1. Selecting initial values: – the number of rows kx , a number of columns ky (the total number of codebook vectors is N = kx × ky ); – the number of training epochs eˆ (each input vector Xl is passed to the network eˆ times, the number of training steps tmax = eˆ × m); ij ij – initial values of codebook vectors Mij = (mij 1 , m2 , . . . , mn ), i = 1, . . . , kx , j = 1, . . . , ky ; as usual they are initialized at random and mij k ∈ (0, 1), k = 1, . . . , n. 2. Network learning: FOR t = 0 TO tmax FOR l = 1 TO m // at each learning step, the input vector Xl is passed to the neural network The Euclidean distances from the input vector Xl to each codebook vector Mij , are calculated. ˆ c with the minimal Euclidean distance to Xl is designated as a The vector M winner, here c = arg min{||Xl − Mij ||}. The neuron (vector) Mij is adapted i,j
according to the learning rule:
32
O. Kurasova and A. Molyt˙e
Mij (t + 1) = Mij (t) + hcij (t)(Xl − Mij (t)), where hcij is the so-called neighbourhood function, hcij (t) → 0, as t → ∞. END // end of passing of the input vectors END // end of learning Then the networks are trained, the quantization error EQE is calculated by the formula: m 1 ˆ c EQE = (1) Xl − M . m l=1
ˆ c is a winner for the vector Xl , M ˆ c = W1 in the neural gas method. where M 2.2
Multidimensional Scaling
Multidimensional scaling (MDS) refers to a group of methods that are widely used for dimensionality reduction and visualization (mapping into 2D or 3D space) of multidimensional data [5]. The starting point of the MDS is a matrix consisting of pairwise proximities of the data. In general, the proximities need not be distances in the mathematically strict sense. Let us take the vectors Xi = (xi1 , xi2 , ..., xin ), i = 1, . . . , m, whose dimensionality is equal to n, Xi ∈ Rn . The problem is to reduce the dimensionality n to p (p < n). It is needed to find lower-dimensional vectors Yi ∈ Rp corresponding to the vectors Xi , i = 1, . . . , m. If p = 2, we get two-dimensional vectors Y1 , Y2 , . . . , Ym ∈ R2 (Yi = (yi1 , yi2 ), i = 1, . . . , m), and can present them in Cartesian coordinate system as a scatter plot. Thus, we visualize the multidimensional data. Denote the distance between the vectors Xi and Xj by d(Xi , Xj ), and the distance between the corresponding vectors in a lower-dimensional space (Yi and Yj ) by d(Yi , Yj ). The goal of multidimensional scaling (MDS) is to find lower-dimensional vectors Yi , i = 1, . . . , m, such that the distances between the vectors in the lower-dimensional space were be as close to the original distances (or other proximities) as possible. If the square-error cost is used, the objective function (stress) to be minimized can be written as EMDS = wij (d(Xi , Xj ) − d(Yi , Yj )). (2) i<j
There exists a multitude of variants of MDS with different weight wij and optimization algorithms: simple gradient descent method, conjugate gradient, quasiNewton method, simulated annealing, combination of the genetic algorithm and quasi-Newtons descent algorithm [6], two-level minimization [7]. In this paper, we use the SMACOF (Scaling by majorization of a complicated function) algorithm for the stress function EMDS (2) minimization. It is one of the best optimization algorithms for this type of the minimization problem. This method is simple and powerful, because it guarantees a monotone convergence of the stress function [5]. This algorithm of MDS for finding two-dimensional vectors is summarized as follows:
Combination of Vector Quantization and Visualization
33
1. Set the initial values of the two-dimensional vector Y , as t = 0. 2. Compute the value of the stress function EMDS (Y (t)); here the twodimensional vectors Y are set in t = 0. 3. Increase the iteration counter t by one. 4. Compute the Guttman transform Y (t) by (3). 5. Compute EMDS (Y (t)) by (2). 6. If EMDS (Y (t − 1)) − EMDS (Y (t)) < or t is equal to the maximum number of iterations, then stop ( is a small positive constant), else go to Step 3. Formula (3) is called the Guttman transform: Y (t + 1) = m−1 B(Y (t))Y (t),
(3)
where B(Y (t)) has the elements d(X ,X ) m − d(Yii ,Yjj) , for i = j and d(Yi , Yj ) =0 bij = , bii = − bij . 0, for i = j and d(Yi , Yj ) = 0 j=1, j =i Quantitative Criteria of Mapping. A way of comparing the mapping results is to calculate and estimate the value of the stress function EMDS (2). Error ˆMDS (Stress-1) (4) is often used instead of EMDS , because the inclusion of E the normalized parameter (d(Xi , Xj ))2 gives a clear interpretation of the i<j
mapping quality that does not depend on the scale and the number of distances ˆMDS rather than the squared in an n-dimensional space. The reason for using E 2 ˆ2 ˆMDS error EˆMDS is that E is almost always very small in practice, so E MDS values are easier to discriminate [5]. 2 i<j (d(Xi , Xj ) − d(Yi , Yj )) ˆ EMDS = . (4) 2 i<j d(Xi , Xj ) The problem of objective comparison of the mapping results arises when the multidimensional data are visualized using various methods that optimize different criteria of the mapping quality. Some criteria are described and used for the comparison in [8]. In this paper, we use a topology preserving measure introduced by Konig in [9]. This measure was applied in [10]. The μ nearest vectors (neighbour) of each n-dimensional vector Xj , j ∈ (1, . . . , m), are denoted by NijX , i ∈ (1, . . . , μ), the μ nearest vectors of each twodimensional vector Yj , j ∈ (1, . . . , m), corresponding to Xj are denoted by NijY , i ∈ (1, . . . , μ). A topology preserving measure for the jth vector and the ith neighbour is calculated as follows: ⎧ 3, if NijX = NijY ⎪ ⎪ ⎨ 2, if NijX = NljY , l ∈ (1, . . . , μ), i =l ETijMP = , X Y 1, if N = N , t ∈ (μ + 1, . . . , ν), μ =ν ⎪ ij tj ⎪ ⎩ 0, else
34
O. Kurasova and A. Molyt˙e
The general measure ET P M is calculated as follows: ij 1 E 3μ × m i=1 j=1 T P M μ
ET P M =
m
(5)
Typical parameter settings are μ = 4 and ν = 10 [10]. The range of ET P M is between 0 and 1, where ET P M = 0 indicates a poor neighbourhood preservation, and ET P M = 1 indicates a perfect neighbourhood preservation. 2.3
Combination of Neural Gas and Self-Organizing Map with Multidimensional Scaling
After a large number of training steps, the neural gas network and the selforganizing map have been organized and n-dimensional input vectors X1 , X2 , . . . , Xm have been mapped - each input vector is related to the nearest neuron. Some neurons may remain unrelated with any vector of the set X = {X1 , X2 , . . . , Xm }, but there may occur neurons related with some input vectors. The neurons related with the input vectors are called neurons-winners. The number of neurons-winners r is usually smaller than that of all neurons N (r ≤ N ). In the case of the rectangular topology of SOM, we can draw a table with cells corresponding to the neurons. However, the table does not answer the question, how much the vectors of the neighbouring cells are close in the ndimensional space. A natural idea arises to apply the distance-preserving method to an additional mapping of the neurons-winners in the SOM. MDS-type methods may be used for this purpose. In [8], [11], some combinations of SOM and the MDS-type method are proposed and discussed. Neurons-winners, obtained by the neural gas method, may be visualized by MDS-type methods, too. The cross-entropy approach to visualize the neurons-winners, obtained by the neural gas, is proposed in [10]. The method is compared there with some other combinations. In this paper, we propose a combination of the neural gas and the self-organizing map with the multidimensional scaling based on SMACOF minimization. A scheme of mapping (visualization) of the neurons-winners by MDS is presented in Fig. 1.
Fig. 1. Scheme of visualization of the neuron-winners
Combination of Vector Quantization and Visualization
Table 1. Data sets No
Name
m
n
Number Description of classes
1 2 3 4 5 6
chainlink hepta target auto MPG rand clust10 rand clust5
1000 212 770 392 100 100
3 3 2 7 10 5
2 7 6 4 5 5
linear not separable clearly defined clusters outliers linear not separable random clusters random clusters
Fig. 2. Computing time of NG (left) and SOM (right)
35
36
O. Kurasova and A. Molyt˙e
Fig. 3. Number of vectors-winners in NG and SOM
3 3.1
Experimental Investigations Data Sets
Some data sets are used in the experimental investigations (see Table 1). Each data set has some specific characteristics. The first three data sets are taken from [12], the fourth data set is taken from [13], and the last two data sets are generated by us. 3.2
Dependence of Computing Time on the Number of Neurons
Computing time of the neural gas (NG) and the self-organizing maps (SOM) is investigated. Fig. 2 shows the dependence of the computing time on the number N of neurons. The curves are approximated by the exponential functions precisely enough (R2 ≥ 0.9), it means that the computing time exponentially increases when the number of neurons is increasing. Moreover, the computations in the NG run on much longer than in the SOM. Regarding the limit of the paper, the results of only four data are presented. Analogous results are for other data sets.
Combination of Vector Quantization and Visualization
37
Fig. 4. Quantization error (QE) of NG (left) and SOM (right)
3.3
Numbers of Neurons-Winners
It is of interest to investigate whereby method (NG or SOM) more neurons become winners. The ratios between the number of neurons-winners and all the neurons of the NG and SOM are presented in Fig. 3. It is shown that the ratios of NG are larger than that of the SOM: about 80% of the NG neurons become winners. If the numbers of neurons are large, only about 50% of the SOM neurons become winners. 3.4
Quality of Quantization and Visualization
Quantization error EQE (1) is calculated to estimate the quality of quantization. The quantization error decreases then the number of neurons is increasing (Fig. 4). As we see in Fig. 4, the quantization errors of the NG are smaller than that of the SOM then the number of neurons-winners is approximately equal. It means that the neural gas is more suitable for a vector quantization. In order to estimate the quality of mapping the neurons-vectors, two measures ˆMDS (4) (Stress-1) and topology preserving measure ET P M (5). are calculated: E
38
O. Kurasova and A. Molyt˙e
Fig. 5. Stress-1 values of mapping the NG and SOM neurons-winners
Fig. 6. Topology preserving measure (TPM) of mapping the NG and SOM neuronswinners
Combination of Vector Quantization and Visualization
39
Fig. 7. Mapping images of hepta data obtained by NG (left) and SOM (right)
ˆMDS are presented in Fig. 5. When the neurons-winners of the The values of E ˆMDS are approximately equal, the NG and SOM are mapped, the values of E dimensionality n of chainlink, hepta, and target data being small (n = 2 or ˆMDS n = 3). When the neurons-winners of the SOM are mapped, the values of E ˆMDS , if the neurons-winners of the NG are are smaller than the values of E mapped (auto MPG, rand clust10, rand clust5 data sets). The dimensionality n of these data is larger (n = 7, 5 or 10). The values of the topology preserving measure ET P M of mapping the NG and SOM neurons-winners are calculated and the results are presented in Fig. 6. Fig. 6 shows that the values of ET P M of mapping the NG neurons-winners are smaller than that of the SOM neurons-winners. It means that the SOM preserves neighbourhood better.
40
O. Kurasova and A. Molyt˙e
Fig. 8. Mapping images of chainlink data obtained by NG (left) and SOM (right)
Combination of Vector Quantization and Visualization
Fig. 9. Mapping images of target data obtained by NG (left) and SOM (right)
41
42
3.5
O. Kurasova and A. Molyt˙e
Mapping Images
ˆ 1, M ˆ 2, . . . , M ˆ r are When the neurons-winners that are n-dimensional vectors M mapped by multidimensional scaling, two-dimensional vectors Y1 , Y2 , . . . , Yr are obtained. Here r is the number of neurons-winners. Two-dimensional vectors may be presented in a scatter plot (Figs. 7-9). Numbers near points indicate order numbers of classes to which the points belong. Figs. 7-9 show how the mapping images change when the number of neurons-winners is growing. The data structure is visible even when the number r of neurons-winners obtained by the NG is small enough. If the number r of neurons-winners obtained by the SOM is larger, the data structure is visible, as well.
4
Conclusions
In this paper, some data mining methods are investigated. Two vector quantization methods (neural gas and self-organizing maps) are used for reducing the number of the data sets analyzed. The neurons-winners are visualized by multidimensional scaling. The dependence of computing time on the number of neurons, the number of neuron-winners, quantization and mapping qualities, and preserving of a data structure in the mapping image are investigated. The experiments show that: 1. The computing time of the NG and SOM exponentially increases when the number of neurons is increasing. Moreover, the computations in the NG run on much longer than in the SOM. 2. The ratios between the number of neurons-winners and all the neurons of the NG are larger than that of the SOM: about 80% of the NG neurons become winners. When the numbers of neurons are large, only about 50% of the SOM neurons become winners. 3. The quantization error decreases when the number of neurons is growing. The quantization errors of the NG are smaller than that of the SOM when the number of neurons-winners is approximately equal. It means that the neural gas is more suitable for vector quantization. 4. When the neurons-winners of the NG and SOM are mapped, the values of Stress-1 are approximately equal for the data whose dimensionality n is small. When the neurons-winners of the SOM are mapped, the values of Stress-1 are smaller for the data whose dimensionality n is larger. 5. The values of the topology preserving measure of mapping the NG neuronswinners are smaller than that of mapping the SOM neurons-winners. It means that the SOM preserves neighbourhood better. 6. The data structure is visible in the mapping image even when the number r of neurons-winners of the NG is small enough. If the number r of neuronswinners of the SOM is larger, the data structure is visible, as well.
Combination of Vector Quantization and Visualization
43
Acknowledgments. The research is partly supported by the Lithuanian State Science and Studies Foundation project ”Information technology tools of clinical decision support and citizens wellness for e.Health system, Info Health (No. B-07019)”.
References 1. Kohonen, T.: Self-Organizing Maps, 3rd edn. Springer Series in Information Science, vol. 30. Springer, Berlin (2001) 2. Martinetz, T.M., Schulten, K.J.: A Neural-Gas Network Learns Topologies. In: Kohonen, T., M¨ akisara, K., Simula, O., Kangas, J. (eds.) Artificial Neural Networks, pp. 397–402. North-Holland, Amsterdam (1991) 3. Chen, C., Hardle, W., Unwin, A.: Handbook of Data Visualization. Springer, Berlin (2008) 4. Dzemyda, G., Kurasova, O., Medvedev, V.: Dimension Reduction and Data Visualization Using Neural Networks. In: Maglogiannis, I., Karpouzis, K., Wallace, M., Soldatos, J. (eds.) Emerging Artificial Intelligence Applications in Computer Engineering. Real World AI Systems with Applications in eHealth, HCI, Information Retrieval and Pervasive Technologies. Frontiers in Artificial Intelligence and Applications, vol. 160, pp. 25–49. IOS Press, Amsterdam (2007) 5. Borg, I., Groenen, P.: Modern Multidimensional Scaling. Springer, New York (2005) ˇ 6. Mathar, R., Zilinskas, A.: On Global Optimization in Two-Dimensional Scaling. Acta Aplicandae Mathematicae 33, 109–118 (1993) ˇ ˇ 7. Zilinskas, A., Zilinskas, J.: Two Level Minimization in Multidimensional Scaling. Journal of Global Optimization 38(4), 581–596 (2007) 8. Bernataviˇcien˙e, J., Dzemyda, G., Kurasova, O., Marcinkeviˇcius, V.: Optimal Decisions in Combining the SOM with Nonlinear Projection Methods. European Journal of Operational Research 173, 729–745 (2006) 9. Konig, A.: Interactive Visualization and Analysis of Hierarchical Neural Projections for Data Mining. IEEE Transactions on Neural Networks 11(3), 615–624 (2000) 10. Est´evez, P.A., Figueroa, C.J., Saito, K.: Cross-Entropy Embedding of HighDimensional Data Using the Neural Gas Model. Neural Networks 18, 727–737 (2005) 11. Dzemyda, G., Kurasova, O.: Heuristic Approach for Minimizing the Projection Error in the Integrated Mapping. European Journal of Operational Research 171(3), 859–878 (2006) 12. Fundamental Clustering Problem Suite, http://www.uni-marburg.de/fb12/ datenbionik/data/ 13. Asuncion, A., Newman, D.J.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2007), http://www.ics.uci.edu/~ mlearn/MLRepository.html
Discretization of Target Attributes for Subgroup Discovery Katherine Moreland1 and Klaus Truemper2 1
2
The MITRE Corporation, McLean, VA 22102, U.S.A. Department of Computer Science, University of Texas at Dallas Richardson, TX 75083, U.S.A.
Abstract. We describe an algorithm called TargetCluster for the discretization of continuous targets in subgroup discovery. The algorithm identifies patterns in the target data and uses them to select the discretization cutpoints. The algorithm has been implemented in a subgroup discovery method. Tests show that the discretization method likely leads to improved insight. Keywords: Subgroup Discovery, Logic, Classification, Feature Selection.
1
Introduction
The task of subgroup discovery is defined as follows. Given a data set that is representative of a particular population, find all statistically interesting subgroups of the data set for a given target attribute of interest. Target attributes may be binary, nominal, or continuous. Many subgroup discovery algorithms can handle only binary target attributes, and continuous target attributes must be discretized prior to application of these algorithms. In this paper, we present a new algorithm called TargetCluster for that task as well as a new subgroup discovery method. There are three main goals of target discretization. First, clusters should be densely populated since then they are likely to represent similar cases. Second, clusters should be clearly distinct since two clusters located close together may actually correspond to a similar target group. Finally, isolated points that do not convincingly fall into a cluster should be effectively skipped since they are unlikely part of an interesting target group. Let’s begin by evaluating how well-suited current approaches for unsupervised discretization meet these goals. The approaches can be classified as being one of three types: 1) equal-width-intervals, 2) equal-frequency-intervals, or 3) clustering. The first two cases simply divide the attribute into a series of equal width or frequency intervals. Consider the following case for a target attribute. In Figure 1, both the equal-width and equal-frequency methods identify clusters with lower density that are located very close to neighboring clusters. Existing clustering approaches do not satisfy the third goal of target discretization P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 44–52, 2009. c Springer-Verlag Berlin Heidelberg 2009
Discretization of Target Attributes for Subgroup Discovery
45
Fig. 1. Example target discretization using equal-width- and equal-frequency-intervals
since they assign all points to clusters with the exception of outliers. We conclude that current unsupervised discretization methods are not well suited for target discretization. We propose a new method for target discretization that simultaneously achieves the above three goals of target discretization. The algorithm uses a dynamic programming approach. The next section provides details.
2
Solution Algorithm for Cluster Problem
Let the values of the target attribute to be discretized be v1 , v2 , . . . , vN in ascending order. We separate clusters of these values using intervals containing roughly log2 N consecutive values. We say “roughly” since the actual number of values can be somewhat less. We cover details later, when we also motivate the selected size of intervals. Two intervals are disjoint if they do not contain a common value. A collection of intervals is acceptable if the intervals are pairwise disjoint. Suppose we have a collection of acceptable intervals. If we color the values vi occurring in the intervals red and the remaining values green, then there are green subsequences separated by red subsequences. The values in each green subsequence define a cluster. A collection of intervals is feasible if it is acceptable and if each cluster contains at least two values vi . We want a valuation of feasible collections of intervals. To this end, we define the width wk of the kth cluster, say containing values vi , vi+1 , . . ., vj , to be wk = vj − vi
(1)
The score of a feasible collection c is sc = max wk k
(2)
If the score is small, then all clusters have small width, an attractive situation. Let C be the set of feasible c. An optimal collection c∗ ∈ C has minimum score
46
K. Moreland and K. Truemper
among the c ∈ C, and thus is the most attractive one. Therefore, the score of c∗ is defined via sc∗ = min sc (3) c
We determine an optimal collection c∗ by dynamic programming as follows. For m ≥ 0 and 1 ≤ n ≤ N , we compute optimal solutions for the restricted case of m intervals and for the sequence of values v1 , v2 , . . . , vn . We denote the optimal score under these restrictions by sm n . The stages of the dynamic programming recursion are indexed by m, where m = 0 means that no interval is used at all. In a slight change of notation, denote the width of a cluster starting at vl and terminating at vr by wl,r Since each cluster of a feasible c contains at least two values vi , the scores s0n for the stage m = 0 are given by ∞, if n = 1 0 sn = (4) w1,n , otherwise For statement of the recursive formula for any stage m ≥ 1, let a be any acceptable interval for the sequence v1 , v2 , . . . , vn . Referring again to the coloring of the values, this time based on the values in a being red and other values being green, define the largest v-value that is smaller than all red values to have index al , and define the smallest v-value that is larger than all red values to have index ar . Differently stated, the green values val and var bracket the red values of the interval a. With these definitions, the recursive equation for stage m ≥ 1 can be stated as follows. m−1 sm , war ,n } (5) n = min max{sal a
We still must discuss the choice of the size of the intervals. Suppose the interval is chosen to be very small. Then random changes of values may move a value from one cluster across the interval into another cluster, an unattractive situation. We want to choose the interval large enough so that such movement becomes unlikely. In [2], this situation is investigated for the classification of records into two classes. For the case of N records, the conclusion of the reference is that the interval should have roughly log2 N data points. In the present context, that size can lead to odd choices, given that values may occur in multiplicities. Using some experimentation, the following modification leads to reasonable results. First, a tolerance is defined as = (var − val )/10 (6) Take any value q ≤ N − 1 such that p = q − log2 N ≥ 2. Declare the values vi satisfying vp + ≤ vi ≤ vq − to be the points of an interval associated with index q. By having q take on all possible values, we obtain all intervals of interest for the optimization problem. There is one more refinement of the method based on experimentation. We do not want to obtain clusters that are not dense, where density of a cluster with values vi , vi+1 , . . . , vj is defined to be di,j = wi,j /(j − i + 1)
(7)
Specifically, we do not want to have clusters that have less density than the cluster containing the entire sequence v1 , v2 , . . . , vN . Thus, we declare the width
Discretization of Target Attributes for Subgroup Discovery
47
of any cluster whose density di,j is less than d1,N to be equal to ∞. We omit the simple modification of the above optimization formulas (4) and (5) that accommodate the change.
3
Implementation
We created a subgroup discovery algorithm using the EXARP learning algorithm of [7], which in turn utilizes the Lsquare algorithm of [3,4,8]. Each attribute is of one of three possible types: (1) target attribute only, (2) explanation attribute only, or (3) both target and explanation attribute. The goal is to explain the target attributes using the explanation attributes. The explanations correspond to the potentially interesting subgroups for each target. If an attribute is labeled both a target and explanation attribute, it may be explained since it is a target, and it may be part of explanations for other targets. Subgroup discovery proceeds as follows. The data set is evenly divided into a training and testing set. We set the testing data aside, and for the moment use only the training data. Each target is discretized as described in Section 2. For a given target, this yields j cutpoints with j uncertain intervals, for j = 1, 2, . . . For each such j, we use the intervals to partition the data into two sets, Atrain and Btrain , in two ways. In the first case, for a given interval, records with a target value below the interval make up the Atrain set while records with target value above the interval comprise the Btrain set. If a record’s target value falls within the interval, the record is considered uncertain and is not used in computation of explanations. The second case enumerates all pairs of consecutive intervals, k1 and k2 . For each pair, records with a target value less than interval k1 or larger than interval k2 make up the Atrain set while records with target value larger than interval k1 and less than interval k2 make up the Btrain set. Analogous to the first case, records with a target value that falls within interval k1 or k2 are uncertain and not used to compute the explanations. The testing data is partitioned into an A and B set in the same fashion. The EXARP method of [7] is called to compute two DNF logic formulas that separate the Atrain and Btrain sets using only the explanation attributes. One formula evaluates to True for all records of the Atrain set while the other evaluates to True for records of the Btrain set. Each clause of the resulting formulas describes a potentially interesting target subgroup. Using the discretized testing data, we calculate the significance of subgroups somewhat differently from current methods [1,5,6]. Suppose a formula evaluates to True on the A records and let T be a clause of the formula. We define the subgroup S associated with clause T to be S = {x ∈ A | clause T evaluates to True}
(8)
Let n be the number of records in S. A random process is postulated that constructs records of the A set using the clause T . We skip the details, but mention the following. Let p be the probability that the random process produces at least n records that satisfy the clause. Define q to be the fraction of set B
48
K. Moreland and K. Truemper
for which the clause correctly has the value False. Then the clause significance is given by s = (1 − p + q)/2 (9) To reduce the number of significant explanations in the output to the user, logically equivalent explanations are grouped together in equivalence classes, as follows. Each attribute appearing in a clause is one of four possible types: (1) the attribute has low values (e.g., x < 2.5), (2) high values (e.g., x > 7.5), (3) low or high values (e.g., x < 2.5 | | x > 7.5), or (4) neither low nor high values (e.g., 2.5 < x < 7.5). Two explanations are logically equivalent if they meet the following criteria: (1) The explanations describe subgroups for the same target attribute, (2) the explanations contain the same attributes, and (3) each attribute must be of the same type in both explanations. The explanation in the equivalence class with the highest significance is chosen as the representative for the class. The representative explanations constitute the interesting subgroups presented to the user.
4
Computational Results
For testing of the TargetCluster algorithm, we use the Abalone, Heart Disease, and Wine sets from the UC Irvine Machine Learning Repository as well as two data sets supplied by H. Bickel, Technical University Munich, Germany, and S. K¨ ummel, Charit´e Berlin, Germany. The data set from Technical University Munich concerns Dementia and is called SKT; it contains binary and continuous attributes. The data set from Charit´e Berlin is called Cervical Cancer and contains only continuous attributes. Thus, we have 5 data sets called Abalone, Cervical Cancer, Heart Disease, SKT, and Wine. Table 1 summarizes the data sets. We compare the number and significance of subgroups obtained via TargetCluster discretization with the results found via a combined equal-width-intervals Table 1. Summary of Data Sets Data Set Abalone
No. of No. of Attribute Type Rec’s Attr’s Target Only Explanation Only number of measurements, 4177 9 rings gender
Both none
Cervical Cancer 109
11
none
none
serum levels
Heart Disease
303
14
none
age
patient measurements
SKT
183
20
9 test scores, overall score
patient measurements
none
Wine
178
14
none
none
color intensity, hue, intensity levels
Discretization of Target Attributes for Subgroup Discovery
49
Fig. 2. Abalone Data Set Table 2. TargetCluster vs EWF Significance of Highest Number of Subgroups Identified Ranking Subgroup Significance ≥ 0.98 TargetTargetData Set EWF EWF BOTH TOTAL Cluster Cluster Abalone 0.97 0.98 0 1 0 1 Cervical Cancer 0.95 0.99 0 3 0 3 Heart Disease 0.99 0.98 2 1 0 3 SKT 0.93 0.98 0 1 0 1 Wine 0.99 0.99 2 7 2 11 Average 0.966 0.984 0.8 2.6 0.4 3.8
and equal-frequency-intervals method (EWF). In both cases, the discretization data are input to the subgroup discovery method described in Section 3. The graphs of Figures 2 - 6 illustrate the number of subgroups identified when EWF and TargetCluster are used and different significance thresholds are applied. Except for the Heart Disease case, TargetCluster leads to more subgroups with high significance than EWF. Table 2 shows the significance value of the highest ranking subgroup produced via each of the discretization methods. In all cases, TargetCluster discretization leads to the discovery of at least one additional subgroup with significance ≥ 0.98. For instance, using the Cervical Cancer data set, 3 subgroups with significance ≥ 0.98 are identified with TargetCluster that are not identified using EWF.
50
K. Moreland and K. Truemper
Fig. 3. Cervical Cancer Data Set
Fig. 4. Heart Disease Data Set
On average, TargetCluster results in subgroups with higher significance than EWF. For the SKT data set, EWF does not result in any subgroup with significance higher than 0.93. However, the TargetCluster method leads to a subgroup with significance of 0.98. On average, TargetCluster results in 2.6 additional subgroups per data set not produced when EWF is used.
Discretization of Target Attributes for Subgroup Discovery
51
Fig. 5. SKT Data Set
Fig. 6. Wine Data Set
Based on these findings, the TargetCluster method likely is a useful additional discretization tool for subgroup discovery.
5
Summary
This paper describes the TargetCluster algorithm for discretization of continuous targets in subgroup discovery. The algorithm is based on a clustering approach
52
K. Moreland and K. Truemper
on the real line. TargetCluster has been combined with a subgroup discovery method. In tests, the use of TargetCluster resulted in a number of additional interesting subgroups when compared with standard unsupervised discretization methods.
References 1. Atzmueller, M., Puppe, F., Buscher, H.-P.: Exploiting background knowledge for knowledge-intensive subgroup discovery. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005) (2005) 2. Bartnikowski, S., Granberry, M., Mugan, J., Truemper, K.: Transformation of rational and set data to logic data. In: Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Springer, Heidelberg (2006) 3. Felici, G., Sun, F., Truemper, K.: Learning logic formulas and related error distributions. In: Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Springer, Heidelberg (2006) 4. Felici, G., Truemper, K.: A MINSAT approach for learning in logic domain. INFORMS Journal of Computing 14, 20–36 (2002) 5. Kl¨ osgen, W.: Subgroup discovery. In: Handbook of Data Mining and Knowledge Discovery. Morgan Kaufmann, San Francisco (2002) 6. Lavraˇc, N., Flach, P., Kavsek, B., Todorovski, L.: Adapting classification rule induction to subgroup discovery. In: Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM 2002) (2002) 7. Riehl, K., Truemper, K.: Construction of deterministic, consisten, and stable explanations from numerical data and prior domain knowledge. Working paper (2008) 8. Truemper, K.: Design of Logic-based Intelligent Systems. Wiley, Chichester (2004)
Preserving Privacy in Time Series Data Classification by Discretization Ye Zhu1 , Yongjian Fu1 , and Huirong Fu2 1
Cleveland State University, Cleveland OH. 44115, USA
[email protected],
[email protected] 2 Oakland University, Rochester, MI 48309, USA
[email protected]
Abstract. In this paper, we propose discretization-based schemes to preserve privacy in time series data mining. Traditional research on preserving privacy in data mining focuses on time-invariant privacy issues. With the emergence of time series data mining, traditional snapshot-based privacy issues need to be extended to be multi-dimensional with the addition of time dimension. In this paper, we defined three threat models based on trust relationship between the data miner and data providers. We propose three different schemes for these three threat models. The proposed schemes are extensively evaluated against public-available time series data sets [1]. Our experiments show that proposed schemes can preserve privacy with cost of reduction in mining accuracy. For most data sets, proposed schemes can achieve low privacy leakage with slight reduction in classification accuracy. We also studied effect of parameters of proposed schemes in this paper.
1 Introduction Privacy has been a serious issue with the popularity of data mining. To ease privacy concerns, data mining researchers have proposed privacy-preserving data mining[2]. In privacy-preserving data mining, data is mined in a way such that its privacy is protected. Many approaches have been proposed for privacy-preserving data mining. To protect privacy, some modify data by adding noises, generalization, transformation, and so on. Others split data among multiple parties and they cooperate to find global patterns without sharing data. Mining of time series data has been an interesting topic due to the abundance of time series data. Though many privacy-preserving data mining approaches have been successful, they are either ineffective or inapplicable for time series data such as stock prices, sales, and weather. Privacy is an important issue in time series data mining, which few solutions have been proposed yet. Due to the temporal dimension of time series data, its privacy goes beyond traditional privacy of data. In time series data mining, we need to preserve privacy not only in
This work was partly supported by the National Science Foundation under Grant No. CNS0716527. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 53–67, 2009. c Springer-Verlag Berlin Heidelberg 2009
54
Y. Zhu, Y. Fu, and H. Fu
time domain, i.e., the data values, but also in frequency domain, e.g., periodicity. For example, a store wants to protect not only its sales data, but also the periodicity of its sales, which could reveal sales promotions. Moreover, many existing privacy preserving techniques developed for time series data cannot be directly applied for time series data. For example, if random noise added to a time series is independent of the time series, the noise can be easily filtered out to reveal the time series. In this paper, we propose to use discretization for preserving privacy in time series data mining. In our approach, a time series is discretized such that its domain is segmented into a set of ranges. A representative value is selected for each range. Each individual value is then mapped into a range and replaced by a corresponding representative value. This is similar to generalization where a value is replaced by its range. We choose to use representatives instead of ranges so that the modified data is still a time series. Extensive experiments have been conducted using real data sets. The rest of the paper is organized as follows: In Section 2, we review related work in privacy preserving and time series data mining. Section 3 identifies privacy issues in time series data mining. We describe the system model and the threat model in Section 4. In Section 5, we present discretization-based schemes to preserve privacy in time series data mining. We evaluate proposed schemes in Section 6 in terms of privacy leakage and mining accuracy. In Section 7, we discuss related issues. We conclude our paper in Section 8.
2 Related Work In this section, we first briefly summarize research in privacy preserving data mining. We then discuss related work in privacy preserving in time series data mining. Many approaches have been proposed for privacy preserving data mining. These approaches can be broadly classified into two categories. The first category of approaches can be labeled as data modification approaches, in which data is modified to protect privacy and mining is performed on modified data. Modification techniques in this category include adding random noise [2,3], transformation [4], aggregation [5], generalization [6,7,4,8], randomization [9,10,11], and suppression [8,4]. The second category of approaches does not modify data. Instead, data is split among multiple parties, who uses secure multi-party computing to find global patterns without sharing local data, therefore protecting privacy. Most of them use encryption as well. We call the second category data distribution approaches. Secure multi-party computing was introduced as a technique for privacy preserving decision tree construction in [12]. It had been used for association rule mining [13,14], clustering with k-means[15,16], and learning Bayesian network structures [17]. Some recent papers address specifically privacy preserving in time series data mining. A privacy preserving algorithm for mining frequent patterns in time series data has been proposed by Silva and Klusch [18]. They assume that time series data is distributed among multiple parties. Each time series is first segmented, then discretized into a sequence of symbols. A frequent pattern is a sequence which occurs frequently in the time series. The algorithm uses encryption and secure multi-party computing to find global frequent patterns without breaching the privacy of individual parties.
Preserving Privacy in Time Series Data Classification by Discretization
55
Time series data privacy has been studied by Papadimitriou et. al. [19]. They argue that time series data has unique characteristics in terms of privacy. In order to preserve privacy, they propose two perturbation methods based on Fourier and wavelet transformations respectively. It is shown that white noise perturbation does not preserve privacy while the proposed methods are effective. In [20], we also point out that privacy of time series data is more than just the identity of data. We argue that existing privacy preserving techniques such as aggregation and white noise are ineffective for preserving privacy in time series data. We present a privacy attack that can separate individual time series from aggregate data or data with white noise. Though our paper addresses privacy issues in time series data mining, our approach is different from the previous papers. We propose to use discretization to preserve privacy instead of secure multi-party computing as proposed in [18]. Besides, [19] and [20] do not consider the effects of privacy preserving on data mining results. We investigate the relationship between privacy and classification accuracy.
3 Privacy Issues of Time Series In this section, we identify privacy issues for time series data in addition to traditional privacy issues in data mining. Time series data from a data source can be regarded as a time-domain signal. All the characteristics of a time-domain signal can be potentially regarded as private information by the data provider. Below, we list common characteristics in time series data that a data provider may need to keep confidential. – Amplitude: Amplitude indicates the strength of a signal. This is the same as in traditional privacy research. – Average: The average signal strength over time. For example, for a series of sales data, average amplitude indicates the average sales. – Peak and trough: Peak and trough indicate extreme situations. The information is usually considered confidential as it may disclose extreme changes in underlying causes such as difficulties in money flow. – Trend: By observing trends of time series data, an adversary may predict future changes of time series data. Thus trend information should be protected from competitors as well. – Periodicity: Periodical changes in time series data indicate existence of periodically changing factors. For sales data of a store, the factor can be periodical changes in marketing strategies such as promotions which are usually regarded as confidential information for stores. Unlike the previous characteristics which are in time domain, periodicity is in frequency domain.
4 System Model and Threat Model In this section, we describe the system model and the threat model.
56
Y. Zhu, Y. Fu, and H. Fu
Fig. 1. Privacy-Preserving Time Series Data Classification Model
4.1 System Model Based on discretization approaches, the privacy-preserving data classification system is as shown in Figure 1. In this model, data providers first discretize their own time series data to protect privacy and then transmit discretized time series data to the data miner. The data miner constructs classification models and builds the classifier. Data miner can use the classifier to analyze available time series data. Data miner can also use the classifier to analyze data provided by classification customer. The classification customer can be one of the data providers since classification based on models built on time series data provided by more data providers is generally more accurate or more interesting to data providers. The classification customer can also be an outsider, meaning not one of the data providers, but the classification customer wants to analyze data against models built from data contributed by other data providers. The data miner may provide the classification service to “outside” customers for profit. 4.2 Threat Model In this paper, we assume the participants of classification systems are semi-honest. In other words, the participants will follow the protocols strictly. But the participants can record the communication. Especially the data miner can record the data supplied by data providers and analyze these data for other purpose such as searching sensitive information from these data.
Preserving Privacy in Time Series Data Classification by Discretization
57
Based on the trust relationship between data providers and the data miner, the threat models can be divided into following three categories: – Low-Trust Threat Model: In this model, data providers are willing to share discretized time series data with the data miner. Data providers are not interested in data analysis through models built by the data miner. Data providers in this model mainly care their data privacy: They are willing to share discretized if their privacy can be preserved. – High-Trust Threat Model: In this model, data providers are willing to share discretized time series data and are interested in utilizing models built by the data miner. Data providers in this model care about both their data privacy and accuracy of the models partially built from the shared data. – Full-Trust Threat Model: In this model, data providers have full trust on the data miner. Data provides are willing to share original time series data with the data miner without discretization. The data miner may sell the models built from the data supplied by data providers so that an “outsider” can obtain the models and classify data by herself. The data miner cares about accuracy of the models and has the responsibility to protect data providers’ privacy. To protect data providers’ privacy, the data miner will discretize the original time series data collected from data providers and build the models from discretized data. So the system model corresponding to this threat model is slightly different from the one shown in Figure 1: The discretization is done at data miner’s side. All three models are valid in real-world applications. For example, low-trust threat model can be used when data providers are competitors to each other. High-trust models can be used when data providers wants to do data mining based on global data for accuracy. Full-trust model can be used when data providers have full trust on the data miner. The full-trust relationship can be found between company branches and the company headquarter: Company branches provide data for mining at the company headquarter.
5 Preserving Privacy by Discretization In this section, we first formally generalize discretization method from previous definitions and then introduce our schemes. 5.1 Generalization of Discretization Discretization is a process to convert continuous values to discrete values. More formally, the discretization process can be described as follows: First the possible value space is partitioned into N consecutive intervals: [−∞, p1 ), [p1 , p2 ), [p2 , p3 ), · · ·, [pN −1 , +∞], where pi > pj , if i > j. The ith data sample in original time series s(t) is discretized as d(i) = vm , if s(i) ∈ [pm−1 , pm ] ,
(1)
where d(i) denotes the ith sample in the discretized time series d(t), m is the index of the interval, vm is the discretized value and vm ∈ [pm−1 , pm ]. In the language of data
58
Y. Zhu, Y. Fu, and H. Fu
compression, index m is also called as discretization level, the vector [v1 , v2 , · · · , vN ] is called as codebook. Obviously partition parameters p1 , · · · , pN −1 and discrete values v1 , · · · , vN determine discretization performance. In previous studies, these parameters are mostly chosen to minimize discretization error. The discretization error is usually defined as mean square error between original time series and discretized time series, i.e., δ=
l
(si − di )2 .
(2)
i=1
In our research, the discretization parameters are chosen for preserving privacy and minimizing classification error. 5.2 Privacy-Preserving Schemes Based on Discretization We introduce privacy-preserving schemes for all the three threat models below.
Algorithm 1. Scheme for Low-Trust Threat Model Data Miner Side: 1. Broadcast a request message containing classification accuracy limit (δlimit ) to data providers ; 2. Wait for messages containing maximum variance from all the data providers; 3. Select maximum variance reported by data providers and calculate N , the number of discretization levels; 4. Broadcast the number of discretization levels (N ) to data providers; 5. Waiting for data messages from data providers; 6. Upon receiving a data message, forward the message to model construction component as shown in Figure 1. Data Provider Side: input : privlimit , the maximum acceptable privacy leakage of the data provider 1. Upon receiving a request message from the data miner, send back the maximum variance of original time series; 2. Wait for the message containing the number of discretization levels (N ) from the data miner; 3. Calculate minimum possible privacy leakage (privmin ) based on δlimit and original time series; 4. if privmin > privlimit 5. Goto 1; 6. else 7. Discretize Original Time Series to achieve minimum possible privacy leakage (privmin ); 8. Send a data message containing the codebook and discretization levels of each time series to the data miner; 9. end if
Preserving Privacy in Time Series Data Classification by Discretization
59
Scheme for Low-Trust Threat Model. Based on the system model shown in Figure 1, the communication protocol between the data miner and data providers is shown in Algorithm 1. Before explaining details of the scheme, we would like to first introduce metrics and parameters used in the scheme. The first metric to be introduced is privacy leakage. It is defined as absolute value of the correlation between an original time series and its corresponding discretized time series, i.e., privleak (s(t), d(t)) = |corr(s(t), d(t))| (3) where corr() denotes correlation function. The privacy leakage is within the range [0, 1]. A larger value of the metric privl eak means less effectiveness in privacy preserving. As introduced in Section 3, all the characteristics about time series can be potentially regarded as private information. So our privacy leakage metric is based on correlation, a general metric used to measure similarity. The important parameter for data providers is the parameter privlimit , the maximum acceptable privacy leakage. The larger privlimit is, the more private information is allowed to be disclosed. So a data provider caring a lot about data privacy can choose a small privlimit to protect her privacy. The important parameter for data miner is discretization error limit δlimit . It is used by the data miner to ensure accuracy of models built on discretized time series, i.e., classification accuracy. Without δlimit , data providers may choose discretization parameters only to minimize privacy leakage so that the classifier built on the discretized time series may cause a large classification error. Before discretizing time series data and sending discretized data to the data miner, data providers need to exchange parameters with the data miner to ensure both classification accuracy and privacy protection. The exchange begins with broadcast of discretization error limit δlimit from the data miner to data providers. Data providers will respond the request message with the maximum variance of their own original time series. The data miner selects the overall maximum variance among maximum variances reported by data providers and calculates the number of discretization levels (N ). Then the data miner broadcasts N to data providers. After receiving N , the data provider calculate the minimum possible privacy leakage given the constraints of N and δlimit . The calculation will be introduced in the rest of this section. If the minimum possible privacy leakage is less than privlimit , one data provider discretizes her own time series data to minimize privacy leakage. Otherwise, the data provider stops and waits for request message from the data miner. A data provider shares discretized time series with the data miner by transmitting the codebook [v1 , v2 , · · · , vN +1 ] and discretization levels [m1 , m2 , · · · , ml ] of each sample in original time series. The discretized time series can be recovered at the data miner’s side from the codebook and discretization levels. Calculation of Discretization Level N : The data miner needs to calculate the discretization level based on the maximum variance σ 2 and discretization error limit δlimit . Before introducing the details of the calculation, we would like to introduce the following theorem.
60
Y. Zhu, Y. Fu, and H. Fu
Theorem 1. The minimum number of discretization levels required to achieve the dis√ 2 2σ cretization error limit δlimit is δlimit . Proof. From rate distortion theorem of a Gaussian source [21], the rate distortion √ funcσ2 2σ2 tion is R = 12 log δlimit . So the number of discretization level required is at least δlimit for all possible distributions. The minimum number of discretization levels in Theorem 1 can not be directly used in our scheme since it can only be achieved by very long time series according to the rate distortion theorem. In our scheme, the number of discretization levels is chosen σ2 heuristically: N = r δlimit . The scaling factor r is larger than 1. We will analyze the scaling factor in Section 6. Calculation of Minimum Possible Privacy Leakage: Finding minimum possible privacy leakage can be formulated as an optimization problem. The cost function is min
{[p1 ,p2,···, pN −1 ],[v1 ,v2 ,···vN ]}
k
privleak (si (t), di (t))
(4)
i=1
where k is number of time series to be sent to the data miner. The optimization is over all the possible discretization parameters. The constraint of the optimization problem is k 1 (si (t) − di (t))2 ≤ δlimit . k i=1
(5)
Algorithm 2. Scheme for High-Trust Threat Model Data Miner Side: 1. Broadcast a request message containing the number of discretization levels (N ) and the local classification accuracy threshold (Tacc ); 2. Waiting for data messages from data providers; 3. Upon receiving a data message, forward the message to model construction component as shown in Figure 1. Data Provider Side: 1. Upon receiving a request message from the data miner, optimize cost function (4) with the constraint of local classification accuracy threshold (Tacc ) ; 2. if the optimization problem is not solvable; 3. Goto 1; 4. else 5. Discretize Original time series with optimal discretization parameters found in Step 1; 6. Send a data message containing the codebook and discretization levels of each time series to the data miner.
Scheme for High-Trust Threat Model. The communication protocol of the scheme for high-trust threat model is shown in Algorithm 2. In this scheme, a local classification accuracy threshold (Tacc) is broadcast from the data miner to data providers. Data providers optimize discretization of time series in terms of privacy leakage. The
Preserving Privacy in Time Series Data Classification by Discretization
61
constraint to the optimization problem is the local classification accuracy threshold. In other words, data providers are required to guarantee that mining on discretized local times series can achieve classification accuracy Tacc . Formally, the optimization problem can be defined as follows: (a) The cost function is as shown in (4). (b) The constraint is Pacc ≥ Tacc where Pacc denotes the classification accuracy on discretized local time series. Scheme for Full-Trust Threat Model. The communication protocol for this scheme is simple: First the data miner broadcast request a message to data providers. Data providers respond the request message with original time series since data providers fully trust the data miner. After receiving original time series from data providers, the data miner will discretize all time series. The discretization parameters are chosen to optimize the cost function 4. The constraint of local classification accuracy in scheme for high-trust model changes to the constraint on global classification accuracy, i.e., classification accuracy on all discretized time series in this scheme. The data miner will build the classifier from the discretized time series. Comparing the scheme for high-trust threat model and this scheme, the major differences are: (a) In the scheme for high-trust threat model, discretization is done by data providers. In this scheme, discretization is done by the data miner. (b) In the scheme for high-trust threat model, optimization is run on time series owned by each data provider. In this scheme, optimization is run on all the time series.
6 Experiment Results In this section, we investigate the performance of proposed schemes. We chose 15 1 time series data sets from the UCR Time Series Classification/Clustering data collection [1]. Each data set has a training set and a test set. These data sets are quite different in terms of size, length of time series, number of classes, and one-nearest-neighbor (1-nn) Euclidean distance. Unless stated otherwise, our experiments are conducted using all 15 data sets and the results are averaged across all data sets. In our experiments, we simulate three data providers 2 . The time series in each class are randomly and evenly assigned to each data provider. Each experiment result is averaged over 10 different random combinations. For fair comparison, the number of discretization levels is set to be 16 for all the experiment results. In our experiments, the optimization problems in schemes are solved by the simulated annealing method [22]. 6.1 Overall Performance Table 1 shows the performance of all three schemes. In the table, NA means not applicable. In other words, it means that the optimization problem has no solutions. 1
2
Five data sets are not included in our experiments because of not enough time series of one or more classes. We assume there are three data providers so that all selected data sets have at least one time series in each class for each data provider. We get similar results in experiments of large data sets with more data providers.
62
Y. Zhu, Y. Fu, and H. Fu Table 1. Scheme Performance for Low-Trust, High-Trust, and Full-Trust Models
Original Low-Trust Classification Accuracy3 Classification Privacy Accuracy Leakage Beef 0.53 0.36 0.32 Adiac 0.61 NA NA CBF 0.85 0.37 0.20 Coffee 0.75 0.63 0.94 ECG200 0.88 0.52 0.11 FaceAll 0.71 0.42 0.54 Fish 0.78 NA NA Gun Point 0.91 0.56 0.31 Lighting2 0.75 0.45 0.05 Lighting7 0.58 0.23 0.21 OliverOil 0.87 0.41 0.15 OSULeaf 0.52 NA NA SwedishLeaf 0.79 NA NA Synthetic Control 0.88 0.55 0.44 Trace 0.76 0.25 0.04 Average 0.75 0.41 0.22 Dataset
High-Trust
Full-Trust
Classification Privacy Accuracy Leakage 0.47 0.55 0.31 0.96 NA NA 0.56 0.11 NA NA 0.38 0.56 NA NA 0.76 0.97 0.45 0.05 0.16 0.33 0.70 0.96 NA NA NA NA 0.81 0.94 0.66 0.93 0.58 0.73
Classification Privacy Accuracy Leakage 0.43 0.12 NA NA 0.69 0.39 0.61 0.06 0.71 0.17 0.57 0.39 0.63 0.48 0.74 0.33 0.61 0.19 0.47 0.46 0.70 0.25 0.41 0.52 0.63 0.63 0.71 0.47 0.61 0.11 0.61 0.34
For the scheme proposed for low-trust model, optimization solutions may not be found when classification accuracy limit δlimit and the number of discretization levels N are too small: So that it is impossible to discretize time series with a given classification accuracy limit. For the schemes proposed for high-trust and full-trust models, optimization solutions may not be found when classification accuracy threshold Tacc is too high: So that it is impossible to achieve the required classification accuracy with certain number of discretization levels. Datasets with NA results for any of these three models are excluded from average calculation in Table 1. Experiments results in the table are obtained with following parameters4: – Scheme for Low-Trust Model: The parameter δlimit is calculated with the scaling factor r to be 20 as described in the previous section. The parameter privlimit is set to one meaning optimization is over all possible discretization parameters. We set the parameter privl imit to one because we would like to investigate the tradeoff between privacy leakage and classification accuracy. In practice, data providers can choose their own privlimit to determine whether to share data with the data miner. – Scheme for High-Trust Model: Since different dataset has different original classification accuracy, we set local classification accuracy threshold Tacc to be 80 percent of original classification accuracy. – Scheme for Full-Trust Model: Similar as high-trust model, the classification accuracy threshold is set to be 80 percent of original classification accuracy. 4
Further experiments on these parameters are described in the rest of this section.
Preserving Privacy in Time Series Data Classification by Discretization
63
From Table 1, we can make following observations: – Proposed schemes can preserve privacy at the cost of reduction in classification accuracy. For several data sets, the privacy leakage can reach less than 0.1. Performance in terms of privacy leakage and classification accuracy for different data sets can vary a lot. For example, for the Beef data set, with cost of 6 percent reduction in classification accuracy, it is possible achieve 0.55 privacy leakage using the scheme for the high-trust model. For the Gun point data set, 0.97 privacy leakage is at the cost of 25 percent reduction in classification accuracy. The difference is mainly because of different amount of redundancy in different data sets: If redundancy is high in one data set, it is possible to have less reduction in the distance between different cluster after discretization, i.e., less reduction in classification accuracy. – The scheme for low-trust model can protect privacy when classification accuracy limit is large. On average, privacy leakage is 0.22 for the scheme proposed for lowtrust model. But its average classification accuracy is 0.41, about 34 percent less than original classification accuracy. – For the scheme proposed for high-trust model, privacy leakage can achieve 0.73 with the cost of 17 percent reduction in classification accuracy. – In general, the scheme for full-trust model performs better than the scheme for high-trust model in terms of both privacy leakage and classification accuracy: The scheme for high-trust model can achieve 0.34 privacy leakage with the cost of 14 percent reduction in classification accuracy. The performance difference between these two scheme because the scheme for full-trust model optimizes discretization over all time series and the scheme for high-trust model only optimizes discretization over local time series. – The scheme for high-trust model can not guarantee the classification accuracy is 80 percent of original classification accuracy. For example, the scheme for hightrust model can get classification accuracy of 0.31 for Adiac dataset, almost half of original classification accuracy. It is because local optimization performed by data providers in this scheme can only guarantee the classification accuracy of discretized local time series. The scheme for full-trust model can guarantee classification accuracy to be at least 80 percent of original classification accuracy because of the global optimization. 6.2 Effect of Scaling Factor r In this set of experiments, we study the scaling effect r for the scheme proposed for low-trust model. We use the coffee data set from the UCR time series database [1] in this set of experiments. Similar results are observed on other data sets in the time series database. We fix the number of discretization levels to be 16. So the change in scaling factor r reflects the change of the parameter δlimit . Experiment results are shown in Figure 2. From Figure 2(a), we can observe that privacy leakage decreases when the scaling factor r increases. A larger scaling factor means larger classification accuracy limit δlimit . In turn, it means larger range for the min operator in the cost function (4). So the minimal information leakage decreases when the scaling factor r increases. Figure 2(b) shows that classification accuracy fluctuates between 0.63 and 0.78. The fluctuation is because optimization in the scheme for
64
Y. Zhu, Y. Fu, and H. Fu
0.95
0.8
0.9
0.78 0.76
Classification Accuracy
Privacy Leakage
0.85
0.8
0.75
0.7
0.65
0.72 0.7 0.68 0.66
0.6
0.55 20
0.74
0.64
30
40
50
60 70 Scaling Factor (r)
80
90
0.62 20
100
(a) Privacy Leakage
30
40
50
60 70 Scaling Factor (r)
80
90
100
(b) classification accuracy Fig. 2. Effect of Scaling Factor (r)
1.00 0.90
0.75
Scheme for HighŦTrust Model Scheme for FullŦTrust Model
0.80
Scheme for HighŦTrust Model Scheme for FullŦTrust Model
0.70
Classification Accuracy
Privacy Leakage
0.70 0.60 0.50 0.40
0.65
0.60
0.30 0.20
0.55
0.10 0.00 0.08
0.15
0.23
0.30 0.38 0.45 0.53 Classification Accuracy Threshold (Tacc)
(a) Privacy Leakage
0.60
0.68
0.75
0.50 0.08
0.15
0.23
0.30 0.38 0.45 0.53 Classification AccuracyThreshold (Tacc)
0.60
0.68
0.75
(b) classification accuracy
Fig. 3. Effect of classification accuracy Threshold Tacc
low-trust model focuses on privacy leakage only. The original classification accuracy for coffee data set is 0.75. So we can observe that it is possible have classification accuracy slightly better than the original classification accuracy when the scaling factor r is large. 6.3 Effect of Classification Accuracy Threshold Tacc In this set of experiments, we focus on the classification accuracy threshold Tacc used in schemes for high-trust and full-trust models. The data set used in this set of experiments is again the coffee data set from the UCR time series database [1]. Similar experiment results can also be observed for other data sets.
Preserving Privacy in Time Series Data Classification by Discretization
65
From Figure 3, we can make following observations: – In Figure 3(a), the privacy leakage for both schemes increases as the classification accuracy threshold increases. It is because a larger classification accuracy threshold means tighter constraints on the optimization in both schemes for high-trust and full-trust models. So less privacy leakage can be achieved when the classification accuracy threshold is small. – We can also observe that for coffee data set, the privacy leakage approaches zero when the classification accuracy is 40 percent less than original classification accuracy. The cost of almost perfect privacy preservation is about 20 percent reduction in classification accuracy. – In terms of privacy leakage, the scheme for full-trust model outperforms the scheme for high-trust model. Simply because optimization in scheme for full-trust model is over time series contributed by all data providers and optimization in scheme for high-trust model is over local time series. – The scheme for high-trust model can guarantee overall classification accuracy is above classification accuracy threshold. For the scheme proposed for full-trust model, the overall classification accuracy can be lower than local classification accuracy threshold. For example, when local classification accuracy threshold is 0.65, the schemes for high-trust model and full-trust model have overall classification accuracy of 0.56 and 0.67 respectively. It is caused by the difference between the global optimization in scheme for full-trust model and the local optimization in scheme for high-trust model.
7 Discussion We focus on privacy leakage and classification accuracy in our investigation of discretization-based schemes to preserve privacy in time series data mining. Another obvious benefit of discretization-based schemes is data compression. We use 16 discretization levels in our experiments to discretize time series data. In other words, four bits are enough to represent each value in times series. Although the proposed schemes need extra bits to send codebooks, the schemes can save a large amount of bits in data storage and data transmission in comparison with mining with original time series data, especially for long time series. The scheme for low-trust model has the parameter privlimit , the maximal acceptable privacy leakage for data providers. This parameter can also be used in the scheme for high-trust model so that data providers can determine whether to share discretized time series based on this parameter.
8 Conclusion In this paper, we propose discretization-based schemes to preserve privacy in time series data mining. Based on trust relationship between the data miner and data providers, we defined three threat models. We propose three different schemes for these three threat models. The proposed schemes are extensively evaluated against public-available time
66
Y. Zhu, Y. Fu, and H. Fu
series data sets [1]. Our experiments show that proposed schemes can preserve privacy with cost of reduction in mining accuracy. For most data sets, proposed schemes can achieve low privacy leakage with slight reduction in classification accuracy. We also studied effect of parameters of proposed schemes in this paper.
Acknowledgment We thank Professor Keogh for providing us the data sets used in our experiments.
References 1. Keogh, E., Xi, X., Wei, L., Ratanamahatana, C.A.: The ucr time series classification/clustering homepage (2006), http://www.cs.ucr.edu/˜eamonn/time_series_data/ 2. Agrawal, R., Srikant, R.: Privacy-preserving data mining. In: SIGMOD Conference, pp. 439– 450 (2000) 3. Evfimievski, A.V., Srikant, R., Agrawal, R., Gehrke, J.: Privacy preserving mining of association rules. In: SIGKDD, pp. 217–228 (2002) 4. Iyengar, V.S.: Transforming data to satisfy privacy constraints. In: KDD 2002: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 279–288. ACM, New York (2002) 5. Zhang, N., Zhao, W.: Privacy-preserving data mining systems. Computer 40(4), 52–58 (2007) 6. Bayardo, R.J., Agrawal, R.: Data privacy through optimal k-anonymization. In: ICDE 2005: Proceedings of the 21st International Conference on Data Engineering, Washington, DC, USA, pp. 217–228. IEEE Computer Society, Los Alamitos (2005) 7. LeFevre, K., DeWitt, D.J., Ramakrishnan, R.: Workload-aware anonymization. In: KDD 2006: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 277–286. ACM, New York (2006) 8. Fung, B.C.M., Wang, K.: Anonymizing classification data for privacy preservation. IEEE Trans. on Knowl. and Data Eng. 19(5), 711–725 (2007); Fellow-Philip S. Yu 9. Du, W., Zhan, Z.: Using randomized response techniques for privacy-preserving data mining. In: SIGKDD, pp. 505–510 (2003) 10. Huang, Z., Du, W., Chen, B.: Deriving private information from randomized data. In: SIGMOD Conference, pp. 37–48 (2005) 11. Zhu, Y., Liu, L.: Optimal randomization for privacy preserving data mining. In: SIGKDD, pp. 761–766 (2004) 12. Lindell, Y., Pinkas, B.: Privacy preserving data mining. In: Bellare, M. (ed.) CRYPTO 2000. LNCS, vol. 1880, pp. 36–54. Springer, Heidelberg (2000) 13. Vaidya, J., Clifton, C.: Privacy preserving association rule mining in vertically partitioned data. In: SIGKDD, pp. 639–644 (2002) 14. Kantarcioglu, M., Clifton, C.: Privacy-preserving distributed mining of association rules on horizontally partitioned data. IEEE Trans. Knowl. Data Eng. 16(9) (2004) 15. Vaidya, J., Clifton, C.: Privacy-preserving k-means clustering over vertically partitioned data. In: SIGKDD, pp. 206–215 (2003) 16. Jagannathan, G., Wright, R.N.: Privacy-preserving distributed k-means clustering over arbitrarily partitioned data. In: SIGKDD, pp. 593–599 (2005)
Preserving Privacy in Time Series Data Classification by Discretization
67
17. Wright, R.N., Yang, Z.: Privacy-preserving bayesian network structure computation on distributed heterogeneous data. In: SIGKDD, pp. 713–718 (2004) 18. da Silva, J.C., Klusch, M.: Privacy-preserving discovery of frequent patterns in time series. In: Industrial Conference on Data Mining, pp. 318–328 (2007) 19. Papadimitriou, S., Li, F., Kollios, G., Yu, P.S.: Time series compressibility and privacy. In: VLDB 2007: Proceedings of the 33rd international conference on Very large data bases, pp. 459–470. VLDB Endowment (2007) 20. Zhu, Y., Fu, Y., Fu, H.: On privacy in time series data mining. In: Washio, T., Suzuki, E., Ting, K.M., Inokuchi, A. (eds.) PAKDD 2008. LNCS, vol. 5012, pp. 479–493. Springer, Heidelberg (2008) 21. Cover, T.M., Thomas, J.A.: Elements of information theory. Wiley-Interscience, New York (1991) 22. Ingber, L.: Adaptive simulated annealing (asa). Technical report, Pasadena, CA (1993)
Using Resampling Techniques for Better Quality Discretization Taimur Qureshi and Djamel A. Zighed University of Lyon 2 Laboratory ERIC - 5, Avenue Pierre Mendes 69676 Bron Cedex - France {taimur.qureshi,abdelkader.zighed}@eric.univ-lyon2.fr
Abstract. Many supervised induction algorithms require discrete data, however real data often comes in both discrete and continuous formats. Quality discretization of continuous attributes is an important problem that has effects on accuracy, complexity, variance and understandability of the induction model. Usually, discretization and other types of statistical processes are applied to subsets of the population as the entire population is practically inaccessible. For this reason we argue that the discretization performed on a sample of the population is only an estimate of the entire population. Most of the existing discretization methods, partition the attribute range into two or several intervals using a single or a set of cut points. In this paper, we introduce two variants of a resampling technique (such as bootstrap) to generate a set of candidate discretization points and thus, improving the discretization quality by providing a better estimation towards the entire population. Thus, the goal of this paper is to observe whether this type of resampling can lead to better quality discretization points, which opens up a new paradigm to construction of soft decision trees. Keywords: Bootstrap, discretization, resampling.
1
Introduction
Discretization is a general purpose preprocessing method that can be used for data exploration or data preparation in data mining. While they are critical in the case of decision tree methods, discretization methods can also be used for bayesian classifiers as in [14] and others. Many real-world classification algorithms are hard to solve unless the continuous attributes are discretized and the choice of discretization technique has important consequences on the induction model used. In the top-down induction of decision trees family, a simple unsupervised discretization procedure divides the range of a continuous variable into equalwidth intervals or equal-frequency intervals. While, supervised methods use information quality or statistical quality based measures to determine the interval P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 68–81, 2009. c Springer-Verlag Berlin Heidelberg 2009
Using Resampling Techniques for Better Quality Discretization
69
boundary points. Y.Yang et al [12] and Liu et al. [9] classified discretization methods from into different viewpoints: supervised vs. unsupervised, static vs. dynamic, global vs. local, top-down vs. bottom-up, Parameterized vs. Unparameterized, Disjoint vs. Non-disjoint, Univariate vs. Multivariate, primary vs. composite and Split vs. Merge. Out of these, Top-down methods as FUSBIN and MDLPC [5,6] start with one interval and split intervals in the process of discretization. While, bottom-up methods like FUSINTER[5] and Chi-Merge [4] split completely all the continuous values of the attribute and merge intervals in the process of discretization. Apart from these local discretization methods, global discretization [19], such as binning, produce a mesh over the entire ndimensional continuous instance space, where each feature is partitioned into regions independent of the other attributes. In this article, we focus on local topdown and bottom-up strategies for providing comparisons in terms of quality and prediction rates [1]. Our goal is to find a way to produce better discretization points. Previously, various studies have been done to estimate the discretization points from learning samples taken from the population. Because of inaccessibility of entire populations, we usually try to estimate statistical processes such as discretization from samples rather than the population. Significantly, in [1], a set of learning samples are used to approximate the best discretization points of the whole population. They argue that the learning sample is just an approximation of the whole population, so the optimal solution built on a single sample set is not necessarily the global one. Taking this point into consideration, in this paper we try to provide a better estimate toward the entire population. Our interpretation of the above problem leads us to use a resampling approach [2] to determine better distributions of the discretization points. In [18], the authors use various approaches including bootstrap to decrease discretization variance as preprocessing for decision trees. In this paper, we focus on obtaining discretization points where each candidate point has a higher probability to be the ”‘better”’ discretization point toward the whole population. By doing so, we attempt to improve on the predication accuracy of discretization and better estimation of the discretization points of the entire population, thus, treating the discretization problem in the statistical area with new results. We use ordinary bootstrap [11] as a method for resampling in our approach which tries to improve on the above mentioned problem. We argue that the recent increase in processing power of computers has allowed us to use extensive resampling analysis in order to find better estimates of the larger population. In this paper we focus on supervised discretization, however, unsupervised discretization can also be applied in the same way. In our approach, we estimate the discretization point distribution over an attribute Xi by repeatedly resampling and performing discretization on each bootstrap sample using any of the discretization approaches and thus, creating a histogram density function of the obtained candidate discretization points. Then, we apply a moving average filter to this function with a window size ws and try to obtain distinct regions (peaks) in the distribution most likely to be the exact discretization points towards the
70
T. Qureshi and D.A. Zighed
entire population. From this distribution we extract discretization points using two variant techniques explained later. The number of intervals of the discretization is determined by using a most frequent interval method. To conclude, we compare our approach with different top-down and bottom-up strategies (without resampling) in terms of discretization quality (defined later) and we see how close we get to an optimal solution (discretization points). This solution is build by applying a dynamic programming based technique known as fishers algorithm on a finite and known data set. In section 2, we lay out the framework for discretization and define the various terminologies. We also discuss the various data sets used in our experiments. In 3, we briefly describe various existing top-down and bottom-up discretization strategies used in our paper. Then, in section 4, we give a detailed illustration of our proposed scheme and then in 5, we show our results by applying the explained techniques to 10 benchmarking data sets. We also compare different discretization strategies to ours and at the end we conclude with observations, deductions and proposals for future work.
2 2.1
Terminologies and Definitions Framework and Formulation
Let X(.) be an attribute value on the real set . For each example ω of a learning set Ω, X(ω) is the value taken by the attribute X(.) at ω. The attribute C(.) is called the endogenous variable or class and is usually symbolic and if an example belongs to a class c, we have C(ω) = c. We also suppose that C(ω) is known for all ω of the learning sample set Ω. Thus, we try to build a model, denoted by Φ, such that ideally we have: C(.) = Φ(X1 (.), ..., Xp (.))). The discretization of X(.) consists in splitting the domain Dx of continuous attribute X(.), into k intervals Ij , j = 1, ...., k, with k ≥ 1. We denote Ij = [dj−1 , dj ] with the dj s called the discretization points which, are determined by taking into account the particular attribute C(.). The purpose of the method is to build a model which can predict class C(.) relative to X(.). 2.2
Resampling
We use an Ordinary Bootstrap method [11] for resampling in which the learning and test sets overlap. In this, a prediction rule is built on a bootstrap sample and tested on the original sample, averaging the misclassification rates across all bootstrap replications gives the ordinary bootstrap estimate. 2.3
Smoothing
We use a moving average filter as a smoothing technique which is the unweighted mean of the previous n data points.
Using Resampling Techniques for Better Quality Discretization
2.4
71
Quality of Discretization
We use three factors to analyze the quality of the obtained discretization. Prediction Accuracy. The goal of the discretization is to make the class C(.) predictable by an attribute X(.). To measure this prediction rate we define a notion of prediction accuracy of the achieved discretization as follows: The discretization of the attribute Xj from a sample Ωs , provides k intervals denoted Iij ; i = 1, .., k. For each ω taken from the test sample Ωt we denote Iij the interval to which it belongs after discretization of the sample Ωs . The point ω will be labeled C(ω) = c∗ if the majority of the points in the Ωt that are in Iij have the class c∗ . This corresponds to a bayesian decision rule with a matrix of symmetrical costs and prior probabilities of the classes estimated by the proportion of the individuals belonging to each class in the Ωt . We measure the quality of the discretization by the rate of good predictions: τj =
ˆ card{ω∈Ωt /C(ω)=C(ω) } card{Ωt }
We denote by τj the good prediction rate resulting from the discretization of Xj obtained by applying a method on the sample Ωs . Complexity. In measuring the complexity of a discretization we take into account the number of intervals Inumber obtained in the resulting discretization. Large number of intervals increase the complexity of the induction algorithm that use the discretized input and also because a large number of intervals are likely to increase the discretization bias and variance. Higher discretization variance effects the quality of discretization even if the prediction accuracy is high. This property is explained by Yang et al [12]. Robustness. We introduce a concept of robustness as previously used in [14]. This is equal to the prediction accuracy in terms of the training sample divided by the predication accuracy of the whole population (which is known in our experiments). This measures the degree of accurate estimation of the population from a small training sample.
3
Evaluated Discretization Techniques
We have evaluated 7 different discretization methods. Out of these 3 methods use top-down technique such as (MDLPC, Fusbin, BalancedGain) and 3 use bottomup (Fusinter,Chimerge,MODL), while 1 is based on an optimal algorithm (Fisher). A brief description of these methods is given below, while we describe 2 of these methods in detail. The MDLPC method is a greedy top-down split method, whose evaluation criterion is based on the Minimum Description Length Principle [16]. At each
72
T. Qureshi and D.A. Zighed
step of the algorithm, the MDLPC evaluates two hypotheses (to cut or not to cut the interval) and chooses the hypothesis whose total encoding cost (model plus exceptions) is the lowest. The Balanced-Gain method exploits a criterion similar to the Gain-Ratio criterion [10]: it divides the entropy-based Information Gain criterion by the log of the arity of the partition in order to penalize excessive multi-splits. Fusbin is a top-down method whereas, the Fusinter method is a greedy bottom-up method. Both exploit an uncertainty measure sensitive to the sample size. Its criterion employs a quadratic entropy term to evaluate the information in the intervals and is regularized by a second term in inverse proportion of the interval frequencies. The Chi-Merge [4] method is a greedy bottom-up merge method that locally exploits the chi-square criterion to decide whether two adjacent intervals are similar enough to be merged. MODL [14] algorithm is based on a bayesian approach. It defines a criterion which is minimal for the bayes optimal discretization. Fisher’s algorithm is a dynamic programming algorithm that tries to find the optimal partition given a criterion, introduced by W.Fisher [21]. The criterion that is used in this paper for Fisher’s algorithm is based on Fusinter’s uncertainty principle, which is presented in detail by Zighed et al [1].
4 4.1
Proposed Scheme Using Resampling Assumptions
1. In each interval, the distribution of the class values is defined by the frequencies of the class values in this interval. 2. The attributes the conditionally independent of each other given the class. 3. Thus, we discretize one attribute at a time with the class (in two dimensions). 4.2
Our Approach
Earlier, we argued that the learning sample is just an approximation of the whole population, so the optimal discretization solution built on a single sample set is not necessarily the global one. Thus, our approach is based on finding a better discretization estimate toward the entire population in terms of discretization quality (as discussed above), using a sample selected randomly from that population and then resampling it. We use a ordinary bootstrap based resampling technique to achieve this objective. These two solutions are explained below: Resampling based Smoothed Discretization Based Approach (RSD). This technique is carried out in three phases. In the first phase we estimate a discretization point distribution over an attribute Xi by repeated resampling n times and performing discretization on each bootstrap sample Ωbs using an entropy based MDLPC method and thus, creating a histogram density function of the resulting candidate points as shown in fig 1a. Note that we can use any discretization method in place of MDLPC but in our experiments we happen to use
Using Resampling Techniques for Better Quality Discretization
73
Fig. 1. a) Discretization point frequency distribution of variable 1 of the waveform data where the x-axis represents the attribute’s continuous values and the y-axis represents their frequencies. b). The distribution of fig 1a after smoothing with a ws = 7 and the resulting peaks.
this technique. During the discretization performed on each bootstrap sample, we record the number of intervals for each sample and build the histogram density function for the number of intervals obtained for n bootstraps. Then from this interval distribution we select the most probable interval obtained denoted as Ibest . This is illustrated in figure 2. In the second phase we smooth over the discretization point distribution function by applying a moving average filter with a window size ws. The resulting smoothed curve is shown in fig 1b. We can see distinct regions (or peaks) where the probability of the candidate points to be the exact discretization point is higher. As shown in fig 1b, the straight line that runs parallel with the x-axis is called the threshold parameter T , which is set as the median of the obtained frequency values. By taking into account this threshold T we define peaks Preg as the regions which lie above this threshold line. The reason for defining such a threshold is to help to elaborate only the regions of higher probabilities (most frequent). The ws is determined as follows: We start by setting ws = 3 and we
74
T. Qureshi and D.A. Zighed
Fig. 2. Number of intervals (interval frequencies) distributed in an example data
calculate the number of peaks Preg obtained. We continue to increment the ws until the the number of peaks Preg obtained approach the most probable interval number Ibest as above. In the third phase, we extract candidate discretization points from this smoothed function. For this phase we have two variations; (1) applying a voting procedure to the peaks(fig 1b) or (2) averaging the peaks. 1. From each peak Preg we select or vote the most probable or frequent point as shown in fig 1b. 2. For each peak Preg we take the average of all the points that lie in the peak region and select the point on the x-axis that corresponds to that average.
5
Results - Analysis and Comparisons
In our experimentations we apply our approaches RSD1 and RSD2 using the MDLPC discretization obtained from n = 100 bootstrap samples and try to build a near optimal solution. Then, we compare this solution to MDLPC original and other 6 discretization methods defined above. The MODL, MDLPC and BalancedGain methods have an automatic stopping rule and do not require any parameter setting. For the Fusinter criterion, we use the regularization parameters recommended in [17]. For the ChiMerge method, the significance level is set to 0.95 for the chi-square test threshold. For Fisher’s optimal algorithm we choose the criteria used in FusInter [1]. In order to find the add-value of our resampling based discretization techniques and compare them with the above mentioned top-down and bottom-up strategies, we measure the quality of discretization and comparing the strategies considering 10 different datasets taken from U.C. Irvine repository [15] shown in figure 3. The 10 data sets of fig 2 have 85 continuous attributes denoted as (X1 (ω), ..., X8 5(ω)) and a label C(ω). Note that the waveform dataset contains significant noise as compared to the others. We used a 10-fold cross validation to measure
Using Resampling Techniques for Better Quality Discretization
75
Fig. 3. Data sets and their summary
Fig. 4. Comparison of accuracy between the methods
the geometric mean of number of intervals μInumber , accuracy rate, and robustness from discretization of all the 85 variables using all the 7 methods discussed. Then we calculated RSD1 and RSD2 as described above. The resulting measures and their results are explained in the following subsections along with figures 4 to 8. Predication Accuracy. Figure 4 shows the cumulative accuracy rates of all the discretization methods plotted against the 85 continuous attributes. It shows
76
T. Qureshi and D.A. Zighed
Fig. 5. Bi-criteria evaluation of the methods for the accuracy and number of intervals, using datasets geometric means
the curve of RSD1 and RSD2, being above all the curves of the other methods including MDLPC from which our methods are originally built (by resampling and obtaining selected discretization points) as explained earlier. As we have plotted the curves using all the attributes so the graphical representation might not be very clear, thus we summarize this comparison in the table of fig 7. However, we note that for attributes 5 to 26 there is a lot of variations between the accuracy curves of the methods. These attributes belong to the waveform dataset and contain significant noise, causing the most of the discretization methods to deteriorate. Here, RSD1 and RSD2 perform significantly much better than the others as compared to the other attributes where at times the difference is small. Bi-criteria Evaluations. In multi-criteria analysis, a solution dominates (or is non-inferior to) another one if it is better for all criteria. A solution that cannot be dominated is Pareto optimal: any improvement of one of the criteria causes a deterioration on another criterion. The Pareto surface is the set of all the Pareto optimal solutions. In order to analyze both the accuracy and robustness results, we report the dataset geometric means on a two-criteria plan in Figure 5a, with the accuracy on the x-coordinate and the robustness on the y-coordinate. Similarly, we report the accuracy and the number of intervals in Figure 5b. Each point in these figures represents the summary of all the experiments. The multi-criteria figures are thus reliable and informative: they allow us to clearly differentiate the behavior of almost all the methods. Accuracy is certainly the most important parameter to distinguish a discretization method so we have grouped it in both the analysis of fig 5 and 6.
Using Resampling Techniques for Better Quality Discretization
77
Fig. 6. Bi-criteria evaluation of the methods for the accuracy and robustness, using datasets geometric means
Fig. 7. Comparison of the Robustness between all the methods
Complexity - Number of Intervals. In this section we compare the number of intervals generated by each method. Here, we point out that the number of intervals produced by RSD1 and RSD2 are the same so we don’t show them separately in fig 5. In illustrated in fig 5, RSD outperforms the other methods in terms of accuracy and complexity. The number of intervals are significantly better than MDLPC. Chimerge, Balancedgain are outperformed by Fusbin, Fusinter, MDLPC and MODL. Chimerge shows a relatively bad result.
78
T. Qureshi and D.A. Zighed
Fig. 8. Comparison of the critical area between all the methods
Robustness. The robustness is an interesting criterion that allows to estimate whether the performance on the sample train data is a good prediction of the performance on the whole data set (Population). The higher is the robustness, the most accurate will be a ranking of attributes based on their sample accuracy. This can be critical in the case of classifiers such as decision trees that incorporate an attribute selection method to build the next node of the tree. We calculated a higher mean robustness of RSD in comparison to other methods, with upto 6 to 7 percent better performance in comparison with Balanced-gain and Chi-merge, while 2 percent better performance in case of MDLPC, MODL and FUSINTER as shown in fig 8. This property exhibits the quality of our method as in a better estimation of the discretization points of the entire data set from a smaller data sample. Fig 6 clearly shows that RSD1 and RSD2 clearly outperforms all the other methods in accuracy and robustness. MDLPC and MODL are very close to each other with Fusinter not far behind. Balanced-gain has the worst overall robustness but chimerge has the worst accuracy. Comparison Summary. Instead of comparing the mean accuracy of all the methods, we measure the critical area t∗ of the difference of each method. The methods are compared two by two according to the following statistical procedure. Let u and v be two methods to compare. We form the difference Γuv between the rates of well ordered elements of the methods u and v. This difference is a random attribute which is roughly normal with parameters (μ, σ). We conclude that u is better than v if μ is significantly superior to 0. We have n = 85 * 10 observations. The estimated mean value μ and mean standard deviation σ are: 11,21 1 uv uv u v μuv = 850 j=1,s=1 γjs ; where, γjs = γjs − γjs σuv =
1 850
11,21 j=1,s=1
∗ ∗ uv − μ γjs uv and t is: t =
μuv√ σuv / n
> t1−α
Using Resampling Techniques for Better Quality Discretization
79
with t1−α the critical value at the rate α of a Student’s law with (n−1) degrees of freedom. Since, n is large, we have for α = 0.05, t1−α/2 = 1.96. The computed t∗ results are reported in figure 8. Positive values of t∗ indicate that the method in the row is better than the method in the column. Aside from Chi-Merge method whose results are relatively bad, all the other methods have relatively smaller differences. However, RSD1 and RSD2 significantly, report much better results and draw much closer to Fishers optimal results. Time Complexity. In terms of time complexity among those methods MDLPC seemed to be the best with a much lesser time complexity in comparison to Fisher’s optimal algorithm which is the most computationally intensive. The time complexity of RSD1 depends mainly on the number of bootstrap samples generated and discretized. The more bootstrap samples, the better the discretization quality. This is a trade-off between time complexity and quality. But with vast improvements in computing speeds, we argue that quality could be a much valuable commodity. Discretization Variance. By building a discretization point distribution by bootstrap and using techniques such as smoothing and averaging we try to lower the discretization variance that is obvious in the distribution of the discretization points. This significantly effects classifiers such as decision trees. In addition, this work has contributed to a fuzzy or soft discretization [20] technique which tries to cater to this problem of high discretization variance, where we smooth the edges of discretization boundaries.
6
Conclusion
The learning sample is an approximation of the whole population, so the optimal discretization built on a single sample set is not necessarily the global optimal one. Our Resampling based approaches tends to give a better discretization estimate in terms of achieving better discretization quality. Applying our schemes, significantly improves the discretization quality and thus, aiming to arrive nearer to a global optimal solution. Except for Chi-Merge and Balancedgain, the other methods provide small variations in terms of accuracy. MDLPC performs the best in terms of number of intervals and time complexity. RSD1 and RSD2 are computation intensive but seem to provide much better discretization quality and also perform well for noisy data. As future work, we shall apply this discretization approach in the context of decision trees, to see whether it improves the global performance or not. We are also aiming to apply this approach in leading to potential discretization points in the context of fuzzy or soft discretization in decision trees. But, at the same time carrying out this approach needs to answer some other questions such as the optimal number of bootstrap samples, optimal window size and time complexity.
80
T. Qureshi and D.A. Zighed
References 1. Zighed, D.A., Rabas´eda, S., Rakotomalala, R.: Discretization Methods in Supervised Learning. Encyclopedia of Computer Science and Technology 40, 35–45 (1998) 2. Wehenkel, L.: An Information Quality Based Decision Tree Pruning Method. In: Valverde, L., Bouchon-Meunier, B., Yager, R.R. (eds.) IPMU 1992. LNCS, vol. 682. Springer, Heidelberg (1993) 3. Fayyad, U.M., Irani, K.B.: On the Handling of Continuous-Valued Attributes in Decision Tree Generation. Machine Learning 8, 87–102 (1992) 4. Kerber, R.: Discretization of Numeric Attributes. In: Proceedings of the Tenth National Conference on Artificial Intelligence, pp. 123–128. MIT Press, Cambridge (1992) 5. Zighed, D.A., Rakotomalala, R., Rabas´eda, S.: Discretization Method for Continuous Attributes in Induction Graphs. In: Proceeding of the 13th European Meetings on Cybernetics and System Research, pp. 997–1002 (1996) 6. Fayyad, U.M., Irani, K.: Multi-interval Discretization of Continuous-Valued Attributes for Classification Learning. In: Proceedings of the 13th International Joint Conference on Artificial Intelligence, pp. 1022–1027. Morgan Kaufmann, San Mateo (1993) 7. Zighed, D.A., Rickotomalala, R.: A Method for Non Arborescent Induction Graphs. Technical Report, Laboratory ERIC, University of Lyon 2 (1996) 8. Ching, J.Y., Wong, A.K.C., Chan, K.C.C.: Class-dependent discretization for inductive learning from continuous and mixed mode data. IEEE Trans. on Pattern Analysis and Machine Intelligence 17(7), 641–651 (1995) 9. Liu, H., Hussain, F., Tan, C.L., Dash, M.: Discretization: An enabling technique. Data Mining and Knowledge Discovery 6(4), 393–423 (2002) 10. Quinlan, J.R.: Improved use of continuous attributes in c4.5. Journal of Artificial Intelligence Research 4, 77–90 (1996) 11. Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman and Hall, Boca Raton (1998) 12. Yang, Y., Webb, G.I.: Discretization for naive-bayes learning: managing discretization bias and variance. Technical Report 2003/131, School of Computer Science and Software Engineering, Monash University (2003) 13. Hsu, C.N., Huang, H.J., Wong, T.T.: Why discretization works for naive Bayesian classifiers. In: Proceedings of the Seventeenth International Conference on Machine Learning, pp. 309–406 (2000) 14. MODL: A Bayes optimal discretization method for continuous attributes. Journal of Machine Learning, 131–165 (2006) 15. Blake, C.L., Merz, C.J.: UCI Repository of machine learning databases. University of California, Department of Information and Computer Science, Irvine, CA (1998), http://www.ics.uci.edu/~ mlearn/MLRepository.html 16. Rissanen, J.: Modeling by shortest data description. Automatica 14, 465–471 (1978) 17. Zighed, D.A., Rabaseda, S., Rakotomalala, R.: Fusinter: a method for discretization of continuous attributes for supervised learning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6(33), 307–326 (1998) 18. Geurts, P., Wehenkel, L.: Investigation and reduction of discretization variance in decision tree induction. In: Lopez de Mantaras, R., Plaza, E. (eds.) ECML 2000. LNCS, vol. 1810, pp. 162–170. Springer, Heidelberg (2000)
Using Resampling Techniques for Better Quality Discretization
81
19. Chmielewski, M.R., Grzymala Busse, J.W.: Global discretization of continuous attributes as preprocessing for machine learning. In: Third International Workshop on Rough Sets and Soft Computing, pp. 294–301 (1994) 20. Peng, Y., Flach, P.: Soft Discretization to Enhance the Continuous Decision Tree Induction. In: Giraud-Carrier, C., Lavrac, N., Moyle, S. (eds.) Integrating Aspects of Data Mining, Decision Support and Meta-Learning, September 2001. ECML/PKDD 2001 workshop notes, pp. 109–118 (2001) 21. Fischer, W.D.: On grouping for maximum of homogeneity. Journal of the American Statistical Association 53, 789–798 (1958)
A Large Margin Classifier with Additional Features Xinwang Liu1, Jianping Yin1, En Zhu1, Guomin Zhang1, Yubin Zhan1, and Miaomiao Li2 1
School of Computer Science, National University of Defense Technology, Changsha, 410073, Hunan, China
[email protected] 2 College of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, 650216, Yunnan, China
[email protected]
Abstract. We consider the problem of learning classifiers from samples which have additional features that are absent due to noise or corruption of measurement. The common approach for handling missing features in discriminative models is first to complete their unknown values, and then a standard classification algorithm is employed over the completed data. In this paper, an algorithm which aims to maximize the margin of each sample in its own relevant subspace is proposed. We show how incomplete data can be classified directly without completing any missing features in a large-margin learning framework. Moreover, according to the theory of optimal kernel function, we proposed an optimal kernel function which is a convex composition of a set of linear kernel function to measure the similarity between additional features of each two samples. Based on the geometric interpretation of the margin, we formulate an objective function to maximize the margin of each sample in its own relevant subspace. In this formulation, we make use of the structural parameters trained from existing features and optimize the structural parameters trained from additional features only. A two-step iterative procedure for solving the objective function is proposed. By avoiding the pre-processing phase in which the data is completed, our algorithm could offer considerable computational saving. We demonstrate our results on a large number of standard benchmarks from UCI and the results show that our algorithm can achieve better or comparable classification accuracy compared to the existing algorithms. Keywords: Large Margin Framework, Incremental Missing Features Learning, Support Vector Machine, Kernel Method.
1 Introduction The task of learning classifiers with samples which have additional features and some of these features are absent due to measurement noise or corruption has been widely used in real world applications. For example, in fingerprint recognition, new samples may contain many additional features which can characterize fingerprints from other aspects. Another example is in Intrusion Detection field where many additional P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 82–95, 2009. © Springer-Verlag Berlin Heidelberg 2009
A Large Margin Classifier with Additional Features
83
features are contained in new samples as the diverse means of attack. However, the set of additional features may vary among samples in real world tasks, i.e. some of additional features are missing. Common methods for classification with absent features assume that the features exist but their values are unknown. The approach usually taken is a two-step procedure known as imputation. First the values of missing features are filled during a preprocessing phase, and then a standard classification procedure is applied to the completed data [1-2]. Imputation makes sense when the features are known to exist, while their values are absent due to noise, especially the values of the features are missing at random (when the missing pattern is conditionally independent of the unobserved features given in the observations), or missing completely at random (when it is independent of both observed and unobserved measurements). In the common practice of imputation application, absent features in continuous are often filled with zero, with the average of all of the samples, or using the k nearest neighbors (kNN) of each sample to find a filled value of its missing features. Another imputation method builds probabilistic generative models of the features using maximum likelihood or algorithms such as expectation maximization (EM) to find the most probable completion [3]. Such model-based methods allow the designer to introduce prior knowledge about the distribution of features, and are extremely useful when such knowledge can be explicitly modeled [4]. These methods have been shown to work very well for missing at random (MAR) data settings, because they assume that the missing features are generated by the same model that generates the observed features. However, model-based approaches can be computationally expensive and time consuming, and require significant prior knowledge about the domain. What’s more, they may produce meaningless completions for non-existing features, which will likely decrease classification performance. Motivated by reference [5], we propose a Large-Margin Learning Algorithm to tackle with data which have additional features and some of these features are missing. In this algorithm, incomplete data can be classified directly without any completing the missing features using a max-margin learning framework. Moreover, according to the theory of optimal kernel function, we proposed an optimal kernel function which is a convex composition of a set of linear kernel function to measure the similarity between additional features of two samples. We formulate an objective function, based on the geometric interpretation of the margin, which aims to maximize the margin of each sample in its own relevant subspace. In this formulation, we make use of the structural parameters trained from existing features and only optimize the structural parameters trained from additional features. A two-step iterative procedure is proposed for solving the objective function. In this two-step iterative procedure, the first step is to compute the Lagrange Multipliers corresponding to each constraints and the second step is to optimize the optimal weight corresponding to each feature. The two steps iterative until converge to a given criteria. This paper is organized as follows. In the next section we will present a MaxMargin formulation for incremental missing features. Section 3 will firstly give a large-margin learning algorithm with incremental absent features, and then the algorithm with an optimal kernel version is proposed. In section 4 we present the experiments comparing our approach to existing ones. We conclude with a discussion in Section 5.
84
X. Liu et al.
2 A Max-Margin Formulations for Incremental Missing Features Given a training set of samples with input data and , we can train these samples with Support Vector Machine and get and is the structural parameters and , where Lagrange multiplier corresponding to each constraint. As the tasks of detection, recognition and decision get further, new samples in classification may contain additional features and some of these additional features may be absent due to measurement noise or corruption. Traditional methods to deal with this classification task are to discard the original samples containing features and retrain a classifier with the new samples. Apparently, this treatment is timeconsuming and it discards the information learned from the original samples. Motivated by reference [5], we propose an incremental large-margin Learning Algorithm to tackle with data which have additional features and some of these features are missing. We formulate an objective function, based on the geometric interpretation of the margin, which aims to maximize the margin of each sample in its own relevant subspace. In this formulation, we make use of the structural parameters trained from existing features and only optimize the structural parameters trained from additional features. We formulate the proposed problem as follows. Considering a training set of samples
, where
and
represent existing and
sample, respectively. Let denote the set of indices of additional features of the and each is characterized by a subset of additional features of can therefore be viewed features from a full set of size m. Each as embedded in the relevant subspace . Each sample also has a . Importantly, since samples share features, the binary class label classifier we learn should have parameters that are consistent across samples, even if those samples do not lie in the same subspace. We address the problem of finding an optimal classifier in the max-margin framework. In the classical SVM method of reference [10], we learn the structural parameter of a linear classifier by maximizing the margin, defined as . Motivated by the ideal of incremental learning, we consider to learning the structural parameters from existing and additional features. Let and denote the structural parameters trained from already existing features and additional absent features, respectively. Consequently, the structural parameters trained from can be denoted as
. Consider now learning
such a classifier in the presence of samples which have additional features and some of these features are absent. In this case, we can make use of the structural parameters trained from already existing features and train the additional features in the max-margin framework only.
A Large Margin Classifier with Additional Features
85
To address this problem we treat the margin of each sample in its own relevant for the sample as subspace. We define the sample margin
(1)
is a vector obtained by taking the entries of that are relevant for , Where namely those for which the additional features has valid features. Its norm is . We now consider our new geometric margin to be the minimum over all sample margins, problem for the additional missing-features case
and get a new optimization
(2) For this optimization problem, since different margin terms are normalized by differin Equation (2), the denominator can no longer be taken out ent norms of the minimization. In addition, each of the terms is non-convex in , which makes it difficult to solve in a direct way. Section 3 will discuss how the geometric margin of Equation (2) can be optimized.
3 A Large Margin Classifier with Additional Features This section will firstly give a large margin classifier with additional features in the original space, and then we propose a kernel version of this algorithm in which the samples firstly are mapped into a high dimensional feature space and secondly a kernel is adopted to substitute the inner product in this high dimensional feature space. Finally, we propose two two-step iterative produces for the above cases, respectively. 3.1 A Large Margin Classifier with Additional Features in the Original Space Motivated by reference [5], we proposed a two-step iterative algorithm in original space for solving Equation (2). Firstly, we introduce a scaling coefficient and rewrite Equation (2) as
(3)
86
X. Liu et al.
Secondly, the following invariance is used: for every solution , there exists a solution that achieves the same target function value and with a margin that equals 1. This allows us to rewrite Equation (3) as a constrained optimization problem over and as follows.
(4)
Equation (4) is equivalent to the problem of minimizing same constraints. After introducing the slack variables therefore translates into
with the , Equation (4)
(5)
Equation (5) is not a quadratic problem. However, Equation (5) is a standard quadratic optimization problem over with constraints after is given, and then new is calculated using the resulting . To solve the quadratic program, we derive the dual problem for given .
(6)
Where the inner product is taken only over features that are valid for both . Table 1 provides a pseudo code for the two-step optimization algorithm. and 3.2 A Large Margin Classifier with Additional Features in the Feature Space In order to tackle with data which can not be linearly classified, the data in the original space is mapped to a so-called feature space by an implicit nonlinear mapping function
A Large Margin Classifier with Additional Features
87
Table 1. Pseudo code for the two-step iterative optimization algorithm in original space
Iterative Optimization Algorithm in Original Space Initialization: Initialize , the maximum iterative times 2. Iteration times and get the optimal . 2.1 Solve Equation (6) for the current
1.
2.2 Using 2.3 If else go to 2.1.
or
to update , stop.
.
and a hyper plane is constructed in this high dimension feature space. The inner product between two vectors in the feature space is computed by a kernel function. According to the theory of optimal kernel function [6], we proposed an optimal kernel function, which is a convex composition of a set of linear kernel function, to substitute the inner product in Equation (6). The optimal kernel function in this paper is as follows in Equation (7) (7) denote the weight of the feature, represents the feature of , Where is the number of all attributes, is a basic kernel function. Using Equation (7) substitutes the inner product in Equation (6), Equation (6) therefore translates into
(8)
For given
and , Equation (8) is a quadratic problem. can be optimized by Equation (9) (9)
Where we get a quadratic objective (10) to optimize
. Substitute .
to Equation (9),
88
X. Liu et al.
(10)
where product between
and and
is the inner
.
and , Equation (10) is a quadratic problem. We imFor given prove the iterative optimization algorithm in Table 1 as above. is set to linear function, which In our experiments, the basic kernel function means that the optimal kernel function is the convex composition of these basic linear , the optimal functions. When all the feature weight equals, i.e. kernel function degenerates to a linear function. This demonstrates that the iterative optimization algorithm in original space is a special case of the iterative optimization algorithm in feature space. Table 2. Pseudo code for the two-step iterative optimization algorithm in feature space
Iterative Optimization Algorithm in Feature Space Initialization: and the maximum iterative times Initialize . 2. Iteration 2.1 Solving Lagrange multipliers . 2.1.1 Solving Equation (8) for the current and to get . 2.1.2 Using to update .
1.
2.2 Solve Equation (10) for the current . or , stop. 2.3 If else go to 2.1.
and
to get the optimal
4 Experiment Results and Analyses In this section, we compare the performance of our algorithm with that of zeroimputation (the values of absent features are filled with zero), mean-imputation (the values of absent features are filled with the average of all of the samples) and maxmargin algorithm on 14 benchmark data sets which come from UCI Repository. Detailed information of these data sets is summarized in Table 3.
A Large Margin Classifier with Additional Features
89
Table 3. The information of data sets used in experiments Data set
#samples
#features
#existing features
#additional features
#training samples
#test samples
bupa
345
6
3
3
230
115
glass
146
9
5
4
98
48
ionospere
351
34
24
10
234
117
iris
100
4
2
2
66
34
monks
124
6
3
3
83
41
475
166
100
66
317
158
musk pima
768
8
4
4
512
256
promoter
106
57
45
12
71
35
tic-tac-toc
958
9
5
4
639
319
voting
435
16
9
7
290
145
wdbc
569
30
18
12
379
190
wine
130
13
7
6
87
43
spambase
4601
57
32
25
3068
1533
waveform
3345
40
20
20
2330
1115
In Table 3, for Glass and Waveform data set, we only take two-class samples. In experiments, the optimal kernel which is the convex of a serial of linear kernel functions is adopted. For each data set with samples and features, a random matrix is generated in each experiment. For a given missing threshold , is set to if . To overcome the influence of the random matrix to classification prediction, we get the statistical results by repeating each experiment 30 times. The performance of this algorithm is measured by means of prediction accuracy. The reported results are got by 3-fold cross validation. In zero-filling algorithm and mean-filling algorithm, the absent features are completed with zero and the mean of observed features, respectively. Then, a standard SVM classification algorithm is applied to the completed data. We denote the iterative optimization algorithm in original space and the iterative optimization algorithm in feature space as Algorithm1 and Algorithm2, respectively. The following part demonstrates the experiment results of Algorithm1 and Algorithm2, respectively. 4.1 The Experiment Results of Algorithm1 and Other Algorithms This subsection firstly illustrates the classification accuracy of Algorithm1, zerofilling algorithm and mean-filling algorithm, and then plots the classification accuracy curve of the three algorithms varying with the missing ratio.
90
X. Liu et al. Table 4. Prediction accuracy of three algorithms Data set
Algorithm1
zero-filling
mean-filling
bupa
57.5731±1.2633
56.8713±0.7272
60.5702±0.9386
glass
66.1458±3.7539
61.0069±6.6696
59.8611±6.3791
ionospere
82.1510±5.8258
81.2536±2.5666
80.9687±2.3525
iris
99.0625±1.2170
97.9167±2.3152
98.8542±1.6205
monks
54.7500±5.8559
54.1667±5.8926
55.3750±5.2510 54.7046±1.0988
musk
58.5970±5.0868
54.1772±1.0077
pima
74.0850±3.3262
75.4314±1.8304
75.1373±2.7241
promoter
62.9902±13.8341
60.9804±12.7562
60.4412±14.6031
tic-tac-toc
53.5168±8.4123
41.8553±8.1801
41.9916±7.8059 95.5977±1.3504
voting
95.5057±1.9797
95.8276±1.3666
wdbc
91.4638±2.1704
91.1640±2.8884
92.4515±3.2891
wine
91.5476±2.1318
90.7937±4.5905
89.6032±6.3846
spambase
84.4412±1.1707
85.0283±1.7815
85.1348±1.7421
waveform
94.1420±0.7427
94.0996±0.5444
94.1077±0.5481
The prediction accuracy of our algorithm, zero-filling algorithm and mean-filling algorithm is summarized in Tables 4, where the missing threshold . Each result is denoted as mean accuracy±stand error. Best method for each data set is marked with bold face. Form table 4, we see that in most cases our algorithm can outperform the other two algorithms in prediction accuracy and in other cases our algorithm can achieve comparable predication accuracy with the other three algorithms. In order to investigate the influence of missing ratio on Algorithm1 and other algorithms, we plot curves of the prediction accuracy as the missing ratio varied with different additional features in Figure1, Figure2, Figure3 and Figure4, respectively. Number of New Addition Featuresequals 5 on Waveform 92.45 Algorithm1 Zero-filling Mean-filling
Accuracy of Classification
92.4
92.35
92.3
92.25
92.2
0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
0.6
Fig. 1. The accuracy with different missing ratio(#(additional feature)=5)
A Large Margin Classifier with Additional Features
91
Number of New Addition Featuresequals 10 on Waveform 92.3
Accuracy of Classification
92.25
92.2
92.15
92.1
92.05
92
Algorithm1 Zero-filling Mean-filling 0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
0.6
Fig. 2. The accuracy with different missing ratio(#(additional feature)=10)
Considering the fact that the missing ratio is no more than 0.6 in real world applications, the missing varied from 0 to 0.6 with an increment 0.1. From Figure1, Figure2, Figure3 and Figure4, one can see that our algorithm shows higher prediction accuracy than that of the other two algorithms with the increment of missing ratio. We also can see that our algorithm demonstrates better performance as the number of additional features increases, which indicates that the performance of our algorithm is more robust than that of the other two algorithms. 4.2 The Experiment Results of Algorithm2 and Other Algorithms This subsection illustrates the classification accuracy of Algorithm2, zero-filling algorithm, mean-filling algorithm and the max-margin feature algorithm and then plots the predictive accuracy curve of four algorithms varying with the missing ratio. Number of New Addition Featuresequals 15 on Waveform 92.25 92.2
Accuracy of Classification
92.15 92.1 92.05 92 91.95 91.9 Algorithm1 Zero-filling Mean-filling
91.85
0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
Fig. 3. The accuracy with different missing ratio(#(additional feature)=15)
0.6
92
X. Liu et al. Number of New Addition Featuresequals 20 on Waveform 92.3
Accuracy of Classification
92.2
92.1
92
91.9
91.8
91.7
Algorithm1 Zero-filling Mean-filling 0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
0.6
Fig. 4. The accuracy with different missing ratio(#(additional feature)=20) Table 5. Prediction accuracy of four algorithms Data set bupa
Alogithm2
zero-filling
mean-filling
Max-margin
75.4721±1.0532
56.8713±0.7272
60.5702±0.9386
68.3512±1.2356
glass
67.3458±3.2546
61.0069±6.6696
59.8611±6.3791
52.7519±3.3486
ionospere
91.3547±5.6743
81.2536±2.5666
80.9687±2.3525
81.9488±2.5578
iris
99.3675±1.1350
97.9167±2.3152
98.8542±1.6205
99.3547±1.7365
monks
56.2500±4.7379
54.1667±5.8926
55.3750±5.2510
54.8768±5.4976
musk
69.7987±4.3812
54.1772±1.0077
54.7046±1.0988
56.8467±5.4936
pima
73.2854±2.4274
75.4314±1.8304
75.1373±2.7241
77.3697±3.8723
60.9804±12.7562
60.4412±14.6031
59.9471±12.7227
41.8553±8.1801
41.9916±7.8059
60.7687±7.4071
promoter
62.9902±13.8341
tic-tac-toc
57.4188±7.7183
voting
97.5997±1.6587
95.8276±1.3666
95.5977±1.3504
96.7976±1.4734
wdbc
91.4638±3.0908
91.1640±2.8884
92.4515±3.2891
92.7507±3.1211
wine
94.7487±2.2742
90.7937±4.5905
89.6032±6.3846
89.9602±5.8826
spambase
89.5671±1.7231
85.0283±1.7815
85.1348±1.7421
87.8339±1.3649
waveform
94.1420±0.8456
94.0996±0.5444
94.1077±0.5481
93.3903±0.5857
The prediction accuracy of our algorithm, zero-filling algorithm, mean-filling algorithm and the max-margin absent feature algorithm is summarized in Tables 3, where the missing threshold . Each result is denoted as mean accuracy±stand error. In order to investigate the influence of missing ratio on the four algorithms, we plot curves of the prediction accuracy as the missing ratio which varies from zero to 0.6 with increment 0.1 in Figure5 and Figure6 on Waveform and Spambase with additional features, respectively.
A Large Margin Classifier with Additional Features
93
Number of New Addition Featuresequals 20 on Waveform 96.5 Algorithm2 Zero-Filling Mean-Filling Max-Margin
96
Accuracy of Classification
95.5 95 94.5 94 93.5 93 92.5
0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
0.6
Fig. 5. The accuracy with different missing ratio on Waveform(#(additional feature)=20) Number of New Addition Featuresequals 25 on Spambase 91
Accuracy of Classification
90
89
88
87
Algorithm2 Zero-Filling Mean-Filling Max-Margin
86
85
0
0.1
0.2
0.3 Missing Ratio
0.4
0.5
0.6
Fig. 6. The accuracy with different missing ratio on Spambase(#(additional feature)=25)
From Figure5 and Figure6, we can see that as the missing ratio increases, the prediction accuracy of four algorithms decreases. Moreover, Algorithm2 shows higher prediction accuracy than that of the other three algorithms with the increment of missing ratio, which indicates that the performance of our algorithm is more robust than that of the other three algorithms. Algorithm1 and Algorithm2 can reduce training time by making use of the structural parameters trained from existing features and only training the additional features in a large margin framework. Moreover, by avoiding completing the data in the pre-processing phase, our algorithm could offer considerable computational saving. However, Algorithm2 cost much more computation resources than that of other three algorithms in optimizing feature weights , especially when samples contains a large number of features.
94
X. Liu et al.
Another fact, which should be emphasized, is that we have not analyzed the convergence of Algorithm1 and Algorithm2 in theory. However, in all experiments, Algorithm1 and Algorithm2 can all achieve an optimal solution.
5 Conclusions and Feature Work We propose an algorithm to learn classifiers from samples which have additional features and some of these additional features are absent due to measurement noise or corruption using a max-margin learning framework. We formulate an objective function, based on the geometric interpretation of the margin, which aims to maximize the margin of each sample in its own relevant subspace. In this formulation, we make use of the structural parameters trained from existing features and only optimize the structural parameters trained from additional features. A two-step iterative procedure for solving the objective function is proposed. We conduct experiments to evaluate the performance of the proposed algorithm, zero-filling algorithm and mean-filling algorithm on a large number of UCI data sets. In general, our algorithm can achieve better prediction accuracy and reduce the training time. Our cfuture work is to select a suitable kernel function for absent features and extend Algorithm2 to tackle with the problem that there are some features are absent in existing features. Algorithm2 cost much more computation resources in optimizing feature weight than that of other three algorithms, so efficient algorithm is our future work. Moreover, the convergence analysis of Alorithm1 and Algorithm2 in theory and apply our algorithms to real world applications is our other future work.
Acknowledgement This work is supported by the National Natural Science Foundation of China (No.60603015), the Foundation for the Author of National Excellent Doctoral Dissertation, the Scientific Research Fund of Hunan Provincial Education (the Foundation for the Author of Hunan Provincial Excellent Doctoral Dissertation ). We thank the reviewers for their constructive and insightful comments on the first submitted version of this paper.
References [1] Little, R.J.A., Rubin, D.B.: Statistical Analysis with Missing Data. Wiley, New York (1987) [2] Roth, P.: Missing data: A conceptual review for applied psychologists. Personnel Psychology 47(3), 537–560 (1994) [3] Ghahramani, Z., Jordan, M.I.: Supervised learning from incomplete data via an EM approach. In: Cowan, J.D., Tesauro, G., Alspector, J. (eds.) Advances in Neural Information Processing Systems, vol. 6, pp. 120–127. Morgan Kaufmann Publishers, Inc., San Francisco (1994)
A Large Margin Classifier with Additional Features
95
[4] Kapoor, A.: Learning Discriminative Models with Incomplete Data. PhD thesis, MIT Media Lab, Feburary (2006) [5] Chechik, G., Heitz, G., Elidan, G., Abbeel, P., Koller, D.: Max-margin Classification of Data with Absent Features. Journal of Machine Learning Research 9, 1–21 (2008) [6] Ong, C.S., Smola, A.J., Williamson, R.C.: Learning the Kernel with Hyperkernels. Journal of Machine Learning Research 6, 1043–1071 (2005) [7] Crammer, K., Keshet, J., Singer, Y.: Kernel design using boosting. In: Advances in Neural Information Processing Systems, vol. 15, pp. 537–544 (2002) [8] Ong, C.S., Smola, A.J.: Machine learning using hyperkernels. In: Proceedings of the International Conference on Machine Learning, pp. 568–575 (2003) [9] Liu, X., Zhang, G.: An Incremental Feature Learning Algorithm Based on Least Square Support Vector Machine. In: Preparata, F.P., Wu, X., Yin, J. (eds.) FAW 2008. LNCS, vol. 5059, pp. 330–338. Springer, Heidelberg (2008) [10] Kazushi, I., Takemasa, Y.: Incremental support vector machines and their geometrical analyses. Neuro-computing, 2528–2533 (2007) [11] Wang, L., Yang, C., Feng, J.: On learning with dissimilarity functions. In: Proceedings of the 24th international conference on machine learning, pp. 991–998 (2007) [12] Dick, U., Haider, P., Scheffer, T.: Learning from Incomplete Data with Infinite Imputations. In: Proceedings of the 25th international conference on machine learning, pp. 232– 239 (2008) [13] Williams, D., Carin, L.: Analytical kernel ma-trix completion with incomplete multi-view data. In: Proceedings of the ICML Workshop on Learning With Multiple Views (2005) [14] Williams, D., Liao, X., Xue, Y., Carin, L.: Incomplete-data classification using logistic regression. In: Proceedings of the 22nd International Conference on Machine learning (2005) [15] Chen, Z., Li, J.: Least Square Support Vector Machine. In: Proceedings of the 2006 International Conference on Computational Intelligence and Security, pp. 176–179 (2006) [16] Asuncion, A., Newman, D.J.: UCI machine learning repository [OL] (2009), http://archive.ics.uci.edu/ml/
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model Based Classifier Bashar Awwad Shiekh Hasan and John Q. Gan School of Computer Science and Electronic Engineering University of Essex Wivenhoe Park, CO4 3SQ, UK {bawwad,jqgan}@essex.ac.uk
Abstract. In this paper we present a sequential expectation maximization algorithm to adapt in an unsupervised manner a Gaussian mixture model for a classification problem. The goal is to adapt the Gaussian mixture model to cope with the non-stationarity in the data to classify and hence preserve the classification accuracy. Experimental results on synthetic data show that this method is able to learn the time-varying statistical features in data by adapting a Gaussian mixture model online. In order to control the adaptation method and to ensure the stability of the adapted model, we introduce an index to detect when the adaptation would fail.
1
Introduction
Gaussian mixture model (GMM) is a successful and simple clustering method that is widely used in many application domains. In GMM configuration the data are assumed to be generated from a finite number of Gaussian distributions. The data is then modeled by a probability density function. p(x) =
K
πk N (x|μk , Σk ) .
(1)
k=1
Where K is the number of Gaussian components, π1 , . . . , πK are the mixing coefficients, N (x|μk , Σk ) is a Gaussian distribution with μk mean and Σk variance. πk should satisfy the following conditions 0 ≤ πk ≤ 1 . and
K
πk = 1 .
(2)
(3)
k=1
In order to better model the data, a training method is required to estimate the model parameters. One well-known and widely-used method is ExpectationMaximization(EM). EM works by alternating between two steps: the E step that P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 96–106, 2009. c Springer-Verlag Berlin Heidelberg 2009
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model
97
uses the values of the current model parameters to calculate the responsibilities of the Gaussian components, and the M step in which the responsibilities are used to re-estimate the model parameters [1]. After a number of iterations the method converges to a model that maximizes the log likelihood of the data points given the model. EM is a batch method that provides a very good estimate of the clusters when the number of clusters is carefully chosen. In [2] the infinite GMM was introdcued to sidestep the difficult problem of deciding the “right” number of mixture components. The inference in the model is done using an efficient parameter-free Markov chain that relies entirely on Gibbs sampling. This results in adding new components to the mixture as new data arrives, which can work well to adapt to unforeseen data but it does not work within a classification configuration. In[3] and [4] Bayesian Information Criterion based incremental methods were developed to build a GMM with the best number of components. Those methods focus only on clustering problems and they still require all the data offline. Some customized adaptive methods for GMM can be found in the literature, like in [5], where the adaptation scheme is based on a constant learning rate that is used to update the GMM and the number of components is changing. This method works only when the data represent several clusters of one class. In this paper we introduce a sequential version of EM to train a GMM in the case where the statistical features of the data are changing over time. The GMM built here is meant to be used for classification which introduces more constraints on how to adapt the components in a way to maintain/improve the classification accuracy. We also introduce a method to detect misrepresentation of the classes.
2 2.1
Method Gaussian Mixture Model and EM
Here we briefly introduce a formulation of Gaussian mixtures in terms of discrete latent variables as it was introduced and deeply discussed in [1]. Let us introduce a K-dimensional binary random variable z having a 1-ofK representation in which a particular element zk is equal to 1 and all other elements are equal to 0. We will define the joint distribution p(x, z) in terms of a marginal distribution p(z) and a conditional distribution p(x|z). The marginal distribution over z is specified in terms of the mixing coefficients zk πk , with p(zk = 1) = πk . p(z) = K k=1 πk . The conditional distribution of x given a particular value of z is given as p(x|zk = 1) = N (x|μk , Σk ) and K p(x|z) = k=1 N (x|μk , Σk )zk . The joint distribution is then given by p(z)p(x|z), and p(x) =
z
p(z)p(x|z) =
K k=1
πk N (x|μk , Σk ) .
(4)
98
B.A.S. Hasan and J.Q. Gan
The introduction of z helps in calculating the responsibility of each component in the mixture γ(zk ) = p(zk = 1|x). The EM algorithm is shown in Method1., where γ(znk ) is the responsibility associated with data point xn , and γ(zk ) is the responsibilities associated with all the data points x. Method 1. The standard EM for GMM E-STEP: Evaluate responsibilities using current parameter values πk N (x|μk , Σk ) K j=1 πj N (x|μj , Σj )
γ(zk ) = p(zk = 1|x) =
.
(5)
M-STEP: Re-estimate parameters using current responsibilities
μnew = k Σknew =
1 Nk
1 Nk
γ(z
γ(z N
N
N n=1
(6)
nk )(xn
− μk )(xn − μk )T .
(7)
n=1
πknew = where Nk =
.
nk )xn
n=1
Nk . N
(8)
γ(znk )
The initial values for the model parameters can be calculated using a simple clustering method. In this study k-means algorithm is used. 2.2
Classification with Unsupervised Training
In the literature Gaussian mixture models were used for classification in a supervised way[6][7]. A GMM is built for each class and then a Bayesian classifier is used to classify new data(when the classes have the same prior the likelihood is enough for classification). This approach is not suitable for unsupervised adaptation as the labels are always required for training/adaptation. In order to overcome this problem we took a different approach. One GMM was built for all training/offline data and then the available labels were used to calculate p(c = classi |zk ), the probability of the class to be classi when the data point is generated from component zk . To calculate the probability of the data point x belonging to class classi , p(c = classi |x) is calculated as follows: p(c = classi |x) =
K k=1
p(c = classi |zk , x)p(zk |x) .
(9)
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model
99
where p(zk |x) is the responsibility of component zk to generate the point x. Assuming p(c = classi |zk , x) is independent of x then p(c = classi |x) =
K
p(c = classi |zk )p(zk |x) .
(10)
k=1
2.3
Sequential EM(SEM)
EM as an optimization method uses a batch of the available data to tune the model parameters and hence better model the data. In the case where we have an incoming stream of non-stationary data, the model built on the training/offline data will not be able to cope with these changes and this will cause a drop in the classification accuracy because of the misrepresentation of the new data by the old model. To tackle this problem a sequential/online version of EM method for GMM is introduced here. The main idea behind this method is to rewrite each parameter/ hyper-parameter as a convex combination of old and new data. This is valid here as all the parameters/hyper-parameters used by EM are sufficient statistics of the distribution. Method2. outlines the sequential EM method, where t is the current time point. Method 2. The sequential EM for GMM E-STEP: Evaluate responsibility using parameters at t-1 and x at t
γ(zkt ) =
t−1 πkt−1 N (xt |μt−1 ) k , Σk
K j=1
πjt−1 N (xt |μt−1 , Σjt−1 ) j
.
(11)
M-STEP: adapt the model parameters 1 (N t−1 μt−1 + γ(zkt )xt ) . k Nkt k
(12)
1 (N t−1 Σkt−1 + γ(zkt )(xt − μtk )(xt − μtk )T ) . Nkt k
(13)
Nkt . t
(14)
μtk = Σkt =
πkt = where Nkt = Nkt−1 + γ(zkt )
In the E-Step we calculate the responsibility γ(zkt ) associated with a new data point xt at time t based on the model parameters estimated at time t − 1 and using the new data point xt only. In the M-Step, the mean μtk is estimated based on the previous estimation t−1 μk and the new data point xt , similarly Σkt and Nkt are calculated based on the previously estimated parameters Σkt−1 , Nkt−1 , and xt .
100
B.A.S. Hasan and J.Q. Gan
γ(zkt ) controls the adaptation process, so the cluster that is closer to the new sample will adapt more than a further away one. The sequential method is initialized by a model that has been generated by the standard method trained on offline training data. The presented SEM is a parametric adaptive method, which does not take into consideration the problem of the optimal number of clusters that represent the data. The assumption is that the model trained over the offline data has already been chosen to optimally represent the data, using cross validation methods for example. 2.4
Adaptation Failure Detection
As the data shift over time the Gaussian components using SEM will try to keep track of changes in the new model, but it is still bounded by the old data. So when a dramatic change in the new data occurs the adaptation method will most likely fail in presenting the new data. This will affect the classification accuracy and most likely the system will end up with fewer dominant components. The other components still exist but they are not playing any actual role in representing/classifying the new data. Figure 1 shows the components’ responsibilities (summed over all data in the adaptation window), the straight lines are components that represent class1 and the boxed lines are components that represent class2. It is clear that one component becomes dominant over time and hence badly affect the classification accuracy. Another cause of failure of adaptation is due to the changes in the probability of class given a Gaussian component p(c = classi |zk ). The model might still be adapted well to the new data but because of the overlapping between the data of the two classes the Gaussian components might shift to represent one class better than the other(s). In order to enhance the system performance, a failure detection method is necessary to know when to stop adaptation, or re-train the system. We have used a simple and efficient method for failure detection based on the responsibility of the components to generate the new data p(zk |X), X = {x1 , . . . , xN } and the probability of the class given the component p(c = classi |zk ). Let cci = {k : p(c = classi |zk ) = max(p(c = classj |zk )), j = 1, 2} which contains the indexes of components that represent classi . We then define cli =
N
αin .
(15)
n=1
where
p(zk |xn ) : p(zk |xn ) = max(p(zj |xn )), j = 1, . . . , K and k ∈ cci . 0 otherwise (16) 4 cli gives an index of how probable classi is among the data X. The failure detection index is then defined as αin =
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model
101
F DI = cl1 /cl2 .
(17)
When the value of F DI > ul = 2.0 or F DI < ll = 0.5 then one of the classes has a dominant component(s) and the adaptive model has failed. For a more strict/loose constraint the upper limit ul and lower limit ll values can be changed accordingly. FDI deals with the first adaptation failure cause. Detecting the second problem is much harder as the labels are not available online. Instead of trying to detect the shifts of the components between the classes, we try to re-calculate the probability p(c = classi |zk ) after several adaptation batches. This is not a very efficient method though and it might actually affect the classification results especially if the classes are too overlapped.
components responsibilities over a window of data 2000 1800 1600
responsibility
1400 1200 1000 800 600 400 200 0
1
2
3
4
5
6 time
7
8
9
10
11
Fig. 1. Responsibilities of the adaptive GMM components
2.5
The Algorithm
The method starts by modeling the offline data using a predefined number of Gaussian distributions. Then the adaptation is done on a fixed sized windows of the online data, after each adaptation window the FDI is calculated to check the adapted system. If the adaptation has failed, re-training is used to build a model for the new data. Algorithm 1. shows an outline of the adaptive algorithm, where classifyComponents is a method that calculates p(c = classi |zk ). It should be pointed out that step 12 in the algorithm is not mandatory, one might want to stop the adaptation , see the discussion for more about this.
102
B.A.S. Hasan and J.Q. Gan
Algorithm 1. Adaptive GMM 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:
3 3.1
model = EMGMM(offlineData) classProb = classifyComponents(model,offlineData,labels) previousModel = model newModel = model while there is new data do classify(newDataPoint,previousModel,classProb) newModel = SEMGMM(newModel,newDataPoint) if size(newData)=adaptationWindow then FDI = calculateFDI(newModel,newData) if F DI > ul||F DI < ll then newModel = EMGMM(newData) classProb= classifyComponents(model,newData,newLabels) end if previousModel=newModel end if end while
Experimental Results Introduction
In order to objectively test the proposed method, synthesized data were used instead of real-life data. The data generated in a way to satisfy a number of criterions – the data present two-class problem – number of data points is balanced between the two classes – the data points are generated from a Gaussian mixture model with equal number of components for each class – linear separability of the data (between 80%-90% in our case) – the data is non-stationary over time – the non-stationary change in the data follow a pre-defined path Those criterions are necessary to make sure that the generated data represents the problem we are trying to solve here, and they make it possible to evaluate the method objectively. 3.2
Synthetic Data Generation
Here we detail the data generation method that complies with the previously stated criterions. The data were generated from a Gaussian mixture model with 6 components that represents 2 classes (3 components per class). The components’ mean were selected randomly on either side of a linear hyperplan, the covariances were selected randomly as well. 2000 data points were sampled from the original model and used as the offline data to train a Gaussian mixture model with 6
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model
103
components. The original model, for generating the data, was then shifted on a curve from θ = 0 to θ = π/2 and on 10 consequent steps, as shown in Fig.2. In each of these steps the covariances were scaled randomly and then another 2000 data points were sampled from the shifted model, this insures a control over the non-stationarity in the data. As the probability of both classes are assumed to be the same, the simulated data are balanced and the ongoing streams use small widows of samples by alternating between the two classes. The assumption of balanced online data does not affect the generality of the method, as this can be satisfied in most application areas by using a proper adaptation window that covers data from both classes, using some prior knowledge of the domain. Means of original data model shifted with time 6
5
4
3
2
1
0 −2
−1
0
1
2
3
4
5
6
Fig. 2. Path of the change in data means
3.3
Results
To test the adaptation method, first we provide the results without the use of FDI, so steps 9-13 are removed from the algorithm. The adaptation window is taken as 2000. The performance of the adapted model is tested on the data from the following window. Here we present the results from 10 data sets. Table1 shows the results using the model built on the offline data. Table2 demonstrates the results using the adaptive method. Figure3 shows the change of average accuracy over all the data sets. In the same figure we added the p values calculated using Wilcoxon signed-rank test as suggested in[8]. The signed-rank test results show significant enhancement of accuracy when the adaptive method is applied. Although the focus in this work is on unsupervised adaptation. We here present the results achieved using re-training after a failure detection using FDI to show it’s usability,as in Table3.
104
B.A.S. Hasan and J.Q. Gan Table 1. Results using the static model
Dataset T1 DS1 0.9370 DS2 0.9040 DS3 0.9095 DS4 0.9790 DS5 0.912 DS6 0.925 DS7 0.75 DS8 0.6935 DS9 0.901 DS10 0.82
T2 0.7610 0.8415 0.8985 0.9745 0.879 0.908 0.657 0.549 0.832 0.785
T3 0.6840 0.8470 0.9060 0.9805 0.8685 0.8835 0.6095 0.599 0.736 0.738
T4 0.6980 0.7705 0.8920 0.9945 0.882 0.7815 0.6055 0.5925 0.594 0.743
T5 0.6575 0.8105 0.8665 0.9510 0.8845 0.733 0.6565 0.562 0.552 0.654
T6 0.7325 0.8375 0.7890 0.9625 0.696 0.6305 0.647 0.706 0.4435 0.604
T7 0.6760 0.8260 0.6485 0.9145 0.559 0.5955 0.6775 0.718 0.4265 0.5825
T8 0.6770 0.7335 0.5865 0.8485 0.5405 0.5385 0.71 0.565 0.3825 0.5725
T9 0.6740 0.5630 0.5400 0.7435 0.521 0.506 0.604 0.544 0.479 0.5605
T10 0.7195 0.4670 0.4980 0.6565 0.5125 0.5 0.472 0.511 0.7445 0.499
Table 2. Results using the adaptive algorithm without re-training Dataset T1 DS1 0.937 DS2 0.904 DS3 0.9095 DS4 0.979 DS5 0.9115 DS6 0.925 DS7 0.75 DS8 0.6925 DS9 0.901 DS10 0.82
4
T2 0.956 0.8535 0.9015 0.985 0.921 0.915 0.6265 0.8385 0.849 0.8005
T3 0.9475 0.8855 0.9285 0.973 0.921 0.918 0.5 0.895 0.874 0.767
T4 0.9615 0.81 0.9165 0.991 0.9285 0.9065 0.498 0.869 0.8535 0.8065
T5 0.9595 0.8285 0.9235 1 0.926 0.8685 0.499 0.8765 0.842 0.7195
T6 0.963 0.8095 0.9095 0.9915 0.887 0.748 0.494 0.8635 0.8395 0.6635
T7 0.9785 0.8235 0.8415 0.992 0.8915 0.706 0.501 0.814 0.8 0.627
T8 0.9655 0.829 0.869 0.982 0.8715 0.6155 0.536 0.852 0.816 0.58
T9 0.9645 0.831 0.811 0.9875 0.8315 0.5235 0.49 0.77 0.877 0.4955
T10 0.961 0.8445 0.763 0.962 0.815 0.5135 0.464 0.682 0.839 0.483
Discussion and Conclusion
In this paper we have presented a sequential EM method to adapt a Gaussian mixture model in a classification configuration. In addition we have defined an adaptation failure detection index. This method is suitable for cases where the system data statistics suffers shifts over time, EMG and EEG data are examples of such a case. The data is expected to change slowly over time, sudden changes can be much harder to capture in our method. In[9], Neal presented a justification for online variant of EM. He showed that an online EM based on sequential E-step can converge faster than the standard EM. Sato [10] showed that the online EM algorithm can be considered as a stochastic approximation method to find the maximum likelihood estimator. Although we did not present a discount factor similar to the one presented by Sato, the online EM method for GMM presented here follows the general scheme of online EM in the literature. Here we did not deal with the problem of the optimal number of Gaussian components that represents each class. Some work on incremental addition/removal
Sequential EM for Unsupervised Adaptive Gaussian Mixture Model
105
average accuracies over 10 data sets p=0.5 0.85
accuracy
0.8
p=0.037109 p=0.048828 p=0.027344 p=0.037109 p=0.037109
0.75
p=0.037109 p=0.019531
0.7
p=0.019531 0.65
p=0.013672
0.6
0.55
2
4
6
8
10
12
time
Fig. 3. Average accuracies over time using static model(continuous line) and adaptive model (discrete line) Table 3. Results using the adaptive algorithm with re-training Dataset T1 DS1 0.937 DS2 0.904 DS3 0.9095 DS4 0.979 DS5 0.9115 DS6 0.925 DS7 0.75 DS8 0.6925 DS9 0.901 DS10 0.82
T2 0.956 0.8535 0.9015 0.985 0.921 0.915 0.8355 0.8385 0.849 0.8005
T3 0.9475 0.8855 0.9285 0.973 0.921 0.918 0.6835 0.895 0.874 0.767
T4 0.9615 0.81 0.9165 0.991 0.9285 0.9065 0.69 0.869 0.8535 0.8605
T5 0.9595 0.8285 0.9235 1 0.926 0.8685 0.7285 0.8765 0.842 0.8275
T6 0.963 0.8095 0.9095 0.9915 0.887 0.748 0.8475 0.8635 0.8395 0.8135
T7 0.9785 0.8235 0.8415 0.992 0.8915 0.925 0.83 0.814 0.8 0.788
T8 0.9655 0.829 0.869 0.982 0.8715 0.8395 0.7405 0.852 0.816 0.675
T9 0.9645 0.831 0.811 0.9875 0.8315 0.928 0.9855 0.77 0.877 0.8805
T10 0.961 0.8445 0.997 0.962 0.815 0.803 0.838 0.682 0.839 0.755
of Gaussian components was presented in[3][4], but in a classification configuration this is a very difficult problem. It is hard to know online ,in unsupervised way,the probabilities p(c = classi |zn ) where zn is the newly added component, so we assumed the number of components is static and only the model parameters are adapted. In the algorithm presented, Algorithm1., p(c = classi |zk ) are considered static and calculated only when building the original model. These probability distributions can be updated with time between sessions based on the current model classification. The re-training step mentioned in the algorithm can only be used if labels are available/partially available online. In a totally unsupervised adaptation scheme
106
B.A.S. Hasan and J.Q. Gan
FDI ,which is a fast and reliable measure of the adaptation, indicates when the adaptation fails and then one might stop adaptation and use the last known stable model. The size of adaptation window might have a considerable effect on the performance of the adaptation method. Small window might not change the model enough while longer window means larger drop in the ongoing classification until the new adapted model is used. The selection of the window size is determined mostly by the chosen application. An important feature of such a window is that it provides balanced number of examples from the two classes. This is important to protect the adaptation method from adapting to one class over the other. In [11], we have applied the proposed method in the field of Brain-Computer Interface (BCI). The experimental results showed the usefulness of this approach in building adaptive BCIs.
Acknowledgment The authors would like to thank Prof. Stephen Roberts for his useful input. This work is part of the project “Adaptive Asynchronous Brain Actuated Control” funded by UK EPSRC. Bashar’s study is funded by Aga Khan Foundation.
References 1. Bishop, C.: Pattern recognition and machine learning. Springer, Heidelberg (2006) 2. Rasmussen, C.E.: The infinite Gaussian mixture model. In: Advances in Neural Information Processing Systems, vol. 12, pp. 554–560 (2000) 3. Cheng, S., Wang, H., Fu, H.: A model-selection-based self-splitting Gaussian mixture learning with application to speaker identification. EURASIP Journal on Applied Signal Processing 17, 2626–2639 (2004) 4. Fraley, C., Raftery, A., Wehrensy, R.: Incremental model-based clustering for large datasets with small clusters. Tech. Rep. 439 (2003) 5. Shimada, A., Arita, D., Taniguchi, R.: Dynamic control of adaptive mixture-ofGaussians background model. In: AVSS 2006. Proceedings of the IEEE International Conference on Video and Signal Based Surveillance, vol. 5 (2006) 6. Marques, J., Moreno, P.J.: A study of musical instrument classification using Gaussian mixture models and support vector machines. Tech. Rep. CRL 99/4 (1999) 7. Millan, J.R.: On the need for on-line learning in brain-computer interfaces. In: Proc. IEEE International Joint Conference on Neural Networks, vol. 4, pp. 2877–2882 (2004) 8. Desmar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006) 9. Neal, R.M., Hinton, G.E.: A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models, 355–368 (1998) 10. Sato, M., Ishii, S.: On-line EM algorithm for the normalized Gaussian network. Neural Comp. 12(2), 407–432 (2000) 11. Awwad Shiekh Hasan, B., Gan, J.Q.: Unsupervised adaptive GMM for BCI. In: International IEEE EMBS Conf. on Neural Engineering, Antalya, Turkey (2009)
Optimal Double-Kernel Combination for Classification Feng Wang and Hongbin Zhang College of Computer Science, Beijing University of Technology, Beijing 100124, China
[email protected],
[email protected]
Abstract. Traditional kernelised classification methods could not perform well sometimes because of the using of a single and fixed kernel, especially on some complicated data sets. In this paper, a novel optimal double-kernel combination (ODKC) method is proposed for complicated classification tasks. Firstly, data sets are mapped by two basic kernels into different feature spaces respectively, and then three kinds of optimal composite kernels are constructed by integrating information of the two feature spaces. Comparative experiments demonstrate the effectiveness of our methods. Keywords: Classification, kernel learning, pattern recognition.
1 Introduction The success of SVM[1][2] has extended the use of kernel method to other linear classification algorithms, such as Kernel Fisher discriminant (KFD) [3], and Kernel Minimum Squared Error (KMSE) [4], called traditional kernelised classification methods in this paper. The key to deliver high performance is that through feature mapping, the geometrical structure of the mapped data is able to adapt well to corresponding linear algorithms or at least possesses a better linear separability than that in the input space. Unfortunately, traditional kernelised classification methods could not perform well sometimes, especially on some complicated data sets. The reason is that only by using a single and fixed kernel, traditional methods are not able to adapt well to complicated data sets. Furthermore, performance of traditional methods could be even worse if an inappropriate kernel or inappropriate kernel parameters are used [5] [6]. In order to overcome the disadvantage of traditional kernelised method, a kind of method of constructing objective kernel function using label information for some specific tasks has been focused recently, such as [7][8][9][10]. Compared with traditional kernelised classification methods, these methods above make progress on the adaptability of kernels to various target data sets. But since restricting themselves to shallow architectures [11] like traditional kernel method, they are still limited in their performance on complicated data sets. Among flexible learning algorithms, [11] establishes a distinction between shallow architectures and deep architectures. Shallow architectures are best exemplified by kernel machines and neural networks with only one hidden layer. And deep architectures are perhaps best exemplified by multi-layer neural networks with several hidden layers. [11] presents mathematical and empirical evidence suggesting that shallow architectures are very inefficient for representing complex, highly-varying functions in terms of required P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 107–122, 2009. © Springer-Verlag Berlin Heidelberg 2009
108
F. Wang and H. Zhang
number of computational component and the training data, while deep architectures are proposed as a way to escape from the limitation. Although a number of learning algorithms for deep architectures have been available, training such architectures is still largely perceived as a big challenge. As a trade-off between traditional kernel methods and deep architectures algorithms, a novel optimal double-kernel combination method (ODKC) is proposed in this paper. [12] points out most kernel combination methods take a risk of losing information in the original kernel matrices and develops a method without losing that information. But the size of its kernel matrix is greatly enlarged so as to cause high computational complexity. ODKC possesses low complex property, because its kernel matrixes have the same size of original kernel matrices and it is easier to be constructed. Also, better than single kernel machine, ODKC is composed of two different basic kernels, which can take the advantages of various kernels to better adapt to complicated tasks. Meanwhile, ODKC possesses non-iterative property, since the problem of optimization reduces to a generalized eigenvalue problem. The paper is organized as follows. Firstly, basic idea is proposed in Section 2. Section 3 presents our method in detail. Following that, our algorithm is described in Section 4. In Section 5, experimental results on five datasets are shown. Finally, we give the conclusion remarks in Section 6.
2 Basic Idea , ,…, , of points , with corres, where m is the number of different classes, our basic
Given a training data set ponding labels 1,2, … , idea is illustrated in Figure 1.
Φ = φ1 o φ0
φ1 = QTφ2 (S )
φ0
Feature Space Input Space • Ο • Ο Ο Ο • • Ο Ο • •
Ο • Ο • •
Ο
• Ο Ο Ο • •
• •
• •••
ΟΟΟ ΟΟ Ο
Fig. 1. Feature mapping Φ · embeds the data of two classes ‘•’and ‘o’ from input space (a) into a feature space (c)
The aim of this work is to seek the optimal objective kernel mapping Φ , called ODKC kernel mapping composed of two basic kernel mappings, and . is chosen as a feature mapping corresponding to a valid basic kernel such as Polynomial Kernel, Gauss Kernel or ANOVA kernel etc. [13]. Data mapped by , illustrated in Figure 1(b), may be still nonlinearly separating, especially for some
Optimal Double-Kernel Combination for Classification
109
complicated data sets. So it is necessary to do another feature mapping in order to reshape geometrical structure of the data in favor of linear classification algorithms. We · as , where is a linear operator, and is defined as choose operator three forms in Section 3.3, just in order to form a unified framework, in which we can discuss the construction of the three kinds of composite kernels easily. How to define and how to optimize the linear operator according to a criterion function are two main focuses in this paper. The criterion function is introduced in Section 2.1. 2.1 The Criterion Function For the classification task, we aim to seek the optimal objective kernel mapping Φ according to a criterion function increasing intra-class compactness and inter-class separability. We, firstly, give the definition of the membership matrix before the criterion function is introduced. is called membership matrix, if the element is:
Definition 2.1.1: matrix
0, 1,
,
。
where f(x) is called membership function, f(xj) = i if and only if i is the label of xj Under appropriate constraints, a reasonable criterion is to minimize the following objective function min
,
,Φ
+
Φ
+
(1)
Φ
,
(2)
Where 1 |
|
Φ
is the centre of data of the k-th class in the feature space corresponding to feature mapping Φ · , |f -1(k)| is the number of data in the k-th class, max
Φ
is the square of the maximum distance between data in the i-th class and its centre in the feature space corresponding to feature mappingΦ · , and|| || is Euclidean norm. incurs a heavy The objective function with our choice of weights penalty if the points Φ are close to the centers of other classes. How to choose weights is to be discussed in detail in Section 3.4. It is obvious that the hypersphere , , Φ that solves the minimization problem is the hypersphere in the feature space containing all the data in the i-th class with the smallest radius .
110
F. Wang and H. Zhang
· and
3 Computations of
Let denote ODKC kernel matrix corresponding to ODKC kernel mapping Φ Based on Eq(2), we will discuss the computations Φ · and in detail below. 3.1 Solve the Optimization Problem of Eq. (2) After the criterion function is proposed, we are going to seek the optimal solution of Φ in the minimization problem of Eq. (2). Firstly, let’s predigest and transform Eq. (2). Let 1 |
|
,the second term of the right-hand side of Eq. (2) can be
and matrix written
(3) +
trace Q
Q
,
where, ,
,…, ,
, ,…,
, ,
+
is a diagonal matrix whose entries are row sums of matrix F, and is a diagonal matrix whose entries are column sums of matrix F. Substitute Eq. (3) into the objective function Eq. (2) and use the constraint 1 we get the minimization problem reduced to
,
min ,
Subject to
+ 1.
Optimal Double-Kernel Combination for Classification
111
Following that, we can solve the constrained and simplified optimization problem by 1, … , for each defining a Lagrangian involving one Lagrange multiplier constraint. ,
,
+
1 .
1, … ,
We then solve by setting partial derivatives of L with respect to 1, … , equal to zero ∂ ∂
2
0
1
(4) and
(5)
0
(6)
1, … , are just the eigenvalues of Eq. (5) shows that Lagrange multipliers 1, … , are the eigenvectors of matrix A. From Eq. (5) and matrix A and Eq. (6), we can see the optimal solution of Lagrangian function is given by the smallest s eigenvalues and corresponding eigenvectors of A. In fact, the symmetric matrix A is indefinite, the eigenvalues of A may be positive, negative or zero. To minimize function L, we let s equal to the number of the negative eigenvalues, and select these negative eigenvalues and corresponding eigenvectors as the solution. As a consequence ,
,
,
0
Hence, the optimal objective kernel mapping we seek is ·
Φ ·
,
· ,
,
where , , are the eigenvectors corresponding to the negative eignvalues of matrix A. · is presented implicitly in the definition of However, because feature mapping A, it is impossible to construct matrix A and compute its eigenvectors 1, … , . In fact, to compute Φ · and , we do not need to construct A and compute its eigenvectors . 3.2 Computations of
· and
∈
Define vector vi RN by the k-th element of vi as and define matrix
,
,…,
,
0,
, we have
and +
Substitute Eq. (7) into the Eq. (5), we have
|
|
.
(7)
112
F. Wang and H. Zhang
+
·
.
(8)
So, 1
+ ,
.
(9)
where 1
+
.
Substitute Eq. (9) into the Eq. (8) and both sides of Eq. (8) are left multiplied , we have with +
where
,
(10)
is a Gram matrix corresponding to feature
. mapping Therefore, given a training set
,
,…,
Φ x ,Φ x ,…,Φ x
in input space, we get
,
,…,
α ,…,α
(11)
are s generalized ,… in the feature space after optimal embeddingΦ · , where eigenvectors of Eq. (10). So, the task of looking for qj has become the generalized eigenvalues problem of Eq. (10). As for a test point , we have Φ , ,
,…,
,
,
.
(12)
And Φ
,Φ ,
,…,Φ ,…,
Φ ,
,…,
,Φ
,…,Φ .
(13)
Optimal Double-Kernel Combination for Classification
113
As can be seen from equation (13), we are not able to compute before we know what matrix is. The next Section will discuss the construction of and computation of . and Computation of
3.3 The Construction of
is proposed just to form a unified framework in which three ODKC kernels can be discussed easily. In this section, we will see that ODKC kernels are easily constructed, corresponding to three kinds of definitions of . Let S be a finite set of N training data , , , , …, , kernels and correspond respectively to the feature mappings and , and and to these points, the corresponding kernel matrices obtained by restricting x . the kernel function corresponding to feature mapping is and chosen as a feature mapping corresponding to a valid and basic kernel such as Polynomial Kernel, Gauss Kernel or ANOVA kernel, different from , then: is defined by:
Definition 3.3.1: The non-linear mapping of ,
:
,
where [v1,v2]=(v11,v12,…v1m,v21,v22,…,v2n)T stands for the concatenation of vector v1 and vector v2. , is as follows: In this case, the computation of ,
, , ,
,
, +
+
, ,
,
So, the kernel matrix + . Figure 2 shows the mapping Φ · in the form of neural network where is defined by Definition 3.3.1. Data is mapped into two different feature spaces in layer-2 by and respectively, and then the objective feature space in layer-3 is created by feature selection from the two feature spaces. is defined by:
Definition 3.3.2: The non-linear mapping of
,
:
for i=1,…,N1, j=1,…,N2, where N1 is the dimension of x, and N2 is that of , is: , the computation of For this choice of ,
.
, ∑ ∑
∑ 1
1
0
,
0
,
∑
2
1
3
3
114
F. Wang and H. Zhang
∗
So, the kernel matrix
.
Figure 3 shows the mapping Φ · in the form of neural network where is defined by Definition 3.3.2. Data is mapped into two different feature spaces in layer-2 by and respectively, then another higher feature space in lay-3 is created by taking into account the correlations between attributions in the two feature spaces in layer-2. Finally, the objective feature space in layer-4 is created by feature selection from the feature space in layer-3. Definition 3.3.3: Let defined as: : So computation of
be chosen as a dot kernel , the non-linear mapping of . , is: ,
is
, , , ,
So, the kernel matrix
.
Figure 4 shows the mapping Φ · in the form of neural network where is defined by Definition 3.3.3. Data mapped by and then by form the feature space in layer-3. The objective feature space in layer-4 is created by feature selection from the feature space in layer-3. For the three options, because and are chosen as valid kernels, it is easy to prove that , is a valid kernel function. ·
φ01( xnew) x1new
φ02 ( xnew)
Q Φ1( xnew)
x
2 n ew
φ0N ( xnew) 1
φ31 ( xnew)
Φs ( xnew)
n xnew
φ3N ( xnew) 2
Fig. 2. Feature mappingΦ · embeds the data of the case of definition 3.3.1
from input space into a feature space in
Optimal Double-Kernel Combination for Classification
φ01( xnew) x
1 new
x
2 new
115
φ01 •φ31 Q
φ02 ( xnew )
Φ1 ( xnew )
φ •φ 1 0
φ0N ( xnew)
N2 3
1
φ31 ( xnew )
n new
φ0N •φ31 1
x
Φs ( xnew )
φ3N ( xnew)
φ0N •φ3N
2
Fig. 3. Feature mapping Φ · embeds the data of the case of definition 3.3.2
1 new
x
2 xnew
n new
x
φ01( xnew)
1
2
from input space into a feature space in
φ31(φ0(xnew))
Q
Φ1 ( xnew )
φ ( xnew) 2 0
Φs ( xnew )
φ0N ( xnew) 1
Fig. 4. Feature mapping Φ · embeds the data of the case of definition 3.3.3
φ3N (φ0(xnew)) 2
from input space into a feature space in
3.4 The Choice of Weighting Coefficients Weights in criteria function incur heavy penalties when the points Φ are close to other centers. The weighting coefficients are chosen as follows
116
F. Wang and H. Zhang
, ∑ where
is the squared Euclidean distance of the point computed in [13] as follow 2
,
|
,
|
from the i-th class centre +
1 |
is the expected squared distance of a point centre, computed in [13] as follow 1 |
,
|
+
,
|
, ;
in the i-th class from its 1 |
|
,
;
and function H(x) is Heaviside function that returns 1 if its argument is greater than 0 and zero otherwise.
4 The Algorithm of ODKC The algorithm of constructing ODKC kernel matrixes can be sum up as follows: Step 1 [Constructing matrixes] Using the label of training data, relative definitions and equations discussed above to construct matrixes below: ,
,
,
,
,
Step 2 [Computing generalized eigenvalues and generalized eigenvectors of Eq. (10)] Compute generalized eigenvalues and generalized eigenvectors of Eq. (10). Let s be the number of the negative eigenvalues and , … be the generalized eigenvectors of Eq. (10), ordered according to s negative eigenvalues: 0 ,Φ ,…,Φ andΦ Step 3 [Optimal feature mapping] Compute Φ Eq. (11) and Eq. (12). ODKC kernel matrix is computed by Eq (13).
using
Step 4 [classification using ODKC kernel matrix] Substitute ODKC kernel matrix for traditional kernel matrix in SVM, Fisher, MSE or KNN and perform these algorithms.
5 Experiments In this Section, we conduct two parts of experiments showing higher performance of ODKC kernels compared to basic kernel. In the first part of experiments, we compared the
Optimal Double-Kernel Combination for Classification
117
Alignment Measure of ODKC kernels with that of gauss basic kernel. In the second part, we conduct classification accuracies of SVM, KFD, KMSE, and k-nearest-neighbor (KNN), when using ODKC kernels and using gauss kernel. In the two parts of experiments, only Gaussian kernel and Polynomial kernel are considered to be used to construct ODKC kernels. In these Tables, symbol P stands for Polynomial kernel , + with with parameters (R, d), and g Gaussian kernel , / parameter . ODKC stands for ODKC kernel constructed by the third definition of ; ODKC stands for ODKC kernel constructed by the first definition of ; ODKC ∗ stands ODKC kernel constructed by the second definition of . stands for gauss kernel. The parameters of ODKC (R, d, and ) are fixed to 4, 2, and 33. And SVM with basic kernel gets its parameters by cross validation. Besides that, other necessary parameters are provided in parentheses in Tables. Five data sets are used in this paper, adopted from the UCI benchmark repository [14]. Basic information about these data sets is shown in table 1. 5.1 Experiments on Alignment Measure The “alignment” measure was introduced by Cristianini et al. [7] for measuring the adaptability of a kernel to the target data. It is defined as a normalized Frobenius inner product between the kernel matrix K and the target label matrix: , ,
,
where y denotes the label vector of the training data. The range of Alignment is [0, 1]. The larger its value is the closer the kernel to the target. Respectively, 80%, 50%, and 20% data was chosen from each class randomly to compute the values of Alignment Measure. Each experiment was repeated 15 times and the mean value is recorded in Table 2, 3, 4 and 5. As can be seen from these Tables, ODKC kernel matrixes have larger Alignment values than traditional ones, showing that ODKC kernel matrixes have much more adaptability to the target data. 5.2 Experiments on Classification Accuracies In this part, we conduct classification algorithms SVM, FDA, MSE, and KNN on the Ionosphere, Wisconsin Breast cancer, Pima Indians diabetes, and Sonar, using ODKC kernels compared with Gaussian kernel. Firstly, two-class problem is considered. Respectively, 80%, 50%, and 20% data was chosen randomly from each class for training and the rest for testing. Each experiment was repeated 15 times and the mean of value is recorded in Table 6, 7, 8 and 9. For multi-classification problem, Table 10 gives classification accuracies for KNN algorithm (k=1 and k=3), using Gaussian kernel and
118
F. Wang and H. Zhang Table 1. Information about the UCI data sets used in this paper
Number of Instances Number of Attributes Number of classes
Ionosphere 351 34 2
Sonar 208 60 2
Breast 699 9 2
Pima 768 8 2
Iris 150 4 3
Table 2. Alignment values in the case of Ionosphere dataset 80%training data ODKC ODKC ODKC
50%training data
0.233 0.180 0.310 0.089
∗
K 33
0.221 0.231 0.316 0.097
20% training data 0.233 0.061 0.331 0.070
Table 3. Alignment values in the case of Sonar dataset 80%training data ODKC ODKC ODKC
∗
K 33
0.124 0.053 0.230 0.035
50%training data 0.165 0.061 0.231 0.044
20%training data 0.267 0.054 0.227 0.049
Table 4. Alignment values in the case of Wisconsin Breast cancer dataset ODKC ODKC ODKC ∗ K 33
80% training data 0.192 0.191 0.155 0.100
50% training data 0.198 0.1478 0.153 0.100
20% training data 0.294 0.299 0.302 0.043
Table 5. Alignment values in the case of Wisconsin Pima Indians dataset ODKC ODKC ODKC ∗ K 33
80% training data 0.095 0.094 0.055 0.040
50% training data 0.092 0.078 0.053 0.051
20%training data 0.020 0.099 0.082 0.097
ODKC kernel on the Iris data set. Respectively, 80% and 60% data was chosen from each class randomly for training and the rest for testing. Each experiment was repeated 15 times and the mean of value is recorded in Table 10. In these Tables, Symbols ‘1’ and ‘3’ in tables stand for 1-nearest neighbor algorithm and 3-nearest neighbor algorithm, respectively. SVM + ODKC stands for linear SVM algorithm with ODKC kernel, FDA + K stands for FDA algorithm with Gaussian kernel, and so on. The results of experiments show the effectiveness of our method.
Optimal Double-Kernel Combination for Classification
119
Table 6. Comparison of classification accuracies (%) for training set and test set for various kernels in the case of the Ionosphere data 80% training data Training Set Test Set SVM + ODKC SVM + ODKC ∗ SVM + ODKC SVM + K FDA + ODKC FDA + ODKC ∗ FDA + ODKC FDA + K 33 MSE + ODKC MSE + ODKC ∗ MSE + ODKC MSE + K 33
95.80 97.43 96.22 94.81 98.30 99.50 99.20 99.29 98.92 97.50 97.20 97.14
95.60 94.89 94.72 93.52 94.95 94.54 94.36 92.95 94.36 93.41 93.21 92.95
50% training data Training Set Test Set 94.32 96.22 95.67 92.60 99.20 99.50 98.20 99.50 96.57 97.24 97.15 96.57
92.68 89.95 89.95 90.10 89.77 90.10 92.61 88.07 92.34 92.40 90.32 91.47
20% training data Training Set Test Set 98.20 98.54 97.52 99.12 98.56 99.12 99.20 99.20 98.02 97.77 98.67 98.02
83.35 86.21 85.70 70.81 85.65 79.81 84.94 83.98 77.82 79.02 80.56 80.21
Table 7. Comparison of classification accuracies (%) for training set and test set for various kernels in the case of the Sonar data 80%training data TrainingSet Test Set SVM + ODKC
100.00
SVM + ODKC
∗
SVM + ODKC
100.00
SVM + K
99.00
FDA + ODKC FDA + ODKC
100.00 ∗
FDA + ODKC
100.00 99.85
FDA + K 33
100.00
MSE + ODKC MSE + ODKC
100.00
100.00 ∗
MSE + ODKC MSE + K 33
100.00 100.00 100.00
96.6 5 96.7 7 96.7 0 95.5 0 95.8 7 96.0 0 95.0 5 92.9 0 97.0 0 97.3 0 97.0 0 90.0 1
50%training data TrainingSet Test Set
20% training data TrainingSet Test Set
99.00
89.28
100.00
82.28
99.00
91.17
100.00
81.77
99.00
91.11
100.00
80.96
99.00
85.65
100.00
80.87
100.00
93.03
100.00
83.00
100.00
94.00
100.00
84.65
98.56
94.73
99.12
84.33
100.00
88.07
100.00
83.98
100.00
97.00
100.00
87.88
100.00
96.20
100.00
89.10
100.00
96.00
100.00
89.07
100.00
92.32
100.00
88.56
5.3 Analysis of Experimental Results Compared with traditional methods, ODKC kernel is composed of two different basic kernels, which can take the advantages of various kernels to better adapt to different target datasets. As can be seen from experiments, the performance of classification algorithm with ODKC kernel is better than those with basic kernel. Speaking specifically, ODKC can provide significant improvement of the classification performance in the case of complicated datasets such as Ionosphere and Sonar datasets, and a limited
120
F. Wang and H. Zhang
improvement in the case of simple datasets like Wisconsin Breast cancer, Pima Indians diabetes and Iris, compared with traditional method. Table 8. Comparison of classification accuracies (%) for training set and test set for various kernels in the case of Wisconsin Breast cancer data 80%training data Training Set Test Set SVM + ODKC SVM + ODKC ∗ SVM + ODKC SVM + K FDA + ODKC SVM + ODKC ∗ SVM + ODKC FDA + K 33 MSE + ODKC MSE + ODKC ∗ MSE + ODKC MSE + K 33
96.87 96.53 95.55 96.47 99.89 99.53 99.12 98.58 99.87 99.84 99.84 98.58
50%training data Training Set Test Set
95.77 94.43 95.63 93.57 97.02 95.46 96.91 96.77 97.56 97.72 97.88 96.70
95.76 95.44 94.34 95.56 97.57 96.23 98.34 97.71 99.45 98.54 99.71 97.42
20% training data Training Set Test Set
91.55 92.66 93.62 91.91 95.77 94.66 96.52 95.98 98.43 96.62 97.42 95.98
97.22 95.55 95.25 96.53 99.02 98.15 99.53 100.00 98.23 96.50 98.58 98.58
85.70 87.65 88.67 85.40 87.77 89.56 88.78 86.07 96.56 92.54 95.53 86.25
Table 9. Comparison of classification accuracies (%) for training set and test set for test data in the case of Wisconsin Pima Indians data 80%training data Training Set Test Set SVM + ODKC SVM + ODKC ∗ SVM + ODKC SVM + K FDA + ODKC FDA + ODKC ∗ FDA + ODKC FDA + K 33 MSE + ODKC MSE + ODKC ∗ MSE + ODKC MSE + K 33
85.45 87.56 86.12 85.62 86.46 85.45 88.76 83.16 90.32 88.54 89.66 89.43
50%training data Training Set Test Set
84.63 81.75 81.22 78.15 81.65 84.34 82.13 80.04 83.65 84.76 83.12 80.60
84.32 83.64 84.52 82.68 89.74 85.53 89.51 83.85 88.33 87.75 89.73 83.54
20%training data TrainingSet Test Set
76.88 77.87 77.60 76.82 77.34 77.54 77.41 76.56 79.45 82.33 79.14 78.64
83.57 84.27 83.66 84.67 91.03 90.45 90.02 83.65 85.45 86.65 88.11 83.58
71.47 73.54 70.50 72.64 77.76 76.65 75.12 74.95 77.07 76.66 75.04 76.50
Table 10. Comparison of classification accuracies (%) for KNN algorithm for various kernels in the case of Iris data in multi-class problem Testing Set (%) 20% 40%
1
96.00 94.65
∗
95.65 93.54
1
1
96.00 93.77
3
+
96.12 95.13
3
∗
94.55 96.45
3
96.12 95.13
KNN 1
95.12 93.69
KNN3
95.67 94.53
Comparing each form of kernel ODKC defined in Section 3.3, algorithm using and for classification kernel ∗ are better than that with kernel accuracies on Ionosphere and Sonar. The reason is for high-dimension data sets like Ionosphere and Sonar, there is lots of correlation information between attributions useful for classification, where kernel ∗ uses the information well. However, cannot perform better than and on some kernel ∗
Optimal Double-Kernel Combination for Classification
121
low-dimension data sets like Wisconsin Breast cancer and Pima Indians diabetes and , which contain little correlation information. As for with deeper architecture can learn more complex decision function than , while depends on the good linear-separability of data in the the good performance of feature space of lagyer-2, such as Iris data set.
6 Conclusion In this paper, three kinds of composite kernels are constructed under a unified framework by different definition of . We also compared classification algorithms when using traditional kernel and these three kinds of ODKC kernels on five datasets. Finally, we gave analysis of experimental results. Only double-kernel combination is considered in this paper. Actually, it is easy to promote the case of double-kernel to multi-kernel according to Section 3.2. So, another attempt in the future is to analysis and study advantages and disadvantages of optimal multi-kernel combinations. Acknowledgments. The work on this paper was supported in part by the National Natural Science Foundation of China (NSFC) under grant 60775011.
References 1. Cristianini, N., Shawe-Taylor, J.: An introduction to support vector machines. Cambridge Univ. Press, Cambridge (2000) 2. Burges, C.J.C., Scholkopf, B.: Improving the accuracy and speed of support vector learning machines. In: Mozer, M., Jordan, M., Pestsche, T. (eds.) Advance in neural information processing systems, vol. 9, pp. 375–381. MIT Press, Cambridge (1997) 3. Baudat, G., Anousr, F.: Generalized discriminant analysis using a kernel approach. Neural Computation 12(10), 2385–2404 (2000) 4. Ruiz, A., Lopez-de Teruel, P.E.: Nonlinear kernel-based statistical pattern analysis. IEEE Trans. Neural Network 12(1), 16–32 (2001) 5. Lanckriet, G.R.G., Cristianini, N., Bartlett, P., Ghaoui, L.E., Jordan, M.I.: Learning the Kernel Matrix with Semidefinite Programming. J. Machine Learning Research 5, 27–72 (2004) 6. Bach, F., Lanckriet, G.R.G., Jordan, M.I.: Multiple Kernel Learning, Conic Duality, and the SMO Algorithm. In: Proc. 21st Int’l Conf. Machine Learning (2004) 7. Cristianini, N., Shawe-Taylor, J., Elisseeff, A., Kandola, J.: On kernel-target alignment. In: Advance in neural information processing systems. MIT Press, Cambridge (2001) 8. Nguye, C.H., Ho, T.B.: An efficient kernel matrix evaluation measure. Pattern Recognition 41(11), 3366–3372 (2008) 9. Xiong, H., Swamy, M.N.S.: Optimizing the kernel in the empirical feature space. IEEE transactions on neural networks 16(2), 460–474 (2005) 10. Chen, B., Liu, H., Bao, Z.: Optimizing the data-dependent kernel under unified kernel optimization framework. Pattern recognition 41(6), 2107–2119 (2007) 11. Bottou, L., Chapelle, O., DeCoste, D., Weston, J.: Large-Scale Kernel Machines. MIT Press, Cambridge (2007)
122
F. Wang and H. Zhang
12. Lee, W.-J., Verzakov, S., Duin, R.P.W.: Kernel Combination Versus Classifier Combination. In: Haindl, M., Kittler, J., Roli, F. (eds.) MCS 2007. LNCS, vol. 4472, pp. 22–31. Springer, Heidelberg (2007) 13. Shawe-Taylor, J., Cristianini, N.: Kernel methods for pattern analysis. Cambridge University Press, Cambridge (2004) 14. Blake, C., Keogh, E., Merz, C.J.: UCI Repository of Machine Learning Databases (1998), http://www.ics.uci.edu/mlearn
Efficient AdaBoost Region Classification M. Moed and E.N. Smirnov Department of Knowledge Engineering, Maastricht University, The Netherlands
[email protected],
[email protected]
Abstract. The task of region classification is to construct class regions containing the correct classes of the objects being classified with an error probability ε ∈ [0, 1]. To turn a point classifier into a region classifier, the conformal framework is employed [11,14]. However, to apply the framework we need to design a non-conformity function. This function has to estimate the instance’s non-conformity for the point classifier used. This paper introduces a new non-conformity function for AdaBoost. The function has two main advantages over the only existing non-conformity function for AdaBoost. First, it reduces the time complexity of computing class regions with a factor equal to the size of the training data. Second, it results in statistically better class regions.
1 Introduction Most of the research in data mining focuses on point classifiers: classifiers capable of estimating the correct class of an object being classified [2,5]. However, in critical applications with high misclassification costs, region classifiers are needed. We define region classifiers as those classifiers that are capable of providing a region of classes that contains the correct class of the object being classified with a probability of error ε ∈ [0, 1]. Point classifiers can be turned into region classifiers using the conformal framework [11,14]. This framework is valid if the data are drawn from the same unknown exchangeability distribution and if the learning process is online. Given an object space X, a class set Y, training data D over X × Y, and an object x ∈ X to be classified, the framework first computes for each class y ∈ Y the non-conformity scores of the instances in D and the labeled instance (x, y). The non-conformity score for an instance estimates how unusual this instance is in D ∪ {(x, y)} relative to the other instances in D ∪{(x, y)}. The p-value py of class y for the object x is computed as the proportion of instances in D ∪ {(x, y)} of which the non-conformity scores are greater than or equal to that of the instance (x, y). Hence, the class y is added to the final class region for the object x if py ≥ ε. The main problem in applying the conformal framework for a point classifier is to design a non-conformity function. This non-conformity function has to estimate the instance’s non-conformity scores for that classifier. So far, non-conformity functions have been proposed for support vector machines, k-nearest neighbor classifiers, linear-discriminant classifiers, naive Bayes classifier, boosting classifiers, and kernel perceptrons [12,13,14]. P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 123–136, 2009. © Springer-Verlag Berlin Heidelberg 2009
124
M. Moed and E.N. Smirnov
This paper focuses on the problem of efficiently applying the conformal framework for AdaBoost point classifiers [10]. We first consider the only existing non-conformity function for AdaBoost proposed in [8,14], called the general non-conformity function. We show that if we use this function, the time complexity to construct one class region is very high: it is equal to O(|Y||D|Tab ) where Tab is the time complexity of the AdaBoost algorithm. To reduce this complexity we propose a new non-conformity function for AdaBoost,+ that is based on the weights of the training instances calculated by the AdaBoost algorithm. Since the weights indicate the classification “difficulty” of the instances, they are used as non-conformity values. Thus, our function returns for each training instance the weight of this instance. This results in time complexity to construct one class region equal to O(|Y|Tab ), i.e. we reduce the time complexity with a factor |D|. In addition, we analyze the class regions based on our weights-based non-conformity function. We show that they are statistically better than the class regions based on the general non-conformity function if the AdaBoost point classifiers are neither underfitted nor overfitted. The remaining of the paper is organized as follows. Section 2 formalizes the tasks of point classification and region classification. The conformal framework and AdaBoost are introduced in Sections 3 and 4, respectively. Section 5 considers the general nonconformity function and weights-based non-conformity function. The experiments are presented and discussed in Section 6. Finally, Section 7 concludes the paper.
2 Point and Region Classification Let X be an object space and Y a class set. The instance space Z over X and Y is defined as X × Y. The training data Dn is a bag z1 , z2 , ..., zn of n instances zi ∈ Z drawn from the same unknown probability distribution Q. Given training data Dn and object xn+1 ∈ X to be classified, the point-classification task is to provide an estimate yˆ of the real class yr ∈ Y of the object xn+1 . To provide such a class estimation for any object x ∈ X we identify a point classifier h(Dn , x) in a space H of point classifiers h (h : Z(∗) × X → 2R )1 . We assume that h(Dn , x) outputs for x a posterior distribution of scores {sy }y∈Y over all the classes in Y. The class y with the highest posterior score sy is the estimated class yˆ for x. The region-classification task is different from the point-classification task. Formally, given significance level ε ∈ [0, 1], training data Dn , and object xn+1 ∈ X to be classified, the region-classification task is to find class region Γ ε (Dn , xn+1 ) ⊆ Y that contains the correct class y ∈ Y of xn+1 with probability at least 1 − ε 2 . In order to do so, first a posterior distribution of probabilities {py }y∈Y over Y is generated by normalizing the scores in the posterior distribution {sy }y∈Y . Then, the class region Γ ε (Dn , x) ⊆ Y is defined as a set consisting of those classes y ∈ Y of which the posterior probabilities py are greater than or equal to ε. Since in general the posterior probabilities of point classifiers are not calibrated [15], this approach fails to produce correct class regions. 1 2
Z(∗) denotes the set of all bags defined in Z. A naive approach to the region-classification task is to employ a point classifier h(Dn , x).
Efficient AdaBoost Region Classification
125
In the next section we briefly sketch a framework for constructing classifiers for the region-classification task. For short they will be called region classifiers.
3 The Conformal Framework This section briefly sketch the conformal framework for constructing region classifiers [11,14]. Subsection 3.1 describes formally the framework. Then, Subsection 3.2 discusses options when the framework is being applied. Finally, Subsection 3.3 introduces metrics for evaluating region classifiers based on the conformal framework. 3.1 Formal Description The conformal framework was proposed for turning point classifiers into region classifiers [11,14]. The framework is proven to be valid when the training data and objects to be classified are drawn from the same unknown distribution under the exchangeability assumption. The exchangeability assumption holds when different orderings of training instances are equally likely. Applying the conformal framework is a two-stage process. Given a point classifier h(Dn , x), we first construct a non-conformity function for h(Dn , x) capable of measuring how unusual an instance looks relative to the other instances in the data. Then, we apply the conformal algorithm that employs the non-conformity function to compute the class regions. Formally, the non-conformity function is a function of type A : Z(∗) ×Z → R∪{∞}. Given a bag Dn ∈ Z(∗) of training instances and instance z ∈ Z, it returns a value α in the range of [0, R ∪ {∞}] indicating how unusual the instance z with respect to the instances in Dn . In general, the function A returns different scores for instance z depending on whether z is in the bag Dn (added prediction) or not (deleted prediction): if z ∈ Dn , then the score is lower; otherwise it is higher. Since there is no consensus on this issue [14], care must be taken when choosing between added or deleted prediction. The general non-conformity function was defined in [14] for any point classifier (∗) h(Dn , x). Given a bag Dn ∈ Z of training instances and instance z equal to (x, yr ), the function outputs the sum y∈Y,y =yr sy where sy is the score for class y ∈ Y produced by h(Dn , x). The conformal algorithm is presented in Figure 1. Given significance level ε ∈ [0, 1], training data Dn of n instances, object xn+1 ∈ X to be classified, and the nonconformity function A for a point classifier h(Dn , x), the algorithm constructs class region Γ ε (Dn , xn+1 ) ⊆ Y for the object xn+1 . To decide whether to include a class y ∈ Y in the class region Γ ε (Dn , xn+1 ) the object xn+1 and class y are first combined into instance zn+1 . The instance zn+1 is added to the training data which, resulting in a data set of n + 1 instances denoted by Dn+1 . Then, the algorithm computes the nonconformity score αi for each instance zi ∈ Dn+1 using the non-conformity function A 3 . The non-conformity scores are used for computing the p-value py of the class y for the object xn+1 . More precisely, py is computed as the proportion of instances in 3
We note that depending on the option chosen (added/deleted prediction), the instance zi is included in or excluded from the data used when the non-conformity function A is called.
126
M. Moed and E.N. Smirnov
Significance level , Training data Dn equal to the bag z1 , z2 , ..., zn , Object xn+1 to be classified, Non-conformity function A for a point classifier h(Dn , x). Output: Class region Γ (Dn , xn+1 ). 1: Γ (Dn , xn+1 ) := ∅; 2: for each class y ∈ Y do 3: zn+1 := (xn+1 , y); 4: Dn+1 := Dn zn+1 ; 5: for i := 1 to n + 1 do 6: if using deleted prediction then 7: αi := A(Dn+1 \ zi , zi ); 8: else if using added prediction then 9: αi := A(Dn+1 , zi ); 10: end if 11: end for #{i=1,...,n+1|αi ≥αn+1 } 12: py := ; n+1 13: Include y in Γ (Dn , xn+1 ) if py > ; 14: end for 15: Output Γ (Dn , xn+1 ). Input:
Fig. 1. The conformal algorithm
the training data Dn+1 of which the non-conformity scores αi are greater or equal to that of the instance zn+1 . Once py is set, the algorithm includes the class y in the class region Γ ε (Dn , xn+1 ) if py ≥ ε. The conformal algorithm was originally designed for the online learning setting. This setting assumes an initially empty data set Dn . Then for each integer n from 0 to +∞ we first construct class region Γ ε (Dn , xn+1 ) for the new object xn+1 being classified, and then add the instance (xn+1 , yr ) to Dn where yr is the correct class of xn+1 . In this context we note that the conformal algorithm is proven to be valid [11,14] (i.e. it constructs for any object xn+1 a class region Γ ε (Dn , xn+1 ) ⊆ Y containing the correct class y ∈ Y for xn+1 with a probability of at least 1 − ε) if: – the data are drawn from the same unknown distribution under the exchangeability assumption; – the learning setting is online. 3.2 Possible Options To apply the conformal framework we have to make a set of choices concerning the non-conformity function used and the learning setting. The conformal algorithm outputs valid class regions for any real-valued function used as non-conformity function [11]. However, the class region will be efficient (small) if the function estimates well the difference of any instance with respect to the training data. In this context we note that the general non-conformity function is not always the
Efficient AdaBoost Region Classification
127
most efficient. Therefore, one of the main issues when applying the conformal framework is how to design specific a non-conformity function for the point classifier used. As stated above, the conformal algorithm is valid when the learning setting is online [11]. However, it was shown experimentally in [12,13] that the algorithm produces valid class regions in the offline (batch) learning setting4 . Thus, the conformal algorithm can be used either in the online setting or in the offline setting. In this paper we apply the algorithm in the offline setting only. 3.3 Evaluation Metrics Any class region Γ ε (B, xn+1 ) is valid if it contains the correct class y ∈ Y of the object xn+1 ∈ X being classified with probability of at least 1 − ε. To evaluate experimentally the validity of the class regions provided by the conformal algorithm we introduce the error metric E. E is defined as the proportion of the class regions that do not contain the correct class5 . Thus, in order to prove experimentally that the conformal algorithm is valid we have to show that for any significance level ε ∈ [0, 1] the error E is less than or equal to ε. Any class region Γ ε (B, xn+1 ) is efficient if it is non-empty and small. Thus, to evaluate experimentally the efficiency of the class regions provided by the conformal algorithm, we introduce three metrics: the percentage Pe of empty-class regions, the percentage Ps of single-class regions, and the percentage Pm of multiple-class regions. The empty-class regions, single-class regions, and multiple-class regions can be characterized by their own errors. The percentage Pe of empty-class regions is essentially an error, since the correct classes are not in the class regions. The error Es on single-class regions is defined as the proportion of the invalid single-class regions among all class regions. The error Em on multiple-class regions is defined as the proportion of invalid multiple-class regions among all the class regions. The errors Pe , Es , and Em are components of the error E. More precisely, it is easy to prove that E = Pe + Es + Em . The error E has its own upper bound E u representing the worst-case case when we are not able to pick up correct classes from valid multiclass regions. In this case we will err on all the multi-class regions and, thus, E u is defined equal to Pe + Es + Pm . We note that for any significance level ε ∈ [0, 1] there is no guarantee that E u is less than or equal to ε unless Pm = 0.
4 Boosting Boosting is a general approach to constructing point classifiers from weak point classifiers (defined as just better-than-random classifiers) [10]. Boosting operates by sequentially training the weak point classifiers on re-weighted versions of the training data. The final point classifiers are constructed as weighted-majority voters of the resulting weak classifiers. 4
5
The offline learning setting is the traditional setting in machine learning. In this setting the classified instances are not added to the training data. We note that the computing the evaluation metrics introduced in this subsection depends on the learning setting use (online/offline).
128
M. Moed and E.N. Smirnov
Training data Dn equal to the bag z1 , z2 , . . . , zn , Type h(Dn , x) of weak point classifier, Number of iterations T . Output: AdaBoost point classifier hT (x). 1: for i := 1 to n do 2: wi = n1 ; 3: end for 4: for t := 1 to T do 5: Train weak classifier ht (Dn , x) on Dn using weights wi ; 6: Calculate the error of ht (Dn , x) on Dn using weights wi ; 7: if = 0 or ≥ 12 then 8: T := T − 1; 9: Abort loop 10: end if 11: βt = − log2 ( ); 1− 12: for i := 1 to n do 13: if ht (xi ) = yi then 14: wi := wi ∗ 1− ; 15: end if 16: end for 17: Normalize the weights wi ; 18: end for 19: Output Adaboost classifier hT (Dn , x) equal to the βt -weighted sum of the classifiers ht (Dn , x). Input:
Fig. 2. The AdaBoost algorithm
Adaboost (Adaptive Boosting) is the first efficient boosting algorithm [4]. Due to its straightforward implementation and good generalization performance, it is one of the most widely adopted boosting algorithms to date. The Adaboost algorithm is given in Figure 2. The input is formed by: training data Dn with n instances, type h(Dn , x) of a weak point classifier, and iteration number T . The output consists of a AdaBoost point classifier hT (B, x) based on weak classifiers of type h(Dn , x). The Adaboost algorithm considers the training data Dn as weighted data. This means that each instance zi ∈ Dn is associated with a weight wi indicating the classification “difficulty” of the instance. Since no classification information is provided a priori, the weights wi are initialized uniformly. The Adaboost algorithm itself is iterative. For each iteration t ∈ 1 . . . T , AdaBoost first trains a weak point classifier ht (Dn , x) on the weighted training data Dn . Then, the algorithm calculates the weighted error of ht (Dn , x). If = 0 or ≥ 12 , the current point classifier ht (Dn , x) is viewed as bad. In this case the algorithm deletes ht (Dn , x) and does not perform any more iterations. If = 0 and < 12 , the algorithm computes the weight βt of the weak point classifier ht (Dn , x) equal to − log2 ( ). Then, it re-computes the weights wi of the train1− ing instances so that the weights of the correctly classified instances are decreased by factor 1− . The final weights wi are normalized so that they indicate the classification
Efficient AdaBoost Region Classification
129
“difficulty” of the instances in the training data Dn for the weak point classifiers from the previous t iterations. When all T weak point classifiers have been generated, the algorithm outputs the Adaboost point classifier hT (Dn , x). It is equal to the sum of the weak point classifiers ht (Dn , x) weighted according to their weights βt . In other words: given an object x ∈ X to be classified the classifier hT (Dn , x) outputs score sy for each class y ∈ Y T equal to t=1 βt st (y) where st (y) is the score for object x and class y produced by the t-th weak point classifier with weight βt . The final class scores of the classifier hT (Dn , x) can be normalized to produce distribution of probabilities {py }y∈Y over all the classes in Y. However, these probabilities are not calibrated [7]; i.e. the AdaBoost point classifier alone cannot be used for region classification.
5 AdaBoost Non-conformity Functions As stated in Section 4, the AdaBoost classifiers are point classifiers. To turn these classifiers into region classifiers we need first to construct a non-conformity function, and then apply the conformal algorithm. In this section we focus on non-conformity functions for the AdaBoost classifiers. We first analyze the non-conformity function for AdaBoost proposed in [14]. We show that the time complexity of the conformal algorithm based on this function is very high. To avoid this problem we then propose our new non-conformity function for AdaBoost. We show that the time complexity of the conformal algorithm based on our function is reduced with a factor equal to the number of training instances. Later in the experimental section, we show that our non-conformity function results in better class regions. 5.1 The General Non-conformity Function The first non-conformity function proposed for AdaBoost is the general non-conformity function [8,14]. Given a bag Dn ∈ Z (∗) of n training instances and instance z equal to (x, yr ), the non-conformity function equals A(Dn , z) = y∈Y,y =yr sy , where sy is the score for class y ∈ Y produced by the AdaBoost point classifier hT (B, x). In this context we point out one property of hT (B, x): if the instance z = (x, yr ) is included in the training data Dn , then the score syr increases with the number of the AdaBoost iterations while the scores sy for the remaining classes y ∈ Y \ {yr } decrease. This implies that the function tends to produce non-conformity values close to 0; i.e. it cannot distinguish well the non-conformity of the instances. Thus, the general non-conformity function is applied for AdaBoost only if the instance z = (x, yr ) for which the nonconformity score is being calculated is deleted from the training data Dn . This means that when the conformal algorithm has to be used with the general non-conformity function in combination with AdaBoost, deleted prediction has to be used (see Figure 1). We note that in this case, computing p-value py for one class y ∈ Y requires |Dn | runs of the AdaBoost algorithm. Thus, the time complexity for constructing one class region becomes O(|Y||Dn |Tab ), where Tab is the time complexity of the AdaBoost algorithm.
130
M. Moed and E.N. Smirnov
5.2 Weights-Based Non-conformity Function To reduce the time complexity for constructing one class region, we propose a new non-conformity function for AdaBoost. This new function is called the weights-based non-conformity function. As the name suggests, it is based on the weights of the training instances calculated by the AdaBoost algorithm. Since they indicate the classification “difficulty” of the instances, we interpret them as non-conformity values. Formally, the weights-based non-conformity function is defined as follows: given a bag Dn of training instances and instance z equal to (x, yr ) so that z ∈ Dn , the function returns the weight wT of (x, yr ) calculated by the AdaBoost algorithm after T iterations, i.e. A(Dn , z) = wT . We note that since the AdaBoost algorithm computes weights only for training instances, the instance z has to belong to the data set Dn . This implies that when using the weights-based non-conformity function, added prediction has to be used (see Figure 1). We note that in this case computing p-value py for one class y ∈ Y requires only one run of the AdaBoost algorithm. Thus, the time complexity for constructing one class region becomes O(|Y|Tab ), where Tab is the time complexity of the AdaBoost algorithm. Comparing the time complexities of the conformal algorithm for the general nonconformity function and for the weights-based non-conformity function, we conclude that the latter is |Dn | times faster than the former. To complete our study, we investigate the validity and efficiency of the class regions based on these two functions in Section 6.
6 Experiments and Discussion This section presents our experiments with the conformal algorithm. The algorithm was tested for the general non-conformity function and for the weights-based nonconformity function for AdaBoost. The performance of the algorithm for these two functions are expressed in terms of the validity and efficiency of the class regions. We carried out two series of experiments. In the first one we experimentally compared the performance of the conformal algorithm for the general non-conformity function and for the weights-based non-conformity function. In the second series we investigated the influence of the model complexity of the AdaBoost point classifiers on the class regions based on these two functions. 6.1 Setup The experiments were performed with the conformal algorithm in combination with AdaBoost. As a base classifier, AdaBoost employs the decision stump classifier. The class regions of the conformal algorithm were evaluated using five metrics (defined in Subsection 3.3): the error E, the upper-bound error E u , the percentage Pe of emptyclass regions, the percentage Ps of single-class regions, and the percentage Pm of multiple-class regions. The method of evaluation was 10-fold cross validation. To compare the metrics that correspond to the non-conformity functions investigated we used a paired-t test on significance level 0.05.
Efficient AdaBoost Region Classification
131
6.2 Non-conformity Functions Comparison The conformal algorithm for the general and weights-based non-conformity functions was run on ten UCI data sets [1]. For each data set, the number T of AdaBoost iterations was set equal to the number Tmax of iterations for which the 10-fold cross-validation accuracy of the AdaBoost classifier is maximized. The results are given in Table 1 given below. The table presents the validity and efficiency of the class regions of the conformal algorithm for the general and weights-based non-conformity functions. Both functions have been evaluated for probabilities of error ε = 0.1 and ε = 0.05. Table 1 shows that the class regions computed by the conformal algorithm for both non-conformity functions are valid. This is due to the fact that the error E is always lower than or equal to the probability of error ε. In addition, we observe that the error E is higher for the weights-based non-conformity function. This is due to the fact that for this function: (1) the percentage Ps of single-class regions is higher (8 times significantly) and the percentage Pm of multiple-class regions is lower (8 times significantly). In other words: the weights-based non-conformity function has higher error E because we have much more desired single class regions than undesired multiple class regions. This explains why the upper-bound error E u is always lower for the weightsbased non-conformity function (8 times significantly). From this we conclude that if the AdaBoost classifiers are neither underfitted nor overfitted6, the class regions based on the weights-based non-conformity function are statistically better than those of the general non-conformity function in terms of the percentage Ps of single-class regions, the percentage Pm of multiple-class regions, and the upper-bound error E u .
6.3 AdaBoost Complexity Influence The AdaBoost number T of iterations controls the complexity of the final point classifier hT (B, x). In this subsection we investigate the influence of this complexity on the class regions based on the general and weights-based non-conformity functions. We provide the experimental results for one data set for three values of the iteration number T : 1, Tmax , and 10 × Tmax 7 . These values bound the whole spectrum of complexity of the AdaBoost classifiers. When T = 1 the AdaBoost classifier is underfitted; i.e. it is rather simple since it consists of only one decision stump classifier. When T = Tmax the AdaBoost classifier is optimized - the 10-fold cross validation accuracy is maximized. When T = 10 × Tmax the AdaBoost classifier is overfitted; i.e. it is rather complex. Table 2 and Figure 3 present the results for the iteration number T equal to 1, Tmax , and 10 × Tmax for the hepatitis training data. The observations and explanations are as follows: 6
7
In our experiments the number of AdaBoost iterations maximizes the 10-fold cross-validation accuracy of the AdaBoost classifier. We performed the same experiment on the remaining 9 UCI data sets. Since the results are similar, due to the length restriction we consider in detail only the experiment for the hepatitis data set.
132
M. Moed and E.N. Smirnov
Table 1. Paired t-test performance statistics of the class regions of the conformal algorithm based on the weights-based non-conformity function (Weights) and the general non-conformity function (General). The symbol ‘*’ means that the result is statistically better.
Weight breast-cancer General Weight colic General Weight heart-c General Weight heart-h General Weight heart-statlog General Weight hepatitis General Weight iris General Weight labor General Weight lymph General Weight sonar General
E 0.04 0.06 0.09 0.02∗ 0.08 0.01∗ 0.09 0.00∗ 0.09 0.01∗ 0.09 0.04 0.04 0.00 0.03 0.00 0.04 0.07 0.08 0.00∗
Pe 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.03 0.00 0.00 0.00 0.02 0.00
ε = 0.1 Ps Pm 0.23 0.77 0.23 0.77 0.72∗ 0.28∗ 0.19 0.81 0.74∗ 0.26∗ 0.22 0.78 0.75∗ 0.25∗ 0.22 0.78 0.72∗ 0.28∗ 0.21 0.79 0.76∗ 0.24∗ 0.23 0.77 0.89∗ 0.09∗ 0.31 0.69 0.76∗ 0.21∗ 0.32 0.68 0.07 0.93 0.23 0.77 0.70∗ 0.27∗ 0.23 0.77
Eu 0.81 0.83 0.37∗ 0.83 0.34∗ 0.79 0.33∗ 0.78 0.37∗ 0.79 0.33∗ 0.82 0.11∗ 0.69 0.21∗ 0.68 0.94 0.81 0.33∗ 0.77
E 0.03 0.04 0.05 0.01∗ 0.05 0.00∗ 0.05 0.00∗ 0.04 0.01∗ 0.05 0.03 0.01 0.00 0.02 0.00 0.02 0.05 0.05 0.00
ε = 0.05 Pe Ps Pm 0.00 0.16 0.84 0.00 0.18 0.82 0.00 0.51∗ 0.49∗ 0.00 0.14 0.86 0.00 0.59∗ 0.41∗ 0.00 0.16 0.84 0.00 0.68∗ 0.32∗ 0.00 0.17 0.83 0.00 0.56∗ 0.44∗ 0.00 0.16 0.84 0.00 0.61∗ 0.39∗ 0.00 0.16 0.84 0.00 0.87∗ 0.13∗ 0.00 0.31 0.69 0.02 0.69∗ 0.30∗ 0.00 0.24 0.76 0.00 0.03 0.97 0.00 0.21∗ 0.79∗ 0.01 0.62∗ 0.37∗ 0.00 0.18 0.82
Eu 0.86 0.86 0.54∗ 0.86 0.46∗ 0.84 0.37∗ 0.83 0.49∗ 0.85 0.44∗ 0.87 0.14∗ 0.69 0.30∗ 0.76 0.98 0.83∗ 0.41∗ 0.82
Table 2. The hepatitis data set: paired t-test performance statistics of the class regions of the conformal algorithm based on the weights-based non-conformity function (Weights) and the general non-conformity function (General) for the number T of AdaBoost iterations equal to 1, Tmax , and 10 × Tmax . The symbol ‘*’ means that the result is statistically better.
T =1 T = Tmax T = 10 × Tmax
Weights General Weights General Weights General
E 0.00 0.08 0.09 0.04 0.06 0.10
Pe 0.00 0.00 0.00 0.00 0.00 0.00
= 0.1 Ps Pm 0.01 0.99 0.60∗ 0.40∗ 0.76∗ 0.24∗ 0.23 0.77 0.42 0.58 0.76∗ 0.24∗
Eu 0.99 0.48∗ 0.33∗ 0.82 0.64 0.33∗
E 0.00 0.05 0.05 0.03 0.03 0.06
Pe 0.00 0.00 0.00 0.00 0.00 0.00
= 0.05 Ps Pm 0.01 0.99 0.48∗ 0.52∗ 0.61∗ 0.39∗ 0.16 0.84 0.15 0.85 0.54∗ 0.46∗
Eu 0.99 0.57∗ 0.44∗ 0.87 0.88 0.51∗
(1) when T = 1 the error E is lower for the weights-based non-conformity function. This is due to the fact that: (1) the percentage Ps of single-class regions is lower and (2) the percentage Pm of multiple-class regions is higher. Thus, the upper-bound
Efficient AdaBoost Region Classification
133
error E u is higher for the weights-based non-conformity function. We explain these results with the AdaBoost weights. When T = 1 the instance weights take on only two possible values: one value for correctly classified instances and one value for incorrectly classified instances. Thus, the weights-based non-conformity function returns implausible nonconformity values. This contrasts with the general non-conformity function which, for T = 1, employs the class scores of the base classifier. Since the scores of the base classifier used (decision stump) are considered reasonable [6], the general non-conformity function outputs better non-conformity values. (2) when T = Tmax the error E is higher for the weights-based non-conformity function. This is due to: (1) the percentage Ps of single-class regions is higher and (2) the percentage Pm of multiple-class regions is lower. Thus, the upper-bound error E u is lower for the weights-based non-conformity function. We explain these good results with the AdaBoost weights. When the AdaBoost point classifier is neither underfitted nor overfitted, the weights plausibly 3 represent the classification difficulty of the training instances [3] that we consider as non-conformity values. This contrasts with the general non-conformity function which, for T = Tmax , employs the class scores of the AdaBoost classifier. Since the class scores of the AdaBoost point classifier are known to be uncalibrated [7], this results in relatively worse non-conformity values. (3) when T = 10 × Tmax the error E is lower for the weights-based non-conformity function. This is due to the fact that: (1) the percentage Ps of single-class regions is lower and (2) the percentage Pm of multiple-class regions is higher. Thus, the upper-bound error E u is higher for the weights-based non-conformity function. These results are also due to the AdaBoost algorithm. When T = 10 × Tmax this algorithm concentrates too much on training instances that were consistently classified incorrectly. As a result these instances receive higher weights while the remaining instances receive lower weights. When the iteration number T receives a very high number, most of the training instances receive a weight of 0 (this is demonstrated experimentally for the hepatitis data set in Figure 4, presenting the weight distributions for T = 10 and T = 10.000). Thus, the weights-based non-conformity function returns implausible non-conformity values. This contrasts with the general non-conformity function which, for a high number of iterations T , employs the class scores of the AdaBoost point classifier. These class scores result in better non-conformity values. From (1), (2), and (3) we conclude again that the class regions based on the weightsbased non-conformity function are statistically better than those of the general non-conformity function when the accuracy of the AdaBoost point classifier used is maximized; i.e. the classifier is neither underfitted nor overfitted. In this context we note that when the AdaBoost point classifier is either underfitted or overfitted, the general non-conformity function results in better class regions, but they are still worse than those of the weights-based non-conformity function when the AdaBoost point classifier has maximum accuracy.
134
M. Moed and E.N. Smirnov
T = 1 (general)
T = 1 (weights−based)
1
1 Pm
Pm
E Ps
0.8
E Ps
0.8
Pe Performance
Performance
Pe 0.6
0.4
0.2
0 0
0.6
0.4
0.2
0.2
0.4 0.6 Significance level ε
0.8
0 0
1
0.2
T = Tmax (general)
0.4 0.6 Significance level ε
0.8
T = Tmax (weights−based)
1
1 Pm
Pm
E Ps
0.8
E Ps
0.8
P
P
e
Performance
Performance
e
0.6
0.4
0.2
0 0
0.6
0.4
0.2
0.2
0.4 0.6 Significance level ε T = 10 × T
max
0.8
0 0
1
0.2
0.4 0.6 Significance level ε
T = 10 × T
(general)
max
1
0.8
0.8
= 200 (weights−based)
P
P
E P
E P
m
0.8
s
s
Pe Performance
Pe Performance
1
1 m
0.6
0.4
0.2
0 0
1
0.6
0.4
0.2
0.2
0.4 0.6 Significance level ε
0.8
1
0 0
0.2
0.4 0.6 Significance level ε
0.8
1
Fig. 3. Performance graphs of the conformal algorithm based on the general non-conformity function and the weights-based non-conformity function on the hepatitis data set. Note that the diagonal line (0.0,0.0)-(1.0,1.0) is the calibration line. Ideally, the error E should be smaller than or equal to this calibration line for all ε ∈ [0, 1].
Efficient AdaBoost Region Classification
135
Instance weights over several iterations 0.08
10.000 iterations 10 iterations
0.07 0.06
Weight
0.05 0.04 0.03 0.02 0.01 0
20
40
60 80 Instance number
100
120
140
Fig. 4. Weight distribution for a single run of the AdaBoost algorithm on the hepatitis data set for 10 and 10.000 iterations. Note that the weights are sorted.
7 Conclusions This paper introduced the weights-based non-conformity function for AdaBoost. This function reduces the time complexity of computing class regions with a factor equal to the size of the training data compared to the general non-conformity function. Using the weights-based non-conformity function results in statistically better class regions than those produced by the general non-conformity function, if the AdaBoost classifiers employed are neither underfitted nor overfitted. Thus, we conclude that the weights-based non-conformity function allows construction of better AdaBoost region classifiers that can be trained on larger data sets. To increase the practical use, our future research will concentrate on the problem of adapting the weights-based non-conformity function so that the conformal framework for AdaBoost can be applied to strong classifiers. The key idea is to employ a soft margin version of AdaBoost [9] that enables control of complexity by means of a regularization parameter C. In this way, our adapted function will return weights again but this time controlled by C. Thus, we can employ strong classifiers for the AdaBoost algorithm and we can overcome the weight skewness that can be observed in Figure 4.
136
M. Moed and E.N. Smirnov
References [1] Asuncion, A., Newman, D.J.: UCI machine learning repository (2007) [2] Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006) [3] Caprile, B., Furlanello, C., Merler, S.: Highlighting hard patterns via adaboost weights evolution. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS (LNAI), vol. 2364, pp. 72–80. Springer, Heidelberg (2002) [4] Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: International Conference on Machine Learning, pp. 148–156 (1996) [5] Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2002) [6] Holte, R.C.: Very simple classification rules perform well on most commonly used datasets. Machine Learning 11, 63–91 (1993) [7] Niculescu-Mizil, A., Caruana, R.: Obtaining calibrated probabilities from boosting. In: Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, Edinburgh, Scotland, July 26-29, pp. 413–418 (2005) [8] Proedrou, K.: Rigorous Measures of Confidence for Pattern Recognition and Regression. PhD thesis, Royal Holloway College, University of London, UK (2003) [9] R¨atsch, G., Onoda, T., M¨uller, K.-R.: Soft margins for adaboost. Machine Learning 42(3), 287–320 (2001) [10] Schapire, R.: The boosting approach to machine learning: An overview. In: MSRI Workshop on Nonlinear Estimation and Classification, Berkeley, CA (March 2001) [11] Shafer, G., Vovk, V.: A tutorial on conformal prediction. Journal of Machine Learning Research 9, 371–421 (2008) [12] Smirnov, E.N., Vanderlooy, S., Sprinkhuizen-Kuyper, I.G.: Meta-typicalness approach to reliable classification. In: Proceedings of the 17th European Conference on Artificial Intelligence, Riva del Garda, Italy, August 28 - September 1, pp. 810–811. IOS Press, Amsterdam (2006) [13] Vanderlooy, S., van der Maaten, L., Sprinkhuizen-Kuyper, I.: Off-line learning with transductive confidence machines: an empirical evaluation. In: Perner, P. (ed.) MLDM 2007. LNCS (LNAI), vol. 4571, pp. 310–323. Springer, Heidelberg (2007) [14] Vovk, V., Gammerman, A., Shafer, G.: Algorithmic learning in a random world. Springer, Heidelberg (2005) [15] Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In: Proceedings of the 18th International Conference on Machine Learning, pp. 609–616. Morgan Kaufmann, San Francisco (2001)
A Linear Classification Method in a Very High Dimensional Space Using Distributed Representation Takao Kobayashi and Ikuko Shimizu Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology, 2-24-16 Nakacho, Koganei-shi, 184-8588 Japan {tkobaya, ikuko}@cc.tuat.ac.jp
Abstract. We have proposed a fast learning and classification method by using distributed representation of vectors. In this paper, first, we shows that our method provides faster and better performance than 1-NN method by introducing a definition of a similarity concerned with LSH scheme. Next we compare our method with the Naive Bayes with respect to the number of dimensions of features. While the Naive Bayes requires a considerably large dimensional feature space, our method achieves higher performance even where the number of dimensions of a feature space of our method is much smaller than that of Naive Bayes. We explain our method by formalizing as a linear classifier in a very high dimensional space and show it is a special case of Naive Bayes model. Experimental results show that our method provides superior classification rates with small time complexity of learning and classification and is applicable to large data set. Keywords: Locality sensitive hashing, 1-NN method, Naïve Bayes, Bagging.
1 Introduction In recent years, huge amount of data sets have been occurred and stored in various scenes in the society. These data sets include transaction data, observation data, web pages and various contents of texts, images, sounds and videos. Because of the background, attention for the technology of leaning and classification against large data sets is led. Some typical methods take an enormous amount of time for learning so that they fail to finish in practical time if a large number of samples are involved. Also the classification problem for a large class set is not solved sufficiently. For example, generic object recognition [1] and handwritten Chinese character recognition involve more than several thousand classes. To classify patterns into a large number of classes we also have to manage huge amount of samples in most cases. On the other hand the memory size deployed in a computer is increasing every year, so new algorithms using large memory space become feasible. We have proposed and evaluated a learning and classification method using large memory size [2]. The key idea of the method is that a feature vector is represented by a set of
P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 137–147, 2009. © Springer-Verlag Berlin Heidelberg 2009
138
T. Kobayashi and I. Shimizu
vectors and a similarity function for classification is defined using it. We call it as “distributed representation” and also call the classification method as “DR classification (DRC).” In this paper, we present excellences of DRC by clarifying relationships to other well-known methods. Chapter 2 introduces a similarity by distributed representation and considers relations to locality sensitive hashing scheme including comparison with the 1-NN method. Chapter 3 shows DRC is a special variant of Naïve Bayes classification. And also shows that Bagging applied for DR classifiers is almost the same to Naïve Bayes classifier by DR. Chapter 4 discusses about meaning of linear classification in a very high dimensional space. Chapter 5 draws a conclusion.
2 Classification by Distributed Representation 2.1 Definition of a Similarity by Distributed Representation Charikar introduced a definition of the hash family F as locality sensitive hashing (LSH) scheme as follows: [3]
sim( x, y ) = PrH ∈F (H ( x) = H ( y ) )
(1)
where sim(x, y) is some similarity function between two points x and y. In other words, the locality sensitive hash family has a property such that two points are nearer if the probability of collision of these two hash codes is higher. Thus the distance between x and y can be estimated by calculating ∑ δ (H l ( x), H l ( y ) ) using many Hl , where δ is the Kronecker delta. On the other hand, we introduce a new definition in the reverse way. Namely, we prepare L hash functions H1, …, HL in advance, then the similarity between two vectors is defined as follows: L
sim( x, y ) = ∑ δ (H l ( x) = H l ( y ) ) .
(2)
l =1
Therefore the distance between two vectors is self-evidently defined as
dist ( x, y ) = L − sim( x, y ) .
(3)
It is easily proved that it satisfies the axiom of metric. Then we describe concrete hash functions. To construct them there exists the way by dividing a space into Voronoi cells using an appropriate point set. We give a redefinition of Eq.3 with L discrete point sets Q1, … , QL prepared in advance as follows: L
sim DR ( x, y ) = ∑ Ql ( x) ∩ Ql ( y ) l =1
(4)
A Linear Classification Method in a Very High Dimensional Space
139
where we use following notations. For a discrete point set Q, Q(x) is a set of the nearest neighbor in Q from a vector x (i.e. Q(x) is a set having only one element). Thus a vector x is represented as (Q1(x), …, QL(x)) and this manner can be used to measure a distance or to classify patterns. We call this form as distributed representation (DR). In this idea, we are implicitly using mathematical properties of high dimensional spaces. Generally our approach is not good for lower dimension (e.g. less than 10). However, in the case of higher dimension (e.g. more than 20), we can construct good algorithms by using proper data structures such as hash tables although required memory size is apt to large. Here we illustrate a property of high dimensional spaces with an instance of hash families. Let x and y be vectors in N dimensional spherical space (i.e. |x| = |y| = 1). For arbitrary N×N orthogonal matrices R1, R2, … the expected value of {−1,+1}N ( Rl x) ∩ {−1,+1}N ( Rl y ) depends on the distance between x and y. Fig.1 shows relation between the distance and the expected value. As the number of dimensions becomes higher, the value becomes more locally sensitive. In this case, the number of elements | { 1, +1}N | is 2N which becomes very large as N grows.
-
Expected value 1.0
Dimension=5 Dimension=10 Dimension=20 Dimension=40
0.8
0.6
0.4
0.2
0.0 0
30
60
Distance (degree)
90
Fig. 1. Relation between the distance between two vectors and the expected value of |Q (x)∩Q (y)|. In this case Q = {Rb | b {-1, +1}N, R is an arbitrary orthogonal matrix} is adopted for hash functions.
∈
2.2 Relationship between DR Classification and the 1-NN Method We have proposed a definition of a similarity between an unknown vector x and a sample set Y = {y(0), …, y(M-1)} belonging to a class, which is called DR classification (DRC), as follows [2]:
140
T. Kobayashi and I. Shimizu
L
simDR ( x, Y ) = ∑ Ql ( x) ∩ l =1
∪Q ( y
( j)
l
y
( j)
) .
(5)
∈Y
Thus classifying x into the most suitable class Ck* is formalized as Eq.6 using sample sets Y1, …, YK belonging each class C1, …, CK .
k* = arg max{simDR ( x, Yk )} .
(6)
k
Clearly, ∀
{
}
k , sim DR ( x, Yk ) ≥ max sim DR ( x, y k( j ) ) . j
(7)
And the right hand side is exactly the 1-NN method. Therefore the performance of DRC may have some relation with the 1-NN method. In fact, we have confirmed that classification rates of the DR classifier can excel ones of the 1-NN method under practical conditions of the numbers of dimensions and learning samples [4]. An experimental result will be shown in next section. 2.3 Experiment for Comparing Classification Rates We present an experiment for comparison of DRC (Eq.5) and the 1-NN method with the similarity of Eq.4 employing an off-line handwritten numerical data set BIRDS [5] which includes more than 190,000 patterns. It has so many patterns that we can observe relations between the number of learning samples and classification rates. For this experiment we adopted the way mentioning in [2] as a distributed representation, i.e. the feature space is Facto(N) = {x = (x1, …, xN) | x1…xN is a permutation of N integers 2i-(N-1) (i = 0, …, N-1)}, and N-1 sets are used as Q1, … , QN-1, where Ql = {a = (a1, …, aN) | ai = -1,+1, ∑δ(ai, +1) = l} (l = 1, …, N-1). Thus the element a such that Ql(x) = {a} is determined as follows:
⎧+ 1 ( xi ≥ N + 1 − 2l ) a = ( ai ), x = ( xi ), ai = ⎨ (i = 1, …, N ) . (otherwise) ⎩− 1 For reference, rates of 1-NN method with Euclidean metric in Facto(N) (simEUC) are also measured. The result is shown in Fig.2. The rates of DRC are higher than ones of the 1-NN with simDR while they are lower than ones of 1-NN with simEUC. However, the 1-NN method has a critical drawback, that is, its classification time becomes extremely larger as the number of learning samples is increased because no fast algorithm is known for classification of the 1-NN method. However DRC has fast algorithms for both of learning and classification. To mention the time complexity of DRC, here let M be the number of whole learning samples, N be the number of dimensions of a feature space and K be the number of classes, then there exists an algorithm using O(MN) learning time and O(NK) classification time. And the memory size of a classifier is lower than O(MN). Namely, classification time of DRC is independent of the number of learning samples.
A Linear Classification Method in a Very High Dimensional Space
141
Therefore we can take a simple strategy that we have only to add sufficient learning samples in order to improve the classification rate.
0.99
0.99
tea 0.97 r no 0.95 tia icif 0.93 ss laC 0.91
et ar 0.95 no 0.93 tia icif 0.91 ss 0.89 laC 0.87
0.97
Dimension = 24
DRC 1-NN (simDR) 1-NN (Euclid)
0.89 0.87 0.85
Dimension = 35
DRC 1-NN (simDR) 1-NN (Euclid)
0.85 0.83 0.81
0
50000
100000
150000
Number of learning samples
0
50000
100000
150000
Number of learning samples
Fig. 2. Classification rates of DRC excel ones of 1-NN with simDR although they are lower than ones of 1-NN with Euclidean metric at the same number of learning samples. The rates of all methods become gradually higher as increasing learning samples.
3 Relation between Naïve Bayes and DRC In this chapter we introduce a new format of Naïve Bayes classification. Then we formalize both Naïve Bayes and DRC as high dimensional multi-class linear classifiers. A multi-class linear classifier is designed as a format of the dot products of an input vector and each weight vector corresponding to a class. Weight vectors are determined by learning samples. We formalize weight vectors of these classifiers in the unified style. We also show Bagging of DRC becomes a linear classifier and is very similar to Naïve Bayes. 3.1 Orthodox Naïve Bayes Classifier
The conditional probability of a class Ck with occurrence of an N dimensional vector x = (x1, …, xN) can be written using Bayes theorem: Pr (Ck x1 , … , x N ) = Pr (x1 , … , x N C k ) Pr (C k ) Pr (x1 , … , x N ) .
(8)
For simplicity, we assume that prior probabilities Pr(Ck) are the same for all k. Thus classification is formalized simply as
arg max{Pr (x1 ,…, xN Ck )} . k
(9)
For Naïve Bayes model the assumption that each feature is conditionally independent is used. Then the probability is expressed by the form of the product:
142
T. Kobayashi and I. Shimizu
Pr(x1 ,…, xN Ck ) = ∏ Pr (xi Ck ) . N
(10)
i =1
Each Pr(xi|Ck) can be estimated from a sample set. Finally, a Naïve Bayes classifier is formalized as Eq.11, where Laplace correction ε is added so as to avoid the case such that an estimated probability is zero.
⎫ ⎧N arg max ⎨∑ log(Pr (xi Ck ) + ε )⎬ . k ⎭ ⎩ i=1
(11)
Here we try to apply the method of Eq.11 to classification for vectors used in section 2.3. Where a feature vector x is in Facto(N) and the number of dimensions N is from 24 to 63. The experimental result shows in Fig.3. However, classification rates do not reach 80%, so that they are far from those of DRC. We will discuss good conditions for Naïve Bayes classification in chapter 4. 0.80 0.78
et ar no 0.76 tia cif 0.74 is sa lC0.72
dimension = 24 dimension = 35 dimension = 48 dimension = 63
0.70 0.68 0
50000
100000
150000
Number of learning samples
Fig. 3. Classification rates by Naïve Bayes model are measured in the cases of dimensions 24, 35, 48 and 63. For this experiment the same data set and the same feature vectors as that in section 2.3 are used.
3.2 Naïve Bayes Using Distributed Representation
In this section we introduce a new format of Naïve Bayes classification by distributed representation. As shown in Eq.5 a DR classifier uses L attributes (Q1(x), …, QL(x)) derived from a input vector x. Here we assume the following equation
Pr((Q1 ( x),…, QL ( x)) Ck ) = ∏ Pr((l , Ql ( x)) C k ) . L
l =1
Then a Naïve Bayes classifier is constructed as follows
A Linear Classification Method in a Very High Dimensional Space
⎧L ⎫ arg max⎨∑ log(Pr((l , Ql ( x)) Ck ) + ε )⎬ . k ⎩ l =1 ⎭
143
(12)
Each attribute Ql(x) (l = 1, .., L) takes |Ql| kinds of value respectively. Therefore the inside of large brackets in Eq.12 can be written as Eq.13 with a bijective function F that maps (l, Ql(x)) to {1, …, ∑|Ql|} . (See Appendix.)
∑ Ql ⎛ L ⎞ ⎜ ∑ δ ( F (l , Ql ( x)), u ) × log(Pr(u Ck ) + ε )⎟ . ∑ u =1 ⎝ l =1 ⎠
(13)
Each probability Pr(u|Ck) for all u and k is estimated by a learning sample set. Namely it is calculated as Yk
L
Pr(u C k ) = ∑∑ δ ( F (l , Ql ( y k( j ) )), u ) j =1 l =1
Yk .
Note that |Yk| is a constant, then a similarity function between an input vector x and a learning sample set Yk can be defined as follows: sim NB ( x, Yk ) =
∑ Ql ⎛ L ⎛ Yk L ⎞⎞ ⎜ ∑ δ ( F (l , Ql ( x )), u ) × log⎜ ∑∑ δ ( F (l , Ql ( y k( j ) )), u ) + ε ′ ⎟ ⎟ ∑ ⎜ ⎟⎟ ⎜ u =1 ⎝ l =1 ⎝ j =1 l =1 ⎠⎠
(14)
where the left side of the product, ∑δ(F(l, Ql (x)), u) is a component of a ∑|Ql| dimensional sparse binary vector determined by an input vector, and the right side ⎞ ⎛ Yk L log⎜⎜ ∑∑ δ ( F (l , Ql ( y k( j ) )), u ) + ε ′ ⎟⎟ ⎠ ⎝ j =1 l =1
is a component of a ∑|Ql| dimensional vector with real value determined by learning samples. It can be regarded as a weight vector. The similarity simNB is obtained by calculating the dot product of these two vectors. Namely this operation means linear classification in ∑|Ql| dimensional space. To construct a good classifier, |Ql| must be big because Ql must have a high resolution in a feature space. So ∑|Ql| necessarily becomes very big. For example, in experiments in section 2.3 and 3.4 ∑|Ql| is in the range of 224 and 248. DRC shown in Eq.5 is also represented in the form of linear classification as
simDR ( x, Yk ) =
∑ Ql ⎛ L ⎞ ⎜ ∑ δ ( F (l , Ql ( x)), u ) × w(u, k ) ⎟ ∑ u =1 ⎝ l =1 ⎠
(15)
⎧ Yk L ⎫ where, w(u, k ) = min ⎨ ∑∑ δ ( F (l , Ql ( y k( j ) ), u )), 1 ⎬ . ⎩ j =1 l =1 ⎭ The difference between simNB (Eq.14) and simDR (Eq.15) is the value of components of weight vectors. The weight vectors by simDR are binary vectors. Thus simDR is
144
T. Kobayashi and I. Shimizu
simpler than simNB for program implementation. To measure these performances an experiment will be shown in section 3.4. 3.3 Bagging of DC Classifiers
In this section we lead a form similar to Eq.14 in a different way described in [7]. A form of Bagging [6] with T DC classifiers learned with m samples for each class Ck is formalized as follows:
sim Bagging ( x, Yk ) =
1 T ∑ simDR ( x, Yk(t ) ) T t =1
=
Q ⎞⎤ 1⎡T ∑ l⎛ L ⎢∑ ∑ ⎜ ∑ δ ( F (l , Ql ( x)), u ) × wt (u, k ) ⎟⎥ T ⎣⎢ t =1 u =1 ⎝ l =1 ⎠⎦⎥
=
∑ Ql ⎛ L 1 T ⎞ ⎜ ∑ δ ( F (l , Ql ( x)), u ) × ∑ wt (u, k ) ⎟ ∑ T u =1 ⎝ l =1 t =1 ⎠
(16)
where, t-th learning sample set belonging class Ck is Yk(t) (t = 1, …, T) (|Yk(t)|= m), and
⎧⎪ wt (u, k ) = min ⎨ ⎪⎩
Yk( t )
⎫⎪
L
∑∑ δ ( F (l , Ql ( y k(t , j ) ), u )), 1 ⎬ . ⎪⎭
j =1 l =1
We assume that learning samples are available inexhaustibly and let T→∞, then
1 T ∑ wt (u, Ck ) → E[w(u, k )] . T t =1
(17)
This is the average weight of classifiers learned with randomly taken m samples. Suppose p is a average value of w(u, k) learned with one sample, then average value learned with m samples is 1-(1-p)m. Therefore we can estimate E[w(u, k )] from actual whole learning sample set Yk and lead following equation:
sim Bagging ( x, Yk ) = ∑ Ql ⎛ ⎜ ∑ ⎜ u =1 ⎜ ⎝
⎛ ⎛ 1 δ ( F (l , Ql ( x)), u ) × ⎜⎜1 − ⎜⎜1 − ∑ Y l =1 k ⎝ ⎝ L
Yk
L
∑ ∑ δ ( F (l, Q ( y j =1 l =1
l
( j) k
⎞ )), u ) ⎟ ⎟ ⎠
m
⎞ ⎟ ⎟ ⎠
⎞ ⎟ ⎟⎟ ⎠
(18)
This equation shows that creating a Bagging classifier results in creating only one linear classifier with adjusted weights. The optimal value of m in Eq.18 can be determined by cross-validation. The weight vector of Eq.18 seems to be similar to that of Eq.14.
A Linear Classification Method in a Very High Dimensional Space
145
3.4 Experimental Results
We conduct an experiment to measure classification rates of DRC (simDR; Eq.5 or Eq.15), Naïve Bayes by DR (simNB; Eq.14) and Bagging of DR classifiers (simBagging; Eq.18). The same dataset as that in section 2.2 is used for this experiment. Fig 4 shows the results. In the case of dimensions 24, simNB and simBagging become higher than simDR when the number of learning samples is increased. While in the cases of 35 and 48, there are only slight differences between simDR and the others. It is natural that simBagging is higher than simDR because simBagging is ensemble of classifiers of simDR. It is also observed that the rates of simNB and simBagging are almost the same. This result is acceptable because forms of weight expression in Eq.14 and Eq.18 are similar. 0.98
0.97
Dimension = 24
0.96
simDR simNB
0.95
simBagging
0.94 0
50000
100000
150000
0.98
0.98
0.97
0.97
0.96
Dimension = 35
0.96 0.95
simDR
0.94
0.95
simNB
0.93
simBagging
0.92
0.93
Dimension = 48
0.94
simDR simNB simBagging
0.91 0.90
0.92 0
50000
100000
150000
0
50000
100000
150000
Fig. 4. Comparison of classification rates of DRC (simDR), Naïve Bayes by DR (simNB) and Bagging of DR classifiers (simBagging). Is is observed that all have strong correlations.
4 Discussion By well-known phenomenon as the curse of dimensionality, classification becomes harder as the number of dimensions of a feature space becomes larger for the limited
146
T. Kobayashi and I. Shimizu
number of learning samples because the distribution of feature vectors becomes extremely sparse in the space. However as we observed in Chapter 2 and 3, classification by distributed representation is the method such that the feature vector space is mapped to a very high dimensional space where linear classification is performed. Furthermore its performance is excellent. Linear classification in very high dimensional space reminds us of SVM using kernels. SVM attains a hyperplane such that leaning sample sets of each class are divided by the iterative method. Its learning process is so time-consuming that learning for a large amount of samples is not practical. Further, applying SVM to a large class set is also unacceptable. On the other hand, a learner of DRC creates weight vectors for each class which classify all learning samples directly. Therefore DRC can be applied to classification problems involving very large data sets. Naïve Bayes classification by DR is similar to DRC because their difference is only for weights and an experimental result show their classification rates are close. DRC can be therefore regarded as a variant of Naïve Bayes classification. Further, Bagging of DR classifier is very similar to Naïve Bayes by DR, thus we can say reversely that they are variants of DRC. By the way, the conventional Naïve Bayes model is implemented as linear classification in the original feature vector space. In this case, in fact, a relatively higher dimension space (e.g. several hundred or several ten thousand) is used. For example, in the case of document classification, Naïve Bayes model using feature vectors of which attributes are frequencies of each word is applied. In [8], 38500 dimensional feature vectors are used for 20,000 learning samples and 20 classes. The reason why Naïve Bayes classification acts well seems to be that a high dimensional space is used.
5 Conclusion We formalized the learning and classification method using distributed representation and considered relation to LSH scheme, 1-NN, Naïve Bayes and Bagging. We presented that distributed representation model is a metamorphosis of the 1NN method using a similarity defined by hash functions and confirmed that its classification rates are higher than ones of the 1-NN method by an experiment using a real dataset. We also formalized DRC as a linear classifier in a very high dimensional space and considered relations to Naïve Bayes and Bagging. These results will accelerate further progress concerning these methods. DR learner and classifier works so fast that we can expect to solve various classification problems involving huge data set.
References 1. Yanai, K.: Current state and future directions on generic object recognition. IPSJ Transaction on Computer Vision and Image Media 48(SIG) (CVIM19), 1–24 (2007) (in Japanese) 2. Kobayashi, T., Nakagawa, M.: A pattern classification method of linear-time learning and constant-time classification. Transactions of IEICE J89-A(11), 981–992 (2006) (in Japanese)
A Linear Classification Method in a Very High Dimensional Space
147
3. Charikar, M.S.: Similarity Estimation Techniques from Rounding Algorithms. In: Proceedings of the 34th Annual ACM Symposium on Theory of Computing 2002 (2002) 4. Kobayashi, T., Shimizu, I., Nakagawa, M.: Theoretical studies of the Power Space Similarity method: a fast learning and classification algorithm. In: Proceedings of the 3rd Korea-Japan Joint Workshop on Pattern Recognition, November 2008, pp. 29–30 (2008) 5. http://www.geocities.jp/onex_lab/birdsdb/birdsdb.html 6. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996) 7. Kobayashi, T., Nakagawa, M.: Pattern recognition by distributed coding: test and analysis of the power space similarity method. In: Proc. 9th IWFHR, October 2004, pp. 389–394 (2004) 8. Joachims, T.: A probabilistic analysis of the rocchio algorithm with TD.IDF for text categorization. Technical Report CMU-CS-96-118, Carnegie-Mellon Institute (1996)
Appendix: Bijective Function F For any x, Ql(x) for each l (l = 1, …, L) becomes |Ql| kinds respectively. Consequently (l, Ql(x)) for all l becomes ∑|Ql| kinds. A bijective function L ⎧ ⎫ F : (l , Ql ( x)) → ⎨1,…, ∑ Ql ⎬ l =1 ⎩ ⎭
can be defined in the following manner, for instance:
F : { (1, {q}) q ∈ Q1 } → {1, …, Q1 }
F : { (2,{q}) q ∈ Q2 } → { Q1 + 1, …, Q1 + Q2 } l ⎧ l −1 ⎫ F : { (l , {q}) q ∈ Ql } → ⎨ ∑ Qi + 1, …, ∑ Qi ⎬ i =1 ⎩ i =1 ⎭
L ⎧ L−1 ⎫ F : { ( L,{q}) q ∈ QL } → ⎨ ∑ Qi + 1, …, ∑ Qi ⎬ i =1 ⎩ i =1 ⎭
PMCRI: A Parallel Modular Classification Rule Induction Framework Frederic Stahl, Max Bramer, and Mo Adda University of Portsmouth, Buckingham Building, Lion Terrace, Portsmouth PO1 3HE, United Kingdom {Frederic.Stahl, Max.Bramer, Mo.Adda}@port.ac.uk
Abstract. In a world where massive amounts of data are recorded on a large scale we need data mining technologies to gain knowledge from the data in a reasonable time. The Top Down Induction of Decision Trees (TDIDT) algorithm is a very widely used technology to predict the classification of newly recorded data. However alternative technologies have been derived that often produce better rules but do not scale well on large datasets. Such an alternative to TDIDT is the PrismTCS algorithm. PrismTCS performs particularly well on noisy data but does not scale well on large datasets. In this paper we introduce Prism and investigate its scaling behaviour. We describe how we improved the scalability of the serial version of Prism and investigate its limitations. We then describe our work to overcome these limitations by developing a framework to parallelise algorithms of the Prism family and similar algorithms. We also present the scale up results of a first prototype implementation.
1 Introduction The growing interest and importance of commercial knowledge discovery and data mining techniques has led to a growing interest in the area of classification rule induction from data samples to enable the classification of previously unseen data. Research in classification rule induction can be traced back at least to the 1960s [1].A very widely used method to induce classification rules is TDIDT [2] which has given rise to a variety of implementations such as C5.0. However alternative algorithms exist such as the Prism algorithm [3]. Prism produces more generalised rules than TDIDT and thus tends to perform better on noisy datasets. As a result Prism has been used in areas where datasets are naturally noisy such as image recognition [4] or text classification. Prism is also the base for further data mining algorithms such as PrismTCS [5], N-Prism [6]. A free implementation of Prism can be found in the WEKA package [7] and also in the Inducer workbench[8]. The increase in performance of computer hardware such as CPU power and disc storage and sensors to record data such as CCTV cameras enables companies and researchers to generate and store larger and larger datasets to which they still wish to apply classification rule induction algorithms. This has led to the exploration of a new niche in data mining, parallel and distributed data mining. So far, work on distributed and parallel classification rule induction has been focused on the well-established TDIDT approach. Notable developments are the SLIQ [9] and its successor the SPRINT [10] algorithm. The latter P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 148–162, 2009. © Springer-Verlag Berlin Heidelberg 2009
PMCRI: A Parallel Modular Classification Rule Induction Framework
149
achieves an almost linear scale up with respect to the number of CPUs and the sample size. However, very little work has been done on scaling up alternative algorithms such as Prism. One approach to scaling a data mining algorithm is to sample the data before the algorithm is applied. Catlett’s work [11] showed that sampling of data results in a loss of accuracy in the induced classifier. However Catlett’s research was conducted 17 years ago and the datasets he used were fairly small compared with those used today. Frey and Fisher found in 1999 that the rate of increase of accuracy slows down with the increase of the sample size [12]. This resulted in seeking optimized methods for sampling massive datasets such as progressive sampling [13]. Whereas sampling might be an option for predictive modelling, scaling up data mining algorithms is still desirable in applications that are concerned with the discovery of new knowledge. Chan and Stolfo considered a way to scale up classification rule induction by dividing the data into subsets that fit in a single computer's memory and then generating a classifier on each data subset in parallel on several machines[14, 15]. The different classifiers generated are then combined by using various algorithms in order to achieve a final classifier. Despite the significant reduction of run times of the classification rule induction process, Chan and Stolfo's studies also showed that this approach does not achieve the accuracy of a single classifier induced on the same training data. In order to meet the need for a well scaling, more generalised and thus noise tolerant classifier, we investigate and improve PrismTCS’s scaling behaviour and derive a parallel approach to inducing classification rules in parallel for algorithms based on the Prism family. We present a framework that induces modular classification rules in parallel based on the PrismTCS algorithm and evaluate its scaling behaviour.
2 Inducing Modular Classification Rules The main drawback of the TDIDT approach, also often called the divide and conquer approach, lies in the intermediate representation of its classification rules in the form of a decision tree. Rules such as: IF a = 1 AND b = 1 THEN class = 1 IF c = 1 AND d = 1 THEN class = 0 which have no attribute in common, could not be induced directly using the TDIDT approach. In such cases, TDIDT will first need to introduce additional tests that are logically redundant simply to force the rules into a form suitable for combining into a tree structure. This will inevitably lead to unnecessarily large and confusing decision trees.Cendrowska designed the original Prism algorithm to induce directly sets of 'modular' rules that generally will not fit conveniently into a tree structure, thus avoiding the redundant terms that result when using the TDIDT approach. Prism generally induces rule sets that tend to overfit less compared with TDIDT, especially if it is applied to noisy datasets or datasets with missing values [6]. Cendrowska's Prism algorithm follows the separate-and-conquer approach which learns a rule that explains a certain part of the training data. It then separates the data explained by the rule induced and induces the next rule using the remaining data. Thus it recursively
150
F. Stahl, M. Bramer, and M. Adda
“conquers” until no training data is left. This strategy can be traced back to the AQ learning system [16]. The basic separate and conquer algorithm can be described as follows: Rule_Set = []; While Stopping Criterion not satisfied{ Rule = Learn_Rule; Remove all data instances covered from Rule; } The algorithm specific procedure Learn_Rule learns the best rule for the current training data subset. After each induced rule all data instances that are not covered are deleted and the next rule is learned from the remaining data instances. This is done until a Stopping Criterion is fulfilled. Also the Stopping Criterion is an algorithm specific one that differs from algorithm to algorithm. PrismTCS (Prism with Target Class, Smallest first) a version of Prism that attempts to scale up Prism to larger datasets has been developed by one of the present authors [5]. Whereas in PrismTCS the separate-and-conquer approach is applied only once, in the original Prism algorithm it is applied for each class in turn. PrismTCS has a comparable predictive accuracy to that of Prism [5]. Our implementation of PrismTCS for continuous data only is summarised in the following pseudo code: (a)
(b)
(c) (d) (e) (f)
(g)
working dataset W = restore Dataset; delete all records that match the rules that have been derived so far and select the ; target class i = class that covers the fewest instances in W; For each attribute A in W - sort the data according to A; - for each possible split value v of attribute A calculate the probability that the class is i for both subsets A < v and A ≥ v; Select the attribute that has the subset S with the overall highest probability; build a rule term describing S; W = S; Repeat b to e until the dataset contains only records of class i. The induced rule is then the conjunction of all the rule terms built at step d; restore Dataset = restore Dataset – W; Repeat a to f until W only contains instances of class i or is empty;
The following approaches and the parallel classification rule induction algorithm presented in this paper are explained in the context of PrismTCS. However, our approaches can be applied to Prism and all its descendants analogously.
PMCRI: A Parallel Modular Classification Rule Induction Framework
151
2.1 Speeding Up PrismTCS We identified two major overheads in Prism and PrismTCS that lower its computational efficiency considerably. The overheads comprise sorting for continuous attributes in step b of the algorithm and the frequent deletion of data instances and resetting of the training dataset in step a, e and g of the algorithm. With respect to the sorting overhead we removed the innermost loop of sorting in step b by employing a presorting strategy. The training dataset is pre-sorted by building attribute lists of the structure
similar to the SPRINT algorithm [10, 17]. These attribute lists can be sorted before the first iteration of the Prism algorithm and remain sorted during the whole duration of the algorithm. With respect to frequent restoring of the training dataset an efficient data compression algorithm has been developed. When we talk about compression we mean an efficient way to delete and restore instances of the training dataset, while maintaining the sorted nature of the attributes, which is needed frequently in the algorithms of the Prism family. For example regarding the PrismTCS pseudo code, data instances are deleted in step e and g and restored in step a. The challenge here is to find a way of data efficient compression that takes account of the pre-sorted attribute lists. One way to implement this would be to keep two copies of each attribute list in memory, one list for resetting purposes and one list to work with, analogously to the “working dataset W” and the “restore Dataset” in the PrismTCS pseudo code in section 2. Attribute lists would be restored by replacing the working set of lists with a restore set of lists. However this approach involves a considerable overhead of memory usage by keeping two copies of the whole training dataset in the form of attribute lists in memory. A further overhead in processing time is caused by frequently creating deep copies of attribute list records. We derived a more memory and time efficient algorithm for deleting and restoring data which involves having the dataset only stored once in the memory. We do that by only working with the record ids of each attribute list record which are stored in an integer array. This array is used to reference attribute values and class values which are stored in separate double precision and character arrays. Thus when pre-sorting the attribute list we only need to sort the integer array with record ids. Also when deleting list records we only need to delete the references in the record ids array. Thus the attribute values array and class values array are left untouched during the whole duration of the Prism algorithm. However we also need to avoid expensive resizing of the record ids array due to deletion and resetting of ids. We do that by replacing ids that need to be deleted in the array by the next id that does not need to be deleted and thus the size of the actual array stays the same. If the number of ids that are not deleted is n then PrismTCS will only take record ids stored between indices 0 and n-1 into account and ignore the rest. Thus PrismTCS is required to update n whenever ids are deleted or reset. The pseudo code below shows the basic compression algorithm: int numbRelevant;// number of relevant ids in the array boolean[] remove;// each index in the array corresponds //to a actual id value that needs to be //deleted removal(numbRelevant, remove){
152
F. Stahl, M. Bramer, and M. Adda
int i,j; j = 0; FOR(i=0; i
y = 0.002x2 + 9.036x (R² = 1) y = 0.002x2 + 1.158x (R² = 1) y = 3.620x (R² = 0.998)
PMCRI: A Parallel Modular Classification Rule Induction Framework
153
Fig. 1. Scale up of PrismTCS
3 PMCRI: A Parallel Modular Classification Rule Induction Framework There have been several attempts to scale up classification rule induction via parallelisation. In the area of TDIDT we have already mentioned the SPRINT [10] algorithm. We can distinguish two types of parallel processing in data mining: fine grained parallelisation and loosely coupled distributed data mining [19]. Whereas fine grained parallelisation makes use of “shared memory multiprocessor” machines (SMP), loosely coupled distributed data mining makes use of “shared nothing” or “massively parallel processors” (MPP) [20]. We will focus here on parallelising modular classification rule induction using an MPP approach. Our reasoning is that MPP can be represented by a network of workstations and thus is a cheap way of running parallel algorithms for organisations with limited budgets. We want to use an MPP infrastructure to parallelise the modular classification rule induction of Prism by using the Cooperating Data mining Model (CDM) [19]. The CDM model is illustrated in figure 2. It is partitioned into different sections which we call layers. In the first layer of the CDM model is the sample selection procedure which partitions the data sample S into n sub samples where n is the number of workstations available. There are n Learning algorithms L1,…,Ln in the second layer that run on the corresponding subsets and generate concept descriptions. C1, …. Cn. In the third layer these concept descriptions are then merged by a combining procedure to a final concept description Cfinal. The final concept description in the case of classification rule induction would be a set of classification rules. We developed a parallel modular classification rule induction framework for the Prism family and tested it on PrismTCS, the PMCRI (Parallel Modular Classification Rule Induction) framework [21], which applies to the CDM model. Parallelisation in the first layer is achieved by distributing all attribute lists evenly over n processors and processing them locally by algorithms L1 to Ln, which induce rule terms. To implement the second layer in the CDM model we used a distributed blackboard system architecture based on the DARBS distributed blackboard system [22].
154
F. Stahl, M. Bramer, and M. Adda
Fig. 2. Cooperating Data Mining
Fig. 3. The architecture of the PMCRI framework using a distributed blackboard system in order to parallelise the induction of modular rule terms. It is divided into two logical partitions: one to submit local rule term information and one to retrieve global information about the algorithm’s status.
A blackboard system can be imagined as a physical blackboard which is observed by several experts with different knowledge domains, having a common problem to solve. Each expert will use its knowledge domain plus knowledge written on the blackboard in order to infer new knowledge about the problem and advertise it to the other experts by writing it on the blackboard. In the software model such a blackboard system can be represented by a client server architecture. Figure 3 shows the basic communication pattern of PMCRI. The expertise of each expert machine is determined by the attribute lists it holds. Thus loosely speaking, each expert can induce the “locally best rule term”. The experts then use then the “Local Rule Term Partition” on the blackboard to exchange information in order to find the “globally best rule term”. The winning expert then will communicate the ids of the instances that are uncovered by this rule term to the other waiting experts using the “Global Information Partition” on the blackboard system. Now the next rule term can be induced in the same way. In
PMCRI: A Parallel Modular Classification Rule Induction Framework
155
Fig. 4. The combining procedure of the CDM is realised by the moderator program, which will assemble the complete rules after each expert submits its globally best rule terms together with information about the rule for which they were induced
PMCRI the attribute lists decrease in size at the same rate, thus the workload on each expert machine stays in the same proportion for all remaining experts. The following steps describe how PMCRI induces one rule in the context of PrismTCS[23]: Step 1 Moderator (PrismTCS) writes on “Global Information Partition” the command to induce locally best rule terms. Step 2 All Experts induce the locally best rule term and write the rule terms plus its covering probability and the number of list records covered on the “local Rule Term Partition” Step 3 Moderator (PrismTCS) compares all rule terms written on the “Local Rule Term Partition”; adds best term to the current rule; writes the name of the Expert that induced the best rule term on the Global Information Partition Step 4 Expert retrieves name of winning expert. IF Expert is winning expert { derive by last induced rule term uncovered ids and write them on the “Global Information Partition” and delete uncovered list records } ELSE IF Expert is not winning expert { wait for by best rule term uncovered ids being available on the “Global Information Partition”, download them and delete list records matching the retrieved ids. }
156
F. Stahl, M. Bramer, and M. Adda
In order to induce the next rule term, PMCRI would loop back to step one. For PMCRI to know when to stop the rule it needs to know when the remaining list records on the expert machines are either empty or consist only of instances of the current target class. This information is communicated between the winning expert and the moderator program using the Global Information Partition. It is possible to implement all the descendants of the original Prism algorithm simply by adapting the learner algorithm within this framework. In layer 3 at the end of the PMCRI execution each expert machine will hold a set of terms for each rule. The implementation of the combining procedure in layer three in the CDM model is realised by communicating all the rule terms locally stored at the expert machines to the blackboard. Each rule term is associated with information about the rule and the class for which the terms were induced. The moderator program then simply appends each rule term to its corresponding rule as illustrated in figure 4.
4 Performance of PMCRI To evaluate PMCRI we used the yeast dataset from the UCI repository [18]. To create a larger dataset and thus a higher (more challenging) workload for our system, we first appended it to itself in a horizontal direction until the dataset comprised a total number of 50 attributes. We used this base dataset to evaluate the system's performance. We then appended the data to itself in a vertical direction in order to increase the number of instances and thus increase the system's workload. Please note that the learner algorithms of PMCRI are based on PrismTCS and produce exactly the same rules as PrismTCS would induce. There is therefore no need to concern ourselves with issues concerning the comparative quality of rules generated by the different algorithms. Also please note that all datasets used were based on the yeast dataset, thus the classifiers induced in each experiment were identical and issues relating to differing numbers of rules and rule terms do not arise. This enables us to focus on issues of workload only. We made scale up experiments to evaluate PMCRI's performance with respect to its number of processors, speed up experiments to evaluate its performance with respect to the number of processors together with the processors' workload and size up experiments in order to evaluate its performance concerning the system's total workload. The hardware we used comprised ten identical machines where each machine had one Pentium processor with 2.8 GHz and one GB memory. We run lightweight xUbuntu Linux Systems on each machine. 4.1 Scale Up Scale up is used to observe the system's ability to be enlarged. For examining the scale up of PMCRI we observe how the response time changes if the number of
PMCRI: A Parallel Modular Classification Rule Induction Framework
157
Fig. 5. Scale up of PMCRI
processors is increased while the workload per machine stays constant. In the ideal case the response time of PMCRI would stay constant since the total workload of each processor stays the same. We studied three scale up experiments with a workload corresponding to 130,000, 300,000 and 850,000 instances per processor. Please note due to the distribution of the data via attribute lists each processor only holds a part of each data instance, for example the 130k instances workload for each processor in the case of two processors are actually 260k instances with half the attributes of each instance. In order to simplify matters we just refer to 130k training instances. The results of the scale up experiments are shown in figure 5. The results show a nice scale up. The drop in scale up with adding more processors can be explained by the additional communication overhead in the LAN, as more processors need to synchronise by communicating information about covered and uncovered list ids, while the amount of data per processor stays constant, the number of ids that need to be communicated increases with an increasing number of total instances. However as figure 5 also shows the effect of communication overhead can be lowered by increasing workload per processor. Loosely speaking the higher the overall workload the more the system profits from using additional processors. 4.2 Speed Up Speed up is used to compare how much a parallel algorithm is faster than the serial version of it. However we are limited with workload of the serial version of PrismTCS by the size of the memory of the computer used. However as can be seen in section 4.3 all parallel versions of PrismTCS using PMCRI are faster than the serial version thus an absolute speed up compared with the serial PrismTCS would be positive. However we are able to determine the relative speed up of PMCRI in the context of PrismTCS by basing it to a two processor configuration. By examining the speed up characteristics of PMCRI we observe how the response time changes with the number of processors while the total workload stays constant. We studied three speed up experiments with a total number 600k, 1,000k and 2,000k data instances for configurations of 2, 5 and 10 processors. The results of the speedup experiments are shown in figure 6.
158
F. Stahl, M. Bramer, and M. Adda
Fig. 6. Speed up of PMCRI
As we can expect the speedup increases with the size of the dataset. The ideal speedup would be if the amount of processors is doubled then the response time is reduced by half. However this behaviour is contradicted for the same reason as the scale up behaviour, by a communication overhead. Especially for small datasets, the additional communication overhead contradicts the benefit of more processors considerably, but for larger datasets the benefit from having more CPU power strongly outweighs the communication overhead. Again we observe that PrismTCS parallelised using PMCRI is faster the more processors we use especially for higher data workloads. 4.3 Size Up In size up experiments we examine how PMCRI performs on a fixed processor configuration. We do that by increasing the size of the data and leaving the number of processors constant. Figure 7 shows the size up for three different processor configurations of PMCRI and our serial version of PrismTCS. We increased the dataset size from 17k up to 8,000k training instances.
Fig. 7. Size up of PMCRI
PMCRI: A Parallel Modular Classification Rule Induction Framework
159
Fig. 8. Size up using relative response time
Generally we observe a linear size up for PrismTCS and all PMCRI configurations, thus the processing time is a linear function of' the size of the dataset. The equations of a linear regression that prove the linear behaviour are shown below where x is the number of training instances and y the runtime in ms: PrismTCS: PMCRI with 2 processors: PMCRI with 5 processors: PMCRI with 10 processors:
y = 3.840x (R² = 0.994 ) y = 2.019x (R² = 0.997 ) y = 1.065x (R² = 0.995 ) y = 0.775x (R² = 0.997 )
Please note that all experiments from figure 7 were sized up to their maximum number of training instances. The maximum number of data instances is limited by the total amount of memory in the system. Buffering of attribute lists to the hard disc would be possible in order to overcome memory limitations; however frequent I/O operations are unwanted due to their time expensive behaviour. We can clearly see that if we use double the amount of memory we can hold roughly double the amount of training instances into memory. Figure 8 shows the size up using the relative response time for the same configurations as in figure 7, however this time we also added the ideal size up into the figure. The ideal size up would be that if we have double the amount of training instances on the same number of processors then we will need double the amount of time to train our classifier. We can observe that the serial version of PrismTCS clearly scales worse than the ideal runtime however for all parallel versions we observe a better sizing up behaviour than ideal, in particular the more processors we use the better the size up result compared with the ideal size up behaviour. Thus we can say PMCRI shows a superior size up performance.
5 Conclusions We presented the work and first results of the PMCRI framework, a Parallel Modular Classification Rule Induction algorithm. PMCRI harvests the computational power of
160
F. Stahl, M. Bramer, and M. Adda
a network in order to make modular classification rule induction scaling better on large datasets. We started the paper by discussing the limitations of the more popular TDIDT algorithm and then we discussed the Prism algorithm family as an alternative that tries to overcome TDIDT's limitations naturally. However Prism's downside is that it does not scale well on large datasets. We discovered the frequent deletion of and restoring of data records and frequent sorting operations for continuous attributes as bottlenecks of Prism. The latter bottleneck has been tackled by using attribute lists in order to keep the training data sorted during the whole duration of the algorithm. We addressed the bottleneck of the frequent deletion and restoring of data records by proposing an algorithm that efficiently deletes attribute list instances without having to resize the underlying data structure. We achieve a linear scaling behaviour of PrismTCS by implementing these two approaches. We then described our work on a parallel classification rule induction algorithm based on the same rule generalisation method as PrismTCS, the PMCRI algorithm. The parallelisation in PMCRI is achieved by distributing attribute lists evenly over the machines in the network. Each machine induced rule terms independently that are locally the best. A global view of the algorithm is achieved by each machine by exchanging information about the local status of each machine using a distributed blackboard architecture. As both PMCRI and PrismTCS employ the same rule generalisation strategy the rule sets produced by both on the same training data are identical. We then experimentally analysed PMCRI's performance. With respect to its scale up behaviour we observed that the more processors we use the higher the synchronisation overhead due to communicating local information between processors. However the synchronisation overhead is contradicted by the processors' workload, thus the higher the workload the closer PMCRI scales up to its ideal behaviour. With respect to PMCRI speed up we again observed a speed up below its ideal speed up, which again can be explained by a communication overhead between processors, however we could observe that the higher the workload the closer the actual speed up performance is to its ideal. With respect to PMCRI size up we observed a linear behaviour on a fixed processor configuration with respect to its workload. Generally for PMCRI we observed superior size up behaviour to its ideal. We also stretched the boundaries in order to find the maximum workload of the system. The maximum workload was limited by the total memory available thus the more machines we used the more data we were able to use in order to train our classifier. However we can generally say that the benefit of using more processors in PMCRI slows down due to an additional communication overhead per processor. However the communication overhead can be outweighed by adjusting the total workload of the system. Thus, loosely speaking, the user of the PMCRI system should balance the number of processors used with the total workload, thus for a smaller workload a configuration with less processors or even the serial version might be more beneficial concerning the system's runtime. However for large workloads a configuration with more processors is more likely to be beneficial. We are currently developing a more intelligent workload balancing strategy for PMCRI that takes into account that computers might have different CPU speeds and memory sizes available.
PMCRI: A Parallel Modular Classification Rule Induction Framework
161
References 1. Hunt, E.B., Marin, J., Stone, P.J.: Experiments in Induction. Academic Press, London (1966) 2. Quinlan, J.R.: Induction of decision trees. Machine Learning 1, 81–106 (1986) 3. Cendrowska, J.: PRISM: an Algorithm for Inducing Modular Rules. International Journal of Man-Machine Studies 27, 349–370 (1987) 4. Shu-Ching, C., Mei-Ling, S., Schengcui, Z.: Detection of Soccer Goal Shots Using Joint Multimedia Features and classification Rules. In: Fourth International Workshop on Multimedia Data Mining, Washington, DC, USA, pp. 36–44 (2003) 5. Bramer, M.: An Information-Theoretic Approach to the Pre-pruning of Classification Rules. In: Proceedings of the IFIP Seventeenth World Computer Congress - TC12 Stream on Intelligent Information Processing, pp. 201–212. Kluwer, B.V., Dordrecht (2002) 6. Bramer, M.: Automatic Induction of Classification Rules from Examples Using N-Prism. In: Research and Development in Intelligent Systems XVI (2000) 7. Garner, S.: Weka: The Waikato Environment for Knowledge Analysis. In: New Zealand Computer Science Research Students Conference, pp. 57–64 (1995) 8. Bramer, M.: Inducer: a public domain workbench for data mining. International Journal of Systems Science 36(14), 909–919 (2005) 9. Metha, M., Agrawal, R., Rissanen, J.: SLIQ: A Fast Fcalable Classier for Data Mining. In: Apers, P.M.G., Bouzeghoub, M., Gardarin, G. (eds.) EDBT 1996. LNCS, vol. 1057. Springer, Heidelberg (1996) 10. Shafer, J.C., Agrawal, R., Mehta, M.: SPRINT: A Scalable Parallel Classifier for Data Mining. In: Twenty-second International Conference on Very Large Data Bases (1996) 11. Catlett, J.: Megainduction: Machine learning on very large databases. University of Technology, Sydney (1991) 12. Frey, L.J., Fisher, D.H.: Modelling Decision Tree Performance with the Power Law. In: Evelenth International Workshop on Artificial Intelligence and Statistics (1999) 13. Provost, F., Jensen, D., Oates, T.: Efficient Progressive Sampling. In: Geoffrey, I. (ed.) Knowledge Discovery and Data Mining, pp. 23–32 (1999) 14. Chan, P.K., Stolfo, S.J.: Experiments on Multistrategy Learning by Meta Learning. In: Second International Conference on Information and Knowledge Management, pp. 314– 323 (1993) 15. Chan, P.K., Stolfo, S.J.: Meta-Learning for Multistrategy and Parallel Learning. In: Second International Workshop on Multistrategy Learning, pp. 150–165 (1993) 16. Michalski, R.S.: On the quasi-minimal solution of the general covering problem. In: Proceedings of the Fifth International Symposium on Information Processing, Bled, Yugoslavia, pp. 125–128 (1969) 17. Zaki, M.J., Ho, C.T., Agrawal, R.: Parallel Classification for Data Mining on Shared Memory Multiprocessors. In: Fifteenth International conference on Data Mining (1999) 18. Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. University of California, Irvine, Department of Information and Computer Sciences (1998) 19. Provost, F.: Distributed Data Mining: Scaling up and Beyond. In: Kargupta, P.C.H. (ed.) Advances in Distributed and Parallel Knowledge Discovery. AAAI Press / The MIT Press (2000) 20. Kamath, C., Musik, R.: Scalable Data Mining through Fine-Grained Parallelism. In: Kargupta, P.C.H. (ed.) Advances in Distributed and Parallel Knowledge Discovery. AAAI Press / The MIT Press (2000) 21. Stahl, F., Bramer, M.: P-Prism: A Computationally Efficient Approach to Scaling up Classification Rule Induction. In: IFIP International Conference on Artificial Intelligence. Springer, Milan (2008)
162
F. Stahl, M. Bramer, and M. Adda
22. Nolle, L., Wong, K.C.P., Hopgood, A.: DARBS: A Distributed Blackboard System. In: Twenty-first SGES International Conference on Knowledge Based Systems (2001) 23. Stahl, F., Bramer, M.: Parallel Induction of Modular Classification Rules. In: Twentyeighth SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence. Springer, Cambridge (2008)
Dynamic Score Combination: A Supervised and Unsupervised Score Combination Method Roberto Tronci, Giorgio Giacinto, and Fabio Roli DIEE Department of Electrical and Electronic Engineering, University of Cagliari, 09123 Cagliari, Italy {roberto.tronci, giacinto, roli}@diee.unica.it
Abstract. In two-class score-based problems the combination of scores from an ensemble of experts is generally used to obtain distributions for positive and negative patterns that exhibit a larger degree of separation than those of the scores to be combined. Typically, combination is carried out by a “static” linear combination of scores, where the weights are computed by maximising a performance function. These weights are equal for all the patterns, as they are assigned to each of the expert to be combined. In this paper we propose a “dynamic” formulation where the weights are computed individually for each pattern. Reported results on a biometric dataset show the effectiveness of the proposed combination methodology with respect to “static” linear combinations and trained combination rules.
1
Introduction
The combination of experts is largely used in the literature to improve the reliability and the performance of a recognition system [1,2]. The combination can be made at different levels and in different ways depending on the classification problem and the data of interest. In the case of two-class problems, the classes are usually denoted with the terms positive class, and negative class (in some specific domains such as biometry the terms are genuine and impostor, respectively). Usually, a two-class expert computes for each pattern a similarity (dissimilarity) score with respect to prototypes of the positive class. The decision is taken by setting a decision threshold on the score produced, so that the outcome is positive if the score is greater than a decision threshold, while it is negative otherwise. If the scores produced are not easily separable through a decision threshold the patterns can be subdivided into “true” and “false” patterns within the positive and negative classes. The aim of a two-class problem is to maximise the true positive rate (i.e. the number of positive patterns classified as positive over the total number of positive patterns), while minimising the false positive rate (i.e. the number of negative patterns classified as positive over the total number of negative patterns). Typically this tradeoff is measured through the use of the Receiver Operating Characteristic curve (ROC), which plots the true positive rate against the false positive rate as the decision threshold th varies along the score range [3,4]. The Area Under P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 163–177, 2009. c Springer-Verlag Berlin Heidelberg 2009
164
R. Tronci, G. Giacinto, and F. Roli
the ROC Curve (AUC), and the Equal Error Rate (EER) are two performance measure derived from the ROC that are typically used to compare the performance of different experts. The AUC is the area under the ROC curve, the larger the AUC, the better the ROC. The EER represents the point where the false positive rate and the false negative rate are equal, and a good system should keep this value as small as possible. The ROC, AUC, and EER are illustrated in Figure 1.
Fig. 1. An example of a ROC curve, its AUC and its EER
As said previously, different methodologies to combine the experts had been developed. In the case of two-class problem the combination is performed at the score level. The combination of scores aims to obtain a distribution of combined scores whose distributions for positive and negative patterns exhibit a larger degree of separation than those produced by individual experts. In the literature different score combination methods have been proposed [5,6,7,8,9], the vast majority of them are based on a linear combination of scores where the weights of the combination are computed through the maximisation of a performance function. These weight are “static”, that is they are computed once from a training set, and they are equal for all the patterns of the application we are dealing with. In this paper we propose a formulation of linear score combination that is “dynamic” in the sense that the computed weights for the combination are not fixed for all the patterns, but they vary according to the pattern to be classified. The proposed formulation allows devising a number of combination techniques, that can be both unsupervised and supervised. In addition, this formulation is general enough so that typical “static” combination rules can be considered as special cases.
A Supervised and Unsupervised Score Combination Method
165
The paper is organised as follows: Section 2 describes the proposed formulation, while some consideration concerning its generality are reported in Section 3. Different techniques based on supervised and unsupervised estimation of the parameters are presented in Section 4. Section 5 reports the experimental phase aimed at assessing and comparing the performances of the proposed methods. Finally, conclusions are outlined in Section 6.
2
Dynamic Score Combination
Let E = {E1 , E2 , . . . Ej . . . EN } be a set of N experts and X = {xi } be the set of patterns. Let also fj (·) be the function associated to expert Ej that produces a score for each pattern xi , sij = fj (xi ). In order to improve the performance of a pattern recognition system, a linear combination of the scores produced by the N experts for each pattern xi can be employed [5] N s∗i = αj · sij (1) j=1
where the values of the weights αj are usually computed by maximising some performance measure (e.g. AUC, error rates, etc.) [5,6,8]. To this end, a training set is used, and the resulting weights are fixed for all the patterns to be classified. This approach can be considered as “suboptimal” in the sense that the weights are computed so that the performance are increased on average. On the other hand, an “optimal” solution should produce a set of weights for each pattern. This formulation leads to the following equation: s∗i =
N
αij · sij
(2)
j=1
if we were able to compute the weights αij the result of the combination should better fit the pattern xi . Thus, ideally the weights αij can be adapted both to the characteristics of the pattern, and to the performance of the experts for that pattern (“dynamic” combination), instead of having a set of weights αj based on the global performance of the experts on the training samples (“static” combination). It is easy to see that in the case of Equation 1 we face the problem of estimating the value of N variables, while in the case of Equation 2 we face the problem of estimating N · M variables, where M is the number of patterns to be classified. As a consequence, it is easy to see that the high number of variables in the latter formulation makes it hard to devise a solution to the weights’ estimation problem. In the following, we propose an equivalent formulation of the problem, with a reduced number of variables to estimate. Let us consider the “dynamic” linear combination of experts in Equation 2, with N experts E, and let the scores for pattern xi be sorted from the smaller to the larger: si1 ≤ si2 ≤ · · · ≤ siN
166
R. Tronci, G. Giacinto, and F. Roli
The formula in Equation 2, can be built in a incremental way. If we consider the first three experts, we have the following expression αi1 si1 + αi2 si2 + αi3 si3 the previous expression is equivalent to αi1 si1 + si2 + αi3 si3 and taking into account that si2 is a constant. we have the following equivalent expression αi1 si1 + αi3 si3 If we consider four experts we have αi1 si1 + αi2 si2 + αi3 si3 + αi4 si4 and, according to the previous reasoning, this is equivalent to αi1 si1 + αi2 si2 + αi4 si4 and if we iterate the reasoning we obtain that the above expression is equivalent to α i1 si1 + αi4 si4 It can be seen that if we apply the same reasoning for all the N experts we then obtain that the dynamic combination is equivalent to: βi1 · min(sij ) + βi2 · max(sij ) , ∀ xi j
j
Consequently, Equation 2 can be rewritten as s∗i = βi1 · min(sij ) + βi2 · max(sij ) j
(3)
j
If we further add the constrain that the weights should sum to one, then Equation 3 can be rewritten in the following form: s∗i = (1 − βi ) · min(sij ) + βi · max(sij ) j
j
, ∀ xi
(4)
This constrain is inherited from linear combination. A consequence of this constrain is that the result of the combination is in the range [minj (sij ), maxj (sij )]. On the other hand, if this constrain is not added, the result of the combination can be a generic real value. Let us consider Equation 4 from now on. As stated before, the aim of combination is to find the ideal combined score values s∗i for each pattern xi to be classified. Then, taking into account Equations 2 and 4 the following relations hold N αij · sij = s∗i j=1
A Supervised and Unsupervised Score Combination Method
167
(1 − βi ) · min(sij ) + βi · max(sij ) = s∗i N βi =
j
j=1
j
αij · sij − minj (sij )
maxj (sij ) − min(sij )
, ∀ xi
(5)
The last relation implies that exists an optimal value of the weight βi that allows obtaining the ideal combined score value s∗i .
3
General Nature of the Dynamic Score Combination
In this section we focus on showing the generality of Equation 4, while in the next section we are going to present some methods to compute the value of βi . We named the formulation expressed in Equation 4 as Dynamic Score Combination (DSC) s∗i = (1 − βi ) · min(sij ) + βi · max(sij ) j
j
, ∀ xi
as the weights are computed dynamically for each pattern. This formulation can be considered a general “linear” score combination function. It is easy to see that this formulation embeds the following types of combination: – the “static” score linear combination: weights can be estimated either in supervised or unsupervised way, and it is a particular case of the “dynamic” linear score combination. This can be easily proven from the expressions reported in the previous section s∗i =
N
αj · sij
j=1
and for each pattern xi the βi have to be computed N j=1 αj · sij − minj (sij ) βi = , ∀ xi maxj (sij ) − min(sij ) – the Sum/Mean rule: this is a particular case of a “static” linear combination, where the weights are equal to a constant value s∗i =
N j=1
sij
or s∗i =
N 1 sij N j=1
and for each pattern xi the value of βi is given by the following expressions N j=1 sij − minj (sij ) sum βi = maxj (sij ) − min(sij ) N 1 j=1 sij − minj (sij ) N mean βi = maxj (sij ) − min(sij )
168
R. Tronci, G. Giacinto, and F. Roli
– the Max rule: when βi = 1, the maximum score is used s∗i = Max{sij } – the Min rule: when βi = 0, the minimum score is used s∗i = Min{sij } – the Dynamic Score Selection (DSS): the selection of one of the available scores from N experts for each pattern xi is ideally performed by choosing between the maximum and the minimum scores for a positive or negative pattern respectively [7]. This choice can be made using an estimation of the probability of the pattern to belong to the positive class. As an alternative, to account for the uncertainty in the estimation of the probability, the following expression can be used to obtain a score in the range [minj (sij ), maxj (sij )] [10] s∗i = P (pos|xi ) · max(sij ) + P (neg|xi ) · min(sij ) (6) j
j
Moreover, if we does not limit the value of the combined score in the range [minj (sij ), maxj (sij )], we can use a different interval such as the “ideal” values of scores for a positive and negative pattern (e.g. 1 and 0 respectively). Consequently, Equation 4 can be modified by using the ideal values for positive and negative patterns in place of the maximum and the minimum scores, respectively. Finally, Equation 3 can be used to obtain combined scores that take values in .
4
Ways to Estimate βi
The main problem with Equation 4 is to estimate the value of βi for each pattern xi . As it was shown in the previous section, Equation 4 embeds different combination methods that suggest different possible strategies of estimating βi . One way to estimate βi is that of using the Dynamic Score Selection (DSS) approach illustrated in Equation 6, thus attaining a supervised Dynamic Score Combination. It is worth noting that βi = P (pos|xi ) in this case. As illustrated in [10], a classifier C is trained on a training set made up of patterns represented on the space of scores produced by the N experts. Afterwards, this classifier is used to estimate the P (pos|xi ) for each pattern xi , and finally the combined score is computed according to Equation 6. We will refer to this formulation both as Dynamic Score Selection, and as supervised Dynamic Score Combination. The previous procedure requires that a training set is available. Unfortunately this training set is not always available, or a classifier can’t be used due to requests of low computational time and complexity. One way to estimate βi is to obtain the P (pos|xi ) (or βi ) in an unsupervised way. According to [1] the P (pos|xi ) of a combined system can be computed using the Mean rule, the Max rule, or the Min rule. When dealing with similarity scores, these rules are usually used directly on the scores instead of on the probability estimations. Thus, we propose to compute the βi for each pattern xi using the following rules:
A Supervised and Unsupervised Score Combination Method
– the Mean rule:
169
N 1 βi = sij N j=1
– the Max rule: βi = max{sij } – the Min rule: βi = min{sij } in this way the values of βi are “dynamic” because they are computed for each pattern xi , and are unsupervised because they don’t require a training set to be estimated, and can be computed on the fly.
5
Experiments
The aim of the following experiments is to show that the proposed formulation of the Dynamic Score Combination allows attaining good performance with respect to other combination techniques. When dealing with scores, one of the most relevant application of two-class classifiers is the biometric authentication. Thus, the experiments have been performed using a multi-modal biometric authentication dataset. The dataset is the Biometric Scores Set Release 1 of the National Institute of Standards and Technology1 (NIST). This dataset contains similarity scores from 4 experts related to face and fingerprint recognition system. The scores were normalised using the Min-Max rule. In the experiments we compared the Dynamic Score Combination (DSC) based on the Mean rule, the Max rule, and the Min rule, with the related “fixed” rules, to evaluate the performance of the proposed formulation when the value of βi is estimated in an unsupervised way. Moreover, results are also compared with the ideal score selector [7], the optimal “static” linear combiner, a “static” linear combination whose weights are computed by Linear Discriminant Analysis (LDA) [11,12], and the Dynamic Score Selection (DSS) [7,10], where the value of βi is estimated using a classifier. The classifiers used for the DSS are the k-Nearest Neighbour (k-NN), the Linear Discriminant Classifier (LDC), the Quadratic Discriminant Classifier (QDC) and the Support Vector Machine (SVM). The performance of the optimal “static” linear combiner are estimated by performing an exhaustive search on the value of the combination weight αj . The dataset was subdivided into 4 subsets: one subset at a time was used as the training set (for the classifiers and LDA), and the other three as the test set. In Tables 1 and 3 the performance are assessed in terms of average and standard deviation of Area Under the ROC Curve (AUC), Equal Error Rate (EER), and the d . The AUC is computed using the Wilcoxon-Mann-Whitney (WMW) statistic [13] n+ n− pos pos neg 1 sp,j > sneg p=1 q=1 I(sp,j , sq,j ) pos neg q,j AU C = , I(sp,j , sq,j ) = neg 0 spos n+ · n− p,j < sq,j 1
Web site: http://www.itl.nist.gov/iad/894.03/biometricscores/
170
R. Tronci, G. Giacinto, and F. Roli
where n+ is the number of positive patterns and n− is the number of negative patterns. The EER is the point of the ROC where the false positive rate and the false negative rate are equal. This measure is largely used in the biometric field, the smaller the EER, the better the performance. The d is a measure of the separation of the distributions of positive and negative score, the larger the d , the higher the separation [12,14] |μpos − μneg | d = 2 2 σpos σneg 2 + 2 In Tables 2 and 4 performance are assessed in terms of False Matching Rate (FMR, i.e. the false positive rate) and False Not Matching Rate (FNMR, i.e. the false negative rate) where F M Rj (th) = P (sij > th|xi ∈ negative) F N M Rj (th) = P (sij ≤ th|xi ∈ positive) In particular, the tables report the F M R 1% (F N M R 1%) values, that are the FNMR (FMR) values when F M R = 1% (F M R = 0%), and the F M R 0% (F N M R 0%) values, that are the FNMR (FMR) values when the F M R = 0% (F M R = 0%).
Table 1. Average and standard deviation of AUC, EER and d’ for the combination methods in Finger-Face dataset using all possible score combinations of 3 experts Ideal Score Selector Optimal Linear Combiner Mean Max Min DSC Mean DSC Max DSC Min LDA DSS k-NN DSS ldc DSS qdc DSS svm
AUC 1.0000(±0.0000) 0.9997(±0.0004) 0.9982(±0.0013) 0.9892(±0.0022) 0.9708(±0.0085) 0.9986(±0.0011) 0.9960(±0.0015) 0.9769(±0.0085) 0.9945(±0.0040) 0.9987(±0.0016) 0.9741(±0.0087) 0.9964(±0.0039) 0.9996(±0.0004)
EER 0.0000(±0.0000) 0.0050(±0.0031) 0.0096(±0.0059) 0.0450(±0.0048) 0.0694(±0.0148) 0.0064(±0.0030) 0.0214(±0.0065) 0.0634(±0.0158) 0.0296(±0.0123) 0.0104(±0.0053) 0.0642(±0.0149) 0.0147(±0.0092) 0.0048(±0.0026)
d’ 25.4451(±8.7120) 3.1231(±0.2321) 3.6272(±0.4850) 3.0608(±0.3803) 2.0068(±0.1636) 3.8300(±0.5049) 3.8799(±0.2613) 2.3664(±0.2371) 2.3802(±0.2036) 6.9911(±0.9653) 2.7654(±0.2782) 9.1452(±3.1002) 4.8972(±0.4911)
Tables 1 and 2 show the average performance attained for all the possible combinations of three experts, and for all the 4 subdivisions of the dataset. The Tables show that the proposed DSC formulation, where the values of βi are estimated by the Mean rule (DSC Mean), outperforms the Mean rule in terms of the AUC, the EER, the d’, the FMR 1%, and the FNMR 1%. The DSC
A Supervised and Unsupervised Score Combination Method
171
Table 2. Average and standard deviation of FMR 1%, FNMR 1%, FMR 0%, and FNMR 0% for the combination methods in Finger-Face dataset using all possible score combinations of 3 experts Ideal Score Selector Optimal Linear Combiner Mean Max Min DSC Mean DSC Max DSC Min LDA DSS k-NN DSS ldc DSS qdc DSS svm
FMR 1% 0.0000(±0.0000) 0.0027(±0.0047) 0.0090(±0.0075) 0.0851(±0.0118) 0.1044(±0.0229) 0.0048(±0.0035) 0.0311(±0.0102) 0.0909(±0.0232) 0.0483(±0.0270) 0.0100(±0.0067) 0.0969(±0.0225) 0.0160(±0.0129) 0.0034(±0.0034)
FNMR 1% 0.0000(±0.0000) 0.0025(±0.0037) 0.0175(±0.0245) 0.2304(±0.0841) 0.6994(±0.1220) 0.0028(±0.0036) 0.0746(±0.0680) 0.5159(±0.1603) 0.1300(±0.1026) 0.0354(±0.0563) 0.6461(±0.1195) 0.1140(±0.1760) 0.0019(±0.0023)
FMR 0% 0.0000(±0.0000) 0.0741(±0.0328) 0.0530(±0.0210) 0.5055(±0.0839) 0.2471(±0.0643) 0.0553(±0.0249) 0.3516(±0.0884) 0.2261(±0.0613) 0.1575(±0.0867) 0.0791(±0.0391) 0.2079(±0.0561) 0.3906(±0.1591) 0.0551(±0.0210)
FNMR 0% 0.0000(±0.0000) 0.0920(±0.1232) 0.4899(±0.4020) 0.8172(±0.2128) 0.9964(±0.0093) 0.5002(±0.4168) 0.7133(±0.3363) 0.9725(±0.0533) 0.5325(±0.3085) 0.2481(±0.3227) 0.9788(±0.0346) 0.3800(±0.3001) 0.1142(±0.1217)
where the values of βi are estimated on the Max, and the Min rule, outperform the Max and the Min rule, respectively, in terms of all considered measures of performance. In particular, the larger improvement is exhibited by the DSC formulation of the Max rule. It is also worth noting that the DSC Mean, and the DSC Max generally obtain better performance than those obtained by LDA, that is a trained linear combiner. Thus the proposed unsupervised Dynamic Combination mechanism allows outperforming supervised linear combination mechanisms.
Table 3. Average and standard deviation of AUC, EER and d’ for the combination methods in Finger-Face dataset using all possible score combinations 4 experts Ideal Score Selector Optimal Linear Combiner Mean Max Min DSC Mean DSC Max DSC Min LDA DSS k-NN DSS ldc DSS qdc DSS svm
AUC 1.0000(±0.0000) 1.0000(±0.0000) 0.9996(±0.0003) 0.9898(±0.0026) 0.9700(±0.0054) 0.9996(±0.0002) 0.9957(±0.0016) 0.9739(±0.0048) 0.9977(±0.0019) 0.9998(±0.0001) 0.9760(±0.0051) 0.9991(±0.0008) 0.9998(±0.0002)
EER 0.0000(±0.0000) 0.0017(±0.0010) 0.0045(±0.0013) 0.0427(±0.0035) 0.0742(±0.0080) 0.0052(±0.0019) 0.0207(±0.0030) 0.0716(±0.0060) 0.0200(±0.0068) 0.0071(±0.0025) 0.0632(±0.0098) 0.0097(±0.0044) 0.0040(±0.0024)
d’ 31.1087(±0.6621) 3.3325(±0.3414) 3.9444(±0.0599) 3.0807(±0.1952) 1.9082(±0.0421) 3.9402(±0.0763) 3.7414(±0.1634) 2.1957(±0.0564) 2.5846(±0.0686) 8.3496(±1.0404) 2.9149(±0.2305) 11.4145(±2.9062) 5.5261(±0.3032)
172
R. Tronci, G. Giacinto, and F. Roli
Table 4. Average and standard deviation of FMR 1%, FNMR 1%, FMR 0%, and FNMR 0% for the combination methods in Finger-Face dataset using all possible score combinations of 4 experts Ideal Score Selector Optimal Linear Combiner Mean Max Min DSC Mean DSC Max DSC Min LDA DSS k-NN DSS ldc DSS qdc DSS svm
FMR 1% 0.0000(±0.0000) 0.0000(±0.0000) 0.0039(±0.0026) 0.0800(±0.0122) 0.1122(±0.0121) 0.0039(±0.0015) 0.0322(±0.0074) 0.1044(±0.0119) 0.0251(±0.0108) 0.0058(±0.0039) 0.0980(±0.0160) 0.0097(±0.0061) 0.0032(±0.0025)
FNMR 1% 0.0000(±0.0000) 0.0003(±0.0003) 0.0001(±0.0001) 0.2096(±0.1180) 0.6942(±0.0925) 0.0002(±0.0002) 0.0463(±0.0177) 0.5359(±0.1129) 0.0743(±0.0578) 0.0010(±0.0011) 0.6042(±0.0686) 0.0061(±0.0069) 0.0001(±0.0000)
FMR 0% 0.0000(±0.0000) 0.0354(±0.0226) 0.0251(±0.0044) 0.5442(±0.0509) 0.2186(±0.0107) 0.0348(±0.0065) 0.4198(±0.0672) 0.2070(±0.0167) 0.0825(±0.0075) 0.0554(±0.0194) 0.1941(±0.0352) 0.4011(±0.1851) 0.0264(±0.0013)
FNMR 0% 0.0000(±0.0000) 0.0014(±0.0007) 0.1525(±0.0985) 0.9060(±0.1821) 0.9981(±0.0038) 0.1424(±0.0820) 0.8435(±0.3075) 0.9779(±0.0442) 0.2342(±0.2240) 0.0426(±0.0188) 0.9923(±0.0003) 0.1759(±0.1204) 0.0745(±0.0626)
Tables 3 and 4 show the results obtained using all the possible combinations of the four experts at time. The average and standard deviation of the results are related to the 4 subdivisions of the dataset. The behaviour of the DSC Max and the DSC Min is the same of that obtained in the previous experiments. On the other hand, in this case the performance of the DSC Mean, and that of the Mean rule are nearly the same. In Figure 2 a graphical comparison between the DSC based on the Mean, Max, and the Min rule and the Mean, Max, and Min rule themselves is reported. Results are related to all the possible combinations of 3 and 4 experts among the 4 subdivisions. In Figure 2a the AUC obtained using the DSC formulation is compared to the AUC obtained using the fixed rules. The graph reports the number of times that the AUC obtained with an unsupervised DSC is greater then the AUC obtained with the correspondant “fixed” rule. In Figure 2b an analogous graph compares the EER obtained using the DSC to the EER obtained using the fixed rules. These figures clearly show that the Dynamic Score Combination performs generally better than the “fixed” rules. Figure 3 shows the same comparison as in Figure 2, where the performance are evaluated in terms of FMR 1% (Fig. 3a), FNMR 1% (Fig. 3b), FMR 0% (Fig. 3c), and FNMR 0% (Fig. 3d). These graphs clearly show that the DSC formulations based on the Max, and the Min rule allow to obtain better performance than those of the “fixed” rules also when these performance measures are considered. This statement holds also in the case of Mean rule when the FMR 1% and FMR 0% are taken into account (Figures 3a and 3c), while when the FNMR 1% and FNMR 0% are considered the performance are nearly the same. Figures 4 and 5 show the comparison between the DSC based on the Mean rule (the best unsupervised DSC) and the Dynamic Score Selection based on k-NN, LDC, QDC, and SVM (the supervised version of the DSC). As for the
A Supervised and Unsupervised Score Combination Method
AUC
>= AUC
AUC
> AUC
0.7
0.8
DSC Fixed
173
Fixed
DSC
Min
Max
Mean
0
0.1
0.2
0.3
0.4
0.5
0.6
0.9
1
(a)
EER
<= EER
EER
< EER
0.7
0.8
DSC Fixed
Fixed
DSC
Min
Max
Mean
0
0.1
0.2
0.3
0.4
0.5
0.6
0.9
1
(b) Fig. 2. Comparison between the DSC based on “fixed” rules and the “fixed” rules themselves in terms of AUC (a), and the EER (b). In black are reported the number of times that the AUC, or the EER, obtained with the DSC, outperform those obtained using the related “fixed” rule, while in white is reported the opposite case.
previous Figures, the graphs show the number of times were the DSC Mean outperforms the DSS. In Figure 4a the AUC obtained using the DSC Mean is
174
R. Tronci, G. Giacinto, and F. Roli
compared to the AUC obtained using the DSS. The graph shows that the DSC Mean is pretty equivalent to the DSS k-NN, and the DSS QDC. Moreover the DSC Mean outperforms the DSS LDC, while it is outperformed by the DSS SVM. In Figure 4b the comparison is shown in terms of EER. In this case the DSC Mean outperforms all the DSS methods, but the one based on SVM. In Figure 5 the DSC Mean outperforms the DSS based on k-NN, LDC, and QDC, but not the DSS based on SVM. By analysing the Tables 1-4, and the Figures 4 and 5 it is clear that the supervised version of the Dynamic Score Combination (the DSS) does not always outperform the unsupervised ones. In particular, if the DSC Mean is considered, the DSS LDC always exhibits lower performance, the DSS k-NN and DSS QDC provide similar performance, while the SVM outperforms all other methods. These results allows to conclude that the advantages of going supervised strongly depend on the classifier chosen.
FMR
DSC
1% <= FMR
Fixed
1%
FNMR
DSC
FMRFixed 1% < FMRDSC 1% Min
Min
Max
Max
Mean
Mean
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
(a)
0.5
0.6
DSC
0% <= FMR
Fixed
0%
DSC
Min
Max
Max
Mean
Mean
0.3
0.4
0.5
(c)
1%
0.8
0.9
1
0.6
0.7
0.8
0.9
0% <= FNMR
Fixed
0%
FNMRFixed 0% < FNMRDSC 0%
Min
0.2
0.7
FNMR
FMRFixed 0% < FMRDSC 0%
0.1
Fixed
(b) FMR
0
1% <= FNMR
FNMRFixed 1% < FNMRDSC 1%
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(d)
Fig. 3. Comparison between the DSC based on “fixed” rules and the “fixed” rules themselves in terms of FMR 1% (a), FNMR 1% (b), FMR 0% (c), and FNMR 0% (d). In black are reported the number of times that the DSC outperforms the related “fixed” rule, while in white is reported the opposite case.
A Supervised and Unsupervised Score Combination Method
175
SVM AUC
>= AUC
AUC
> AUC
0.7
0.8
DSC DSS
DSS
DSC
QDC
LDC
k−NN
0
0.1
0.2
0.3
0.4
0.5
0.6
0.9
1
(a)
SVM EER
<= EER
EER
< EER
DSC DSS
DSS
DSC
QDC
LDC
k−NN
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(b) Fig. 4. Comparison between the DSC based on the Mean rule and the DSS in terms of AUC (a), and the EER (b). In black are reported the number of times that the AUC, or the EER, obtained with the DSC Mean is better than that obtained using the DSS, while in white is reported the opposite case.
176
R. Tronci, G. Giacinto, and F. Roli
SVM
SVM FMR
DSC
1% <= FMR
DSS
1%
FNMR
DSC
FMRDSS 1% < FMRDSC 1%
QDC
QDC
LDC
LDC
k−NN
k−NN
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1% <= FNMR
DSS
1%
FNMRDSS 1% < FNMRDSC 1%
1
0
0.1
0.2
0.3
0.4
(a)
0.5
0.6
0.7
0.8
0.9
1
(b)
SVM
SVM FMR
DSC
0% <= FMR
DSS
0%
FNMR
DSC
FMRDSS 0% < FMRDSC 0%
QDC
QDC
LDC
LDC
k−NN
k−NN
0
0.1
0.2
0.3
0.4
0.5
(c)
0.6
0.7
0.8
0.9
0% <= FNMR
DSS
0%
FNMRDSS 0% < FNMRDSC 0%
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(d)
Fig. 5. Comparison between the DSC based on the Mean rule and the DSS in terms of FMR 1% (a), FNMR 1% (b), FMR 0% (c), and FNMR 0% (d). In black are reported the number of times that the DSC Mean outperforms the DSS, while in white is reported the opposite case.
6
Conclusions
In this paper we propose a general formulation of the linear combination of scores called Dynamic Score Combination. This formulation is general because it embeds into itself different formulations: the “static” linear combination, the “dynamic” linear combination, and the Dynamic Score Selection. Moreover this formulation can be both used in supervised and unsupervised mode. The experiments show that the unsupervised use is generally preferable with respect to some “fixed” combination rules with a similar computational complexity. Moreover, when the supervised techniques are considered, their performance strongly depends on the classifier used, and in many cases they are very close to those provided by the best unsupervised Dynamic Score Combination.
A Supervised and Unsupervised Score Combination Method
177
In conclusion the proposed formulation can be exploited both in supervised, and unsupervised mode. The unsupervised DSC based on the Mean rule has the advantages to have low computational time and complexity (comparable to those of a “fixed” combination rule). The supervised DSC (the DSS), depending on the classifier used, can allow obtaining better performance, but with an increase of computational time and complexity.
References 1. Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998) 2. Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons, Inc., Chichester (2004) 3. Bradley, A.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30(7), 1145–1159 (1997) 4. Fawcett, T.: An introduction to ROC analysis. Pattern Recognition Letters 27(8), 861–874 (2006) 5. Marrocco, C., Duin, R., Tortorella, F.: Maximizing the area under the ROC curve by pairwise feature combination. Pattern Recognition 41(6), 1961–1974 (2008) 6. Ross, A., Nandakumar, K., Jain, A.: Handbook of Multibiometrics. Springer, Heidelberg (2006) 7. Tronci, R., Giacinto, G., Roli, F.: Dynamic Score Selection for Fusion of Multiple Biometric Matchers. In: ICIAP 2007, pp. 15–20 (2007) 8. Nandakumar, K., Chen, Y., Dass, S., Jain, A.: Likelihood ratio-based biometric score fusion. IEEE Trans. on PAMI 30(2), 342–347 (2008) 9. Marcialis, G., Roli, F.: Fusion of multiple fingerprint matchers by single-layer perceptron with class-separation loss function. Pattern Recognition Letters 26(12), 1830–1839 (2005) 10. Tronci, R., Giacinto, G., Roli, F.: Dynamic score combination of binary experts. In: 19th International Conference on Pattern Recognition (ICPR 2008), pp. 1–4. IEEE Computer Society, Los Alamitos (2008) 11. Ross, A., Jain, A.K.: Information fusion in biometrics. Pattern Recognition Letters 24, 2115–2125 (2003) 12. Duda, R., Hart, P., Stork, D.: Pattern Classification. John Wiley & Sons, Inc., Chichester (2001) 13. Hanley, J., McNeil, B.: The meaning and the use of the area under a receiver operanting charateristic curve. Radiology 143, 29–36 (1982) 14. Tronci, R., Giacinto, G., Roli, F.: Selection of experts for the design of multiple biometric systems. In: Perner, P. (ed.) MLDM 2007. LNCS (LNAI), vol. 4571, pp. 795–809. Springer, Heidelberg (2007)
ODDboost: Incorporating Posterior Estimates into AdaBoost Olga Barinova and Dmitry Vetrov Moscow State University, dept. of Computational Mathematics and Cybernetics, 119992 Moscow, Russia [email protected], [email protected]
Abstract. Boosting methods while being among the best classification methods developed so far, are known to degrade performance in case of noisy data and overlapping classes. In this paper we propose a new upper generalization bound for weighted averages of hypotheses, which uses posterior estimates for training objects and is based on reduction of binary classification problem with overlapping classes to a deterministic problem. If we are given accurate posterior estimates, proposed bound is lower than existing bound by Schapire et al [25]. We design an AdaBoost-like algorithm which optimizes proposed generalization bound and show that incorporated with good posterior estimates it performs better than the standard AdaBoost on real-world data sets.
1 Introduction Boosting is one of the best-performing classification algorithms developed so far, in experimental comparison of classifiers performed by Caruana et al. [6] it demonstrated the best result. There are many different views on the nature of boosting, but the most theoretically solid explanation of boosting generalization capabilities is margin theory [25]. Margin theory provides an upper generalization bound independent of number of iterations made by boosting. This bound suggests that boosting may not overfit even if ran for many rounds. However, different researchers reported that boosting is very sensitive to noise and suffers from overfitting after considerable amount of iterations, although much less then one should expect from such general model [9][16]. It is still an open question why boosting actually overfits in practice even with constant complexity base learner (for example, stump). In this paper we consider the problem of worsening performance of boosting in case of binary classification problems with overlapping classes. By overlapping classes we mean that for a considerable amount of instances x both objects ( x, +1)
and ( x, −1) can occur with positive probabilities. We propose a method for reduction of overlapping classes’ problem to deterministic problem, where each instance x is associated with a label с ( x ) . We also introduce a new upper generalization bound for weighted averages of hypotheses based on this reduction, which is independent on number of iterations made by boosting. Proposed bound has a similar form to the one provided by margin theory [25], but uses estimates of posterior probabilities for P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 178–190, 2009. © Springer-Verlag Berlin Heidelberg 2009
ODDboost: Incorporating Posterior Estimates into AdaBoost
179
training samples as an input. When incorporated with accurate posterior estimates, it has a lower extra term comparing to upper bound provided by margin theory. Then we propose an AdaBoost-like algorithm, which is proven to minimize the introduced upper bound. We incorporate developed algorithm with posterior estimation method proposed by Vezhnevets et al. [29] and perform experiments on real-world datasets from UCI repository [3]. This paper is organized as follows. In section 2. we consider related work, in section 3. new upper-bounds for weighted averages of hypotheses are introduced. Section 4. describes an AdaBoost-like algorithm which was designed specially for minimizing proposed generalization bound. In section 5. we describe an algorithm for estimation of posterior probabilities for training instances, which uses only training set. Section 6 shows experimental results comparing proposed algorithm with standard AdaBoost on real-world datasets. The last section is left for conclusion.
2 Related Work A large body of research addresses the problem of explaining and avoiding the phenomenon of boosting overfitting. Domingos [10] showed the relation between margin theory explanation of boosting and bias-variance explanation. He also made an interesting statement that reducing variance (increasing margins) is beneficial only for unbiased samples, while for biased samples it is preferable to have high variance (lower margin). Biased sample is a sample, for which the optimal prediction, for a given loss and the family of classifiers, differs from its current label. Many works were devoted to explaining and avoiding degradation of boosting performance in case of noisy datasets. Several authors reported that boosting tends to increase the weights of few hard-to-learn samples, which leads to overfitting. Several modifications of the reweighting scheme were proposed that make weights change more smoothly. Domingo et al. [11] propose a modification of AdaBoost in which the weights of the examples are kept bounded by its initial value, however the authors admit no significant difference between AdaBoost and its modification in experiments on noisy data. Friedman [15] suggests that shrinking the weights of base hypothesis would increase boosting generalization capability. The property of concentrating on few hard-to-learn patterns can be interpreted in terms of margin maximization; this view leads to regularized modifications. Ratsch et al. [22] view boosting as minimization of cost functional through an approximate gradient descent with respect to a margin. The authors propose regularization methods to achieve “soft margins” for AdaBoost which should avoid boosting from concentrating on misclassified samples with large negative margins. Friedman [15] also considers regularization methods through introducing proportional shrinkage into gradient boosting. Other authors see unboundness of loss function as the major source of overfitting. Viewing boosting as a gradient descent search for a good fit in function space allows modifying loss functions. Mason et al. [19] views boosting as gradient descent on an appropriate cost functional in a suitable inner product space. Proposed modification of boosting algorithm with normalized sigmoid cost function was reported to outperform
180
O. Barinova and D. Vetrov
AdaBoost when boosting decision stumps, particularly in the presence of label noise. Friedman [15] considers Huber loss function. Rosset [24] proposes an approach of weight decay for observation weights which is equivalent to "robustifying" the underlying loss function. However the author admits that in experiments on real-world data there is no consistent winner between the nondecayed and the decayed versions of boosting algorithm. A large body of research addresses learning in the presence of classification noise. Krause & Singer [18] suggest employing EM algorithm and changing loss function to make boosting tolerant to known level of noise. Takenouchi & Eguchi [27] develop a modification of AdaBoost for classification noise case. The authors modify loss function and show that proposed method moderates the overweighting for outliers using a uniform weight distribution. Other researchers see the main source of boosting overfitting not in an inappropriateness of particular loss function, but in general concept of average loss minimization and illustrate it’s inappropriateness for the case of overlapping classes [29]. The authors use posterior estimates for eliminating samples form the training set which belong to opposite class with greater probability than to their own class. They show that this approach can lead to better boosting generalization performance and lower test error. In this work we extend their approach by estimating the probabilities of the objects to belong to each class. We then use these probabilities for recomputing the weights of training objects in order to make boosting procedure give more attention to the object which classification is confident and less attention to the objects whose classification is unclear (i.e. the probabilities to belong to each class are close to 0.5). The closest work to ours is the work by Schapire et al. [27]. The authors describe a modification to boosting algorithm that permits the incorporation of prior human knowledge as a means of compensating for a shortage of training data. They base on a boosting-style algorithm by Collins et al. [7] and modify target distribution in order to capture both class labels of the training samples and their posterior estimates. They show that this modification can lead to better classification performance. In fact there are many cases when class probabilities for each object of the training set are known. Imagine a situation when class labels are assigned to training objects by more than one expert. Then the rate of experts who voted that the given object belongs to the first class can be considered as a probability that the object belongs to the first class. Taking into account this additional information may potentially improve boosting performance by weakening the influence of the objects for which the experts have different opinions. In the paper we develop the framework for incorporating this knowledge into standard Discreet AdaBoost algorithm [12]. We show that such extension leads to new generalization bounds which are tighter than the ones derived in classical case. Furthermore we suggest a way how the proposed approach can be used even in cases when we do not have any prior knowledge about probabilities to belong to each of the classes. Our experiments show that in general such boosting incorporated with posterior estimates tends to outperform AdaBoost on real-world problems.
ODDboost: Incorporating Posterior Estimates into AdaBoost
181
3 Proposed Generalization Bounds Let T = ( xi , yi ) , i = 1,..., n be the training set, where xi ∈ X is the vector of attributes and yi ∈ {−1, +1} is the class label. We consider binary classification problems. We take the assumption of statistical learning framework, and assume that the pairs (x,y) are random variables distributed according to an unknown distribution P ( x, y ) . Later we’ll use a notion of margin of an object (x,y) relative to classifier f(x), which is by definition a value of yf(x). Support vector machine (SVM) was the first classifier which exploited largemargin approach [0]. Boosting was shown to have large margin properties only afterwards. The application of margin theory to boosting introduced by Schapire et al. in [25] still remains the most theoretically solid explanation for boosting generalization capabilities. Below we remind the most important results of margin theory. 1 1 m m Let’s denote PS [ χ ( x, y )] = ∑ i =1 I [ χ ( x , y )] and ES [ z ( x, y )] = ∑ i =1 z ( xi , yi ) . i i m m Theorem (Schapire et, al) Upper generalization bound based on cumulative distribution of margins Let P be a distribution over X × {+1, −1} and S be a sample of m examples chosen independently at random according to P . Assume that the base-classifier space H is finite and let δ > 0 . Then with probability at least 1 − δ over the random choice of training set S , every weighted average of hypotheses f where weights sum up to one, satisfies the following bound for all θ > 0 :
P [ yf ( x) ≤ 0] ≤ PS [ yf ( x) ≤ θ ] + ε ( m,| H |, θ , δ ) , 1/ 2 ⎛ 1 ⎛ log m log | H | ⎞ ⎞ where ε ( m,| H |,θ , δ ) = O ⎜ log(1/ ) δ + ⎟ ⎟⎟ ⎜ m ⎜⎝ θ2 ⎠ ⎠ ⎝
This theorem states that classification error of any weighted average of hypotheses can be upper-bounded by the value of cumulative margins distribution on training set plus an extra term, which depends on the number of training objects, number of weak hypotheses and two user-defined constants. The most important property of this extra term is it’s independence of the number of averaged hypotheses and their coefficients. If we are given a fixed training set for calculating this upper bound, the extra term is constant. Thus we can govern the value of the bound by choosing hypotheses and coefficients in weighted average in order to minimize the first term. AdaBoost algorithm [25] was proven to do exactly the right thing. Theorem (Schapire et, al) Large margin properties of AdaBoost Suppose the base learning algorithm, when called by AdaBoost, generates classifiers with weighted training errors ξ1 ,..., ξT . Then for any θ , we have that T
PS [ yf ( x) ≤ θ ] ≤ 2T ∏ ε t1−θ (1 − ε t )1+θ . t =1
182
O. Barinova and D. Vetrov
In this work, following the logic of margin theory, we introduce a new generalization bound for weighted averages and propose AdaBoost-like algorithm for minimization of this new bound. Below is given a sketch of considerations leading to proposed generalization bound. First work on boosting [12] operated with PAC-learning framework [17] for binary classification problem. Unlike statistical learning where both -1 and +1 class labels can be associated with an instance x ( p(+1|x) and p(-1|x) can be both positive), PAClearning assumes that label of each instance x is defined by target function c(x) and is fixed. Freund & Schapire [13] in their discussion on Friedman’s paper suggest that for overlapping classes (when Bayesian error is not zero) “AdaBoost is not the optimal method in this case”. It should be noted that in real world applications it is a rare case if Bayesian error is zero and classes are perfectly separable due to imperfect feature vector representation of objects, limitations of measuring equipment and noise. Our main idea is to bring boosting back from statistical learning to PAC learning framework. Further we will need the following notation: pmax ( x ) max ( p ( +1 | x ), p ( −1 | x ) ) pmin ( x ) min ( p ( +1 | x ), p ( −1 | x ) )
E y ( z ( x, y ) ) = p ( +1 | x ) × z ( x, +1) + p ( −1 | x ) × z ( x, −1) . The next statement shows how any statistical learning problem can be reduced to an equivalent PAC-learning problem. Here we operate with true distributions. Theorem 1 Let P be a distribution over X × {+1, −1} . Consider distribution
P ( x ) = ( pmax ( x ) − pmin ( x ) ) P ( x ) / (1−2 B ) and target function c ( x ) = arg max y ( p ( +1 | x ), p ( −1 | x ) ) . Consider classifier space H . Then for any classifier f(x)holds an inequality: P [ yf ( x ) ≤ 0] ≤ B + (1 − 2 B ) × P [c ( x) f ( x ) ≤ 0] ,
where B stands for the error of Bayesian classifier. And thus solutions of statistical learning problem with distribution P and PAClearning problem with distribution P ( x ) and target function c ( x ) coincide: A rg min H P ( h ( x )≠ c ( x ) ) = A rg min H P ( h ( x )≠ y ) Proof: The proof of this and the following theorems can be found in [0]. Using this fact, previous theorem and generalization bound from Schapire et al. we can upperbound classification error of any weighted average using a sample S from modified distribution P . Theorem 2 Let P be a distribution over X × {+1, −1} and let P stand for corresponding distribution and c(x) for corresponding target function in PAC-learning framework. Let S
ODDboost: Incorporating Posterior Estimates into AdaBoost
183
be a sample of m examples chosen independently at random according to P where labels are chosen according to c(x). Assume that the base-classifier space H is finite and let δ > 0 . Then with probability at least 1 − δ over the random choice of training set S , every weighted average of hypotheses f where weights sum up to one, satisfies the following bound for all θ > 0 : P [ yf ( x ) ≤ 0] ≤ B + (1 − 2 B ) × PS [c ( x ) f ( x ) ≤ θ ] + (1 − 2 B ) × ε ( m, | H |, θ , δ ) ,
where B is an error of Bayesian classifier over the distribution ε ( m, | H |, θ , δ ) is taken from Schapire’s theorem. This bound looks much like Schapire’s one but the extra term here is smaller. The problem is that instead of sample S from modified distribution P we are given only a sample S from distribution P . But still we can approximate cumulative distribution of margins on sample S by weighted cumulative distribution of margins on sample S . This approximation causes some increase of extra term, but still it remains smaller than the one from Schapire’s theorem. Noticing also that the condition c ( x ) f ( x ) < θ implies that E y ( yf ( x ) ) < θ , we obtain theorem 3. Theorem 3 Let P be a distribution over X × {+1, −1} Let S be a sample of m examples chosen independently at random according to P . Assume that the base-classifier space H is finite and let δ > 0 . Then with probability at least 1 − δ over the random choice of training set S , every weighted average of hypotheses f where weights sum up to one, satisfies the following bound for all θ > 0 : ⎛ p ( x) − pmin ( x) ⎞ P [ yf ( x) ≤ 0] ≤ B + (1 − 2 Bˆ ) × ES ⎜ max I[c ( x ) f ( x ) ≤θ ] ⎟ + ε ( m,| H |,θ , δ ) 1 − 2 Bˆ ⎝ ⎠ 1 m where B stands for the error of Bayesian classifier, Bˆ = ∑ pmin ( xi ) , m i =1
and ε ( m, | H |, θ , δ ) = ε ( m,| H |, θ , δ ) − 2 B
log | H |
compared to the term in
m
Schapire’s theorem. This upper bound can be calculated only in case when exact values of posteriors for every training example are known. This limits practical importance of theorem 3. The following theorem states the upper bound of classification error for the case when only rough estimates of posteriors are given. Theorem 4 Let P be a distribution over X × {+1, −1} Let S be a sample of m examples chosen independently at random according to P . Assume that the base-classifier space H is finite and let δ > 0 . Suppose we are given estimates of posteriors for all training samples, that obey a
condition pˆ max ( xi ) − pmax ( xi ) < α , i = 1,..., M .
184
O. Barinova and D. Vetrov
Then with probability at least 1 − δ over the random choice of training set S , every weighted average of hypotheses f where weights sum up to one, satisfies the following bound for all θ > 0 : 2α ⎛ pˆ − pˆ ⎞ + ε ( m,| H |, θ − β (α ), δ ) P [ yf ( x ) < 0] ≤ B + (1 − 2 B) × ES ⎜ max min I[ Eˆ yf ( x )<θ + β (α )] ⎟ + y ⎝ 1 − 2B ⎠ 1− 2B
where B stands for the error of Bayesian classifier, 1 m B = ∑ pˆ min ( xi ) , m i =1
β (α ) is a continuous function of α , and ε ( m, | H |, θ , δ ) = ε ( m,| H |, θ , δ ) − 2 B
log | H |
compared to the term in
m
Schapire’s theorem. Theorems 3 and 4 provide new upper generalization bounds for weighted average of hypotheses. These bounds are closer to the true error than the bound provided by margin theory, although not significantly. It is known that lower values of generalization bounds usually correspond to better classifier performance even if those bounds are loose [4]. This fact allows expecting to achieve better classifier performance when minimizing proposed generalization bounds instead of conventional generalization bound provided by margin theory. Bounds from theorem 3 and 4 cannot be calculated directly because they require knowledge of Bayesian error on distribution P. However theorem 4 allows deriving new AdaBoost-like algorithm which reduces generalization upper bound by first estimating posteriors with further minimization of second term in upper bound while keeping all other terms unchanged. The corresponding algorithm is described in the next section.
4 ODDboost Algorithm Following the line of margin theory we designed an algorithm that takes a training sample and posteriors for each sample and minimizes the value of the bound from Theorem3 (if exact posteriors are known) or the bound from Theorem 4 (if we know only rough estimates of posteriors). The following theorem shows that this algorithm minimizes upper generalization bounds from theorem 3 and theorem 4. Theorem 5 Suppose that proposed algorithm, when called by ODDboost, generates classifiers m
with weighted training errors ξ1 ,..., ξT : ξt = ∑ Dt (i ) p ( y ≠ ht ( xi ) | xi ) . Then for any
θ , we have that
i =1
ODDboost: Incorporating Posterior Estimates into AdaBoost
185
T ⎛ ( pˆ ( x ) − pˆ min ( x) ) ⎞ ES ⎜ max I ⎡ E ( yf ( x ) ) ≤θ ⎤ ⎟ ≤ 2T ∏ ε t1−θ (1 − ε t1+θ ) y ⎣ ⎦ 1 − 2B t =1 ⎝ ⎠
ODDboost algorithm 1) Estimate posteriors for each training instance pˆ (+1| xi ), i = 1,..., N using training set pˆ ( x ) − pˆ min ( xi ) 2) Assign weights to training instances D1 (i ) = max i , i = 1,..., M 1 − 2B 3) For t = 1: T a) Obtain weak hypothesis ht ( x) ∈ {−1, +1} , using weights Dt (i ) and posterior
estimates pˆ (+1 | xi ) and pˆ (−1 | xi ) by minimizing m
ε t = ∑ Dt (i ) p ( y ≠ ht ( xi ) | xi ) i =1
M
b) Calculate et ( x) = ∑ Dt (i ) p ( y ≠ ht ( xi ) | xi ) i =1
c)
Calculate weight of current weak learner α t =
d) Update weights of training instances Dt +1 (i ) =
1 ln ( (1 − et ) / et ) 2 Dt (i ) × Ε y ( exp(− yα t ht ( xi ) ) Zt
,
M
where Z t = ∑ Dt +1 (i ) i =1
T
4) Final classifier sign∑ α t ht ( x) t =1
ODDboost algorithm has much in common with standard AdaBoost, the main differences are the following. While in original AdaBoost initial weights of training examples are equal, in proposed algorithm they are taken proportional to a value ( pˆ max ( x) − pˆ min ( x) ) , this allows to concentrate on must-learn examples, which belong
to arg max y ( pˆ ( y | x) ) class with high confidence.
Second, in proposed algorithm weak learner is trained using posterior estimates and m
ε t = ∑ Dt (i ) p ( y ≠ ht ( xi ) | xi ) is minimized instead of simply minimizing training error i =1 m
ε t = ∑ Dt (i ) I[ y ≠ h ( x )| x ] made by AdaBoost. This property can be useful because incorpoi =1
i
t
i
i
rating posterior estimates in this step can help to choose better weak hypotheses. The last difference between ODDboost and AdaBoost is in the formula for updating weights of training instances. AdaBoost multiplies weight of a training instance by exp(− yiα t ht ( xi ) and thus uses a label of training object from training set, ODDboost
186
O. Barinova and D. Vetrov
updates training weights by multiplying them by Ε y ( exp(− yα t ht ( xi ) ) and thus takes into account the fact that training instance can belong to each class with known probabilities. This property helps ODDboost to avoid concentrating on hard-to-learn patterns, which AdaBoosting is known to do. It’s worth to mention that AdaBoost and ODDboost have the same computational complexity. In approach presented by Schapire et al. [27] a new training set with weight over example-label pairs is created. This new training set contains three copies of each training instance with different weights and different labels. Each training instance xi is replicated twice as ( xi , +1) and as ( xi , −1) . In their work each replicate of training example has a weight wl = η pˆ (l | xi ) , where l is an assigned class label of a new object, η is a user-defined parameter of the algorithm, and pˆ (l | xi ) is an estimated conditional probability of a class l given feature vector xi . We could view our algorithm in a similar way and say that ODDboost algorithm includes two copies of each training instance into теуц training set and set initial weights of training instances equal to wl = pˆ (l | xi ) ( pˆ max ( xi ) − pˆ min ( xi ) ) 1 − 2 B ,
(
)
1 m where B = ∑ pˆ min ( xi ) . So our method differs from the one proposed by Scahpire m i =1
et al. in a way of initializing training examples weights. Another difference is that we modify the formula for weights update while in [27] this step is kept unchanged.
5 Posteriors Estimation Our goal is to roughly estimate the conditional probabilities of training set samples labels and to incorporate them into proposed boosting procedure. We used a method for posteriors estimations proposed in [29]. Its general idea is based on the use of Platt scaling [20] to obtain calibrated probabilities from boosting. Platt’s scaling method fits the sigmoid into raw classifier output, thus approximating posterior as: P ( y | x) ≈
1 1 + exp( A ⋅ F ( x) + B )
where constants A and B are set to maximize likelihood of special calibration set. To estimate posteriors we construct an iterative process; during iterations we randomly split data into 3 parts. We train a boosted committee on the first part, calibrate probabilities on the second and estimate posterior for samples labels in the third part (using the built classifier and estimated calibration parameters). We repeat this procedure several times. At each step we acquire an estimate of class probabilities for training instances. Posterior estimates for every sample are averaged. In [29] the authors experimented with Real AdaBoost algorithm [26] and used posterior estimates for eliminating noisy samples from the data. In this work we modify Discrete AdaBoost [12] in order to incorporate posterior estimates into the algorithm. Averaging of estimates was extensively studied in context of regression tasks [28]. Perrone [21] proved that averaging estimates always generates improved estimate in
ODDboost: Incorporating Posterior Estimates into AdaBoost
187
the sense of any convex optimization measure even without independence assumption on the estimates. Brieman [5] experimented with averaging class probability estimates obtained by learning decision trees on bootstrap replicates of the training set. He showed that averaging decreases error of class probability estimation.
6 Experiments In order to study the behavior of proposed ODDboost algorithm we have conducted a set of experiments on real world data. We compare the performance of AdaBoost and ODDboost with stumps as a base learner. Stumps were chosen as base learners to avoid possible issues with base learner complexity [23]. Table 1. Test error (%) of AdaBoost and ODDboost trained on various data sets Dataset
100 iterations
1 iteration
AdaBoost
ODDboost
AdaBoost
ODDboost
13.2±0.5
14.2±0.2
12.1±0.1
16.0±0.1
13.1±0.1
15.0±0.5
15.0±0.5
14.5±0.2
13.0±0.1
17.4±0.1
13.0±0.1
BREAST
8.5±0.3
7.5±0.3
4.5±0.1
4.2±0.0
5.4±0.0
4.4±0.1
CHECKER9
47.0±0.4
46.0±0.4
28.8±0.4
27.5±0.2
29.5±0.4
27.0±0.2
GERMAN
39.0±0.3
38.0±0.3
23.7±0.2
26.6±0.2
25.7±0.3
25.0±0.1
MADELON
37.5±0.3
37.6±0.3
40.1±0.5
38.7±0.2
41.5±0.3
38.0±0.1
MAGIC04
26.4±0.2
26.4±0.2
17.0±0.5
17.6±0.2
17.9±0.1
16.4±0.1
PAGE
6.3±0.1
6.4±0.1
3.4±0.2
3.6±0.1
3.3±0.1
2.9±0.1
PIMA
26.3±0.3
25.2±0.3
25.5±0.5
24.2±0.2
26.2±0.1
24.7±0.2
LETTER T
31.9±0.3
31.8±0.3
15.0±0.2
14.1±0.1
16.2±0.2
14.6±0.0
VOTE
4.4±0.2
4.4±0.2
3.7±0.2
4.3±0.2
4.6±0.2
4.3±0.2
AUSTRA AUSTRALIA
Total :
AdaBoost
ODDboost
13.2±0.5
1000 iterations
2
1
6
12
1
14
We used the following experimental strategy. First we estimated posterior probabilities by running 100 iterations of procedure described in section 5. We used AdaBoost classifier as initial classifier for posteriors estimation. The obtained posteriors were then used as inputs for ODDboost algorithm. We measured its 5x2 cross-validation error [8] after 1, 100, and 1000 iterations respectively. In order to measure the performance of our algorithm on real world data we have selected 11 datasets from UCI-repository [3]. In our experiments we split the dataset
188
O. Barinova and D. Vetrov
in two subsets of equal size and use one subset for training and the other for testing and vise-versa. We measured the test error. This procedure was repeated 5 times for a total of 10 runs (5x2 cross-validation). To compare our approach with standard boosting we run AdaBoost algorithm and estimated its error rate in a similar way.
Fig. 1. Crossvalidation error depending on number of iterations made by boosting. From top left to bottom right: test error on Australian, pima-indians-diabetes, madelon and breast cancer datasets. Error of standard AdaBoost are shown in red and errors of proposed ODDboost are shown in blue.
Table 1 presents the results of our experiments. The best overall result is shown in bold, the second-best is shown in italic. The last line shows overall scores: if algorithm performed the best 2 is added to it’s score, second-best result adds 1 to the the score. These results allow us to make the following conclusions: ODDboost generally performs better than AdaBoost both after 100 and 1000 iterations. As the number of learning iterations increases, AdaBoost tends to overfit on most datasets, while ODDboost still keeps high generalization ability. The typical test errors w.r.t. the number of iterations for AdaBoost and ODDboost are shown in figure 1. It can be seen that ODDboost tends to be more stable. We suppose that it happens because we deliberately reduce the weights of noisy objects thus concentrating mostly on reliable objects which do not lead to overfitting. The reweighting of training objects can be considered as a special kind of regularization which can potentially be used for any classification algorithm.
ODDboost: Incorporating Posterior Estimates into AdaBoost
189
On the other hand the obvious drawback of such approach is its requirement of posterior estimates at the time of training. Posterior estimates are not always available for training samples so special procedure for estimation of posteriors is needed. Not only this is difficult problem by itself, but it also requires more computational efforts.
7 Conclusion We considered the problem of poor performance of boosting in case of binary classification problems with overlapping classes. We introduced a new upper generalization bound for weighted averages of hypotheses based on reduction of overlapping classes’ problem to a deterministic problem. Introduced bound doesn’t depend on number of iterations made by boosting. Proposed bound uses estimates of posterior probabilities for training samples as an input and has a similar form to the upper bound provided by margin theory. When incorporated with accurate posterior estimates, an extra of proposed bound is lower than the one from margin theory. Based on the new upper bound we proposed ODDBoost algorithm, which is proven to minimize the second term in the bound while keeping all other terms unchanged. We incorporated developed algorithm with posterior estimation method proposed in [29] and performed experiments on real-world datasets from UCI repository, which confirm that proposed algorithm generally performs better than standard AdaBoost.
Acknowledgments Authors would like to thank D. Kropotov for useful discussions. The work was partly supported by grants of Russian Foundation for basic research (projects ##08-0100883, 08-01-00405).
References 1. Barinova, O., Vetrov, D.: ODDboost: Incorporating posterior estimates into AdaBoost, Technical Report 1422 (2009), http://graphics.cs.msu.ru/en/publications/text/oddboost.pdf 2. Boser, B., Guyon, I., Vapnik, V.: A training algorithm for optimal margin classifiers. In: Fifth Annual Workshop on Computational Learning Theory. ACM Press, Pittsburgh (1992) 3. Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998) 4. Burges, C.J.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge 2, 121–167 (1998) 5. Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1996) 6. Caruana, R., Niculescu-Mizil, A.: An Empirical Comparison of Supervised Learning Algorithms. In: Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)(2006) 7. Collins, M., Schapire, R., Singer, Y.: Logistic regression, AdaBoost and Bregman distances. Machine Learning 48(1/2/3) (2002) 8. Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1895–1923 (1998)
190
O. Barinova and D. Vetrov
9. Dietterich, T.G.: An experimental comparison of three methods for constructing ensebles of decision trees: Bagging, boosting, and randomization. Machine Learning 40(2) (1999) 10. Domingos, P.: A Unified Bias-Variance Decomposition for Zero-One and Squared Loss. In: Proc. of the 17th National Conference on Artificial Intelligence (2000) 11. Domingo, C., Watanabe, O.: Madaboost: A modication of adaboost. In: 13th Annual Conference on Comp. Learning Theory (2000) 12. Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1997) 13. Freund, Y., Schapire, R.: Discussion of the paper “Additive logistic regression: a statistical view of boosting” by J. Friedman, T. Hastie and R. Tibshirani. The Annals of Statistics 38(2), 391–393 (2000) 14. Friedman, J., Hastie, T., Tibshirani, R.: Additive Logistic Regression: a Statistical View of Boosting. The Annals of Statistics 28(2), 337–407 (2000) 15. Friedman, J.: Greedy function approximation: A gradient boosting machine. Annals of Statistics 29(5) (2001) 16. Grove, A.J., Schuurmans, D.: Boosting in the limit: Maximizing the margin of learned ensembles. In: Proceedings of the Fifteenth National Conference on Artifical Intelligence (1998) 17. Kearns, M., Vazirani, U.: An introduction to computational learning theory. MIT Press, Cambridge (1994) 18. Krause, N., Singer, Y.: Leveraging the Margin More Carefully. In: ACM International Conference Proceeding Series, vol. 69 (2004) 19. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Boosting algorithms as gradient descent. In: Neural Information Processing Systems, vol. 12, pp. 512–518. MIT Press, Cambridge (2000) 20. Niculescu-Mizil, A., Caruana, R.: Obtaining Calibrated Probabilities from Boosting. In: Proc. of 21st Conference on Uncertainty in Artificial Intelligence (2005) 21. Perrone, M.: Improving regression estimation: Averaging methods for Variance reduction with extension to General Convex Measure Optimization, Ph.D. Thesis, Brown University (1993) 22. Ratsch, G.: Robust Boosting and Convex Optimization. Doctoral dissertation, University of Potsdam (2001) 23. Reyzin, L., Schapire, R.: How boosting the margin can also boost classifier complexity. In: Proceedings of the 23rd International Conference on Machine Learning (2006) 24. Rosset, S.: Robust Boosting and Its Relation to Bagging. In: KDD 2005 (2005) 25. Schapire, R., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: A new explanation for the effectiveness of voting methods. In: Machine Learning: Proceedings of the Fourteenth International Conference (1997) 26. Schapire, R., Singer, Y.: Improved boosting algorithm using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999) 27. Schapire, R., Rochery, M., Rahim, M., Gupta, N.: Incorporating prior knowledge into boosting. In: Machine Learning: Proceedings of the Nineteenth International Conference, pp. 538–545 (2002); Takenouchi T., Eguchi S.: Robustifying AdaBoost by adding the naive error rate. Neural Computation 16 (2004) 28. Taniguchi, M., Tresp, V.: Averaging Regularized Estimators. Neural Computation (1997) 29. Vezhnevets, A., Barinova, O.: Avoiding boosting overfitting by removing confusing samples. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS, vol. 4701, pp. 430–441. Springer, Heidelberg (2007)
Ensemble Learning: A Study on Different Variants of the Dynamic Selection Approach Jo˜ ao Mendes-Moreira1,2, Alipio Mario Jorge2,3, Carlos Soares2,4, and Jorge Freire de Sousa5 1
Faculdade de Engenharia, Universidade do Porto, DEI, Portugal 2 LIAAD-INESC Porto L.A., Portugal 3 Faculdade de Ciˆencias, Universidade do Porto, Portugal 4 Faculdade de Economia, Universidade do Porto, Portugal 5 Faculdade de Engenharia, Universidade do Porto, DEIG, Portugal [email protected], [email protected], [email protected], [email protected]
Abstract. Integration methods for ensemble learning can use two different approaches: combination or selection. The combination approach (also called fusion) consists on the combination of the predictions obtained by different models in the ensemble to obtain the final ensemble prediction. The selection approach selects one (or more) models from the ensemble according to the prediction performance of these models on similar data from the validation set. Usually, the method to select similar data is the k-nearest neighbors with the Euclidean distance. In this paper we discuss other approaches to obtain similar data for the regression problem. We show that using similarity measures according to the target values improves results. We also show that selecting dynamically several models for the prediction task increases prediction accuracy comparing to the selection of just one model.
1
Introduction
Ensemble learning is an important research field in different research communities. It consists in the use of a set of prediction models, instead of just one, to accomplish the prediction task. The interest in ensemble learning is mainly due to the improvement of accuracy and robustness when compared to the use of just one model [10]. It has two main phases: ensemble generation and ensemble integration. When the same induction algorithm is used to generate all the models of the ensemble, it is said that the ensemble generation is homogeneous, otherwise it is called heterogeneous. Ensemble integration can use two different approaches [16]: combination (also called fusion) or selection. The former consists in the combination of the predictions obtained by the different models in the ensemble to obtain the final prediction. The selection approach, in particular its dynamic version, selects one (or more) models from the ensemble according to the prediction performance of these models on similar data from the validation set. In this paper we focus on the dynamic selection approach which is relatively less popular. Well known ensemble methods such as bagging [3], boosting [9], P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 191–205, 2009. c Springer-Verlag Berlin Heidelberg 2009
192
J. Mendes-Moreira et al.
random forests [4], among others, use the combination approach. However, there are some indications that dynamic selection can obtain better results [29,27] than combination using the above mentioned ensemble methods. In this paper we present a study on dynamic selection, testing some alternatives to the usual selection methods described in the literature. We test algorithms (never previously used in the dynamic selection framework) that measure similarity between data according to the outputs. We also compare different integration functions in order to investigate whether there are benefits from selecting more than one predictor from the ensemble comparing to the selection of only one predictor. We start by reviewing the related work on the selection approach. Then, we propose alternative methods and present the experiments done, followed by a discussion on the results. Our results show that using similarity measures according to the target values improves results as well as using several models for the prediction task instead of just one.
2
Dynamic Selection
The assumption behind the use of dynamic selection is that each model/predictor in the ensemble has different prediction abilities on different subsets of the input space. What follows assumes that base models are already available as well as the estimates of the errors of these models on the validation data as described by Merz [19]. Dynamic selection (Figure 1) consists of the following tasks:
validation data
(1) Find similar data
similar data
(2) Select model(s)
F
fˆ1 , fˆ2 ,..., fˆk1 x
(3) Model(s) prediction(s)
(4) Ensemble prediction
fˆ1 (x), fˆ2 (x),..., fˆk1 (x)
fˆ f ( fˆ1 (x), fˆ2 (x),..., fˆk1 (x), vd , sd )
Fig. 1. The dynamic selection process
1. given an input value x, select similar data from the validation set; 2. select the model(s) from the ensemble according to their performance on the selected similar data; 3. predict fˆi (x) for the given input value, for each selected model i; 4. obtain the ensemble prediction fˆf . This is straightforward if just one model is selected, otherwise, it is necessary to combine results.
Ensemble Learning: A Study on Different Variants
193
F = {fˆi , i = 1, ..., k} represents the ensemble of k predictors, and k1 represents the number of selected predictors from the ensemble. vd and sd are, respectively, the validation data and the selected data. Dynamic selection typically uses three different data sets: the training set to obtain the k base predictors of the ensemble F ; the validation set to assess base predictors’ generalization error; and the test set to assess ensemble predictor’s generalization error. While task (3) is straightforward, the others are not. In this section we describe each one of the remaining three tasks, with particular emphasis on the methods used to obtain similar data. 2.1
Obtaining Similar Data
The standard method for obtaining similar data (task 1) in the context of ensemble learning is the well known k-nearest neighbors algorithm with Euclidean distance [30]. One limitation of this method is that it weighs all input variables equally even if there are input variables with different levels of relevance in the explanation of the target variable. If the data set has many variables with a small impact on the target variable and a small subset of dominant input variables, this problem can significantly affect the results. Didaci & Giacinto propose the use of DANN - Discriminant Adaptive Nearest Neighbor [12], which locally reshapes the nearest neighborhood according to the target values. In practice, some of the explanatory variables are discarded, reducing the dimensionality of the problem. Experiments done by Didaci & Giacinto show that this approach as well as a dynamic choice of the number k of neighbors can improve meaningfully the results when compared with the standard Euclidean distance [7]. In the context of random forests [4] there are also solutions that measure similarity according to the target values [28]. The k-nearest examples are the k examples with the highest proportion of trees from the forest where the example appears in the same leaf node as the test example. All these approaches use the predictions of the base models for each example in the training set, as it is fully described in [19]. 2.2
Selecting the Models
The simplest method of selecting models (task 2) is to pick the one with the best performance according to a given metric [30,11]. However, the dynamic selection approach can use more than one model [19]. The dynamic voting with selection [29] and its regression version, called dynamic weighting with selection [24], use half of the models from the ensemble, selecting the ones which are locally (i.e., on the selected examples) more accurate. 2.3
Combining the Predictions of Base Models
When more than one model is selected, their results are combined (task 4). The dynamic voting (weighting) methods [22,29,24] obtain the final prediction by a
194
J. Mendes-Moreira et al. Table 1. Data sets Data set Diabetes
Nr of examples Nr of input vars 43
2
Pyrimidines
74
27
Breast cancer
194
32
Machine CPU
209
6
Boston housing
506
13
weighted average of the base models’ predictions, where the weights are proportional to the accuracy on similar data. The dynamic voting with selection gives better results than simple dynamic voting, for both regression and classification problems. Many other methods can be used to combine results [20].
3
Our Study
In this section we study different approaches on ensemble selection in order to identify major advantages/ disadvantages of each approach. We only consider the regression problem. In all the experiments, we use the same set of base predictors, the same regression data sets obtained from the repository of Torgo [26] (Table 1), and the same experimental setup. All the variables in the data sets are numerical. All the experiments were carried out using the R-project [25] statistical package. The ensemble F (see Figure 1) has four models: – – – –
the linear model (function lm); CART (function rpart), using the default parameters; 1-nearest neighbor (function nn); 20-nearest neighbor (function nn).
For the construction of the ensemble, the only concern was to select models that are expected to make diverse errors. This motivated the use of 1 and 20 nearest neighbors. While with 1 nearest neighbor the prediction is unstable (i.e., small changes in the data may cause significant changes to the predictions), with 20 neighbors it is much smoother [32]. In our experimental setup we use two cross-validation cycles, illustrated in Figure 2. In the inner cycle, which is part of the training phase, we estimate the performance of the base-level algorithms and generate the ensemble. In the outer cycle, which is part of the testing phase, we estimate the performance of the ensemble. Firstly, the data is randomly split in 10 parts of approximately the same size. Then, a 10-fold cross-validation cycle (i.e., the outer cycle) is carried out, using nine subsets (training set ) to build the ensemble and the other subset (test set ) to estimate the performance of the ensemble. The generation of the ensemble consists of a 9-fold cross-validation cycle (i.e., the inner cycle) on the
Ensemble Learning: A Study on Different Variants
1
2
3
4
Folds 5 6
7
8
9
195
10 1st run 2nd run 3rd run 4th run 5th run 6th run 7th run 8th run 9th run 10th run
This process is repeated using each fold (cell) as test set. Training set Validation set
Test set Not used
Fig. 2. Experimental setup
nine subsets that are provided as training set. Out of these nine subsets, eight are used to obtain the base models and the other subset (validation set ) to estimate their performance. The process is repeated by using each of the nine subsets as validation set. Finally, the models that are part of the ensemble are obtained by applying the base algorithms to the whole training set and the ensemble is evaluated on the test set. The outer cycle is repeated by using each of the 10 subsets as test set. The reason for using 10-fold cross-validation is the small number of examples in some of the data sets used, which would make the performance estimates obtained using a simpler method (e.g., simple hold-out) unreliable [8]. The main advantage is to train the base predictors with more examples (it can be critical for small data sets) but it has the disadvantage of increasing the computational cost. The process can be repeated several times (we do it ten times) to obtain different sample values for the evaluation criterion. We evaluate the performance of the models using the variation index, calculated as the ratio between the squared root of the mean squared error (mse) and the average of the corresponding target values, 1 ˆ (f (xi ) − f (xi ))2 , n i=1 √ mse Variation Index = . y n
mse =
(1) (2)
Its main advantage is to express the dispersion metric as a percentage of the average, which simplifies interpretation. Using the described experimental setup, three different experiments were executed each one of them with a different purpose:
196
J. Mendes-Moreira et al.
– The first experiment compares different similarity searching approaches; – The second one tests the effect of the number of similar examples; – And the last experiment compares different approaches to accomplish tasks 2 and 4, i.e., different methods for both model selection (task 2) and model combination (task 4).
3.1
Testing Methods to Obtain Similar Data
The first experiment tests different methods for the selection of examples that are similar to the test case (task 1). We tested four methods. The last three are tested in the dynamic selection framework for the first time. The first three use the k-nearest neighbor (knn) algorithm (ten nearest neighbors were used in all the three methods) but with different distance functions. The four methods we have tested are: – Knn using the Euclidean distance (knn-euc): the Euclidean distance is obtained from the function dist of the R [25] statistical package. – Knn using the kd-tree (knn-kdt): the kd-tree [2] is a generalization of the binary search tree. Just one input variable is tested at each split. It is particularly suited for high-dimensional data sets. Its main advantage is the low computational cost since it reduces the search space with little accuracy loss. We use the function nn from the library knnF inder [13] of the R-project [25]. This is an implementation of the approximate nearest neighbor algorithm [1]. This algorithm differs from the original kd-tree because instead of returning the k nearest neighbors, it returns an approximate number of k, more precisely k1 nearest neighbors where k1 ≤ k. This approach reduces meaningfully the computational cost with a low loss in accuracy. – Knn using the RReliefF algorithm (knn-RRF): the RReliefF algorithm [23] (see details in appendix A) weighs each one of the input variables. These weights are used to obtain a weighted distance attribute. The array of weights W returned by the RReliefF algorithm (one weight for each input variable) is used to calculate the distance between the test example and each one of the examples in the training set using a weighted average. The disadvantage of RReliefF is the high computational cost when compared with the other two distance functions. – CART’s leaf node examples (CART-ln): the fourth method to obtain similar data uses the well known CART algorithm [5]. After obtaining a tree with the CART algorithm on the training set, we apply the tree on the test example. The selected data are all the examples from the leaf node where the test example falls. We have used, as a CART implementation, the rpart function in R [25]. The pseudo-code for this method is, program leaf.node.approach(trainingSet, testExample) begin tree := rpart(trainingSet) leafNode := findleafnode(tree, testExample)
Ensemble Learning: A Study on Different Variants
197
members := getmembers(tree, leafNode) return(members) end. An important limitation of this method is that there is not an upper bound to the number of similar examples; it is possible to define a minimum number of examples in the leaf nodes, but not a maximum. This problem can be solved, but implies additional effort, such as, using a distance measure to select k examples from the leaf node, or by controlling in a different way the pruning process during the construction of the tree. In the first method, knn-euc, the distance between the examples only takes into account the value of the independent variables, ignoring the values of the target. In the remaining three methods, the distance between examples is affected also by the value of the target variable in the training examples. For instance, RReliefF weighs the input variables according to how much they explain the target values for the selected neighbors. For model selection (task 2) and model combination (task 4), we tested the following alternatives: – Selection of the model which obtains the best performance on the selected examples. In this case, only a single model is selected, so there is no need for model combination. The prediction of the ensemble is simply the prediction of the selected model. – Forward selection with replacement (FSwR) [6]. This method iteratively selects the model that, when included in the ensemble, maximizes the reduction in the error on the selected examples. It stops when performance of the ensemble does not improve further on the validation set. It is important to note that, although the predictions of the models are combined using a simple average (task 4), in practice it is not, because the same model can be selected more than once, so the weight of each model depends on the number of times this model is selected [6]. Results on Methods to Obtain Similar Data. Each one of the methods for model selection is tested with each one of the methods to obtain the final prediction. Results obtained by the base predictors (LM, CART, nn-1, nn-20) and the simple average of the base predictions (mean) are also presented for baseline comparison (Table 2). The ratio of the time each method takes to obtain similar data when compared with knn-euc is presented in Table 3. Several conclusions can be taken from the results: – Contrary to previous results [29,28], ours indicate that dynamic selection does not improve results when compared to a simple combination method of base models, such as the mean prediction. However, our results do show that dynamic selection obtains better overall results than any of the individual base-models. – By comparing the dynamic selection methods among themselves, we observe that although there is no absolute winner, in general, knn-RRF obtains the
198
J. Mendes-Moreira et al.
Table 2. The variation index (Eq. 2) using different methods to obtain similar data Base models Dynamic selection LM CART nn-1 nn-20 mean CART-ln knn-euc knn-kdt knn-RRF Diabetes 0.133 0.133 0.177 0.136 0.131 0.140 0.139 0.133 0.133 Best 0.139 0.137 0.133 0.131 FSwR Pyrim 0.261 0.166 0.143 0.162 0.149 0.171 0.170 0.169 0.166 Best 0.164 0.168 0.168 0.161 FSwR R wpbc 0.728 0.824 0.971 0.706 0.717 0.743 0.756 0.756 0.749 Best 0.731 0.731 0.731 0.736 FSwR M-CPU 0.654 0.932 0.587 0.592 0.591 0.745 0.743 0.706 0.725 Best 0.706 0.705 0.697 0.678 FSwR Housing 0.217 0.215 0.318 0.316 0.206 0.300 0.292 0.289 0.284 Best 0.294 0.280 0.279 0.274 FSwR
Table 3. Relative time compared to the knn with the Euclidean distance
Diabetes Pyrim R wpbc M-CPU Housing
CART-ln knn-euc knn-kdt knn-RRF 1.4 1.0 1.6 660.3 1.4 1.0 1.7 54072.6 1.5 1.0 1.9 27072.2 2.5 1.0 2.1 1297.7 0.6 1.0 1.3 1106.6
best results and knn-CART obtains the worst ones. We also observe that knn-kdt and knn-RRF obtain comparable or better results than knn-euc, although with some extra computational cost, particularly in the latter case. – By comparing the combination methods, these results indicate that the use of forward selection with replacement is systematically better than selecting the best model, although the differences are not very large; – Additionally, the simple combination method (mean) is the best overall method, although in some cases another algorithm may obtain slightly better results. It gives the best results when all the base predictors are reasonably accurate. This observation provides partial support to the ambiguity decomposition where it is stated that the base predictors must be accurate and with uncorrelated errors [15]. 3.2
The Effect of the Number of Similar Examples
The aim of the following experiment is to study of the effect of the neighborhood size on the final accuracy (Tables 4 and 51 ). We used just one method to select the similar data for the dynamic selection approach (task 1), the k-nearest neighbor with the Euclidean distance. This experiment uses as neighborhood size k ∈ 1
Tables 2 (left part), 5 and 7 contain results of the same set of methods on the same data sets, the values are slightly different because each experiment uses a different seed for pseudo-random number generation.
Ensemble Learning: A Study on Different Variants
199
Table 4. The variation index (Eq. 2) using different sizes of the similar data
2-nn Diabetes 0.147 0.145 Pyrim 0.174 0.168 R wpbc 0.800 0.787 M-CPU 0.735 0.718 Housing 0.285 0.279
4-nn 0.145 0.141 0.168 0.165 0.780 0.764 0.712 0.691 0.284 0.277
6-nn 0.140 0.139 0.163 0.160 0.763 0.752 0.734 0.722 0.289 0.280
8-nn 0.139 0.137 0.179 0.162 0.757 0.742 0.788 0.755 0.290 0.282
Dynamic Selection 10-nn 12-nn 14-nn 16-nn 0.136 0.136 0.133 0.133 0.134 0.134 0.133 0.135 0.161 0.161 0.161 0.160 0.161 0.160 0.158 0.158 0.748 0.745 0.740 0.725 0.734 0.732 0.723 0.720 0.794 0.781 0.797 0.814 0.753 0.732 0.746 0.752 0.292 0.293 0.295 0.296 0.282 0.282 0.282 0.286
18-nn 0.135 0.135 0.161 0.160 0.719 0.717 0.817 0.753 0.295 0.287
20-nn 25-nn 0.136 0.135 0.136 0.134 0.162 0.163 0.161 0.161 0.719 0.716 0.712 0.712 0.829 0.837 0.737 0.756 0.296 0.303 0.288 0.289
30-nn 0.135 0.135 0.163 0.162 0.714 0.710 0.862 0.770 0.305 0.293
Best FSwR Best FSwR Best FSwR Best FSwR Best FSwR
Table 5. The variation index (Eq. 2) using different sizes of the similar data (continuation)
LM Diabetes 0.132 Pyrim 0.249 R wpbc 0.727 M-CPU 0.672 Housing 0.217
Base models CART nn-1 nn-20 0.133 0.180 0.135 0.166 0.146 0.163 0.813 0.970 0.706 0.954 0.589 0.596 0.213 0.320 0.314
mean 0.131 0.149 0.714 0.616 0.206
{2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 25, 30}. The same two methods for the local choice of the predictors (task 2) are also used in this experiment as well as the apparent simple average for task 4 when more than one model is selected in task 2. Like in the first experiment, the average without dynamic selection is also presented for baseline comparison, as well as the results using the base models. The best neighborhood size, k, as well as the sensitivity to its value, are problem dependent, as expected. For each data set, the best value of k is the same for both tested methods, even if without statistical validation. 3.3
Testing Different Methods for Tasks 2 and 4
The goal of this section is to test different methods for: the selection of the models (task 2); and the combination of the predictions, when more than one model is selected (task 4). Here we consider a more extensive set of methods than the ones tested in Sect. 3.1. The methods selected previously obtained good results for both regression [24] and classification [29,28]. For task 1 we use, for each data set, the two most promising combinations according to the previous experiments, because our goal is to identify the methods that obtain the best possible performance for each data set. From the first set of experiments we select a data selection method for each data set and for each one of the two methods of choosing the models (Best and FSwR). From
200
J. Mendes-Moreira et al.
the second set of experiments we select the best value of k. When the CART-ln approach is used for data selection, the k value is determined automatically, as explained earlier. The methods we describe next accomplish both tasks 2 and 4: – Best and FSwR, already described in Sect. 3.1; – Dynamic weighting (DW), already mentioned in Sect. 2: it uses the vector w of weights (appendix B presents how w is calculated), and the matrix sqe with the squared error of the models for each similar data point, in order to obtain the weight for each model [22,24] according to: √ I 1 (w ∗sqei,k ) i=1 i wmk = K √ I 1 k1=1
,
(3)
i=1 (wi ∗sqei,k1 )
where i is the example’s index, k and k1 are the model’s indexes, and K is the ensemble size; – Dynamic weighting with selection (DWS) selects a subset of the models before executing DW. The selected models are the ones with a mean squared error not higher than a percentage threshold related to the best model. A similar approach is already known [29,24] but for a fixed threshold. We test 5 different values for the threshold: 10%, 30%, 50%, 70% and 90%. Results are presented in Tables 6 and 7. The best methods for each data set are in bold face. If more than one method is in bold face it means that the paired t-test does not reject the null hypothesis, i.e., the difference is not significantly different for α = 0.05. The cells with numbers formatted like 0.000 instead of 0.000, are the ones where both the KS-Lilliefors normality test [17] and the skewness test for small samples [31] were rejected also for α = 0.05. Consequently, for these cells, the paired t-test was not done. More runs of the experimental setup were needed. Several conclusions can be drawn: – DW and DWS are clearly better than the two methods used in earlier experiments for tasks 2 and 4. The results obtained by dynamic selection with the former two methods are also competitive with the simple combination method of base models (mean). – DW in combination with knn-RRF may be the best choice because it obtains similar or better results than DWS and has no parameters; – These results show that it is possible to improve accuracy using more than one model when compared to the use of just the best model on the similar data set; – It seems that there is a tendency for the knn-RRF approach to obtain better results in problems with more input variables. Although this is not surprising, it should be explored for problems with higher dimension.
Ensemble Learning: A Study on Different Variants
201
Table 6. The variation index (Eq. 2) using different methods for tasks 2 and 4
Best FSwR Diabetes 0.135 0.136 Pyrim 0.166 0.164 R wpbc 0.734 0.716 M-CPU 0.706 0.782 Housing 0.277 0.272
0.136 0.136 0.166 0.161 0.732 0.710 0.694 0.771 0.271 0.265
Dynamic selection DWS 10% 30% 50% 70% 90% 0.132 0.131 0.131 0.131 0.131 0.137 0.136 0.134 0.133 0.134 0.164 0.155 0.151 0.149 0.149 0.158 0.152 0.148 0.146 0.146 0.736 0.708 0.706 0.708 0.709 0.709 0.707 0.709 0.711 0.711 0.606 0.567 0.559 0.556 0.553 0.720 0.672 0.660 0.661 0.669 0.258 0.232 0.220 0.215 0.212 0.252 0.230 0.221 0.217 0.213
DW 0.131 0.133 0.149 0.146 0.711 0.711 0.539 0.630 0.207 0.208
knn-kdt-14 knn-RRF-14 CART-ln knn-RRF-16 CART-ln knn-kdt-30 knn-kdt-4 knn-RRF-4 knn-kdt-4 knn-RRF-4
Table 7. The variation index (Eq. 2) using different methods for tasks 2 and 4 (continuation)
LM Diabetes 0.132 Pyrim 0.256 R wpbc 0.728 M-CPU 0.674 Housing 0.216
4
Base models CART nn-1 nn-20 0.135 0.179 0.135 0.166 0.145 0.162 0.809 0.983 0.706 0.980 0.588 0.595 0.213 0.314 0.314
mean 0.132 0.149 0.717 0.622 0.205
Conclusions and Future Research Directions
This paper describes a study on different variants of the dynamic selection approach to ensemble learning. In particular we focus on the following tasks: example selection (task 1 in Section 2) and model selection and combination (tasks 2 and 4). For task 1, we have introduced three different methods (CART-ln, knn-kdt and knn-RRF) in order to test whether selection methods that take into account the value of the target in the training examples yields better results in comparison to methods which are only based on the independent variables (knn with Euclidean distance). Knn-RRF [23] was the most promising one. The experiments also confirm in all the five data sets we have tested that the dynamic combination of several models is better than selecting a single one. Despite this result has been claimed previously in [14] for the classification problem, this is the first experience using regression data sets. It seems that the dichotomy “selection vs. fusion”, commonly discussed in the ensemble learning literature [16], does not make sense because it is not proved that it is better to dynamically choose just one model instead of several models (Sect. 3.1). The
202
J. Mendes-Moreira et al.
reduction in the variance achieved by averaging several results can explain why the selection of several models is better than the selection of just one [28]. Although this is not the focus of this paper, some of our results can be used to compare constant and non-constant weighting functions. For which type of problems are non-constant weighting functions more adapted than constant weighting functions? This dichotomy was often used in the past [20], but has fallen in disuse, unfortunately. Maybe this was due to the possibility of better defining the ensemble members when using constant weighting functions. It can be said that instead of choosing the best method to combine the predictions, research was driven into the generation of the best ensemble (knowing, a priori, the integration function to use: typically the simple average). Some of the most successful approaches for ensemble generation, namely bagging [3], random forests [4] and negative correlation learning [18], use the simple average for integration. Our results provide further evidence in the sense that simple weighting provides competitive results. However, a more definitive answer to the question above needs further research. The use of non-constant weighting functions in concept drifting problems [28], i.e., problems changing with time, is a possible direction of research. In the experiments described in this paper there was no particular concern with ensemble generation. The characteristics that the ensemble should have are particularly well known for regression problems when using a constant weighting function as integration method [15]. Ensembles using non constant weighting functions are much less studied. However, given the nature of these methods, an analytical decomposition of the generalization error may not be possible (because the integration function is not known since it is non-constant) but, at least empirically, it is possible to obtain some insights. A possible approach to explore in the future is meta-learning [21].
Acknowledgments This work was partially supported by Programa de Financiamento Plurianual de Unidades de I&D.
References 1. Arya, S., Mount, D.M., Netanyahu, N.S., Silverman, R., Wu, A.Y.: An optimal algorithm for approximate nearest neighbor searching. Journal of the ACM 45(6), 891–923 (1998) 2. Bentley, J.L.: Multidimensional binary search trees used for associative searching. Communications of the ACM 18(9), 509–517 (1975) 3. Breiman, L.: Bagging predictors. Machine Learning 26, 123–140 (1996) 4. Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001) 5. Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and regression trees. Chapman and Hall/CRC, Boca Raton (1984) 6. Caruana, R., Niculescu-Mozil, A., Crew, G., Ksikes, A.: Ensemble selection from libraries of models. In: International Conference on Machine Learning (2004)
Ensemble Learning: A Study on Different Variants
203
7. Didaci, L., Giacinto, G.: Dynamic classifier selection by adaptive k-nearest neighbourhood rule. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 174–183. Springer, Heidelberg (2004) 8. Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation 10, 1895–1923 (1998) 9. Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: International Conference on Machine Learning, pp. 148–156 (1996) 10. Garc´ıa-Pedrajas, N., Herv´ as-Mart´ınez, C., Ortiz-Boyer, D.: Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Transactions on Evolutionary Computation 9(3), 271–302 (2005) 11. Giacinto, G., Roli, F.: Adaptive selection of image classifiers. In: Del Bimbo, A. (ed.) ICIAP 1997. LNCS, vol. 1310, pp. 38–45. Springer, Heidelberg (1997) 12. Hastie, T., Tibshirani, R.: Discriminant adaptive nearest neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6), 607–616 (1996) 13. Kemp, S.E.: knnfinder: Fast near neighbour search. R package version 1.0 14. Ko, A.H.-R., Sabourin, R., Britto Jr., A.d.S.: From dynamic classifier selection to dynamic ensemble selection. Pattern Recognition 41, 1718–1731 (2008) 15. Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Advances in Neural Information Processing Systems, vol. 7, pp. 231– 238 (1995) 16. Kuncheva, L.I.: Switching between selection and fusion in combining classifiers: an experiment. IEEE Transactions on Systems, Man, and Cybernetics-Part B 32(2), 146–156 (2002) 17. Lilliefors, H.W.: On the kolmogorov-smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association 62(318), 399– 402 (1967) 18. Liu, Y., Yao, X., Higuchi, T.: Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation 4(4), 380–387 (2000) 19. Merz, C.J.: Dynamical selection of learning algorithms. In: Fisher, D., Lenz, H.J. (eds.) International Workshop on Artificial Intelligence and Statistics. Learning from Data: Artificial Intelligence and Statistics, vol. V. Springer, Heidelberg (1996) 20. Merz, C.J.: Classification and regression by combining models. Phd thesis, University of California, USA (1998) 21. Prudˆencio, R.B.C., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137 (2004) 22. Puuronen, S., Terziyan, V., Tsymbal, A.: A dynamic integration algorithm for an ensemble of classifiers. In: Ra´s, Z.W., Skowron, A. (eds.) ISMIS 1999. LNCS, vol. 1609, pp. 592–600. Springer, Heidelberg (1999) ˘ 23. Robnik-Sikonja, M., Kononenko, I.: Theoretical and empirical analysis of relieff and rrelieff. Machine Learning 53(1-2), 23–69 (2003) 24. Rooney, N., Patterson, D., Anand, S., Tsymbal, A.: Dynamic integration of regression models. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 164–173. Springer, Heidelberg (2004) 25. R. D. C. Team. R: A language and environment for statistical computing. Technical report, R Foundation for Statistical Computing (2006) ISBN 3-900051-07-0 26. Torgo, L.: Regression data repository 27. Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Dynamic integration with random forests. Tech. Report TCD-CS-2006-23, The University of Dublin, Trinity College (2006)
204
J. Mendes-Moreira et al.
28. Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Dynamic integration with random forests. In: F¨ urnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 801–808. Springer, Heidelberg (2006) 29. Tsymbal, A., Puuronen, S.: Bagging and boosting with dynamic integration of ˙ classifiers. In: Zighed, D.A., Komorowski, J., Zytkow, J.M. (eds.) PKDD 2000. LNCS (LNAI), vol. 1910, pp. 116–125. Springer, Heidelberg (2000) 30. Woods, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(4), 405–410 (1997) 31. Xycoon. Statistics - econometrics - forecasting 32. Yankov, D., DeCoste, D., Keogh, E.: Ensembles of nearest neighbor forecasts. In: F¨ urnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 545–556. Springer, Heidelberg (2006)
A
The RReliefF Algorithm
Our implementation of RReliefF follows [23], Program RReliefF(R, τ, m, k) {where each element of R (a vector of size M) is the input vector of each example from the training set; τ is a vector with the target values corresponding to R; m is the number of iterations; and k is the number of nearest examples} begin set NdC , NdA [A], NdC&dA [A], W [A] to 0, where A = 1, 2, · · · , a for (i:=1 to m) randomly select example Ri from R select k examples I nearest to Ri for (j:=1 to k) NdC := NdC + dif f (τ (.), Ri , I[j]) × d(i, j) for(A:=1 to a) NdA [A] := NdA [A] + dif f (A, Ri , I[j]) × d(i, j) NdC&dA [A] := NdC&dA[A] + dif f (τ (.), Ri , I[j])× dif f (A, Ri , I[j]) × d(i, j) end for end for end for for (A=1 to a) W [A] := NdC&dA[A]/NdC − (NdA [A] − NdC&dA[A]) × (m − NdC ) end for return(W) end. The variable a in the algorithm is the number of input variables of the data set. The parameter m is the number of iterations. Its choice is very important. The right value is problem dependent. The larger the m value is, more stable
Ensemble Learning: A Study on Different Variants
205
the weights estimations are. Additionally the computational cost also increases. The authors refer that the stability of the results is obtained, typically, for values between 20 and 50 iterations. We use m = 50. The value used for k (the number of nearest examples) was 10, as suggested by the authors. From the several distance functions proposed by the authors, we use one that quadratically increases the cost of the distance:
d(i, j) = k
d1(i, j)
l=1 (d1(i, l))
,
1 d1(i, j) = a , and ( l=1 dif f (Al , R[i], I[j]))2 ⎧ 0 : d ≤ teq ⎨ 1 : d > tdif f , dif f (A, I1 , I2 ) = ⎩ d−teq : teq ≤ tdif f tdif f −teq
(4) (5)
(6)
where the values of teq and tdif f are, respectively, 5% and 10% of the length of the input variable’s value interval, as suggested by the authors; and the value d represents the absolute difference of the input variable A for the two examples, I1 and I2 .
B
The Calculation of the Vector w of Weights for DW and DWS
The estimation of the error for each predictor weighs the error of each similar data point by the inverse of its distance to the input value, i.e., the weight used for each example is: wi = I
1 disti
i1=1
1 disti1
,
(7)
where dist is the vector of distances of the similar data examples to the input example and I is the number of examples, i.e., the size of the similar data set.
Relevance and Redundancy Analysis for Ensemble Classifiers Rakkrit Duangsoithong and Terry Windeatt Center for Vision, Speech and Signal Processing University of Surrey Guildford, United Kingdom GU2 7XH {r.duangsoithong,t.windeatt}@surrey.ac.uk
Abstract. In machine learning systems, especially in medical applications, clinical datasets usually contain high dimensional feature spaces with relatively few samples that lead to poor classifier performance. To overcome this problem, feature selection and ensemble classification are applied in order to improve accuracy and stability. This research presents an analysis of the effect of removing irrelevant and redundant features with ensemble classifiers using five datasets and compared with floating search method. Eliminating redundant features provides better accuracy and computational time than removing irrelevant features of the ensemble. Keywords: Feature selection, Ensemble classification, Redundant feature, Irrelevant feature.
1
Introduction
With improvements in information and healthcare technology, many medical information databases have been created. However, clinical data usually contains high dimensional features with small sample size and leads to degradation in accuracy and efficiency of system by curse of dimensionality. As Bellman stated in 1961 [1], high dimensional datasets lead to the degradation of classifier system performance because the more features, the more complexity, harder to train classifier and longer computational time. Over-fitting is another problem in high dimension features spaces. It usually occurs when the number of features is high compared to the number of instances. The resulting classifier works very well with training data but very poorly on testing data. In order to deal with high dimensional features, dimensionality should be reduced. Generally, there are two methods to reduce the dimension: feature extraction and feature selection. Feature extraction transforms or projects original features to fewer dimensions without using prior knowledge. Nevertheless, it lacks comprehensibility and uses all original features which may be impractical in large feature spaces. On the other hand, feature selection aims to select optimal feature subsets from original features by removing irrelevant and redundant P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 206–220, 2009. c Springer-Verlag Berlin Heidelberg 2009
Relevance and Redundancy Analysis for Ensemble Classifiers
207
features. It has the ability to reduce over-fitting, increase classification accuracy, reduce complexity, computational time and improve comprehensibility by preserving original semantic of datasets. Basically, feature selection can be divided into four categories: Wrapper, Filter, Hybrid and Embedded methods [2],[3],[4]. Wrapper method uses predetermined learning algorithm to evaluate selected feature subsets that are optimum for the learning process. This method has high accuracy but is computationally expensive. Filter method is faster than Wrapper method and is independent from learning method used in the classification process. It uses measurement techniques such as correlation, distance and consistency measurement to find a good subset from entire features. Nevertheless, the selected subset may or may not be appropriate with the learning method. Hybrid method combines advantage of both Filter and Wrapper method together. It evaluates features by using an independent measure to find the best subset and then using a learning algorithm to find the final best subset. Finally, Embedded method interacts with learning algorithm but it is more efficient than Wrapper method because the filter algorithm has been built with the classifier. Example of Embedded method is Recursive Feature Elimination (RFE) that is embedded with Support Vectors Machines. As has been illustrated by Liu and Yu(2005) [3], feature selection has four basic processes: Subset generation, subset evaluation, stopping criterion and subset validation. Subset generation produces candidate subset by complete (exhaustive), sequential (heuristic) or random search with three directions: forward (adding feature to selected subset that begin with empty set), backward (eleminate features from selected subset that begins with full original set) and bidirectional (both adding and removing features). After that, the candidate subset is evaluated based on criteria such as distance, dependency and information gain and consistency measurement. The process will stop when it reaches the stopping criterion. Finally, the selected subset is validated with validation data. There are many applications that applied feature selection as an important pre-processing step to improve systems efficiency, such as web text mining and e-mail classification, intrusion detection, biomedical informatics, gene selection in micro array data, medical data mining, and clinical decision support systems. An ensemble classifier or multiple classifier system (MCS) is another wellknown technique to improve system accuracy [5]. It is a group of base classifiers that learn a target function by combining their prediction together. Ensemble has ability to increase accuracy of system by combining output of multiple experts, improve efficiency by decomposing complex problem into multiple sub problems and improve reliability by reducing uncertainty. To increase accuracy, each classifier in the ensemble should be diverse or unique in order to reduce total error such as starting with different input, initial weight, random features and random classes [6]. There are many approaches to ensemble classifiers such as averaging, Bagging [7] and Boosting [8].
208
R. Duangsoithong and T. Windeatt
Fig. 1. Block diagram of Feature Selection with Ensemble classifier
Figure 1 presents the system block diagram of this research. The original dataset is sent to feature selection in order to remove redundant or irrelevant features. After that, selected features are passed through ensemble classifier using ensemble algorithm for training and predicting output. 1.1
Related Research
Feature selection and ensemble classification have interested researchers for many years. At the beginning of feature selection history, most researchers focused only on removing irrelevant features such as ReliefF [9], FOCUS [10] and Correlationbased Feature Selection(CFS) [11] In Yu and Liu (2004) [12], Fast Correlation-Based Filter (FCBF) algorithm was proposed to remove both irrelevant and redundant features by using Symmetrical Uncertainty (SU) measurement and was successful for reducing high dimensional features while maintaining high accuracy. Another technique of removing both irrelevant and redundant features using discriminant and redundancy analysis was proposed by Malarvili et al. (2007) [13] for Neonatal Seizure Detection. The technique is based on area under Receiver Operating Characteristic (ROC) curve for discriminant analysis and modified FCBF algorithm for redundancy analysis by meaning of predominant features based on discriminant power. According to Deisy et al. (2007) [14], SU does not have enough accuracy to quantify the dependency among features and does not take into account the effect of two features on the class label during redundancy analysis. Decision Independent Correlation (DIC) and Decision Dependent Correlation (DDC) were proposed instead of using SU to remove irrelevant and redundant features, respectively. DIC and DDC provide better performance than FCBF algorithm in terms of number of selected features, computational time and accuracy.
Relevance and Redundancy Analysis for Ensemble Classifiers
209
In Chou et al. (2007) [15], modified FCBF algorithm was used to eliminate both redundant and irrelevant features for intrusion detection. In redundancy analysis, they proposed to calculate SU between features and all original features. They found that FCBF algorithm possibly keeps redundant features in the final optimal subset because it considers only SU between selected features and the rest of features at a time. Biesiada and Duch (2008) [16] used SU to remove irrelevant features and proposed to use Pearson X 2 test to eliminate redundant features for biomedical data analysis. Moreover, Kolmogorov-Smirnov was also proposed [17] to reduce both redundant and irrelevant features. Although feature selection is widely used, there has been little work devoted to handling feature selection explicitly in the context of ensemble classifiers. Most previous approaches have focused on determining selecting optimal features, but rarely to combine with ensemble classification. Moreover, it is usual to remove both irrelevant and redundant features in contrast to the approach proposed in this paper. Moreover, generally, the number of features in feature selection analysis can be divided into three categories: small scale (the number of features is less than 19), medium scale (the number of features is between 20 and 49) and large scale (the number of features is equal or higher than 50 features) [18],[19]. The main purpose of this research is to find methods that can scale up to deal with hundreds or thousands of features.
2
Theoretical Approach
In our research as shown in Figure 1, Fast Correlation-Based Filter (FCBF) [12], an effective feature selection algorithm that removes both irrelevant and redundant features, is compared with Correlation-based Feature Selection with Sequential Forward Floating Search (CFS+SFFS) [11],[20]. Bagging [7] algorithm for ensemble classifiers, described in Section 2.2, is experimentally compared with different learning algorithms. 2.1
Feature Selection
FCBF algorithm has two stages, relevance analysis and redundancy analysis. Relevance Analysis. Irrelevant features are removed from original features and Correlation is widely used to analyze relevance. In linear systems, correlation can be measured by linear correlation coefficient (r) (xi − xi )(yi − yi ) r = i (1) 2 2 (x i i − xi ) i (yi − yi ) However, in real world applications, most systems are non-linear. We can measure correlation in non-linear systems by using Symmetrical Uncertainty (SU). SU = 2
IG(X|Y ) H(X)H(Y )
(2)
210
R. Duangsoithong and T. Windeatt
IG(X, Y ) = H(X) − H(X|Y ) H(X) = − P (xi )log2 P (xi )
(3) (4)
i
where IG(X|Y ) is the Information Gain of X after observing variable Y . H(X) and H(Y ) are the entropy of variable X and Y , respectively. P (xi ) is the probability of variable x. SU is the modified version of Information Gain that has range between 0 and 1. FCBF removes irrelevant features by ranking correlation (SU) between feature and class. If SU between feature and class equal to 1, it means that this feature is completely related to that class. On the other hand, if SU equal to 0, the features are irrelevant to this class. Redundancy Analysis. After ranking relevant features, FCBF eliminates redundant features from selected features based on SU between feature and class and between feature and feature. Redundant features can be defined from meaning of predominant feature and approximate Markov Blanket. In Yu and Liu (2004) [12], a feature is predominant (both relevant and non redundant feature) if it does not have any approximate Markov blanket in the current set. Approximate Markov blanket: For two relevant features Fi and Fj (i = j), Fj forms an approximate Markov blanket for Fi if SUj,c ≥ SUi,c and SUi,j ≥ SUi,c
(5)
where SUi,c is a correlation between any feature and the Class. SUi,j is a correlation between any pair of feature Fi and Fj (i = j) Correlation-Based Feature Selection (CFS). CFS [11] is one of well-known techniques to rank the relevance of features by measuring correlation between features and classes and between features and other features. Given number of features k and classes C, CFS defined relevance of features subset by using Pearson’s correlation equation M erits =
krkc k + (k − 1)rkk
(6)
Where M erits is relevance of feature subset, rkc is the average linear correlation coefficient between these features and classes and rkk is the average linear correlation coefficient between different features. Normally, CFS adds (forward selection) or deletes (backward selection) one feature at a time, however, in this research, we used Sequential Forward Floating Search (SFFS) as the search direction. Sequential Forward Floating Search (SFFS). SFFS [20] is one of a classic heuristic searching method. It is a variation of bidirectional search and sequential forward search (SFS) that has dominant direction on forward search. SFFS
Relevance and Redundancy Analysis for Ensemble Classifiers
211
removes features (backward elimination) after adding features (forward selection). The number of forward and backward step is not fixed but dynamically controlled depending on the criterion of the selected subset and therefore, no parameter setting is required. 2.2
Ensemble Classifier
Bagging. Bagging [7] or Bootstrap aggregating is one of the earliest, simplest and most popular for ensemble based classifiers. Bagging uses Bootstrap that randomly samples with replacement and combines with majority vote. Bootstrap is the most well-known strategy for injecting randomness to improve generalization performance in multiple classifier systems and provides out-of-bootstrap estimate for selecting classifier parameters [5]. Randomness is desirable since it increases diversity among the base classifiers, which is known to be a necessary condition for improved performance. However, there is an inevitable trade off between accuracy and diversity known as the accuracy/diversity dilemma [5].
3 3.1
Experimental Setup Dataset
The medical datasets used in this experiment were taken from UCI machine learning repository [21]: heart disease, hepatitis, diabetes, lung cancer and Parkinson dataset. Heart Disease Dataset was contributed by Cleveland Clinic foundation has 303 samples, 13 attributes with 138 samples presenting for heart disease class and 165 samples for absent class. Hepatitis Dataset was donated by G.Gong from Carnegie-Mellon University contains 155 instances, 19 attributes with live or die classes. Diabetes Dataset. Prima Indians Diabetes dataset was donated by John Hopkins University has 768 samples, 8 numeric features with tested positive and tested negative classes. Lung Cancer Dataset. Pathological lung cancer has 32 samples, 56 nominal features with three classes. Parkinson Dataset. Parkinson’s disease dataset is the speech signals recorded by Max Little from University of Oxford collaborated with the National Centre for Voice and Speech, Denver, Colorado. It has 197 samples, 23 features with two classes (healthy and Parkinson’s patient).
212
R. Duangsoithong and T. Windeatt
Fig. 2. FCBF algorithm
Fig. 3. Proposed FCBF algorithm
3.2
Proposed Algorithm
Our research separated FCBF results independently into relevance analysis (removing irrelevant features) and redundancy analysis (eliminating redundant features) as shown in Figure 2 and 3 and compared with original dataset and CFS+SFFS algorithm. 3.3
Evaluation
To evaluate feature selection process we use four widely used classifiers: NaiveBayes(NB), Multilayer Perceptron (MLP), Support Vector Machines (SVM) and Decision Trees (DT). The parameters of each classifier were chosen based on the highest accuracy of the experiment datasets of base classifier. MLP has one hidden layer with 16 hidden nodes, learning rate 0.2, momentum 0.3, 125 iterations and uses backpropagation algorithm with sigmoid transfer function. SVMs uses linear kernel and set the regularization value to 0.7 and Decision Trees use pruned C4.5 algorithm. The number of classifiers in Bagging is varied from 10, 20, 50, 100 to 250 classifiers. The threshold value of FCBF algorithm in our research is set at zero for heart disease, diabetes and Parkinson and 1.4 and 2.8 for hepatitis and lung cancer, respectively. The classifier results were validated by 10 fold cross validation with 10 repetitions for each experiment and evaluated by percent of test set accuracy and training CPU computational time.
Relevance and Redundancy Analysis for Ensemble Classifiers
4
213
Experimental Result
Table 1 shows the number of selected features in each analysis. For example, in heart disease dataset, redundancy analysis removed six redundant features, relevance analysis removed two irrelevant features and CFS+SFFS eliminates six weakly relevant [22] features . The relationship between number of selected features and threshold setting of FCBF algorithm with relevance and redundancy analysis of the five datasets is shown in figure 4. Figure 5 presents the average accuracy of the five datasets using four base classifiers with threshold setting of FCBF algorithm and figure 6 is the average accuracy of four base classifiers and threshold setting. According to these figures, zero threshold setting provides the best accuracy for heart disease, diabetes and Parkinson datasets. For hepatitis and lung cancer dataset, FCBF algorithm provides the best accuracy when the threshold is set at 0.14 and 0.28, respectively. Table 1. Number of selected features Dataset Heart Disease Hepatitis Diabetes Lung Cancer Parkinson
Original Feature Redundancy Analysis 13 7 19 3 8 4 56 2 23 5
Relevance Analy- CFS+SFFS sis 11 7 17 10 8 4 56 11 23 10
Selected Features 10 Heart Disease Hepatitis Diabetes Lung Cancer Parkinson
Selected Features
8
6
4
2
0 0
0.05
0.1
0.15 0.2 Threshold
0.25
Fig. 4. Selected Feature vs. Threshold
0.3
0.35
214
R. Duangsoithong and T. Windeatt MultipleLayer Perceptron
Percent Accuracy
Percent Accuracy
Naive Bayes 90 80 70 60 50 40 30 20 10 0 0
0.05
0.1
0.15
0.2
0.25
0.3
90 80 70 60 50 40 30 20 10 0
0.35
Heart Disease Hepatits Diabetes Lung cancer Parkinson 0
0.05
0
0.05
0.1 0.15 0.2 0.25 FCBF Threshold
0.15
0.2
0.25
0.3
0.35
0.1 0.15 0.2 0.25 FCBF Threshold
0.3
0.35
Decision Tree
90 80 70 60 50 40 30 20 10 0
Percent Accuracy
Percent Accuracy
Support Vector Machines
0.1
0.3
0.35
90 80 70 60 50 40 30 20 10 0 0
0.05
Fig. 5. Average Accuracy of four classifiers vs. Threshold Average Accuracy vs FCBF Threshold for five medical datasets 85 80 75
Average Accuracy
70 65 60 55 50 45 Heart Disease Hepatitis Diabetes Lung Cancer Parkinson
40 35 0
0.05
0.1
0.15 0.2 Threshold
0.25
0.3
0.35
Fig. 6. Average Accuracy vs. Threshold
The two examples of selected features of Heart disease and Hepatitis dataset with FCBF and CFS+SFFS compared to Polat and Gunes research [23] are shown in Figure 7-8, respectively. In [23], the features were selected by using
Relevance and Redundancy Analysis for Ensemble Classifiers
215
Fig. 7. Selected Features for Heart disease dataset
expert knowledge system (decision tree) without ranking them. In FCBF, the features were ranked and eliminated both irrelevant and redundant features while in CFS+SFFS, the features were selected based on merit value. According to figure 7, although the selected features for each analysis are not exactly the
Fig. 8. Selected Features for Hepatitis dataset
216
R. Duangsoithong and T. Windeatt
Naive Bayes
MultipleLayer Perceptron 80
Original Redundancy Relevance CFS+SFFS
78
Percent Accuracy
Percent Accuracy
80
76 74 72 70
78 76 74 Original Redundancy Relevance CFS+SFFS
72 70
1
10
50
100
250
1
80
78
78
76 74
Original Redundancy Relevance CFS+SFFS
72
50
100
250
100
250
Decision Tree
80 Percent Accuracy
Percent Accuracy
Support Vector Machines
10
70
76 74
Original Redundancy Relevance CFS+SFFS
72 70
1
10
50
Number of Classifiers
100
250
1
10
50
Number of Classifiers
Fig. 9. Average accuracy of five datasets for each classifier
same, there are some common eliminated features for heart disease dataset of three algorithms are fbs (fast blood sugar), trestbps (blood pressure) and chol (cholesterol). In hepatitis dataset, the common removed features of three algorithms are antirivals, fatigue, liver firm and spleen palpable. Figure 9 presents the average test accuracy of the five datasets using four base classifiers comparing with ensembles of size 10, 50, 100 and 250 (in logarithmic scale). Solid line presents original data set, short-dashed line is the result of removing irrelevant features, eliminating redundant features is shown as dashed line and CFS+SFFS result is presented in dotted line. Figure 10 shows the average computational time of the five datasets. Using all features in original set takes longest time to compute while using feature selection reduces the computational time. Ensemble classifiers take longer time to calculate than the base classifier and depend on number of classifiers. Figures 11-12 show the overall average accuracy and computational time of five datasets and four classifiers, respectively. Moreover, statistical significance (T-Test) using base classifier for one of the example datasets, lung cancer dataset, is shown in figure 13. From the number of significant win of column compare to row, redundancy analysis has more significance than CFS+SFFS and original dataset respectively. (There are no irrelevant features in this dataset.)
Relevance and Redundancy Analysis for Ensemble Classifiers
Naive Bayes
MultipleLayer Perceptron 250
Original Redundancy Relevance CFS+SFFS
0.12 0.1
Computation time (sec)
Computation time (sec)
0.14
0.08 0.06 0.04 0.02 0
Original Redundancy Relevance CFS+SFFS
200 150 100 50 0
1
10
50 100
250
1
Support Vector Machines 10 9 8 7 6 5 4 3 2 1 0
10
50 100
250
50 100
250
Decision Tree 2.5
Original Redundancy Relevance CFS+SFFS
Computation time (sec)
Computation time (sec)
217
Original Redundancy Relevance CFS+SFFS
2 1.5 1 0.5 0
1
10
50
100
250
1
Number of Classifiers
10
Number of Classifiers
Fig. 10. Average computational time of five datasets for each classifier
Average Accuracy for five medical datasets and four classifiers 77
76.5
Percent Accuracy
76
75.5
75
74.5
74
Original Redundancy Relevance CFS+SFFS
73.5
73 1
10 Number of Classifiers
50
100
Fig. 11. Average accuracy of five datasets and four classifiers
250
218
R. Duangsoithong and T. Windeatt
Average Computational Time for five medical datasets and four classifiers 60
Computation time (sec)
50
40
30
20
10 Original Redundancy Relevance CFS+SFFS 0 1
10 Number of Classifiers
50
100
250
Fig. 12. Average computational time of five datasets and four classifiers
Fig. 13. Example statistic significance of lung cancer dataset
5
Discussion
From our experimental result, redundancy analysis provides better accuracy than CFS+SFFS, relevance analysis and original dataset, respectively. Both relevance and redundancy analysis clearly reduce complexity and calculation time compared to full original dataset. As the number of classifiers is increased, the MLP makes beneficial use of irrelevant features. It would appear that the diversity induced by irrelevant features assists the MLP optimization algorithm. Given
Relevance and Redundancy Analysis for Ensemble Classifiers
219
that this effect is classifier-dependant, it is proposed that the out-of-bootstrap error estimate could be used to determine when eliminating irrelevant features is desirable [6].
6
Conclusion
In this paper, we introduce a comparison between redundancy analysis and relevance analysis combined with ensemble classifier for five medical datasets. From experimental result, redundancy analysis is more important than relevance analysis for ensembles. Further work is aimed at increasing the number of datasets and reducing training set size to investigate small sample size effects.
References 1. Bellman, R.E.: Adaptive Control Processes: A Guided Tour. Princeton University Press, Princeton (1961) 2. Liu, H., Dougherty, E., Dy, J., Torkkola, K., Tuv, E., Peng, H., Ding, C., Long, F., Berens, M., Parsons, L., Zhao, Z., Yu, L., Forman, G.: Evolving feature selection. IEEE Intelligent Systems 20(6), 64–76 (2005) 3. Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17(4), 491–502 (2005) 4. Saeys, Y., Inza, I., Larranaga, P.: A review of feature selection techniques in bioinformatics. Bioinformatics 23(19), 2507–2517 (2007) 5. Windeatt, T.: Ensemble MLP Classifier Design. LNCS, vol. 137, pp. 133–147. Springer, Heidelberg (2008) 6. Windeatt, T.: Accuracy/diversity and ensemble MLP classifier design. IEEE Transactions on Neural Networks 17(5), 1194–1211 (2006) 7. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996) 8. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Proceedings of the Thirteenth International Conference on Machine Learning, pp. 148– 156. Morgan Kaufmann, San Francisco (1996) 9. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005) 10. Almuallim, H., Dietterich, T.G.: Learning with many irrelevant features. In: Proceedings of the Ninth National Conference on Artificial Intelligence, pp. 547–552. AAAI Press, Menlo Park (1991) 11. Hall, M.A.: Correlation-based feature selection for discrete and numeric class machine learning. In: Proceeding of the 17th International Conference on Machine Learning, pp. 359–366. Morgan Kaufmann, San Francisco (2000) 12. Yu, L., Liu, H.: Efficient feature selection via analysis of relevance and redundancy. J. Mach. Learn. Res. 5, 1205–1224 (2004) 13. Malarvili, M., Mesbah, M.: Hrv feature selection based on discriminant and redundancy analysis for neonatal seizure detection. In: 6th International Conference on Information, Communications and Signal Processing, p. 15 (2007) 14. Deisy, C., Subbulakshmi, B., Baskar, S., Ramaraj, N.: Efficient dimensionality reduction approaches for feature selection. In: International Conference on Computational Intelligence and Multimedia Applications, vol. 2, pp. 121–127 (2007)
220
R. Duangsoithong and T. Windeatt
15. Chou, T., Yen, K., Luo, J., Pissinou, N., Makki, K.: Correlation-based feature selection for intrusion detection design. In: IEEE on Military Communications Conference, MILCOM 2007, pp. 1–7 (2007) 16. Biesiada, J., Duch, W.: Feature Selection for High- Dimensional Data - A Pearson Redundancy Based Filter, vol. 45, pp. 242–249. Springer, Heidelberg (2008) 17. Biesiada, J., Duch, W.: A Kolmogorov-Smirnov Correlation-Based Filter for Microarray Data. In: Ishikawa, M., Doya, K., Miyamoto, H., Yamakawa, T. (eds.) ICONIP 2007, Part II. LNCS, vol. 4985, pp. 285–294. Springer, Heidelberg (2008) 18. Kudo, M., Sklansky, J.: Comparison of algorithms that select features for pattern classifiers. Pattern Recognition 33, 25–41 (2000) 19. Zhang, H., Sun, G.: Feature Selection using Tabu search. Pattern Recognition 35, 701–711 (2002) 20. Pudil, P., Novovicova, J., Kitler, J.: Floating Search Methods in Feature Selection. Pattern Recognition Letters 15, 1119–1125 (1994) 21. Asuncion, A., Newman, D.: UCI machine learning repository (2007), http://www.ics.uci.edu/mlearn/MLRepository.html 22. John, G., Kohavi, R., Pfleger, K.: Irrelevant features and the subset selection problem, pp. 121–129. Morgan Kaufmann, San Francisco (1994) 23. Polat, K., Gunes, S.: A hybrid approach to medical decision support systems: Combining feature selection, fuzzy weighted pre-processing and airs. Computer Methods and Program in Biomedicine 88(2), 164–174 (2007)
Drift-Aware Ensemble Regression Frank Rosenthal, Peter Benjamin Volk, Martin Hahmann, Dirk Habich, and Wolfgang Lehner Technische Universit¨ at Dresden Database Technology Group 01062 Dresden, Germany [email protected]
Abstract. Regression models are often required for controlling production processes by predicting parameter values. However, the implicit assumption of standard regression techniques that the data set used for parameter estimation comes from a stationary joint distribution may not hold in this context because manufacturing processes are subject to physical changes like wear and aging, denoted as process drift. This can cause the estimated model to deviate significantly from the current state of the modeled system. In this paper, we discuss the problem of estimating regression models from drifting processes and we present ensemble regression, an approach that maintains a set of regression models— estimated from different ranges of the data set—according to their predictive performance. We extensively evaluate our approach on synthetic and real-world data. Keywords: Ensemble Method, Regression, Process Drift.
1
Introduction
Regression models are important tools in scientific, industrial, and many other application areas. They are used to learn a functional dependency between a numeric target variable and other variables of a given data set. Knowledge of such dependencies is often required to control industrial production processes. A concrete example in semiconductor manufacturing is etching, a process where material is physically removed with means like acid or plasma to create the layout of integrated circuits. In this setting, the time that is required to etch a specific amount of material is an important process parameter. The time heavily influences the width of the structures that are etched, which defines the electrical properties of the final integrated circuit. Hence, the correct etch time has to be determined to attain a product that fulfills defined specifications on this most important quality measure. Naturally, etch times that deviate from the ideal time will result in lower product quality. Semiconductor manufacturing is technologically growing fast. Companies enhance their manufacturing processes continuously while creating products of smaller and smaller structure widths. Maintaining production processes that operate in such small P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 221–235, 2009. c Springer-Verlag Berlin Heidelberg 2009
222
F. Rosenthal et al.
dimensions—e.g., 22nm—requires extensive use of regression models for control, since theoretical research alone cannot provide the required models [1,2,3]. Creating regression models for process control requires a sufficiently large data set. In single-lot semiconductor manufacturing, where only few expensive products of a certain design are manufactured in complex and long-running processes, this creates a new challenge. Since the acquisition of the data set expands over a long period of real time, elements of the production process change in the meantime and the examples basically represent different processes. In our example of etching, chemicals used in the process are subject to aging. Hence, an acid might react more slowly, thereby removing less material per unit of time. A control model that predicts etch time therefore ideally has to consider the age of the chemical. However, the knowledge to describe such aging processes is typically not readily available. In this paper, we consider drifting processes, i.e., processes that observably change over time. Aging is an example of a continuous drift, since it continuously changes the process. Estimating a regression model from examples of a drifting process violates the basic assumption that the data set has been drawn from one joint distribution of the involved variables. If regression is done anyway, the resulting model will not reflect the current state of the process but an average state over time. Note that such drifts occur in many industrial processes, since every machine wears and ages. In the context of single-lot semiconductor manufacturing, drift may result in direct monetary loss, since even small errors in the estimates of process parameters are critical. Besides continuous drift, there are also abrupt drifts. In our example, they may result from a change of the etching chemical. The effect on estimated regression models is similar to the effect of continuous drift, i.e., the estimated model does not reflect the current state of the modeled system. One basic approach to handle drift is to restrict the data set to the most recent examples, which represent the current state of the system best. However, this restricted data set may not contain enough information to estimate complex functional dependencies, like those that occur in semiconductor manufacturing. Therefore, a tradeoff between a current model and a stable model, in terms of average prediction error, must be found. In this paper, we describe drifting processes and how drift influences regression models that have been estimated from examples generated by such processes (Section 2). In Section 3, we then propose ensemble regression, an approach that learns a composite regression model that reduces the prediction error when used as a model of the current state of the system. Ensemble regression uses a set of regression models estimated from different ranges of the complete data set and maintains this set according to the predictive performance of the models. Predictions are drawn from the ensemble using a weighted average of the predictions of all ensemble members. Our approach can be used in connection with any regression method. We extensively evaluate the effects of certain types of drift as well as the predictive performance of models estimated by our approach on synthetic and real-world data in Section 4. Section 5 describes related work and Section 6 concludes the paper.
Drift-Aware Ensemble Regression
2
223
Drifting Processes and Regression
We consider the problem of regression on data from drifting processes. Regression is the task of estimating or learning a function yˆ = f (x) from a data set D consisting of n examples di = (yi , xj,i ), j = 1, . . . , k, i = 1, . . . , n where y is the numeric label (the dependent variable) and xj are the k features (the independent variables). In our etching example, the time of etching is the dependent variable, while the structural width—and thereby the amount of material—is the independent variable. Each time the etching process is executed, we can acquire an example of an etch time and the structural width that has been reached. The basic assumption in regression is that the data set has been drawn from exactly one joint distribution of dependent and independent variables [4] that captures the dependencies between them. If the joint distribution is stationary when the examples are acquired, the examples represent exactly one functional dependence, i.e., one function, disregarding the uncertainty introduced by noise that may be introduced by measurement equipment. In this paper, we consider the case when the data set is acquired from a drifting process. Such a data set is not distributed according to just one distribution. Instead, the distribution function is a function of time and drift is the change that occurs over time. Therefore, the examples acquired from the process represent a set of functional dependencies between dependent and independent variables. Each of these functions governed the process at a time when an example was drawn. Therefore, we denote a system that yields data according to a changing underlying functional dependency as a drifting process. The effect of drift is that the same input values may result in different output values when determined at different points in time. In terms of the governing function, drift may be viewed as a change in functional form or as a change in the parameters. We now illustrate this view with a simple example from physics. The electrical resistance of an object is a measure of the amount of opposition against an electric current, which is described by Ohm’s law: R = U I −1 . If we want to determine the electrical resistance of an object experimentally, one way is to put it into an electrical circuit, set a certain voltage U and measure the current I. Performing this repeatedly for different values of U enables the estimation of R using linear regression, where U = y and I = x. However, electrical resistance also depends on the temperature of the resistor; higher temperatures will result in higher resistance. Therefore, if the temperature during the experiment is not constant, the examples (U, I) will represent different underlying functions U = Ri I, where Ri is the resistance at the time a particular measurement was taken. In physics, this problem has been solved by analyzing the impact of temperature on the electrical resistance and extending the basic dependence in a form that treats the resistance as a function of temperature R(θ). However, deriving such laws requires significant insight into the observed system and is not feasible for complex real-world processes like single-lot semiconductor manufacturing. Even if the influence factors and their functional form were all known, some factors might not be observable at all or not observable with acceptable cost. Additionally, real-world drifts may behave non-deterministically with abrupt
224
F. Rosenthal et al.
●
1st underlying function
20
2nd underlying function model from complete dataset
10
model from window
y
● ●
0
● ●
−10
●
−4
−2
0
2
4
6
8
x
Fig. 1. Effect of an abrupt drift from the first to the second true dependency on a regression model estimated from mixed samples
changes and may be subject to random fluctuations. In our example of etching, these obstacles circumvent the modeling of the behavior of the drift. Therefore, an approach is required to estimate regression models from the data of drifting processes while minimizing the error introduced by the drift. We now present the characteristics of this error using another simple example that is depicted in Figure 1. Consider a linear functional dependence y = f (x) = ax + b from univariate input x to univariate output y. Suppose further that the functional form stays constant, but drift occurs in parameter a. Hence, a(t) is a function of time and y = ft (x) = a(t)x + b. Figure 1 depicts two such functions at different times t. An abrupt drift has occurred while gathering the examples. For the first five examples (the circles) a(t) = 1 and for the last five examples (the triangles) a(t) = 3. For presentation purposes, we excluded the presence of any measurement noise. In general, n examples are acquired that can be indexed by 1, . . . , n in the order of the time they have been drawn from the process. When using the complete data set, i.e., d1 , . . . , dn , to estimate the parameters a ˆ and ˆb of the linear model, an average over the underlying functions is the result, since the discrepancies between the examples from the different underlying functions are implicitly discarded as noise by the regression method. Applying this approach to our example results in the dotted line in Figure 1. The estimated model and the current underlying function (line with triangles) are different and there will be an error when using the estimated model for prediction. The dashed line in Figure 1 represents a model that has been estimated from a window of size 6, i.e., the more recent half of the data set and the last example from the first underlying function. This model resembles the recent state much closer. The sum of the squared prediction errors of this model is about a tenth
Drift-Aware Ensemble Regression
225
of the error of the model that had been estimated from the whole data set. This example gives an indication that using older data may result in a greater prediction error. We can minimize the prediction error by using only the last example. However, this is not feasible when the functional dependence is of a more complex nature (e.g., polynomial) or when non-parametric regression is used.
3
Ensemble Regression for Drifting Processes
We are given a data set D = {di }, i = 1, . . . , n, where di are examples from a drifting process and di = (yi , xj,i ), j = 1, . . . , k, y is the numeric label and xj are the k features to be used in the calculation of the prediction yˆ. The core idea to reduce the effect of drift on the estimated model is to restrict the data set used for parameter estimation to a window of examples, thereby excluding older examples that do not represent the current state of the modeled system. The basic approach consists of estimating a regression model from a window of examples d(n−w) , . . . , dn , where n is the index of the most recent example and w is the number of examples in the window. Whenever a new example is acquired, n is incremented to represent the expanded data set. The right border of the estimation window is fixed to the current value of i, since the most recent example represents the current state of the modeled system best. Since n increases with each example, the whole window is shifted and therefore older data is excluded. The difficulty with this approach lies in determining the window size w that results in a model with a small prediction error. An analytic solution requires knowledge about the characteristics of the drift of the modeled process, i.e., rate of drift or cycle length. In our application setting—single-lot semiconductor manufacturing—this knowledge is often not available. Additionally, drift may be discontinuous, e.g., abrupt and random, which causes the optimal window size to change as well within the lifetime of a prediction model. Therefore, we need an adaptive solution, where the window size is set as part of the training process. Changes in the characteristics of the drift can then be compensated. Since an exact determination of the window size is not possible in our setting, we avoid determining one best-effort window size. Instead, we use several windows of different size for training several regression models that are maintained according to predictive performance, i.e., when a model is not sufficient anymore, it is replaced. To maintain this set of models, the following statistics are needed for each model modelm . First of all, there is lm , the left border of the window of examples for estimating the associated model. Second, there is the number of positive posm and negative negm predictions that were made with the associated model. The counters are based on tests performed on new examples and a local threshold te for the acceptable error. Finally, there is the weight wm , which is used for calculating overall predictions from the ensemble. We describe the statistics more detailed in the following. The statistics and the model form a member Mm that can be uniquely identified by an index m. A set of members forms an ensemble E = {Mm }, m = 1, . . . , |E|.
226
F. Rosenthal et al.
Algorithm 1. Training algorithm for ensemble regression. Require: emax : the maximum number of members in the ensemble Require: te : threshold for prediction error Require: tr : threshold for the performance ratio 1: E ← ∅; D ← ∅; n ← 1 2: for each new d do 3: n ← n + 1; dn ← d 4: D ← D ∪ dn 5: for each Mm ∈ E do 6: if not is stable(modelm ) then 7: modelm ← train(lm , n); next 8: end if 9: err ← error(yn , predict(modelm , xn )) 10: if err < te then 11: posm ← posm + 1 12: else 13: negm ← negm + 1 14: end if 15: if posm (posm + negm )−1 < tr then 16: E ← E\M 17: else 18: modelm ← train(lm , n) 19: end if 20: end for 21: posT = |E| m=1 posm 22: for each stable Mm ∈ E do 23: wm ← posm pos−1 T 24: end for 25: if |E| < emax then 26: E ← E ∪ new member(dn ) 27: end if 28: end for
We now present the algorithm (see Algorithm 1) that is used to maintain an ensemble by describing several iterations and the resulting state of the ensemble after each of them. The example is based on the data set depicted in Figure 1, which contained an abrupt drift after the fifth of ten examples. As initialization, we start with an empty data set D and an empty ensemble E. The main loop of the algorithm (line 2) is executed whenever a new example d is acquired. The index for the most recent example n is increased by one and the example is added to the data set D as dn . Hence, in the first iteration, D = {d1 }. Since the ensemble is empty, lines 5 to 24 have no effect and the first member is added to the ensemble in line 26. Members are added to the ensemble as long as the maximum number of members emax is not reached. This is a technical parameter to limit the required computational resources. The method new member(dn ) initializes the statistics of the new member and estimates the associated regression model from a minimal window containing only the most recent example dn .
Drift-Aware Ensemble Regression
227
The current value of n will be stored as left border lm of the estimation window. After this first iteration, E = {M1 }. When the next example d is acquired, it is added to the total data set as d2 and therefore D = {d1 , d2 }. Since the ensemble now contains one member, line 6 is executed for member M1 . The method is stable(modelm ) tests whether a member may be considered as stable, i.e whether the estimation window contains enough examples. The exact definition for stability depends on the used regression method. For linear regression, a stable fit requires at least as many examples as coefficients to be estimated. In our example, we use linear regression and two coefficients have to be estimated. Hence, M1 is still unstable in step two, because the current window for estimation contains only one example. In line 7, the model is now estimated from a larger window [lm , n] = [1, 2]. In general, the window associated with a member grows by one example for each example acquired. After this training, all other steps involving M1 are skipped. The second iteration of the algorithm is completed after adding the second member M2 , which is associated with a model that was estimated from d2 . Hence, E = {M1 , M2 }. In the third step, M1 has finally become stable and therefore the following steps from lines 9 to 19 are performed on it. First, M1 is tested by using the associated model modelm to predict the known label of the new example (line 9). Then, the prediction error is calculated using a given metric error, e.g., the root mean squared error. We use this testing scheme, since the most recent example represents the current underlying function best and since the most recent example has not been used in model estimation so far. In our example, no drift has occurred in this third iteration. Therefore, the error is err = 0. In lines 10 to 14, the error determined in the test is classified as positive or negative and the according counter (posm or negm ) is incremented. The parameter te that defines this local error threshold is intuitive to set. A larger threshold will result in more predictions classified as positive. In some application areas, like semiconductor manufacturing, this parameter is derivable from application knowledge, like quality specifications. In our example, the number of positive tests for M1 is pos1 = 1 in the third iteration, since err = 0 and because we set te = 1. In line 15, the test for eviction is performed. We propose a threshold-based criterion, where a member is evicted when posm (posm + negm )−1 < tr , i.e., when the ratio of positive test predictions to all test predictions falls below a given value 0 ≤ tr ≤ 1. This criterion has several important properties. It is resistant to outliers, since single negative predictions, e.g., those caused by noise, have limited influence when a member has reached a stable state. A member may even recover from a series of negative predictions, e.g., in cases of cyclic drift. Most importantly, members that have accumulated a large number of positives, and therefore have been in the ensemble for a long time, are harder to evict than members with fewer positives, since they can accumulate more negatives before fulfilling the eviction criterion. Therefore, mature members are favored but can still be evicted if a permanent drift occurs that makes them unusable.
228
F. Rosenthal et al.
P N
P N
1 0
1
3 0
1
0 0
2
2 0
2
0 0
3
1 0
3
0 0
4
0 0
5
(a) State at n=3 P N
(b) State at n=5 P N
3 2
1
2 1
5
2 2
2
2 0
6
1 0
5
1 0
7
0 0
6
0 0
8
7
0 0
0 0
(c) State at n=7
9
(d) State at n=9
Fig. 2. Window size, number of positive and negative predictions of the ensemble members. Data set from Figure 1.
Note that the prediction weight of such a member can decrease even while it is still in the ensemble, which accounts for the fact of decreasing predictive performance. In our example, M1 stays in the ensemble since the ratio of positive test predictions to all test predictions is one and we set tr = 0.5. Members with sufficient predictive performance remain in the ensemble and the associated model is trained on the window [lm , n], thereby including the new example. The threshold on the performance ratio influences how fast the ensemble reacts to drift, whereby a larger ratio implies that few negative predictions are tolerated and eviction may happen early. Figure 2 depicts the further evolution of the ensemble. White squares represent examples from the first underlying function, while gray squares represent examples from the second underlying function. The examples are ordered from left to right with ascending index. The rectangles below depict the window that each member in the ensemble used to estimate its associated model. The number in the rectangle is the member identification m and at the left, the column P shows posm , while N shows negm . Figure 2(a) shows the state of the ensemble after the completion of the third iteration. Three members have been added so far. M1 uses the largest window and is the only stable member. Figure 2(b) depicts the state after the fifth example has been incorporated. There are five members now, which means that the set emax is reach. No further member will be added. Each member has been estimated from a different window, with the first member still using the largest window. It also has the largest number of positive predictions, since no drift has occurred yet. Member M5 is still considered unstable. Member M4 is stable but has not been tested yet on a new example, since the fourth and fifth examples were used in parameter estimation. No member has made any negative predictions.
Drift-Aware Ensemble Regression
229
Figure 2(c) depicts the state after the seventh example has been incorporated. There are again five members, but members M4 and M3 have been replaced by the new members M6 and M7 . Member M4 was evicted in the previous step, while member M3 was evicted in the seventh step. Interestingly, member M5 is still in the ensemble, although its window covers an example from the previous underlying function. Figure 2(d) depicts the state after the ninth example has been incorporated. The members M1 and M2 have been evicted, since they were unable to perform any more positive predictions, thereby dropping below the required performance ratio. Member M5 remained in the ensemble, since more and more current examples helped to improve the fit of the underlying function. So far we have not discussed the determination of overall predictions yˆ from the ensemble. yˆ is determined by calculating yˆm = predict(modelm , x) for each member in the ensemble and then calculating a weighted average yˆ = |E| m=1 wm yˆm , whereby the weights wm are determined as part of Algorithm 1 in two steps. First, in line 21, the total number of positive predictions posT = |E| m=1 posm of the members in the ensemble is determined. Then, this sum is used to determine the prediction weight of each member in lines 22 to 24. It is defined as the ratio of positive predictions of the member to the sum of positive predictions of all members: posm posT −1 . Hence, if a member has made only a small number of positive predictions, while other members have made large numbers of positive predictions, its weight will be low and vice versa. Mature members are therefore favored. Most importantly, the weight of a member can decrease if it does not make positive predictions while other members do (e.g., newer members).
4
Evaluation
In this section, we evaluate our approach on several types of synthetic data as well as a real-world data set. Our goal is to determine the prediction error when using different approaches to estimate regression models from drifting processes. We compare ensemble regression using different weighting schemes with the baseline (performing regression on the complete data set) as well as with the sliding window approach. A simulation environment with these algorithms was implemented in R. We used lm, included in the standard stats package, to estimate linear regression models. Since we restricted our experiments to the use of linear models in the ensembles, we also restricted our experiments to linear underlying functions. This ensures that the models are capable of representing the governing functions and no error can be induced from an improper functional form. Our systematic evaluation is therefore only valid for data sets from drifting processes that are governed by a linear dependency. However, we also present results based on a non-linear, real-world data set that indicate that our approach performs well in that setting, too. Formally, our Ksynthetic data sets were generated using a governing function y = f (x) = k=1 wk (t)xk + N (0, σ 2 ) where K is the number of independent
230
F. Rosenthal et al.
variables, wk (t) is the time-dependent weight of attribute xk and N (0, σ 2 ) is a normal random variable that represents measurement noise. In this setting, drift manifests itself in changing values of wk (t), while the functional form of f (x) stays fixed. We define three different functional forms of wk (t): Linear Drift. wk (t) = at + b, where slope a determines the speed of the drift. We fix b since it has no influence on the prediction error, while a is varied. Results are presented in Figures 3(a) and 3(b). Autoregressive Drift. wk (t) = wk (t − 1) + N (μ, σ 2 ); wk (0) = N (μ, σ 2 ), where wk (t − 1) is the weight determined for the previous example and wk (0) is the random starting weight. N (μ, σ 2 ) is a normally distributed random variable with mean μ and variance σ 2 . In our experiments, we vary μ while we fix σ. Results are presented in Figures 3(c) and 3(d). Mixture Drift. wk (t) = wk (t − 1) + a + I(p)N (μ, σ 2); wk (0) = N (μ, σ 2 ), where wk (t − 1) is the weight determined for the previous example, wk (0) is the random starting weight and a is the slope. I(p) yields either 1 with probability p or 0 with probability (1 − p), i.e., I(p) follows a binomial distribution with n and k fixed to 1. I(p)N (μ, σ 2 ) represents abrupt drift of random magnitude. This drift contains a fixed linear and a stochastic component and is therefore more realistic than the other two. In our experiments, we vary a and μ while we fix σ and p. Results are presented in Figures 3(e) and 3(f). For each of these three types of drift, we fixed a set of values for the parameters that are varied, which resulted in 97 parameter sets. For each of them, 10 data sets were created, with each data set containing 1,000 examples. The values of the xn were drawn from a uniform distribution, while the wk (t) were calculated and the weighted sum yielded the associated yn as defined above. We then simulated the repeated estimation of a regression model from each of these data sets and recorded the root squared error for overall predictions from the ensemble. The results are distributions of the error per regression approach over all parameter sets and they are depicted in Figure 3 with one row of box plots per drift type. Since the variance in the error differs significantly, the results are shown on two scales. The maximum displayed root squared error RSEmax is 1, 000 and 100 in the left and right column respectively. In each diagram, five box plots are shown. The labels on the x-axis correspond to the following five approaches: bl. This is the baseline approach of estimating a regression model from the complete data set whenever a new example is added. This approach is clearly the worst because of the large median error and large error variance for all three drift types. This approach is neither accurate nor stable. However, even for this approach small errors were observed, which can be explained by the dependence of the error on the value of the independent variables xn . This was illustrated in Figure 1, where the smallest error for the model estimated from the whole data set is zero at the intersection with the most recent underlying function. win. The windowing approach achieved very good results on the linear data set, where the median and the first quartile are the lowest of all approaches.
Drift-Aware Ensemble Regression
231
This changes for the autoregressive data set where the median and the first quartile are equal to our proposed approach (labeled as ens-g). For the mixture drift, window is the second-best approach. These results also confirm our assumption that a restriction of the data set used for estimation can result in a smaller prediction error. ens-e. This approach is identical to our proposed ensemble regression scheme, except that it does not use the calculated weights. Instead, the overall predictions are determined as an equally weighted linear combination. The approach outperforms baseline on the linear and the autoregressive data sets, but it is worse on the mixture drift data set. This can be explained by the relatively strong changes caused by the abrupt component in the mixture drift, which invalidates members very fast. Since it takes some time until the members are evicted, they contribute a large error that is propagated into the final prediction, since all weights are equal. ens-l. This approach is identical to our proposed ensemble regression scheme, except that it employs the local performance ratio posm (posm + negm )−1 as weight, which seems to be an intuitive choice. However, the variance of the prediction error is even larger than when using equal weights. ens-g. This is the approach proposed in this paper and it uses the global weight posm pos−1 T . It yields the best overall performance, since it has the lowest error variance for all three drift types. For linear and autoregressive drift, the median error is slightly worse or equal to the median error of the windowing approach. However, the median error is significantly lower for the mixture drift. In a second set of examples, we examined the influence of the maximum number of ensemble members emax and the eviction ratio tr on the distribution of the prediction error. We executed the simulation using different values for emax and tr on the data sets of all three drift types. The results were similar with respect to the observed influence. We therefore show only the error distribution based on the linear drift data sets. In Figure 4(a) the error distribution using different values of emax is depicted. It can be seen that emax has only a limited influence on the prediction error, although the median error and the error variance seem to increase slightly with increasing emax . However, we cannot conclude that using fewer members is a good strategy, since a smaller value of emax causes an increased chance of a completely unstable ensemble. This situation occurs when all stable members are evicted at once and new unstable members—being introduced one by one— still have to gather examples to be regarded as stable. In our experiments, we defined members with a window size smaller than 4 as unstable, since we used linear regression and since 3 coefficients had to be estimated. Otherwise, the underlying equation system would be under-determined and the resulting model insufficient. Using larger emax can reduce the likelihood of this situation. We can only conclude that future work is required for application settings were this behavior of our approach is unacceptable. In the first set of experiments, depicted in Figure 3, emax was set to 10.
232
F. Rosenthal et al.
(a) Linear drift; RSEmax = 1, 000
(b) Linear drift; RSEmax = 100
(c) Autoreg. drift; RSEmax = 1, 000
(d) Autoreg. drift; RSEmax = 100
(e) Mixture drift; RSEmax = 1, 000
(f) Mixture drift; RSEmax = 100
Fig. 3. Distribution of root squared error for different types of drift
Drift-Aware Ensemble Regression
(a) Influence of emax
233
(b) Influence of tr
Fig. 4. Parameter influence on the distribution of root squared error
● ● ●
● ● ●
baseline
ensemble
−10
0
5
10 15 20
In Figure 4(b), the influence of the eviction ratio is depicted. Obviously, a higher value of tr results in a lower median error and a decreased error variance. In the first set of experiments, depicted in Figure 3, tr was set to 0.2. We did not study the impact of te further, since—as we argued in Section 3—if possible, it should be set using application knowledge. In the first set of experiments, we fixed it to be 0.1% of the total spread of the values of y, which translates to an absolute value of te = 5. In Figure 5, we report results on a real-world data set from semiconductor manufacturing. To prevent any deductions about the underlying processes, we just report signed, relative errors. The data set is high-dimensional, non-linear and contains a relatively small continuous drift. The baseline approach yields a median error of about 1.3%, which is reduced to −0.1% by using ensemble regression . The model yielded by ensemble regression is therefore much better centered, while the error variance is identical.
Fig. 5. Distribution of relative error for a real-world data set
234
5
F. Rosenthal et al.
Related Work
In this paper, we consider the problem of estimating a regression model from data of a drifting process, whereby the most recent underlying function is to be approximated. To the best of our knowledge, this problem has not been considered so far. Current approaches in industrial applications can often use specially designed control models [5] to correct process drift and therefore eliminate the cause of the type of error investigated in this paper. Hence, standard regression techniques can be applied there. However, these approaches are not feasible in our application setting because of the indeterministic nature of drift and still ongoing theoretical research on the underlying physical phenomenons [1,2,3]. Concept drift is a related problem in classification and was introduced in [6]. Concept drift occurs when a hidden context exists and changes while examples are gathered. Changes in the hidden context can induce changes in the target concept. Analogous, in our problem setting, the underlying function changes and therefore influences the regression models estimated from a yielded data set. A number of approaches for handling concept drift exist. In [7] a system is presented that uses a sliding window of examples to construct concept descriptions using a description language. In [8] an algorithm for mining decision trees from a sliding window on a continuously changing data stream was presented. Adaptation to concept drift is performed by replacing subtrees or by building alternative subtrees. All these approaches enhance particular classification methods to be able to adapt to concept drift. In [9,10] an approach is presented where ensembles of base classifiers are built from sequential chunks of data. Using such jumping windows is not appropriate in our application context, since the newest examples always have to be incorporated. In [11] dynamic weighted majority was proposed for tracking concept drift. It uses an ensemble of base classifiers that are built from growing windows. However, the different learning task made it necessary to use a different eviction criterion and weighting scheme. Regarding the latter, members start with a fixed weight and each false prediction is penalized by reducing the weight by a constant fraction and normalizing the weights afterwards. In this approach, new members have a larger influence than in our approach, where the predictive weight is gathered slowly.
6
Conclusion and Outlook
Regression is an important, widely used tool, although the implicit assumption of a stationary joint distribution may not be met. Using regression on data from drifting processes results in a model that does not represent the current state of the process well. In application areas like semiconductor manufacturing, this induces an error that can be critical. In this paper, we presented ensemble regression, an approach that uses a set of regression models estimated from different ranges of the complete data set. The ensemble is maintained according to the predictive performance of the members, thereby yielding a low prediction error and a low error variance. In future work, we plan to widen our empirical study
Drift-Aware Ensemble Regression
235
to the examination of the effects of drift and the performance of our approach on non-linear dependencies as well. This will be accompanied by the use of nonlinear and nonparametric regression techniques as base learners. Additionally, we plan to investigate the use of sophisticated heuristics for adding new members to the ensemble.
References 1. Spitzlsperger, G., Schmidt, C., Ernst, G., Strasser, H., Speil, M.: Fault detection for a via etch process using adaptive multivariate methods. IEEE Transactions on Semiconductor Manufacturing 18(4), 528–533 (2005) 2. Bunday, B.D., Bishop, M., Donald, W., McCormack, J., Villarrubia, J.S., Vladar, A.E., Dixson, R., Vorburger, T.V., Orji, N.G., Allgair, J.A.: Determination of optimal parameters for cd-sem measurement of line-edge roughness. Metrology, Inspection, and Process Control for Microlithography XVIII 5375(1), 515–533 (2004) 3. Yue, H.H., Qin, S.J., Wiseman, J., Toprac, A.: Plasma etching endpoint detection using multiple wavelengths for small open-area wafers. Journal of Vacuum Science & Technology A: Vacuum, Surfaces, and Films 19(1), 66–75 (2001) 4. Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning. Springer, Heidelberg (2003) 5. DiRaddo, R., Girard, P., Chang, S.: Process drift and model-based control of forming operations. In: American Control Conference, 2002. Proceedings of the 2002, vol. 5, pp. 3588–3593 (2002) 6. Schlimmer, J.C., Granger, R.H.: Incremental learning from noisy data. Machine Learning 1, 317 (1986) 7. Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Machine Learning 23(1), 69–101 (1996) 8. Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: KDD 2001: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 97–106 (2001) 9. Street, W.N., Kim, Y.: A streaming ensemble algorithm (sea) for large-scale classification. In: KDD 2001: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 377–382. ACM, New York (2001) 10. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams using ensemble classifiers. In: KDD 2003: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 226–235. ACM, New York (2003) 11. Kolter, J., Maloof, M.: Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Third IEEE International Conference on Data Mining (ICDM), November 2003, pp. 123–130 (2003)
Concept Drifting Detection on Noisy Streaming Data in Random Ensemble Decision Trees Peipei Li1,2 , Xuegang Hu1 , Qianhui Liang2 , and Yunjun Gao2,3 1 2
School of Computer Science and Information Technology, Hefei University of Technology, China, 230009 School of Information Systems, Singapore Management University, Singapore, 178902 3 College of Computer Science, Zhejiang University, China, 310027
Abstract. Although a vast majority of inductive learning algorithms has been developed for handling of the concept drifting data streams, especially the ones in virtue of ensemble classification models, few of them could adapt to the detection on the different types of concept drifts from noisy streaming data in a light demand on overheads of time and space. Motivated by this, a new classification algorithm for Concept drifting Detection based on an ensembling model of Random Decision Trees (called CDRDT) is proposed in this paper. Extensive studies with synthetic and real streaming data demonstrate that in comparison to several representative classification algorithms for concept drifting data streams, CDRDT not only could effectively and efficiently detect the potential concept changes in the noisy data streams, but also performs much better on the abilities of runtime and space with an improvement in predictive accuracy. Thus, our proposed algorithm provides a significant reference to the classification for concept drifting data streams with noise in a light weight way. Keywords: Data Streams, Ensemble Decision Trees, Concept Drift, Noise.
1
Introduction
As the definition of data streams described in [23], it is an ordered sequence of tuples with certain time intervals. And as compared with the traditional data source, it always presents various new characteristics as being open-ended, continuous and high-volume etc.. It is hence a challenge to learn from these streaming data for most of traditional inductive models or classification algorithms[18,19,9]. Especially, it is intensively challenging for them oriented to the issues of concept drifts and noise contamination in the real applications, such as web search, online shopping or stock market and alike. To handle these problems, massive models and algorithms of classification have been proposed. The representative ones are based on ensemble learning, including an early ensemble algorithm of SEA[1] addressed the concept drift of data streams, a general framework P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 236–250, 2009. c Springer-Verlag Berlin Heidelberg 2009
Concept Drifting Detection on Noisy Streaming Data
237
for mining concept-drifting data streams using weighted ensemble classifiers[2], a discriminative model based on the EM framework for fast mining of noisy data streams[4], decision tree algorithms for concept drifting data streams with noise[5,11] and a boosting-like method for adaptation to different kinds of concept drifts[6] etc.. However, for these algorithms referred above, the limitations mainly rely that on one hand, little attention is paid to handle various types of concept drifts in data streams impacted from noise. On the other hand, the overheads of space and runtime are probably demanded heavily while without a prominent improvement on predictive accuracy. Therefore, to address the aforementioned issues, we present a light-weighted ensemble classification algorithm of CDRDT for Concept Drifting data streams with noise. It is based on random decision trees evolved from semi-random decision trees in [14]. Namely, it adopts the strategy of random selection to solve split-test for the nodes with numerical attributes instead of the heuristic method. In comparison to other ensembling model of random decision trees for concept drifting data streams, there are four significant contributions in CDRDT: i) the basic classifiers are constructed incrementally with small various chunks of streaming data. ii) the inequality of Hoeffding Bounds[7] is adopted to specify two thresholds, which are used in the concept drifting detection from noise. It benefits distinguishing the different types of concept drifts from noise. iii) the sizes of data chunks are adjusted dynamically with the bound limit to adapt to the concept drifts. It is beneficial to avoid the disadvantage of too large or too small sizes of data chunks in the detection of data distribution, especially in the case with the classification method of majority class. iv) the effectiveness and efficiency of CDRDT in the detection on concept drifts from noisy data streams are estimated and contrasted with other algorithms, including the state-of-the-art algorithm of CVFDT[10] and the new ensemble algorithms of MSRT (Multiple Semi-Random decision Trees)[11] based on semi-random decision trees. And the experimental results show that CDRDT performs in a light demand on the overheads of time and space with higher predictive accuracy. The rest of the paper is organized as follows. Section 2 reviews related work based on ensemble classifiers of random decision trees learning from concept drifting data streams. Our algorithm of CDRDT for the concept drifting detection from noisy data streams is described in details at Section 3. Section 4 provides the experimental evaluations and Section 5 is the conclusion.
2
Related Work
Since the model of Random Decision Forests[12] was first proposed by Ho in 1995, the random selection strategy of split-features has been applied into the model of decision trees popularly. And many developed or new random decision trees have appeared, such as [24, 25, 17]. However, it is not suitable for them to handle data streams directly. Sub-sequentially, a random decision tree ensembling method[3] for streaming data was proposed by Fan in 2004. It adopts the crossvalidation estimation for higher classification accuracy. Hu et al. designed an incremental algorithm of Semi-Random Multiple Decision Trees for Data Streams
238
P. Li et al.
(SRMTDS )[14] in 2007. It uses the inequality of Hoeffding bounds with a heuristic method to implement split-test. In the following year, an extended algorithm of MSRT in [11] was further introduced by authors to reduce the impact from noise in the concept-drifting detection. At the same year, H. Abdulsalam et al. proposed a stream-classification algorithm of Dynamic Streaming Random Forests[13]. It is able to handle evolving data streams with the underlying class boundaries drift using an entropy-based drift-detection technique. In contrast with the algorithms based on decision trees ensembling mentioned above, our classification algorithm of CDRDT for concept drifting data streams proposed here behaves with four prominent characteristics. Firstly, the ensemble models of random decision trees developed from semi-random decision trees are generated incrementally in variable sizes of streaming data chunks. Secondly, to avoid the oversensitivity to the concept drifts and reduce the noise contamination, two thresholds are specified to partition their bounds in the inequality of Hoeffding Bound. Thirdly, the check period are adjusted dynamically for adaptation to concept drifts. Lastly, it presents better performances on the abilities of space, time and predictive accuracy.
3
3.1
Concept Drifting Detection Algorithm Based on Random Ensemble Decision Trees Algorithm Description
The classification algorithm of CDRDT to be proposed in this section is for the detection of concept drifts from the data streams with noise. It first generates multiple classifiers of random decision trees incrementally with variable chunks of data streams. After seeing all streaming data in a chunk (i.e., the check period is reached), a concept drifting detection is installed in this ensembling model. By means of the pre-defined thresholds in the Hoeffding Bound inequality, the difference of the average error rates classified in the method of Na¨ıve Bayes or majority-class at leaves are taken to measure the distribution changes of streaming data. Further different types of concept drifts from noise are distinguished. Once a concept drift is detected, we correspondingly adjust the check period to adapt to the concept drift. Finally, a majority-class voting or Na¨ıve Bayes is utilized to classify the test instances. Generally, the process flow of CDRDT mentioned above could be partitioned into three major components: i) the incremental generation of random decision trees in the function of GenerateClassifier. ii) the concept drifting detection methods adopted in ComputeClassDistribution. iii) the adaptation strategies to concept drifts and noise in CheckConceptChange. The related details will be illustrated as follows respectively. Ensemble Classifiers of Random Decision Trees In different from the previous algorithms involved in [11, 14], on one hand, CDRDT utilizes various magnitudes of streaming data chunks to generate ensemble classifiers of random decision trees. Here, random indicates that the split-test
Concept Drifting Detection on Noisy Streaming Data
239
Input: Training set: DSTR; Test set: DSTE ; Attribute set: A; Initial height of tree: h 0 ; The number of minimum split-examples: n min ; Split estimator function: H (·); The number of trees: N ; The set of classifiers: CT ; Memory Constraint: MC and Check Period: CP. Output: The error rate of classification Procedure CDRDT {DSTR, DSTE, A, h 0 , n min , H (·), N, CT, MC, CP } 1. For each chunk of training data streams S j ∈ DSTR (|CP | =|Sj |, j ≥ 1) 2. For each classifier of CT k (1 ≤ k ≤ N ) 3. GenerateClassifier (CT k , S j , MC, CP ); 4. If all streaming data in S j are observed 5. averageError = ComputeClassDistribution(); 6. If the current chunk is the first one 7. fError = averageError ; 8. Else 9. sError = averageError ; 10. If ( j ≥ 2 ) 11. CheckConceptChange(fError, sError, CP, S j ); 12. fError = sError ; 13. For each test instance in DSTE 14. For each classifier of CT k 15. Travel the tree of CT k from its root to a leaf; 16. Classify with the method of majority class or Na¨ıve Bayes in CT k ; 17. Return the error rate of voting classification.
method adopted in our algorithm selects an index of the discretization intervals consisted in ordered values of a numerical attribute randomly and sets the mean value of this interval to a cut-point. On the other hand, it won’t split continuously for nodes with the discrete attributes until the count of instances collected meets the specified threshold (a default value is initialized to two). However, the remainder details of trees’ growing are similar to the descriptions in [11, 14]. Concept Drifting Detection In this subsection, we first introduce several basic concepts relevant to concept drift. Definition 1. A concept signifies either a stationary distribution of class labels in a set of instances at the current data streams or a similar distribution rule about the attributes in the given instances. According to the divergence of concept drifting patterns, the change modes of a concept could be divided into three types of concept drift, concept shift and sampling change as involved in [15].
240
P. Li et al.
Definition 2. The types of concept drift and concept shift belong to the pattern with distinct change speed in the attribute values or class labels of databases. The first one refers to the gradual change and the other one indicates the rapid change. Definition 3. sampling change is mostly attributed to the pattern change in the data distribution of class labels (in this paper all changes are called concept drifts instead.). In CDRDT, a concept drifting detection on the distribution changes of streaming data is installed after a data chunk traverses all of random decision trees. And various types of concept drifts are distinguished from noise in virtue of the relation between the difference of average error rates of classification at leaves and the specified thresholds. Here, the thresholds are specified in the inequality of Hoeffding Bound, whose detailed description is given below: Consider a real-valued random variable r whose range is R. Suppose we have made n independent observations of this variable, and computed their mean r¯, which shows that, with probability 1 - δ , the true mean of the variable is at least r¯ - ε. P (r ≥ r¯ - ε) = 1 - δ, ε=
R2 ln(1/δ)/2n
(1)
Where R is defined as log(M (classes)) and M (classes) indicates the count of total class labels in the current database, the value of n refers to the size of the current streaming data chunk, the random variable of r specifies the expectation error rate classified in the method of Na¨ıve Bayes or majority-class at leaves over all classifiers of random decision trees in CDRDT. Suppose the target object of r¯ is the history classification result in the i th -chunk (denoted as e¯f ) and the current observation object refers to the estimation result of classification in the (i+1)th chunk (marked as e¯s ). The detailed definition of e¯f (¯ es ) is formalized below. k k Mleaf Mleaf e¯f (¯ es ) = 1/N · N [p · n / ki ki k=1 i=1 i=1 nki ]
(2)
In this formula, N signifies the number of total trees, Mkleaf refers to the count of leaves at the k th classifier, nki is the count of instances at the i th leaf in the classifier of CT k and pki is the error rate estimated in 0-1 loss function at the ith leaf in CT k . In terms of Formula (2), we utilize the difference between e¯s and e¯f ( i.e., Δe = e¯s − e¯f ) to discover the distribution changes of class labels. More specifically, if the value of Δe is nonnegative, a potential concept drift is taken into account. Otherwise, it is regarded as a case without any concept drift. This is based on the statistics theory, which guarantees that for stationary distribution of the instances, the online error of Na¨ıve Bayes will decrease; when the distribution function of the instances changes, the online error of the Na¨ıve Bayes at the node will increase[16]. However, for the classification results in the method of majority-class, a similar rule could be concluded from the distribution changes
Concept Drifting Detection on Noisy Streaming Data
241
of class labels in small chunks of streaming data but with sufficient instances as well (In this paper, the minimum size of a data chunk marked as n min is set to 0.2k, 1k = 1000. This is obtained from the conclusion in [22].). It is also verified in our experiments on the tracking of concept drifts in Section 4. Hence, Eq.(1) could be transformed into Eq.(3). P (¯ es - e¯f ≥ ε0 ) = 1 - δ0 , ε0 =
R2 ln(1/δ0 )/2n
(3)
To distinguish diverse concept drifts from noise, it is necessary to specify different values of ε0 to partition their bounds, which refer to the tolerant bounds of deviation between the current error rate and the reference error rate. Evidently, the larger the variance of ε0 the higher the drifting likelihood is. In other words, it is more probable that the previous model won’t adapt to the current data streams due to the deficiency in the accuracy of classification. Correspondingly, the value of δ0 will decrease while the confidence of 1-δ0 will increase. Therefore, with the evocation from [8], two thresholds are defined in the inequality of Hoeffding Bound to control the classification deviation of error rates, i.e., T max and T min . Considering the demand on the predictive ability of the current models, their values are specified as follows. P (¯ es - e¯f ≥ Tmax ) = 1 - δmin , Tmax = 3ε0 δmin = 1 / exp[Tmax 2 ·2n/R 2 ] P (¯ es - e¯f ≥ Tmin ) = 1 - δmax , Tmin = ε0 δmax = 1 / exp(Tmin 2 ·2n/R 2 )
(4)
(5)
Adaptation to Concept Drifts Contaminated by the Noise In accordance with the related analysis mentioned above and the definitions of thresholds specified in Eqs.(4) and (5), four types of concept drifting states would be partitioned, including the ones of a non-concept drift, a potential concept drift, a plausible concept drift and a true concept drift. Namely, if the value of Δe is negative, it is taken as a non-concept drift. Otherwise, it is in a case of other three possible concept drifts. More precisely, if the value of Δe is less than T min , a potential concept drift is considered (potential indicates that the slower or much slower concept drift is probably occurring). And if greater than T max , a true concept drift is taken into account, which is resulted from a potential concept drift or an abrupt concept drift. Otherwise, it is attributed to the state of plausible concept drift considering the effect from the noise contamination. It spans the transition interval between a potential concept drift and a true concept drift. As regards this fuzzy status, it is beneficial to reduce the impact from the noise in data streams and avoid over-sensitivity to the concept drifts. Correspondingly, different strategies are adopted to handle various types of concept drifts. More specifically, for the case of non-concept drift, maintain the size of the current data chunk in a default value (e.g., n min ). For the potential concept drift, increase the chunk size by the number of instances-m min (e.g., m min = n min = 0.2k). However, for a plausible concept drift, shrink the size of
242
P. Li et al.
streaming data chunk and the check period by one third respectively. Because it is necessary to further observe the change of data streams for a deterministic type of concept drift. Otherwise, for a true concept drift, reduce the sizes into a half of the original values. Regarding the disadvantages of streaming data chunks with too large or too small sizes, the maximum bound (e.g., m max = 10*n min ) and the minimum one (e.g., m min ) are specified to control the change magnitude of a data chunk for better adaption to the concept changes. It indicates that if a bound is reached, the check period remains invariable until a new concept drift occurs. Furthermore, to improve the utility of each tree, those sub-branches whose error rates of classification are lower than the average level (e.g., 50%) will be pruned. 3.2
Analysis
Generation Error Rate for the Concept Drifting Data Streams According to the theorem of generation error analyzed in [17], as the number of trees increases, for almost surely all sequences Θ1 . . ., the generation error of PE will converge to P X,Y (P Θ (h(X, Θ)=Y ) - maxj =Y (P Θ (h(X, Θ)=j ))<0)
(6)
where X is the training set; Y means the class label, Θ specifies the random vector of feature generated from the attribute set, P (X, Y ) indicates the probability over the X, Y space and h(X, Θ) refers to the classifier. Eq.(6) is concluded on the assumption that the sequences of Θ1 . . . are independent identically distributed random vectors. However, regarding the concept drifting data streams, it is probable that the streaming data distributions are not uniform any more as the time flows. As a result, it is improper to judge the convergence of generation error in this case. Therefore, in the analysis of our ensembling model, we give an infimum bound of generation error. Because each detection on concept drift is installed every a certain period of instances, the training data are divided into small sequences, i.e., θt (t ∈{1, 2, ..., |B|}, |B| is the maximum index of sequence.). In an overall consideration, each generation error of our model in a chunk of θt could be expressed below: PE Tθtt = P θt (V (T t , θt ) = Y )-Maxj=Y P θt (V (T t , θt )=j )
(7)
where T t specifies the current decision tree ensemble and each tree is generated or updated with the data chunks composed of {θk , 1 ≤ k ≤ t }; V (·) signifies the voting function, which acts on the data chunk of θt classified by the current ensemble random decision trees. Considering the worst case, the generation error would be defined in Formula (8). Tt t PE ∗ = Max(P (Tt ,θt ) (PE T θt <0)) ≥ P (Tt ,θt ) (PE θt <0)
(8)
Concept Drifting Detection on Noisy Streaming Data
243
Due to M axj=Y Pθt (V (Tt , θt ) = j ) ≤ 1 − Pθt (V (Tt , θt ) = Y ), Eq.(8) hence could be written as Eq.(9): PE ∗ ≥ P (Tt ,θt ) (P θt (V (T t , θt )=Y )≤ 0.5)
(9)
Based on the analysis on the probability of optimal ensemble model in [14], i.e., h0 P (M (Attr ), N, h 0 ) = 1-(1-1/M (Attr ))N·2 −1 , we take it as an estimation on the
classification accuracy. Therefore, the generation error referring to the probability of P (M (Attr ), N, h 0 ) ≤ 0.5 would be formalized into Eq.(10), PE ∗ ≥ P (P (M (Attr ), N, h 0 )≤0.5)
(10)
It clearly shows that the higher the optimal probability for an ensemble model the less the generation error rate is. As a consequence, we could adjust the number of trees or the heights of trees to improve the predictive accuracy for the adaptation to concept drifts.
4
Experiments
To verify the efficiency and effectiveness of CDRDT in the detection on different types of concept drifts from noisy data streams, extensive experiments are conducted on the diverse benchmark concept drifting databases and the real streaming data obtained from Yahoo! Shopping Web Service. And the experimental study presents that not only could CDRDT detect concept changes timely and effectively with a certain resilience to the noise, but also outperforms on the abilities of runtime & space and the predictive accuracy as compared with CVFDT and MSRT. Therefore, this section would be divided into two parts correspondingly. The first one discusses the characteristics of all conceptdrifting databases used in our experiments. And the second one analyzes the drifting track in CDRDT and the performances on runtime, space and predictive accuracy (all experiments referred here are performed on a P4, 3.00GHz PC with 1G main memory, running Windows XP Professional. Furthermore, all algorithms used in our experiments are written in Visual C++.). Due to the limited space, only partial experimental results are given as follows. 4.1
Data Source
Synthetic Data HyperPlane. HyperPlane is a benchmark database of data streams with the gradual concept drift, which has been used in a numerous of references included [10, 3, 11, 2]. A HyperPlane in a d -dimensional space (d =50) is denoted by equa tion: d i=1 w i x i =w 0 . Each vector of variables (x 1 , x 2 , · · ·, x d ) in this database is a randomly generated instance and is uniformly distributed in the multidimen sional space[0, 1]d . If d i=1 w i x i ≥ w 0 , the class label is 1, or else is 0. The bound
244
P. Li et al.
of coefficient w i is limited to [-10, 10]. For a weight of w i , each initial value is generated at random. And then it increases or decreases continuously by the value of Δw i = 0.005 till it is up or down to the boundary, further changes the direction with the probability of p w =10%). Meanwhile, in order to simulate the concept drifting case, we select 5-dimension to change their weights in the database with the noise rate of r =10%. SEA. The artificial data of SEA first described in [1] is a well-known data set of concept shift with numerical attributes only. It is composed of 60k random points in a three-dimensional feature space with two classes. All three features have values between 0 and 10 but only the first two features are relevant. Those points are divided into four chunks with different concepts. In each chunk, a data point belongs to class 1 if f 1 +f 2 ≤ θ, where f 1 and f 2 represent the first two features and θ is a threshold value between these two classes. In this database, there are four thresholds of 8, 9, 7 and 9.5 to divide data chunks. Each chunk reserves 2.5k-sized records as test sets containing 10% class noise for the different concepts. And the rest 50k-sized points are treated as the training data, in which each of concept shift appears every 12.5k-sized instances. KDDCup99. The KDDCup99 database[20] is a database for network intrusion detection, which is selected here because it has been simulated as streaming data with sampling change in [15]. In this database, the count of attributes is 41 dimensions with 34-dimensions of numerical attributes and the number of class labels is 24 totaly. Due to the skew distribution of class labels, the data with minor rates of class labels (i.e., the total number is lower than the value of n min ) are taken as the noise data. Hence, the data set without noise contains 490k-sized instances with 12 class labels. Real Data Yahoo! Shopping Data. The web shopping data used in our experiments are obtained via the interface of Yahoo! web services. They are sampled from Yahoo! shopping databases relevant to catalog listing, product search and merchant search. The basic feature of this data set contains 17 dimensions of attributes and the total number of numerical ones is 10 dimensions. Meanwhile, it is composed of the product information with the attribute set of (NnumeratingofProduct, AverageRating, etc.) and the related information of merchants included attributes of (NumRatingsofMerchant, Price-SatisfactionRating, OverallRating etc.). The correlation between a product and a merchant is connected by the catalog listing (see [21] for more details). To mine the relation between the credibility of merchants and possible factors, the attribute of OverallRating with different scores are defined as our class labels, which are divided into five class labels. By the label distribution, we extract the number of 84k-sized instances randomly from the obtained records as a training set and the rest 28k-instances as a test set. 4.2
Experimental Evaluation on Synthetic Databases
Before introducing the experimental results evaluated on synthetic databases, several symbols involved in our experiments would be given firstly in the above
Concept Drifting Detection on Noisy Streaming Data
245
Symbol Max/Bayes
Description the classification method of majority-class/Na¨ıve Bayes. “∗ ∗ ∗” refers to the name of a database. The whole symbol refers that the ex∗ ∗ ∗-Max/Bayes perimental results are classified by the method of Max/Bayes. Error rate the error rate estimated in a test set or a specified data chunk, Unit: (%). The value in the former is averaged over 20 runs classified by 20*N -trees. the training + test time (algorithms based on ensemble decision trees run in T+C time a simulated parallel environment of multiple PCs. Thus, the time overhead is calculated by the largest one of N trees, Unit: (s). But for CDRDT, the “T” time is computed by the total generation time of N -trees here. Memory the total memory consumption of all trees, Unit: (M). All results are averaged over 20 runs as similarly as the results with Error rate. structure: the name + the size of training data + the size of test data + the Database name database type (i.e., C: numerical, D: discrete, CD: hybrid) + the number of attribute dimensions, e.g., SEA-50K-2.5K-C-3. Drift-Track the process of detection on the concept drifts in a certain time-interval. for KDDCup99, it refers to the index of different class labels, marked as ClaDrift-Level ss label; For HyperPlane, it stands for the difference rate of class labels in comparison with an original database after introducing the noise, marked as Drift-rate, Unit: (%). A unified name is called Drift Level. Period-Change a short for the size of a data chunk in the concept drifting detection, i.e., the count of instances for each detection, Unit: (k).
table. Secondly, on the setting of parameters, the parameters in CVFDT and MSRT still follow the original definitions in [10, 11] respectively. While for CDRDT, the parameter values are specified below: N = 10, h 0 = M (attr)/2 as defined in [22], an initial size of a streaming data chunk of |S j | = 0.2k (i.e., |CP |), MC = 500k and δmax = 0.1 (ε0 and δmin are calculated in Eqs.(5) & (4) respectively). Now, the experimental details will be described as follows. Tracking Concept Drifts The tracking curves of concept drifts are plotted in Figures 1∼4, which present various detection cases on the databases with different drifting characteristics. In these figures, the curves of Drift-Track are drawn in solid lines with the scale of the left y-axis and the Drift-Level curves are described in dotted lines with the scale of the right y-axis. While for the changing values of Period-Change, we use a plus sign of “+” to represent in the corresponding figures. Its definition in scale is specified below: the lowest value starts from 1k and the basic unit (denoted as BU ) is set to 0.5k based on the scale of the left y-axis. In Figure 1-a, due to the different magnitudes of Drift-Level, the corresponding tracking curve fluctuates variously. Especially at the beginning of detection, there are larger discrepancies owing to the insufficiency of training data. Furthermore, when the current drifting level transfers to another one, a jump will appear in the tracking curve from a local minimum point to a local maximum one. However, due to the gradual concept drifts in HyperPlane, little deviation occurs between the adjacent detection results. As a result, most of check periods are maintained stably. Moreover, with the increasing of streaming data, the fluctuation trend is gradually converging. Considering the concept shifting detection on SEA, the tracking results fluctuate with three shifting occurrences as shown in Figure 2. As compared with the detection cases in the methods of Max and Bayes, there are several common characteristics: i) the fluctuations are frequent at the beginning of the training
246
P. Li et al.
Fig. 1. (left:) Drift track on HyperPlane (right:) Classification results
SEA-database-Max 36.0 35.4
2.0
37.6
1.8
37.0
) % (
34.8
Drift-Track
e t a r r o r r E
34.2 33.6 33.0
Period-Change
32.4
1.6
36.4
1.4
35.8
1.2
35.2
) k 1.0 (
34.6
d o 0.8 i r
34.0
e p 0.6
31.8
k c 0.4 e h C 0.2
31.2 30.6 30.0
0.0 1
9
17
25
33
41
49
Index of data chunk Fig. 4-a
57
65
73
81
SEA-database-Bayes
2.0 1.8
) % (
1.6
e t a r r o r r
1.4 1.2
E
Drift-Track
) k 1.0 ( d o 0.8 i r e p 0.6
Period-Change
33.4
k c 0.4 e h C 0.2
32.8 32.2 31.6
0.0 1
9
17
25
33
41
49
57
65
73
81
89
97
105
Index of data chunk Fig. 4-b
Fig. 2. Drift track over sequential data chunks for SEA
even if without any concept change. This is similar to the case in HyperPlane; ii) the upward and downward trends in curves alternately take place with the shifting of the concepts. This is mainly resulted from the distribution changes with only two class labels in this database, which contains four concepts in total varying from the rate of 1.8: 1 to 0.85:1 (In this figure, both of the minimum period and the unit of period refer to 0.2k-sized instances, which take the left y-axis as the scale axis). With respect to the detection on the sampling change, the tracking curves drawn in Figures 3∼4 describe the detection results on KDDCup99 in the classification methods of Max and Bayes respectively. In the observations, we can see that the more the frequent changes in the distribution of class labels the fiercer the fluctuations occur. It is demonstrated noticeably in the curve segment spanning the interval from the 1st data chunk to the 121th one (For KDDCup99, the default value of check period is set to 1k and the unit of BU is 0.2k. In Figure 3, the distance of two basic units in the left y-axis signifies one BU for a conspicuous description while a unit refers to one BU in Figure 4.). However, if the stable distribution of a class label is reached, the trend of tracking curves will fall down, such as during the curve segment between the 121th chunk and
Concept Drifting Detection on Noisy Streaming Data
5.6
4.8 4.4 4.0 3.6
22
KDDCup99-database-Max
e t a r ) r % o ( r r E
5.2
247
s s a l C
l e b a l
14
Drift-Level
2.8
12 10
Period-Change
2.4
18 16
Drift-Track
3.2
20
2.0
8
1.6
6
1.2
4
0.8
2
Index of data chunk
0.4 0.0
0 1
41
81
121
161
201
241
281
321
361
401
441
Fig. 3. Drift track over sequential data chunks classified with Max for KDDCup99 3.9
3.3 3.0 2.7
22
KDDCup99-database-Bayes
e t a ) r r % o ( r r E
3.6
s s la
C
l 20 e b 18 la 16
Drift-Track
2.4
14
Drift-Level
2.1
12
Period-Change
1.8
10
1.5
8
1.2
6
0.9 4
0.6 0.3 0.0
2
Index of data chunk 1
41
81
121
161
201
241
281
321
361
401
441
0
Fig. 4. Drift track over sequential data chunks classified with Bayes for KDDCup99
the 321th one. Hence, the check period maintains invariably because the state of non-concept drift is considered in this case. Predictive Accuracy and Overheads of Time and Space Except of the drifting detection on the training data, we also evaluate the classification abilities of CDRDT on the different test sets in comparison to CVFDT and MSRT. To be concrete, firstly, the results classified on Hyperplane are plotted in Figure 1-b, including the overheads of runtime & space and the mean error rates of classification added variances. It is clear that CDRDT outperforms other two algorithms on all abilities involved above. Secondly, the classification results on SEA and KDDCup99 are also concluded in Tables 1 and 2 respectively. As shown in Table 1, though in the Max classification method, CDRDT does not perform as well as other algorithms on the predictive accuracy, the error rate could be reduced by 6.15% at least if adopting the method of Bayes. Furthermore, as similar to the case in HyperPlane, the overheads of space and runtime are lowest, which the largest deviation is up to dozens of times. However, in Table 2, the predictive accuracy in CDRDT is improved largely by 14.76% and 36.61% respectively in Max as compared with CVFDT and MSRT. Meanwhile, in the case with the Bayes method, the superiority of predictive accuracy in CDRDT is also prominent. Moreover, the performances on both of the runtime and space consumption are much better as well.
248
P. Li et al. Table 1. Classification results on SEA-50k-2.5k-C-3 Algorithm
CDRDT MSRT CVFDT
Error rate(%) Max Bayes mean variance mean variance 44.32 3.762 13.53 1.354 25.98 2.770 19.66 2.344 25.24 /
(T+C)time(s)
Memory(M)
Max 0+0 0+0 8+0
Max <1
Bayes 0+0 0+0 /
41
Bayes 5 7*5 /
Table 2. Classification results on KDDCup99-490k-310k-CD-41 Algorithm
CDRDT MSRT CVFDT
4.3
Error rate(%) Max Bayes mean variance mean variance 8.87 0.169 9.06 0.488 44.48 22.350 28.60 19.896 23.48 /
(T+C)time(s)
Memory(M)
Max 32+19 93+18 75+18
Max Bayes 6 7 65 ∗ 5 20 /
Bayes 41+123 92+504 /
Experimental Evaluation on Web-Shopping Data
For the real-world data streams, it is hard to judge whether the current data streams carry a potential concept drift or when a concept drift occurs. Meanwhile, it is inevitable to be affected from the noise. Therefore, to verify the feasibility and utility of our algorithm, we conduct several comparison experiments on real data streams of Yahoo! shopping data with other algorithms as well. The experimental results listed in Table 3 show that CDRDT is superior to CVFDT and MSRT in the predictive accuracy and the overheads of runtime and space. For instance, the predictive accuracy in CDRDT is highest, which is improved by 9.32% averagely even though in the worst case. And for the training time, it is reduced by the times of 3/2 at least with an approximate overhead on the test time in contrast to other two algorithms. In addition, on the space consumption, the maximum rate only takes a half of the space consumption in CVFDT while the minimum one is only 1/25 of MSRT. Table 3. Classification on Yahoo!-shopping-data-84k-28k-CD-16 Algorithm
CDRDT MSRT CVFDT
5
Error rate(%) Max Bayes mean variance mean variance 14.02 5.545 4.45 1.936 33.52 17.891 44.42 39.166 23.34 /
(T+C)time(s)
Memory(M)
Max 2+1 11+1 10+3
Max Bayes <1 6 30 ∗ 5 13 /
Bayes 6+5 15+20 /
Conclusion
In this paper, we have proposed an ensembling classification algorithm named CDRDT for Concept Drifting detection from noisy data streams, which is based on Random Decision Trees. In contrast to previous efforts on ensemble classifiers of decision trees or random decision trees, small data chunks with unfixed sizes are adopted in CDRDT to generate the classifiers of random decision tress incrementally. To effectively distinguish different types of concept drifts from noise,
Concept Drifting Detection on Noisy Streaming Data
249
two thresholds are defined in virtue of the inequality of Hoffeding Bounds. Furthermore, for better adaptation to the concept drifts, the check period is adjusted dynamically and timely. Moreover, extensive experiments are conducted on three types of synthetic concept drifting databases and a real-world database of Yahoo! shopping data. And the experimental results demonstrate that CDRDT could adapt to various concept drifts efficiently in the noisy data streams. In addition, as compared with the state-of-the-art algorithm of CVFDT and an ensemble algorithm of MSRT, it outperforms on the abilities of runtime & space and the predictive accuracy. Hence, a conclusion is drawn in the study that CDRDT is a light-weighted ensembling algorithm of classification. It would provide an efficient method for the detection on a variety of concept drifts in data stream. However, how to model the noise data to discern the concept drifts from noise accurately and how to deal with the cases with the skewed distribution of class labels in data streams are still challenging and interesting issues for our future work.
References 1. Street, W., Kim, Y.: A streaming ensemble algorithm (sea) for large-scale classification. In: 7th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD 2001, pp. 377–382. ACM Press, New York (2001) 2. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining Concept-Drifting Data Streams Using Ensemble Classifiers. In: 9th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD 2003, pp. 226–235. ACM Press, New York (2003) 3. Fan, W.: Streamminer: a classifier ensemble-based engine to mine concept-drifting data streams. In: 30th international conference on Very Large Data Bases, VLDB 2004, pp. 1257–1260. VLDB Endowment (2004) 4. Chu, F., Wang, Y., Zaniolo, C.: An adaptive learning approach for noisy data streams. In: 4th IEEE International Conference on Data Mining, pp. 351–354. IEEE Computer Science, Los Alamitos (2004) 5. Gama, J., Fernandes, R., Rocha, R.: Decision trees for mining data streams. Intelligent Data Analysis 10, 23–45 (2006) 6. Scholz, M., Klinkenberg, R.: Boosting Classifiers for Drifting Concepts. Intelligent Data Analysis (IDA), Special Issue on Knowledge Discovery from Data Streams 11(1), 3–28 (2007) 7. Hoeffding, W.: Probability inequalities for sums of bounded random variabless. Journal of the American Statistical Association 58(301), 13–30 (1963) 8. Castillo, G., Gama, J., Medas, P.: Adaptation to Drifting Concepts. In: Pires, F.M., Abreu, S.P. (eds.) EPIA 2003. LNCS (LNAI), vol. 2902, pp. 279–293. Springer, Heidelberg (2003) 9. Gehrke, J., Ganti, V., Ramakrishnan, R., Loh, W.Y.: Boat-optimistic decision tree construction. In: 1999 ACM SIGMOD International Conference on Management of Data, pp. 169–180. ACM Press, New York (1999) 10. Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: 7th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD 2001, pp. 97–106 (2001)
250
P. Li et al.
11. Li, P., Hu, X., Wu, X.: Mining concept-drifting data streams with multiple semirandom decision trees. In: Tang, C., Ling, C.X., Zhou, X., Cercone, N.J., Li, X. (eds.) ADMA 2008. LNCS, vol. 5139, pp. 733–740. Springer, Heidelberg (2008) 12. Ho, T.K.: Random decision forests. In: 3rd International Conference on Document Analysis and Recognition, pp. 278–282. IEEE Computer Society, Los Alamitos (1995) 13. Abdulsalam, H., Skillicorn, D.B., Martin, P.: Classifying Evolving Data Streams Using Dynamic Streaming Random Forests. In: Bhowmick, S.S., K¨ ung, J., Wagner, R. (eds.) DEXA 2008. LNCS, vol. 5181, pp. 643–651. Springer, Heidelberg (2008) 14. Hu, X., Li, P., Wu, X., Wu, G.: A semi-random multiple decision-tree algorithm for mining data streams. Journal of Computer Science and Technology 22(5), 711–724 (2007) 15. Yang, Y., Wu, X., Zhu, X.: Combining Proactive and Reactive Predictions for Data Streams. In: 11th ACM SIGKDD international conference on Knowledge Discovery in Data mining, KDD 2005, pp. 710–715. ACM Press, New York (2005) 16. Mitchell, T.: Machine Learning. McGraw Hill, New York (1997) 17. Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001) 18. Quinlan, R.J.: C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc., San Francisco (1993) 19. Shafer, J., Agrawal, R., Mehta, M.: Sprint: A scalable parallel classifier for data mining. In: 22th International Conference on Very Large Data Bases, VLDB 1996, pp. 544–555. Morgan Kaufmann, San Francisco (1996) 20. KDDCUP 1999 DataSet, http://kdd.ics.uci.edu//databases/kddcup99/kddcup99.html 21. Yahoo! Shopping Web Services, http://developer.yahoo.com/everything.html 22. Li, P., Liang, Q., Wu, X., Hu, X.: Parameter Estimation in Semi-Random Decision Tree Ensembling on Streaming Data. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, T.-B. (eds.) PAKDD 2009. LNCS (LNAI), vol. 5476, pp. 376–388. Springer, Heidelberg (2009) 23. Wikipedia, http://en.wikipedia.org/wiki/Data_stream 24. Amit, Y., Geman, D.: Shape quantization and recognition with randomized trees. Neural Computation 9, 1545–1588 (1997) 25. Ho, T.K.: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)
Mining Multiple Level Non-redundant Association Rules through Two-Fold Pruning of Redundancies Corrado Loglisci and Donato Malerba Dipartimento di Informatica Universita’ degli Studi di Bari Via Orabona 4, 70125, Bari - Italy {loglisci,malerba}@di.uniba.it
Abstract. Association rules (AR) are a class of patterns which describe regularities in a set of transactions. When items of transactions are organized in a taxonomy, AR can be associated with a level of the taxonomy since they contain only items at that level. A drawback of multiple level AR mining is represented by the generation of redundant rules which do not add further information to that expressed by other rules. In this paper, a method for the discovery of non-redundant multiple level AR is proposed. It follows the usual two-stepped procedure for AR mining and it prunes redundancies in each step. In the first step, redundancies are removed by resorting to the notion of multiple level closed frequent itemsets, while in the second step, pruning is based on an extension of the notion of minimal rules. The proposed technique has been applied to a real case of analysis of textual data. An empirical comparison with the Apriori algorithm proves the advantages of the proposed method in terms of both time-performance and redundancy reduction. Keywords: Association Rules, Multiple Level Rules, Redundant Information.
1
Introduction
Association rules are a class of patterns which describe regularities or co-occurrence relationships in a set of data (e.g., transactions) [1]. Formally, an association rule is expressed in form of A⇒C, where both the antecedent A and the consequent C are sets of items (or itemsets) such that A∩C =. The meaning of an association rule is quite intuitive: a transaction which contains A is likely to contain C as well. To quantify this likelihood, two statistical parameters are usually used, namely support and confidence. The former, denoted as s(A ⇒ C), indicates the portion of the transactions where the conjunction A∪C occurs, and hence estimates the probability P(A∪C). The latter, denoted as c(A ⇒ C), indicates the portion of the transactions where C occurs out of the transactions in which A is present, thus it estimates the posterior probability P(C|A). The problem of association P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 251–265, 2009. c Springer-Verlag Berlin Heidelberg 2009
252
C. Loglisci and D. Malerba
rule mining consists of finding out all rules whose support and confidence values exceed two user-defined thresholds, called minsup and minconf respectively. The blueprint for most association rule mining algorithms proposed in the literature is a two-stepped procedure, according to which given a dataset D of transactions and the values of minsup and minconf, the method 1. finds out all frequent itemsets, i.e., the itemsets with support greater than or equal to minsup; 2. generates all association rules X ⇒ Y − X, where Y is a frequent itemset, X ⊂ Y and the confidence is greater than or equal to minconf. At the end of the second step, the rules generated are called valid (or strong) rules. Association rules are largely applied to many domains, but their usage remains problematic because of the huge number of association rules typically discovered and the resulting difficulty met in their analysis and interpretation. This problem is exacerbated in presence of redundant rules, i.e., AR which convey the same information conveyed by other rules of the same usefulness and the same relevance [5]. This important issue has attracted the attention of the research community (see [12] for a recent overview) and several methods have been proposed to obtain non redundant sets of rules which are in some way representative of the whole space of valid rules. Two main approaches are well identifiable: i) the usage of user subjective criteria and ii) the selection of relevant rules on the basis of statistical metrics. The former is followed, for instance, by Baralis et al. [4] who propose a framework to extract rules on the basis of user specifications described in form of templates. In [11] the user criteria are rather used to define a notion of interestingness which filters out all the uninteresting rules. The second approach resorts to post-processing techniques which return compact subsets with respect to some heuristic or statistical measures [9,3]. A different but not deeply investigated research line is that followed by the current work, that is, eliminating redundancies during the mining process, namely when frequent itemsets and/or valid rules are generated: discussion on the works proposed in the literature is postponed in Section 7. The problem of the redundancy in the AR is even more serious in the case of multiple level (or generalized) association rules [7],[2],[14], which represent an important extension of the AR and are discovered by exploiting a pre-defined taxonomic arrangement of the items. However, the organization of the items on several taxonomic levels leads to a combinatorial increase of discovered rules, many of which do not add further information to that expressed by a subset of them. More importantly, redundant rules makes the amount of work of the end user unbearable and, in ultimate analysis, it can cause the failure of a data mining project. In this paper, a novel method for the discovery of non-redundant AR at several taxonomic levels is proposed. It follows the usual two-stepped procedure for AR mining and prunes redundancies in each of two steps as sketched in the following. In the first step, a set of non-redundant frequent itemsets, named as multiple
Mining Multiple Level Non-redundant Association Rules
253
level closed frequent itemsets, is found out by extending the concept of closed itemsets. In the second step, the set of minimal association rules is produced on the basis of frequent itemsets previously found. The rest of the paper is organized as follows. In the next section, we illustrate the motivation of this work and explain our contribution. In Section 3 the problem of redundancy reduction is re-formulated two sub-problems: pruning the itemsets and pruning the set of generated rules. An algorithm which solves the former is proposed in Section 4, while a solution to the latter is reported in Section 5. In Section 6, we present the application to a task of textual data analysis where the multiple level non-redundant rules are exploited for discovering meaningful associations of biomedical concepts from biomedical literature. In Section 7 works related to ours are shortly presented and discussed. Finally, some conclusions close this paper.
2
Motivation and Contribution
Most approaches to keeping association rule redundancy under control operate either by pruning the set of rules which are deemed irrelevant according to some subjective notions of interestingness, or by preventing the generation of redundant rules during the mining process. Poor attention has been paid to the problem of removing redundant information when a taxonomic arrangement of the items is available. A taxonomy categorizes the items over several hierarchical levels by groups or classes. Its consideration in AR mining permits the generation of rules with hierarchically related items (multiple level ARs). Algorithms for multiple level AR mining do not face the problem of redundant information: they are basically focused on the strategies to scan the taxonomy. Our contribution aims to fill this gap through an approach that revises the usual two-stepped procedure. More precisely, we introduce two additional pruning criteria: – For the first step, by extending the concept of closed itemsets [13] to the case of hierarchically organized items. An itemset is closed if none of its supersets has the same support. For instance, if two itemsets A, B and A, B, C have the same support, then A, B is not closed, while if A, B, C has a strictly lower support than A, B, then A, B is closed. The interest toward closed itemsets is that they preserve information and help to keep computational complexity under control. – For the second step, by applying the notion of minimal rules [5]. For instance, if the following three rules R1 : A ⇒ B, C, R2 : A ⇒ B, C, D, and R3 : A ⇒ B, C, D, E have identical support and confidence, then R3 is a minimal rule in the set of three, since R1 and R2 can be derived from by R3 and convey no additional information. In the following sections we first provide a formal description of the problem, then we describe our two-fold contribution to redundancy removal during the mining process.
254
3
C. Loglisci and D. Malerba
Formal Statement
Let I: {x1 ,. . . , xn , y1 ,. . . ,ym , z} be a set of distinct literals called items and X be an itemset that contains k items (k-itemset), |X|=k. Given D a transaction set, each transaction T supports an itemset X if X⊆ T. The portion of transactions T⊆ D supporting an itemset X is the support of X, s(X). X is called frequent itemset if s(X)≥ minsup.
Fig. 1. Representation in form of hierarchical structure of a taxonomic arrangement of the items
Let G be a taxonomy represented in form of hierarchical structure over the items in I (see Figure 1) and organised by subtype-supertype (or is-a) relationships. The items at lowest levels are named as leaf items, those at intermediate levels as inner items, while the items at highest level as root item. An edge xi ← yj in G denotes a subtype relationship: the item xi is the direct subtype of yj , or conversely, the item yj is the direct supertype of xi . An item z is a supertype of xi if there is a path from xi to z (xi is a subtype of z). A transaction T supports an item yj ∈ I when yj ∈ T or ∃ xi ∈ T such that yj is a supertype of xi . An itemset Y is called multiple level itemset if it does not contain an item and its supertypes as well. An itemset Y is a supertype itemset of an itemset X, if Y can be obtained by replacing one or more items in X with one of their supertypes and |X| = |Y| (conversely X is a subtype itemset of Y). Given an itemset Y and Z one of its subsets, Z⇒ Y-Z is called multiple level association rule if no one of the items contained in the antecedent Z (conversely, consequent Y-Z) is supertype of any item contained in the consequent Y-Z (conversely, antecedent Z). Interestingly, rules containing items from the upper taxonomic levels represent more generalized information, conversely, those containing items from the lower levels represent more specialized information. Now we introduce some preliminary notions for explaining our contribution: – dXi be a function which maps I into a set of positive integer numbers, where dXi (xi )=1 iff xi is the root item in G, otherwise dXi (xi )= dXi (direct supertype (xi )) +1; – dX be a function which maps I into a set of positive integer numbers, where dX (X)= max(dXi (xi )), xi in X. – h be the composition f˚g where f and g are defined as follows:
Mining Multiple Level Non-redundant Association Rules
255
• f(D’): ℘(D’)→ ℘(I’), which associates with D’ the items common to all the transactions tk ∈ R and returns the set { xi ∈ I’ | ∀ tk ∈ D’, (tk , xi ) ∈ R }; • g(I’): ℘(I’) → ℘ (D’), which associates with I’ the transactions related to all items xi ∈ I’ and returns the set { tk ∈ D’ | ∀ xi ∈ I’, (tk , xi ) ∈ R }; where R⊆ D×I is a binary relation with I’⊆ I, D’⊆ D. The function h is commonly called Galois closure operator [6],[13]. Now it is worth of reminding that: i. an itemset X ⊆ I is closed iff h(X)=X. Moreover from [13] it follows that X is closed if none of its supersets has the same support as X (i.e., s(h(X))=s(X)); ii. given an itemset X and xi ∈ I: g(X)⊆ g( xi ) ⇔ xi ∈ h(X) [12]; The concept of closed itemset is fundamental in this work: we extend it to the case of taxonomically organized items for removing redundant multiple level itemsets. In formal terms we have: Definition 1 (Multiple Level Closed Itemset). Let Y be a multiple level itemset, Y= y1 , y2 ,. . . , yj ,. . . ,yh , it is called multiple level closed itemset(mCI) iff: 1. item w, w ∈ I, w ∈ / Y, w is not supertype of yj ∈Y; 2. dX (Y) ≥ dXi (w); 3. | g(Y) - g(w )| ≤ , positive user-defined threshold (when =0 g(Y) ⊆ g(w )). It can be argued in this way. Assume =0 (for the sake of simplicity) and suppose an item w which is neither contained in Y and nor supertype of yj (yj ∈Y), and Y y1 , y2 ,. . . , yh a closed itemset where g(Y)⊆g(w). This means that an itemset Y’ so composed y1 , y2 ,. . . , yh , w exists and meets the relationships g(Y’)=g( y1 , y2 ,. . . , yh , w)= g(Y) ∩ g(w). Since g(Y) ∩ g(w)=g(Y) it follows that s(Y’)= s(Y), and then, according to the relation s(h(Y))=s(Y), Y is not closed itemset: this is in contradiction with the initial hypothesis, and then validates the conditions in Definition 1. Moreover, by the third condition of Definition 1 we know that an itemset X is redundant w.r.t Y if Y is supported by at least the same number of transactions that supports X, and if X and Y are supported from common transactions (respect to the threshold). An explanatory toy example follows. Example 1. Consider the taxonomy in Figure 2, the transaction set in Table 1, as 0 and the item w=B. It can be derived that the itemset A is a mCI while the itemset A11 ,A12 does not. Indeed, since g(A) is {1,2,3,4,5,6} and g(B) is {1,2,3,5} the relationship g(A)⊆ g(B) does not hold, hence B ∈ / h(A), finally A is a mCI (point ii presented above). On the contrary, g(A11 ,A12 )={3,5}, hence the relationship g(A11 ,A12 )⊆ g(B) does hold, then B ∈ / h(A11 ,A12 ), finally A11 ,A12 is not a mCI. Trivially it holds for any subtype of B.
256
C. Loglisci and D. Malerba
Table 1. A toy set of boolean transaction on leaf items. The taxonomic arrangement is reported in Figure 2. Transaction 1 2 3 4 5 6
A11 1 0 1 1 1 0
A12 0 1 1 0 1 0
B11 1 0 1 0 1 0
B12 1 0 1 0 1 0
A1 1 1 1 1 1 0
A2 1 0 0 1 1 1
B1 1 0 1 0 1 0
B2 1 1 1 0 1 0
A 1 1 1 1 1 1
B 1 1 1 0 1 0
Fig. 2. Representation of the taxonomy over the items for the example in Table 1
Considering the formal framework thus far described, the problem of interest in this work can be formulated as follows: Given: D transaction set, I set of items, G a taxonomy over the set I, minsup, minconf user-defined minimal thresholds for support and confidence, and as user-defined value. Goal: 1. Finding out the set of multiple level closed frequent itemsets (mCFI) w.r.t. to such that the support of each mCFI exceeds minsup; 2. Generating the set of multiple level association rules (mAR) exceeding the value of minconf. In the following two sections we present the computational solutions to these sub-problems.
4
An Algorithm for Finding Out the Multiple Level Closed Frequent Itemsets
The first sub-problem is here solved with an algorithm, named DELIS (DEscending Levels and Increasing Size) which scans the taxonomy in top-down way while, at each level, generates mCFIs with increasing size (or length). In particular the scanning strategy is based on a suitable order relation ”≺” over the items, which meets the following:
Mining Multiple Level Non-redundant Association Rules
given a set of items
1
257
I, a taxonomy G over I, a function O: I→ {1,2,. . . ,|I|},
i. xi ≺ xj (∀ xi ,xj ∈ I) iff dXi (xi ) = dXi (xj ) and O(xi )< O(xj ) (e.g., in Figure 2 O(A1 )=2, O(A2 )=6: A1 ≺ A2 ); ii. xi ≺ xik ≺ xj (∀ xi ,xj ∈ I, ∀ xik direct subtype of xi ) iff dXi (xi )= dXi (xj ), dXi (xik )= dXi (xi )+1 and O(xi ) < O(xik )< O(xj ) (e.g., in Figure 2 O(A1 )=2, O(A2 )=6, O(A11 )=3: A1 ≺ A11 ≺ A2 ); m iii. given two itemsets X1 , X2 , |X1 | = |X2 | = n, dX (X1 )≤ dX (X2 ), ∃ xm i xj , m m m≤n, the m-th items of X1 , X2 respectively, xi ≺ xj : X1 ≺ X2 . As we will see in the description of the algorithm 1 such a order relation permits: – to return frequent itemsets only if their supertypes are frequent too; – to generate frequent k-itemsets by joining frequent (k-1)-itemsets; – to consider closed frequent itemsets only if their supertypes are closed frequent too. At this aim the algorithm exploits the theorem reported below. Theorem 1. Let X1 = x1 , x2 ,. . . , xj ,. . . ,xh be an itemset, X’1 = x1 , x2 ,. . . , x’j ,. . . ,xh its subtype itemset, xj supertype of x’j , X1 is not a closed itemset: X’1 is not a closed itemset. Proof : Assume as 0 and X1 is not a closed itemset. By resorting to the point ii in the previous section it is possible considering that it exists an item of I xk ∈ / X1 such that the relationship g(X1 ) ⊆ g(xk ) holds and xk ∈ h(X1 ). Hence it follows that g(X’1 ) ⊆ g(X1 ) ⊆ g(xk ), namely g(X’1 ) ⊆ g(xk ) holds, where xk ∈ h(X’1 ) and xk ∈ / X’1 : it results that X’1 is not a closed itemset. Finally we report an high-level description of DELIS algorithm (see Algorithm 1), where – h represents the size of the current multiple level frequent itemsets; – k indicates the current level of the taxonomy; – mFIh,k represents a set containing the multiple level frequent itemsets with size h generated at the level k ; – mCFI represents a set containing the multiple level closed frequent itemsets; – lowerLevel(G,k) returns the lower level of the taxonomy G w.r.t. the level k ; – frequentItemsGeneration(k) returns frequent items at level k. It also exploits the set of frequent items in mFI1,k−1 if k=lowerLevel(G,); – join(Xi ,Xj ) joins a la apriori Xi ,Xj [1], where Xi ≺ Xj ; – frequentItemsetsGeneration(mFIh,k−1 ) returns the multiple level frequent itemsets with size h generated at the level k: ∀ Xi ∈ mF I h,k−1 ’ Xi = join(X’i ,X”i ), |X’i |=|X”i |=h-1, dX (X’i )≤k-1 and dX (X”i )≤k-1, it joins Xdi with X”i , Xdi with X’i and Xdi with Xdi where dX (Xdi )=k or dX (Xdi )=k, Xdi subtype itemset of X’i , Xdi subtype itemset of X”i respectively; 1
Without generality loss, we assume that the set of transactions D contains only leaf-items of the taxonomy.
258
C. Loglisci and D. Malerba
– getClosed(mFIh,k ) returns the multiple level closed frequent itemsets from mFI by exploiting Definition 1 and Theorem 1; – getGenerator(mCFIh,k ) returns the generator frequent itemsets from mFIh,k . We provide the definition of generator in the next section, where it is also used for the rule generation; – CG indicates the total set of generators; – leafItemLevel represents the lowest level of the taxonomy. A trace of the algorithm 1 follows. Consider the toy example reported in Table 1, minsup=0.4 and for the sake of simplicity =0. First (step 4), given h=1 and k=taxonomy level 1, mFIh,k = {A,B} is determined. Then (steps 5-12), at the current level of taxonomy, the frequent itemsets with larger length are determined (i.e., mFIh,k ={A,B}), moreover, by exploiting the Definition 1, the step 10 finds the closed ones (i.e., {A,A,B}) that will be also used at k=taxonomy level 2. From the step 13 the algorithm iteratively goes down the lower levels of taxonomy by scanning only the direct subtypes of closed frequent itemsets: at the step 15, given h=1 and k=taxonomy level 2, mFIh,k ={A1 , A2 }, B2 } where A2 , B2 are not closed because of A1 . Then, at the step 21 with h=2 mFIh,k consists of {A1 ,A2 ,A1 ,B2 }, A2 , B2 }, and at the step 23 the algorithm updates mFIh,k with {A,B2 ,A1 ,B,A2 ,B}. By exploiting Definition 1 at the step 24 only the frequent itemsets which are subtype of closed itemsets have to be proven as closed. Finally, given h=3 mFIh,k consists of {A1 ,A2 ,B2 ,A1 ,A2 ,B}.
5
Operative Definitions for Generating Multiple Level Minimal Association Rules
The algorithm DELIS generates the mCFIs which indeed constitute a subset of the complete set of mFIs: in [5] has been proven that mFIs can be inferred from mCFIs. Moreover, the rules generated with mCFIs (mAR) are indeed a minimal set of those discovered with mFIs, or, in other words, they are a non-redundant subset. In formal terms the notion of minimal rule [5] states that Definition 2 (Minimal Rule). An association rule R1 :A1 ⇒C1 is called minimal iff there does not exist R2 :A2 ⇒C2 with s(R1 )=s(R2 ), c(R1 )=c(R2 ), A2 ⊆ A1 , C1 ⊆ C2 . Now, our interest is to generate the rules from mCFIs and at this aim we provide a set of operative definitions. In literature it has also been proven that the mAR can be seen as the union of two sets, namely ExactRuleSet and ApproximateRuleSet respectively, which are computed by exploiting the following: Definition 3 (ExactRuleSet and ApproximateRuleSet). Let mCFI be the set of multiple level closed frequent itemsets and CG be the set of generators: the sets of rules ExactRuleSet and ApproximateRuleSet consist of rules with confidence respectively equals to 1 and smaller than 1. They correspond to the sets
Mining Multiple Level Non-redundant Association Rules
259
{Y⇒X-Y| X ∈ mCFI ∧ Y∈ CG ∧ Y generator of X ∧ s(X) = s(Y)} {Z⇒X-Z| X ∈ mCFI ∧ Z ∈ CG ∧ h(Z)⊆h(X)} A generator is defined as: Definition 4 (Generator Itemset). Let X be a closed itemset, and h the Galois closure operator: an itemset Y, |Y| >1, Y⊆I, is called generator of X (closed itemset) iff h(Y)=X and Y’⊆I with Y’⊂ Y such that h(Y’)=X and s(Y’)=s(Y). Operatively ExactRuleSet is produced with rules of the form Y⇒X-Y where the relationships s(X)=s(Y) and h(X)=h(Y) hold, given c(Y⇒X-Y)= s(X∪Y)/ s(Y)=1, Y⊂ X and X as a mCI. Moreover, rules of the form Z ⇒X-Z can be derived from X and Y⇒X-Y considering that for each Z, where Y⊂ Z⊂ X, the relationship s(Z)=s(X)=s(Y) has to hold. Analogously, ApproximateRuleSet is composed of rules of the form Y⇒X-Y where the relationships s(X)<s(Y) and h(Y)⊆h(X)=X hold, given c(Y⇒X-Y)=s(X∪Y)/s(Y)<1, Y⊂ X. Additionally, rules of the form Z⇒X-Z can be considered as approximate if h(Z)⊆h(X) is true for each Z, Z⊂ X. By implementing this framework of definitions, our two-fold approach returns a subset of the total set of valid rules: it contains only those non-redundant, namely the rules that represent the greater set. More interestingly, the latter can be derived from the former.
6
Experiments
In this section we illustrate the application of the mining of non-redundant rules for discovering meaningful, and non-superfluous relationships among named entities reported in biomedical literature. The arrangement of the biomedical entities (also called MeSH terms2 , MEdical Subject Headings) in several is-a hierachies (see Figure 3) makes suitable the discovery of multiple level AR. In particular, association rules have been discovered on two datasets obtained by retrieving publication abstracts through PubMed-Medline3 with two queries: Neuropathy Ataxia Retinitis Pigmentosa and Alzheimer Drug Interactions: the first query returned 455 abstracts (NARP dataset) while the second 135 abstracts (ADI dataset). The returned abstracts have been initially annotated by the BioTeKSIBM UIM Architecture [?]. For each set a relational database was created and fed with MeSHs occurring in the corresponding set of retrieved abstracts. In particular, each transaction of a single table was associated to an individual abstract and it was described in terms of items that correspond to MeSHs. The simplest representation, namely the boolean representation, was adopted in order to represent the occurrence of a MeSH term in an abstract: NARP dataset was more sparse than ADI. 2 3
http://www.nlm.nih.gov/mesh/ http://www.ncbi.nlm.nih.gov/entrez/query.fcgi
260
C. Loglisci and D. Malerba
Fig. 3. An excerpt of the pre-defined taxonomies on the MeSH items: the lower levels are more indented than the higher ones
In order to evaluate the ’goodness’ of our approach we performed a comparison with the well-known Apriori algorithm [1] along three different criteria: running times for generating both frequent itemset and association rules, the number of itemsets and association rules, and the information expressed by the rules. Frequent itemsets and valid multiple level AR were generated for both algorithms by varying minsup value (while only for DELIS algorithm, but for the sake of brevity here we report the simpliest case: =0).
Fig. 4. Running times on the varying of the minimum support
Execution time of both algorithms for the two dataset is reported in Figure 4 where we can observe that the Apriori algorithm requires more execution time on the sparse datasets while running times of DELIS seem to be not affected by the dataset. Data sparseness usually implies a low correlation in data and however, the correlation is more evident on the lowest level of the taxonomies. Apriori does not take into account this information since it basically combines frequent itemsets. On the contrary, DELIS generates itemsets at low levels of the taxonomies and it proceeds by generating subtype itemsets from their supertypes at more and more deep levels of the taxonomies. This allows DELIS to gain time when the mining process is performed on sparse datasets. Considering the number of frequent itemsets that have been generated (see Figure 5), we find that by decreasing minsup in both data sets, DELIS generates a lower number of frequent itemsets than Apriori. Moreover, in the case of ADI, DELIS generates less itemsets than in the case of NARP. This is due to the the fact that ADI is more dense than NARP, hence the positive effect of the closed itemsets generation is more evident. Actually, the mechanism subsumed by the closed itemset discovery allows to generate compact subsets of itemsets by preserving information on correlation among items. Consequently, the number of
Mining Multiple Level Non-redundant Association Rules
261
Fig. 5. Number of generated itemsets on the varying of the minimum support
association rules that have been generated is also reduced (see Figure 6). For instance, by scanning the rules reported below discovered for ADI (where the effect of redundancy elimination can be more evident) benefits coming from the use of the DELIS algorithm can be observed. Apriori rules: (minsup=0.3 and minconf=0.5 ) R82 : Alzheimer Disease ⇒ Chemical and drugs, s=0.53, c=0.79 R1343 : Alzheimer Disease ⇒ Biological sciences,Chemical and drugs, s= 0.35, c=0.52 R6004 : Alzheimer Disease, Mental disorders ⇒ Biological sciences, Chemical and drugs, s= 0.35, c=0.52 R6009 : Alzheimer Disease ⇒ Biological sciences,Chemical and drugs,Mental disorder, s=0.35, c=0.52 R6253 : Alzheimer Disease, Tauopathies ⇒ Biological sciences,Chemical and drugs, s=0.35, c=0.52 R6259 : Alzheimer Disease ⇒ Biological sciences,Chemical and drugs,Tauopathies, s=0.35, c=0.52 R10739 : Alzheimer Disease,Mental disorders,Tauopathies ⇒ Biological sciences,Chemical and drugs, s= 0.35, c=0.52 R10751 : Alzheimer Disease,Tauopathies ⇒ Biological sciences,Chemical and drugs,Mental disorders, s=0.35, c=0.52 R10752 : Alzheimer Disease,Mental disorders ⇒ Biological sciences,Chemical and drugs,Tauopathies, s=0.35, c=0.52 R10758 : Alzheimer Disease ⇒ Biological sciences,Chemical and drugs,Mental disorders,Tauopathies, s=0.35, c=0.52
DELIS rules: (minsup=0.3 and minconf=0.5 ) R5145 : Alzheimer Disease⇒Chemical and drugs,Mental disorders,Tauopathies, s=0.53, c=0.79 R5759 : Alzheimer Disease⇒Biological sciences, Chemical and drugs, Mental disorders,Tauopathies, s=0.35, c=0.52
As we can see, Apriori discovers a set of the rules which convey the same information: R1343 , R6009 , R6259 can be subsumed by R10758 which however expresses additional information. Differently, DELIS algorithm mines a reduced set of rules than the previous ({R5759 }), prunes the redundant rules and returns only the more informative one. Moreover, by reminding that association rules are monotone with respect to pruning on the consequent side (i.e., if X ⇒ Y ∪ {A} is valid, then the rule X ⇒ Y is valid too) some considerations can be drawn. Rules R1343 , R6004 , R6009 , R6253 , R6259 , R10739 , R10751 , R10752 extracted by Apriori can be obtained by the rule R10758 , since they have the same support and confidence and this rule results to have the shortest antecedent and the longest consequent. This means that moving an item from the consequent side to the antecedent side the meaning of the rule is not affected when the resulting rule has the same support
262
C. Loglisci and D. Malerba
Fig. 6. Number of the discovered rules on the varying of the minimum support
Algorithm 1. Top-level description of DELIS algorithm. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27:
input: D, G, minsup; output:mCF I, CG; mCF I ← ; CG ← ; mF I h,k ←; k ←lowerLevel(G,); h ←1; mF I h,k ← f requentItemsGeneration(k); while mF I h,k = do h ← h + 1; mF I h,k ←; for each Xi , Xj ∈ mF I h−1,k and Xi = Xj do mF I h,k ← mF I h,k ∪ join(Xi ,Xj ); end for mCF I ← mCF I ∪ getClosed(mF I h,k ); CG ← CG ∪ getGenerator(mF I h,k ); end while while k ≤ leaf ItemLevel do h ← 1; k ← lowerLevel(G, k); mF I h,k ← f requentItemsGeneration(k); mCF I ← mCF I ∪ getClosed(mF I h,k ); CG ← CG ∪ getGenerator(mF I h,k ); while mF I h,k = do h ← h + 1; mF I h,k ←; for each Xi , Xj ∈ mF I h−1,k and Xi = Xj do mF I h,k ← mF I h,k ∪ join(Xi ,Xj ); end for mF Ih,k ← mF Ih,k ∪ f requentItemsetsGeneration(mF Ih,k−1); mCF I ← mCF I ∪ getClosed(mF I h,k ); CG ← CG ∪ getGenerator(mF I h,k ); end while end while
and confidence. Therefore, by running DELIS, a single rule, that is the R5759 rule, is obtained and it is representative as well. On the contrary, in the case of the rule R82 , although the consequent of the rule is contained in the consequent of the rule R10758 , both of them should be preserved since the confidence and support values are not different, and hence contains different information. For this reason, DELIS discovers another rule, R5145 , that has the same support
Mining Multiple Level Non-redundant Association Rules
263
and confidence of the R82 extracted by Apriori, and as consequent a superset of the rule R82 . By compacting the consequent of the R5145 rule the rule R82 is obtained.
7
Related Work
Literature shows that much work has been dedicated for eliminating redundancies in the mining process only after that the all possible AR (included those redundant) have been discovered. Pruning redundancies during the generation of AR is a strategy which has attracted the attention of few researchers. First of all, Pasquier et al.[13] solve only the problem of reducing the number of frequent itemsets. They resort to the Galois closure mechanism for creating a smaller itemsets lattice and searching within those non-redundant. This allows to reduce the set of final frequent itemsets. The same authors exploit that idea for the step of rule generation [5]: here the set of valid rules is generated considering two base sets which contain the most informative rules, namely those with maximal antecedent and minimal consequent. Interesting alternatives for the second step of AR mining have been proposed by Zaki[16] and Kanda et al.[10]. In the former the non redundant rules are discovered as those most general: rules which have less items w.r.t. others with identical support and confidence are returned. Although more compact, however these rules often do not represent the most informative ones. Kanda et al. [10] focus on the rule generation and propose a concept of approximate redundancy: the non-redundant AR are those rules from which all valid ones can be approximately reconstructed. However, in these works the problem of redundancy is basically studied only for the second step of AR mining, and, moreover, the analysis is not extended to the case more general of items organized over taxonomies. Indeed, this aspect is considered in [15] only for finding out frequent itemsets. Here the concept of generalized closed itemsets is introduced to deal with items at several levels of generalization. Thus, the authors describe a technique for extracting generalized frequent closed itemsets under some constraints and conditional properties, while nothing is provided about the removing of redundancies on the generalized rules. Differently, in our work redundancy is pruned on both generalized rules and frequent itemsets.
8
Conclusions
The presence of redundant information in the association rules (or more generally, in the patterns) can be misleading for the task of the end-user and, thus, can cause the failure of a data mining project. In this paper we faced this issue operating on the process of discovery, namely directly in each of the two steps of AR mining. Moreover, the proposed computational solution takes into account of situations more general and often real, namely when background information on the items is available. The basic idea was of pruning redundancies on both frequent itemsets and association rules while these are found out. Experiments showed that the advantages of our approach are reflected in the generation of
264
C. Loglisci and D. Malerba
reduced sets of itemsets and rules, the determination of more significant itemsets and rules, and finally the lowering of computational costs. This could be beneficial for the understandability of the results of a data mining process. Acknowledgments. This work is supported in partial fulfillment of the research objectives of “Ateneo 2009” project “Modelli e metodi computazionali per la scoperta di conoscenza in dati biomedici“ and “FAR” project “MBLab Laboratorio di bioinformatica per la biodiversita’ molecolare. The authors are grateful to the anonymous reviewers for their helpful comments.
References 1. Agrawal, R., Srikant, R.: Fast algorithms for mining association rules. In: Proceedings of the 20th Int’l Conference on Very Large Data Bases, pp. 478–499 (1994) 2. Agrawal, R., Srikant, R.: Mining Generalized Association Rules. In: Proc. of the 21st Int’l Conf. on Very Large Databases, pp. 407–419 (1995) 3. Ashrafi, M., Taniar, D., Smith, K.: Redundant Association Rules Reduction Techniques. In: Zhang, S., Jarvis, R.A. (eds.) AI 2005. LNCS, vol. 3809, pp. 254–263. Springer, Heidelberg (2005) 4. Baralis, E., Psaila, G.: Designing Templates for Mining Association Rules. J. Intell. Inf. Syst. 9(1), 7–32 (1997) 5. Bastide, Y., Pasquier, N., Taouil, R., Stumme, G., Lakhal, L.: Mining minimal non-redundant association rules using frequent closed itemsets. In: Palamidessi, C., Moniz Pereira, L., Lloyd, J.W., Dahl, V., Furbach, U., Kerber, M., Lau, K.-K., Sagiv, Y., Stuckey, P.J. (eds.) CL 2000. LNCS, vol. 1861, pp. 972–986. Springer, Heidelberg (2000) 6. Davey, B.A., Priestley, H.A.: Lattices Theory. Introduction to Lattices and Order, 4th edn. Cambridge University Press, Cambridge (1994) 7. Han, J., Fu, Y.: Mining Multiple-Level Association Rules in Large Databases. IEEE Trans. Knowl. Data Eng. 11(5), 798–804 (1999) 8. Hernandez Palancar, J., Fraxedas Tormo, O., Feston Cardenas, J., HernandezLeon, R.: Distributed and Shared Memory Algorithm for Parallel Mining of Association Rules. In: Perner, P. (ed.) MLDM 2007. LNCS (LNAI), vol. 4571, pp. 349–363. Springer, Heidelberg (2007) 9. Hilderman, R.J., Hamilton, H.J.: Knowledge Discovery and Interest Measures. Kluwer Academic, Boston (2002) 10. Kanda, K., Haraguchi, M., Okubo, Y.: Constructing Approximate Informative Basis of Association Rules. Discovery Science, 141–154 (2001) 11. Klemettinen, M., Mannila, H., Ronkainen, P., Toivonen, H., Verkamo, A.I.: Finding Interesting Rules from Large Sets of Discovered Association Rules. In: Proc. of the 3rd Int. Conf. on Information and Knowledge Management, pp. 401–407 (1994) 12. Kotsiantis, S., Kanellopoulos, D.: Association Rules Mining: A Recent Overview. GESTS Int. Trans. on Computer Science and Eng. 32(1), 71–82 (2006) 13. Pasquier, N., Bastide, Y., Taouil, R., Lakhal, L.: Discovering frequent closed itemsets for association rules. In: Proc. of 7th Int. Conf. on DB Theory, pp. 398–416 (1999)
Mining Multiple Level Non-redundant Association Rules
265
14. Sharma, L.K., Vyas, O.P., Tiwary, U.S., Vyas, R.: A Novel Approach of Multilevel Positive and Negative Association Rule Mining for Spatial Databases. In: Perner, P., Imiya, A. (eds.) MLDM 2005. LNCS (LNAI), vol. 3587, pp. 620–629. Springer, Heidelberg (2005) 15. Sriphaew, K., Theeramunkong, T.: Mining Generalized Closed Frequent Itemsets of Generalized Association Rules. In: Proc. of 7th Int. Conf. on Knowledge-Based Intell. Inf. and Eng. Systems, pp. 476–484 (2003) 16. Zaki, M.J.: Generating non-redundant association rules. In: Proc. of The Ninth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 34–43 (2000)
Pattern Mining with Natural Language Processing: An Exploratory Approach Ana Cristina Mendes1 and Cl´ audia Antunes2 1
Spoken Language Systems Laboratory - L2 F/INESC-ID Instituto Superior T´ecnico, Technical University of Lisbon R. Alves Redol, 9 - 2o – 1000-029 Lisboa, Portugal [email protected] 2 Department of Computer Science and Engineering Instituto Superior T´ecnico, Technical University of Lisbon Av. Rovisco Pais 1 – 1049-001 Lisboa, Portugal [email protected] Abstract. Pattern mining derives from the need of discovering hidden knowledge in very large amounts of data, regardless of the form in which it is presented. When it comes to Natural Language Processing (NLP), it arose along the humans’ necessity of being understood by computers. In this paper we present an exploratory approach that aims at bringing together the best of both worlds. Our goal is to discover patterns in linguistically processed texts, through the usage of NLP state-of-the-art tools and traditional pattern mining algorithms. Articles from a Portuguese newspaper are the input of a series of tests described in this paper. First, they are processed by an NLP chain, which performs a deep linguistic analysis of text; afterwards, pattern mining algorithms Apriori and GenPrefixSpan are used. Results showed the applicability of sequential pattern mining techniques in textual structured data, and also provided several evidences about the structure of the language.
1
Introduction
Recent years have witnessed a great increase of the information available to the main public: information that was formely only accessible on books is now everywhere, and digitally available to everyone. This new paradigm, mainly due to the leverage of the Web, triggered the emergence of several research areas, namely: Information Extraction, Information Retrieval and Question-Answering, in the field of the Natural Language Processing (NLP), and Text Mining in the Knowledge Discovery and Data Mining (KDD) field. Bridging the gap between humans and computers has always been the goal behind NLP, by attempting to provide the laters with the ability of understanding and interacting using natural language. Even if this goal was not completely achieved, yet, a significant difference can be perceived between techiques that use NLP and other data processing techniques: the first ones have a much deeper understanding of language. P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 266–279, 2009. c Springer-Verlag Berlin Heidelberg 2009
Pattern Mining with Natural Language Processing
267
The need for discovering unknown knowledge in text documents set off the interest in Text Mining. Also known as Knowledge Discovery in Text (KDT), it aims at discovering and extracting knowledge from textual data, presented in a semi-strutured and unstructured way, and “automatically identifying interesting patterns and relationships” [1]. Text Mining techniques have a surface knowledge about the language in which data is presented (for instance, in terms of its stopwords and stemming procedures); usually, they do not possess or deal with any knowledge about the structure of the language. In this paper we study the application of pattern mining algorithms in textual stuctured data. We aim at going a step further from what it is nowadays accomplished with Text Mining. Our hypothesis is that pattern mining over written data will benefit if we take advantage of the deeper understanding of the language that NLP provides. Our approach relies on pattern discovery over the processed texts using state-of-the-art NLP tools; therefore, in this paper we present the results of a set of studies concerning the usage of typical pattern mining algorithms over text data to which was included linguistic knowledge. Since NLP is the starting point in our pipeline, traditional pre-processing techniques, based on bag-of-words approaches, can not be applied; raw text is used instead. Natural language is ambiguous and profits from an incomparable richness and variety; hence, the usage of NLP can imply the introduction of certain errors, namely misclassification of words in morphological classes and wrong division of sentences into its syntatic constituents. Performance, however, should not be considered an obstacle to NLP anymore due to the massification of applications implemented in distributed environments. The remainder of this paper is organized as follows: Section 2 makes a literature review on this topic; Section 3 formalizes the problem of mining patterns in textual structured data; Section 4 describes the studies we performed on a real-world dataset. The paper finishes in Section 5, where conclusions are drawn and future work directions are presented.
2
Literature Review
Data Mining is an interdisciplinary field that brings together techniques from statistics, database, artificial intelligence and machine learning. Data Mining is all about mining patterns in large databases in order to extract hidden knowledge. It gave its first steps in the business domain, when companies experienced an enormous increase of their structured data digitally stored. A different perspective to pattern mining arose with the outgrowth of large amounts of unstructured text data: Text Mining. Although common approaches to Text Mining are based on a flat representation of words (usually with little linguistic processing), the close interaction between Text Mining and NLP is an indesputable fact. They share a strong common ground: text presented in an unstrutured way, written in natural language.
268
A.C. Mendes and C. Antunes
Recent discussion suggests that the integration of linguistic concepts and NLP techniques to benefit Text Mining applications is still in its fresh start. Not long ago, in 2005, the usefulness of NLP in Text Mining was an open issue [2,3] and an hard question to be answered. The general conclusion arose that more experiments were still necessary. Researchers could not agree on a common opinion about whether NLP helps or not the results already achieved by Text Mining. There is a significant amount of approaches to the problem of Text Mining stated in the literature that claim the use of results from NLP. For instance, NLP tags are used in the InFact [4] system for text analysis and search. Linguistic knowledge, namely grammatical and syntactic roles, is used to build subjectaction-object triples, which are then utilized as indexes. The subsequent search is based on a linguistically enriched query language and allows users to search for actions, entities, relationships and events. However, and unlike the InFact system, common approaches do not make direct manipulation of NLP morpho-syntactic tags to benefit Text Mining and search. They rather employ techniques borrowed from Information Extraction (IE) or are related with Text Classification. In what concerns IE, it regards the extraction of specific data from natural language texts, often by applying NLP techniques, like named entities recognition. The usage of IE in Text Mining can be seen especially in the field of bioinformatics [5,6]. Concerning the Text Classification task, it aims at assigning a correct class to previously unseen records, given a collection of already labeled records. Text Classification is studied, for instance, in [7], in which pattern mining over syntactic trees is used to make an opinion analysis of documents. Text Mining techniques for document classification are applied mainly to spam-filtering. Although the previously cited works only regard the implications of NLP in pattern mining, it is worth to mention that the interaction between both is, in fact, bidirectional. The work by [8,9], for instance, devise methods for named entity recognition and co-reference resolution using association rules.
3
Problem Definition
Natural languages can be formally modelled by Context-Free Grammars (CFG). A CFG is a tuple G = (N, Σ, R, S), where: N is a set of non-terminal symbols; Σ is a set of terminal symbols; R is a set of rules, each of the form A → β, in which A is a non-terminal and β is a string of symbols from the set (Σ ∪ N ); and S ∈ N is the start symbol. A CFG can be thought of in two ways: a generator of sentences or a device that assigns a structure to a sentence [10]. Besides, notice that, in the NLP domain, the set of terminals is related with the lexicon, or vocabulary, constituted by the words and symbols of a given natural language. Also, non-terminals represent either lexical classes of words (Part-Of-Speech (PoS)1 ) or multi-word constituents of the language 1
Traditional grammar classifies words according to eight PoS, or lexical classes: verb, noun, pronoun, adjective, adverb, preposition, conjunction, and interjection.
Pattern Mining with Natural Language Processing
269
Grammar G0 S→ NP → VP → PP →
NP VP | NP VP PP P ronoun | N oun | Determiner N oun | Determiner Adjective N oun V erb | V erb NP P reposition NP
Lexicon for G0 N oun → P ronoun → V erb → Adjective → Determiner → P reposition →
car | man | plane | morning | panter | airport I | me | you | it saw | drove | want | like big | pink | green | tall the | a | an | this on | to | f rom
Fig. 1. Set of rules for the grammar G0 and its lexicon
(chunks). 2 Figure 1 shows a lexicon and a set rules that belong to a sample grammar G0 3 . A labeled rooted tree is defined as an acyclic connected graph, denoted as the quintuple T = (V, E, Σ, L, v0 ) where: V is the set of nodes; E is the set of edges that connect nodes; Σ is the set of labels; L : V → Σ is the labeling function, that assigns labels from Σ to nodes in V ; and v0 is the root node. The level of a node v is the length of the shortest path from v to the root node v0 . Each node can be either a leaf or an internal node: leafs have no children and internal nodes have one or more child nodes. An ordered labeled rooted tree is a labeled rooted tree where the children of each internal node are ordered. A natural language sentence can be generated by a sequence of rule expansions, commonly represented by a parse tree. Thus, a parse tree is defined as an ordered labeled rooted tree τ , in which v0 = S and tree nodes are labeled according to the morphological and syntactic classes of each corresponding unit. Figure 2 shows the tree structure of the sentence “I saw a pink plane on the airport” according to G0 . Mining patterns using algorithms which do not focus on this specific data structure (like sequential pattern mining algorithms) implies a transformation of parse trees into other structures with different characteristics. In this context, the l-sequence of an ordered tree is the preorder transversal label sequence of all nodes; the preorder transversal label-level sequence, l2 -sequence, is the cannonical representation of a tree, composed not only by the label of each node, but also by its level in the tree. Both concepts are introduced in [11], where authors also prove the uniqueness of l2 -sequences. 2 3
For the sake of simplicity, we will refer to the terminal and non-terminal symbols of a given sentence as lexical units and morpho-syntactic units, respectively. Symbols NP (noun phrase), VP (verb phrase) and PP (prepositional phrase) are language constituents (chunks) with a specific syntactic function within a sentence. Readers interested on more information about this topic are suggested to refer to [10].
270
A.C. Mendes and C. Antunes
S
NP
Pronoun
VP
PP
NP
Verb
Determiner Adjective
I
saw
a
pink
Preposition
Noun
plane
on
NP
Determiner
Noun
the
airport
Fig. 2. Parse tree of “I saw a pink plane on the airport” according to G0
Knowledge
Natural Language Processing Raw Text Data
Textual Structured Data
Pattern Mining Patterns
Fig. 3. Our exploratory approach to discovering patterns in text
In the l-sequence of a parse tree each item has a linguistic meaning and is related with a morpho-syntactic or semantic property of a determined unit within a sentence. Likewise, the l2 -sequence of a parse tree provides the knowledge and a unique representation of its structure. Therefore, transforming sentences into l-sequences (or l2 -sequences) represents the transformation of the unstructured dataset into its corresponding structured version. Given an input sentence S, the preorder transversal sequence of its associated parse tree is a new sentence S provided with a deeper linguistic knowledge, which captures the morpho-syntactic structure of S. We define the sentence S as a textual structured sentence since its base constituents refer to natural language units, either lexical (words) or morpho-syntactic (PoS and/or chunks). Given a database D of transactions and a user-specified minimum support σ, a pattern is said to be frequent if it is contained in, at least, σ transactions in D. Problem statement. The application of pattern mining algorithms over textual structured data provides deeper knowledge about a natural language inner organization than tradicional Text Mining techniques. Given a text database and a user-specified minimum support, the problem of mining textual structured data is to discover the frequent structures of a natural language. Being so, this exploratory study attempts to bring the understanding of the natural language structure that NLP provides to pattern mining. We propose to test typical algorithms on textual structured data in order to discover patterns on structured data. The high-level view of the proposed approach is shown in Figure 3.
Pattern Mining with Natural Language Processing
4
271
Case Study
Experiments described in this section were conducted on a dataset composed by news articles from a Portuguese daily newspaper, P´ ublico, dated from years 1994 and 1995. Since the news domain is far too broad, only the news belonging to the category “National News” were used. Moreover, from these, paragraphs with less than 100 characters were discarded, since these often refer to articles’ title. The characteristics4 of the dataset are shown in Table 1. Table 1. Characteristics of the dataset National News Size (MB) ≈ 27
Words (#) ≈ 3 000 000
Distinct Words (#) 85 770
Paragraphs (#) 58 744
Sentences (#) 176 447
Text data was processed using the Spoken Language Systems Laboratory (L2 F) NLP chain for Portuguese, running on a computacional grid [12]. The chain involves the following processing steps: morphological analysis (by Palavroso [13]), morphological disambiguation (by MARv [14]), multi-word contraction and term spliting (by RuDriCo [15]) and, finally, chunking, named entities recognition and dependencies extraction (by XIP [16]). The chain is responsible for creating the parse tree, morpho-syntactically disambiguated, for each sentence received as input. An illustrative example of the output of the NLP chain for the input sentence “O primeiro-ministro Jos´e S´ocrates inicia uma visita oficial `a Madeira.”5 is depicted in Figure 4. Thus, the parse tree used in our experiments is defined as τ , and nodes in τ are labeled according to a function that marks a node v with a PoS or a chunk if v is an internal node, or with a language word or a language symbol if v is a leaf. In addition, PoS belong to the set of symbols: {NOUN, VERB, ART, ADJ, ADV, CONJ, PRON, INTERJ, PREP, PUNCT, PASTPART, REL, FOREIGN, SYMBOL, NUM}; a chunk is one of {TOP, VP, NP, ADVP, AP, PP, SC}. Considering the reference sentence, its associated textual structured sentence is the sequence: TOP NP ART O NOUN NOUN primeiro-ministro NOUN Jos´e NOUN S´ ocrates VP VERB inicia NP ART uma NOUN visita AP ADJ oficial PP PREP a ART a NOUN Madeira PUNCT . Since the NLP chain introduce many features in each tree node, associated with the morphological, syntactic and semantic analysis (some of which are shown in small rounded boxes in Figure 4), those unwanted are discarded by 4
5
Notice that the Paragraphs column concerns a rough text spliting by carriage return, while the Sentences column is related with the correct natural language sentences identified by the morpho-syntactic analyser. Thus, one paragraph might contain several sentences. “Prime-minister Jos´e S´ ocrates starts official visit to Madeira.” For clarity reasons, this sentence will be referred to and used as reference sentence throughout the rest of this paper.
272
A.C. Mendes and C. Antunes
Fig. 4. Parse tree of “O primeiro-ministro Jos´e S´ ocrates inicia uma visita oficial ` a Madeira”
a post-processing stage. This post-processing aims at building the alphabet for the mining algorithms with only the relevant symbols for our experiments. If all features were used, a much larger alphabet would be created. However, this would lever the discovery of many frequent noisy patterns, not serving the purpose of our current research studies. 4.1
Pattern Mining Algorithms over Textual Structured Data
There are several algorithms to mine patterns in data. They can follow specific categorizations depending on the characteristics of the input data: items in a basket, sequences, tree structures, and so forth. In these studies, however, we slightly transformed our dataset to be used as input for two pattern mining algorithms, which belong to different strands on KDD. The first algorithm employed in the experiments was Apriori [17]. Apriori is based on a candidate generation and test philosophy, following the anti-monotone property, according to which if a pattern is not frequent, then none of its supersets is frequent. In this case, each textual structured sentence is seen as a different transaction in the database and each of it components is considered is a different item. The second algorithm used was GenPrefixSpan [18]. This algorithm for sequential pattern mining represents a generalization of PrefixSpan [19] to deal with gap constraints, while maintaining a pattern-growth philosophy: avoids the candidate generation step altogether, and focus the search on a specific set of the initial database. Here, and like with Apriori, the textual structured sentences are transactions and its components are items. However, in this case, the position of each component is related with a discrete time instant; being so, each component occurs immediately after its left side neighbour. This algorithm was used to mine sequences of contiguous components in textual structured sentences.
Pattern Mining with Natural Language Processing
273
A sample input for each pattern mining algorithm is presented in Table 2. Each line contains a textual structured sentence, configured for the correspondent algorithm, as previously defined in Section 4. Since we will present results from tests over different input data, we will refer to this input configuration as the default configuration. Table 2. Input data for each pattern mining algorithm (default configuration) Algorithm
Input
Apriori
TOP NP ART O NOUN NOUN primeiro-ministro NOUN, Jos´e NOUN S´ ocrates VP VERB inicia ...
GenPrefixSpan
TOP(1) NP(2) ART(3) O(4) NOUN(5) NOUN(6) primeiro-ministro(7) NOUN(8) Jos´e(9) NOUN(10) S´ ocrates(11) VP(12) VERB(13) inicia(14) ...
4.2
Results
The results of each pattern mining algorithm, Apriori and GenPrefixSpan, are displayed on Table 3 and Table 4, respectively. A detailed analysis of the results allow us to draw further conclusions about the language. It is worth to remind, however, that these experiments were conducted on a very distinctive dataset, composed by news articles focused on a particular domain: the “National News”. These are commonly written in a well-formed language, with reliable and formal discourse structures and few spelling errors. Tests on other datasets with different characteristics would probably lead to different results. One may argue that, regardless the text’s characteristics, it is still Portuguese we are dealing with, and its structure remains unchangeable; however, some conclusions we draw shall not be extended to the language as a whole, as it is spoken and written in other real-world situations. Table 3. Results of the Apriori algorithm Support (Transactions) Min (%) Max (%) 75 50 25 25 10
100 100 100 50 25
Discovered Patterns (#) 1 41 1 454 1 413 28 721
227 079 135 056 028
Regarding the results from the Apriori algorithm, the great increase of patterns discovered as the values of minimum support decrease should be pinpointed and deserves a careful analysis. Actually, since language PoS and chunks are not in great amount (≈ 20), some of which populating every or almost every sentence in the textual structured data (consider, for instance, constituents TOP, the top-level chunk that refers to a sentence, and PUNCT, the PoS that refers
274
A.C. Mendes and C. Antunes
to punctuation), it is not a surprise to verify that a significant part of the frequent patterns are composed only by morpho-syntactic units and stopwords. This, however, does not seem to bring any reliable knowlegde about language’s structure. Also, most of these patterns have no linguistic meaning or sense at all, since they lack on order. Table 4. Results of the GenPrefixSpan algorithm (default configuration) Min Support (Transactions) % # 50 25 10 5 2.5 1 0.5 0.25 0.1
88 44 17 8 4 1
224 112 645 822 411 764 882 441 176
Discovered Sequences (#) Total Size > 1
2 4 11 36
42 95 235 401 790 139 962 875 375
1 4 10 33
20 66 197 346 698 885 405 715 807
In what concerns the results from the GenPrefixSpan algorithm, the number of discovered sequences is much smaller when compared to the number of discovered patterns discovered with Apriori, given the same support. Indeed, the philosophy behind GenPrefixSpan avoids the generation of patterns with no linguistic meaning. This algorithm mined only subsequences of our textual structure sentences; therefore with linguistic meaning. Also, it is worth to mention that in 50% of our dataset only 22 one-unitsize sequences were discovered. From these 22, only 6 were language symbols or language words, namely: comma, dot, determiners “a” and “o” (English word “the”), preposition “em” (corresponding to English words “in”, “on” and “at”) and preposition “de” (English word “of”). All chunks are frequent given a mininum support of 50% and, having in mind that the NLP chain labels nodes according to 15 different PoS, the following lexical classes were not considered as being frequent: INTERJ, interjections, PARTPART, words in past participle, REL, relative pronouns, FOREIGN, words written in other language than Portuguese, SYMBOL, special symbols, and NUM, numerals. A sample output of both algorithms is shown in Table 5. Besides the default configuration, similar other experiments were conducted on the same dataset with GenPefixSpan. These, however, differ on the amount of encoded linguistic knowledge, in order to discover how mined patterns evolve given different domain conditions. The tested configurations are listed bellow: (I) The textual structured sentences contain the level of each unit; the cannonical representation of each tree was used: TOP-L0 NP-L1 ART-L2 O-L3 NOUN-L2 NOUN-L3 primeiro-ministro-L4 NOUN-L3 Jos´e-L4 NOUN-L3
Pattern Mining with Natural Language Processing
275
Table 5. Patterns discovered by each pattern mining algorithm (default configuration) Algorithm
Pattern
Support (%)
Apriori
SC ADVP a de PP . PUNCT CONJ AP a PRON PP . NP TOP AP ADJ a PREP VP VERB NOUN PUNCT TOP o . PREP VP VERB
25.8 31.6 50.1 65.4
GenPrefixSpan
NP NOUN NOUN dur~ ao NOUN barroso VP VERB era NP PREP em NOUN lisboa que VP VERB
0.5 1.0 1.1 22.8
S´ ocrates-L4 VP-L1 VERB-L2 inicia-L3 NP-L1 ART-L2 uma-L3 NOUN-L2 visitaL3 AP-L1 ADJ-L2 oficial-L3 PP-L1 PREP-L2 a-L3 ART-L2 a-L3 NOUN-L2 Madeira-L3 PUNCT-L1 .-L2 (II) The same as the default configuration, however PoS components are removed. Therefore, each textual structured sentence is composed only by language words and symbols and chunks: TOP NP O primeiro-ministro Jos´e S´ ocrates VP inicia NP uma visita AP oficial PP a a Madeira . (III) The same as the default configuration, however only language words and symbols are used. Being so, instead of textual structured data, only the linguistically processed text is used as input: O primeiro-ministro Jos´e S´ocrates inicia uma visita oficial a a Madeira . (IV) Only typical Text Mining pre-processing techniques are applied: stopwords and punctuation removal. The dataset is not linguistically processed; therefore, no linguistic knowledge is included at all. Moreover, albeit the dataset was the same, the input for the pattern mining algorithm is slighlty different: instead of each transaction having a direct correspondence with a correct natural language sentence, an entire paragraph is now related with a transaction, since the NLP chain is not used: primeiroministro Jos´e S´ocrates inicia visita oficial Madeira encontra-se amanh˜a presidente governo regional The two graphs in Figure 5 compare the results of applying the algorithm GenPrefixSpan with different input configurations, either in terms of total frequent sequences and in terms of frequent n-unit-size sequences (n > 1). As it can be perceived from the analysis of both graphs, no significant differences exist between the number of discovered patterns using configurations Default and I. However, the cardinality of the alphabets used in the two configurations is fairly distinct: 85 792 and 116 634 symbols, respectively. This can bring us to the conclusion that the structure of the language in our dataset is somewhat static, and frequent nodes do not often change their level in the parse tree. A careful analysis of the results showed that the discovered sequences are indeed almost the same in both configurations. Differences are mainly due to two main reasons: for a determined minimum support value x, one sequence in
276
A.C. Mendes and C. Antunes Sequences
Sequences (Size > 1)
100000
10000
1000
100
Default I
10
II III IV
1
50
25
10
5 2,5 1 Minimum Support (%)
0,5
0,25
0,1
50
25
10
5 2,5 1 Minimum Support (%)
0,5
0,25
0,1
Fig. 5. Number of discovered frequent sequences given different input configurations to the GenPrefixSpan algorithm
the default configuration is associated with several sequences in configuration I, and their support is higher that x; for a determined support value x, one sequence in the default configuration is associated with more than one sequence in configuration I, but their support is lower that x. For instance, [NP PRON] (support 47.0%) is a sequence in the default configuration associated with sequences [NP-L1 PRON-L2] (39.7%) and [NP-L2 PRON-L3] (13.0%) in configuration I. For a minimum support of 10% the later sequence is frequent; this does not occur, however, for a minimum support of 25%. The comparison of one-unit-size frequent sequences between configurations default and I provides some clues about the level of frequent words, PoS tags and chunks in the parse tree. Ultimately, and although this is not this work’s main purpose, these results can be used to uncover potential errors in the textual structured data, introduced by the morpho-syntactic analyser: for instance, when it comes to conjunctions, for a minimum support of 0.1%, they are frequent oneunit-size sequences in more than 60% of the transactions ([CONJ] (61.2%)); if we consider also the levels of nodes, conjunctions are frequent when placed in levels 1, 2, 3 and 4 of the parse tree ([CONJ-L1] (54.9%), [CONJ-L2] (15.0%), [CONJ-L3] (1.0%) and [CONJ-L4] (0.2%), respectively). Given these results, one might wonder whether the last pattern is a misclassification of words in their morphological class or an error in the sintactic division of sentences, introduced by the NLP chain when processing the raw text dataset. Regarding the other configurations, greater differences exist among results. In every configuration in which the NLP chain was applied, the number of discovered sequences got smaller with the decrease of linguistic knowledge in the input data. For instance, with a minimum support of 2.5%, 790 sequences were discovered in the default configuration, 263 in II and 131 in III. Configuration II allow us to understand how sentences are composed regarding its bigger substructures: – in terms of those substructures inner organization, for instance: • a Noun Phrase usually starts with determiners “o” and “a” ⇒ Frequent patterns: [NP o] (support 45.2%) and [NP a] (36.1%).
Pattern Mining with Natural Language Processing
277
• the sequence of words “o presidente de a rep´ ublica” (“the president of the republic”) is a frequent Noun Phrase ⇒ Frequent pattern: [NP o presidente de a rep´ ublica] (0.4%) – in terms of those substructures interaction with other sentence constituents, for instance: • in almost 40% of the cases, a sentence begins with a Noun Phrase ⇒ Frequent pattern: [TOP NP] (38.8%). • it is more likely that a Noun Phrase appears after the inflected form of the verb to be “´e”, than a Verb Phrase ⇒ Frequent patterns: [´ e NP] (6.0%) and [´ e VP] (2.0%). Although a strict comparison can not be made between configuration IV and the others (after all, transactions are different), results show similar numbers in terms of discovered sequences for configurations III and IV; however, the number of sequences with size bigger than one is much lower when the stopwords and the punctuation were removed. Indeed, the usage of sentences without any pre-processing technique, led to the appearance of many n-unit-size sequences composed uniquely by those small and frequent words, with no content or semantic value (stopwords) and by punctuation. 4.3
Critical Analysis
The application of pattern mining algorithms over textual structured data brought us diverse clues about the natural language inner organization: similar results could not be achieved if only a flat representation of words were used. Also, recall that the algorithms were applied on the output of an already mature and stettled NLP chain. No other knowledge was manual or automatically inserted on our test data, besides some small modifications we made in order to test the various input configurations. Moreover, these studies prove the fact that some algorithms better apply in this specific domain than others: Apriori led to a combinatorial explosion of results, since it does not keep track of the input’s order; on the contrary, GenPrefixSpan seemed to discover reliable frequent language substructures. However, the discretization of parse trees into sentences and subsequent application of a sequential pattern mining algorithm implies the discovery of incomplete substructures (like, for instance, the sequence PP PREP a ART). In fact, other mining algorithms should be further explored in this specific domain, given the output of state-of-the-art linguistic analysers.
5
Conclusions and Future Work
In this paper we presented a series of exploratory studies we conducted in order to discover patterns on text data. The notion of textual structured data was introduced as being a database of text sequences which contain specific knowlegde about a sentence’s morpho-syntactic structure.
278
A.C. Mendes and C. Antunes
We took advantage of the knowledge of the language that NLP provides and used it as input for two different pattern mining algorithms: Apriori and GenPrefixSpan. On one hand, Apriori discovered a large amount of frequent patterns; however, these had little meaning since the order between the units within the natural language sentences was not preserved. On the other hand, GenPrefixSpan output several sequences that led to a better understanding of the natural language, Portuguese in this case. As future work we intend to explore the syntactic organization of the language and employ pattern mining algorithms over tree structures. We would like to test how these algorithms apply to the NLP domain. In addition, we aim at investigating the impact of including other morphological features (for example, words’ gender, number and lemma) in the mining process. Also, our results are confined to the output of an NLP chain, that includes itself the understanding of the language in the form of a grammar (for Portuguese); therefore, the results reflect the structure of that grammar. Since the application of pattern mining algorithms to textual structured data is a language independent approach, we would like to test it on other languages, with distinct morpho-syntactic structures.
Acknowledgments Authors would like to thank Prof. Lu´ısa Coheur for the thorough reviews during the writing of this paper, and Tiago Lu´ıs for the technical support and shared knowledge on parallel computing. Ana Cristina Mendes is supported by a PhD fellowship from Funda¸c˜ao para a Ciˆencia e a Tecnologia (SFRH/BD/43487/2008).
References 1. Feldman, R., Sanger, J.: The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge University Press, Cambridge (2006) 2. Kao, A., Poteet, S.: Report on KDD Conference 2004 Panel Discussion Can Natural Language Processing Help Text Mining? SIGKDD Exp. Newsl. 6(2), 132–133 (2004) 3. Kao, A., Poteet, S.: Text mining and natural language processing: introduction for the special issue. SIGKDD Explor. Newsl. 7(1), 1–2 (2005) 4. Liang, J., Koperski, K., Nguyen, T., Marchisio, G.: Extracting Statistical Data Frames from Text. SIGKDD Explor. Newsl. 7(1), 67–75 (2005) 5. Leser, U., Hakenberg, J.: What Makes a Gene Name? Named Entity Recognition in the Biomedical Literature. Briefings in Bioinformatics 6(4), 357–369 (2005) 6. Otasek, D., Brown, K., Jurisica, I.: Confirming protein-protein interactions by text mining. In: SIAM Conference on Text Mining (2006) 7. Matsumoto, S., Takamura, H., Okumura, M.: Sentiment Classification Using Word Sub-sequences and Dependency Sub-trees. In: Ho, T.B., Cheung, D., Li, H. (eds.) PAKDD 2005. LNCS, vol. 3518, pp. 301–311. Springer, Heidelberg (2005)
Pattern Mining with Natural Language Processing
279
8. Budi, I., Bressan, S.: Association rules mining for name entity recognition. In: WISE 2003: Proceedings of the Fourth International Conference on Web Information Systems Engineering, Washington, DC, USA, p. 325. IEEE Computer Society, Los Alamitos (2003) 9. Budi, I., Bressan, S., Nasrullah: Co-reference resolution for the indonesian language using association rules. In: Kotsis, G., Taniar, D., Pardede, E., Ibrahim, I.K. (eds.) iiWAS, vol. 214, pp. 117–126. Austrian Computer Society (2006) 10. Jurafsky, D., Martin, J.H.: 12. In: Speech and Language Processing: An Introduction to Natural Language Processing. In: Computational Linguistics and Speech Recognition. Prentice Hall, Englewood Cliffs (2008) 11. Wang, C., Hong, M., Pei, J., Zhou, H., Wang, W., Shi, B.: Efficient pattern-growth methods for frequent tree pattern mining. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS, vol. 3056, pp. 441–451. Springer, Heidelberg (2004) 12. Lu´ıs, T.: Paraleliza¸ca ˜o de Algoritmos de Processamento de L´ıngua Natural em Ambientes Distribu´ıdos. Master’s thesis, Instituto Superior T´ecnico, Universidade T´ecnica de Lisboa, Portugal (2008) 13. Medeiros, J.C.: An´ alise morfol´ ogica e correc¸ca ˜o ortogr´ afica do portuguˆes. Master’s thesis, Instituto Superior T´ecnico, Universidade T´ecnica de Lisboa, Portugal (1995) (in Portuguese) 14. Rodrigues, D.J.: Uma evolu¸ca ˜o no sistema ShRep: optimizac˜ ao, interface gr´ afica e integrac˜ ao de mais duas ferramentas”. Master’s thesis, Instituto Superior T´ecnico, Universidade T´ecnica de Lisboa, Portugal (2007) (in Portuguese) 15. Paulo, J.: Extrac¸ca ˜o Semi-Autom´ atica de Termos. Master’s thesis, Instituto Superior T´ecnico, Universidade T´ecnica de Lisboa, Portugal (2001) (in Portuguese) 16. A¨ıt-Mokhtar, S., Chanod, J.P., Roux, C.: A multi-input dependency parser. In: IWPT. Tsinghua University Press (2001) 17. Agrawal, R., Srikant, R.: Fast algorithms for mining association rules in large databases. In: VLDB 1994: Proceedings of the 20th International Conference on Very Large Data Bases, pp. 487–499. Morgan Kaufmann Publishers Inc., San Francisco (1994) 18. Antunes, C.M.: Pattern Mining over Nominal Event Sequences using Constraint Relaxations. PhD thesis, Instituto Superior T´ecnico, Universidade T´ecnica de Lisboa, Portugal (2005) 19. Pei, J., Han, J., Mortazavi-Asl, B., Pinto, H., Chen, Q., Dayal, U., Hsu, M.C.: Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth. In: ICDE 2001: Proceedings of the 17th International Conference on Data Engineering, Washington, DC, USA, pp. 215–226. IEEE Computer Society, Los Alamitos (2001)
Is the Distance Compression Effect Overstated? Some Theory and Experimentation Stephen France1 and Douglas Carroll2 1
Lubar School of Business, UW – Milwaukee, 3202 N. Maryland Avenue., Milwaukee, Wisconsin, 53201-0742 2 Rutgers University, Graduate School of Management, Newark, New Jersey, 07102-3027 [email protected], [email protected]
Abstract. Previous work in the document clustering literature has shown that the Minkowski-p distance metrics are unsuitable for clustering very high dimensional document data. This unsuitability is put down to the effect of “compression” of the distances created using the Minkowski-p metrics on high dimensional data. Previous experimental work on distance compression has generally used the performance of clustering algorithms on distances created by the different distance metrics as a proxy for the quality of the distance representations created by those metrics. In order to separate out the effects of distances from the performance of the clustering algorithms we tested the homogeneity of the latent classes with respect to item neighborhoods rather than testing the homogeneity of clustering solutions with respect to latent classes. We show the theoretical relationships between the cosine, correlation, and Euclidean metrics. We posit that some of the performance differential between the cosine and correlation metrics and the Minkowski-p metrics is due to the inbuilt normalization of the cosine and correlation metrics. The normalization effect decreases with increasing dimensionality and the distance compression effect increases with increasing dimensionality. For document datasets with dimensionality up to 20,000, the normalization effect dominates the distance compression effect. We propose a methodology for measuring the relative normalization and distance compression effects. Keywords: Dimensionality, Minkowski metrics, normalization, clustering, latent classes.
1 Introduction There has been much recent work in the data mining field on the effects of high dimensional data on various data mining algorithms and on the problems of using the Euclidean metric to calculate distances in high dimensions. This is due to the fact that nearest neighbors may be ill defined when using the Euclidean metric to calculate distances in high dimensional space. Much, but not all of this research has focused on the performance of clustering algorithms on distances calculated using different distance metrics [3][12]. Non-clustering work includes [15], which describes how high dimensional data affect neural-network based algorithms. P. Perner (Ed.): MLDM 2009, LNAI 5632, pp. 280–294, 2009. © Springer-Verlag Berlin Heidelberg 2009
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
281
The document data sets described in [3] and [12] have a very specific format. Documents are items to be clustered and words correspond to data dimensions. Each ‘word’ contained within at least one document constitutes a dimension, and the number of occurrences of a word in a document is the value for that document on the corresponding word dimension. Each document tested is a member of some underlying latent class. The latent class is often the document category; e.g., for documents that are news stories, the classes may include sports, crime, business, home, international, etc. Work on document clustering, such as [3], often uses the homogeneity of clusters with respect to underlying classes as a proxy for the quality of the clustering solutions. Solutions where clusters correspond to latent classes are considered to be of high quality. We perform a similar analysis to determine the quality of neighborhood recovery for pair-wise distances between items. High quality distance representations should have more homogenous class labels for pairs of items with short distances between them, and less homogenous class labels for pairs of items with large distances between them. For many clustering and dimensionality reduction applications where the input data are in the form of object × stimuli data, the Euclidean metric is used to calculate input distances. The Euclidean metric is a special case of the more general Minkowski metric given in (1), with p = 2. If p = 1 the metric is known as the city-block or Manhattan metric. 1/ p
p ⎞ ⎛ d d ( x i , x j ) = ⎜ ∑ xi , k − x j , k ⎟ ⎝ k =1 ⎠
(1)
Despite the fact that the Euclidean metric is the most common metric used in the clustering literature, it is not always the most appropriate metric to use. Recent data mining research suggests that the concept of nearest neighbors of points calculated using Euclidean distances becomes meaningless as the dimensionality of the data increases. For these data, higher dimensional distances should not be calculated using the Euclidean metric. [1] extends [2], and shows that for an arbitrary distribution of n points embedded in a space of dimensionality d, ⎡ D max dp − D min dp ⎤ C p ≤ lim d →∞ E ⎢ ⎥ ≤ ( n − 1) C p , d 1 p −1 2 ⎣ ⎦
(2)
where p is the Minkowski metric, Cp is an arbitrary constant dependent on the metric, and Dmax and Dmin are the maximum and minimum distances between any pair of the n points. Thus, D max dp − D min dp increases at a rate of d 1 p −1 2 , independent of the distribution. As the dimensionality d increases, for p = 1 (city-block) d 1 p −1 2 increases at a rate of d 1 2 , for p = 2 (Euclidean) d 1 p −1 2 = 1 , and for p > 2 d 1 p −1 2 converges to 0. Thus, the higher the value of p, the less relative distance there is between nearest and furthest neighbor. [1] states that the city-block p = 1 metric is preferable to the Euclidean distance metric (for high dimensional data mining applications). [12] tests several clustering procedures on high dimensional document data, using a variety of different distance metrics. The paper suggests that cosine, extended
282
S. France and D. Carroll
Jaccard, and correlation distances are the most appropriate metrics for dealing with high dimensional data. The cosine distance between two items is based upon the difference in angles with which the item vectors meet with the origin. The cosine distance is given in (3). d ( xi , x j ) = 1 −
xi ⋅ x′j
( xi ⋅ x′i )
1/ 2
(x
⋅ x′j )
(3)
1/ 2
j
The correlation distance is the correlation subtracted from 1, giving a maximum correlation distance of 2 (when the correlation is -1) and a minimum correlation distance of 0 (when the correlation is 1). The formula for the correlation distance is given in (4). d ( xi , x j ) = 1 −
( xi − 1.xi ) ( x j − 1.x j )′ ⎡ x − 1.x )( x − 1.x )′ ⎤ i i i ⎥ ⎢⎣( i ⎦
1/2
⎡ ′⎤ ⎢⎣( x j − 1.x j )( x j − 1.x j ) ⎥⎦
1/2
(4)
The extended Jaccard distance is a continuous extension of the discrete Jaccard distance, which measures the ratio of the number of attributes shared by both item vectors to the number of attributes possessed by either item. The formula for the extended Jaccard distance is given in (5).
d ( xi , x j ) = 1 −
xi ⋅ x′j
( xi ⋅ xi′ ) + ( x j ⋅ x′j ) − xi ⋅ x′j
(5)
2 Experimentation In this section we begin by testing the usage of a wide range of distance metrics on high dimensional data sets. We find somewhat unexpected results and posit that that the performance differential between the Euclidean metric and the cosine and correlation metrics, previously ascribed to “distance compression” is in fact due to the inbuilt normalization in the latter two metrics. High quality distance representations should have more homogenous class labels for pairs of items with short distances between them, and less homogenous class labels for pairs of items with large distances between them. We test this homogeneity directly using neighborhood recovery measures rather than testing the performance of clustering algorithms on the data. We do this in order to decouple the effects of the different distance metrics tested from the performance effects of the clustering algorithms used. Consider a situation where distances are calculated for very high dimensional data and the homogeneity of the resulting distance representations is tested with respect to the underlying latent classes. We consider the following research questions: Q1: How does the use of different distance metrics affect the discriminatory power of distances for very high dimensional data? Based on the previously described
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
283
literature, one would expect that the use of Euclidean distances would cause problems with discrimination between nearest neighbors in the higher dimensional space. Testing how different distance metrics recover homogeneity between items with respect to the underlying latent classes should provide a measure of the usefulness of the distance metrics for use with high dimensional data. Q2: Which Minkowski metric is most appropriate for calculating distances for very high dimensional data? Based upon the theoretical results from the data mining literature, the city-block metric should perform better on high dimensional data sets than the Euclidean metric or Minkowski metric with p > 2. In order to test Q1 and Q2 we calculated distances between items on high dimensional data sets and then calculated neighborhood recovery with respect to the underlying latent classes. We considered six different data sets. Four of the data sets were high dimensional document data sets where each word corresponds to a single dimension in the data set. Given the large number of possible words relative to the size of the documents, the document data are rather sparse, with a small number of non-zero entries relative to the total number of entries. Two non-document data sets were included in order to compare between types of data sets. All six data sets had items grouped into underlying latent classes. The four high dimensional document data sets were taken from the CLUTO clustering data repository [9]. The k1a data set (see [3] for a description) originated from the WebACE project and describes webpage documents. The documents are split into classes using Yahoo subject categories. The fbis and la data sets are taken from [13]. The fbis data set is from the Foreign Broadcast Information Service and each latent class is a category of broadcast. For the la data set, each document is a news story in the Los Angeles Times and the documents are categorized into classes using the ‘desk’ of the paper from which the article was sourced. The OHSUM data set is taken from the OHSUMED database of medical journal articles. The articles are categorized using medical document categories. The data are described in [8]. The data sets used are summarized in Table 1. The isolet and SEG data sets are intended to provide comparison with the highdimensional document data sets. These data sets contain dimensions that are not sparse (i.e., have non-zero values for most items) and that measure continuous attributes. The isolet data set has a relatively high dimensionality (617) for the type of data set and the SEG data set has a lower dimensionality (18). The isolet data set is taken from a speech recognition classification task. Each item in the data is the recording of a letter being spoken. The latent classes for the data are the 26 letters of the alphabet. Each dimension is a sensor reading for the recording. A description of the data set is given in [6]. The SEG data set is taken from a data set stored at the UCI Machine Learning Repository and donated by the Vision Group at the University of Massachusetts [4]. Each item in the SEG data set contains measurements taken on a subset of a certain image. The different images from which the measurements are taken correspond to the latent classes for the data set and each dimension corresponds to a measurement on the image subset.
284
S. France and D. Carroll Table 1. Data set summary
Data fbis isolet k1a la OHSUM SEG
Dimensions 2463 617 21839 31472 11465 18
Type Sparse Count real (non-sparse) Sparse Count Sparse Count Sparse Count real (non-sparse)
Data set description [13] [6] [3] [13] [8] [4]
For each data set, we took four sets of 500 items and derived distance matrices from the data. We calculated separate distance matrices for the Euclidean, city-block, Minkowski for p = 3, 4, 10, cosine, correlation, and extended Jaccard metrics. In total, 6 × 4 × 7 = 168 distance matrices were calculated. From the distances we tested the quality of the distance representations with respect to the underlying latent classes. In order to determine the quality of the distance representations we tested the homogeneity of item neighborhoods with respect to the underlying latent classes. As mentioned previously, one would expect that high quality distance representations should have more homogenous class labels for items with short distances between pairs of items, and less homogenous class labels for items with large distances between pairs of items. We test this homogeneity using three measures, the nearest neighbor, the decile lift, and the GINI coefficient. When describing these measures, it is assumed that the k nearest neighbors of a point i are the k points with the smallest values of the distance d(i,k) using the distance metric to be tested. Many data analysis methods, including many partitioning and hierarchical clustering algorithms, use the concept of nearest or furthest neighbor from either an item to be clustered or from a cluster centroid. Thus, an accurate representation of item neighborhoods is a necessary condition for the strong performance of these methods. The nearest neighbor (NN) is the simplest of the approaches described and measures the latent class homogeneity with respect to the single nearest neighbor of each point. Consider each point i in the configuration and set ai to be 1 if the nearest neighbor of i has the same latent class as i, otherwise set ai to be 0. The value of the nearest neighbor metric is given in (6). The value of the metric varies between 0 and 1, with higher values indicating strong neighborhood recovery with respect to the latent classes.
NN =
1 n ∑ ai n i =1
(6)
Both the decile lift (DL) metric and the GINI coefficient are often used as measures of classification performance, e.g., [10]. A modified version of the decile lift for calculating medium range neighborhood recovery is given in (7). The decile lift measures the increase in the proportion of the labels of the 10% nearest neighbors of an item having the same latent class as the item over the proportion of the latent class across all items (except the tested point) in the configuration. Let ai,j be 1 if the jth nearest neighbor of item i has the same latent class as point i and be 0 otherwise. The decile lift over all n points is given in (7).
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
DL =
1 n ∑ n i =1
⎡ ⎣⎢ n −1 10⎦⎥ ⎤ ⎢ ∑ ai , j ⎣⎢ n − 1 10 ⎦⎥ ⎥ ⎢⎣ j =1 ⎥⎦ n −1
∑ ai, j n − 1
285
(7)
j =1
The GINI coefficient [5] was originally developed to measure a nation’s level of income equality/inequality. Consider a graph with the cumulative share of population plotted on the horizontal axis and the cumulative share of income plotted on the vertical axis. A straight line indicates an equal distribution of income and a line forming a convex function below the straight line indicates some income inequality (assuming that people are ordered in terms of increasing wealth). The GINI coefficient for a given convex income line is the proportion of area below the linearly increasing income equality line that is above the convex income line. We use the GINI coefficient as a measure of neighborhood recovery throughout item neighborhoods of all sizes. For an item i, the other n – 1 items are ordered in decreasing order of distance from item i. Taking each item i, ci,j is the sum of items with the same class label as i from items 1 to j, with items ordered in decreasing distance from i. j
ci , j = ∑ ai
(8)
i =1
Given some homogeneity of items with respect to latent classes, it would be expected that the value of ci,j will increase slowly for the first items, which are further away from item i than for the later items, which are closer. Greater “inequality” of the cumulative distribution leads to a more strongly convex graph and thus a larger GINI coefficient. It is possible that most items with the same latent class as item i have the largest distances from the item. This would lead to a concave graph and a negative value of the GINI coefficient. The equation for the GINI coefficient, aggregated for all items, is given in (9). ⎡ ⎛ ⎛ n −1 ⎢ ⎜ ⎜ ∑ ( n − j ) ci , j 1 ⎢ 1 ⎜ j =1 G(S ) = ∑ n − 2⎜ n −1 ⎜ n i =1 ⎢ n − 1 ⎜ ci , j ⎢ ∑ ⎜ ⎜ j =1 ⎢⎣ ⎝ ⎝ n
n −1 ⎞ ⎞⎤ ⎡ ⎤ 2∑ j ⋅ ci , j ⎟ ⎟⎥ ⎥ n ⎢ 1 n j =1 ⎟ ⎟⎥ = ∑ ⎢ ⎥ − n −1 ⎟ ⎟ ⎥ n i =1 ⎢ n − 1⎥ n 1 j c − ⋅ ) ∑ i, j ⎟ ⎟⎥ ⎢( ⎥ j =1 ⎠ ⎠ ⎥⎦ ⎣ ⎦
(9)
In order to test Q1 and Q2, the values of the nearest neighborhood, decile lift, and GINI coefficients were calculated for each combination of data set and distance metric. Based upon the previously described theoretical work of [1] and the experimental work of [12], one would expect that distances calculated using the cosine distance, correlation distance, or extended Jaccard distance would be the better at recovering neighborhood structure than those calculated with the Euclidean metric or Minkowski-p metrics with higher values of p. One would expect distances calculated with the city-block metric to be intermediate to the results of the other distance metrics.
286
S. France and D. Carroll
2.1 Results
The results for the 6 files, 4 item sets per file, and 7 distance metrics per item set are summarized in Tables 2-4. Table 2. Summary of results for nearest neighbor
Avg of NN Metric CB Corr Cos Euclid Mi10 Mi3 Mi4 ExJac
File fbis 0.4715 0.709 0.7095 0.6145 0.5825 0.6185 0.6005 0.696
isolet 0.7245 0.7625 0.754 0.7585 0.646 0.779 0.767 0.7545
k1 0.191 0.439 0.4395 0.251 0.372 0.334 0.3555 0.4465
la12 0.441 0.4765 0.4735 0.3935 0.238 0.376 0.34 0.7635
OH 0.176 0.4815 0.481 0.301 0.331 0.3485 0.3405 0.4805
segmt 0.945 0.9245 0.931 0.925 0.9 0.914 0.9105 0.9245
Table 3. Summary of results for decile lift
Avg of DL Metric CB Corr Cos Euclid Mi10 Mi3 Mi4 ExJac
File fbis 2.2883 4.7988 4.7623 2.8080 2.9050 2.9458 2.9488 4.4288
isolet 6.8522 7.1040 7.0248 7.0870 5.9859 7.2547 7.2652 7.0240
k1 1.1573 2.1508 2.1611 1.1873 1.4411 1.2628 1.3139 2.4117
la12 1.9740 1.9641 1.9633 1.6225 1.0461 1.2474 1.1683 2.4538
OH 1.2121 2.7693 2.7669 1.3514 1.6257 1.4811 1.5324 2.7628
segmt 4.7414 4.4367 4.5034 4.5801 4.1044 4.4317 4.3405 4.5785
Table 4. Summary of results for GINI coefficient
Avg of GINI Metric CB Corr Cos Euclid Mi10 Mi3 Mi4 ExJac
File fbis 0.3700 0.6910 0.6851 0.3646 0.3618 0.3621 0.3619 0.6025
isolet 0.7949 0.7975 0.7938 0.8005 0.7413 0.8027 0.8026 0.7937
k1 0.3688 0.4799 0.4800 0.3764 0.3614 0.3720 0.3686 0.4786
la12 0.4189 0.4260 0.4254 0.3637 0.3376 0.3397 0.3388 0.4107
OH 0.3853 0.5106 0.5085 0.3766 0.3596 0.3688 0.3660 0.5078
segmt 0.7004 0.7032 0.7128 0.6370 0.5058 0.5799 0.5570 0.7137
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
287
The data were analyzed using the MANOVA technique. Both the distance metric and the file used were considered to be factors, with each of the four random data sets considered as a single response for a combination of metric and file. The values of the nearest neighbor, the decile lift, and the GINI coefficient were taken as dependent variables. Using Wilks’s Lamba, both factors and the interaction between factors were significant, with p = 0.000 to 3.s.f for all values. Tests of between subject effects, were significant for all combinations of dependent variables and factors, again with p = 0.000 for all values. Post hoc tests were carried out on the marginal means of the different distance metrics for all three dependent variables. The Scheffé test [11] was used, as it is a fairly conservative and robust post-hoc test. These post-hoc tests show strong performance of the cosine and correlation metrics relative to the Minkowski-p distances, but we note that as the data sets are biased towards high dimensional data sets. We aim to gain more insight into the performance of the metrics by plotting, for each dependent variable, the marginal means of each file and metric. The metrics are given on the horizontal axis, the marginal means are given on the vertical axis, and separate series are plotted for each file. The plots for nearest neighbor, decile lift, GINI coefficient are given in Fig.’s 1, 2, and 3 respectively. We wish only to compare plots for individual files. To compare performance between series would not be appropriate as the different data sets have different numbers and distributions of latent classes. Multiple series are only displayed on a single graph in order to aid parsimony. The cosine, correlation, and extended Jaccard metrics outperform the Minkowski metrics. All three graphs show several clear trends. For three of the four high dimensional document data sets, the cosine and correlation metrics have superior performance to the Minkowski metrics. The one data set that does not conform to this trend is the la data set, for which case the city-block metric produces virtually identical performance to the correlation and cosine metrics. In fact, the results are not as expected, as the outperformance of the cosine and correlation metrics over the cityblock and Euclidean metrics on the high-dimensional document data sets is not related to the dimensionality of the data. For example, the fbis data set with a dimensionality of 2463, which has the lowest dimensionality of all of the high dimensional document data sets, has the greatest performance increase for the correlation and cosine metrics over the Euclidean metric. There is also little difference between the performances of the Minkowski-p metrics for p ≥ 2. Particularly for the high-dimensional document data sets, there is little difference in performance between p = 2, 3, and 4. The exceptions to this are the very high dimensional la and k1a data sets, where this effect appears to be present. There is a significant decrease in performance for some of the data sets with the p = 10 metric, particularly using the larger neighborhood measures, the decile lift and the GINI coefficient. It would be expected, given past experimental work and equation (2), that there would be a decreasing level of performance as p increases from 2 to 4, but that is not the case for all data sets. We explore possible reasons for these results in the theoretical discussion in the next section.
288
S. France and D. Carroll
Estimated Marginal Means of Nearest Neighbor 1
0.9
0.8
0.7
fbis isolet
Means
0.6
k1 la12 OH segmt
0.5
0.4
0.3
0.2
0.1
0 ExJac
Corr
Cos
CB
Euclid
Mi3
Mi4
Mi10
Metric
Fig. 1. Estimated marginal means of nearest neighborhood for high-dimensional distances
Estimated Marginal Means of Decile Lift 8
7
6
Means
5
fbis isolet k1 la12 OH segmt
4
3
2
1
0 ExJac
Corr
Cos
CB
Euclid
Mi3
Mi4
Mi10
Metric
Fig. 2. Estimated marginal means of decile lift for high-dimensional distances
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
289
Estimated Marginal Means of GINI Coefficient 0.9
0.8
0.7
Means
0.6 fbis isolet k1 la12 OH segmt
0.5
0.4
0.3
0.2
0.1
0 ExJac
Corr
Cos
CB
Euclid
Mi3
Mi4
Mi10
Metric
Fig. 3. Estimated marginal means of GINI coefficient for high-dimensional distances
3 Theoretical Discussion Past theoretical work has shown that for the family of Minkowski-p metrics, distances between points become increasingly compressed as the dimensionality of data increases, which leads to an inability to distinguish between neighborhoods of points. Past empirical work has tested the performance of both Minkowski-p and nonMinkowski-p distance metrics. In these experiments the homogeneity of clustering solutions with respect to latent classes is tested. The results from these experiments show that the cosine, correlation, and extended Jaccard metrics outperform the Minkowski-p distance metrics on these data. It was concluded from these experiments that the cause of this performance disparity is due to the previously described phenomenon of distance compression on high dimensional data. Here the homogeneity of the clustering solutions relative to a set of latent classes is used as a proxy for the neighborhood recovery of points relative to the latent classes. There are several aspects of our results that do not mesh with the previously described work. These are listed below: The difference in performance between the cosine and correlation metrics and the Minkowski-p metrics is highest for the data set with the lowest dimensionality (fbis) and lowest for the data set with the highest dimensionality (la). Given the inequality (2), one would expect the opposite. Except for the very high dimensional data set, there is little difference in performance of the Minkowski-p metrics for p = 2, 3, and 4. As there is a large difference in performance between the Minkowski-p metrics and the non-Minkowski metrics ascribed to (2), one would expect a strong difference in performance between
290
S. France and D. Carroll
the Minkowski metrics. With increasing dimensionality and given (2), there is a constant bound on the differences between distances for p = 2 and a converging bound for p > 2, so one would expect differences in performance that in the experimental results do not occur or only occur slightly. We account for these inconsistencies as follows. We believe that there are two effects at work. These effects are a normalization effect and a dimensionality effect. The normalization effect is due to the nature of the document data tested and was misinterpreted as a dimensionality effect in previous studies due to the document data being the only type of high dimensional data tested in these studies. The normalization effect is strongest in the document data sets with lower dimensionality and less strong in document data sets with very high dimensionality. We found evidence of the dimensionality effect in the high dimensional la data set, but believe that the dimensionality effect is overstated for data sets with dimensionality between 2,000 and 20,000 and that most of the performance difference between the cosine and correlation metrics and the Minkowski-p metrics is due to the normalization effect. So what is meant by normalization and how does this affect the Minkowski-p distance metrics relative to the cosine, correlation, and extended Jaccard metrics? Consider a document data set consisting of newpaper stories, with latent classes corresponding to the story categories (e.g., news, finance, sport, house & home, business, etc). Consider two stories, one a 2000 word house & home feature detailing the revitalization of a city neighborhood and the other a 200 word sports bulletin. Both stories may mention the word tennis ‘two times’, the first in the context of the local sports facilities and the second in the context of the sports results. However, this similarity is misleading as only a very small proportion of the first article is concerned with tennis and the article belongs to a different latent class than does the sports bulletin. If the data were proportioned or normalized by dividing each word count by either the total number of words in the document or by the Euclidean norm for the story vector, then the similarities/distances between the story vectors may provide stronger predictors of the story latent class. The cosine and correlation metrics are strongly related to the Euclidean metric. In fact one can describe these metrics as special cases of the squared Euclidean metric on appropriately normalized data. The squared Euclidean metric is monotone increasing with respect to the Euclidean metric, so it can be shown that the Euclidean metric with appropriately normalized data produces the same item neighborhoods as the cosine metric. Theorem 1. Let X be an m × n data matrix with m items and n dimensions. Let X be normalized to Xnorm so that each row xi is normalized to having xi = 1 for all i =
1⋅⋅⋅m. Let D1 be the m × m distance matrix where D1ij is the cosine distance between row vectors xi and xj of X. Let D2 be the m × m distance matrix where D2ij is the Euclidean distance between row vectors of Xnorm, so that x*i = xi x and x*j = x j x . Then D1 and D2 share the same neighborhoods, i.e., for any point i and some permutation P of the remaining n – 1 points ordered so that D1iP1 ≤ D1iP2 ≤ ⋅⋅⋅ D1iP(n – 2) ≤ D1iP(n – 1) and some permutation Q of the remaining n – 1 points ordered so that D2iQ1 ≤ D2iQ2 ≤ ⋅⋅⋅ D2iQ(n – 2) ≤ D2iQ(n – 1) then P = Q.
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
291
Proof. Take any arbitrary point i in the configuration Xnorm. Arrange the remaining n - 1 points in ascending order of Euclidean distances from point i. The permutation of points in order of distances is given as D2iQ1 ≤ D2iQ2 ≤ ⋅⋅⋅ D2iQ(n – 2) ≤ D2iQ(n – 1). By the distance axioms, all distances are nonnegative, so squared Euclidean distances are monotone increasing with Euclidean distances, and (10) holds. D2iQ1 ≤ D2iQ 2 ≤ ⋅⋅⋅ ≤ D2iQ1( n − 2) ≤ D2iQ ( n −1) ⇔ D22iQ1 ≤ D22iQ 2 ≤ ⋅⋅⋅ ≤ D22iQ1( n − 2) ≤ D22iQ ( n −1)
(10)
The squared Euclidean distances for Xnorm can be expressed as Dij2 = ( x*i − x*j )( x*i − x*j )′ = x*i
2
+ x*j
2
− 2 x*i , x*j ,
(11)
where x*i and x*j are row vectors of Xnorm. From the definition of x*i and x*j , x*i = x*j = 1 , giving
(
Dij2 = 2 − 2 x*i , x*j = 2 1 − x*i , x*j
).
(12)
.
(13)
Again, from the definition x*i and x*j , x*i , x*j =
x i xi , xi x j
=
xi , x j xi x j
Putting (12) into (13) and applying the cosine rule to the angle between the two vectors xi and xj, gives (14), which is the definition of the cosine distance.
Dij2 2
= 1−
xi , x j xi x j
= 1 − cos θ .
(14)
By (14) half of the squared Euclidean distances on the Xnorm are equal to the cosine distances on X, so have the same neighborhoods and as Dij2 2 is monotone with respect to Dij2 then squared Euclidean distances on the Xnorm have the same neighborhoods as the cosine distances on X. By (10) the cosine distances on X have the same neighborhoods as the Euclidean distances on Xnorm. Theorem 2. Given the mean centered matrix X• = X - 1m x , then Euclidean distances on the normed mean centered matrix X•norm produce the same neighborhoods as the correlation distances applied to X. Proof: The proof follows as for Theorem 1, except that (13) becomes, x•i , x•j =
xi − x x j − x , xi − x x j − x
=
xi − x , x j − x xi − x x j − x
= r ( xi , x j ) ,
(15)
292
S. France and D. Carroll
where r ( xi , x j ) is the correlation between xi and xj. It follows that Dij2
2
= 1 − xi• , x•j = 1 − r ( xi , x j ) .
(16)
The remainder of the proof is the same as that for Theorem 1. Thus, the Euclidean metric applied to Xnorm and X•norm , respectively produces the same item neighborhoods as the cosine and correlation metric applied to X. Given both the experimental and theoretical results, we believe that most of the difference between the performance of the Minkowski-p metrics and the correlation and cosine metrics is due to the inbuilt normalization in the latter two metrics. We cannot discount completely the possibility that the compression of high dimensional distances shown in (2) affects normalized data and non-normalized data differently, but the fact that the outperformance of the cosine and correlation metrics relative to the Minkowski-p metrics is inversely proportional to dimensionality suggests that distance compression is not a major factor in the difference in performance between the metrics. The conclusion, in previous experimental work, that the difference in performance between the Minkowski-p metrics and the cosine and correlation metrics is due to distance compression is probably due to a couple of reasons. The first is that real world data sets of very high dimensionality are rare. Most of the high dimensional data sets are document data sets, in which words are considered as dimensions. In previous research these data sets are often taken as a proxy for all high dimensional data. These data are sparse count data, which cannot be taken as being typical of all high dimensional data. Also, previous research tested both the clustering/classification algorithm and the distance metrics in a single experiment. Thus it is much harder to detect patterns in results than if neighborhood recovery is tested directly. We do believe that the compression of distances shown in (2) is a factor in the relative performance of distance metrics in high dimensional space, and we make no attempt to question the validity of the work described in [1] and [2]. We note that some differences between the performances of the Minkowski-p distance metrics become apparent in the 21,839 dimensional k1a and 31472 dimensional la data sets. We also note some degradation in the performance of the cosine and correlation distance metrics relative to the extended Jaccard metric on the l1a data set. This may be because of distance compression affecting neighborhood recovery for the Euclidean metric and thus the neighborhood recovery for the cosine and correlation metrics due to the neighborhood agreement shown in Theorems 1 and 2. We conclude that the use of Minkowski-p metrics on non-normalized document data is not appropriate for applications reliant on the preservation of neighborhood structure. It is appropriate to use the Euclidean metric on normalized data or mean centered and normalized data. The resulting neighborhood structures are identical to those produced by the cosine and correlation distances respectively. For very high dimensional data it may be more appropriate to use the extended Jaccard metric rather than the previously mentioned metrics.
Is the Distance Compression Effect Overstated? Some Theory and Experimentation
293
4 Future Work In this paper we have proposed that the difference in performance of distance metrics for the preservation of neighborhood structure is due to two main factors, that of normalization and that of distance compression. Future work could help clarify several aspects of this performance relationship. Firstly, the only high dimensional data sets tested in either this paper or in previous high dimensional work are document data sets. The SEG and isolet data sets were not of this type but were of lower dimensionality. Both these data sets had continuous valued dimensions, with each dimension measured on a different scale. Thus, the data were standardized before applying the Euclidean metric. If high dimensional continuous data, with dimensions measured on different scales, are not available then it may be possible to artificially generate the data. Futher testing on further high dimensional document data sets would aid the discovery of the relationship between the normalization and distance compression effects. By testing the differences between the performances of the Minkowski-p metrics, a measure of the effects of distance compression on neighborhood preservation could be gained. The shape of the curv