“00-FM-SA272” 18/9/2008 page iv
Academic Press is an imprint of Elsevier 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK
⬁ This book is printed on acid-free paper. Copyright © 2009, Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then “Obtaining Permissions.” Library of Congress Cataloging-in-Publication Data Application submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. ISBN: 978-1-59749-272-0 For information on all Academic Press publications visit our Web site at www.books.elsevier.com Printed in the United States of America 09 10 11 12 13 14 15 16 5 4 3 2 1
“02-Preface-SA272” 17/9/2008 page xv
Preface This book is the outgrowth of our teaching advanced undergraduate and graduate courses over the past 20 years. These courses have been taught to different audiences, including students in electrical and electronics engineering, computer engineering, computer science, and informatics, as well as to an interdisciplinary audience of a graduate course on automation. This experience led us to make the book as self-contained as possible and to address students with different backgrounds. As prerequisitive knowledge, the reader requires only basic calculus, elementary linear algebra, and some probability theory basics. A number of mathematical tools, such as probability and statistics as well as constrained optimization, needed by various chapters, are treated in four Appendices. The book is designed to serve as a text for advanced undergraduate and graduate students,and it can be used for either a one- or a two-semester course. Furthermore,it is intended to be used as a self-study and reference book for research and for the practicing scientist/engineer. This latter audience was also our second incentive for writing this book, due to the involvement of our group in a number of projects related to pattern recognition.
SCOPE AND APPROACH The goal of the book is to present in a unified way the most widely used techniques and methodologies for pattern recognition tasks. Pattern recognition is in the center of a number of application areas, including image analysis, speech and audio recognition, biometrics, bioinformatics, data mining, and information retrieval. Despite their differences, these areas share, to a large extent, a corpus of techniques that can be used in extracting, from the available data, information related to data categories, important “hidden”patterns, and trends. The emphasis in this book is on the most generic of the methods that are currently available. Having acquired the basic knowledge and understanding, the reader can subsequently move on to more specialized application-dependent techniques, which have been developed and reported in a vast number of research papers. Each chapter of the book starts with the basics and moves, progressively, to more advanced topics’and reviews up-to-date techniques. We have made an effort to keep a balance between mathematical and descriptive presentation. This is not always an easy task. However, we strongly believe that in a topic such as pattern recognition, trying to bypass mathematics deprives the reader of understanding the essentials behind the methods and also the potential of developing new techniques, which fit the needs of the problem at hand that he or she has to tackle. In pattern recognition, the final adoption of an appropriate technique and algorithm is very much a problem-dependent task. Moreover, according to our experience, teaching pattern recognition is also a good “excuse” for the students to refresh and solidify
xv
“02-Preface-SA272” 17/9/2008 page xvi
xvi
Preface
some of the mathematical basics they have been taught in earlier years. “Repetitio est mater studiosum.”
NEW TO THIS EDITION The new features of the fourth edition include the following. ■
MATLAB codes and computer experiments are given at the end of most chapters.
■
More examples and a number of new figures have been included to enhance the readability and pedagogic aspects of the book.
■
New sections on some important topics of high current interest have been added, including: • Nonlinear dimensionality reduction • Nonnegative matrix factorization • Relevance feedback • Robust regression • Semi-supervised learning • Spectral clustering • Clustering combination techniques
Also, a number of sections have been rewritten in the context of more recent applications in mind.
SUPPLEMENTS TO THE TEXT Demonstrations based on MATLAB are available for download from the book Web site, www.elsevierdirect.com/9781597492720. Also available are electronic figures from the text and (for instructors only) a solutions manual for the end-of-chapter problems and exercises. The interested reader can download detailed proofs, which in the book necessarily, are sometimes, slightly condensed. PowerPoint presentations are also available covering all chapters of the book. Our intention is to update the site regularly with more and/or improved versions of the MATLAB demonstrations. Suggestions are always welcome. Also at this Web site a page will be available for typos, which are unavoidable, despite frequent careful reading. The authors would appreciate readers notifying them about any typos found.
“02-Preface-SA272” 17/9/2008 page xvii
Preface
ACKNOWLEDGMENTS This book would have not been written without the constant support and help from a number of colleagues and students throughout the years. We are especially indebted to Kostas Berberidis, Velissaris Gezerlis, Xaris Georgion, Kristina Georgoulakis, Leyteris Kofidis, Thanassis Liavas, Michalis Mavroforakis, Aggelos Pikrakis,Thanassis Rontogiannis, Margaritis Sdralis, Kostas Slavakis, and Theodoros Yiannakoponlos. The constant support provided by Yannis Kopsinis and Kostas Thernelis from the early stages up to the final stage, with those long nights, has been invaluable. The book improved a great deal after the careful reading and the serious comments and suggestions of Alexandros Bölnn. Dionissis Cavouras, Vassilis Digalakis, Vassilis Drakopoulos, Nikos Galatsanos, George Glentis, Spiros Hatzispyros, Evagelos Karkaletsis, Elias Koutsoupias, Aristides Likas, Gerassimos Mileounis, George Monstakides, George Paliouras, Stavros Perantonis, Takis Stamatoponlos, Nikos Vassilas, Manolis Zervakis, and Vassilis Zissimopoulos. The book has greatly gained and improved thanks to the comments of a number of people who provided feedback on the revision plan and/or comments on revised chapters: Tulay Adali, University of Maryland; Mehniet Celenk, Ohio University; Rama Chellappa, University of Maryland; Mark Clements, Georgia Institute of Technology; Robert Duin, Delft University of Technology; Miguel Figneroa,Villanueva University of Puerto Rico; Dimitris Gunopoulos, University of Athens; Mathias Kolsch, Naval Postgraduate School; Adam Krzyzak, Concordia University; Baoxiu Li,Arizona State University; David Miller, Pennsylvania State University; Bernhard Schölkopf, Max Planck Institute; Hari Sundaram, Arizona State University; Harry Wechsler, George Mason University; and Alexander Zien, Max Planck Institute. We are greatly indebted to these colleagues for their time and their constructive criticisms. Our collaboration and friendship with Nikos Kalouptsidis have been a source of constant inspiration for all these years. We are both deeply indebted to him. Last but not least, K. Koutroumbas would like to thank Sophia, DimitrisMarios, and Valentini-Theodora for their tolerance and support and S.Theodoridis would like to thank Despina, Eva, and Eleni, his joyful and supportive “harem.”
xvii
“03-Ch01-SA272” 17/9/2008 page 1
CHAPTER
Introduction
1
1.1 IS PATTERN RECOGNITION IMPORTANT? Pattern recognition is the scientific discipline whose goal is the classification of objects into a number of categories or classes. Depending on the application, these objects can be images or signal waveforms or any type of measurements that need to be classified. We will refer to these objects using the generic term patterns. Pattern recognition has a long history, but before the 1960s it was mostly the output of theoretical research in the area of statistics. As with everything else, the advent of computers increased the demand for practical applications of pattern recognition, which in turn set new demands for further theoretical developments. As our society evolves from the industrial to its postindustrial phase, automation in industrial production and the need for information handling and retrieval are becoming increasingly important. This trend has pushed pattern recognition to the high edge of today’s engineering applications and research. Pattern recognition is an integral part of most machine intelligence systems built for decision making. Machine vision is an area in which pattern recognition is of importance. A machine vision system captures images via a camera and analyzes them to produce descriptions of what is imaged. A typical application of a machine vision system is in the manufacturing industry, either for automated visual inspection or for automation in the assembly line. For example, in inspection, manufactured objects on a moving conveyor may pass the inspection station, where the camera stands, and it has to be ascertained whether there is a defect. Thus, images have to be analyzed online, and a pattern recognition system has to classify the objects into the “defect” or“nondefect”class. After that,an action has to be taken,such as to reject the offending parts. In an assembly line, different objects must be located and “recognized,” that is, classified in one of a number of classes known a priori. Examples are the “screwdriver class,” the “German key class,” and so forth in a tools’ manufacturing unit. Then a robot arm can move the objects in the right place. Character (letter or number) recognition is another important area of pattern recognition, with major implications in automation and information handling. Optical character recognition (OCR) systems are already commercially available and more or less familiar to all of us. An OCR system has a “front-end” device consisting of a light source, a scan lens, a document transport, and a detector. At the output of
1
“03-Ch01-SA272” 17/9/2008 page 2
2
CHAPTER 1 Introduction
the light-sensitive detector, light-intensity variation is translated into “numbers” and an image array is formed. In the sequel, a series of image processing techniques are applied leading to line and character segmentation. The pattern recognition software then takes over to recognize the characters—that is, to classify each character in the correct “letter, number, punctuation” class. Storing the recognized document has a twofold advantage over storing its scanned image. First, further electronic processing, if needed, is easy via a word processor, and second, it is much more efficient to store ASCII characters than a document image. Besides the printed character recognition systems, there is a great deal of interest invested in systems that recognize handwriting. A typical commercial application of such a system is in the machine reading of bank checks. The machine must be able to recognize the amounts in figures and digits and match them. Furthermore, it could check whether the payee corresponds to the account to be credited. Even if only half of the checks are manipulated correctly by such a machine, much labor can be saved from a tedious job. Another application is in automatic mail-sorting machines for postal code identification in post offices. Online handwriting recognition systems are another area of great commercial interest. Such systems will accompany pen computers, with which the entry of data will be done not via the keyboard but by writing. This complies with today’s tendency to develop machines and computers with interfaces acquiring human-like skills. Computer-aided diagnosis is another important application of pattern recognition, aiming at assisting doctors in making diagnostic decisions. The final diagnosis is, of course, made by the doctor. Computer-assisted diagnosis has been applied to and is of interest for a variety of medical data,such as X-rays,computed tomographic images, ultrasound images, electrocardiograms (ECGs), and electroencephalograms (EEGs). The need for a computer-aided diagnosis stems from the fact that medical data are often not easily interpretable, and the interpretation can depend very much on the skill of the doctor. Let us take for example X-ray mammography for the detection of breast cancer. Although mammography is currently the best method for detecting breast cancer, 10 to 30% of women who have the disease and undergo mammography have negative mammograms. In approximately two thirds of these cases with false results the radiologist failed to detect the cancer, which was evident retrospectively. This may be due to poor image quality, eye fatigue of the radiologist, or the subtle nature of the findings. The percentage of correct classifications improves at a second reading by another radiologist. Thus, one can aim to develop a pattern recognition system in order to assist radiologists with a “second” opinion. Increasing confidence in the diagnosis based on mammograms would, in turn, decrease the number of patients with suspected breast cancer who have to undergo surgical breast biopsy, with its associated complications. Speech recognition is another area in which a great deal of research and development effort has been invested. Speech is the most natural means by which humans communicate and exchange information. Thus, the goal of building intelligent machines that recognize spoken information has been a long-standing one for scientists and engineers as well as science fiction writers. Potential applications of such machines are numerous. They can be used, for example, to improve efficiency
“03-Ch01-SA272” 17/9/2008 page 3
1.1 Is Pattern Recognition Important?
in a manufacturing environment, to control machines in hazardous environments remotely, and to help handicapped people to control machines by talking to them. A major effort, which has already had considerable success, is to enter data into a computer via a microphone. Software, built around a pattern (spoken sounds in this case) recognition system, recognizes the spoken text and translates it into ASCII characters, which are shown on the screen and can be stored in the memory. Entering information by “talking” to a computer is twice as fast as entry by a skilled typist. Furthermore, this can enhance our ability to communicate with deaf and dumb people. Data mining and knowledge discovery in databases is another key application area of pattern recognition. Data mining is of intense interest in a wide range of applications such as medicine and biology, market and financial analysis, business management, science exploration, image and music retrieval. Its popularity stems from the fact that in the age of information and knowledge society there is an ever increasing demand for retrieving information and turning it into knowledge. Moreover,this information exists in huge amounts of data in various forms including,text, images, audio and video, stored in different places distributed all over the world. The traditional way of searching information in databases was the description-based model where object retrieval was based on keyword description and subsequent word matching. However, this type of searching presupposes that a manual annotation of the stored information has previously been performed by a human. This is a very time-consuming job and, although feasible when the size of the stored information is limited, it is not possible when the amount of the available information becomes large. Moreover, the task of manual annotation becomes problematic when the stored information is widely distributed and shared by a heterogeneous “mixture”of sites and users. Content-based retrieval systems are becoming more and more popular where information is sought based on “similarity” between an object, which is presented into the system, and objects stored in sites all over the world. In a content-based image retrieval CBIR (system) an image is presented to an input device (e.g., scanner). The system returns“similar”images based on a measured“signature,” which can encode, for example, information related to color, texture and shape. In a music content-based retrieval system, an example (i.e., an extract from a music piece), is presented to a microphone input device and the system returns “similar” music pieces. In this case, similarity is based on certain (automatically) measured cues that characterize a music piece, such as the music meter, the music tempo, and the location of certain repeated patterns. Mining for biomedical and DNA data analysis has enjoyed an explosive growth since the mid-1990s. All DNA sequences comprise four basic building elements; the nucleotides: adenine (A), cytosine (C), guanine (G) and thymine (T). Like the letters in our alphabets and the seven notes in music, these four nucleotides are combined to form long sequences in a twisted ladder form. Genes consist of,usually, hundreds of nucleotides arranged in a particular order. Specific gene-sequence patterns are related to particular diseases and play an important role in medicine. To this end, pattern recognition is a key area that offers a wealth of developed tools for similarity search and comparison between DNA sequences. Such comparisons
3
“03-Ch01-SA272” 17/9/2008 page 4
4
CHAPTER 1 Introduction
between healthy and diseased tissues are very important in medicine to identify critical differences between these two classes. The foregoing are only five examples from a much larger number of possible applications. Typically, we refer to fingerprint identification, signature authentication, text retrieval, and face and gesture recognition. The last applications have recently attracted much research interest and investment in an attempt to facilitate human–machine interaction and further enhance the role of computers in office automation, automatic personalization of environments, and so forth. Just to provoke imagination, it is worth pointing out that the MPEG-7 standard includes a provision for content-based video information retrieval from digital libraries of the type: search and find all video scenes in a digital library showing person “X” laughing. Of course, to achieve the final goals in all of these applications, pattern recognition is closely linked with other scientific disciplines, such as linguistics, computer graphics, machine vision, and database design. Having aroused the reader’s curiosity about pattern recognition, we will next sketch the basic philosophy and methodological directions in which the various pattern recognition approaches have evolved and developed.
1.2 FEATURES, FEATURE VECTORS, AND CLASSIFIERS Let us first simulate a simplified case “mimicking” a medical image classification task. Figure 1.1 shows two images, each having a distinct region inside it. The two regions are also themselves visually different. We could say that the region of Figure 1.1a results from a benign lesion, class A, and that of Figure 1.1b from a malignant one (cancer), class B. We will further assume that these are not the only patterns (images) that are available to us, but we have access to an image database
(a)
(b)
FIGURE 1.1 Examples of image regions corresponding to (a) class A and (b) class B.
“03-Ch01-SA272” 17/9/2008 page 5
1.2 Features, Feature Vectors, and Classifiers
⫹ ⫹ ⫹ ⫹ ⫹ ⫹ ⫹⫹
FIGURE 1.2 Plot of the mean value versus the standard deviation for a number of different images originating from class A () and class B (⫹). In this case, a straight line separates the two classes.
with a number of patterns, some of which are known to originate from class A and some from class B. The first step is to identify the measurable quantities that make these two regions distinct from each other. Figure 1.2 shows a plot of the mean value of the intensity in each region of interest versus the corresponding standard deviation around this mean. Each point corresponds to a different image from the available database. It turns out that class A patterns tend to spread in a different area from class B patterns. The straight line seems to be a good candidate for separating the two classes. Let us now assume that we are given a new image with a region in it and that we do not know to which class it belongs. It is reasonable to say that we measure the mean intensity and standard deviation in the region of interest and we plot the corresponding point. This is shown by the asterisk (∗) in Figure 1.2. Then it is sensible to assume that the unknown pattern is more likely to belong to class A than class B. The preceding artificial classification task has outlined the rationale behind a large class of pattern recognition problems. The measurements used for the classification,the mean value and the standard deviation in this case,are known as features. In the more general case l features xi , i ⫽ 1, 2, . . . , l, are used, and they form the feature vector x ⫽ [x1 , x2 , . . . , xl ]T
where T denotes transposition. Each of the feature vectors identifies uniquely a single pattern (object). Throughout this book features and feature vectors will be treated as random variables and vectors, respectively. This is natural, as the measurements resulting from different patterns exhibit a random variation. This is due partly to the measurement noise of the measuring devices and partly to
5
“03-Ch01-SA272” 17/9/2008 page 6
6
CHAPTER 1 Introduction
the distinct characteristics of each pattern. For example, in X-ray imaging large variations are expected because of the differences in physiology among individuals. This is the reason for the scattering of the points in each class shown in Figure 1.1. The straight line in Figure 1.2 is known as the decision line, and it constitutes the classifier whose role is to divide the feature space into regions that correspond to either class A or class B. If a feature vector x, corresponding to an unknown pattern, falls in the class A region, it is classified as class A, otherwise as class B. This does not necessarily mean that the decision is correct. If it is not correct, a misclassification has occurred. In order to draw the straight line in Figure 1.2 we exploited the fact that we knew the labels (class A or B) for each point of the figure. The patterns (feature vectors) whose true class is known and which are used for the design of the classifier are known as training patterns (training feature vectors). Having outlined the definitions and the rationale, let us point out the basic questions arising in a classification task. ■
How are the features generated? In the preceding example, we used the mean and the standard deviation, because we knew how the images had been generated. In practice,this is far from obvious. It is problem dependent,and it concerns the feature generation stage of the design of a classification system that performs a given pattern recognition task.
■
What is the best number l of features to use? This is also a very important task and it concerns the feature selection stage of the classification system. In practice, a larger than necessary number of feature candidates is generated, and then the “best” of them is adopted.
■
Having adopted the appropriate, for the specific task, features, how does one design the classifier? In the preceding example the straight line was drawn empirically, just to please the eye. In practice, this cannot be the case, and the line should be drawn optimally, with respect to an optimality criterion. Furthermore,problems for which a linear classifier (straight line or hyperplane in the l-dimensional space) can result in acceptable performance are not the rule. In general, the surfaces dividing the space in the various class regions are nonlinear. What type of nonlinearity must one adopt, and what type of optimizing criterion must be used in order to locate a surface in the right place in the l-dimensional feature space? These questions concern the classifier design stage.
■
Finally, once the classifier has been designed, how can one assess the performance of the designed classifier? That is, what is the classification error rate? This is the task of the system evaluation stage.
Figure 1.3 shows the various stages followed for the design of a classification system. As is apparent from the feedback arrows, these stages are not independent. On the contrary,they are interrelated and,depending on the results,one may go back
“03-Ch01-SA272” 17/9/2008 page 7
1.3 Supervised, Unsupervised, and Semi-Supervised Learning
patterns sensor
feature generation
feature selection
classifier design
system evaluation
FIGURE 1.3 The basic stages involved in the design of a classification system.
to redesign earlier stages in order to improve the overall performance. Furthermore, there are some methods that combine stages, for example, the feature selection and the classifier design stage, in a common optimization task. Although the reader has already been exposed to a number of basic problems at the heart of the design of a classification system, there are still a few things to be said.
1.3 SUPERVISED, UNSUPERVISED, AND SEMI-SUPERVISED LEARNING In the example of Figure 1.1, we assumed that a set of training data were available, and the classifier was designed by exploiting this a priori known information. This is known as supervised pattern recognition or in the more general context of machine learning as supervised learning. However, this is not always the case, and there is another type of pattern recognition tasks for which training data, of known class labels, are not available. In this type of problem, we are given a set of feature vectors x and the goal is to unravel the underlying similarities and cluster (group) “similar” vectors together. This is known as unsupervised pattern recognition or unsupervised learning or clustering. Such tasks arise in many applications in social sciences and engineering, such as remote sensing, image segmentation, and image and speech coding. Let us pick two such problems. In multispectral remote sensing, the electromagnetic energy emanating from the earth’s surface is measured by sensitive scanners located aboard a satellite, an aircraft,or a space station. This energy may be reflected solar energy (passive) or the reflected part of the energy transmitted from the vehicle (active) in order to “interrogate” the earth’s surface. The scanners are sensitive to a number of wavelength bands of the electromagnetic radiation. Different properties of the earth’s surface contribute to the reflection of the energy in the different bands. For example, in the visible–infrared range properties such as the mineral and moisture contents of soils, the sedimentation of water, and the moisture content of vegetation are the main contributors to the reflected energy. In contrast, at the thermal end of the infrared, it is the thermal capacity and thermal properties of the surface and near subsurface that contribute to the reflection. Thus, each band measures different properties
7
“03-Ch01-SA272” 17/9/2008 page 8
8
CHAPTER 1 Introduction
of the same patch of the earth’s surface. In this way, images of the earth’s surface corresponding to the spatial distribution of the reflected energy in each band can be created. The task now is to exploit this information in order to identify the various ground cover types, that is, built-up land, agricultural land, forest, fire burn, water, and diseased crop. To this end, one feature vector x for each cell from the “sensed” earth’s surface is formed. The elements xi , i ⫽ 1, 2, . . . , l, of the vector are the corresponding image pixel intensities in the various spectral bands. In practice, the number of spectral bands varies. A clustering algorithm can be employed to reveal the groups in which feature vectors are clustered in the l-dimensional feature space. Points that correspond to the same ground cover type, such as water, are expected to cluster together and form groups. Once this is done, the analyst can identify the type of each cluster by associating a sample of points in each group with available reference ground data, that is, maps or visits. Figure 1.4 demonstrates the procedure. Clustering is also widely used in the social sciences in order to study and correlate survey and statistical data and draw useful conclusions, which will then lead to the right actions. Let us again resort to a simplified example and assume that we are interested in studying whether there is any relation between a country’s gross national product (GNP) and the level of people’s illiteracy, on the one hand, and children’s mortality rate on the other. In this case, each country is represented by a three-dimensional feature vector whose coordinates are indices measuring the quantities of interest. A clustering algorithm will then reveal a rather compact cluster corresponding to countries that exhibit low GNPs, high illiteracy levels, and high children’s mortality expressed as a population percentage.
x2
forest
soil soil
vegetation
water
water
vegetation
forest
forest
soil (a)
(b)
x1
FIGURE 1.4 (a) An illustration of various types of ground cover and (b) clustering of the respective features for multispectral imaging using two bands.
“03-Ch01-SA272” 17/9/2008 page 9
1.5 MATLAB Programs
A major issue in unsupervised pattern recognition is that of defining the “similarity” between two feature vectors and choosing an appropriate measure for it. Another issue of importance is choosing an algorithmic scheme that will cluster (group) the vectors on the basis of the adopted similarity measure. In general, different algorithmic schemes may lead to different results, which the expert has to interpret. Semi-supervised learning/pattern recognition for designing a classification system shares the same goals as the supervised case, however now, the designer has at his or her disposal a set of patterns of unknown class origin, in addition to the training patterns, whose true class is known. We usually refer to the former ones as unlabeled and the latter as labeled data. Semi-supervised pattern recognition can be of importance when the system designer has access to a rather limited number of labeled data. In such cases, recovering additional information from the unlabeled samples, related to the general structure of the data at hand, can be useful in improving the system design. Semi-supervised learning finds its way also to clustering tasks. In this case, labeled data are used as constraints in the form of must-links and cannot-links. In other words, the clustering task is constrained to assign certain points in the same cluster or to exclude certain points of being assigned in the same cluster. From this perspective, semi-supervised learning provides an a priori knowledge that the clustering algorithm has to respect.
1.4 MATLAB PROGRAMS At the end of most of the chapters there is a number of MATLAB programs and computer experiments. The MATLAB codes provided are not intended to form part of a software package, but they are to serve a purely pedagogical goal. Most of these codes are given to our students who are asked to play with and discover the “secrets”associated with the corresponding methods. This is also the reason that for most of the cases the data used are simulated data around the Gaussian distribution. They have been produced carefully in order to guide the students in understanding the basic concepts. This is also the reason that the provided codes correspond to those of the techniques and algorithms that, to our opinion, comprise the backbone of each chapter and the student has to understand in a first reading. Whenever the required MATLAB code was available (at the time this book was prepared) in a MATLAB toolbox, we chose to use the associated MATLAB function and explain how to use its arguments. No doubt,each instructor has his or her own preferences, experiences,and unique way of viewing teaching. The provided routines are written in a way that can run on other data sets as well. In a separate accompanying book we provide a more complete list of MATLAB codes embedded in a user-friendly Graphical User Interface (GUI) and also involving more realistic examples using real images and audio signals.
9
“03-Ch01-SA272” 17/9/2008 page 10
10
CHAPTER 1 Introduction
1.5 OUTLINE OF THE BOOK Chapters 2–10 deal with supervised pattern recognition and Chapters 11–16 deal with the unsupervised case. Semi-supervised learning is introduced in Chapter 10. The goal of each chapter is to start with the basics, definitions, and approaches, and move progressively to more advanced issues and recent techniques. To what extent the various topics covered in the book will be presented in a first course on pattern recognition depends very much on the course’s focus, on the students’ background, and, of course, on the lecturer. In the following outline of the chapters, we give our view and the topics that we cover in a first course on pattern recognition. No doubt, other views do exist and may be better suited to different audiences. At the end of each chapter, a number of problems and computer exercises are provided. Chapter 2 is focused on Bayesian classification and techniques for estimating unknown probability density functions. In a first course on pattern recognition, the sections related to Bayesian inference, the maximum entropy, and the expectation maximization (EM) algorithm are omitted. Special focus is put on the Bayesian classification,the minimum distance (Euclidean and Mahalanobis),the nearest neighbor classifiers, and the naive Bayes classifier. Bayesian networks are briefly introduced. Chapter 3 deals with the design of linear classifiers. The sections dealing with the probability estimation property of the mean square solution as well as the bias variance dilemma are only briefly mentioned in our first course. The basic philosophy underlying the support vector machines can also be explained, although a deeper treatment requires mathematical tools (summarized in Appendix C) that most of the students are not familiar with during a first course class. On the contrary,emphasis is put on the linear separability issue, the perceptron algorithm, and the mean square and least squares solutions. After all, these topics have a much broader horizon and applicability. Support vector machines are briefly introduced. The geometric interpretation offers students a better understanding of the SVM theory. Chapter 4 deals with the design of nonlinear classifiers. The section dealing with exact classification is bypassed in a first course. The proof of the backpropagation algorithm is usually very boring for most of the students and we bypass its details. A description of its rationale is given, and the students experiment with it using MATLAB. The issues related to cost functions are bypassed. Pruning is discussed with an emphasis on generalization issues. Emphasis is also given to Cover’s theorem and radial basis function (RBF) networks. The nonlinear support vector machines, decision trees, and combining classifiers are only briefly touched via a discussion on the basic philosophy behind their rationale. Chapter 5 deals with the feature selection stage, and we have made an effort to present most of the well-known techniques. In a first course we put emphasis on the t-test. This is because hypothesis testing also has a broad horizon, and at the same time it is easy for the students to apply it in computer exercises. Then, depending on time constraints, divergence, Bhattacharrya distance, and scattered matrices are presented and commented on, although their more detailed treatment
“03-Ch01-SA272” 17/9/2008 page 11
1.5 Outline of The Book
is for a more advanced course. Emphasis is given to Fisher’s linear discriminant method ( LDA) for the two-class case. Chapter 6 deals with the feature generation stage using transformations. The Karhunen–Loève transform and the singular value decomposition are first introduced as dimensionality reduction techniques. Both methods are briefly covered in the second semester. In the sequel the independent component analysis (ICA),nonnegative matrix factorization and nonlinear dimensionality reduction techniques are presented. Then the discrete Fourier transform (DFT), discrete cosine transform (DCT), discrete sine transform (DST), Hadamard, and Haar transforms are defined. The rest of the chapter focuses on the discrete time wavelet transform. The incentive is to give all the necessary information so that a newcomer in the wavelet field can grasp the basics and be able to develop software, based on filter banks, in order to generate features. All these techniques are bypassed in a first course. Chapter 7 deals with feature generation focused on image and audio classification. The sections concerning local linear transforms, moments, parametric models, and fractals are not covered in a first course. Emphasis is placed on first- and secondorder statistics features as well as the run-length method. The chain code for shape description is also taught. Computer exercises are then offered to generate these features and use them for classification for some case studies. In a one-semester course there is no time to cover more topics. Chapter 8 deals with template matching. Dynamic programming (DP) and the Viterbi algorithm are presented and then applied to speech recognition. In a two-semester course, emphasis is given to the DP and the Viterbi algorithm. The edit distance seems to be a good case for the students to grasp the basics. Correlation matching is taught and the basic philosophy behind deformable template matching can also be presented. Chapter 9 deals with context-dependent classification. Hidden Markov models are introduced and applied to communications and speech recognition. This chapter is bypassed in a first course. Chapter 10 deals with system evaluation and semi-supervised learning. The various error rate estimation techniques are discussed, and a case study with real data is treated. The leave-one-out method and the resubstitution methods are emphasized in the second semester,and students practice with computer exercises. Semi-supervised learning is bypassed in a first course. Chapter 11 deals with the basic concepts of clustering. It focuses on definitions as well as on the major stages involved in a clustering task. The various types of data encountered in clustering applications are reviewed, and the most commonly used proximity measures are provided. In a first course, only the most widely used proximity measures are covered (e.g., lp norms, inner product, Hamming distance). Chapter 12 deals with sequential clustering algorithms. These include some of the simplest clustering schemes, and they are well suited for a first course to introduce students to the basics of clustering and allow them to experiment with
11
“03-Ch01-SA272” 17/9/2008 page 12
12
CHAPTER 1 Introduction
the computer. The sections related to estimation of the number of clusters and neural network implementations are bypassed. Chapter 13 deals with hierarchical clustering algorithms. In a first course, only the general agglomerative scheme is considered with an emphasis on single link and complete link algorithms, based on matrix theory. Agglomerative algorithms based on graph theory concepts as well as the divisive schemes are bypassed. Chapter 14 deals with clustering algorithms based on cost function optimization, using tools from differential calculus. Hard clustering and fuzzy and possibilistic schemes are considered,based on various types of cluster representatives,including point representatives, hyperplane representatives, and shell-shaped representatives. In a first course, most of these algorithms are bypassed, and emphasis is given to the isodata algorithm. Chapter 15 features a high degree of modularity. It deals with clustering algorithms based on different ideas,which cannot be grouped under a single philosophy. Spectral clustering, competitive learning, branch and bound, simulated annealing, and genetic algorithms are some of the schemes treated in this chapter. These are bypassed in a first course. Chapter 16 deals with the clustering validity stage of a clustering procedure. It contains rather advanced concepts and is omitted in a first course. Emphasis is given to the definitions of internal, external, and relative criteria and the random hypotheses used in each case. Indices, adopted in the framework of external and internal criteria, are presented, and examples are provided showing the use of these indices. Syntactic pattern recognition methods are not treated in this book. Syntactic pattern recognition methods differ in philosophy from the methods discussed in this book and, in general, are applicable to different types of problems. In syntactic pattern recognition, the structure of the patterns is of paramount importance, and pattern recognition is performed on the basis of a set of pattern primitives, a set of rules in the form of a grammar, and a recognizer called automaton. Thus, we were faced with a dilemma: either to increase the size of the book substantially, or to provide a short overview (which, however, exists in a number of other books), or to omit it. The last option seemed to be the most sensible choice.
“04-Ch02-SA272” 18/9/2008 page 13
CHAPTER
Classifiers Based on Bayes Decision Theory
2
2.1 INTRODUCTION This is the first chapter, out of three, dealing with the design of the classifier in a pattern recognition system. The approach to be followed builds upon probabilistic arguments stemming from the statistical nature of the generated features. As has already been pointed out in the introductory chapter, this is due to the statistical variation of the patterns as well as to the noise in the measuring sensors. Adopting this reasoning as our kickoff point,we will design classifiers that classify an unknown pattern in the most probable of the classes. Thus, our task now becomes that of defining what “most probable” means. Given a classification task of M classes,1 , 2 , . . . , M ,and an unknown pattern, which is represented by a feature vector x, we form the M conditional probabilities P(i |x), i ⫽ 1, 2, . . . , M. Sometimes, these are also referred to as a posteriori probabilities. In words, each of them represents the probability that the unknown pattern belongs to the respective class i , given that the corresponding feature vector takes the value x. Who could then argue that these conditional probabilities are not sensible choices to quantify the term most probable? Indeed, the classifiers to be considered in this chapter compute either the maximum of these M values or, equivalently, the maximum of an appropriately defined function of them. The unknown pattern is then assigned to the class corresponding to this maximum. The first task we are faced with is the computation of the conditional probabilities. The Bayes rule will once more prove its usefulness! A major effort in this chapter will be devoted to techniques for estimating probability density functions (pdf ), based on the available experimental evidence, that is, the feature vectors corresponding to the patterns of the training set.
2.2 BAYES DECISION THEORY We will initially focus on the two-class case. Let 1 , 2 be the two classes in which our patterns belong. In the sequel, we assume that the a priori probabilities
13
“04-Ch02-SA272” 18/9/2008 page 14
14
CHAPTER 2 Classifiers Based on Bayes Decision Theory
P(1 ), P(2 ) are known. This is a very reasonable assumption, because even if they are not known, they can easily be estimated from the available training feature vectors. Indeed, if N is the total number of available training patterns, and N1 , N2 of them belong to 1 and 2 , respectively, then P(1 ) ≈ N1 /N and P(2 ) ≈ N2 /N . The other statistical quantities assumed to be known are the class-conditional probability density functions p(x|i ), i ⫽ 1, 2, describing the distribution of the feature vectors in each of the classes. If these are not known, they can also be estimated from the available training data, as we will discuss later on in this chapter. The pdf p(x|i ) is sometimes referred to as the likelihood function of i with respect to x. Here we should stress the fact that an implicit assumption has been made. That is, the feature vectors can take any value in the l-dimensional feature space. In the case that feature vectors can take only discrete values,density functions p(x|i ) become probabilities and will be denoted by P(x|i ). We now have all the ingredients to compute our conditional probabilities, as stated in the introduction. To this end, let us recall from our probability course basics the Bayes rule (Appendix A) P(i |x) ⫽
p(x|i )P(i ) p(x)
(2.1)
where p(x) is the pdf of x and for which we have (Appendix A) p(x) ⫽
2
p(x|i )P(i )
(2.2)
i⫽1
The Bayes classification rule can now be stated as If If
P(1 |x) ⬎ P(2 |x), P(1 |x) ⬍ P(2 |x),
x is classified to 1 x is classified to 2
(2.3)
The case of equality is detrimental and the pattern can be assigned to either of the two classes. Using (2.1), the decision can equivalently be based on the inequalities p(x|1 )P(1 ) ≷ p(x|2 )P(2 )
(2.4)
p(x) is not taken into account, because it is the same for all classes and it does not affect the decision. Furthermore, if the a priori probabilities are equal, that is, P(1 ) ⫽ P(2 ) ⫽ 1/2, Eq. (2.4) becomes p(x|1 ) ≷ p(x|2 )
(2.5)
Thus, the search for the maximum now rests on the values of the conditional pdfs evaluated at x. Figure 2.1 presents an example of two equiprobable classes and shows the variations of p(x|i ), i ⫽ 1, 2, as functions of x for the simple case of a single feature (l ⫽ 1). The dotted line at x0 is a threshold partitioning the feature space into two regions,R1 and R2 . According to the Bayes decision rule,for all values of x in R1 the classifier decides 1 and for all values in R2 it decides 2 . However, it is obvious from the figure that decision errors are unavoidable. Indeed, there is
“04-Ch02-SA272” 18/9/2008 page 15
2.2 Bayes Decision Theory
p(x|1)
p(x|)
p(x|2)
x0
x R2
R1
FIGURE 2.1 Example of the two regions R1 and R2 formed by the Bayesian classifier for the case of two equiprobable classes.
a finite probability for an x to lie in the R2 region and at the same time to belong in class 1 . Then our decision is in error. The same is true for points originating from class 2 . It does not take much thought to see that the total probability, Pe , of committing a decision error for the case of two equiprobable classes, is given by Pe ⫽
1 2
x0 p(x|2 ) dx ⫹ ⫺⬁
1 2
⫹⬁ p(x|1 ) dx
(2.6)
x0
which is equal to the total shaded area under the curves in Figure 2.1. We have now touched on a very important issue. Our starting point to arrive at the Bayes classification rule was rather empirical, via our interpretation of the term most probable. We will now see that this classification test, though simple in its formulation, has a sounder mathematical interpretation.
Minimizing the Classification Error Probability We will show that the Bayesian classifier is optimal with respect to minimizing the classification error probability. Indeed, the reader can easily verify, as an exercise, that moving the threshold away from x0 , in Figure 2.1, always increases the corresponding shaded area under the curves. Let us now proceed with a more formal proof. Proof. Let R1 be the region of the feature space in which we decide in favor of 1 and R2 be the corresponding region for 2 . Then an error is made if x ∈ R1 , although it belongs to 2 or if x ∈ R2 , although it belongs to 1 . That is, Pe ⫽ P(x ∈ R2 , 1 ) ⫹ P(x ∈ R1 , 2 )
(2.7)
15
“04-Ch02-SA272” 18/9/2008 page 16
16
CHAPTER 2 Classifiers Based on Bayes Decision Theory
where P(·, ·) is the joint probability of two events. Recalling, once more, our probability basics (Appendix A), this becomes Pe ⫽ P(x ∈ R2 |1 )P(1 ) ⫹ P(x ∈ R1 |2 )P(2 ) ⫽ P(1 ) p(x|1 ) dx ⫹ P(2 ) p(x|2 ) dx R2
or using the Bayes rule
Pe ⫽
(2.8)
R1
P(1 |x)p(x) dx ⫹
R2
P(2 |x)p(x) dx
(2.9)
R1
It is now easy to see that the error is minimized if the partitioning regions R1 and R2 of the feature space are chosen so that R1 : P(1 |x) ⬎ P(2 |x) R2 : P(2 |x) ⬎ P(1 |x)
(2.10)
Indeed,since the union of the regions R1 , R2 covers all the space,from the definition of a probability density function we have that
P(1 |x)p(x) dx ⫹ R1
P(1 |x)p(x) dx ⫽ P(1 )
(2.11)
R2
Combining Eqs. (2.9) and (2.11), we get
(P(1 |x) ⫺ P(2 |x)) p(x) dx
Pe ⫽ P(1 ) ⫺
(2.12)
R1
This suggests that the probability of error is minimized if R1 is the region of space in which P(1 |x) ⬎ P(2 |x). Then, R2 becomes the region where the reverse is true. So far, we have dealt with the simple case of two classes. Generalizations to the multiclass case are straightforward. In a classification task with M classes, 1 , 2 , . . . , M ,an unknown pattern,represented by the feature vector x,is assigned to class i if P(i |x) ⬎ P(j |x)
᭙j ⫽ i
(2.13)
It turns out that such a choice also minimizes the classification error probability ( Problem 2.1).
Minimizing the Average Risk The classification error probability is not always the best criterion to be adopted for minimization. This is because it assigns the same importance to all errors. However, there are cases in which some wrong decisions may have more serious implications than others. For example, it is much more serious for a doctor to make a wrong decision and a malignant tumor to be diagnosed as a benign one, than the other way round. If a benign tumor is diagnosed as a malignant one, the wrong decision will be cleared out during subsequent clinical examinations. However, the results
“04-Ch02-SA272” 18/9/2008 page 17
2.2 Bayes Decision Theory
from the wrong decision concerning a malignant tumor may be fatal. Thus, in such cases it is more appropriate to assign a penalty term to weigh each error. For our example, let us denote by 1 the class of malignant tumors and as 2 the class of the benign ones. Let, also, R1 , R2 be the regions in the feature space where we decide in favor of 1 and 2 , respectively. The error probability Pe is given by Eq. (2.8). Instead of selecting R1 and R2 so that Pe is minimized, we will now try to minimize a modified version of it, that is,
r ⫽ 12 P(1 )
p(x|1 )dx ⫹ 21 P(2 )
R2
p(x|2 )dx
(2.14)
R1
where each of the two terms that contributes to the overall error probability is weighted according to its significance. For our case, the reasonable choice would be to have 12 ⬎ 21 . Thus errors due to the assignment of patterns originating from class 1 to class 2 will have a larger effect on the cost function than the errors associated with the second term in the summation. Let us now consider an M-class problem and let Rj , j ⫽ 1, 2, . . . , M,be the regions of the feature space assigned to classes j , respectively. Assume now that a feature vector x that belongs to class k lies in Ri , i ⫽ k. Then this vector is misclassified in i and an error is committed. A penalty term ki , known as loss, is associated with this wrong decision. The matrix L, which has at its (k, i) location the corresponding penalty term, is known as the loss matrix.1 Observe that in contrast to the philosophy behind Eq. (2.14), we have now allowed weights across the diagonal of the loss matrix (kk ), which correspond to correct decisions. In practice, these are usually set equal to zero, although we have considered them here for the sake of generality. The risk or loss associated with k is defined as rk ⫽
M
ki
i⫽1
p(x|k ) dx
(2.15)
Ri
Observe that the integral is the overall probability of a feature vector from class k being classified in i . This probability is weighted by ki . Our goal now is to choose the partitioning regions Rj so that the average risk r⫽
M
rk P(k )
k⫽1
⫽
M i⫽1 R
i
M
ki p(x|k )P(k ) dx
(2.16)
k⫽1
is minimized. This is achieved if each of the integrals is minimized, which is equivalent to selecting partitioning regions so that x ∈ Ri
if
li ≡
M k⫽1
1 The
ki p(x|k )P(k ) ⬍ lj ≡
M k⫽1
terminology comes from the general decision theory.
kj p(x|k )P(k )
᭙j ⫽ i
(2.17)
17
“04-Ch02-SA272” 18/9/2008 page 18
18
CHAPTER 2 Classifiers Based on Bayes Decision Theory
It is obvious that if ki ⫽ 1 ⫺ ␦ki , where ␦ki is Kronecker’s delta (0 if k ⫽ i and 1 if k ⫽ i ), then minimizing the average risk becomes equivalent to minimizing the classification error probability. The two-class case. For this specific case we obtain l1 ⫽ 11 p(x|1 )P(1 ) ⫹ 21 p(x|2 )P(2 ) l2 ⫽ 12 p(x|1 )P(1 ) ⫹ 22 p(x|2 )P(2 )
(2.18)
We assign x to 1 if l1 ⬍ l2 , that is, (21 ⫺ 22 )p(x|2 )P(2 ) ⬍ (12 ⫺ 11 )p(x|1 )P(1 )
(2.19)
It is natural to assume that ij ⬎ ii (correct decisions are penalized much less than wrong ones). Adopting this assumption, the decision rule (2.17) for the two-class case now becomes x ∈ 1 (2 )
if
l12 ≡
P(2 ) 21 ⫺ 22 p(x|1 ) ⬎ (⬍) p(x|2 ) P(1 ) 12 ⫺ 11
(2.20)
The ratio l12 is known as the likelihood ratio and the preceding test as the likelihood ratio test. Let us now investigate Eq. (2.20) a little further and consider the case of Figure 2.1. Assume that the loss matrix is of the form 0 L⫽ 21
12 0
If misclassification of patterns that come from 2 is considered to have serious consequences, then we must choose 21 ⬎ 12 . Thus, patterns are assigned to class 2 if p(x|2 ) ⬎ p(x|1 )
12 21
where P(1 ) ⫽ P(2 ) ⫽ 1/2 has been assumed. That is, p(x|1 ) is multiplied by a factor less than 1 and the effect of this is to move the threshold in Figure 2.1 to the left of x0 . In other words, region R2 is increased while R1 is decreased. The opposite would be true if 21 ⬍ 12 . An alternative cost that sometimes is used for two class problems is the NeymanPearson criterion. The error for one of the classes is now constrained to be fixed and equal to a chosen value ( Problem 2.6). Such a decision rule has been used, for example, in radar detection problems. The task there is to detect a target in the presence of noise. One type of error is the so-called false alarm—that is, to mistake the noise for a signal (target) present. Of course, the other type of error is to miss the signal and to decide in favor of the noise (missed detection). In many cases the error probability of false alarm is set equal to a predetermined threshold.
“04-Ch02-SA272” 18/9/2008 page 19
2.3 Discriminant Functions and Decision Surfaces
Example 2.1 In a two-class problem with a single feature x the pdfs are Gaussians with variance 2 ⫽ 1/2 for both classes and mean values 0 and 1, respectively, that is, 1 p(x|1 ) ⫽ √ exp(⫺x 2 ) 1 p(x|2 ) ⫽ √ exp(⫺(x ⫺ 1)2 ) If P(1 ) ⫽ P(2 ) ⫽ 1/2, compute the threshold value x0 (a) for minimum error probability and (b) for minimum risk if the loss matrix is 0 0.5 L⫽ 1.0 0 Taking into account the shape of the Gaussian function graph (Appendix A), the threshold for the minimum probability case will be x0 : exp(⫺x 2 ) ⫽ exp(⫺(x ⫺ 1)2 ) Taking the logarithm of both sides, we end up with x0 ⫽ 1/2. In the minimum risk case we get x0 : exp(⫺x 2 ) ⫽ 2 exp(⫺(x ⫺ 1)2 ) or x0 ⫽ (1 ⫺ ln 2)/2 ⬍ 1/2; that is, the threshold moves to the left of 1/2. If the two classes are not equiprobable, then it is easily verified that if P(1 ) ⬎ (⬍) P(2 ) the threshold moves to the right (left). That is, we expand the region in which we decide in favor of the most probable class, since it is better to make fewer errors for the most probable class.
2.3 DISCRIMINANT FUNCTIONS AND DECISION SURFACES It is by now clear that minimizing either the risk or the error probability or the Neyman-Pearson criterion is equivalent to partitioning the feature space into M regions, for a task with M classes. If regions Ri , Rj happen to be contiguous, then they are separated by a decision surface in the multidimensional feature space. For the minimum error probability case, this is described by the equation P(i |x) ⫺ P(j |x) ⫽ 0
(2.21)
From the one side of the surface this difference is positive, and from the other it is negative. Sometimes, instead of working directly with probabilities (or risk functions), it may be more convenient, from a mathematical point of view, to work with an equivalent function of them, for example, gi (x) ≡ f (P(i |x)), where f (·) is a monotonically increasing function. gi (x) is known as a discriminant function. The decision test (2.13) is now stated as classify x in i
if
gi (x) ⬎ gj (x)
᭙j ⫽ i
(2.22)
The decision surfaces, separating contiguous regions, are described by gij (x) ≡ gi (x) ⫺ gj (x) ⫽ 0,
i, j ⫽ 1, 2, . . . , M,
i ⫽ j
(2.23)
19
“04-Ch02-SA272” 18/9/2008 page 20
20
CHAPTER 2 Classifiers Based on Bayes Decision Theory
So far,we have approached the classification problem via Bayesian probabilistic arguments and the goal was to minimize the classification error probability or the risk. However, as we will soon see, not all problems are well suited to such approaches. For example, in many cases the involved pdfs are complicated and their estimation is not an easy task. In such cases, it may be preferable to compute decision surfaces directly by means of alternative costs, and this will be our focus in Chapters 3 and 4. Such approaches give rise to discriminant functions and decision surfaces, which are entities with no (necessary) relation to Bayesian classification, and they are, in general, suboptimal with respect to Bayesian classifiers. In the following we will focus on a particular family of decision surfaces associated with the Bayesian classification for the specific case of Gaussian density functions.
2.4 BAYESIAN CLASSIFICATION FOR NORMAL DISTRIBUTIONS 2.4.1 The Gaussian Probability Density Function One of the most commonly encountered probability density functions in practice is the Gaussian or normal probability density function. The major reasons for its popularity are its computational tractability and the fact that it models adequately a large number of cases. One of the most celebrated theorems in statistics is the central limit theorem. The theorem states that if a random variable is the outcome of a summation of a number of independent random variables, its pdf approaches the Gaussian function as the number of summands tends to infinity (see Appendix A). In practice,it is most common to assume that the sum of random variables is distributed according to a Gaussian pdf, for a sufficiently large number of summing terms. The one-dimensional or the univariate Gaussian, as it is sometimes called, is defined by p(x) ⫽ √
(x ⫺ )2 exp ⫺ 2 2 2 1
(2.24)
The parameters and 2 turn out to have a specific meaning. The mean value of the random variable x is equal to , that is, ⫹⬁ ⫽ E[x] ≡ xp(x)dx
(2.25)
⫺⬁
where E[·] denotes the mean (or expected) value of a random variable. The parameter 2 is equal to the variance of x, that is, ⫹⬁ ⫽ E[(x ⫺ ) ] ≡ (x ⫺ )2 p(x)dx 2
2
⫺⬁
(2.26)
“04-Ch02-SA272” 18/9/2008 page 21
2.4 Bayesian Classification for Normal Distributions
p(x)
p(x)
1
1
x
0 (a)
0 1 ( b)
x
FIGURE 2.2 Graphs for the one-dimensional Gaussian pdf. (a) Mean value ⫽ 0, 2 ⫽ 1, (b) ⫽ 1 and 2 ⫽ 0.2. The larger the variance the broader the graph is. The graphs are symmetric, and they are centered at the respective mean value.
Figure 2.2a shows the graph of the Gaussian function for ⫽ 0 and 2 ⫽ 1, and Figure 2.2b the case for ⫽ 1 and 2 ⫽ 0.2. The larger the variance the broader the graph, which is symmetric, and it is always centered at (see Appendix A, for some more properties). The multivariate generalization of a Gaussian pdf in the l-dimensional space is given by p(x) ⫽
1 (2)l/2 |⌺|1/2
1 exp ⫺ (x ⫺ )T ⌺⫺1 (x ⫺ ) 2
(2.27)
where ⫽ E[x] is the mean value and ⌺ is the l ⫻ l covariance matrix (Appendix A) defined as ⌺ ⫽ E[(x ⫺ )(x ⫺ )T ]
(2.28)
where |⌺| denotes the determinant of ⌺. It is readily seen that for l ⫽ 1 the multivariate Gaussian coincides with the univariate one. Sometimes, the symbol N (, ⌺) is used to denote a Gaussian pdf with mean value and covariance ⌺. To get a better feeling on what the multivariate Gaussian looks like, let us focus on some cases in the two-dimensional space, where nature allows us the luxury of visualization. For this case we have
x1 ⫺ 1
⌺⫽E x1 ⫺ 1 , x2 ⫺ 2 x2 ⫺ 2 12 12 ⫽ 12 22
(2.29)
(2.30)
where E[xi ] ⫽ i , i ⫽ 1, 2, and by definition 12 ⫽ E[(x1 ⫺ 1 )(x2 ⫺ 2 )], which is known as the covariance between the random variables x1 and x2 and it is a measure
21
“04-Ch02-SA272” 18/9/2008 page 22
22
CHAPTER 2 Classifiers Based on Bayes Decision Theory
of their mutual statistical correlation. If the variables are statistically independent, their covariance is zero (Appendix A). Obviously, the diagonal elements of ⌺ are the variances of the respective elements of the random vector. Figures 2.3–2.6 show the graphs for four instances of a two-dimensional Gaussian probability density function. Figure 2.3a corresponds to a Gaussian with a diagonal covariance matrix
⌺⫽
3 0
0 3
p(x) x2
x1 x1 x2 (a)
(b)
FIGURE 2.3 (a) The graph of a two-dimensional Gaussian pdf and (b) the corresponding isovalue curves for a diagonal ⌺ with 12 ⫽ 22 . The graph has a spherical symmetry showing no preference in any direction.
p(x) x2
x1 x1 x2 (a)
(b)
FIGURE 2.4 (a) The graph of a two-dimensional Gaussian pdf and (b) the corresponding isovalue curves for a diagonal ⌺ with 12 ⬎⬎ 22 . The graph is elongated along the x1 direction.
“04-Ch02-SA272” 18/9/2008 page 23
2.4 Bayesian Classification for Normal Distributions
p(x) x2
x1 x1 x2 (a)
(b)
FIGURE 2.5 (a) The graph of a two-dimensional Gaussian pdf and (b) the corresponding isovalue curves for a diagonal ⌺ with 12 ⬍⬍ 22 . The graph is elongated along the x2 direction. p(x) x2
x1 x1 x2 (a)
(b)
FIGURE 2.6 (a) The graph of a two-dimensional Gaussian pdf and (b) the corresponding isovalue curves for a case of a nondiagonal ⌺. Playing with the values of the elements of ⌺ one can achieve different shapes and orientations.
that is, both features, x1 , x2 have variance equal to 3 and their covariance is zero. The graph of the Gaussian is symmetric. For this case the isovalue curves (i.e., curves of equal probability density values) are circles (hyperspheres in the general l-dimensional space) and are shown in Figure 2.3b. The case shown in Figure 2.4a corresponds to the covariance matrix
⌺⫽
12 0
0 22
with 12 ⫽ 15 ⬎⬎ 22 ⫽ 3. The graph of the Gaussian is now elongated along the x1 -axis, which is the direction of the larger variance. The isovalue curves, shown
23
“04-Ch02-SA272” 18/9/2008 page 24
24
CHAPTER 2 Classifiers Based on Bayes Decision Theory
in Figure 2.4b, are ellipses. Figures 2.5a and 2.5b correspond to the case with 12 ⫽ 3 ⬍⬍ 22 ⫽ 15. Figures 2.6a and 2.6b correspond to the more general case where
12 ⌺⫽ 12
12 22
and 12 ⫽ 15, 22 ⫽ 3, 12 ⫽ 6. Playing with 12 , 22 and 12 one can achieve different shapes and different orientations. The isovalue curves are ellipses of different orientations and with different ratios of major to minor axis lengths. Let us consider, as an example, the case of a zero mean random vector with a diagonal covariance matrix. To compute the isovalue curves is equivalent to computing the curves of constant values for the exponent, that is, ⎡ ⫺1
x ⌺ T
x ⫽ [x1 , x2 ]
1 2 ⎣ 1
0
⎤ ⎦ x1 ⫽ C 1 x2 2
0
(2.31)
2
or x12 x22 ⫹ ⫽C 12 22
(2.32)
for some constant C. This is the equation of an ellipse whose axes are determined by the the variances of the involved features. As we will soon see, the principal axes of the ellipses are controlled by the eigenvectors/eigenvalues of the covariance matrix. As we know from linear algebra (and it is easily checked), the eigenvalues of a diagonal matrix, which was the case for our example, are equal to the respective elements across its diagonal.
2.4.2 The Bayesian Classifier for Normally Distributed Classes Our goal in this section is to study the optimal Bayesian classifier when the involved pdfs, p(x|i ), i ⫽ 1, 2, . . . , M (likelihood functions of i with respect to x), describing the data distribution in each one of the classes, are multivariate normal distributions, that is, N (i , ⌺i ), i ⫽ 1, 2, . . . , M. Because of the exponential form of the involved densities, it is preferable to work with the following discriminant functions, which involve the (monotonic) logarithmic function ln(·): gi (x) ⫽ ln( p(x|i )P(i )) ⫽ ln p(x|i ) ⫹ ln P(i )
(2.33)
1 gi (x) ⫽ ⫺ (x ⫺ i )T ⌺⫺1 i (x ⫺ i ) ⫹ ln P(i ) ⫹ ci 2
(2.34)
or
where ci is a constant equal to ⫺(l/2) ln 2 ⫺ (1/2) ln|⌺i |. Expanding, we obtain gi (x) ⫽ ⫺
1 T ⫺1 1 1 T ⫺1 1 T ⫺1 x ⌺i x ⫹ x T ⌺⫺1 i i ⫺ i ⌺i i ⫹ i ⌺i x 2 2 2 2
⫹ ln P(i ) ⫹ ci (2.35)
“04-Ch02-SA272” 18/9/2008 page 25
2.4 Bayesian Classification for Normal Distributions
In general, this is a nonlinear quadratic form. Take, for example, the case of l ⫽ 2 and assume that
⌺i ⫽
i2
0
0
i2
Then (2.35) becomes gi (x) ⫽ ⫺
1 1 2 1 2 2 x ⫹ x22 ⫹ 2 (i1 x1 ⫹ i2 x2 ) ⫺ 2 i1 ⫹ ln P(i ) ⫹ ci ⫹ i2 2i2 1 i 2i
(2.36)
and obviously the associated decision curves gi (x) ⫺ gj (x) ⫽ 0 are quadrics (i.e., ellipsoids, parabolas, hyperbolas, pairs of lines). That is, in such cases, the Bayesian classifier is a quadratic classifier, in the sense that the partition of the feature space is performed via quadric decision surfaces. For l ⬎ 2 the decision surfaces are hyperquadrics. Figure 2.7a shows the decision curve corresponding to P(1 ) ⫽ P(2 ), 1 ⫽ [0, 0]T and 2 ⫽ [4, 0]T . The covariance matrices for the two classes are
0.3 ⌺1 ⫽ 0.0
0.0 , 0.35
1.2 ⌺2 ⫽ 0.0
0.0 1.85
For the case of Figure 2.7b the classes are also equiprobable with 1 ⫽ [0, 0]T , 2 ⫽ [3.2, 0]T and covariance matrices
0.1 ⌺1 ⫽ 0.0
0.0 , 0.75
0.75 ⌺2 ⫽ 0.0
0.0 0.1
Figure 2.8 shows the two pdfs for the case of Figure 2.7a. The red color is used for class 1 and indicates the points where p(x|1 ) ⬎ p(x|2 ). The gray color is similarly used for class 2 . It is readily observed that the decision curve is an ellipse, as shown in Figure 2.7a. The setup corresponding to Figure 2.7b is shown in Figure 2.9. In this case, the decision curve is a hyperbola. x2 3
x2 4
2 1
0
1
1
1
⫺2
2
⫺3 ⫺3
⫺2
⫺1 (a)
0
x1
⫺5 ⫺10
⫺5
0
5 x1
(b)
FIGURE 2.7 Examples of quadric decision curves. Playing with the covariance matrices of the Gaussian functions, different decision curves result, that is, ellipsoids, parabolas, hyperbolas, pairs of lines.
25
“04-Ch02-SA272” 18/9/2008 page 26
26
CHAPTER 2 Classifiers Based on Bayes Decision Theory
0.25
0.2
0.15
0.1
0.05
0 8 6 4 2 0 22 24 26 28 28
26
24
22
0
2
4
6
8
FIGURE 2.8 An example of the pdfs of two equiprobable classes in the two-dimensional space. The feature vectors in both classes are normally distributed with different covariance matrices. In this case, the decision curve is an ellipse and it is shown in Figure 2.7a. The coloring indicates the areas where the value of the respective pdf is larger.
Decision Hyperplanes The only quadratic contribution in (2.35) comes from the term x T ⌺⫺1 i x. If we now assume that the covariance matrix is the same in all classes, that is, ⌺i ⫽ ⌺, the quadratic term will be the same in all discriminant functions. Hence, it does not enter into the comparisons for computing the maximum, and it cancels out in the decision surface equations. The same is true for the constants ci . Thus, they can be omitted and we may redefine gi (x) as gi (x) ⫽ w Ti x ⫹ wi0
(2.37)
“04-Ch02-SA272” 18/9/2008 page 27
2.4 Bayesian Classification for Normal Distributions
FIGURE 2.9 An example of the pdfs of two equiprobable classes in the two-dimensional space. The feature vectors in both classes are normally distributed with different covariance matrices. In this case, the decision curve is a hyperbola and it is shown in Figure 2.7b.
where w i ⫽ ⌺⫺1 i
(2.38)
and wi0 ⫽ ln P(i ) ⫺
1 T ⫺1 ⌺ i 2 i
(2.39)
Hence gi (x) is a linear function of x and the respective decision surfaces are hyperplanes. Let us investigate this a bit more.
27
“04-Ch02-SA272” 18/9/2008 page 28
28
CHAPTER 2 Classifiers Based on Bayes Decision Theory
■
Diagonal covariance matrix with equal elements: Assume that the individual features, constituting the feature vector, are mutually uncorrelated and of the same variance (E[(xi ⫺ i )(xj ⫺ j )] ⫽ 2 ␦ij ). Then, as discussed in Appendix A, ⌺ ⫽ 2 I , where I is the l-dimensional identity matrix, and (2.37) becomes gi (x) ⫽
1 T x ⫹ wi0 2 i
(2.40)
Thus, the corresponding decision hyperplanes can now be written as (verify it) gij (x) ≡ gi (x) ⫺ gj (x) ⫽ w T (x ⫺ x 0 ) ⫽ 0
(2.41)
w ⫽ i ⫺ j
(2.42)
where
and x0 ⫽
i ⫺ j 1 P(i ) (i ⫹ j ) ⫺ 2 ln 2 P(j ) i ⫺ j 2
(2.43)
where x ⫽ x12 ⫹ x22 ⫹ · · · ⫹ xl2 denotes the Euclidean norm of x. Thus, the decision surface is a hyperplane passing through the point x 0 . Obviously, if P(i ) ⫽ P(j ), then x 0 ⫽ 12 (i ⫹ j ), and the hyperplane passes through the average of i , j , that is, the middle point of the segment joining the mean values. On the other hand, if P(j ) ⬎ P(i ) (P(i ) ⬎ P(j )) the hyperplane is located closer to i (j ). In other words, the area of the region where we decide in favor of the more probable of the two classes is increased. The geometry is illustrated in Figure 2.10 for the two-dimensional case and for two cases, that is, P(j ) ⫽ P(i ) (black line) and P(j ) ⬎ P(i ) (red line). We observe that for both cases the decision hyperplane (straight line) is orthogonal to i ⫺j . Indeed,for any point x lying on the decision hyperplane, the vector x ⫺ x 0 also lies on the hyperplane and gij (x) ⫽ 0 ⇒ w T (x ⫺ x 0 ) ⫽ (i ⫺ j )T (x ⫺ x 0 ) ⫽ 0 That is, i ⫺ j is orthogonal to the decision hyperplane. Furthermore, if 2 is small with respect to i ⫺ j , the location of the hyperplane is rather insensitive to the values of P(i ), P(j ). This is expected, because small variance indicates that the random vectors are clustered within a small radius around their mean values. Thus a small shift of the decision hyperplane has a small effect on the result. Figure 2.11 illustrates this. For each class, the circles around the means indicate regions where samples have a high probability, say 98%,
“04-Ch02-SA272” 18/9/2008 page 29
2.4 Bayesian Classification for Normal Distributions
x2
i ⴚ j
i
j
x0
x1
FIGURE 2.10 Decision lines for normally distributed vectors with ⌺ ⫽ 2 I . The black line corresponds to the case of P(j ) ⫽ P(i ) and it passes through the middle point of the line segment joining the mean values of the two classes. The red line corresponds to the case of P(j ) ⬎ P(i ) and it is closer to i , leaving more “room” to the more probable of the two classes. If we had assumed P(j ) ⬍ P(i ), the decision line would have moved closer to j .
x2
x2
i
j
i
(a)
x1
j
(b)
x1
FIGURE 2.11 Decision line (a) for compact and (b) for noncompact classes. When classes are compact around their mean values, the location of the hyperplane is rather insensitive to the values of P(1 ) and P(2 ). This is not the case for noncompact classes, where a small movement of the hyperplane to the right or to the left may be more critical.
of being found. The case of Figure 2.11a corresponds to small variance, and that of Figure 2.11b to large variance. No doubt the location of the decision hyperplane in Figure 2.11b is much more critical than that in Figure 2.11a.
29
“04-Ch02-SA272” 18/9/2008 page 30
30
CHAPTER 2 Classifiers Based on Bayes Decision Theory
■
Nondiagonal covariance matrix: Following algebraic arguments similar to those used before, we end up with hyperplanes described by gij (x) ⫽ w T (x ⫺ x 0 ) ⫽ 0
(2.44)
w ⫽ ⌺⫺1 (i ⫺ j )
(2.45)
where and x0 ⫽
i ⫺ j 1 P(i ) (i ⫹ j ) ⫺ ln 2 P(j ) i ⫺ j 2 ⫺1
(2.46)
⌺
where x⌺⫺1 ≡ (x T ⌺⫺1 x)1/2 denotes the so-called ⌺⫺1 norm of x. The comments made before for the case of the diagonal covariance matrix are still valid, with one exception. The decision hyperplane is no longer orthogonal to the vector i ⫺ j but to its linear transformation ⌺⫺1 (i ⫺ j ). Figure 2.12 shows two Gaussian pdfs with equal covariance matrices, describing the data distribution of two equiprobable classes. In both classes, the data are distributed around their mean values in exactly the same way and the optimal decision curve is a straight line.
Minimum Distance Classifiers We will now view the task from a slightly different angle. Assuming equiprobable classes with the same covariance matrix, gi (x) in (2.34) is simplified to 1 gi (x) ⫽ ⫺ (x ⫺ i )T ⌺⫺1 (x ⫺ i ) 2
(2.47)
where constants have been neglected. ■
⌺ ⫽ 2 I : In this case maximum gi (x) implies minimum Euclidean distance: d⑀ ⫽ x ⫺ i
(2.48)
Thus, feature vectors are assigned to classes according to their Euclidean distance from the respective mean points. Can you verify that this result ties in with the geometry of the hyperplanes discussed before? Figure 2.13a shows curves of equal distance d⑀ ⫽ c from the mean points of each class. They are obviously circles of radius c (hyperspheres in the general case). ■
Nondiagonal ⌺: For this case maximizing gi (x) is equivalent to minimizing the ⌺⫺1 norm, known as the 1/2 Mahalanobis distance: dm ⫽ (x ⫺ i )T ⌺⫺1 (x ⫺ i )
(2.49)
In this case, the constant distance dm ⫽ c curves are ellipses (hyperellipses). Indeed, the covariance matrix is symmetric and, as discussed in Appendix B, it can always be diagonalized by a unitary transform ⌺ ⫽ ⌽⌳⌽T
(2.50)
“04-Ch02-SA272” 18/9/2008 page 31
2.4 Bayesian Classification for Normal Distributions
0.014 0.012 0.01 0.008 0.006 0.004 0.002 0 40 30 20 10 0 ⫺10 0 ⫺10 ⫺20 ⫺20 ⫺30 ⫺30 ⫺40 ⫺40
10
20
30
40
FIGURE 2.12 An example of two Gaussian pdfs with the same covariance matrix in the two-dimensional space. Each one of them is associated with one of two equiprobable classes. In this case, the decision curve is a straight line.
where ⌽T ⫽⌽⫺1 and ⌳ is the diagonal matrix whose elements are the eigenvalues of ⌺. ⌽ has as its columns the corresponding (orthonormal) eigenvectors of ⌺ ⌽ ⫽ [v 1 , v 2 , . . . , v l ]
(2.51)
Combining (2.49) and (2.50), we obtain (x ⫺ i )T ⌽⌳⫺1 ⌽T (x ⫺ i ) ⫽ c 2
x⬘ ⫽ ⌽T x.
(2.52)
v Tk x, k ⫽ 1, 2, . . . , l, that
Define The coordinates of x⬘ are equal to is, the projections of x onto the eigenvectors. In other words, they are the coordinates of x with respect to a new coordinate system whose axes are determined by v k , k ⫽ 1, 2, . . . , l. Equation (2.52) can now be written as (x1⬘ ⫺ i1⬘)2 (x⬘l ⫺ il⬘)2 ⫹ ··· ⫹ ⫽ c2 1 l
(2.53)
31
“04-Ch02-SA272” 18/9/2008 page 32
32
CHAPTER 2 Classifiers Based on Bayes Decision Theory
x2
x2 2cv2 2 !W 1cv1 2 !W
1 2 (a)
x1
(b)
x1
FIGURE 2.13 Curves of (a) equal Euclidean distance and (b) equal Mahalanobis distance from the mean points of each class. In the two-dimensional space, they are circles in the case of Euclidean distance and ellipses in the case of Mahalanobis distance. Observe that in the latter case the decision line is no longer orthogonal to the line segment joining the mean values. It turns according to the shape of the ellipses.
This is the equation of a hyperellipsoid in the new coordinate system. Figure 2.13b shows the l ⫽ 2 case. The center of mass of the ellipse is at i ,and the principal √ axes are aligned with the corresponding eigenvectors and have lengths 2 k c, respectively. Thus, all points having the same Mahalanobis distance from a specific point are located on an ellipse. Example 2.2 In a two-class, two-dimensional classification task, the feature vectors are generated by two normal distributions sharing the same covariance matrix 1.1 0.3 ⌺⫽ 0.3 1.9 and the mean vectors are 1 ⫽ [0, 0]T , 2 ⫽ [3, 3]T , respectively. (a) Classify the vector [1.0, 2.2]T according to the Bayesian classifier. It suffices to compute the Mahalanobis distance of [1.0, 2.2]T from the two mean vectors. Thus, 2 dm (1 , x) ⫽ (x ⫺ 1 )T ⌺⫺1 (x ⫺ 1 ) 0.95 ⫺0.15 1.0 ⫽ [1.0, 2.2] ⫽ 2.952 ⫺0.15 0.55 2.2
Similarly,
2 (2 , x) ⫽ [⫺2.0, ⫺0.8] dm
0.95 ⫺0.15
⫺0.15 0.55
⫺2.0 ⫽ 3.672 ⫺0.8
(2.54)
Thus, the vector is assigned to the class with mean vector [0, 0]T . Notice that the given vector [1.0, 2.2]T is closer to [3, 3]T with respect to the Euclidean distance.
“04-Ch02-SA272” 18/9/2008 page 33
2.4 Bayesian Classification for Normal Distributions
(b) Compute the principal axes of the ellipse centered at [0, 0]T that corresponds to a constant √ Mahalanobis distance dm ⫽ 2.952 from the center. To this end, we first calculate the eigenvalues of ⌺. 1.1 ⫺ 0.3 det ⫽ 2 ⫺ 3 ⫹ 2 ⫽ 0 0.3 1.9 ⫺ or 1 ⫽ 1 and 2 ⫽ 2. To compute the eigenvectors we substitute these values into the equation (⌺ ⫺ I )v ⫽ 0 and we obtain the unit norm eigenvectors ⎤ ⎡ v1 ⫽
√3 ⎣ 10 ⎦ , ⫺ √1 10
⎡ v2 ⫽
⎤
√1 ⎣ 10 ⎦ √3 10
It can easily be seen that they are mutually orthogonal. The principal axes of the ellipse are parallel to v 1 and v 2 and have lengths 3.436 and 4.859, respectively.
Remarks ■
In practice, it is quite common to assume that the data in each class are adequately described by a Gaussian distribution. As a consequence,the associated Bayesian classifier is either linear or quadratic in nature, depending on the adopted assumptions concerning the covariance matrices. That is, if they are all equal or different. In statistics, this approach to the classification task is known as linear discriminant analysis (LDA) or quadratic discriminant analysis (QDA), respectively. Maximum likelihood is usually the method mobilized for the estimation of the unknown parameters that define the mean values and the covariance matrices (see Section 2.5 and Problem 2.19).
■
A major problem associated with LDA and even more with QDA is the large number of the unknown parameters that have to be estimated in the case of high-dimensional spaces. For example, there are l parameters in each of the mean vectors and approximately l 2 /2 in each (symmetric) covariance matrix. Besides the high demand for computational resources,obtaining good estimates of a large number of parameters dictates a large number of training points, N . This is a major issue that also embraces the design of other types of classifiers, for most of the cases, and we will come to it in greater detail in Chapter 5. In an effort to reduce the number of parameters to be estimated, a number of approximate techniques have been suggested over the years, including [Kimu 87, Hoff 96, Frie 89, Liu 04]. Linear discrimination will be approached from a different perspective in Section 5.8.
■
LDA and QDA exhibit good performance in a large set of diverse applications and are considered to be among the most popular classifiers. No doubt, it is hard to accept that in all these cases the Gaussian assumption provides a reasonable modeling for the data statistics. The secret of the success seems
33
“04-Ch02-SA272” 18/9/2008 page 34
34
CHAPTER 2 Classifiers Based on Bayes Decision Theory
to lie in the fact that linear or quadratic decision surfaces offer a reasonably good partition of the space, from the classification point of view. Moreover, as pointed out in [Hast 01], the estimates associated with Gaussian models have some good statistical properties (i.e., bias variance trade-off, Section 3.5.3) compared to other techniques.
2.5 ESTIMATION OF UNKNOWN PROBABILITY DENSITY FUNCTIONS So far, we have assumed that the probability density functions are known. However, this is not the most common case. In many problems, the underlying pdf has to be estimated from the available data. There are various ways to approach the problem. Sometimes we may know the type of the pdf (e.g.,Gaussian,Rayleigh),but we do not know certain parameters, such as the mean values or the variances. In contrast, in other cases we may not have information about the type of the pdf but we may know certain statistical parameters, such as the mean value and the variance. Depending on the available information, different approaches can be adopted. This will be our focus in the next subsections.
2.5.1 Maximum Likelihood Parameter Estimation Let us consider an M-class problem with feature vectors distributed according to p(x|i ), i ⫽ 1, 2, . . . , M. We assume that these likelihood functions are given in a parametric form and that the corresponding parameters form the vectors i which are unknown. To show the dependence on i we write p(x|i ; i ). Our goal is to estimate the unknown parameters using a set of known feature vectors in each class. If we further assume that data from one class do not affect the parameter estimation of the others, we can formulate the problem independent of classes and simplify our notation. At the end, one has to solve one such problem for each class independently. Let x 1 , x 2 , . . . , x N be random samples drawn from pdf p(x; ). We form the joint pdf p(X; ), where X ⫽ {x 1 , . . . , x N } is the set of the samples. Assuming statistical independence between the different samples, we have p(X; ) ≡ p(x 1 , x 2 , . . . , x N ; ) ⫽
N
p(x k ; )
(2.55)
k⫽1
This is a function of , and it is also known as the likelihood function of with respect to X. The maximum likelihood (ML) method estimates so that the likelihood function takes its maximum value, that is, ˆ ML ⫽ arg max
N k⫽1
p(x k ; )
(2.56)
“04-Ch02-SA272” 18/9/2008 page 35
2.5 Estimation of Unknown Probability Density Functions
A necessary condition that ˆ ML must satisfy in order to be a maximum is the gradient of the likelihood function with respect to to be zero, that is ⭸
N
k⫽1 p(x k ; )
⭸
⫽0
(2.57)
Because of the monotonicity of the logarithmic function, we define the loglikelihood function as L() ≡ ln
N
p(x k ; )
(2.58)
k⫽1
and (2.57) is equivalent to N N ⭸L() ⭸ ln p(x k ; ) 1 ⭸p(x k ; ) ⫽ ⫽ ⫽0 ⭸ ⭸ p(x k ; ) ⭸ k⫽1
(2.59)
k⫽1
Figure 2.14 illustrates the method for the single unknown parameter case. The ML estimate corresponds to the peak of the log-likelihood function. Maximum likelihood estimation has some very desirable properties. If 0 is the true value of the unknown parameter in p(x; ), it can be shown that under generally valid conditions the following are true [Papo 91]. ■
The ML estimate is asymptotically unbiased, which by definition means that lim E[ˆ ML ] ⫽ 0
(2.60)
N →⬁
Alternatively,we say that the estimate converges in the mean to the true value. The meaning of this is as follows. The estimate ˆ ML is itself a random vector, because for different sample sets X different estimates will result. An estimate is called unbiased if its mean is the true value of the unknown parameter. In the ML case this is true only asymptotically (N → ⬁). p(X;)
ML
FIGURE 2.14 The maximum likelihood estimator ML corresponds to the peak of p(X; ).
35
“04-Ch02-SA272” 18/9/2008 page 36
36
CHAPTER 2 Classifiers Based on Bayes Decision Theory
■
The ML estimate is asymptotically consistent, that is, it satisfies lim prob{ˆ ML ⫺ 0 ⱕ ⑀} ⫽ 1
N →⬁
(2.61)
where ⑀ is arbitrarily small. Alternatively,we say that the estimate converges in probability. In other words, for large N it is highly probable that the resulting estimate will be arbitrarily close to the true value. A stronger condition for consistency is also true: lim E[ˆ ML ⫺ 0 2 ] ⫽ 0
N →⬁
(2.62)
In such cases we say that the estimate converges in the mean square. In words, for large N , the variance of the ML estimates tends to zero. Consistency is very important for an estimator,because it may be unbiased, but the resulting estimates exhibit large variations around the mean. In such cases we have little confidence in the result obtained from a single set X. ■
■
The ML estimate is asymptotically efficient; that is, it achieves the Cramer–Rao lower bound (Appendix A). This is the lowest value of variance, which any estimate can achieve. The pdf of the ML estimate as N → ⬁ approaches the Gaussian distribution with mean 0 [Cram 46]. This property is an offspring of (a) the central limit theorem (Appendix A) and (b) the fact that the ML estimate is related to the sum of random variables, that is, ⭸ ln(p(x k ; ))/⭸ (Problem 2.16).
In summary, the ML estimator is unbiased, is normally distributed, and has the minimum possible variance. However, all these nice properties are valid only for large values of N .
Example 2.3 Assume that N data points, x1 , x2 , . . . , xN , have been generated by a one-dimensional Gaussian pdf of known mean, , but of unknown variance. Derive the ML estimate of the variance. The log-likelihood function for this case is given by N N (xk ⫺ )2 1 L( 2 ) ⫽ ln p(xk ; 2 ) ⫽ ln √ √ exp ⫺ 2 2 2 k⫽1 k⫽1 2 or L( 2 ) ⫽ ⫺
N N 1 (xk ⫺ )2 ln(2 2 ) ⫺ 2 2 2 k⫽1
Taking the derivative of the above with respect to
2
and equating to zero, we obtain
N N 1 ⫺ 2 ⫹ (xk ⫺ )2 ⫽ 0 4 2 2 k⫽1
“04-Ch02-SA272” 18/9/2008 page 37
2.5 Estimation of Unknown Probability Density Functions
and finally the ML estimate of 2 results as the solution of the above, 2 ˆ ML ⫽
N 1 (xk ⫺ )2 N
(2.63)
k⫽1
2 in Eq. (2.63) is a biased estimate of the variance. Indeed, Observe that, for finite N , ˆ ML 2 ]⫽ E[ˆ ML
N 1 N ⫺1 2 E[(xk ⫺ )2 ] ⫽ N N k⫽1
where have
2
is the true variance of the Gaussian pdf. However, for large values of N , we
1 2 ) ≈ 2 N which is in line with the theoretical result of asymptotic consistency of the ML estimator. 2 E[ˆ ML ] ⫽ (1 ⫺
Example 2.4 Let x 1 , x 2 , . . . , x N be vectors stemmed from a normal distribution with known covariance matrix and unknown mean, that is, 1 1 T ⫺1 p(x k ; ) ⫽ exp ⫺ (x k ⫺ ) ⌺ (x k ⫺ ) (2)l/2 |⌺|1/2 2 Obtain the ML estimate of the unknown mean vector. For N available samples we have L() ≡ ln
N k⫽1
p(x k ; ) ⫽ ⫺
N N 1 ln((2)l |⌺|) ⫺ (x k ⫺ )T ⌺⫺1 (x k ⫺ ) 2 2
(2.64)
k⫽1
Taking the gradient with respect to , we obtain ⎡ ⭸L ⎤ ⭸1
⎢ ⭸L ⎥ ⎢ ⎥ N ⭸2 ⎥ ⭸L () ⎢ ⎥⫽ ≡⎢ ⌺⫺1 (x k ⫺ ) ⫽ 0 ⎢ . ⎥ ⭸ ⎢ .. ⎥ k⫽1 ⎣ ⎦
(2.65)
⭸L ⭸l
or ˆ ML ⫽
N 1 xk N
(2.66)
k⫽1
That is, the ML estimate of the mean, for Gaussian densities, is the sample mean. However, this very “natural approximation” is not necessarily ML optimal for non-Gaussian density functions.
37
“04-Ch02-SA272” 18/9/2008 page 38
38
CHAPTER 2 Classifiers Based on Bayes Decision Theory
p(X|)
p(X|)
p()
p()
(a)
(b)
FIGURE 2.15 ML and MAP estimates of will be approximately the same in (a) and different in (b).
2.5.2 Maximum a Posteriori Probability Estimation For the derivation of the maximum likelihood estimate, we considered as an unknown parameter. In this subsection we will consider it as a random vector, and we will estimate its value on the condition that samples x 1 , . . . , x N have occurred. Let X ⫽ {x 1 , . . . , x N }. Our starting point is p(|X). From our familiar Bayes theorem we have p()p(X|) ⫽ p(X)p(|X)
(2.67)
or p(|X) ⫽
p()p(X|) p(X)
(2.68)
The maximum a posteriori probability (MAP) estimate ˆ MAP is defined at the point where p(|X) becomes maximum, ⭸ p(|X) ⫽ 0 ˆ MAP : ⭸
or
⭸ ( p()p(X|)) ⫽ 0 ⭸
(2.69)
Note that p(X) is not involved since it is independent of . The difference between the ML and the MAP estimates lies in the involvement of p() in the latter case. If we assume that this obeys the uniform distribution, that is, is constant for all , both estimates yield identical results. This is also approximately true if p() exhibits small variation. However, in the general case, the two methods yield different results. Figures 2.15a and 2.15b illustrate the two cases. Example 2.5 Let us assume that in Example 2.4 the unknown mean vector is known to be normally distributed as 1 ⫺ 0 2 1 exp ⫺ p() ⫽ (2)l/2 l 2 2
“04-Ch02-SA272” 18/9/2008 page 39
2.5 Estimation of Unknown Probability Density Functions
The MAP estimate is given by the solution of N ⭸ ln p(x k |)p() ⫽ 0 ⭸ k⫽1
or, for ⌺ ⫽
2I , N 1 1 (x k ⫺ ) ˆ ⫺ 2 ( ˆ ⫺ 0 ) ⫽ 0 ⇒ 2
k⫽1
ˆ MAP ⫽
0 ⫹
2 2
1⫹
N
k⫽1 x k 2 N 2
2 2
⬎⬎ 1, that is, the variance 2 is very large and the corresponding We observe that if Gaussian is very wide with little variation over the range of interest, then ˆ MAP ≈ ˆ ML ⫽
N 1 xk N k⫽1
Furthermore, observe that this is also the case for N →⬁, regardless of the values of the variances. Thus, the MAP estimate tends asymptotically to the ML one. This is a more general result. For large values of N , the likelihood term N k⫽1 p(x k |) becomes sharply peaked around the true value (of the unknown parameter) and is the term that basically determines where the maximum occurs. This can be better understood by mobilizing the properties of the ML estimate given before.
2.5.3 Bayesian Inference Both methods considered in the preceding subsections compute a specific estimate of the unknown parameter vector . In the current method, a different path is adopted. Given the set X of the N training vectors and the a priori information about the pdf p(), the goal is to compute the conditional pdf p(x|X). After all, this is what we actually need to know. To this end, and making use of known identities from our statistics basics, we have the following set of relations at our disposal: p(x|X) ⫽
p(x|)p(|X) d
(2.70)
with p(|X) ⫽
p(X|) ⫽
p(X|)p() p(X|)p() ⫽ p(X) p(X|)p() d N k⫽1
p(x k |)
(2.71)
(2.72)
39
“04-Ch02-SA272” 18/9/2008 page 40
40
CHAPTER 2 Classifiers Based on Bayes Decision Theory
The conditional density p(|X) is also known as the a posteriori pdf estimate, since it is updated “knowledge”about the statistical properties of , after having observed the data set X. Once more, Eq. (2.72) presupposes statistical independence among the training samples. In general, the computation of p(x|X) requires the integration in the right-hand side of (2.70). However, analytical solutions are feasible only for very special forms of the involved functions. For most of the cases, analytical solutions for (2.70), as well as for the denominator in (2.71), are not possible, and one has to resort to numerical approximations. To this end, a large research effort has been invested in developing efficient techniques for the numerical computation of such statistical quantities. Although a detailed presentation of such approximation schemes is beyond the scope of this book, we will attempt to highlight the main philosophy behind these techniques in relation to our own problem. Looking more carefully at (2.70) and assuming that p(|X) is known,then p(x|X) is nothing but the average of p(x|) with respect to , that is, p(x|X) ⫽ E p(x|)
If we assume that a large enough number of samples i , i ⫽ 1, 2 . . . , L, of the random vector are available, one can compute the corresponding values p(x|i ) and then approximate the expectation as the mean value 1 p(x|i ) L L
p(x|X) ≈
i⫽1
The problem now becomes that of generating a set of samples, i , i ⫽ 1, 2 . . . , L. For example,if p(|X) were a Gaussian pdf,one could use a Gaussian pseudorandom generator to generate the L samples. The difficulty in our case is that, in general, the exact form of p(|X) is not known, and its computation presupposes the numerical integration of the normalizing constant in the denominator of (2.71). This difficulty is bypassed by a set of methods known as Markov chain Monte Carlo (MCMC) techniques. The main rationale behind these techniques is that one can generate samples from (2.71) in a sequential manner that asymptotically follow the distribution p(|X), even without knowing the normalizing factor. The Gibbs sampler and the Metropolis-Hastings algorithms are two of the most popular schemes of this type. For more details on such techniques, the interested reader may consult, for example, [Bish 06]. Further insight into the Bayesian methods can be gained by focusing on the Gaussian one-dimensional case. Example 2.6 Let p(x|) be a univariate Gaussian N (, 2 ) with unknown parameter the mean, which is also assumed to follow a Gaussian N (0 , 02 ). From the theory exposed before we have p(|X) ⫽
N p(X|)p() 1 p(xk |)p() ⫽ p(X) ␣ k⫽1
“04-Ch02-SA272” 18/9/2008 page 41
2.5 Estimation of Unknown Probability Density Functions
where for a given training data set, X, p(X) is a constant denoted as ␣, or N 1 (xk ⫺ )2 1 1 ( ⫺ 0 )2 p(|X) ⫽ exp ⫺ exp ⫺ √ √ ␣ 2 202 2 20 k⫽1 It is a matter of some algebra (Problem 2.25) to show that, given a number of samples, N , p(|X) turns out to be also Gaussian, that is, 1 ( ⫺ N )2 p(|X) ⫽ √ exp ⫺ (2.73) 2N2 2N with mean value N ⫽ and variance
N 02 x¯ N ⫹ 2 0 N 02 ⫹ 2
(2.74)
2 02 N 02 ⫹ 2
(2.75)
N2 ⫽
where x¯ N ⫽ N1 N k⫽1 xk . Letting N vary from 1 to ⬁, we generate a sequence of Gaussians N (N , N2 ), whose mean values move away from 0 and tend, in the limit, to the sample mean, which, asymptotically, becomes equal to the true mean value. Furthermore, their variance keeps decreasing at the rate 2 /N for large N . Hence, for large values of N , p(|X) becomes sharply peaked around the sample mean. Recall that the latter is the ML estimate of the mean value. Once p(|X) has been computed, it can be shown, by substituting (2.73) into (2.70) (problem 2.25), that 1 1 (x ⫺ N )2 p(x|X) ⫽ exp ⫺ 2 2 ⫹ N2 2( 2 ⫹ N2 ) which is a Gaussian pdf with mean value N and variance 2 ⫹ N2 . Observe that as N tends to infinity, the unknown mean value of the Gaussian tends to the ML estimate x¯ N (and asymptotically to the true mean) and the variance to the true value 2 . For finite values of N , the variance is larger than 2 to account for our extra uncertainty about x due to the unknown value of the mean . Figure 2.16 shows the posterior pdf estimate p(|X) obtained for different sizes of the training data set. Data were generated using a pseudorandom number generator following a Gaussian pdf with mean value equal to ⫽ 2 and variance 2 ⫽ 4. The mean value was assumed to be unknown, and the prior pdf was adopted to be Gaussian with 0 ⫽ 0 and 02 ⫽ 8. We observe that as N increases p(|X) gets narrower (in accordance to (2.75)). The respective mean value estimate (Eq. (2.74)) depends on N and x¯ N . For small values of N , the ML estimate of the mean, x¯ N , can vary a lot, which has a direct effect in moving around the centers of the Gaussians. However, as N increases, x¯ N tends to the true value of the mean ( ⫽ 2) with a decreasing variance. It can be shown (Problem 2.27) that the results of this example can be generalized for the case of multivariate Gaussians. More specifically, one can show that Eqs. (2.74) and (2.75) are generalized to the following p(|X) ∼ N (N , ⌺N )
(2.76)
41
“04-Ch02-SA272” 18/9/2008 page 42
42
CHAPTER 2 Classifiers Based on Bayes Decision Theory
p( |X)
4 3.5
N 5 400
3
N 5 200
2.5 2 1.5 N 5 50
N 5 100
1
N 5 10
0.5 0
1
2
3
4
5
6
7
8
9
FIGURE 2.16 A sequence of the posterior pdf estimates (Eq. (2.73)), for the case of Example 2.6. As the number of training points increases, the posterior pdf becomes more spiky (the ambiguity decreases) and its center moves toward the true mean value of the data.
where N ⫽ N ⌺0 [N ⌺0 ⫹ ⌺]⫺1 x¯ N ⫹ ⌺[N ⌺0 ⫹ ⌺]⫺1 0
(2.77)
⌺N ⫽ ⌺0 [N ⌺0 ⫹ ⌺]⫺1 ⌺
(2.78)
p(x|X) ∼ N (N , ⌺ ⫹ ⌺N )
(2.79)
and and also
Remarks ■
■
If p(|X) in Eq. (2.71) is sharply peaked at a ˆ and we treat it as a delta ˆ that is, the parameter estimate function, Eq. (2.70) becomes p(x|X) ≈ p(x|); is approximately equal to the MAP estimate. This happens, for example, if p(X|) is concentrated around a sharp peak and p() is broad enough around this peak. Then the resulting estimate approximates the ML one. The latter was also verified by our previous example. This is a more general property valid for most of the pdfs used in practice, for which the posterior probability of the unknown parameter vector p(|X) tends to a delta function as N tends to ⫹⬁. Thus, all three methods considered so far result, asymptotically, in the same estimate. However, the results are different for small numbers N of training samples. An obvious question concerns the choice of the prior p(). In practice, the choice depends on the form of the likelihood function p(x|), so that the posterior pdf p(|X) can be of a tractable form. The set of prior distributions
“04-Ch02-SA272” 18/9/2008 page 43
2.5 Estimation of Unknown Probability Density Functions
for which the adopted model p(x|) is of the same functional form as the posterior distribution p(|X) is known as conjugate with respect to the model. Some commonly used forms of the conjugate priors are discussed,for example, in [Bern 94]. ■
For data sets of limited length, ML and MAP estimators are simpler to use, and they result in a single estimate of the unknown parameters vector,which is the outcome of a maximization procedure. On the other hand, Bayesian methods make use of more information and, provided that this information is reliable, these techniques are expected to give better results, albeit at the expense of higher complexity. Due to the advances in computer technology, Bayesian methods have gained a lot of popularity over the recent years.
2.5.4 Maximum Entropy Estimation The concept of entropy is known from Shannon’s information theory. It is a measure of the uncertainty concerning an event and, from another viewpoint, a measure of randomness of the messages (feature vectors in our case) occurring at the output of a system. If p(x) is the density function, the associated entropy H is given by
H ⫽⫺
p(x) ln p(x) dx
(2.80)
x
Assume now that p(x) is unknown but we know a number of related constraints (mean value, variance, etc.). The maximum entropy estimate of the unknown pdf is the one that maximizes the entropy,subject to the given constraints. According to the principle of maximum entropy, stated by Jaynes [Jayn 82], such an estimate corresponds to the distribution that exhibits the highest possible randomness, subject to the available constraints. Example 2.7 The random variable x is nonzero for x1 ⱕ x ⱕ x2 and zero otherwise. Compute the maximum entropy estimate of its pdf. We have to maximize (2.80) subject to the constraint x2 p(x) dx ⫽ 1 (2.81) x1
Using Lagrange multipliers (Appendix C), this is equivalent to maximizing x2 HL ⫽ ⫺ p(x)(ln p(x) ⫺ ) dx
(2.82)
x1
Taking the derivative with respect to p(x), we obtain x2 ⭸HL ⫽⫺ ln p(x) ⫺ ⫹ 1 dx ⭸p(x) x1
(2.83)
43
“04-Ch02-SA272” 18/9/2008 page 44
44
CHAPTER 2 Classifiers Based on Bayes Decision Theory
Equating to zero, we obtain ˆ p(x) ⫽ exp( ⫺ 1)
(2.84)
To compute , we substitute this into the constraint equation (2.81), and we get exp( ⫺ 1) ⫽ 1 x2 ⫺x1 . Thus ⎧ 1 ⎨ x ⫺x if x1 ⱕ x ⱕ x2 2 1 ˆ (2.85) p(x) ⫽ ⎩0 otherwise That is, the maximum entropy estimate of the unknown pdf is the uniform distribution. This is within the maximum entropy spirit. Since we have imposed no other constraint but the obvious one, the resulting estimate is the one that maximizes randomness and all points are equally probable. It turns out that if the mean value and the variance are given as the second and third constraints, the resulting maximum entropy estimate of the pdf, for ⫺⬁ ⬍ x ⬍ ⫹⬁, is the Gaussian (Problem 2.30).
2.5.5 Mixture Models An alternative way to model an unknown p(x) is via a linear combination of density functions in the form of p(x) ⫽
J
p(x|j)Pj
(2.86)
p(x|j) dx ⫽ 1
(2.87)
j⫽1
where J j⫽1
Pj ⫽ 1, x
In other words, it is assumed that J distributions contribute to the formation of p(x). Thus, this modeling implicitly assumes that each point x may be “drawn” from any of the J model distributions with probability Pj , j ⫽ 1, 2, . . . , J . It can be shown that this modeling can approximate arbitrarily closely any continuous density function for a sufficient number of mixtures J and appropriate model parameters. The first step of the procedure involves the choice of the set of density components p(x|j) in parametric form, that is, p(x|j; ), and then the computation of the unknown parameters, and Pj , j ⫽ 1, 2, . . . , J , based on the set of the available training samples x k . There are various ways to achieve this. A typical maximum likelihood formulation, maximizing the likelihood function k p(x k ; , P1 , P2 , . . . , PJ ) with respect to and the Pj ’s,is a first thought. The difficulty here arises from the fact that the unknown parameters enter the maximization task in a nonlinear fashion; thus,nonlinear optimization iterative techniques have to be adopted (Appendix C).A review of related techniques is given in [Redn 84]. The source of this complication is the lack of information concerning the labels of the available training samples, that is, the specific mixture from which each sample is contributed. This is the issue that makes the current problem different from the ML case treated in Section 2.5.1. There, the class labels were known, and this led to a separate ML problem for each
“04-Ch02-SA272” 18/9/2008 page 45
2.5 Estimation of Unknown Probability Density Functions
of the classes. In the same way, if the mixture labels were known, we could collect all data from the same mixture and carry out J separate ML tasks. The missing label information makes our current problem a typical task with an incomplete data set. In the sequel, we will focus on the so-called EM algorithm, which has attracted a great deal of interest over the past few years in a wide range of applications involving tasks with incomplete data sets.
The Expectation Maximization (EM) Algorithm This algorithm is ideally suited for cases in which the available data set is incomplete. Let us first state the problem in more general terms and then apply it to our specific task. Let us denote by y the complete data samples, with y ∈ Y ⊆ Rm , and let the corresponding pdf be py (y; ), where is an unknown parameter vector. The samples y, however, cannot be directly observed. What we observe instead are samples x ⫽ g(y) ∈ Xob ⊆ Rl , l ⬍ m. We denote the corresponding pdf px (x; ). This is a many-to-one mapping. Let Y (x) ⊆ Y be the subset of all the y’s corresponding to a specific x. Then the pdf of the incomplete data is given by
px (x; ) ⫽
py (y; ) dy
(2.88)
Y (x )
As we already know, the maximum likelihood estimate of is given by ˆ ML :
⭸ ln(py (y k ; )) ⭸
k
⫽0
(2.89)
However, the y’s are not available. So, the EM algorithm maximizes the expectation of the log-likelihood function, conditioned on the observed samples and the current iteration estimate of . The two steps of the algorithm are: ■
E-step: At the (t ⫹ 1)th step of the iteration, where (t) is available, compute the expected value of Q(; (t)) ≡ E
ln(py (y k ; |X; (t))
(2.90)
k
This is the so-called expectation step of the algorithm. ■
M-step: Compute the next (t ⫹ 1)th estimate of by maximizing Q(; (t)), that is, (t ⫹ 1):
⭸Q(; (t)) ⫽0 ⭸
(2.91)
This is the maximization step, where, obviously, differentiability has been assumed. To apply the EM algorithm, we start from an initial estimate (0), and iterations are terminated if (t ⫹ 1) ⫺ (t) ⱕ ⑀ for an appropriately chosen vector norm and ⑀.
45
“04-Ch02-SA272” 18/9/2008 page 46
46
CHAPTER 2 Classifiers Based on Bayes Decision Theory
Remark ■
It can be shown that the successive estimates (t) never decrease the likelihood function. The likelihood function keeps increasing until a maximum (local or global) is reached and the EM algorithm converges. The convergence proof can be found in the seminal paper [Demp 77] and further discussions in [Wu 83, Boyl 83]. Theoretical results as well as practical experimentation confirm that the convergence is slower than the quadratic convergence of Newton-type searching algorithms (Appendix C), although near the optimum a speedup may be possible. However, the great advantage of the algorithm is that its convergence is smooth and is not vulnerable to instabilities. Furthermore, it is computationally more attractive than Newtonlike methods, which require the computation of the Hessian matrix. The keen reader may obtain more information on the EM algorithm and some of its applications from [McLa 88, Titt 85, Moon 96].
Application to the Mixture Modeling Problem In this case, the complete data set consists of the joint events (x k , jk ), k ⫽ 1, 2, . . . , N ,and jk takes integer values in the interval [1, J ],and it denotes the mixture from which x k is generated. Employing our familiar rule, we obtain p(x k , jk ; ) ⫽ p(x k |jk ; )Pjk
(2.92)
Assuming mutual independence among samples of the data set, the log-likelihood function becomes L() ⫽
N
ln p(x k |jk ; )Pjk
(2.93)
k⫽1
Let P ⫽ [P1 , P2 , . . . , PJ ]T . In the current framework,the unknown parameter vector is QT ⫽ [T , P T ]T . Taking the expectation over the unobserved data, conditioned on the training samples and the current estimates,Q(t),of the unknown parameters, we have
E-step: Q( Q; Q(t)) ⫽ E
N
ln( p(x k | jk ; )Pjk )
k⫽1
⫽
N
E[ln( p(x k | jk ; )Pjk )]
(2.94)
k⫽1
⫽
J N
P(jk |x k ; Q(t)) ln(p( x k | jk ; )Pjk )
(2.95)
k⫽1 jk ⫽1
The notation can now be simplified by dropping the index k from jk . This is because, for each k, we sum up over all possible J values of jk and these are the same for all k. We will demonstrate the algorithm for the case of Gaussian mixtures with diagonal
“04-Ch02-SA272” 18/9/2008 page 47
2.5 Estimation of Unknown Probability Density Functions
covariance matrices of the form ⌺j ⫽ j2 I , that is,
1
p(x k |j; ) ⫽
2j2
l/2 exp ⫺
x k ⫺ j 2
(2.96)
2j2
Assume that besides the prior probabilities, Pj , the respective mean values j as well as the variances j2 , j ⫽ 1, 2, . . . , J , of the Gaussians are also unknown. Thus, is a J (l ⫹ 1)-dimensional vector. Combining Eqs. (2.95) and (2.96) and omitting constants, we get E-step:
J N
l 1 Q( Q; Q(t)) ⫽ P( j|x k ; Q(t)) ⫺ ln j2 ⫺ x k ⫺ j 2 ⫹ ln Pj 2 2 2 j k⫽1 j⫽1
(2.97)
M-step: Maximizing the above with respect to j , j2 , and Pj results in (Problem 2.31) N j (t ⫹ 1) ⫽
k⫽1 P(
N
k⫽1 P(
N j2 (t ⫹ 1) ⫽ Pj (t ⫹ 1) ⫽
j|x k ; Q(t))x k j|x k ; Q(t))
j|x k ; Q(t))x k ⫺ j (t ⫹ 1)2 l N k⫽1 P( j|x k ; Q(t))
k⫽1 P(
N 1 P( j|x k ; Q(t)) N
(2.98)
(2.99)
(2.100)
k⫽1
For the iterations to be complete we need only to compute P(j|x k ; Q(t)). This is easily obtained from P(j|x k ; Q(t)) ⫽ p(x k ; Q(t)) ⫽
p(x k |j; (t))Pj (t) p(x k ; Q(t)) J
p(x k |j; (t))Pj (t)
(2.101)
(2.102)
j⫽1
Equations (2.98)–(2.102) constitute the EM algorithm for the estimation of the unknown parameters of the Gaussian mixtures in (2.86). The algorithm starts with valid initial guesses for the unknown parameters. Valid means that probabilities must add to one. Remark ■
Modeling unknown probability density functions via a mixture of Gaussian components and the EM algorithm has been very popular in a number of applications. Besides some convergence issues associated with the EM algorithm,
47
“04-Ch02-SA272” 18/9/2008 page 48
CHAPTER 2 Classifiers Based on Bayes Decision Theory
as previously discussed, another difficulty may arise in deciding about the exact number of components, J . In the context of supervised learning, one may use different values and choose the model that results in the best error probability. The latter can be computed by employing an error estimation technique (Chapter 10). Example 2.8 Figure 2.17a shows N ⫽ 100 points in the two-dimensional space, which have been drawn from a multimodal distribution. The samples were generated using two Gaussian random generators N (1 , ⌺1 ), N (2 , ⌺2 ), with 1.0 2.0 1 ⫽ , 2 ⫽ 1.0 2.0 and covariance matrices
⌺1 ⫽ ⌺2 ⫽
0.1 0.0
0.0 0.1
respectively. Each time a sample x k , k ⫽ 1, 2, . . . , N , is to be generated a coin is tossed. The corresponding probabilities for heads or tails are P(H) ≡ P ⫽ 0.8, P(T ) ⫽ 1 ⫺ P ⫽ 0.2, respectively. If the outcome of the coin flip is heads, the sample x k is generated from N (1 , ⌺1 ). Otherwise, it is drawn from N (2 , ⌺2 ). This is the reason that in Figure 2.17a the space around the point [1.0, 1.0]T is more densely populated. The pdf of the data set can obviously be written as p(x) ⫽ g(x; 1 , 12 )P ⫹ g(x; 2 , 22 )(1 ⫺ P)
(2.103)
denotes the Gaussian pdf with parameters the mean value and where a diagonal covariance matrix, ⌺ ⫽ diag{ 2 }, having 2 across the diagonal and zeros g(·; , 2 )
x2 3
⫺200
Log-likelihood
48
2
1
0
0
1
2 (a)
3
⫺300
⫺400
⫺500 x1
0
10 Iterations (b)
20
FIGURE 2.17 (a) The data set of Example 2.8 and (b) the log-likelihood as a function of the number of iteration steps.
“04-Ch02-SA272” 18/9/2008 page 49
2.5 Estimation of Unknown Probability Density Functions
elsewhere. Equation (2.103) is a special case of the more general formulation given in (2.86). The goal is to compute the maximum likelihood estimate of the unknown parameters vector QT ⫽ [P, T1 , 12 , T2 , 22 ] based on the available N ⫽ 100 points. The full training data set consists of the sample pairs (x k , jk ), k ⫽ 1, 2, . . . , N , where jk ∈ {1, 2}, and it indicates the origin of each observed sample. However, only the points x k are at our disposal, with the “label” information being hidden from us. To understand this issue better and gain more insight into the rationale behind the EM methodology, it may be useful to arrive at Eq. (2.95) from a slightly different route. Each of the random vectors, x k , can be thought of as the result of a linear combination of two other random vectors; namely, x k ⫽ ␣k x 1k ⫹ (1 ⫺ ␣k )x 2k where x 1k is drawn from N (1 , ⌺1 ) and x 2k from N (2 , ⌺2 ). The binary coefficients ␣k ∈ {0, 1} are randomly chosen with probabilities P(1) ⫽ P ⫽ 0.8, P(0) ⫽ 0.2. If the values of the ␣k s, k ⫽ 1, 2, . . . , N , were known to us, the log-likelihood function in (2.93) would be written as N N ␣k ln g(x k ; 1 , 12 )P ⫹ (1 ⫺ ␣k ) ln g(x k ; 2 , 22 )(1 ⫺ P) (2.104) L(Q; ␣) ⫽ k⫽1
k⫽1
since we can split the summation in two parts, depending on the origin of each sample x k . However, this is just an “illusion” since the ␣k s are unknown to us. Motivated by the spirit ˆ behind the EM algorithm, we substitute in (2.104) the respective mean values E[␣k |x k ; Q], ˆ of the unknown parameter vector. For the needs of our example we given an estimate, Q, have ˆ ⫽ 1 ⫻ P(1|x k ; Q) ˆ ⫹ 0 ⫻ (1 ⫺ P(1|x k ; Q)) ˆ ⫽ P(1|x k ; Q) ˆ E[␣k |x k ; Q]
(2.105)
Substitution of (2.105) into (2.104) results in (2.95) for the case of J ⫽ 2. We are now ready to apply the EM algorithm [Eqs. (2.98)–(2.102)] to the needs of our example. The initial values were chosen to be 1 (0) ⫽ [1.37, 1.20]T ,
2 (0) ⫽ [1.81, 1.62]T ,
12 ⫽ 22 ⫽ 0.44,
P ⫽ 0.5
Figure 2.17b shows the log-likelihood as a function of the number of iterations. convergence, the obtained estimates for the unknown parameters are 1 ⫽ [1.05, 1.03]T , 2 ⫽ [1.90, 2.08]T , 12 ⫽ 0.10, 22 ⫽ 0.06, P ⫽ 0.844
After
(2.106)
2.5.6 Nonparametric Estimation So far in our discussion a pdf parametric modeling has been incorporated, in one way or another, and the associated unknown parameters have been estimated. In
49
“04-Ch02-SA272” 18/9/2008 page 50
50
CHAPTER 2 Classifiers Based on Bayes Decision Theory
the current subsection we will deal with nonparametric techniques. These are basically variations of the histogram approximation of an unknown pdf, which is familiar to us from our statistics basics. Let us take, for example, the simple onedimensional case. Figure 2.18 shows two examples of a pdf and its approximation by the histogram method. That is, the x-axis (one-dimensional space) is first divided into successive bins of length h. Then the probability of a sample x being located in a bin is estimated for each of the bins. If N is the total number of samples and kN of these are located inside a bin, the corresponding probability is approximated by the frequency ratio P ≈ kN /N
(2.107)
This approximation converges to the true P as N → ⬁ (Problem 2.32). The corresponding pdf value is assumed constant throughout the bin and is approximated by ˆ ˆ x) p(x) ≡ p( ˆ ≈
1 kN , h N
|x ⫺ x| ˆ ⱕ
h 2
(2.108)
where xˆ is the midpoint of the bin. This determines the amplitude of the histogram curve over the bin. This is a reasonable approximation for continuous p(x) and small enough h so that the assumption of constant p(x) in the bin is sensible. It can ˆ be shown that p(x) converges to the true value p(x) as N → ⬁ provided: ■
hN → 0
■
kN → ⬁
■
kN N
→0
where hN is used to show the dependence on N . These conditions can be understood from simple reasoning, without having to resort to mathematical details. The first has already been discussed. The other two show the way that kN must grow
p(x)
p(x)
(a)
x
(b)
x
FIGURE 2.18 Probability density function approximation by the histogram method with (a) small and (b) large-size intervals (bins).
“04-Ch02-SA272” 18/9/2008 page 51
2.5 Estimation of Unknown Probability Density Functions
to guarantee convergence. Indeed, at all points where p(x) ⫽ 0 fixing the size hN , however small, the probability P of points occurring in this bin is finite. Hence, kN ≈ PN and kN tends to infinity as N grows to infinity. On the other hand, as the size hN of the bin tends to zero, the corresponding probability also goes to zero, justifying the last condition. In practice, the number N of data points is finite. The preceding conditions indicate the way that the various parameters must be chosen. N must be “large enough,” hN “small enough,” and the number of points falling in each bin “large enough” too. How small and how large depend on the type of the pdf function and the degree of approximation one is satisfied with. Two popular approaches used in practice are described next.
Parzen Windows In the multidimensional case, instead of bins of size h, the l-dimensional space is divided into hypercubes with length of side h and volume hl . Let x i , i ⫽ 1, 2, . . . , N , be the available feature vectors. Define the function (x) so that (x i ) ⫽
⎧ ⎨1
for |xij | ⱕ 1/2
⎩0
otherwise
(2.109)
where xij , j ⫽ 1, . . . , l, are the components of x i . In words, the function is equal to 1 for all points inside the unit side hypercube centered at the origin and 0 outside it. This is shown in Figure 2.19(a). Then (2.108) can be “rephrased” as ˆ p(x) ⫽
1 hl
N 1 xi ⫺ x N h
(2.110)
i⫽1
The interpretation of this is straightforward. We consider a hypercube with length of side h centered at x, the point where the pdf is to be estimated. This is illustrated in Figure 2.19(b) for the two-dimensional space. The summation equals kN , that is, the number of points falling inside this hypercube. Then the pdf estimate results from dividing kN by N and the respective hypercube volume hl . However, viewing Eq. (2.110) from a slightly different perspective,we see that we try to approximate a continuous function p(x) via an expansion in terms of discontinuous step functions (·). Thus,the resulting estimate will suffer from this“ancestor’s sin.”This led Parzen [Parz 62] to generalize (2.110) by using smooth functions in the place of (·). It can be shown that, provided
(x) ⱖ 0 (x) dx ⫽ 1
and
(2.111) (2.112)
x
the resulting estimate is a legitimate pdf. Such smooth functions are known as kernels or potential functions or Parzen windows. A typical example is the Gaussian N (0, I ), kernel. For such a choice, the approximate expansion of the unknown
51
“04-Ch02-SA272” 18/9/2008 page 52
52
CHAPTER 2 Classifiers Based on Bayes Decision Theory
x2 x21 0.5h
x
x2
0.5
x22 0.5h
20.5
0.5 x1 20.5 x120.5h x1 x110.5h
(b)
(a)
FIGURE 2.19 In the two-dimensional space (a) the function (x i ) is equal to one for every point, x i , inside the square of unit side centered at the origin and equal to zero for every point outside it. length, (b) The function x i h⫺x is equal to unity for every point x i inside the square with side length equal to h, centered at x and zero for all the other points.
p(x) will be ˆ p(x) ⫽
N 1 1 (x ⫺ x i )T (x ⫺ x i ) exp ⫺ l N 2h2 l i⫽1 (2) 2 h
In other words,the unknown pdf is approximated as an average of N Gaussians,each one centered at a different point of the training set. Recall that as the parameter h becomes smaller, the shape of the Gaussians becomes narrower and more “spiky” (Appendix A) and the influence of each individual Gaussian is more localized in the feature space around the area of its mean value. On the other hand, the larger the value of h, the broader their shape becomes and more global in space their influence is. The expansion of a pdf in a sum of Gaussians was also used in 2.5.5. However, here, the number of Gaussians coincides with the number of points, and the unknown parameter, h, is chosen by the user. In the EM algorithm concept, the number of Gaussians is chosen independently of the number of training points, and the involved parameters are computed via an optimization procedure. In the sequel, we will examine the limiting behavior of the approximation. To this end, let us take the mean value of (2.110) ˆ E[p(x)] ⫽
1 hl
≡ x⬘
N 1 x i ⫺ x E N h i⫽1
1 x⬘ ⫺ x p(x⬘) dx⬘ hl h
(2.113)
“04-Ch02-SA272” 18/9/2008 page 53
2.5 Estimation of Unknown Probability Density Functions
Thus, the mean value is a smoothed version of the true pdf p(x). However as 1 ⬘⫺x x tends to the delta function ␦(x⬘ ⫺ x). Indeed, h→0 the function hl h its amplitude goes to infinity, its width tends to zero, and its integral from (2.112) remains equal to one. Thus, in this limiting case and for well-behaved continuous ˆ pdfs, p(x) is an unbiased estimate of p(x). Note that this is independent of the size N of the data set. Concerning the variance of the estimate (Problem 2.38), the following remarks are valid: ■
For fixed N , the smaller the h the higher the variance, and this is indicated by the noisy appearance of the resulting pdf estimate, for example, Figures 2.20a and 2.21a as well as Figures 2.22c and 2.22d. This is because p(x) is approximated by a finite sum of ␦-like spiky functions, centered at the training sample ˆ points. Thus, as one moves x in space the response of p(x) will be very high near the training points, and it will decrease very rapidly as one moves away, leading to this noiselike appearance. Large values of h smooth out local variations in density.
■
For a fixed h, the variance decreases as the number of sample points N increases. This is illustrated in Figures 2.20a and 2.20b as well as in Figures 2.22b and 2.22c. This is because the space becomes dense in points, and the spiky functions are closely located. Furthermore, for a large enough number of samples, the smaller the h the better the accuracy of the resulting estimate, for example, Figures 2.20b and 2.21b.
■
It can be shown, for example, [Parz 62, Fuku 90] that, under some mild conditions imposed on (·), which are valid for most density functions, if h tends to zero but in such a way that hN → ⬁, the resulting estimate is both unbiased and asymptotically consistent.
p(x)
p(x)
0.12
0.12
0.06
0.06
0 0
10 (a)
20 x
0
0
10 (b)
20 x
FIGURE 2.20 Approximation (full-black line) of a pdf (dotted-red line) via Parzen windows, using Gaussian kernels with (a) h ⫽ 0.1 and 1,000 training samples and (b) h ⫽ 0.1 and 20,000 samples. Observe the influence of the number of samples on the smoothness of the resulting estimate.
53
“04-Ch02-SA272” 18/9/2008 page 54
54
CHAPTER 2 Classifiers Based on Bayes Decision Theory
p(x)
p(x)
0.12
0.12
0.06
0.06
0 0
10 (a)
20 x
0
0
10 (b)
20 x
FIGURE 2.21 Approximation (full-black line) of a pdf (dotted-red line) via Parzen windows, using Gaussian kernels with (a) h ⫽ 0.8 and 1,000 training samples and (b) h ⫽ 0.8 and 20,000 samples. Observe that, in this case, increasing the number of samples has little influence on the smoothness as well as the approximation accuracy of the resulting estimate. 0.12
0.12
0.08
0.08
0.04
0.04
0 6
0 6
1 ⫺4
⫺4
⫺2
0 (a)
2
4
6
1 ⫺4 ⫺4
0.12
0.12
0.08
0.08
0.04
0.04
0 6 1 ⫺4 ⫺4
⫺2
0 (c)
2
4
6
0 6 1 ⫺4 ⫺4
⫺2
0 (b)
2
4
6
⫺2
0 (d)
2
4
6
FIGURE 2.22 Approximation of a two-dimensional pdf, shown in (a), via Parzen windows, using twodimensional Gaussian kernels with (b) h ⫽ 0.05 and N ⫽ 1000 samples, (c) h ⫽ 0.05 and N ⫽ 20000 samples and (d) h ⫽ 0.8 and N ⫽ 20000 samples. Large values of h lead to smooth estimates, but the approximation accuracy is low (the estimate is highly biased), as one can observe by comparing (a) with (d). For small values of h, the estimate is more noisy in appearance, but it becomes smoother as the number of samples increases, (b) and (c). The smaller the h and the larger the N , the better the approximation accuracy.
“04-Ch02-SA272” 18/9/2008 page 55
2.5 Estimation of Unknown Probability Density Functions
Remarks ■
In practice, where only a finite number of samples is possible, a compromise between h and N must be made. The choice of suitable values for h is crucial, and several approaches have been proposed in the literature, for example, [Wand 95]. A straightforward way is to start with an initial estimate of h and then modify it iteratively to minimize the resulting misclassification error. The latter can be estimated by appropriate manipulation of the training set. For example, the set can be split into two subsets, one for training and one for testing. We will say more on this in Chapter 10.
■
Usually,a large N is necessary for acceptable performance. This number grows exponentially with the dimensionality l. If a one-dimensional interval needs, say, N equidistant points to be considered as a densely populated one, the corresponding two-dimensional square will need N 2 , the three-dimensional cube N 3 ,and so on. We usually refer to this as the curse of dimensionality. To our knowledge, this term was first used by Bellman in the context of Control theory [Bell 61]. To get a better feeling about the curse of dimensionality problem, let us consider the l-dimensional unit hypercube and let us fill it randomly with N points drawn from a uniform distribution. It can be shown ([Frie 89]) that the average Euclidean distance between a point and its nearest neighbor is given by d(l, N ) ⫽ 2
l⌫(l/2)
1 l
l 2 2 N
where ⌫(·) is the gamma function (AppendixA). In words,the average distance 1 to locate the nearest neighbor to a point, for fixed l, shrinks as N ⫺ l . To get a more quantitative feeling, let us fix N to the value N ⫽ 1010 . Then for l ⫽ 2, 10, 20 and 40, d(l, N ) becomes 10⫺5 , 0.18, 0.76, and 1.83, respectively. Figure 2.23a shows 50 points lying within the unit-length segment in the one-dimensional space. The points were randomly generated by the uniform distribution. Figure 2.23b shows the same number of points lying in the unitlength square. These points were also generated by a uniform distribution in the two-dimensional space. It is readily seen that the points in the onedimensional segment are, on average, more closely located compared to the same number of points in the two-dimensional square. The large number of data points required for a relatively high-dimensional feature space to be sufficiently covered puts a significant burden on complexity requirements, since one has to consider one Gaussian centered at each point. To this end,some techniques have been suggested that attempt to approximate the unknown pdf by using a reduced number of kernels, see, for example, [Babi 96]. Another difficulty associated with high-dimensional spaces is that, in practice, due to the lack of enough training data points, some regions in the feature space may be sparsely represented in the data set. To cope with
55
“04-Ch02-SA272” 18/9/2008 page 56
56
CHAPTER 2 Classifiers Based on Bayes Decision Theory
x2
1
0.8 0.6 0.4 0.2 0 0
0.2
0.4
0.6
0.8
0.2
0
1x
(a)
0.4
0.6
0.8
1x 1
(b)
FIGURE 2.23 Fifty points generated by a uniform distribution lying in the (a) one-dimensional unit-length segment and (b) the unit-length square. In the two-dimensional space the points are more spread compared to the same number of points in the one-dimensional space.
such scenarios, some authors have adopted a variable value for h. In regions where data are sparse, a large value of h is used, while in more densely populated areas a smaller value is employed. To this end, a number of mechanisms for adjusting the value of h have been adopted, see, for example, [Brei 77, Krzy 83, Terr 92, Jone 96]. Application to classification:
test in (2.20) becomes assign x to 1 (2 )
if
On the reception of a feature vector x the likelihood ⎛
l12 ≈ ⎝
1 N1 h l 1 N2 h l
⎞
x i ⫺x P(2 ) i⫽1 h N2 x i ⫺x ⎠ ⬎ (⬍) P(1 ) i⫽1 h N 1
21 ⫺ 22 12 ⫺ 11
(2.114)
where N1 , N2 are the training vectors in class 1 , 2 , respectively. The risk-related terms are ignored when the Bayesian minimum error probability classifier is used. For large N1 , N2 this computation is a very demanding job, in both processing time and memory requirements.
k Nearest Neighbor Density Estimation In the Parzen estimation of the pdf in (2.110), the volume around the points x was considered fixed (hl ) and the number of points kN , falling inside the volume, was left to vary randomly from point to point. Here we will reverse the roles. The number of points kN ⫽ k will be fixed, and the size of the volume around x will be adjusted each time, to include k points. Thus, in low-density areas the volume will be large and in high-density areas it will be small. We can also consider more general types of regions, besides the hypercube. The estimator can now be
“04-Ch02-SA272” 18/9/2008 page 57
2.5 Estimation of Unknown Probability Density Functions
written as ˆ p(x) ⫽
k NV (x)
(2.115)
where the dependence of the volume V (x) on x is explicitly shown. Again it can be shown [Fuku 90] that asymptotically (lim k ⫽ ⫹⬁, lim N ⫽ ⫹⬁, lim(k/N ) ⫽ 0) this is an unbiased and consistent estimate of the true pdf, and it is known as the k Nearest Neighbor (kNN) density estimate. Results concerning the finite k, N case have also been derived; see [Fuku 90, Butu 93]. A selection of seminal papers concerning NN classification techniques can be found in [Dasa 91]. From a practical point of view, at the reception of an unknown feature vector x, we compute its distance d, for example, Euclidean, from all the training vectors of the various classes, for example, 1 , 2 . Let r1 be the radius of the hypersphere, centered at x, that contains k points from 1 and r2 the corresponding radius of the hypersphere containing k points from class 2 (k may not necessarily be the same for all classes). If we denote by V1 , V2 the respective hypersphere volumes, the likelihood ratio test becomes assign x to 1 (2 )
if l12 ≈
kN2 V2 P(2 ) 21 ⫺ 22 ⬎ (⬍) kN1 V1 P(1 ) 12 ⫺ 11 N1 P(2 ) 21 ⫺ 22 V2 ⬎ (⬍) V1 N2 P(1 ) 12 ⫺ 11
(2.116)
If the Mahalanobis distance is alternatively adopted, we will have hyperellipsoids in the place of the hyperspheres. The volume of a hyperellipsoid, corresponding to Mahalanobis distance equal to r, is given by ([Fuku 90]) 1
V ⫽ V0 |⌺| 2 r l
(2.117)
where V0 is the volume of the hypersphere of unit radius given by V0 ⫽
⎧ ⎨
l
2 /(l/2)!,
⎩2l l⫺1 2 ( l⫺1 )!/l!, 2
l even
(2.118)
l odd
Verify that Eq. (2.117) results to 4r 3 /3 for the volume of a sphere of radius r in the three-dimensional space. Remark ■
The nonparametric probability density function estimation techniques, discussed in this section,are among the techniques that are still in use in practical applications. It is interesting to note that, although the performance of the methods, as density estimators, degrades in high-dimensional spaces due to the lack of sufficient data, their performance as classifiers may be sufficiently good. After all, lack of enough training data points affects, in one way or another, all the methods.
57
“04-Ch02-SA272” 18/9/2008 page 58
58
CHAPTER 2 Classifiers Based on Bayes Decision Theory
More recently, the so-called probabilistic neural networks have been suggested as efficient implementations for the computation of the classifier given in (2.114), by exploiting the intrinsic parallelism of the neural network architectures and will be discussed in Chapter 4.
Example 2.9 The points shown in Figure 2.24 belong to either of two equiprobable classes. Black points belong to class 1 and red points belong to class 2 . For the needs of the example we assume that all points are located at the nodes of a grid. We are given the point denoted by a “star”, with coordinates (0.7, 0.6), which is to be classified in one of the two classes. The Bayesian (minimum error probability) classifier and the k-nearest neighbor density estimation technique, for k ⫽ 5, will be employed. Adopting the Euclidean distance, we find the five nearest neighbors to the unknown point (0.7, 0.6) from all the points in class 2 . These are the points (0.8, 0.6), (0.7, 0.7), (0.6, 0.5), (0.6, 0.6), (0.6, 0.7). The full line circle encircles the five nearest neighbors, and its radius is equal to the distance of the point that is furthest from (0.7, 0.6), that is, √ √ ⫽ 0.12 ⫹ 0.12 ⫽ 0.1 2. In the sequel, we repeat the procedure for the points in class 1 . The nearest points are the ones with coordinates (0.7, 0.5), (0.8, 0.4), (0.8, 0.7),
x2 1.3
0.9
0.6
0.3
0
0
0.4
0.7
1
1.4
x1
FIGURE 2.24 The setup for the example 2.9. The point denoted by a “star” is classified to the class 2 of the red points. The k ⫽ 5 nearest neighbors from this class lie within a smaller area compared to the five nearest neighbors coming from the other class.
“04-Ch02-SA272” 18/9/2008 page 59
2.5 Estimation of Unknown Probability Density Functions
(0.9, 0.6), √ (0.9, 0.8). The dotted circle is the one that encircles all five points, and its radius √ is equal to 0.22 ⫹ 0.22 ⫽ 0.2 2 ⫽ 2. There are N1 ⫽ 59 points in class 1 and N2 ⫽ 61 in class 2 . The areas (volumes) of the two circles are V1 ⫽ 42 and V2 ⫽ 2 , respectively, for the two classes. Hence, according to Eq. (2.116) and ignoring the risk related terms, we have V2 2 ⫽ ⫽ 0.25 V1 42 and since 0.25 is less than 59/61 and the classes are equiprobable, the point (0.7, 0.6) is classified to class 2 .
2.5.7 The Naive-Bayes Classifier The goal in this section, so far, was to present various techniques for the estimation of the probability density functions p(x|i ), i ⫽ 1, 2, . . . , M, required by the Bayes classification rule, based on the available training set, X. As we have already stated, in order to safeguard good estimates of the pdfs the number of training samples, N , must be large enough. To this end, the demand for data increases exponentially fast with the dimension, l, of the feature space. Crudely speaking, if N could be regarded as a good number of training data points for obtaining sufficiently accurate estimates of a pdf in an one-dimensional space, then N l points would be required for an l-dimensional space. Thus, large values of l make the accurate estimation of a multidimensional pdf a bit of an “illusion” since in practice data is hard to obtain. Loosely speaking, data can be considered to be something like money. It is never enough! Accepting this reality, one has to make concessions about the degree of accuracy that is expected from the pdf estimates. One widely used approach is to assume that individual features xj , j ⫽ 1, 2, . . . , l, are statistically independent. Under this assumption, we can write p(x|i ) ⫽
l
p(xj |i ),
i ⫽ 1, 2, . . . , M
j⫽1
The scenario is now different. To estimate l one-dimensional pdfs, for each of the classes, lN data points would be enough in order to obtain good estimates, instead of N l . This leads to the so-called naive-Bayes classifier, which assigns an unknown sample x ⫽ [x1 , x2 , . . . , xl ]T to the class m ⫽ arg max i
l
p(xj |i ),
i ⫽ 1, 2, . . . , M
j⫽1
It turns out that the naive-Bayes classifier can be very robust to violations of its independence assumption, and it has been reported to perform well for many realworld data sets. See, for example, [Domi 97].
59
“04-Ch02-SA272” 18/9/2008 page 60
60
CHAPTER 2 Classifiers Based on Bayes Decision Theory
Example 2.10 The discrete features case: In Section 2.2, it was stated that in the case of discrete-valued features the only required change in the Bayesian classification rule is to replace probability density functions with probabilities. In this example, we will see how the associated with the naive Bayes classifier assumption of statistical independence among the features simplifies the Bayesian classification rule. Consider the feature vector x ⫽ [x1 , x2 , . . . , xl ]T with binary features, that is, xi ∈ {0, 1}, i ⫽ 1, 2, . . . , l. Also let the respective class-conditional probabilities be P(xi ⫽ 1|1 ) ⫽ pi and P(xi ⫽ 1|2 ) ⫽ qi . According to the Bayesian rule, given the value of x, its class is decided according to the value of the likelihood ratio P(1 )P(x|1 ) ⬎ (⬍)1 P(2 )P(x|2 )
(2.119)
for the minimum probability error rule (the minimum risk rule could also be used). The number of values that x can take, for all possible combinations of xi , amounts to 2l . If we do not adopt the independence assumption, then one must have enough training data in order to obtain probability estimates for each one of these values (probabilities add to one, thus 2l ⫺ 1 estimates are required). However, adopting statistical independence among the features, we can write l
P(x|1 ) ⫽
pixi (1 ⫺ pi )1⫺xi
i⫽1
and l
P(x|2 ) ⫽
qixi (1 ⫺ qi )1⫺xi
i⫽1
Hence, the number of required probability estimates is now 2l, that is, the pi ’s and qi ’s. It is interesting to note that, taking the logarithm of both sides in (2.119), one ends up with a linear discriminant function similar to the hyperplane classifier of Section 2.4, that is, g(x) ⫽
l i⫽1
pi 1 ⫺ pi xi ln ⫹ (1 ⫺ xi ) ln qi 1 ⫺ qi
⫹ ln
P(1 ) P(2 )
(2.120)
which can easily be brought into the form of g(x) ⫽ w T x ⫹ w0
(2.121)
where &
pl (1 ⫺ ql ) p1 (1 ⫺ q1 ) , . . . , ln w ⫽ ln q1 (1 ⫺ p1 ) ql (1 ⫺ pl ) and w0 ⫽
l i⫽1
ln
1 ⫺ pi P(1 ) ⫹ ln 1 ⫺ qi P(2 )
'T
“04-Ch02-SA272” 18/9/2008 page 61
2.6 The Nearest Neighbor Rule
Binary features are used in a number of applications where one has to decide based on the presence or not of certain attributes. For example, in medical diagnosis, 1 can represent a normal value in a medical test and a 0 an abnormal one.
2.6 THE NEAREST NEIGHBOR RULE A variation of the kNN density estimation technique results in a suboptimal, yet popular in practice, nonlinear classifier. Although this does not fall in the Bayesian framework, it fits nicely at this point. In a way, this section could be considered as a bridge with Chapter 4. The algorithm for the so-called nearest neighbor rule is summarized as follows. Given an unknown feature vector x and a distance measure, then: ■
Out of the N training vectors, identify the k nearest neighbors, regardless of class label. k is chosen to be odd for a two class problem, and in general not to be a multiple of the number of classes M.
■
Out of these k samples, identify the number of vectors, ki , that belong to class i , i ⫽ 1, 2, . . . , M. Obviously, i ki ⫽ k.
■
Assign x to the class i with the maximum number ki of samples.
Figure 2.25 illustrates the k-NN rule for the case of k ⫽ 11. Various distance measures can be used, including the Euclidean and Mahalanobis distance. The simplest version of the algorithm is for k ⫽ 1,known as the nearest neighbor (NN) rule. In other words, a feature vector x is assigned to the class of its nearest neighbor! Provided that the number of training samples is large enough, this simple
FIGURE 2.25 Using the 11-NN rule, the point denoted by a “star” is classified to the class of the red points. Out of the eleven nearest neighbors seven are red and four are black. The circle indicates the area within which the eleven nearest neighbors lie.
61
“04-Ch02-SA272” 18/9/2008 page 62
62
CHAPTER 2 Classifiers Based on Bayes Decision Theory
rule exhibits good performance. This is also substantiated by theoretical findings. It can be shown [Duda 73, Devr 96] that, as N →⬁, the classification error probability, for the NN rule, PNN , is bounded by PB ⱕ PNN ⱕ PB 2 ⫺
M PB ⱕ 2PB M ⫺1
(2.122)
where PB is the optimal Bayesian error. Thus, the error committed by the NN classifier is (asymptotically) at most twice that of the optimal classifier. The asymptotic performance of the kNN is better than that of the NN, and a number of interesting bounds have been derived. For example, for the two-class case it can be shown, for example, [Devr 96] that 1 PB ⱕ PkNN ⱕ PB ⫹ √ ke
( or PB ⱕ PkNN ⱕ PB ⫹
2PNN k
(2.123)
Both of these suggest that as k → ⬁ the performance of the kNN tends to the optimal one. Furthermore, for small values of Bayesian errors, the following approximations are valid [Devr 96]: PNN ≈ 2PB P3NN ≈ PB ⫹ 3(PB )2
(2.124) (2.125)
Thus, for large N and small Bayesian errors, we expect the 3NN classifier to give performance almost identical to that of the Bayesian classifier. As an example, let us say that the error probability of the Bayesian classifier is of the order of 1%; then the error resulting from a 3NN classifier will be of the order of 1.03%! The approximation improves for higher values of k. A little thought can provide justification for this without too much mathematics. Under the assumption of large N , the radius of the hypersphere (Euclidean distance) centered at x and containing its k nearest neighbors tends to zero [Devr 96]. This is natural, because for very large N we expect the space to be densely filled with samples. Thus, the k (a very small portion of N ) neighbors of x will be located very close to it, and the conditional class probabilities, at all points inside the hypersphere around x, will be approximately equal to P(i |x) (assuming continuity). Furthermore, for large k (yet an infinitesimally small fraction of N ), the majority of the points in the region will belong to the class corresponding to the maximum conditional probability. Thus,the kNN rule tends to the Bayesian classifier. Of course, all these are true asymptotically. In the finite sample case there are even counterexamples (Problem 2.34) where the kNN results in higher error probabilities than the NN. However, in conclusion, it can be stated that the nearest neighbor techniques are among the serious candidates to be adopted as classifiers in a number of applications. A comparative study of the various statistical classifiers, considered in this chapter as well as others, can be found in [Aebe 94]. Remarks ■
A serious drawback associated with (k)NN techniques is the complexity in search of the nearest neighbor(s) among the N available training samples.
“04-Ch02-SA272” 18/9/2008 page 63
2.6 The Nearest Neighbor Rule
Brute-force searching amounts to operations proportional to kN (O(kN )).2 The problem becomes particularly severe in high-dimensional feature spaces. To reduce the computational burden, a number of efficient searching schemes have been suggested; see, for example, [Fuku 75, Dasa 91, Brod 90, Djou 97, Nene 97, Hatt 00, Kris 00, Same 08]. In [Vida 94, Mico 94] a preprocessing stage is suggested that computes a number of base prototypes that are in some sense maximally separated from among the set of training feature vectors. A summary of efficient searching techniques and a comparative study is given in [McNa 01]. ■
■
Although, due to its asymptotic error performance, the kNN rule achieves good results when the data set is large (compared to the dimension of the feature space), the performance of the classifier may degrade dramatically when the value of N is relatively small [Devr 96]. Also, in practice, one may have to reduce the number of training patterns due to the constraints imposed by limited computer resources. To this end, a number of techniques, also known as prototype editing or condensing, have been proposed. The idea is to reduce the number of training points in a way that a cost related to the error performance is optimized; see, for example, [Yan 93, Huan 02, Pare 06a] and the references therein. Besides computational savings, reducing the size of a finite set appropriately may offer performance improvement advantages, by making the classifier less sensitive to outliers. A simple method, which also makes transparent the reason for such a potential improvement, has been suggested in [Wils 72]. This editing procedure tests a sample using a kNN rule against the rest of the data. The sample is discarded if it is misclassified. The edited data set is then used for a NN classification of unknown samples. A direction to cope with the performance degradation associated with small values of N is to employ distance measures that are optimized on the available training set. The goal is to find a data-adaptive distance metric that leads to an optimal performance, according to an adopted cost. Such trained metrics can be global ones (i.e., the same at every point), class-dependent (i.e., shared by all points of the same class), and/or locally dependent (i.e., the metric varies according to the position in the feature space); see, for example, [Hast 96, Dome 05, Pare 06] and the references therein. An in depth treatment of the topic is given in [Frie 94]. When the k ⫽ 1 nearest neighbor rule is used, the training feature vectors x i , i⫽1, 2, . . . , N , define a partition of the l-dimensional space into N regions, Ri . Each of these regions is defined by Ri ⫽ {x: d(x, x i ) ⬍ d(x, x j ), i ⫽ j}
(2.126)
that is, Ri contains all points in space that are closer to x i than any other point of the training set, with respect to the distance d. This partition of the
2
O(n) denotes order of n calculations.
63
“04-Ch02-SA272” 18/9/2008 page 64
64
CHAPTER 2 Classifiers Based on Bayes Decision Theory
FIGURE 2.26 An example of Voronoi tessellation in the two-dimensional space and for Euclidean distance.
feature space is known as Voronoi tessellation. Figure 2.26 is an example of the resulting Voronoi tessellation for the case of l ⫽ 2 and the Euclidean distance.
2.7 BAYESIAN NETWORKS In Section 2.5.7 the naive-Bayes classifier was introduced as a means of coping with the curse of dimensionality and to exploiting more efficiently the available training data set. However,by adopting the naive-Bayes classifier,one goes from one extreme (fully dependent features) to another (features mutually independent). Common sense drives us to search for approximations that lie between these two extremes. The essence of the current section is to introduce a methodology that allows one to develop models that can accommodate built-in independence assumptions with respect to the features xi , i ⫽ 1, 2, . . . , l. Recall the well-known probability chain rule [Papo 91, p. 192] p(x1 , x2 , . . . , xl ) ⫽ p(xl |xl⫺1 , . . . , x1 )p(xl⫺1 |xl⫺2 . . . , x1 ) . . . , p(x2 |x1 )p(x1 )
(2.127)
This rule applies always and does not depend on the order in which features are presented. The rule states that the joint probability density function can be expressed in terms of the product of conditional pdfs and a marginal one (p(x1 )).3 This important and elegant rule opens the gate through which assumptions will infiltrate the problem. The conditional dependence for each feature,xi ,will be limited into a subset of the features appearing in each term in the product. Under this assumption,
3
In the study of several random variables, the statistics of each are called marginal.
“04-Ch02-SA272” 18/9/2008 page 65
2.7 Bayesian Networks
Eq. (2.127) can now be written as p(x) ⫽ p(x1 )
l
p(xi |Ai )
(2.128)
Ai ⊆ {xi⫺1 , xi⫺2 , . . . , x1 }
(2.129)
p(x6 |x5 , . . . , x1 ) ⫽ p(x6 |x5 , x4 )
(2.130)
p(x5 |x4 , . . . , x1 ) ⫽ p(x5 |x4 )
(2.131)
i⫽2
where
For example, let l ⫽ 6 and
p(x4 |x3 , x2 , x1 ) ⫽ p(x4 |x2 , x1 )
(2.132)
p(x3 |x2 , x1 ) ⫽ p(x3 |x2 )
(2.133)
p(x2 |x1 ) ⫽ p(x2 )
(2.134)
Then, A6 ⫽ {x5 , x4 }, A5 ⫽ {x4 }, A4 ⫽ {x2 , x1 }, A3 ⫽ {x2 }, A2 ⫽ ∅
where ∅ denotes the empty set. These assumptions are represented graphically in Figure 2.27. Nodes correspond to features. The parents of a feature, xi , are those features with directed links toward xi and are the members of the set Ai . In other words, xi is conditionally independent of any combination of its nondescendants, given its parents. There is a subtle point concerning conditional independence. Take, for example, that p(x3 |x2 , x1 ) ⫽ p(x3 |x2 ). This does not necessarily mean that x3 and x1 are independent. They may be dependent while x2 is unknown, but they become independent once the value of x2 is disclosed to us. This is not surprising since by measuring the value of a random variable part of the randomness is removed. Under the previous assumptions, the problem of estimating the joint pdf has broken into the product of simpler terms. Each of them involves, in general, a much smaller number of features compared to the original number. For example, x1
x4
x2
x3 x5
x6
FIGURE 2.27 Graphical model illustrating conditional dependencies.
65
“04-Ch02-SA272” 18/9/2008 page 66
66
CHAPTER 2 Classifiers Based on Bayes Decision Theory
for the case of Eqs. (2.130)–(2.134) none of the products involves more than three features. Hence, the estimation of each pdf term in the product takes place in a low-dimensional space and the problems arising from the curse of dimensionality can be handled easier. To get a feeling for the computational size reduction implied by the independence assumptions, encoded in the graphical model of Figure 2.27, let us assume that variables xi , i ⫽ 1, 2, . . . , 6, are binary. Then the pdfs in (2.127)– (2.134) become probabilities. Complete knowledge of P(x1 , . . . , x6 ) requires the estimation of 63 (2l ⫺ 1) probability values. It is 63 and not 64 due to the constraint that probabilities must add to one. This is also suggested by the right-hand side of Eq. (2.127). The number of the required probability values is 2l⫺1 ⫹2l⫺2 ⫹· · ·⫹1 ⫽ 2l ⫺ 1. In contrast to that, the assumptions in (2.130)–(2.134) reduce the number of the required probability values to be estimated to 13 (Why?). For large values of l, such a saving can be very significant. The naive-Bayes classifier is a special case for which Ai ⫽ ∅, i ⫽ 2, . . . , l, and the product in (2.128) becomes a product of marginal pdfs. Examples of classifiers that exploit the idea of conditional independence with respect to a subset of features are given in, for example, [Frie 97, Webb 05, Roos 05]. Although our original goal was to seek for ways for the approximate estimation of a joint pdf,it turns out that the adopted assumptions (nicely condensed in a graphical representation such as in Figure 2.27), have much more interesting consequences. For the rest of the section and for the sake of simplicity, we will assume that the features can only take values from a discrete set. Thus, pdfs give their place to probabilities. Definition: A Bayesian network is a directed acyclic graph (DAG) where the nodes correspond to random variables (features). Each node is associated with a set of conditional probabilities, P(xi |Ai ), where xi is the variable associated with the specific node and Ai is the set of its parents in the graph. Acyclic means that there are no cycles in the graph. For example, the graph in Figure 2.27 is an acyclic one, and it will cease to be so if one draws an arc directed from x6 to, say, x1 . The complete specification of a Bayesian network requires knowledge of (a) the marginal probabilities of the root nodes (those without a parent) and (b) the conditional probabilities of the nonroot nodes, given their parents for all possible combinations of their values. The joint probability of the variables can now be obtained by multiplying all conditional probabilities with the prior probabilities of the root nodes. All that is needed is to perform a topological sorting of the random variables;that is,to order the variables such that every variable comes before its descendants in the related graph. Bayesian networks have been used in a variety of applications. The network in Figure 2.28 corresponds to an example inspired by the discipline of medical diagnosis, a scientific area where Bayesian networks have been very popular. S stands for smokers, C for lung cancer, and H for heart disease. H1 and H2 are heart disease medical tests, and C1 and C2 are cancer medical tests. The table of the root node shows the population percentage (probability) of smokers (True) and
“04-Ch02-SA272” 18/9/2008 page 67
2.7 Bayesian Networks
P(S) True False 0.40 0.60 P(H|S) True False S True 0.40 0.60 0.85 False 0.15
S
S True False
H
C
H1 H True False
P(C|S) True False 0.20 0.80 0.11 0.89
C1
P(H1|H) True False 0.95 0.05 0.01 0.99
H2
P(H2|H) H True False True 0.98 0.02 False 0.05 0.95
P(C1|C) True False C True 0.99 0.01 False 0.10 0.90
C2
P(C2|C) C True False True 0.98 0.02 False 0.05 0.95
FIGURE 2.28 Bayesian network modeling conditional dependencies for an example concerning smokers (S), tendencies to develop cancer (C), and heart disease (H), together with variables corresponding to heart (H1, H2) and cancer (C1, C2) medical tests.
nonsmokers (False). The tables along the nodes of the tree are the respective conditional probabilities. For example, P(C : True|S : True) ⫽ 0.20 is the probability of a smoker (True) to develop cancer (True). (The probabilities used in Figure 2.28 may not correspond to true values having resulted from statistical studies.) Once a DAG has been constructed, the Bayesian network allows one to calculate efficiently the conditional probability of any node in the graph, given that the values of some other nodes have been observed. Such questions arise in the field of artificial intelligence closely related to pattern recognition. The computational efficiency stems from the existing probability relations encoded in the graph. A detailed treatment of the topic is beyond the scope of this book; the interested reader may consult more specialized texts, such as [Neap 04]. The remainder of this section aims at providing the reader with a flavor of the related theory. Probability Inference: This is the most common task that Bayesian networks help us to solve efficiently. Given the values of some of the variables, known as evidence, the goal is to compute the conditional probabilities for some (or all) of the other variables in the graph, given the evidence.
67
“04-Ch02-SA272” 18/9/2008 page 68
68
CHAPTER 2 Classifiers Based on Bayes Decision Theory
Example 2.11 Let us take the simple Bayesian network of Figure 2.29. For notational simplicity we avoid subscripts, and the involved variables are denoted by x, y, z, w. Each variable is assumed to be binary. We also use the symbol x1 instead of x ⫽ 1 and x0 instead of x ⫽ 0, and similarly for the rest of the variables. The Bayesian network is fully specified by the marginal probabilities of the root node (x) and the conditional probabilities shown in Figure 2.29. Note that only the values above the graph need to be specified. Those below the graph can be derived. Take, for example, the y node. P( y1) ⫽ P( y1|x1)P(x1) ⫹ P( y1|x0)P(x0) ⫽ (0.4)(0.6) ⫹ (0.3)(0.4) ⫽ 0.36 P( y0) ⫽ 1 ⫺ P( y1) ⫽ 0.64 Also, P(y0|x1) ⫽ 1 ⫺ P(y1|x1) The rest are similarly derived. Note that all of these parameters should be available prior to performing probability inference. Suppose now that: (a) x is measured and let its value be x1 (the evidence). We seek to compute P(z1|x1) and P(w0|x1). (b) w is measured and let its value be w1. We seek to compute P(x0|w1) and P(z1|w1). To answer (a), the following calculations are in order. P(z1|x1) ⫽ P(z1|y1, x1)P(y1|x1) ⫹ P(z1|y0, x1)P(y0|x1) ⫽ P(z1|y1)P(y1|x1) ⫹ P(z1|y0)P(y0|x1) ⫽ (0.25)(0.4) ⫹ (0.6)(0.6) ⫽ 0.46
(2.135)
Though not explicitly required, P(z0|x1) must also be evaluated, as we will soon realize. P(z0|x1) ⫽ 1 ⫺ P(z1|x1) ⫽ 0.54
P(x1) 5 0.60
(2.136)
P( y 1|x 1) 5 0.40 P(z 1|y 1) 5 0.25 P(w 1|z 1) 5 0.45 P( y 1|x 0) 5 0.30 P(z 1|y 0) 5 0.60 P(w 1|z 0) 5 0.30
P(x0)5 0.40
P( y 0|x 1) 5 0.60 P(z 0|y 1) 5 0.75 P(w 0|z 1) 5 0.55 P( y 0|x 0) 5 0.70 P(z 0|y 0) 5 0.40 P(w 0|z 0) 5 0.70 P( y 1) 5 0.36
P(z 1) 5 0.47
P(w 1) 5 0.37
P( y 0) 5 0.64
P(z 0) 5 0.53
P(w 0) 5 0.63
FIGURE 2.29 A simple Bayesian network where conditional dependencies are restricted to a single variable.
“04-Ch02-SA272” 18/9/2008 page 69
2.7 Bayesian Networks
In a similar way, we obtain P(w0|x1) ⫽ P(w0|z1, x1)P(z1|x1) ⫹ P(w0|z0, x1)P(z0|x1) ⫽ P(w0|z1)P(z1|x1) ⫹ P(w0|z0)P(z0|x1) ⫽ (0.55)(0.46) ⫹ (0.7)(0.54) ⫽ 0.63
(2.137)
We can think of the algorithm as a process that passes messages (i.e., probabilities) downward from one node to the next. The first two computations, (2.135) and (2.136), “are performed in node z” and then “passed” to the last node, where (2.137) is performed. To answer (b), the direction of “message propagation” is reversed since, now, the evidence is provided from node w and the required information, P(x0|w1), P(z1|w1) concerns nodes x and z, respectively. (0.45)(0.47) P(w1|z1)P(z1) ⫽ ⫽ 0.57 P(z1|w1) ⫽ P(w1) 0.37 The activity is then passed to node y, where the following needs to be performed. P(w1|y1)P(y1) P(y1|w1) ⫽ P(w1) P(w1|y1) is unknown and can be computed as discussed in the “downward” message propagation. That is, P(w1|y1) ⫽ P(w1|z1, y1)P(z1|y1) ⫹ P(w1|z0, y1)P(z0|y1) ⫽ P(w1|z1)P(z1|y1) ⫹ P(w1|z0)P(z0|y1) ⫽ (0.45)(0.25) ⫹ (0.3)(0.75) ⫽ 0.34 In a similar way, P(w1|y0) ⫽ 0.39 is obtained. These values are then “passed” over to node x, and it is left as an exercise to show that P(x0|w1) ⫽ 0.4. This idea can be carried out to any net of any size of the form given in Figure 2.29.
For Bayesian networks that have a tree structure,probability inference is achieved via a combination of downward and upward computations propagated through the tree. A number of algorithms have been proposed for the general case of Bayesian networks based on this “message-passing” philosophy. See, for example, [Pear 88, Laur 96]. For the case of singly connected graphs, these algorithms have complexity that is linear in the number of nodes. A singly connected graph is one that has no more than one path between any two nodes. For example, the graph in Figure 2.27 is not singly connected since there are two paths connecting x1 and x6 . An alternative approach to derive efficient algorithms for probability inference, which exploits the structure of the DAG, has been taken in [Li 94]. Although it is beyond our scope to focus on algorithmic details, it is quite instructive to highlight the basic idea around which this type of algorithm evolves. Let us take as an example the DAG shown in Figure 2.30,with nodes corresponding to the variables s, u, v, x, y, w, z with the joint probability P(s, u, v, x, y, w, z). This can be obtained, as we have already stated, as the product of all conditional
69
“04-Ch02-SA272” 18/9/2008 page 70
70
CHAPTER 2 Classifiers Based on Bayes Decision Theory
x u
y
v
w
z
s
FIGURE 2.30 A Bayesian network with a tree structure.
probabilities defining the network. Suppose one wishes to compute the conditional probability P(s|z ⫽ z0 ), where z ⫽ z0 is the evidence. From the Bayes rule we have P(s|z ⫽ z0 ) ⫽
P(s, z ⫽ z0 ) P(s, z ⫽ z0 ) ⫽ P(z ⫽ z0 ) s P(s, z ⫽ z0 )
(2.138)
To obtain P(s, z ⫽ z0 ), one has to marginalize (Appendix A) the joint probability over all possible values of u, v, x, y, w; that is, P(s, z ⫽ z0 ) ⫽
P(s, u, v, x, y, w, z ⫽ z0 )
(2.139)
u,v,x,y,w
Assuming, for simplicity, that each of the discrete variables can take, say, L values, the complexity of the previous computations amounts to L5 operations. For more variables and a large number of values, L, this can be a prohibitively large number. Let us now exploit the structure of the Bayesian network in order to reduce this computational burden. Taking into account the relations implied by the topology of the graph shown in Figure 2.30 and the Bayes chain rule in (2.128) (for probabilities), we obtain
P(s, u, v, x, y, w, z ⫽ z0 ) ⫽
u,v,x,y,w
P(s)P(u|s)P(v|s)P(w|v)P(x|u)P(y|u)P(z ⫽ z0 |y) ⫽
u,v,x,y,w
P(s)
P(u|s)P(v|s)
u,v
w
) )
P(w|v) *+ v
,)
x
P(x|u) *+ u
,)
y
P(y|u)P(z ⫽ z0 |y) *+ u
*+
(2.140)
, ,
s
or u,v,x,y,w
P(s, u, v, x, y, w, z ⫽ z0 ) ⫽ P(s)
u,v
P(u|s)P(v|s)1 (v)2 (u)3 (u) (2.141)
“04-Ch02-SA272” 18/9/2008 page 71
2.8 Problems
where the definitions of i (·), i ⫽ 1, 2, 3, are readily understood by inspection. Underbraces indicate what variable the result of each summation depends on. To obtain 3 (u) for each value of u, one needs to perform L operations (products and summations). Hence, a total number of L2 operations is needed to compute 3 (u) for all possible values of u. This is also true for the 2 (u), 1 (v). Thus, the total number of operations required to compute (2.141) is, after the factorization, of the order of L2 , instead of the order of L5 demanded for the brute-force computation in (2.139). This procedure could be viewed as an effort to decompose a “global” sum into products of “local” sums to make computations tractable. Each summation can be viewed as a processing stage that removes a variable and provides as output a function. The essence of the algorithm given in [Li 94] is to search for the factorization that requires the minimal number of operations. This algorithm also has linear complexity in the number of nodes for singly connected networks. In general, for multiply connected networks the probability inference problem is NP-hard [Coop 90]. In light of this result, one tries to seek approximate solutions, as in [Dagu 93]. Training: Training of a Bayesian network consists of two parts. The first is to learn the network topology. The topology can either be fixed by an expert who can provide knowledge about dependencies or by use of optimization techniques based on the training set. Once the topology has been fixed, the unknown parameters (i.e., conditional probabilities and marginal probabilities) are estimated from the available training data points. For example, the fraction (frequency) of the number of instances that an event occurs over the total number of trials performed is a way to approximate probabilities. In Bayesian networks, other refined techniques are usually encountered. A review of learning procedures can be found in [Heck 95]. For the reader who wishes to delve further into the exciting world of Bayesian networks, the books of [Pear 88, Neap 04, Jens 01] will prove indispensable tools.
2.8 PROBLEMS 2.1 Show that in a multiclass classification task, the Bayes decision rule minimizes the error probability. Hint: It is easier to work with the probability of correct decision. 2.2 In a two-class one-dimensional problem, the pdfs are the Gaussians N (0, 2 ) and N (1, 2 ) for the two classes, respectively. Show that the threshold x0 minimizing the average risk is equal to x0 ⫽ 1/2 ⫺ 2 ln
21 P(2 ) 12 P(1 )
where 11 ⫽ 22 ⫽ 0 has been assumed. 2.3 Consider a two equiprobable class problem with a loss matrix L. Show that if ⑀1 is the probability of error corresponding to feature vectors from class 1
71
“04-Ch02-SA272” 18/9/2008 page 72
72
CHAPTER 2 Classifiers Based on Bayes Decision Theory
and ⑀2 for those from class 2 , then the average risk r is given by r ⫽ P(1 )11 ⫹ P(2 )22 ⫹ P(1 )(12 ⫺ 11 )⑀1 ⫹ P(2 )(21 ⫺ 22 )⑀2
2.4 Show that in a multiclass problem with M classes the probability of classification error for the optimum classifier is bounded by Pe ⱕ
M ⫺1 M
Hint: Show first that for each x the maximum of P(i |x), i ⫽ 1, 2, . . . , M, is greater than or equal to 1/M. Equality holds if all P(i |x) are equal. 2.5 Consider a two (equiprobable) class, one-dimensional problem with samples distributed according to the Rayleigh pdf in each class, that is, p(x|i ) ⫽
⎧ ⎪ ⎨
x i2
exp
⎪ ⎩0
⫺x 2 2i2
xⱖ0 x⬍0
Compute the decision boundary point g(x) ⫽ 0. 2.6 In a two-class classification task, we constrain the error probability for one of the classes to be fixed, that is, ⑀1 ⫽ ⑀. Then show that minimizing the error probability of the other class results in the likelihood test decide x in 1 if
P(1 |x) ⬎ P(2 |x)
where is chosen so that the constraint is fulfilled. This is known as the Neyman–Pearson test, and it is similar to the Bayesian minimum risk rule. Hint: Use a Lagrange multiplier to show that this problem is equivalent to minimizing the quantity q ⫽ (⑀1 ⫺ ⑀) ⫹ ⑀2
2.7 In a three-class, two-dimensional problem the feature vectors in each class are normally distributed with covariance matrix 1.2 ⌺⫽ 0.4
0.4 1.8
The mean vectors for each class are [0.1, 0.1]T , [2.1, 1.9]T , [⫺1.5, 2.0]T . Assuming that the classes are equiprobable, (a) classify the feature vector [1.6, 1.5]T according to the Bayes minimum error probability classifier; (b) draw the curves of equal Mahalanobis distance from [2.1, 1.9]T . 2.8 In a two-class, three-dimensional classification problem, the feature vectors in each class are normally distributed with covariance matrix ⎡
0.3 ⎢ ⌺ ⫽ ⎣0.1 0.1
0.1 0.3 ⫺0.1
⎤ 0.1 ⎥ ⫺0.1⎦ 0.3
“04-Ch02-SA272” 18/9/2008 page 73
2.8 Problems
The respective mean vectors are [0, 0, 0]T and [0.5, 0.5, 0.5]T . Derive the corresponding linear discriminant functions and the equation describing the decision surface. 2.9 In a two equiprobable class classification problem, the feature vectors in each class are normally distributed with covariance matrix ⌺,and the corresponding mean vectors are 1 , 2 . Show that for the Bayesian minimum error classifier, the error probability is given by ⫹⬁ PB ⫽ (1/2)dm
1 exp(⫺z 2 /2) dz √ 2
where dm is the Mahalanobis distance between the mean vectors. Observe that this is a decreasing function of dm . Hint: Compute the log-likelihood ratio u ⫽ ln p(x|1 ) ⫺ ln p(x|2 ). Observe 2 , d 2 ) if that u is also a random variable normally distributed as N ((1/2)dm m 2 , d 2 ) if x ∈ . Use this information to compute x ∈ 1 and as N (⫺(1/2)dm 2 m the error probability. 2.10 Show that in the case in which the feature vectors follow Gaussian pdfs, the likelihood ratio test in (2.20) x ∈ 1 (2 )
if
l12 ≡
p(x|1 ) ⬎ (⬍) p(x|2 )
is equivalent to 2 2 dm (1 , x|⌺1 ) ⫺ dm (2 , x|⌺2 ) ⫹ ln
|⌺1 | ⬍ (⬎) ⫺ 2 ln |⌺2 |
where dm (i , x|⌺i ) is the Mahalanobis distance between i and x with respect to the ⌺⫺1 i norm. 2.11 If ⌺1 ⫽ ⌺2 ⫽ ⌺, show that the criterion of the previous problem becomes (1 ⫺ 2 )T ⌺⫺1 x ⬎ (⬍)⌰
where ⌰ ⫽ ln ⫹ 1/2(1 ⌺⫺1 ⫺ 2 ⌺⫺1 )
2.12 Consider a two-class, two-dimensional classification task, where the feature vectors in each of the classes 1 , 2 are distributed according to p(x|1 ) ⫽
1
1 T 2 exp ⫺ 2 (x ⫺ 1 ) (x ⫺ 1 ) 21 2
21 p(x|2 ) ⫽
1
1 T 2 exp ⫺ 2 (x ⫺ 2 ) (x ⫺ 2 ) 22 2
22
73
“04-Ch02-SA272” 18/9/2008 page 74
74
CHAPTER 2 Classifiers Based on Bayes Decision Theory
with 1 ⫽ [1, 1]T ,
2 ⫽ [1.5, 1.5]T ,
12 ⫽ 22 ⫽ 0.2
Assume that P(1 ) ⫽ P(2 ) and design a Bayesian classifier (a) that minimizes the error probability (b) that minimizes the average risk with loss matrix 0 ⌳⫽ 0.5
1 0
Using a pseudorandom number generator, produce 100 feature vectors from each class, according to the preceding pdfs. Use the classifiers designed to classify the generated vectors. What is the percentage error for each case? Repeat the experiments for 2 ⫽ [3.0, 3.0]T . 2.13 Repeat the preceding experiment if the feature vectors are distributed according to p(x|i ) ⫽
1 1 T ⫺1 (x ⫺ exp ⫺ ) ⌺ (x ⫺ ) i i 2|⌺|1/2 2
with ⌺⫽
1.01 0.2
0.2 1.01
and 1 ⫽ [1, 1]T , 2 ⫽ [1.5, 1.5]T . Hint: To generate the vectors, recall from [Papo 91, p. 144] that a linear transformation of Gaussian random vectors also results in Gaussian vectors. Note also that
1.01 0.2
0.2 1 ⫽ 1.01 0.1
0.1 1 1 0.1
0.1 1
2.14 Consider a two-class problem with normally distributed vectors with the same ⌺ in both classes. Show that the decision hyperplane at the point x 0 ,Eq. (2.46), is tangent to the constant Mahalanobis distance hyperellipsoids. Hint: (a) Compute the gradient of Mahalanobis distance with respect to x. (b) Recall from vector analysis that ⭸f⭸(xx) is normal to the tangent of the surface f (x) ⫽ constant. 2.15 Consider a two-class, one-dimensional problem with p(x|1 ) being N (, 2 ) and p(x|2 ) a uniform distributionbetween a and b. Show that the Bayesian a⫺ ⫺ G error probability is bounded by G b⫺ , where G(x) ≡ P(y ⱕ x) and y is N (0, 1). 2.16 Show that the mean value of the random vector
⭸ ln(p( x :)) ⭸
is zero.
“04-Ch02-SA272” 18/9/2008 page 75
2.8 Problems
2.17 In a heads or tails coin-tossing experiment the probability of occurrence of a head (1) is q and that of a tail (0) is 1 ⫺ q. Let xi , i ⫽ 1, 2, . . . , N , be the resulting experimental outcomes, xi ∈ {0, 1}. Show that the ML estimate of q is qML ⫽
N 1 xi N i⫽1
Hint: The likelihood function is P(X : q) ⫽
N
q xi (1 ⫺ q)(1⫺xi )
i⫽1
Then show that the ML results from the solution of the equation
q
i xi (1
⫺ q)(N ⫺
i xi )
i
xi
q
⫺
N ⫺ i xi ⫽0 1⫺q
2.18 The random variable x is normally distributed N (, 2 ),where is considered unknown. Given N measurements of the variable, compute the Cramer–Rao 2 bound ⫺E ⭸ ⭸L() (Appendix A). Compare the bound with the variance of 2 the resulting ML estimate of . Repeat this if the unknown parameter is the variance 2 . Comment on the results. 2.19 Show that if the likelihood function is Gaussian with unknowns the mean as well as the covariance matrix ⌺, then the ML estimates are given by ˆ ⫽
N 1 xk N k⫽1
N ˆ ⫽ 1 ⌺ (x k ⫺ )(x ˆ ˆ T k ⫺ ) N k⫽1
2.20 Prove that the covariance estimate ⌺ˆ ⫽
N 1 (x k ⫺ )(x ˆ ˆ T k ⫺ ) N ⫺1 k⫽1
is an unbiased one, where ˆ ⫽
N 1 xk N k⫽1
2.21 Prove that the ML estimates of the mean value and the covariance matrix (Problem 2.19) can be computed recursively, that is, ˆ N ⫹1 ⫽ ˆ N ⫹
1 (x N ⫹1 ⫺ ˆ N ) N ⫹1
75
“04-Ch02-SA272” 18/9/2008 page 76
76
CHAPTER 2 Classifiers Based on Bayes Decision Theory
and ⌺ˆ N ⫹1 ⫽
N ˆ N ⌺N ⫹ (x N ⫹1 ⫺ ˆ N )(x N ⫹1 ⫺ ˆ N )T N ⫹1 (N ⫹ 1)2
where the subscript in the notation of the estimates, ˆ N , ⌺ˆ N indicates the number of samples used for their computation. 2.22 The random variable x follows the Erlang pdf p(x; ) ⫽ 2 x exp(⫺x)u(x)
where u(x) is the unit-step function, u(x) ⫽
. 1
if x ⬎ 0
0
if x ⬍ 0
Show that the maximum likelihood estimate of , given N measurements, x1 , . . . , xN , of x, is 2N ˆ ML ⫽ N
k⫽1 xk
2.23 In the ML estimation, the zero of the derivative of the log pdf derivative was computed. Using a multivariate Gaussian pdf, show that this corresponds to a maximum and not to a minimum. 2.24 Prove that the sum z ⫽ x ⫹ y of two independent random variables, x and y, where x ∼ N (x , x2 ) and y ∼ N (y , y2 ), is also a Gaussian one with mean value and variance equal to x ⫹ y and x2 ⫹ y2 , respectively. 2.25 Show relations (2.74) and (2.75). Then show that p(x|X) is also normal with mean N and variance 2 ⫹ N2 . Comment on the result. 2.26 Show that the posterior pdf estimate in the Bayesian inference task, for independent variables, can be computed recursively, that is, p(|x 1 , . . . , x N ) ⫽
p(x N |)p( |x 1 , . . . , x N ⫺1 ) p(x N |x 1 , . . . , x N ⫺1 )
2.27 Show Eqs. (2.76)–(2.79). 2.28 The random variable x is normally distributed as N (, 2 ), with being the unknown parameter described by the Rayleigh pdf p() ⫽
exp(⫺2 /22 ) 2
Show that the maximum a posteriori probability estimate of is given by ˆ MAP
( Z 4R 1⫹ 1⫹ 2 ⫽ 2R Z
“04-Ch02-SA272” 18/9/2008 page 77
2.8 Problems
where Z⫽
N 1 xk , 2
R⫽
k⫽1
N 1 ⫹ 2 2
2.29 Show that for the lognormal distribution p(x) ⫽
(ln x ⫺ )2 1 exp ⫺ , √ 2 2 x 2
x⬎0
the ML estimate is given by N 1 ˆ ML ⫽ ln xk N k⫽1
2.30 Show that if the mean value and the variance of a random variable are known, that is, ⫹⬁ ⫽ xp(x) dx, ⫺⬁
⫹⬁ ⫽ (x ⫺ )2 p(x) dx 2
⫺⬁
the maximum entropy estimate of the pdf is the Gaussian N (, 2 ). 2.31 Show Eqs. (2.98), (2.99), and (2.100). Hint: For the latter, note that the probabilities add to one; thus a Lagrangian multiplier must be used. 2.32 Let P be the probability of a random point x being located in a certain interval h. Given N of these points, the probability of having k of them inside h is given by the binomial distribution prob{k} ⫽
N! P k (1 ⫺ P)N ⫺k k!(N ⫺ k)!
Show that E[k/N ] ⫽ P and that the variance around the mean is 2 ⫽ E[(k/N ⫺ P)2 ] ⫽ P(1 ⫺ P)/N . That is, the probability estimator P ⫽ k/N is unbiased and asymptotically consistent. 2.33 Consider three Gaussian pdfs: N (1.0, 0.1), N (3.0, 0.1), and N (2.0, 0.2). Generate 500 samples according to the following rule. The first two samples are generated from the second Gaussian, the third sample from the first one, and the fourth sample from the last Gaussian. This rule repeats until all 500 samples have been generated. The pdf underlying the random samples is modeled as a mixture 3
N (i , i2 )Pi
i⫽1
Use the EM algorithm and the generated samples to estimate the unknown parameters i , i2 , Pi .
77
“04-Ch02-SA272” 18/9/2008 page 78
78
CHAPTER 2 Classifiers Based on Bayes Decision Theory
2.34 Consider two classes 1 , 2 in the two-dimensional space. The data from class 1 are uniformly distributed inside a circle of radius r. The data of class 2 are also uniformly distributed inside another circle of radius r. The distance between the centers of the circles is greater than 4r. Let N be the number of the available training samples. Show that the probability of error of the NN classifier is always smaller than that of the kNN, for any k ⱖ 3. 2.35 Generate 50 feature vectors for each of the two classes of Problem 2.12, and use them as training points. In the sequel, generate 100 vectors from each class and classify them according to the NN and 3NN rules. Compute the classification error percentages. 2.36 The pdf of a random variable is given by p(x) ⫽
⎧ ⎨1
for 0 ⬍ x ⬍ 2
⎩0
otherwise
2
Use the Parzen window method to approximate it using as the kernel function the Gaussian N (0, 1). Choose the smoothing parameter to be (a) h ⫽ 0.05 and (b) h ⫽ 0.2. For each case, plot the approximation based on N ⫽ 32, N ⫽ 256, and N ⫽ 5000 points, which are generated from a pseudorandom generator according to p(x). 2.37 Repeat the preceding problem by generating N ⫽ 5000 points and using k nearest neighbor estimation with k ⫽ 32, 64, and 256, respectively. 2.38 Show that the variance N2 (x) of the pdf estimate, given by Eq. (2.110), is upper bounded by: N2 (x) ⱕ
ˆ sup()E[p(x)] Nhl
where sup(·) is the supremum of the associated function. Observe that for large values of h the variance is small. On the other hand, we can make the variance small for small values of h, provided N tends to infinity and if, also, the product Nhl tends to infinity. 2.39 Recall Equation (2.128) p(x) ⫽ p(x1 )
l
p(xi |Ai )
i⫽2
Assume l ⫽ 6 and p(x6 |x5 , . . . , x1 ) ⫽ p(x6 |x5 , x1 )
(2.142)
p(x5 |x4 , . . . , x1 ) ⫽ p(x5 |x4 , x3 )
(2.143)
p(x4 |x3 , x2 , x1 ) ⫽ p(x4 |x3 , x2 , x1 ) p(x3 |x2 , x1 ) ⫽ p(x3 )
(2.144) (2.145)
“04-Ch02-SA272” 18/9/2008 page 79
MATLAB Programs and Exercises
p(x2 |x1 ) ⫽ p(x2 )
(2.146)
Write the respective sets Ai , i ⫽ 1, 2, . . . , 6, and construct the corresponding DAG. 2.40 In the DAG defined in Figure 2.29, assume that the variable z is measured to be z0. Compute P(x1|z0) and P(w0|z0). 2.41 In the example associated with the tree-structured DAG of Figure 2.28,assume that the patient undergoes the medical test H1 and that this turns out to be positive (True). Based on this test, compute the probability that the patient has developed cancer. In other words, compute the conditional probability P(C ⫽ True|H1 ⫽ True).
MATLAB PROGRAMS AND EXERCISES Computer Exercises A number of MATLAB functions are provided,which will help the interested reader to experiment on some of the most important issues discussed in the present chapter. Needless to say that there may be other implementations of these functions. Short comments are also given along with the code. In addition,we have used the symbols m and S to denote the mean vector (given as a column vector) and the covariance matrix, respectively, instead of the symbols and ⌺, which are used in the text. In the following, unless otherwise stated, each class is represented by an integer in {1, . . . , c} where c is the number of classes. 2.1 Gaussian generator. Generate N l-dimensional vectors from a Gaussian distribution with mean m and covariance matrix S, using the mvnrnd MATLAB function.
Solution Just type mvnrnd(m,S,N)
2.2 Gaussian function evaluation. Write a MATLAB function that computes the value of the Gaussian distribution N (m, S), at a given vector x.
Solution function z=comp_gauss_dens_val(m,S,x) [l,q]=size(m); % l=dimensionality z=(1/((2*pi)^ (l/2)*det(S)^ 0.5) )... *exp(-0.5*(x-m)'*inv(S)*(x-m));
79
“04-Ch02-SA272” 18/9/2008 page 80
80
CHAPTER 2 Classifiers Based on Bayes Decision Theory
2.3 Data set generation from Gaussian classes. Write a MATLAB function that generates a data set of N l-dimensional vectors that stem from c different Gaussian distributions N (mi , Si ), with corresponding a priori probabilities Pi , i ⫽ 1, . . . , c.
Solution In the sequel: ■ m is an l ⫻ c matrix, the i-th column of which is the mean vector of the i-th class distribution. ■
S is an l ⫻ l ⫻ c (three-dimensional) matrix, whose ith two-dimensional l ⫻ l component is the covariance of the distribution of the ith class. In MATLAB S(:, :, i) denotes the i-th two-dimensional l ⫻ l matrix of S.
P is the c dimensional vector that contains the a priori probabilities of the classes. mi , Si , Pi , and c are provided as inputs. The following function returns: ■ A matrix X with (approximately) N columns, each column of which is an l-dimensional data vector. ■
■
A row vector y whose ith entry denotes the class from which the ith data vector stems.
function [X,y]=generate_gauss_classes(m,S,P,N) [l,c]=size(m); X=[]; y=[]; for j=1:c % Generating the [p(j)*N)] vectors from each distribution t=mvnrnd(m(:,j),S(:,:,j),fix(P(j)*N)); % The total number of points may be slightly less than N % due to the fix operator X=[X t]; y=[y ones(1,fix(P(j)*N))*j]; end
2.4 Plot of data. Write a MATLAB function that takes as inputs: (a) a matrix X and a vector y defined as in the previous function, (b) the mean vectors of c class distributions. It plots: (a) the data vectors of X using a different color for each class, (b) the mean vectors of the class distributions. It is assumed that the data live in the two-dimensional space.
Solution % CAUTION: This function can handle up to % six different classes
“04-Ch02-SA272” 18/9/2008 page 81
MATLAB Programs and Exercises
function plot_data(X,y,m) [l,N]=size(X); % N=no. of data vectors, l=dimensionality [l,c]=size(m); % c=no. of classes if(l ~ =2) fprintf('NO PLOT CAN BE GENERATED\n') return else pale=['r.'; 'g.'; 'b.'; 'y.'; 'm.'; 'c.']; figure(1) % Plot of the data vectors hold on for i=1:N plot(X(1,i),X(2,i),pale(y(i),:)) end % Plot of the class means for j=1:c plot(m(1,j),m(2,j),'k+') end end
2.5 Bayesian classifier ( for Gaussian Processes). Write a MATLAB function that will take as inputs: (a) the mean vectors, (b) the covariance matrices of the class distributions of a c-class problem, (c) the a priori probabilities of the c classes,and (d) a matrix X containing column vectors that stem from the above classes. It will give as output an N -dimensional vector whose ith component contains the class where the corresponding vector is assigned,according to the Bayesian classification rule.
Solution Caution: While inserting the following function, do not type the labels (A), (B) and (C). They are used to serve as references, as we will see later on. (A) function z=bayes_classifier(m,S,P,X) [l,c]=size(m); % l=dimensionality, c=no. of classes [l,N]=size(X); % N=no. of vectors for i=1:N for j=1:c (B) t(j)=P(j)*comp_gauss_dens_val(m(:,j),... S(:,:,j),X(:,i)); end % Determining the maximum quantity Pi*p(x|wi) (C) [num,z(i)]=max(t); end
81
“04-Ch02-SA272” 18/9/2008 page 82
82
CHAPTER 2 Classifiers Based on Bayes Decision Theory
2.6 Euclidean distance classifier. Write a MATLAB function that will take as inputs: (a) the mean vectors, and (b) a matrix X containing column vectors that stem from the above classes. It will give as output an N -dimensional vector whose ith component contains the class where the corresponding vector is assigned, according to the minimum Euclidean distance classifier.
Solution The requested function may be obtained by the bayes_classifier function by replacing (A), (B), and (C) with ■
function z=euclidean_classifier(m,X)
■
t(j)=sqrt((X(:,i)-m(:,j))'*(X(:,i)-m(:,j)));
(computation of the Euclidean distances from all class representatives) ■
[num,z(i)]=min(t);
(determination of the closest class mean), respectively. 2.7 Mahalanobis distance classifier. Write a MATLAB function that will take as inputs: (a) the mean vectors,(b) the covariance matrix of the class distributions of a c-class problem, and (c) a matrix X containing column vectors that stem from the above classes. It will give as output an N -dimensional vector whose ith component contains the class where the corresponding vector is assigned according to the minimum Mahalanobis distance classifier.
Solution The requested function may be obtained by the bayes_classifier function by replacing (A), (B) and (C) with ■
function z=mahalanobis_classifier(m,S,X)
■
t(j)=sqrt((X(:,i)-m(:,j))'*inv(S(:,:,j))*... (X(:,i)-m(:,j)));
(computation of the Mahalanobis distances from all class representatives) ■
[num,z(i)]=min(t);
(determination of the closest class mean), respectively. 2.8 k-nearest neighbor classifier. Write a MATLAB function that takes as inputs: (a) a set of N1 vectors packed as columns of a matrix Z, (b) an N1 -dimensional vector containing the classes where each vector in Z belongs, (c) the value for the parameter k of the classifier,(d) a set of N vectors packed as columns in the
“04-Ch02-SA272” 18/9/2008 page 83
MATLAB Programs and Exercises
matrix X. It returns an N -dimensional vector whose ith component contains the class where the corresponding vector of X is assigned, according to the k-nearest neighbor classifier.
Solution function z=k_nn_classifier(Z,v,k,X) [l,N1]=size(Z); [l,N]=size(X); c=max(v); % The number of classes % Computation of the (squared) Euclidean distance % of a point from each reference vector for i=1:N dist=sum((X(:,i)*ones(1,N1)-Z).^ 2); %Sorting the above distances in ascending order [sorted,nearest]=sort(dist); % Counting the class occurrences among the k-closest % reference vectors Z(:,i) refe=zeros(1,c); %Counting the reference vectors per class for q=1:k class=v(nearest(q)); refe(class)=refe(class)+1; end [val,z(i)]=max(refe); end
2.9 Classification error evaluation. Write a MATLAB function that will take as inputs: (a) an N -dimensional vector, each component of which contains the class where the corresponding data vector belongs and (b) a similar N dimensional vector each component of which contains the class where the corresponding data vector is assigned from a certain classifier. Its output will be the percentage of the places where the two vectors differ (i.e., the classification error of the classifier).
Solution function clas_error=compute_error(y,y_est) [q,N]=size(y); % N= no. of vectors c=max(y); % Determining the number of classes clas_error=0; % Counting the misclassified vectors for i=1:N if(y(i)~ =y_est(i)) clas_error=clas_error+1; end end
83
“04-Ch02-SA272” 18/9/2008 page 84
84
CHAPTER 2 Classifiers Based on Bayes Decision Theory
% Computing the classification error clas_error=clas_error/N;
Computer Experiments Notes: In the sequel, it is advisable to use the command randn(‘seed’,0)
before generating the data sets, in order to initialize the Gaussian random number generator to 0 (or any other fixed number). This is important for the reproducibility of the results. 2.1 a. Generate and plot a data set of N ⫽ 1,000 two-dimensional vectors that stem from three equiprobable classes modeled by normal distributions T , m ⫽ [7, 7]T , m ⫽ [15, 1]T and with mean vectors m1 ⫽&[1, 1]' 2 & 3& ' ' 12 0 8 3 2 0 ,S ⫽ ,S ⫽ . covariance matrices S1 ⫽ 0 1 2 3 2 3 0 2 b. Repeat (a) when the a priori probabilities of the classes are given by the vector P ⫽ [0.6, 0.3, 0.1]T .
Solution Figure (2.31)a–b display the vectors from each class. Note the “shape” of the clusters formed by the vectors of each class. This is directly affected by the corresponding covariance matrix. Also note that, in the first case, each class has roughly the same number of the vectors, while in the latter case,
x2
x2
14
14
10
10
6
6
2
2
⫺2
⫺2
⫺6 ⫺5
0
5 (a)
10
15
x1
⫺6
⫺5
0
5 (b)
10
15
FIGURE 2.31 (a) The equiprobable classes case. (b) The case where the a-priori probabilities differ.
x1
“04-Ch02-SA272” 18/9/2008 page 85
MATLAB Programs and Exercises
the leftmost and the rightmost classes are more “dense” and more “sparse” compared to the previous case, respectively. 2.2 a. Generate a data set X1 of N ⫽ 1,000 two-dimensional vectors that stem from three equiprobable classes modeled by normal distributions with mean vectors m1 ⫽ [1, 1]T , m2 ⫽ [12, 8]T , m3 ⫽ [16,1]T and covariance matrices S1 ⫽ S2 ⫽ S3 ⫽ 4I , where I is the 2 ⫻ 2 identity matrix. b. Apply the Bayesian, the Euclidean, and the Mahalanobis classifiers on X1 . c. Compute the classification error for each classifier. 2.3 a. Generate a data set X2 of N ⫽ 1,000 two-dimensional vectors that stem from three equiprobable classes modeled by normal distributions with mean vectors m1 ⫽ [1, 1]&T , m2'⫽ [14, 7]T , m3 ⫽ [16, 1]T and covariance 5 3 . matrices S1 ⫽ S2 ⫽ S3 ⫽ 3 4 (b)–(c) Repeat steps b) and (c) of experiment 2.2, for X2 . 2.4 a. Generate a data set X3 of N ⫽ 1,000 two-dimensional vectors that stem from three equiprobable classes modeled by normal distributions with mean vectors m1 ⫽ [1, 1]T , m2 ⫽ [8, 6]T , m3 ⫽ [13, 1]T and covariance matrices S1 ⫽ S2 ⫽ S3 ⫽ 6I , where I is the 2 ⫻ 2 identity matrix. (b)–(c) Repeat (b) and (c) from experiment 2.2, for X3 . 2.5 a. Generate a data set X4 of N ⫽ 1,000 two-dimensional vectors that stem from three equiprobable classes modeled by normal distributions with T , m ⫽ [10, 5]T , m ⫽ [11, 1]T and covariance mean vectors m1 ⫽ [1, 1]& 2 ' 3 7 4 . matrices S1 ⫽ S2 ⫽ S3 ⫽ 4 5 (b)–(c) Repeat steps (b) and (c) of experiment 2.2, for X4 . 2.6 Study carefully the results obtained by experiments (2.2)–(2.5) and draw your conclusions. 2.7 a. Generate two data sets X5 and X5⬘ of N ⫽ 1,000 two-dimensional vectors each that stem from three classes modeled by normal distributions with mean vectors m1 ⫽ [1, 1]T , m2 ⫽ [4, 4]T , m3 ⫽ [8, 1]T and covariance matrices S1 ⫽ S2 ⫽ S3 ⫽ 2I . In the generation of X5 , the classes are assumed to be equiprobable, while in the generation of X5⬘ , the a priori probabilities of the classes are given by the vector P ⫽ [0.8, 0.1, 0.1]T . b. Apply the Bayesian and the Euclidean classifiers on both X5 and X5⬘ . c. Compute the classification error for each classifier for both data sets and draw your conclusions.
85
“04-Ch02-SA272” 18/9/2008 page 86
86
CHAPTER 2 Classifiers Based on Bayes Decision Theory
2.8 Consider the data set X3 (from experiment (2.4)). Using the same settings, generate a data set Z, where the class from which a data vector stems is known. Apply the k nearest neighbor classifier on X3 for k ⫽ 1 and k ⫽ 11 using Z as the training set and draw your conclusions.
REFERENCES [Aebe 94] Aeberhard S., Coomans D., Devel O. “Comparative analysis of statistical pattern recognition methods in high dimensional setting,” Pattern Recognition,Vol. 27(8), pp. 1065–1077, 1994. [Babi 96] Babich G.A., Camps O.I. “Weighted Parzen windows for pattern classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 18(5), pp. 567–570, 1996. [Bell 61] Bellman R. Adaptive Control Processes: A Guided Tour, Princeton University Press, 1961. [Bern 94] Bernardo J.M., Smith A.F.M Bayesian Theory, John Wiley, 1994. [Bish 06] Bishop C.M. Pattern Recognition and Machine Learning, Springer, 2006. [Boyl 83] Boyles R.A. “On the convergence of the EM algorithm,”J. Royal Statistical Society B,Vol. 45(1), pp. 47–55, 1983. [Brei 77] Breiman L., Meisel W., Purcell E. “Variable kernel estimates of multivariate densities,” Technometrics,Vol. 19(2), pp. 135–144, 1977. [Brod 90] Broder A. “Strategies for efficient incremental nearest neighbor search,” Pattern Recognition,Vol. 23, pp. 171–178, 1990. [Butu 93] Buturovic L.J. “Improving k-nearest neighbor density and error estimates,” Pattern Recognition,Vol. 26(4), pp. 611–616, 1993. [Coop 90] Cooper G.F. “The computational complexity of probabilistic inference using Bayesian belief networks,”Artifical Intelligence,Vol. 42, pp. 393–405, 1990. [Cram 46] Cramer H. Mathematical Methods of Statistics, Princeton University Press, 1941. [Dagu 93] Dagum P., Chavez R.M. “Approximating probabilistic inference in Bayesian belief networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15(3), pp. 246–255, 1993. [Dasa 91] Dasarasthy B. Nearest Neighbor Pattern Classification Techniques, IEEE Computer Society Press, 1991. [Demp 77] Dempster A.P., Laird N.M., Rubin D.B. “Maximum likelihood from incomplete data via the EM algorithm,” J. Royal Statistical Society,Vol. 39(1), pp. 1–38, 1977. [Devr 96] Devroye L.,Gyorfi L.,Lugosi G. A Probabilistic Theory of Pattern Recognition,SpringerVerlag, 1996. [Djou 97] Djouadi A., Bouktache E. “A fast algorithm for the nearest neighbor classifier,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19(3), pp. 277–282, 1997. [Dome 05] Domeniconi C., Gunopoulos D., Peng J. “Large margin nearest neighbor classifiers,” IEEE Transactions on Neural Networks,Vol. 16(4), pp. 899–909, 2005.
“04-Ch02-SA272” 18/9/2008 page 87
References
[Domi 97] Domingos P., Pazzani M. “Beyond independence: Conditions for the optimality of the simple Bayesian classifier,” Machine Learning,Vol. 29, pp. 103–130, 1997. [Duda 73] Duda R., Hart P.E. Pattern Classification and Scene Analysis, John Wiley & Sons, 1973. [Frie 94] Friedman J.H.“Flexible metric nearest neighbor classification,”Technical Report, Department of Statistics, Stanford University, 1994. [Frie 89] Friedman J.H. “Regularized discriminant analysis,” Journal of American Statistical Association,Vol. 84(405), pp. 165–175, 1989. [Frie 97] Friedman N.,Geiger D.,Goldszmidt M.“Bayesian network classifiers,”Machine Learning, Vol. 29, pp. 131–163, 1997. [Fuku 75] Fukunaga F., Narendra P.M. “A branch and bound algorithm for computing k-nearest neighbors,” IEEE Transactions on Computers,Vol. 24, pp. 750–753, 1975. [Fuku 90] Fukunaga F. Introduction to Statistical Pattern Recognition, 2nd ed.,Academic Press, 1990. [Hast 96] Hastie T., Tibshirani R. “Discriminant adaptive nearest neighbor classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 18(6), pp. 607–616, 1996. [Hast 01] Hastie T.,Tibshirani R., Friedman J. The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer, 2001. [Hatt 00] Hattori K.,Takahashi M.“A new edited k-nearest neighbor rule in the pattern classification problem,” Pattern Recognition,Vol. 33, pp. 521–528, 2000. [Heck 95] Heckerman D. “A tutorial on learning Bayesian networks,” Technical Report #MSR-TR95-06, Microsoft Research, Redmond,Washington, 1995. [Hoff 96] Hoffbeck J.P., Landgrebe D.A. “Covariance matrix estimation and classification with limited training data,”IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 18(7), pp. 763–767, 1996. [Huan 02] Huang Y.S., Chiang C.C., Shieh J.W., Grimson E. “Prototype optimization for nearestneighbor classification,” Pattern Recognition,Vol. 35, pp. 1237–1245, 2002. [Jayn 82] Jaynes E.T. “On the rationale of the maximum entropy methods,” Proceedings of the IEEE,Vol. 70(9), pp. 939–952, 1982. [Jens 01] Jensen F.V. Bayesian Networks and Decision Graphs, Springer, 2001. [Jone 96] Jones M.C., Marron J.S., Seather S.J. “A brief survey of bandwidth selection for density estimation,” Journal of the American Statistical Association,Vol. 91, pp. 401–407, 1996. [Kimu 87] Kimura F.,Takashina K.,Tsuruoka S., Miyake Y. “Modified quadratic discriminant functions and the application to Chinese character recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 9(1), pp. 149–153, 1987. [Kris 00] Krishna K.,Thathachar M.A.L., Ramakrishnan K.R. “Voronoi networks and their probability of misclassification,”IEEE Transactions on Neural Networks,Vol. 11(6), pp. 1361–1372, 2000. [Krzy 83] Krzyzak A. “Classification procedures using multivariate variable kernel density estimate,” Pattern Recognition Letters,Vol. 1, pp. 293–298, 1983. [Laur 96] Lauritzen S.L. Graphical Models, Oxford University Press, 1996. [Li 94] Li Z., D’Abrosio B. “Efficient inference in Bayes’ networks as a combinatorial optimization problem,” International Journal of Approximate Inference,Vol. 11, 1994.
87
“04-Ch02-SA272” 18/9/2008 page 88
88
CHAPTER 2 Classifiers Based on Bayes Decision Theory
[Liu 04] Liu C.-L., Sako H., Fusisawa H. “Discriminative learning quadratic discriminant function for handwriting recognition,” IEEE Transactions on Neural Networks, Vol. 15(2), pp. 430–444, 2004. [McLa 88] McLachlan G.J.,Basford K.A. Mixture Models: Inference and Applications to Clustering, Marcel Dekker, 1988. [McNa 01] McNames J. “A Fast nearest neighbor algorithm based on principal axis search tree,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23(9), pp. 964–976, 2001. [Mico 94] Mico M.L., Oncina J., Vidal E. “A new version of the nearest neighbor approximating and eliminating search algorithm (AESA) with linear preprocessing time and memory requirements,” Pattern Recognition Letters,Vol. 15, pp. 9–17, 1994. [Moon 96] Moon T. “The expectation maximization algorithm,” Signal Processing Magazine, Vol. 13(6), pp. 47–60, 1996. [Neap 04] Neapolitan R.D. Learning Bayesian Networks, Prentice Hall, 2004. [Nene 97] Nene S.A., Nayar S.K. “A simple algorithm for nearest neighbor search in high dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19(9), pp. 989–1003, 1997. [Papo 91] Papoulis A. Probability Random Variables and Stochastic Processes, 3rd ed., McGrawHill 1991. [Pare 06] Paredes R., Vidal E. “Learning weighted metrics to minimize nearest neighbor classification error,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28(7), pp. 1100–1111, 2006. [Pare 06a] Paredes R., Vidal E. “Learning prototypes and distances: A prototype reduction technique based on nearest neighbor error minimization,” Pattern Recognition, Vol. 39, pp. 180–188, 2006. [Parz 62] Parzen E. “On the estimation of a probability density function and mode,” Ann. Math. Stat.Vol. 33, pp. 1065–1076, 1962. [Pear 88] Pearl J. Probabilistic Reasoning in Intelligent Systems, Morgan Kaufmann, 1988. [Redn 84] Redner R.A.,Walker H.F.“Mixture densities,maximum likelihood and the EM algorithm,” SIAM Review,Vol. 26(2), pp. 195–239, 1984. [Roos 05] Roos T., Wettig H., Grunwald P., Myllymaki P., Tirri H. “On discriminative Bayesian network classifiers and logistic regression,” Machine Learning,Vol. 59, pp. 267–296, 2005. [Same 08] Samet H. “k-Nearest neighbor finding using MaxNearestDist,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 30(2), pp. 243–252, 2008. [Terr 92] Terrell G.R., Scott D.W. “Variable kernel density estimation,” Annals of Statistics, Vol. 20(3), pp. 1236–1265, 1992. [Titt 85] Titterington D.M., Smith A.F.M., Makov U.A. Statistical Analysis of Finite Mixture Distributions, John Wiley & Sons, 1985. [Vida 94] Vidal E. “New formulation and improvements of the nearest neighbor approximating and eliminating search algorithm (AESA),” Pattern Recognition Letters, Vol. 15, pp. 1–7, 1994. [Wand 95] Wand M., Jones M. Kernel Smoothing, Chapman & Hall, London, 1995. [Webb 05] Webb G.I., Boughton J.R., Wang Z. “Not so naive Bayes: Aggregating one dependence estimators,” Machine Learning,Vol. 58, pp. 5–24, 2005.
“04-Ch02-SA272” 18/9/2008 page 89
References
[Wils 72] Wilson D.L. “Asymptotic properties of NN rules using edited data,” IEEE Transactions on Systems, Man, and Cybernetics,Vol. 2, pp. 408–421, 1972. [Wu 83] Wu C. “On the convergence properties of the EM algorithm,” Annals of Statistics, Vol. 11(1), pp. 95–103, 1983. [Yan 93] Yan H. “Prototype optimization for nearest neighbor classifiers using a two layer perceptron,” Pattern Recognition,Vol. 26(2), pp. 317–324, 1993.
89
“05-Ch03-SA272” 17/9/2008 page 91
CHAPTER
3
Linear Classifiers
3.1 INTRODUCTION Our major concern in Chapter 2 was to design classifiers based on probability density or probability functions. In some cases, we saw that the resulting classifiers were equivalent to a set of linear discriminant functions. In this chapter, we will focus on the design of linear classifiers, regardless of the underlying distributions describing the training data. The major advantage of linear classifiers is their simplicity and computational attractiveness. The chapter starts with the assumption that all feature vectors from the available classes can be classified correctly using a linear classifier, and we will develop techniques for the computation of the corresponding linear functions. In the sequel we will focus on a more general problem, in which a linear classifier cannot correctly classify all vectors, yet we will seek ways to design an optimal linear classifier by adopting an appropriate optimality criterion.
3.2 LINEAR DISCRIMINANT FUNCTIONS AND DECISION HYPERPLANES Let us once more focus on the two-class case and consider linear discriminant functions. Then the respective decision hypersurface in the l-dimensional feature space is a hyperplane, that is g(x) ⫽ w T x ⫹ w0 ⫽ 0
(3.1)
where w ⫽ [w1 , w2 , . . . , wl ]T is known as the weight vector and w0 as the threshold. If x 1 , x 2 are two points on the decision hyperplane, then the following is valid 0 ⫽ w T x 1 ⫹ w0 ⫽ w T x 2 ⫹ w0 w T (x 1 ⫺ x 2 ) ⫽ 0
⇒ (3.2)
91
“05-Ch03-SA272” 17/9/2008 page 92
92
CHAPTER 3 Linear Classifiers
Since the difference vector x 1 ⫺ x 2 obviously lies on the decision hyperplane (for any x 1 , x 2 ), it is apparent from Eq. (3.2) that the vector w is orthogonal to the decision hyperplane. Figure 3.1 shows the corresponding geometry (for w1 ⬎ 0, w2 ⬎ 0, w0 ⬍ 0). Recalling our high school math, it is easy to see that the quantities entering in the figure are given by |w0 | d⫽ w12 ⫹ w22
(3.3)
|g(x)| z⫽ w12 ⫹ w22
(3.4)
and
In other words, |g(x)| is a measure of the Euclidean distance of the point x from the decision hyperplane. On one side of the plane g(x) takes positive values and on the other negative. In the special case that w0 ⫽ 0,the hyperplane passes through the origin.
x2
2
w0 w2
x wT5[w1 , w2] z
w
d 2
w0 w1 2
x1 1
FIGURE 3.1 Geometry for the decision line. On one side of the line it is g(x) ⬎ 0(⫹) and on the other g(x) ⬍ 0(⫺).
“05-Ch03-SA272” 17/9/2008 page 93
3.3 The Perceptron Algorithm
3.3 THE PERCEPTRON ALGORITHM Our major concern now is to compute the unknown parameters wi , i ⫽ 0, . . . , l, defining the decision hyperplane. In this section, we assume that the two classes 1 , 2 are linearly separable. In other words, we assume that there exists a hyperplane, defined by w ∗T x ⫽ 0, such that w ∗T x ⬎ 0
᭙x ∈ 1
w ∗T x
᭙x ∈ 2
⬍0
(3.5)
The formulation above also covers the case of a hyperplane not crossing the origin, that is, w ∗T x ⫹ w0∗ ⫽ 0, since this can be brought into the previous formulation by defining the extended (l ⫹ 1)-dimensional vectors x⬘ ≡ [x T , 1]T , w⬘ ≡ [w ∗T , w0∗ ]T . Then w ∗T x ⫹ w0∗ ⫽ w⬘T x⬘. We will approach the problem as a typical optimization task (Appendix C). Thus we need to adopt (a) an appropriate cost function and (b) an algorithmic scheme to optimize it. To this end, we choose the perceptron cost defined as J (w) ⫽
(␦x w T x)
(3.6)
x ∈Y
where Y is the subset of the training vectors, which are misclassified by the hyperplane defined by the weight vector w. The variable ␦x is chosen so that ␦x ⫽ ⫺1 if x ∈ 1 and ␦x ⫽ ⫹1 if x ∈ 2 . Obviously, the sum in (3.6) is always positive, and it becomes zero when Y becomes the empty set, that is, if there are not misclassified vectors x. Indeed, if x ∈ 1 and it is misclassified, then w T x ⬍ 0 and ␦x ⬍ 0, and the product is positive. The result is the same for vectors originating from class 2 . When the cost function takes its minimum value, 0, a solution has been obtained, since all training feature vectors are correctly classified. The perceptron cost function in (3.6) is continuous and piecewise linear. Indeed, if we change the weight vector smoothly, the cost J (w) changes linearly until the point at which there is a change in the number of misclassified vectors ( Problem 3.1). At these points the gradient is not defined, and the gradient function is discontinuous. To derive the algorithm for the iterative minimization of the cost function, we will adopt an iterative scheme in the spirit of the gradient descent method (Appendix C), that is, w(t ⫹ 1) ⫽ w(t) ⫺ t
⭸J (w) ⭸w w⫽w(t)
(3.7)
where w(t) is the weight vector estimate at the tth iteration step and t is a sequence of positive real numbers. However, we must be careful here. This is not
93
“05-Ch03-SA272” 17/9/2008 page 94
94
CHAPTER 3 Linear Classifiers
defined at the points of discontinuity. From the definition in (3.6), and at the points where this is valid, we get ⭸J (w) ⫽ ␦x x ⭸w
(3.8)
x ∈Y
Substituting (3.8) into (3.7), we obtain
w(t ⫹ 1) ⫽ w(t) ⫺ t
␦x x
(3.9)
x ∈Y
The algorithm is known as the perceptron algorithm and is quite simple in its structure. Note that Eq. (3.9) is defined at all points. The algorithm is initialized from an arbitrary weight vector w(0), and the correction vector x ∈Y ␦x x is formed using the misclassified features. The weight vector is then corrected according to the preceding rule. This is repeated until the algorithm converges to a solution, that is, all features are correctly classified. A pseudocode for the perceptron algorithm is given below. The Perceptron Algorithm ■
Choose w(0) randomly
■
Choose 0
■
t⫽0
■
Repeat • Y ⫽∅ • For i ⫽ 1 to N If ␦ w(t)T x ⱖ 0 then Y ⫽ Y ∪ {x } xi i i • End {For} • w(t ⫹ 1) ⫽ w(t) ⫺ t
x ∈Y
␦x x
• Adjust t • t ⫽t ⫹1 ■
Until Y ⫽ ∅
Figure 3.2 provides a geometric interpretation of the algorithm. It has been assumed that at step t there is only one misclassified sample, x, and t ⫽ 1. The perceptron algorithm corrects the weight vector in the direction of x. Its effect is to turn the corresponding hyperplane so that x is classified in the correct class 1 . Note that in order to achieve this, it may take more than one iteration step, depending on the value(s) of t . No doubt, this sequence is critical for the convergence. We will now show that the perceptron algorithm converges
“05-Ch03-SA272” 17/9/2008 page 95
3.3 The Perceptron Algorithm
x2
x w(t 1 1)
w* x1 w(t) w1 w2
FIGURE 3.2 Geometric interpretation of the perceptron algorithm. The update of the weight vector is in the direction of x in order to turn the decision hyperplane to include x in the correct class.
to a solution in a finite number of iteration steps, provided that the sequence t is properly chosen. The solution is not unique, because there are more than one hyperplanes separating two linearly separable classes. The convergence proof is necessary because the algorithm is not a true gradient descent algorithm and the general tools for the convergence of gradient descent schemes cannot be applied.
Proof of the Perceptron Algorithm Convergence Let ␣ be a positive real number and w ∗ a solution. Then from (3.9) we have w(t ⫹ 1) ⫺ ␣w ∗ ⫽ w(t) ⫺ ␣w ∗ ⫺ t
␦x x
(3.10)
x ∈Y
Squaring the Euclidean norm of both sides results in w(t ⫹ 1) ⫺ ␣w ∗ 2 ⫽ w(t) ⫺ ␣w ∗ 2 ⫹ t2
␦x x2
x ∈Y
⫺ 2t
x ∈Y
␦x (w(t) ⫺ ␣w ∗ )T x
(3.11)
95
“05-Ch03-SA272” 17/9/2008 page 96
96
CHAPTER 3 Linear Classifiers
T x ∈Y ␦x w (t) x ⬍ 0. Hence
But ⫺
w(t ⫹ 1) ⫺ ␣w ∗ 2 ⱕ w(t) ⫺ ␣w ∗ 2 ⫹ t2 ⫹ 2t ␣
␦x x2
x ∈Y
␦x w ∗T x
(3.12)
x ∈Y
Define 2 ⫽
max
⊆ 1 ∪ 2 Y
␦x x2
(3.13)
x ∈Y
That is, 2 is the maximum value that the involved vector norm can take by considering all possible (nonempty) subsets of the available training feature vectors. Similarly, let ␥⫽
max
⊆1 ∪ 2 Y
␦x w ∗T x
(3.14)
x ∈Y
Recall that the summation in this equation is negative; thus, its maximum value over all possible subsets of x’s will also be a negative number. Hence, (3.12) can now be written as w(t ⫹ 1) ⫺ ␣w ∗ 2 ⱕ w(t) ⫺ ␣w ∗ 2 ⫹ t2 2 ⫺ 2t ␣|␥|
Choose ␣ ⫽
2 2|␥|
(3.15)
and apply (3.15) successively for steps t, t ⫺ 1, . . . , 0. Then ∗ 2
∗ 2
w(t ⫹ 1) ⫺ ␣w ⱕ w(0) ⫺ ␣w ⫹ 
2
t
k⫽0
k2
⫺
t
k
(3.16)
k⫽0
If the sequence t is chosen to satisfy the following two conditions:
lim
t→⬁
lim
t→⬁
t
k ⫽ ⬁
(3.17)
k2 ⬍ ⬁
(3.18)
k⫽0 t k⫽0
then there will be a constant t0 such that the right-hand side of (3.16) becomes nonpositive. Thus 0 ⱕ w(t0 ⫹ 1) ⫺ ␣w∗ ⱕ 0
(3.19)
“05-Ch03-SA272” 17/9/2008 page 97
3.3 The Perceptron Algorithm
or w(t0 ⫹ 1) ⫽ ␣w∗
(3.20)
That is, the algorithm converges to a solution in a finite number of steps. An example of a sequence satisfying conditions (3.17), (3.18) is t ⫽ c/t, where c is a constant. In other words, the corrections become increasingly small. What these conditions basically state is that t should vanish as t→⬁ [Eq. (3.18)] but on the other hand should not go to zero very fast [Eq. (3.17)]. Following arguments similar to those used before, it is easy to show that the algorithm also converges for constant t ⫽ , provided is properly bounded ( Problem 3.2). In practice, the proper choice of the sequence t is vital for the convergence speed of the algorithm. Example 3.1 Figure 3.3 shows the dashed line x1 ⫹ x2 ⫺ 0.5 ⫽ 0 corresponding to the weight vector [1, 1, ⫺0.5]T , which has been computed from the latest iteration step of the perceptron algorithm (3.9), with t ⫽ ⫽ 0.7. The line classifies correctly x2 1
0.5
0
20.5 20.5
0
0.5
1x
1
FIGURE 3.3 An example of the perceptron algorithm. After the update of the weight vector, the hyperplane is turned from its initial location (dotted line) to the new one (full line), and all points are correctly classified.
97
“05-Ch03-SA272” 17/9/2008 page 98
98
CHAPTER 3 Linear Classifiers
all the vectors except [0.4, 0.05]T weight vector will be ⎡ 1 ⎢ w(t ⫹ 1) ⫽ ⎣ 1
and [⫺0.20, 0.75]T . According to the algorithm, the next ⎤
⎡
⎤ ⫺0.2 ⎥ ⎢ ⎥ ⎥ ⎢ ⎦ ⫺ 0.7(⫺1) ⎣0.05⎦ ⫺ 0.7(⫹1) ⎣ 0.75 ⎦ 1 ⫺0.5 1
or
⎡
0.4
⎤
⎡
⎤
1.42
⎢ ⎥ w(t ⫹ 1) ⫽ ⎣ 0.51 ⎦ ⫺0.5 The resulting new (solid) line 1.42x1 ⫹ 0.51x2 ⫺ 0.5 ⫽ 0 classifies all vectors correctly, and the algorithm is terminated.
Variants of the Perceptron Algorithm The algorithm we have presented is just one form of a number of variants that have been proposed for the training of a linear classifier in the case of linearly separable classes. We will now state another simpler and also popular form. The N training vectors enter the algorithm cyclically, one after the other. If the algorithm has not converged after the presentation of all the samples once, then the procedure keeps repeating until convergence is achieved—that is, when all training samples have been classified correctly. Let w(t) be the weight vector estimate and x (t) the corresponding feature vector, presented at the tth iteration step. The algorithm is stated as follows: w(t ⫹ 1) ⫽ w(t) ⫹ x (t)
if x (t) ∈ 1 and w T (t)x (t) ⱕ 0
(3.21)
w(t ⫹ 1) ⫽ w(t) ⫺ x (t)
if x (t) ∈ 2 and w T (t)x (t) ≥ 0
(3.22)
w(t ⫹ 1) ⫽ w(t)
otherwise
(3.23)
In other words, if the current training sample is classified correctly, no action is taken. Otherwise, if the sample is misclassified, the weight vector is corrected by adding (subtracting) an amount proportional to x (t) . The algorithm belongs to a more general algorithmic family known as reward and punishment schemes. If the classification is correct, the reward is that no action is taken. If the current vector is misclassified, the punishment is the cost of correction. It can be shown that this form of the perceptron algorithm also converges in a finite number of iteration steps (Problem 3.3). The perceptron algorithm was originally proposed by Rosenblatt in the late 1950s. The algorithm was developed for training the perceptron, the basic unit used for modeling neurons of the brain. This was considered central in developing powerful models for machine learning [Rose 58, Min 88].
“05-Ch03-SA272” 17/9/2008 page 99
3.3 The Perceptron Algorithm
Example 3.2 Figure 3.4 shows four points in the two-dimensional space. Points (⫺1, 0), (0, 1) belong to class 1 , and points (0, ⫺1), (1, 0) belong to class 2 . The goal of this example is to design a linear classifier using the perceptron algorithm in its reward and punishment form. The parameter is set equal to one, and the initial weight vector is chosen as w(0) ⫽ [0, 0, 0]T in the extended three-dimensional space. According to (3.21)–(3.23), the following computations are in order: Step 1.
⎤ ⫺1 ⎥ ⎢ w T (0) ⎣ 0 ⎦ ⫽ 0, 1 ⎡
Step 2.
Step 3.
Step 4.
⎤ ⎤ ⎡ ⫺1 ⫺1 ⎥ ⎥ ⎢ ⎢ w(1) ⫽ w(0) ⫹ ⎣ 0 ⎦ ⫽ ⎣ 0 ⎦ 1 1 ⎡
⎤ 0 ⎥ ⎢ w T (1) ⎣ 1 ⎦ ⫽ 1 ⬎ 0, 1 ⎡
⎤ 0 ⎥ ⎢ w T (2)⎣ ⫺1 ⎦ ⫽ 1 ⬎ 0, 1 ⎡
w(2) ⫽ w(1)
⎤ ⎤ ⎡ ⫺1 0 ⎥ ⎥ ⎢ ⎢ w(3) ⫽ w(2) ⫺ ⎣ ⫺1 ⎦ ⫽ ⎣ 1 ⎦ 0 1 ⎡
⎤ 1 ⎥ ⎢ w T (3) ⎣ 0 ⎦ ⫽ ⫺1 ⬍ 0, 1 ⎡
w(4) ⫽ w(3)
x2 1
21
1
x1
21
FIGURE 3.4 The setup for Example 3.2. The line x1 ⫽ x2 is the resulting solution.
99
“05-Ch03-SA272” 17/9/2008 page 100
100
CHAPTER 3 Linear Classifiers
Step 5.
⎤ ⫺1 ⎥ ⎢ w T (4)⎣ 0 ⎦ ⫽ 1 ⬎ 0, 1 ⎡
Step 6.
⎤ 0 ⎥ ⎢ w T (5) ⎣ 1 ⎦ ⫽ 1 ⬎ 0, 1
w(5) ⫽ w(4)
⎡
Step 7.
w(6) ⫽ w(5)
⎤ 0 ⎥ ⎢ w T (6)⎣⫺1 ⎦ ⫽ ⫺1 ⬍ 0, 1 ⎡
w(7) ⫽ w(6)
Since for four consecutive steps no correction is needed, all points are correctly classified and the algorithm terminates. The solution is w ⫽ [⫺1, 1, 0]T . That is, the resulting linear classifier is ⫺x1 ⫹ x2 ⫽ 0, and it is the line passing through the origin shown in Figure 3.4.
The Perceptron Once the perceptron algorithm has converged to a weight vector w and a threshold w0 , our next goal is the classification of an unknown feature vector to either of the two classes. Classification is achieved via the simple rule If
w T x ⫹ w0 ⬎ 0
assign x to 1
If
w x ⫹ w0 ⬍ 0
assign x to 2
T
(3.24)
A basic network unit that implements the operation is shown in Figure 3.5a. x1 x2
x1 w1 w2
x2 Σ
wl (a)
w2
f
w0
xl
w1
wl
w0
xl (b)
FIGURE 3.5 The basic perceptron model. (a) A linear combiner is followed by the activation function. (b) The combiner and the activation function are merged together.
“05-Ch03-SA272” 17/9/2008 page 101
3.3 The Perceptron Algorithm
The elements of the feature vector x1 , x2 , . . . , xl are applied to the input nodes of the network. Then each one is multiplied by the corresponding weights wi , i ⫽ 1, 2, . . . , l. These are known as synaptic weights or simply synapses. The products are summed up together with the threshold value w0 . The result then goes through a nonlinear device, which implements the so-called activation function. A common choice is a hard limiter; that is, f (·) is the step function [f (x) ⫽ ⫺1 if x ⬍ 0 and f (x) ⫽ 1 if x ⬎ 0]. The corresponding feature vector is classified in one of the classes depending on the sign of the output. Besides ⫹1 and ⫺1, other values (class labels) for the hard limiter are also possible. Another popular choice is 1 and 0, and it is achieved by choosing the two levels of the step function appropriately. This basic network is known as a perceptron or neuron. Perceptrons are simple examples of the so-called learning machines—that is, structures whose free parameters are updated by a learning algorithm, such as the perceptron algorithm, in order to “learn” a specific task, based on a set of training data. Later on we will use the perceptron as the basic building element for more complex learning networks. Figure 3.5b is a simplified graph of the neuron where the summer and nonlinear device have been merged for notational simplification. Sometimes a neuron with a hard limiter device is referred to as a McCulloch–Pitts neuron. Other types of neurons will be considered in Chapter 4.
The Pocket Algorithm A basic requirement for the convergence of the perceptron algorithm is the linear separability of the classes. If this is not true, as is usually the case in practice, the perceptron algorithm does not converge. A variant of the perceptron algorithm was suggested in [Gal 90] that converges to an optimal solution even if the linear separability condition is not fulfilled. The algorithm is known as the pocket algorithm and consists of the following two steps ■
■
Initialize the weight vector w(0) randomly. Define a stored (in the pocket!) vector w s . Set a history counter hs of the w s to zero. At the tth iteration step compute the update w(t ⫹ 1), according to the perceptron rule. Use the updated weight vector to test the number h of training vectors that are classified correctly. If h ⬎ hs replace w s with w(t ⫹ 1) and hs with h. Continue the iterations.
It can be shown that this algorithm converges with probability one to the optimal solution, that is, the one that produces the minimum number of misclassifications [Gal 90, Muse 97]. Other related algorithms that find reasonably good solutions when the classes are not linearly separable are the thermal perceptron algorithm [Frea 92], the loss minimization algorithm [Hryc 92], and the barycentric correction procedure [Poul 95].
Kesler’s Construction So far we have dealt with the two-class case. The generalization to an M-class task is straightforward. A linear discriminant function w i , i ⫽ 1, 2, . . . , M, is defined for
101
“05-Ch03-SA272” 17/9/2008 page 102
102
CHAPTER 3 Linear Classifiers
each of the classes. A feature vector x (in the (l ⫹ 1)-dimensional space to account for the threshold) is classified in class i if w Ti x ⬎ w Tj x,
᭙j ⫽ i
(3.25)
This condition leads to the so-called Kesler’s construction. For each of the training vectors from class i , i ⫽1, 2, . . . , M, we construct M⫺1 vectors x ij ⫽ [0T , 0T , . . . , x T , . . . , ⫺x T , . . . , 0T ]T of dimension (l ⫹ 1)M ⫻ 1. That is, they are block vectors having zeros everywhere except at the ith and jth block positions, where they have x and ⫺x, respectively, for j ⫽ i. We also construct the block vector w ⫽ [w T1 , . . . , w TM ]T . If x ∈ i , this imposes the requirement that w T x ij ⬎ 0, ᭙j ⫽ 1, 2, . . . , M, j ⫽ i. The task now is to design a linear classifier, in the extended (l ⫹ 1)M-dimensional space, so that each of the (M ⫺ 1)N training vectors lies in its positive side. The perceptron algorithm will have no difficulty in solving this problem for us, provided that such a solution is possible—that is, if all the training vectors can be correctly classified using a set of linear discriminant functions. Example 3.3 Let us consider a three-class problem in the two-dimensional space. The training vectors for each of the classes are the following: 1 : [1, 1]T , [2, 2]T , [2, 1]T 2 : [1, ⫺1]T , [1, ⫺2]T , [2, ⫺2]T 3 : [⫺1, 1]T , [⫺1, 2]T , [⫺2, 1]T This is obviously a linearly separable problem, since the vectors of different classes lie in different quadrants. To compute the linear discriminant functions, we first extend the vectors to the threedimensional space, and then we use Kesler’s construction. For example, For [1, 1]T we get
[1, 1, 1, ⫺1, ⫺1, ⫺1, 0, 0, 0]T
and
[1, 1, 1, 0, 0, 0, ⫺1, ⫺1, ⫺1]T For [1, ⫺2]T we get [⫺1, 2, ⫺1, 1, ⫺2, 1, 0, 0, 0]T
and
[0, 0, 0, 1, ⫺2, 1, ⫺1, 2, ⫺1]T For [⫺2, 1]T we get [2, ⫺1, ⫺1, 0, 0, 0, ⫺2, 1, 1]T
and
[0, 0, 0, 2, ⫺1, ⫺1, ⫺2, 1, 1]T Similarly, we obtain the other twelve vectors. To obtain the corresponding weight vectors w 1 ⫽ [w11 , w12 , w10 ]T w 2 ⫽ [w21 , w22 , w20 ]T w 3 ⫽ [w31 , w32 , w30 ]T
“05-Ch03-SA272” 17/9/2008 page 103
3.4 Least Squares Methods
we can run the perceptron algorithm by requiring w T x ⬎ 0, w ⫽ [w T1 , w T2 , w T3 ]T , for each of the eighteen 9-dimensional vectors. That is, we require all the vectors to lie on the same side of the decision hyperplane. The initial vector of the algorithm w(0) is computed using the uniform pseudorandom sequence generator in [0, 1]. The learning sequence t was chosen to be constant and equal to 0.5. The algorithm converges after four iterations and gives w 1 ⫽ [5.13, 3.60, 1.00]T w 2 ⫽ [⫺0.05, ⫺3.16, ⫺0.41]T w 3 ⫽ [⫺3.84, 1.28, 0.69]T
3.4 LEAST SQUARES METHODS As we have already pointed out, the attractiveness of linear classifiers lies in their simplicity. Thus, in many cases, although we know that the classes are not linearly separable, we still wish to adopt a linear classifier, despite the fact that this will lead to suboptimal performance from the classification error probability point of view. The goal now is to compute the corresponding weight vector under a suitable optimality criterion. The least squares methods are familiar to us, in one way or another, from our early college courses. Let us then build upon them.
3.4.1 Mean Square Error Estimation Let us once more focus on the two-class problem. In the previous section, we saw that the perceptron output was ⫾1, depending on the class ownership of x. Since the classes were linearly separable, these outputs were correct for all the training feature vectors, after, of course, the perceptron algorithm’s convergence. In this section we will attempt to design a linear classifier so that its desired output is again ⫾1, depending on the class ownership of the input vector. However, we will have to live with errors; that is, the true output will not always be equal to the desired one. Given a vector x, the output of the classifier will be w T x (thresholds can be accommodated by vector extensions). The desired output will be denoted as y(x) ≡ y ⫽⫾1. The weight vector will be computed so as to minimize the mean square error (MSE) between the desired and true outputs, that is, J (w) ⫽ E[|y ⫺ x T w|2 ]
(3.26)
w ˆ ⫽ arg min J (w)
(3.27)
w
The reader can easily check that J (w) is equal to J (w) ⫽ P(1 )
(1 ⫺ x T w)2 p(x|1 ) dx ⫹ P(2 )
(1 ⫹ x T w)2 p(x|2 ) dx
(3.28)
103
“05-Ch03-SA272” 17/9/2008 page 104
104
CHAPTER 3 Linear Classifiers
Minimizing (3.27) easily results in
Then where
⭸J (w) ⫽ 2E[x(y ⫺ x T w)] ⫽ 0 ⭸w
(3.29)
w ˆ ⫽ R⫺1 x E[xy]
(3.30)
⎡
E[x1 x1 ] ⎢E[x x ] 2 1 ⎢ Rx ≡ E[xx T ] ⫽ ⎢ .. ⎢ ⎣ . E[xl x1 ]
⎤ · · · E[x1 xl ] · · · E[x2 xl ]⎥ ⎥ .. ⎥ .. ⎥ . ⎦ . · · · E[xl xl ]
(3.31)
is known as the correlation or autocorrelation matrix and is equal to the covariance matrix, introduced in the previous chapter, if the respective mean values are zero. The vector ⎡⎡
⎤⎤ x1 y ⎢ ⎢ . ⎥⎥ ⎢ ⎥⎥ E[xy] ⫽ E ⎢ ⎣ ⎣ .. ⎦⎦ xl y
(3.32)
is known as the cross-correlation between the desired output and the (input) feature vectors. Thus, the mean square optimal weight vector results as the solution of a linear set of equations, provided, of course, that the correlation matrix is invertible. It is interesting to point out that there is a geometrical interpretation of this solution. Random variables can be considered as points in a vector space. It is straightforward to see that the expectation operation E[xy] between two random variables satisfies the properties of the inner product. Indeed, E[x 2 ] ⱖ 0, E[xy] ⫽ E[yx], E[x(c1 y ⫹ c2 z)] ⫽ c1 E[xy] ⫹ c2 E[xz]. In such a vector space w T x ⫽ w1 x1 ⫹ · · · ⫹ wl xl is a linear combination of vectors, and thus it lies in the subspace defined by the xi ’s. This is illustrated by an example in Figure 3.6. Then, if we want to approximate y by this linear combination, the resulting error is y ⫺ w T x. Equation (3.29) states that the minimum mean square error solution results if the error is orthogonal to each xi ; thus it is orthogonal to the vector subspace spanned by xi , i ⫽ 1, 2, . . . , l—in other words,if y is approximated by its orthogonal projection on the subspace (Figure 3.6). Equation (3.29) is also known as the orthogonality condition.
Multiclass Generalization In the multiclass case, the task is to design the M linear discriminant functions gi (x) ⫽ w Ti x according to the MSE criterion. The corresponding desired output responses (i.e., class labels) are chosen so that yi ⫽ 1 if x ∈ i and yi ⫽ 0 otherwise. This is in agreement with the two-class case. Indeed, for such
“05-Ch03-SA272” 17/9/2008 page 105
3.4 Least Squares Methods
y y 2 wTx
x1 wTx
x2
FIGURE 3.6 Interpretation of the MSE estimate as an orthogonal projection on the input vector elements’ subspace.
a choice and if M ⫽ 2, the design of the decision hyperplane w T x ≡ (w 1 ⫺ w 2 )T x corresponds to ⫾1 desired responses, depending on the respective class ownership. Let us now define y T ⫽ [y1 , . . . , yM ], for a given vector x, and W ⫽ [w 1 , . . . , w M ]. That is, matrix W has as columns the weight vectors w i . The MSE criterion in (3.27) can now be generalized to minimize the norm of the error vector y ⫺ W T x, that is, ˆ ⫽ arg min E[y ⫺ W x ] ⫽ arg min E W T
W
2
W
M
yi ⫺ w Ti x
2
(3.33)
i⫽1
This is equivalent to M MSE independent minimization problems of the (3.27) type, with scalar desired responses. In other words, in order to design the MSE optimal linear discriminant functions, it suffices to design each one of them so that its desired output is 1 for vectors belonging to the corresponding class and 0 for all the others.
3.4.2 Stochastic Approximation and the LMS Algorithm The solution of (3.30) requires the computation of the correlation matrix and crosscorrelation vector. This presupposes knowledge of the underlying distributions, which in general are not known. After all, if they were known, why not then use
105
“05-Ch03-SA272” 17/9/2008 page 106
106
CHAPTER 3 Linear Classifiers
Bayesian classifiers? Thus, our major goal now becomes to see if it is possible to solve (3.29) without having this statistical information available. The answer has been provided by Robbins and Monro [Robb 51] in the more general context of stochastic approximation theory. Consider an equation of the form E[F (x k , w)] ⫽ 0, where x k , k ⫽ 1, 2, . . . , is a sequence of random vectors from the same distribution, F (·, ·) a function, and w the vector of the unknown parameters. Then adopt the iterative scheme w(k) ˆ ⫽ w(k ˆ ⫺ 1) ⫹ k F (x k , w(k ˆ ⫺ 1))
(3.34)
In other words,the place of the mean value (which cannot be computed due to lack of information) is taken by the samples of the random variables resulting from the experiments. It turns out that under mild conditions the iterative scheme converges in probability to the solution w of the original equation, provided that the sequence k satisfies the two conditions ⬁
k → ⬁
(3.35)
k2 ⬍ ⬁
(3.36)
k⫽1 ⬁ k⫽1
and which implies that k → 0
(3.37)
lim prob{w(k) ˆ ⫽ w} ⫽ 1
(3.38)
That is, k→⬁
The stronger, in the mean square sense, convergence is also true lim E[w(k) ˆ ⫺ w2 ] ⫽ 0
k→⬁
(3.39)
Conditions (3.35), (3.36) have already been met before and guarantee that the corrections of the estimates in the iterations tend to zero. Thus, for large values of k (in theory at infinity) iterations freeze. However, this must not happen too early (first condition) to make sure that the iterations do not stop away from the solution. The second condition guarantees that the accumulated noise, due to the stochastic nature of the variables, remains finite and the algorithm can cope with it [Fuku 90]. The proof is beyond the scope of the present text. However, we will demonstrate its validity via an example. Let us consider the simple equation E[xk ⫺ w] ⫽ 0. For k ⫽ 1/k the iteration becomes w(k) ˆ ⫽ w(k ˆ ⫺ 1) ⫹
1 1 (k ⫺ 1) [xk ⫺ w(k w(k ˆ ⫺ 1) ⫹ xk ˆ ⫺ 1)] ⫽ k k k
“05-Ch03-SA272” 17/9/2008 page 107
3.4 Least Squares Methods
For large values of k it is easy to see that w(k) ˆ ⫽
k 1 xr k r⫽1
That is, the solution is the sample mean of the measurements. Most natural! Let us now return to our original problem and apply the iteration to solve (3.29). Then (3.34) becomes w(k) ˆ ⫽ w(k ˆ ⫺ 1) ⫹ k x k yk ⫺ x Tk w(k ˆ ⫺ 1)
(3.40)
where (yk , x k ) are the desired output (⫾1)–input training sample pairs,successively presented to the algorithm. The algorithm is known as the least mean squares (LMS) or Widrow–Hoff algorithm, after those who suggested it in the early 1960s [Widr 60, Widr 90]. The algorithm converges asymptotically to the MSE solution. A number of variants of the LMS algorithm have been suggested and used. The interested reader may consult, for example, [Hayk 96, Kalou 93]. A common variant is to use a constant in the place of k . However, in this case the algorithm does not converge to the MSE solution. It can be shown, for example, [Hayk 96], that if 0 ⬍ ⬍ 2/trace{Rx } then E[w(k)] ˆ → w MSE
and E[w(k) ˆ ⫺ w MSE 2 ] → constant
(3.41)
where w MSE denotes the MSE optimal estimate and trace{·} the trace of the matrix. That is, the mean value of the LMS estimate is equal to the MSE solution, and also the corresponding variance remains finite. It turns out that the smaller the ,the smaller the variance around the desired MSE solution. However,the smaller the ,the slower the convergence of the LMS algorithm. The reason for using Constant in place of a vanishing sequence is to keep the algorithm “alert” to track variations when the statistics are not stationary but are slowly varying, that is, when the underlying distributions are time dependent. Remarks ■
Observe that in the case of the LMS, the parameters’ update iteration step, k, coincides with the index of the current input sample x k . In case k is a time index, LMS is a time-adaptive scheme, which adapts to the solution as successive samples become available to the system.
■
Observe that Eq. (3.40) can be seen as the training algorithm of a linear neuron, that is, a neuron without the nonlinear activation function. This type of training, which neglects the nonlinearity during training and applies the desired response just after the adder of the linear combiner part of the neuron (Figure 3.5a),was used byWidrow and Hoff. The resulting neuron architecture
107
“05-Ch03-SA272” 17/9/2008 page 108
108
CHAPTER 3 Linear Classifiers
is known as adaline (adaptive linear element). After training and once the weights have been fixed, the model is the same as in Figure 3.5, with the hard limiter following the linear combiner. In other words, the adaline is a neuron that is trained according to the LMS instead of the perceptron algorithm.
3.4.3 Sum of Error Squares Estimation A criterion closely related to the MSE is the sum of error squares or simply the least squares (LS) criterion defined as J (w) ⫽
N
yi ⫺ x Ti w
2
≡
i⫽1
N
ei2
(3.42)
i⫽1
In other words, the errors between the desired output of the classifier (⫾1 in the two-class case) and the true output are summed up over all the available training feature vectors, instead of averaging them out. In this way, we overcome the need for explicit knowledge of the underlying pdfs. Minimizing (3.42) with respect to w results in N
x i yi ⫺ x Ti w ˆ
⫽0⇒
i⫽1
N
x i x Ti
w ˆ ⫽
i⫽1
N
(x i yi )
(3.43)
i⫽1
For the sake of mathematical formulation let us define ⎡
x T1
⎤
⎡
x11 ⎢ T ⎥ ⎢x ⎢ x 2 ⎥ ⎢ 21 ⎥ ⎢ X ⫽⎢ ⎢ .. ⎥ ⫽ ⎢ .. ⎣ . ⎦ ⎣ . xN 1 x TN
x12 x22 .. . xN 2
... ... .. . ...
⎤ x1l x2l ⎥ ⎥ .. ⎥ ⎥, . ⎦ xNl
⎡ ⎢ ⎢ y⫽⎢ ⎢ ⎣
⎤ y1 y2 ⎥ ⎥ .. ⎥ ⎥ . ⎦ yN
(3.44)
That is, X is an N ⫻ l matrix whose rows are the available training feature vectors, consisting desired responses. Then N and yT is a vector Nof the corresponding T X and also T y. Hence, (3.43) can now be x x ⫽ X x y ⫽ X i i i i⫽1 i⫽1 i written as (X T X)w ˆ ⫽ XT y ⇒ w ˆ ⫽ (X T X)⫺1 X T y
(3.45)
Thus, the optimal weight vector is again provided as the solution of a linear set of equations. Matrix X T X is known as the sample correlation matrix. Matrix X ⫹ ≡ (X T X)⫺1 X T is known as the pseudoinverse of X, and it is meaningful only if X T X is invertible, that is, X is of rank l. X ⫹ is a generalization of the inverse of an
“05-Ch03-SA272” 17/9/2008 page 109
3.4 Least Squares Methods
invertible square matrix. Indeed, if X is an l ⫻ l square and invertible matrix, then it is straightforward to see that X ⫹ ⫽ X ⫺1 . In such a case the estimated weight vector is the solution of the linear system X w ˆ ⫽ y. If, however, there are more equations than unknowns, N ⬎ l, as is the usual case in pattern recognition, there is not, in general, a solution. The solution obtained by the pseudoinverse is the vector that minimizes the sum of error squares. It is easy to show that (under mild assumptions) the sum of error squares tends to the MSE solution for large values of N (Problem 3.8). Remarks ■
■
So far we have restricted the desired output values to be ⫾1. Of course, this is not necessary. All we actually need is a desired response positive for 1 and negative for 2 . Thus, in place of ⫾1 in the y vector we could have any positive (negative) values. Obviously, all we have said so far is still applicable. However, the interesting aspect of this generalization would be to compute these desired values in an optimal way,in order to obtain a better solution. The Ho–Kashyap algorithm is such a scheme solving for both the optimal w and optimal desired values yi . The interested reader may consult [Ho 65, Tou 74]. Generalization to the multi-class case follows the same concept as that introduced for the MSE cost, and it is easily shown that it reduces to M equivalent problems of scalar desired responses, one for each discriminant function (Problem 3.10).
Example 3.4 Class 1 consists of the two-dimensional vectors [0.2, 0.7]T , [0.3, 0.3]T , [0.4, 0.5]T , [0.6, 0.5]T , [0.1, 0.4]T and class 2 of [0.4, 0.6]T , [0.6, 0.2]T , [0.7, 0.4]T , [0.8, 0.6]T , [0.7, 0.5]T . Design the sum of error squares optimal linear classifier w1 x1 ⫹ w2 x2 ⫹ w0 ⫽ 0. We first extend the given vectors by using 1 as their third dimension and form the 10 ⫻ 3 matrix X, which has as rows the transposes of these vectors. The resulting sample correlation 3 ⫻ 3 matrix X T X is equal to ⎡
2.8 ⎢ X T X ⫽ ⎣2.24 4.8
2.24 2.41 4.7
⎤ 4.8 ⎥ 4.7⎦ 10
The corresponding y consists of five 1’s and then five ⫺1’s and ⎤ ⫺1.6 ⎥ ⎢ X T y ⫽ ⎣ 0.1⎦ 0.0 ⎡
109
“05-Ch03-SA272” 17/9/2008 page 110
110
CHAPTER 3 Linear Classifiers
x2
1
1
2
0.5
0
0
0.5
1 x1
FIGURE 3.7 Least sum of error squares linear classifier. The task is not linearly separable. The linear LS classifier classifies some of the points in the wrong class. However, the resulting sum of error squares is minimum.
Solving the corresponding set of equations results in [w1 , w2 , w0 ] ⫽ [⫺3.218, 0.241, 1.431]. Figure 3.7 shows the resulting geometry.
3.5 MEAN SQUARE ESTIMATION REVISITED 3.5.1 Mean Square Error Regression In this subsection, we will approach the MSE task from a slightly different perspective and in a more general framework. Let y, x be two random vector variables of dimensions M ⫻ 1 and l ⫻ 1, respectively, and assume that they are described by the joint pdf p(y, x). The task of interest is to estimate the value of y, given the value of x that is obtained from an experiment. No doubt the classification task falls under this more general formulation. For example, when we are given a feature vector x, our goal is to estimate the value of the class label y, which is ⫾1 in the two-class case. In a more general setting, the values of y may not be discrete. Take, as an example, the case where y ∈ R is generated by an unknown rule, i.e., y ⫽ f (x) ⫹ ⑀
where f (·) is some unknown function and ⑀ is a noise source. The task now is to estimate (predict) the value of y, given the value of x. Once more, this is a problem
“05-Ch03-SA272” 17/9/2008 page 111
3.5 Mean Square Estimation Revisited
of designing a function g(x), based on a set of training data points ( yi , x i ), i ⫽ 1, 2, . . . , N , so that the predicted value yˆ ⫽ g(x)
to be as close as possible to the true value y in some optimal sense. This type of problem is known as a regression task. One of the most popular optimality criteria for regression is the mean square error (MSE). In this section, we will focus on the MSE regression and highlight some of its properties. The mean square estimate yˆ of the random vector y, given the value x, is defined as yˆ ⫽ arg min E[y ⫺ y ˜ 2] y˜
(3.46)
Note that the mean value here is with respect to the conditional pdf p( y|x). We will show that the optimal estimate is the mean value of y, that is, yˆ ⫽ E[ y|x] ≡
⬁
⫺⬁
yp( y|x) dy
(3.47)
Proof. Let y˜ be another estimate. It will be shown that it results in higher mean square error. Indeed, E[y ⫺ y ˜ 2 ] ⫽ E[y ⫺ yˆ ⫹ yˆ ⫺ y ˜ 2 ] ⫽ E[y ⫺ y ˆ 2] ˆ T ( yˆ ⫺ y)] ˜ ⫹ E[yˆ ⫺ y ˜ 2 ] ⫹ 2E[( y ⫺ y)
(3.48)
where the dependence on x has been omitted for notational convenience. Note now that yˆ ⫺ y˜ is a constant. Thus, E[y ⫺ y ˜ 2 ] ⱖ E[y ⫺ y ˆ 2 ] ⫹ 2E[( y ⫺ y) ˆ T ]( yˆ ⫺ y) ˜
(3.49)
and from the definition of yˆ ⫽ E[y] it follows that E[y ⫺ y ˜ 2 ] ⱖ E[y ⫺ y ˆ 2]
(3.50)
Remark ■
This is a very elegant result. Given a measured value of x, the best (in the MSE sense) estimate of y is given by the function y(x) ≡ E[ y|x]. In general,this is a nonlinear vector-valued function of x (i.e.,g(·) ≡ [g1 (·), . . . , gM (·)]T ),and it is known as the regression of y conditioned on x. It can be shown (Problem 3.11) that if ( y, x) are jointly Gaussian, then the MSE optimal regressor is a linear function.
111
“05-Ch03-SA272” 17/9/2008 page 112
112
CHAPTER 3 Linear Classifiers
3.5.2 MSE Estimates Posterior Class Probabilities In the beginning of the chapter we promised to “emancipate” ourselves from the Bayesian classification. However, the nice surprise is that a Bayesian flavor still remains, although in a disguised form. Let us reveal it—it can only be beneficial. We will consider the multiclass case. Given x, we want to estimate its class label. Let gi (x) be the discriminant functions to be designed. The cost function in Eq. (3.33) now becomes M 2 J ⫽E (gi (x) ⫺ yi ) ≡ E[g(x) ⫺ y2 ]
(3.51)
i⫽1
where the vector y consists of zeros and a single 1 at the appropriate place. Note that each gi (x) depends only on x, whereas the yi ’s depend on the specific class to which x belongs. Let p(x, i ) be the joint probability density of the feature vector belonging to the ith class. Then (3.51) is written as J⫽
M ⫹⬁
⫺⬁ j⫽1
M 2 ( gi (x) ⫺ yi ) p(x, j ) dx
(3.52)
i⫽1
Taking into account that p(x, j ) ⫽ P(j |x)p(x), (3.52) becomes J⫽
⬁
⎧ M M ⎨
⫺⬁ ⎩ j⫽1 i⫽1
gi (x) ⫺ yi
2
⎫ ⎬
P(j |x) p(x) dx ⎭
⎤ ⎡ M M 2 gi (x) ⫺ yi P(j |x)⎦ ⫽ E⎣
(3.53)
j⫽1 i⫽1
where the mean is taken with respect to x. Expanding this, we get ⎡ ⎤ M M 2 2 gi (x)P(j |x) ⫺ 2gi (x)yi P(j |x) ⫹ yi P(j |x) ⎦ J ⫽ E⎣
(3.54)
j⫽1 i⫽1
Exploiting the fact that gi (x) is a function of x only and becomes
M
j⫽1 P(j |x)
⫽ 1, (3.54)
⎡ ⎤ M M M gi2 (x) ⫺ 2gi (x) J ⫽ E⎣ yi P(j |x) ⫹ yi2 P(j |x) ⎦ i⫽1
⫽E
j⫽1
M
gi2 (x) ⫺ 2gi (x)E[ yi |x] ⫹ E
i⫽1
j⫽1
yi2 |x
(3.55)
“05-Ch03-SA272” 17/9/2008 page 113
3.5 Mean Square Estimation Revisited
where E[ yi |x] and E[ yi2 |x] are the respective mean values conditioned on x. Adding and subtracting (E[ yi |x])2 , Eq. (3.55) becomes J ⫽E
M
2
gi (x) ⫺ E[ yi |x]
⫹E
i⫽1
M 2 E yi |x ⫺ (E[ yi |x])2
(3.56)
i⫽1
The second term in (3.56) does not depend on the functions gi (x), i ⫽ 1, 2, . . . , M. Thus, minimization of J with respect to (the parameters of ) gi (·) affects only the first of the two terms. Let us concentrate and look at it more carefully. Each of the M summands involves two terms: the unknown discriminant function gi (·) and the conditional mean of the corresponding desired response. Let us now write gi (·) ⫽ gi (·; w i ), to state explicitly that the functions are defined in terms of a set of parameters, to be determined optimally during training. Minimizing J with respect to w i , i ⫽ 1, 2, . . . , M,results in the mean square estimates of the unknown parameters, w ˆ i , so that the discriminant functions approximate optimally the corresponding conditional means—that is, the regressions of yi conditioned on x. Moreover, for the M-class problem and the preceding definitions we have E[yi |x] ≡
M
yi P(j |x)
(3.57)
j⫽1
However yi ⫽ 1(0) if x ∈ i (x ∈ j , j ⫽ i). Hence gi (x, w ˆ i ) is the MSE estimate of P(i |x)
(3.58)
This is an important result. Training the discriminant functions gi with desired outputs 1 or 0 in the MSE sense, Eq. (3.51) is equivalent to obtaining the MSE estimates of the class posterior probabilities, without using any statistical information or pdf modeling! It suffices to say that these estimates may in turn be used for Bayesian classification. An important issue here is to assess how good the resulting estimates are. It all depends on how well the adopted functions gi (·; w i ) can model the desired (in general) nonlinear functions P(i |x). If, for example, we adopt linear models, as was the case in Eq. (3.33), and P(i |x) is highly nonlinear, the resulting MSE optimal approximation will be a bad one. Our focus in the next chapter will be on developing modeling techniques for nonlinear functions. Finally, it must be emphasized that the conclusion above is an implication of the cost function itself and not of the specific model function used. The latter plays its part when the approximation accuracy issue comes into the scene. MSE cost is just one of the costs that have this important property. Other cost functions share this property too, see, for example, [Rich 91, Bish 95, Pear 90, Cid 99]. In [Guer 04] a procedure is developed to design cost functions that provide more accurate estimates of the probability values, taking into account the characteristics of each classification problem.
113
“05-Ch03-SA272” 17/9/2008 page 114
114
CHAPTER 3 Linear Classifiers
3.5.3 The Bias–Variance Dilemma So far we have touched on some very important issues concerning the interpretation of the output of an optimally designed classifier. Also, we saw that a regressor or a classifier can be viewed as learning machines realizing a function or a set of functions g(x), which attempt to estimate the corresponding value or class label y and make a decision based on these estimates. In practice, the functions g(·) are estimated using a finite training data set D ⫽ {(yi , x i ), i ⫽ 1, 2, . . . , N } and a suitable methodology (e.g., mean square error, sum of error squares, LMS). To emphasize the explicit dependence on D, we write g(x; D). This subsection focuses on the capabilities of g(x; D) to approximate the MSE optimal regressor E[y|x] and on how this is affected by the finite size, N , of the training data set. The key factor here is the dependence of the approximation on D. The approximation may be very good for a specific training data set but very bad for another. The effectiveness of an estimator can be evaluated by computing its mean square deviation from the desired optimal value. This is achieved by averaging over all possible sets D of size N , that is, 2 ! ED g(x; D) ⫺ E[ y|x]
(3.59)
If we add and subtract ED [g(x; D)] and follow a procedure similar to that in the proof of (3.47), we easily obtain 2 ! ⫽ (ED [g(x; D)] ⫺ E[ y|x])2 ED g(x; D) ⫺ E[ y|x] ⫹ ED (g(x; D) ⫺ ED [g(x; D)])2
(3.60)
The first term is the contribution of the bias and the second that of the variance. In other words, even if the estimator is unbiased, it can still result in a large mean square error due to a large variance term. For a finite data set, it turns out that there is a trade-off between these two terms. Increasing the bias decreases the variance and vice versa. This is known as the bias–variance dilemma. This behavior is reasonable. The problem at hand is similar to that of a curve fitting through a given data set. If, for example, the adopted model is complex (many parameters involved) with respect to the number N , the model will fit the idiosyncrasies of the specific data set. Thus, it will result in low bias but will yield high variance, as we change from one data set to another. The major issue now is to seek ways to make both bias and variance low at the same time. It turns out that this may be possible only asymptotically, as the number N grows to infinity. Moreover, N has to grow in such a way as to allow more complex models, g, to be fitted (which reduces bias) and at the same time to ensure low variance. However, in practice N is finite, and one should aim at the best compromise. If, on the other hand, some a priori knowledge is available, this must be exploited in the form of constraints that the classifier/regressor has to satisfy. This can lead to lower values of both the
“05-Ch03-SA272” 17/9/2008 page 115
3.5 Mean Square Estimation Revisited
variance and the bias, compared with a more general type of classifier/regressor. This is natural, because one takes advantage of the available information and helps the optimization process. Let us now use two simplified“extreme”example cases, which will help us grasp the meaning of the bias–variance dilemma using common-sense reasoning. Let us assume that our data are generated by the following mechanism y ⫽ f (x) ⫹ ⑀
where f (·) is an unknown function and ⑀ a noise source of zero mean and known variance equal to, say, ⑀2 . Obviously, for any x, the optimum MSE regressor is E[ y|x] ⫽ f (x). To make our point easier to understand, let us further assume that the randomness in the different training sets, D, is due to the yi ’s (whose values are affected by the noise),while the respective points,xi ,are fixed. Such an assumption is not an unreasonable one. Since our goal is to obtain an estimate of f (·),it is sensible for one to divide the interval [x1 , x2 ], in which x lies, in equally spaced points. For 1 example, one can choose xi ⫽ x1 ⫹ xN2 ⫺x ⫺1 (i ⫺ 1), i ⫽ 1, 2, . . . , N . ■
Case 1. Choose the estimate of f (x), g(x; D), to be independent of D, for example, g(x) ⫽ w1 x ⫹ w0
for some fixed values of w1 and w0 . Figure 3.8 illustrates this setup showing a line g(x) and N ⫽ 11 training pairs (yi , xi ), which spread around f (x), x ∈ [0, 1]. Since g(x) is fixed and does not change, we have ED [g(x; D)] ⫽ g(x; D) ≡ g(x), and the variance term in (3.60) is zero. On the other hand, since g(x) has been chosen arbitrarily, in general, one expects the bias term to be large. ■
Case 2. In contrast to g(x), the function g1 (x), shown in Figure 3.8, corresponds to a polynomial of high degree and with a large enough number of free parameters so that,for each one of the different training sets D,the respective graphs of g1 (x) pass through the training points (yi , xi ), i ⫽ 1, 2, . . . , 11. For this case, due to the zero mean of the noise source, we have that ED [g1 (x; D)] ⫽ f (x) ⫽ E[y|x], for any x ⫽ xi . That is, at the training points, the bias is zero. Due to the continuity of f (x) and g1 (x), one expects similar behavior and at the points that lie in the vicinity of the training points xi . Thus, if N is large enough we can expect the bias to be small for all the points in the interval [0, 1]. However, now the variance increases. Indeed, for this case we have that ED
2 !
g1 (x; D) ⫺ ED [ g1 (x; D)]
⫽ ED
f (x) ⫹ ⑀ ⫺ f (x)
2 !
⫽ ⑀2 , for x ⫽ xi , i ⫽ 1, 2, . . . , N
In other words,the bias becomes zero (or approximately zero) but the variance is now equal to the variance of the noise source.
115
“05-Ch03-SA272” 17/9/2008 page 116
116
CHAPTER 3 Linear Classifiers
y
g1(x)
g(x)
f(x)
0
0.5
1
x
FIGURE 3.8 The data points are spread around the f (x) curve. The line g(x) ⫽ 0 exhibits zero variance but high bias. The high degree polynomial curve, g1 (x) ⫽ 0, always passes through the training points and leads to low bias (zero bias at the training points) but to high variance.
The reader will notice that everything that has been said so far applies to both the regression and the classification tasks. The reason that we talked only for regression is that the mean square error is not the best criterion to validate the performance of a classifier. After all, we may have a classifier that results in high mean square error, yet its error performance can be very good. Take, as an example, the case of a classifier g(x) resulting in relatively high mean square error, but predicts the correct class label y for most of the values of x. That is, for all points originating from class 1 (2 ) the predicted values lie, for most of the cases, on the correct side of the classifier, albeit with a lot of variation (for the different training sets) and away from the desired values of ⫾1. From the classification point of view, such a designed classifier would be perfectly acceptable. Concerning the preceding theory, in order to get more meaningful results, for the classification task, one has to rework the previous theory in terms of the probability of error. However, now the algebra gets a bit more involved, and some further assumptions need to be adopted (e.g., Gaussian data), in order to make the algebra more tractable. We will not delve into that, since more recent and elegant theories are
“05-Ch03-SA272” 17/9/2008 page 117
3.6 Logistic Discrimination
now available, which study the trade-off between model complexity and the accuracy of the resulting classifier for finite data sets in a generalized framework (see Chapter 5). A simple and excellent treatment of the bias–variance dilemma task can be found in [Gema 92]. As for ourselves, this was only the beginning. We will come to the finite data set issue and its implications many times throughout this book and from different points of view.
3.6 LOGISTIC DISCRIMINATION In logistic discrimination the logarithm of the likelihood ratios [Eq. (2.20)] is modeled via linear functions. That is, ln
P(i |x) ⫽ wi,0 ⫹ w Ti x, i ⫽ 1, 2, . . . , M ⫺ 1 P(M |x)
(3.61)
In the denominator, any class other than M can also be used. The unknown parameters, wi,0 , w i , i ⫽ 1, 2, . . . , M ⫺ 1, must be chosen to ensure that probabilities add to one. That is, M
P(i |x) ⫽ 1
(3.62)
i⫽1
Combining (3.61) and (3.62), it is straightforward to see that this type of linear modeling is equivalent to an exponential modeling of the a posteriori probabilities P(M |x) ⫽
1
exp wi,0 ⫹ w Ti x exp wi,0 ⫹ w Ti x P(i |x) ⫽ , i ⫽ 1, 2, . . . M ⫺ 1 T 1 ⫹ M⫺1 i⫽1 exp wi,0 ⫹ w i x 1⫹
M⫺1
(3.63)
i⫽1
(3.64)
For the two-class case, the previous equations are simplified to P(2 |x) ⫽
1 1 ⫹ exp(w0 ⫹ w T x)
(3.65)
P(1 |x) ⫽
exp(w0 ⫹ w T x) 1 ⫹ exp(w0 ⫹ w T x)
(3.66)
To estimate the set of the unknown parameters, a maximum likelihood approach is usually employed. Optimization is performed with respect to all parameters, which we can think of as the components of a parameter vector . Let x k , k ⫽ 1, 2, . . . , N , be the training feature vectors with known class labels. Let
117
“05-Ch03-SA272” 17/9/2008 page 118
118
CHAPTER 3 Linear Classifiers
us denote by x (m) ⫽ 1, 2, . . . , Nm , the vectors originating from class m ⫽ k , k 1, 2, . . . , M. Obviously, m Nm ⫽ N . The log-likelihood function to be optimized is given by L( ) ⫽ ln
N 1 "
p(x (1) k |1 ; )
k⫽1
N2 "
p(x (2) k |2 ; ) . . .
k⫽1
NM "
p(x (M) k |M ; )
(3.67)
k⫽1
Taking into account that p(x (m) k |m ; ) ⫽
(m) p(x (m) k )P(m |x k ; ) P(m )
(3.68)
(3.67) becomes L() ⫽
N1
ln P(1 |x (1) k )⫹
k⫽1
N2
ln P(2 |x (2) k ) ⫹ ... ⫹
k⫽1
NM
ln P(M |x (M) k )⫹C
(3.69)
k⫽1
where the explicit dependence on has been suppressed for notational simplicity and C is a parameter independent on equal to #N p(x k ) C ⫽ ln #M k⫽1 Nm m⫽1 P(m )
(3.70)
Inserting Eqs. (3.63) and (3.64) in (3.69), any optimization algorithm can then be used to perform the required maximization (Appendix C). More on the optimization task and the properties of the obtained solution can be found in, for example, [Ande 82, McLa 92]. There is a close relationship between the method of logistic discrimination and the LDA method, discussed in Chapter 2. It does not take much thought to realize that under the Gaussian assumption and for equal covariance matrices across all classes the following holds true. ln
1 P(1 |x) ⫽ (T2 ⌺⫺1 2 ⫺ 1 ⌺⫺1 1 ) ⫹ (1 ⫺ 2 )T ⌺⫺1 x P(2 |x) 2 ≡ w0 ⫹ w T x
Here, the equiprobable two-class case was considered for simplicity. However, LDA and logistic discrimination are not identical methods. Their (subtle) difference lies in the way the unknown parameters are estimated. In LDA, the class probability densities are assumed to be Gaussian and the unknown parameters are, basically, estimated by maximizing (3.67) directly. In this maximization, the marginal probability densities (p(x k )) play their own part, since they enter implicitly into the game. However, in the case of logistic discrimination, marginal densities contribute to C and do not affect the solution. Thus, if the Gaussian assumption is a reasonable one for the problem at hand, LDA is the natural approach since it exploits all available information. On the other hand, if this is not a good assumption, then logistic
“05-Ch03-SA272” 17/9/2008 page 119
3.7 Support Vector Machines
discrimination seems to be a better candidate, since it relies on fewer assumptions. However, in practice it has been reported [Hast 01] that there is little difference between the results obtained by the two methods. Generalizations of the logistic discrimination method to include nonlinear models have also been suggested. See, for example, [Yee 96, Hast 01].
3.7 SUPPORT VECTOR MACHINES 3.7.1 Separable Classes In this section,an alternative rationale for designing linear classifiers will be adopted. We will start with the two-class linearly separable task, and then we will extend the method to more general cases where data are not separable. Let x i , i ⫽ 1, 2, . . . , N , be the feature vectors of the training set, X. These belong to either of two classes, 1 , 2 , which are assumed to be linearly separable. The goal, once more, is to design a hyperplane g(x) ⫽ w T x ⫹ w0 ⫽ 0
(3.71)
that classifies correctly all the training vectors. As we have already discussed in Section 3.3, such a hyperplane is not unique. The perceptron algorithm may converge to any one of the possible solutions. Having gained in experience, this time we will be more demanding. Figure 3.9 illustrates the classification task with
x2
x1
FIGURE 3.9 An example of a linearly separable two-class problem with two possible linear classifiers.
119
“05-Ch03-SA272” 17/9/2008 page 120
120
CHAPTER 3 Linear Classifiers
two possible hyperplane1 solutions. Both hyperplanes do the job for the training set. However, which one of the two would any sensible engineer choose as the classifier for operation in practice, where data outside the training set will be fed to it? No doubt the answer is: the full-line one. The reason is that this hyperplane leaves more “room” on either side, so that data in both classes can move a bit more freely, with less risk of causing an error. Thus such a hyperplane can be trusted more, when it is faced with the challenge of operating with unknown data. Here we have touched a very important issue in the classifier design stage. It is known as the generalization performance of the classifier. This refers to the capability of the classifier, designed using the training data set, to operate satisfactorily with data outside this set. We will come to this issue over and over again. After the above brief discussion, we are ready to accept that a very sensible choice for the hyperplane classifier would be the one that leaves the maximum margin from both classes. Later on, at the end of Chapter 5, we will see that this sensible choice has a deeper justification, springing from the elegant mathematical formulation that Vapnik and Chervonenkis have offered to us. Let us now quantify the term margin that a hyperplane leaves from both classes. Every hyperplane is characterized by its direction (determined by w) and its exact position in space (determined by w0 ). Since we want to give no preference to either of the classes,then it is reasonable for each direction to select that hyperplane which has the same distance from the respective nearest points in 1 and 2 . This is illustrated in Figure 3.10. The hyperplanes shown with dark lines are the selected ones from the infinite set in the respective direction. The margin for direction “1” is 2z1 , and the margin for direction “2” is 2z2 . Our goal is to search for the direction that gives the maximum possible margin. However, each hyperplane is determined within a scaling factor. We will free ourselves from it, by appropriate scaling of all the candidate hyperplanes. Recall from Section 3.2 that the distance of a point from a hyperplane is given by z⫽
|g(x)| w
We can now scale w, w0 so that the value of g(x), at the nearest points in 1 , 2 (circled in Figure 3.10), is equal to 1 for 1 and, thus, equal to ⫺1 for 2 . This is equivalent with 1 ⫹ 1 ⫽ 2 1. Having a margin of w w w 2. Requiring that
1 We
w T x ⫹ w0 ⱖ 1,
᭙x ∈ 1
w T x ⫹ w0 ⱕ ⫺1,
᭙x ∈ 2
will refer to lines as hyperplanes to cover the general case.
“05-Ch03-SA272” 17/9/2008 page 121
3.7 Support Vector Machines
x2
direction 2 z2 z2
z1 z1
direction 1
x1
FIGURE 3.10 An example of a linearly separable two-class problem with two possible linear classifiers.
We have now reached the point where mathematics will take over. For each x i , we denote the corresponding class indicator by yi (⫹1 for 1 , ⫺1 for 2 .) Our task can now be summarized as: Compute the parameters w, w0 of the hyperplane so that to: 1 w2 2
minimize
J (w, w0 ) ≡
subject to
yi (w T x i ⫹ w0 ) ⱖ 1,
(3.72) i ⫽ 1, 2, . . . , N
(3.73)
Obviously, minimizing the norm makes the margin maximum. This is a nonlinear (quadratic) optimization task subject to a set of linear inequality constraints. The Karush–Kuhn–Tucker (KKT) conditions (Appendix C) that the minimizer of (3.72), (3.73) has to satisfy are ⭸ L(w, w0 , ) ⫽ 0 ⭸w
(3.74)
⭸ L(w, w0 , ) ⫽ 0 ⭸w0
(3.75)
i ⱖ 0,
i ⫽ 1, 2, . . . , N
i [yi (w T x i ⫹ w0 ) ⫺ 1] ⫽ 0,
i ⫽ 1, 2, . . . , N
(3.76) (3.77)
121
“05-Ch03-SA272” 17/9/2008 page 122
122
CHAPTER 3 Linear Classifiers
where is the vector of the Lagrange multipliers, i , and L(w, w0 , ) is the Lagrangian function defined as 1 T w w⫺ i [yi (w T x i ⫹ w0 ) ⫺ 1] 2 N
L(w, w0 , ) ⫽
(3.78)
i⫽1
Combining (3.78) with (3.74) and (3.75) results in w⫽
N
i yi x i
(3.79)
i⫽1 N
i yi ⫽ 0
(3.80)
i⫽1
Remarks ■
The Lagrange multipliers can be either zero or positive (Appendix C).Thus,the vector parameter w of the optimal solution is a linear combination of Ns ⱕ N feature vectors that are associated with i ⫽ 0. That is, w⫽
Ns
i yi x i
(3.81)
i⫽1
These are known as support vectors and the optimum hyperplane classifier as a support vector machine (SVM). As it is pointed out in Appendix C,a nonzero Lagrange multiplier corresponds to a so called active constraint. Hence, as the set of constraints in (3.77) suggests for i ⫽ 0, the support vectors lie on either of the two hyperplanes, that is, w T x ⫹ w0 ⫽ ⫾1
(3.82)
In other words, they are the training vectors that are closest to the linear classifier,and they constitute the critical elements of the training set. Feature vectors corresponding to i ⫽ 0 can either lie outside the “class separation band,” defined as the region between the two hyperplanes given in (3.82), or they can also lie on one of these hyperplanes (degenerate case,Appendix C). The resulting hyperplane classifier is insensitive to the number and position of such feature vectors, provided they do not cross the class separation band. ■
Although w is explicitly given, w0 can be implicitly obtained by any of the (complementary slackness) conditions (3.77), satisfying strict complementarity (i.e., i ⫽ 0, Appendix C). In practice, w0 is computed as an average value obtained using all conditions of this type.
“05-Ch03-SA272” 17/9/2008 page 123
3.7 Support Vector Machines
■
The cost function in (3.72) is a strict convex one (Appendix C),a property that is guaranteed by the fact that the corresponding Hessian matrix is positive definite [Flet 87]. Furthermore, the inequality constraints consist of linear functions. As discussed in Appendix C, these two conditions guarantee that any local minimum is also global and unique. This is most welcome. The optimal hyperplane classifier of a support vector machine is unique.
Having stated all these very interesting properties of the optimal hyperplane of a support vector machine, we next need to compute the involved parameters. From a computational point of view this is not always an easy task, and a number of algorithms exist, for example, [Baza 79]. We will move to a path, which is suggested to us by the special nature of our optimization task, given in (3.72) and (3.73). It belongs to the convex programming family of problems, since the cost function is convex and the constraints are linear and define a convex set of feasible solutions. As we discuss in Appendix C, such problems can be solved by considering the socalled Lagrangian duality. The problem can be stated equivalently by its Wolfe dual representation form, that is, maximize L(w, w0 , ) subject to
w⫽
N
i yi x i
(3.83) (3.84)
i⫽1 N
i yi ⫽ 0
(3.85)
i⫽1
ⱖ0
(3.86)
The two equality constraints are the result of equating to zero the gradient of the Lagrangian, with respect to w, w0 . We have already gained something. The training feature vectors enter into the problem via equality constraints and not inequality ones, which can be easier to handle. Substituting (3.84) and (3.85) into (3.83) and after a bit of algebra we end up with the equivalent optimization task ⎛
N
max ⎝ i⫽1 subject to
⎞ 1 T ⎠ i ⫺ i j yi yj x i xj 2
(3.87)
i,j
N
i yi ⫽ 0
(3.88)
i⫽1
ⱖ0
(3.89)
123
“05-Ch03-SA272” 17/9/2008 page 124
124
CHAPTER 3 Linear Classifiers
Once the optimal Lagrange multipliers have been computed, by maximizing (3.87), the optimal hyperplane is obtained via (3.84), and w0 via the complementary slackness conditions, as before. Remarks ■
Besides the more attractive setting of the involved constraints in (3.87),(3.88), there is another important reason that makes this formulation popular. The training vectors enter into the game in pairs, in the form of inner products. This is most interesting. The cost function does not depend explicitly on the dimensionality of the input space! This property allows for efficient generalizations in the case of nonlinearly separable classes. We will return to this at the end of Chapter 4.
■
Although the resulting optimal hyperplane is unique, there is no guarantee about the uniqueness of the associated Lagrange multipliers i . In words, the expansion of w in terms of support vectors in (3.84) may not be unique, although the final result is unique (Example 3.5).
3.7.2 Nonseparable Classes When the classes are not separable, the above setup is no longer valid. Figure 3.11 illustrates the case in which the two classes are not separable. Any attempt to draw a hyperplane will never end up with a class separation band with no x2
x2
FIGURE 3.11 In the nonseparable class case, points fall inside the class separation band.
“05-Ch03-SA272” 17/9/2008 page 125
3.7 Support Vector Machines
data points inside it, as was the case in the linearly separable task. Recall that the margin is defined as the distance between the pair of parallel hyperplanes described by w T x ⫹ w0 ⫽⫾ 1
The training feature vectors now belong to one of the following three categories: ■
Vectors that fall outside the band and are correctly classified. These vectors comply with the constraints in (3.73).
■
Vectors falling inside the band and are correctly classified. These are the points placed in squares in Figure 3.11, and they satisfy the inequality 0 ⱕ yi (w T x ⫹ w0 ) ⬍ 1
■
Vectors that are misclassified. They are enclosed by circles and obey the inequality yi (w T x ⫹ w0 ) ⬍ 0
All three cases can be treated under a single type of constraints by introducing a new set of variables, namely, yi [w T x ⫹ w0 ] ⱖ 1 ⫺ i
(3.90)
The first category of data corresponds to i ⫽ 0, the second to 0 ⬍ i ⱕ 1, and the third to i ⬎ 1. The variables i are known as slack variables. The optimizing task becomes more involved,yet it falls under the same rationale as before. The goal now is to make the margin as large as possible but at the same time to keep the number of points with ⬎ 0 as small as possible. In mathematical terms, this is equivalent to adopting to minimize the cost function 1 w2 ⫹ C I (i ) 2 N
J (w, w0 , ) ⫽
(3.91)
i⫽1
where is the vector of the parameters i and I (i ) ⫽
1 0
i ⬎ 0 i ⫽ 0
(3.92)
The parameter C is a positive constant that controls the relative influence of the two competing terms. However, optimization of the above is difficult since it involves a
125
“05-Ch03-SA272” 17/9/2008 page 126
126
CHAPTER 3 Linear Classifiers
discontinuous function I (·). As it is common in such cases, we choose to optimize a closely related cost function, and the goal becomes 1 w2 ⫹ C i 2 N
J (w, w0 , ) ⫽
minimize
(3.93)
i⫽1
yi [w T x i ⫹ w0 ] ⱖ 1 ⫺ i ,
subject to
i ⱖ 0,
i ⫽ 1, 2, . . . , N
i ⫽ 1, 2, . . . , N
(3.94) (3.95)
The problem is again a convex programming one,and the corresponding Lagrangian is given by 1 w2 ⫹ C i ⫺ i i 2
L(w, w0 , , , ) ⫽
N
⫺
N
N
i⫽1
i⫽1
i [yi (w T x i ⫹ w0 ) ⫺ 1 ⫹ i ]
(3.96)
i⫽1
The corresponding Karush–Kuhn–Tucker conditions are ⭸L ⫽0 ⭸w
or w ⫽
i yi x i
(3.97)
i⫽1
⭸L ⫽0 ⭸w0 ⭸L ⫽0 ⭸i
N
or
N
i yi ⫽ 0
(3.98)
i⫽1
C ⫺ i ⫺ i ⫽ 0,
i ⫽ 1, 2, . . . , N
(3.99)
i [yi (w T x i ⫹ w0 ) ⫺ 1 ⫹ i ] ⫽ 0,
i ⫽ 1, 2, . . . , N
(3.100)
or
i i ⫽ 0, i ⱖ 0,
i ⫽ 1, 2, . . . , N
i ⱖ 0,
i ⫽ 1, 2, . . . , N
The associated Wolfe dual representation now becomes maximize L(w, w0 , , , ) subject to
w⫽
N
i yi x i
i⫽1 N
i yi ⫽ 0
i⫽1
C ⫺ i ⫺ i ⫽ 0, i ⱖ 0, i ⱖ 0,
i ⫽ 1, 2, . . . , N i ⫽ 1, 2, . . . , N
(3.101) (3.102)
“05-Ch03-SA272” 17/9/2008 page 127
3.7 Support Vector Machines
Substituting the above equality constraints into the Lagrangian, we end up with ⎛
N
max ⎝ i⫽1 subject to
⎞ 1 T i ⫺ i j yi yj x i x j ⎠ 2
(3.103)
i,j
0 ⱕ i ⱕ C, N
i ⫽ 1, 2, . . . , N
i yi ⫽ 0
(3.104) (3.105)
i⫽1
Note that the Lagrange multipliers corresponding to the points residing either within the margin or on the wrong side of the classifier, that is, i ⬎ 0, are all equal to the maximum allowable value C. Indeed, at the solution, for i ⫽ 0 the KKT conditions give i ⫽ 0 leading to i ⫽ C. In other words,these points have the largest possible “share” in the final solution w.
3.7.3 The Multiclass Case In all our discussions, so far, we have been involved with the two-class classification task. In an M-class problem, a straightforward extension is to consider it as a set of M two-class problems (one-against-all). For each one of the classes, we seek to design an optimal discriminant function, gi (x), i ⫽ 1, 2, . . . , M, so that gi (x) ⬎ gj (x), ᭙j ⫽ i, if x ∈ i . Adopting the SVM methodology, we can design the discriminant functions so that gi (x) ⫽ 0 to be the optimal hyperplane separating class i from all the others. Thus, each classifier is designed to give gi (x) ⬎ 0 for x ∈ i and gi (x) ⬍ 0 otherwise. Classification is then achieved according to the following rule: assign x in i if i ⫽ arg max {gk (x)} k
This technique, however, may lead to indeterminate regions, where more than one gi (x) is positive (Problem 3.15). Another drawback of the technique is that each binary classifier deals with a rather asymmetric problem in the sense that training is carried out with many more negative than positive examples. This becomes more serious when the number of classes is relatively large. An alternative technique is the one-against-one. In this case, M(M ⫺ 1)/2 binary classifiers are trained and each classifier separates a pair of classes. The decision is made on the basis of a majority vote. The obvious disadvantage of the technique is that a relatively large number of binary classifiers has to be trained. In [Plat 00] a methodology is suggested that may speed up the procedure. A different and very interesting rationale has been adopted in [Diet 95]. The multiclass task is treated in the context of error correcting coding, inspired by the coding schemes used in communications. For a M-class problem a number of, say, L binary classifiers are used, where L is appropriately chosen by the designer. Each class is now represented by a binary code word of length L. During training, for
127
“05-Ch03-SA272” 17/9/2008 page 128
128
CHAPTER 3 Linear Classifiers
the ith classifier, i ⫽ 1, 2, . . . , L, the desired labels, y, for each class are chosen to be either ⫹1 or ⫺1. For each class, the desired labels may be different for the various classifiers. This is equivalent to constructing a matrix M ⫻ L of desired labels. For example, if M ⫽ 4 and L ⫽ 6, such a matrix can be ⎡
⫺1 ⎢⫹1 ⎢ ⎢ ⎣⫹1 ⫺1
⫺1 ⫺1 ⫹1 ⫺1
⫺1 ⫹1 ⫺1 ⫹1
⫹1 ⫹1 ⫺1 ⫺1
⫺1 ⫺1 ⫺1 ⫹1
⎤ ⫹1 ⎥ ⫺1⎥ ⎥ ⫹1⎦ ⫹1
(3.106)
In words, during training, the first classifier (corresponding to the first column of the previous matrix) is designed in order to respond (⫺1,⫹1,⫹1,⫺1) for patterns originating from classes 1 , 2 , 3 , 4 , respectively. The second classifier will be trained to respond (⫺1,⫺1,⫹1,⫺1), and so on. The procedure is equivalent to grouping the classes into L different pairs, and, for each pair, we train a binary classifier accordingly. For the case of our example and for the first binary classifier, class 1 has been grouped together with 4 and class 2 with class 3 . Each row must be distinct and corresponds to a class. For our example, and in the absence of errors, the outputs of the L classifiers for a pattern from class 1 will result in the code word (⫺1,⫺1,⫺1,⫹1,⫺1,⫹1), and so on. When an unknown pattern is presented, the output of each one of the binary classifiers is recorded, resulting in a code word. Then, the Hamming distance (number of places where two code words differ) of this code word is measured against the M code words, and the pattern is classified to the class corresponding to the smallest distance. Here in lies the power of the technique. If the code words are designed so that the minimum Hamming distance between any pair of them is, say, ( d, then ) a correct decision will still be reached even if the decisions of at most d⫺1 , out 2 of the L, classifiers are wrong, where · is the floor operation. For the matrix in (3.106) the minimum Hamming distance, between any pair, is equal to three. In [Diet 95], the method has been applied for numerical digit classification, and the grouping of the ten classes is done in such a way as to be meaningful. For example, one grouping is based on the existence in the numeric digits of a horizontal line (e.g., “4” and “2”), or the existence of a vertical line (e.g., “1” and “4”), and so on. An extension of this method, which is proposed in [Allw 00], takes into consideration the resulting values of the margin (when an SVM or another type of margin classifier, e.g., boosting classifiers discussed in Chapter 4, is used). In [Zhou 08], the composition of the individual binary problems and their number (code word length, L) is the result of a data-adaptive procedure that designs the code words by taking into account the inherent structure of the training data. All previous techniques are appropriate for any classifier. Another alternative, specific for SVMs, is to extentd the two class SVM mathematical formulation to the M-class problem, see, for example, [Vapn 98, Liu 06]. Comparative studies of the various methods for multiclass SVM classification can be found in [Rifk 04, Hsu 02, Fei 06].
“05-Ch03-SA272” 17/9/2008 page 129
3.7 Support Vector Machines
Remarks ■
The only difference between the linearly separable and nonseparable cases lies in the fact that for the latter one the Lagrange multipliers need to be bounded above by C. The linearly separable case corresponds to C→⬁, see Eqs. (3.104) and (3.89). The slack variables,i ,and their associated Lagrange multipliers,i , do not enter into the problem explicitly. Their presence is indirectly reflected through C.
■
A major limitation of support vector machines is the high computational burden required, both during training and in the test phase. A naive implementation of a quadratic programming (QP) solver takes O(N 3 ) operations, and its memory requirements are of the order of O(N 2 ). For problems with a relatively small number of training data, any general purpose optimization algorithm can be used. However, for a large number of training points (e.g., of the order of a few thousands), a naive QP implementation does not scale up well, and a special treatment is required. Training of SVM is usually performed in batch mode. For large problems this sets high demands on computer memory requirements. To attack such problems, a number of procedures have been devised. Their philosophy relies on the decomposition, in one way or another, of the optimization problem into a sequence of smaller ones, for example, [Bose 92, Osun 97, Chan 00]. The main rationale behind such algorithms is to start with an arbitrary data subset (chunk of data, working set) that can fit in the computer memory. Optimization is, then, performed on this subset via a general optimizer. Support vectors remain in the working set while others are replaced by new ones, outside the current working set, that violate severely the KKT conditions. It can be shown that this iterative procedure guarantees that the cost function is decreasing at each iteration step. In [Plat 99, Matt 99], the so called Sequential Minimal Optimization (SMO) algorithm is proposed where the idea of decomposition is pushed to its extreme and each working set consists of only two points. Its great advantage is that the optimization can now be performed analytically. In [Keer 01], a set of heuristics is used for the choice of the pair of points that constitute the working set. To this end, it is suggested that the use of two thresholded parameters can lead to considerable speedups. As suggested in [Plat 99, Platt 98], efficient implementations of such a scheme have an empirical training time complexity that scales between O(N ) and O(N 2.3 ). Theoretical issues related to the algorithm, such as convergence, are addressed in [Chen 06] and the references therein. The parallel implementation of the algorithm is considered in [Cao 06]. In [Joac 98] the working set is the result of a search for the steepest feasible direction. More recently, [Dong 05] suggested a technique to quickly remove most of
129
“05-Ch03-SA272” 17/9/2008 page 130
130
CHAPTER 3 Linear Classifiers
the nonsupport vectors, using a parallel optimization step, and the original problem can be split into many subproblems that can be solved more efficiently. In [Mavr 06], the geometric interpretation of SVMs (Section 3.7.5) is exploited, and the optimization task is treated as a minimum distance points search between convex sets. It is reported that substantial computational savings can be obtained compared to the SMO algorithm. A sequential algorithm, which operates on the primal problem formulation, has been proposed in [Navi 01], where an iterative reweighted least squares procedure is employed and alternates weight optimization with constraint forcing. An advantage of the latter technique is that it naturally leads to online implementations. Another trend is to employ an algorithm that aims at an approximate solution to the problem. In [Fine 01] a low-rank approximation is used in place of the the so-called kernel matrix, which is involved in the computations. In [Tsan 06, Hush 06] the issues of complexity and accuracy of the approximation are considered together. For example, in [Hush 06] polynomial-time algorithms are derived that produce approximate solutions with a guaranteed accuracy for a class of QP problems that include the SVM classifiers. For large problems, the test phase can also be quite demanding, if the number of support vectors is excessively high. Methods that speed up computations have also been suggested, for example, [Burg 97, Nguy 06].
Example 3.5 Consider the two-class classification task that consists of the following points: w1 : [1, 1]T , [1, ⫺1]T w2 : [⫺1, 1]T , [⫺1, ⫺1] Using the SVM approach, we will demonstrate that the optimal separating hyperplane (line) is x1 ⫽ 0 and that this is obtained via different sets of Lagrange multipliers. The points lie on the corners of a square, as shown in Figure 3.12. The simple geometry of the problem allows for a straightforward computation of the SVM linear classifier. Indeed, a careful observation of Figure 3.12 suggests that the optimal line g(x) ⫽ w1 x1 ⫹ w2 x2 ⫹ w0 ⫽ 0 is obtained for w2 ⫽ w0 ⫽ 0 and w1 ⫽ 1, that is, g(x) ⫽ x1 ⫽ 0 Hence for this case, all four points become support vectors, and the margin of the separating line from both classes is equal to 1. For any other direction, e.g., g1 (x) ⫽ 0, the margin is smaller. It must be pointed out that the same solution is obtained if one solves the associated KKT conditions (Problem 3.16.)
“05-Ch03-SA272” 17/9/2008 page 131
3.7 Support Vector Machines
x2
1
21
x1
1
21
g(x) 5 0
g1(x) 5 0
FIGURE 3.12 In this example all four points are support vectors. The margin associated with g1 (x) ⫽ 0 is smaller compared to the margin defined by the optimal g(x) ⫽ 0.
Let us now consider the mathematical formulation of our problem. The linear inequality constraints are w1 ⫹ w2 ⫹ w0 ⫺ 1 ⱖ 0 w 1 ⫺ w2 ⫹ w0 ⫺ 1 ⱖ 0 w 1 ⫺ w2 ⫺ w0 ⫺ 1 ⱖ 0 w 1 ⫹ w2 ⫺ w0 ⫺ 1 ⱖ 0 and the associated Lagrangian function becomes L(w2 , w1 , w0 , ) ⫽
w12 ⫹ w22 ⫺ 1 (w1 ⫹ w2 ⫹ w0 ⫺ 1) 2 ⫺ 2 (w1 ⫺ w2 ⫹ w0 ⫺ 1) ⫺ 3 (w1 ⫺ w2 ⫺ w0 ⫺ 1) ⫺ 4 (w1 ⫹ w2 ⫺ w0 ⫺ 1)
The KKT conditions are given by ⭸L ⫽ 0 ⇒ w1 ⫽ 1 ⫹ 2 ⫹ 3 ⫹ 4 ⭸w1
(3.107)
⭸L ⫽ 0 ⇒ w2 ⫽ 1 ⫹ 4 ⫺ 2 ⫺ 3 ⭸w2
(3.108)
⭸L ⫽ 0 ⇒ 1 ⫹ 2 ⫺ 3 ⫺ 4 ⫽ 0 ⭸w0
(3.109)
131
“05-Ch03-SA272” 17/9/2008 page 132
132
CHAPTER 3 Linear Classifiers
1 (w1 ⫹ w2 ⫹ w0 ⫺ 1) ⫽ 0
(3.110)
2 (w1 ⫺ w2 ⫹ w0 ⫺ 1) ⫽ 0
(3.111)
3 (w1 ⫺ w2 ⫺ w0 ⫺ 1) ⫽ 0
(3.112)
4 (w1 ⫹ w2 ⫺ w0 ⫺ 1) ⫽ 0
(3.113)
1 , 2 , 3 , 4 ⱖ 0
(3.114)
Since we know that the solution for w, w0 is unique, we can substitute the solution w1 ⫽ 1, w2 ⫽ w0 ⫽ 0 into the above equations. Then we are left with a linear system of three equations with four unknowns, that is, 1 ⫹ 2 ⫹ 3 ⫹ 4 ⫽ 1
(3.115)
1 ⫹ 4 ⫺ 2 ⫺ 3 ⫽ 0
(3.116)
1 ⫹ 2 ⫺ 3 ⫺ 4 ⫽ 0
(3.117)
which obviously has more than one solution. However, all of them lead to the unique optimal separating line.
Example 3.6 Figure 3.13 shows a set of training data points residing in the two-dimensional space and divided into two nonseparable classes. The full line in Figure 3.13a is the resulting hyperplane using Platt’s algorithm and corresponds to the value C ⫽ 0.2. Dotted lines meet the conditions given in (3.82) and define the margin that separates the two classes, for those points with
x2 5
x2 5
4
4
3
3
2
2
1
1
0
0
21
21
22 23 22 21 1
0
1 2 (a)
3
4
5 x1
22 23 22 21
0
1 2 (b)
3
4
5 x1
FIGURE 3.13 An example of two nonseparable classes and the resulting SVM linear classifier (full line) with the associated margin (dotted lines) for the values (a) C ⫽ 0.2 and (b) C ⫽ 1000. In the latter case, the location and direction of the classifier as well as the width of the margin have changed in order to include a smaller number of points inside the margin.
“05-Ch03-SA272” 17/9/2008 page 133
3.7 Support Vector Machines
i ⫽ 0. The setting in Figure 3.13b corresponds to C ⫽ 1000 and has been obtained with the same algorithm and the same set of trimming parameters (e.g., stopping criteria). It is readily observed that the margin associated with the classifier corresponding to the larger value of C is smaller. This is because the second term in (3.91) has now more influence in the cost, and the optimization process tries to satisfy this demand by reducing the margin and consequently the number of points with i ⬎ 0. In other words, the width of the margin does not depend entirely on the data distribution, as was the case with the separable class case, but is heavily affected by the choice of C. This is the reason SVM classifiers, defined by (3.91), are also known as soft margin classifiers.
3.7.4 -SVM Example 3.6 demonstrated the close relation that exists between the parameter C and the width of the margin obtained as a result of the optimization process. However, since the margin is such an important entity in the design of SVM (after all, the essence of the SVM methodology is to maximize it), a natural question that arises is why not involve it in a more direct way in the cost function, instead of leaving its control to a parameter (i.e., C) whose relation with the margin, although strong, is not transparent to us. To this end, in [Scho 00] a variant of the soft margin SVM was introduced. The margin is defined by the pair of hyperplanes w T x ⫹ w0 ⫽ ⫾
(3.118)
and ⱖ 0 is left as a free variable to be optimized. Under this new setting, the primal problem given in (3.93)–(3.95) can now be cast as minimize
J (w, w0 , , ) ⫽
N 1 1 w2 ⫺ ⫹ i 2 N
(3.119)
i⫽1
subject to
yi [w T x i ⫹ w0 ] ⱖ ⫺ i ,
i ⫽ 1, 2, . . . , N
(3.120)
i ⱖ 0,
i ⫽ 1, 2, . . . , N
(3.121)
ⱖ0
(3.122)
To understand the role of , note that for i ⫽ 0 the constraints in (3.120) state that 2 the margin separating the two classes is equal to w . In the previous formulation, also known as -SVM, we simply count and average the number of points with i ⬎ 0, whose number is now controlled by the margin variable . The larger the the wider the margin and the higher the number of points within the margin, for a specific direction w. The parameter controls the influence of the second term in the cost function, and its value lies in the range [0, 1]. ( We will revisit this issue later on.)
133
“05-Ch03-SA272” 17/9/2008 page 134
134
CHAPTER 3 Linear Classifiers
The Lagrangian function associated with the task (3.119)–(3.122) is given by L(w, w0 , , , , , ␦) ⫽
N N 1 1 w2 ⫺ ⫹ i ⫺ i i 2 N i⫽1
⫺
N
i⫽1
! i yi (w T x i ⫹ w0 ) ⫺ ⫹ i ⫺ ␦
(3.123)
i⫽1
Adopting similar steps as in Section 3.7.2, the following KKT conditions result: w⫽
N
i yi x i
(3.124)
i⫽1 N
i yi ⫽ 0
(3.125)
i⫽1
i ⫹ i ⫽
1 , N
N
i ⫽ 1, 2, . . . , N
i ⫺ ␦ ⫽
(3.126) (3.127)
i⫽1
! i yi (w T x i ⫹ w0 ) ⫺ ⫹ i ⫽ 0, i i ⫽ 0,
i ⫽ 1, 2, . . . , N
i ⫽ 1, 2, . . . , N ␦ ⫽ 0
i ⱖ 0,
i ⱖ 0,
␦ ⱖ 0,
(3.128) (3.129) (3.130)
i ⫽ 1, 2, . . . , N
(3.131)
The associated Wolfe dual representation is easily shown to be maximize
L(w, w0 , , , , ␦)
subject to
w⫽
N
(3.132)
i yi x i
(3.133)
i⫽1 N
i yi ⫽ 0
(3.134)
i⫽1
i ⫹ i ⫽ N
1 , N
i ⫽ 1, 2, . . . , N
i ⫺ ␦ ⫽
(3.135) (3.136)
i⫽1
i ⱖ 0, i ⱖ 0, ␦ ⱖ 0,
i ⫽ 1, 2, . . . , N
(3.137)
“05-Ch03-SA272” 17/9/2008 page 135
3.7 Support Vector Machines
If we substitute the equality constraints (3.133)–(3.136) in the Lagrangian, the dual problem becomes equivalent to (Problem 3.17 ) ⎛
⎞ 1 T max ⎝⫺ i j yi yj x i x j ⎠ 2
(3.138)
i, j
subject to
0 ⱕ i ⱕ N
1 , N
i ⫽ 1, 2, . . . , N
i yi ⫽ 0
(3.139)
(3.140)
i⫽1 N
i ⱖ
(3.141)
i⫽1
Once more, only the Lagrange multipliers enter into the problem explicitly, and and the slack variables, i , make their presence felt through the bounds appearing in the constraints. Observe that in contrast to (3.103) the cost function is now quadratically homogeneous and the linear term N i⫽1 i is not present. Also, the new formulation has an extra constraint. Remarks ■
■
[Chan 01] shows that the -SVM and the more standard SVM formulation [(3.103)–(3.105)], sometimes referred to as C-SVM, lead to the same solution for appropriate values of C and . Also, it is shown that in order for the optimization problem to be feasible, the constant must lie in a range 0 ⱕ min ⱕ ⱕ max ⱕ 1. Although both SVM formulations result in the same solution, for appropriate choices of and C the -SVM offers certain advantages to the designer. As we will see in the next section, it leads to a geometric interpretation of the SVM task for nonseparable classes. Furthermore, the constant , controlled by the designer, offers itself to serve two important bounds concerning (a) the error rate and (b) the number of the resulting support vectors. At the solution, the points lying either within the margin or outside it but on the wrong side of the separating hyperplane correspond to i ⬎ 0 and hence to i ⫽ 0 [Eq. (3.129)], forcing the respective Lagrange multipliers to be i ⫽ N1 [Eq. (3.126)]. Also, since at the solution, for ⬎ 0, ␦ ⫽ 0 [Eq. (3.130)], it turns out that N i⫽1 i ⫽ [Eq. (3.127)]. Combining these and taking into account that all points that lie in the wrong side of the classifier correspond to i ⬎ 0, the total number of errors can, at
135
“05-Ch03-SA272” 17/9/2008 page 136
136
CHAPTER 3 Linear Classifiers
most, be equal to N . upper-bounded as
Thus, the error rate, Pe , on the training set is Pe ⱕ .
(3.142)
Also, at the solution, from the constraints (3.127) and (3.126) we have that ⫽
N
i ⫽
i⫽1
Ns
i ⱕ
i⫽1
Ns 1 N
(3.143)
i⫽1
or N ⱕ Ns
(3.144)
Thus,the designer,by controlling the value of ,may have a feeling for both the error rate on the training set and the number of the support vectors to result from the optimization process. The number of the support vectors, Ns , is very important for the performance of the classifier in practice. First, as we have already commented, it directly affects the computational load, since large Ns means that a large number of inner products are to be computed for classifying an unknown pattern. Second, as we will see at the end of Section 5.10, a large number of support vectors can limit the error performance of the SVM classifier when it is fed with data outside the training set (this is also known as the generalization performance of the classifier). For more on the -SVM,the interested reader can consult [Scho 00, Chan 01, Chen 03],where implementation issues are also discussed.
3.7.5 Support Vector Machines: A Geometric Viewpoint In this section, we will close the circle around the SVM design task via a path that is very close to what we call common sense. Figure 3.14a illustrates the case of two separable data classes together with their respective convex hulls. The convex hull of a data set X is denoted as conv{X} and is defined as the intersection of all convex sets (see Appendix C.4) containing X. It can be shown (e.g., [Luen 69]) that conv{X} consists of all the convex combinations of the N elements of X. That is, conv{X} ⫽ y : y ⫽
N
i x i : x i ∈ X,
i⫽1 N i⫽1
i ⫽ 1, 0 ⱕ i ⱕ 1, i ⫽ 1, 2, . . . , N
(3.145)
“05-Ch03-SA272” 17/9/2008 page 137
3.7 Support Vector Machines
x2 3
x2 3
2
2
1
1
0
0
21
21
22
22
23 23 22 21
0
1
2 3 (a)
4
5
6
23 23 22 21
7x 1
0
1
2 3 (b)
4
5
6
7
x1
FIGURE 3.14 (a) A data set for two separable classes with the respective convex hulls. (b) The SVM optimal hyperplane bisects the segment joining the two nearest points between the convex hulls.
It turns out that solving the dual optimization problem in (3.87 )–(3.89) for the linearly separable task results in the hyperplane that bisects the linear segment joining two nearest points between the convex hulls of the data classes [Figure 3.14b]. In other words, searching for the maximum margin hyperplane is equivalent to searching for two nearest points between the corresponding convex hulls! Let us investigate this a bit further. Denote the convex hull of the vectors in class 1 as conv{X ⫹ } and the convex hull corresponding to class 2 as conv{X ⫺ }. Following our familiar notation, any ⫹ }, being a convex combination of all the points in , can be point in conv{X 1 written as i:yi ⫽1 i x i , and any point in conv{X ⫺ } as i:yi ⫽⫺1 i x i , provided that i fulfill the convexity constraints in (3.145). Searching for the closest points, it suffices to find the specific values of i , i ⫽ 1, 2, . . . N , such that min i x i ⫺ i x i 2 i:yi ⫽1 i:yi ⫽⫺1 subject to
i ⫽ 1,
i:yi ⫽1
i ⫽ 1
(3.146)
(3.147)
i:yi ⫽⫺1
i ⱖ 0,
i ⫽ 1, 2, . . . N
(3.148)
Elaborating the norm in (3.146) and reshaping the constraints in (3.147), we end up with the following equivalent formulation. minimize
yi yj i j x Ti x j
(3.149)
i,j
subject to
N
yi i ⫽ 0,
i⫽1
i ⱖ 0,
N
i ⫽ 2
(3.150)
i⫽1
i ⫽ 1, 2, . . . N
(3.151)
137
“05-Ch03-SA272” 17/9/2008 page 138
138
CHAPTER 3 Linear Classifiers
It takes a few lines of algebra to show that the optimization task in (3.87)–(3.89) results in the same solution as the task given in (3.149)–(3.151) ([Keer 00] and Problem 3.18). Having established the geometric interpretation of the SVM optimization task, any algorithm that has been developed to search for nearest points between convex hulls (e.g., [Gilb 66, Mitc 74, Fran 03]) can now, in principle, be mobilized to compute the maximum margin linear classifier. It is now the turn of the nonseparable class problem to enter into the game, which, at this point becomes more exciting. Let us return to the -SVM formulation and reparameterize the primal problem in (3.119)–(3.122) by dividing the cost 2 function by 2 and the set of constraints by ([Crisp 99]). Obviously, this has no effect on the solution. The optimization task now becomes minimize
J (w, w0 , , ) ⫽ w2 ⫺ 2 ⫹
N
i
(3.152)
i⫽1
subject to
yi [w T x i ⫹ w0 ] ⱖ ⫺ i , i ⱖ 0,
i ⫽ 1, 2, . . . , N
i ⫽ 1, 2, . . . , N
(3.153) (3.154)
ⱖ0
(3.155)
2 where ⫽ N and we have kept, for economy, the same notation, although the parameters in (3.152)–(3.155) are scaled versions of those in (3.119)–(3.122). That i w0 is, w → w , w0 → , → , i → . Hence, the solution obtained via (3.152)– (3.155) is a scaled version of the solution resulting via (3.119)–(3.122). The Wolfe dual representation of the primal problem in (3.152)–(3.155) is easily shown to be equivalent to
minimize
yi yj i j x Ti x j
(3.156)
i,j
subject to
i
yi i ⫽ 0,
i ⫽ 2
(3.157)
i
0 ⱕ i ⱕ , i ⫽ 1, 2, . . . N
(3.158)
This set of relations is almost the same as those defining the nearest points between the convex hulls in the separable class case, (3.149)–(3.151), with a small, yet significant, difference. The Lagrange multipliers are bounded by , and for ⬍ 1 they are not permitted to span their entire allowable range (i.e., [0, 1]).
3.7.6 Reduced Convex Hulls The reduced convex hull (RCH) of a (finite) vector set, X, is denoted as R(X, ) and is defined as the convex set
“05-Ch03-SA272” 17/9/2008 page 139
3.7 Support Vector Machines
R(X, ) ⫽ y : y ⫽
N
i x i : x i ∈ X,
i⫽1 N
i ⫽ 1, 0 ⱕ i ⱕ , i ⫽ 1, 2, . . . , N
(3.159)
i⫽1
It is apparent from the previous definition that R(X, 1) ≡ conv{X} and that R(X, ) ⊆ conv{X}
(3.160)
Figure 3.15a shows the respective convex hulls for the case of two intersecting data classes. In Figure 3.15b, full lines indicate the convex hulls, conv{X ⫹ } and conv{X ⫺ }, and the dotted lines the reduced convex hulls R(X ⫹ , ), R(X ⫺ , ), for two different values of ⫽ 0.4 and ⫽ 0.1, respectively. It is readily apparent that the smaller the value of , the smaller the size of the reduced convex hull. For small enough values of , one can make R(X ⫹ , ) and R(X ⫺ , ) nonintersecting. Adopting a procedure similar to the one that led to (3.149)–(3.151), it is not difficult to see that finding two nearest points between R(X ⫹ , ) and R(X ⫺ , ) results in the -SVM dual optimization task given in (3.156)–(3.158). Observe that the only difference between the latter and the task for the separable case,defined in (3.149)– (3.151), lies in the range in which the Lagrange multipliers are allowed to be. In the separable class case, the constraints (3.150) and (3.151) imply that 0 ⱕ i ⱕ 1, which in its geometric interpretation means that the full convex hulls are searched for the nearest points. In contrast,in the nonseparable class case a lower upper bound (i.e., x2 5
x2 5
4
4
3
3
2
2
1
1
0
0
1
2
3 (a)
4
5x
0 1
0
1
2
3
4
5x
1
(b)
FIGURE 3.15 (a) Example of a data set with two intersecting classes and their respective convex hulls. (b) The convex hulls (indicated by full lines) and the resulting reduced convex hulls (indicated by dotted lines) corresponding to ⫽ 0.4 and ⫽ 0.1, respectively, for each class. The smaller the value of the smaller the RCH size.
139
“05-Ch03-SA272” 17/9/2008 page 140
140
CHAPTER 3 Linear Classifiers
ⱕ 1) is imposed for the Lagrange multipliers. From the geometry point of view, this means that the search for the nearest points is limited within the respective reduced convex hulls. Having established the geometric interpretation of the -SVM dual representation form,let us follow pure geometric arguments to draw the separating hyperplane. It is natural to choose it as the one bisecting the line segment joining two nearest points between the reduced convex hulls. Let x ⫹ and x ⫺ be two nearest points, with x ⫹ ∈ R(X ⫹ , ) and x ⫺ ∈ R(X ⫺ , ). Also, let i , i ⫽ 1, 2, . . . , N , be the optimal set of multipliers resulting from the optimization task. Then, as can be deduced from Figure 3.16, w ⫽ x⫹ ⫺ x⫺ ⫽
i x i ⫺
i:yi ⫽1
⫽
N
i x i
(3.161)
i:yi ⫽⫺1
i yi x i
(3.162)
i⫽1
This is the same (within a scaling factor) as the w obtained from the KKT conditions associated with the -SVM task [Eq. (3.124)]. Thus, both approaches result in a separating hyperplane pointing in the same direction (recall from Section 3.2 that w defines the direction of the hyperplane). However, it is early to say that the two solutions are exactly the same. The hyperplane bisecting the line segment x2 5
4
3
w
x1 x*
x2
2
1
0
0
1
2
3
4
5x
1
FIGURE 3.16 The optimal linear classifier resulting as the bisector of the segment joining the two closest points between the reduced convex hulls of the classes, for the case of the data set shown in Figure 3.15 and for ⫽ 0.1.
“05-Ch03-SA272” 17/9/2008 page 141
3.7 Support Vector Machines
joining the nearest points crosses the middle of this segment; that is, the point x ∗ ⫽ 12 (x ⫹ ⫹ x ⫺ ). Thus, w T x ∗ ⫹ w0 ⫽ 0
(3.163)
from which we get ⎛ ⎞ 1 T⎝ w0 ⫽ ⫺ w i x i ⫹ i x i ⎠ 2 i:yi ⫽1
(3.164)
i:yi ⫽⫺1
This value for w0 is, in general, different from the value resulting from the KKT conditions in (3.128). In conclusion, the geometric approach in the case of the nonseparable problem is equivalent to the -SVM formulation only to the extent that both approaches result in hyperplanes pointing in the same direction. However, note that the value in Eq. (3.128) can be obtained from that given in Eq. (3.164) in a trivial way [Crisp 99]. Remarks ■
2 The choice of and consequently of ⫽ N must guarantee that the feasible region is nonempty (i.e., a solution exists,Appendix C) and also that the solution is a nontrivial one (i.e., w ⫽ 0). Let N ⫹ be the number of points in X ⫹ and N ⫺ the number of points in X ⫺ , where N ⫹ ⫹ N ⫺ ⫽ N . Let Nmin ⫽ min{N ⫹ , N ⫺ }. Then it is readily seen from the crucial constraint 0 ⱕ i ⱕ and the fact that i i ⫽ 1, in the definition of the reduced convex 1 . This readily suggests that cannot take any value hull, that ⱖ min ⫽ Nmin but must be upper-bounded as
ⱕ max ⫽ 2
Nmin ⱕ1 N
Also, if the respective reduced convex hulls intersect, then the distance between the closest points is zero, leading to the trivial solution (Problem 3.19). Thus, nonintersection is guaranteed for some value max such that ⱕ max ⱕ 1, which leads to ⱖ min ⫽
2 max N
From the previous discussion it is easily deduced that for the feasible region to be nonempty it is required that R(X ⫹ , min ) ∩ R(X ⫺ , min ) ⫽ ∅
If N ⫹ ⫽ N ⫺ ⫽ N2 , it is easily checked out that in this case each of the reduced convex hulls is shrunk to a point, which is the centroid of the respective class
141
“05-Ch03-SA272” 17/9/2008 page 142
142
CHAPTER 3 Linear Classifiers
(e.g., N2 i:yi ⫽1 x i ). In other words, a solution is feasible if the centroids of the two classes do not coincide. Most natural! ■
Computing the nearest points between reduced convex hulls turns out not to be a straightforward extension of the algorithms that have been developed for computing nearest points between convex hulls. This is because,for the latter case, such algorithms rely on the extreme points of the involved convex hulls. However, in this case, extreme points coincide with points in the original data sets, that is, X ⫹ , X ⫺ . This is not the case for the reduced convex hulls, where extreme points are combinations of points of the original data sets. The lower the value of , the higher the number of data samples that contribute to an extreme point in the respective reduced convex hull. A neat solution to this problem is given in [Mavr 05, Mavr 06, Mavr 07, Tao 04, Theo 07]. The developed nearest point algorithms are reported to offer computational savings,which in some cases can be significant,compared to the more classical algorithms in [Plat 99, Keer 01].
3.8 PROBLEMS 3.1 Explain why the perceptron cost function is a continuous piecewise linear function. 3.2 Show that if k ⫽ in the perceptron algorithm,the algorithm converges after 2 ⫺␣w ∗ k0 ⫽ w(0) 2 (2⫺) steps, where ␣ ⫽ |␥| and ⬍ 2. 3.3 Show that the reward and punishment form of the perceptron algorithm converges in a finite number of iteration steps. 3.4 Consider a case in which class 1 consists of the two feature vectors [0, 0]T and [0, 1]T and class 2 of [1, 0]T and [1, 1]T . Use the perceptron algorithm in its reward and punishment form, with ⫽ 1 and w(0) ⫽ [0, 0]T , to design the line separating the two classes. 3.5 Consider the two-class task of Problem 2.12 of the previous chapter with T1 ⫽ [1, 1],
T2 ⫽ [0,0],
12 ⫽ 22 ⫽ 0.2
Produce 50 vectors from each class. To guarantee linear separability of the classes, disregard vectors with x1 ⫹ x2 ⬍ 1 for the [1, 1] class and vectors with x1 ⫹x2 ⬎1 for the [0, 0] class. In the sequel,use these vectors to design a linear classifier using the perceptron algorithm of (3.21)–(3.23). After convergence, draw the corresponding decision line. 3.6 Consider once more the classification task of Problem 2.12. Produce 100 samples for each of the classes. Use these data to design a linear classifier via the LMS algorithm. Once all samples have been presented to the algorithm,
“05-Ch03-SA272” 17/9/2008 page 143
3.8 Problems
draw the corresponding hyperplane to which the algorithm has converged. Use k ⫽ ⫽ 0.01. 3.7 Show, using Kesler’s construction, that the tth iteration step of the reward and punishment form of the perceptron algorithm (3.21)–(3.23), for an x (t) ∈ i , becomes w i (t ⫹ 1) ⫽ w i (t) ⫹ x (t)
if w Ti (t)x (t) ⱕ w Tj (t)x (t) , j ⫽ i
w j (t ⫹ 1) ⫽ w j (t) ⫺ x (t)
if w Ti (t)x (t) ⱕ w Tj (t)x (t) , j ⫽ i
w k (t ⫹ 1) ⫽ w k (t),
᭙k ⫽ j and k ⫽ i
3.8 Show that the sum of error squares optimal weight vector tends asymptotically to the MSE solution. 3.9 Repeat Problem 3.6 and design the classifier using the sum of error squares criterion. 3.10 Show that the design of an M class linear,sum of error squares optimal,classifier reduces to M equivalent ones, with scalar desired responses. 3.11 Show that, if x, y are jointly Gaussian, the regression of y on x is given by ␣y x ␣y x E[ y|x] ⫽ ⫹ y ⫺ , x x
where ⌺ ⫽
x2
␣x y
␣x y
y2
(3.165)
3.12 Let an M class classifier be given in the form of parameterized functions g(x; w k ). The goal is to estimate the parameters w k so that the outputs of the classifier give desired response values, depending on the class of x. Assume that as x varies randomly in each class, the classifier outputs vary around the corresponding desired response values,according to a Gaussian distribution of known variance,assumed to be the same for all outputs. Show that in this case the sum of error squares criterion and the ML estimation result in identical estimates. Hint: Take N training data samples of known class labels. For each of them form yi ⫽ g(x i ; w k )⫺dki ,where dki is the desired response for the kth class of the ith sample. The yi ’s are normally distributed with zero mean and variance 2 . Form the likelihood function using the yi ’s. 3.13 In a two-class problem, the Bayes optimal decision surface is given by g(x) ⫽ P(1 |x)⫺P(2 |x) ⫽ 0. Show that if we train a decision surface f (x; w) in the MSE so as to give ⫹1(⫺1) for the two classes, respectively, this is equivalent to approximating g(·) in terms of f (·; w), in the MSE optimal sense. 3.14 Consider a two-class classification task with jointly Gaussian distributed feature vectors and with the same variance ⌺ in both classes. Design the linear MSE classifier and show that in this case the Bayesian classifier (Problem 2.11) and the resulting MSE one differ only in the threshold value. For simplicity, consider equiprobable classes.
143
“05-Ch03-SA272” 17/9/2008 page 144
144
CHAPTER 3 Linear Classifiers
Hint: To compute the MSE hyperplane w T x ⫹ w0 ⫽ 0,increase the dimension of x by one and show that the solution is provided by
1 E[x] w 2 (1 ⫺ 2 ) ⫽ w0 E[x]T 1 0 R
Then relate R with ⌺ and show that the MSE classifier takes the form 1 (1 ⫺ 2 )T ⌺⫺1 x ⫺ (1 ⫹ 2 ) ⱖ 0 2
3.15 In an M class classification task, the classes can be linearly separated. Design M hyperplanes,so that hyperplane gi (x) ⫽ 0 leaves class i on its positive side and the rest of the classes on its negative side. Demonstrate via an example, for example, M ⫽ 3, that the partition of the space using this rule creates indeterminate regions (where no training data exist) for which more than one gi (x) is positive or all of them are negative. 3.16 Obtain the optimal line for the task of Example 3.5, via the KKT conditions. Restrict the search for the optimum among the lines crossing the origin. 3.17 Show that if the equality constraints (3.133)–(3.136) are substituted in the Lagrangian (3.123), the dual problem is described by the set of relations in (3.138)–(3.141). 3.18 Show that for the case of two linearly separable classes the hyperplane obtained as the SVM solution is the same as that bisecting the segment joining two closest points between the convex hulls of the classes. 3.19 Show that if in the -SVM is chosen smaller than min , it leads to the trivial zero solution. 3.20 Show that if the soft margin SVM cost function is chosen to be N C 2 1 ||w||2 ⫹ i 2 2 i⫽1
the task can be transformed into an instance of the class-separable case problem [Frie 98].
MATLAB PROGRAMS AND EXERCISES Computer Programs 3.1 Perceptron algorithm. Write a MATLAB function for the perceptron algorithm. This will take as inputs: (a) a matrix X containing N l-dimensional column vectors, ( b) an N -dimensional row vector y, whose ith component contains
“05-Ch03-SA272” 17/9/2008 page 145
MATLAB Programs and Exercises
the class (⫺1 or ⫹1) where the corresponding vector belongs, and (c) an initial value vector w_ini for the parameter vector. It returns the estimated parameter vector.
Solution function w=perce(X,y,w_ini) [l,N]=size(X); max_iter=10000; % Maximum allowable number of iterations rho=0.05; % Learning rate w=w_ini; % Initialization of the parameter vector iter=0; % Iteration counter mis_clas=N; % Number of misclassified vectors while (mis_clas>0) && (iter<max_iter) iter=iter+1; mis_clas=0; gradi=zeros(l,1);% Computation of the "gradient" % term for i=1:N if((X(:,i)'*w)*y(i)<0) mis_clas=mis_clas+1; gradi=gradi+rho*(-y(i)*X(:,i)); end end w=w-rho*gradi; % Updating the parameter vector end
3.2 Sum of error squares classifier: Write a MATLAB function that implements the sum of error squares classifier for two classes. This will take as inputs: (a) a matrix X containing N l-dimensional column vectors, and ( b) an N -dimensional row vector y whose ith component contains the class (⫺1 or ⫹1) where the corresponding vector belongs. It returns the estimated parameter vector.
Solution function w=SSErr(X,y) w=inv(X*X')*(X*y');
3.3 LMS algorithm. Write a MATLAB function for the LMS algorithm. This will take as inputs: (a) a matrix X containing N l-dimensional column vectors, (b) an N -dimensional row vector y whose ith component contains the class (⫺1 or ⫹1) where the corresponding vector is assigned,and (c) an initial value vector w_ini for the parameter vector. It returns the estimated parameter vector.
145
“05-Ch03-SA272” 17/9/2008 page 146
146
CHAPTER 3 Linear Classifiers
Solution function w=LMSalg(X,y,w_ini) [l,N]=size(X); rho=0.1; % Learning rate initialization w=w_ini; % Initialization of the parameter vector for i=1:N w=w+(rho/i)*(y(i)-X(:,i)'*w)*X(:,i); end
Computer Experiments Note: In the sequel, it is advisable to use the command randn('seed',0)
before generating the data sets, in order to initialize the Gaussian random number generator to 0 (or any other fixed number). This is important for the reproducibility of the results. 3.1 a. Generate two data sets X1 and X1⬘ of N ⫽ 200 two-dimensional vectors each. The first half of the vectors stem from the normal distribution with m1 ⫽ [⫺5, 0]T and S1 ⫽ I , while the second half of the vectors stem from the normal distribution with m1 ⫽ [5, 0]T and S1 ⫽ I , where I is the identity 2 ⫻ 2 matrix. Append each vector of both X1 and X1⬘ by inserting an additional coordinate, which is set equal to 1. b. Apply the perceptron algorithm, the sum of error squares classifier, and the LMS algorithm on the previous data set, using various initial values for the parameter vector (where necessary). c. Measure the performance of each one of the above methods on both X1 and X1⬘ . d. Plot the data sets X1 and X1⬘ as well as the line corresponding to the parameter vector w. 3.2 Repeat experiment 1 using now the sets X2 and X2⬘ whose first half of their vectors stem from the normal distribution with m1 ⫽ [⫺2, 0]T and S1 ⫽ I , while the second half of their vectors stem from the normal distribution with m1 ⫽ [2, 0]T and S1 ⫽ I . 3.3 Repeat experiment 3.1 using now the sets X3 and X3⬘ whose first half of the vectors stem from the normal distribution with m1 ⫽ [⫺1, 0]T and S1 ⫽ I , while the second half of the vectors stem from the normal distribution with m1 ⫽ [1, 0]T and S1 ⫽ I . 3.4 Discuss the results obtained by the previous experiments.
“05-Ch03-SA272” 17/9/2008 page 147
References
REFERENCES [Allw 00] Allwein E.L., Schapire R.E., Singer Y. “Reducing multiclass to binary: aunifying approach for margin classifiers,” Journal of Machine Learning Research,Vol. 1, pp. 113–141, 2000. [Ande 82] Anderson J.A.“Logistic discrimination,”in Handbook of Statistics (Krishnaiah R.P.,Kanal L.N., eds.), North Holland, 1982. [Baza 79] Bazaraa M.S., Shetty C.M. Nonlinear Programming, John Wiley & Sons, 1979. [Bish 95] Bishop C. Neural Networks for Pattern Recognition, Oxford University Press, 1995. [Bose 92] Bose B.E., Guyon I.M.,Vapnik,V.N. “A training algorithm for optimal margin classifiers,” Proceedings of the 5th Annual Workshop on Computational Learning Theory, pp. 144–152, Morgan Kaufman, 1992. [Burg 97] Burges C.J.C., Schölkoff B. “Improving the accuracy and speed of support vectors learning machines,” in Advances in Neural Information Processing Systems 9 (Mozer M. Jordan M., Petsche T., eds.), pp. 375–381, MIT Press, 1997. [Cao 06] Cao L.J., Keerthi S.S., Ong C.-J., Zhang J.Q., Periyathamby V., Fu X.J., Lee H.P. “Parallel sequential minimal optimization for the training of support vector machines,” IEEE Transactions on Neural Networks,Vol. 17(4), pp. 1039–1049, 2006. [Chan 00] Chang C.C., Hsu C.W., Lin C.J. “The analysis of decomposition methods for SVM,” IEEE Transactions on Neural Networks,Vol. 11(4), pp. 1003–1008, 2000. [Chan 01] Chang C.C., Lin C.J. “Training -support vector classifiers: Theory and algorithms,” Neural Computation Vol. 13(9), pp. 2119–2147, 2001. [Chen 06] Chen P.-H., Fan R.-E., Lin C.-J. “A study on SMO-type decomposition for support vector machines,” IEEE Transactions on Neural Networks,Vol. 17(4), pp. 893–908, 2006. [Chen 03] Chen P-H., Lin C.J., Schölkopf B. “A tutorial on -support vector machines,” Applied Stochastic Models in Business and Industry,Vol. 21, pp. 111–136, 2005. [Cid 99] Cid-Sueiro J.,Arribas J.I., Urban-Munoz S., Figuieras-Vidal A.R. “Cost functions to estimate a-posteriori probabilities in multi-class problems,”IEEE Transactions on Neural Networks,Vol. 10(3), pp. 645–656, 1999. [Crisp 99] Crisp D.J., Burges C.J.C. “A geometric interpretation of -SVM classifiers,” Proceedings of Neural Information Processing,Vol. 12, MIT Press, 1999. [Diet 95] Dietterich T.G., Bakiri G. “Solving multi-class learning problems via error-correcting output codes,” Journal of Artificial Intelligence Research,Vol. 2, pp. 263–286, 1995. [Dong 05] Dong J.X., Krzyzak A., Suen C.Y. “Fast SVM training algorithm with decomposition on very large data sets,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27(4), pp. 603–618, 2005. [Fei 06] Fei B., Liu J. “Binary tree of SVM: A new fast multi-class training and classification algorithm,” IEEE Transactions on Neural Networks,Vol. 17(3), pp. 696–704, 2006. [Fine 01] Fine S., Scheinberg K. “Efficient SVM training using low rank kernel representations,” Journal of Machine Learning Research,Vol. 2, pp. 243–264, 2001. [Flet 87] Fletcher R. Practical Methods of Optimization, 2nd ed., John Wiley & Sons, 1987. [Fran 03] Franc V., Hlaváˇc V.“An iterative algorithm learning the maximal margin classifier,”Pattern Recognition,Vol. 36, pp. 1985–1996, 2003. [Frea 92] Frean M.“A thermal perceptron learning rule,”Neural Computation,Vol. 4,pp. 946–957, 1992.
147
“05-Ch03-SA272” 17/9/2008 page 148
148
CHAPTER 3 Linear Classifiers
[Frie 98] FriessT.T.“The kernel adatron with bias and soft margin,”Technical Report,The University of Sheffield, Dept. of Automatic Control, England, 1998. [Fuku 90] Fukunaga K. Introduction to Statistical Pattern Recognition, 2nd ed.,Academic Press, 1990. [Gal 90] Gallant S.I. “Perceptron based learning algorithms,” IEEE Transactions on Neural Networks,Vol. 1(2), pp. 179–191, 1990. [Gema 92] Geman S.,Bienenstock E.,Doursat R.“Neural networks and the bias/variance dilemma,” Neural Computation,Vol. 4, pp. 1–58, 1992. [Gilb 66] Gilbert E.G. “An iterative procedure for computing the minimum of a quadratic form on a convex set,” SIAM Journal on Control,Vol. 4(1), pp. 61–79, 1966. [Guer 04] Guerrero-Curieses A., Cid-Sueiro J., Alaiz-Rodriguez R., Figueiras-Vidal A.R. “Local estimation of posterior class probabilities to minimize classification errors,”IEEE Transactions on Neural Networks,Vol. 15(2), pp. 309–317, 2004. [Hast 01] Hastie T.,Tibsharini R., Friedman J. The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer, 2001. [Hayk 96] Haykin S. Adaptive Filter Theory, 3rd ed., Prentice Hall, 1996. [Ho 65] Ho Y.H., Kashyap R.L. “An algorithm for linear inequalities and its applications,” IEEE Transactions on Electronic Computers,Vol. 14(5), pp. 683–688, 1965. [Hsu 02] Hsu C.W., Lin C.J. “A comparison of methods for multi-class SVM,” IEEE Transactions on Neural Networks,Vol. 13, pp. 415–425, 2002. [Hryc 92] Hrycej T., Modular Learning in Neural Networks, John Wiley & Sons, 1992. [Hush 06] Hush D.,Kelly P. Scovel C.,Steinwart I.“QP algorithms with guaranteed accuracy and run time for support vector machines,”Journal of Machine Learning Research,Vol. 7,pp. 733–769, 2006. [Joac 98] JoachimsT.“Making large scale support vector machines practical,”inAdvances in Kernel Methods, (Schölkoph B., Burges C.J.C., Smola A. eds.), MIT Press, 1998. [Kalou 93] Kalouptsidis N.,Theodoridis S. Adaptive System Identification and Signal Processing Algorithms, Prentice Hall, 1993. [Keer 00] Keerthi S.S., Shevade S.K., Bhattacharyya C., Murthy K.R.K. “A fast iterative nearest point algorithm for support vector machine classifier design,” IEEE Transactions on Neural Networks,Vol. 11(1), pp. 124–136, 2000. [Keer 01] Keerthi S.S.,Shevade S.K.,Bhattacharyya C.,Murth K.R.K.“Improvements to Platt’s SMO algorithm for SVM classifier design,” Neural Computation,Vol. 13, pp. 637–649, 2001. [Liu 06] Liu Y., You Z., Cao L. “A novel and quick SVM-based multi-class classifier,” Pattern Recognition,Vol. 39(11), pp. 2258–2264, 2006. [Luen 69] Luenberger D.G. Optimization byVector Space Methods,John Wiley & Sons,New York, 1969. [McLa 92] McLachlan G. J. Discriminant Analysis and Statistical Pattern Recognition,JohnWiley & Sons, 1992. [Matt 99] Mattera D., Palmieri F., Haykin S. “An explicit algorithm for training support vector machines,” IEEE Signal Processing Letters,Vol. 6(9), pp. 243–246, 1999. [Mavr 07] Mavroforakis M., Sdralis M., Theodoridis S. “A geometric nearest point algorithm for the efficient solution of the SVM classification task,” IEEE Transactions on Neural Networks, Vol. 18(5), pp. 1545–1550, 2007.
“05-Ch03-SA272” 17/9/2008 page 149
References
[Mavr 06] Mavroforakis M., Theodoridis S. “A geometric approach to Support Vector Machine (SVM) classification,” IEEE Transactions on Neural Networks, Vol. 17(3), pp. 671–682, 2006. [Mavr 05] Mavroforakis M.,Theodoridis S. “Support Vector Machine classification through geometry,” Proceedings of the XII European Signal Processing Conference (EUSIPCO), Antalya, Turkey, 2005. [Mitc 74] Mitchell B.F., Demyanov V.F., Malozemov V.N. “Finding the point of a polyhedron closest to the origin,” SIAM Journal on Control,Vol. 12, pp. 19–26, 1974. [Min 88] Minsky M. L., Papert S.A. Perceptrons, expanded edition, MIT Press, MA, 1988. [Muse 97] Muselli M. “On convergence properties of pocket algorithm,” IEEE Transactions on Neural Networks,Vol. 8(3), pp. 623–629, 1997. [Navi 01] Navia-Vasquez A., Perez-Cuz F., Artes-Rodriguez A., Figueiras-Vidal A. “Weighted least squares training of support vector classifiers leading to compact and adaptive schemes,” IEEE Transactions on Neural Networks,Vol. 12(5), pp. 1047–1059, 2001. [Nguy 06] Nguyen D., Ho T. “A bottom-up method for simplifying support vector solutions,” IEEE Transactions on Neural Networks,Vol. 17(39), pp. 792–796, 2006. [Osun 97] Osuna E., Freund R., Girosi F., “An improved training algorithm for support vector machines,” Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 276–285,Amelia Island, FL, 1997. [Papo 91] Papoulis A. Probability, Random Variables and Stochastic Processes, 3rd ed., McGrawHill, 1991. [Pear 90] Pearlmutter B., Hampshire J. “Equivalence proofs for multilayer perceptron classifiers and the Bayesian discriminant function,”Proceedings Connectionists Models Summer School, Morgan Kauffman, 1990. [Plat 00] Platt J.C., Cristianini N., Shawe-Taylor J. “Large margin DAGs for the multiclass classification,”in Advances in Neural Information Processing, (Smola S.A., LeenT.K., Müller K.R., eds.), Vol. 12, pp. 547–553, MIT Press, 2000. [Plat 99] Platt J. “Fast training of support vector machines using sequential minimal optimization,” in Advances in Kernel Methods: Support Vector Learning (Scholkopf B., Burges C.J.C., Smola A. J., eds), pp. 185–208, MIT Press, 1999. [Platt 98] Platt J. “Sequential minimal optimization: A fast algorithm for training support vector machines,”Technical Report, Microsoft Research, MSR-TR-98-14,April 21, 1998. [Poul 95] Poulard H., “Barycentric correction procedure: A fast method of learning threshold units,” Proc.WCNN ’95,Vol. 1,Washington, DC, pp. 710–713, July, 1995. [Rich 91] Richard M.D., Lippmann R.P. “Neural network classifiers estimate Bayesian a posteriori probabilities,” Neural Computation,Vol. 3, pp. 461–483, 1991. [Rifk 04] Rifkiy R.,KlautauA.“In defense of one-vs-all classification,”Journal of Machine Learning Research,Vol. 5, pp. 101–141, 2004. [Robb 51] Robbins H., Monro S. “A stochastic approximation method,” Annals of Mathematical Statistics,Vol. 22, pp. 400–407, 1951. [Rose 58] Rosenblatt F. “The perceptron: only A probabilistic model for information storage and organization in the brain,” Psychological Review,Vol. 65, pp. 386–408, 1958. [Scho 00] Schölkoph B., Smola A.J.,Williamson R.C., Bartlett P.L. “New support vector algorithms,” Neural Computation,Vol. 12, pp. 1207–1245, 2000.
149
“05-Ch03-SA272” 17/9/2008 page 150
150
CHAPTER 3 Linear Classifiers
[Tao 04] Tao Q., Wu G.-W., Wang J. “A generalized S-K algorithm for learning -SVM classifiers,” Pattern Recognition Letters,Vol. 25(10), pp. 1165–1171, 2004. [Theo 07] Theodoridis S., Mavroforakis M. “Reduced convex hulls: A geometric approach to support vector machines,” IEEE Signal Processing Magazine,Vol. 24(3), pp. 119–122, 2007. [Tou 74] Tou J., Gonzalez R.C. Pattern Recognition Principles,Addison-Wesley, 1974. [Tsan 06] Tsang I.W.-H., Kwok J.T. -Y., Zurada J.M. “Generalized core vector machines,” IEEE Transactions on Neural Networks,Vol. 17(5), pp. 1126–1140, 2006. [Vapn 98] Vapnik V.N. Statistical Learning Theory, John Wiley & Sons, 1998. [Widr 60] Widrow B., Hoff M.E., Jr. “Adaptive switching circuits,” IRE WESCON Convention Record, pp. 96–104, 1960. [Widr 90] Widrow B., Lehr M.A. “30 years of adaptive neural networks: Perceptron, madaline, and backpropagation,” Proceedings of the IEEE,Vol. 78(9), pp. 1415–1442, 1990. [Yee 96] Yee T., Wild C. “Vector generalized additive models,” Journal of the Royal Statistical Society, Series B,Vol. 58, pp. 481–493, 1996. [Zhou 08] Zhou J., Peng H., Suen C.Y. “Data-driven decomposition for multi-class classification,” Pattern Recognition,Vol. 41, pp. 67–76, 2008.
“06-Ch04-SA272” 18/9/2008 page 151
CHAPTER
Nonlinear Classifiers
4
4.1 INTRODUCTION In the previous chapter we dealt with the design of linear classifiers described by linear discriminant functions (hyperplanes) g(x). In the simple two-class case, we saw that the perceptron algorithm computes the weights of the linear function g(x), provided that the classes are linearly separable. For nonlinearly separable classes, linear classifiers were optimally designed, for example, by minimizing the squared error. In this chapter we will deal with problems that are not linearly separable and for which the design of a linear classifier, even in an optimal way, does not lead to satisfactory performance. The design of nonlinear classifiers emerges now as an inescapable necessity.
4.2 THE XOR PROBLEM To seek nonlinearly separable problems one does not need to go into complicated situations. The well-known Exclusive OR (XOR) Boolean function is a typical example of such a problem. Boolean functions can be interpreted as classification tasks. Indeed, depending on the values of the input binary data x ⫽ [x1 , x2 , . . . , xl ]T , the output is either 0 or 1, and x is classified into one of the two classes A(1) or B(0). The corresponding truth table for the XOR operation is shown in Table 4.1. Figure 4.1 shows the position of the classes in space. It is apparent from this figure that no single straight line exists that separates the two classes. In contrast, the other two Boolean functions, AND and OR, are linearly separable. The corresponding truth tables for the AND and OR operations are given in Table 4.2 and the respective class positions in the two-dimensional space are shown in Figure 4.2a and 4.2b. Figure 4.3 shows a perceptron, introduced in the previous chapter, with synaptic weights computed so as to realize an OR gate (verify). Our major concern now is first to tackle the XOR problem and then to extend the procedure to more general cases of nonlinearly separable classes. Our kickoff point will be geometry.
151
“06-Ch04-SA272” 18/9/2008 page 152
152
CHAPTER 4 Nonlinear Classifiers
Table 4.1 Truth Table for the XOR Problem x1
x2
XOR
Class
0
0
0
B
0
1
1
A
1
0
1
A
1
1
0
B
x2
XOR
1
A
B
B
A
0
x1
1
FIGURE 4.1 Classes A and B for the XOR problem.
Table 4.2 Truth Table for AND and OR Problems x2
AND
Class
OR
Class
0
0
0
B
0
B
0
1
0
B
1
A
1
0
0
B
1
A
1
1
1
A
1
A
x1
“06-Ch04-SA272” 18/9/2008 page 153
4.3 The Two-Layer Perceptron
x2
x2 AND
1
B
A
1 A
A
B
A 1 (b)
B
B 0
OR
x1
1 (a)
0
x1
FIGURE 4.2 Classes A and B for (a) the AND and (b) OR problems.
x1 1
1 x2
2
1 2
FIGURE 4.3 A perceptron realizing an OR gate.
4.3 THE TWO-LAYER PERCEPTRON To separate the two classes A and B in Figure 4.1, a first thought that comes to mind is to draw two, instead of one, straight lines. Figure 4.4 shows two such possible lines, g1 (x) ⫽ g2 (x) ⫽ 0, as well as the regions in space for which g1 (x) ≷ 0, g2 (x) ≷ 0. The classes can now be separated. Class A is to the right (⫹) of g1 (x) and to the left (⫺) of g2 (x). The region corresponding to class B lies either to the left or to the right of both lines. What we have really done is to attack the problem in two successive phases. During the first phase, we calculate the position of a feature vector x with respect to each of the two decision lines. In the second phase, we combine the results of the previous phase and we find the position of x with respect to both lines, that is, outside or inside the shaded area. We will now view this from a slightly different perspective, which will subsequently lead us easily to generalizations.
153
“06-Ch04-SA272” 18/9/2008 page 154
154
CHAPTER 4 Nonlinear Classifiers
x2 1 2
A
B
(0,1) 1 2
(1,1)
B
A 1 2
(0,0)
1 2
(1,0) g1(x)
x1 g2(x)
FIGURE 4.4 Decision lines realized by a two-layer perceptron for the XOR problem.
Table 4.3 Truth Table for the Two Computation Phases of the XOR Problem 1st Phase x1
x2
y1
y2
2nd Phase
0
0
0 (⫺)
0 (⫺)
B (0)
0
1
1 (⫹)
0 (⫺)
A (1)
1
0
1 (⫹)
0 (⫺)
A (1)
1
1
1 (⫹)
1 (⫹)
B (0)
Realization of the two decision lines (hyperplanes), g1 (·) and g2 (·), during the first phase of computations is achieved with the adoption of two perceptrons with inputs x1 , x2 and appropriate synaptic weights. The corresponding outputs are yi ⫽ f ( gi (x)), i ⫽ 1, 2, where the activation function f (·) is the step function with levels 0 and 1. Table 4.3 summarizes the yi values for all possible combinations of the inputs. These are nothing else than the relative positions of the input vector x with respect to each of the two lines. From another point of view, the computations during the first phase perform a mapping of the input vector x to a new
“06-Ch04-SA272” 18/9/2008 page 155
4.3 The Two-Layer Perceptron
one y ⫽ [ y1 , y2 ]T . The decision during the second phase is now based on the transformed data; that is,our goal is now to separate [ y1 , y2 ] ⫽ [0, 0] and [ y1 , y2 ] ⫽ [1, 1], which correspond to class B vectors, from the [ y1 , y2 ] ⫽ [1, 0], which corresponds to class A vectors. As is apparent from Figure 4.5, this is easily achieved by drawing a third line g( y),which can be realized via a third neuron. In other words, the mapping of the first phase transforms the nonlinearly separable problem to a linearly separable one. We will return to this important issue later on. Figure 4.6 gives a possible realization of these steps. Each of the three lines is realized via a neuron with appropriate synaptic weights. The resulting multilayer architecture can be considered as a generalization of the perceptron,and it is known as a two-layer perceptron
y2
B (1, 1)
B
2
1
A
(0, 0)
y1
(1, 0)
FIGURE 4.5 Decision line formed by the neuron of the second layer for the XOR problem.
x1
1 1 1
1 2 2
1
x2
22
1
2
FIGURE 4.6 A two-layer perceptron solving the XOR problem.
3 2
2
1 2
155
“06-Ch04-SA272” 18/9/2008 page 156
156
CHAPTER 4 Nonlinear Classifiers
or a two-layer feedforward1 neural network. The two neurons (nodes) of the first layer perform computations of the first phase and they constitute the so-called hidden layer. The single neuron of the second layer performs the computations of the final phase and constitutes the output layer. In Figure 4.6 the input layer corresponds to the (nonprocessing) nodes where input data are applied. Thus, the number of input layer nodes equals the dimension of the input space. Note that at the input layer nodes no processing takes place. The lines that are realized by the two-layer perceptron of the figure are g1 (x) ⫽ x1 ⫹ x2 ⫺
1 ⫽0 2
g2 (x) ⫽ x1 ⫹ x2 ⫺
3 ⫽0 2
g( y) ⫽ y1 ⫺ y2 ⫺
1 ⫽0 2
The multilayer perceptron architecture of Figure 4.6 can be generalized to l-dimensional input vectors and to more than two (one) neurons in the hidden (output) layer. We will now turn our attention to the investigation of the class discriminatory capabilities of such networks for more complicated nonlinear classification tasks.
4.3.1 Classification Capabilities of the Two-Layer Perceptron A careful look at the two-layer perceptron of Figure 4.6 reveals that the action of the neurons of the hidden layer is actually a mapping of the input space x onto the vertices of a square of unit side length in the two-dimensional space (Figure 4.5). For the more general case, we will consider input vectors in the l-dimensional space, that is, x ∈ Rl , and p neurons in the hidden layer (Figure 4.7). For the time being, we will keep one output neuron, although this can also be easily generalized to many. Again employing the step activation function, the mapping of the input space, performed by the hidden layer, is now onto the vertices of the hypercube of unit side length in the p-dimensional space, denoted by Hp . This is defined as Hp ⫽ {[ y1 , . . . , yp ]T ∈ Rp , yi ∈ [0, 1], 1 ⱕ i ⱕ p}
The vertices of the hypercube are all the points [ y1 , . . . , yp ]T of Hp with yi ∈ {0, 1}, 1 ⱕ i ⱕ p. The mapping of the input space onto the vertices of the hypercube is achieved via the creation of p hyperplanes. Each of the hyperplanes is created by a neuron in the hidden layer, and the output of each neuron is 0 or 1, depending on the relevant position of the input vector with respect to the corresponding hyperplane.
1 To distinguish it from other related structures where feedback paths from the output back to the input exist.
“06-Ch04-SA272” 18/9/2008 page 157
4.3 The Two-Layer Perceptron
y1
x1
y2
x2
yp xl
FIGURE 4.7 A two-layer perceptron. g3
1 2
B 111 B 110
A 011 1 2
B 010 A 001 1 2
A 000
B 100
g2 g1
FIGURE 4.8 Polyhedra formed by the neurons of the first hidden layer of a multilayer perceptron.
Figure 4.8 is an example of three intersecting hyperplanes (three neurons) in the two-dimensional space. Each region defined by the intersections of these hyperplanes corresponds to a vertex of the unit three-dimensional hypercube, depending on its position with respect to each of these hyperplanes. The ith dimension of the vertex shows the position of the region with respect to the gi hyperplane. For example, the 001 vertex corresponds to the region that is in the (⫺) side of g1 , in the (⫺) side of g2 , and in the (⫹) side of g3 . Thus, the conclusion we reach is that the first layer of neurons divides the input l-dimensional space into polyhedra,2 which are formed by hyperplane intersections. All vectors located within one of these polyhedral regions are mapped onto a specific vertex of the unit Hp hypercube. The output neuron subsequently realizes another hyperplane, which separates the hypercube into two parts,having some of its vertices on one and some
A polyhedron or polyhedral set is the finite intersection of closed half-spaces of Rl , which are defined by a number of hyperplanes.
2
157
“06-Ch04-SA272” 18/9/2008 page 158
158
CHAPTER 4 Nonlinear Classifiers
111
011
010
110
001
101 y2
000
100
y3
y1
FIGURE 4.9 The neurons of the first hidden layer map an input vector onto one of the vertices of a unit (hyper)cube. The output neuron realizes a (hyper)plane to separate vertices according to their class label.
on the other side. This neuron provides the multilayer perceptron with the potential to classify vectors into classes consisting of unions of the polyhedral regions. Let us consider, for example, that class A consists of the union of the regions mapped onto vertices 000, 001, 011 and class B consists of the rest (Figure 4.8). Figure 4.9 shows the H3 unit (hyper)cube and a (hyper)plane that separates the space R3 into two regions with the (class A) vertices 000, 001, 011 on one side and (class B) vertices 010, 100, 110, 111 on the other. This is the ⫺y1 ⫺ y2 ⫹ y3 ⫹ 0.5 ⫽ 0 plane, which is realized by the output neuron. With such a configuration all vectors from class A result in an output of 1(⫹) and all vectors from class B in 0(⫺). On the other hand, if class A consists of the union 000 ∪ 111 ∪ 110 and class B of the rest, it is not possible to construct a single plane that separates class A from class B vertices. Thus, we can conclude that a two-layer perceptron can separate classes each consisting of unions of polyhedral regions but not any union of such regions. It all depends on the relative positions of the vertices of Hp , where the classes are mapped, and on whether or not these are linearly separable. Before we proceed further to see ways to overcome this shortcoming, it should be pointed out that vertex 101 of the cube does not correspond to any of the polyhedral regions. Such vertices are said to correspond to virtual polyhedra, and they do not influence the classification task.
4.4 THREE-LAYER PERCEPTRONS The inability of the two-layer perceptrons to separate classes resulting from any union of polyhedral regions springs from the fact that the output neuron can realize
“06-Ch04-SA272” 18/9/2008 page 159
4.4 Three-Layer Perceptrons
x1 x2
xl
FIGURE 4.10 Architecture of a multilayer perceptron with two hidden layers of neurons and a single output neuron.
only a single hyperplane. This is the same situation confronting the basic perceptron when dealing with the XOR problem. The difficulty was overcome by constructing two lines instead of one. A similar escape path will be adopted here. Figure 4.10 shows a three-layer perceptron architecture with two layers of hidden neurons and one output layer. We will show, constructively, that such an architecture can separate classes resulting from any union of polyhedral regions. Indeed, let us assume that all regions of interest are formed by intersections of p l-dimensional half-spaces defined by the p hyperplanes. These are realized by the p neurons of the first hidden layer, which also perform the mapping of the input space onto the vertices of the Hp hypercube of unit side length. In the sequel let us assume that class A consists of the union of K of the resulting polyhedra and class B of the rest. We then use K neurons in the second hidden layer. Each of these neurons realizes a hyperplane in the p-dimensional space. The synaptic weights for each of the second-layer neurons are chosen so that the realized hyperplane leaves only one of the Hp vertices on one side and all the rest on the other. For each neuron a different vertex is isolated, that is, one of the K A class vertices. In other words, each time an input vector from class A enters the network, one of the K neurons of the second layer results in a 1 and the remaining K ⫺ 1 give 0. In contrast, for class B vectors all neurons in the second layer output a 0. Classification is now a straightforward task. Choose the output layer neuron to realize an OR gate. Its output will be 1 for class A and 0 for class B vectors. The proof is now complete. The number of neurons in the second hidden layer can be reduced by exploiting the geometry that results from each specific problem—for example, whenever two of the K vertices are located in a way that makes them separable from the rest,using a single hyperplane. Finally,the multilayer structure can be generalized to more than two classes. To this end, the output layer neurons are increased in number, realizing one OR gate for each class. Thus, one of them results in 1 every time a vector from the respective class enters the network, and all the others give 0. The number of second-layer neurons is also affected (why?).
159
“06-Ch04-SA272” 18/9/2008 page 160
160
CHAPTER 4 Nonlinear Classifiers
In summary,we can say that the neurons of the first layer form the hyperplanes, those of the second layer form the regions, and finally the neurons of the output layer form the classes. So far, we have focused on the potential capabilities of a three-layer perceptron to separate any union of polyhedral regions. To assume that in practice we know the regions where the data are located and we can compute the respective hyperplane equations analytically is no doubt wishful thinking, for this is as yet an unrealizable goal. All we know in practice is a set of training points with the respective class labels. As was the case with the perceptron, one has to resort to learning algorithms that learn the synaptic weights from the available training data vectors. We will focus our attention on two major directions. In one of them the network is constructed in a way that classifies correctly all the available training data, by building it as a succession of linear classifiers. The other direction relieves itself of the correct classification constraint and computes the synaptic weights so as to minimize a preselected cost function.
4.5 ALGORITHMS BASED ON EXACT CLASSIFICATION OF THE TRAINING SET The starting point of these techniques is a small architecture (usually unable to solve the problem at hand), which is successively augmented until the correct classification of all N feature vectors of the training set X is achieved. Different algorithms follow different ways to augment their architectures. Thus,some algorithms expand their architectures in terms of the number of layers [Meza 89, Frea 90],whereas others use one or two hidden layers and expand them in terms of the number of their nodes (neurons) [Kout 94, Bose 96]. Moreover, some of these algorithms [Frea 90] allow connections between nodes of nonsuccessive layers. Others allow connections between nodes of the same layer [Refe 91]. A general principle adopted by most of these techniques is the decomposition of the problem into smaller problems that are easier to handle. For each smaller problem, a single node is employed. Its parameters are determined either iteratively using appropriate learning algorithms, such as the pocket algorithm or the LMS algorithm (Chapter 3), or directly via analytical computations. From the way these algorithms build the network, they are sometimes referred to as constructive techniques. The tiling algorithm [Meza 89] constructs architectures with many (usually more than three) layers. We describe the algorithm for the two-class (A and B) case. The algorithm starts with a single node, n(X), in the first layer, which is called the master unit of this layer. This node is trained using the pocket algorithm (Chapter 3), and, after the completion of the training, it divides the training data set X into two subsets X ⫹ and X ⫺ (line 1 in Figure 4.11). If X ⫹ (X ⫺ ) contains feature vectors from both classes, we introduce an additional node, n(X ⫹ ) (n(X ⫺ )), which is called the ancillary
“06-Ch04-SA272” 18/9/2008 page 161
4.5 Algorithms Based on Exact Classification of the Training Set
1 1 2
1 2
3 1 2
2
1
(a)
2
3
(b)
FIGURE 4.11 Decision lines and the corresponding architecture resulting from the tiling algorithm. The black (red) dots correspond to class A (B).
unit. This node is trained using only the feature vectors in X ⫹ (X ⫺ ) (line 2). If one of the X ⫹⫹ , X ⫹⫺ (X ⫺⫹ , X ⫺⫺ ) produced by neuron n(X ⫹ ) (n(X ⫺ )) contains vectors from both classes, more ancillary nodes are added. This procedure stops after a finite number of steps, since the number of vectors a newly added (ancillary) unit has to discriminate decreases at each step. Thus, the first layer consists of a single master unit and, in general, more than one ancillary units. It is easy to show that in this way we succeed so that no two vectors from different classes give the same first-layer outputs. Let X1 ⫽ {y : y ⫽ f1 (x), x ∈ X}, where f1 is the mapping implemented by the first layer. Applying the procedure just described to the set X1 of the transformed y samples,we construct the second layer of the architecture and so on. In [Meza 89] it is shown that proper choice of the weights between two adjacent layers ensures that each newly added master unit classifies correctly all the vectors that are correctly classified by the master unit of the previous layer,plus at least one more vector. Thus, the tiling algorithm produces an architecture that classifies correctly all patterns of X in a finite number of steps. An interesting observation is that all but the first layer treat binary vectors. This reminds us of the unit hypercube of the previous section. Mobilizing the same arguments as before, we can show that this algorithm may lead to correct classification architectures having three layers of nodes at the most. Another family of constructive algorithms builds on the idea of the nearest neighbor classification rule, discussed in Chapter 2. The neurons of the first layer implement the hyperplanes bisecting the line segments that join the training feature vectors [Murp 90]. The second layer forms the regions, using an appropriate number of neurons that implement AND gates, and the classes are formed via the neurons of the last layer, which implement OR gates. The major drawback of this
161
“06-Ch04-SA272” 18/9/2008 page 162
162
CHAPTER 4 Nonlinear Classifiers
technique is the large number of neurons involved. Techniques that reduce this number have also been proposed ([Kout 94, Bose 96]).
4.6 THE BACKPROPAGATION ALGORITHM The other direction we will follow to design a multilayer perceptron is to fix the architecture and compute its synaptic parameters so as to minimize an appropriate cost function of its output. This is by far the most popular approach, which not only overcomes the drawback of the resulting large networks of the previous section but also makes these networks powerful tools for a number of other applications, beyond pattern recognition. However, such an approach is soon confronted with a serious difficulty. This is the discontinuity of the step (activation) function, prohibiting differentiation with respect to the unknown parameters (synaptic weights). Differentiation enters into the scene as a result of the cost function minimization procedure. In the sequel we will see how this difficulty can be overcome. The multilayer perceptron architectures we have considered so far have been developed around the McCulloch–Pitts neuron,employing as the activation function the step function f (x) ⫽
1
x⬎0
0
x⬍0
A popular family of continuous differentiable functions,which approximate the step function, is the family of sigmoid functions. A typical representative is the logistic function f (x) ⫽
1 1 ⫹ exp(⫺ax)
(4.1)
where a is a slope parameter. Figure 4.12 shows the sigmoid function for different values of a, along with the step function. Sometimes a variation of the logistic function is employed that is antisymmetric with respect to the origin, that is, f (⫺x) ⫽ ⫺f (x). It is defined as f (x) ⫽
2 ⫺1 1 ⫹ exp(⫺ax)
(4.2)
It varies between 1 and ⫺1, and it belongs to the family of hyperbolic tangent functions, f (x) ⫽ c
ax 1 ⫺ exp(⫺ax) ⫽ c tanh 1 ⫹ exp(⫺ax) 2
(4.3)
All these functions are also known as squashing functions since their output is limited in a finite range of values. In the sequel, we will adopt multilayer neural architectures like the one in Figure 4.10, and we will assume that the activation
“06-Ch04-SA272” 18/9/2008 page 163
4.6 The Backpropagation Algorithm
f(x) 1
a3 a 2 a 1
a3. a2 . a1
0
x
FIGURE 4.12 The logistic function. The larger the value of the slope parameter, a, the better the approximation of the unit-step function achieved.
functions are of the form given in (4.1)–(4.3). Our goal is to derive an iterative training algorithm that computes the synaptic weights of the network so that an appropriately chosen cost function is minimized. Before going into the derivation of such a scheme, an important point must be clarified. From the moment we move away from the step function, all we have said before about mapping the input vectors onto the vertices of a unit hypercube is no longer valid. It is now the cost function that takes on the burden for correct classification. For the sake of generalization, let us assume that the network consists of a fixed number of L layers of neurons, with k0 nodes in the input layer and kr neurons in the rth layer, for r ⫽ 1, 2, . . . , L. Obviously, k0 equals l. All the neurons employ the same sigmoid activation function. As was the case in Section 3.3,we assume that N training pairs are available ( y(i), x(i)), i ⫽ 1, 2, . . . , N .3 Because we have now assumed kL output neurons, the output is no longer a scalar but a kL -dimensional vector, y(i) ⫽ [ y1 (i), . . . , ykL (i)]T . The input (feature) vectors are k0 -dimensional vectors, x(i) ⫽ [x1 (i), . . . , xk0 (i)]T . During training, when vector x(i) is applied to the input, the output of the network will be y(i), ˆ which is different from the desired value, y(i). The synaptic weights are computed such that an appropriate (for each problem) cost function J , which is dependent on the values y(i) and
3
In contrast to other chapters, we use i in parentheses and not as an index. This is because, for the needs of the chapter, the latter notation can become very cumbersome.
163
“06-Ch04-SA272” 18/9/2008 page 164
164
CHAPTER 4 Nonlinear Classifiers
S
f
S
S
f
S ykr21
wrj k
f
vrj f
S
f
S
f
S
f
S
f
yrj
r
r21
FIGURE 4.13 Definition of variables involved in the backpropagation algorithm.
y(i), ˆ i ⫽ 1, 2, . . . , N , is minimized. It is obvious that J depends, through y(i), ˆ on the weights and that this is a nonlinear dependence, due to the nature of the network itself. Thus, minimization of the cost function can be achieved via iterative techniques. In this section we will adopt the gradient descent scheme (Appendix C), which is the most widely used approach. Let w rj be the weight vector (including the threshold) of the jth neuron in the rth layer,which is a vector of dimension kr⫺1 ⫹ 1 r , wr , . . . , wr T and is defined as (Figure 4.13) w rj ⫽ [wj0 j1 jkr⫺1 ] . The basic iteration step will be of the form w rj (new) ⫽ w rj (old) ⫹ ⌬w rj
with ⌬w rj ⫽ ⫺
⭸J ⭸w rj
(4.4)
where w rj (old) is the current estimate of the unknown weights and ⌬w rj the corresponding correction to obtain the next estimate w rj (new). In Figure 4.13 jr is the weighted summation of the inputs to the jth neuron of the rth layer and yjr the corresponding output after the activation function. In the sequel we will focus our attention on cost functions of the form J⫽
N
E(i)
(4.5)
i⫽1
where E is an appropriately defined function depending on y(i) ˆ and y(i), i ⫽ 1, 2, . . . , N . In other words, J is expressed as a sum of the N values that function E
“06-Ch04-SA272” 18/9/2008 page 165
4.6 The Backpropagation Algorithm
takes for each of the training pairs (y(i), x(i)). For example, we can choose E(i) as the sum of squared errors in the output neurons E(i) ⫽
kL kL 1 1 2 em (i) ≡ ( ym (i) ⫺ yˆ m (i))2 , 2 2 m⫽1
i ⫽ 1, 2, . . . , N
(4.6)
m⫽1
For the computation of the correction term in (4.4) the gradient of the cost function J with respect to the weights is required and, consequently, the evaluation of ⭸E(i)/⭸w rj .
Computation of the Gradients Let ykr⫺1 (i) be the output of the kth neuron, k ⫽ 1, 2, . . . , kr⫺1 , in the (r ⫺ 1)th layer r the current estimate of the corresponding weight for the ith training pair and wjk leading to the jth neuron in the rth layer, with j ⫽ 1, 2, . . . , kr (Figure 4.13). Thus, the argument of the activation function f (·) of the latter neuron will be jr (i) ⫽
k r⫺1
r r wjk ykr⫺1 (i) ⫹ wjo ≡
k r⫺1
k⫽1
r wjk ykr⫺1 (i)
(4.7)
k⫽0
where by definition y0r (i) ≡ ⫹1, ᭙r, i; so as to include the thresholds in the weights. For the output layer, we have r ⫽ L, ykr (i) ⫽ yˆ k (i), k ⫽ 1, 2, . . . , kL , that is, the outputs of the neural network, and for r ⫽ 1, ykr⫺1 (i) ⫽ xk (i), k ⫽ 1, 2, . . . , k0 , that is, the network inputs. As is apparent from (4.7), the dependence of E(i) on w rj passes through jr (i). By the chain rule in differentiation, we have r ⭸E(i) ⭸j (i) ⭸E(i) ⫽ ⭸w rj ⭸jr (i) ⭸w rj
(4.8)
From (4.7) we obtain ⎡ ⎢ ⭸ r ⎢ (i) ≡ ⎢ ⎣ ⭸w rj j
⭸ r r j (i) ⭸wj0
.. . ⭸
r ⭸wjk
r⫺1
jr (i)
⎤ ⎥ ⎥ ⎥ ⫽ y r⫺1 (i) ⎦
(4.9)
where ⎡ ⎢ ⎢ y r⫺1 (i) ⫽ ⎢ ⎢ ⎣
⫹1 y1r⫺1 (i) .. . ykr⫺1 (i) r⫺1
⎤ ⎥ ⎥ ⎥ ⎥ ⎦
(4.10)
165
“06-Ch04-SA272” 18/9/2008 page 166
166
CHAPTER 4 Nonlinear Classifiers
Let us define ⭸E(i) ≡ ␦rj (i) ⭸jr (i)
(4.11)
Then (4.4) becomes ⌬w rj ⫽ ⫺
N
␦rj (i)y r⫺1 (i)
(4.12)
i⫽1
Relation (4.12) is general for any differentiable cost function of the form (4.5). In the sequel we will compute ␦rj (i) for the special case of least squares (4.6). The procedure is similar for alternative cost function choices.
Computation of ␦rj (i) for the Cost Function in (4.6) The computations start from r ⫽ L and propagate backward for r ⫽ L ⫺ 1, L ⫺ 2, . . . , 1. This is why the algorithm that will be derived is known as the backpropagation algorithm. 1. r ⫽ L ␦Lj (i) ⫽ E(i) ≡
⭸E(i) ⭸jL (i)
(4.13)
kL kL 1 1 2 L em (i) ≡ (f (m (i)) ⫺ ym (i))2 2 2 m⫽1
(4.14)
m⫽1
Hence ␦Lj (i) ⫽ ej (i)f ⬘(jL (i))
(4.15)
where f ⬘ is the derivative of f (·). In the last layer, the dependence of E(i) on jL (i) is explicit, and the computation of the derivative is straightforward. This is not true, however, for the hidden layers, where the computations of the derivatives need more elaboration. 2. r ⬍ L. Due to the successive dependence among the layers, the value of jr⫺1 (i) influences all kr (i), k ⫽ 1, 2, . . . , kr , of the next layer. Employing the chain rule in differentiation once more, we obtain kr ⭸E(i) ⭸E(i) ⭸kr (i) ⫽ r⫺1 ⭸kr (i) ⭸jr⫺1 (i) ⭸j (i) k⫽1
(4.16)
and from the respective definition (4.11) ␦r⫺1 (i) ⫽ j
kr k⫽1
␦rk (i)
⭸kr (i) ⭸jr⫺1 (i)
(4.17)
“06-Ch04-SA272” 18/9/2008 page 167
4.6 The Backpropagation Algorithm
But ⭸ ⭸kr (i) ⫽ ⭸jr⫺1 (i)
kr⫺1 r r⫺1 m⫽0 wkm ym (i) ⭸jr⫺1 (i)
(4.18)
with r⫺1 r⫺1 ym (i) ⫽ f (m (i))
(4.19)
⭸kr (i) r ⬘ r⫺1 f (j (i)) ⫽ wkj ⭸jr⫺1 (i)
(4.20)
Hence,
From (4.20) and (4.17 ) the following results: ␦r⫺1 (i) j
⫽
kr
r ␦rk (i)wkj
f ⬘(jr⫺1 (i))
(4.21)
k⫽1
and for uniformity with (4.15) ⬘ r⫺1 ␦r⫺1 (i) ⫽ ejr⫺1 (i)f ( (i)) j j
(4.22)
where ejr⫺1 (i) ⫽
kr
r ␦rk (i)wkj
(4.23)
k⫽1
Relations (4.15), (4.22), and (4.23) constitute the iterations leading to the computation of ␦rj (i), r ⫽ 1, 2, . . . , L, j ⫽ 1, 2, . . . , kr . The only quantity ⬘ For the function in (4.1) we have that is not yet computed is f (·). ⬘ f (x) ⫽ af (x)(1 ⫺ f (x))
The algorithm has now been derived. The algorithmic scheme was first presented in [Werb 74] in a more general formulation.
The Backpropagation Algorithm ■
■
Initialization: Initialize all the weights with small random values from a pseudorandom sequence generator. Forward computations: For each of the training feature vectors x(i), i ⫽ 1, 2, . . . , N , compute all the jr (i), yjr (i) ⫽ f (jr (i)), j ⫽ 1, 2, . . . , kr , r ⫽ 1, 2, . . . , L, from (4.7). Compute the cost function for the current estimate of weights from (4.5) and (4.14).
167
“06-Ch04-SA272” 18/9/2008 page 168
168
CHAPTER 4 Nonlinear Classifiers
■
■
Backward computations: For each i ⫽ 1, 2, . . . , N and j ⫽ 1, 2, . . . , kL com(i) from (4.22) and pute ␦Lj (i) from (4.15) and in the sequel compute ␦r⫺1 j (4.23) for r ⫽ L, L ⫺ 1, . . . , 2, and j ⫽ 1, 2, . . . , kr Update the weights: For r ⫽ 1, 2, . . . , L and j ⫽ 1, 2, . . . , kr w rj (new) ⫽ w rj (old) ⫹ ⌬w rj ⌬w rj ⫽ ⫺
N
␦rj (i)y r⫺1 (i)
i⫽1
Remarks ■
A number of criteria have been suggested for terminating the iterations. In [Kram 89] it is suggested that we terminate the iterations either when the cost function J becomes smaller than a certain threshold or when its gradient with respect to the weights becomes small. Of course, the latter has a direct effect on the rate of change of the weights between successive iteration steps.
■
As with all the algorithms that spring from the gradient descent method, the convergence speed of the backpropagation scheme depends on the value of the learning constant . Its value must be sufficiently small to guarantee convergence but not too small, because the convergence speed becomes very slow. The best choice of depends very much on the problem and the cost function shape in the weight space. Broad minima yield small gradients; thus large values of lead to faster convergence. On the other hand, for steep and narrow minima small values of are required to avoid overshooting the minimum. As we will soon see, scenarios with adaptive are also possible.
■
The cost function minimization for a multilayer perceptron is a nonlinear minimization task. Thus, the existence of local minima in the corresponding cost function surface is an expected reality. Hence, the backpropagation algorithm runs the risk of being trapped in a local minimum. If the local minimum is deep enough, this may still be a good solution. However, in cases in which this is not true, getting stuck in such a minimum is an undesirable situation,and the algorithm should be reinitialized from a different set of initial conditions.
■
The algorithm described in this section updates the weights once all the training (input–desired output) pairs have appeared in the network. This mode of operation is known as the batch mode. A variation of this approach is to update the weights for each of the training pairs. This is known as the pattern or online mode. This is analogous to the LMS, where, according to
“06-Ch04-SA272” 18/9/2008 page 169
4.7 Variations on the Backpropagation Theme
the Robbins–Monro approach, the instantaneous value of the gradient is computed instead of its mean. In the backpropagation case, the sum of ␦rj (i) over all i is substituted with each of them. The algorithm in its pattern mode of operation then becomes w rj (i ⫹ 1) ⫽ w rj (i) ⫺ ␦rj (i)y r⫺1 (i)
Compared with the pattern mode, the batch mode is an inherent averaging process. This leads to a better estimate of the gradient, and thus to more well-behaved convergence. On the other hand, the pattern mode presents a higher degree of randomness during training. This may help the algorithm to avoid being trapped in a local minimum. In [Siet 91] it is suggested that the beneficial effects that randomness may have on training can be further emphasized by adding a (small) white noise sequence in the training data. Another commonly used practice focuses on the way the training data are presented in the network. During training, the available training vectors are used in the update equation more than once until the algorithm converges. One complete presentation of all N training pairs constitutes an epoch. As successive epochs are applied, it is good practice from the convergence point of view to randomize the order of presentation of the training pairs. Randomization can again help the pattern mode algorithm to jump out of regions around local minima, when this occurs. However, the final choice between the batch and pattern modes of operation depends on the specific problem [Hert 91, p. 119]. ■
Once training of the network has been achieved, the values to which the synapses and thresholds have converged are frozen, and the network is ready for classification. This is a much easier task than training. An unknown feature vector is presented in the input and is classified in the class that is indicated by the output of the network. The computations performed by the neurons are of the multiply–add type followed by a nonlinearity. This has led to various hardware implementations ranging from optical to VLSI chip design. Furthermore, neural networks have a natural built-in parallelism, and computations in each layer can be performed in parallel. These distinct characteristics of neural networks have led to the development of special neurocomputers. A number of those are already commercially available; see, for example, [Koli 97].
4.7 VARIATIONS ON THE BACKPROPAGATION THEME Both versions of the backpropagation scheme—the batch and the pattern modes— inherit the disadvantage of all methods built on the gradient descent approach: their convergence to the cost function minimum is slow. Appendix C discusses the fact that this trait becomes more prominent if the eigenvalues of the corresponding
169
“06-Ch04-SA272” 18/9/2008 page 170
170
CHAPTER 4 Nonlinear Classifiers
Hessian matrix exhibit large spread. In such cases, the change of the cost function gradient between successive iteration steps is not smooth but oscillatory, leading to slow convergence. One way to overcome this problem is to use a momentum term that smoothes out the oscillatory behavior and speeds up the convergence. The backpropagation algorithm with momentum term takes the form ⌬w rj (new) ⫽ ␣⌬w rj (old) ⫺
N
␦rj (i)y r⫺1 (i)
(4.24)
i
w rj (new) ⫽ w rj (old) ⫹ ⌬w rj (new)
(4.25)
Compared with (4.4), we see that the correction vector ⌬w rj depends not only on the gradient term but also on its value in the previous iteration step. The constant ␣ is called the momentum factor and in practice is chosen between 0.1 and 0.8. To see the effect of the momentum factor, let us look at the correction term for a number of successive iteration steps. At the tth iteration step we have ⌬w rj (t) ⫽ ␣⌬w rj (t ⫺ 1) ⫺ g(t)
(4.26)
where the last term denotes the gradient. For a total of T successive iteration steps we obtain ⌬w rj (T ) ⫽ ⫺
T ⫺1
␣t g(T ⫺ t) ⫹ ␣T ⌬w rj (0)
(4.27)
t⫽0
Since ␣ ⬍ 1, the last term gets close to zero after a few iteration steps and the smoothing (averaging) effect of the momentum term becomes apparent. Let us now assume that the algorithm is at a low-curvature point of the cost function surface in the weight space. We can then assume that the gradient is approximately constant over a number of iteration steps. Applying this, we can write that ⌬w rj (T ) ⫺(1 ⫹ ␣ ⫹ ␣2 ⫹ ␣3 ⫹ · · · )g ⫽ ⫺
g 1⫺␣
In other words, in such cases the effect of the momentum term is to effectively increase the learning constant. In practice, improvements in converging speed by a factor of 2 or even more have been reported [Silv 90]. A heuristic variation of the previous technique is to use an adaptive value for the learning factor , depending on the cost function values at successive iteration steps. A possible procedure is the following: Let J (t) be the value of the cost at the tth iteration step. If J (t) ⬍ J (t ⫺ 1), then increase the learning rate by a factor of ri . If, on the other hand, the new value of the cost is larger than the old one by a factor c,then decrease the learning rate by a factor of rd . Otherwise use the same value. In summary J (t) ⬍ 1, J (t ⫺ 1)
(t) ⫽ ri (t ⫺ 1)
“06-Ch04-SA272” 18/9/2008 page 171
4.7 Variations on the Backpropagation Theme
J (t) ⬎ c, (t) ⫽ rd (t ⫺ 1) J (t ⫺ 1) 1ⱕ
J (t) ⱕ c, (t) ⫽ (t ⫺ 1) J (t ⫺ 1)
Typical values of the parameters which are adopted in practice are ri ⫽ 1.05, rd ⫽ 0.7, c ⫽ 1.04. For iteration steps where the cost increases, it may be advantageous not only to decrease the learning rate but also to set the momentum term equal to 0. Others suggest that the update of the weighting not to be done at this step. Another strategy for updating the learning factor is followed in the so-called delta-delta rule and in its modification delta-bar-delta rule [Jaco 88]. The idea here is to use a different learning factor for each weight and to increase the particular learning factor if the gradient of the cost function with respect to the corresponding weight has the same sign on two successive iteration steps. Conversely, if the sign changes, this is an indication of a possible oscillation and the learning factor should be reduced. A number of alternative techniques for speeding up convergence have also been suggested. In [Cich 93] a more extensive review of such techniques is provided. The other option for faster convergence is to free ourselves from the gradient descent rationale and to adopt alternative searching schemes,usually at the expense of increased complexity. A number of such algorithmic techniques have appeared in the related literature. For example, [Kram 89, Barn 92, Joha 92] present algorithmic schemes based on the conjugate gradient algorithm; [Batt 92, Rico 88, Barn 92, Watr 88] provide schemes of the Newton family; [Palm 91, Sing 89] propose algorithms based on the Kalman filtering approach; and [Bish 95] a scheme based on the Levenberg–Marquardt algorithm. In many of these algorithms, elements of the Hessian matrix need to be computed, that is, the second derivatives of the cost function with respect to the weights ⭸2 J q r ⭸wjk ⭸wnm
The computations of the Hessian matrix are performed by adopting, once more, the backpropagation concept (see also Problems 4.12, 4.13). More on these issues can be found in [Hayk 99, Zura 92]. A popular scheme that is loosely based on Newton’s method is the quickprop scheme [Fahl 90]. It is a heuristic method and treats the weights as if they were quasi-independent. It then approximates the error surface, as a function of each weight, by a quadratic polynomial. If this has its minimum at a sensible value, the latter is used as the new weight for the iterations; otherwise a number of heuristics are used. A usual form of the algorithm, for the weights in the various layers, is ⌬wij (t) ⫽
⎧ ⎨␣ij (t)⌬wij (t ⫺ 1),
if ⌬wij (t ⫺ 1) ⫽ 0
⎩ ⭸J , ⭸wij
if ⌬wij (t ⫺ 1) ⫽ 0
(4.28)
171
“06-Ch04-SA272” 18/9/2008 page 172
172
CHAPTER 4 Nonlinear Classifiers
where ⎧ ⎨
␣ij (t) ⫽ min
⎩
⭸J ⭸wij (t) ⭸J ⭸wij (t
⫺ 1) ⫺
⭸J ⭸wij (t)
, ␣max
⎫ ⎬ ⎭
(4.29)
with typical values of the involved variables being 0.01 ⱕ ⱕ 0.6, ␣max ≈ 1.75 [Cich 93]. An algorithm similar in spirit to quickprop has been proposed in [Ried 93]. It is reported that it is as fast as quickprop, and it requires less adjustment of the parameters to be stable.
4.8 THE COST FUNCTION CHOICE It will not come as a surprise that the least squares cost function in (4.6) is not the unique choice available to the user. Depending on the specific problem, other cost functions can lead to better results. Let us look, for example, at the least squares cost function more carefully. Since all errors in the output nodes are first squared and summed up, large error values influence the learning process much more than the small errors. Thus, if the dynamic ranges of the desired outputs are not all of the same order, the least squares criterion will result in weights that have “learned”via a process of unfair provision of information. Furthermore, in [Witt 00] it is shown that for a class of problems, the gradient descent algorithm with the squared error criterion can be trapped in a local minimum and fail to find a solution, although (at least) one exists. In the current context, a solution is assumed to be a classifier that classifies correctly all training samples. In contrast, it is shown that there is an alternative class of functions, satisfying certain criteria, which guarantee that the gradient descent algorithm converges to such a solution, provided that one exists. This class of cost functions is known as well-formed functions. We will now present a cost function of this type, which is well suited for pattern recognition tasks. The multilayer network performs a nonlinear mapping of the input vectors x to the output values yˆ k ⫽ k (x; w) for each of the output nodes k ⫽ 1, 2, . . . kL , where the dependence of the mapping on the values of the weights is explicitly shown. In Chapter 3 we have seen that, if we adopt the least squares cost function and the desired outputs yk are binary (belong to or not in class k ), then for the optimal values of the weights w ∗ the corresponding output of the network, yˆ k , is the least squares optimal estimate of the posterior probability P(k |x). (The question of how good or bad this estimate is will be of interest to us soon.) At this point we will adopt this probabilistic interpretation of the real outputs yˆ k as the basis on which our cost function will be built. Let us assume that the desired output values, yk , are independent binary random variables and that yˆ k are the respective posterior probabilities that these random variables are 1 [Hint 90, Baum 88].
“06-Ch04-SA272” 18/9/2008 page 173
4.8 The Cost Function Choice
The cross-entropy cost function is then defined by J ⫽⫺
kL N
( yk (i) ln yˆ k (i) ⫹ (1 ⫺ yk (i)) ln(1 ⫺ yˆ k (i)))
(4.30)
i⫽1 k⫽1
J takes its minimum value when yk (i) ⫽ yˆ k (i), and for binary desired response values the minimum is zero. There are various interpretations of this cost function [Hint 90, Baum 88, Gish 90, Rich 91]. Let us consider, for example, the output vector y(i) when x(i) appears at the input. This consists of a 1 at the true class node and zero elsewhere. If we take into account that the probability of the kth node to be 1(0) is yˆ k (i)(1 ⫺ yˆ k (i)) and by considering nodes independently, then p( y) ⫽
kL
( yˆ k )yk (1 ⫺ yˆ k )1⫺yk
(4.31)
k⫽1
where the dependence on i has been suppressed for notational convenience. Then it is straightforward to check that J results from the negative log-likelihood of the training sample pairs. If yk (i) were true probabilities in (0, 1) then subtracting the minimum value from J (4.30) becomes J ⫽⫺
kL N
yk (i)ln
i⫽1 k⫽1
yˆ k (i) 1 ⫺ yˆ k (i) ⫹ (1 ⫺ yk (i))ln yk (i) 1 ⫺ yk (i)
(4.32)
For binary valued yk s the above is still valid if we use the limiting value 0 ln 0 ⫽ 0. It is not difficult to show ( Problem 4.5) that the cross-entropy cost function depends on the relative errors and not on the absolute errors, as its least squares counterpart; thus it gives the same weight to small and large values. Furthermore, it has been shown that it satisfies the conditions of the well-formed functions [Adal 97]. Finally, it can be shown that adopting the cross-entropy cost function and binary values for the desired responses, the outputs yˆ k corresponding to the optimal weights w ∗ are indeed estimates of P(k |x), as in the least squares case [Hamp 90]. A major advantage of the cross-entropy cost function is that it diverges if one of the outputs converges to the wrong extreme, hence the gradient descent reacts fast. On the other hand, the squared error cost function approaches a constant in this case, and the gradient descent on the LS will wander on a plateau, even though the error may not be small. This advantage of the cross-entropy cost function is demonstrated in the channel equalization context in [Adal 97]. A different cost function results if we treat yˆ k (i) and yk (i) as the true and desired probabilities, respectively. Then a measure of their similarity is given by the crossentropy function (Appendix A) J ⫽⫺
kL N i⫽1 k⫽1
yk (i)ln
yˆ k (i) yk (i)
(4.33)
173
“06-Ch04-SA272” 18/9/2008 page 174
174
CHAPTER 4 Nonlinear Classifiers
This is also valid for binary target values (using the limiting form). However,although we have interpreted the outputs as probabilities,there is no guarantee that they sum up to unity. This can be imposed onto the network by adopting an alternative activation function for the output nodes. In [Brid 90] the so-called softmax activation function was suggested, given by yˆ k ⫽
exp(vkL ) L k⬘ exp(v ⬘ )
(4.34)
k
This guarantees that the outputs lie in the interval [0, 1] and that they sum up to unity (note that in contrast to (4.32) the output probabilities are not considered independently). It is easy to show, ( Problem 4.7) that in this case the quantity ␦Lj required by the backpropagation is equal to yˆ k ⫺ yk . Besides the cross-entropy cost function in (4.33) a number of alternative cost functions has been proposed. For example, in [Kara 92] a generalization of the quadratic error cost function is utilized with the aim of speeding up convergence. Another direction is to minimize the classification error, which after all is the major goal in pattern recognition. A number of techniques have been suggested with this philosophy [Nede 93, Juan 92, Pado 95], which is known as discriminative learning. The basic potential advantage of discriminative learning is that essentially it tries to move the decision surfaces so as to reduce classification error. To achieve this goal, it puts more emphasis on the largest of the class a posteriori probability estimates. In contrast,the squared error cost function,for example,assigns the same importance to all posterior probability estimates. In other words, it tries to learn more than what is necessary for classification, which may limit its performance for a fixed size network. Most of the discriminative learning techniques use a smoothed version of the classification error, so as to be able to apply differentiation in association with gradient descent approaches. This,of course,presents the danger that the minimization procedure will be trapped in a local minimum. In [Mill 96] a deterministic annealing procedure is employed to train the networks, with an enhanced potential to avoid local minima (see Chapter 15). The final choice of the cost function depends on the specific problem under consideration. However, as is pointed out in [Rich 91], in a number of practical situations the use of alternative, to least squares, cost functions did not necessarily lead to substantial performance improvements. So far, the task of training multilayer perceptrons has been approached via the unconstrained optimization route (Appendix C). However, more recent research has shown that it is often beneficial to incorporate additional knowledge in the learning rule using constrained optimization. This has been shown to lead to the formulation of efficient learning algorithms with accelerated learning properties. The additional knowledge can be encoded in the form of objectives leading to single- or multi-objective optimization criteria that have to be satisfied simultaneously, with the demand for a long-term decrease of the cost function [Pera 00].
“06-Ch04-SA272” 18/9/2008 page 175
4.8 The Cost Function Choice
In this approach, an optimization problem is formulated at each epoch of the learning process. For example, the requirement of partial alignment of current and previous epoch weight vector updates (respectively denoted by ⌬w(t) and ⌬w(t ⫺ 1)), which basically underlies the use of the momentum term in the backpropagation algorithm, is enhanced by requiring maximization of the quantity ⌽ ⫽ ⌬w(t)T ⌬w(t ⫺ 1) and simultaneously allowing for a controlled decrease of the cost function. This leads to a constrained first-order algorithm, and it has been reported that it outperforms backpropagation and several of its variants in different benchmark learning tasks [Pera 95]. Generalizations of the method involving the Hessian matrix have also been proposed and used successfully in several benchmarks and applications [Ampa 02, Huan 04]. Under certain conditions, the solution of the problem for each epoch is provided analytically, leading to a closed formula for the weight update rule [Pera 03].
A Bayesian Framework for Network Training All the cost functions considered so far aim at computing a single set of optimal values for the unknown parameters of the network. An alternative rationale is to look at the probability distribution function of the unknown weights, w, in the weight space. The idea behind this approach stems from the Bayesian inference technique used for the estimating an unknown parametric pdf, as we discussed in Chapter 2. The basic steps followed for this type of network training, known as Bayesian learning, are (e.g., [Mack 92a]): ■
■
■
Assume a model for the prior distribution p(w) of the weights. This must be rather broad in shape in order to provide equal chance to a rather large range of values. Let Y ⫽ { y(i), i ⫽ 1, 2, . . . , N } be the set of the desired output training vectors for a given input data set X ⫽ {x(i), i ⫽ 1, 2, . . . , N }. Assume a model for the likelihood function p(Y |w), for example, Gaussian.4 This basically models the error distribution between the true and desired output values, and it is the stage at which the input training data come into the scene. Using Bayes’s theorem, we obtain p(w|Y ) ⫽
p(Y |w)p(w) p(Y )
(4.35)
where p(Y ) ⫽ p(Y |w)p(w) dw. The resulting posterior pdf will be more sharply shaped around a value w 0 , since it has learned from the available training data. Strictly speaking we should write P(Y |w, X). However, all probabilities and pdf’s are conditioned on X, and we omit it for notational convenience.
4
175
“06-Ch04-SA272” 18/9/2008 page 176
176
CHAPTER 4 Nonlinear Classifiers
■
Interpreting the true outputs of a network, yˆ k ⫽ k (x; w), as the respective class probabilities, conditioned on the input x and the weight vector w, the conditional class probability is computed by averaging over all w [Mack 92b]: P(k |x; Y ) ⫽
k (x; w)p(w|Y ) dw
(4.36)
The major computational cost associated with this type of technique is due to the required integration in the multidimensional space. This is not an easy task, and various practical implementations have been suggested in the literature. Further discussion of these issues is beyond the scope of this book. A good introduction to Bayesian learning, including a discussion of related practical implementations, is provided in [Bish 95].
4.9 CHOICE OF THE NETWORK SIZE In the previous sections, we assumed the number of layers and neurons for each layer to be known and fixed. How one determines the appropriate number of layers and neurons was not of interest to us. This task will become our major focus now. One answer to the problem could be to choose the size of the network large enough and leave the training to decide about the weights. A little thought reveals that such an approach is rather naive. Besides the associated computational complexity problems, there is a major reason why the size of the network should be kept as small as possible. This is imposed by the generalization capabilities that the network must possess. As has already been pointed out in Section 3.7, the term generalization refers to the capability of the multilayer neural network (and of any classifier) to classify correctly feature vectors that were not presented to it during the training phase—that is,the capability of a network to decide upon data unknown to it, based on what it has learned from the training set. Taking for granted the finite (and in many cases small) number N of training pairs, the number of free parameters (synaptic weights) to be estimated should be (a) large enough to learn what makes “similar” the feature vectors within each class and at the same time what makes one class different from the other and ( b) small enough, with respect to N , so as not to be able to learn the underlying differences among the data of the same class. When the number of free parameters is large, the network tends to adapt to the particular details of the specific training data set. This is known as overfitting and leads to poor generalization performance, when the network is called to operate on feature vectors unknown to it. In conclusion, the network should have the smallest possible size to adjust its weights to the largest regularities in the data and ignore the smaller ones, which might also be the result of noisy measurements. Some theoretical touches concerning the generalization aspects of a classifier will
“06-Ch04-SA272” 18/9/2008 page 177
Error
4.9 Choice of the Network Size
Test set
Training set
Epochs
FIGURE 4.14 Trend of the output error versus the number of epochs illustrating overtraining of the training set.
be presented in Chapter 5, when we discuss the Vapnik–Chervonenkis dimension. The bias–variance dilemma, discussed in Chapter 3, is another side of the same problem. Adaptation of the free parameters to the peculiarities of the specific training set may also occur as the result of overtraining (e.g., see [Chau 90]). Let us assume that we can afford the luxury of having a large set of training data. We divide this set into two subsets, one for training and one for test. The latter is known as the validation or test set. Figure 4.14 shows the trend of two curves of the output error as a function of iteration steps. One corresponds to the training set, and we observe that the error keeps decreasing as the weights converge. The other corresponds to the error of the validation set. Initially, the error decreases, but at some later stage it starts increasing. This is because the weights, computed from the training set, adapt to the idiosyncrasies of the specific training set, thus affecting the generalization performance of the network. This behavior could be used in practice to determine the point where the learning process iterations must terminate. This is the point where the two curves start departing. However, this methodology assumes the existence of a large number of data sets, which is not usually the case in practice. Besides generalization, other performance factors also demand to keep the size of a network as small as possible. Small networks are computationally faster and cheaper to build. Furthermore, their performance is easier to understand, which is important in some critical applications.
177
“06-Ch04-SA272” 18/9/2008 page 178
178
CHAPTER 4 Nonlinear Classifiers
In this section we focus on methods that select the appropriate number of free parameters, under certain criteria and for a given dimension of the input vector space. The latter is very important, because the input data dimension is no doubt related to the number of free parameters to be used;thus it also affects generalization properties. We will come to issues related to input space dimension reduction in Chapter 5. The most widely used approaches to selecting the size of a multilayer network come under one of the following categories: ■
Analytical methods. This category employs algebraic or statistical techniques to determine the number of its free parameters.
■
Pruning techniques. A large network is initially chosen for training, and then the number of free parameters is successively reduced, according to a preselected rule.
■
Constructive techniques. A small network is originally selected, and neurons are successively added, based on an appropriately adopted learning rule.
Algebraic Estimation of the Number of Free Parameters We have already discussed in Section 4.3.1 the capabilities of a multilayer perceptron, with one hidden layer and units of the McCulloch–Pitts type, to divide the input l-dimensional space into a number of polyhedral regions. These are the result of intersections of hyperplanes formed by the neurons. In [Mirc 89] it is shown that in the l-dimensional space a multilayer perceptron of a single hidden layer with K neurons can form a maximum of M polyhedral regions with M given by M⫽
l m⫽0
and
K , m
K where ⫽ 0, for K ⬍ m m
(4.37)
K! K ≡ m m!(K ⫺ m)!
For example, if l ⫽ 2, and K ⫽ 2, this results in M ⫽ 4; thus, the XOR problem, with M ⫽ 3(⬍ 4), can be solved with two neurons. The disadvantage of this method is that it is static and does not take into consideration the cost function used as well as the training procedure.
Pruning Techniques These techniques start training a sufficiently large network, and then they remove, in a stepwise procedure, the free parameters that have little influence on the cost function. There are two major methodological directions:
“06-Ch04-SA272” 18/9/2008 page 179
4.9 Choice of the Network Size
Methods Based on Parameter Sensitivity Calculations Let us take for example the technique suggested in [Lecu 90]. Using a Taylor series expansion,the variation imposed on the cost function by parameter perturbations is ␦J ⫽
gi ␦wi ⫹
i
1 1 hii ␦wi2 ⫹ hij ␦wi ␦wj 2 2 i
i,j i⫽j
⫹ higher order terms
where gi ⫽
⭸J , ⭸wi
hij ⫽
⭸2 J ⭸wi ⭸wj
and i, j runs over all the weights. The derivatives can be computed via the backpropagation methodology ( Problem 4.13). In practice,the derivatives are computed after some initial period of training. This allows us to adopt the assumption that a point near a minimum has been reached and the first derivatives can be set equal to zero. A further computational simplification is to assume that the Hessian matrix is diagonal. Under these assumptions, the cost function sensitivity is approximately given by ␦J ⫽
1 hii ␦wi2 2
(4.38)
i
and the contribution of each parameter is determined by the saliency value si ,given approximately by si ⫽
hii wi2 2
(4.39)
where we assume that a weight of value wi is changed to zero. Pruning is now achieved in an iterative fashion according to the following steps: ■
The network is trained using the backpropagation algorithm for a number of iteration steps so that its cost function is reduced to a sufficient percentage.
■
For the current weight estimates, the respective saliency values are computed and weights with small salencies are removed.
■
The training process is continued with the remaining weights,and the process is repeated after some iteration steps. The process is stopped when a chosen stopping criterion is met.
In [Hass 93] the full Hessian matrix has been employed for the pruning procedure. It should be stressed that, although the backpropagation concept is present in this technique, the learning procedure is distinctly different from the backpropagation
179
“06-Ch04-SA272” 18/9/2008 page 180
180
CHAPTER 4 Nonlinear Classifiers
training algorithm of Section 4.6. There, the number of free parameters was fixed throughout the training. In contrast, the philosophy here is exactly the opposite.
Methods Based on Cost Function Regularization These methods achieve the reduction of the originally large size of the network by including penalty terms in the cost function. The cost function now has the form J⫽
N
E(i) ⫹ ␣Ep (w)
(4.40)
i⫽1
The first term is the performance cost function, and it is chosen according to what we have already discussed (e.g., least squares, cross entropy). The second depends on the weight vector, and it is chosen to favor small values for the weights. The constant ␣ is the so-called regularization parameter, and it controls the relative significance of these two terms. A popular form for the penalty term is Ep (w) ⫽
K
h(wk2 )
(4.41)
k⫽1
with K being the total number of weights in the network and h(·) an appropriately chosen differentiable function. According to such a choice, weights that do not contribute significantly in the formation of the network output do not materially affect much the first term of the cost function. Hence, the existence of the penalty term drives them to small values. Thus, pruning is achieved. In practice, a threshold is preselected and weights are compared against it after a number of iteration steps. Weights that become smaller than it are removed,and the process is continued. This type of pruning is known as weight elimination. Function h(·) can take various forms. For example, in [Wein 90] the following is suggested: h(wk2 ) ⫽
w02
wk2 ⫹ wk2
(4.42)
where w0 is a preselected parameter close to unity. Closer observation of this penalty term reveals that it goes to zero very fast for values wk ⬍ w0 ; thus such weights become insignificant. In contrast, the penalty term tends to unity for wk ⬎ w0 . A variation of (4.41) is to include in the regularized cost function another penalty term that favors small values of ykr , that is, small neuron outputs. Such techniques lead to removal of insignificant neurons as well as weights. A summary and discussion of various pruning techniques can be found in [Refe 91, Russ 93]. Keeping the size of the weights small is in line with the theoretical results obtained in [Bart 98]. Assume that a large multilayer perceptron is used as a classifier and that the learning algorithm finds a network with (a) small weights and (b) small squared error on the training patterns. Then, it can be shown that the
“06-Ch04-SA272” 18/9/2008 page 181
4.10 A Simulation Example
generalization performance depends on the size of the weights rather than the number of weights. More specifically, for a two-layer perceptron with sigmoid activation functions if A is an upper bound of the sum of the magnitudes of the weights associated with each neuron, then the associated classification error probability is no√more than a certain error estimate (related to the output squared error) plus A3 (log l)/N . This is a very interesting result indeed. It confronts the generally accepted fact that the generalization performance is directly related to the number of training points and the number of free parameters. It also explains why sometimes the generalization error performance of multilayer perceptrons is good, although the training has been performed with a relatively small (compared to the size of the network) number of training points. Further discussion concerning the generalization performance of a classifier and some interesting theoretical results is found at the end of Chapter 5.
Constructive Techniques In Section 4.5 we have already discussed such techniques for training neural networks. However, the activation function was the unit-step function, and also the emphasis was put on classifying correctly all input training data and not on the generalization properties of the resulting network. In [Fahl 90] an alternative constructive technique for training neural networks, with a single hidden layer and sigmoid activation functions, was proposed, known as cascade correlation. The network starts with input and output units only. Hidden neurons are added one by one and are connected to the network with two types of weights. The first type connects the new unit with the input nodes as well as the outputs of previously added hidden neurons. Each time a new hidden neuron is added in the network, these weights are trained so as to maximize the correlation between the new unit’s output and the residual error signal in the network outputs prior to the addition of the new unit. Once a neuron is added, these weights are computed once and then they remain fixed. The second type of synaptic weights connects the newly added neuron with the output nodes. These weights are not fixed and are trained adaptively,each time a new neuron is installed,in order to minimize a sum of squares error cost function. The procedure stops when the performance of the network meets the prespecified goals. Discussion of constructive techniques with an emphasis on pattern recognition can be found in [Pare 00].
4.10 A SIMULATION EXAMPLE This section demonstrates the capability of a multilayer perceptron to classify nonlinearly separable classes. The classification task consists of two distinct classes, each being the union of four regions in the two-dimensional space. Each region consists of normally distributed random vectors with statistically independent
181
“06-Ch04-SA272” 18/9/2008 page 182
CHAPTER 4 Nonlinear Classifiers
components and each with variance 2 ⫽ 0.08. The mean values are different for each of the regions. Specifically, the regions of the class denoted by red “◦” (see Figure 4.15) are formed around the mean vectors [0.4, 0.9]T , [2.0, 1.8]T , [2.3, 2.3]T , [2.6, 1.8]T
and those of the class denoted by black “⫹” around the values [1.5, 1.0]T , [1.9, 1.0]T , [1.5, 3.0]T , [3.3, 2.6]T
A total of 400 training vectors were generated,50 from each distribution. A multilayer perceptron, with three neurons in the first and two neurons in the second hidden layer,were used,with a single output neuron. The activation function was the logistic one with a ⫽ 1 and the desired outputs 1 and 0, respectively, for the two classes. Two different algorithms were used for the training, namely, the momentum and the adaptive momentum. After some experimentation the algorithmic parameters employed were (a) for the momentum ⫽ 0.05, ␣ ⫽ 0.85 and (b) for the adaptive momentum ⫽ 0.01, ␣ ⫽ 0.85, ri ⫽ 1.05, c ⫽ 1.05, rd ⫽ 0.7. The weights were initialized by a uniform pseudorandom distribution between 0 and 1. Figure 4.15a shows the respective output error convergence curves for the two algorithms as a function of the number of epochs (each epoch consisting of the 400 training feature vectors). The respective curves can be considered typical and the adaptive momentum algorithm leads to faster convergence. Both curves correspond to the batch mode of operation. Figure 4.15b shows the resulting decision surface using 103
x2 3
Sum-squares error
182
2
10
2
101
1
100
1021
0
2000
4000 Epochs (a)
6000
0
0
1
2
3 x1
(b)
FIGURE 4.15 (a) Error convergence curves for the adaptive momentum (dark line) and the momentum algorithms. Note that the adaptive momentum leads to faster convergence. (b) The decision curve formed by the multilayer perceptron.
“06-Ch04-SA272” 18/9/2008 page 183
4.11 Networks with Weight Sharing
x2 1.5
x2 1.5
1
1
0.5
0.5
0
0
0.5
1 (a)
1.5 x1
0
0
0.5
1 (b)
1.5 x1
FIGURE 4.16 Decision curve (a) before pruning and (b) after pruning.
the weights estimated from the adaptive momentum training. Once the weights of the network have been estimated, the decision surface can easily be drawn. To this end, a two-dimensional grid is constructed over the area of interest, and the points of the grid are given as inputs to the network, row by row. The decision surface is formed by the points where the output of the network changes from 0 to 1 or vice versa. A second experiment was conducted in order to demonstrate the effect of the pruning. Figure 4.16 shows the resulting decision surfaces separating the samples of the two classes, denoted by black and red “◦,” respectively. Figure 4.16a corresponds to a multilayer perceptron (MLP), with two hidden layers and 20 neurons in each of them, amounting to a total of 480 weights. Training was performed via the backpropagation algorithm. The overfitting nature of the resulting curve is readily observed. Figure 4.16b corresponds to the same MLP trained with a pruning algorithm. Specifically, the method based on parameter sensitivity was used, testing the saliency values of the weights every 100 epochs and removing weights with saliency value below a chosen threshold. Finally, only 25 of the 480 weights were left, and the curve is simplified to a straight line.
4.11 NETWORKS WITH WEIGHT SHARING One major issue encountered in many pattern recognition applications is that of transformation invariance. This means that the pattern recognition system should classify correctly, independent of transformations performed on the input space, such as translation, rotation, and scaling. For example, the character “5” should “look the same”to an optical character recognition system, regardless of its position,
183
“06-Ch04-SA272” 18/9/2008 page 184
184
CHAPTER 4 Nonlinear Classifiers
orientation, and size. There are a number of ways to approach this problem. One is to choose appropriate feature vectors, which are invariant under such transformations. This will be one of our major goals in Chapter 7. Another way is to make the classifier responsible for it in the form of built-in constraints. Weight sharing is such a constraint, which forces certain connections in the network to have the same weights. One type of network in which the concept of weight sharing has been adopted is the so-called higher order network. These are multilayer perceptrons with activation functions acting on nonlinear, instead of linear, combinations of the input parameters. The outputs of the neurons are now of the form f () ⫽ f w0 ⫹ wi xi ⫹ wjk xj xk i
jk
This can be generalized to include higher order products. Let us now assume that the inputs to the network originate from a two-dimensional grid (image). Each point of the grid corresponds to a specific xi and each pair (xi , xj ) to a line segment. Invariance to translation is built in by constraining the weights wjk ⫽ wrs , whenever the respective line segments, defined by the points (xj , xk ) and (xr , xs ), are of the same gradient. Invariance to rotation can be built in by sharing weights corresponding to segments of equal length. Of course, all these are subject to inaccuracies caused by the resolution coarseness of the grid. Higher order networks can accommodate more complex transformations [Kana 92, Pera 92, Delo 94]. Because of the weight sharing, the number of free parameters for optimization is substantially reduced. However, it must be pointed out that, so far, such networks have not been widely used in practice. A special type of network called the circular backpropagation model results if f () ⫽ f w0 ⫹ wi x i ⫹ w s xi2 i
i
The increase in the number of parameters is now small, and in [Ride 97] it is claimed that the involvement of the nonlinear term offers the network increased representation power without affecting its generalization capabilities. Besides the higher order networks, weight sharing has been used to impose invariance on first-order networks used for specific applications [Fuku 82, Rume 86, Fuku 92, Lecu 89]. The last, for example, is a system for handwritten zip code recognition. It is a hierarchical structure with three hidden layers and inputs the gray levels of the image pixels. Nodes in the first two layers form groups of twodimensional arrays known as feature maps. Each node in a given map receives inputs from a specific window area of the previous layer, known as the receptive field. Translation invariance is imposed by forcing corresponding nodes in the same map, looking at different receptive fields, to share weights. Thus, if an object moves from one input receptive field to the other, the network responds in the same way.
“06-Ch04-SA272” 18/9/2008 page 185
4.12 Generalized Linear Classifiers
4.12 GENERALIZED LINEAR CLASSIFIERS In Section 4.3,dealing with the nonlinearly separable XOR problem,we saw that the neurons of the hidden layer performed a mapping that transformed the problem to a linearly separable one. The actual mapping was x ⫺→y
with
y⫽
f (g1 (x)) y1 ⫽ f (g2 (x)) y2
(4.43)
where f (·) is the activation function and gi (x), i ⫽ 1, 2, the linear combination of the inputs performed by each neuron. This will be our kickoff point for this section. Let us consider our feature vectors to be in the l-dimensional space Rl and assume that they belong to either of the two classes A, B, which are nonlinearly separable. Let f1 (·), f2 (·), . . . , fk (·) be nonlinear (in the general case) functions fi : Rl → R, i ⫽ 1, 2, . . . , k
which define the mapping x ∈ Rl → y ∈ Rk ⎤ f1 (x) ⎢ f (x)⎥ ⎥ ⎢ 2 ⎥ y≡⎢ ⎢ .. ⎥ ⎣ . ⎦ fk (x) ⎡
(4.44)
Our goal now is to investigate whether there is an appropriate value for k and functions fi so that classes A, B are linearly separable in the k-dimensional space of the vectors y. In other words, we investigate whether there exists a k-dimensional space where we can construct a hyperplane w ∈ Rk so that w0 ⫹ w T y ⬎ 0, x ∈ A
(4.45)
w0 ⫹ w y ⬍ 0, x ∈ B
(4.46)
T
Assuming that in the original space the two classes were separable by a (nonlinear) hypersurface g(x) ⫽ 0, relations (4.45), (4.46) are basically equivalent to approximating the nonlinear g(x) as a linear combination of fi (x), that is, g(x) ⫽ w0 ⫹
k
wi fi (x)
(4.47)
i⫽1
This is a typical problem of function approximation in terms of a preselected class of interpolation functions fi (·). This is a well-studied task in numerical analysis,
185
“06-Ch04-SA272” 18/9/2008 page 186
186
CHAPTER 4 Nonlinear Classifiers
x1
f1
y1 w1
x2
f2
y2
w2 Σ
yk xl
wk
w0
fk
FIGURE 4.17 Generalized linear classifier.
and a number of different interpolating functions have been proposed (exponential, polynomial,Tchebyshev,etc.). In the next sections,we will focus on two such classes of functions, which have been widely used in pattern recognition. Once the functions fi have been selected, the problem becomes a typical design of a linear classifier, that is, to estimate the weights wi in the k-dimensional space. This justifies the term generalized linear classification. Figure 4.17 shows the corresponding block diagram. The first layer of computations performs the mapping to the y space; the second layer performs the computation of the decision hyperplane. In other words, (4.47) corresponds to a two-layer network where the nodes of the hidden layer have different activation functions,fi (·), i ⫽ 1, 2, . . . , k. For an M-class problem we need to design M such weight vectors w r , r ⫽ 1, 2, . . . , M, one for each class, and select the rth class according to the maximum output w Tr y ⫹ wr0 . A similar expression to (4.47) expansion of g(x) is known as projection pursuit, introduced in [Fried 81], and it is defined as g(x) ⫽
k
fi (w Ti x)
i⫽1
Observe that the argument in each of the functions fi (·) is not the feature vector x but its projection on the direction determined by the respective w i . Optimization with respect to fi and w i , i ⫽ 1, 2, . . . , k, results in the best choice for directions to project as well as the interpolation functions. If fi (·) are all chosen a priori to be sigmoid functions, the projection pursuit method becomes identical to a neural network with a single hidden layer. Projection pursuit models are not members of the generalized linear models family, and their optimization is carried out iteratively in a two-stage fashion. Given the functions fi (·), w i s are estimated and in the next stage optimization is performed with respect to the fi s. See, for example, [Fried 81]. Although interesting from a theoretical point of view, it seems that in practice projection pursuit methods have been superseded by the multilayer perceptrons.
“06-Ch04-SA272” 18/9/2008 page 187
4.13 The l -Dimensional Space in Linear Dichotomies
In the sequel,we will first try to justify our expectations,that by going to a higher dimensional space the classification task may be transformed into a linear one, and then study popular alternatives for the choice of functions fi (·) in (4.44).
4.13 CAPACITY OF THE l -DIMENSIONAL SPACE IN LINEAR DICHOTOMIES Let us consider N points in the l-dimensional space. We will say that these points are in general position or well distributed if there is no subset of l ⫹ 1 of them that lie on an (l ⫺ 1)-dimensional hyperplane. Such a definition excludes detrimental cases, such as in the two-dimensional space having three points on a straight line (a one-dimensional hyperplane). The number O(N , l) of groupings that can be formed by (l ⫺ 1)-dimensional hyperplanes to separate the N points in two classes, taking all possible combinations, is given by ([Cove 65] and Problem 4.18): l N ⫺1 O(N , l) ⫽ 2 i
(4.48)
i⫽0
where
(N ⫺ 1)! N ⫺1 ⫽ i (N ⫺ 1 ⫺ i)!i!
(4.49)
Each of these two class groupings is also known as a (linear) dichotomy. From the properties of the binomial coefficients, it turns out that for N ⱕ l ⫹ 1, O(N , l) ⫽ 2N . Figure 4.18 shows two examples of such hyperplanes resulting in O(4, 2) ⫽ 14 and O(3, 2) ⫽ 8 two-class groupings, respectively. The seven lines of Figure 4.18a form the following groupings. [(ABCD)], [A,(BCD)], [B,(ACD)], [C,(ABD)], [D,(ABC)], [(AB),(CD)], and [(AC),(BD)]. Each grouping corresponds to two possibilities. For example, (ABCD) can belong to either class 1 or 2 . Thus, the total number of
B A
A
D C C (a)
B
(b)
FIGURE 4.18 Number of linear dichotomies (a) for four and (b) for three points.
187
“06-Ch04-SA272” 18/9/2008 page 188
188
CHAPTER 4 Nonlinear Classifiers
combinations of assigning four points in the two-dimensional space in two linearly separable classes is 14. This number is obviously smaller than the total number of combinations of assigning N points in two classes, which is known to be 2N . This is because the latter also involves nonlinearly separable combinations. In the case of our example, this is 16, which arises from the two extra possibilities of the grouping [(AD), (BC)]. We are now ready to write the probability (percentage) of grouping N points in the l-dimensional space in two linearly separable classes [Cove 65]. This is given by:
PNl ⫽
⎧ ⎪ ⎪ ⎨
1
O(N , l) 2N ⫺1 ⫽ N ⎪ 2 ⎪ ⎩
l
N ⫺1
i⫽0
i
N ⬎l⫹1 (4.50) N ⱕl⫹1
1
A practical way to study the dependence of PNl on N and l is to assume that N ⫽ r(l ⫹ 1) and investigate the probability for various values of r. The curve in Figure 4.19 shows the probability of having linearly separable classes for various values of l. It is readily observed that there are two regions, one to the left of r ⫽ 2, that is, N ⫽ 2(l ⫹ 1), and one to the right. Furthermore, all curves go through the point (PNl , r) ⫽ (1/2, 2), since O(2l ⫹ 2, l) ⫽ 22l⫹1 (Problem 4.19). The transition from one region to the other becomes sharper as l → ⬁. Thus, for large values of l and if N ⬍ 2(l ⫹ 1) the probability of any two groups of the N points being linearly separable approaches unity. The opposite is true if N ⬎ 2(l ⫹ 1). In practice, where we cannot afford the luxury of very large values of N and l, our findings P l5` l1 l2
l1 > l2
1 2
0
1
2
r
FIGURE 4.19 Probability of linearly separable groupings of N ⫽ r (l ⫹ 1) points in the l-dimensional space.
“06-Ch04-SA272” 18/9/2008 page 189
4.14 Polynomial Classifiers
guarantee that if we are given N points, then mapping into a higher dimensional space increases the probability of locating them in linearly separable two-class groupings.
4.14 POLYNOMIAL CLASSIFIERS In this section we will focus on one of the most popular classes of interpolation functions fi (x) in (4.47). Function g(x) is approximated in terms of up to order r polynomials of the x components, for large enough r. For the special case of r ⫽ 2 we have g(x) ⫽ w0 ⫹
l i⫽1
wi xi ⫹
l⫺1 l
wim xi xm ⫹
i⫽1 m⫽i⫹1
l
wii xi2
(4.51)
i⫽1
If x ⫽ [x1 , x2 ]T , then the general form of y will be y ⫽ [x1 , x2 , x1 x2 , x12 , x22 ]T
and g(x) ⫽ w T y ⫹ w0 w T ⫽ [w1 , w2 , w12 , w11 , w22 ]
The number of free parameters determines the new dimension k. The generalization of (4.51) for rth-order polynomials is straightforward,and it will contain products of p p p the form x1 1 x2 2 . . . xl l where p1 ⫹ p2 ⫹ · · · ⫹ pl ⱕ r. For an rth-order polynomial and l-dimensional x it can be shown that k⫽
(l ⫹ r)! r!l!
For l ⫽ 10 and r ⫽ 10 we obtain k ⫽ 184,756 (!!). That is, even for medium-size values of the network order and the input space dimensionality the number of free parameters gets very high. Let us consider, for example, our familiar nonlinearly separable XOR problem. Define ⎡
⎤ x1 ⎢ ⎥ y ⫽ ⎣ x2 ⎦ x1 x2
(4.52)
The input vectors are mapped onto the vertices of a three-dimensional unit (hyper) cube, as shown in Figure 4.20a ((00) → (000), (11) → (111), (10) → (100), (01) → (010)). These vertices are separable by the plane y1 ⫹ y2 ⫺ 2y3 ⫺
1 ⫽0 4
189
“06-Ch04-SA272” 18/9/2008 page 190
190
CHAPTER 4 Nonlinear Classifiers
011
x2
111
010
110 001
101 y2
000
(0,1)
(1, 1)
(0,0)
(1, 0) x1
y3
100
y1
(a)
(b)
FIGURE 4.20 The XOR classification task, via the polynomial generalized linear classifier. (a) Decision plane in the three-dimensional space and (b) decision curves in the original two-dimensional space.
The plane in the three-dimensional space is equivalent to the decision function g(x) ⫽ ⫺
1 ⬎0 ⫹ x1 ⫹ x2 ⫺ 2x1 x2 ⬍0 4
x∈A x∈B
in the original two-dimensional space, which is shown in Figure 4.20b.
4.15 RADIAL BASIS FUNCTION NETWORKS The interpolation functions (kernels) that will be considered in this section are of the general form f ( x ⫺ c i )
That is, the argument of the function is the Euclidean distance of the input vector x from a center c i , which justifies the name radial basis function ( RBF). Function f can take various forms, for example, 1 f (x) ⫽ exp ⫺ 2 x ⫺ c i 2 2i f (x) ⫽
2
2 ⫹ x ⫺ c i 2
(4.53)
(4.54)
The Gaussian form is more widely used. For a large enough value of k, it can be shown that the function g(x) is sufficiently approximated by [Broo 88, Mood 89] g(x) ⫽ w0 ⫹
(x ⫺ c )T (x ⫺ c ) i i wi exp ⫺ 2 2 i i⫽1
k
(4.55)
“06-Ch04-SA272” 18/9/2008 page 191
4.15 Radial Basis Function Networks
That is, the approximation is achieved via a summation of RBFs, where each is located on a different point in the space. One can easily observe the close relation that exists between this and the Parzen approximation method for the probability density functions of Chapter 2. Note, however, that there the number of the kernels was chosen to be equal to the number of training points k ⫽ N . In contrast, in (4.55) k ⬍⬍ N . Besides the gains in computational complexity, this reduction in the number of kernels is beneficial for the generalization capabilities of the resulting approximation model. Coming back to Figure 4.17, we can interpret (4.55) as the output of a network with one hidden layer of RBF activation functions (e.g., (4.53), (4.54)) and a linear output node. As has already been said in Section 4.12, for an M-class problem there will be M linear output nodes. At this point, it is important to stress one basic difference between RBF networks and multilayer perceptrons. In the latter, the inputs to the activation functions, of the first hidden layer, are linear combinations of the input feature parameters wj xj . That is, the output of each neuron is j
the same for all {x: j wj xj ⫽ c}, where c is a constant. Hence, the output is the same for all points on a hyperplane. In contrast, in the RBF networks the output of each RBF node, fi (·), is the same for all points having the same Euclidean distance from the respective center c i and decreases exponentially (for Gaussians) with the distance. In other words, the activation responses of the nodes are of a local nature in the RBF and of a global nature in the multilayer perceptron networks.This intrinsic difference has important repercussions for both the convergence speed and the generalization performance. In general,multilayer perceptrons learn slower than their RBF counterparts. In contrast,multilayer perceptrons exhibit improved generalization properties, especially for regions that are not represented sufficiently in the training set [Lane 91]. Simulation results in [Hart 90] show that, in order to achieve performance similar to that of multilayer perceptrons, an RBF network should be of much higher order. This is due to the locality of the RBF activation functions, which makes it necessary to use a large number of centers to fill in the space in which g(x) is defined, and this number exhibits an exponential dependence on the dimension of the input space (curse of dimensionality) [Hart 90]. Let us now come back to our XOR problem and adopt an RBF network to perform the mapping to a linearly separable class problem. Choose k ⫽ 2, the centers c 1 ⫽ [1, 1]T , c 2 ⫽ [0, 0]T , and f (x) ⫽ exp(⫺ x ⫺ c i 2 ). The corresponding y resulting from the mapping is y ⫽ y(x) ⫽
exp(⫺ x ⫺ c 1 2 ) exp(⫺ x ⫺ c 2 2 )
Hence (0, 0) → (0.135, 1), (1, 1) → (1, 0.135), (1, 0) → (0.368, 0.368), (0, 1) → (0.368, 0.368). Figure 4.21a shows the resulting class position after the mapping
191
“06-Ch04-SA272” 18/9/2008 page 192
192
CHAPTER 4 Nonlinear Classifiers
y2 x2
XOR
(1, 1)
(0, 1)
B 1
(0, 0)
A
(1, 0) x1
B 0
y1
1 (a)
(b)
FIGURE 4.21 Decision curves formed by an RBF generalized linear classifier for the XOR task. The decision curve is linear in the transformed space (a) and nonlinear in the original space (b).
in the y space. Obviously, the two classes are now linearly separable and the straight line g( y) ⫽ y1 ⫹ y2 ⫺ 1 ⫽ 0
is a possible solution. Figure 4.21b shows the equivalent decision curve, g(x) ⫽ exp(⫺ x ⫺ c 1 2 ) ⫹ exp(⫺ x ⫺ c 2 2 ) ⫺ 1 ⫽ 0
in the input vector space. In our example we selected the centers c 1 , c 2 as [0, 0]T and [1, 1]T . The question now is, why these specific ones? This is an important issue for RBF networks. Some basic directions on how to tackle this problem are given in the following.
Fixed Centers Although in some cases the nature of the problem suggests a specific choice for the centers [Theo 95], in the general case these centers can be selected randomly from the training set. Provided that the training set is distributed in a representative manner over all the feature vector space,this seems to be a reasonable way to choose the centers. Having now selected k centers for the RBF functions, the problem has become a typical linear one in the k-dimensional space of the vectors y, ⫺ x ⫺c 1 2 exp ⎢ 212 ⎢ ⎢ .. y⫽⎢ . ⎢ ⎣ ⫺ x ⫺c k 2 exp 2 ⎡
2k
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
“06-Ch04-SA272” 18/9/2008 page 193
4.15 Radial Basis Function Networks
where the variances are also considered to be known, and g(x) ⫽ w0 ⫹ w T y
All methods described in Chapter 3 can now be recalled to estimate w0 and w.
Training of the Centers If the centers are not preselected, they have to be estimated during the training phase along with the weights wi and the variances i2 , if the latter are also considered unknown. Let N be the number of input–desired output training pairs (x( j), y( j), j ⫽ 1, . . . , N ). We select an appropriate cost function of the output error J⫽
N
(e( j))
j⫽1
where (·) is a differentiable function (e.g., the square of its argument) of the error e( j) ⫽ y( j) ⫺ g(x( j))
Estimation of the weights wi , the centers c i , and the variances i2 becomes a typical task of a nonlinear optimization process. For example, if we adopt the gradient descent approach, the following algorithm results: ⭸J !! ! , i ⫽ 0, 1, . . . , k ⭸wi t ⭸J !! c i (t ⫹ 1) ⫽ c i (t) ⫺ 2 i ⫽ 1, 2, . . . , k !, ⭸c i t ⭸J !! i (t ⫹ 1) ⫽ i (t) ⫺ 3 ! , i ⫽ 1, 2, . . . , k ⭸i t
wi (t ⫹ 1) ⫽ wi (t) ⫺ 1
(4.56) (4.57) (4.58)
where t is the current iteration step. The computational complexity of such a scheme is prohibitive for a number of practical situations. To overcome this drawback, alternative techniques have been suggested. One way is to choose the centers in a manner that is representative of the way data are distributed in space. This can be achieved by unraveling the clustering properties of the data and choosing a representative for each cluster as the corresponding center [Mood 89]. This is a typical problem of unsupervised learning,and algorithms discussed in the relevant chapters later in the book can be employed. The unknown weights, wi , are then learned via a supervised scheme (i.e., gradient descent algorithm) to minimize the output error. Thus, such schemes use a combination of supervised and unsupervised learning procedures. An alternative strategy is described in [Chen 91]. A large number of candidate centers is initially chosen from the training vector set. Then, a forward linear regression technique is employed, such as orthogonal least squares, which leads to a
193
“06-Ch04-SA272” 18/9/2008 page 194
194
CHAPTER 4 Nonlinear Classifiers
parsimonious set of centers. This technique also provides a way to estimate the order of the model k. A recursive form of the method, which can lead to computational savings, is given in [Gomm 00]. Another method has been proposed based on support vector machines. The idea behind this methodology is to look at the RBF network as a mapping machine, through the kernels, into a high-dimensional space. Then we design a hyperplane classifier using the vectors that are closest to the decision boundary. These are the support vectors and correspond to the centers of the input space. The training consists of a quadratic programming problem and guarantees a global optimum [Scho 97]. The nice feature of this algorithm is that it automatically computes all the unknown parameters including the number of centers. We will return to it later in this chapter. In [Plat 91] an approach similar in spirit to the constructive techniques,discussed for the multilayer perceptrons, has been suggested. The idea is to start training the RBF network with a few nodes (initially one) and keep growing the network by allocating new ones, based on the “novelty” in the feature vectors that arrive sequentially. The novelty of each training input–desired output pair is determined by two conditions: (a) the input vector to be very far (according to a threshold) from all already existing centers and (b) the corresponding output error (using the RBF network trained up to this point) greater than another predetermined threshold. If both conditions are satisfied, then the new input vector is assigned as the new center. If not, the input–desired output pair is used to update the parameters of the network according to the adopted training algorithm, for example, the gradient descent scheme. A variant of this scheme that allows removal of previously assigned centers has also been suggested in [Ying 98]. This is basically a combination of the constructive and pruning philosophies. The procedure suggested in [Kara 97] also moves along the same direction. However, the assignment of the new centers is based on a procedure of progressive splitting (according to a splitting criterion) of the feature space using clustering or learning vector quantization techniques (Chapter 14). The representatives of the regions are then assigned as the centers of the RBF’s. As was the case with the aforementioned techniques, network growing and training is performed concurrently. In [Yang 06] a weight structure is imposed that binds the weights to a specified probability density function, and estimation is achieved in the Bayesian framework rationale. A number of other techniques have also been suggested. For a review see, for example, [Hush 93]. A comparison of RBF networks with different center selection strategies versus multilayer perceptrons in the context of speech recognition is given in [Wett 92]. Reviews involving RBF networks and related applications are given in [Hayk 96, Mulg 96].
4.16 UNIVERSAL APPROXIMATORS In this section we provide the basic guidelines concerning the approximation properties of the nonlinear functions used throughout this chapter—that is, sigmoid,
“06-Ch04-SA272” 18/9/2008 page 195
4.16 Universal Approximators
polynomial, and radial basis functions. The theorems that are stated justify the use of the corresponding networks as decision surface approximators as well as probability function approximators, depending on how we look at the classifier. In (4.51) the polynomial expansion was used to approximate the nonlinear function g(x). This choice for the approximation functions is justified by theWeierstrass theorem. Theorem Let g(x) be a continuous function defined in a compact (closed) subset S ⊂ Rl , and ⑀ ⬎ 0. Then there are an integer r ⫽ r(⑀) and a polynomial function (x) of order r so that |g(x) ⫺ (x)| ⬍ ⑀,
᭙x ∈ S
In other words,function g(x) can be approximated arbitrarily closely for sufficiently large r. A major problem associated with polynomial expansions is that good approximations are usually achieved for large values of r. That is, the convergence to g(x) is slow. In [Barr 93] it is shown that the approximation error is reduced accord1 ) rule, where O(·) denotes order of magnitude. Thus, the error ing to an O( r 2/l decreases more slowly with increasing dimension l of the input space, and large values of r are necessary for a given approximation error. However, large values of r, besides the computational complexity and generalization issues (due to the large number of free parameters required), also lead to poor numerical accuracy behavior in the computations, because of the large number of products involved. On the other hand, the polynomial expansion can be used effectively for piecewise approximation, where smaller r’s can be adopted. The slow decrease of the approximation error with respect to the system order and the input space dimension is common to all expansions of the form (4.47) with fixed basis functions fi (·). The scenario becomes different if data-adaptive functions are chosen,as is the case with the multilayer perceptrons. In the latter,the argument in the activation functions is f (w T x), with w computed in an optimal fashion from the available data. Let us now consider a two-layer perceptron with one hidden layer, having k nodes with activation functions f (·) and an output node with linear activation. The output of the network is then given by (x) ⫽
k
o wjo f (w hT j x) ⫹ wo
(4.59)
j⫽1
where h refers to the weights, including the thresholds, of the hidden layer and o to the weights of the output layer. Provided that f (·) is a squashing function, the following theorem establishes the universal approximation properties of such a network [Cybe 89, Funa 89, Horn 89, Ito 91, Kalo 97]. Theorem Let g(x) be a continuous function defined in a compact subset S ⊂ Rl and ⑀ ⬎ 0. Then there exist k ⫽ k() and a two-layer perceptron (4.59) so that |g(x) ⫺ (x)| ⬍ ⑀, ᭙x ∈ S
195
“06-Ch04-SA272” 18/9/2008 page 196
196
CHAPTER 4 Nonlinear Classifiers
In [Barr 93] is shows that, in contrast to the polynomial expansion, the approximation error decreases according to an O( k1 ) rule. In other words, the input space dimension does not enter explicitly into the scene and the error is inversely proportional to the system order, that is, the number of neurons. Obviously, the price we pay for it is that the optimization process is now nonlinear, with the associated disadvantage of the potential for convergence to local minima. The question that now arises is whether we gain anything by using more than one hidden layer, since a single one is sufficient for the function approximation. An answer is that using more than one layer may lead to a more efficient approximation; that is, the same accuracy is achieved with fewer neurons in the network. The universal approximation property is also true for the class of RBF functions. For sufficiently large values of k in (4.55) the resulting expansion can approximate arbitrarily closely any continuous function in a compact subset S [Park 91, Park 93].
4.17 PROBABILISTIC NEURAL NETWORKS In Section 2.5.6 we have seen that the Parzen estimate of an unknown pdf, using a Gaussian kernel, is given by ˆ p(x| i) ⫽
Ni 1 1 (x ⫺ x i )T (x ⫺ x i ) exp ⫺ l Ni 2h2 l i⫽1 (2) 2 h
(4.60)
where now we have explicitly included in the notation the class dependence, since decisions according to the Bayesian rule rely on the maximum value, with ˆ respect to i , of P(i )p(x| i ). Obviously, in Eq. (4.60) only the training samples, x i , i ⫽ 1, 2, . . . , Ni , of class i are involved. The objective of this section is to develop an efficient architecture for implementing Eq. (4.60), which is inspired by the multilayer NN rationale. The critical computation involving the unknown feature vector, x, in Eq. (4.60) is the inner product norm (x ⫺ x i )T (x ⫺ x i ) ⫽ ||x||2 ⫹ ||x i ||2 ⫺ 2x Ti x
(4.61)
Let us now normalize all the feature vectors, which are involved in the game, to " l 2 unit norm. This is achieved by dividing each vector x by its norm ||x|| ⫽ i⫽1 xi . After normalization, and combining Eqs. (4.61) and (4.60), Bayesian classification now relies on searching for the maximum of the following discriminant functions Ni x Ti x ⫺ 1 P(i ) g(i ) ⫽ exp Ni h2
(4.62)
i⫽1
where the constant multiplicative weights have been omitted. The above can be efficiently implemented by the network of Figure 4.22, when parallel processing resources are available. The input consists of nodes where an unknown feature
“06-Ch04-SA272” 18/9/2008 page 197
4.17 Probabilistic Neural Networks
y1
1
1
x1 xk1 x2
yk k
xk2 k11
xl
yk+1 2
xkl
yN N
FIGURE 4.22 A probabilistic neural network architecture with N training data points. Each node corresponds to a training point, and it is numbered accordingly. Only the synaptic weights for the kth node are drawn. We have assumed that there are two classes and that the first k points originate from class 1 and the rest from class 2 .
vector x ⫽ [x1 , x2 , . . . , xl ]T is applied.
The number of hidden layer nodes is equal of the number of training data, N ⫽ M i⫽1 Ni , where M is the number of classes. In the figure,for the sake of simplicity,we have assumed two classes,although generalizing to more classes is obvious. The synaptic weights, leading to the kth hidden node, consist of the components of the respective normalized training feature vector x k , i.e., xk,j , j ⫽ 1, 2, . . . , l and k ⫽ 1, 2, . . . , N . In other words, the training of this type of network is very simple and is directly dictated by the values of the training points. Hence, the input presented to the activation function of the kth hidden layer node is given by inputk ⫽
l
xk, j xj ⫽ x Tk x
j⫽1
Using as activation function for each node the Gaussian kernel, the output of the kth node is equal to yk ⫽ exp
inputk ⫺ 1 h2
There are M output nodes, one for each class. Output nodes are linear combiners. Each output node is connected to all hidden layer nodes associated with the respective class. The output for the mth output node, m ⫽ 1, 2, . . . , M, will be outputm ⫽
Nm P(m ) yi Nm i⫽1
197
“06-Ch04-SA272” 18/9/2008 page 198
198
CHAPTER 4 Nonlinear Classifiers
where Nm is the number of hidden layer nodes (number of training points) associated with the mth class. The unknown vector is classified according to the class giving the maximum output value. Probabilistic neural network architectures were introduced in [Spec 90], and they have been used in a number of applications, for example, [Rome 97, Stre 94, Rutk 04].
4.18 SUPPORT VECTOR MACHINES: THE NONLINEAR CASE In Chapter 3, we discussed support vector machines (SVM) as an optimal design methodology of a linear classifier. Let us now assume that there exists a mapping x ∈ Rl ⫺→y ∈ Rk
from the input feature space into a k-dimensional space, where the classes can satisfactorily be separated by a hyperplane. Then, in the framework discussed in Section 4.12, the SVM method can be mobilized for the design of the hyperplane classifier in the new k-dimensional space. However, there is an elegant property in the SVM methodology that can be exploited for the development of a more general approach. This will also allow us for (implicit) mappings in infinite dimensional spaces, if required. Recall from Chapter 3 that, in the computations involved in the Wolfe dual representation the feature vectors participate in pairs, via the inner product operation. Also, once the optimal hyperplane (w, w0 ) has been computed, classification is performed according to whether the sign of g(x) ⫽ w T x ⫹ w0 ⫽
Ns
i yi x Ti x ⫹ w0
i⫽1
is ⫹ or ⫺, where Ns is the number of support vectors. Thus, once more, only inner products enter into the scene. If the design is to take place in the new k-dimensional space, the only difference is that the involved vectors will be the k-dimensional mappings of the original input feature vectors. A naive look at it would lead to the conclusion that now the complexity is much higher, since, usually, k is much higher than the input space dimensionality l, in order to make the classes linearly separable. However, there is a nice surprise just waiting for us. Let us start with a simple example. Assume that ⎡
x12
⎤
⎥ ⎢√ ⎥ x ∈ R2 ⫺→y ⫽ ⎢ ⎣ 2x1 x2 ⎦ x22
“06-Ch04-SA272” 18/9/2008 page 199
4.18 Support Vector Machines: The Nonlinear Case
Then, it is a matter of simple algebra to show that 2 y Ti y j ⫽ x Ti x j
In words, the inner product of the vectors in the new (higher dimensional) space has been expressed as a function of the inner product of the corresponding vectors in the original feature space. Most interesting! Theorem Mercer’s Theorem. Let x ∈ Rl and a mapping x → (x) ∈ H
where H is a Hilbert space.5 The inner product operation has an equivalent representation #
$ (x), (z) ⫽ K (x, z)
(4.63)
where ·, · denotes the inner product operation in H and K (x, z) is a symmetric continuous function satisfying the following condition: K (x, z)g(x)g(z) dx dz ⱖ 0 C
(4.64)
C
for any g(x), x ∈ C ⊂ Rl such that
g(x)2 dx ⬍ ⫹⬁
(4.65)
C
where C is a compact ( finite) subset of Rl . The opposite is always true; that is, for any symmetric, continuous function K (x, z) satisfying (4.64) and (4.65) there exists a space in which K(x, z) defines an inner product! Such functions are also known as kernels and the space H as Reproducing kernel Hilbert space (RKHS) (e.g., [Shaw 04, Scho 02]). What Mercer’s theorem does not disclose to us, however, is how to find this space. That is, we do not have a general tool to construct the mapping (·) once we know the inner product of the corresponding space. Furthermore, we lack the means to know the dimensionality of the space, which can even be infinite. This is the case, for example, for the radial basis (Gaussian) kernel ([Burg 99]). For more on these issues, the mathematically inclined reader is referred to [Cour 53]. Typical examples of kernels used in pattern recognition applications are as follows: Polynomials K (x, z) ⫽ (x T z ⫹ 1)q ,
5
q⬎0
(4.66)
A Hilbert space is a complete linear space equipped with an inner product operation. A finite dimensional Hilbert space is a Euclidean space.
199
“06-Ch04-SA272” 18/9/2008 page 200
200
CHAPTER 4 Nonlinear Classifiers
Radial Basis Functions
K (x, z) ⫽ exp ⫺
x ⫺ z 2 2
(4.67)
Hyperbolic Tangent K (x, z) ⫽ tanh x T z ⫹ ␥
(4.68)
for appropriate values of  and ␥ so that Mercer’s conditions are satisfied. One possibility is  ⫽ 2, ␥ ⫽ 1. In [Shaw 04] a unified treatment of kernels is presented, focusing on their mathematical properties as well as methods for pattern recognition and regression that have been developed around them. Once an appropriate kernel has been adopted that implicitly defines a mapping into a higher dimensional space (RKHS), the Wolfe dual optimization task (Eqs. (3.103)–(3.105)) becomes ⎛
max ⎝
i
subject to
⎞ 1 i ⫺ i j yi yj K (x i , x j )⎠ 2
(4.69)
i,j
0 ⱕ i ⱕ C, i ⫽ 1, 2, . . . , N i yi ⫽ 0
(4.70) (4.71)
i
and the resulting linear (in the RKHS) classifier is assign x in 1 (2 ) if g(x) ⫽
Ns
i yi K (x i , x) ⫹ w0 ⬎ (⬍) 0
(4.72)
i⫽1
Due to the nonlinearity of the kernel function, the resulting classifier is a nonlinear one in the original Rl space. Similar arguments hold true for the -SVM formulation. Figure 4.23 shows the corresponding architecture. This is nothing else than a special case of the generalized linear classifier of Figure 4.17. The number of nodes is determined by the number of support vectors Ns . The nodes perform the inner products between the mapping of x and the corresponding mappings of the support vectors in the high-dimensional space, via the kernel operation. Figure 4.24 shows the resulting SVM classifier for two nonlinearly separable classes, where the Gaussian radial basis function kernel, with ⫽ 1.75, has been used. Dotted lines mark the margin and circled points the support vectors. Remarks ■
Notice that if the kernel function is the RBF, then the architecture is the same as the RBF network architecture of Figure 4.17. However, the approach followed here is different. In Section 4.15,a mapping in a k-dimensional space
“06-Ch04-SA272” 18/9/2008 page 201
4.18 Support Vector Machines: The Nonlinear Case
K(x1,x)
x1
1
y
1
2 y
K(x2 ,x)
x2
2
y NS NS
w0
K(xN ,x)
xl
S
FIGURE 4.23 The SVM architecture employing kernel functions.
x2 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0
0
1
2
3
4
5 x1
FIGURE 4.24 Example of a nonlinear SVM classifier for the case of two nonlinearly separable classes. The Gaussian RBF kernel was used. Dotted lines mark the margin and circled points the support vectors.
201
“06-Ch04-SA272” 18/9/2008 page 202
202
CHAPTER 4 Nonlinear Classifiers
was first performed, and the centers of the RBF functions had to be estimated. In the SVM approach,the number of nodes as well as the centers are the result of the optimization procedure. ■
The hyperbolic tangent function is a sigmoid one. If it is chosen as a kernel, the resulting architecture is a special case of a two-layer perceptron. Once more, the number of nodes is the result of the optimization procedure. This is important. Although the SVM architecture is the same as that of a two-layer perceptron, the training procedure is entirely different for the two methods. The same is true for the RBF networks.
■
A notable characteristic of the support vector machines is that the computational complexity is independent of the dimensionality of the kernel space, where the input feature space is mapped. Thus, the curse of dimensionality is bypassed. In other words, one designs in a high-dimensional space without having to adopt explicit models using a large number of parameters, as this would be dictated by the high dimensionality of the space. This also has an influence on the generalization properties, and indeed, SVMs tend to exhibit good generalization performance. We will return to this issue at the end of Chapter 5.
■
A major limitation of the support vector machines is that up to now there has been no efficient practical method for selecting the best kernel function. This is still an unsolved, yet challenging, research issue. Once a kernel function has been adopted, the so-called kernel parameters (e.g., for the Gaussian kernel) as well as the smoothing parameter,C,in the cost function are selected so that the error performance of the resulting classifier can be optimized. Indeed, this set of parameters, also known as hyperparameters, is crucial for the generalization capabilities of the classifier (i.e.,its error performance when it is “confronted” with data outside the training set). To this end, a number of easily computed bounds, which relate to the generalization performance of the classifier, have been proposed and used for the best choice of the hyperparameters. The most common procedure is to solve the SVM task for different sets of hyperparameters and finally select the SVM classifier corresponding to the set optimizing the adopted bound. See, for example, [Bart 02, Lin 02, Duan 03, Angu 03, Lee 04]. [Chap 02] treats this problem in a minimax framework: maximize the margin over the w and minimize the bound over the hyperparameters. A different approach to the task of data-adaptive kernel tuning, with the same goal of improving the error performance,is to use information geometry arguments [Amar 99]. The basic idea behind this approach is to introduce a conformal mapping into the Riemannian geometry induced by the chosen kernel function, aiming at enhancing the margin. [Burg 99] points out that the feature vectors, which originally lie in the l-dimensional space, after the mapping induced by the kernel function lie in an l-dimensional surface, S,
“06-Ch04-SA272” 18/9/2008 page 203
4.19 Beyond the SVM Paradigm
in the high-dimensional space. It turns out that (under some very general assumptions) S is a Riemannian manifold with a metric that can be expressed solely in terms of the kernel. ■
Support vector machines have been applied to a number of diverse applications, ranging from handwritten digit recognition ([Cort 95]), to object recognition ([Blan 96]), person identification ([Ben 99]), spam categorization ([Druc 99]), channel equalization ([Seba 00]), and medical imaging [ElNa 02, Flao 06]. The results from these applications indicate that SVM classifiers exhibit enhanced generalization performance, which seems to be the power of support vector machines. An extensive comparative study concerning the performance of SVM against sixteen other popular classifiers, using twenty-one different data sets, is given in [Meye 03]. The results verify that SVM classifiers rank at the very top among these classifiers, although there are cases for which other classifiers gave lower error rates.
4.19 BEYOND THE SVM PARADIGM One of the most attractive properties of the support vector machines, which has contributed to their popularity, is that their computational structure allows for the use of a kernel function, as discussed in the previous section. Sometimes this is also known as the kernel trick. This powerful tool makes the design of a linear classifier in the high-dimensional space independent of the dimensionality of this space. Moreover, due to the implicit nonlinear mapping, dictated by the adopted kernel function, the designed classifier is a nonlinear one in the original space. The success of the SVMs in practice has inspired a research effort to extend a number of linear classifiers to nonlinear ones, by embedding the kernel trick in their structure. This is possible if all the computations can be expressed in terms of inner product operations. Let us illustrate the procedure for the case of the classical Euclidean distance classifier. Assume two classes 1 and 2 ,with N1 and N2 training pairs,(yi , x i ),respectively, with yi ⫽⫾1 being the class label of the ith sample. Let K (·, ·) be a kernel function associated with an implicit mapping x → (x) from the original Rl space to a high-dimensional RKHS. Given an unknown x, the Euclidean classifier, in the RKHS, classifies it to the 2 class if ||(x) ⫺ 1 ||2 ⬎ ||(x) ⫺ 2 ||2
(4.73)
or, after some basic algebra, if $ 1 ||2 ||2 ⫺ ||1 ||2 ≡ (x), (2 ⫺ 1 ) ⬎ 2
#
(4.74)
where ·, · denotes the inner product operation in the RKHS and 1 ⫽
1 N1
i : yi ⫽⫹1
(x i )
and
2 ⫽
1 N2
i : yi ⫽⫺1
(x i )
(4.75)
203
“06-Ch04-SA272” 18/9/2008 page 204
204
CHAPTER 4 Nonlinear Classifiers
Combining Eqs. (4.74) and (4.75), we conclude that we assign x in 2 if 1 N2
K (x, x i ) ⫺
i : yi ⫽⫺1
1 N1
K (x, x i ) ⬎
(4.76)
i : yi ⫽⫹1
where 2 ⫽
1 N22
K (x i , x j ) ⫺
i : yi ⫽⫺1 j : yj ⫽⫺1
1 N12
K (x i , x j )
i : yi ⫽⫹1 j : yj ⫽⫹1
The left-hand side in formula (4.76) reminds us of the Parzen pdf estimate. Adopting the Gaussian kernel, the first term can be taken as the pdf estimator associated with the class 2 and the second one with the 1 one. Besides the Euclidean classifier, other classical cases, including Fisher’s linear discriminant (Chapter 5), have been extended to nonlinear ones by employing the kernel trick,see,for example,[Mull 01, Shaw 04]. Another notable and pedagogically attractive example of a “kernelized” version of a linear classifier is the kernel perceptron algorithm. The perceptron rule was introduced in Section 3.3. There it was stated that the perceptron algorithm converges in a finite number of steps, provided that the two classes are linearly separable. This drawback has prevented the perceptron algorithm to be used in realistic practical applications. However, after mapping the original feature space to a high-dimensional (even of infinite dimensionality) space and utilizing Cover’s theorem (Section 4.13), one expects the classification task to be linearly separable, with high probability, in the RKHS space. In this perspective, the kernelized version of the perceptron rule transcends its historical, theoretical, and educational role and asserts a more practical value as a candidate for solving linearly separable tasks in the RKHS. We will choose to work on the perceptron algorithm in its reward and punishment form, given in Eqs. (3.21)–(3.23). The heart of the method is the update given by the Eqs. (3.21), and (3.22). These recursions, take place in the extended RKHS (its dimension is increased by one to account for the bias term), and they are compactly written as
w(t ⫹ 1) w0 (t ⫹ 1)
⫽
w(t) w0 (t)
⫹ y(t)
(x (t) ) 1
# $ each time a misclassification occurs—that is, if y(t) ( w(t), (x (t) ) ⫹ w0 ) ⱕ 0, where the coefficient has been taken to be equal to one. Let ␣i , i ⫽ 1, 2, . . . , N , be a counter corresponding to each one of the training points. The counter ␣i is increased by one every time x (t) ⫽ x i , and a misclassification occurs leading to a respective update of the classifier. If one starts from a zero initial vector, then the solution, after all points have been correctly classified, can be written as w⫽
N i⫽1
␣i yi (x i ), w0 ⫽
N i⫽1
␣i yi
“06-Ch04-SA272” 18/9/2008 page 205
4.19 Beyond the SVM Paradigm
The final resulting nonlinear classifier, in the original feature space, then becomes N N # $ g(x) ≡ w, (x) ⫹ w0 ⫽ ␣i yi K (x, x i ) ⫹ ␣i yi i⫽1
i⫽1
A pseudocode for the kernel perceptron algorithm follows. The Kernel Perceptron Algorithm ■
Set ␣i ⫽ 0, i ⫽ 1, 2, . . . , N
■
Repeat • count_misclas ⫽ 0 • For i ⫽ 1 to N
N N If y i j⫽1 ␣j yj K (x i , x j ) ⫹ j⫽1 ␣j yj ⱕ 0 then — ␣i ⫽ ␣i ⫹ 1 — count_misclas ⫽ count_misclas ⫹ 1 • End {For}
■
Until count_misclas ⫽ 0
4.19.1 Expansion in Kernel Functions and Model Sparsification In this subsection, we will briefly discuss classifiers that resemble to (or, even are inspired by) the SVMs, in an effort to establish bridges among different methodologies. We have already done so in Section 4.18 for the SVM, RBF, and multilayer neural networks. After all,it is a small world! Using the Gaussian kernel in Eq. (4.72), we obtain g(x) ⫽
Ns i⫽1
||x ⫺ x i ||2 ai exp ⫺ 2 2
⫹ w0
(4.77)
where we have used ai ⫽ i yi . Equation (4.77 ) is very similar to the Parzen expansion of a pdf, discussed in Chapter 2. There are a few differences, however. In contrast to the Parzen expansion, g(x) in (4.77 ) is not a pdf function; that is, in general, it does not integrate to unity. Moreover, from the practical point of view, the most important difference lies in the different number of terms involved in the summation. In the Parzen expansion all the training samples offer their contribution to the final solution. In contrast, in the solution provided by the SVM formulation only the support vectors, that is, the points lying either in the margin or in the wrong side of the resulting classifier, are assigned as the “significant” ones and are selected to contribute to the solution. In practice, a small fraction of the training points enter in the summation in Eq. (4.77), that is Ns ⬍⬍ N . In fact, as we
205
“06-Ch04-SA272” 18/9/2008 page 206
206
CHAPTER 4 Nonlinear Classifiers
will discuss at the end of Section 5.10, if the number of support vectors gets large, the generalization performance of the classifier is expected to degrade. If Ns ⬍⬍N , we say that the solution is sparse. A sparse solution spends computational resources only on the most relevant of the training patterns. Besides the computational complexity aspects,having a sparse solution is in line with our desire to avoid overfitting (see also Section 4.9). In real-world data, the presence of noise in regression and the overlap of classes in classification, as well as the presence of outliers, imply that the modeling must be such that to avoid overfitting to the specific training data set. A closer look behind the SVM philosophy reveals that the source of sparsity in the obtained solution is the presence of the margin term in the cost function. Another way to view the term w 2 in the cost function in (3.93), that is, 1 w 2 ⫹ C I (i ) 2 N
J (w, w0 , ) ⫽
i⫽1
is as a regularization term, whose presence satisfies our will to keep the norm of the solution as “small” and “simple” as possible, while, at the same time, trying to N I ( ) minimize the number of margin errors i . This implicitly forces most of i⫽1 the i s in the solution to be zero, keeping only the most significant of the samples, that is, the support vectors. In Section 4.9, regularization was also used in order to keep the size of the neural networks small. For a deeper and an elegant discussion of the use of regularization in the context of regression/classification the interested reader can refer to [Vapn 00]. With the sparsification goal in mind, a major effort has been invested to develop techniques, both for classification and for regression tasks, which lead to classifiers/regressors of the form g(x) ⫽
N
aj K (x, x j )
(4.78)
j⫽1
for an appropriately chosen kernel function. A bias constant term can also be added, but it has been omitted for simplicity. The task is to estimate the unknown weights aj , j ⫽ 1, 2 . . . , N , of the expansion. Functions of the form in (4.78) are justified by the following theorem ([Kime 71, Scho 02]): Representer Theorem Let L(·, ·) : R2 → [0, ⬁) be an arbitrary nonnegative loss function, measuring the deviation between a desired response, y, and the value of g(x). Then the minimizer g(·) ∈ H, where H is a RKHS defined by a kernel function K (·, ·), of the regularized cost N i⫽1
L(g(x i ), yi ) ⫹ ⍀(||g||)
(4.79)
“06-Ch04-SA272” 18/9/2008 page 207
4.19 Beyond the SVM Paradigm
admits a representation of the form in (4.78). In (4.79), (yi , x i ), i ⫽ 1, 2, . . . , N , are the training data, ⍀(·) : [0, ⬁) → R is a strictly monotonic increasing function and || · || is the norm operation in H. For a more mathematical treatment of this result, the interested reader may refer to, for example, [Scho 02]. For those who are not familiar with functional analysis and some of the mathematical secrets behind RKHS, recall that the set of functions Rl → R form a linear space, which can be equipped with an inner product operation to become a Hilbert space. Hence, by restricting g(·) ∈ H, we limit our search for solutions in (4.79) among the points in an RKHS (function space) defined by the specific kernel function. This is an important theorem because it states that, although working in a high- (even infinite) dimensional space, the optimal solution, minimizing (4.79), is expressed as a linear combination of only N kernels placed at the training points! In order to see how this theorem can simplify the search for the optimal solution in practice, let us consider the following example. Example 4.1 The kernel least squares solution. Let (yi , x i ), i ⫽ 1, 2, . . . , N , be the training points. The goal is to design the optimal linear least squares classifier (regressor) in a RKHS space, which is defined by the kernel function K (·, ·). According to the definition of the least squares cost in Section 3.4.3, we have to minimize, with respect to g ∈ H, the cost N
yi ⫺ g(x i )
2
(4.80)
i⫽1
According to the Representer Theorem, we can write g(x) ⫽
N
aj K (x, x j )
(4.81)
j⫽1
Substituting (4.81) into (4.80), we get the equivalent task of minimizing with respect to a finite number of parameters, ai , i ⫽ 1, 2, . . . , N , the cost ⎞2 ⎛ N N ⎝yi ⫺ aj K (x i , x j )⎠ (4.82) J (a) ⫽ i⫽1
j⫽1
The cost in (4.82) can be written in terms of the Euclidean norm in the RN space, that is, J (a) ⫽ ( y ⫺ Ka)T ( y ⫺ Ka) where y ⫽ [ y1 , y2 , . . . , yN defined as
]T
(4.83)
and K is the N ⫻ N matrix known as the Gram matrix, which is K(i, j) ≡ K (x i , x j )
(4.84)
Expanding (4.83) and taking the gradient with respect to a to be equal to zero, we obtain a ⫽ K⫺1 y
(4.85)
207
“06-Ch04-SA272” 18/9/2008 page 208
208
CHAPTER 4 Nonlinear Classifiers
provided that the Gram matrix is invertible. Hence, recalling (4.81), the kernel least squares estimate can be written compactly as g(x) ⫽ a T p ⫽ y T K⫺1 p
(4.86)
p ≡ [K (x, x 1 ), . . . , K (x, x N )]T
(4.87)
where
The Representer Theorem has been exploited in [Tipp 01] in the context of the so-called relevance vector machine (RVM) methodology. Based on (4.78), a conditional probability model is built for the desired response (label) given the values of a. The computation of the unknown weights is carried out in the Bayesian framework rationale (Chapter 2). Sparsification is achieved by constraining the unknown weight parameters and imposing an explicit prior probability distribution over each one of them. It is reported that RVMs lead to sparser solutions compared to the SVMs, albeit at a higher complexity. Memory requirements scale with the square, and the required computational resources scale with the cube of the number of basis functions, which makes the algorithm less practical for large data sets. In contrast,the amount of memory requirements for the SVMs is linear,and the number of computations is somewhere between linear and (approximately) quadratic in the size of the training set ([Plat 99]). A more recent trend is to obtain solutions of the form in Eq. (4.78) in an online time-adaptive fashion. That is, the solution is updated each time a new training pair (yi , x i ) is received. This is most important when the statistics of the involved data are slowly time varying. To this end, in [Kivi 04] a kernelized online LMS-type algorithm (see Section 3.4.2) is derived that minimizes the cost function J ( gt ) ⫽
t
L(gt (x i ), yi ) ⫹ ||gt ||2
(4.88)
i⫽1
where the index t has been used in gt to denote the time dependence explicitly. L(·, ·) is a loss function that quantifies the deviation between the desired output value, yi , and the true one that is provided by the current estimate, gt (·), of the unknown function. The summation accounts for the total number of errors committed on all the samples that have been received up to the time instant t. Sparsification is achieved by regularizing the cost function by the square norm ||gt ||2 of the required solution. Another way to look at the regularization term and better understand how its presence beneficially affects the sparsification process is the following. Instead of minimizing, for example, (4.88) one can choose to work with an alternative formulation of the optimization task, that is, minimize
t
L(gt (x i ), yi )
(4.89)
i⫽1
subject to
||gt ||2 ⱕ s
(4.90)
“06-Ch04-SA272” 18/9/2008 page 209
4.19 Beyond the SVM Paradigm
The use of Lagrange multipliers leads to minimizing J (gt ) in (4.88). It can be shown (see, e.g., [Vapn 00]) that the two problems are equivalent for appropriate choices of the parameters s and . However, formulating the optimization task as in (4.89)–(4.90) makes our desire for constraining the size of the solution explicitly stated. In [Slav 08] an adaptive solution of the cost in Eq. (4.78) is given based on projections and convex set arguments. Sparsification is achieved by constraining the solution to lie within a (hyper)sphere in the RKHS. It is shown that such a constraint becomes equivalent to imposing a forgetting factor that forces the algorithm to forget data in the remote past and adaptation focuses on the most recent samples. The algorithm scales linearly with the number of data corresponding to its effective memory (due to the forgetting factor). An interesting feature of this algorithmic scheme is that it provides as special cases a number of well-known algorithms, such as the kernelized normalized LMS (NLMS) [Saye 04] and the kernelized affine projection [Slav 08a] algorithms. Another welcome feature of this methodology is that it can accommodate differentiable as well as nondifferentiable cost functions,in a unified way,due to the possibility of employing subdifferentials of the cost function, in place of the gradient in the correction term, in each time-update recursion. A different root to the online sparsification is followed in [Enge 04, Slav 08a]. A dictionary of basis functions is adaptively formed. For each received sample, its dependence on the samples that are already contained in the dictionary is tested, according to a predefined criterion. If the dependence measure is below a threshold value, the new sample is included in the dictionary whose cardinality is now increasing by one; otherwise the dictionary remains unaltered. It is shown that the size of the dictionary does not increase indefinitely and that it remains finite. The expansion of the solution is carried out by using only the basis functions associated with the samples in the dictionary. A pitfall of this technique is that the complexity scales with the square of the size of the dictionary, as opposed to the linear complexity of the two previous adaptive techniques. In our discussion so far we have assumed the use of a loss function. The choice of the loss function is user-dependent. Some typical choices that can be used and have frequently been adopted in classification tasks are as follows: ■
Soft margin loss L(g(x), y) ⫽ max(0, ⫺ yg(x))
where defines the margin parameter. In words, a (margin) error is committed if yg(x) cannot achieve a value of at least . For smaller values, the loss function becomes positive, and it is also linearly increasing as the value of yg(x) becomes smaller moving toward negative values. That is, it provides a measure of how far from the margin the estimate lies. Figure 4.25 shows the respective graph for ⫽ 0.
209
“06-Ch04-SA272” 18/9/2008 page 210
CHAPTER 4 Nonlinear Classifiers
■
Exponential loss L(g(x), y) ⫽ exp ⫺yg(x)
As shown in Figure 4.25,this loss function penalizes heavily nonpositive values of yg(x), which lead to wrong decisions. We will use this loss function very soon in Section 4.22. ■
Logistic loss L(g(x), y) ⫽ ln 1 ⫹ exp(⫺yg(x))
The logistic loss function is basically the negative log-likelihood of a logisticlike probabilistic model, discussed in Section 3.6, operating in the RKHS. g(x) as a linear function in the RKHS, that is, g(x) ⫽ #Indeed, interpreting $ w, (x) ⫹ w0 , and modeling the probability of the class label as P(y|x) ⫽
1 # $ 1 ⫹ exp ⫺y(w0 ⫹ w, (x) )
then the logistic loss is the respective negative log-likelihood function. This loss function has also been used in the context of support vector machines, see [Keer 05].
Soft margin Exponential Logistic Classification error
Loss
210
1 y 2 g(x)
FIGURE 4.25 Typical loss functions used in classification tasks. The margin parameter for the soft margin loss has been set equal to 0. The logistic loss has been normalized in order to pass through the [1,0] point, to facilitate the comparison among the different loss functions. The classification error loss function scores a one if an error is committed and zero otherwise.
“06-Ch04-SA272” 18/9/2008 page 211
4.19 Beyond the SVM Paradigm
4.19.2 Robust Statistics Regression The regression task was introduced in Section 3.5.1. Let y ∈ R, x ∈ Rl be two statistically dependent random entities. Given a set of training samples ( yi , x i ), the goal is to compute a function g(x) that optimally estimates the value of y when x is measured. In a number of cases, mean square or least squares type of costs are not the most appropriate ones. For example, in cases where the statistical distribution of the data has long tails, then using the least squares criterion will lead to a solution dominated by a small number of points that have very large values (outliers). A similar situation can occur from incorrectly labeled data. Take, for example,a single training data point whose target value has been incorrectly labeled by a large amount. This point will have an unjustifiably ( by the true statistics of the data) strong influence on the solution. Such situations can be handled more efficiently by using alternative cost functions, which are known as robust statistics loss functions. Typical examples of such loss functions are: ■
Linear ⑀-insensitive loss L( g(x), y) ⫽ | y ⫺ g(x)|⑀ ≡ max(0, | y ⫺ g(x)| ⫺ ⑀)
■
Quadratic ⑀-insensitive loss L( g(x), y) ⫽ | y ⫺ g(x)|2⑀ ≡ max(0, | y ⫺ g(x)|2 ⫺ ⑀)
■
Huber loss L( g(x), y) ⫽
c | y ⫺ g(x)| ⫺ 2 1 2 y ⫺ g(x)
c2 2
if | y ⫺ g(x)| ⬎ c if | y ⫺ g(x)| ⱕ c
where ⑀ and c are user-defined parameters. Huber’s loss function reduces from quadratic to linear the contributions of samples with absolute error values greater than c. Such a choice makes the optimization task less sensitive to outliers. Figure 4.26 shows the curves associated with the previous loss functions. In the sequel, we will focus on the linear ⑀-insensitive loss. We are by now experienced enough to solve for the nonlinear g(x) case by expressing the problem as a linear one in an RKHS. For the linear ⑀-insensitive case, nonzero contributions to the cost have samples with error values | y ⫺ g(x)| larger than ⑀. This setup can be compactly expressed by adopting two slack variables, , ∗ , and the optimization task is now cast as minimize
N N 1 2 ∗ J (w, w0 , , ) ⫽ w ⫹ C i ⫹ i 2 ∗
i⫽1
subject to
# $ yi ⫺ w, (x i ) ⫺ w0 ⱕ ⑀ ⫹ i∗ ,
(4.91)
i⫽1
i ⫽ 1, 2, . . . , N
(4.92)
211
“06-Ch04-SA272” 18/9/2008 page 212
CHAPTER 4 Nonlinear Classifiers
Linear e-insensitive loss Quadratic e-insensitive loss Huber loss
Loss
212
Squared error
21
1
y2g(x)
FIGURE 4.26 Loss functions used for regression tasks. The parameters ⑀ and c have been set equal to one. In Huber’s loss, observe the change from quadratic to linear beyond ⫾ c.
$ w, (x i ) ⫹ w0 ⫺ yi ⱕ ⑀ ⫹ i ,
#
i ≥ 0,
i∗ ≥ 0,
i ⫽ 1, 2, . . . , N
i ⫽ 1, 2, . . . , N
(4.93) (4.94)
# $ The above setup guarantees that i , i∗ are zero if |yi ⫺# w, (x i )$ ⫺ w0 | ⱕ ⑀ and contribution # $ to the cost function occurs if either yi ⫺ w, (x i ) ⫺ w0 ⬎ ⑀ or if yi ⫺ w, (x i ) ⫺w0 ⬍⫺⑀. The presence of the norm ||w|| guards against overfitting, as has already been discussed. Following similar arguments made in Section 3.7.2, it turns out that the solution is given by w⫽
N (∗i ⫺ i )(x i )
(4.95)
i⫽1
where ∗i , i are the Lagrange multipliers associated with the set of constraints in (4.92)–(4.93), respectively. The corresponding KKT conditions are (in analogy of (3.98)–(3.102)) # $ ∗i ( yi ⫺ w, (x i ) ⫺ w0 ⫺ ⑀ ⫺ i∗ ) ⫽ 0, # $ i ( w, (x i ) ⫹ w0 ⫺ yi ⫺ ⑀ ⫺ i ) ⫽ 0, C ⫺ i ⫺ i ⫽ 0, C ⫺ ∗i ⫺ i∗ ⫽ 0,
i ⫽ 1, 2, . . . , N i ⫽ 1, 2, . . . , N
i ⫽ 1, 2, . . . , N
(4.96) (4.97) (4.98)
“06-Ch04-SA272” 18/9/2008 page 213
4.19 Beyond the SVM Paradigm
i i ⫽ 0, i∗ i∗ ⫽ 0,
i ⫽ 1, 2, . . . , N
i ≥ 0, ∗i ≥ 0, i ≥ 0, i∗ ≥ 0, N i⫽1
i ⫽
N
(4.99)
i ⫽ 1, 2, . . . , N
∗i
(4.100) (4.101)
i⫽1
i i∗ ⫽ 0, i ∗i ⫽ 0,
i ⫽ 1, 2, . . . , N
(4.102)
where i∗ , i are the Lagrange multipliers associated with the set of constraints in (4.94). Note that i , i∗ cannot be nonzero simultaneously, and the same applies for the Lagrange multipliers ∗i , i . Furthermore, a careful look at the KKT conditions reveals that: # $ ■ The points with absolute error values strictly less than ⑀,i.e.,| yi ⫺ w, (x i ) ⫺ w0 | ⬍ ⑀ result in zero Lagrange multipliers,i , ∗i . This is a direct consequence of (4.96) and (4.97). These points are the counterparts of the points that lie strictly outside the margin in the SVM classification task. # $ ■ Support vectors are those points satisfying the inequality | yi ⫺ w, (x i ) ⫺ w0 | ≥ ⑀. ■
The points $ associated with errors satisfying the strict inequality | yi ⫺ # w, (x i ) ⫺ w0 | ⬎ ⑀ result in either i ⫽ C or ∗i ⫽ C. This is a consequence of (4.99) and (4.98) and of the fact that, in this case,#i (or i∗ )$is nonzero. For those of the points that equality holds, that is, |yi ⫺ w, (x i ) ⫺ w0 | ⫽ ⑀, the respective i (i∗ ) ⫽ 0 and from (4.97) (or (4.96)) the respective i (∗i ) can be nonzero. Then from (4.99),(4.100),and (4.98) it turns out that 0 ⱕ i (∗i ) ⱕ C.
The Lagrange multipliers can be obtained by writing the problem in its equivalent dual representation form, that is, maximize
N
yi (∗i ⫺ i ) ⫺ ⑀
i⫽1
N (∗i ⫹ i ) ⫺ i⫽1
# $ 1 ∗ (i ⫺ i )(∗j ⫺ j ) (x i ), (x j ) 2
(4.103)
i, j
subject to
0 ⱕ i ⱕ C, N i⫽1
∗i ⫽
N
0 ⱕ ∗i ⱕ C, i
i ⫽ 1, 2, . . . , N
(4.104) (4.105)
i⫽1
where maximization is with respect to the Lagrange multipliers i , ∗i , i ⫽ 1, 2, . . . , N . This optimization task is similar to the problem defined by (3.103) and (3.105 ).
213
“06-Ch04-SA272” 18/9/2008 page 214
214
CHAPTER 4 Nonlinear Classifiers
Once the Lagrange multipliers have been computed, the nonlinear regressor is obtained as )
* N N ∗ g(x) ≡ (x), (i ⫺ i )(x i ) ⫹ w0 ⫽ (∗i ⫺ i )K (x, x i ) ⫹ w0 i⫽1
i⫽1
where w0 is computed from the KKT conditions in (4.97) and (4.96) for 0 ⬍ i ⬍ C, 0 ⬍ ∗i ⬍ C. Sparsification is achieved via the points associated with zero Lagrange multipliers, that is, points resulting in absolute error values strictly less than ⑀. If instead of the linear ⑀-insensitive loss one adopts the quadratic ⑀-insensitive loss or the Huber loss functions, the resulting sets of formulas are similar to the ones derived here, see, for example, [Vapn 00]. Throughout the derivations in this subsection, we kept referring to the optimization of the SVM classification task considered in Section 3.7.2. The similarity is not accidental. Indeed, it is a matter of a few simple arithmetic manipulations to see that if we set ⑀ ⫽ 0 and yi ⫽⫾ 1, depending on the class origin, our regression task becomes the same as the problem considered in Section 3.7.2.
Ridge Regression We will close this section by establishing the connection of the regression task, which was considered before, with the classical regression problem known as ridge regression. This concept has been used extensively in statistical learning and has been rediscovered under different names. If in the quadratic ⑀-insensitive loss we set ⑀ ⫽ 0 and, for simplicity, w0 ⫽ 0, the result is the standard sum of squared errors cost function. Substituting in the associated constraints the inequalities with equalities and slightly rephrasing the cost (to bring it in its classical formulation), we end up with the following minimize
J (w, ) ⫽ C w 2 ⫹
N
i2
(4.106)
i ⫽ 1, 2, . . . , N
(4.107)
i⫽1
subject to
# $ yi ⫺ w, (x i ) ⫽ i ,
1 where C ⫽ 2C . The task defined in (4.106)–(4.107) is a regularized version of the least squares cost function expressed in an RKHS. If we work on the dual Wolfe representation, it turns out that the solution of the kernel ridge regression is expressed in closed form (see Problem 4.25), that is,
w⫽
N 1 i (x i ) 2C
(4.108)
i⫽1
[1 , . . . , N ]T ⫽ 2C (K ⫹ CI )⫺1 y
(4.109)
“06-Ch04-SA272” 18/9/2008 page 215
4.20 Decision Trees
and
# $ g(x) ≡ w, (x) ⫽ y T (K ⫹ CI )⫺1 p
(4.110)
where I is the N ⫻ N identity matrix and K is the N ⫻ N Gram matrix, defined in (4.84) and p the N -dimensional vector defined in (4.87). Observe that the only difference from the kernel least squares solution is the presence of the CI factor. An advantage of the (kernel) ridge regression, compared to the robust statistics regression, is that a neat closed form solution results. However, by having adopted ⑀ ⫽ 0 we have lost in model sparseness. As we have already pointed out for the case of the linear ⑀-insensitive loss (the same is true for the quadratic version), training points that result in error with absolute value strictly less than ⑀ do not contribute in the solution. There is no free lunch in real life! To establish another bridge with Chapter 3, let us employ the linear kernel, that is, K (x i , x j ) ⫽ x Ti x j (which implies that one works in the input low-dimensional space and no mapping in a high-dimensional RKHS is performed), and solve the primal instead of the dual task ridge regression task. It is easy to show (Problem 4.26) that the solution becomes w ⫽ (X T X ⫹ CI )⫺1 X T y
(4.111)
In other words, the solution is the same as the least squares error solution, given in (3.45). The only difference lies in the presence of the CI factor. The latter is the result of the regularization term in the minimized cost function (4.106). In practice, the term CI is used in the LS solution in cases where X T X has a small determinant and matrix inversion problems arise. Adding a small positive value across the diagonal acts beneficially from the numerical stability point of view. As a last touch on this section, let us comment on (4.111) and (4.108)–(4.109). For the linear kernel case,the Gram matrix becomes XX T ,and the solution resulting from the dual formulation is given by w ⫽ X T (XX T ⫹ CI )⫺1 y
(4.112)
Since this is a convex programming task,both solutions,in (4.111) and (4.112),must be the same. This can be verified by simple algebra ( Problem 4.27).
4.20 DECISION TREES In this section we briefly review a large class of nonlinear classifiers known as decision trees. These are multistage decision systems in which classes are sequentially rejected until we reach a finally accepted class. To this end, the feature space is split into unique regions, corresponding to the classes, in a sequential manner. Upon the arrival of a feature vector, the searching of the region to which the feature vector will be assigned is achieved via a sequence of decisions along a path of
215
“06-Ch04-SA272” 18/9/2008 page 216
216
CHAPTER 4 Nonlinear Classifiers
x2 3 2
3 4
1
1 2
1
1 4
3
1 4
1 2
4
3 4
x1
FIGURE 4.27 Decision tree partition of the space.
nodes of an appropriately constructed tree. Such schemes offer advantages when a large number of classes are involved. The most popular decision trees are those that split the space into hyperrectangles with sides parallel to the axes. The sequence of decisions is applied to individual features, and the questions to be answered are of the form “is feature xi ⱕ ␣?” where ␣ is a threshold value. Such trees are known as ordinary binary classification trees (OBCTs). Other types of trees are also possible that split the space into convex polyhedral cells or into pieces of spheres. The basic idea behind an OBCT is demonstrated via the simplified example of Figure 4.27. By a successive sequential splitting of the space, we have created regions corresponding to the various classes. Figure 4.28 shows the respective binary tree with its decision nodes and leaves. Note that it is possible to reach a decision without having tested all the available features. The task illustrated in Figure 4.27 is a simple one in the two-dimensional space. The thresholds used for the binary splits at each node of the tree in Figure 4.28 were dictated by a simple observation of the geometry of the problem. However,this is not possible in higher dimensional spaces. Furthermore, we started the queries by testing x1 against 14 . An obvious question is why to consider x1 first and not another feature. In the general case, in order to develop a binary decision tree, the designer has to consider the following design elements in the training phase:
“06-Ch04-SA272” 18/9/2008 page 217
4.20 Decision Trees
t0 no
x1 >
1 4
yes
1
t1 x2 >
no
1 2
yes
t3
t2 no 3
x1 > 3 4
yes 4
no
x1 >
1 2
yes
2
t4 no
x2 >
3 4
yes
1
3
FIGURE 4.28 Decision tree classification for the case of Figure 4.27.
■
At each node, the set of candidate questions to be asked has to be decided. Each question corresponds to a specific binary split into two descendant nodes. Each node, t, is associated with a specific subset Xt of the training set X. Splitting of a node is equivalent to the split of the subset Xt into two disjoint descendant subsets, XtY , XtN . The first of the two consists of the vectors in Xt that correspond to the answer “Yes” of the question and those of the second to the “No.” The first (root) node of the tree is associated with the training set X. For every split, the following is true: XtY XtY
+ ,
XtN ⫽ ∅ XtN ⫽ Xt
■
A splitting criterion must be adopted according to which the best split from the set of candidate ones is chosen.
■
A stop-splitting rule is required that controls the growth of the tree,and a node is declared as a terminal one (leaf ).
■
A rule is required that assigns each leaf to a specific class.
We are now experienced enough to understand that more than one method can be used to approach each of the above design elements.
217
“06-Ch04-SA272” 18/9/2008 page 218
218
CHAPTER 4 Nonlinear Classifiers
4.20.1 Set of Questions For the OBCT type of trees, the questions are of the form “Is xk ⱕ ␣?” For each feature, every possible value of the threshold ␣ defines a specific split of the subset Xt . Thus in theory, an infinite set of questions has to be asked if ␣ varies in an interval Y␣ ⊆ R. In practice, only a finite set of questions can be considered. For example, since the number, N , of training points in X is finite, any of the features xk , k ⫽ 1, . . . , l, can take at most Nt ⱕ N different values, where Nt is the cardinality of the subset Xt ⊆ X. Thus, for feature xk , one can use ␣kn , n ⫽ 1, 2, . . . , Ntk (Ntk ⱕ Nt ), where ␣kn are taken halfway between consecutive distinct values of xk in the training subset Xt . The same has to be repeated
for all features. Thus in such a case, the total number of candidate questions is lk⫽1 Ntk . However, only one of them has to be chosen to provide the binary split at the current node, t, of the tree. This is selected to be the one that leads to the best split of the associated subset Xt . The best split is decided according to a splitting criterion.
4.20.2 Splitting Criterion Every binary split of a node, t, generates two descendant nodes. Let us denote them by tY and tN according to the “Yes” or “No” answer to the single question adopted for the node t, also referred as the ancestor node. As we have already mentioned, the descendant nodes are associated with two new subsets, that is, XtY , XtN , respectively. In order for the tree growing methodology, from the root node down to the leaves, to make sense, every split must generate subsets that are more “class homogeneous” compared to the ancestor’s subset Xt . This means that the training feature vectors in each one of the new subsets show a higher preference for specific class(es), whereas data in Xt are more equally distributed among the classes. As an example, let us consider a four-class task and assume that the vectors in subset Xt are distributed among the classes with equal probability (percentage). If one splits the node so that the points that belong to 1 , 2 classes form the XtY subset, and the points from 3 , 4 classes form the XtN subset, then the new subsets are more homogeneous compared to Xt or “purer” in the decision tree terminology. The goal, therefore, is to define a measure that quantifies node impurity and split the node so that the overall impurity of the descendant nodes is optimally decreased with respect to the ancestor node’s impurity. Let P(i |t) denote the probability that a vector in the subset Xt , associated with a node t, belongs to class i , i ⫽ 1, 2, . . . , M. A commonly used definition of node impurity, denoted as I (t), is given by I (t) ⫽ ⫺
M
P(i |t) log2 P(i |t)
(4.113)
i⫽1
where log2 is the logarithm with base 2. This is nothing else than the entropy associated with the subset Xt , known from Shannon’s Information Theory. It is not difficult to show that I (t) takes its maximum value if all probabilities are equal to
“06-Ch04-SA272” 18/9/2008 page 219
4.20 Decision Trees
(highest impurity) and it becomes zero (recall that 0 log 0 ⫽ 0) if all data belong to a single class, that is, if only one of the P(i |t) ⫽ 1 and all the others are zero (least impurity). In practice, probabilities are estimated by the respective percentages, Nti /Nt , where Nti is the number of points in Xt that belong to class i . Assume now that performing a split, NtY points are sent into the “Yes” node (XtY ) and NtN into the “No” node (XtN ). The decrease in node impurity is defined as 1 M
⌬ I (t) ⫽ I (t) ⫺
NtY NtN I (tY ) ⫺ I (tN ) Nt Nt
(4.114)
where I (tY ), I (tN ) are the impurities of the tY and tN nodes, respectively. The goal now becomes to adopt, from the set of candidate questions, the one that performs the split leading to the highest decrease of impurity.
4.20.3 Stop-Splitting Rule The natural question that now arises is when one decides to stop splitting a node and declares it as a leaf of the tree. A possibility is to adopt a threshold T and stop splitting if the maximum value of ⌬ I (t), over all possible splits, is less than T . Other alternatives are to stop splitting either if the cardinality of the subset Xt is small enough or if Xt is pure, in the sense that all points in it belong to a single class.
4.20.4 Class Assignment Rule Once a node is declared to be a leaf,then it has to be given a class label. A commonly used rule is the majority rule, that is, the leaf is labeled as j where j ⫽ arg max P(i |t) i
In words, we assign a leaf, t, to that class to which the majority of the vectors in Xt belong. Having discussed the major elements needed for the growth of a decision tree, we are now ready to summarize the basic algorithmic steps for constructing a binary decision tree ■ ■
Begin with the root node, that is, Xt ⫽ X For each new node t • For every feature xk , k ⫽ 1, 2, . . . , l For every value ␣ kn , n ⫽ 1, 2, . . . , Ntk — Generate XtY and XtN according to the answer in the question: is xk (i) ⱕ ␣kn , i ⫽ 1, 2, . . . , Nt — Compute the impurity decrease
End
Choose ␣kn0 leading to the maximum decrease w.r. to xk
219
“06-Ch04-SA272” 18/9/2008 page 220
220
CHAPTER 4 Nonlinear Classifiers
• End • Choose xk0 and associated ␣k0 n0 leading to the overall maximum decrease of impurity • If the stop-splitting rule is met, declare node t as a leaf and designate it with a class label • If not, generate two descendant nodes tY and tN with associated subsets XtY and XtN , depending on the answer to the question: is xk0 ⱕ ␣k0 n0 ■
End
Remarks ■
A variety of node impurity measures can be defined. However, as pointed out in [Brei 84], the properties of the resulting final tree seem to be rather insensitive to the choice of the splitting criterion. Nevertheless, this is very much a problem-dependent task.
■
A critical factor in designing a decision tree is its size. As was the case with the multilayer perceptrons, the size of a tree must be large enough but not too large; otherwise it tends to learn the particular details of the training set and exhibits poor generalization performance. Experience has shown that use of a threshold value for the impurity decreases as the stop-splitting rule does not lead to trees of the right size. Many times it stops tree growing either too early or too late. The most commonly used approach is to grow a tree up to a large size first and then prune nodes according to a pruning criterion. This philosophy is similar to that for pruning multilayer perceptrons. A number of pruning criteria have been suggested in the literature. A commonly used criterion is to combine an estimate of the error probability with a complexity measuring term (e.g., number of terminal nodes). For more on this issue the interested reader may refer to [Brei 84, Ripl 94].
■
A drawback associated with tree classifiers is their high variance. In practice it is not uncommon for a small change in the training data set to result in a very different tree. The reason for this lies in the hierarchical nature of the tree classifiers. An error that occurs in a node high in the tree propagates all the way down to the leaves below it. Bagging ( bootstrap aggregating) [Brei 96, Gran 04] is a technique that can reduce variance and improve the generalization error performance. The basic idea is to create a number of, say, B variants, X1 , X2 , . . . , XB , of the training set, X, using bootstrap techniques, by uniformly sampling from X with replacement (see also Section 10.3). For each of the training set variants, Xi , a tree, Ti , is constructed. The final decision is in favor of the class predicted by the majority of the subclassifiers, Ti , i ⫽ 1, 2, . . . , B.
“06-Ch04-SA272” 18/9/2008 page 221
4.20 Decision Trees
Random forests use the idea of bagging in tandem with random feature selection [Brei 01]. The difference with bagging lies in the way the decision trees are constructed. The feature to split in each node is selected as the best among a set of F randomly chosen features, where F is a user-defined parameter. This extra introduced randomness is reported to have a substantial effect in performance improvement. ■
Our discussion so far was focused on the OBCT type of tree. More general partition of the feature space, via hyperplanes not parallel to the axis, is possible via questions of the type: Is lk⫽1 ck xk ⱕ ␣? This can lead to a better partition of the space. However, the training now becomes more involved; see, for example, [Quin 93].
■
Constructions of fuzzy decision trees have also been suggested, by allowing the possibility of partial membership of a feature vector in the nodes that make up the tree structure. Fuzzification is achieved by imposing a fuzzy structure over the basic skeleton of a standard decision tree; see, for example, [Suar 99] and the references therein.
■
Decision trees have emerged as one of the most popular methods for classification. An OBCT performs binary splits on single variables, and classifying a pattern may only require a few tests. Moreover, they can naturally treat mixtures of numeric and categorical variables. Also, due to their structural simplicity, they are easily interpretable.
Example 4.2 In a tree classification task, the set Xt , associated with node t, contains Nt ⫽ 10 vectors. Four of these belong to class 1 , four to class 2 , and two to class 3 , in a three-class classification task. The node splitting results into two new subsets XtY , with three vectors from 1 , and one from 2 , and XtN with one vector from 1 , three from 2 , and two from 3 . The goal is to compute the decrease in node impurity after splitting. We have that I (t) ⫽ ⫺
4 4 2 4 4 2 log2 ⫺ log2 ⫺ log2 ⫽ 1.521 10 10 10 10 10 10
3 1 1 3 I (tY ) ⫽ ⫺ log2 ⫺ log2 ⫽ 0.815 4 4 4 4 1 3 2 3 2 1 I (tN ) ⫽ ⫺ log2 ⫺ log2 ⫺ log2 ⫽ 1.472 6 6 6 6 6 6 Hence, the impurity decrease after splitting is ⌬I (t) ⫽ 1.521 ⫺
4 6 (0.815) ⫺ (1.472) ⫽ 0.315 10 10
221
“06-Ch04-SA272” 18/9/2008 page 222
222
CHAPTER 4 Nonlinear Classifiers
For further information and a deeper study of decision tree classifiers, the interested reader may consult the seminal book [Brei 84]. A nonexhaustive sample of later contributions in the area is [Datt 85, Chou 91, Seth 90, Graj 86, Quin 93]. A comparative guide for a number of well-known techniques is provided in [Espo 97]. Finally, it must be stated that there are close similarities between the decision trees and the neural network classifiers. Both aim at forming complex decision boundaries in the feature space. A major difference lies in the way decisions are made. Decision trees employ a hierarchically structured decision function in a sequential fashion. In contrast, neural networks utilize a set of soft (not final) decisions in a parallel fashion. Furthermore, their training is performed via different philosophies. However, despite their differences, it has been shown that linear tree classifiers (with a linear splitting criterion) can be adequately mapped to a multilayer perceptron structure [Seth 90, Seth 91, Park 94]. So far, from the performance point of view, comparative studies seem to give an advantage to the multilayer perceptrons with respect to the classification error, and an advantage to the decision trees with respect to the required training time [Brow 93].
4.21 COMBINING CLASSIFIERS The present chapter is the third one concerning the classifier design phase. Although we have not exhausted the list (a few more cases will be discussed in the chapters to follow), we feel that we have presented to the reader the most popular directions currently used for the design of a classifier. Another trend that offers more possibilities to the designer is to combine different classifiers. Thus, one can exploit their individual advantages in order to reach an overall better performance than could be achieved by using each of them separately. An important observation that justifies such an approach is the following. From the different (candidate) classifiers we design in order to choose the one that fits our needs, one results in the best performance; that is, minimum classification error rate. However, different classifiers may fail (to classify correctly) on different patterns. That is, even the “best” classifier can fail on patterns that other classifiers succeed on. Combining classifiers aims at exploiting this complementary information that seems to reside in the various classifiers. This is illustrated in Figure 4.29. Many interesting design issues have now come onto the scene. What is the strategy that one has to adopt for combining the individual outputs in order to reach the final conclusion? Should one combine the results following the product rule, the sum rule, the min rule, the max rule, or the median rule? Should all classifiers be fed with the same feature vectors, or must different feature vectors be selected for the different classifiers? Let us now highlight some of these issues a bit further.
“06-Ch04-SA272” 18/9/2008 page 223
4.21 Combining Classifiers
Input pattern
Classifier #1
Classifier #L
Classifier #2
Combiner
Output
FIGURE 4.29 L classifiers are combined in order to provide the final decision for an input pattern. The individual classifiers may operate in the same or in different feature spaces.
Assume that we are given a set of L classifiers, which have already been trained (in one way or another) to provide as outputs the class a posteriori probabilities. For a classification task of M classes, given an unknown feature vector x each classifier produces estimates of the a posteriori class probabilities; that is, Pj (i |x), i ⫽ 1, 2, . . . , M, j ⫽ 1, 2, . . . , L. Our goal is to devise a way to come up with an improved estimate of a “final” a posteriori probability P(i |x) based on all the resulting estimates from the individual classifiers, Pj (i |x), j ⫽ 1, 2, . . . , L. An elegant way is to resort to information theoretic criteria [Mill 99] by exploiting the Kullback–Leibler (KL) (Appendix A) probability distance measure.
4.21.1 Geometric Average Rule According to this rule, one chooses P(i |x) in order to minimize the average KL distance between probabilities. That is, Dav ⫽
L 1 Dj L
(4.115)
j⫽1
where Dj ⫽
M i⫽1
P(i |x) ln
P(i |x) Pj (i |x)
(4.116)
223
“06-Ch04-SA272” 18/9/2008 page 224
224
CHAPTER 4 Nonlinear Classifiers
Taking into account that M
Pj (i |x) ⫽ 1
i⫽1
and employing Lagrange multipliers,optimization of (4.115) with respect to P(i |x) results in ( Problem 4.23) P(i |x) ⫽
L 1 1 (Pj (i |x)) L C
(4.117)
j⫽1
where C is a class-independent quantity C⫽
M L
1
(Pj (i |x)) L
i⫽1 j⫽1
All products are raised into the same power 1/L, independently of the class i . Thus, neglecting all the terms common to all classes, the classification rule becomes equivalent to assigning the unknown pattern to the class maximizing the product. That is, max i
L
Pj (i |x)
(4.118)
j⫽1
4.21.2 Arithmetic Average Rule As pointed out in Appendix A, the KL probability dissimilarity cost is not a true distance measure (according to the strict mathematical definition), in the sense that it is not symmetric. A different (from the product) combination rule results if we choose to measure the probability distance via the alternative KL distance formulation. That is, Dj ⫽
M
Pj (i |x) ln
i⫽1
Pj (i |x) P(i |x)
(4.119)
Using (4.119) in (4.115), optimization leads to (Problem 4.24) 1 Pj (i |x) L L
P(i |x) ⫽
(4.120)
j⫽1
There is no theoretical basis for preferring to maximize (4.120) instead of (4.118) with respect to i , and it has been reported (e.g., [Mill 99]), that although the product rule often produces better results than the sum rule, it may lead to less reliable results when the outputs of some of the classifiers result in values close to zero.
“06-Ch04-SA272” 18/9/2008 page 225
4.21 Combining Classifiers
4.21.3 Majority Voting Rule The product and the summation schemes of combining classifiers belong to the so-called soft type rules. Hard type combination rules are also very popular, owing to their simplicity and their robust performance. According to the majority vote scheme, one decides in favor of the class for which there is a consensus, or when at least lc of the classifiers agree on the class label of the unknown pattern, where lc ⫽
⎧ ⎨ L ⫹ 1, 2
L even
⎩ L⫹1 ,
L odd
2
(4.121)
Otherwise, the decision is rejection (i.e., no decision is taken). In other words, the combined decision is correct when the decisions of the majority of classifiers are correct, and it is wrong when the decisions of the majority of classifiers are wrong and they agree on the wrong label. A rejection is considered neither correct nor wrong. Assume now that we are given L individually trained classifiers,as previously, and in addition 1. The number L is odd. 2. Each classifier has the same probability p of correct classification. 3. The decision of each classifier is taken independently of the others. Of these three assumptions the third is the strongest. In reality,the decisions cannot be independent. The other two assumptions can (fairly) easily be relaxed (e.g., [Mill 99, Lam 97]). Let Pc (L) be the probability of correct decision, after the majority vote. Then, this is given by the binomial distribution (see [Lam 97]) Pc (L) ⫽
L m⫽lc
L m
pm (1 ⫺ p)L⫺m
where lc is defined in (4.121). Assuming L ≥ 3, then the following are true. ■
If p ⬎ 0.5, Pc (L) is monotonically increasing in L and Pc (L) → 1 as L → ⬁.
■
If p ⬍ 0.5, Pc (L) is monotonically decreasing in L and Pc (L) → 0 as L → ⬁.
■
If p ⫽ 0.5, Pc (L) ⫽ 0.5 for all L.
In other words, using L classifiers combined with the majority vote scheme and under the assumptions made the probability of correct classification increases with L and tends to 1, provided p ⬎ 0.5. This is slightly surprising! Does it mean that by combining classifiers with the majority vote scheme one can do better than the optimal Bayesian classifier? The answer is obviously no, and the secret lies in the last of the three assumptions, which is not valid in practice. On the contrary, one can deduce that even if this is approximately valid for small values of L as L increases the independence assumption becomes more and more unrealistic. However,
225
“06-Ch04-SA272” 18/9/2008 page 226
226
CHAPTER 4 Nonlinear Classifiers
the previously cited analysis provides a theoretical framework that justifies the general trend observed from experimental studies; that is, increasing the number of classifiers increases the probability of a correct decision. In [Kunc 03] it is pointed out that, in the case of combining dependent classifiers, there is no guarantee that a majority vote combination improves performance. Based on an artificially generated data set,an upper and a lower limit for the accuracy of the majority vote combiner are derived,in terms of the accuracy p of the individual classifiers, the number L of the classifiers, and the degree of dependence among the individual classifiers. Furthermore, it is shown that dependency is not necessarily detrimental and that training-dependent classifiers with a certain pattern of dependency may be beneficial. Similar results have been obtained in [Nara 05], where lower and upper bounds for the performance of combining classifiers through the majority voting rule have been theoretically derived for the binary classification problem. The analysis involves no assumptions about independence, and the majority voting problem is treated as a constraint optimization task. Other attempts to extend the theory to deal with dependent voters have also appeared in the literature, for example,[Berg 93, Bola 89],and some results are available that give performance predictions closer to the experimental evidence [Mill 99]. In practice, a number of scenarios have been proposed aiming to make the decisions of the individual classifiers more independent. One approach is to train individual classifiers using different data points that reside in the same feature space. This can be done using various resampling techniques from the original training set, such as bootstrapping. Bagging (Section 4.20) belongs to this family of methods. These types of combination approaches are most appropriate for unstable classifiers—that is, classifiers whose output(s) exhibit large variations for small changes in their input data. Tree classifiers and large (with respect to the number of training points) neural networks are typical examples of unstable classifiers. Stacking [Wolpe 92] is an alternative attempt toward independence that constructs the combiner using for its training the outputs of the individual classifiers. However, these outputs correspond to data points that have been excluded from the training set, which was used to train the classifiers. This is done in a rotated fashion. Each time, different points are excluded from the training set and are kept to be used for testing. The outputs of the classifiers, obtained from these tests, are then employed to train the combiner. The rationale is basically the same as that behind the leave-one-out method, discussed in Chapter 10. An alternative route that takes us closer to independence is to let each classifier operate in a different feature subspace. That is, each classifier is trained by employing a different subset from an original set of selected features (e.g., [Ho 98]). The majority vote scheme needs no modification in operating under such a scenario, in that all that is required is a counting of hard decisions. In contrast, the situation is different for the soft-type combination rules considered previously. Now, each pattern is represented in each classifier by a different input vector, and the resulting class posterior probabilities at the outputs of the classifiers can no longer be considered estimates of the same functional value, as is the case for Eqs. (4.118)
“06-Ch04-SA272” 18/9/2008 page 227
4.21 Combining Classifiers
and (4.120). The classifiers operate on different feature spaces. In [Kitt 98], a Bayesian framework is adopted to justify soft-type combination rules for such scenarios.
4.21.4 A Bayesian Viewpoint Let x i , i ⫽ 1, 2, . . . , L, be the feature vector representing the same pattern at the input of the ith classifier, with x i ∈ Rli , where li is the dimensionality of the respective feature space, which may not be the same for all feature vectors. The task is now cast in the Bayesian rationale. Given L measurements, x i , i ⫽ 1, 2, . . . , L, compute the maximum a posteriori joint probability M
P(i |x 1 , . . . , x L ) ⫽ max P(k |x 1 , . . . , x L )
(4.122)
P(k )p(x 1 , . . . , x L |k ) p(x 1 , . . . , x L )
(4.123)
k⫽1
However, P(k |x 1 , . . . , x L ) ⫽
For the problem to become tractable, we will once more adopt the statistical independence assumption, thus p(x 1 , . . . , x L |k ) ⫽
L
p(x j |k )
(4.124)
j⫽1
Combining Eqs. (4.122) through (4.124) and dropping out the class-independent quantities, the classification rule becomes equivalent to M
P(i |x 1 , . . . , x L ) ⫽ max P(k ) k⫽1
L
p(x j |k )
(4.125)
j⫽1
Substituting in the previous the Bayes rule p(x j |k ) ⫽
P(k |x j )p(x j ) P(k )
and getting rid of the class-independent terms,the classification rule finally becomes equivalent to searching for M
max(P(k ))1⫺L k⫽1
L
P(k |x j )
(4.126)
j⫽1
If we adopt the assumption that each class posterior probability, P(k |x j ), k ⫽ 1, 2, . . . , M, is provided (as an estimate) at the output of the respective classifier, then Eq. (4.126) is, once more, the product rule, this time with a Bayesian“blessing.” Although such an approach seems to provide the optimal classifier, it is built on two
227
“06-Ch04-SA272” 18/9/2008 page 228
228
CHAPTER 4 Nonlinear Classifiers
assumptions. The first is that of statistical independence. The second is that the true P(k |x j ) is approximated sufficiently well by the output of the jth classifier. The accuracy of the final result depends on how good this approximation is. A sensitivity analysis in [Kitt 98] shows that in many cases the product rule is very sensitive in such approximation errors. In contrast, the sum rule seems to be more resilient to errors and in many cases in practice outperforms the product rule. It can easily be shown that the sum rule can be obtained from Eq. (4.126) by assuming that P(k |x) ≈ P(k )(1 ⫹ ␦), where ␦ is a small value. This is a very strong assumption, since it implies that the a posteriori probability is approximately equal to the a priori one. This implicitly states that the classification task is very hard and no extra information about the class label is gained after the value of x becomes available. No doubt, from a theoretical point of view this is not a very pleasing assumption! An alternative viewpoint of the summation rule is given in [Tume 95],through the bias–variance dilemma looking glass. Assuming that the individual classifiers result in low bias estimates of the a posteriori class probabilities and under the mutual independence assumption, averaging of the outputs reduces variance, leading to a reduction in error rate. This point of view tempts one to choose large classifiers ( large number of free parameters) with respect to the number of training data, N , since such a choice favors low bias at the expense of high variance (Section 3.5.3), which is then reduced by the action of averaging. These results have been extended to include the weighted average case in [Fume 05]. In addition to the product and sum,other combination rules have been suggested in the literature, such as the max, min, and median rules. These rules are justified by the valid inequalities L j⫽1
L
P(k |x j ) ⱕ min P(k |x j ) ⱕ j⫽1
L 1 L P(k |x j ) ⱕ max P(k |x j ) j⫽1 L j⫽1
and classification is achieved by maximizing the respective bounds instead of the product or the summation [Kitt 98]. In some cases, the existence of outliers may lead the sum average value to be very wrong, since the value of the outlier is the dominant one in the summation. In such cases, the median value is well known to provide a more robust estimate, and the combination rule decides in favor of the class that gives the maximum median value. That is, M
max median{P(k |x j )} k⫽1
In the published literature, a number of variants of the previous methods have also been suggested, such as in [Kang 03, Lin 03, Ho 94, Levi 02]. The choice of the specific combination rule is, in general, problem dependent. In [Jain 00], a set of experiments is reported concerning the results of combining twelve different classifiers using five different combination rules. For the same data set ( handwritten numerals ∅⫺9),six different feature sets were generated. Each classifier was trained separately for each of the six feature sets, resulting in six different variants. Two types of combinations were performed: (a) all classifiers,trained on the same feature set, were combined using the five different combiners and (b) the outputs of the six
“06-Ch04-SA272” 18/9/2008 page 229
4.21 Combining Classifiers
variants of the same classifier were also combined via the five different combiners. The results show the following. 1. There is not a single type of combination (e.g., product rule, majority voting) that scores best for all cases. Each case seems to “prefer” its own combining rule. 2. For every case, some, out of the five, combining rules result in a higher error rate compared to that obtained by the best individual classifier. This means that combining does not necessarily lead to improved performance. 3. There are cases where none of the combining rules does better than the best individual classifier. 4. Improvements obtained by combining the variants of the same classifier,each trained on a different feature set, are substantially better than those obtained by combining different classifiers but trained on the same set. This seems to be a more general trend. That is, training each individual classifier on a different feature set offers the combiner better chances for improvements. In practice, one tries to combine classifiers that are as “diverse” as possible, expecting to improve performance by exploiting the complementary information residing in the outputs of the individual classifiers. Take, for example, the extreme case where all classifiers agree on their predictions. Any attempt to combine the classifiers for improving the overall performance would obviously be meaningless. As there is no formal definition of classifier diversity, a number of different measures have been suggested to quantify diversity for the purpose of classifier combining. For example, in [Germ 92] the variance is adopted as a diversity measure. For the case of hard decisions, let i (x j ) be the class label predicted by the ith classifier ¯ j ) also be the respective “mean” class label, computed over for pattern x j . Let (x all classifiers. The mean must be defined in a meaningful way. For hard decisions, one possibility is to adopt as the mean value the most frequent one among all (L) classifiers. Define
1 0
d i (x j ), (x ¯ j) ⫽
if i (x j ) ⫽ (x ¯ j) otherwise
The variance of the combined classifiers can be computed as 1 d i (x j ), (x ¯ j) NL N
V ⫽
L
j⫽1 i⫽1
A large variance is taken to be indicative for large diversity. Besides the variance, other measures have also been suggested and used. For example, in [Kang 00] the mutual information among the outputs of the classifiers is used, and in [Kunc 03] the Q statistics test is employed. For a review of diversity measures and comparative studies see, for example, [Kunc 03a, Akse 06]. In [Rodr 06] the issue of designing
229
“06-Ch04-SA272” 18/9/2008 page 230
230
CHAPTER 4 Nonlinear Classifiers
diverse classifiers is considered together with the issue of accuracy. A methodology, called Rotation Forest, is proposed, which aims at designing classifiers that are both accurate and diverse. The classifiers are then combined to boost the overall performance. Experimental comparative studies that demonstrate the performance improvement that may be gained by combining classifiers can be found in [Mill 99, Kitt 98, Tax 00, Dzer 04]. It seems that the sum average and the majority vote rules are more popular and used the most frequently. Which of the two is to be adopted depends on the application. In [Kitt 03], it is shown that for normally distributed error probabilities the sum rule outperforms the voting rule. In contrast, for heavy tail error distributions the voting scheme may be better. More theoretical results, concerning combinations of classifiers, can be found in [Kunc 02] and [Klei 90]. However,it is true that most of the available theoretical results have been developed under rather restrictive assumptions. More recently, a “fresh”look at the theoretical study of the performance of combiners has been presented in [Evge 04] and theoretical nonasymptotic bounds on the combiner’ s generalization error are derived, for the case of combining SVM classifiers, via weighted averaging. The so called no panacea theorem is stated in [Hu 08]. It is shown that if the combination function is continuous and diverse one can always construct probability density distributions that describe the data and lead the combination scheme to poor performance. In other words,this theorem points out that combining classifiers has to be considered carefully. The common characteristic of all combination techniques presented so far is that the individual classifiers are separately trained and the combiner relies on a simple rule. Besides these techniques, a number of other schemes have been developed, which rely on optimizing the combiner and in some cases jointly with the individual classifiers, for example, [Ueda 97, Rose 96, Kunc 01]. The price one pays for such procedures obviously is complexity, which in some cases can become impractical, see [Rose 96]. Moreover, there is no guarantee that optimization leads to improved performance compared to the simpler nonoptimal methods considered previously. More recently, Bayesian approaches ([Tres 01]) and Bayesian networks ([Garg 02, Pikr 08]) have been mobilized to construct combiners. A game-theoretic approach has been adopted in [Geor 06]. The so-called mixture of experts [ Jaco 91, Hayk 96, Avni 99] are structures that share some of the ideas exposed in this section. The rationale behind such models is to assign different classifiers for different regions in space and then use an extra “gating”network, which also sees the input feature vector, to decide which classifier (expert) should be used each time. All classifiers as well as the gating network are jointly trained.
4.22 THE BOOSTING APPROACH TO COMBINE CLASSIFIERS Boosting is a general approach to improve the performance of a given classifier and is one of the most powerful techniques, together with the support vector machines,
“06-Ch04-SA272” 18/9/2008 page 231
4.22 The Boosting Approach to Combine Classifiers
that blossomed in the 1990s. Although boosting can be considered an approach to combine classifiers, it is conceptually different from the techniques presented in the previous section, and it deserves a separate treatment. The roots of boosting go back to the original work of Viliant and Kearns [Vali 84, Kear 94], who posed the question whether a “weak” learning algorithm (i.e., one that performs just slightly better than a random guessing) can be boosted into a “strong” algorithm with good error performance. At the heart of a boosting method lies the so-called base classifier, which is a weak classifier. A series of classifiers is then designed iteratively, employing each time the base classifier but using a different subset of the training set, according to an iteratively computed distribution, or a different weighting over the samples of the training set. At each iteration, the computed weighting distribution gives emphasis to the “hardest” (incorrectly classified) samples. The final classifier is obtained as a weighted average of the previously hierarchically designed classifiers. It turns out that given a sufficient number of iterations the classification error of the final combination measured on the training set can become arbitrarily low [Scha 98]. This is very impressive indeed. Using a weak classifier as the base, one can achieve an arbitrarily low training error rate by appropriate manipulation of the training data set in harmony with the performance of the sequence of the designed classifiers (Problem 4.28). In this section we will focus on one such algorithm, the so-called AdaBoost (adaptive boosting), which is sometimes known as the discrete AdaBoost, to emphasize the fact that it returns a binary discrete label. This is the most popular algorithm of the family and one that has been extensively studied. The treatment follows the approach introduced in [Frie 00]. We concentrate on the two-class classification task and let the set of the training data be {(x 1 , y1 ), (x 2 , y2 ) . . . , (x N , yN )} with yi ∈ {⫺1, 1}, i ⫽ 1, 2, . . . , N . The goal is to construct an optimally designed classifier of the form f (x) ⫽ sign {F (x)}
(4.127)
where F (x) ⫽
K
␣k (x; k )
(4.128)
k⫽1
where (x; ) denotes the base classifier that returns a binary class label; that is, (x; ) ∈ {⫺1, 1}. The base classifier is described by the corresponding parameter vector , whose value is allowed to be different in each of the summand terms, as will become apparent soon. The values of the unknown parameters result from the following optimization. arg
min
␣k ;k ,k:1,K
N
exp ⫺yi F (x i )
(4.129)
i⫽1
This cost function is common in learning theory. It penalizes the samples that are wrongly classified ( yi F (x i ) ⬍ 0) much more heavily than those correctly classified
231
“06-Ch04-SA272” 18/9/2008 page 232
232
CHAPTER 4 Nonlinear Classifiers
( yi F (x i ) ⬎ 0). However, direct optimization of (4.129) is a highly complex task. A suboptimal method commonly employed in optimization theory for complex problems is to carry out the optimization in a stage-wise fashion. At each step, a new parameter is considered and optimization is carried out with respect to this parameter, leaving unchanged the previously optimized ones. To this end, let us define Fm (x) to denote the result of the partial sum up to m terms. That is, Fm (x) ⫽
m
␣k (x; k ), m ⫽ 1, 2, . . . , K
(4.130)
k⫽1
Based on the this definition, the following recursion becomes obvious. Fm (x) ⫽ Fm⫺1 (x) ⫹ ␣m (x; m )
(4.131)
Let us now employ a stage-wise optimization in our problem. At step m, Fm⫺1 (x) is the part that has been optimized in the previous step, and the current task is to compute the optimal values for ␣m , m . In other words, the task at step m is to compute (␣m , m ) ⫽ arg min J (␣, ) ␣,
where the cost function is defined as J (␣, ) ⫽
N
exp ⫺yi (Fm⫺1 (x i ) ⫹ ␣(x i ; ))
(4.132)
i⫽1
Once more,optimization will be carried out in two steps. First, ␣ will be considered constant, and the cost will be optimized with respect to the base classifier (x; ). That is, the cost to be minimized is now simplified to m ⫽ arg min
N
wi(m) exp ⫺yi ␣(x i ; )
(4.133)
i⫽1
where wi(m) ≡ exp ⫺yi Fm⫺1 (x i )
(4.134)
Since each wi(m) depends neither on ␣ nor on (x i ; ), it can be regarded as a weight associated with the sample point x i . Due to the binary nature of the base classifier ((x; ) ∈ {⫺1, 1}),it is easy to see that minimizing (4.133) is equivalent to designing the optimal classifier (x; m ) so that the weighted empirical error (the fraction of the training samples that are wrongly classified) is minimum. That is, m ⫽ arg min Pm ⫽
N i⫽1
wi(m) I (1 ⫺ yi (x i ; ))
(4.135)
“06-Ch04-SA272” 18/9/2008 page 233
4.22 The Boosting Approach to Combine Classifiers
Function I (·) is either 0 or 1, depending on its argument, whether it is zero or positive, respectively. To guarantee that the value of the weighted empirical error rate remains in the interval [0, 1], the weights must sum to one. This is easily achieved by normalization; that is, dividing each weight by the respec appropriate (m) w , which does not affect the optimization and can easily be tive sum, N i⫽1 i incorporated in the final iterative algorithm. Having computed the optimal classifier at step m, (x; m ), the following are easily established from the respective definitions.
wi(m) ⫽ Pm
(4.136)
wi(m) ⫽ 1 ⫺ Pm
(4.137)
yi ( x i ;m )⬍0
yi ( x i ;m )⬎0
Combining Eqs. (4.137) and (4.136) with (4.134) and (4.132), the optimum value, ␣m , results from ␣m ⫽ arg min {exp(⫺␣)(1 ⫺ Pm ) ⫹ exp(␣)Pm } ␣
(4.138)
Taking the derivative with respect to ␣ and equating to zero, we obtain ␣m ⫽
1 1 ⫺ Pm ln 2 Pm
(4.139)
Once ␣m and (x; m ) have been computed, the weights for the next step are readily available via the iteration wi(m⫹1)
wi(m) exp ⫺yi ␣m (x i ; m ) exp(⫺yi Fm (x i )) ⫽ ⫽ Zm Zm
(4.140)
where Zm is the normalizing factor Zm ≡
N
wi(m) exp ⫺yi ␣m (x i ; m )
(4.141)
i⫽1
Observe that the value of the weight corresponding to sample x i is increased (decreased) with respect to its value at the previous iteration step if the classifier (x i ; m ) fails (wins) at the respective point. Moreover, the percentage of the increase or decrease depends on the value of ␣m , which also controls the relative importance of the term (x ; m ) in building up the final classifier F (x) in (4.128). Hard examples (i.e., samples that fail to be classified correctly by a number of successive classifiers) gain an increased importance in the weighted empirical error rate as they insist on failing! A pseudocode for the AdaBoost algorithm follows.
233
“06-Ch04-SA272” 18/9/2008 page 234
234
CHAPTER 4 Nonlinear Classifiers
The AdaBoost Algorithm ■
Initialize: wi(1) ⫽
■
Initialize: m ⫽ 1
■
1 N,
i ⫽ 1, 2 . . . , N
Repeat • Compute optimum m in (·; m ) by minimizing Pm ; (4.135) • Compute the optimum Pm ; (4.135) • ␣m ⫽
1 2
ln
1⫺Pm Pm
• Zm ⫽ 0.0 • For i ⫽ 1 to N
wi(m⫹1) ⫽ wi(m) exp ⫺yi ␣m (x i ; m ) Zm ⫽ Zm ⫹ wi(m⫹1)
• End{For} • For i ⫽ 1 to N
wi(m⫹1) ⫽ wi(m⫹1) /Zm
• End {For} • K ⫽m • m⫽m⫹1 ■ ■
Until a termination criterion is met.
f (·) ⫽ sign( K k⫽1 ␣k (·, k ))
One of the main and very interesting properties of boosting is its relative immunity to overfitting, which was defined in Section 4.9. In practice, it has been verified that, although the number of terms, K , and consequently the associated number of parameters can be quite high,the error rate on a test set does not increase but keeps decreasing and finally levels off at a certain value. It has been observed that the test error continues to decrease long after the error on the training set has become zero. A mathematically pleasing explanation is offered in [Scha 98], where an upper bound for the error probability (also known as generalization error) is derived in terms of the margins of the training points with respect to the designed classifier. Note that the test error rate is an estimate of the error probability (more formal definitions of these quantities are provided in Section 5.9). The bound is independent of the number of iterations, K . More specifically, it is shown that with high probability the generalization error is upper-bounded by the quantity .
prob{marginf (x, y) ⬍ ␥} ⫹ O
Vc N ␥2
(4.142)
“06-Ch04-SA272” 18/9/2008 page 235
4.22 The Boosting Approach to Combine Classifiers
for ␥ ⬎ 0, where Vc is a parameter measuring the complexity of the base classifier and is known as the Vapnic–Chervonenkis dimension (we will discuss it later in the book). The margin of a training example with respect to a classifier f [Eq. (4.127)] is defined as yF (x) y marginf (x, y) ⫽ K ⫽ ␣ k⫽1 k
K
k⫽1 ␣k k (x; k )
K k⫽1 ␣k
The margin lies in the interval [⫺1, 1] and is positive if and only if the respective pattern is classified correctly. The bound implies that if (a) the margin probability is small for large values of margin ␥ and ( b) N is large enough with respect to Vc , one expects the generalization error to be small, and this does not depend on the number of iterations that were used to design f (x). The bound suggests that if for most of the training points the margin is large, the generalization error is expected to be small. This is natural, since the magnitude of the margin can be interpreted as a measure of confidence about the decision of the classifier with respect to a sample. Hence, if for a large training data set the resulting margin for most of the training points is large, it is not beyond common sense to expect that a low training error rate may also suggest a low generalization error. Furthermore, as pointed out in [Maso 00, Scha 98], boosting is particularly aggressive at improving the margin distribution, since it concentrates on examples with the smallest margins, as one can readily dig out by looking carefully at the cost (4.129). From this point of view, there is an affinity with the support vectors machines, which also try to maximize the margin of the training samples from the decision surface. See,for example,[Scha 98, Rats 02]. The major criticism about the bound in (4.142) lies in the fact that it is very loose (unless the number, N , of the training points is very large; i.e., of the order of tens of thousands!), so it can only be used as a qualitative rather than as a quantitative explanation of the commonly encountered experimental evidence. Another explanation for this overfitting immunity associated with the boosting algorithms could be that parameter optimization is carried out in a stage-wise fashion, each time with respect to a single parameter. Some very interesting and enlightening discussions, among leading experts in the field, regarding the overfitting as well as other issues concerning boosting and related algorithmic families can be found in the papers [Frie 00, Brei 98]. A comparative study of the performance of boosting and other related algorithms can be found in [Baue 99]. Remarks ■
Obviously,Adaboost is not the only boosting algorithm available. For example, one can come up with other algorithms by adopting alternatives to (4.129) cost functions or growing mechanisms to build up the final classifier. In fact, it has been observed that in difficult tasks corresponding to relatively high Bayesian error probabilities (i.e., attained by using the optimal Bayesian classifier), the performance of the AdaBoost can degrade dramatically. An
235
“06-Ch04-SA272” 18/9/2008 page 236
236
CHAPTER 4 Nonlinear Classifiers
explanation for it is that the exponential cost function over-penalizes “bad” samples that correspond to large negative margins, and this affects the overall performance. More on these issues can be obtained from [Hast 01, Frie 00] and the references therein. A variant of the AdaBoost has been proposed in [Viol 01] and later generalized in [Yin 05]. Instead of training a single base classifier, a number of base classifiers are trained simultaneously, each on a different set of features. At each iteration step, the classifier (·) results by combining these base classifiers. In principle, any of the combination rules can be used. [Scha 05] presents a modification of the AdaBoost that allows for incorporation of prior knowledge into boosting as a means of compensating for insufficient data. The so called AdaBoost∗ version was introduced in [Rats 05], where the margin is explicitly brought into the game and the algorithm maximizes the minimum margin of the training setup. The algorithm incorporates a current estimate of the achievable margin which is used for computation of the optimal combining coefficients of the base classifiers. Multiple additive regression trees (MART) is a possible alternative that overcomes some of the drawbacks related to AdaBoost. In this case, the additive model in (4.128) consists of an expansion in a series of classification trees (CART), and the place of the exponential cost in (4.129) can be taken by any differentiable function. MART classifiers have been reported to perform well in a number of real cases, such as in [Hast 01, Meye 03]. ■
For the multiclass case problem there are several extensions of AdaBoost. A straightforward extension is given in [Freu 97, Eibl 06]. However, this extension fails if the base classifier results in error rates higher than 50%. This means that the base classifier will not be a weak one, since in the multiclass case random guessing means a success rate equal to M1 , where M is the number of classes. Thus, for large M 50% rate of correct classification can be a strong requirement. To overcome this difficulty, other (more sophisticated) extensions have been proposed. See [Scha 99, Diet 95].
Example 4.3 Let us consider a two-class classification task. The data reside in the 20-dimensional space and obey a Gaussian distribution of unit covariance matrix and mean values √ [⫺a, ⫺a, . . . , ⫺a]T , [a, a, . . . , a]T , respectively, for each class, where a ⫽ 2/ 20. The training set consists of 200 points (100 from each class) and the test set of 400 points (200 from each class) independently generated from the points of the training set. To design a classifier using the AdaBoost algorithm, we chose as a seed the weak classifier known as stump. This is a very “naive” type of tree, consisting of a single node, and classification of a feature vector x is achieved on the basis of the value of only one of its features, say, xi . Thus, if xi ⬍ 0, x is assigned to class A. If xi ⬎ 0, it is assigned to class B. The decision
“06-Ch04-SA272” 18/9/2008 page 237
4.23 The Class Imbalance Problem
0.45 0.4 0.35
Error
0.3 0.25 0.2 Training set 0.15 Test set 0.1 0.05 0
0
200
400
600
800 1000 1200 1400 Number of base classifiers
1600
1800
2000
FIGURE 4.30 Training and test error rate curves as functions of the number of iteration steps for the AdaBoost algorithm, using a stump as the weak base classifier. The test error keeps decreasing even after the training error becomes zero.
about the choice of the specific feature, xi , to be used in the classifier was randomly made. Such a classifier results in a training error rate slightly better than 0.5. The AdaBoost algorithm was run on the training data for 2000 iteration steps. Figure 4.30 verifies the fact that the training error rate converges to zero very fast. The test error rate keeps decreasing even after the training error rate becomes zero and then levels off at around 0.05. Figure 4.31 shows the margin distributions, over the training data points, for four different training iteration steps. It is readily observed that the algorithm is indeed greedy in increasing the margin. Even when only 40 iteration steps are used for the AdaBoost training, the resulting classifier classifies the majority of the training samples with large margins. Using 200 iteration steps, all points are correctly classified (positive margin values), and the majority of them with large margin values. From then on, more iteration steps further improve the margin distribution by pushing it to higher values.
4.23 THE CLASS IMBALANCE PROBLEM In practice there are cases in which one class is represented by a large number of training points while another by only of few. This is usually referred to as the class imbalance problem. Such situations occur in a number of applications
237
“06-Ch04-SA272” 18/9/2008 page 238
CHAPTER 4 Nonlinear Classifiers
1 1000 base classifiers
0.9 0.8 cumulative distribution
238
0.7 0.6 0.5 0.4 0.3
40 base classifiers 200 base classifiers
0.2 0.1 0 21
5 base classifiers
20.8
20.6
20.4
20.2
0 margin
0.2
0.4
0.6
0.8
1
FIGURE 4.31 Margin distribution for the AdaBoost classifier corresponding to different numbers of training iteration steps. Even when only 40 iteration steps are used, the resulting classifier classifies the majority of the training samples with large margins.
such as text classification, diagnosis of rare medical conditions, and detection of oil spills in satellite imaging. It is by now well established that class imbalances may severely hinder the performance of a number of standard classifiers, for example, decision trees, multilayer neural networks, SVMs, and boosting classifiers. This does not come as a surprise, since our desire for a good generalization performance dictates the design of classifiers that are as “simple” as possible. A simple hypothesis, however, will not pay much attention to the rare cases in imbalanced data sets. A study of the imbalance class problem is given in [Japk 02]. There it is stated that the class imbalance may not necessarily be a hindrance to the classification, and it has to be considered in relation to the number of training points as well as the complexity and the nature of the specific classification task. For example, a large class imbalance may not be a problem in the case of an easy to learn task, for example, well separable classes, or in cases where a large training data set is available. On the other hand, there are cases where a small imbalance may be very harmful in difficult-to-learn tasks with overlapping classes and/or in the absence of a sufficient number of training points. To cope with this problem, a number of approaches have been proposed that evolve along two major directions.
“06-Ch04-SA272” 18/9/2008 page 239
4.24 Discussion
Data-level Approaches The aim here is to “rebalance” the classes by either oversampling the small class and/or undersampling the large class. Resampling can be either random or focused. The focus can be on points that lie close to the boundaries of the decision surfaces (oversampling) or far away (undersampling); see, for example, [Chaw 02, Zhou 06]. A major problem with this method is how to decide the class distribution given the data set; see, for example, [Weis 03].
Cost-sensitive Approaches According to this line of “thought”, standard classifiers are modified appropriately to account for the unfair data representation in the training set. For example, in SVMs, one way is to use different parameters C in the cost function for the two classes, for example, [Lin 02a]. According to the geometric interpretation, given in Section 3.7.5,this is equivalent to reducing the convex hulls at a different rate paying more respect to the smaller class, for example, [Mavr 07]. In [Sun 07], cost-sensitive modifications of the AdaBoost algorithm are proposed, where, during the iterations, samples from the small class are more heavily weighted than those coming from the more prevalent class. Class imbalance is a very important issue in practice. The designer of any classification system must be aware of the problems that may arise and alert of the ways to cope with it.
4.24 DISCUSSION The number of available techniques is large, and the user has to choose what is more appropriate for the problem at hand. There are no magic recipes. A large research effort has been focused on comparative studies of various classifiers in the context of different applications; see also the review in [Jain 00]. One of the most extensive efforts was the Statlog project [Mich 94], in which a wide range of classifiers was tested using a large number of different data sets. Furthermore, research effort has been devoted to unraveling relations and affinities between the different techniques. Many of these techniques have their origin in different scientific disciplines. Therefore, until a few years ago, they were considered independently. Recently, researchers have started to recognize underlying similarities among various approaches. For readers who want to dig a bit deeper into these questions, the discussions and results presented in [Chen 94, Ripl 94, Ripl 96, Spec 90, Holm 97, Josh 97, Reyn 99] will be quite enlightening. In [Zhan 00] a survey on applications of neural networks in pattern recognition is presented,and links between neural and more conventional classifiers are discussed. In summary, the only tip that can be given to the designer is that all of the techniques presented in this book are still serious players in the classifier
239
“06-Ch04-SA272” 18/9/2008 page 240
240
CHAPTER 4 Nonlinear Classifiers
design game.The final choice depends on the specific task. The proof of the pudding is in the eating!
4.25 PROBLEMS 4.1 We are given 10 feature vectors that originate from two classes 1 and 2 as follows 1 : [0.1, ⫺0.2]T , [0.2, 0.1]T , [⫺0.15, 0.2]T , [1.1, 0.8]T , [1.2, 1.1]T 2 : [1.1, ⫺0.1]T , [1.25, 0.15]T , [0.9, 0.1]T , [0.1, 1.2]T , [0.2, 0.9]T
Check whether these are linearly separable and, if not, design an appropriate multilayer perceptron with nodes having step function activation to classify the vectors in the two classes. 4.2 Using the computer, generate four two-dimensional Gaussian random sequences with covariance matrices 0.01 ⌺⫽ 0.0
0.0 0.01
and mean values 1 ⫽ [0, 0]T , 2 ⫽ [1, 1]T , 3 ⫽ [0, 1]T , 4 ⫽ [1, 0]T . The first two form class 1 , and the other two class 2 . Produce 100 vectors from each distribution. Use the batch mode backpropagation algorithm of Section 4.6 to train a two-layer perceptron with two hidden neurons and one in the output. Let the activation function be the logistic one with a ⫽ 1. Plot the error curve as a function of iteration steps. Experiment yourselves with various values of the learning parameter . Once the algorithm has converged, produce 50 more vectors from each distribution and try to classify them using the weights you have obtained. What is the percentage classification error? 4.3 Draw the three lines in the two-dimensional space x1 ⫹ x2 ⫽ 0 x2 ⫽
1 4
x1 ⫺ x 2 ⫽ 0
For each of the polyhedra that are formed by their intersections, determine the vertices of the cube into which they will be mapped by the first layer of a multilayer perceptron, realizing the preceding lines. Combine the regions into two classes so that (a) a two-layer network is sufficient to classify them and (b) a three-layer network is necessary. For both cases compute analytically the corresponding synaptic weights. 4.4 Show that if x 1 and x 2 are two points in the l-dimensional space, the hyperplane bisecting the segment with end points x 1 , x 2 , leaving x 1 at its
“06-Ch04-SA272” 18/9/2008 page 241
4.25 Problems
positive side, is given by (x 1 ⫺ x 2 )T x ⫺
1 1 x 1 2 ⫹ x 2 2 ⫽ 0 2 2
4.5 For the cross-entropy cost function of (4.33) ■
Show that its minimum value for binary desired response values is zero and it occurs when the true outputs are equal to the desired ones.
■
Show that the cross-entropy cost function depends on the relative output errors.
4.6 Show that if the cost function, optimized by a multilayer perceptron, is the cross entropy (4.33) and the activation function is the sigmoid (4.1), then the gradient ␦Lj (i) of (4.13) becomes ␦Lj (i) ⫽ a(1 ⫺ yˆ j (i))yj (i)
4.7 Repeat Problem 4.6 for the softmax activation function and show that ␦Lj (i) ⫽ yˆ j (i) ⫺ yj (i). 4.8 Show that for the cross-entropy cost function (4.30) the outputs of the network, corresponding to the optimal weights, approximate the conditional probabilities P(i |x). 4.9 Using formula (4.37), show that if l ≥ K then M ⫽ 2K . 4.10 Develop a program to repeat the simulation example of Section 4.10. 4.11 For the same example start with a network consisting of six neurons in the first hidden layer and nine neurons in the second. Use a pruning algorithm to reduce the size of the network. 4.12 Let the sum of error squares J⫽
N kL 1 ( yˆ m (i) ⫺ ym (i))2 2 i⫽1 m⫽1
be the minimized function for a multilayer perceptron. Compute the elements of the Hessian matrix ⭸2 J r ⭸wr ⬘ ⭸wkj k⬘ j ⬘
Show that near a minimum, this can be approximated by ⭸2 J ⬘
r ⭸wr ⭸wkj ⬘
k j⬘
⫽
kL N ⭸ˆym (i) ⭸ˆym (i) r ⬘ ⭸wkj ⭸wr i⫽1 m⫽1
k⬘ j ⬘
241
“06-Ch04-SA272” 18/9/2008 page 242
242
CHAPTER 4 Nonlinear Classifiers
Thus, the second derivatives can be approximated by products of the first-order derivatives. Following arguments similar to those used for the derivation of the backpropagation algorithm, show that ⭸ˆym (i) ⫽ ␦ˆ rjm ykr⫺1 r ⭸wkj
where ⭸ˆym (i) ␦ˆ rjm ⫽ ⭸jr (i)
Its computation takes place recursively in the backpropagation philosophy. This has been used in [Hass 93]. 4.13 In Section 4.4 it was pointed out that an approximation to the Hessian matrix, which is often employed in practice, is to assume that it is diagonal. Prove that under this assumption ⭸2 E
r ⭸wkj
2
is propagated via a backpropagation concept according to the formulas: ⭸2 E
(1)
r ⭸wkj
⭸2 E
(2)
⭸v
⭸2 E r⫺1 2 2 yk ⭸jr
2 ⫽
Lj 2
2 ⫽ f ⬘⬘ Lj ej ⫹ f ⬘ Lj
kl kr 2 r 2 ⭸2 E r ␦r wkj 2 ⫹ f ⬘⬘ jr⫺1 ⫽ f ⬘ jr⫺1 wkj k 2 ⭸jr⫺1 ⭸jr k⫽1 k⫽1
(3)
⭸2 E
where all off-diagonal terms of the Hessian matrix have been neglected and the dependence on i has been suppressed for notational convenience. 4.14 Derive the full Hessian matrix for a simple two-layer network with two hidden neurons and an output one. Generalize to the case of more than two hidden neurons. 4.15 Rederive the backpropagation algorithm of Section 4.6 with activation function f (x) ⫽ c tanh(bx)
4.16 In [Dark 91] the following scheme for adaptation of the learning parameter has been proposed: ⫽ 0
1 1⫹
t t0
“06-Ch04-SA272” 18/9/2008 page 243
4.25 Problems
Verify that, for large enough values of t0 (e.g., 300 ⱕ t0 ⱕ 500), the learning parameter is approximately constant for the early stages of training (small values of iteration step t) and decreases in inverse proportion to t for large values. The first phase is called search phase and the latter converge phase. Comment on the rationale of such a procedure. 4.17 Use a two-layer perceptron with a linear output unit to approximate the function y(x) ⫽ 0.3 ⫹ 0.2 cos(2x), x ∈ [0, 1]. To this end, generate a sufficient number of data points from this function for the training. Use the backpropagation algorithm in one of its forms to train the network. In the sequel produce 50 more samples, feed them into the trained network, and plot the resulting outputs. How does it compare with the original curve? Repeat the procedure for a different number of hidden units. 4.18 Show Eq. (4.48). Hint: Show first that the following recursion is true: O(N ⫹ 1, l) ⫽ O(N , l) ⫹ O(N , l ⫺ 1)
To this end, start with the N points and add an extra one. Then show that the difference in the dichotomies, as we go from N to N ⫹ 1 points, is due to the dichotomizing hyperplanes (for the N points case) that could have been drawn via this new point. 4.19 Show that if N ⫽ 2(l ⫹ 1) the number of dichotomies is given by 2N ⫺1 . Hint: Use the identity that J J ⫽ 2J i i⫽0
and recall that
2n ⫹ 1 2n ⫹ 1 ⫽ n⫺i⫹1 n⫹i
4.20 Repeat the experiments of Problem 4.17 using an RBF network. Select the k centers regularly spaced within [0, 1]. Repeat the experiments with different numbers of Gaussian functions and . Estimate the unknown weights using the least squares method. 4.21 Using your experience from the previous problem, repeat the procedure for the two-dimensional function y(x1 , x2 ) ⫽ 0.3 ⫹ 0.2 cos(2x1 ) cos(2x2 )
4.22 Let the mapping from the input space to a higher dimensional space be x ∈ R ⫺→y ≡ (x) ∈ R2k⫹1
243
“06-Ch04-SA272” 18/9/2008 page 244
244
CHAPTER 4 Nonlinear Classifiers
where
0T / 1 (x) ⫽ √ , cos x, cos 2x, . . . , cos kx, sin x, sin 2x, . . . , sin kx 2
Then show that the corresponding inner product kernel is y Ti y j ⫽ K (xi , xj ) sin k ⫹ 12 xi ⫺ xj ⫽ x ⫺x 2 sin i 2 j
4.23 Show (4.117). 4.24 Show (4.120). 4.25 Prove Eqs. (4.108) and (4.109) for the ridge regression task in its dual representation form. 4.26 Show that if the linear kernel is used, the primal ridge regression task results in Eq. (4.111). 4.27 Prove that Eqs. (4.111) and (4.112) are equivalent. 4.28 Prove that the the error rate on the training set corresponding to the final boosting classifier tends to zero exponentially fast.
MATLAB PROGRAMS AND EXERCISES Computer Programs 4.1 Data generator. Write a MATLAB function named data_generator that generates a two-class, two-dimensional data set using four normal distributions, with covariance matrices Si ⫽ s ∗ I , i ⫽ 1, . . . , 4, where I is the 2 ⫻ 2 identity matrix. The vectors that stem from the first two distributions belong to class ⫹1, while the vectors originating from the other two distributions belong to class ⫺1. The inputs for this function are: (a) a 2 ⫻ 4 matrix, m, whose ith column is the mean vector of the ith distribution, (b) the variance parameter s, mentioned before, and (c) the number of the points, N , which will be generated from each distribution. The output of the function consists of (a) an array, X, of dimensionality 2 ⫻ 4 ∗ N , whose first group of N vectors stem from the first distribution, the second group from the second distribution and so on, (b) a 4 ∗ N dimensional row vector y with values ⫹1 or ⫺1, indicating the classes to which the corresponding data vectors in X belong.
Solution function [x,y]=data_generator(m,s,n) S = s*eye(2);
“06-Ch04-SA272” 18/9/2008 page 245
MATLAB Programs and Exercises
[l,c] = size(m); x = []; % Creating the training set for i = 1:c x = [x mvnrnd(m(:,i)',S,N)']; end y=[ones(1,N) ones(1,N) -ones(1,N) -ones(1,N)];
4.2 Neural network training. Write a MATLAB function, named NN _training, which uses the least squares criterion to train a two-layer feed-forward neural network with a single node in the output layer. The activation function for all the nodes is the hyperbolic tangent one. For training, one may select one of the following algorithms: a) the standard gradient descent backpropagation algorithm (code 1), ( b) the backpropagation algorithm with momentum (code 2), and (c) the backpropagation algorithm with adaptive learning rate (code 3). The inputs of this function are: (a) The data set (X, y), where the ith column of the array matrix X is the data vector and the ith element of the row vector y contains the class label (⫺1 or ⫹1), indicating the corresponding class to which the ith data vector belongs. (b) The number of first layer nodes. (c) The code number of the training method to be adopted. (d) The number of iterations for which the algorithm will run. (e) A parameter vector that contains the values of the parameters required for the adopted training method. This has the form [lr, mc, lr_inc, lr_dec, max_perf _inc] where lr is the learning rate, mc is the momentum parameter and the remaining three parameters, which are used in the backpropagation algorithm with variable learning rate, correspond to ri , rd , and c, as defined in Section 4.7, respectively. For the standard backpropagation algorithm, the last four components of the parameter vector are 0, for the momentum variant the last three parameters are 0, while for the adaptive learning rate case only the second component is 0. The output of the network is the object net that corresponds to the trained neural network. To make the results reproducible for comparison purposes, ensure that every time this function is called it begins from the same initial condition.
Solution function net = NN_training(x,y,k,code,iter,par_vec) rand('seed',0) % Initialization of the random number % generators
245
“06-Ch04-SA272” 18/9/2008 page 246
246
CHAPTER 4 Nonlinear Classifiers
randn('seed',0) % for reproducibility of net initial % conditions % List of training methods methods_list = {'traingd'; 'traingdm'; 'traingda'}; % Limits of the region where data lie limit = [min(x(:,1)) max(x(:,1)); min(x(:,2)) max(x(:,2))]; % Neural network definition net = newff(limit,[k 1],{'tansig','tansig'},... methods_list{code,1}); % Neural network initialization net = init(net); % Setting parameters net.trainParam.epochs = iter; net.trainParam.lr=par_vec(1); if(code == 2) net.trainParam.mc=par_vec(2); elseif(code == 3) net.trainParam.lr_inc = par_vec(3); net.trainParam.lr_dec = par_vec(4); net.trainParam.max_perf_inc = par_vec(5); end % Neural network training net = train(net,x,y); %NOTE: During training, the MATLAB shows a plot of the % MSE vs the number of iterations.
4.3 Write a MATLAB function, named NN _evaluation, which takes as inputs: (a) a neural network object and (b) a data set (X, y) and returns the probability of error that this neural network gives when it runs over this data set. The two-class case (⫺1 and ⫹1) is considered.
Solution function pe = NN_evaluation(net,x,y) y1 = sim(net,x); %Computation of the network outputs pe=sum(y.*y1<0)/length(y);
4.4 Write a MATLAB function, named plot_dec_regions, that plots the decision regions produced by a neural network trained on data sets produced by the data_generator function. The inputs to this function are: (a) a neural network object, (b) the lower and the upper bounds in the horizontal and vertical directions (lh, uh, lv, uv, respectively) of the region of space where the decision regions will be plotted as well as the resolution parameters (rh, rv) in both directions (the lower their value the finer the plot) and (c) the matrix m with the mean vectors of the normal distributions. Its
“06-Ch04-SA272” 18/9/2008 page 247
MATLAB Programs and Exercises
output is a decision region plot, where each region is marked with a red star or with a blue circle according to whether it belongs to class ⫹1 or ⫺1, respectively.
Solution For an implementation of this function see in the book website www. elsevierdirect.com/9781597492720. 4.5 Write a MATLAB function, named SVM_clas, which (i) generates an SVM classifier that uses RBF kernels, utilizing a given training set and (ii) measures its performance on both the training set and a given test set. The function takes as inputs (a) a training set (X1, y1), where each column of X1 is a data vector and each element of the row vector y1 indicates the category of the corresponding data vector in X1 (⫺1 or ⫹1 in this case), (b) a similarly defined test set (X2, y2), (c) a tolerance parameter that is used in the termination condition of the SVM training procedure, (d) the parameter C in the SVM cost function (Section 3.7.2), (e) the parameter sigma for the RBF kernels. The function returns as outputs (a) the structure of the SVM classifier in the SVMstruct object, (b) the indices of the support vectors in the svIndex vector, (c) the error probability for the training set pe_tr, and (d) the error probability of the test set pe_te.
Solution function [SVMstruct,svIndex,pe_tr,pe_te]=... SVM_clas(X1,y1,X2,y2,tol,C,sigma) options = svmsmoset('TolKKT',tol,'Display','iter',... 'MaxIter',20000,'KernelCacheLimit',10000); %Training and Ploting parameters [SVMstruct,svIndex]=svmtrain(X1', y1','... KERNEL_FUNCTION','rbf',... 'RBF_SIGMA',sigma,'BOXCONSTRAINT',C,'showplot',true,... 'Method','SMO','SMO_Opts',options); %Computation of the error probability train_res=svmclassify(SVMstruct,X1'); pe_tr=sum(y1'~=train_res)/length(y1); test_res=svmclassify(SVMstruct,X2'); pe_te=sum(y2'~=test_res)/length(y2);
Computer Experiments 4.1
(a) After initializing the seed of the randn function of MATLAB to, say, 0 to create the data (randn(‘seed ⬘ , 0)), use the / data_generator function 0 ⫺5 ⫹5 ⫹5 ⫺5 set (X1, y1), with m ⫽ , s ⫽ 2 and N ⫽ 100. ⫹5 ⫺5 ⫹5 ⫺5
247
“06-Ch04-SA272” 18/9/2008 page 248
248
CHAPTER 4 Nonlinear Classifiers
(b) Initialize the seed of randn to,say,10 and repeat (a) to produce the data set (X2, y2). (c) Repeat the above two steps using the corresponding seeds for randn (although this is not necessary), for s ⫽ 5, and produce the (X3, y3) and (X4, y4) data sets, respectively (m and N are as before). (d) Plot the data sets. 4.2
(a) Run the standard backpropagation algorithm with lr ⫽ 0.01 and 2, 4, and 15 first layer nodes, for 1000 iterations, using the data set (X1, y1) as training set. (b) Evaluate the performance of the designed neural networks for both (X1, y1) (training set) and (X2, y2) (test set) and plot the decision regions (use lh ⫽ lv ⫽ ⫺10, uh ⫽ uv ⫽ 10, rh ⫽ rv ⫽ 0.2). (c) Comment on the results.
4.3
(a) Run the backpropagation algorithm with 4 first layer nodes with (i) lr ⫽ 0.01 for 300 iterations, (ii) lr ⫽ 0.001, for 300 iterations, (iii) lr ⫽ 0.01, for 1000 iterations and (iv) lr ⫽ 0.001, for 1000 iterations, using the (X1, y1) as training set. (b) Evaluate the performance of the designed neural networks for both (X1, y1) (training set) and (X2, y2) (test set) and plot the decision regions. (c) Comment on the results.
4.4
(a) Run the adaptive learning rate variation of the backpropagation algorithm with lr ⫽ 0.001, lr_inc ⫽ 1.05, lr_dec ⫽ 0.7, max_perf _inc ⫽ 1.04, for 300 iterations. (b) Evaluate the performance of the designed neural networks for both (X1, y1) (training set) and (X2, y2) (test set) and plot the decision regions. (c) Compare the above results with whose obtained for the standard backpropagation algorithm with lr ⫽ 0.001, for 300 iterations.
4.5 Repeat 4.2–4.4 using the (X3, y3) and (X4, y4) as training and test sets, respectively. 4.6
(a) Generate SVM classifiers using (X1, y1) and (X2, y2) defined in 4.1 as training and test sets, respectively, for C ⫽ 1, 100, 1000, sigma ⫽ 0.5, 1, 2, 4, and tolerance parameter equal to 0.001. (b) For each case, evaluate the respective classifier’ s performance on the training and the test sets. (c) Comment on the number of support vectors produced in each case.
“06-Ch04-SA272” 18/9/2008 page 249
References
4.7 Repeat 4.6 where now (X1, y1), (X2, y2) are replaced by (X3, y3) (X4, y4), respectively.
REFERENCES [Adal 97] Adali T., Liu X., Sonmez K. “Conditional distribution learning with neural networks and its application to channel equalization,” IEEE Transactions on Signal Processing, Vol. 45(4), pp. 1051–1064, 1997. [Akse 06] Aksela M., Laaksonen J. “Using diversity of errors for selecting members of a committee classifier,” Pattern Recognition,Vol. 39, pp. 608–623, 2006. [Amar 99] Amari S., Wu S. “Improving support vector machines by modifying kernel functions,” Neural Networks,Vol. 12(6), pp. 783–789, 1999. [Ampa 02] Ampazis N., Perantonis S.J. “Two highly efficient second order algorithms for feedforward networks,” IEEE Transactions on Neural Networks, Vol. 13(5), pp. 1064–1074, 2002. [Angu 03] Anguita D.,Ridella S.,Rivieccio F.,Zunino R.“Hyperparameter design criteria for support vector classifiers,” Neurocomputing,Vol. 55, pp. 109–134, 2003. [Avni 99] Avnimelech R., Intrator N. “Boosted mixture of experts: An ensemble learning scheme,” Neural Computation,Vol. 11, pp. 475–490, 1999. [Barn 92] Barnard E. “Optimization for training neural networks,” IEEE Transactions on Neural Networks,Vol. 3(2), pp. 232–240, 1992. [Barr 93] Barron A.R. “Universal approximation bounds for superposition of a sigmoid function,” IEEE Transactions on Information Theory,Vol. 39(3), pp. 930–945, 1993. [Bart 02] Bartlett P., Boucherou S., Lugosi G. “Model selection and error estimation,” Machine Learning,Vol. 48, pp. 85–113, 2002. [Bart 98] Bartlett P.L. “The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network,” IEEE Transactions on Information Theory,Vol. 44(2), pp. 525–536, 1998. [Batt 92] Battiti R. “First and second order methods for learning: Between steepest descent and Newton’s method,” Neural Computation,Vol. 4, pp. 141–166, 1992. [Baue 99] Bauer E., Kohavi R. “An empirical comparison of voting classification algorithms: Bagging, boosting, and variants,” Machine Learning,Vol. 36, pp. 105–139, 1999. [Baum 88] Baum E.B., Wilczek F. “Supervised learning of probability distributions by neural networks,” in Neural Information Processing Systems (Anderson D., ed.), pp. 52–61, American Institute of Physics, New York, 1988. [Ben 99] Ben-Yakoub S.,Abdeljaoued,Mayoraj E.“Fusion of face and speech data for person identity verification,” IEEE Transactions on Neural Networks,Vol. 10(5), pp. 1065–1075, 1999. [Berg 93] Berg S. “Condorcet’s jury theorem, dependency among jurors,” Social Choice Welfare, Vol. 10, pp. 87–95, 1993. [Bish 95] Bishop C.M. Neural Networks for Pattern Recognition, Oxford University Press, 1995. [Blan 96] Blanz V., Schölkopf B., Bülthoff H., Burges C.,Vapnik V., Vetter T. “Comparison of view based object recognition using realistic 3D models,”Proceedings of International Conference on Artificial Neural Networks, pp. 251–256, Berlin, 1996.
249
“06-Ch04-SA272” 18/9/2008 page 250
250
CHAPTER 4 Nonlinear Classifiers
[Bola 89] Boland P.J. “Majority systems and the Condorcet jury theorem,” Statistician, Vol. 38, pp. 181–189, 1989. [Bose 96] Bose N.K., Liang P. Neural Network Fundamentals with Graphs, Algorithms and Applications, McGraw-Hill, 1996. [Brei 84] Breiman L., Friedman J., Olshen R., Stone C. Classification and Regression Trees, Wadsworth International, pp. 226–239, Belmont, CA, 1984. [Brei 96] Breiman L. “Bagging predictors,” Machine Learning,Vol. 24, pp. 123–140, 1996. [Brei 98] Breiman L. “Arcing classifiers,”The Annals of Statistics,Vol. 26(3), pp. 801–849, 1998. [Brei 01] Breiman L. “Random Forests,” Machine Learning,Vol. 45, pp. 5–32, 2001. [Brid 90] Bridle J.S.“Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation parameters,”in Neural Information Processing Systems 2 (Touretzky D.S., ed.), pp. 211–217, Morgan Kaufmann, 1990. [Broo 88] Broomhead D.S., Lowe D. “Multivariable functional interpolation and adaptive networks,” Complex Systems,Vol. 2, pp. 321–355, 1988. [Brow 93] Brown D., Corrnble V., Pittard C.L. “A comparison of decision tree classifiers with backpropagation neural networks for multimodal classification problems,” Pattern Recognition, Vol. 26(6), pp. 953–961, 1993. [Burg 99] Burges C.J.C.“Geometry and invariance in kernel based methods,”in Advances in Kernel Methods: Support Vector Learning (Schólkopf B., Burges C.J.C., Smola A.J., eds.), MIT Press, 1999. [Chap 02] Chapelle O.,VapnikV.,Bousquet O.,Mukherjee S.“Choosing multiple parameters for support vector machines,” Machine Learning,Vol. 46, pp. 131–159, 2002. [Chau 90] Chauvin Y. “Generalization performance of overtrained backpropagation networks,” in Neural Networks, Proc. EURASIP Workshop (Almeida L.B., Wellekens C.J., eds.), pp. 46–55, 1990. [Chaw 02] Chawla N.V., Bowyer K., Hall L., Kelgemeyer W.P. “SMOTE: Synthetic minority oversampling technique,” Journal of Artificial Intelligence Research,Vol. 16, pp. 321–357, 2002. [Chen 91] Chen S.,Cowan C.F.N.,Grant P.M.“Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Transactions on Neural Networks,Vol. 2, pp. 302–309, 1991. [Chen 94] Cheng B.,Titterington D.M. “Neural networks: A review from a statistical perspective,” Statistical Science,Vol. 9(1), pp. 2–30, 1994. [Chou 91] Chou P.“Optimal partitioning for classification and regression trees,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 13, pp. 340–354, 1991. [Cich 93] Cichocki A., Unbehauen R. Neural Networks for Optimization and Signal Processing, John Wiley & Sons, 1993. [Cort 95] Cortes C., Vapnik V.N. “Support vector networks,” Machine Learning, Vol. 20, pp. 273–297, 1995. [Cour 53] Courant R., Hilbert D. Methods of Mathematical Physics, Interscience, 1953. [Cove 65] Cover T.M. “Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition,” IEEE Transactions on Electronic Computers, Vol. 14, pp. 326–334, 1965. [Cybe 89] Cybenko G. “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems,Vol. 2, pp. 304–314, 1989.
“06-Ch04-SA272” 18/9/2008 page 251
References
[Czyz 04] Czyz J., Kittler J.,Vandendorpe L.“Multiple classifier combination for face-based identity verification,” Pattern Recognition,Vol. 37, pp. 1459–1469, 2004. [Dark 91] Darken C., Moody J. “Towards faster stochastic gradient search,” Advances in Neural Information Processing Systems 4, pp. 1009–1016, Morgan Kaufmann, San Mateo, CA, 1991. [Datt 85] Dattatreya G.R.,Kanal L.N.“Decision trees in pattern recognition,”in Progress in Pattern Recognition 2 (Kanal L.N., Rosenfeld A., eds.), North Holland, 1985. [Delo 94] Delopoulos A.,Tirakis A., Kollias S.“Invariant image classification using triple correlation based neural networks,” IEEE Transactions on Neural Networks,Vol. 5, pp. 392–408, 1994. [Diet 95] Dietterich T.G., Bakiri G. “Solving multiclass learning problems via error-correcting output codes,” Journal of Artificial Intelligence Research,Vol. 2, pp. 263–286, 1995. [Druc 99] Drucker H.,Wu D.,Vapnik,V.N.“Support vector machines for spam categorization,”IEEE Transactions on Neural Networks,Vol. 10(5), pp. 1048–1055, 1999. [Duan 03] Duan K., Keerthi S.S., Poo A.N. “Evaluation of simple performance measures for tuning SVM hyperparameters,” Neurocomputing,Vol. 51, pp. 41–59, 2003. ˇ [Dzer 04] Dˇzeroski S., Zenko B. “Is combining classifiers with stacking better than selecting the best one?” Machine Learning,Vol. 54, pp. 255–273, 2004. [Eibl 06] Eibl G., Pfeifer K.P. “Multiclass boosting for weak classifiers,” Journal of Machine Learning Research,Vol. 6, pp. 189–210, 2006. [ElNa 02] El-Naga I.,Yang Y.,Wernick M.N., Galatsanos N. “A support vector machine approach for detection of microcalcifications,” IEEE Transactions on Medical Imaging,Vol. 21, pp. 1552– 1563, 2002. [Enge 04] Engel Y., Mannor S. “The kernel least-squares algorithm,” IEEE Transactions on Signal Processing,Vol. 52(8), pp. 2275–2285, 2004. [Espo 97] Esposito F., Malebra D., Semeraro G. “A comparative analysis of methods for pruning decision trees,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 19(5), pp. 476–491, 1997. [Evge 04] Evgeniou T., Pontil M., Elisseeff A. “Leave one out error, stability, and generalization of voting combinations of classifiers,” Machine Learning,Vol. 55, pp. 71–97, 2004. [Fahl 90] Fahlman S.E., Lebiere C. “The cascade-correlation learning architecture,” in Advances in Neural Information Processing Systems 2 (Touretzky D.S., ed.), pp. 524–532, Morgan Kaufmann, 1990. [Flao 06] Flaounas I.N., Iakovidis D.K., Marvoulis D.E. “Cascading SVMs as a tool for medical diagnosis using multi-class gene expression data,” International Journal of Artificial Intelligence Tools. [Frea 90] Frean M. “The Upstart algorithm: A method for constructing and training feedforward networks,” Neural Computation,Vol. 2(2), pp. 198–209, 1990. [Freu 97] Freund Y., Schapire R.E. “A decision theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences,Vol. 55(1), pp. 119–139, 1997. [Frie 00] Friedman J., Hastie T., Tibshirani R. “Additive logistic regression: A statistical view of boosting,”The Annals of Statistics,Vol. 28(2), pp. 337–407, 2000. [Fried 81] Friedman J., Stuetzle W.“Projection pursuit regression,”Journal of American Statistical Association,Vol. 76, pp. 817–823, 1981.
251
“06-Ch04-SA272” 18/9/2008 page 252
252
CHAPTER 4 Nonlinear Classifiers
[Fuku 92] Fukumi M.,Omatu S.,Takeda F.,KosakaT.“Rotational invariant neural pattern recognition system with application to coin recognition,” IEEE Transactions on Neural Networks,Vol. 3, pp. 272–279, 1992. [Fuku 82] Fukushima K., Miyake S.,“Neogognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position,” Pattern Recognition, Vol. 15(6), pp. 445–469, 1982. [Fume 05] Fumera G., Roli F. “A theoretical and experimental analysis of linear combiners for multiple classifier systems,”IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27(6), pp. 942–956, 2005. [Funa 89] Funahashi K. “On the approximate realization of continuous mappings by neural networks,” Neural Networks,Vol. 2(3), pp. 183–192, 1989. [Gall 90] Gallant S.I. “Perceptron-based learning algorithms,” IEEE Transactions on Neural Networks,Vol. 1, pp. 179–191, 1990. [Garg 02] Garg A., Pavlovic V., Huang T.S. “Bayesian networks as enseble of classifiers,” Proceedings 16th International Conference on Pattern Recognition,Vol. 2, pp. 779–784, 2002. [Geor 06] Georgion H., Mavroforakis M.,Theodoridis S. “A game theoretic approach to weighted majority voting for combining SVM classifiers,”in Proceedings of the International Conference on Artificial Neural Networks, pp. 284–292, LNCS 4131, Springer, 2006. [Germ 92] German S., Vienenstock E., Doursat R. “Neural networks and bias/variance dilemma,” Neural Computation,Vol. 4(1), pp. 1–58, 1992. [Gibs 90] Gibson G.J., Cowan C.F.N. “On the decision regions of multilayer perceptrons,” Proceedings of the IEEE,Vol. 78(10), pp. 1590–1594, 1990. [Gish 90] Gish H. “A probabilistic approach to the understanding and training neural classifiers,” in Proceedings of the IEEE Conference on Acoustics Speech and Signal Processing, pp. 1361–1364,April 1990. [Gomm 00] Gomm J.B., Yu D.L. “Selecting radial basis function network centers with recursive orthogonal least squares training,”IEEE Transactions on Neural Networks,Vol. 11(2),pp. 306– 314, 2000. [Graj 86] Grajki K.A. et al. “Classification of EEG spatial patterns with a tree structured methodology,” IEEE Transactions on Biomedical Engineering,Vol. 33(12), pp. 1076–1086, 1986. [Gran 04] Grandvalet Y. “Bagging equalizes influence,” Machine Learning, Vol. 55, pp. 251–270, 2004. [Hamp 90] Hampshire J.B. II, Perlmutter B.A. “Equivalence proofs for multilayer perceptron classifiers and the Bayesian discriminant function,” in Proceedings of the 1990 Connectionist Models Summer School (Touretzky D, et al., eds.), Morgan Kaufmann, 1990. [Hart 90] Hartman E.J., Keeler J.D., Kowalski J.M. “Layered neural networks with Gaussian hidden units as universal approximations,” Neural Computations,Vol. 2(2), pp. 210–215, 1990. [Hass 93] Hassibi B., Stork D.G.,Wolff G.J. “Optimal brain surgeon and general network pruning,” Proceedings IEEE Conference on Neural Networks,Vol. 1, pp. 293–299, San Francisco, 1993. [Hast 01] Hastie T.,Tibshirani R., Friedman J. The Elements of Statistical Learning, Springer, 2001. [Hayk 96] Haykin S. “Neural networks expand SP’s horizons,” IEEE Signal Processing Magazine, Vol. 13(2), pp. 24–49, 1996. [Hayk 99] Haykin S. Neural Networks: A Comprehensive Foundation,2nd ed.,Prentice Hall,1999.
“06-Ch04-SA272” 18/9/2008 page 253
References
[Hert 91] Hertz J., Krogh A., Palmer R.G. Introduction to the Theory of Neural Computation, Addison-Wesley, 1991. [Hint 90] Hinton G.E. “Connectionist learning procedures,” in Machine Learning: Paradigms and Methods (Carbonell J.G., ed.), pp. 185–234, MIT Press, 1990. [Ho 98] Ho T.K. “The random subspace method for constructing decision forests,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 20(8), pp. 832–844, 1998. [Ho 94] Ho T.K., Hull J.J., Srihari S.N. “Decision combination in multiple classifier design,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 16(1), pp. 66–75, 1994. [Holm 97] Holmstrom L., Koistinen P., Laaksonen J., Oja E. “Neural and statistical classifiers— taxonomy and two case studies,” IEEE Transactions on Neural Networks,Vol. 8(1), pp. 5–17, 1997. [Horn 89] Hornik K., Stinchcombe M., White H. “Multilayer feedforward networks are universal approximators,” Neural Networks,Vol. 2(5), pp. 359–366, 1989. [Hu 08] Hu R.,Damper R.I.“A no panacea theorem for classifier combination,”Pattern Recognition, Vol. 41, pp. 2665–2673, 2008. [Huan 04] Huang D-S., Ip H., Chi Z. “A neural root finder of polynomials based on root moments.” Neural Computation,Vol. 16, pp. 1721–1762, 2004. [Hush 93] Hush D.R.,Horne B.G.“Progress in supervised neural networks,”IEEE Signal Processing Magazine,Vol. 10(1), pp. 8–39, 1993. [Ito 91] Ito Y. “Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory,” Neural Networks, Vol. 4(3), pp. 385–394, 1991. [Jaco 88] Jacobs R.A.“Increased rates of convergence through learning rate of adaptation,”Neural Networks,Vol. 2, pp. 359–366, 1988. [ Jaco 91] Jacobs R.A., Jordan M.I., Nowlan S.J., Hinton G.E. “Adaptive mixtures of local experts,” Neural Computation,Vol. 3, pp. 79–87, 1991. [Jain 00] Jain A.K., Duin P.W., Mao J. “Statistical pattern recognition: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 22(1), pp. 4–37, 2000. [Japk 02] Japkowicz N., Stephen S. “The class imbalance problem: a systematic study,” Intelligent Data Analysis Journal,Vol. 6(5), pp. 429–450, 2002. [Joha 92] Johansson E.M., Dowla F.U., Goodman D.M. “Backpropagation learning for multilayer feedforward neural networks using conjugate gradient method,” International Journal of Neural Systems,Vol. 2(4), pp. 291–301, 1992. [Josh 97] Joshi A., Ramakrishman N., Houstis E.N., Rice J.R. “On neurobiological, neuro-fuzzy, machine learning, and statistical pattern recognition techniques,” IEEE Transactions on Neural Networks,Vol. 8(1), pp. 18–31, 1997. [Juan 92] Juang B.H., Katagiri S. “Discriminative learning for minimum error classification,” IEEE Transactions on Signal Processing,Vol. 40(12), pp. 3043–3054, 1992. [Kalo 97] Kalouptsidis N. Signal Processing Systems, Theory and Design,JohnWiley & Sons,1997. [Kana 92] Kanaoka T., Chellapa R., Yoshitaka M., Tomita S. “A higher order neural network for distortion invariant pattern recognition,” Pattern Recognition Letters, Vol. 13, pp. 837–841, 1992.
253
“06-Ch04-SA272” 18/9/2008 page 254
254
CHAPTER 4 Nonlinear Classifiers
[Kang 00] Kang H., Lee S. “An information-theoretic strategy for constructing multiple classifier systems,” Proceedings of the 15th International Conference on Pattern Recognition (ICPR), Vol. 2, pp. 483–486, 2000. [Kang 03] Kang H.J. “Combining multiple classifiers based on third-order dependency for handwritten numeral recognition,” Pattern Recognition Letters, Vol. 24, pp. 3027–3036, 2003. [Kara 92] Karayiannis N.B., Venetsanopoulos A.N. “Fast learning algorithm for neural networks,” IEEE Transactions on Circuits and Systems,Vol. 39, pp. 453–474, 1992. [Kara 97] Karayiannis N.B., Mi G.W. “Growing radial basis neural networks. Merging supervised and unsupervised learning with network growth techniques,” IEEE Transactions on Neural Networks,Vol. 8(6), pp. 1492–1506, 1997. [Kear 94] Kearns M., Valiant L.G. “Cryptographic limitations on learning Boolean formulae and finite automata,” Journal of the ACM,Vol. 41(1), pp. 67–95, 1994. [Keer 05] Keerthi S.S., Duan K.B., Shevade S.K., Poo A.N. “A fast dual algorithm for kernel logistic regression,” Machine Learning,Vol. 61, pp. 151–165, 2005. [Kime 71] Kimeldorf G.S., Wahba G. “Some results on Tchebycheffian spline functions,” Journal of Mathematical Analysis and Applications,Vol. 33, pp. 88–95, 1971. [Kitt 03] Kittler J.,Alkoot F.M. “Sum versus vote fusion in multiple classifiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 25(1), pp. 110–115, 2003. [Kitt 98] Kittler J., Hatef M., Duin R. Matas J. “On combining classifiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 20(3), pp. 226–234, 1998. [Kivi 04] Kivinen J.K., Smola A.L., Williamson R.C. “Online learning with kernels,” IEEE Transactions on Signal Processing,Vol. 52(8), pp. 2165–2176, 2004. [Klei 90] Kleinberg R.M. “Stochastic discrimination,” Annals of Mathematics and Artificial Intelligence,Vol. 1, pp. 207–239, 1990. [Koli 97] Kolinummi P.,HamalainenT.,Kaski K.“Designing a digital neurocomputer,”IEEE Circuits and Systems Magazine,Vol. 13(2), pp. 19–27, 1997. [Kout 94] Koutroumbas K., Kalouptsidis N. “Nearest neighbor pattern classification neural networks,” Proc. of IEEE World Congress of Computational Intelligence, pp. 2911–2915, Orlando, FL, July 1994. [Kram 89] Kramer A.H., Sangiovanni-Vincentelli A. “Efficient parallel learning algorithms for neural networks,” in Advances in Neural Information Processing Systems 1 (Touretzky D.S., ed.), pp. 40–48, Morgan Kaufmann, 1989. [Kunc 01] Kuncheva L.I., Bezdek J.C., Duin R.P.W. “Decision templates for multiple classifier fusion: an experimental comparison,” Pattern Recognition,Vol. 34, pp. 299–314, 2001. [Kunc 02] Kunchera L.I. “A theoretical study on six classifier fusion strategies,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 24(2), pp. 281–286, 2002. [Kunc 03] Kuncheva L.I., Whitaker C.J., Shipp C.A., Duin R.P.W. “Limits on the majority vote accuracy in classifier fusion,” Pattern Analysis and Applications,Vol. 6, pp. 22–31, 2003. [Kunc 03a] Kuncheva L.I.,Whitaker C.J. “Measures of diversity in classifier ensembles,” Machine Learning,Vol. 51, pp. 181–207, 2003. [Lam 97] Lam L., Suen Y. “Application of majority voting to pattern recognition: An analysis of its behaviour and performance,” IEEE Transactions on Systems, Man, and Cybernetics Vol. 27(5), pp. 553–568, 1997.
“06-Ch04-SA272” 18/9/2008 page 255
References
[Lane 91] Lane S.H., Flax M.G., Handelman D.A., Gelfand J.J. “Multilayer perceptrons with B-spline receptive field functions,” in Advances in Neural Information Processing Systems 3 (Lippmann R.P., Moody J.,Touretzky D.S., eds.), pp. 684–692, Morgan Kaufmann, 1991. [Lecu 89] Le Cun Y., Boser B., Denker J.S., Henderson D., Howard R.E., Hubbard W., Jackel L.D. “Backpropagation applied to handwritten zip code recognition,” Neural Computation, Vol. 1(4), pp. 541–551, 1989. [Lecu 90] Le Cun Y., Denker J.S., Solla S.A. “Optimal brain damage,” in Advances in Neural Information Systems 2 (Touretzky D.S., ed.), pp. 598–605, Morgan Kaufmann, 1990. [Lee 04] Lee M.S., Keerthi S.S., Ong C.J. “An efficient method for computing Leave-One-Out error in support vector machines with Gaussian kernels,” IEEE Transactions on Neural Networks, Vol. 15(3), pp. 750–757, 2004. [Levi 02] Levitin G. “Evaluating correct classification probability for weighted voting classifiers with plurality voting,”European Journal of Operational Research,Vol. 141,pp. 596–607,2002. [Lin 03] Lin X., Yacoub S., Burns J., Simske S. “Performance analysis of pattern classifier combination by plurality voting,” Pattern Recognition Letters,Vol. 24, pp. 1959–1969, 2003. [Lin 02] Lin Y., Wahba G., Zhang H., Lee Y. “Statistical properties and adaptive tuning of support vector machines,” Machine Learning,Vol. 48, pp. 115–136, 2002. [Lin 02a] Lin Y., Lee Y., Wahba G. “Support vector machines for classification in nonstandard situations,” Machine Learning,Vol. 46, pp. 191–202, 2002. [Lipp 87] Lippmann R.P. “An introduction to computing with neural networks,” IEEE ASSP Magazine,Vol. 4(2), pp. 4–22, 1987. [Mack 92a] MacKay D.J.C. “A practical Bayesian framework for backpropagation networks,” Neural Computation,Vol. 4(3), pp. 448–472, 1992. [Mack 92b] MacKay D.J.C. “The evidence framework applied to classification networks,” Neural Computation,Vol. 4(5), pp. 720–736, 1992. [Maso 00] Mason L., Baxter J., Bartleet P., Frean M. “Boosting algorithms as gradient descent,” in Neural Information Processing Systems,Vol. 12, 2000. [Mavr 07] Mavroforakis M., Sdralis M.,Theodoridis S.“A geometric nearest point algorithm for the efficient solution of the SVM classification task,” IEEE Transactions on Neural Networks,Vol. 18(5), pp. 1545–1550, 2007. [Meye 03] Meyer D., Leisch F., Hornik K. “The support vector machine under test,” Neurocomputing,Vol. 55, pp. 169–186, 2003. [Meza 89] Mezard M., Nadal J.P. “Learning in feedforward layered networks: The tilling algorithm,” Journal of Physics,Vol. A 22, pp. 2191–2203, 1989. [Mich 94] Michie D., Spiegelhalter D.J., Taylor C.C., eds. Machine Learning, Neural, and Statistical Classification, Ellis Horwood Ltd., London, 1994. [Mill 96] Miller D., Rao A., Rose K., Gersho A. “A global optimization technique for statistical classifier design,” IEEE Transactions on Signal Processing,Vol. 44(12), pp. 3108–3122, 1996. [Mill 99] Miller D.J.,Yan L. “Critic-driven ensemble classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 47(10), pp. 2833–2844, 1999. [Mirc 89] Mirchandini G., Cao W. “On hidden nodes in neural nets,”IEEE Transactions on Circuits and Systems,Vol. 36(5), pp. 661–664, 1989. [Mood 89] Moody J., Darken C.J. “Fast learning in networks of locally tuned processing units,” Neural Computation,Vol. 6(4), pp. 281–294, 1989.
255
“06-Ch04-SA272” 18/9/2008 page 256
256
CHAPTER 4 Nonlinear Classifiers
[Mulg 96] Mulgrew B. “Applying radial basis functions,” IEEE Signal Processing Magazine, Vol. 13(2), pp. 50–65, 1996. [Mull 01] Müller R.M., Mika S., Rätsch G.,Tsuda K., Schölkopf B. “An introduction to kernel-based learning algorithms,” IEEE Transactions on Neural Networks, Vol. 12(2), pp. 181–201, 2001. [Murp 90] Murphy O.J. “Nearest neighbor pattern classification perceptrons,” Proceedings of the IEEE,Vol. 78(10), October 1990. [Nara 05] Narasimhamurthy A. “Theoretical bounds of majority voting performance for a binary classification problem,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27(12), pp. 1988–1995, 2005. [Nede 93] Nedeljkovic V. “A novel multilayer neural networks training algorithm that minimizes the probability of classification error,” IEEE Transactions on Neural Networks,Vol. 4(4), pp. 650–659, 1993. [Pado 95] Pados D.A., Papantoni-Kazakos P. “New non least squares neural network learning algorithms for hypothesis testing,” IEEE Transactions on Neural Networks, Vol. 6, pp. 596–609, 1995. [Palm 91] Palmieri F., Datum M., Shah A., Moiseff A. “Sound localization with a neural network trained with the multiple extended Kalman algorithm,” International Joint Conference on Neural Networks,Vol. 1, pp. 125–131, Seattle, 1991. [Pare 00] Parekh R., Yang J., Honavar V. “Constructive neural network learning algorithms for pattern classification,” IEEE Transactions on Neural Networks,Vol. 11(2), pp. 436–451, 2000. [Park 91] Park J., Sandberg I.W. “Universal approximation using radial basis function networks,” Neural Computation,Vol. 3(2), pp. 246–257, 1991. [Park 93] Park J., Sandberg I.W. “Approximation and radial basis function networks,” Neural Computation,Vol. 5(2), pp. 305–316, 1993. [Park 94] ParkY.“A comparison of neural net classifiers and linear tree classifiers: Their similarities and differences,” Pattern Recognition,Vol. 27(11), pp. 1493–1503, 1994. [Pera 03] Perantonis S.J. “Neural networks: nonlinear optimization for constrained learning and its applications,” Proceedings of NOLASC 2003, pp. 589–594, Athens, Greece, December 27–29, 2003. [Pera 00] Perantonis S.J., Ampazis N., Virvilis V. “A learning framework for neural networks using constrained optimization methods,” Annals of Operations Research, Vol. 99, pp. 385–401, 2000. [Pera 95] Perantonis S.J., Karras D.A. “An efficient learning algorithm with momentum acceleration,” Neural Networks,Vol. 8, pp. 237–249, 1995. [Pera 92] Perantonis S.J., Lisboa P.J.G. “Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiers,” IEEE Transactions on Neural Networks,Vol. 3(2), pp. 241–251, 1992. [Pikr 08] Pikrakis A., Ganakopoulos T., Theodoridis S. “A speech/music discriminator of radio recordings based on dynamic programming and Bayesian networks. IEEE Transactions mulitmedia,Vol. 10(5), pp. 846–856, 2008. [Plat 91] Platt J. “A resource allocating network for function interpolation,” Neural Computation, Vol. 3, pp. 213–225, 1991.
“06-Ch04-SA272” 18/9/2008 page 257
References
[Plat 99] Platt J. “Fast training of support vector machines using sequential minimal optimization,” in Advances in Kernel Methods: Support Vector Learning (Scholkopf B., Burges C.J.C., Smola A.J. eds.), pp. 185–208, MIT Press, 1999. [Quin 93] Quinlan J.R. C4.5: Programs for Machine Learning, Morgan Kaufmann, 1993. [Rats 02] Rätsch G., Mika S., Schölkopf B., Müller K.R. “Constructing boosting algorithms from SVMS:An application to one class classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 24(9), pp. 1184–1199, 2002. [Rats 05] Rätsch G., Warmuth M.K. “Efficient margin maximizing with boosting,” Journal of Machine Learning Research,Vol. 6, pp. 2131–2152, 2005. [Refe 91] Refenes A., Chen L. “Analysis of methods for optimal network construction,” University College London Report, CC30/080:DCN, 1991. [Reyn 99] Reyneri L. “Unification of neural and wavelet networks and fuzzy systems,” IEEE Transactions on Neural Networks,Vol. 10(4), pp. 801–814, 1999. [Rich 91] Richard M., Lippmann R.P. “Neural network classifiers estimate Bayesian a posteriori probabilities,” Neural Computation,Vol. 3, pp. 461–483, 1991. [Rico 88] Ricotti L.P., Ragazinni S., Martinelli G. “Learning of word stress in a suboptimal second order backpropagation neural network,”in Proceedings of the IEEE International Conference on Neural Networks,Vol. 1, pp. 355–361, San Diego, 1988. [Ride 97] Ridella S., Rovetta S., Zunino R. “Circular backpropagation networks for classification,” IEEE Transactions on Neural Networks,Vol. 8(1), pp. 84–97, 1997. [Ried 93] Riedmiller M., Brau H. “A direct adaptive method for faster backpropagation learning: The rprop algorithm,”Proceedings of the IEEE Conference on Neural Networks,San Francisco, 1993. [Ripl 94] Ripley B.D. “Neural networks and related methods for classification,” Journal of Royal Statistical Society,Vol. B, 56(3), pp. 409–456, 1994. [Ripl 96] Ripley B.D. Pattern Recognition and Neural Networks, Cambridge University Press, 1996. [Rodr 06] Rodriguez J.J., Kuncheva L.I., Alonso C.J. “Rotation forests: A new classifier ensemble method,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28(10), pp. 1619–1631, 2006. [Rome 97] Romero R.D., Touretzky D.S., Thibadeau G.H. “Optical character recognition using probabilistic neural networks,” Pattern Recognition,Vol. 3, pp. 1279–1292, 1997. [Rose 96] Rosen B.E. “Ensemble learning using decorrelated neural networks,” Connections Science Vol. 8(3), pp. 373–384, 1996. [Rume 86] Rumelhart D.E., Hinton G.E.,Williams R.J. “Learning internal representations by error propagation,” Parallel Distributed Processing: Explorations in the Microstructures of Cognition (Rumelhart D.E., McClelland J.L., eds.),Vol. 1, pp. 318–362, MIT Press, 1986. [Russ 93] Russell R. “Pruning algorithms. A survey,” IEEE Transactions on Neural Networks, Vol. 4(5), pp. 740–747, 1993. [Rutk 04] Rutkowski L. “Adaptive probabilistic neural networks in time-varying environments,” IEEE Transactions on Neural Networks,Vol. 15, pp. 811–827, 2004. [Saye 04] Sayed A. Fundamentals of Adaptive Filtering, John. Wiley & Sons, 2003. [Scha 98] Schapire R.E., Freund V., Bartlett P., Lee W.S. “Boosting the margin: A new explanation for the effectiveness of voting methods,”The Annals of Statistics,Vol. 26(5), pp. 1651–1686, 1998.
257
“06-Ch04-SA272” 18/9/2008 page 258
258
CHAPTER 4 Nonlinear Classifiers
[Scha 05] Schapire R.E., Rochery M., Rahim M., Gupta N. “Boosting with prior knowledge for call classification,” IEEE Transactions on Speech and Audio Processing, Vol. 13(2), pp. 174–181, 2005. [Scha 99] Schapire R.E., Singer Y. “Improved boosting algorithms using confidence-rated predictions,” Machine Learning,Vol. 37(3), pp. 297–336, 1999. [Scho 97] Schölkopf B., Sung K.-K., Burges C.J.C., Girosi F., Niyogi P., Poggio T.,Vapnic V. “Comparing support vector machines with Gaussian kernels to RBF classifiers,” IEEE Transactions on Signal Processing,Vol. 45(11), pp. 2758–2766, 1997. [Scho 02] Schölkoph B., Smola A.J. Learning with Kernels, MIT Press , 2002. [Seba 00] Sebald D.J.,Bucklew J.A.“Support vector machine techniques for nonlinear equalization,” IEEE Transactions on Signal Processing,Vol. 48(11), pp. 3217–3227, 2000. [Seth 90] Sethi I.K. “Entropy nets: From decision trees to neural networks,” Proceedings of the IEEE,Vol. 78, pp. 1605–1613, 1990. [Seth 91] Sethi I.K. “Decision tree performance enhancement using an artificial neural network interpretation,” in Artificial Neural Networks and Statistical Pattern Recognition (Sethi I., Jain A., eds.), Elsevier Science Publishers, 1991. [Shaw 04] Shawe-Taylor J., Cristianini N. Kernel Methods for Pattern Analysis, Cambridge University Press, 2004. [Siet 91] Sietsma J., Dow R.J.F. “Creating artificial neural networks that generalize,” Neural Networks,Vol. 4, pp. 67–79, 1991. [Silv 90] Silva F.M., Almeida L.B. “Accelaration technique for the backpropagation algorithm,” Proceedings of the EURASIP Workshop on Neural Networks (Almeida L.B. et al., eds.), pp. 110–119, Portugal, 1990. [Sing 89] Singhal S.,Wu L. “Training feedforward networks with the extended Kalman filter,” Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing, pp. 1187–1190, Glasgow, 1989. [Slav 08] Slavakis K., Theodoridis S., I. Yamada “Online Kernel-Based Classification and Adaptive Projection Algorithms,” IEEE Transactions on Signal Processing, Vol. 56(7), pp. 2781–2797, 2008. [Slav 08a] Slavakis K., Theodoridis S. “Sliding Window Generalized Kernel Affine Projection Algorithm using Projection Mappings,” EURASIP Journal on Advances on Signal Processing, JASP, To appear 2008. [Spec 90] Specht D. “Probabilistic neural networks,” Neural Networks,Vol. 3, pp. 109–118, 1990. [Stre 94] Streit R.L.,LuginbuhlT.E.“Maximum likelihood training of probabilistic neural networks,” IEEE Transactions on Neural Networks,Vol. 5, pp. 764–783, 1994. [Spec 90] Specht D.F.“Probabilistic neural networks,”Neural Networks,Vol. 3, pp. 109–118, 1990. [Suar 99] Suarez A., Lutsko J.F. “Globally optimal fuzzy decision trees for classification and regression,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21(12), pp. 1297–1311, 1999. [Sun 07] Sun Y., Kamel M.S., Wong A.K.C., Wang T. “Cost-effective boosting for classification of imbalanced data,” Pattern Recognition,Vol. 40, pp. 3358–3378, 2007. [Tax 00] Tax D.M.J., Breukelen M., Duin R.P.W., Kittler J. “Combining multiple classifiers by averaging or by multiplying?” Pattern Recognition,Vol. 33, pp. 1475–1485, 2000.
“06-Ch04-SA272” 18/9/2008 page 259
References
[Theo 95] Theodoridis S., Cowan C.F.N., Callender C., Lee C.M.S. “Schemes for equalization in communication channels with nonlinear impairments,”IEE Proceedings on Communications, Vol. 61(3), pp. 268–278, 1995. [Tipp 01] Tipping M.E. “Sparse Bayesiay learning and the relevance vector machine,” Journal of Machine Learning Research,Vol. 1, pp. 211–244, 2001. [Tres 01] Tresp V. “Committee Machines,” in Handbook for Neural Network Signal Processing (Hu Y.H., Hwang J.N., eds), CRC Press, 2001. [Tume 95] Tumer K., Ghosh J. “Analysis of decision boundaries in linearly combined classifiers,” Pattern Recognition,Vol. 29(2), pp. 341–348, 1995. [Ueda 97] Ueda N. “Optimal linear combination of neural networks for improving classification performance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22(2), pp. 207–215, 2000. [Vali 84] Valiant L.G. “A theory of the learnable,” Communications of the ACM, Vol. 27(11), pp. 1134–1142, 1984. [Vapn 00] Vapnik V.N. The Nature of Statistical Learning Theory, Springer Verlag, 2000. [Viol 01] Viola P., Jones M. “Robust real-time object detection,” Proceedings IEEE Workshop on Statistical and Computational Theories of Vision,Vancouver, Canada, 2001. [Watr 88] Watrous R.L. “Learning algorithms for connectionist networks: Applied gradient methods of nonlinear optimization,” in Proceedings of the IEEE International Conference on Neural Networks,Vol. 2, pp. 619–627, San Diego, 1988. [Wein 90] Weigend A.S., Rumelhart D.E., Huberman B.A. “Backpropagation, weight elimination and time series prediction,” in Proceedings of the Connectionist Models Summer School (Touretzky D., Elman J., Sejnowski T., Hinton G., eds.), pp. 105–116, 1990. [Weis 03] Weiss G, Provost F. “Learning when training data are costly: the effect of class distribution on tree induction,” Journal of Artificial Intelligence Research, Vol. 19, pp. 315–354, 2003. [Werb 74] Werbos P.J. “Beyond regression: New tools for prediction and analysis in the behavioral sciences,” Ph.D. Thesis, Harvard University, Cambridge, MA, 1974. [Wett 92] Wettschereck D., Dietterich T. “Improving the performance of radial basis function networks by learning center locations,” in Advances in Neural Information Processing Systems, 4th ed. (Moody J.E.,Hanson S.J.,Lippmann R.P.,eds.),pp. 1133–1140,Morgan Kaufmann,1992. [Witt 00] Witten I., Frank E. Data Mining: Practical Machine Learning Tools and Techniques with JAVA Implementations, Morgan Kaufmann, 2000. [Wolpe 92] Wolpet D.H. “Stacked generalization,” Neural Networks,Vol. 5(2), pp. 241–260, 1992. [Yang 06] Yang Z.R. “A novel radial basis function neural network for discriminant analysis,” IEEE Transactions on Neural Networks,Vol. 17(3), pp. 604–612, 2006. [Yin 05] Yin X.C., Liu C.P., Han Z. “Feature combination using boosting,” Pattern Recognition Letters,Vol. 25(14), pp. 2195–2205, 2005. [Ying 98] Yingwei L., Sundararajan N., Saratihandran P. “Performance evaluation of a sequential minimal RBF neural network learning algorithm,” IEEE Transactions on Neural Networks, Vol. 9(2), pp. 308–318, 1998.
259
“06-Ch04-SA272” 18/9/2008 page 260
260
CHAPTER 4 Nonlinear Classifiers
[Zhan 00] Zhang G.P. “Neural networks for classification: A survey,” IEEE Transactions on Systems Man and Cybernetics – Part C,Vol. 30(4), pp. 451–462, 2000. [Zhou 06] Zhou Z.H., Liu X.Y. “Training cost sensitive neural networks with methods addressing the class imbalance problem,” IEEE Transactions on Knowledge Data Engineering, Vol. 18(1), pp. 63–77, 2006. [Zura 92] Zurada J. Introduction to Artificial Neural Networks, West Publishing Company, St. Paul, MN., 1992.
“07-Ch05-SA272” 17/9/2008 page 261
CHAPTER
Feature Selection
5
5.1 INTRODUCTION In all previous chapters, we considered the features that should be available prior to the design of the classifier. The goal of this chapter is to study methodologies related to the selection of these variables. As we pointed out very early in the book, a major problem associated with pattern recognition is the so-called curse of dimensionality (Section 2.5.6). The number of features at the disposal of the designer of a classification system is usually very large. As we will see in Chapter 7, this number can easily reach the order of a few dozens or even hundreds. There is more than one reason to reduce the number of features to a sufficient minimum. Computational complexity is the obvious one. A related reason is that, although two features may carry good classification information when treated separately, there is little gain if they are combined into a feature vector because of a high mutual correlation. Thus, complexity increases without much gain. Another major reason is that imposed by the required generalization properties of the classifier, as discussed in Section 4.9 of Chapter 4. As we will state more formally at the end of this chapter,the higher the ratio of the number of training patterns N to the number of free classifier parameters, the better the generalization properties of the resulting classifier. A large number of features are directly translated into a large number of classifier parameters (e.g., synaptic weights in a neural network, weights in a linear classifier). Thus, for a finite and usually limited number N of training patterns, keeping the number of features as small as possible is in line with our desire to design classifiers with good generalization capabilities. Furthermore, the ratio N /l enters the scene from another nearby corner. One important step in the design of a classification system is the performance evaluation stage,in which the classification error probability of the designed classifier is estimated. We not only need to design a classification system,but we must also assess its performance. As is pointed out in Chapter 10, the classification error estimate improves as this ratio becomes higher. In [Fine 83] it is pointed out that in some cases ratios as high as 10 to 20 were considered necessary.
261
“07-Ch05-SA272” 17/9/2008 page 262
262
CHAPTER 5 Feature Selection
The major task of this chapter can now be summarized as follows. Given a number of features, how can one select the most important of them so as to reduce their number and at the same time retain as much as possible of their class discriminatory information? The procedure is known as feature selection or reduction. This step is very crucial. If we selected features with little discrimination power, the subsequent design of a classifier would lead to poor performance. On the other hand, if information-rich features are selected, the design of the classifier can be greatly simplified. In a more quantitative description,we should aim to select features leading to large between-class distance and small within-class variance in the feature vector space. This means that features should take distant values in the different classes and closely located values in the same class. To this end, different scenarios will be adopted. One is to examine the features individually and discard those with little discriminatory capability. A better alternative is to examine them in combinations. Sometimes the application of a linear or nonlinear transformation to a feature vector may lead to a new one with better discriminatory properties. All these paths will be our touring directions in this chapter. Finally, it must be pointed out that there is some confusion in the literature concerning the terminology of this stage. In some texts the term feature extraction is used, but we feel that this may be confused with the feature generation stage treated in Chapter 7. Others prefer to call it a preprocessing stage. We have kept the latter term to describe the processing performed on the features prior to their utilization. Such processing involves removing outliers, scaling of the features to safeguard comparable dynamic range of their respective values, treating missing data, and so forth.
5.2 PREPROCESSING 5.2.1 Outlier Removal An outlier is defined as a point that lies very far from the mean of the corresponding random variable. This distance is measured with respect to a given threshold, usually a number of times the standard deviation. For a normally distributed random variable, a distance of two times the standard deviation covers 95% of the points, and a distance of three times the standard deviation covers 99% of the points. Points with values very different from the mean value produce large errors during training and may have disastrous effects. These effects are even worse when the outliers are the result of noisy measurements. If the number of outliers is very small, they are usually discarded. However, if this is not the case and they are the result of a distribution with long tails, then the designer may have to adopt cost functions that are not very sensitive in the presence of outliers. For example, the least squares criterion is very sensitive to outliers, because large errors dominate the cost function due to the squaring of the terms. A review of related techniques that attempt to address such problems is given in [Hube 81].
“07-Ch05-SA272” 17/9/2008 page 263
5.2 Preprocessing
5.2.2 Data Normalization In many practical situations a designer is confronted with features whose values lie within different dynamic ranges. Thus, features with large values may have a larger influence in the cost function than features with small values,although this does not necessarily reflect their respective significance in the design of the classifier. The problem is overcome by normalizing the features so that their values lie within similar ranges. A straightforward technique is normalization via the respective estimates of the mean and variance. For N available data of the kth feature we have x¯ k ⫽
N 1 xik , N
k ⫽ 1, 2, . . . , l
i⫽1
k2 ⫽
N 1 (xik ⫺ x¯ k )2 N ⫺1 i⫽1
xˆ ik ⫽
xik ⫺ x¯ k k
In words, all the resulting normalized features will now have zero mean and unit variance. This is obviously a linear method. Other linear techniques limit the feature values in the range of [0, 1] or [⫺1, 1] by proper scaling. Besides the linear methods, nonlinear methods can also be employed in cases in which the data are not evenly distributed around the mean. In such cases transformations based on nonlinear (i.e., logarithmic or sigmoid) functions can be used to map data within specified intervals. The so-called softmax scaling is a popular candidate. It consists of two steps y⫽
xik ⫺ x¯ k 1 , xˆ ik ⫽ rk 1 ⫹ exp(⫺y)
(5.1)
This is basically a squashing function limiting data in the range of [0, 1]. Using a series expansion approximation, it is not difficult to see that for small values of y this is an approximately linear function with respect to xik . The range of values of xik that correspond to the linear section depends on the standard deviation and the factor r, which is user defined. Values away from the mean are squashed exponentially.
5.2.3 Missing Data In practice, certain features may be missing from some feature vectors. Such incomplete-data cases are common in social sciences due, for example, to partial response in surveys. Remote sensing is another area where incomplete-data may occur when certain regions are covered by a subset of sensors. Sensor networks, which rely on a set of distributed information sources and on the data fusion from a number of sensors, is also a discipline where incomplete-data may arise. The most traditional techniques in dealing with missing data include schemes that “complete”the missing values by (a) zeros or (b) the unconditional mean, computed from the available values of the respective feature or (c) the conditional
263
“07-Ch05-SA272” 17/9/2008 page 264
264
CHAPTER 5 Feature Selection
mean, if one has an estimate of the pdf of the missing values given the observed data. Completing the missing values in a set of data is also known as imputation. Another approach is to discard feature vectors with missing values. Although such a technique can be useful in cases of large data sets, in most cases it is considered a “luxury” to afford to drop available information. Since the mid-1970s ([Rubi 76]), a large research effort has been invested to cope efficiently with the missing data task, and a number of sophisticated methods have been developed and used successfully. A popular alternative to the previously exposed, rather naive approaches is known as imputing from a conditional distribution. The idea here is to impute by respecting the statistical nature of the missing values. Under this rationale, missing values are not replaced by statistical means or zeros but by a random draw from a distribution. Let us denote the complete feature vector as x com and assume that some of its components are missing (x mis ) and the rest are observed (x obs ). Then the complete feature vector is written as
x com
x obs ⫽ x mis
Under the assumption that the probability of missing a value does not depend on the value itself (this is known as the missing at random (MAR) assumption), imputing from a conditional distribution means to simulate a draw from the following conditional pdf p(x mis |x obs ; ) ⫽
where
p(x obs , x mis ; ) p(x obs ; )
(5.2)
p(x obs ; ) ⫽
p(x com ; )dx mis
(5.3)
where is an unknown set of parameters. In practice, an estimate ˆ of must first be obtained from x obs . The celebrated EM algorithms (Chapter 2) is a popular choice for parameter estimation under the missing data setting, for example, [Ghah 94, Tsud 03]. The multiple imputation (MI) procedure ([Rubi 87]) is one step beyond the previous methodology, usually referred to as single imputation (SI). In MI, for each missing value, m ⬎ 1 samples are generated. The results are then combined appropriately so that certain statistical properties are fulfilled. MI can be justified as an attempt to overcome uncertainties associated with the estimation of the parameˆ one can use ter . Hence, instead of drawing a single point from p(x mis |x obs ; ) different parameters, ˆ i , i ⫽ 1, 2, . . . , m, and draw the m samples from p(x mis |x obs ; ˆ i ), i ⫽ 1, 2, . . . , m
A way to approach this problem is via Bayesian inference arguments (Chapter 2), where the unknown parameter vector is treated as a random one described by a posterior probability, see, for example, [Gelm 95].
“07-Ch05-SA272” 17/9/2008 page 265
5.3 The Peaking Phenomenon
In a more recent paper ([Will 07]), the missing data problem is treated in the context of logistic regression classification (Section 3.6) and explicit imputation is bypassed. This is achieved by integrating out the missing values and predicting the binary class label, yi , of the ith pattern based on the value of
P( yi |x i,obs ) ⫽
P( yi |x i,obs , x i,mis )p(x i,mis |x i,obs )dx i,mis
Under the assumption that p(x i,mis |x i,obs ) is sufficiently modeled by a Gaussian mixture model (Section 2.5.5) the previous integration can be performed analytically. For more on the problem of missing data the interested reader may refer to the excellent review article [Scha 02]. We will return to the missing data problem in Chapter 11.
5.3 THE PEAKING PHENOMENON As stated in the introduction of this chapter, in order to design a classifier with good generalization performance, the number of training points, N , must be large enough with respect to the number of features, l, that is, the dimensionality of the feature space. Take as an example the case of designing a linear classifier, w T x ⫹ w0 . The number of the unknown parameters is l ⫹ 1. In order to get a good estimate of these parameters, the number of data points must be larger than l ⫹ 1. The larger the N the better the estimate, since we can filter out the effects of the noise and also minimize the effects of the outliers. In [Trun 79], an elegant simple example has been given that reveals the interplay between the number of features and the size of the training data set and elucidates the way these two parameters influence the performance of a classifier. Consider a two-class classification task with equal prior probabilities,P(1 ) ⫽ P(2 ) ⫽ 12 ,in the l-dimensional space. Both classes, 1 , 2 , are described by Gaussian distributions of the same covariance matrix ⌺ ⫽ I , where I is the identity matrix and mean values and ⫺, respectively, where ⫽
1, √1 , √13 , . . . , √1 2
l
T
(5.4)
Since the features are jointly Gaussian and ⌺ ⫽ I ,the involved features are statistically independent (see Appendix A.9). Moreover, the optimal Bayesian rule is equivalent to the minimum Euclidean distance classifier. Given an unknown feature vector x, we classify it to, say, 1 if ||x ⫺ ||2 ⬍ ||x ⫹ ||2
or after performing the algebra, if z ≡ xT ⬎ 0
If z ⬍ 0, we decide in favor of the class 2 . Thus, the decision relies on the value of the inner product z. In the sequel we will consider two cases.
265
“07-Ch05-SA272” 17/9/2008 page 266
266
CHAPTER 5 Feature Selection
Known Mean Value The inner product z, being a linear combination of independent Gaussian variables, is also a Gaussian variable (see, e.g., [Papo 91]) with mean value E[z] ⫽ ||||2 ⫽ li⫽1 1i and variance z2 ⫽ ||||2 (Problem 5.1). The probability of committing an error turns out to be equal to (Problem 5.1) Pe ⫽
⬁
bl
z2 1 exp ⫺ dz √ 2 2
(5.5)
where l 1 bl ⫽ i
(5.6)
i⫽1
Note that the series in (5.6) tends to infinity as l ⫺→⬁; hence the probability of error tends to zero as the number of features increases.
Unknown mean value In this case, the mean value has to be estimated from the training data set. Adopting the maximum likelihood estimate we obtain ˆ ⫽
N 1 sk x k N k⫽1
where sk ⫽ 1 if x k ∈ 1 and sk ⫽ ⫺1 if x k ∈ 2 . Decisions are taken depending ˆ However, z is no more a Gaussian variable, since on the inner product z ⫽ x T . ˆ is not a constant but a random vector. By the definition of the inner product, z ⫽ li⫽1 xi ˆ i and for large enough l and the central limit theorem (Appendix A) z can be considered approximately Gaussian. Its mean value and variance are given by (Problem 5.2) E[z] ⫽
l 1 i⫽1
i
(5.7)
and
l l 1 1 ⫹ z2 ⫽ 1 ⫹ N i N
(5.8)
i⫽1
The probability of error is given by (5.5) with bl ⫽
E[z] z
(5.9)
It can now be shown that bl ⫺→0 as l ⫺→⬁ and the probability of error tends to for any finite number N (Problem 5.2).
1 2
“07-Ch05-SA272” 17/9/2008 page 267
5.3 The Peaking Phenomenon
The above example demonstrates that: ■
If for any l the corresponding pdf is known,then we can perfectly discriminate the two classes by arbitrarily increasing the number of features.
■
If the pdfs are not known and the associated parameters must be estimated using a finite training set, then the arbitrary increase of the number of features leads to the maximum possible value of the error rate, that is, Pe ⫽ 0.5. This implies that under a limited number of training data we must try to keep the number of features to a relatively low number.
In practice, for a finite N , by increasing the number of features one obtains an initial improvement in performance, but after a critical value further increase of the number of features results in an increase of the probability of error. This phenomenon is also known as the peaking phenomenon. Figure 5.1 illustrates the general trend that one expects to experience in practice by playing with the number of features,l,and the size of the training data set,N . For N2 ⬎⬎ N1 ,the error values corresponding to N2 are lower than those resulting for N1 , and the peaking phenomenon occurs for a value l2 ⬎ l1 . For each value of N , the probability of error starts decreasing with increasing l till a critical value where the error starts increasing. The minimum in the curves occurs at some number l ⫽ N␣ , where ␣, usually, takes values in the range of 2 to 10. Consequently, in practice, for a small number of training data, a small number of features must be used. If a large number of training data is available, a larger number of features can be selected that yield better performance. Although the above scenario covers a large set of “traditional” classifiers, it is not always valid. We have already seen that adopting an appropriate kernel function to design a nonlinear SVM classifier implies a mapping to a high-dimensional space,
Pe N ⫽ N1
l1
l2
N ⫽ N2
l
FIGURE 5.1 For a given value of N , the probability of error decreases as the number of features increases till a critical value. Further increase in the number of features forces the error probability to increase. If the number of points increases, N2 ⬎⬎ N1 , the peaking phenomenon occurs for larger values, l 2 ⬎ l1 .
267
“07-Ch05-SA272” 17/9/2008 page 268
268
CHAPTER 5 Feature Selection
which can even be of infinite dimension. In spite of the fact that one now works in almost empty spaces (N is much less than the dimensionality of the space), the generalization performance of the SVM classifiers can be very good. The secret to that was disclosed to us fairly recently. It is not the number of parameters that really controls the generalization performance, under finite N , but another quantity. For some types of classifiers,this quantity is directly related to the number of parameters to be estimated and the dimensionality of the feature space. However, for some classifiers, such as the SVM, this quantity can be controlled independent of the dimensionality of the feature space. These issues are discussed at the end of this chapter. More on the peaking phenomenon and the small sample size problem can be found in for example, [Raud 80, Raud 91, Duin 00].
5.4 FEATURE SELECTION BASED ON STATISTICAL HYPOTHESIS TESTING A first step in feature selection is to look at each of the generated features independently and test their discriminatory capability for the problem at hand. Although looking at the features independently is far from optimal, this procedure helps us to discard easily recognizable “bad” choices and keeps the more elaborate techniques, which we will consider next, from unnecessary computational burden. Let x be the random variable representing a specific feature. We will try to investigate whether the values it takes for the different classes, say 1 , 2 , differ significantly. To give an answer to this question, we will formulate the problem in the context of statistical hypothesis testing. That is, we will try to answer which of the following hypotheses is correct: H1 : The values of the feature differ significantly H0 : The values of the feature do not differ significantly H0 is known as the null hypothesis and H1 as the alternative hypothesis. The decision is reached on the basis of experimental evidence supporting the rejection or not of H0 . This is accomplished by exploiting statistical information,and obviously any decision will be taken subject to an error probability. We will approach the problem by considering the differences of the mean values corresponding to a specific feature in the various classes, and we will test whether these differences are significantly different from zero. Let us first, however, refresh our memory with some basics from the statistics related to hypothesis testing.
5.4.1 Hypothesis Testing Basics Let x be a random variable with a probability density function, which is assumed to be known within an unknown parameter . As we have already seen in Chapter 2,
“07-Ch05-SA272” 17/9/2008 page 269
5.4 Statistical Hypothesis Testing
in the case of a Gaussian, this parameter may be the mean value or its variance. Our interest here lies in the following hypothesis test: H1 : ⫽ 0 H0 : ⫽ 0
The decision on this test is reached in the following context. Let xi , i ⫽ 1, 2, . . . , N , be the experimental samples of the random variable x. A function f (·, . . . , ·) is selected, depending on the specific problem, and let q ⫽ f (x1 , x2 , . . . , xN ). The function is selected so that the probability density function of q is easily parameterized in terms of the unknown , that is, pq (q; ). Let D be the interval of ¯ be its q in which it has a high probability of lying under hypothesis H0 . Let D complement, that is, the interval of low probability, also under hypothesis H0 . Then, if the value of q that results from the available samples, xi , i ⫽ 1, 2, . . . , N , ¯ we will reject it. D is known as lies in D we will accept H0 , and if it lies in D ¯ as the critical interval. The variable q is known the acceptance interval and D as test statistic. The obvious question now refers to the probability of reaching a wrong decision. Let H0 be true. Then the probability of an error in our decision is ¯ 0 ) ≡ . This probability is obviously the integral of pq (q|H0 ) (pq (q; o )) P(q ∈ D|H ¯ (Figure 5.2). In practice, we preselect this value of , which is known over D as the significance level, and we sometimes denote the corresponding critical ¯ (D ). Let us now apply this procedure in the case in (acceptance) interval as D which the unknown parameter is the mean of x.
The Known Variance Case Let x be a random variable and xi , i ⫽ 1, 2, . . . , N ,the resulting experimental samples, which we will assume to be mutually independent. Let E[x] ⫽ E[(x ⫺ )2 ] ⫽ 2 pq(q H0 )
D
D
D
q
FIGURE 5.2 Acceptance and critical regions for hypothesis testing. The area of the shaded region is the probability of an erroneous decision.
269
“07-Ch05-SA272” 17/9/2008 page 270
270
CHAPTER 5 Feature Selection
A popular estimate of based on the known samples is the sample mean x¯ ⫽
N 1 xi N i⫽1
Using a different set of N samples, a different estimate will result. Thus, x¯ is also a ¯ random variable, and it is described in terms of a probability density function px¯ ( x). The corresponding mean is N N 1 1 E[x] ¯ ⫽ E xi ⫽ E[xi ] ⫽ N N i⫽1
(5.10)
i⫽1
Thus, x¯ is an unbiased estimate of the mean of x. The variance x2¯ of x¯ is ⎡
N 1 E[( x¯ ⫺ ) ] ⫽ E ⎣ xi ⫺ N 2
2 ⎤ ⎦
i⫽1
⫽
N 1 1 E[(xi ⫺ )2 ] ⫹ 2 E[(xi ⫺ )(xj ⫺ )] N2 N i⫽1
i
j⫽i
The statistical independence of the samples dictates E[(xi ⫺ )(xj ⫺ )] ⫽ E[xi ⫺ ]E[xj ⫺ ] ⫽ 0
Hence x2¯ ⫽
1 2 N
(5.11)
In words, the larger the number of measurement samples, the smaller the variance of x¯ around the true mean . Having now acquired the necessary ingredients, let us assume that we are given a value ˆ and we have to decide upon H1 : E[x] ⫽ ˆ H0 : E[x] ⫽ ˆ
To this end we define the test statistic q⫽
x¯ ⫺ ˆ √ / N
(5.12)
Recalling the central limit theorem from Appendix A, the probability density 2 ˆ is (approximately) the Gaussian N ( , ˆ N ) function of x¯ under H0 (i.e., given ) √ N
px¯ ( x) ¯ ⫽ √
N ( x¯ ⫺ ) ˆ 2 exp ⫺ 2 2
“07-Ch05-SA272” 17/9/2008 page 271
5.4 Statistical Hypothesis Testing
Table 5.1 Acceptance Intervals [⫺x , x ] Corresponding to Various Probabilities for an N (0, 1) Normal Distribution 1⫺ x
0.8
0.85
0.9
0.95
0.98
0.99
0.998
0.999
1.282
1.440
1.645
1.967
2.326
2.576
3.090
3.291
Hence, the probability density function of q under H0 is approximately N (0, 1). For a significance level the acceptance interval D ≡ [⫺x , x ], is chosen as the interval in which the random variable q lies with probability 1⫺ ( the probability ¯ This is readily provided from available tables. of being in D). An example for normally distributed N (0, 1) variables is given in Table 5.1. The decision on the test hypothesis can now be reached by the following steps. ■
Given the N experimental samples of x, compute x¯ and then q.
■
Choose the significance level .
■
■
Compute from the corresponding tables for N (0, 1) the acceptance interval D ⫽ [⫺x , x ], corresponding to probability 1 ⫺ . If q ∈ D decide H0 , if not decide H1 .
Basically,all we say is that we expect the resulting value q to lie in the high-percentage 1⫺ interval. If it does not, then we decide that this is because the assumed mean value is not “correct.” Of course, the assumed mean value may be correct, but it so happens that the resulting q lies in the least probable area because of the specific set of experimental samples available. In such a case our decision is erroneous, and the probability of committing such an error is . Example 5.1 Let us consider an experiment with a random variable x of ⫽ 0.23, and assume N to be equal to 16 and the resulting x¯ equal to 1.35. Adopt the significance level ⫽ 0.05. We will test if the hypothesis ˆ ⫽ 1.4 is true. From Table 5.1 we have x¯ ⫺ ˆ prob ⫺1.97 ⬍ ⬍ 1.97 ⫽ 0.95 0.23/4 prob ⫺0.113 ⬍ x¯ ⫺ ˆ ⬍ 0.113 ⫽ 0.95 Thus, since the value of , ˆ which we have assumed, is in the interval 1.237 ⫽ 1.35 ⫺ 0.113 ⬍ ˆ ⬍ 1.35 ⫹ 0.113 ⫽ 1.463 we accept it, as there is no evidence at the 5% level that the mean value is not equal to . ˆ The interval [1.237, 1463] is also known as the confidence interval at the 1⫺ ⫽ 0.95 level.
271
“07-Ch05-SA272” 17/9/2008 page 272
272
CHAPTER 5 Feature Selection
The Unknown Variance Case If the variance of x is not known, it must be estimated. The estimate ˆ 2 ≡
N 1 (xi ⫺ x) ¯ 2 N ⫺1
(5.13)
i⫽1
is an unbiased estimate of the variance. Indeed, E[ˆ 2 ] ⫽
N 1 E[(xi ⫺ x) ¯ 2] N ⫺1 i⫽1
⫽
N 1 E ((xi ⫺ ) ⫺ ( x¯ ⫺ ))2 N ⫺1 i⫽1
⫽
N 2 1 2 ⫹ ⫺ 2E[(xi ⫺ )( x¯ ⫺ )] N ⫺1 N i⫽1
Due to the independence of the experimental samples E[(xi ⫺ )( x¯ ⫺ )] ⫽
1 2 E (xi ⫺ ) (x1 ⫺ ) ⫹ · · · ⫹ (xN ⫺ ) ⫽ N N
Thus, E[ˆ 2 ] ⫽
N N ⫺1 2 ⫽ 2 N ⫺1 N
The test statistic is now defined as q⫽
x¯ ⫺ √ / ˆ N
(5.14)
However, this is no longer a Gaussian variable. Following Appendix A, and if we assume that x is a Gaussian random variable, then q is described by the socalled t-distribution with N ⫺ 1 degrees of freedom. Table 5.2 shows the confidence interval D ⫽ [⫺x , x ] for various significance levels and degrees of freedom of the t-distribution. Example 5.2 For the case of Example 5.1 let us assume that the estimate of the standard deviation ˆ is 0.23. Then, according to Table 5.2 for 15 degrees of freedom (N ⫽ 16) and significance level ⫽ 0.025 x¯ ⫺ ˆ prob ⫺2.49 ⬍ ⬍ 2.49 ⫽ 0.975 0.23/4 and the confidence interval for ˆ at the 0.975 level is 1.207 ⬍ ˆ ⬍ 1.493
“07-Ch05-SA272” 17/9/2008 page 273
5.4 Statistical Hypothesis Testing
Table 5.2 Interval Values at Various Significance Levels and Degrees of Freedom for a t-Distribution Degrees of Freedom
1⫺
0.9
0.95
0.975
0.99
0.995
10
1.81
2.23
2.63
3.17
3.58
11
1.79
2.20
2.59
3.10
3.50
12
1.78
2.18
2.56
3.05
3.43
13
1.77
2.16
2.53
3.01
3.37
14
1.76
2.15
2.51
2.98
3.33
15
1.75
2.13
2.49
2.95
3.29
16
1.75
2.12
2.47
2.92
3.25
17
1.74
2.11
2.46
2.90
3.22
18
1.73
2.10
2.44
2.88
3.20
19
1.73
2.09
2.43
2.86
3.17
20
1.72
2.09
2.42
2.84
3.15
5.4.2 Application of the t -Test in Feature Selection We will now see how all of this is specialized for the case of feature selection in a classification problem. Our major concern now will be to test, against zero, the difference 1 ⫺2 between the means of the values taken by a feature in two classes. Let xi , i ⫽ 1, 2, . . . , N , be the sample values of the feature in class 1 with mean 1 . Correspondingly, for the other class 2 we have yi , i ⫽ 1, 2, . . . , N , with mean 2 . Let us now assume that the variance of the feature values is the same in both classes, 12 ⫽ 22 ⫽ 2 . To decide about the closeness of the two mean values, we will test for the hypothesis H1 : ⌬ ⫽ 1 ⫺ 2 ⫽ 0 H0 : ⌬ ⫽ 1 ⫺ 2 ⫽ 0
(5.15)
To this end, let z ⫽x⫺y
(5.16)
where x, y denote the random variables corresponding to the values of the feature in the two classes 1 , 2 , respectively, for which statistical independence has been assumed. Obviously, E[z] ⫽ 1 ⫺ 2 , and due to the independence assumption z2 ⫽ 2 2 . Following arguments similar to those used before, we now have z¯ ⫽
N 1 (xi ⫺ yi ) ⫽ x¯ ⫺ y¯ N i⫽1
(5.17)
273
“07-Ch05-SA272” 17/9/2008 page 274
274
CHAPTER 5 Feature Selection
2
and for the known variance case z¯ follows the normal N (1 ⫺ 2 , 2 N ) distribution for large N . Thus,Table 5.1 can be used to decide about (5.15). If the variance is not known, then we choose the test statistic q⫽
( x¯ ⫺ y¯ ) ⫺ (1 ⫺ 2 ) sz N2
(5.18)
where
sz2 ⫽
N N 1 (xi ⫺ x) ¯ 2⫹ (yi ⫺ y¯ )2 2N ⫺ 2 i⫽1
i⫽1
s2 (2N ⫺2)
It can be shown that z 2 follows a chi-square distribution with 2N ⫺ 2 degrees of freedom (Appendix A and Problem 5.3). As is pointed out in Appendix A, if x, y are normally distributed variables of the same variance 2 , then the random variable q turns out to follow the t-distribution with 2N ⫺ 2 degrees of freedom. Thus, Table 5.2 has to be adopted for the test. When the available number of samples is not the same in all classes, a slight modification is required (Problem 5.4). Furthermore, in practice the variances may not be the same in the two classes. Sometimes this becomes the object of another hypothesis test, ˆ 2
concerning the ratio F ⫽ ˆ 12 , to check whether it is close to unity. It can be 2 shown that F , being the ratio of two chi-square distributed variables, follows the so-called F -distribution and the related tables should be used [Fras 58]. Finally, if the Gaussian assumption about x is not a valid one, other criteria can be used to check the equality hypothesis of the means, such as the Kruskal–Wallis statistic [Walp 78, Fine 83]. Example 5.3 The sample measurements of a feature in two classes are class 1 : class 2 :
3.5 3.2
3.7 3.6
3.9 3.1
4.1 3.4
3.4 3.0
3.5 3.4
4.1 2.8
3.8 3.1
3.6 3.3
3.7 3.6
The question is to check whether this feature is informative enough. If not, it will be discarded during the selection phase. To this end, we will test whether the values of the feature in the two classes differ significantly. We choose the significance level ⫽ 0.05. From the foregoing we have 1 : x¯ ⫽ 3.73
ˆ 12 ⫽ 0.0601
2 : y¯ ⫽ 3.25
ˆ 22 ⫽ 0.0672
“07-Ch05-SA272” 17/9/2008 page 275
5.5 The Receiver Operating Characteristics (ROC) Curve
For N ⫽ 10 we have sz2 ⫽
1 2 ( ˆ ⫹ ˆ 22 ) 2 1
q⫽
( x¯ ⫺ y¯ ⫺ 0) sz N2
q ⫽ 4.25 From Table 5.2 and for 20 ⫺ 2 ⫽ 18 degrees of freedom and significance level 0.05, we obtain D ⫽ [⫺2.10, 2.10]. Since 4.25 lies outside the interval D, we decide in favor of H1 ; that is, the mean values differ significantly at the 0.05 level. Hence, the feature is selected.
5.5 THE RECEIVER OPERATING CHARACTERISTICS (ROC) CURVE The hypothesis tests we have presented offer statistical evidence about the difference of the mean values of a single feature in the various classes. However, although this is useful information for discarding features, if the corresponding mean values are closely located, this information may not be sufficient to guarantee good discrimination properties of a feature passing the test. The mean values may differ significantly yet the spread around the means may be large enough to blur the class distinction. We will now focus on techniques providing information about the overlap between the classes. Figure 5.3a illustrates an example of two overlapping probability density functions describing the distribution of a feature in two classes, together with a threshold (one pdf has been inverted for illustration purposes). We decide class 1 for values on the left of the threshold and class 2 for the values on the right. This decision is associated with an error probability, a, of reaching a wrong decision concerning class 1 (the probability of a correct decision is 1 ⫺ a). This is equal to the 1⫺ 1
1⫺a 
a 1⫺ 0
(a)
1
0
a
(b)
FIGURE 5.3 Example of (a) overlapping pdfs of the same feature in two classes and (b) the resulting ROC curve. The larger the shaded area in (b) the less the overlap of the respective pdfs.
275
“07-Ch05-SA272” 17/9/2008 page 276
276
CHAPTER 5 Feature Selection
shaded area under the corresponding curve. Similarly,let  (1⫺) be the probability of a wrong (correct) decision concerning class 2 . By moving the threshold over “all” possible positions, different values of a and  result. It takes little thought to realize that if the two distributions have complete overlap, then for any position of the threshold we get a ⫽ 1 ⫺ . Such a case corresponds to the straight line in Figure 5.3b,where the two axes are a and 1⫺. As the two distributions move apart, the corresponding curve departs from the straight line,as Figure 5.3b demonstrates. Once more, a little thought reveals that the less the overlap of the classes, the larger the area between the curve and the straight line. At the other extreme of two completely separated class distributions,moving the threshold to sweep the whole range of values for a in [0, 1], 1 ⫺  remains equal to unity. Thus, the aforementioned area varies between zero, for complete overlap, and 1/2 (the area of the upper triangle), for complete separation, and it is a measure of the class discrimination capability of the specific feature. In practice, the ROC curve can easily be constructed by sweeping the threshold and computing percentages of wrong and correct classifications over the available training feature vectors. Other related criteria that test the overlap of the classes have also been suggested (see Problem 5.7). More recently, the area under the receiver operating characteristics curve (AUC) has been used as an effective criterion to design classifiers. This is because larger AUC values indicate on average better classifier performance, see, for example, [Brad 97, Marr 08, Land 08].
5.6 CLASS SEPARABILITY MEASURES The emphasis in the previous section was on techniques referring to the discrimination properties of individual features. However, such methods neglect to take into account the correlation that unavoidably exists among the various features and influences the classification capabilities of the feature vectors that are formed. Measuring the discrimination effectiveness of feature vectors will now become our major concern. This information will then be used in two ways. The first is to allow us to combine features appropriately and end up with the “best” feature vector for a given dimension l. The second is to transform the original data on the basis of an optimality criterion in order to come up with features offering high classification power. In the sequel we will first state class separability measures, which will be used subsequently in feature selection procedures.
5.6.1 Divergence Let us recall our familiar Bayes rule. Given two classes 1 and 2 and a feature vector x, we select 1 if P(1 |x) ⬎ P(2 |x)
As pointed out in Chapter 2, the classification error probability depends on the dif1 |x ) ference between P(1 |x) and P(2 |x), e.g., Eq. (2.12). Hence, the ratio P( P(2 |x ) can
“07-Ch05-SA272” 17/9/2008 page 277
5.6 Class Separability Measures
convey useful information concerning the discriminatory capabilities associated with an adopted feature vector x, with respect to the two classes 1 , 2 . Alternatively (for given values of P(1 ), P(2 )), the same information resides in the ratio x |1 ) ln p( p( x |2 ) ≡ D12 (x),and this can be used as a measure of the underlying discriminating information of class 1 with respect to 2 . Clearly, for completely overlapped classes, we get D12 (x) ⫽ 0. Since x takes different values, it is natural to consider the mean value over class 1 , that is, D12
⫹⬁ p(x|1 ) dx ⫽ p(x|1 ) ln p(x|2 )
(5.19)
⫺⬁
Similar arguments hold for class 2 , and we define D21
⫹⬁ p(x|2 ) dx ⫽ p(x|2 ) ln p(x|1 )
(5.20)
⫺⬁
The sum d12 ⫽ D12 ⫹ D21
is known as the divergence and can be used as a separability measure for the classes 1 , 2 , with respect to the adopted feature vector x. For a multiclass problem, the divergence is computed for every class pair i , j dij ⫽ Dij ⫹ Dji ⫹⬁ p(x|i ) dx ⫽ (p(x|i ) ⫺ p(x|j )) ln p(x|j )
(5.21)
⫺⬁
and the average class separability can be computed using the average divergence d⫽
M M
P(i )P(j )dij
i⫽1 j⫽1
Divergence is basically a form of the Kullback–Leibler distance measure between density functions [Kulb 51] (Appendix A). The divergence has the following easily shown properties: dij ⱖ 0 dij ⫽ 0
if i ⫽ j
dij ⫽ dji
If the components of the feature vector are statistically independent, then it can be shown (Problem 5.10) that dij (x1 , x2 , . . . , xl ) ⫽
l r⫽1
dij (xr )
277
“07-Ch05-SA272” 17/9/2008 page 278
278
CHAPTER 5 Feature Selection
Assuming now that the density functions are Gaussians N (i , ⌺i ) and N (j , ⌺j ), respectively, the computation of the divergence is simplified, and it is not difficult to show that dij ⫽
⫺1 1 1 ⫺1 T ⫺1 trace{⌺⫺1 (i ⫺ j ) i ⌺j ⫹ ⌺j ⌺i ⫺ 2I } ⫹ (i ⫺ j ) ⌺i ⫹ ⌺j 2 2
(5.22)
For the one-dimensional case this becomes 1 1 2 1 j 1 ⫹ i2 ⫺ 2 ⫹ (i ⫺ j )2 2 ⫹ 2 2 2 i 2 j i j 2
dij ⫽
As already pointed out, a class separability measure cannot depend only on the difference of the mean values; it must also be variance dependent. Indeed, divergence does depend explicitly on both the difference of the means and the respective variances. Furthermore, dij can be large even for equal mean values, provided the variances differ significantly. Thus, class separation is still possible even if the class means coincide. We will come to this later on. Let us now investigate (5.22). If the covariance matrices of the two Gaussian distributions are equal, ⌺i ⫽ ⌺j ⫽ ⌺, then the divergence is further simplified to dij ⫽ (i ⫺ j )T ⌺⫺1 (i ⫺ j )
which is nothing other than the Mahalanobis distance between the corresponding mean vectors. This has another interesting implication. Recalling Problem 2.9 of Chapter 2, it turns out that in this case we have a direct relation between the divergence dij and the Bayes error—that is, the minimum error we can achieve by adopting the specific feature vector. This is a most desirable property for any class separability measure. Unfortunately, such a direct relation of the divergence with the Bayes error is not possible for more general distributions. Furthermore, in [Swai 73, Rich 95] it is pointed out that the specific dependence of the divergence on the difference of the mean vectors may lead to misleading results, in the sense that small variations in the difference of the mean values can produce large changes in the divergence, which, however, are not reflected in the classification error. To overcome this, a variation of the divergence is suggested, called the transformed divergence: dˆij ⫽ 2 1 ⫺ exp(⫺dij /8)
In the sequel, we will try to define class separability measures with a closer relationship to the Bayes error.
5.6.2 Chernoff Bound and Bhattacharyya Distance The minimum attainable classification error of the Bayes classifier for two classes 1 , 2 can be written as: ⬁ Pe ⫽ ⫺⬁
min P(i )p(x|i ), P(j )p(x|j ) dx
(5.23)
“07-Ch05-SA272” 17/9/2008 page 279
5.6 Class Separability Measures
Analytic computation of this integral in the general case is not possible. However, an upper bound can be derived. The derivation is based on the inequality min[a, b] ⱕ as b1⫺s
a, b ≥ 0, and 0 ⱕ s ⱕ 1
(5.24)
p(x|i )s p(x|j )1⫺s dx ≡ ⑀CB
(5.25)
for
Combining (5.23) and (5.24), we get ⬁ Pe ⱕ P(i )s P(j )1⫺s ⫺⬁
⑀CB is known as the Chernoff bound. The minimum bound can be computed by minimizing ⑀CB with respect to s. A special form of the bound results for s ⫽ 1/2: Pe ⱕ ⑀CB
⫽ P(i )P(j )
⬁
p(x|i )p(x|j ) dx
(5.26)
⫺⬁
For Gaussian distributions N (i , ⌺i ), N (j , ⌺j ) and after a bit of algebra, we obtain ⑀CB ⫽
P(i )P(j ) exp (⫺B)
where ⌺ ⫹⌺
B⫽
⌺ ⫹ ⌺ ⫺1 | i j | 1 1 i j (i ⫺ j )T (i ⫺ j ) ⫹ ln 2 8 2 2 | ⌺i || ⌺j |
(5.27)
and | · | denotes the determinant of the respective matrix. The term B is known as the Bhattacharyya distance, and it is used as a class separability measure. It can be shown (Problem 5.11) that it corresponds to the optimum Chernoff bound when ⌺i ⫽ ⌺j . It is readily seen that in this case the Bhattacharyya distance becomes proportional to the Mahalanobis distance between the means. In [Lee 00] an equation that relates the optimal Bayesian error and the Bhattacharyya distance is proposed, based on an empirical study involving normal distributions. This was subsequently used for feature selection in [Choi 03]. A comparative study of various distance measures for feature selection in the context of multispectral data classification in remote sensing can be found in [Maus 90]. A more detailed treatment of the topic is given in [Fuku 90].
Example 5.4 Assume that P(1 ) ⫽ P(2 ) and that the corresponding distributions are Gaussians N (, 12 I ) and N (, 22 I ). The Bhattacharyya distance becomes 2 2 l 1 ⫹2 2 1 1 2 ⫹ 22 l B ⫽ ln ⫽ ln 1 (5.28) 2 2 21 2 2l 2l 1
2
For the one-dimensional case l ⫽ 1 and for 1 ⫽ 102 , B ⫽ 0.8097 and Pe ⱕ 0.2225
279
“07-Ch05-SA272” 17/9/2008 page 280
280
CHAPTER 5 Feature Selection
1
0.8
0.6
0.4
0.2
0 ⫺15
⫺10
⫺5
5
0
10
15
FIGURE 5.4 Gaussian pdfs with the same mean and different variances.
If 1 ⫽ 1002 , B ⫽ 1.9561 and Pe ⱕ 0.0707 Thus, the greater the difference of the variances, the smaller the error bound. The decrease is bigger for higher dimensions due to the dependence on l. Figure 5.4 shows the pdfs for the same mean and 1 ⫽ 1, 2 ⫽ 0.01. The figure is self-explanatory as to how the Bayesian classifier discriminates between two classes of the same mean and significantly different variances. Furthermore, as 2 /1 ⫺→0, the probability of error tends to zero (why?)
5.6.3 Scatter Matrices A major disadvantage of the class separability criteria considered so far is that they are not easily computed, unless the Gaussian assumption is employed. We will now turn our attention to a set of simpler criteria, built upon information related to the way feature vector samples are scattered in the l-dimensional space. To this end, the following matrices are defined: Within-class scatter matrix Sw ⫽
M
Pi ⌺i
i⫽1
where ⌺i is the covariance matrix for class i ⌺i ⫽ E[(x ⫺ i )(x ⫺ i )T ]
“07-Ch05-SA272” 17/9/2008 page 281
5.6 Class Separability Measures
and Pi the a priori probability of class i . That is,Pi ni /N ,where ni is the number of samples in class i , out of a total of N samples. Obviously, trace {Sw } is a measure of the average, over all classes, variance of the features. Between-class scatter matrix Sb ⫽
M
Pi (i ⫺ 0 )(i ⫺ 0 )T
i⫽1
where 0 is the global mean vector 0 ⫽
M
Pi i
i
Trace{Sb } is a measure of the average (over all classes) distance of the mean of each individual class from the respective global value. Mixture scatter matrix Sm ⫽ E[(x ⫺ 0 )(x ⫺ 0 )T ]
That is, Sm is the covariance matrix of the feature vector with respect to the global mean. It is not difficult to show (Problem 5.12) that Sm ⫽ Sw ⫹ Sb
Its trace is the sum of variances of the features around their respective global mean. From these definitions it is straightforward to see that the criterion J1 ⫽
trace{Sm } trace{Sw }
takes large values when samples in the l-dimensional space are well clustered around their mean, within each class, and the clusters of the different classes are well separated. Sometimes Sb is used in place of Sm . An alternative criterion results if determinants are used in the place of traces. This is justified for scatter matrices that are symmetric positive definite, and thus their eigenvalues are positive (Appendix B). The trace is equal to the sum of the eigenvalues, while the determinant is equal to their product. Hence, large values of J1 also correspond to large values of the criterion J2 ⫽
|Sm | ⫺1 ⫽ |Sw Sm | |Sw |
A variant of J2 commonly encountered in practice is ⫺1 J3 ⫽ trace{Sw Sm }
As we will see later on, criteria J2 and J3 have the advantage of being invariant under linear transformations, and we will adopt them to derive features in an optimal way.
281
“07-Ch05-SA272” 17/9/2008 page 282
282
CHAPTER 5 Feature Selection
In [Fuku 90] a number of different criteria are also defined by using various combinations of Sw , Sb , Sm in a “trace” or “determinant” formulation. However, whenever a determinant is used, one should be careful with Sb , since |Sb | ⫽ 0 for M ⬍ l. This is because Sb is the sum of M l ⫻ l matrices, of rank one each. In practice, all three matrices are approximated by appropriate averaging using the available data samples. These criteria take a special form in the one-dimensional, two-class problem. In this case,it is easy to see that for equiprobable classes |Sw | is proportional to 12 ⫹22 and |Sb | proportional to (1 ⫺ 2 )2 . Combining Sb and Sw , the so-called Fisher’s discriminant ratio ( FDR) results FDR ⫽
(1 ⫺ 2 )2 12 ⫹ 22
FDR is sometimes used to quantify the separability capabilities of individual features. It reminds us of the test statistic q appearing in the hypothesis statistical tests dealt with before. However, here the use of FDR is suggested in a more “primitive” fashion, independent of the underlying statistical distributions. For the multiclass case, averaging forms of FDR can be used. One possibility is FDR1 ⫽
M M (i ⫺ j )2 i2 ⫹ j2 i j⫽i
where the subscripts i, j refer to the mean and variance corresponding to the feature under investigation for the classes i , j , respectively. Example 5.5 Figure 5.5 shows three cases of classes at different locations and within-class variances. The resulting values for the J3 criterion involving the Sw and Sm matrices are 164.7, 12.5, and 12
12
12
8
8
8
4
4
8 (a)
12
4
4
8 (b)
12
4
4
8 (c)
12
FIGURE 5.5 Classes with (a) small within-class variance and small between-class distances, (b) large withinclass variance and small between-class distances, and (c) small within-class variance and large between-class distances.
“07-Ch05-SA272” 17/9/2008 page 283
5.7 Feature Subset Selection
620.9 for the cases in Figures 5.5a, b, and c, respectively. That is, the best is for distant well-clustered classes and the worst for the case of closely located classes with large withinclass variance.
5.7 FEATURE SUBSET SELECTION Having defined a number of criteria, measuring the classification effectiveness of individual features and/or feature vectors, we come to the heart of our problem, that is, to select a subset of l features out of m originally available. There are two major directions to follow.
5.7.1 Scalar Feature Selection Features are treated individually. Any of the class separability measuring criteria can be adopted, for example, ROC, FDR, one-dimensional divergence, and so on. The value of the criterion C(k) is computed for each of the features, k ⫽ 1, 2, . . . , m. Features are then ranked in order of descending values of C(k). The l features corresponding to the l best values of C(k) are then selected to form the feature vector. All the criteria we have dealt with in the previous sections measure the classification capability with respect to a two-class problem. As we have already pointed out in a couple of places, in a multiclass situation a form of average or “total” value, over all classes, is used to compute C(k). However, this is not the only possibility. In [Su 94] the one-dimensional divergence dij was used and computed for every pair of classes. Then, for each of the features, the corresponding C(k) was set equal to C(k) ⫽ min dij i,j
that is,the minimum divergence value over all class pairs,instead of an average value. Thus, selecting the features with the largest C(k) values is equivalent to choosing features with the best “worst-case” class separability capability, giving a “maxmin” flavor to the feature selection task. Such an approach may lead to more robust performance in certain cases. The major advantage of dealing with features individually is computational simplicity. However, such approaches do not take into account existing correlations between features. Before we proceed to techniques dealing with vectors, we will comment on some ad hoc techniques that incorporate correlation information combined with criteria tailored for scalar features. Let xnk ,n ⫽ 1, 2, . . . , N and k ⫽ 1, 2, . . . , m,be the kth feature of the nth pattern. The cross-correlation coefficient between any two of them is given by N xni xnj ij ⫽ n⫽1 N N 2 2 n⫽1 xni n⫽1 xnj
(5.29)
283
“07-Ch05-SA272” 17/9/2008 page 284
284
CHAPTER 5 Feature Selection
It can be shown that |ij |ⱕ1 (Problem 5.13). The selection procedure evolves along the following steps: ■
Select a class separability criterion C and compute its values for all the available features xk , k ⫽ 1, 2, . . . , m. Rank them in descending order and choose the one with the best C value. Let us say that this is xi1 .
■
To select the second feature,compute the cross-correlation coefficient defined in Eq. (5.29) between the chosen xi1 and each of the remaining m⫺1 features, that is, i1 j , j ⫽ i1 .
■
Choose the feature xi2 for which i2 ⫽ arg max ␣1 C( j) ⫺ ␣2 |i1 j | , j
for all j ⫽ i1
where ␣1 , ␣2 are weighting factors that determine the relative importance we give to the two terms. In words, for the selection of the next feature, we take into account not only the class separability measure C but also the correlation with the already chosen feature. This is then generalized for the kth step ■
Select xik , k ⫽ 3, . . . , l, so that !
k⫺1 ␣2 ik ⫽ arg max ␣1 C( j) ⫺ |ir j | j k⫺1
for j ⫽ ir ,
r⫽1
r ⫽ 1, 2, . . . , k ⫺ 1
That is, the average correlation with all previously selected features is taken into account. There are variations of this procedure. For example, in [Fine 83] more than one criterion is adopted and averaged out. Hence, the best index is found by optimizing !
k⫺1 ␣3 |ir j | ␣1 C1 ( j) ⫹ ␣2 C2 ( j) ⫺ k⫺1 r⫽1
5.7.2 Feature Vector Selection Treating features individually, that is, as scalars, has the advantage of computational simplicity but may not be effective for complex problems and for features with high mutual correlation. We will now focus on techniques measuring classification capabilities of feature vectors. It does not require much thought to see that computational burden is the major limiting factor of such an approach. Indeed, if we want to act according to what “optimality” suggests, we should form all possible vector combinations of l features out of the m originally available. According to the
“07-Ch05-SA272” 17/9/2008 page 285
5.7 Feature Subset Selection
type of optimality rule that one chooses to work with, the feature selection task is classified into two categories: Filter approach. In this approach, the optimality rule for feature selection is independent of the classifier,which will be used in the classifier design stage. For each combination we should use one of the separability criteria introduced previously (e.g., Bhattacharrya distance, J2 ) and select the best feature vector combination. Recalling our combinatorics basics, we obtain the total number of vectors as m! m ⫽ l l!(m ⫺ l)!
(5.30)
This is a large number even for small values of l, m. Indeed, for m ⫽ 20, l ⫽ 5, the number equals 15,504. Furthermore, in many practical cases the number l is not even known a priori. Thus, one has to try feature combinations for different values of l and select the“best”value for it ( beyond which no gain in performance is obtained) and the corresponding “best” l-dimensional feature vector. Wrapper approach. As we will see in Chapter 10, sometimes it is desirable to base our feature selection decision not on the values of an adopted class separability criterion but on the performance of the classifier itself. That is, for each feature vector combination the classification error probability of the classifier has to be estimated and the combination resulting in the minimum error probability is selected. This approach may increase the complexity requirements even more, depending, of course, on the classifier type. For both approaches, in order to reduce complexity, a number of efficient searching techniques have been suggested. Some of them are suboptimal and some optimal (under certain assumptions or constraints).
Suboptimal Searching Techniques Sequential Backward Selection We will demonstrate the method via an example. Let m ⫽ 4, and the originally available features are x1 , x2 , x3 , x4 . We wish to select two of them. The selection procedure consists of the following steps: ■
Adopt a class separability criterion, C, and compute its value for the feature vector [x1 , x2 , x3 , x4 ]T .
■
Eliminate one feature and for each of the possible resulting combinations, that is, [x1 , x2 , x3 ]T , [x1 , x2 , x4 ]T , [x1 , x3 , x4 ]T , [x2 , x3 , x4 ]T , compute the corresponding criterion value. Select the combination with the best value, say [x1 , x2 , x3 ]T .
■
From the selected three-dimensional feature vector eliminate one feature and for each of the resulting combinations, [x1 , x2 ]T , [x1 , x3 ]T , [x2 , x3 ]T , compute the criterion value and select the one with the best value.
285
“07-Ch05-SA272” 17/9/2008 page 286
286
CHAPTER 5 Feature Selection
Thus, starting from m, at each step we drop out one feature from the “best” combination until we obtain a vector of l features. Obviously, this is a suboptimal searching procedure, since nobody can guarantee that the optimal twodimensional vector has to originate from the optimal three-dimensional one. The number of combinations searched via this method is 1 ⫹ 1/2((m ⫹ 1)m ⫺ l (l ⫹ 1)) (Problem 5.15), which is substantially less than that of the full search procedure.
Sequential Forward Selection Here, the reverse to the preceding procedure is followed: ■
Compute the criterion value for each of the features. Select the feature with the best value, say x1 .
■
Form all possible two-dimensional vectors that contain the winner from the previous step, that is, [x1 , x2 ]T , [x1 , x3 ]T , [x1 , x4 ]T . Compute the criterion value for each of them and select the best one, say [x1 , x3 ]T .
If l ⫽ 3,then the procedure must continue. That is,we form all three-dimensional vectors springing from the two-dimensional winner,that is,[x1 , x3 , x2 ]T ,[x1 , x3 ,x4 ]T , and select the best one. For the general l, m case, it is simple algebra to show that the number of combinations searched with this procedure is lm ⫺ l(l ⫺ 1)/2. Thus, from a computational point of view,the backward search technique is more efficient than the forward one for l closer to m than to 1.
Floating Search Methods The preceding two methods suffer from the so-called nesting effect. That is, once a feature is discarded in the backward method, there is no possibility for it to be reconsidered again. The opposite is true for the forward procedure; once a feature is chosen, there is no way for it to be discarded later on. In [Pudi 94] a technique is suggested that offers the flexibility to reconsider features previously discarded and, vice versa, to discard features previously selected. The technique is called the floating search method. Two schemes implement this technique. One springs from the forward selection, and the other from the backward selection rationale. We will focus on the former. We consider a set of m features,and the idea is to search for the best subset of k of them for k ⫽ 1, 2, . . . , l ⱕ m so that a cost criterion C is optimized. Let Xk ⫽ {x1 , x2 , . . . , xk } be the set of the best combination of k of the features and Ym⫺k the set of the remaining m ⫺ k features. We also keep all the lower dimension best subsets X2 , X3 , . . . , Xk⫺1 of 2, 3, . . . , k ⫺ 1 features, respectively. The rationale at the heart of the method is summarized as follows: At the next step the k ⫹ 1 best subset Xk⫹1 is formed by “borrowing” an element from Ym⫺k . Then, return to the previously selected lower dimension subsets to check whether the inclusion of this new element improves the criterion C. If it does,the new element replaces one of the
“07-Ch05-SA272” 17/9/2008 page 287
5.7 Feature Subset Selection
previously selected features. The steps of the algorithm, when maximization of C is required are: ■
■
Step I: Inclusion xk⫹1 ⫽ arg maxy∈Ym⫺k C({Xk , y}); that is, choose that element from Ym⫺k which, combined with Xk , results in the best value of C. Xk⫹1 ⫽ {Xk , xk⫹1 } Step II: Test 1. xr ⫽ arg maxy∈Xk⫹1 C(Xk⫹1 ⫺ {y}); that is, find the feature that has the least effect on the cost when it is removed from Xk⫹1 . 2. If r ⫽ k ⫹ 1, change k ⫽ k ⫹ 1 and go to step I. 3. If r ⫽ k ⫹ 1 AND C(Xk⫹1 ⫺ {xr }) ⬍ C(Xk ) go to step I;that is, if removal of xr does not improve upon the cost of the previously selected best group of k, no further backward search is performed. 4. If k ⫽ 2 put Xk ⫽ Xk⫹1 ⫺ {xr } and C(Xk ) ⫽ C(Xk⫹1 ⫺ {xr });go to step I.
■
Step III: Exclusion 1. Xk⬘ ⫽ Xk⫹1 ⫺ {xr }; that is, remove xr . 2. xs ⫽ arg maxy∈X ⬘ C(Xk⬘ ⫺ {y}); that is, find the least significant feature k in the new set. 3. If C(Xk⬘ ⫺ {xs }) ⬍ C(Xk⫺1 ) then Xk ⫽ Xk⬘ and go to step I; no further backward search is performed. ⬘ 4. Put Xk⫺1 ⫽ Xk⬘ ⫺ {xs } and k ⫽ k ⫺ 1.
5. If k ⫽ 2 put Xk ⫽ Xk⬘ and C(Xk ) ⫽ C(Xk⬘ ) and go to step I. 6. Go to step III.1. The algorithm is initialized by running the sequential forward algorithm to form X2 . The algorithm terminates when l features have been selected. Although the algorithm does not guarantee finding all the best feature subsets, it results in substantially improved performance compared with its sequential counterpart, at the expense of increased complexity. The backward floating search scheme operates in the reverse direction but with the same philosophy.
Optimal Searching Techniques These techniques are applicable when the separability criterion is monotonic, that is, C(x1 , . . . , xi ) ⱕ C(x1 , . . . , xi , xi⫹1 )
This property allows identifying the optimal combination but at a considerably reduced computational cost with respect to (5.30). Algorithms based on the dynamic programming concept (Chapter 8) offer one possibility to approaching
287
“07-Ch05-SA272” 17/9/2008 page 288
288
CHAPTER 5 Feature Selection
the problem. A computationally more efficient way is to formulate the problem as a combinatorial optimization task and employ the so-called branch and bound methods to obtain the optimal solution [Lawe 66, Yu 93]. These methods compute the optimal value without involving exhaustive enumeration of all possible combinations. A more detailed description of the branch and bound methods is given in Chapter 15 and can also be found in [Fuku 90]. However, the complexity of these techniques is still higher than that of the previously mentioned suboptimal techniques. Remark ■
The separability measures and feature selection techniques presented above, although they indicate the major directions followed in practice, do not cover the whole range of methods that have been suggested. For example,in [Bati 94, Kwak 02, Leiv 07] the mutual information between the input features and the classifier’s outputs is used as a criterion. The features that are selected maximize the input–output mutual information. In [Sind 04] the mutual information between the class labels of the respective features and those predicted by the classifier is used as a criterion. This has the advantage that only discrete random variables are involved. The existence of bounds that relate the probability of error to the mutual information function,for example, [Erdo 03, Butz 05], could offer a theoretically pleasing flavor to the adoption of information theoretic criteria for feature selection. In [Seti 97] a feature selection technique is proposed based on a decision tree by excluding features one by one and retraining the classifier. In [Zhan 02] the tabu combinatorial optimization technique is employed for feature selection.
Comparative studies of various feature selection searching schemes can be found in [Kitt 78, Devi 82, Pudi 94, Jain 97, Brun 00, Wang 00, Guyo 03]. The task of selection bias, when using the wrapper approach and how to overcome it is treated in [Ambr 02]. This is an important issue, and it has to be carefully considered in practice in order to avoid biased estimates of the error probability.
5.8 OPTIMAL FEATURE GENERATION So far, the class separability measuring criteria have been used in a rather “passive” way, that is, to measure the classification effectiveness of features generated in some way. In this section we will employ these measuring criteria in an “active” manner, as an integral part of the feature generation process itself. From this point of view, this section can be considered as a bridge between this chapter and the following one. The method goes back to the pioneering work of Fisher ([Fish 36]) on linear discrimination, and it is also known as linear discriminant analysis (LDA). We will first focus on the simplest form of the method in order to get a better feeling and physical understanding of its basic rationale.
“07-Ch05-SA272” 17/9/2008 page 289
5.8 Optimal Feature Generation
The Two-class Case Let our data points,x,be in the m-dimensional space and assume that they originate from two classes. Our goal is to generate a feature y as a linear combination of the components of x. In such a way, we expect to “squeeze” the classificationrelated information residing in x in a smaller number (in this case only one) of features. In this section, this goal is achieved by seeking the direction w in the mdimensional space, along which the two classes are best separated in some way. This is not the only possible path for generating features via linear combination of measurements, and a number of alternative techniques will be studied in the next chapter. Given an x ∈ Rm the scalar y⫽
wT x ||w||
(5.31)
is the projection of x along w. Since scaling all our feature vectors by the same factor does not add any classification-related information, we will ignore the scaling factor ||w||. We adopt the Fisher’s discriminant ratio (FDR) (Section 5.6.3) FDR ⫽
(1 ⫺ 2 )2 12 ⫹ 22
(5.32)
where 1 , 2 are the mean values and 12 , 22 the variances of y in the two classes 1 and 2 , respectively, after the projection along w. Using the definition in (5.31) and omitting ||w||, it is readily seen that i ⫽ w T i , i ⫽ 1, 2
(5.33)
where i , i ⫽ 1, 2, is the mean value of the data in i in the m-dimensional space. Assuming the classes to be equiprobable and recalling the definition of Sb in Section 5.6.3, it is easily shown that (1 ⫺ 2 )2 ⫽ w T (1 ⫺ 2 )(1 ⫺ 2 )T w ⬀ w T Sb w
(5.34)
where ⬀ denotes proportionality. We now turn our attention to the denominator of (5.32). We have i2 ⫽ E[(y ⫺ i )2 ] ⫽ E[w T (x ⫺ i )(x ⫺ i )T w] ⫽ w T ⌺i w
(5.35)
where for each i ⫽ 1, 2,samples y (x) from the respective class i have been used. ⌺i is the covariance matrix corresponding to the data of class i in the m-dimensional space. Recalling the definition of Sw from Section 5.6.3, we get 12 ⫹ 22 ⬀ w T Sw w
(5.36)
Combining (5.36), (5.34), and (5.32), we end up that the optimal direction is obtained by maximizing Fisher’s criterion FDR(w) ⫽
w T Sb w w T Sw w
(5.37)
289
“07-Ch05-SA272” 17/9/2008 page 290
290
CHAPTER 5 Feature Selection
with respect to w. This is the celebrated generalized Rayleigh quotient, which, as it is known from linear algebra (Problem 5.16), is maximized if w is chosen such that Sb w ⫽ Sw w
(5.38)
⫺1 S . However, for our simple case we do not where is the largest eigenvalue of Sw b have to worry about any eigen decomposition. By the definition of Sb we have that
Sw w ⫽ (1 ⫺ 2 )(1 ⫺ 2 )T w ⫽ ␣(1 ⫺ 2 )
where ␣ is a scalar. Solving the previous equation with respect to w, and since we are only interested in the direction of w, we can write ⫺1 w ⫽ Sw (1 ⫺ 2 )
(5.39)
assuming, of course, that Sw is invertible. As has already been discussed, in practice, Sw and Sb are approximated by averaging using the available data samples. Figures 5.6a and 5.6b correspond to two examples for the special case of the twodimensional space (m ⫽ 2). In both cases,the classes are assumed equiprobable and have the same covariance matrix ⌺. Thus Sw ⫽ ⌺. In Figure 5.6a, ⌺ is diagonal with equal diagonal elements, and w turns out to be parallel to 1 ⫺ 2 . In Figure 5.6b, ⌺ is no more diagonal, and the data distribution does not have a spherical symmetry. In this case, the optimal direction for projection (the line on the left) is no more parallel to 1 ⫺ 2 , and its direction changes in order to account for the shape of the data distribution. This simple example once again demonstrates that the right choice of the features is of paramount importance. Take as an example the case of generating a feature by projecting along the direction of the line on the right in Figure 5.6b. Then, the values that this feature takes for the two classes exhibit a heavy overlap.
12 10 8
1 – 2
1 – 2
5 4 0 ⫺5
0
0
4
8
12 (a)
16
20
⫺4
0
4
8 12 (b)
16
FIGURE 5.6 (a) The optimal line resulting from Fisher’s criterion, for two Gaussian classes. Both classes share the same diagonal covariance matrix, with equal elements on the diagonal. The line is parallel to 1 ⫺ 2 . (b) The covariance matrix for both classes is nondiagonal. The optimal line is on the left. Observe that it is no more parallel to 1 ⫺ 2 . The line on the right is not optimal and the classes, after the projection, overlap.
“07-Ch05-SA272” 17/9/2008 page 291
5.8 Optimal Feature Generation
Thus, we have reduced the number of features from m to 1 in an optimal way. Classification can now be performed based on y. Optimality guarantees that the class separability, with respect to y, is as high as possible, as this is measured by the FDR criterion. In the case where both classes are described by Gaussian pdfs with equal covariance matrices, Eq. (5.39) corresponds to nothing else but the optimal Bayesian classifier with the exception of a threshold value (Problem 2.11 and Eqs. (2.44)– (2.46)). Moreover, recall from Problem 3.14 that this is also directly related to the linear MSE classifier. In other words, although our original goal was to generate a single feature ( y) by linearly combining the m components of x,we obtained something extra for free. Fisher’s method performed feature generation and at the same time the design of a (linear) classifier; it combined the stages of feature generation and classifier design into a single one. The resulting classifier is ⫺1 g(x) ⫽ (1 ⫺ 2 )T Sw x ⫹ w0
(5.40)
However, Fisher’s criterion does not provide a value for w0 , which has to be determined. For example, for the case of two Gaussian classes with the same covariance matrix the optimal classifier is shown to take the form (see also Problem 3.14)
P(2 ) 1 ⫺1 g(x) ⫽ (1 ⫺ 2 )T Sw x ⫺ (1 ⫹ 2 ) ⫺ ln 2 P(1 )
(5.41)
It has to be emphasized, however, that in the context of Fisher’s theory the Gaussian assumption was not necessary to derive the direction of the optimal hyperplane. In practice, sometimes the rule in (5.41) is used even if we know that the data are non-Gaussian. Of course, other values of w0 may be devised, according to the problem at hand.
Multiclass Case The previous results, obtained for the two-class case, are readily generalized for the case of M ⬎ 2 classes. The multiclass LDA has been adopted as a tool for optimal feature generation in a number of applications, including biometrics and bioinformatics, where an original large number of features has to be compactly reduced. Our major task can be summarized as follows: If x is an m-dimensional vector of measurement samples, transform it into another l-dimensional vector y so that an adopted class separability criterion is optimized. We will confine ourselves to linear transformations, y ⫽ AT x
where AT is an l ⫻ m matrix. Any of the criteria exposed so far can be used. Obviously, the degree of complexity of the optimization procedure depends heavily on the chosen criterion. We will demonstrate the method via the J3 scattering matrix criterion, involving Sw and Sb matrices. Its optimization is straightforward,
291
“07-Ch05-SA272” 17/9/2008 page 292
292
CHAPTER 5 Feature Selection
and at the same time it has some interesting implications. Let Sxw , Sxb be the withinclass and between-class scatter matrices of x. From the respective definitions, the corresponding matrices of y become Syw ⫽ AT Sxw A, Syb ⫽ AT Sxb A
Thus, the J3 criterion in the y subspace is given by J3 (A) ⫽ trace{(AT Sxw A)⫺1 (AT Sxb A)}
Our task is to compute the elements of A so that this is maximized. Then A must necessarily satisfy ⭸J3 (A) ⫽0 ⭸A
It can be shown that (Problem 5.17) ⭸J3 (A) ⫽ ⫺2Sxw A(AT Sxw A)⫺1 (AT Sxb A)(AT Sxw A)⫺1 ⫹ 2Sxb A(AT Sxw A)⫺1 ⭸A ⫽0
or ⫺1 ⫺1 (Sxw Sxb )A ⫽ A(Syw Syb )
(5.42)
An experienced eye will easily identify the affinity of this with an eigenvalue problem. It suffices to simplify its formulation slightly. Recall from Appendix B that the matrices Syw , Syb can be diagonalized simultaneously by a linear transformation BT Syw B ⫽ I , BT Syb B ⫽ D
(5.43)
which are the within- and between-class scatter matrices of the transformed vector yˆ ⫽ BT y ⫽ BT AT x
B is an l ⫻ l matrix and D an l ⫻ l diagonal matrix. Note that in going from y to yˆ there is no loss in the value of the cost J3 . This is because J3 is invariant under linear transformations, within the l-dimensional subspace. Indeed, T ⫺1 T J3 ( y) ˆ ⫽ trace{Sy⫺1 ˆ w Syˆ b } ⫽ trace{(B Syw B) (B Syb B)} ⫺1 Syb B} ⫽ trace{B⫺1 Syw ⫺1 ⫽ trace{Syw Syb BB⫺1 } ⫽ J3 (y)
“07-Ch05-SA272” 17/9/2008 page 293
5.8 Optimal Feature Generation
Combining (5.42) and (5.43), we finally obtain ⫺1 (Sxw Sxb )C ⫽ CD
(5.44)
where C ⫽ AB is an m ⫻ l dimensional matrix. Equation (5.44) is a typical eigenvalue–eigenvector problem, with the diagonal matrix D having the eigenval⫺1 S ues of Sxw xb on its diagonal and C having the corresponding eigenvectors as its ⫺1 S is an m ⫻ m matrix, and the question is which l out of a columns. However, Sxw xb total of m eigenvalues we must choose for the solution of (5.44). From its definition, matrix Sxb is of rank M⫺1, where M is the number of classes (Problem 5.18). Thus, ⫺1 S Sxw xb is also of rank M⫺1 and there are M⫺1 nonzero eigenvalues. Let us focus on the two possible alternatives separately. ■
l ⫽ M ⫺ 1: We first form matrix C so that its columns are the unit norm M ⫺ 1 ⫺1 S . Then we form the transformed vector eigenvectors of Sxw xb yˆ ⫽ C T x
(5.45)
This guarantees the maximum J3 value. In reducing the number of data from m to M ⫺ 1, there is no loss in class separability power, as this is measured by J3 . Indeed, recalling from linear algebra that the trace of a matrix is equal to the sum of its eigenvalues, we have ⫺1 J3, x ⫽ trace{Sxw Sxb } ⫽ 1 ⫹ · · · ⫹ M⫺1 ⫹ 0
(5.46)
J3, yˆ ⫽ trace{(C T Sxw C)⫺1 (C T Sxb C)}
(5.47)
Also
Rearranging (5.44), we get C T Sxb C ⫽ C T Sxw CD
(5.48)
Combining (5.47) and (5.48), we obtain J3, yˆ ⫽ trace{D} ⫽ 1 ⫹ · · · ⫹ M⫺1 ⫽ J3,x
(5.49)
It is most interesting to view this from a slightly different perspective. Let us recall the Bayesian classifier for an M class problem. Of the M conditional class probabilities, P(i |x), i ⫽ 1, 2, . . . , M, only M ⫺ 1 are independent, since they all add up to one. In general, M ⫺ 1 is the minimum number of discriminant functions needed for an M-class classification task (Problem 5.19). The linear ˆ can be seen as operation C T x, which computes the M ⫺ 1 components of y, an optimal linear rule that provides M ⫺ 1 discriminant functions, where optimality is with respect to J3 . This was clearly demonstrated in the twoclass case, where Fisher’s method was also used as a classifier (subject to an unknown threshold).
293
“07-Ch05-SA272” 17/9/2008 page 294
294
CHAPTER 5 Feature Selection
Investigating the specific form that Eq. (5.45) takes for the two-class problem, one can show that for M ⫽ 2 there is only one nonzero eigenvalue, and it turns out that (Problem 5.20) ⫺1 yˆ ⫽ (1 ⫺ 2 )T Sxw x
which is our familiar Fisher’s linear discriminant. ■
l ⬍ M⫺1: In this case C is formed from the eigenvectors corresponding to the ⫺1 S . The fact that J is given as the sum of the corl largest eigenvalues of Sxw xb 3 responding eigenvalues guarantees its maximization. Of course, in this case there is loss of the available information because now J3,ˆy ⬍ J3,x .
A geometric interpretation of (5.45) reveals that yˆ is the projection of the original ⫺1 S . It must be vector x onto the subspace spanned by the eigenvectors v i of Sw b pointed out that these are not necessarily mutually orthogonal. Indeed, although ⫺1 S are not; thus, the matrices Sw , Sb (Sm ) are symmetric, products of the form Sw b eigenvectors are not mutually orthogonal (Problem 5.21). Furthermore, as we saw during the proof, once we decide on which subspace to project (by selecting the appropriate combination of eigenvectors) the value of J3 remains invariant under any linear transformation within this subspace. That is, it is independent of the coordinate system,and its value depends only on the particular subspace. In general, projection of the original feature vectors onto a lower dimensional subspace is associated with some information loss. An extreme example is shown in Figure 5.7, where the two classes coincide after projection on the v2 axis. On the other hand, from all possible projection directions, Fisher’s linear discrimination rule leads to
v1
x2 v2
x1
FIGURE 5.7 Geometry illustrating the loss of information associated with projections in lower dimensional subspaces. Projecting onto the direction of the principle eigenvector, v 1 , there is no loss of information. Projection on the orthogonal direction results in a complete class overlap.
“07-Ch05-SA272” 17/9/2008 page 295
5.8 Optimal Feature Generation
the choice of the one-dimensional subspace v1 , which corresponds to the optimal J3 value,that guarantees no loss of information for l ⫽ M ⫺ 1 ⫽ 1 (as this is measured by the J3 criterion). Thus, this is a good choice, provided that J3 is a good criterion for the problem of interest. Of course, this is not always the case; it depends on the specific classification task. For example, in [Hams 08] the criterion used is the probability of error for a multiclass task involving normally distributed data. A more extensive treatment of the topic, also involving other optimizing criteria, can be found in [Fuku 90]. Remarks ■
If J3 is used with another combination of matrices, such as Sw and Sm , then, in general, the rank of the corresponding matrix product involved in the trace is m and there are m nonzero eigenvalues. In such cases, the transformation matrix C is formed so that its columns are the eigenvectors corresponding to the l largest eigenvalues. According to (5.49), this guarantees the maximum value of J3 .
■
In practice,one may encounter cases in which Sw is not invertible. This occurs in applications where the available size of the training set, N , is smaller than the dimensionality,m,of the original feature space. In such cases the resulting estimate of Sw , which is obtained as the mean of N outer vector products, has rank lower than m; hence it is singular. This is known as the small sample size (SSS) problem. Web document classification, face recognition, and disease classification based on gene-expression profiling are some examples where the small sample size problem occurs frequently in practice. ⫹ in place One way to overcome this difficulty is to use the pseudoinverse Sw ⫺1 of Sw [Tian 86]. However, now, there is no guarantee that the J3 criterion is ⫹ S corresponding to the largest maximized by selecting the eigenvectors of Sw b eigenvalues. An alternative route is to employ regularization techniques, in one way or another, for example, [Frie 89, Hast 95]. For example, Sw may be replaced by Sw ⫹ ⍀, where ⍀ can be any positive definite and symmetric matrix. The specific choice depends on the problem. The choice of is also a critical factor here. Another drawback of these techniques is that they do not scale well for problems with large dimensionality. For example, in certain tasks of face recognition, the resulting covariance matrices can be as high as a few thousand making matrix inversion a computationally thirsty task. Another way to deal with the small sample size problem is to adopt a twostage approach. One such technique is the so-called PCA+LDA technique. In the first stage, principle component analysis (PCA, see Chapter 6) is performed to reduce, appropriately, the dimensionality of the feature space and linear discriminant analysis (LDA) is then performed in the low-dimensional space, for example, [Belh 97]. A drawback of this technique is that during the dimension reduction phase part of the discriminatory information may be lost.
295
“07-Ch05-SA272” 17/9/2008 page 296
296
CHAPTER 5 Feature Selection
In [Yang 02] the mixture scatter matrix, Sm , is used in the J criterion in the place of Sw . It is shown that in this case, applying first a PCA on Sm , to reduce the dimensionality to the value of the rank of Sm , followed by an LDA in the reduced space, does not lead to any loss of information. In [Chen 00] the null space of the within-class scatter matrix is brought into the game. It has been observed that the null space, of Sw contains useful discriminant information. The method first projects onto the null space and, then, in the projected space the transformation that maximizes the between-class scatter is computed. A disadvantage of this approach is that it may lose information by considering the null space instead of Sw . A second problem is that the complexity of determining the null space of Sw is very high. Computational difficulties of the method are addressed in [Cevi 05]. In [Ye 05],in the first stage,dimensionality reduction is achieved by maximizing the between-class cluster (Sb ), via a QR decomposition technique. In the second stage,a refinement is achieved by focusing on the within-class scatter issue, following arguments similar to the classical LDA. A unifying treatment of a number from the previous techniques is considered in [Zhan 07]. A different approach is proposed in [Li 06]. Instead of the J3 criterion, another criterion is introduced that involves the trace of the difference of the involved matrices, thus bypassing the need for inversions. Besides the small sample size problem, another issue associated with the LDA is that the number of features that can be generated is at most one less than the number of classes. As we have seen, this is due to the rank of the ⫺1 S . For an M-class problem, there are only M ⫺ 1 nonzero matrix product Sw b eigenvalues. All the J3 related discriminatory information can be recovered by projecting onto the subspace generated by the eigenvectors associated with these nonzero eigenvalues. Projecting on any other direction adds no information. Good insight into it can be gained through geometry by considering a simple example. Let us assume, for simplicity, a two-class task with classes normally distributed with covariance matrices equal to the identity matrix. Then by its definition,Sw is also an identity matrix. It is easy to show ( Problem 5.20) that in this case the eigenvector corresponding to the only nonzero eigenvalue is equal to 1 ⫺ 2 . The (Euclidean) distance between the mean values of the projection points in the (nonzero) eigenvector direction is the same as the distance between the mean values of the classes in the original space, i.e., ||(1 ⫺ 2 )||. This can easily be deduced by visual inspection of Figure 5.7, which corresponds to a case such as is discussed our example. Projecting on the orthogonal direction adds no information since the classes coincide. All the scatter information, with respect to both classes, is obtained from a single direction. Due to the previous drawback,there are cases where the number of classes M is small, and the resulting number of, at most, M ⫺ 1 features is insufficient. An attempt to overcome this difficulty is given in [Loog 04]. The main
“07-Ch05-SA272” 17/9/2008 page 297
5.8 Optimal Feature Generation
idea is to employ a different to Sb measure to quantify the between-class scatter. The Chernoff distance (closely related to the Bhattacharyya distance of Section 5.6.2) is employed. This change offers the possibility of reducing the dimensionality to any dimension l smaller than the original m. A different path is followed in [Kim 07]. From the original m features, the authors build a number of so-called composite vectors. Each vector consists of a subset of the m features. Different composite vectors are allowed to share some of the original features. LDA is then performed on this new set of feature vectors. This procedure enhances the range of the rank of the involved matrix product beyond M ⫺ 1. In [Nena 07], the shortcomings of LDA are overcome by defining a new class-separability measure based on an information-theoretic cost inspired by the concept of mutual information. ■
No doubt, scattering matrix criteria are not the only ones that can be used to compute the optimal transformation matrix. For example, [Wata 97] suggested using ta different transformation matrix for each class and optimizing with respect to the classification error. This is within the spirit of the recent trend, to optimize directly with respect to the quantity of interest, which is the classification error probability. For the optimization, smooth versions of the error rate are used to guarantee differentiability. Other ways to compute the transformation matrix will be discussed in the next chapter.
■
Besides the linear nonlinear transformations can also be employed for optimal feature selection. For example, in [Samm 69] a nonlinear technique is proposed that attempts to preserve maximally all the distances between vectors. Let x i , y i , i ⫽ 1, 2, . . . , N , be the feature vectors in the original m-dimensional and the transformed l-dimensional space, respectively. The transformation into the lower dimensional space is performed so as to maximize N ⫺1
1
J ⫽ N ⫺1 N i⫽1
j⫽i⫹1 d
o (i, j)
N
i⫽1 j⫽i⫹1
d o (i, j) ⫺ d(i, j) d o (i, j)
2 (5.50)
where d o (i, j), d(i, j) are the (Euclidean) distances between vectors x i , and x j in the original space and y i , y j in the transformed space,respectively. ■
Another nonlinear generalization of the method consists of two (implicit) steps. First, one employs a nonlinear vector function to transform the input feature space into a higher-dimensional one, which can even be of infinite dimension. Then, the linear discriminant method is applied in this highdimensionality space. However, the problem formulation is done so that vectors appear only via inner products. This allows the use of kernel functions to facilitate computations, as was the case with the nonlinear support vector machines presented in Chapter 4 [Baud 00, Ma 03].
297
“07-Ch05-SA272” 17/9/2008 page 298
298
CHAPTER 5 Feature Selection
5.9 NEURAL NETWORKS AND FEATURE GENERATION/SELECTION Recently, efforts have been made to use neural networks for feature generation and selection. A possible solution is via the so-called auto-associative networks. A network is employed having m input and m output nodes and a single hidden layer with l nodes with linear activations. During training, the desired outputs are the same as the inputs. That is, E(i) ⫽
m
( yˆ k (i) ⫺ xk (i))2
k⫽1
where the notation of the previous chapter has been adopted. Such a network has a unique minimum, and the outputs of the hidden layer constitute the projection of the input m-dimensional space onto an l-dimensional subspace. In [Bour 88] it is shown that this is basically a projection onto the subspace spanned by the l principal eigenvectors of the input correlation matrix, a topic that will be our focus in the next chapter. An extension of this idea is to use three hidden layers [Kram 91]. Such a network performs a nonlinear principal component analysis. The major drawback of such an architecture is that nonlinear optimization techniques have to be used for the training. Besides the computational load, the risk of being trapped in local minima is always present. An alternative is to use neural networks, or any other (non)linear structure, to exploit properties of the LS cost function. In Chapter 3,we saw that the outputs of a network approximate posterior probabilities, provided that the weights have been trained so that the outputs match, in the LS sense, the class labels. In [Lowe 91] it is pointed out that, besides this property, another very interesting one is also valid. A multilayer perceptron was considered with linear output nodes. The network was trained to minimize the squared error between the actual and desired responses (i.e., class labels 1 and 0). It was shown that minimizing the squared error is equivalent to maximizing the criterion ⫺1 J ⫽ trace{Sm Sb }
(5.51)
where Sm is the mixture scatter matrix of the vectors formed by the outputs of the last hidden layer nodes and Sb the corresponding between-class scatter matrix in a weighted form (Problem 5.22). If the inverse of Sm does not exist,it is replaced by its pseudoinverse. In other words,such a network can be used as a J -optimal nonlinear transformer of the input m-dimensional vectors into l-dimensional vectors, where l is the number of nodes in the last hidden layer. Another approach is to employ neural networks to perform the computations associated with the optimization of various class separability criteria discussed in this chapter. Although these techniques do not necessarily provide new approaches, the incorporation of neural networks offers the capability of adaptation in case the statistics of the input data are slowly varying. In [Chat 97, Mao 95] a number of such techniques are developed. The idea behind most of these techniques is to use
“07-Ch05-SA272” 17/9/2008 page 299
5.10 A Hint on Generalization Theory
a network that iteratively computes eigenvectors of correlation matrices,a step that, as we have seen, is at the heart of a number of optimality criteria. An alternative technique has been suggested in [Lee 93, Lee 97]. They have shown that the discriminantly informative feature vectors have a component that is normal to the decision surface at least at one point on the surface. Furthermore, the less informative vectors are orthogonal to a vector normal to the decision surface at every point of the surface. This is natural, because vectors that do not have a component normal to the decision surface cannot cross it (and hence change classes) whatever their value is. Based on this observation, they estimate normal vectors to the decision boundary, using gradient approximation techniques, which are then used to formulate an appropriate eigenvalue–eigenvector problem leading to the computation of the transformation matrix. Finally, pruning a neural network is a form of feature selection integrated into the classifier design stage. Indeed, the weights of the input nodes corresponding to less important features are expected to be small. As discussed in Chapter 4, the incorporation of appropriate regularization terms in the cost function encourages such weights to converge to zero and ultimately to be eliminated. This approach was followed, for example, in [Seti 97].
5.10 A HINT ON GENERALIZATION THEORY So far in this book, two major issues have occupied us: the design of the classifier and its generalization capabilities. The design of the classifier involved two stages: the choice of the classifier type and the choice of the optimality criterion. The generalization capabilities led us to seek ways to reduce the feature space dimensionality. In this section we will point out some important theoretical results that relate the size N of the training data set and the generalization performance of the designed classifier. To this end, let us summarize a few necessary basic steps and definitions. ■
■
Let F be the set of all the functions f that can be realized by the adopted classifier scheme. For example, if the classifier is a multilayer perceptron with a given number of neurons, then F is the set of all the functions that can be realized by the specific network structure. Functions f are mappings from Rl → {0, 1}. Thus, the response is either 1 or 0; that is, the two-class problem is considered, and the mapping is either f (x) ⫽ 1 or f (x) ⫽ 0. Let PeN ( f ) be the empirical classification error probability, based on the available input—desired output training pairs (x i , yi ), i ⫽ 1, 2, . . . , N , which are considered to be independent and identically distributed (i.i.d.). Thus, PeN ( f ) is the fraction of training samples for which an error occurs,that is,f (x i ) ⫽ yi . Obviously, this depends on the specific function f and the size N . The optimal function that results from minimizing this empirical cost is denoted by f ∗ and belongs to the set F.
299
“07-Ch05-SA272” 17/9/2008 page 300
300
CHAPTER 5 Feature Selection
■
Pe ( f ) is the true classification error probability when a function f is realized. The corresponding empirical PeN (f ) can be very small, even zero, since a classifier can be designed to classify all training feature vectors correctly. However, Pe ( f ) is the important performance measure, because it measures error probability based on the statistical nature of the data and not on the specific training set only. For a classifier with good generalization capabilities, we expect the empirical and the true error probabilities to be close. Pe ( f ) is sometimes known as the generalization error probability.
■
Pe denotes the minimum error probability over all the functions of the set, that is, Pe ⫽ minf ∈F Pe ( f ).1 Again, in practice we would like the optimal empirical error PeN ( f ∗ ) to be close to Pe .
The Vapnik–Chervonenkis theorem is as follows. Theorem Let F be the class of functions of the form Rl →{0, 1}. Then the empirical and true error probabilities corresponding to a function f in the class, satisfy prob{ max |PeN ( f ) ⫺ Pe ( f )| ⬎ ⑀} ⱕ 8S(F, N ) exp(⫺N ⑀2 /32) f ∈F
(5.52)
The term S(F, N ) is called the shatter coefficient of the class F. This is defined as the maximum number of dichotomies of N points that can be formed by the functions in F. From our combinatorics basics,we know that the maximum number of dichotomies on a set of N points (separating them into two distinct subsets) is 2N . However, not all these combinations can be implemented by a function f : Rl → {0, 1}. For example,we know that,in the two-dimensional space,the set of functions realized by a perceptron (hyperplane) can form only fourteen distinct dichotomies on four points out of the 16 ⫽ 24 possibilities. The two XOR combinations cannot be realized. However, the class of functions realized by the perceptron can form all possible 8 ⫽ 23 dichotomies for N ⫽ 3 points. This leads us to the following definition Definition 1. The largest integer k ⱖ 1 for which S(F, k) ⫽ 2k is called the Vapnik–Chervonenkis, or VC dimension of the class F, and is denoted by Vc . If S(F, N ) ⫽ 2N for every N , then the VC dimension is infinite. Thus,in the two dimensional space,the VC dimension of a single perceptron is 3. In the general l-dimensional space case, the VC dimension of a perceptron is l ⫹ 1, as is easily verified from Section 4.13. It will not come as a surprise to say that the VC dimension and the shatter coefficient are related, because they have common origins. Indeed, this is true. It turns out that if the VC dimension is finite, then the following bound is valid S(F, N ) ⱕ N Vc ⫹ 1
1
Strictly speaking, in this section inf must be used instead of min and sup instead of max.
(5.53)
“07-Ch05-SA272” 17/9/2008 page 301
5.10 A Hint on Generalization Theory
That is, the shatter coefficient is either 2N or is bounded as given in (5.53). This bound has a very important implication for the usefulness of (5.52). Indeed, for finite VC dimensions (5.53) guarantees that for large enough N the shatter coefficient is bounded by polynomial growth. Then the bound in (5.52) is dominated by its exponential decrease, and it tends to zero as N →⬁. In words, for large N the probability of having large differences between the empirical and the true probability errors is very small! Thus, the network guarantees good generalization performance for large N . Furthermore, the theory guarantees another strong result [Devr 96] prob{Pe ( f ∗ ) ⫺ min Pe (f ) ⬎ ⑀} ⱕ 8S(F, N ) exp(⫺N ⑀2 /128) f ∈F
(5.54)
That is,for large N we expect with high probability the performance of the empirical error optimal classifier to be close to the performance of the optimal one, over the specific class of functions. Let us look at these theoretical results from a more intuitive perspective. Consider two different networks with VC dimensions Vc1 ⬍⬍ Vc2 . Then if we fix N and ⑀, we expect the first network to have better generalization performance, because the bound in (5.52) will be much tighter. Thus, the probability that the respective empirical and true errors will differ more than the predetermined quantity will be much smaller. We can think of the VC dimension as an intrinsic capacity of a network, and only if the number of training vectors exceeds this number sufficiently can we expect good generalization performance. Learning theory is rich in bounds that have been derived and that relate quantities such as the empirical error, the true error probability, the number of training vectors, and the VC dimension or a VC related quantity. In his elegant theory of learning,Valiant [Vali 84] proposed to express such bounds in the flavor of statistical tests. That is, the bounds involve an error ⑀, such as in Eqs. (5.52) and (5.54), and a confidence probability level that the bound holds true. Such bounds are known as PAC bounds, which stands for Probably (the probability of the bound to fail is small) Approximately Correct (when the bound holds, the error is small). A very interesting (for our purposes) bound that can be derived refers to the minimum number of training points that guarantee, with high probability, the design of a classifier with good error performance. Let us denote this minimum number of points as N (⑀, ). It can be shown that if
N (⑀, ) ⱕ max
k1 Vc k2 Vc k3 8 ln 2 , 2 ln ⑀2 ⑀ ⑀
(5.55)
then for any number of training points N ⱖN (⑀, ) the optimal classifier,f ∗ ,resulting by minimizing the empirical error probability PeN (f ) satisfies the bound P{Pe ( f ∗ ) ⫺ Pe ⬎ ⑀} ⱕ
(5.56)
where k1 , k2 , k3 are constants [Devr 96]. In other words,for small values of ⑀ and ,if N ⱖN (⑀, ),the performance of the optimum empirical error classifier is guaranteed,
301
“07-Ch05-SA272” 17/9/2008 page 302
302
CHAPTER 5 Feature Selection
with high probability, to be close to the optimal classifier in the class of functions F, realized by the specific classification scheme. The number N (⑀, ) is also known as sample complexity. Observe that the first of the two terms in the bound has a linear dependence on theVC dimension and an inverse quadratic dependence on the error ⑀. Doubling,for example,theVC dimension roughly requires that we need to double the number of training points in order to keep the same ⑀ and confidence level. On the other hand, doubling the accuracy (i.e., ⑀/2) requires us to quadruple the size of the training set. The confidence level has a little influence on the bound, due to its logarithmic dependence. Thus, high VC dimension sets high demands on the number of training points required to guarantee, with high probability, a classifier with good performance. Another related bound of particular interest to us that holds with a probability at least 1 ⫺ is the following:
Pe ( f
) ⱕ PeN (
Vc f)⫹ N
(5.57)
where Vc is the VC dimension of the corresponding class and
Vc N
≡
V ln 2N ⫹ 1 ⫺ ln(/4)
c Vc N
(5.58)
The interested reader may obtain more bounds and results concerning the Vapnik–Chervonenkis theory from [Devr 96, Vapn 95]. It will take some effort, but it is worth it! In some of the published literature, the constants in the bounds are different. This depends on the way the bounds are derived. However, this is not of major practical importance, since the essence of the theory remains the same. Due to the importance of the VC dimension, efforts have been made to compute it for certain classes of networks. In [Baum 89] it has been shown that the VC dimension of a multilayer perceptron with hard limiting activation functions in the nodes is bounded by " 2
# Knh l ⱕ Vc ⱕ 2Kw log2 (eKn ) 2
(5.59)
where Knh is the total number of hidden layer nodes, Kn the total number of nodes, Kw the total number of weights, l the input space dimension, e the base of the natural logarithm, and [·] the floor operator that gives the largest integer less than its argument. The lower bound holds only for networks with a single hidden layer and full connectivity between the layers. A similar upper bound is true for RBF networks too. Looking at this more carefully, one can say that for such networks the VC dimension is roughly given by the number of weights of the network, that is, the number of its free parameters to be determined! In practice, good generalization performance is expected if the number of training samples is a few times the VC dimension. A good rule of thumb is to choose N to be of the order of 10 times the VC dimension or more [Hush 93].
“07-Ch05-SA272” 17/9/2008 page 303
5.10 A Hint on Generalization Theory
Besides theVapnik–Chervonenkis theory,the published literature is rich in results concerning aspects of designing various classifiers using a finite data set N . Although they lack the elegance of the generality of theVapnik–Chervonenkis theory,they provide further insight into this important task. For example, in [Raud 91] asymptotic analytic results are derived for a number of classifiers (linear, quadratic, etc.) under the Gaussian assumption of the involved densities. The classification error probability of a classifier designed using a finite set of N training samples is larger, by an amount ⌬N , than the error of the same classifier designed using an infinite (N → ⬁) set of data. It is shown that the mean of ⌬N (over different design sets) decreases as N tends to infinity. The rate of decrease depends on the specific type of classifier, on the dimensionality of the problem, and also on the value of the asymptotic (N → ⬁) error. It turns out that in order to keep the mismatch ⌬N within certain limits,the number N of design samples must be a number of times larger than the dimension l. Depending on the type of classifier, this proportionality constant can range from low values (e.g., 1.5) up to a few hundred! Furthermore, in [Fuku 90] it is shown that keeping N constant and increasing l, beyond a point, results in an increase of the classification error. This is known as the Hughes phenomenon, and it was also discussed in Section 5.3. All these theoretical results provide useful guidelines in selecting appropriate values for N and l for certain types of classifiers. Moreover, they make crystal clear the urge to keep the number of features as small as possible with respect to N and the importance of the feature selection stage in the design of a classification system. In the fringes of this theoretical “happening,” a substantial research effort has also been devoted to experimental comparisons, involving different classifiers, with respect to their generalization capabilities;see,for example,[Mama 96] and the references therein. In practice, however, experience and empirical results retain an important part in the final decision. Engineering still has a flavor of art!
Structural Risk Minimization In our discussion so far, we have focused on the effects of the finite size of the training data set, N , for a given class of functions, that is, a given classifier structure. Let us now touch on another important issue. If we allow N to grow indefinitely, does this luxury provide us with the means not only to have good generalization properties but also to improve our classification error so as to approach the optimal Bayesian performance? Recall that as N grows, we can expect to obtain the optimal performance with respect to all allowable sets of classifiers that can be implemented by the chosen network structure. However, the error rate of the corresponding optimal classifier may still be very far from that of the Bayesian classifier. Let us denote by PB the Bayesian error probability. Then we can write Pe ( f ∗ ) ⫺ PB ⫽ Pe ( f ∗ ) ⫺ Pe ⫹ (Pe ⫺ PB )
(5.60)
A diagrammatic interpretation of Eq. (5.60) is given in Figure 5.8. The right-hand side in Eq. (5.60) consists of two conflicting terms. If the class F is too small, then
303
“07-Ch05-SA272” 17/9/2008 page 304
304
CHAPTER 5 Feature Selection
Set of functions F f * {Pe ( f *)} Pe ( f *) – Pe Optimal Bayesian rule {PB}
fopt {Pe} Pe – PB
FIGURE 5.8 Diagrammatic interpretation of Eq. (5.60). The optimal function over the set F , associated with the minimum error Pe , is denoted as fopt , and f ∗ is the optimal function resulting from the empirical cost for a given N .
the first term is expected to be small, but the second term is likely to be large. If, on the other hand, the class of functions F is large, then the second term is expected to be small but the first term is probably large. This is natural, because the larger the set of functions, the higher the probability of including in the set a good approximation of the Bayesian classifier. Moreover, the smaller the class, the less the variation between its members. This reminds us of the bias–variance dilemma we discussed in Chapter 3. A little thought suffices to reveal that the two problems are basically the same, seen from a different viewpoint. Then the natural question arises once more, can we make both terms small and how? The answer is that this is possible only asymptotically, provided that at the same time the size of the class F grows appropriately. An elegant strategy to achieve this has been suggested by Vapnik and Chervonenkis [Vapn 82]. Let F (1) , F (2) , . . . be a sequence of nested classes of functions, that is, F (1) ⊂ F (2) ⊂ F (3) ⊂ · · ·
(5.61)
with an increasing, yet finite,VC dimension, Vc,F (1) ⱕ Vc,F (2) ⱕ Vc,F (3) ⱕ · · ·
(5.62)
Also let lim
inf Pe ( f ) ⫽ PB
i→⬁ f ∈ F (i)
(5.63)
For each N and class of functions F (i) , i ⫽ 1, 2, . . . , compute the optimum,fN∗ ,i ,with respect to the empirical error using the N training pairs of input–output samples. Vapnik and Chervonenkis suggest choosing for each N the function fN∗ according to the structural risk minimization principle (SRM). This consists of the following two steps. First we select the classifier fN∗ ,i from every class F (i) that minimizes
“07-Ch05-SA272” 17/9/2008 page 305
5.10 A Hint on Generalization Theory
the corresponding empirical error over the class of functions. Then, from all these classifiers, we choose the one that minimizes the upper bound in (5.57), over all i. More precisely, form the so-called guaranteed error bound, P˜ e ( fN∗ ,i ) ≡ PeN (fN∗ ,i ) ⫹
Vc,F (i)
(5.64)
N
and choose fN∗ ⫽ arg min P˜ e ( fN∗ ,i )
(5.65)
i
Then,as N → ⬁,Pe ( fN∗ ) tends to PB with probability one. Note that the second term V
(i)
, is a complexity penalty term that increases as in the minimized bound, c,F N the network complexity increases (i.e., with the size of the class of functions and Vc,F (i) ). If on one hand the classifier model is too simple, the penalty term is small but the empirical error term will be large in (5.64). On the other hand, if the model is complex, the empirical error is small but the penalty term large. The structural risk minimization criterion aims at achieving the best trade-off between these two terms. This is illustrated in Figure 5.9. From this point of view, the structural risk minimization principle belongs to a more general class of approaches that try to estimate the order of a system,by considering simultaneously the model complexity and a performance index. Depending
Guaranteed error bound
Optimal empirical error
Complexity penalty term
Vc,F (1)
Vc,*F
Vc,F (k)
VC dimension
FIGURE 5.9 For a fixed N , the complexity penalty term increases and the optimal empirical error decreases as the VC dimension of the model increases. Choosing the model according to the SRM principle aims at achieving the best trade-off between these two terms that corresponds to the minimum of the guaranteed error bound. Note that, Vc,F (1) ⬍ V ∗c,F ⬍ Vc,F (k) , which implies F (1) ⊂ F ∗ ⊂ F (k) .
305
“07-Ch05-SA272” 17/9/2008 page 306
306
CHAPTER 5 Feature Selection
on the function used to measure the model complexity and the corresponding performance index, different criteria result. For example, in the Akaike Information Criterion [Akai 74], the place of the empirical error is taken by the value of the log-likelihood function, corresponding to the maximum likelihood estimates of the unknown parameters, and the complexity term is proportional to the number of free parameters to be estimated. See also Sections 5.11 and 16.4.1. An alternative interpretation of the SVM cost function in Eq. (3.93) is given in [Lin 02]. It is treated as a typical regularization method with two components a 2 data fit functional term i I (i ) and a regularization penalty term (||w|| ). The latter is the complexity-related component and is used to guard against overfitting. In general, the data-fit term approaches a limiting functional as N → ⬁. Under some general conditions, the estimate resulting from the regularization method is the minimizer of this data-fit-limiting functional, as N → ⬁. It turns out that in the SVM case the minimizer of the limiting data-fit functional is the Bayes optimal rule and the SVM solution approaches it, as N → ⬁, provided that the kernel choice guarantees a rich enough space ( RKHS) and the smoothing parameter, C, is chosen appropriately. This interesting result nicely ties the SVM and the Bayesian optimal rule. Would it be an exaggeration to say that a good theory is like a good piece of art, in the sense that both support more than one interpretation? Remarks ■
The SRM procedure provides a theoretical guideline for constructing a classifier that converges asymptotically to the optimal Bayesian one. However, the bound in (5.57), which is exploited in order to reach this limit, must not be misinterpreted. For any bound to be useful in practice, one needs an extra piece of information. Is this bound loose or tight? In general, until now, no result has provided this information. We can construct classifiers whose VC dimension is large,yet their performance can be good. A typical example is the nearest neighbor (NN ) classifier. Its VC dimension is infinite. Indeed, since we know the class label of all the N training points, the NN classifier classifies correctly all training points and the corresponding shatter coefficient is 2N . Yet, it is generally accepted that the generalization performance of this classifier can be quite good in practice. In contrast, one can build a classifier with finite VC dimension, yet whose performance is always bad ([Burg 98]). Concluding this remark, we have to keep in mind that if two classifiers have the same empirical error, it does not, necessarily, mean that the one with the smaller VC dimension leads to better performance.
■
Observe that in all bounds given previously no assumptions have been made about the statistical distribution underlying the data set. That is, they are distribution-free bounds.
“07-Ch05-SA272” 17/9/2008 page 307
5.10 A Hint on Generalization Theory
Support Vector Machines: A Last Touch We have already discussed that the VC dimension of a linear classifier in the l-dimensional space is l ⫹ 1. However, hyperplanes that are constrained to leave the maximum margin between the classes may have a smaller VC dimension. Let us assume that r is the radius of the smallest ( hyper)sphere that encloses all the data (Problem 5.23), that is, x i ⱕ r, i ⫽ 1, 2, . . . , N
Then if a hyperplane satisfies the conditions in Eq. (3.73) and w 2 ⱕ c
where c is a constant, then its VC dimension, Vc , is bounded by ([Vapn 98]) Vc ⱕ min(r 2 c, l) ⫹ 1
(5.66)
That is, the capacity of the classifier can be controlled independently of the dimensionality of the feature space. This is very interesting indeed. It basically states that the capacity of a classifier may not, necessarily, be related to the number of unknown parameters! This is a more general result. To emphasize it further, note that it is possible to construct a classifier with only one free parameter, yet with infinite VC dimension; see, for example, [Burg 98]. Let us now consider a sequence of bounds c1 ⬍ c 2 ⬍ c 3 ⬍ . . .
This defines the following sequence of classifiers: $ % F i : w T x ⫹ w0 : w 2 ⱕ ci
(5.67)
where F i ⊂ F i⫹1
If the classes are separable, then the empirical error is zero. Minimizing the norm w is equivalent to minimizing the VC dimension (to be fair, the upper bound of the VC dimension). Thus, we can conclude that, the design of an SVM classifier senses the spirit of the SRM principle. Hence, keeping the VC dimension minimum suggests that we can expect support vector machines to exhibit good generalization performance. More on these issues can be found in [Vapn 98, Burg 98]. The essence of all formulas and discussion in this section is that the generalization performance and accuracy of a classifier depend heavily on two parameters: the VC dimension and the number of the available feature vectors used for the training. The VC dimension may or may not be related to the number of free parameters describing the classifier. For example, in the case of the perceptron linear classifier the VC dimension coincides with the number of free parameters. However, one can
307
“07-Ch05-SA272” 17/9/2008 page 308
308
CHAPTER 5 Feature Selection
construct nonlinear classifiers whose VC dimension can be either lower or higher than the number of free parameters [Vapn 98, p. 159]. The design methodology of the SVM allows one to“play”with theVC dimension (by minimizing w ,Eq. (5.66)), leading to good generalization performance,although the design may be carried out in a high- (even infinite) dimensional space. Digging this fertile ground in a slightly different direction, using tools from the PAC theory of learning one can derive a number of distribution-free and dimensionfree bounds. These bounds bring into the surface a key property underlying the SVM design; that is, that of the maximum margin (SVMs are just one example of this larger family of classifiers, which are designed with an effort to maximize the margin the training points leave from the corresponding decision surface). (See also the discussion at the end of Chapter 4.) Although a more detailed treatment of this topic is beyond the scope of this book, we will provide two related bounds that reinforce this,at first surprising,property of the“emancipation”of the generalization performance from the feature space dimensionality. Assume that all available feature vectors lie within a sphere of radius R (i.e., x ⱕ R). Let, also, the classifier be a linear one, normalized so that w ⫽ 1, designed using N randomly chosen training vectors. If the resulting classifier has a margin of 2␥ (according to the margin definition in Section 3.7.1) and all training vectors lie outside the margin, the corresponding true error probability (generalization error) is no more than c N
R2 2 1 ln N ⫹ ln ␥2
(5.68)
where c is a constant, and this bound holds true with a probability at least 1 ⫺ . Thus, adjusting the margin, as the SVM does, to be maximum we improve the bound, and this can be carried out even in an infinite dimensional space if the adopted kernel so dictates [Bart 99, Cris 00]. This result is logical. If the margin is large on a set of training points randomly chosen,this implies a classifier with large confidence, thus leading with high probability to good performance. The bound given previously was derived under the assumption that all training points are correctly classified. Furthermore, the margin constraint implies that for all training points yi f (x i ) ⱖ ␥, where f (x) denotes the linear classifier (the decision is taken according to sign( f (x)). A very interesting related bound refers to the more realistic case, where some of the training points are misclassified. Let k be the number of points with yi f (x i ) ⬍ ␥. (The product yf (x) is also known as the functional margin of the pair ( y, x) with respect to classifier f ( x)) Obviously, this also allows for negative values of the product. It can be shown that with probability at least 1 ⫺ the true error probability is upper bounded by ([Bart 99, Cris 00]) & k ⫹ N
c N
1 R2 2 ln N ⫹ ln ␥2
(5.69)
“07-Ch05-SA272” 17/9/2008 page 309
5.11 The Bayesian Information Criterion
Another bound relates the error performance of the SVM classifier with the number of support vectors. It can be shown [Bart 99] that if N is the number of training vectors and Ns the number of support vectors, the corresponding true error probability is bounded by 1 N ⫺ Ns
eN N Ns log2 ⫹ log2 Ns
(5.70)
where e is the base of the natural logarithm and the bound holds true with a probability at least 1 ⫺ . Note that this bound is also independent of the dimension of the feature space, where the design takes place. The bound increases with Ns and this must make the user, who has designed an SVM that results in a relatively large number (with respect to N ) of support vectors, cautious and“suspicious”about the performance of the resulting SVM classifier. The previous three bounds indicate that the error performance is controlled by both Ns and ␥. In practice, one may end up, for example, with a large number of support vectors and at the same time with a large margin. In such a case, the error performance could be assessed, with high confidence, depending on which of the two bounds has lower value.
5.11 THE BAYESIAN INFORMATION CRITERION The structural risk minimization principle,discussed in the previous section,belongs to a more general class of methods that estimate the order of a system by considering, simultaneously, the model complexity and a performance index. Depending on the function used to measure the model complexity and the corresponding performance index,different criteria result. In this section we will focus on one of such criteria, which provides a Bayesian theory flavor to the model selection problem. Moreover, it has a structural form that resembles a number of other popular criteria that have been proposed over the years. Although the criteria of this “family” lack the elegance and generality of the SRM principle, they can be useful in a number of cases. Furthermore, they shed light on the “performance versus complexity” trade-off task from another perspective. Let D ⫽ {(x i , yi ), i ⫽ 1, 2, . . . , N } be the training data set. We will focus on the Bayesian classification problem, and the goal is to adopt a parametric model for the class posterior probabilities; that is, P( yi |x; ), yi ∈ {1, 2, . . . , M} for an M class task. The cost used for the optimal choice of the unknown parameter is the log-likelihood function computed over the training set; that is, L() ⫽ ln p(D|). Let Mm denote one of the possible models described by the set of parameters m , where m runs over all candidate models. Let us also assume that for each model we know the prior information with respect to the distribution of m , that is, the pdf p(m |Mm ). Our goal is to choose the model for which the posterior probability
309
“07-Ch05-SA272” 17/9/2008 page 310
310
CHAPTER 5 Feature Selection
P(Mm |D) becomes maximum,over all candidate models. Using the Bayes theorem, we have P(Mm |D) ⫽
P(Mm )p(D|Mm ) p(D)
(5.71)
If we further assume that all models are equiprobable, P(Mm ) can be dropped out. The joint data pdf p(D), which is the same for all models, can also be neglected and our goal becomes to maximize p(D|Mm ) ⫽
p(D|m , Mm )p(m |Mm )dm
(5.72)
Employing a series of assumptions (e.g., Gaussian distribution for m ) and the so-called Laplacian approximation to the integral ([Schw 79, Ripl 96]), and taking the logarithm of both sides in Eq. (5.72) results in ln p(D|Mm ) ⫽ L( ˆ m ) ⫺
Km ln N 2
(5.73)
where L( ˆ m ) is the log-likelihood function computed at the ML estimate, ˆ m , and Km is the number of free parameters (i.e., the dimensionality of m ). Equivalently, one can search for the minimum of the quantity BIC ⫽ ⫺2L( ˆ m ) ⫹ Km ln N
(5.74)
The criterion is known as the Bayesian information criterion (BIC) or the Schwartz criterion. In other words, the best model indicated by this criterion depends (a) on the value of the log-likelihood function at its maximum (i.e., the adopted performance index) and (b) on a term that depends on the complexity of the model and the number of data points. If the model is too simple to describe the distribution underlying the given data set, the first term in the criterion will have a large value, since the probability of having obtained the set D from such a model will be small. On the other hand, if the model is complex, with a large number of free parameters that can adequately describe the data, the first term will have a small value, which however, is penalized by a large value of the second term. BIC provides a trade-off between these two terms. It can be shown that BIC is asymptotically consistent. This means that if the family of the candidate models contains the true one, then as N ⫺→⬁ the probability that BIC will select the correct model tends to one. The Akaike information criterion (AIC) [Akai 74], though derived in a different way, has similar structure, and the only difference lies in the second term, which is 2Km instead of Km ln N (see also Section 16.4.1). In practice, it is not clear which model is to be used. It has been reported that for large values of N AIC tends to choose models that are too complex. On the other hand, for smaller values of N BIC tends to choose models that are too simple [Hast 01]. Besides the previous two criteria, a number of alternatives have also been suggested, such as [Riss 83, Mood 92, Leth 96, Wang 98]. For a review of such techniques, see, for example, [Ripl 96, Hast 01, Stoi 04a, Stoi 04b].
“07-Ch05-SA272” 17/9/2008 page 311
5.12 Problems
5.12 PROBLEMS 5.1 In Trunk’s example, discussed in Section 5.3, prove that the mean value and the variance of the variable z are given by E[z] ⫽ ||||2 ⫽ li⫽1 1i and z2 ⫽ ||||2 respectively. Also show that the probability of error is Pe ⫽
⬁
bl
z2 1 exp ⫺ dz √ 2 2
(5.75)
where l 1 bl ⫽ i
(5.76)
i⫽1
5.2 In Trunk’s example, as in Problem 5.1, show that the mean value and variance of z, in the case of unknown mean value, are given the Eqs. (5.7) and (5.8), respectively. Derive the formula for the probability of error and show that it tends to 0.5 as the number of features tends to infinity. 5.3 If xi , yi , i ⫽ 1, 2, . . . , N are independent samples of two Gaussian distributions (2N ⫺2)sz2 of the same variance 2 , show that the random variable , where 2 sz2
N N 1 2 2 ⫽ (xi ⫺ x) ¯ ⫹ ( yi ⫺ y¯ ) 2N ⫺ 2 i⫽1
i⫽1
where x, ¯ y¯ are the respective sample mean values,is chi-square distributed with 2N ⫺ 2 degrees of freedom. 5.4 Let N1 , N2 be the available values of a feature in two classes, respectively. The feature is assumed to follow a Gaussian distribution with the same variance in each class. Define the test statistic q⫽
( x¯ ⫺ y¯ ) ⫺ (1 ⫺ 2 ) sz N11 ⫹ N12
(5.77)
where sz2
N N2 1 1 2 2 ⫽ (xi ⫺ x) ¯ ⫹ ( yi ⫺ y¯ ) N1 ⫹ N 2 ⫺ 2 i⫽1
i⫽1
and 1 , 2 are the respective true mean values. Show that q follows the t-distribution with N1 ⫹ N2 ⫺ 2 degrees of freedom.
311
“07-Ch05-SA272” 17/9/2008 page 312
312
CHAPTER 5 Feature Selection
5.5 Show that the matrix ⎡
⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ A⫽⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣
√1 n
√1 n
√1 n
...
√1 n
⫺1 √ 2
√1 2
0
...
0
⫺1 √ 6
⫺1 √ 6
√2 6
...
0
.. .
.. .
.. .
.. .
.. .
√ ⫺1 n(n⫺1)
√ ⫺1 n(n⫺1)
√ ⫺1 n(n⫺1)
...
√ n⫺1 n(n⫺1)
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
is orthogonal, that is, AAT ⫽ I . 5.6 Show that if xi , i ⫽ 1, 2, . . . , l, are jointly Gaussian, then the l variables yi , i ⫽ 1, 2, . . . , l, resulting from a linear transformation of them are also jointly Gaussian. Furthermore, if xi are mutually independent and the transformation is orthogonal, then yi are also mutually independent and Gaussian. 5.7 Let i , i ⫽ 1, 2, . . . , M,be the classes for a classification task. Divide the interval of the possible values of a feature into subintervals ⌬j , j ⫽ 1, 2, . . . , K . If P(⌬j ) is the probability of having values in the respective subinterval and P(i |⌬j ), the probability of occurrence of i in this interval, show that the so-called ambiguity function A⫽⫺
i
P(⌬j )P(i |⌬j ) logM (P(i |⌬j ))
j
is equal to 1 for completely overlapped distributions and is equal to 0 for perfectly separated ones. For all other cases it takes intermediate values. Thus, it can be used as a distribution overlap criterion [Fine 83]. 5.8 Show that if dij (x1 , x2 , . . . , xm ) is the class divergence based on m features, adding a new one xm⫹1 cannot decrease the divergence, that is, dij (x1 , x2 , . . . , xm ) ⱕ dij (x1 , x2 , . . . , xm , xm⫹1 )
5.9 Show that if the density functions are Gaussian in both classes with the same covariance matrix ⌺, then on adding a new feature xm⫹1 to the feature vector the new divergence is recursively computed by dij (x1 , . . . , xm⫹1 ) ⫽ dij (x1 , . . . , xm ) ⫹
[(i ⫺ j ) ⫺ (i ⫺ j )T ⌺⫺1 r]2 2 ⫺ r T ⌺⫺1 r
where i , j are the mean values of xm⫹1 for the two classes, 2 is its variance, r is its cross-covariance vector with the other elements of x, and i , j are the mean vectors of x prior to xm⫹1 . If xm⫹1 is now uncorrelated with the previously selected features x1 , . . . , xm , then this becomes dij (x1 , . . . , xm⫹1 ) ⫽ dij (x1 , . . . , xm ) ⫹
(i ⫺ j )2 2
“07-Ch05-SA272” 17/9/2008 page 313
5.12 Problems
5.10 Show that if the features are statistically independent, then the divergence is given by dij (x1 , x2 , . . . , xl ) ⫽
l
dij (xi )
i⫽1
5.11 Show that in the case of Gaussian distributions the Chernoff bound becomes ⑀CB ⫽ exp(⫺b(s))
where b(s) ⫽
s(1 ⫺ s) (i ⫺ j )T [s⌺j ⫹ (1 ⫺ s)⌺i ]⫺1 (i ⫺ j ) 2 1 | s⌺j ⫹ (1 ⫺ s)⌺i | ⫹ ln 2 | ⌺j |s | ⌺i |1⫺s
Then take the derivative with respect to s and show that for equal covariance matrices the optimum is achieved for s ⫽ 1/2. Thus, in this case b(s) equals the Bhattacharyya distance. 5.12 Show that the mixture scatter matrix is the sum of the within-class and between-class scatter matrices. 5.13 Show that the cross-correlation coefficient in (5.29) lies in the interval [⫺1, 1]. Hint: Use Schwartz’s inequality |x T y|ⱕ x y . 5.14 Show that for a two-class problem and Gaussian distributed feature vectors, with the same covariance matrix in the two classes, which are assumed equiprobable, the divergence is equal to ⫺1 trace{Sw Sb }
5.15 Show that the number of combinations to be searched using the backward search technique is given by 1 ⫹ 1/2((m ⫹ 1)m ⫺ l(l ⫹ 1))
5.16 Show that the optimal solution of the generalized Rayleigh quotient in (5.37) satisfies (5.38). 5.17 Show that ⭸ trace{(AT S1 A)⫺1 (AT S2 A)} ⫽ ⫺2S1 A(AT S1 A)⫺1 (AT S2 A)(AT S1 A)⫺1 ⭸A ⫹ 2S2 A(AT S1 A)⫺1
5.18 Show that for an M-class problem the matrix Sb is of rank M ⫺ 1. Hint: Recall that 0 ⫽ i Pi i .
313
“07-Ch05-SA272” 17/9/2008 page 314
314
CHAPTER 5 Feature Selection
5.19 Show that if fi (x), i ⫽ 1, . . . , M, are the discriminant functions of an M-class problem, we can construct from them M ⫺ 1 new functions that are, in principle, sufficient for the classification. Hint: Consider the differences fi (x) ⫺ fj (x). ⫺1 S is 5.20 Show that for a two-class problem the nonzero eigenvalue of matrix Sw b equal to ⫺1 1 ⫽ P1 P2 (1 ⫺ 2 )T Sxw (1 ⫺ 2 )
and the corresponding eigenvector ⫺1 v 1 ⫽ Sxw (1 ⫺ 2 )
where P1 , P2 are the respective class probabilities. 5.21 Show that if matrices ⌺1 , ⌺2 are two covariance matrices, then the eigenvectors of ⌺⫺1 1 ⌺2 are orthogonal with respect to ⌺1 , that is, v Ti ⌺1 v j ⫽ ␦ij
Hint: Use the fact that ⌺1 , ⌺2 can be simultaneously diagonalized (Appendix B). 5.22 Show that in a multilayer perceptron with a linear output node, minimizing the squared error is equivalent to maximizing (5.51). Hint: Assume the weights of the nonlinear nodes fixed and compute first the LS optimal weights driving the linear output nodes. Then substitute these values into the sum of error squares cost function. 5.23 Compute the minimal enclosure (hyper)sphere, that is, the radius as well as its center, of a set of points x i , i ⫽ 1, 2, . . . , N .
MATLAB PROGRAMS AND EXERCISES Computer Programs 5.1 Scatter matrices. Write a MATLAB function named scatter_mat that computes (a) the within-class (Sw), (b) the between-class (Sb) and the mixture (Sm) scatter matrices for a c-class classification problem, taking as inputs (a) an l ⫻ N dimensional matrix X, whose ith row is the ith data vector and (b) an N dimensional row vector y whose ith element contains the class label for the ith vector in X (the jth class is denoted by the integer j, j ⫽ 1, . . . , c).
Solution function [Sw,Sb,Sm]=scatter_mat(X,y) [l,N]=size(X);
“07-Ch05-SA272” 17/9/2008 page 315
MATLAB Programs and Exercises
c=max(y); %Computation of class mean vectors, a priori prob. and %Sw m=[]; Sw=zeros(l); for i=1:c y_temp=(y==i); X_temp=X(:,y_temp); P(i)=sum(y_temp)/N; m(:,i)=(mean(X_temp'))'; Sw=Sw+P(i)*cov(X_temp'); end %Computation of Sb m0=(sum(((ones(l,1)*P).*m)'))'; Sb=zeros(l); for i=1:c Sb=Sb+P(i)*((m(:,i)-m0)*(m(:,i)-m0)'); end %Computation of Sm Sm=Sw+Sb;
5.2 J 3 criterion. Write a MATLAB function named J 3_comp that takes as inputs the within-class (Sw) and the mixture (Sm) scatter matrices and returns the value of the J 3 criterion
Solution function J3=J3_comp(Sw,Sm) J3=trace(inv(Sw)*Sm);
5.3 Best features combination. Write a MATLAB function named features_ best_combin that takes as inputs (a) an l ⫻ N dimensional matrix X, whose ith row is the ith data vector, (b) an N dimensional row vector y, whose ith element contains the class label for the ith vector in X (the jth class is denoted by the integer j, j ⫽ 1, . . . , c), and (c) an integer q, the number of required features. It returns the best combination of q, out of the l, available features, according to the J 3 criterion.
Solution function id=features_best_combin(X,y,q) [l,N]=size(X); J3_max=0; id=[]; combin=nchoosek(1:l,q); for j=1:size(combin,1)
315
“07-Ch05-SA272” 17/9/2008 page 316
316
CHAPTER 5 Feature Selection
X1=X(combin(j,:),:); [Sw,Sb,Sm]=scatter_mat(X1,y); J3=J3_comp(Sw,Sm) if(J3>J3_max) J3_max=J3; id=combin(j,:); end end
5.4 FDR criterion. Write a MATLAB function named FDR_comp that returns the FDR index for a c class problem taking as inputs (a) an l ⫻ N dimensional matrix X, whose ith row is the ith data vector, (b) an N dimensional row vector y, whose ith element contains the class label for the ith vector in X (the jth class is denoted by the integer j, j ⫽ 1, . . . , c), and (c) the index ind of the feature over which the FDR will be computed.
Solution function FDR=FDR_comp(X,y,ind) [l,N]=size(X); c=max(y); for i=1:c y_temp=(y==i); X_temp=X(ind,y_temp); m(i)=mean(X_temp); vari(i)=var(X_temp); end a=nchoosek(1:c,2); q=(m(a(:,1))-m(a(:,2))).^ 2 ./ (vari(a(:,1))+vari(a(:,2)))'; FDR=sum(q);
Computer Experiments 5.1 a. Generate N 1 ⫽ 100 random numbers from the zero mean unit variance normal distribution and another N 2 ⫽ 100 random numbers from the unit variance normal distribution with mean value equal to 2. Assume that these numbers correspond to the values a specific feature takes in the framework of a two-class problem. Use the t-test to check whether or not the hypothesis that the mean values for this feature, for the two classes, differ significantly, at a 5% significance level. b. Repeat (a) when the mean value for the second distribution is 0.2. c. Repeat (a) and (b) when N 1 ⫽ 150 and N 2 ⫽ 200. Comment on the results.
“07-Ch05-SA272” 17/9/2008 page 317
MATLAB Programs and Exercises
Hint: Use the normrnd MATLAB function to generate the random numbers and the ttest2, to perform the t-test. 5.2 a. (i) Generate four sets, each one consisting of 100 two-dimensional vectors, from the normal distributions with mean values [⫺10, ⫺10]T , [⫺10, 10]T , [10, ⫺10]T , [10, 10]T and covariance matrices equal to 0.2 ∗ I . These sets constitute the data set for a four-class two-dimensional classification problem (each set corresponds to a class). a. (ii) Compute the Sw, Sb, and Sm scatter matrices. a. (iii) Compute the value for the criterion J 3. b. Repeat (a) when the mean vectors of the normal distributions that generate the data are [⫺1, ⫺1]T , [⫺1, 1]T , [1, ⫺1]T , [1, 1]T . c. Repeat (a) when the covariance matrices of the normal distributions that generate the data are equal to 3 ∗ I . 5.3 Generate two sets, each one consisting of 100 five-dimensional vectors, from the normal distributions with mean values [0, 0, 0, 0, 0]T and [0, 2, 2, 3, 3]T and covariance matrices equal to ⎡
0.5 ⎢0 ⎢ ⎢ ⎢0 ⎢ ⎣0 0
0 0.5 0 0 0
0 0 1 0 0
0 0 0 1 0
⎤ 0 0 ⎥ ⎥ ⎥ 0 ⎥. ⎥ 0 ⎦ 1.5
Their composition forms the data set for a two-class two-dimensional classification problem (each set corresponds to a class). Using the J3 criterion find the best combination of features if: a. they are considered individually. b. they are considered in pairs. c. they are considered in triples. d. Justify the results. 5.4 a. (i) Generate two sets, each one consisting of 100 two-dimensional vectors, from the normal distributions with mean values [2, 4]T and [2.5, 10]T and covariance matrices equal to the 2 ⫻ 2 identity matrix I . Their composition forms the data set for a two class two dimensional classification problem (each set corresponds to a class). a. (ii) Compute the value of the FDR index for both features. b. Repeat (a) when the covariance matrices of the normal distributions that generate the data are both equal to 0.25 ∗ I . c. Discuss the results.
317
“07-Ch05-SA272” 17/9/2008 page 318
318
CHAPTER 5 Feature Selection
REFERENCES [Akai 74] Akaike H. “A new look at the statistical model identification,” IEEE Transactions on Automatic Control,Vol. 19(6), pp. 716–723, 1974. [Ambr 02] Ambroise C., McLachlan G.J. “Selection bias in gene extraction on the basis of microarray gene-expression data,” Proceedings of the National Academy of Sciences, Vol. 99(10), pp. 6562–6566, 2002. [Bart 99] Bartlett P., Shawe-Taylor J. “Generalization performance of support vector machines and other pattern classifiers,” in Advances in Kernel Methods: Support Vector Learning (Schcölkopf S., Burges J.C., Smola A., eds.), MIT Press, 1999. [Bati 94] Batiti R. “Using mutual information for selecting features in supervised neural network learning,” IEEE Transactions on Neural Networks,Vol. 5(8), pp. 537–550, 1994. [Baud 00] Baudat G., Anouar F. “Generalized discriminant analysis using a kernel approach,” Neural Computation,Vol. 12(10), pp. 2385–2404, 2000. [Baum 89] Baum E.B., Haussler D.“What size net gives valid generalization,”Neural Computation, Vol. 1(1), pp. 151–160, 1989. [Belh 97] Belhumeour P.N., Hespanha J.P., Kriegman D.J. “Eigenfaces vs Fisherfaces: Recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 19(7), pp. 711–720, 1997. [Bish 95] Bishop C. Neural Networks for Pattern Recognition, Oxford University Press, 1995. [Bour 88] Bourland H., Kamp Y. “Auto-association by multilayer perceptrons and singular value decomposition,” Biological Cybernetics,Vol. 59, pp. 291–294, 1988. [Brad 97] Bradley A. “The use of the area under the ROC curve in the evaluation of machine learning algorithms,” Pattern Recognition,Vol. 30 (7), pp. 1145–1159, 1997. [Brun 00] Brunzell H., Erikcson J. “Feature reduction for classification of multidimensional data,” Pattern Recognition,Vol. 33, pp. 1741–1748, 2000. [Burg 98] Burges C.J.C. “A tutorial on support vector machines for pattern recognition,” Data Mining and Knowledge Discovery,Vol. 2(2), pp. 1–47, 1998. [Butz 05] Butz T.,Thiran J.P. “From error probability to information theoretic (multi-modal) signal processing,” Signal Processing),Vol. 85(5), pp. 875–902, 2005. [Cevi 05] Cevikalp H., Neamtu M., Wilkes M., Barkana A. “Discriminative common vectors for face recognition,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 27(1), pp. 4–13, 2005. [Chat 97] Chatterjee C., Roychowdhury V. “On self-organizing algorithms and networks for class-separability features,” IEEE Transactions on Neural Networks, Vol. 8(3), pp. 663–678, 1997. [Chen 00] Chen L.-F., Liao H.-Y.M., Ko M.-T., Lin J.-C.,Yu G.-J. “A new LDA-based face recognition system which can solve the small sample size problem,” Pattern Recognition, Vol. 33(10), pp. 1713–1726, 2000. [Choi 03] Choi E., Lee C. “Feature extraction based on the Bhattacharyya distance,” Pattern Recognition Letters,Vol. 36, pp. 1703–1709, 2003. [Cris 00] Cristianini N., Shawe-Taylor J. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press, Cambridge, MA, 2000. [Devi 82] Devijver P.A., Kittler J. Pattern Recognition;A Statistical Approach, Prentice Hall, 1982.
“07-Ch05-SA272” 17/9/2008 page 319
References
[Devr 96] Devroye L.,Gyorfi L.,Lugosi G. A Probabilistic Theory of Pattern Recognition,SpringerVerlag, 1996. [Duin 00] Duin R.P.W. “Classifiers in almost empty spaces,” Proceedings of the 15th Int. Conference on Pattern Recognition (ICPR), vol. 2, Pattern Recognition and Neural Networks, IEEE Computer Society Press, 2000. [Erdo 03] Erdogmus D., Principe J. “Lower and upper bounds for misclassification probability based on Renyi’ s information,” Journal of VLSI Signal Processing, 2003. [Fine 83] Finette S., Bleier A., Swindel W. “Breast tissue classification using diagnostic ultrasound and pattern recognition techniques: I. Methods of pattern recognition,” Ultrasonic Imaging, Vol. 5, pp. 55–70, 1983. [Fish 36] Fisher R.A. “The use of multiple measurements in taxonomic problems,” Annals of Eugenics,Vol. 7, pp. 179–188, 1936. [Fras 58] Fraser D.A.S. Statistics: An Introduction, John Wiley & Sons, 1958. [Frie 89] Friedman J.H. “Regularized discriminant analysis,” Journal of American Statistical Association,Vol. 84, pp. 165–175, 1989. [Fuku 90] Fukunaga K. Introduction to Statistical Pattern Recognition, 2nd ed.,Academic Press, 1990. [Gelm 95] Gelman A., Rubin D.B., Carlin J., Stern H. Bayesian Data Analysis, Chapman & Hall, London, 1995. [Ghah 94] Ghaharamani Z., Jordan M.I. “Supervised learning from incomplete data via the EM approach,” in Advances in Neural Information Processing Systems (Cowan J.D.,Tesauro G.T., Alspector J., eds),Vol. 6, pp. 120–127, Morgan Kaufmann, San Mateo, CA, 1994. [Guyo 03] Guyon I, Elisseeff A. “An introduction to variable and feature selection,” Journal of Machine Learning Research,Vol. 3, pp. 1157–1182, 2003. [Hams 08] Hamsici O. C.,Martinez A. M.“ Bayes optimality in LDA,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 30(4), pp. 647–657, 2008. [Hast 95] Hastie T., Tibshirani R. “Penalized discriminant analysis,” Annals of Statistics, Vol. 23, pp. 73–102, 1995. [Hast 01] Hastie T.,Tibshirani R., Friedman J. The Elements of Statistical Learning, Springer, 2001. [Hube 81] Huber P.J. Robust Statistics, John Wiley & Sons, 1981. [Hush 93] Hush D.R., Horne B.G. “Progress in supervised neural networks,” Signal Processing Magazine,Vol. 10(1), pp. 8–39, 1993. [Jain 97] Jain A., Zongker D. “Feature selection: Evaluation, application, and small sample performance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19(2), pp. 153–158, 1997. [Kim 07] Kim C., Choi C.-H. “ A discriminant analysis using composite features for classification problems,” Pattern Recognition,Vol. 40(11), pp. 2958–2967, 2007. [Kitt 78] Kittler J. “Feature set search algorithms,” in Pattern Recognition and Signal Processing (Chen C.H., ed.), pp. 41–60, Sijthoff and Noordhoff, Alphen aan den Rijn, The Netherlands, 1978. [Kram 91] Kramer M.A. “Nonlinear principal component analysis using auto-associative neural networks,”AIC Journal,Vol. 37(2), pp. 233–243, 1991. [Kulb 51] Kullback S., Liebler R.A. “On information and sufficiency,” Annals of Mathematical Statistics,Vol. 22, pp. 79–86, 1951.
319
“07-Ch05-SA272” 17/9/2008 page 320
320
CHAPTER 5 Feature Selection
[Kwak 02] Kwak N., Choi C.-H. “Input feature selection for classification problems,” IEEE Transactions on Neural Networks,Vol. 13(1), pp. 143–159, 2002. [Land 08] Landgrebe T. C. W., Duiy R. P. W. “Efficient multiclasss ROC approximation by decomposing via confusion matrix pertubation analysis,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 30(5), pp. 810–822, 2008. [Lawe 66] Lawer E.L.,Wood D.E. “Branch and bound methods: A survey,” Operational Research, Vol. 149(4), 1966. [Lee 00] Lee C., Choi E. “Bayes error evaluation of the Gaussian ML classifier,” IEEE Transactions on Geoscience Remote Sensing,Vol. 38(3), pp. 1471–1475, 2000. [Lee 93] Lee C., Landgrebe D.A. “Decision boundary feature extraction for nonparametric classifiers,” IEEE Transactions on Systems Man and Cybernetics,Vol. 23, pp. 433–444, 1993. [Lee 97] Lee C., Landgrebe D. “Decision boundary feature extraction for neural networks,” IEEE Transactions on Neural Networks,Vol. 8(1), pp. 75–83, 1997. [Leiv 07] Leiva-Murillo J.M., Artes-Rodriguez A. “Maximization of mutual information for supervised linear feature extraction,” IEEE Transactions on Neural Networks, Vol. 18(5), pp. 1433–1442, 2007. [Leth 96] Lethtokanga S.M., Saarinen J., Huuhtanen P., Kaski K. “Predictive minimum description length criterion for time series modeling with neural networks,” Neural Computation,Vol. 8, pp. 583–593, 1996. [Li 06] Li H., Jiang T., Zhang K. “Efficient and robust extraction by maximum margin criterion,” IEEE Transactions on Neural Networks,Vol. 17(1), pp. 157–165, 2006. [Lin 02] Lin Y., Wahba G., Zhang H., Lee Y. “Statistical properties and adaptive tuning of support vector machines,” Machine Learning,Vol. 48, pp. 115–136, 2002. [Loog 04] Loog M., Duin P.W. “Linear Dimensionality reduction via a heteroscedastic extension of LDA: The Chernoff criterion,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 26(6), pp. 732–739, 2004. [Lowe 90] Lowe D., Webb A.R. “Exploiting prior knowledge in network optimization: An illustration from medical prognosis,” Network: Computation in Neural Systems, Vol. 1(3), pp. 299–323, 1990. [Lowe 91] Lowe D., Webb A.R. “Optimized feature extraction and the Bayes decision in feedforward classifier networks,”IEEE Transactions in Pattern Analysis and Machine Intelligence, Vol. 13(4), pp. 355–364, 1991. [Ma 03] Ma J.,Jose L. S.,Ahalt S.“Nonlinear multiclass discriminant analysis,”IEEE Signal Processing Letters,Vol. 10(33), pp. 196–199, 2003. [Mama 96] Mamamoto Y., Uchimura S., Tomita S. “On the behaviour of artificial neural network classifiers in high dimensional spaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 18(5), pp. 571–574, 1996. [Mao 95] Mao J., Jain A.K. “Artificial neural networks for feature extraction and multivariate data projection,” IEEE Transactions on Neural Networks,Vol. 6(2), pp. 296–317, 1997. [Marr 08] Marroco C., Duin R. P. W., Tortorella F. “Maximizing the area under the ROC curve by pairwise feature combination,” Pattern Recognition,Vol. 41, pp. 1961–1974, 2008. [Maus 90] Mausel P.W., Kramber W.J., Lee J.K. “Optimum band selection for supervised classification of multispectra data,” Photogrammetric Engineering and Remote Sensing Vol. 56, pp. 55–60, 1990.
“07-Ch05-SA272” 17/9/2008 page 321
References
[Mood 92] Moody J.E. “The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems” in Advances in Neural Computation (Moody J.E., Hanson S.J., Lippman R.R., eds.), pp. 847–854, Morgan Kaufman, San Mateo, CA, 1992. [Nena 07] Nenadic Z. “Information discriminant analysis: feature extraction with an informationtheoretic objective,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29(8), pp. 1394–1408, 2007. [Papo 91] Papoulis A. Probability Random Variables and Stochastic Processes, 3rd ed., McGrawHill, 1991. [Pudi 94] Pudil P., Novovicova J., Kittler J. “Floating search methods in feature selection,” Pattern Recognition Letters,Vol. 15, pp. 1119–1125, 1994. [Raud 91] Raudys S.J., Jain A.K. “Small size effects in statistical pattern recognition: Recommendations for practitioners,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 13(3), pp. 252–264, 1991. [Raud 80] Raudys S.J.,PikelisV.“On dimensionality,sample size,classification error,and complexity of classification algorithms in pattern recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 2(3), pp. 243–251, 1980. [Rich 95] Richards J. Remote Sensing Digital Image Analysis, 2nd ed., Springer-Verlag, 1995. [Ripl 96] Ripley B.D. Pattern Recognition And Neural Networks, Cambridge University Press, Cambridge, MA, 1996. [Riss 83] Rissanen J.“A universal prior for integers and estimation by minimum description length,” The Annals of Statistics,Vol. 11(2), pp. 416–431, 1983. [Rubi 76] Rubin D.B. “Inference and missing data,” Biometrika,Vol. 63, pp. 581–592, 1976. [Rubi 87] Rubin D.B. Multiple Imputation for Nonresponse in Surveys, John Wiley & Sons, 1987. [Samm 69] Sammon J.W. “A nonlinear mapping for data structure analysis,” IEEE Transactions on Computers,Vol. 18, pp. 401–409, 1969. [Scha 02] Schafer J., Graham J. “Missing data: Our view of the state of the art,” Psychological Methods, vol. 7(2), pp. 67–81, 2002. [Schw 79] Schwartz G. “Estimating the dimension of the model,” Annals of Statistics, Vol. 6, pp. 461–464, 1978. [Seti 97] Setiono R., Liu H. “Neural network feature selector,” IEEE Transactions on Neural Networks,Vol. 8(3), pp. 654–662, 1997. [Sind 04] Sindhwami V., Rakshit S., Deodhare D., Erdogmus D., Principe J.C., Niyogi P. “Feature selection in MLPs and SVMs based on maximum output information,” IEEE Transactions on Neural Networks,Vol. 15(4), pp. 937–948, 2004. [Stoi 04b] Stoica P., Moses R. Spectral Analysis of Signals, Prentice Hall, 2004. [Stoi 04a] Stoica P., Selén Y. “A review of information criterion rules,”Signal Processing Magazine, Vol. 21(4), pp. 36–47, 2004. [Su 94] Su K.Y, Lee C.H. “Speech recognition using weighted HMM and subspace projection approaches,” IEEE Transactions on Speech and Audio Processing, Vol. 2(1), pp. 69–79, 1994.
321
“07-Ch05-SA272” 17/9/2008 page 322
322
CHAPTER 5 Feature Selection
[Swai 73] Swain P.H., King R.C. “Two effective feature selection criteria for multispectral remote sensing,” Proceedings of the 1st International Conference on Pattern Recognition, pp. 536–540, 1973. [Tian 86] Tian Q., Marbero M., Gu Z.H., Lee S.H. “Image classification by the Folley-Sammon transform,” Optical Engineering,Vol. 25(7), pp. 834–840, 1986. [Tou 74] Tou J., Gonzalez R.C. Pattern Recognition Principles,Addison-Wesley, 1974. [Trun 79] Trunk G.V. “A problem of dimensionality: A simple example,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 1(3), pp. 306–307, 1979. [Tsud 03] Tsuda K.,Akaho S.,Asai K.“The EM algorithm for kernel matrix completion with auxiliary data,” Journal of Machine Learning Research,Vol. 4, pp. 67–81, 2003. [Vali 84] Valiant L. “A theory of the learnable,” Communications of the ACM, Vol. 27(11), pp. 1134–1142, 1984. [Vapn 82] Vapnik V.N. Estimation of Dependencies Based on Empirical Data, Springer-Verlag, 1982. [Vapn 95] Vapnik V.N. The Nature of Statistical Learning Theory, Springer-Verlag, 1995. [Vapn 98] Vapnik,V.N. Statistical Learning Theory, John Wiley & Sons, 1998. [Walp 78] Walpole R.E., Myers R.H. Probability and Statistics for Engineers and Scientists, Macmillan, 1978. [Wang 00] Wang W., Jones P., Partridge D. “A comparative study of feature salience ranking techniques,” Neural Computation,Vol. 13(7), pp. 1603–1623, 2000. [Wang 98] Wang Y., Adali T., Kung S.Y., Szabo Z. “Quantization and segmentation of brain tissues from MR images: A probabilistic neural network approach,” IEEE Transactions on Image Processing,Vol. 7(8), 1998. [Wata 97] Watanabe H., Yamaguchi T., Katagiri S. “Discriminative metric for robust pattern recognition,” IEEE Transactions on Signal Processing, Vol. 45(11), pp. 2655–2663, 1997. [Will 07] Williams D., Liao X., Xue Y., Carin L., Krishnapuram B. “On classification with incomplete data,” IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29(3), pp. 427–436, 2007. [Yang 02] Yang J., Yang J.-Y. “Why can LDA be performed in PCA transformed space?” Pattern Recognition,Vol. 36, pp. 563–566, 2002. [Ye 05] Ye J., Li Q. “A two stage linear discriminant analysis via QR decomposition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27(6), pp. 929–941, 2005. [Yu 93] Yu B.,Yuan B. “A more efficient branch and bound algorithm for feature selection,”Pattern Recognition,Vol. 26(6), pp. 883–889, 1993. [Zhan 02] Zhang H., Sun G. “Feature selection using tabu search method,” Pattern Recognition, Vol. 35, pp. 701–711, 2002. [Zhan 07] Zhang S., Sim T. “Discriminant subspace analysis: A Fukunaga-Koontz approach,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 29(10),pp. 1732–1745,2007.
“08-Ch06-SA272” 18/9/2008 page 323
CHAPTER
Feature Generation I: Data Transformation and Dimensionality Reduction
6
6.1 INTRODUCTION Feature generation is of paramount importance in any pattern recognition task. Given a set of measurements, the goal is to discover compact and informative representations of the obtained data. A similar process is also taking place in the human perception apparatus. Our mental representation of the world is based on a relatively small number of perceptually relevant features. These are generated after processing a large amount of sensory data, such as the intensity and the color of the pixels of the images sensed by our eyes, and the power spectra of the sound signals sensed by our ears. The basic approach followed in this chapter is to transform a given set of measurements to a new set of features. If the transform is suitably chosen, transform domain features can exhibit high information packing properties compared with the original input samples. This means that most of the classification-related information is “squeezed” in a relatively small number of features, leading to a reduction of the necessary feature space dimension. Sometimes we refer to such processing tasks as dimensionality reduction techniques. The basic reasoning behind transform-based features is that an appropriately chosen transform can exploit and remove information redundancies, which usually exist in the set of samples obtained by the measuring devices. Let us take for example an image resulting from a measuring device, for example, X-rays or a camera. The pixels (i.e., the input samples) at the various positions in the image have a large degree of correlation, due to the internal morphological consistencies of real-world images that distinguish them from noise. Thus, if one uses the pixels as features, there will be a large degree of redundant information. Alternatively, if one obtains the Fourier transform, for example, of a typical real-world image, it turns out that most of the energy lies in the low-frequency components, due to the high correlation between the pixels’ gray levels. Hence, using the Fourier coefficients as features seems a reasonable choice, because the low-energy, high-frequency coefficients can be neglected, with little loss of information. In this chapter we will
323
“08-Ch06-SA272” 18/9/2008 page 324
324
CHAPTER 6 Feature Generation I
see that the Fourier transform is just one of the tools from a palette of possible transforms.
6.2 BASIS VECTORS AND IMAGES Let x(0), x(1), . . . , x(N ⫺ 1) be a set of input samples and x be the N ⫻ 1 corresponding vector, x T ⫽ [x(0), . . . , x(N ⫺ 1)]
Given a unitary N ⫻ N matrix A,1 we define the transformed vector y of x as ⎡ ⎢ y ⫽ AH x ≡ ⎢ ⎣
aH 0 .. .
⎤ ⎥ ⎥x ⎦
(6.1)
aH N ⫺1
where H denotes the Hermitian operation, that is, complex conjugation and transposition. From (6.1) and the definition of unitary matrices we have x ⫽ Ay ⫽
N ⫺1
(6.2)
y(i)a i
i⫽0
The columns of A, a i , i ⫽ 0, 1, . . . , N ⫺ 1, are called the basis vectors of the transform. The elements y(i) of y are nothing but the projections of x onto these basis vectors. Indeed, taking the inner product of x with a j we have a j , x ≡ a H j x ⫽
N ⫺1 i⫽0
y(i)a j , a i ⫽
N ⫺1
y(i)␦ij ⫽ y( j)
(6.3)
i⫽0
This is due to the unitary property of A, that is, AH A ⫽ I or a i , a j ⫽ a H i a j ⫽ ␦ij . In many problems, such as in image analysis, the input set of samples is a twodimensional sequence X(i, j), i, j ⫽ 0, 1, . . . , N ⫺ 1, defining an N ⫻ N matrix X instead of a vector. In such cases, one can define an equivalent N 2 vector x, for example, by ordering the rows of the matrix one after the other (lexicographic ordering) x T ⫽ [X(0, 0), . . . , X(0, N ⫺ 1), . . . , X(N ⫺ 1, 0), . . . , X(N ⫺ 1, N ⫺ 1)]
and then transform this equivalent vector. However, this is not the most efficient way to work. The number of operations required to multiply an N 2 ⫻ N 2 square matrix (A) with an N 2 ⫻ 1 vector x is of the order of O(N 4 ), which is prohibitive 1 A complex matrix is called unitary if A⫺1 ⫽ AH . Real matrices are equivalently called orthogonal if A⫺1 ⫽ AT .
“08-Ch06-SA272” 18/9/2008 page 325
6.2 Basis Vectors and Images
for many applications. An alternative possibility is to transform matrix X via a set of basis matrices or basis images. Let U and V be unitary N ⫻ N matrices. Define the transformed matrix Y of X as Y ⫽ U H XV
(6.4)
X ⫽ UYV H
(6.5)
or
The number of operations is now reduced to O(N 3 ). Equation (6.5) can alternatively be written ( Problem 6.1) as X⫽
N ⫺1 N ⫺1
Y (i, j)ui v H j
(6.6)
i⫽0 j⫽0
where ui are the column vectors of U and v j the column vectors of V . Each of the outer products ui v H j is an N ⫻ N matrix ⎡
ui v H j
∗ ui0 vj0 ⎢ .. ⫽⎢ . ⎣ ∗ uiN ⫺1 vj0
⎤ ∗ ... ui0 vjN ⫺1 ⎥ .. .. ⎥ ≡ Aij . . ⎦ ∗ . . . uiN ⫺1 vjN ⫺1
and (6.6) is an expansion of matrix X in terms of these N 2 basis images (matrices). The ∗ denotes complex conjugation. Furthermore, if Y turns out to be diagonal, then (6.6) becomes X⫽
N ⫺1
Y (i, i)ui v H i
i⫽0
and the number of basis images is reduced to N . An interpretation similar to (6.3) is also possible. To this end, let us define the inner product between two matrices as A, B ≡
N ⫺1 N ⫺1
A∗ (m, n)B(m, n)
(6.7)
m⫽0 n⫽0
Then it is not difficult to show that ( Problem 6.1) Y (i, j) ⫽ Aij , X
(6.8)
In words, the (i, j) element of the transformed matrix results from multiplying each element of X by the conjugate of the corresponding element of Aij and summing up all products. Transformations of the type (6.4) are also known as separable ( Problem 6.2). The reason is that one can look at them as a succession of one-dimensional transforms, first applied on column vectors and then on row vectors. For example, the intermediate result in (6.4), Z ⫽ U H X, is equivalent to N transforms applied to the column
325
“08-Ch06-SA272” 18/9/2008 page 326
326
CHAPTER 6 Feature Generation I
vectors of X, and (U H X)V ⫽ (V H Z H )H is equivalent to a second sequence of N transforms acting upon the rows of Z. All the two-dimensional transforms that we will deal with in this chapter are separable ones. Example 6.1 Given the image X and the orthogonal transform matrix U
1 1 1 1 2 X⫽ , U ⫽ √ 2 3 2 1 ⫺1 the transformed image Y ⫽ U T XU is
1 1 1 1 Y ⫽ 2 1 ⫺1 2
2 3
1 1
1 4 ⫽ ⫺1 ⫺1
⫺1 0
The corresponding basis images are
1 1 1 1 1 [1,1] ⫽ A00 ⫽ , 2 1 2 1 1
1 1 1 1 ⫺1 [1, ⫺1] ⫽ A11 ⫽ 1 2 ⫺1 2 ⫺1 and similarly A01 ⫽
AT10
1 1 ⫽ 2 1
⫺1 ⫺1
Now verify that the elements of Y are obtained via the matrix inner products Aij , X.
6.3 THE KARHUNEN–LOÈVE TRANSFORM In Section 5.8 the problem of the linear transformation of a feature vector was considered in the spirit of linear discriminant analysis ( LDA). The class labels of the feature vectors were assumed known, and this information was optimally exploited to compute the transformation matrix. The linear transform task will also be considered in this section but from a different perspective. Here, the computation of the transformation matrix will exploit the statistical information describing the data, and it will take place in an unsupervised mode. The Karhunen–Loève transform or principal component analysis ( PCA), as it is also known, is one of the most popular methods for feature generation and dimensionality reduction in pattern recognition. Though an old technique, it is still in use, and it forms the basis for a number of more advanced approaches. Let x be the vector of input samples. In the case of an image array, x may be formed by lexicographic ordering of the array elements. In order to simplify the
“08-Ch06-SA272” 18/9/2008 page 327
6.3 The Karhunen–Loève Transform
presentation, we will assume that the data samples have zero mean. If this is not the case, we can always subtract the mean value. We have already mentioned that a desirable property of the generated features is to be mutually uncorrelated in an effort to avoid information redundancies. We begin this section by first developing a method that generates mutually uncorrelated features, that is, E[ y(i)y( j)] ⫽ 0, i ⫽ j. Let2 y ⫽ AT x
(6.9)
Since we have assumed that E[x] ⫽ 0, it is readily seen that E[ y] ⫽ 0. From the definition of the correlation matrix we have Ry ≡ E[ yy T ] ⫽ E[AT xx T A] ⫽ AT Rx A
(6.10)
In practice, Rx is estimated as an average over the given set of training vectors. For example, if we are given n data vectors x k , k ⫽ 1, 2, . . . , n, then Rx ≈
n 1 x k x Tk n
(6.11)
k⫽1
Note that Rx is a symmetric matrix, and hence its eigenvectors are mutually orthogonal (Appendix B). Thus, if matrix A is chosen so that its columns are the orthonormal eigenvectors a i , i ⫽ 0, 1, . . . , N ⫺ 1, of Rx , then Ry is diagonal (Appendix B) Ry ⫽ AT Rx A ⫽ ⌳
(6.12)
where ⌳ is the diagonal matrix having as elements on its diagonal the respective eigenvalues i , i ⫽ 0, 1, . . . , N ⫺ 1, of Rx . ( Recall that in Section 5.8 a linear transform of the form in (6.9) was also considered, but there the elements of matrix A were computed so that a class separability criterion could be optimized.) Furthermore, assuming Rx to be positive definite (Appendix B) the eigenvalues are positive. The resulting transform is known as the Karhunen–Loève (KL) transform, and it achieves our original goal of generating mutually uncorrelated features. The KL transform was introduced in [Karh 46] in the context of representing a random process in terms of orthogonal functions and in the discrete form used in this section in [Hote 33]. Other classical references of the topic are [Diam 96, Joll 86]. It has to be emphasized that the solution provided by the KL transform is not a unique one, and it was obtained by imposing an orthogonal structure on matrix A (AT A ⫽ I ). Also, note that for zero mean variables the correlation matrix R coincides with the covariance matrix ⌺. As a matter of fact, a direct consequence of the respective definitions is that ⌺x ⫽ Rx ⫺ E[x]E[x]T
2 We
deal with real data. The complex case is a straightforward extension.
327
“08-Ch06-SA272” 18/9/2008 page 328
328
CHAPTER 6 Feature Generation I
In case the zero mean assumption is not valid, the condition for uncorrelated variables becomes E[( y(i) ⫺ E[ y(i)])( y( j) ⫺ E[ y( j)])] ⫽ 0, i ⫽ j,and the problem results in the eigendecomposition of the covariance matrix, that is, ⌺y ⫽ AT ⌺x A ⫽ ⌳
(6.13)
Although our starting point was to generate mutually uncorrelated features, the KL transform turns out to have a number of other important properties, which provide different ways for its interpretation and also the secret for its popularity.
Mean Square Error Approximation From Eqs. (6.2) and (6.3) we have x⫽
N ⫺1
and
y(i)a i
y(i) ⫽ a Ti x
(6.14)
i⫽0
Let us now define a new vector in the m-dimensional subspace xˆ ⫽
m⫺1
(6.15)
y(i)a i
i⫽0
where only m of the basis vectors are involved. Obviously, this is nothing but the projection of x onto the subspace spanned by the m (orthonormal) eigenvectors involved in the summation. If we try to approximate x by its projection x, ˆ the resulting mean square error is given by
E x ⫺ x ˆ
2
⎡
2 ⎤ ⫺1
N
⫽ E ⎣ y(i)a i ⎦
(6.16)
i⫽m
Our goal now is to choose the eigenvectors that result in the minimum MSE. From (6.16) and taking into account the orthonormality property of the eigenvectors, we have ⎡ ⎤ ⎡
2 ⎤ ⫺1
N
E ⎣ y(i)a i ⎦ ⫽ E ⎣ ( y(i)a Ti )( y( j)a j )⎦
i⫽m
i
⫽
N ⫺1
E[ y2 (i)] ⫽
i⫽m
(6.17)
j
N ⫺1
a Ti E[xx T ]a i
(6.18)
i⫽m
Combining this with (6.16) and the eigenvector definition, we finally get ⫺1 N ⫺1 N E x ⫺ x ˆ 2 ⫽ a Ti i a i ⫽ i i⫽m
(6.19)
i⫽m
Thus, if we choose in (6.15) the eigenvectors corresponding to the m largest eigenvalues of the correlation matrix, then the error in (6.19) is minimized, being the sum of the N ⫺ m smallest eigenvalues. Furthermore, it can be shown
“08-Ch06-SA272” 18/9/2008 page 329
6.3 The Karhunen–Loève Transform
(Problem 6.3) that this is also the minimum MSE, compared with any other approximation of x by an m-dimensional vector. This is the reason that the KL transform is also known as principal component analysis ( PCA). A difficulty in practice is how to choose the m principal components. One way is to rank the eigenvalues in descending order, 0 ⱖ 1 ⱖ . . . ⱖ m⫺1 ⱖ m ⱖ . . . ⱖ N ⫺1 , and determine m so that the gap between the values m⫺1 and m is “large.” For more on this issue, see [Jack 91]. Note that the previous analysis concerning the MSE property of the KL transform, after projecting onto the m principal components of Rx , is still valid even if the mean of the data is not zero. However, in this case, although a minimum MSE solution is obtained, the approximation is not, in general, a good one (why?) In such cases, one tries to find the optimum m-dimensional subspace, so that the MSE between x and its following approximation xˆ ⫽
m⫺1 i⫽0
y(i)aˆ i ⫹
N ⫺1
bi aˆ i ,
y(i) ≡ aˆ Ti x
(6.20)
i⫽m
to be minimum,where bi , i ⫽ m, . . . , N ⫺1,are constants independent of x. It turns out (Problem 6.4) that the resulting orthonormal basis consists of the eigenvectors of the covariance matrix, ⌺x , where aˆ i , i ⫽ 0, 2, . . . , m ⫺ 1, correspond to the principal eigenvalues of ⌺x , and the constants are equal to bi ⫽ E[ y(i)] ⫽ aˆ Ti E[x],
i ⫽ m, . . . , N ⫺ 1
In other words, x is projected onto the subspace spanned by the m principal components of ⌺x and the rest N ⫺ m components are frozen to the respective mean values, in order to bring the estimate closer to its mean. Note, however, that the number of free parameters remains equal to m. The optimality of the KL transform, with respect to the MSE approximation, leads to excellent information packing properties and offers us a tool to select the m dominant features out of N measurement samples. However, although this may be a good criterion, in many cases it does not necessarily lead to maximum class separability in the lower dimensional subspace. This is reasonable, since the dimensionality reduction is not optimized with respect to class separability, as was, for example, the case with the scattering matrix criteria of the previous chapter. This is demonstrated via the example of Figure 6.1. The feature vectors in the two classes follow the Gaussian distribution with the same covariance matrix. The ellipses show the curves of constant pdf values. We have computed the eigenvectors of the overall correlation matrix, and the resulting eigenvectors are shown in the figure. Eigenvector a 0 is the one that corresponds to the largest eigenvalue. It does not take time for someone to realize that projection on a 0 makes the two classes almost coincide. However, projecting on a 1 keeps the two class separable.
Total Variance Let E[x] be zero. Let y be the KL transformed vector of x. From the respective 2 ≡ E[ y2 (i)] ⫽ . That is, the eigenvalues of the input definitions we have that y(i) i
329
“08-Ch06-SA272” 18/9/2008 page 330
330
CHAPTER 6 Feature Generation I
x2
a0 a1 x1
FIGURE 6.1 The KL transform is not always best for pattern recognition. In this example, projection on the eigenvector with the larger eigenvalue makes the two classes coincide. On the other hand, projection on the other eigenvector keeps the classes separated.
correlation matrix are equal to the variances of the transformed features. Thus, T selecting those features, y(i) ≡ a i x, corresponding to the m largest eigenvalues makes their sum variance i i maximum. In other words, the selected m features retain most of the total variance associated with the original random variables x(i). , which we know from linear algebra Indeed, the latter is equal to the trace of Rx ⫺1 to be equal to the sum of the eigenvalues N i⫽0 i [Stra 80]. It can be shown that this is a more general property. That is, from all possible sets of m features, obtained via any orthogonal linear transformation on x, the ones resulting from the KL transform have the largest sum variance (Problem 6.3). If the mean value is not zero, to maximize the sum variance, we use ⌺x in place of Rx .
Entropy We know from Chapter 2 that the entropy of a process is defined as Hy ⫽ ⫺E[ln py (y)]
and it is a measure of the randomness of the process. For a zero mean Gaussian multivariable m-dimensional process, the entropy becomes Hy ⫽
1 1 m E[y T R⫺1 ln |Ry | ⫹ ln(2) y y] ⫹ 2 2 2
However, T ⫺1 ⫺1 T E[y T R⫺1 y y] ⫽ E[trace{y Ry y}] ⫽ E[trace{Ry yy }] ⫽ trace(I ) ⫽ m
(6.21)
“08-Ch06-SA272” 18/9/2008 page 331
6.3 The Karhunen–Loève Transform
and using the known property from linear algebra the determinant is ln|Ry | ⫽ ln(0 1 . . . m⫺1 )
In words, selection of the m features that correspond to the m largest eigenvalues maximizes the entropy of the process. This is expected because variance and randomness are directly related.
Dimensionality Reduction PCA achieves a linear transformation of a high-dimensional input vector into a low-dimensional one whose components are uncorrelated. As already stated, we can assume that E[x] is zero, without loss of generality. Assuming that the N ⫺ m smallest eigenvalues of the correlation matrix are zero, then Eq. (6.19) suggests that x ⫽ x. ˆ In other words, vector x lies in an m-dimensional subspace ([Eq. (6.15)]) of the original N -dimensional space. This brings us to the notion of intrinsic dimensionality. A data set X ⊂ RN is said to have intrinsic dimensionality (ID) m ⬍ N , if X can be described in terms of m free parameters. For example, if X consists of vectors whose components are functions of m random variables, xi ⫽ gi (u1 , u2 , . . . , um ), i ⫽ 1, 2, . . . , N , ui ∈ R, then the intrinsic dimensionality of X is m. The geometric interpretation of this is that the entire data set lies on a m-dimensional hypersurface (manifold) in RN . Take as an example the case of a random variable and the functions x1 ⫽ r cos , x2 ⫽ r sin
It does not take time to see that x ⫽ [x1 , x2 ]T lies on the perimeter of the circle with radius equal to r. This is a one-dimensional surface (manifold) since one parameter suffices to describe the data (the length across the circumference from a point, origin, on the perimeter of the circle). From a statistical point of view, the fact that the intrinsic or “effective” dimension is smaller than the “apparent” one means that the features in the data set are correlated. The PCA method has been used extensively for dimensionality reduction and for estimation of the ID of a data set. If ID ⫽ m ⬍ N , then in theory there will be N ⫺ m zero eigenvalues. In practice, one has to ignore the eigenvalues with small values, and thus an approximation of the ID is obtained. PCA, being a linear projection method, works well if the data points are distributed, more or less, throughout a hyperplane. The eigenvalue–eigenvector decomposition of the correlation (covariance) matrix reveals the dimensionality of this hyperplane across which data are spread; in other words, dimensionality is a measure of the number of underlying modes of variability. Recall that PCA projects across the directions of maximum variance. For more general cases, however where the generation mechanism of the data is highly nonlinear and they lie on more complicated manifolds, PCA fails, and it tends to overestimate the true value of the ID. For example,for the case considered before,
331
“08-Ch06-SA272” 18/9/2008 page 332
332
CHAPTER 6 Feature Generation I
where all the data points lie on the perimeter of a circle in the two-dimensional space, PCA would result in ID ⫽ 2, although the true value is ID ⫽ 1. For such cases, nonlinear dimensionality reduction techniques have been developed and used. For example, in [Karh 94], a special type of neural network with three hidden layers is proposed to perform nonlinear PCA. We will return to this issue in Section 6.7. As we have seen in Chapter 5, LDA is another linear method that has been used for dimensionality reduction. However, in the case of LDA, this is achieved in a supervised manner; that is, the lower dimensional space is chosen in order to preserve a class separability measure. Another linear technique used to project in a lower dimensional space, while respecting certain constraints, is the metric multidimensional scaling (MDS). Given the set X ⊂ RN , the goal is to project into a lower dimensional space, Y ⊂ Rm , so that inner products are optimally preserved, that is, E⫽
x Ti x j ⫺ y Ti y j
i
2
j
is minimized, where y i is the image of x i and the sum runs over all the training points in X. The problem is similar to the PCA, and it can be shown that the solution is given by the eigendecomposition of the Gram matrix,whose elements are defined as K(i, j) ⫽ x Ti x j
Another side of the same coin is to require the Euclidean distances, instead of the inner products, to be optimally preserved. A Gram matrix, consistent with the squared Euclidean distances can then be formed, leading to the same solution as before. It turns out that the solutions obtained by PCA and MDS are equivalent. We can see this by the following simple reasoning. PCA performs the eigen decomposition of the correlation matrix Rx , which is approximated by Rx ⫽ E[xx T ] ≈
n 1 1 x k x Tk ⫽ X T X n n
(6.22)
k⫽1
where, as we have defined in (3.44), X T ⫽ [x 1 , x 2 , . . . , x n ]
On the other hand, the Gram matrix can also be written as K ⫽ XX T
However,as we will see in more detail in Section 6.4,the two matrices X T X and XX T are of the same rank and have the same eigenvalues. Their eigenvectors, although different, are related. More on these issues can be found in, for example, [Cox 94, Burg 04]. As we will see in Section 6.6, the main idea behind MDS of preserving the distances is used, in one way or another, in a number of more recently developed nonlinear dimensionality reduction techniques.
“08-Ch06-SA272” 18/9/2008 page 333
6.3 The Karhunen–Loève Transform
Remarks ■
The concept of principal eigenvector subspace has also been exploited as a classifier. First, the sample mean of the whole training set is subtracted from the feature vectors. For each class, i , the correlation matrix Ri is estimated and the principal m eigenvectors (corresponding to the m largest eigenvalues) are computed. A matrix Ai is then formed using the respective eigenvectors as columns. An unknown feature vector x is then classified in the class j for which ATj x ⬎ ATi x , ᭙i ⫽ j
(6.23)
that is, the class corresponding to the maximum norm subspace projection of x [ Wata 73]. From the Pythagoras theorem this is equivalent to classifying a vector in its nearest class subspace. The decision surfaces are hyperplanes if all the subspaces have the same dimension or quadric surfaces in the more general case. Subspace classification integrates the stages of feature generation/selection and classifier design. If this approach results in a relatively high classification error, the performance may be improved by suitable modifications known as learning subspace methods. For example, one can iteratively rotate the subspaces to adjust the lengths of the projections of the training vectors. The basic idea is to increase the length of a projection in the subspace of the correct class and decrease it for the rest. Such techniques have been applied successfully in a number of applications, such as speech recognition, texture classification, and character recognition. The interested reader may consult, for example, [Oja 83, Koho 89, Prak 97]. ■
For the computation of the correlation matrix eigenvectors, a number of iterative schemes have been developed. The computation is performed working directly with the vectors, without having to estimate the corresponding correlation matrix, using neural network concepts [Oja 83, Diam 96].
Example 6.2 The correlation matrix of a vector x is given by ⎡ 0.3 0.1 ⎢ Rx ⫽ ⎣0.1 0.3 0.1 ⫺0.1
⎤ 0.1 ⎥ ⫺0.1⎦ 0.3
Compute the KL transform of the input vector. The eigenvalues of Rx are 0 ⫽ 1 ⫽ 0.4, 2 ⫽ 0.1. Since the matrix Rx is symmetric, we can always construct orthonormal eigenvectors. For this case we have ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 2 1 1 ⎢ ⎥ 1 ⎢ ⎥ 1 ⎢ ⎥ a 0 ⫽ √ ⎣1⎦, a 1 ⫽ √ ⎣ 1⎦, a 2 ⫽ √ ⎣⫺1⎦ 3 2 6 1 ⫺1 ⫺1
333
“08-Ch06-SA272” 18/9/2008 page 334
334
CHAPTER 6 Feature Generation I
The KL transform is then given by ⎡ √ 2/ 6 ⎢ ⎥ ⎢ ⎣ y(1)⎦ ⫽ ⎣ 0 √ y(2) 1/ 3 ⎡
y(0)
⎤
√ 1/ 6 √ 1/ 2 √ ⫺1/ 3
√ ⎤⎡ ⎤ x(0) 1/ 6 √ ⎥⎢ ⎥ ⫺1/ 2 ⎦ ⎣x(1)⎦ √ x(2) ⫺1/ 3
where y(0), y(1) correspond to the two largest eigenvalues.
Example 6.3 Figure 6.2 shows 100 points in the two-dimensional space. The points spread around the x2 ⫽ x1 line, and they have been generated by the model x2 ⫽ x1 ⫹ ⑀, where ⑀ is a noise source following the uniform distribution in [⫺0.5, 0.5]. We first compute the covariance matrix and perform an eigendecomposition. The resulting eigenvectors are a 0 ⫽ [0.7045, 0.7097]T ,
a 1 ⫽ [⫺0.7097, 0.7045]T
corresponding to the eigenvalues 0 ⫽ 17.26,
1 ⫽ 0.04
respectively. Observe that 0 ⬎⬎ 1 . Figure 6.2 shows the two eigenvectors. a 0 , which correspond to the largest eigenvalue, points in the direction where data show maximum variability. Projecting along this direction retains most of the variance. Moreover, according to PCA, the dimensionality of the set is approximately one, due to the large gap between 0 and 1 , which is the correct answer. Also, note, that a 0 , is (approximately) parallel to the line x2 ⫽ x1 . x2 4 a0 2
0 a1 ⫺2 ⫺4 ⫺6
⫺4
⫺2
0
2
4
6 x1
FIGURE 6.2
Points around the x 2 ⫽ x 1 line. The eigenvectors of the associated covariance matrix are a 0 and a 1 . The principal eigenvector a 0 points in the direction of maximum variance.
“08-Ch06-SA272” 18/9/2008 page 335
6.4 The Singular Value Decomposition
6.4 THE SINGULAR VALUE DECOMPOSITION The singular value decomposition of a matrix is one of the most elegant and powerful algorithms in linear algebra, and it has been extensively used for rank and dimension reduction in pattern recognition and information retrieval applications. Given a l ⫻ n matrix X of rank r (obviously r ⱕ min{l, n}), we will show that there exist unitary matrices U and V of dimensions l ⫻ l and n ⫻ n, respectively, so that
X ⫽U
1
⌳2 O
O 0
V
H
or
Y ≡
1
⌳2 O
O 0
⫽ U H XV
(6.24)
√ 1 where ⌳ 2 is the r ⫻ r diagonal matrix with elements i , and i are the r nonzero eigenvalues of the associated matrix X H X. O denotes a zero element matrix. In other words, there exist unitary matrices U and V that transform X into the special diagonal structure of Y . If ui , v i denote the column vectors of matrices U and V , respectively, then Eq. (6.24) is written as ⎤⎡
⎡ √ 0 ⎢ ⎢ X ⫽ [u0 , u1 , . . . , ur⫺1 ] ⎢ ⎢ ⎣
√ 1 ..
.
√ r⫺1
vH 0
⎥⎢ ⎥ ⎢ vH 1 ⎥⎢ ⎥ ⎢ .. ⎦⎣ . vH r⫺1
⎤ ⎥ ⎥ ⎥ ⎥ ⎦
(6.25)
or X⫽
r⫺1
i ui v H i
(6.26)
i⫽0
Sometimes, the above is also written as 1
X ⫽ Ur ⌳ 2 VrH
(6.27)
where Ur denotes the l ⫻ r matrix that consists of the first r columns of U and Vr the r ⫻ n matrix formed by using the first r columns of V . More precisely, ui , v i are the eigenvectors corresponding to the nonzero eigenvalues of the matrices XX H and X H X, respectively. The eigenvalues i are known as singular values of X and the expansion in (6.26) as the singular value decomposition (SVD) of X or the spectral representation of X. Proof. Given a matrix X of rank r, it is known from linear algebra [Stra 80] that the n ⫻ n matrix X H X as well as the l ⫻ l matrix XX H are of the same rank r. Furthermore, both matrices have the same nonzero eigenvalues but different (yet related) eigenvectors ( Problem 6.5), XX H ui ⫽ i ui
(6.28)
X Xv i ⫽ i v i
(6.29)
H
335
“08-Ch06-SA272” 18/9/2008 page 336
336
CHAPTER 6 Feature Generation I
Since both matrices are Hermitian and nonnegative (i.e., (XX H )H ⫽ XX H ), they have nonnegative real eigenvalues and orthogonal eigenvectors (Appendix B). The eigenvectors, given in (6.28) and (6.29), can also be normalized to become H orthonormal, that is, uH i u i ⫽ 1 and v i v i ⫽ 1. It is straightforward to see from (6.28) and (6.29) that 1 ui ⫽ √ Xv i , for i ⫽ 0 i
(6.30)
Indeed, premultiplying (6.29) by X results in (XX H )Xv i ⫽ i Xv i
That is, ui ⫽ ␣Xv i , where (without loss of generality) the scaling factor ␣ can be taken as positive and it is found from 1 H 2 2 ui 2 ⫽ 1 ⫽ ␣2 v H i X Xv i ⫽ ␣ i v i ⇒ ␣ ⫽ √ i
Let us now assume that ui , v i , i ⫽ 0, 1, . . . , r ⫺ 1, are the eigenvectors corresponding to the nonzero eigenvalues and ui , i ⫽ r, . . . , l ⫺ 1, v i , i ⫽ r, . . . , n ⫺ 1, to the zero ones. Then, for the latter case we have H 2 X H Xv i ⫽ 0 ⇒ v H i X Xv i ⫽ 0 ⇒ Xv i ⫽ 0
Hence Xv i ⫽ 0, i ⫽ r, . . . , n ⫺ 1
(6.31)
In a similar way one can show that X H ui ⫽ 0, i ⫽ r, . . . , l ⫺ 1
(6.32)
Combining (6.30) and (6.31), we show that the right-hand side of (6.26) is r⫺1
r⫺1
i⫽0
i⫽0
i ui v H i ⫽X
n⫺1 1 i √ v i v H ⫽ X vi v H i i i i⫽0
(6.33)
Let us now define a matrix V that has as columns the orthonormal eigenvectors v i , V ⫽ [v 0 , . . . , v n⫺1 ]
Orthonormality of the columns results in V H V ⫽ I ; that is, V is unitary and hence VV H ⫽ I . Thus, it turns out that ⎡ ⎢ I ⫽ VV H ⫽ [v 0 , . . . , v n⫺1 ] ⎢ ⎣
vH 0 .. . vH n⫺1
⎤ ⎥ n⫺1 ⎥⫽ vi v H i ⎦ i⫽0
(6.34)
“08-Ch06-SA272” 18/9/2008 page 337
6.4 The Singular Value Decomposition
From (6.33) and (6.34) we obtain X⫽
r⫺1
i ui v H i
(6.35)
i⫽0
and X can be written as
X ⫽U
1
⌳2 O
O 0
VH
(6.36)
where U is the unitary matrix with columns the orthonormal eigenvectors ui .
Low Rank Approximation The expansion in (6.26) is an exact representation of matrix X. A very interesting implication occurs if one uses less than r (the rank of X) terms in the summation. Let X be approximated by X Xˆ ⫽
k⫺1
i ui v H i , kⱕr
(6.37)
i⫽0
ˆ being the sum of k ⱕ r rank-one independent l ⫻ n matrices, is of rank k. Matrix X, If the k largest eigenvalues are involved, it can be shown that the squared error ⑀2 ⫽
l⫺1 n⫺1
ˆ j)|2 |X(i, j) ⫺ X(i,
(6.38)
i⫽0 j⫽0
is the minimum one with respect to all rank-k l ⫻ n matrices. The square root ˆ F of the right-hand side in (6.38) is also known as the Frobenius norm X ⫺ X ˆ of the difference matrix X ⫺ X. The error in the approximation turns out to be (Problem 6.6) ⑀2 ⫽
r⫺1
i
(6.39)
i⫽k
Hence, if we order the eigenvalues in descending order, 0 ≥ 1 ≥ · · · ≥ r⫺1 , then for a given number of k terms in the expansion, the SVD leads to the minimum square error. Thus, Xˆ is the best rank-k approximation of X in the Frobenius norm sense. This reminds us of the Karhunen–Loève expansion. However, in the latter case the optimality was with respect to the mean square error. This is a major difference in philosophy between SVD and KL. The former is related to a single set of samples and the latter to an ensemble of them.
Dimensionality Reduction SVD has been used extensively for dimension reduction in pattern recognition and information retrieval, and it forms the basis of what is known as latent semantics indexing, see, for example, [Berr 95]. Adopting the notation used in (6.25) and
337
“08-Ch06-SA272” 18/9/2008 page 338
338
CHAPTER 6 Feature Generation I
(6.27), Eq. (6.37) can be written as
⎡
⎢ ⎢ X Xˆ ⫽ [u0 , u1 , . . . , uk⫺1 ] ⎢ ⎢ ⎣
√ 0 v H 0 √ 1 v H 1 .. . √ k⫺1 v H k⫺1
⎤ ⎥ ⎥ ⎥ ⎥ ⎦
⫽ Uk [a 0 , a 1 , . . . , a n⫺1 ]
(6.40)
where Uk consists of the first k columns of U and the k-dimensional vectors a i , 1
i ⫽ 0, 1, . . . , n⫺1,are the column vectors of the k⫻n product matrix ⌳k2 VkH ,where 1
VkH consists of the first k rows of V H and ⌳k2 is the diagonal matrix having elements the square roots of the respective k singular values. Figure 6.3 gives a diagrammatic interpretation of the matrix products involved in SVD. The formulation given in (6.40) suggests that each column vector, x i of X, is approximated as x i Uk a i ⫽
k⫺1
i ⫽ 0, 2, . . . , n ⫺ 1
um ai (m),
(6.41)
m⫽0
where ai (m), m ⫽ 0, 1, . . . , k ⫺ 1, denote the elements of the respective vector a i . In words,the l-dimensional vector x i is approximated by the k-dimensional vector a i , lying in the subspace spanned by ui , i ⫽ 0, 1, . . . , k ⫺ 1 (a i is the projection of x i on this subspace; Problem 6.6.) Furthermore, due to the orthonormality of the columns ui , i ⫽ 0, 1, . . . , k ⫺ 1, of Uk it is straightforward to see that ||x i ⫺ x j || ||Uk (a i ⫺ a j )|| ⫽ ||
k⫺1
um (ai (m) ⫺ aj (m))||
m⫽0
⫽ ||a i ⫺ a j ||, k
i, j ⫽ 0, 1, . . . , n ⫺ 1
(6.42)
k 1 2
r
k
H
Lk
k
Vk
r
l 1 2
H
Lr Uk r
Ur
1 2
H
1 2 k
H
r
Vr n
X 5Ur L r V r
X 5Uk L V k
FIGURE 6.3 Diagrammatic interpretation of the matrix products involved in SVD. In the approximation of X ˆ the first k columns of Ur and the first k rows of VrH are involved. by X,
“08-Ch06-SA272” 18/9/2008 page 339
6.4 The Singular Value Decomposition
where || · || represents the Euclidean norm of a vector. That is, using the previous projection and assuming the approximation to be reasonably good, the Euclidean distance between x i and x j in the high l-dimensional space is (approximately) preserved under the projection in the lower k-dimensional subspace. The previous observation has important implications in applications such as information retrieval. Let us take as an example the simple case where we are given a set of n patterns each represented by a l-dimensional feature vector. These patterns constitute the available database. Given an unknown pattern, the goal is to search for and recover from the database the pattern that is most similar to the unknown one, by computing its Euclidean distance from each vector in the database. When l and n are large numbers this can be a very time-consuming task. A procedure to simplify computations is the following. We form the l ⫻ n data matrix, X,3 having as columns the n feature vectors. Perform a SVD on X and represent each feature vector, x i , by its lower dimensional projection, a i , as described before. Given the unknown vector, one projects it on the subspace spanned by the columns of Uk and performs Euclidean distance computations in the k-dimensional space. Since Euclidean distances are approximately preserved, one can decide about the proximity of vectors by working in a lower dimensional space. If k ⬍⬍ l substantial computational savings are obtained (see, e.g., [Berr 95, Deer 90, Sebr 03]). SVD builds upon global information spread over all the data vectors in X. Indeed, a crucial part of the algorithm is the computation of the eigenvalues of X H X or XX H , which,for zero mean data,is directly related to the respective covariance matrix (Eq. 6.22). Hence, the performance of the SVD, as a dimensionality reduction technique, is most effective for cases where data can sufficiently be described in terms of the covariance matrix, for example, to be Gaussian-like distributed. In [Cast 03] a modification of the simple SVD is suggested to account for data with a clustered structure. In Section 6.7, nonlinear dimensionality techniques will be reviewed, where more than a simple linear projection on a subspace is required to reduce dimensionality. Remarks
3
■
Due to its optimal approximation properties,the SVD transform also has excellent “information packing” properties, and an image array can be represented efficiently by a few of its singular values. Thus, SVD is a natural candidate as a tool for feature generation/selection in classification.
■
Performing SVD of large matrices is a computationally expensive task. In order to overcome this drawback, a number of computationally efficient schemes have been developed, see, for example, [Ye 04, Achl 01].
Note that X is defined here as the transpose of the data matrix in (3.44),to comply with the notation used in latent semantics indexing.
339
“08-Ch06-SA272” 18/9/2008 page 340
340
CHAPTER 6 Feature Generation I
Example 6.4 Consider the matrix
⎡
6 ⎢0 ⎢ X ⫽⎢ ⎣4 0
⎤ 6 ⎥ 1⎥ ⎥ 0⎦ 6
The goal is to compute its singular value decomposition. ■
Step 1: Find the eigenvalues and eigenvectors of
52 36 T X X⫽ 36 73 These are 0 ⫽ 100, 1 ⫽ 25, and the corresponding eigenvectors are v 0 ⫽ [0.6, 0.8]T , v 1 ⫽ [0.8, ⫺0.6]T .
■
Step 2: Compute the eigenvectors of XX T . This is a 4 ⫻ 4 matrix of rank 2. The eigenvectors corresponding to the nonzero eigenvalues 0 , 1 are computed via (6.30), that is, u0 ⫽ 0.1Xv 0 , u1 ⫽ 0.2Xv 1 or [0.84, 0.08, 0.24, 0.48]T and [0.24, ⫺0.12, 0.64, ⫺0.72]T respectively.
■
Step 3: Compute the SVD of X X ⫽ 10[0.84, 0.08, 0.24, 0.48]T [0.6, 0.8] ⫹ 5[0.24, ⫺0.12, 0.64, ⫺0.72]T [0.8, ⫺0.6] If we keep the first of the two terms, then the resulting approximation is the best, in the Frobenius sense, rank-1 approximation of X.
Example 6.5 The goal of this example is to demonstrate the power of the SVD as a dimensionality reduction tool, in the context used in latent semantics indexing in information retrieval. (a) Let our data set consist of 1000 three-dimensional vectors x i ⫽ [x1 (i), x2 (i), x3 (i)]T ,
i ⫽ 1, 2, . . . , 1000
This set of points comprises our database. We form the 3 ⫻ 1000 matrix X having these data vectors as columns. For the needs of this example, the components x1 (i), x2 (i) are randomly generated using the uniform distribution in [⫺10, 10]. The value of the third dimension of each point is given by x3 (i) ⫽ ⫺x1 (i) ⫺ x2 (i) ⫹ , where is a noise source following the uniform distribution in [⫺1, 1]. In other words, our data are crowded around the plane H : x1 ⫹ x2 ⫹ x3 ⫽ 0
(6.43)
“08-Ch06-SA272” 18/9/2008 page 341
6.4 The Singular Value Decomposition
Performing SVD analysis, it turns out that the singular values are 0 ⫽ 158.43, 1 ⫽ 89.01, 2 ⫽ 10.50. The relatively small value of 2 is the consequence of the fact that our data are approximately two-dimensional. Recall that the singular values are eigenvalues of XX T , which is the same (within a scaling factor) with the estimate of the correlation matrix used in PCA (Eq. (6.11)). The corresponding (orthonormal) eigenvectors, which are also the column vectors of the U matrix, are (after rounding to the second decimal point) ⎤ ⎤ ⎤ ⎡ ⎡ ⎡ ⫺0.39 0.71 0.58 ⎥ ⎥ ⎥ ⎢ ⎢ ⎢ u0 ⫽ ⎣ ⫺0.42 ⎦, u1 ⫽ ⎣ ⫺0.70 ⎦, u2 ⫽ ⎣ 0.58 ⎦ 0.82 ⫺0.01 0.57 It is not difficult to verify that the plane (subspace) formed by the two principal eigenvectors is H1 : 14.26x1 ⫹ 14.10x2 ⫹ 13.95x3 ⫽ 0 which is very close to the hyperplane H in (6.43), around which our data cluster. (b) We now select randomly six of the points in the data set X and project them along the H1 plane. The 6 ⫻ 6 distance matrix D whose (i, j) element is the squared Euclidean distance between the ith and jth points for i, j ⫽ 1, 2, . . . , 6, is ⎡
0
⎢ ⎢ 26.17 ⎢ ⎢ 24.70 ⎢ D⫽⎢ ⎢112.25 ⎢ ⎢ ⎣ 11.92 4.81
26.17
24.70
112.25
11.92
0
61.46
43.96
38.33
61.46
0
107.97
4.34
43.96
107.97
0
88.72
38.33
4.34
88.72
0
49.25
14.51
140.18
9.95
4.81
⎤
⎥ 49.25⎥ ⎥ 14.51⎥ ⎥ ⎥ 140.18⎥ ⎥ 9.95 ⎥ ⎦ 0
The corresponding distance matrix, D ⬘ , for the respective projections on H1 is ⎡
0
25.85
24.32
112.21
11.72
0
61.46
43.83
37.29
61.46
0
107.80
3.20
⎢ ⎢ 25.85 ⎢ ⎢ 24.32 ⎢ ⬘ D ⫽⎢ ⎢112.21 ⎢ ⎢ ⎣ 11.72
43.83
107.80
0
88.29
37.29
3.20
88.29
0
4.57
49.24
14.49
140.10
9.06
4.57
⎤
⎥ 49.24⎥ ⎥ 14.49⎥ ⎥ ⎥ 140.10⎥ ⎥ 9.06 ⎥ ⎦ 0
which is in close agreement with D. Note that this good agreement is a consequence of the fact that the true dimensionality of the data is very close to 2. Increasing the variance of the noise source , the spread of the data around the plane H would increase and the data would become more and more “three-dimensional.” In other words, the higher the variance of , the less the agreement between D and D ⬘ that one expects to get.
341
“08-Ch06-SA272” 18/9/2008 page 342
342
CHAPTER 6 Feature Generation I
6.5 INDEPENDENT COMPONENT ANALYSIS As we have already seen, the principal component analysis (PCA) performed by the Karhunen–Loève transform produces features y(i), i ⫽ 0, 1, . . . , N ⫺ 1, that are mutually uncorrelated. The solution obtained by the KL transform solution is optimal when dimensionality reduction is the goal and one wishes to minimize the approximation mean square error. However, for certain applications, such as the one illustrated in Figure 6.1,the obtained solution falls short of the expectations. In contrast, the more recently developed independent component analysis (ICA) theory, for example, [Hyva 01, Como 94, Jutt 91, Hayk 00, Lee 98], tries to achieve much more than simple decorrelation of the data. The ICA task is casted as follows: Given the set of input samples x, determine an N ⫻ N invertible matrix W such that the entries y(i), i ⫽ 0, 1, . . . , N ⫺ 1, of the transformed vector y ⫽ Wx
(6.44)
are mutually independent. The goal of statistical independence is a stronger condition than the uncorrelatedness required by the PCA. The two conditions are equivalent only for Gaussian random variables. Searching for independent rather than uncorrelated features gives us the means of exploiting a lot more information, hidden in the higher order statistics of the data. As the example of Figure 6.1 suggests, constraining the search by digging information in the second-order statistics only results in the least interesting, for our problem, projection direction, that is, that of a 0 . However, a 1 is, no doubt, the most interesting direction from the class separation point of view. In contrast, employing ICA can unveil from the higher order statistics of the data the piece of information that points a 1 as the most interesting one. Furthermore, searching for statistically independent features is in line with the way nature builds up the“cognitive”maps of the outside world in the brain, by processing the (input) sensory data. Barlow [Barl 89], in the so-called Barlow’s hypothesis, suggests that the outcome of the early processing performed in our visual cortical feature detectors might be the result of a redundancy reduction process. In other words, the neural outputs are mutually as statistically independent as possible, conditioned, of course, on the received sensory messages. The interested reader can find more on issues related to redundancy reduction and also to a number of methodologies inspired by it in [Atti 92, Fiel 94, Deco 95, Bell 00]. The potential of the ICA as an optimal feature generator technique, in the context of pattern recognition, has been demonstrated in [Cao 03, Bell 97, Hoy 00, Jang 99, Bart 02, Kwon 04]. Before we proceed to develop techniques for performing ICA,we need to be sure that such a problem is well defined and has a solution and under what conditions. To this end,let us assume that our input random data vector x is indeed generated by a linear combination of statistically independent and stationary in the strict sense components (sources), that is, x ⫽ Ay
(6.45)
“08-Ch06-SA272” 18/9/2008 page 343
6.5 Independent Component Analysis
The task now is under what conditions a matrix, W , can be computed so as to recover the components of y from Eq. (6.44), by exploiting information hidden in x. Usually A is known as the mixing and W as the demixing matrix, respectively. The following condition is proved in [Como 94].
Identifiability Condition of the ICA Model All independent components y(i), i ⫽ 1, 2, . . . , N , with the possible exception of one, must be non-Gaussian. A second condition is that matrix A must be invertible. In the more general case where A is a nonsquare l ⫻N matrix,then l must be greater than N and A must be of full column rank. In other words, in contrast to PCA which can always be performed, ICA is meaningful only if the involved random variables are non-Gaussian. Indeed, as has already been stated, for Gaussian random variables independence is equivalent to uncorrelatedness and PCA suffices. From a mathematical point of view, the ICA problem is ill-posed for Gaussian processes. Indeed, if we assume that the obtained independent components y(i), i ⫽ 0, 1, . . . , N ⫺ 1, are all Gaussian, then a linear transformation of them by any unitary matrix will also be a solution (see Problem 5.4). PCA achieves a unique solution by imposing a specific orthogonal structure onto the transformation matrix. Under the above stated conditions,it can be shown that each one of the resulting independent components is uniquely estimated up to a multiplicative constant, which is a rather insignificant indeterminacy associated with the method. This is the reason that many times the components are considered to be of unit variance. Finally, it is interesting that the independent components result in no specific ordering, in contrast to the PCA, where a specific ordering is associated with the values of the corresponding eigenvalues. However, in practice, some form of ordering can be adopted. For example, the components can be ordered according to the degree of “non-Gaussianity,” measured by an appropriate index, for example, the fourth-order cumulant (see Appendix A). Although such an index may seem a bit strange to a newcomer, its physical interpretation will become clearer as we go on. After all, from a common-sense point of view, a Gaussian pdf must be the least interesting one. Recall from Chapter 2 that maximizing the entropy, constraining the solution to be within the family of random variables with given mean and variance, the result is a Gaussian pdf. That is, the Gaussian is the most “random” of all the pdfs describing this family of random variables and from this point of view the least informative one with respect to the underlying structure of the data. In contrast, distributions that have the “least resemblance” to the Gaussian are more interesting since they display some structure associated with the data. This observation is at the heart of a closely related, to ICA, family of techniques known as projection pursuit; see also Section 4.12. The essence behind such techniques is to search for directions (subspaces) in the feature space so that the corresponding data vector projections are described by “interesting” non-Gaussian distributions. For a more rigorous discussion on such issues the reader may refer to, for example, [Hube 85, Jone 87].
343
“08-Ch06-SA272” 18/9/2008 page 344
344
CHAPTER 6 Feature Generation I
6.5.1 ICA Based on Second- and Fourth-Order Cumulants This approach in performing ICA is a direct generalization of the PCA technique. The Karhunen–Loève transform focuses on the second-order statistics and demands the cross-correlations E[y(i)y(j)] to be zero. Since in ICA we demand that the components of y be statistically independent, this is equivalent to demanding that all the higher order cross-cumulants to be zero (see Appendix A). In [Como 94] it is suggested that restricting ourselves up to the fourth-order cumulants is sufficient for many applications. Appendix A shows the first three cumulants are equal to the first three moments, that is, 1 ( y(i)) ⫽ E[ y(i)] ⫽ 0 2 ( y(i)y( j)) ⫽ E[ y(i)y( j)] 3 ( y(i)y( j)y(k)) ⫽ E[ y(i)y( j)y(k)]
and the fourth-order cumulants are given by 4 ( y(i)y( j)y(k)y(r)) ⫽ E[ y(i)y( j)y(k)y(r)] ⫺ E[ y(i)y( j)]E[ y(k)y(r)] ⫺ E[ y(i)y(k)]E[ y( j)y(r)] ⫺ E[ y(i)y(r)]E[ y( j)y(k)]
where zero mean processes have been assumed. Another assumption that is usually encountered in practice,and will be adopted here,is that the associated pdfs are symmetric. This makes all odd order cumulants zero. Thus the problem has now been reduced to finding a matrix, W , so that the second-order (cross-correlations) and fourth-order cross-cumulants of the transformed variables are zero. In [Como 94] this is achieved by the following steps: Step 1: Perform a PCA on the input data, that is, yˆ ⫽ AT x
(6.46)
A is our familiar unitary transformation matrix of the Karhunen–Loève transform. The components of the transformed random vector yˆ are thus uncorrelated. ˆ Step 2: Compute another unitary matrix, A,so that the fourth-order cross-cumulants of the components of the transformed random vector y ⫽ Aˆ T yˆ
(6.47)
are zero. This is equivalent to searching for a matrix Aˆ that makes the sum of the squares of the fourth-order auto-cumulants maximum, that is, ˆ ≡ max ⌿( A)
Aˆ Aˆ T ⫽I
N ⫺1 i⫽0
4 ( y(i))2
(6.48)
“08-Ch06-SA272” 18/9/2008 page 345
6.5 Independent Component Analysis
Step 2 is justified as follows. It can be shown [Como 94] that the sum of the squares of the fourth-order cumulants is invariant under a linear transformation by a unitary matrix. Therefore, since the sum of squares of the fourth-order cumulants is fixed for y, ˆ maximizing the sum of squares of the auto-cumulants of y will force the corresponding cross-cumulants to zero. Observe that this is basically a diagonalization problem of the fourth-order cumulant multidimensional array. In practice, this is achieved by generalizing the method of Givens rotations, used for matrix diagonalization [Como 94]. Note that the right hand side in Eq. (6.48) is a function of (a) ˆ the elements of the unknown matrix A,(b) the elements of the known (for this step) matrix A, and (c) the cumulants of the random components of the input data vector x, which have to be estimated prior to the application of the method. In practice, it may turn out that the nulling of cross-cumulants is only approximately achieved. This is because (a) the input data may not obey the linear model of Eq. (6.45); (b) the input data are corrupted by noise, which has not been taken into account; and (c) the cumulants of the input are only approximately known, since they are estimated by the available input data set. Once the two steps have been completed, the final feature vector with (approximately) independent components is given by the combined transform ˆ Tx ≡ Wx y ⫽ (AA)
(6.49)
Notice that since Aˆ is unitary, the uncorrelatedness achieved in the first step is inherited by the elements of y, which now has its second- and fourth-order crosscumulants (at least approximately) zero.
6.5.2 ICA Based on Mutual Information The approach based on nulling the second- and fourth-order cross-cumulants,though one of the most widely used in practice, somehow lacks in generality and also imposes, externally, a structure in the transformation matrix. An alternative, theoretically more pleasing approach is estimating W by minimizing the mutual information between the transformed random variables. The mutual information, I ( y), between the components of y is defined as I ( y) ⫽ ⫺H( y) ⫹
N ⫺1
H( y(i))
(6.50)
i⫽0
where H( y(i)) is the associated entropy of y(i), defined as ([Papo 91]) H( y(i)) ⫽ ⫺
pi ( y(i)) ln pi ( y(i)) dy(i)
(6.51)
where pi ( y(i)) is the marginal pdf of y(i). In Appendix A, it is shown that I ( y) is equal to the Kullback–Leibler probability distance between the joint pdf p( y) ⫺1 and the product of the respective marginal probability densities N i⫽0 pi ( y(i)). This distance (and hence the associated mutual information I ( y)) is zero if the components y(i) are statistically independent. This is because only in this case is the
345
“08-Ch06-SA272” 18/9/2008 page 346
346
CHAPTER 6 Feature Generation I
joint pdf equal to the product of the corresponding marginal pdfs and the Kullback– Leibler distance becomes zero. Hence, what is more natural than trying to compute W so as to force I ( y) to be minimum, since this will make the components of y as independent as possible? Combining Eqs. (6.44), (6.50), and (6.51) and taking into account the formula that relates the two pdfs associated with x and y (y is a function of x), e.g., [Papo 91], we end up with I ( y) ⫽ ⫺H(x) ⫺ ln |det(W )| ⫺
N ⫺1
pi ( y(i)) ln pi ( y(i)) dy(i)
(6.52)
i⫽0
where det(W ) denotes the determinant of W . The elements of the unknown matrix W are hidden in the marginal pdfs of the transformed variables, y(i). However, it is not easy to express this dependence explicitly. An approach currently used is to expand each of the marginal probabilities around the Gaussian pdf, g( y), following Edgeworth’s expansion (Appendix A), and truncate the series to a reasonable approximation. For example, keeping the first two terms in the Edgeworth expansion, we have 1 1 p( y) ⫽ g( y) 1 ⫹ 3 ( y)H3 ( y) ⫹ 4 ( y)H4 ( y) 3! 4!
(6.53)
where Hk ( y) is the Hermite polynomial of order k (Appendix A). To obtain an approximate expression for I ( y) in terms of cumulants of y(i) and W , we can (a) insert in Eq. (6.52) the pdf approximation in Eq. (6.53),(b) adopt the approximation ln(1 ⫹ y) y ⫺ y2 ,and (c) perform the integrations. This is no doubt a rather painful task! For the case of Eq. (6.53) and constraining W to be unitary, the following is obtained ([Hyva 01]): I ( y) C ⫺
N ⫺1 i⫽0
1 2 1 2 7 4 1 ( y(i)) ⫹ (y(i)) ⫹ ( y(i)) ⫺ 23 ( y(i))4 ( y(i)) 12 3 48 4 48 4 8
(6.54)
where C is a variable independent of W . Under the assumption that the pdfs are symmetric (thus, third-order cumulants are zero,) it can be shown that minimizing the approximate expression of the mutual information in Eq. (6.54) is equivalent to maximizing the sum of the squares of the fourth-order cumulants. Of course, the unitary W constraint is not necessary,and in this case other approximate expressions for I ( y) result, e.g., [Hayk 99]. Minimization of I ( y) in Eq. (6.54) can be carried out by a gradient descent technique (Appendix C), where the involved expectations (associated with the cumulants) are replaced by the respective instantaneous values. Although a detailed treatment of the optimization procedure is beyond the scope of this book,it is worth pointing out some of its aspects. Before we apply the approximations, let us go back to Eq. (6.52). Since H(x) does not depend on W , minimizing I ( y) is equivalent to maximization of J (W ) ⫽ ln |det(W )| ⫹ E
N ⫺1 i⫽0
ln pi ( y(i))
(6.55)
“08-Ch06-SA272” 18/9/2008 page 347
6.5 Independent Component Analysis
Taking the gradient of the cost function with respect to W results in ⭸J (W ) ⫽ W ⫺T ⫺ E[( y)x T ] ⭸W
(6.56)
p⬘N ⫺1 ( y(N ⫺ 1)) T p⬘0 ( y(0)) ,...,⫺ ( y) ≡ ⫺ p0 ( y(0)) pN ⫺1 ( y(N ⫺ 1))
(6.57)
where
and p⬘i ( y(i)) ≡
dpi ( y(i)) dy(i)
(6.58)
Obviously, the derivatives of the marginal probability densities depend on the type of approximation adopted in each case. The general gradient ascent scheme at the tth iteration step can now be written as W (t) ⫽ W (t ⫺ 1) ⫹ (t) W ⫺T (t ⫺ 1) ⫺ E[( y)x T ] W (t) ⫽ W (t ⫺ 1) ⫹ (t) I ⫺ E[( y)y T ] W ⫺T (t ⫺ 1)
(6.59)
In practice, the expectation operator is neglected, in the spirit of the stochastic approximation rationale (Section 3.4.2). Remarks ■
From the gradient in Eq. (6.56) it is easy to see that at a stationary point the following is true: ⭸J (W ) T W ⫽ E[I ⫺ ( y)y T ] ⫽ 0 ⭸W
(6.60)
In other words,what we achieve with ICA is a nonlinear generalization of PCA. Recall that for the latter, the uncorrelatedness condition can be written as E[I ⫺ yy T ] ⫽ 0
(6.61)
The presence of the nonlinear function (·) takes us beyond simple uncorrelatedness, and brings the cumulants into the scene. In fact, Eq. (6.60) was the one that inspired the early pioneering work on ICA, as a direct nonlinear generalization of PCA [ Jutt 91]. ■
The updated equation in Eq. (6.59) involves the inversion of the transpose of the current estimate of W . Besides the computational complexity issues, there is no guarantee of invertibility in the process of adaptation. Use of the so-called natural gradient [Doug 00], instead of the gradient in Eq. (6.56), results in W (t) ⫽ W (t ⫺ 1) ⫹ (t) I ⫺ E[( y)y T ] W (t ⫺ 1)
(6.62)
347
“08-Ch06-SA272” 18/9/2008 page 348
348
CHAPTER 6 Feature Generation I
which does not involve matrix inversion and at the same time improves convergence. A more detailed treatment of this issue is beyond the scope of the present book. Just to give an incentive to the mathematically inclined reader for indulging more deeply this field, it suffices to say that our familiar gradient, that is, Eq. (6.56), points to the steepest ascent direction if the space is Euclidean. However, in our case the parameter space consists of all the nonsingular N ⫻ N matrices, which is a multiplicative group. The space is Riemannian, and it turns out that the natural gradient, pointing to the steepest ascent direction, results if we multiply the gradient in Eq. (6.56) by W T W , which is the corresponding Riemannian metric tensor [Doug 00].
6.5.3 An ICA Simulation Example The example is a realization of the case shown in Figure 6.4. A total of 1024 samples of a two-dimensional normal distribution were generated. The mean and covariance matrix of the normal pdf were ⫽ [⫺2.6042, 2.5]T ,
⌺⫽
10.5246
9.6313
9.6313
11.3203
Similarly, 1024 samples from a second normal pdf were generated with the same covariance and mean ⫺. For the ICA, the method based on the second- and x2
10
5 a0
a1 0
⫺5 ⫺10 ⫺15
⫺10
⫺5
0
5
10
15
x1
FIGURE 6.4 The setup for the ICA simulation example. The two vectors point to the projection directions resulting from the analysis. The optimal direction for projection, resulting from the ICA analysis, is that of a 1 .
“08-Ch06-SA272” 18/9/2008 page 349
6.6 Nonnegative Matrix Factorization
fourth-order cumulants, presented in this section, was used. transformation matrix W is W ⫽
⫺0.7088
0.7054
0.7054
0.7088
≡
a T1
The resulting
a T0
The vectors a 0 and a 1 point in the principal and minor axis directions, respectively, obtained from the PCA analysis. However, the most interesting direction for projection, according to the ICA analysis, is that of a 1 and not of a 0 . Indeed, the kurtosis of the obtained transformed variables [ y1 , y2 ]T ⫽ W x is 4 (y1 ) ⫽ ⫺1.7 4 (y2 ) ⫽ 0.1
Thus, projection in the principal axis direction results in a variable with a pdf close to a Gaussian. The projection to the minor axis direction results in a variable with a pdf that deviates from the Gaussian (Figures 6.1, 6.4) and is more appropriate from the classification point of view.
6.6 NONNEGATIVE MATRIX FACTORIZATION In the PCA as well as the SVD analysis, the underlying constraints were the orthogonality of the involved basis vectors, in the PCA, and of the column vectors in the U and V matrices in the SVD. This becomes crystal clear from the problem 6.3. PCA is formulated as a task minimizing the mean square error subject to the orthogonality constraint of the basis vectors. Although, in some cases, the resulting expansion is useful, for some other cases such a constraint turns out to be very “weak”in representing the data. More recently, a new matrix factorization was suggested in [Paat 91, Paat 94], which guarantees the nonnegativity of the elements of the resulting matrix factors. Such a constraint is enforced in certain applications since negative elements contradict physical reality. For example, in image analysis the intensity values of the pixels cannot be negative. Also, probability values cannot be negative. The resulting factorization is known as nonnegative matrix factorization (NMF) and it has been used successfully in a number of applications including document clustering ([Xu 03]), molecular pattern discovery ([Brun 04]), image analysis ([Lee 01]), clustering ([Szym 06]), music transcription and music instrument classification ([Smar 03, Benn 06]) and face verification ([Zafe 06]). Given a l ⫻ n matrix X, the task of NMF consists of finding an approximate factorization of X, that is, X ≈ WH
(6.63)
where W and H are l ⫻ r and r ⫻ n matrices, respectively, r ⬍ min(n, l) and all the matrix elements are nonnegative, that is, W (i, k) ⱖ 0, H(k, j) ⱖ 0,
349
“08-Ch06-SA272” 18/9/2008 page 350
350
CHAPTER 6 Feature Generation I
i ⫽ 1, 2, . . . , l, k ⫽ 1, 2, . . . , r, j ⫽ 1, 2, . . . , n. Clearly, matrices W and H are of rank at most r and their product is a low rank, at most r, approximation of X. The significance of the above is that every column vector in X is represented by the expansion xi ⫽
r
H(k, i)w k ,
i ⫽ 1, 2, . . . , n
k⫽1
where w k , k ⫽ 1, 2, . . . , r, are the column vectors of W and constitute the basis of the expansion. The number of vectors in the basis is less than the dimensionality of the vector itself. Hence, NMF can also be seen as a method for dimensionality reduction. To get a good approximation in (6.63) one can adopt different costs. The most conventional cost is the Frobenius norm of the error matrix. In such a setting, the NMF task is casted as follows: minimize ||X ⫺ WH||F ≡
l n
X(i, j) ⫹ [WH](i, j)
2
(6.64)
i⫽1 j⫽1
subject to W (i, k) ⱖ 0, H(k, j) ⱖ 0.
H(k, j) ⱖ 0 ᭙i, k, j
(6.65)
where [WH](i, j) is the (i, j) element of matrix WH. Minimization is performed with respect to W and H. Another cost function has also been suggested, which is a close relative of the Kullback–Leibler distance (see Appendix A) and the task now becomes minimize
l n
X(i, j) ln
i⫽1 j⫽1
subject to
X(i, j) ⫺ X(i, j) ⫹ [WH](i, j) [WH](i, j)
W (i, k) ⱖ 0, H(k, j) ⱖ 0 ᭙i, k, j
(6.66) (6.67)
It that if X ⫽ WH the previous cost becomes zero. Also,observe that if is readily seen X(i, j) ⫽ i,j i, j [WH](i, j) ⫽ 1 then the cost becomes identical to the Kullback– Leibler (KL) distance formulation. Note, however, that the previous KL-like cost is not well defined if either X(i, j) or [WH](i, j) are zero. For more information on this topic the interested reader may consult, for example, [Sra 06]. Once the problem has been formulated, the major issue rests on the solution of the optimization task. To this end, a number of algorithms have been proposed, for example, Newton–type or gradient descent type (Appendix C). Such algorithmic issues, as well as a number of related theoretic ones, are beyond the scope of the current book, and the interested reader may consult [Chu 04, Dono 04, Trop 03].
6.7 NONLINEAR DIMENSIONALITY REDUCTION All the techniques that have been discussed so far in this chapter, as well as the LDA in the previous chapter, share a common goal: dimensionality reduction. In
“08-Ch06-SA272” 18/9/2008 page 351
6.7 Nonlinear Dimensionality Reduction
other words, given a high-dimensional data set X ⫽ {x 1 , x 2 , . . . , x n } ⊂ RN of input patterns,4 the goal is to compute n corresponding patterns, Y ⫽ {y 1 , y 2 , . . . , y n } ⊂ Rm , m ⬍ N , that provide an “informative” representation of the input patterns. The word “informative” is interpreted in a different way for different methods; for example, PCA and ICA adopt different views on the issue. Another common characteristic of all the previous methods is that they respect linearity. Once a transformation matrix is computed, for each method, points in Y are obtained by projecting the points in X along the rows of this matrix. The emphasis of dimensionality reduction has, so far, focussed to the domain of feature generation, in order to bypass the curse of dimensionality and cope efficiently with the generalization aspects of a classifier. However, the significance of such techniques goes much beyond and embraces a number of other applications. Data visualization is an area where dimensionality reduction techniques are employed in order to transform the original data from a high-dimensional into two or three dimensions, thereby offering additional insight into the problem at hand. As stated in [Tene 00], the human brain confronts the same problem extracting from the high-dimensional sensory system (i.e., 106 optic nerve fibers) a reduced number of perceptually relevant features. Data mining and information retrieval is another area where searching can be substantially facilitated if it is performed in a lower dimensional space. This is a typical area where data usually lie in a very high-dimensional space, although its intrinsic dimensionality is low. Dimensionality reduction has been used extensively in clustering and in semi-supervised learning, an area that is gaining in importance over the last years. So, in a way, this section is a bridge to the chapters dedicated to these topics. The aim of this section is to discuss nonlinear dimensionality reduction techniques. We will discuss the main directions that are currently popular, and we will not delve into many details.
6.7.1 Kernel PCA As its name suggests, this is a kernelized version of the classical PCA, introduced in [Scho 98]. Given the data set, X, we make an implicit mapping into a RKHS H, x ∈ X → (x) ∈ H
Let x i , i ⫽ 1, 2, . . . , n,be the available training points. We will work with an estimate of the correlation matrix in H obtained as an average over the known sample points5 R⫽
n 1 (x i )(x i )T n
(6.68)
i⫽1
4 To
serve the specific needs of this chapter, we have reserved the symbol N to denote dimensionality of the input space. For the rest of the book, N denotes the number of data points. 5 If the dimensionality of H is infinite, the definition of the correlation matrix needs a special interpretation, but we will not bother about it.
351
“08-Ch06-SA272” 18/9/2008 page 352
352
CHAPTER 6 Feature Generation I
The goal is to perform the eigendecomposition of R, that is, Rv ⫽ v
n
(6.69)
Let us make the assumption that the data are centered i⫽1 (x i ) ⫽ 0. This assumption is only to simplify the discussion, and it can be relaxed. By the definition of R, it can be shown that v lies in the span of {(x 1 ), (x 2 ), . . . , (x n )}. Indeed,
v ⫽
1 (x i )(x i )T n n
v⫽
i⫽1
n 1 (x i )T v (x i ) n i⫽1
and for ⫽ 0 we can write v⫽
n
a(i)(x i )
(6.70)
i⫽1
Combining (6.69) and (6.70), it turns out ([Scho 98]) that the problem is equivalent to performing an eigendecomposition of the Gram matrix Ka ⫽ na
(6.71)
a ≡ [a(1), a(2), . . . , a(n)]T
(6.72)
where
As we already know (Section 4.19.1), the elements of the Gram matrix are K(i, j) ⫽ K (x i , x j ), with K (·, ·) being the adopted kernel function. Thus, the kth eigenvector of R, corresponding to the kth (nonzero) eigenvector of K in (6.71), is expressed as vk ⫽
n
ak (i)(x i ), k ⫽ 1, 2, . . . , p
(6.73)
i⫽1
where 1 ⱖ 2 ⱖ . . . ⱖ p denote the respective eigenvalues in descending order and p is the smallest nonzero one and a Tk ≡ [ak (1), . . . , ak (n)] is the kth eigenvector of the Gram matrix. The latter is assumed to be normalized so that v k , v k ⫽ 1, k ⫽ 1, 2, . . . , p, where ·, · is the dot product in the Hilbert space H. This imposes an equivalent normalization on the respective a k ’s, resulting from 1 ⫽ v k , v k ⫽
n
ak (i)(x i ),
i⫽1
⫽
n n
n
ak (j)(x j )
j⫽1
ak (i)ak (j)K(i, j)
i⫽1 j⫽1
⫽ a Tk Ka k ⫽ nk a Tk a k ,
k ⫽ 1, 2, . . . , p
(6.74)
“08-Ch06-SA272” 18/9/2008 page 353
6.7 Nonlinear Dimensionality Reduction
We are now ready to summarize the basic steps for performing a kernel PCA. Given a vector x ∈ RN and a kernel function K (·, ·): ■ ■
Compute the Gram matrix K(i, j) ⫽ K (x i , x j ), i, j ⫽ 1, 2, . . . , n. Compute the m dominant eigenvalues/eigenvectors k , a k , k ⫽ 1, 2, . . . , m, of K (Eq. (6.71)).
■
Perform the required normalization (Eq. (6.74)).
■
Compute the m projections onto each one of the dominant eigenvectors, n y(k) ≡ v k , (x) ⫽ ak (i)K (x i , x), k ⫽ 1, 2, . . . , m
(6.75)
i⫽1
The operations given in (6.75) correspond to a nonlinear mapping in the input space. Note that, in contrast to the linear PCA, the dominant eigenvectors v k , k ⫽ 1, 2, . . . , m, are not computed explicitly. All we know are the respective (nonlinear) projections, y(k) along them. After all, this is what we are finally interested in. Remarks ■
Kernel PCA is equivalent to performing a standard PCA in the RKHS H. It can be shown that all the properties associated with the dominant eigenvectors, as discussed for the PCA, are still valid for the kernel PCA. That is, (a) the dominant eigenvector directions optimally retain most of the variance, (b) the MSE in approximating a point in H in terms of the m dominant eigenvectors is minimal, with respect to any other m directions, (c) projections onto the eigenvectors are uncorrelated, and (d) the entropy (under Gaussian assumption) is maximized ([Scho 98]).
■
Recall from Section 6.3 that the eigendecomposition of the Gram matrix was required for the metric multidimensional scaling (MDS) method. Hence, kernel PCA can be considered to be a kernelized version of MDS, where inner products in the input space have been replaced by kernel operations in the Gram matrix.
■
Note that the kernel PCA method does not explicitly consider the underlying structure of the manifold on which the data reside.
6.7.2 Graph-Based Methods Laplacian eigenmaps The starting point of this method is the assumption that the data points lie on a smooth manifold (hypersurface) M ⊃ X, whose intrinsic dimension is equal to m ⬍ N and it is embedded in RN , that is, M ⊂ RN . The dimension m is given as a parameter by the user. In contrast, this is not required in the kernel PCA, where m is the number of dominant components, which, in practice, is determined so that the gap between m and m⫹1 has a “large” value.
353
“08-Ch06-SA272” 18/9/2008 page 354
354
CHAPTER 6 Feature Generation I
The main philosophy behind the method is to compute the low-dimensional representation of the data so that local neighborhood information in X ⊂ M is optimally preserved. In this way, one attempts to get a solution that reflects the geometric structure of the manifold. To achieve this, the following steps are in order: Step 1: Construct a graph G ⫽ (V , E), where V ⫽ {vi , i ⫽ 1, 2, . . . , n} is a set of vertices and E ⫽ {eij } is a set of edges connecting vertices (vi , vj ) (see also Section 13.2.5). Each node vi of the graph corresponds to a point x i in our data set X. We connect vi , vj , that is, insert the edge eij between the respective nodes, if points x i , x j are “close” to each other. According to the method, there are two ways of quantifying “closeness.” Vertices vi , vj are connected with an edge if: 1. ||x i ⫺ x j ||2 ⬍ ⑀, for some user-defined parameter ⑀, where || · || is the Euclidean norm operation in RN , or 2. x j is among the k-nearest neighbors of x i or x i is among the k-nearest neighbors of x j , where k is a user-defined parameter and neighbors are chosen according to the Euclidean distance in RN . The use of Euclidean distance is justified by the smoothness of the manifold that allows one to approximate, locally, manifold geodesics by Euclidean distances in the space where the manifold is embedded. The latter is a known result from differential geometry. For those who are unfamiliar with such concepts, think of a sphere embedded in three-dimensional space. If somebody is constrained to live on the surface of the sphere, the shortest path to go from one point to another is the geodesic between these two points. Obviously, this is not a straight line but rather an arc across the surface of the sphere. However, if these points are close enough, their geodesic distance can be approximated by their Euclidean distance, computed in the threedimensional space. Step 2: Each edge, eij , is associated with a weight, W (i, j). For nodes that are not connected, the respective weights are zero. Each weight, W (i, j), is a measure of the “closeness” of the respective neighbors, x i , x j . A typical choice is
W (i, j) ⫽
⎧ ⎨exp ⫺ ||x i ⫺x j ||2 ,
if vi , vj correspond to neighbors
⎩
0
2
otherwise
where 2 is a user-defined parameter. We form the n ⫻ n weight matrix W having as elements the weights W (i, j). Note that W is symmetric, and it is sparse since, in practice, many of its elements turn out to be zero.
“08-Ch06-SA272” 18/9/2008 page 355
6.7 Nonlinear Dimensionality Reduction
Step 3: Define the diagonal matrix D with elements Dii ⫽ j W (i, j), i ⫽ 1, 2, . . . , n, and also the matrix L ⫽ D ⫺ W . The latter is known as the Laplacian matrix of the graph G(V , E). Perform the generalized eigendecomposition Lv ⫽ Dv
Let 0 ⫽ 0 ⱕ 1 ⱕ 2 ⱕ . . . ⱕ m be the smallest m ⫹ 1 eigenvalues.6 Ignore the v o eigenvector corresponding to 0 ⫽ 0 and choose the next m eigenvectors v 1 , v 2 , . . . , v m . Then map x i ∈ RN → y i ∈ Rm ,
i ⫽ 1, 2, . . . , n
where y Ti ⫽ [v 1 (i), v 2 (i), . . . , v m (i)],
i ⫽ 1, 2, . . . , n
(6.76)
The computational complexity of a general eigendecomposition solver amounts to O(n3 ) operations. However, for sparse matrices, such as L, efficient schemes, e.g., the Lanczos algorithm ([Golu 89]), can be employed to reduce complexity to subquadratic in n. We will prove the statement of step 3 for the case of m ⫽ 1. That is, the low dimensional space is the real axis. Our path evolves along the lines adopted in [Belk 03]. The goal is to compute yi ∈ R, i ⫽ 1, 2, . . . , n, so that connected points (in the graph, i.e., neighbors) stay as close as possible after the mapping onto the one-dimensional subspace. The criterion used to satisfy the closeness after the mapping is EL ⫽
n n ( yi ⫺ yj )2 W (i, j)
(6.77)
i⫽1 j⫽1
to become minimum. Observe that if W (i, j) has a large value (i.e., x i , x j are close in RN ), then if the respective yi , yj are far apart in R it incurs a heavy penalty in the cost function. Also, points that are not neighbors do not affect the minimization since the respective weights are zero. For the more general case where 1 ⬍ m ⬍ N the cost function becomes EL ⫽
n n
||y i ⫺ y j ||2 W (i, j)
i⫽1 j⫽1
6 In contrast to the notation used for PCA, the eigenvalues here are marked in ascending order. This is because, in this subsection, we are interested in determining the smallest values, and such a choice is notationally more convenient.
355
“08-Ch06-SA272” 18/9/2008 page 356
356
CHAPTER 6 Feature Generation I
Let us now reformulate (6.77). After some obvious algebra, we obtain EL ⫽
yi2
i
⫽
W (i, j) ⫹
j
yi2 Dii ⫹
i
yj2
j
W (i, j) ⫺ 2
i
yj2 Djj ⫺ 2
j
i
i
yi yj W (i, j)
j
yi yj W (I , j)
j
⫽ 2y T Ly
(6.78)
where L≡D⫺W
(6.79)
and y T ⫽ [ y1 , y2 , . . . , yn ]. The Laplacian matrix L is symmetric and nonnegative definite. The latter is readily seen from the definition in (6.78), where EL is always a nonnegative scalar. Note that the larger the value of Dii , the more “important” is the sample x i . This is because it implies large values for W (i, j), j ⫽ 1, 2, . . . , n, and plays a dominant role in the minimization process. Obviously, the minimum of EL is achieved by the trivial solution yi ⫽ 0, i ⫽ 1, 2, . . . , n. To avoid this, as it is common in such cases,we constrain the solution to a prespecified norm. Hence our problem now becomes minimize
y T Ly
subject to y T Dy ⫽ 1
Although we can work directly on the previous task, we will reshape slightly it in order to use tools that are more familiar to us. Define z ⫽ D 1/2 y
(6.80)
L˜ ⫽ D ⫺1/2 LD ⫺1/2
(6.81)
and
which is known as the normalized graph Laplacian matrix. It is now readily seen that our optimization problem becomes minimize
z T L˜ z
subject to z z ⫽ 1 T
(6.82) (6.83)
Using Lagrange multipliers and equating the gradient of the Lagrangian to zero (Appendix C) it turns out that the solution is given by L˜ z ⫽ z
(6.84)
In other words,computing the solution becomes equivalent to solving an eigenvalue problem. By substituting (6.84) into the cost function (6.82) and taking into account the constraint (6.83), it turns out that the value of the cost associated with the optimal z is equal to . Hence, the solution is the eigenvector corresponding to
“08-Ch06-SA272” 18/9/2008 page 357
6.7 Nonlinear Dimensionality Reduction
the minimum eigenvalue. However, the minimum eigenvalue of L˜ is zero and the corresponding eigenvector corresponds to a trivial solution. Indeed, observe that L˜ D 1/2 1 ⫽ D ⫺1/2 LD ⫺1/2 D 1/2 1 ⫽ D ⫺1/2 (D ⫺ W )1 ⫽ 0
where 1 is the vector having all its elements equal 1. In words, z ⫽ D 1/2 1 is an eigenvector corresponding to the zero eigenvalue, and it results to the trivial solution, yi ⫽ 1, i ⫽ 1, 2, . . . , n. That is, all the points are mapped onto the same point in the real line. To exclude this undesired solution, recall that L˜ is a nonnegative matrix, and, hence, 0 is its smallest eigenvalue (if the graph is connected, that is, at least one path (see Section 13.2.5) connects any pair of vertices, D 1/2 1 is the only eigenvector associated with the zero eigenvalue, 0 , [Belk 03]. This is an assumption we adopt here.) Also, since L˜ is a symmetric matrix, we know (Appendix B) that its eigenvectors are orthogonal to each other. In the sequel, we impose an extra constraint, and we now require the solution to be orthogonal to D 1/2 1. Constraining the solution to be orthogonal to the eigenvector corresponding to the smallest (zero) eigenvalue drives the solution to the next eigenvector corresponding to the next smallest (nonzero) eigenvalue 1 . Note that the eigendecomposition of L˜ is equivalent to what we called generalized eigendecomposition of L in step 3 earlier. For the more general case of m ⬎ 1, we have to compute the m eigenvectors associated with 1 ⱕ. . .ⱕm . For this case,the constraints prevent us from mapping into a subspace of dimension less than the desired m. For example, we do not want to project in a three-dimensional space and the points to lie on a two-dimensional plane or on an one-dimensional line. For more details, the interested reader is referred to the insightful paper [Belk 03].
Local Linear Embedding (LLE) As was the case with the Laplacian eigenmap method, LLE assumes that our data rest on a smooth enough manifold of dimension m, which is embedded in the RN subspace, with m ⬍ N ([Rowe 00]). The smoothness assumption allows us to further assume that, provided there is sufficient data and the manifold is “well” sampled, nearby points lie on (or close to) a “locally” linear patch of the manifold. The algorithm in its simplest form is summarized in the following three steps: Step 1: For each point x i , i ⫽ 1, 2, . . . , n, search for its nearest neighbors. Step 2: Compute the weights W (i, j), i, j ⫽ 1, 2, . . . , n, that best reconstruct each point, x i , from its nearest neighbors, so that to minimize the cost arg min EW ⫽ W
n i⫽1
||x i ⫺
n
W (i, j)x ij ||2
(6.85)
j⫽1
where, x ij denotes the jth neighbor of the ith point. The weights are constrained: (a) to be zero for points that are not neighbors and ( b) the
357
“08-Ch06-SA272” 18/9/2008 page 358
358
CHAPTER 6 Feature Generation I
rows of the weight matrix add to one, that is, n
W (i, j) ⫽ 1
(6.86)
j⫽1
That is, the sum of the weights, over all neighbors, must be equal to one. Step 3: Use the weights obtained from the previous step to compute the corresponding points y i ∈ Rm , i ⫽ 1, 2, . . . , n, so that to minimize the cost with respect to the unknown points Y ⫽ { y i , i ⫽ 1, 2, . . . , n} arg min EY ⫽ Y
n i⫽1
||y i ⫺
W (i, j)y j ||2
(6.87)
j
The above minimization takes place subjectto two constraints so as to avoid degenerate results:(a) the outputs are centered, i y i ⫽ 0 and (b) the outputs have unit covariance matrix ([Saul 01]). The nearest points, in step 1, are searched in the same way as carried out for the Laplacian eigenmap method. Once again, use of the Euclidean distance is justified by the smoothness of the manifold, as long as the search is limited “locally” among neighboring points. For the second step, the method exploits the local linearity of a smooth manifold and tries to predict linearly each point by its neighbors using the least squares error criterion. Minimizing the cost subject to the constraint given in (6.86) results in a solution that satisfies the following three properties: 1. Rotation invariance. 2. Scale invariance. 3. Translation invariance. The first two properties can easily be verified by the form of the cost function and the third one is the consequence of the constraint equation. The implication of this is that the computed weights encode information about the intrinsic characteristics of each neighborhood and they do not depend on the particular point. The resulting weights, W (i, j), reflect the intrinsic properties of the local geometry underlying the data. Since our goal is to retain the local information after the mapping, these weights are used to reconstruct each point in the Rm subspace by its neighbors. As is nicely stated in [Saul 01], it is as if we take a pair of scissors to cut small linear patches of the manifold and place them in the low-dimensional subspace. It turns out that solving for (6.87) for the unknown points y i , i ⫽ 1, 2, . . . , n, is equivalent to: ■
Performing an eigendecomposition of the matrix (I ⫺ W )T (I ⫺ W ).
■
Discarding the eigenvector that corresponds to the smallest eigenvalue.
■
Taking the eigenvectors that correspond to the next ( lower) eigenvalues. These yield the low–dimensional outputs y i , i ⫽ 1, 2, . . . , n.
“08-Ch06-SA272” 18/9/2008 page 359
6.7 Nonlinear Dimensionality Reduction
Once again, the involved matrix W is sparse, and if this is taken into account, the eigenvalue problem scales relatively well to large data sets with complexity subquadratic in n. The complexity for step 2 scales as O(nk3 ), and it is contributed by the solver of the linear set of equations with k unknowns for each point. The method requires that the user provides two parameters, the number of nearest neighbors, k (or ⑀) and the dimensionality m. The interested reader can find more on the LLE method in [Saul 01].
Isometric Mapping (ISOMAP) In contrast to the two previous methods that unravel the geometry of the manifold on a local basis, the ISOMAP algorithm adopts the view that only the geodesic distances between all pairs of the data points can reflect the true structure of the manifold. Euclidean distances between points in a manifold cannot represent it properly, since points that lie far apart, as measured by their geodesic distance, may be close when measured in terms of their Euclidean distance, (see Figure 6.5). ISOMAP is basically a variant of the MDS algorithm in which the Euclidean distances are substituted by the respective geodesic distances along the manifold. The essence of the method is to estimate geodesic distances between points that lie faraway. To this end, a two-step procedure is adopted: Step 1: For each point, x i , i ⫽ 1, 2, . . . , n, compute the nearest neighbors and construct a graph G(V , E) whose vertices represent input patterns and the edges connect nearest neighbors (nearest neighbors are computed with x2 0.06
0.02
20.02
20.06
20.1 20.1
20.05
0
0.05
0.1
x1
FIGURE 6.5
The point denoted by a “star” is deceptively closer to the point denoted by a “dot” than to the point denoted by a “box,” if distance is measured in terms of the Euclidean distance. However, if one is constrained to travel along the spiral, the geodesic distance is the one that determines closeness and it is the “box” point that is closer to the “star.”
359
“08-Ch06-SA272” 18/9/2008 page 360
360
CHAPTER 6 Feature Generation I
either of the two alternatives used for the Laplacian eigenmap method. The parameters k or ⑀ are user-defined parameters.) The edges are assigned weights based on the respective Euclidean distance. ( For nearest neighbors this is a good approximation of the respective geodesic distance.) Step 2: Compute the pairwise geodesic distances among all pairs (i, j), i, j ⫽ 1, 2, . . . , n, along shortest paths through the graph. The key assumption is that the geodesic between any two points on the manifold can be approximated by the shortest path connecting the two points along the graph G(V , E). To this end, efficient algorithms can be used to achieve it with complexity O(n2 ln n ⫹ n2 k) (e.g., Djikstar’s algorithm, [Corm 01]). This cost can be prohibitive for large values of n. Having estimated the geodesic distances between all pairs of point, the MDS method is mobilized. Thus, the problem becomes equivalent to performing the eigendecomposition of the respective Gram matrix and selecting the m most significant eigenvectors to represent the low-dimensional space. After the mapping, Euclidean distances between points in the low-dimensional subspace match the respective geodesic distances on the manifold in the original high-dimensional space. As is the case in PCA and MDS, m is estimated by the number of significant eigenvalues. It can be shown that ISOMAP is guaranteed asymptotically (n ⫺→ ⬁) to recover the true dimensionality of a class of nonlinear manifolds [Tene 00, Dono 04]. All three graph-based methods share a common step for computing nearest neighbors in a graph. This is a problem of complexity O(n2 ) but more efficient search techniques can be used by employing a special type of data structures, for example, [Beyg 06]. A notable difference between the ISOMAP on the one side and the Laplacian eigenmap and LLE methods on the other is that the latter two approaches rely on the eigendecomposition of sparse matrices as opposed to the ISOMAP that relies on the eigendecomposition of the dense Gram matrix. This gives a computational advantage to the Laplacian eigenmap and LLE techniques. Moreover, the calculation of the shortest paths in the ISOMAP is another computationally demanding task. Finally, it is of interest to note that the three graph-based techniques perform the goal of dimensionality reduction while trying to unravel, in one way or another, the geometric properties of the manifold on which the data (approximately) lie. In contrast, this is not the case with the kernel PCA, which shows no interest in any manifold learning. However, as the world is very small, in [Ham 04], it is pointed out that the graph-based techniques can be seen as special cases of the kernel PCA! This becomes possible if data-dependent kernels, derived from graphs encoding neighborhood information, are used in the place of predefined kernel functions. The goal of this section was to present some of the most basic directions that have been suggested for nonlinear dimensionality reduction. Besides the previous basic schemes, a number of improved updates have been proposed in the literature, for example, [Desi 03, Sha 05, Beng 04]. In [Lafo 06, Qui 07], the low-dimensional embedding is achieved to preserve certain measures that reflect the connectivity
“08-Ch06-SA272” 18/9/2008 page 361
6.7 Nonlinear Dimensionality Reduction
of the graph G(V , E). In [He 03, Cai 05, Koki 07] the idea of preserving the local information in the manifold has been carried out to define linear transforms of the form y ⫽ AT x, and the optimization is now carried out with respect to the elements of A. The task of incremental manifold learning for dimensionality reduction was more recently considered in [Law 06]. In [Wein 05, Sun 06], the maximum variance unfolding method is introduced. The variance of the outputs is maximized under the constraint that ( local) distances and angles are preserved among neighbors in the graph. Like the ISOMAP, it turns out that the top eigenvectors of a Gram matrix have to be computed, albeit avoiding the computationally demanding step of estimating geodesic distances, as required by the ISOMAP. In [Shui 07] a general framework, called graph embedding, is presented that offers a unified view for understanding and explaining a number of known (including PCA and nonlinear PCA) dimensionality reduction techniques, and it also offers a platform for developing new ones. In [Lin 08] a manifold learning technique is adopted that constructs coordinate charts for a given Riemannian manifold. For a more detailed and insightful treatment of the topic the interested reader is referred to [Burg 04]. A review of nonlinear dimensionality reduction techniques can be found in [Cama 03]. Example 6.6 A data set consists of 30 points in the two-dimensional space. The points result from sampling the spiral of Archimedes (see Figure 6.6a), described by x1 ⫽ a cos ,
x2 ⫽ a sin
The points of the data set correspond to the values ⫽ 0.5, and 0.7, 0.9, . . . , 2.05 ( is expressed in radians) and a ⫽ 0.1. For illustration purposes and in order to keep track of the “neighboring” information, we have used a sequence of six symbols, that is, “⫻”, “⫹”, “?”, “2”, “♦”, “◦”—with black color—followed by the same sequence of symbols in red color, repeatedly. To study the performance of PCA for this case, where data lie on a nonlinear manifold, we first performed the eigendecomposition of the covariance matrix, estimated from the data set. The resulting eigenvalues are 0 ⫽ 0.089
and
1 ⫽ 0.049
Observe that, in contrast to the linear case of Example 6.3, here the eigenvalues are comparable in size. Thus, if one would trust the “verdict” coming from PCA, the answer concerning the dimensionality of the data would be that it is equal to 2. Moreover, after projecting along the direction of the principal component (the straight line in Figure 6.6b), corresponding to 0 , neighboring information is lost since points from different locations are mixed together. Next, the Laplacian eigenmap technique for dimensionality reduction is employed, with √ ⫽ 0.2 and ⫽ 0.5. The results obtained are shown in Figure 6.6c. Looking from right to left, we see that the Laplacian method nicely “unfolds” the spiral in an one-dimensional straight line. Furthermore, neighboring information is retained in this one-dimensional
361
“08-Ch06-SA272” 18/9/2008 page 362
362
CHAPTER 6 Feature Generation I
x2
x2
0.1
0.1
20.1
20.1
20.3
20.3
20.5 20.3 20.1
0.1 (a)
20.05
0.3
0.5 x1
0.05
20.5 20.3 20.1
0.15
0.1 0.3 (b)
0.5 x1
0.25
(c)
FIGURE 6.6
(a) A spiral of Archimedes in the two-dimensional space. (b) The previous spiral together with the projections of the sampled points on the direction of the first principal component, resulting from PCA. It is readily seen that neighboring information is lost after the projection and points corresponding to different parts of the spiral overlap. (c) The one-dimensional map of the spiral using the Laplacian method. In this case, the neighboring information is retained after the nonlinear projection and the spiral nicely unfold to a one-dimensional line.
representation of the data. Black and red areas are succeeding each other in the right order, and also, observing the symbols, one can see that neighbors are mapped to neighbors.
Example 6.7 Figure 6.7 shows samples from a three-dimensional spiral, parameterized as x1 ⫽ a cos , x2 ⫽ a sin , and sampled at ⫽ 0.5, 0.7, 0.9, . . . , 2.05 ( is expressed in radians), a ⫽ 0.1 and x3 ⫽ ⫺1, ⫺0.8, ⫺0.6, . . . , 1. For illustration purposes and in order to keep track of the “identity” of each point, we have used red crosses and dots interchangeably, as we move upward in the x3 dimension. Also, the first, the middle and the last points for each level of x3 are denoted by black “♦,” black “?” and black “2,” respectively. Basically, all points at the same level lie on a two-dimensional spiral. Figure 6.8 shows the two-dimensional mapping of the three-dimensional spiral using the Laplacian method for dimensionality reduction, with parameter values ⑀ ⫽ 0.35 and √ ⫽ 0.5. Comparing Figures 6.7 and 6.8, we see that all points corresponding to the same level x3 are mapped across the same line, with the first point being mapped to the first one and so on. That is, as was the case in Example 6.6, the Laplacian method unfolds the three-dimensional spiral into a two-dimensional surface, while retaining neighboring information.
“08-Ch06-SA272” 18/9/2008 page 363
6.8 The Discrete Fourier Transform (DFT)
1 0.6 0.2 20.2 20.6 20.5 0
21 1
0.6
20.2
0.2
0.5
FIGURE 6.7
Samples from a three-dimensional spiral. One can think of it as a number of two-dimensional spirals one above the other. Different symbols have been used in order to track neighboring information.
0
⫺0.01
⫺0.02
⫺0.03 ⫺0.02
⫺0.01
0
0.01
0.02
FIGURE 6.8
Two-dimensional mapping of the spiral of Figure 6.7 using the Laplacian eigenmap method. The three-dimensional structure is unfolded to the two-dimensional space by retaining the neighboring information.
6.8 THE DISCRETE FOURIER TRANSFORM (DFT) We have already seen that the basis vectors/images for the KL and SVD expansions are not fixed but are “problem dependent” and are the result of an optimization
363
“08-Ch06-SA272” 18/9/2008 page 364
364
CHAPTER 6 Feature Generation I
process. This is the reason for their optimality with respect to the decorrelation and information-packing properties. At the same time, this accounts for their major disadvantage,that is,their high computational complexity. For the rest of the chapter we will be concerned with transforms that use fixed basis vectors/images. Their suboptimality with respect to decorrelation and information packing properties is most often compensated by their low computational requirements.
6.8.1 One-Dimensional DFT Given N input samples x(0), x(1), . . . , x(N ⫺ 1), their DFT is defined as N ⫺1 1 2 kn , y(k) ⫽ √ x(n) exp ⫺j N N n⫽0
k ⫽ 0, 1, . . . , N ⫺ 1
(6.88)
and the inverse DFT as N ⫺1 1 2 kn , x(n) ⫽ √ y(k) exp j N N k⫽0
where j ≡
n ⫽ 0, 1, . . . , N ⫺ 1
(6.89)
√ ⫺1. Sometimes, (6.88) is given without the normalizing factor
In such cases the normalizing factor in (6.89) becomes y(k) together into two N ⫻ 1 vectors and defining
1 N.
√1 . N
Collecting all x(n) and
2 WN ≡ exp ⫺j N
(6.90)
(6.88) and (6.89) are written in a matrix form as y ⫽ W H x, x ⫽ W y
(6.91)
where ⎡ 1 WH ⫽ √ N
⎢ ⎢ ⎢ ⎢ ⎣
... ... .. .
1 1 .. .
1 WN .. .
1 WN2 .. .
1
WNN ⫺1
WN2(N ⫺1)
1 WNN ⫺1 .. .
. . . WN(N ⫺1)(N ⫺1)
It is not difficult to see that W is a unitary and symmetric matrix W ⫺1 ⫽ W H ⫽ W ∗
The basis vectors are the columns of W. For example, for N ⫽ 4 ⎡
W ⫽
1⎢ ⎢ ⎢ 2⎣
1 1 1 j 1 ⫺1 1 ⫺j
1 ⫺1 1 ⫺1
1 ⫺j ⫺1 j
⎤ ⎥ ⎥ ⎥ ⎦
⎤ ⎥ ⎥ ⎥ ⎥ ⎦
(6.92)
“08-Ch06-SA272” 18/9/2008 page 365
6.8 The Discrete Fourier Transform (DFT)
and the basis vectors are 1 [1, 1, 1, 1]T 2 1 w 1 ⫽ [1, j, ⫺1, ⫺j]T 2 1 w 2 ⫽ [1, ⫺1, 1, ⫺1]T 2 1 w 3 ⫽ [1, ⫺j, ⫺1, j]T 2
w0 ⫽
and 3
x⫽
y(i)w i
i⫽0
The direct computation of (6.91) requires O(N 2 ) computations. However, taking advantage of the specific structure of the matrix W , a substantial saving in computations is possible via the celebrated Fast Fourien Transform (FFT) algorithm, which computes each equation of (6.91) in O(N log2 N ) operations [Proa 92]. So far,the DFT has been introduced as a special type of a linear unitary transform of one vector to another. Another point of view, which will be useful later in this chapter, is to see the DFT as a means of expanding a sequence x(n) into a set of N basis sequences hk (n) x(n) ⫽
N ⫺1
y(k)hk (n)
k⫽0
where
# hk (n) ⫽
√1 N
exp( j 2 N kn), 0,
n ⫽ 0, 1, . . . , N ⫺ 1 otherwise
and y(k) are the coefficients of the expansion. The DFT basis sequences belong to a more general class of sequences known as orthonormal, that is, hl (n), hk (n) ≡
hk (n)hl∗ (n) ⫽ ␦kl
(6.93)
n
where . , . is known as the inner product of the sequences hk (n), hl (n). For the DFT expansion, we have hk (n), hl (n) ⫽
N ⫺1 2 2 1 kn exp ⫺j ln exp j N N N n⫽0
⫽
N ⫺1 2 1 (k ⫺ l)n , exp j N N n⫽0
k, l ⫽ 0, 1, . . . , N ⫺ 1
365
“08-Ch06-SA272” 18/9/2008 page 366
366
CHAPTER 6 Feature Generation I
However, it can easily be shown ( Problem 6.8) that # N ⫺1 2 1 1, l ⫽ k ⫹ rN , r ⫽ 0, ⫾1, ⫾2, . . . exp j (k ⫺ l)n ⫽ 0, otherwise N N
(6.94)
n⫽0
Hence hk (n), hl (n) ⫽ ␦kl
6.8.2 Two-Dimensional DFT Given an N ⫻ N matrix/image, its two-dimensional DFT is defined as Y (k, l) ⫽
N ⫺1 N ⫺1 1 X(m, n)WNkm WNln N
(6.95)
N ⫺1 N ⫺1 1 Y (k, l)WN⫺km WN⫺ln N
(6.96)
m⫽0 n⫽0
and the inverse DFT as X(m, n) ⫽
k⫽0 l⫽0
It is readily seen that this can be written in a compact form as Y ⫽ W H XW H , X ⫽ WYW
(6.97)
Thus, the two-dimensional DFT is a separable transform with basis images w i w Tj , i, j ⫽ 0, 1, . . . , N ⫺ 1. It is apparent from (6.97) that the number of operations required for the respective computations is O(N 2 log2 N ), that is, the number of additions and multiplications needed for 2N one-dimensional DFTs, via the FFT algorithm.
6.9 THE DISCRETE COSINE AND SINE TRANSFORMS Given N input samples x(0), x(1), . . . , x(N ⫺ 1) their discrete cosine transform ( DCT ) is defined as y(k) ⫽ ␣(k)
N ⫺1
x(n) cos
n⫽0
(2n ⫹ 1)k 2N
, k ⫽ 0, 1, . . . , N ⫺ 1
(6.98)
, n ⫽ 0, 1, . . . , N ⫺ 1
(6.99)
and the inverse DCT is given by x(n) ⫽
N ⫺1 k⫽0
␣(k)y(k) cos
(2n ⫹ 1)k 2N
“08-Ch06-SA272” 18/9/2008 page 367
6.9 The Discrete Cosine and Sine Transforms
where
⎧% ⎪ ⎨ N1 , k ⫽ 0 ␣(k) ⫽ % ⎪ ⎩ 2 , k ⫽ 0 N
In vector form the transform is written as y ⫽ CT x
where the elements of the matrix C are given by 1 C(n, k) ⫽ √ , N & C(n, k) ⫽
k ⫽ 0, 0 ⱕ n ⱕ N ⫺ 1
(2n ⫹ 1)k 2 cos , N 2N
1 ⱕ k ⱕ N ⫺ 1, 0 ⱕ n ⱕ N ⫺ 1
Matrix C has real elements, and it is easy to see that it is orthogonal, C ⫺1 ⫽ C T
The two-dimensional DCT is the separable transform defined as Y ⫽ C T XC,
X ⫽ CYC T
(6.100)
The discrete sine transform ( DST ) is defined via the transform matrix & S(k, n) ⫽
(k ⫹ 1)(n ⫹ 1) 2 sin , N ⫹1 N ⫹1
k, n ⫽ 0, 1, . . . , N ⫺ 1
and it is also an orthogonal transform. The DCT and DST belong to the family of transforms that can be computed via a fast method in O(N log2 N ) operations [Jain 89, Lim 90]. Remark ■
The DCT and DST have very good information-packing properties for most of the images, in the sense that they concentrate most of the energy in a few coefficients. An explanation for this property is that both offer a close approximation to the (KL) transform for a class of random signals, known as first-order Markov processes, which can approximately model a number of real-world images [Jain 89]. Figure 6.9 shows an image and the resulting DFT (magnitude), DST, and DCT transforms. It is apparent from the figure that the high-intensity coefficients of the transforms (dark) are concentrated in a small area, whose size depends on the energy compaction properties of the respective transform. This area is smaller for DCT and DST than for DFT.
367
“08-Ch06-SA272” 18/9/2008 page 368
368
CHAPTER 6 Feature Generation I
FIGURE 6.9 Example of an image and its magnitude DFT, DST, and DCT transforms, shown from top left to bottom left in the clockwise sense.
6.10 THE HADAMARD TRANSFORM The Hadamard transform and the Haar transform, to be considered in the next section, share a serious computational advantage over the previously considered DFT, DCT, and DST transforms. Their unitary matrices consist of ⫾1, and the transforms are computed via additions and subtractions only, with no multiplications being involved. Hence, for processors for which multiplication is a time-consuming operation, a substantial saving is obtained. The Hadamard unitary matrix of order n is the N ⫻ N matrix, N ⫽ 2n , generated by the following iteration rule: Hn ⫽ H1 ⊗ Hn⫺1
where
1 1 H1 ⫽ √ 2 1
(6.101)
1 ⫺1
(6.102)
“08-Ch06-SA272” 18/9/2008 page 369
6.11 The Haar Transform
and ⊗ denotes the Kronecker product of two matrices ⎡
A(1, 1)B ⎢ .. ⎢ A⊗B ⫽⎣ . A(N , 1)B
A(1, 2)B .. . A(N , 2)B
⎤ . . . A(1, N )B ⎥ .. .. ⎥ . . ⎦ . . . A(N , N )B
where A(i, j) is the (i, j) element of A, i, j ⫽ 1, 2, . . . , N . Thus, according to (6.101), (6.102), it is ⎡
1 1⎢ ⎢1 H2 ⫽ H1 ⊗ H1 ⫽ ⎢ 2 ⎣1 1
and for n ⫽ 3
1 ⫺1 1 ⫺1
1 1 ⫺1 ⫺1
1 H2 H3 ⫽ H1 ⊗ H2 ⫽ √ 2 H2
⎤ 1 ⎥ ⫺1⎥ ⎥ ⫺1⎦ 1
H2 ⫺H2
It is not difficult to show the orthogonality of Hn , n ⫽ 1, 2, . . . , that is, Hn⫺1 ⫽ HnT ⫽ Hn
For a vector x of N samples and N ⫽ 2n the transform pair is y ⫽ Hn x, x ⫽ Hn y
(6.103)
The two-dimensional Hadamard transform is given by Y ⫽ Hn XHn ,
X ⫽ Hn YHn
(6.104)
The Hadamard transform has good to very good energy packing properties. Fast algorithms for its computation in O(N log2 N ) subtractions and/or additions are also available [Jain 89]. Remark ■
Experimental results using the DCT, DST, and Hadamard transforms for texture discrimination have shown that the performance obtained was close to that of the optimal KL transform [Unse 86, Unse 89]. At the same time, this near-optimal performance is obtained at substantially reduced complexity,due to the availability of fast computational schemes as reported before.
6.11 THE HAAR TRANSFORM The starting point for the definition of the Haar transform is the Haar functions hk (z), which are defined in the closed interval [0, 1]. The order k of the function is
369
“08-Ch06-SA272” 18/9/2008 page 370
370
CHAPTER 6 Feature Generation I
Table 6.1 Parameters for the Haar functions k
0
1
2
3
4
5
6
7
p
0
0
1
1
2
2
2
2
q
0
1
1
2
1
2
3
4
uniquely decomposed into two integers p, q k ⫽ 2p ⫹ q ⫺ 1,
k ⫽ 0, 1, . . . , L ⫺ 1, and L ⫽ 2n
where 0 ⱕ p ⱕ n ⫺ 1, 0 ⱕ q ⱕ 2p
for
p ⫽ 0 and q ⫽ 0 or 1 for p ⫽ 0
Table 6.1 summarizes the respective values for L ⫽ 8. The Haar functions are 1 h0 (z) ≡ h00 (z) ⫽ √ , L
z ∈ [0, 1]
⎧ p ⎪ ⎪ 22 ⎪ ⎪ 1 ⎨ p hk (z) ≡ hpq (z) ⫽ √ ⫺2 2 L⎪ ⎪ ⎪ ⎪ ⎩ 0
(6.105)
q⫺1 2p
ⱕz⬍
q⫺ 12 2p
q⫺ 12 2p
ⱕz⬍
q 2p
(6.106)
otherwise in [0, 1]
The Haar transform matrix of order L consists of rows resulting from the preceding functions computed at the points z ⫽ m L , m ⫽ 0, 1, 2, . . . , L ⫺ 1. For example, the 8 ⫻ 8 transform matrix is ⎡
1 ⎢ ⎢ 1 ⎢√ ⎢ 2 ⎢ 1 ⎢ ⎢ 0 H⫽ √ ⎢ 2 8⎢ ⎢ ⎢ 0 ⎢ ⎢ ⎣ 0 0
⎤ 1 1 1 1 1 1 1 ⎥ 1 1 1 ⫺1 ⫺1 ⫺1 ⫺1⎥ √ √ √ ⎥ 2 ⫺ 2 ⫺ 2 0 0 0 0⎥ √ √ √ √ ⎥ 0 0 0 2 2 ⫺ 2 ⫺ 2⎥ ⎥ ⎥ ⫺2 0 0 0 0 0 0⎥ ⎥ 0 2 ⫺2 0 0 0 0⎥ ⎥ ⎥ 0 0 0 2 ⫺2 0 0⎦ 0 0 0 0 0 2 ⫺2
(6.107)
It is not difficult to see that H ⫺1 ⫽ H T
that is, H is orthogonal. The energy packing properties of the Haar transform are not very good. However, its importance for us lies beyond that. We will use it as the vehicle to take us from the world of unitary transforms to that of multiresolution analysis. To this end, let us look carefully at the Haar transform matrix. We readily
“08-Ch06-SA272” 18/9/2008 page 371
6.12 The Haar Expansion Revisited
observe its sparse nature with a number of zeros, whose location reveals an underlying cyclic shift mechanism. To satisfy our curiosity as to why this happens, let us look at the Haar transform from a different perspective.
6.12 THE HAAR EXPANSION REVISITED Let us split our original set of N input samples (N even) x(0), x(1), . . . , x(N ⫺ 1) into successive blocks of two, that is, (x(2k), x(2k ⫹ 1)), k ⫽ 0, 1, . . . , N2 ⫺ 1, and apply the Haar transform of order L ⫽ 2. For each pair of input samples, a pair of transformed samples is obtained,
1 1 ⫽ √ y0 (k) 2 1
y1 (k)
1 x(2k) ⫺1
x(2k ⫹ 1)
, k ⫽ 0, 1, . . . ,
N ⫺1 2
(6.108)
That is, 1 y1 (k) ⫽ √ (x(2k) ⫹ x(2k ⫹ 1)) 2
(6.109)
N 1 y0 (k) ⫽ √ (x(2k) ⫺ x(2k ⫹ 1)), k ⫽ 0, 1, . . . , ⫺ 1 2 2
(6.110)
This can be interpreted as the action—on the sequence of N input samples— of two (noncausal) filters with impulse responses (h1 (0) ⫽ √1 , h1 (⫺1) ⫽ √1 ) and 2
2
(h0 (0) ⫽ √1 , h0 (⫺1) ⫽ ⫺ √1 ), respectively. The corresponding transfer functions 2 2 (Appendix D) are 1 H1 (z) ⫽ √ (1 ⫹ z) 2
(6.111)
1 H0 (z) ⫽ √ (1 ⫺ z) 2
(6.112)
In other words, the order L ⫽ 2 Haar transform computes the output samples of the two filters when they are fed with the input sequence x(n), n ⫽ 0, 1, 2, . . . , N ⫺ 1. Furthermore, the output sequence samples are computed for every other sample of the input sequence, at even time instants 0, 2, 4, . . . , as (6.109) and (6.110) suggest. This operation is portrayed in Figure 6.10b. The operation at the output of the two filters is known as subsampling by M, in this case M ⫽ 2, and it is defined in Figure 6.10a. In other words, from the samples generated at the filter output we keep one every M(⫽ 2). In the time domain and for an input sequence consisting of eight samples, the output, y0 (k), of the H0 branch of Figure 6.10b will consist of
371
“08-Ch06-SA272” 18/9/2008 page 372
372
CHAPTER 6 Feature Generation I
y
M
y
y(k) 5 y (kM) (a)
H0
2
H1
2
y0
...x (2), x (1), x (0)
( b)
FIGURE 6.10 (a) Subsampling operation and (b) filtering interpretation of the Haar transform.
four samples given by ⎡
x(0)
⎤
⎢ ⎥ ⎢x(1)⎥ ⎢ ⎥ ⎢ –– ⎥ ⎢ ⎥ ⎥ ⎤⎢ ⎢x(2)⎥ ⎡ ⎤ ⎡ √1 √1 ⎢ ⎥ ⫺ | 0 0| 0 0| 0 0 y0 (0) 2 2 ⎥ ⎥⎢ ⎢x(3)⎥ ⎢ ⎥ ⎢ 1 1 ⎥ ⎢ √ √ ⫺ | 0 0| 0 0 0 0| ⎢ ⎢ y0 (1)⎥ ⎢ ⎥ ⎥ 2 2 ⎢ ⎥ ⎥ ⎥⎢ 1 1 ⎢ –– ⎥ ⎢ y (2)⎥ ⫽ ⎢ √ √ ⎥ ⎢ ⫺ | 0 0 0 0| 0 0| ⎥ ⎣ 0 ⎦ ⎣ 2 2 ⎦⎢ ⎢x(4)⎥ ⎥ 0 0| 0 0| 0 0| √1 ⫺ √1 ⎢ y0 (3) 2 2 ⎢x(5)⎥ ⎢ ⎥ ⎢ ⎥ ⎢ –– ⎥ ⎢ ⎥ ⎢x(6)⎥ ⎣ ⎦ x(7)
(6.113)
This is nothing other than the action of the last four rows of the 8 ⫻ 8 Haar transform in (6.107)! What about the rest? Let us carry on the splitting of Figure 6.10b one step further, as shown in Figure 6.11. Using the easily shown Noble identity illustrated in Figure 6.12a ( Problem 6.17 ), the structure of Figure 6.11 turns out to be equivalent to that of Figure 6.12b. Taking into account the subsampling operation of the lower branch after the filters H0 and H1 , the Noble identity leads to Fˆ 1 (z) ⫽
1 1 (1 ⫹ z)(1 ⫺ z 2 ) ⫽ (1 ⫹ z ⫺ z 2 ⫺ z 3 ) 2 2
(6.114)
Fˆ 2 (z) ⫽
1 1 (1 ⫹ z)(1 ⫹ z 2 ) ⫽ (1 ⫹ z ⫹ z 2 ⫹ z 3 ) 2 2
(6.115)
“08-Ch06-SA272” 18/9/2008 page 373
6.12 The Haar Expansion Revisited
H0
y0
2
. . .x (2), x (1), x (0) H1
H0
2
H1
2
y1
2 y2
FIGURE 6.11 Two-stage filtering followed by subsampling operation.
M
5
H(z)
M H(z )
M
(a)
F0
2
F1
4
F2
4
. . .x(2), x(1), x(0)
y0
F0(z) 5 H0(z)
y1
F1(z) 5 H0(z )H1(z)
2
y2
2
F2(z) 5 H1(z )H1(z)
( b)
FIGURE 6.12 (a) Noble identity I and (b) equivalent filter bank of Figure 6.11.
From the transfer function Fˆ 1 (z) and taking into account the subsampling by 4 (2 ⫻ 2) operation, the first two samples of the y1 (k) sequence are given by ⎡
1 y1 (0) 2 ⫽ y1 (1) 0
1 2
⫺ 12
⫺ 12
0
0
0
0
0
0
1 2
1 2
⫺ 12
⎤ x(0)
⎢ ⎥ 0 ⎢x(1)⎥ ⎢ . ⎥ ⎢ . ⎥ ⫺ 12 ⎣ . ⎦
(6.116)
x(7)
This is nothing but the action of the third and fourth rows of the 8 ⫻ 8 Haar transform on the input vector. If we now carry on the splitting one step further, as in Figure 6.13, it is straightforward to show by repeating the preceding arguments that ⎡
1 ' y2 (0) ⫽ √ 1 8
1
1
1
⫺1
⫺1
⫺1
⎤ x(0) ⎥ (⎢ ⎢x(1)⎥ ⎥ . ⫺1 ⎢ ⎢ . ⎥ ⎣ . ⎦ x(7)
(6.117)
373
“08-Ch06-SA272” 18/9/2008 page 374
374
CHAPTER 6 Feature Generation I
H0
2
y0
...x (2), x (1), x (0)
H0 H1
2
y1
2 H1
H0
2
H1
2
y2
2 y3
FIGURE 6.13 Tree-structured filter bank. H() (dB) 0
25 H1()
H0() 210
215
220
2
0
FIGURE 6.14 Magnitude of the frequency response for the two Haar filters. H1 is a low-pass and H0 high-pass filter.
and ⎡
1 ' y3 (0) ⫽ √ 1 8
1
1
1
1
1
1
⎤ x(0) ⎥ (⎢ ⎢x(1)⎥ ⎥ . 1 ⎢ ⎢ . ⎥ ⎣ . ⎦
(6.118)
x(7)
These equations are the actions of the second and first rows of the Haar transform on the input vector. The structure of Figure 6.13 is known as a (three-level) tree-structured filter bank generated by the filters H0 (z) and H1 (z). Figure 6.14 shows the frequency responses of these two filters. One (H0 (z)) is a high-pass and the other a low-pass filter. Herein lies the importance of the filter bank interpretation of the Haar transform. The input sequence x(n) is first split into two
“08-Ch06-SA272” 18/9/2008 page 375
6.13 Discrete Time Wavelet Transform (DTWT)
versions of lower resolution with respect to the original one: a low-pass (average) coarser resolution version and a high-pass (difference) detailed resolution one. In the sequel the coarser resolution version is further split into two versions, and so on. This leads to a number of versions with a hierarchy of resolutions. This decomposition is known as multiresolution decomposition. The idea of multiresolution decomposition has been around for some time [Burt 83, Akan 93] and has been exploited in various applications and for a number of reasons. Its popularity as a tool in pattern recognition is mainly due to the information compaction capabilities associated with such a decomposition, provided filters H0 , H1 are properly designed. For many types of signals, such as speech and images, most of the information is localized in certain resolution levels. Thus, most of the energy is concentrated in a (relatively) small number of samples, which carry most of the necessary information [Este 77, Mall 89]. Some fundamental issues related to the multiresolution decomposition and the design of filter banks will be highlighted next.
6.13 DISCRETE TIME WAVELET TRANSFORM (DTWT) The goal of this section is twofold. We first free ourselves from the Haar functions, and we seek the possibility of using other filters in place of H0 , H1 . There is more than one reason for this generalization. An obvious reason is that the frequency responses of the Haar filters are far from ideal. If our aim is to split the original sequence into a hierarchy of “coarse” and “detailed” versions, we should require the filters that perform the splitting to be as close as possible to the ideal low/highpass ones (Figure 6.15). Our next concern in this section is the inversion problem. |H1()| 1
2
2
(a) |H0()| 1
( b)
FIGURE 6.15 Ideal frequency responses for (a) low-pass and (b) high-pass filters.
375
“08-Ch06-SA272” 18/9/2008 page 376
376
CHAPTER 6 Feature Generation I
That is,if we know the lower resolution versions,can we obtain the original sequence, x(n), as was the case with the unitary transforms? We will show that under certain constraints in the design of H0 and H1 this is indeed possible.
The Two-Band Case Let us start with the simple two-band case of Figure 6.10b, where we now assume that the filters are not the Haar ones. If h0 (k), h1 (k) are the respective impulse responses, then we can write y0 (k) ⫽
x(l)h0 (n ⫺ l)|n⫽2k
l
y1 (k) ⫽
x(l)h1 (n ⫺ l)|n⫽2k
l
where y1 (k) is the output of the lower branch of Figure 6.10b. Collecting all y0 (k), y1 (k), k ⫽ 0, 1, 2, . . . , together in a vector, we have ⎡ ⎡ . ⎤ .. .. . . ⎢ . ⎢ . ⎥ ⎢ ⎥ ⎢. . . h0 (2) ⎢ ⎢y0 (0)⎥ ⎢ ⎥ ⎢ ⎢ ⎢y1 (0)⎥ ⎢. . . h1 (2) ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ ⎢y0 (1)⎥ ⎢. . . ... ⎥ ⎢ ⎢ ⎢y (1)⎥ ⫽ ⎢ ... ⎢ 1 ⎥ ⎢. . . ⎥ ⎢ ⎢ ⎢y0 (2)⎥ ⎢. . . . .. ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ ⎢y1 (2)⎥ ⎢. . . ... ⎦ ⎢ ⎣ ⎣ . .. .. .. . .
.. .
.. .
.. .
.. .
h0 (1)
h0 (0)
h0 (⫺1)
h0 (⫺2)
h1 (1)
h1 (0)
h1 (⫺1)
h1 (⫺2)
...
h0 (2)
h0 (1)
h0 (0)
...
h1 (2)
h1 (1)
h1 (0)
...
...
...
h0 (2)
... .. .
... .. .
... .. .
h1 (2) .. .
⎤ .. . ⎥ ⎥⎡ ⎤ . . .⎥ . ⎥ ⎥ ⎢ .. ⎥ . . .⎥ ⎥ ⎥⎢ ⎥ ⎢x(0)⎥ ⎥ . . .⎥ ⎢ ⎥ ⎥⎢ x(1) ⎢ ⎥ ⎥ ⎥ . . .⎥ ⎢ ⎥ ⎥⎢ x(2) ⎢ ⎥ . . .⎥ ⎥⎣ . ⎦ ⎥ .. . . .⎥ ⎥ .. ⎦ .
or y ⫽ Ti x
(6.119)
Here, we have assumed that the filters can be noncausal, but they are of finite impulse response ( FIR) (the impulse response has a finite nonzero number of terms). The latter assumption is imposed here in order to avoid issues of convergence of infinite series. Observe the structure of Ti . It basically consists of two rows, one with the impulse response of H0 and the other with that of H1 , which are then shifted each time by two to the right, to form the rest of the rows. This is the result of the subsampling, by two, operation. Figure 6.16b shows a structure that ˆ The symbol combines y0 (k), y1 (k), through the filters G0 , G1 , to form a sequence x. at the input of the filters denotes the upsampling by M operation, which is defined in Figure 6.16a. In this case M ⫽ 2. In other words, this is equivalent to stuffing M ⫺ 1 zeros between every two samples. That is, the input sequences of the filters G0 , G1 will be . . . 0 y0 (0) 0 y0 (1) 0 y0 (2) 0 . . . . . . 0 y1 (0) 0 y1 (1) 0 y1 (2) 0 . . .
“08-Ch06-SA272” 18/9/2008 page 377
6.13 Discrete Time Wavelet Transform (DTWT)
y
y
M
y(k) 5 y(l ), k 5 lM y(k) 5 0
y0
y1
otherwise (a)
2
G0
2
G1
x0
x1
x ( b)
FIGURE 6.16 (a) The upsampling operation. (b) Tree-structured synthesis filter bank.
respectively. Thus, every other sample of the impulse response hits a zero and x0 (n) ⫽
y0 (k)g0 (n ⫺ 2k)
k
x1 (n) ⫽
y1 (k)g1 (n ⫺ 2k)
k
x(n) ˆ ⫽ x0 (n) ⫹ x1 (n)
Filters Gi are known as the synthesis filters and the corresponding Hi , of Figure 6.10b, as the analysis filters. Collecting all x(n) ˆ together, it is not difficult to see that ⎡ .. ⎤ .. . ⎢ ⎢ . ⎥ ⎢ ⎢ ⎥ ⎢. . . ⎢x(0) ⎥ ⎢ ⎢ ˆ ⎥ ⎢. . . ⎢ ⎥ ⎢ ˆ ⎥⫽⎢ ⎢x(1) ⎢ ⎥ ⎢. . . ⎢x(2) ⎥ ⎢ ⎢ˆ ⎥ ⎢ ⎣ . ⎦ ⎢. . . ⎣ .. .. . ⎡
.. .
.. .
.. .
.. .
g0 (0)
g1 (0)
g0 (⫺2)
g1 (⫺2)
g0 (1)
g1 (1)
g0 (⫺1)
g1 (⫺1)
g0 (2)
g1 (2)
g0 (0)
g1 (0)
g0 (3) .. .
g1 (3) .. .
g0 (1) .. .
g1 (1) .. .
⎤⎡ ⎤ . .. . ⎥ ⎢ .. ⎥ ⎥⎢ ⎥ . . .⎥ ⎢y0 (0)⎥ ⎥⎢ ⎥ ⎢ ⎥ . . .⎥ ⎥ ⎢y1 (0)⎥ ⎥⎢ ⎥ ⎢ ⎥ . . .⎥ ⎥ ⎢y0 (1)⎥ ⎥⎢ ⎥ . . .⎥ ⎢y1 (1)⎥ ⎦⎣ ⎦ .. .. . .
or xˆ ⫽ To y
(6.120)
In order that xˆ ⫽ x we require that [Vett 92] To Ti ⫽ I ⫽ Ti To
(6.121)
377
“08-Ch06-SA272” 18/9/2008 page 378
378
CHAPTER 6 Feature Generation I
Multiplying rows of Ti with columns of To , (6.121) becomes equivalent to
hi (2k ⫺ n)gj (n ⫺ 2l) ⫽ ␦kl ␦ij ,
i, j ⫽ 0, 1
(6.122)
n
or according to the definition of the inner product in (6.93), hi (2k ⫺ n),gj (n ⫺ 2l) ⫽ ␦kl ␦ij
If (6.122) is satisfied, we say that the two-band filter bank is a perfect reconstruction one and x(n) ˆ ⫽ x(n). Thus, x(n) ⫽
y0 (k)g0 (n ⫺ 2k) ⫹
k
y1 (k)g1 (n ⫺ 2k)
(6.123)
k
Equation (6.123) can also be seen from a different perspective. It is an expansion of x(n) into a set of basis sequences {g0 (n ⫺ 2k), g1 (n ⫺ 2k)}, k ∈ Z
where Z is the set of the integer numbers. From such a point of view y0 (k), y1 (k) are the respective coefficients of the expansion. This is known as the discrete time wavelet transform ( DTWT ) and the coefficients y0 (k), y1 (k) as the discrete time wavelet coefficients. Thus, given a perfect reconstruction two-band filter bank (i.e., condition (6.122) is satisfied), the following transform pair is defined: yi (k) ⫽ x(n) ⫽
x(n)hi (2k ⫺ n) (a)
n 1
yi (k)gi (n ⫺ 2k) (b)
(6.124)
i⫽0 k
Remarks ■
Two sets of basis functions are involved, namely, hi (2k ⫺ n) ≡ ik (n), gj (n ⫺ 2l) ≡ jl (n)
i, j ⫽ 0, 1 and k, l ∈ Z
Equation (6.122) is an orthogonality condition between ik (n) and jl (n), that is, ik (n), jl (n) ⫽ ␦ij ␦kl
and it is known as the biorthogonality condition. The discrete time wavelet transform pair in (6.124) is a biorthogonal expansion. ■
The basis sequences ik (n) and jl (n) of the expansion are shifts by an even number of samples of four basic mother sequences g0 (n), g1 (n), h0 (⫺n), h1 (⫺n), which are the impulse responses of the synthesis and the timereversed analysis filters. For the recovery of x(n) from its discrete time wavelet coefficients, each coefficient yi (k) weighs and adds a copy of the mother sequences gi (n) shifted by 2k.
“08-Ch06-SA272” 18/9/2008 page 379
6.13 Discrete Time Wavelet Transform (DTWT)
■
When the sequences ik (n) ⫽ hi (2k ⫺ n) are themselves orthogonal, that is,
hji (2k ⫺ n)hj (2l ⫺ n) ⫽ ␦kl ␦ij ,
i, j ⫽ 0, 1 and k, l ∈ Z
n
then gi (n) ⫽ hi (⫺n)
That is, the synthesis filters are the time reverse of the analysis ones. Such a filter bank is known as orthogonal or paraunitary, and we have the same set of mother sequences (hi only) involved in both equations of the discrete time wavelet transform (6.124). ■
A number of orthogonal and biorthogonal perfect reconstruction filter pairs have been proposed in the literature [Daub 90, Vett 95]. Table 6.2 gives the coefficients for the first four of Daubechies’s maximally flat orthogonal filters. The low-pass version h1 (n) is shown. The high-pass versions are obtained as h0 (n) ⫽ (⫺1)n h1 (⫺n ⫹ 2L ⫺ 1), where L is the length of the filters.
■
Besides the case of wavelet basis sequences with predefined values, a large research effort has been devoted to constructing such sequences that are optimized to the specific problem of interest. This has also been used in pattern recognition applications. For example, in [Mall 97] it is proposed to design the filters of the bank to optimize a class discriminant criterion. A different approach is followed in [Szu 92], where an optimal linear combination of predefined bases is sought for classification of speech signals.
■
When implementing filter banks in practice, noncausal filters have to be appropriately delayed to make them realizable (Appendix D). This makes it
Table 6.2 h1 (0)
Daubechies’s Low-pass Filters of Length 4, 6, 8, and 10 0.4829629
0.33267
0.2303778
0.1601024
h1 (1)
0.8365163
0.806891
0.7148466
0.6038293
h1 (2)
0.2241439
0.459877
0.6308808
0.7243085
h1 (3)
⫺0.1294095
⫺0.135011
⫺0.0279838
0.1384281
h1 (4)
⫺0.08544
⫺0.1870348
⫺0.2422949
h1 (5)
0.03522
0.0308414
⫺0.0322449
h1 (6)
0.0328830
0.0775715
h1 (7)
⫺0.0105974
⫺0.0062415
h1 (8)
⫺0.0125807
h1 (9)
0.0033357
379
“08-Ch06-SA272” 18/9/2008 page 380
380
CHAPTER 6 Feature Generation I
necessary to involve certain delay elements at different points, in order to safeguard the perfect reconstruction property of the analysis–synthesis bank (Problem 6.19). ■
In practice, the number of input samples x(n) is finite, that is, n ⫽ 0, 1, . . . , N ⫺ 1. Thus, for the computation of (6.124) some initial conditions are required. Zero, periodic, or symmetric extensions of the data are popular alternatives. Such implementation issues as well as algorithms for the efficient computation of the DTWT coefficients are discussed in [Vett 95, Chapter 6].
Many Bands Case Figure 6.17 shows the synthesis part corresponding to the analysis bank of Figure 6.13, and it is a generalization of the two-band synthesis concept. Using the easily shown Noble identity given in Figure 6.18 (Problem 6.17), we end up with the equivalent structure of the synthesis part, shown in Figure 6.19. Let fi (n) be the impulse responses of the Fi filters. It is easy to see that the respective ˆ is contribution of each yi (k) sequence to the output x(n) xi (n) ⫽
yi (k) fi (n ⫺ 2i⫹1 k)
i ⫽ 0, 1, . . . , J ⫺ 2
k
xJ ⫺1 (n) ⫽
yJ ⫺1 (k) fJ ⫺1 (n ⫺ 2 J ⫺1 k)
k
y0
y2
G0
2
2
G1
G1
2 y3
G0
G0
2
y1
2
G1
2
x
FIGURE 6.17 Tree-structured synthesis filter bank.
H(z)
FIGURE 6.18 Noble identity II.
M
⫽
M
H(zM)
“08-Ch06-SA272” 18/9/2008 page 381
6.13 Discrete Time Wavelet Transform (DTWT)
y0
y1
y2
y3
2
F0
4
F1
8
F2
8
F3
x0
F0(z) ⫽ G0(z)
x1
F1(z) ⫽ G0(z2)G1(z)
x2
F2(z) ⫽ G0(z4)G1(z2)G1(z)
x3
F3(z) ⫽ G1(z4)G1(z2)G1(z) x
FIGURE 6.19 Equivalent of the tree-structured filter bank of Figure 6.17.
x(n) ˆ ⫽
J ⫺1
xi (n)
i⫽0
where J is the number of bands, with J ⫽ 4 in the case of Figures 6.17 and 6.19. It can be shown [Vaid 93] that if the mother filters G0 , G1 , which generate the synthesis part, and the mother filters H0 , H1 of the analysis part satisfy the biorthogonality condition (6.122),then the J -level analysis–synthesis bank is also a perfect reconstruction filter bank, that is, x(n) ˆ ⫽ x(n) ⫽
J ⫺2
yi (k) fi (n ⫺ 2i⫹1 k) ⫹
i⫽0 k
yJ ⫺1 (k) fJ ⫺1 (n ⫺ 2 J ⫺1 k)
(6.125)
k
where yi (k) ⫽
x(n) fˆi (2i⫹1 k ⫺ n),
i ⫽ 0, 1, . . . , J ⫺ 2
(6.126)
n
yJ ⫺1 (k) ⫽
x(n) fˆJ ⫺1 (2 J ⫺1 k ⫺ n)
(6.127)
n
with fˆi (k) being the impulse responses of the corresponding analysis band, in analogy with Figure 6.12b. To summarize our findings, let us define ik (n) ⫽ fi (n ⫺ 2i⫹1 k), ( J ⫺1)k (n) ⫽ fJ ⫺1 (n ⫺ 2
J ⫺1
i ⫽ 0, 1, . . . , J ⫺ 2 k)
381
“08-Ch06-SA272” 18/9/2008 page 382
382
CHAPTER 6 Feature Generation I
Table 6.3 yi (k) ⫽
The Discrete Time Wavelet Transform
DTWT
n x(n)ik (n) x(n) ⫽ i k yi (k)ik (n) n ik (n)jl (n) ⫽ ␦kl ␦ij
Inverse DTWT
ik (n) ⫽ ik (n) n ik (n)jl (n) ⫽ ␦kl ␦ij
Orthonormal expansion
Biorthogonal expansion
ik (n) ⫽ fˆi (2i⫹1 k ⫺ n) ( J ⫺1)k (n) ⫽ fˆJ ⫺1 (2
J ⫺1
i ⫽ 0, 1, . . . , J ⫺ 2
k ⫺ n)
Then from (6.125), (6.126), and (6.127 ) we obtain Table 6.3. Remarks ■
A notable characteristic of the DTWT is that the basis sequences for each level i are the power of two shifts of a corresponding mother sequence: ik (n) ⫽ i0 (n ⫺ 2r k),
r ⫽i⫹1 or r ⫽ J ⫺ 1 ik (n) ⫽ i0 (n ⫺ 2r k), r ⫽ i ⫹ 1 or r ⫽ J ⫺ 1
for i for i for i for i
⫽ J ⫽J ⫽ J ⫽J
⫺1 ⫺1 ⫺1 ⫺1
In the more elegant theory of continuous wavelet transform, all analysis (synthesis) basis functions are produced from a single analysis (synthesis) mother function by dilations (time scaling) and shifts [Meye 93, Daub 90, Vett 95]. ■
The magic number 2, whose powers determine the shifts in the mother basis sequences, results from the successive splitting by two in the tree-structured filter banks, which we have adopted to introduce the DTWT. Filter banks of this type are known as octave-band filter banks. Their characteristic is that the bandwidth of each of the filters in the bank is the same in a logarithmic scale. Sometimes they are also called constant-Q filter banks to stress the fact that the ratio of the filters’ bandwidth to the respective central frequency is constant. Generalizations of DTWT with another integer M in place of 2 can also be defined and used [Stef 93].
Example 6.8 The Haar Transform—The Epilogue We have already seen that the Haar transform is equivalent to a tree-structured analysis filter bank. Let us now look at the synthesis problem. For the 8 ⫻ 8 Haar transform and after a row reshuffling of the corresponding Haar matrix, we have
“08-Ch06-SA272” 18/9/2008 page 383
6.13 Discrete Time Wavelet Transform (DTWT)
⎤ ⎡ ⎤ 2 ⫺2 0 0 0 0 0 0 y0 (0) ⎥ ⎢ ⎢ ⎥ 0 2 ⫺2 0 0 0 0⎥ ⎡ ⎢ 0 ⎢y0 (1)⎥ ⎤ ⎥ x(0) ⎢ ⎢ ⎥ ⎥ ⎢ 0 ⎢y0 (2)⎥ 0 0 0 2 ⫺2 0 0 ⎥⎢ ⎥ ⎢ ⎢ ⎥ ⎢x(1)⎥ ⎢y (3)⎥ 2 ⫺2⎥ 1 ⎢ ⎥⎢ ⎥ ⎢√0 √0 √0 √0 0 0 ⎢ 0 ⎥ ⎥⎢ . ⎥ ⎢ ⎥⫽ √ ⎢ ⎢ 2 2 ⫺ 2 ⫺ 2 0 0 ⎢y1 (0)⎥ 0 0⎥ ⎢ .. ⎥ 8 ⎢ ⎢ ⎥ ⎦ √ √ √ √ ⎥⎣ ⎢ 0 ⎢y1 (1)⎥ 0 0 0 2 2 ⫺ 2 ⫺ 2⎥ ⎥ x(7) ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ 1 1 1 ⫺1 ⫺1 ⫺1 ⫺1⎦ ⎣ 1 ⎣y2 (0)⎦ 1 1 1 1 1 1 1 1 y3 (0) ⎡
or ˆ y ⫽ Hx Thus the 8 ⫻ 8 Haar transform gives four coefficients at the finest resolution level 0, two at level 1 and one for each of the coarsest resolution levels 2 and 3. We will now design the corresponding synthesis bank to obtain x(n) from these coefficients. The impulse responses of the Haar analysis filters are ⎧ ⎨ √1 n ⫽ 0 or n ⫽ ⫺1 2 h1 (n) ⫽ ⎩ 0 otherwise ⎧ √1 ⎪ n⫽0 ⎪ 2 ⎪ ⎨ h0 (n) ⫽ ⫺ √1 n ⫽ ⫺1 2 ⎪ ⎪ ⎪ ⎩ 0 otherwise It is readily seen that
hi (2k ⫺ n)hj (2l ⫺ n) ⫽ ␦ij ␦kl ,
i, j ⫽ 0, 1
n
That is, the Haar filter bank is paraunitary . Thus, the synthesis filters can be defined as gi (n) ⫽ hi (⫺n),
i ⫽ 0, 1
Hence 1 G0 (z) ⫽ √ (1 ⫺ z ⫺1 ) 2 1 G1 (z) ⫽ √ (1 ⫹ z ⫺1 ) 2 From the equivalent structure of the synthesis bank of Figure 6.19 we have 1 1 ⫹ z ⫺1 ⫺ z ⫺2 ⫺ z ⫺3 F1 (z) ⫽ G1 (z)G0 z 2 ⫽ 2 and the respective impulse response is
f1 (n) ⫽
⎧ ⎪ ⎪ ⎪ ⎨
1 2
n ⫽ 0, 1
⫺1 ⎪ 2
n ⫽ 2, 3
⎪ ⎪ ⎩ 0
otherwise
383
“08-Ch06-SA272” 18/9/2008 page 384
384
CHAPTER 6 Feature Generation I
Following similar arguments, we have ⎧ 1 √ ⎪ ⎪ 8 ⎪ ⎨ 1 f2 (n) ⫽ ⫺ √ 8 ⎪ ⎪ ⎪ ⎩ 0 ⎧ ⎨ √1 8 f3 (n) ⫽ ⎩ 0
n ⫽ 0, 1, 2, 3 n ⫽ 4, 5, 6, 7 otherwise n ⫽ 0, 1, . . . , 7 otherwise
If we now insert these values in (6.125) and collect the values of x(n) together, we get ⎡
2 0 0 ⎢ ⫺2 0 0 ⎢ ⎤ ⎡ ⎢ ⎢ 0 x(0) 2 0 ⎢ ⎢x(1)⎥ ⎢ ⎥ ⎢ 0 ⫺2 0 ⎢ . ⎥ ⫽ √1 ⎢ ⎢ ⎢ . ⎥ 0 0 2 8⎢ ⎣ . ⎦ ⎢ ⎢ 0 0 ⫺2 x(7) ⎢ ⎢ 0 0 ⎣ 0 0 0 0
√ 0 2 0 √ 0 2 0 √ 0 ⫺ 2 0 √ 0 ⫺ 2 0 √ 0 0 2 √ 0 0 2 √ 2 0 ⫺ 2 √ ⫺2 0 ⫺ 2
1 1 1 1 ⫺1 ⫺1 ⫺1 ⫺1
⎤ 1 ⎡y (0)⎤ ⎥ 0 1⎥ ⎢y0 (1)⎥ ⎥⎢ ⎥ ⎢ ⎥ 1⎥ ⎥ ⎢y0 (2)⎥ ⎥ ⎢ 1⎥ ⎢y0 (3)⎥ ⎥ ⎥⎢ ⎥ 1⎥ ⎢y1 (0)⎥ ⎥⎢ ⎥ ⎢ ⎥ 1⎥ ⎥ ⎢y1 (1)⎥ ⎥ ⎣y2 (0)⎦ 1⎦ y3 (0) 1
or x ⫽ Hˆ T y That is, we reobtain the inverse (within a permutation) Haar transform. Hence, we can now state that the Haar transform and its inverse form a DTWT pair, using the orthogonal Haar sequences as the basis for the wavelet expansion.
6.14 THE MULTIRESOLUTION INTERPRETATION The goal of this section is to highlight, without resorting to mathematical details, an important aspect of the wavelet transform that accounts for its success as a tool in pattern recognition as well as in numerous other applications. Let us assume for simplicity that the two filters in the analysis–synthesis bank of a paraunitary filter bank are ideal low/high pass. Figure 6.20 shows the magnitude responses of the respective filters in the equivalent of the tree-structured octave band filter bank of Figure 6.19. The width of the frequency response (bandwidth) is halved for each level of the tree (Figure 6.20d). That is, the “detail” resolution (high-pass) filters have a wide bandwidth, and the “coarse” resolution (low-pass) filters are of narrow bandwidth. Filters F3 and F2 , the two coarser resolution ones, are of the same bandwidth. These observations are true for any octave band filter bank of any
“08-Ch06-SA272” 18/9/2008 page 385
6.14 The Multiresolution Interpretation
G0(z)
(a) 1
8
8
4
8
4
F3(z) F2(z)
8
4
8
2
2
F1(z)
2
8
1
4
2
2
2
G1(z2)
1
G0(z4)
1
(d) 1
2
G0(z2)
(b) 1
(c)
4
G1(z)
1
4
G1(z4)
8
4
F0(z)
FIGURE 6.20 Filter bandwidths in an octave band filter bank.
number of levels J . That is, the width of Fi (z) is half of the width of Fi⫺1 (z), and the widths of FJ ⫺2 and FJ ⫺1 are equal. This multiresolution viewpoint of the DTWT is a source of its power as a tool, and it is worth spending some time on it. It is known (uncertainty principle) that filters with narrow bandwidth have long impulse responses and that filters with wide bandwidth have short impulse responses. Let us take for example the Haar filter bank. The filter impulse response at level zero, h0 (n) ⫽ fˆ0 (n), is ( √1 , ⫺ √1 ) and at level three [i.e., corresponding to 2 2 Eq. (6.118)], fˆ3 (n), is √1 (1, 1, 1, 1, 1, 1, 1, 1). Since the output of each filter of the 8 analysis bank,that is,the DTWT coefficients,is the convolution of the input sequence with the respective impulse response,the filter tends to spread out the input activity. For example, a single impulse in the input of the Fˆ 3 (z) filter produces a sequence of eight samples at the output. Thus, if our goal is to identify sudden (short in time) changes in an input signal, it is apparent that one should use filters of short impulse response to be able to obtain good time locality. Otherwise, the sudden activity will spread in time. Hence, for sudden changes in time (rich in high-frequency components), one needs a “detailed”analysis filter, that is, a short impulse response. For slowly time-varying activities,rich in low frequencies,one needs a“less detailed”
385
“08-Ch06-SA272” 18/9/2008 page 386
386
CHAPTER 6 Feature Generation I
analysis filter, so as to be able to “see” the longer scale variations. Thus, detail is not of interest here, and long impulse responses are required. In other words, the resolution should match the scale of the activity under investigation. The wavelet transform provides the means of analyzing the input signal into a number of different resolution levels in a hierarchical fashion. This is also known as multiresolution analysis. Thus, signal components corresponding to different physical activities can be best represented at different resolution levels: short high-frequency activities at the finer resolution and long low-frequency activities at coarser resolution levels. It turns out that this coarse-to-fine analysis strategy is appropriate for a number of pattern recognition tasks. On the synthesis part, the signal can be reconstructed from its multiresolution components. See, for example, Figure 6.19. The sequence x(n) is synthesized first by its coarser component x3 (n), and then higher frequency (detailed) components are added, resulting in a successively finer approximation. When the component of the finest detail, x0 (n), is added, the original signal is obtained. This philosophy is at the heart of a number of signal compression schemes.
Remarks ■
The analysis of a signal in a number of components via a filter bank is not new and goes back to the work of Gabor in the 1940s. It is directly related to the short-time Fourier transform defined as [Gabo 46, Vett 95] Xs (, m) ⫽
⬁
x(n)(n ⫺ m) exp(⫺jn)
(6.128)
n⫽⫺⬁
where (n) is a window sequence, whose center is successively moved to the different points m. Thus, each time, the part of the sequence x(n) around m (depending on the window’s effective width) is selected and Fourier transformed. It can be shown that this is equivalent to filtering the signal x(n) by a bank of filters, each centered at a different frequency but all of them having the same bandwidth (Problem 6.20). This is its drawback, because low- and high-frequency signal components are “looked” at through the same window in time, resulting in poor overall localization of the events. What is really needed is a long window to analyze slowly time-varying low-frequency components and a narrow window to detect high-frequency short-time activities. As we saw, this is offered by a tree-structured octave-band filter bank, associated with the DTWT. ■
All we have said about wavelet transforms and multiresolution analysis is just a glimpse of the whole story, a story that is really worth further effort; see, for example, [Daub 90].
“08-Ch06-SA272” 18/9/2008 page 387
6.15 Wavelet Packets
6.15 WAVELET PACKETS The DTWT has been introduced via an octave-band filter bank, and the wavelet coefficients result at the outputs of the bank, when its input is fed with the signal of interest. The octave-band filter bank is constructed by successively splitting by two the lowest frequency band (leaf) of the tree-structured bank (Figure 6.13). However, in many cases most of the activity is not in the low-frequency band but in the middle or high-frequency parts of the spectrum. In such cases, it may be useful to be able to allocate finer frequency bandwidths in the bands where the activity occurs. As we will see later on in the chapter, this can boost the discriminatory power of our system from a classification point of view, which is always our main interest. Figure 6.21a shows an example of a tree-structured filter bank but with the finer frequency splitting occurring at a midfrequency band. Figure 6.21b shows the resulting bandwidths for each of the (ideal) filters in the bank ( f -axis) and the respective window length of the impulse responses in the time domain (n-axis). In other words, filters 2 and 3 have half the bandwidth and twice the impulse response of 4. Furthermore, they have one fourth of the bandwidth and a four times longer impulse response than that of 1. For comparison, Figure 6.22 shows
H0
2
H0 H1
H0
2
1 2
H1
2
3
2
2 H1
4
2
(a) f
1
1
1
1
2 3 4
4 n (b)
FIGURE 6.21 (a) Wavelet packet tree structure and (b) the corresponding frequency versus time resolution.
387
“08-Ch06-SA272” 18/9/2008 page 388
388
CHAPTER 6 Feature Generation I
f
f 1 1
1
1
1 2
2
2
3
3 4
4 n (a)
n (b)
FIGURE 6.22 Frequency versus time resolution for (a) octave band and (b) equal bandwidth filter banks.
the frequency–time resolution plots for an octave-band filter bank (a) and for a bank with equal bandwidths (b), associated with the DTWT and the short-time Fourier transform, respectively. Having freed ourselves from the octave-band tree structure, filter banks can be constructed by various tree growth strategies, with that of Figure 6.21 being just one possibility. As was the case with the octave-band philosophy, these arbitrary tree structures also lead to a set of basis sequences for discrete signal expansions [Coif 92] called wavelet packets. Following arguments similar to those in Section 6.13 and using filters with the perfect reconstruction property, the basis sequences for the wavelet packets result from the respective impulse responses of the synthesis bank, after the appropriate, for each level, power of two time shifts.
6.16 A LOOK AT TWO-DIMENSIONAL GENERALIZATIONS All the concepts discussed so far can be carried over in the two-dimensional case. No doubt, the task is even more challenging now. How can one define subsampling here? The straightforward way is via the “separable” philosophy. That is, we first transform (filter) the columns of the two-dimensional sequence and then the resulting rows. This leads to the subsampling shown in Figure 6.23. In other words, we leave out every other row and every other column (for subsampling by 2). Figure 6.24 shows the filter bank structure that complies with this philosophy. The image sequence I (m, n) appears in the filters of stage 1 column after column, and the respective outputs are subsampled by 2. The resulting subsampled images are in turn filtered at stage two, but now they are fed into the filters row after row. Assuming H0 to be the (ideal) high-pass and H1 the low-pass filter, the four frequency bands that are formed by the previous procedure are illustrated in Figure 6.25a. The area H1 H1 corresponds to low-pass columns and rows, H1 H0 to low-pass columns and high-pass rows, and so forth. Figure 6.25b shows the
“08-Ch06-SA272” 18/9/2008 page 389
6.16 A Look at Two-Dimensional Generalizations
FIGURE 6.23 Separable subsampling by 2 for images.
H0
H1
H0
2
H1
2
H0
2
H1
2
2
2
FIGURE 6.24 Basic element for a two-dimensional filter bank, leading to separable subsampling by 2.
resulting segmentation of the frequency domain when the low-pass area H1 H1 is successively split by repeating the procedure. Example 6.9 Figure 6.26 shows a 64 ⫻ 64 image of a triangle. The three “line” images are the 32 ⫻ 32 images that result when passing the triangle image through the structure of Figure 6.24. In the columnwise filtering of the first stage, the vertical line goes through the low-pass H1 filter (no variation across it) and the horizontal and diagonal lines go through the high-pass H0 . This is because in a columnwise filtering, these appear as impulses in each column, and thus are rich in high frequencies. In the row scanning of the second stage, it will be the horizontal line that will go through the low-pass filter. Similar reasoning explains the position of the various parts of the triangle in the different bands. Although this is obviously a very simplified example, it is quite instructive. It demonstrates how the original image can be obtained from its multiresolution components and also how different characteristics (directional in this case) of the whole may be isolated at different bands.
389
“08-Ch06-SA272” 18/9/2008 page 390
390
CHAPTER 6 Feature Generation I
fv
fv H0 H1
H0 H0
H1 H1
H1 H0
fh
(a)
fh
(b)
FIGURE 6.25 (a) Frequency domain division corresponding to the filter bank of Figure 6.24 and (b) the result of a successive division of the low-pass part of the spectrum.
H0 H1
H0 H0
H1 H0
FIGURE 6.26 A triangle image and its filtered versions through the filter bank of Figure 6.24.
6.17 APPLICATIONS All the transforms we have studied in this chapter are good candidates for feature generation, and they have been used extensively in various pattern recognition tasks. However, the wavelet transform offers an extra advantage, which in some cases can be beneficially exploited. Its multiresolution properties conform to the way perception is achieved by humans, through their hearing and visual systems. The human ear exhibits decreasing resolution at higher frequencies, in a
“08-Ch06-SA272” 18/9/2008 page 391
6.17 Applications
way that is uniform on a logarithmic scale (octave bands) [Flan 72]. Experiments in psychophysics and physiology show that the human visual cortex perceives by decomposing the stimuli in a number of frequency bands [Camp 68, Levi 85],which are also dependent on the spatial orientation [Camp 66]. Experimental results in [Nach 75] indicate that these frequency bands have the approximate bandwidth of an octave. The similarities between the mechanism with which human perception systems treat the respective stimuli and the processing techniques that split the signal into various (spatial) frequency bands, in a way similar to the wavelet transform, justify the use of the latter in pattern recognition tasks [Mall 89]. The following examples come from two of the most important areas of interest in today’s pattern recognition applications.
Recognition of Handwritten Characters The development of OCR systems is of particular importance in various application areas. One of the most challenging among them is the recognition of handwritten characters. Figure 6.27 shows the character“3”as well as its boundary contour after the application of a contour tracing algorithm [Pita 92]. The task now becomes one of shape recognition. As discussed in more detail in Chapter 7, the boundary can
FIGURE 6.27 Wavelet coefficients corresponding to the curvature of the boundary of number “3.”
391
“08-Ch06-SA272” 18/9/2008 page 392
392
CHAPTER 6 Feature Generation I
be represented as a closed parametric curve in the complex plane u(n) ⫽ x(n) ⫹ jy(n),
0ⱕnⱕN ⫺1
(6.129)
with N being the number of samples (pixels) found tracing the contour and x(n), y(n) the corresponding coordinates. The first point (x(0), y(0)) of the sequence is considered as the origin. Fourier methods have been used extensively in such classification tasks, by obtaining the DFT of u(n) and keeping a sufficient number,from the total of N ,of Fourier components as features. An alternative way is to extract the features from the wavelet domain. In other words, x(n) and y(n) are independently filtered through a tree-structured filter bank of appropriate resolution depth,and the resulting wavelet coefficients are used as features. The low-frequency components account for the basic shape of the character and are less sensitive to varying writing styles. The high-frequency components account for the details and are more sensitive to the specific handwriting style. In [Wuns 95], a comparative study was carried out using the same number of DTWT coefficients and Fourier-based features, with a neural network classifier. The experiments showed that classification based on wavelet coefficients resulted in reduced error rates. Furthermore, it was pointed out that Fourier-based features exhibited larger withinclass variance and weaker between-class separation than the wavelet-based ones. A major disadvantage associated with the wavelet coefficients is that they are not shift invariant. That is, if we rotate/translate a character, the resulting coefficients will not be the same. This is a consequence of the subsampling process [Mall 89] (Problem 6.21). In other words, if x⬘(n) ⫽ x(n ⫺ n0 )
and y⬘(n), y(n) are the sequences of wavelet coefficients of x⬘(n) and x(n), respectively, then, in general y⬘(n) ⫽ y(n ⫺ n0 )
This has led to research in designing filter banks that seek to overcome this property [Marc 95, Hui 96]. The shift dependence obviously also makes the wavelet coefficients sensitive to the choice of the initial point from which the contour is traced, as already described. As we will see in more detail in Chapter 7, the Fourier coefficients are also dependent on shifts but in a deterministic way. Thus, various normalizing techniques exist that result in shift-invariant feature parameters [Crim 82, Arbt 90]. To overcome the problems associated with the choice of the initial point within the contour, in order to minimize its effects on the wavelet coefficients, a number of techniques have been suggested and used in practice. A simple method is to select a specific point resulting during the scanning process, for example, the first pixel when scanning the character from left to right. Other more “intelligent” ways have also been suggested [Wuns 95, Chua 96, Pun 03]. An alternative approach to that of Eq. (6.129) is to describe a contour in terms of the arc length between a given point and the origin, within the contour. As arc
“08-Ch06-SA272” 18/9/2008 page 393
6.17 Applications
length t at a given point, we define the number of consecutive pixels between the given point and the point considered as the origin. The contour description can now be achieved via a one-parameter real-valued function,the arc tangent angle (t) or the corresponding curvature (t), defined as dy(t) (t) ⫽ tan⫺1 dx(t) (t) ⫽
d(t) dt
where x(t), y(t) are the coordinates of the respective point as a function of length t from the origin and dt ⫽ dy2 ⫹ dx 2 . A further discussion of this is provided in Chapter 7. [Kapo 96] suggests wavelet transforming the curvature of the contour and using the corresponding wavelet coefficients as features. Figure 6.27 shows the coefficients resulting from the wavelet analysis of the curvature (bottom left) of the boundary contour of number 3. The wavelet basis used for the analysis was Daubechies’s biorthogonal pairs (3,9) [Vett 95]. Six successive resolution levels are shown, the finest being on top and the coarsest at the bottom. Extensive experimentation with a number of different characters shows that the use of wavelet coefficients from more than six resolution levels adds no further discriminatory information to the system. Thus, each of the resulting feature vectors has 32 components. A different philosophy is followed in [Geze 00, Geze 02], in the context of an OCR system for the Greek Orthodox Byzantine music notation. The wavelet transform is applied to the vector combining the four projections (horizontal, vertical,left-diagonal,right-diagonal) of the characters. It turns out that such an approach leads to an efficient coding of the directional properties of the characters.
Texture Classification Texture characterization in image analysis tasks is another area where the wavelet transform, as well as the other transforms discussed in this chapter, has been heavily utilized. The basic approach is similar to that in the OCR example. However, because texture is a property of the image region and not of its boundary, the two-dimensional variants of the transforms are used. The two-dimensional wavelet transform offers the advantages of spatial frequency and orientation selectivity, provided the appropriate bases are chosen. The information compaction properties result from the fact that most of the energy activity is concentrated in certain resolution levels, as was the case with the OCR example considered earlier. The wavelet coefficients of these levels are then selected as features to form the featurevectors. Sometimes a function of the features is employed, such as the energy, i yi2 , or the entropy, i yi2 log yi2 , where yi are the respective wavelet coefficients at each resolution level [Lain 93]. In many cases, the underlying image texture exhibits a great deal of activity in middle or high-frequency bands.
393
“08-Ch06-SA272” 18/9/2008 page 394
394
CHAPTER 6 Feature Generation I
(a)
(b)
FIGURE 6.28 An example of (a) a textured image and (b) its corresponding wavelet packet transform.
Figure 6.28a is an example of a textured image taken from [Brod 66]. In such cases, one must pay attention to the bands of high energy, instead of looking with fine frequency bandwidths at low-frequency bands of low energy. This leads to the adoption of wavelet packets, discussed in Section 6.15. There, we saw that there are a number of choices for splitting the frequency bands, leading to different wavelet packets. In [Chan 93] a dynamic procedure is suggested, depending on the particular texture image. A threshold C is selected prior to the analysis. If the output energy in a band is less than C, no splitting of the band is carried out. If it is higher than C, the band is split further. Splitting terminates if subimages, after filtering and subsampling, become small, for example, 16 ⫻ 16. It is then suggested to use as features the energy values at the J (a preselected number) most dominant (energywise) bands. In [Mojs 00] the effects of the properties of the analysis band on texture characterization are considered. It is demonstrated that the choice of the analysis filters may have a significant influence on the classification performance. In [Unse 95, Lain 96] another variation of the DTWT is adopted, called the discrete wavelet frame. The difference from DTWT is that the filter outputs in the bank are not subsampled. Although this leads to a redundant representation, it results in a texture description tolerant to translations. A procedure similar in concept to the DTWT one is to employ a bank of twodimensional Gabor filters to perform the splitting of the image into a number of frequency bands. The impulse response (point spread function, Appendix D) of the complex two-dimensional Gabor filter is given as the product of a Gaussian low-pass filter with a complex exponential, that is [Bovi 91], h(x, y) ⫽ g⬘(x, y) exp( j(x x ⫹ y y))
where g⬘(x, y) ⫽
1 g 2
1 x 2 ⫹ y2 x⬘ y⬘ , , g(x, y) ⫽ exp ⫺ 2 2
(6.130)
(6.131)
“08-Ch06-SA272” 18/9/2008 page 395
6.17 Applications
and x⬘ ⫽ x cos ⫹ y sin y⬘ ⫽ ⫺x sin ⫹ y cos
(6.132)
That is, g⬘(x, y) is a version of the Gaussian g(x, y) that is spatially scaled and rotated by . The parameter is the spatial scaling, which controls the width of the filter impulse response, and defines the aspect ratio of the filter, which determines the directionality of the filter, which is no longer circularly symmetric. The orientation angle is usually chosen to be equal to the direction of the filter’s center circular frequency ⫽
% 2x ⫹ 2y
(6.133)
That is, ⫽ tan⫺1
y x
(6.134)
By varying the free parameters , , , and , filters of arbitrary orientation and bandwidth characteristics are obtained (Problem 6.22). Figure 6.29a shows the magnitude of the complex Gabor filter for ⫽ 1.0, ⫽ 0.3. Figure 6.29b shows the magnitude of the corresponding spatial frequency response. The choice of the Gabor filters is justified by the fact that these filters offer the optimal tradeoff between spectral bandwidth and spatial localization. In Section 6.14 we saw that the shorter the filter’s impulse response (spatial localization), the wider its frequency bandwidth and vice versa,according to the uncertainty principle [Papo 91],
|H(u, v)|
|h(x, y)|
1
0.6
0 2
0 1
x ⫺2
y 2 (a)
⫺2
u ⫺1 1 (b)
v
⫺1
FIGURE 6.29 Plot of (a) the magnitude of point spread function of a Gabor filter and (b) the magnitude of its Fourier transform.
395
“08-Ch06-SA272” 18/9/2008 page 396
396
CHAPTER 6 Feature Generation I
that is, 1 2 1 ⌬ y⌬y ⱖ 2
⌬ x⌬x ⱖ
(6.135)
In [Daug 85] it has been shown that the two-dimensional Gabor filters attain the minimum uncertainty bound. For digital images, a sampling of the above functions has to be performed, which introduces aliasing errors regardless of the sampling interval. This happens because Gabor filters are not bandlimited, but have a Gaussian-shaped frequency response (Problem 6.23). The topic is treated in [Bovi 90]. Gabor filter banks for analyzing the image in a number of bands, in the context of texture classification, have been successfully employed in a number of cases, [Jain 91, Turn 86, Bovi 90, Hale 95, Hale 99, Weld 96]. A set of Gabor filters centered at different frequencies and having different orientations are used to cover the frequency range of interest, using various frequency and orientation bandwidths. Images are then filtered through this set of filters, and the features are generated from the resulting output samples. For example, the output energies of the Gabor filters may be chosen to be the respective features. Thus, using this strategy, the generated features encode classification information related to the spatial frequency as well as the orientation activity of the various textures. In order to grasp most of the textural information, techniques have been proposed to place the centers of the Gabor filters in the most “important” image frequencies [Pich 96]. In [Chan 93, Grig 02] a comparative study of various transform-based features is provided in the context of texture classification.
6.18 PROBLEMS 6.1 Show the equivalence (a) between (6.5) and (6.6) and (b) between (6.7) and (6.8). 6.2 Consider the separable transform Y ⫽ UXV T . Then show that if Y , X are turned into the row-ordered vectors y, x, respectively, then y ⫽ (U ⊗ V )x, where ⊗ denotes the Kronecker product of two matrices. 6.3 Let e i , i ⫽ 0, 1, . . . , N ⫺ 1, be any orthonormal basis in the N -dimensional space. Show that the MSE between an N -dimensional vector and an mdimensional projection of it is minimized if (a) the basis consists of the eigenvectors of Rx and (b) the m-dimensional subspace is the one spanned by the eigenvectors corresponding to the m largest eigenvalues. Furthermore, the projection onto the latter subspace is the one that maximizes the sum of the variances of its components. Hint: Minimize the mean square error E[ ⑀ 2 ] subject to the constraint e Ti e i ⫽ 1.
“08-Ch06-SA272” 18/9/2008 page 397
6.18 Problems
6.4 Consider an N -dimensional random vector x, which is approximated by xˆ ⫽
m⫺1
yi e i ⫹
N ⫺1
i⫽0
ci e i
i⫽m
where ci are nonrandom constants and e i , i ⫽ 0, 1, 2, . . . , N ⫺ 1, constitute an orthonormal basis. Show that the minimum mean square error E x ⫺ x ˆ 2 is achieved if (a) ci ⫽ E[ yi ], i ⫽ m, . . . , N ⫺ 1; (b) the orthonormal basis consists of the eigenvectors of ⌺x ; and (c) e i , i ⫽ m, . . . , N ⫺ 1, correspond to the N ⫺ m smallest eigenvalues. 6.5 If X is a rank r matrix, show that the two square matrices XX H and X H X have the same nonzero eigenvalues. 6.6 (a) Show Eq. (6.39). (b) Show that the expansion in the right-hand side of (6.41) is the projection of x i on the subspace spanned by the first k columns of U . 6.7 Given the matrix ⎤ 2 ⎥ 1⎦ 3
⎡
1 ⎢ ⎣2 1
compute its SVD representation. 6.8 Show the orthogonality of the DFT matrix W and also identity (6.94). 6.9 Given the image array ⎡
1 ⎢ ⎣0 2
2 1 1
⎤ 1 ⎥ 1⎦ 2
compute its two-dimensional DFT transform. 6.10 For one of the images available at the Web site of the book, write a program to compute its DFT transform. Use a routine to implement the fast Fourier transform and plot the magnitude of the resulting Fourier transform. 6.11 Show the orthogonality of the DCT transform. 6.12 Compute the DCT of the image array of Problem 6.9. 6.13 Develop a program to compute the DCT for one of the images available from the Web site of the book. 6.14 Show the orthogonality of the Hadamard transform. 6.15 Compute the Hadamard transform for a 2 ⫻ 2 submatrix of the matrix of Problem 6.9.
397
“08-Ch06-SA272” 18/9/2008 page 398
398
CHAPTER 6 Feature Generation I
6.16 Show the orthogonality of the Haar transform. 6.17 Show the two Noble identities of Figures 6.12a and 6.18. 6.18 Show the equivalence between the tree structure of Figure 6.11 and that of Figure 6.12b. 6.19 Consider the perfect reconstruction two-band Haar bank. Show that a. If we make the analysis filters causal by delaying each of them by one sample, then the reconstructed sequence x(n) ˆ is delayed by one sample, that is, x(n) ˆ ⫽ x(n ⫺ 1). In the more general case, if both the analysis filters have to be delayed by L, then the output of the band is also delayed by L. b. If this is repeated with the more general N -band case,show that the output is delayed as x(n) ˆ ⫽ x(n ⫺ (2N ⫺1 ⫺ 1)L) and a delay must be inserted in each band to safeguard perfect reconstruction. Figure 6.30 shows the three-band case. In the general case, the delay element in each band is N ⫺i⫺1 ⫺1)L z ⫺(2 , i ⫽ 0, 1, . . . , N ⫺ 1. 6.20 Show that the short-time Fourier transform defined in (6.128) is equal to ⬁
Xs (, m) ⫽ exp(⫺jm)
x(n)(n ⫺ m) exp( j(m ⫺ n))
n⫽⫺⬁
Verify that this is equivalent to filtering the sequence x(n) with different filters of the same bandwidth but centered at different frequencies. 6.21 Show that the process of filtering and then subsampling is equivalent to the action of a linear but time-varying system. In fact, it is a periodically timevarying linear system. The same is true for the process of upsampling followed by linear filtering.
ANALYSIS
H0
2
H1
2
x(n)
SYNTHESIS ⫺3L Z
H0
⫺L Z
2 H0
H1
y0(n) y1(n) 2
y2(n)
2
2 H1
2 y3(n)
2
2
G0
2
G1
G0
2
G0
2
G1
G1 ^ x(n)
FIGURE 6.30 A three-band perfect reconstruction filter bank with causal analysis and synthesis filters.
“08-Ch06-SA272” 18/9/2008 page 399
MATLAB Programs and Exercises
6.22 Show that the frequency (octave) and orientation (radians) half-peak bandwidths for Gabor filters are given by Bf , B , respectively, where ⫹
√
2 ln 2 √ ⫺ 2 ln 2 √ 2 ln 2 B ⫽ 2 tan⫺1
Bf ⫽ log2
Compute these for different values of , . 6.23 Show that the Fourier transform of the two-dimensional Gabor filter response h(x, y) is given by 2 H(u, v) ⫽ exp ⫺ {(u⬘ ⫺ x ⬘)2 2 ⫹ (v⬘ ⫺ y ⬘)2 } 2
where u⬘ ⫽ u cos ⫹ v sin v⬘ ⫽ ⫺u sin ⫹ v cos
and x ⬘, y ⬘ the corresponding versions of x , y rotated by . Draw its magnitude versus frequency (u, v) for different values of and .
MATLAB PROGRAMS AND EXERCISES Computer Programs 6.1 Generation of points around an (l ⫺ 1)-dimensional hyperplane: Write a MATLAB function named generate_hyper that generates randomly ldimensional points x i ⫽ [x1 (i), x2 (i), . . . , xl (i)]T around an (l⫺1)-dimensional hyperplane H: w T x ⫹ w0 ⫽ 0, where w ⫽ [w1 , w2 , . . . , wl ]T . More specifically, the function takes as inputs: (a) the parameter (column) vector w for H (wl ⫽ 0), (b) the offset w0 for H, (c) a positive parameter a that defines the range [⫺a, a], where each one of the first (l ⫺ 1) coordinates of the points is uniformly distributed, (d) the positive parameter e that defines the range [⫺e, e] of a uniformly distributed noise source, which is added to the term (⫺w0 ⫺ l⫺1 i⫽0 wi xi )/wl to produce the lth coordinate, (e) the number N of points to be generated, and (f) the seed sed for the rand MATLAB function. It returns an l ⫻ N dimensional matrix, X, whose columns contain the generated data points. In addition, the function plots the data points for l ⫽ 2, 3.
Solution function X=generate_hyper(w,w0,a,e,N,sed) l=length(w);
399
“08-Ch06-SA272” 18/9/2008 page 400
400
CHAPTER 6 Feature Generation I
t=(rand(l-1,N)-.5)*2*a; t_last=-(w(1:l-1)/w(l))'*t + 2*e*(rand(1,N)-.5)-(w0/w(l)); X=[t; t_last]; %Plots for the 2d and 3d case if(l==2) figure(1), plot(X(1,:),X(2,:),'.b') elseif(l==3) figure(1), plot3(X(1,:),X(2,:),X(3,:),'.b') end figure(1), axis equal
6.2 PCA analysis: Write MATLAB commands to compute the principal components of the covariance matrix of an l ⫻ N dimensional data matrix X as well as the corresponding variances.
Solution Just write [pc,variances]=pcacov(cov(X'))
In the above command (a) cov(X ⬘ ) computes the covariance matrix of X ⬘ and (b) pcacov returns the principal components (eigenvectors of the covariance matrix) in the columns of pc as well as the corresponding variances (eigenvalues) in the column vector variances (note that pcacov assumes the the data vectors lie in the rows of corresponding data matrix). 6.3 Distance matrix computation: Write a MATLAB function named compute_ distances that takes as input an l ⫻ N matrix X and returns the N ⫻ N dimensional matrix distX whose (i, j) entry contains the squared Euclidean distance between the ith and jth column vectors of X.
Solution function distX=compute_distances(X) [l,N]=size(X); distX=zeros(N); for i=1:N for j=i+1:N distX(i,j)=(X(:,i)-X(:,j))'*(X(:,i)-X(:,j)); distX(j,i)=distX(i,j); end end
6.4 Singular Value Decomposition: Write MATLAB commands to perform SVD on an l ⫻ N dimensional data matrix X whose columns are the data vectors.
“08-Ch06-SA272” 18/9/2008 page 401
MATLAB Programs and Exercises
Solution Just write [U,S,V]=svd(X)
The above command returns: (a) a diagonal matrix S of the same size with X, containing in its diagonal the singular values of X in decreasing order and (b) the unitary matrices U , V such that U ∗ S ∗ V ⬘ ⫽ X. 6.5 Dimensionality reduction using SVD: Write a MATLAB function named SVD_eval that evaluates the performance of the SVD method when applied on a data matrix X. More specifically, this function takes as inputs: (a) an l ⫻ N dimensional matrix X, whose columns contain the data vectors, (b) the dimensionality k(⬍ l) of the reduced space (k-dimensional hyperplane), h, generated by the k column vectors of the matrix Ur , which correspond to the k largest singular values of X. It returns: (a) an l ⫻ 1 column vector containing the singular values of X, (b) the corresponding Ur matrix, denoted by Ur, (c) the 1 ⫻ l dimensional parameter vector w of h, (d) the offset w0 of h, (e) the distance matrix distX for X, and (f) the distance matrix distX_proj of the projections of the vectors of X on h.
Solution function [s,Ur,w,w0,distX,distX_proj]=SVD_eval(X,k) [l,N]=size(X); [Ur,S,Vr] = svd(X); s=diag(S(1:l,1:l)); a=S(1:k,1:k)*Vr(:,1:k)'; X_proj=Ur(:,1:k)*a; % Deterimnation of the estimated by the SVD hypeprlane P=X_proj(:,1:l)'; w=[]; for i=1:l w=[w (-1)^(i+1)*det([P(:,1:i-1) P(:,i+1:l) ones(l,1)])]; end w0=(-1)^(l+2)*det(P(:,1:l)); % Computation of distances distX=compute_distances(X); distX_proj=compute_distances(X_proj);
Computer Experiments 6.1 a. Generate an l ⫻ N dimensional matrix X (l ⫽ 2 and N ⫽ 1000), whose columns are two-dimensional points lying around the line h: x1 ⫹ x2 ⫽ 0
401
“08-Ch06-SA272” 18/9/2008 page 402
402
CHAPTER 6 Feature Generation I
(i.e., w ⫽ [1, 1]T and w0 ⫽ 0), using the generate_hyper function with parameters a ⫽ 10, e ⫽ 1 and sed ⫽ 0. b. Compute the principal components of the covariance of X as well as the corresponding variances (eigenvalues). Compare the direction of the first principal component with the direction vector of h (which is perpendicular to w) and draw your conclusions. 6.2 Repeat 6.1 with e ⫽ 5. 6.3 Repeat 6.1 and 6.2 where now l ⫽ 3 and the line h is replaced by the plane H: x1 ⫺ 5x2 ⫹ 2x3 ⫽ 0. 6.4 a. Generate an l ⫻ N dimensional matrix X (l ⫽ 3 and N ⫽ 1000), whose columns are two-dimensional points lying around the three dimensional hyperplane with w ⫽ [1, 1, 1]T ,w0 ⫽ 0,using the generate_hyper function, with parameters a ⫽ 10, e ⫽ 1 and sed ⫽ 0. b. Use the SVD_eval function to compute (a) the singular values of X,(b) k ⫽ 2 column vectors of the matrix Ur , that correspond to the two largest singular values of X. Also, compare the distances between the points of X and the distances of their corresponding projections on h. c. Repeat (b) for k ⫽ 1. 6.5 Repeat 6.4 with e ⫽ 6. Comment on the results.
REFERENCES [Achl 01] Achlioptas D., McSherry F. “Fast computation of low rank approximations,” Proceedings of the ACM STOC Conference, pp. 611–618, 2001. [Akan 93] Akansu A.N., Hadda R.A. Multiresolution Signal Decomposition, Academic Press, 1992. [Arbt 90] Arbter K., Snyder W.E., Burkhardt H., Hirzinger G. “Application of affine-invariant Fourier descriptors to recognition of 3-D objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12(7), pp. 640–647, 1990. [Atti 92] Attick J.J. “Entropy minimization: A design principle for sensory perception,” International Journal of Neural Systems, Vol. 3, pp. 81–90, 1992. [Barl 89] Barlow H.B. “Unsupervised learning,” Neural Computation, Vol. 1, pp. 295–311, 1989. [Bart 02] Bartlett M.S., Movellan J.R., Sejnowski T.J. “Face recognition by independent component analysis,” IEEE Transactions on Neural Networks, Vol. 13(6), pp. 1450–1464, 2002. [Bell 00] Bell A.J. “Information theory, independent component analysis, and applications,” in Unsupervised Adaptive Filtering, Part I: Blind Source Separation (Haykin S., ed.), pp. 237–264, John Wiley & Sons, 2000.
“08-Ch06-SA272” 18/9/2008 page 403
References
[Bell 97] Bell A.J., Sejnowski T.J. “The independent components of natural scenes are edge filters,” Vision Research, Vol. 37(23), pp. 3327–3338, 1997. [Belk 03] Belikn M., Niyogi P. “Laplacian eigenmaps for dimensionality reduction and data representation,” Neural Computation, Vol. 15(6), pp. 1373–1396, 2003. [Beng 04] Bengio Y., Paiement J.-F., Vincent P., Delalleau O., Le Roux N., Quimet M. “Out of sample extensions for LLE, Isomap, MDS, Eigenmaps and Spectral clustering,” Advances in Neural Information Processing Systems Conference, (Thrun S., Saul L., Schölkopf B., eds.), MIT Press, 2004. [Benn 06] Benetos E., Kotti M., Kotropoulos C. “Applying supervised classifiers based on nonnegative matrix factorization to musical instrument classification,” Proceedings IEEE Intl. Conference on Multimedia and Expo, pp. 2105–2108, Toronto, Canada, 2006. [Berr 95] Berry M., Dumais S., O’Brie G. “Using linear algebra for intelligent information retrieval,” SIAM Review, Vol. 37, pp. 573–595, 1995. [Beyg 06] Beygelzimer A., Kakade S., Langford J. “Cover trees for nearest neighbor,” Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, 2006. [Bovi 91] Bovic A.C. “Analysis of multichannel narrow-band filters for image texture segmentation,” IEEE Transactions on Signal Processing, Vol. 39(9), pp. 2025–2044, 1991. [Bovi 90] Bovic A.C., Clark M., Geisler W.S. “Multichannel texture analysis using localized spatial filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12(1), pp. 55–73, 1990. [Brod 66] Brodatz P. Textures: A Photographic Album for Artists and Designers, Dover, 1966. [Brun 04] Brunet J.-P. Tamayo P., Golub T.R., Mesirov J.P. “Meta-genes and molecular pattern discovery using matrix factorization,” Proceedings of the National Academy of Science, Vol. 101(2), pp. 4164–4169, 2004. [Burg 04] Burges C.J.C. “Geometric methods for feature extraction and dimensional reduction: A guided tour,” Technical Report MSR-TR-2004-55, Microsoft Research, 2004. [Burt 83] Burt P.J., Adelson E.H. “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications, Vol. 31(4), pp. 532–540, 1983. [Cai 05] Cai D., He X. “Orthogonal locally preserving indexing,” Proceedings 28th Annual International Conference on Research and Development in Information Retrieval, 2005. [Cama 03] Camastra F. “Data dimensionality estimation methods: A survey,” Pattern Recognition, Vol. 36, pp. 2945–2954, 2003. [Camp 66] Campell F., Kulikowski J. “Orientation selectivity of the human visual system,” Journal of Physiology, Vol. 197, pp. 437–441, 1966. [Camp 68] Campell F., Robson J. “Application of Fourier analysis to the visibility of gratings,” Journal of Physiology, Vol. 197, pp. 551–566, 1968. [Cao 03] Cao J.J., Chua K.S., Chong W.K., Lee H.P., Gu Q.M. “ A comparison of PCA, KPCA and ICA for dimensionality reduction,” Neurocomputing, Vol. 55, pp. 321–336, 2003. [Cast 03] Casteli V., Thomasian A., Li C.-S. “CSVD: Clustering and singular value decomposition for approximate similarity searches in high-dimensional space,” IEEE Transactions on Knowledge and Data Engineering, Vol. 15(3), pp. 671–685, 2003. [Chan 93] Chang T., Kuo C.C.J. “Texture analysis and classification with tree structured wavelet transform,” IEEE Transactions on Image Processing, Vol. 2(4), pp. 429–442, 1993.
403
“08-Ch06-SA272” 18/9/2008 page 404
404
CHAPTER 6 Feature Generation I
[Chu 04] Chu M., Diele F., Plemmons R., Ragni S. “Optimality, computation and interpretation of the nonnegative matrix factorization,” available at http://www.wfu.edu/˜ plemmons, 2004. [Chua 96] Chuang G.C.H., Kuo C.C.J. “Wavelet descriptor of planar curves: Theory and applications,” IEEE Transactions on Image Processing, Vol. 5(1), pp. 56–71, 1996. [Coif 92] Coifman R.R., Meyer Y., Wickerhauser M.V. “Wavelet analysis and signal processing,” in Wavelets and Their Applications (Ruskai M.B. et al., eds.), pp. 153–178, Jones and Barlett, 1992. [Como 94] Comon P. “Independent component analysis—A new concept?” Signal Processing, Vol. 36, pp. 287–314, 1994. [Corm 01] Cormen T.H., Leiserson C.E., Rivest R.L., Stein C. Introduction to Algorithms, Second Edition, MIT Press and McGraw-Hill, 2001. [Cox 94] Cox T., Cox M. Multidimensional Scaling, Chapmay & Hall, London, 1994. [Crim 82] Crimmins T.R. “A complete set of Fourier descriptors for two dimensional shapes,” IEEE Transactions on Systems, Man Cybernetics, Vol. 12(6), pp. 848–855, 1982. [Daub 90] Daubechies I. Ten Lectures on Wavelets, SIAM, Philadelphia, 1991. [Daug 85] Daugman J.G. “Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two dimensional visual cortical filters,” Journal of Optical Society of America, Vol. 2, pp. 1160–1169, 1985. [Deco 95] Deco G., Obradovic D. “Linear redundancy reduction learning,” Neural Networks, Vol. 8(5), pp. 751–755, 1995. [Deer 90] Deerwester S., Dumais S., Furnas G., Landauer T., Harshman R. “Indexing by latent semantic analysis,” Journal of the Society for Information Science, Vol. 41, pp. 391–407, 1990. [Desi 03] De Silva V., Tenenbaum J.B. “Global versus local methods in nonlinear dimensionality reduction,” in Advances in Neural Information Processing Systems Becker S., Thrun S., Obermayer K. (eds.), Vol. 15, pp. 721–728, MIT Press, 2003. [Diam 96] Diamantaras K.I., Kung S.Y. Principal Component Neural Networks, John Wiley Sons, 1996. [Dono 02] Donoho D.L., Grimes C.E. “When does ISOMAP recover the natural parameterization of families of articulated images?” Technical Report 2002-27, Department of Statistics, Stanford University, 2002. [Dono 04] Donoho D., Stodden V. “When does nonnegative matrix factorization give a correct decomposition into parts?” in Advances in Neural Information Processing Systems (Thrun S., Saul L., Schölkopf B., eds.), MIT Press, 2004. [Doug 00] Douglas S.C., Amari S. “Natural gradient adaptation,” in Unsupervised Adaptive Filtering, Part I: Blind Source Separation (Haykin S., ed.), pp. 13–61, John Wiley & Sons, 2000. [Este 77] Esteban D., Galand C. “Application of quadrature mirror filters to split band voice coding schemes,” Proceedings of the IEEE Conference on Acoustics Speech and Signal Procesing, pp. 191–195, May 1977. [Fiel 94] Field D.J. “What is the goal of sensory coding?” pp. 559–601, 1994.
Neural Computation, Vol. 6,
[Flan 72] Flanagan J.L. Speech Analysis, Synthesis and Perception, Springer-Verlag, New York, 1972.
“08-Ch06-SA272” 18/9/2008 page 405
References
[Fuku 90] Fukunaga K. Introduction to Statistical Pattern Recognition, 2nd ed., Academic Press, 1990. [Gabo 46] Gabor D. “Theory of communications,” Journal of the Institute of Elec. Eng., Vol. 93, pp. 429–457, 1946. [Geze 00] Gezerlis V., Theodoridis, S. “An optical music recognition system for the notation of Orthodox Hellenic Byzantine music,” Proceedings of the International Conference on Pattern Recognition (ICPR), Barcelona, 2000. [Geze 02] Gezerlis V., Theodoridis S. “Optical character recognition of the Orthodox Hellenic Byzantine music,” Pattern Recognition, Vol. 35(4), pp. 895–914, 2002. [Golu 89] Golub G.H., Van Loan C.F. Matrix Computations, Johns Hopkins Press, 1989. [Grig 02] Grigorescu S.E., Petkov N., Kruizinga P. “Comparison of texture features based on Gabor filters,” IEEE Transactions on Image Processing, Vol. 11(10), pp. 1160–1167, 2002. [Hale 95] Haley G., Manjunath B.S. “Rotation-invariant texture classification using modified Gabor filters,” IEEE International Conference on Image Processing, pp. 262–265, 1995. [Hale 99] Haley G., Manjunath B.S. “Rotation-invariant texture classification using complete space frequency model,” IEEE Transactions on Image Processing, Vol. 8(2), pp. 255–269, 1999. [Ham 04] Ham J., Lee D.D., Mika S., Scho¨ lkopf B. “A kernel view of the dimensionality reduction of manifolds,” Proceedings of the 21st International Conference on Machine Learning, pp. 369–376, Banff, Canada, 2004. [Hayk 99] Haykin S. Neural Networks—A Comprehensive Foundation, 2nd ed., Prentice Hall, 1999. [Hayk 00] Haykin S. (ed.) Unsupervised Adaptive Filtering, Part I: Blind Source Separation, John Wiley & Sons, 2000. [He 03] He X., Niyogi P. “Locally preserving projections,” Proceedings Advances in Neural Information Processing Systems Conference, 2003. [Hote 33] Hotelling H. “Analysis of a complex of statistical variables into principal components,” Journal of Educational Psychology, Vol. 24, pp. 417–441, 1933. [Hoy 00] Hoyer P.O., Hyvärien A. “Independent component analysis applied to feature extraction from color and stereo images,” Network: Comput. Neural Systems, Vol. 11(3), pp. 191–210, 2000. [Hube 85] Huber P.J. “Projection pursuit,”The Annals of Statistics,Vol. 13(2), pp. 435–475, 1985. [Hui 96] Hui Y., Kok C.W., Nguyen T.Q. “Theory and design of shift invariant filter banks,” Proceeding of IEEE TFTS’96, June 1996. [Hyva 01] Hyvärien A., Karhunen J., Oja E. Independent Component Analysis, Wiley Interscience, 2001. [Jack 91] Jackson J.E A User’s Guide to Principle Components, John Wiley & Sons, 1991. [Jain 89] Jain A.K. Fundamentals of Digital Image Processing, Prentice Hall, 1989. [Jain 91] Jain A.K., Farrokhnia F. “Unsupervised texture segmentation using Gabor filters,” Pattern Recognition, Vol. 24(12), pp. 1167–1186, 1991. [Jang 99] Jang G.J., Yun S.J., Hwan Y. “Feature vector transformation using independent component analysis and its application to speaker identification,” Proceedings of Eurospeech, pp. 767–770, Hungary, 1999.
405
“08-Ch06-SA272” 18/9/2008 page 406
406
CHAPTER 6 Feature Generation I
[Joll 86] Jollife I.T. Principal Component Analysis, Springer-Verlag, 1986. [Jone 87] Jones M.C., Sibson R. “What is projection pursuit?” Journal of the Royal Statistical Society, Ser. A, Vol. 150, pp. 1–36, 1987. [ Jutt 91] Jutten C., Herault J. “Blind separation of sources, Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Processing, Vol. 24, pp. 1–10, 1991. [Kann 04] Kannan R., Vempala S., Vetta A. “On clustering: good, bad and spectral,” Journal of the ACM, Vol. 51(3), pp. 497–515, 2004. [Kapo 96] Kapogiannopoulos G., Papadakis M. “Character recognition using biorthogonal discrete wavelet transform,” Proceedings of the 41st Annual SPIE Meeting, Vol. 2825, August 1996. [Karh 46] Karhunen K. “Zur spektraltheorie stochastischer prozesse,” Annales Academiae Scientiarum Fennicae, Vol. 37, 1946. [Karh 94] Karhunen J., Joutsensalo J. “Representation and separation of signals using nonlinear PCA type learning,” Neural Networks, Vol. 7(1), pp. 113–127, 1994. [Koho 89] Kohonen T. Self-Organization and Associative Memory, 3rd ed., Springer-Verlag, 1989. [Koki 07] Kokiopoulou E., Saad Y. “Orthogonal neighborhood preserving projections: A projection-based dimensionality reduction technique,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29(12), pp. 2143–2156, 2007. [Kwon 04] Kwon O.W., Lee T.W. “Phoneme recognition using the ICA-based feature extraction and transformation,” Signal Processing, Vol. 84(6), pp. 1005–1021, 2004. [Lafo 06] Lafon S., Lee A.B. “Diffusion maps and coarse-graining:A unified framework for dimensionality reduction, graph partitioning and data set parameterization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28(9), pp. 1393–1403, 2006. [Lain 93] Laine A., Fan J. “Texture classification by wavelet packet signatures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15(11), pp. 1186–1191, 1993. [Lain 96] Laine A., Fan J. “Frame representations for texture segmentation,” IEEE Transcaction on Image Processing, Vol. 5(5), pp. 771–780, 1996. [Law 06] Law M.H.C., Jain A.K. “Incremental nonlinear dimensionality reduction by manifold learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28(3), pp. 377–391, 2006. [Lee 98] Lee T.-W. Independent Component Analysis, Kluwer Academic Publishers, 1998. [Lee 01] Lee D.D., Seung S. “Learning the parts of objects by nonnegative matrix factorization,” Nature, Vol. 401, pp. 788–791, 1999. [Levi 85] Levine M.D. Vision in Man and Machine, McGraw-Hill, 1985. [Lim 90] Lim J.S. Two-Dimensional Signal Processing, Prentice Hall, 1990. [Lin 08] Lin T., Zha H. “Riemannian manifold learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30(5), pp. 796–810, 2008. [Mall 89] Mallat S. “Multifrequency channel decompositions of images and wavelet models,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 37(12), pp. 2091–2110, 1989. [Mall 97] Mallet Y., Coomans D., Kautsky J., De Vel O. “Classification using adaptive wavelets for feature extraction,” IEEE Transactions for Pattern Analysis and Machine Intelligence, Vol. 19(10), pp. 1058–1067, 1997.
“08-Ch06-SA272” 18/9/2008 page 407
References
[Marc 95] Marco S.D., Heller P.N., Weiss J. “An M-band two dimensional translation-invariant wavelet transform and its applivations,” Proceedings of the IEEE Conference on Acoustics Speech and Signal Processing, pp. 1077–1080, 1995. [Meye 93] Meyer Y. Wavelets, Algorithms and Applications, SIAM, Philadelphia, 1993. [Mojs 00] Mojsilovic A., Popovic M.V., Rackov D.M. “On the selection of an optimal wavelet basis for texture characterization,” IEEE Transactions Image Processing, Vol. 9(12), 2000. [Nach 75] Nachmais J., Weber A. “Discrimination of simple and complex gratings,” V ision Research, Vol. 15, pp. 217–223, 1975. [Oja 83] Oja E. Subspace Methods for Pattern Recognition, Res. Studies Press, Letchworth, U.K., 1983. [Paat 91] Paatero P., Tapper U., aalto R., Kulmala M. “Matrix factorization methods for analysis diffusion battery data,” Journal of Aerosol Science,Vol. 22 (Supplement 1), pp. 273–276, 1991. [Paat 94] Paatero P., Tapper U. “Positive matrix factor model with optimal utilization of error,” Environmetrics, Vol. 5, pp. 111–126, 1994. [Papo 91] Papoulis A. Probability, Random Variables, and Stochastic Processes, 3rd ed., McGraw-Hill, 1991. [Pich 96] Pichler O., Teuner A., Hosticka, B. “A comparison of texture feature extraction using adaptive Gabor filtering, pyramidal and tree structured wavelet transforms,” Pattern Recognition, Vol. 29(5), pp. 733–742, 1996. [Pita 92] Pitas I. Digital Image Processing Algorithms, Prentice Hall, 1992. [Prak 97] Prakash M., Murty M.N. “Growing subspace pattern recognition methods and their neural network models,” IEEE Transactions on Neural Networks,Vol. 8(1), pp. 161–168, 1997. [Proa 92] Proakis J., Manolakis D. Digital Signal Processing, 2nd ed., Macmillan, 1992. [Pun 03] Pun C.-M., Lee M.-C. “Log-Polar wavelet energy signatures for rotation and scale invariant texture classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25(5), pp. 590–603, 2003. [Qui 07] Qui H., Hancock E.R. “Clustering and embedding using commute times,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29(11), pp. 1873–1890, 2007. [Rowe 00] S.T. Roweis S.T., Saul L.K. “Nonlinear dimensionality reduction by locally linear embedding,” Science, Vol. 290, pp. 2323–2326, 2000. [Saul 01] Saul L.K., Roweis S.T. “An introduction to locally linear embedding,” http://www.cs.toronto.edu/˜ roweis/lle/papers/lleintro.pdf [Scho 98] Schölkopf B., Smola A., Muller K.R. “Nonlinear component analysis as a kernel eigenvalue problem,” Neural Computation, Vol. 10, pp. 1299–1319, 1998. [Sebr 03] Sebro N., Jaakola T. “Weighted low-rank approximations,” Proceedings of the ICML Conference, pp. 720–727, 2003. [Sha 05] Sha F., Saul L.K. “Analysis and extension of spectral methods for nonlinear dimensionality reduction,” Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany, 2005. [Shui 07] Shuicheng Y., Xu D., Zhang B., Zhang H.-J., Yang Q., Lin S. “Graph embedding and extensions: A general framework for dimensionality reduction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29(1), pp. 40–51, 2007. [Shaw 04] Shawe-Taylor J., Cristianini N. Kernel Methods for Pattern Analysis, Cambridge University Press, Cambridge, MA, 2004.
407
“08-Ch06-SA272” 18/9/2008 page 408
408
CHAPTER 6 Feature Generation I
[Smar 03] Smaragdis P., Brown J.C. “Nonnegative matrix factorization for polyphonic music transcription,” Proceedings IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2003. [Sra 06] Sra S., Dhillon I.S. “Non-negative matrix approximation: Algorithms and applications,” Technical Report TR-06-27, University of Texas at Austin, 2006. [Stef 93] Steffen P., Heller P.N., Gopinath R.A., Burrus C.S. “Theory of regular M-band wavelet bases,” IEEE Tansactions on Signal Processing, Vol. 41(12), pp. 3497–3511, 1993. [Stra 80] Strang G. Linear Algebra and Its Applications, 2nd ed., Harcourt Brace Jovanovich, 1980. [Szu 92] Szu H.H., Telfer B.A., Katambe S. “Neural network adaptive wavelets for signal representation and classification,” Optical Eng., Vol. 31, pp. 1907–1916, 1992. [Szym 06] Szymkowiak-Have A., Girolami M.A., Larsen J. “Clustering via kernel decomposition,” IEEE Transactions on Neural Networks, Vol. 17(1), pp. 256–264, 2006. [Sun 06] Sun J., Boyd S., Xiao L., Diaconis P. “The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem,” SIAM Review, Vol. 48(4), pp. 681–699, 2006. [Tene 00] Tenenbaum J.B., De Silva V., Langford J.C. “A global geometric framework for dimensionality reduction,” Science, Vol. 290, pp. 2319–2323, 2000. [Trop 03] Tropp J.A. “Literature survey: Nonnegative matrix factorization,” Unpublished note, http://www-personal.umich.edu/˜jtropp/, 2003. [Turn 86] Turner M.R. “Texture discrimination by Gabor functions,” Biol. Cybern., Vol. 55, pp. 71–82, 1986. [Unse 86] Unser M. “Local linear transforms for texture measurements,” Signal Processing, Vol. 11(1), pp. 61–79, 1986. [Unse 95] Unser M. “Texture classification and segmentation using wavelet frames,” IEEE Transactions on Image Processing, Vol. 4(11), pp. 1549–1560, 1995. [Unse 89] Unser M., Eden M. “Multiresolution feature extraction and selection for texture segmentation,” IEEE Transations on Pattern Analysis and Machine Intelligence, Vol. 11(7), pp. 717–728, 1989. [Vaid 93] Vaidyanathan P.P. Multirate Systems and Filter Banks, Prentice Hall, 1993. [Vett 92] Vetterli M., Herley C. “Wavelets and filter banks: Theory and design,” IEEE Transactions on Signal Processing, Vol. 40(9), pp. 2207–2232, 1992. [Vett 95] Vetterli M., Kovacevic J. Wavelets and Subband Coding, Prentice Hall, 1995. [ Wata 73] Watanabe S., Pakvasa N. “Subspace method in pattern recognition,” Proceedings of the International Joint Conference on Pattern Recognition, pp. 25–32, 1973. [Weld 96] Weldon T., Higgins W., Dunn D. “Efficient Gabor filter design for texture segmentation,” Pattern Recognition, Vol. 29(2), pp. 2005–2025, 1996. [Wein 05] Weinberger K.Q., Saul L.K. “Unsupervised learning of image manifolds by semidefinite programming,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 988–995, Washington D.C., USA, 2004. [Wuns 95] Wuncsh P., Laine A. “Wavelet descriptors for multiresolution recognition of handwritten characters,” Pattern Recognition, Vol. 28(8), pp. 1237–1249, 1995.
“08-Ch06-SA272” 18/9/2008 page 409
References
[Xu 03] Xu W., Liu X., Gong Y. “Document clustering based on nonnegative matrix factorization,” Proceedings 26th Annual International ACM SIGIR Conference, pp. 263–273, ACM Press, 2003. [Ye 04] Ye J. “Generalized low rank approximation of matrices,” Proceedings of the 21st International Conference on Machine Learning, pp. 887–894, Banff, Alberta, Canada, 2004. [Zafe 06] Zafeiriou S., Tefas A., Buciu I., Pitas I. “Exploiting discriminant information in non-negative matrix factorization with application to frontal face verification,” IEEE Transactions on Neural Networks, Vol. 17(3), pp. 683–695, 2006.
409
“09-Ch07-SA272” 18/9/2008 page 411
CHAPTER
Feature Generation II
7
7.1 INTRODUCTION In the previous chapter we dealt with the task of feature generation via linear or nonlinear transformation techniques. This is just one of the possibilities available to the designer. There are a number of alternatives, however, that are very much application dependent. Although similarities among various applications do exist, there are also major differences. We will start by focusing on one major application area; that of image analysis. Clearly, we cannot review all techniques that have been suggested and used. Their number is really large. Instead, we will focus on basic directions, with a wide range of applications in mind, such as medical imaging, remote sensing, robot vision, and optical character recognition. The major goal may be summarized as follows: given an image, or a region within an image, generate the features that will subsequently be fed to a classifier in order to classify the image in one of the possible classes. A digital (monochrome) image is usually the result of a discretization process (sampling) of a continuous image function I (x, y) and is stored in the computer as a twodimensional array I (m, n) with m ⫽ 0, 1, . . . , Nx ⫺ 1 and n ⫽ 0, 1, . . . , Ny ⫺ 1. That is, it is stored as an Nx ⫻ Ny array. Every (m, n) element of the array corresponds to a pixel (picture element or image element) of the image, whose brightness or intensity is equal to I (m, n). Furthermore, when the intensity I (m, n) is quantized in Ng discrete (gray) levels Ng is known as the depth of the image. Then, the gray-level sequence I (m, n) can take one of the integer values 0, 1, . . . , Ng ⫺ 1. The depth Ng is usually a power of 2 and can take large values (e.g., 64, 256) when the image is stored in the computer. However,for the human eye it is difficult to discern detailed intensity differences, and in practice Ng ⫽ 32 or 16 is a sufficient choice for image representation. The need for feature generation stems from our inability to use the raw data. Even for a small 64 ⫻ 64 image the number of pixels is 4096. For most classification tasks this number is too large, raising computational as well as generalization problems, as discussed in earlier chapters. Feature generation is a procedure that computes new variables that in one way or another originate from the stored values
411
“09-Ch07-SA272” 18/9/2008 page 412
412
CHAPTER 7 Feature Generation II
of the image array I (m, n). The goal is to generate features that exhibit high information-packing properties, from the class separability point of view. Because we cannot use the raw data I (m, n) directly, the features should encode efficiently the relevant information residing in the original data. The other application area (discussed at the end of this chapter) is that of audio classification. Although some years ago image and audio analysis were considered to a large extent to be two scientific disciplines with different and distinct application areas, this is no longer the case. In a multimedia document its semantics are embedded in multiple forms that are usually complementary to each other. Thus, effective indexing for efficient handling (browsing, searching, manipulation, and information retrieval) requires a multimodal approach, in which either the most appropriate modality is selected or the different modalities are used in an integrated fashion. The visual modality contains everything that can be sensed by the eye (i.e., images that are naturally or artificially generated). The auditory modality contains the speech, music, and environmental sounds that can be heard in a video document. The focus of the last part of this chapter will be on typical features used to characterize and classify audio information. Some of the feature generation techniques can be considered common and can be applicable in both visual and audio modalities. On the other hand, a large number of features are the result of different approaches to exploit the specific nature of the signals and encode the required classification information in a more efficient way.
7.2 REGIONAL FEATURES 7.2.1 Features for Texture Characterization The texture of an image region is determined by the way the gray levels are distributed over the pixels in this region. Although there is no clear definition of “texture,” we are all in a position to describe an image by the look of it as fine or coarse, smooth or irregular, homogeneous or inhomogeneous, and so forth. Our goal in this subsection is to generate appropriate features that somehow quantify these properties of an image region. These features will emerge by exploiting space relations underlying the gray-level distribution.
First-Order Statistics Features Let I be the random variable representing the gray levels in the region of interest. The first-order histogram P(I ) is defined as P(I ) ⫽
number of pixels with gray-level I total number of pixels in the region
(7.1)
That is, P(I ) is the fraction of pixels with gray-level I . Let Ng be the number of possible gray levels. Based on (7.1), the following quantities are defined.
“09-Ch07-SA272” 18/9/2008 page 413
7.2 Regional Features
Moments: Ng ⫺1
mi ⫽ E[I i ] ⫽
i ⫽ 1, 2, . . .
I i P(I ),
(7.2)
I ⫽0
Obviously m0 ⫽ 1 and m1 ⫽ E[I ], the mean value of I . Central moments: Ng ⫺1
i ⫽ E[(I ⫺ E[I ])i ] ⫽
(I ⫺ m1 )i P(I )
(7.3)
I ⫽0
The most frequently used central moments are 2 , 3 , and 4 . 2 ⫽ 2 is the variance, and 3 is known as the skewness (sometimes and is normalized by 3 ) and 4 as the kurtosis (sometimes is normalized by 4 ) of the histogram. The variance is a measure of the histogram width, that is, a measure of how much the gray levels differ from the mean. Skewness is a measure of the degree of histogram asymmetry around the mean,and 4 is a measure of the histogram sharpness. Depending on the value of 4 , the resulting histogram is called platykurtic, for large values, leptokurtic, for small values, and mesokurtic otherwise. The normal distribution is a mesokurtic one. Figure 7.1 shows six variations of the same image (with 16 gray levels) with their corresponding histograms. We can observe the difference between the platykurtic and the leptokurtic one. In the latter, only the middle gray levels are present (with no I ⫽ 0 or I ⫽ 15), in contrast to the platykurtic one, where all gray levels are present. For the two asymmetric cases, one corresponds to a majority of low gray levels and the other to a majority of high levels. The resulting values of 3
Original
Normal
⫺Skewed
⫹Skewed
FIGURE 7.1 Examples of images and corresponding histograms.
Platykurtic
Leptokurtic
413
“09-Ch07-SA272” 18/9/2008 page 414
414
CHAPTER 7 Feature Generation II
and 4 from left to right are 3 : 4 :
587 16609
0 7365
⫺169 7450
169 7450
0 9774
0 1007
Other quantities that result from the first-order histogram are: Absolute moments: Ng ⫺1
ˆ i ⫽ E[| I ⫺ E[I ] |i ] ⫽
| I ⫺ E[I ] |i P(I )
(7.4)
I ⫽0
Entropy: Ng ⫺1
H ⫽ ⫺E[log2 P(I )] ⫽ ⫺
P(I ) log2 P(I )
(7.5)
I ⫽0
Entropy is a measure of histogram uniformity. The closer to the uniform distribution (P(I ) ⫽ constant), the higher the H. For the six images of Figure 7.1 the corresponding values are H:
4.61
4.89
4.81
4.81
4.96
4.12
Second-Order Statistics Features—Co-occurrence Matrices The features resulting from the first-order statistics provide information related to the gray-level distribution of the image, but they do not give any information about the relative positions of the various gray levels within the image. Are all low-value gray levels positioned together, or are they interchanged with the high-value ones? This type of information can be extracted from the second-order histograms, where the pixels are considered in pairs. Two more parameters now enter into the scene. These are the relative distance among the pixels and their relative orientation. Let d be the relative distance measured in pixel numbers (d ⫽ 1 for neighboring pixels, etc.). The orientation is quantized in four directions: horizontal, diagonal, vertical, and antidiagonal (0◦ , 45◦ , 90◦ , 135◦ ), as shown in Figure 7.2. For each combination of d and a two-dimensional histogram is defined 0◦ : P I (m, n) ⫽ I1 , I (m ⫾ d, n) ⫽ I2 ⫽
number of pairs of pixels at distance d with values ( I1 , I2 ) total number of possible pairs
In a similar way 45◦ : P I (m, n) ⫽ I1 , I (m ⫾ d, n ⫿ d) ⫽ I2 90◦ : P I (m, n) ⫽ I1 , I (m, n ⫿ d) ⫽ I2 135◦ : P I (m, n) ⫽ I1 , I (m ⫾ d, n ⫾ d) ⫽ I2
(7.6)
“09-Ch07-SA272” 18/9/2008 page 415
7.2 Regional Features
m
90˚
5˚
45
˚
13
0˚
n
FIGURE 7.2 The four orientations used to construct co-occurrence matrices.
For each of these histograms an array is defined, known as the co-occurrence or spatial dependence matrix. Let, for example, an image array I (m, n) be ⎡ 0 ⎢1 ⎢ I ⫽⎢ ⎣3 3
0 1 2 2
⎤ 2 ⎥ 0⎥ ⎥ 3⎦ 2
2 0 3 2
(7.7)
which corresponds to a 4 ⫻ 4 image. We have also assumed that Ng ⫽ 4 (I (m, n) ∈ {0, 1, 2, 3}). The co-occurrence matrix for a pair (d, ) is defined as the Ng ⫻ Ng matrix ⎡ A⫽
(0, 0)
⎢ (1, 0) 1⎢ ⎢ ⎢ R ⎣(2, 0) (3, 0)
(0, 3)
⎤
(0, 1)
(0, 2)
(1, 1)
(1, 2)
(2, 1)
(2, 2)
⎥ (1, 3)⎥ ⎥ (2, 3)⎥ ⎦
(3, 1)
(3, 2)
(3, 3)
where (I1 , I2 ) is the number of pixel pairs, at relative position (d, ), which have gray-level values I1 , I2 , respectively. R is the total number of possible pixel pairs. Hence R1 (I1 , I2 ) ⫽ P(I1 , I2 ). For the image of (7.7) and relative pixel position (1, 0◦ ) we have ⎡
4 ⎢1 1 ⎢ A0 (d ⫽ 1) ⫽ ⎢ 24 ⎣1 0
1 2 0 0
1 0 6 3
⎤ 0 ⎥ 0⎥ ⎥ 3⎦ 2
In words, for each of the intensity pairs, such as (0, 0), we count the number of pixel pairs at relative distance d ⫽ 1 and orientation ⫽ 0◦ that take these values.
415
“09-Ch07-SA272” 18/9/2008 page 416
416
CHAPTER 7 Feature Generation II
For our example this is 4. Two of them result from searching in the positive direction and two in the negative. According to the definition (7.6), these pixel pairs have coordinates (m, n) and (m ⫾ 1, n) and gray levels I1 ⫽ 0, I2 ⫽ 0. The total number of pixel pairs for this case is 24. Indeed, for each row there are Nx ⫺ 1 pairs and there are Ny rows. Thus,the total number for both positive and negative directions is 2(Nx ⫺ 1)Ny ⫽ 2(3 ⫻ 4) ⫽ 24. For the diagonal direction 45◦ and d ⫽ 1 for each row we have 2(Nx ⫺ 1) pairs, except the first (or last) one, for which no pairs exist. Thus, the total number is 2(Nx ⫺ 1)(Ny ⫺ 1) ⫽ 2(3 ⫻ 3) ⫽ 18. For d ⫽ 1 and 90◦ we have 2(Ny ⫺ 1)Nx pairs, and finally for d ⫽ 1 and 135◦ 2(Nx ⫺ 1)(Ny ⫺ 1). For our example image and (d ⫽ 1, ⫽ 45◦ ), we obtain ⎡
0 ⎢1 1 ⎢ A45 (d ⫽ 1) ⫽ ⎢ 18 ⎣2 1
1 0 1 1
⎤ 1 ⎥ 1⎥ ⎥ 3⎦ 0
2 1 0 3
From the definition of the co-occurrence matrix, it is apparent that it is a symmetric one, something that can be used to reduce subsequent computations. Having defined the probabilities of occurrence of gray levels with respect to relative spatial pixel position, we will go ahead to define the corresponding features. Some of them have a direct physical interpretation with respect to texture, for example, to quantify coarseness, smoothness, and so on. On the other hand, others do not possess such a property, but they still encode texture-related information with high discriminatory power. ■
Angular second moment Ng ⫺1 Ng ⫺1
ASM ⫽
i⫽0
(P(i, j))2
(7.8)
j⫽0
This feature is a measure of the smoothness of the image. Indeed, if all pixels are of the same gray-level I ⫽ k, then P(k, k) ⫽ 1 and P(i, j) ⫽ 0, i ⫽ k or j ⫽ k, and ASM ⫽ 1. At the other extreme, if we could have all possible pairs of gray levels with equal probability R1 , then ASM ⫽ RR2 ⫽ R1 . The less smooth the region is, the more uniformly distributed P(i, j) and the lower the ASM (Problem 7.5). ■
Contrast Ng ⫺1
CON ⫽
n⫽0
n2
⎧ Ng ⫺1 ⎪ ⎪ ⎨ ⎪ ⎪ ⎩
⎫ ⎪ ⎪ P(i, j) ⎬
j⫽0
⎪ ⎪ ⎭
Ng ⫺1
i⫽0 |i⫺j|⫽n
(7.9)
“09-Ch07-SA272” 18/9/2008 page 417
7.2 Regional Features
This is a measure of theimage contrast—that is, a measure of local graylevel variations. Indeed, i j P(i, j) is the percentage of pixel pairs whose intensity differs by n. The n2 dependence weighs the big differences more; thus, CON takes high values for images of high contrast. ■
Inverse difference moment Ng ⫺1 Ng ⫺1
IDF ⫽
i⫽0
j⫽0
P(i, j) 1 ⫹ (i ⫺ j)2
(7.10)
This feature takes high values for low-contrast images due to the inverse (i⫺j)2 dependence. ■
Entropy Ng ⫺1 Ng ⫺1
Hxy ⫽ ⫺
i⫽0
P(i, j) log2 P(i, j)
(7.11)
j⫽0
Entropy is a measure of randomness and takes low values for smooth images. These features are only a few from a larger set that can be derived. In the classical [Hara 73] paper, fourteen of those are summarized. They are repeated in Table 7.1. Px (Py ) (and related quantities) refer to the statistics with respect to the x(y)-axis. All features in the table are functions of the distance d and the orientation . Thus, if an image is rotated, the values of the features will be different. In practice, for each d the resulting values for the four directions are averaged out. In this way, we make these textural features rotation tolerant. Besides the previous list of features, a number of other statistics-related features have been proposed. For example, in [Tamu 78] textural features are generated with an emphasis on the human visual perception. A set of features is suggested corresponding to texture coarseness, contrast, regularity, and so on. In [Davi 79] features based on a generalized definition of co-occurrence matrices are suggested, which are more appropriate for textures with long scale variations (macrotextures). An extensive treatment of texture is given in [Petr 06]. Example 7.1 Figure 7.3 shows two texture images, one coarse, known as grass [Brod 66], and the other smooth. Table 7.2 summarizes the values of some of the features for both of them.
Features Using Gray-Level Run Lengths A gray-level run is a set of consecutive pixels having the same gray-level value. The length of the run is the number of pixels in the run [Gall 75, Tang 98]. Run length features encode textural information related to the number of times each gray-level,
417
“09-Ch07-SA272” 18/9/2008 page 418
418
CHAPTER 7 Feature Generation II
Table 7.1
Features for Texture Characterization
Angular Second Moment: f1 ⫽
(P(i, j))2 i
Sum Entropy: f8 ⫽ ⫺
j
Entropy: f9 ⫽ ⫺
Correlation: { i j (ij)P(i, j)} ⫺ x y x y
f10 ⫽
j
2Nx ⫺2 (Ng ⫺1)
iPx⫹(⫺)y (i)
i⫽0
Sum Variance: 2N g ⫺2
N g ⫺1
Px⫺y (i) log Px⫺y (i)
Information Measure I:
P(i, j) 1 ⫹ (i ⫺ j)2
Sum (Difference) Average:
f7 ⫽
f11 ⫽ ⫺
i⫽0
Inverse Difference Moment:
f6 ( fˆ6 ) ⫽
(i ⫺ fˆ6 )2 Px⫺y (i)
Difference Entropy:
j
i
N g ⫺1 i⫽0
(i ⫺ )2 P(i, j)
f5 ⫽
P(i, j) log P(i, j) ≡ Hxy
j
Difference Variance:
Variance:
i
i
|i⫺j|⫽n
f4 ⫽
Px⫹y (i) log Px⫹y (i)
i⫽0
Contrast: ⎫ ⎧ ⎪ ⎪ ⎬ ⎨ Ng ⫺1 2 n P(i, j) f2 ⫽ n⫽0 ⎪ ⎪ j ⎭ ⎩i
f3 ⫽
2N g ⫺2
(i ⫺ f6 )2 Px⫹y (i)
f12 ⫽
1 Hxy ⫺ Hxy
max{Hx , Hy }
Information Measure II: f13 ⫽
2 ⫺ H )) 1 ⫺ exp(⫺2(Hxy xy
Maximal Correlation Coefficient: 1
f14 ⫽ (2nd largest eigenvalue of Q)2
i⫽0
P(i, k)P( j, k) Px (i)Py (k) k 1 ⫽⫺ Hxy P(i, j)
Definitions: Q(i, j) ⫽
i
j
log(Px (i)Py ( j)) Px (i) ⫽ P(i, j) j Px⫾y (k) ⫽ P(i, j)
2 ⫽⫺ Hxy
j
Px (i)Py ( j)
i
log(Px (i)Py ( j)) Py ( j) ⫽ P(i, j) i
, x , y , x , y ; Hx , Hy
i j,|i⫾j|⫽k
means, st. deviations and entropies.
“09-Ch07-SA272” 18/9/2008 page 419
7.2 Regional Features
(a)
(b)
FIGURE 7.3 Examples of (a) coarse and (b) smooth images.
Table 7.2 Second-Order Histogram Features for the Two Images of Figure 7.3 Coarse
Smooth
ASM
0.0066
0.0272
CON
989.5
0.613
IDF
0.117
0.783
Hxy
8.352
5.884
for example, “1,” appears in the image by itself, the number of times it appears in pairs, and so on. Take, for example, the image ⎡ 0 ⎢1 ⎢ I ⫽⎢ ⎣3 3
0 1 2 2
2 0 3 2
⎤ 2 ⎥ 0⎥ ⎥ 3⎦ 2
with four possible levels of gray (Ng ⫽ 4). For each of the four directions (0◦ , 45◦ , 90◦ , 135◦ ) we define the corresponding run length matrix QRL . Its (i, j) element gives the number of times a gray-level i ⫺ 1, i ⫽ 1, . . . , Ng , appears in the image with run length j, j ⫽ 1, 2, . . . , Nr . This is an Ng ⫻ Nr array, where Nr is the largest possible run length in the image. For 0◦ we obtain ⎡
0 ⎢0 ⎢ ◦ QRL (0 ) ⫽ ⎢ ⎣1 2
2 1 1 1
0 0 1 0
⎤ 0 ⎥ 0⎥ ⎥ 0⎦ 0
(7.12)
419
“09-Ch07-SA272” 18/9/2008 page 420
420
CHAPTER 7 Feature Generation II
The first element of the first row of the matrix is the number of times gray-level “0” appears by itself (0 for our example), the second element is the number of times it appears in pairs (2 in the example), and so on. The second row provides the same information for gray-level “1” and so on. For the 45◦ direction we have ⎡
4 ⎢2 ⎢ QRL (45◦ ) ⫽ ⎢ ⎣6 4
0 0 0 0
0 0 0 0
⎤ 0 ⎥ 0⎥ ⎥ 0⎦ 0
(7.13)
Based on the preceding definition of the run length matrix, the following features are defined. ■
Short-run emphasis Ng Nr
i⫽1 j⫽1 (QRL (i, j)/j Ng Nr i⫽1 j⫽1 QRL (i, j)
SRE ⫽
2)
(7.14)
The denominator is the total number of run lengths in the matrix, 9 for (7.12) and 16 for (7.13). This feature emphasizes small run lengths,due to the division by j 2 . ■
Long-run emphasis Ng Nr LRE ⫽
2 i⫽1 j⫽1 (QRL (i, j)j ) N g N r i⫽1 j⫽1 QRL (i, j)
(7.15)
This gives emphasis to long-run lengths. Thus, we expect SRE to be large for coarser and LRE to be large for smoother images. ■
Gray-level nonuniformity Ng Nr GLNU ⫽
2
i⫽1 j⫽1 QRL (i, j) Ng Nr i⫽1 j⫽1 QRL (i, j)
(7.16)
The term in the brackets is the total number of run lengths for each graylevel. Large run length values contribute a great deal because of the square. When runs are uniformly distributed among the gray levels, GNLU takes small values. ■
Run length nonuniformity Nr Ng RLN ⫽
2
j⫽1 i⫽1 QRL (i, j) Ng Nr i⫽1 j⫽1 QRL (i, j)
In a similar way, RLN is a measure of run length nonuniformity.
(7.17)
“09-Ch07-SA272” 18/9/2008 page 421
7.2 Regional Features
Table 7.3 Run Length Features for the Images of Figure 7.3
■
Coarse
Smooth
SRE
0.932
0.563
LRE
1.349
16.929
GLNU
255.6
71.6
RLN
3108
507
RP
0.906
0.4
Run percentage Ng Nr RP ⫽
j⫽1 QRL (i, j)
i⫽1
L
(7.18)
where L is the total possible number of runs in the image, if all runs had length equal to one, that is, the total number of pixels. RP takes low values for smooth images. Example 7.2 For the two images of Figure 7.3 the values of Table 7.3 have resulted.
7.2.2 Local Linear Transforms for Texture Feature Extraction Second-order statistics features were introduced in order to exploit the spatial dependencies that characterize the texture of an image region. We will now focus on an alternative possibility, which has been used extensively in practice. Let us consider a neighborhood of size N ⫻ N centered at pixel location (m, n). Let x mn be the vector with elements the N 2 points within the area, arranged in a row-byrow mode. A local linear transform or local feature extractor is defined as ⎡ y mn ⫽ A x mn T
a T1
⎤
⎢ T ⎥ ⎢a ⎥ ⎢ 2 ⎥ ≡ ⎢ . ⎥ x mn ⎢ .. ⎥ ⎣ ⎦ a TN 2
(7.19)
The respective correlation matrices are related via the N 2 ⫻ N 2 nonsingular transformation matrix A as Ry ≡ E[y mn y Tmn ] ⫽ AT Rx A
(7.20)
421
“09-Ch07-SA272” 18/9/2008 page 422
CHAPTER 7 Feature Generation II
m
1 ymn
mask 9
mask 2
n
mask 1
422
2 ymn
9 ymn
FIGURE 7.4 Filtering the image with each of the masks results in new transformed images/channels.
From these definitions it is readily seen that each element of y contains information about all the elements of x. This becomes clearer if we look more closely at the way the two correlation matrices are related. Indeed,the diagonal elements of Ry are the respective variances of the elements of y. These are first-order statistics quantities, yet their values contain information about the spatial dependencies (second-order statistics) of the original image. Here lies the essence of the technique. Texturerelated spatial dependencies of an image can be accommodated in the first-order statistics of the transformed image. Using appropriately defined local transform matrices, various aspects of texture properties can be extracted. Of course, the philosophy does not change if instead of transforming vectors we use two-dimensional (separable) transforms of the corresponding subimage region. One way to look at (7.19) is to interpret it as a series of N 2 filtering operations (convolutions, Appendix D), with a common input vector, x mn , that is, the N ⫻ N subimage centered at (m, n). The elements of y mn are the respective filter output samples. This is illustrated in Figure 7.4, where the N ⫻ N subimage (N ⫽ 3) is filtered through 9 equivalent two-dimensional filters, each characterized by a different coefficient matrix, known as mask. In [Laws 80] it is suggested that the corresponding masks be constructed from three basic vectors, namely, [1, 2, 1]T , [⫺1, 0, 1]T , [⫺1, 2, ⫺1]T , for N ⫽ 3. The first corresponds to a local averaging operator, the second to an edge detection operator, and the third to a spot detector. These form a complete (nonorthogonal) set of vectors in the R3 space. The respective nine masks are formed by their cross-products, that is, ⎡
1 ⎢ ⎣2 1
2 4 2
⎤⎡ ⫺1 1 ⎥⎢ 2⎦ ⎣⫺2 ⫺1 1
0 0 0
⎤⎡ ⫺1 1 ⎥⎢ 2⎦ ⎣⫺2 ⫺1 1
2 4 2
⎤ ⫺1 ⎥ ⫺2⎦ ⫺1
“09-Ch07-SA272” 18/9/2008 page 423
7.2 Regional Features
⎡
⫺1 ⎢ ⎣ 0 1 ⎡ ⫺1 ⎢ ⎣ 2 ⫺1
⫺2 0 2 ⫺2 4 ⫺2
⎤⎡ 1 ⫺1 ⎥⎢ 0⎦ ⎣ 0 ⫺1 1 ⎤⎡ 1 ⫺1 ⎥⎢ 2⎦ ⎣⫺2 1 ⫺1
0 0 0 0 0 0
⎤⎡ 1 ⫺1 ⎥⎢ 0⎦ ⎣ 0 ⫺1 1 ⎤⎡ 1 ⫺1 ⎥⎢ 2⎦ ⎣⫺2 1 ⫺1
⫺2 0 2 ⫺2 4 ⫺2
⎤ 1 ⎥ 0⎦ ⫺1 ⎤ 1 ⎥ ⫺2⎦ 1
Each element of the vector y mn is the result of filtering the local image neighborhood centered at (m, n) with each of the masks. By moving the masks around at the various (m, n) positions, nine different images, channels, will be obtained, each encoding different aspects of the texture of the original image. First-order-statistics quantities, such as variance and kurtosis, computed from each of these images, can then be used as features for texture classification. Masks larger than 3 ⫻ 3 have also been used. In some cases, an attempt to optimize the masks has been made, so that the resulting variances of the channels for the different classes are as different as possible [Unse 86]. This turns out to be an eigenvalue–eigenvector task, similar to the ones we have already met in Chapter 5. A comparative study of a number of optimal or suboptimal local linear transforms, including orthogonal ones, such as DCT, DST, and Karhunen-Loève, is given in [Unse 86, Unse 89, Rand 99]. Finally, it must be pointed out that all these techniques are closely related to the Gabor filtering approach of the previous chapter.
7.2.3 Moments Geometric Moments Let I (x, y) be a continuous image function. Its geometric moment of order p ⫹ q is defined as
mpq ⫽
⬁
⬁
⫺⬁ ⫺⬁
x p yq I (x, y) dx dy
(7.21)
Geometric moments provide rich information about the image and are popular features for pattern recognition. Their information content stems from the fact that moments provide an equivalent representation of an image, in the sense that an image can be reconstructed from its moments (of all orders) [Papo 91,p. 115]. Thus, each moment coefficient conveys a certain amount of the information residing in an image. It is by now commonplace to state that a desirable property in pattern recognition is invariance in geometric transformations. Moments, as defined in (7.21), depend on the coordinates of the object of interest within an image; thus, they lack the invariance property. This problem can be circumvented by defining appropriate combinations of normalized versions of the moments. Specifically, our goal will be to define moments that are invariant to: Translations: x⬘ ⫽ x ⫹ a,
y⬘ ⫽ y ⫹ b
423
“09-Ch07-SA272” 18/9/2008 page 424
424
CHAPTER 7 Feature Generation II
Scaling: x⬘ ⫽ ␣x,
y⬘ ⫽ ␣y
Rotations: x⬘ cos ⫽ y⬘ ⫺ sin
x y
sin cos
To this end, let us define Central moments: pq ⫽
I (x, y)(x ⫺ x) ¯ p (y ⫺ y¯ )q dx dy
(7.22)
where x¯ ⫽
m10 , m00
y¯ ⫽
m01 m00
Central moments are invariant to translations. Normalized central moments: pq ⫽
pq
␥ , 00
␥⫽
p⫹q⫹2 2
(7.23)
These are easily shown to be invariant to both translation and scaling (Problem 7.6).
The Seven Moments of Hu Hu [Hu 62] has defined a set of seven moments that are invariant under the actions of translation, scaling, and rotation. These are p⫹q ⫽2 1 ⫽ 20 ⫹ 02 2 2 ⫽ (20 ⫺ 02 )2 ⫹ 411
p⫹q ⫽3 3 ⫽ (30 ⫺ 312 )2 ⫹ (03 ⫺ 321 )2 4 ⫽ (30 ⫹ 12 )2 ⫹ (03 ⫹ 21 )2 5 ⫽ (30 ⫺ 312 )(30 ⫹ 12 )[(30 ⫹ 12 )2 ⫺ 3(21 ⫹ 03 )2 ] ⫹ (03 ⫺ 321 )(03 ⫹ 21 )[(03 ⫹ 21 )2 ⫺ 3(12 ⫹ 30 )2 ] 6 ⫽ (20 ⫺ 02 )[(30 ⫹ 12 )2 ⫺ (21 ⫹03 )2 ] ⫹411 (30 ⫹ 12 )(03 ⫹ 21)
“09-Ch07-SA272” 18/9/2008 page 425
7.2 Regional Features
7 ⫽ (321 ⫺ 03 )(30 ⫹ 12 )[(30 ⫹ 12 )2 ⫺ 3(21 ⫹ 03 )2 ] ⫹ (30 ⫺ 312 )(21 ⫹ 03 )[(03 ⫹ 21 )2 ⫺ 3(30 ⫹ 12 )2 ]
The first six of these moments are also invariant under the action of reflection, while 7 changes sign. The values of these quantities can be quite different. In practice, in order to avoid precision problems, the logarithms of their absolute values are usually used as features. A number of other moment-based features that are invariant to more general transformations have also been proposed [Reis 91, Flus 93, Flus 94]. The case of moment invariants in the general l-dimensional space is treated in [Mami 98]. For a digital image I (i, j), with i ⫽ 0, 1, . . . , Nx ⫺ 1, j ⫽ 0, 1, . . . , Ny ⫺ 1, the preceding moments can be approximated by replacing integrals by summations, mpq ⫽
i
I (i, j)i p j q
(7.24)
j
In order to keep the dynamic range of the moment values consistent for differentsized images, normalization of the x ⫺ y axis can be performed, prior to computation of the moments. The moments are then approximated by mpq ⫽
p q
I (xi , yi )xi yi
(7.25)
i
where the sum is over all image pixels. Then xi , yi are the coordinates of the center point of the ith pixel and are no longer integers but real numbers in the interval xi ∈ [⫺1, ⫹1], yi ∈ [⫺1, ⫹1]. For digital images, the invariance properties of the moments we have defined are only approximately true. An analysis in [Liao 96] reveals that the approximation error increases with the coarseness of the sampling grid as well as with the order of the moments. Example 7.3 Figure 7.5 shows the Byzantine music symbol known as “petasti,” resulting from a scanner, in scaled and various rotated versions. From left to right in the clockwise sense we have the original version, the scaled, the mirrored, and the rotated by 15◦ , 90◦ , and 180◦ versions, respectively. Table 7.4 shows the resulting Hu moments for each of the version. The (approximate) invariance of the moments is apparent. Note the minus sign in 7 for the reflected (mirror) version.
Zernike Moments The geometric moments defined in (7.21) can also be viewed as projections (Chapter 6) of I (x, y) on the basis functions formed by the monomials x p yq . These monomials are not orthogonal; thus, the resulting geometric moment features are not optimal from an information redundancy point of view. In this subsection we will derive moments based on alternative complex polynomial functions, known as
425
“09-Ch07-SA272” 18/9/2008 page 426
426
CHAPTER 7 Feature Generation II
(a)
(b)
(c)
(d)
(e)
(f)
FIGURE 7.5 The Byzantine symbol “petasti” in various scaled and rotated versions, from (a) to (f).
Table 7.4 The Invariant Moments of Hu for the Versions of the “Petasti” Symbol Moments
0◦
Scaled
180◦
15◦
Mirror
90◦
1
93.13
91.76
93.13
94.28
93.13
93.13
2
58.13
56.60
58.13
58.59
58.13
58.13
3
26.70
25.06
26.70
27.00
26.70
26.70
4
15.92
14.78
15.92
15.83
15.92
15.92
5
3.24
2.80
3.24
3.22
3.24
3.24
6
10.70
9.71
10.70
10.57
10.70
10.70
7
0.53
0.46
0.53
0.56
⫺0.53
0.53
Zernike polynomials. These form a complete orthogonal set over the interior of the unit circle x 2 ⫹ y2 ⱕ 1 (Problem 7.7) and are defined as Vpq (x, y) ⫽ Vpq (, ) ⫽ Rpq () exp( jq)
where: p is a nonnegative integer q is an integer subject to the constraint p ⫺ |q| even, |q| ⱕ p ⫽ x 2 ⫹ y2 ⫽ tan⫺1
y x
“09-Ch07-SA272” 18/9/2008 page 427
7.2 Regional Features
(p⫺|q|)/2
Rpq () ⫽
s⫽0
(⫺1)s [(p ⫺ s)!]p⫺2s p ⫺ |q| p ⫹ |q| ⫺s ! ⫺s ! s! 2 2
The Zernike moments of a function I (x, y) are given by Apq ⫽
p⫹1
x 2 ⫹y2 ⱕ1
I (x, y)V ∗ (, ) dx dy
where the ∗ denotes complex conjugation. For a digital image, the respective Zernike moments are computed as Apq ⫽
p⫹1 I (xi , yi )V ∗ (i , i ), xi2 ⫹ yi2 ⱕ 1 i
where i runs over all the image pixels. The computation of the corresponding moments of an image considers the center of the image as the origin and pixels are mapped into the unit circle, that is, xi2 ⫹ yi2 ⱕ 1. The pixels falling outside the unit circle are not taken into consideration. The magnitude of the Zernike moments is invariant to rotations [Teag 80] (Problem 7.8). Translation and scaling invariance is treated in [Khot 90a, Chon 03]. A drawback of the Zernike moments is the computational complexity associated with the computation of the radial polynomials. A common approach used in reducing complexity includes the application of recurrence relations between successive radial polynomials and coefficients. Computational aspects of the Zernike moments are examined in [Muku 95, Wee 06, Huan 06]. Numerical error issues associated with the computations of the Zernike moments are treated in [Sing 06]. Comparative studies of the performance of the Zernike moments against the moments of Hu, in the context of character recognition, have demonstrated that the former behave better, especially in noisy environments [Khot 90b]. In [Wang 98], Zernike moments are used to cope with both geometry and illumination invariance, in the context of multispectral texture classification. Variants of the Zernike moments, called pseudo-Zernike moments, have also been proposed and used. Comparative studies can be found in [Teh 88, Heyw 95]. Besides Zernike moments, other types of moments have also been suggested and used, such as the Fourier–Mellin moments and moments based on Legendre polynomials, as in [Kan 02, Chon 04, Muku 98].
7.2.4 Parametric Models So far, in various parts of the book, we have treated the gray levels as random variables and looked at aspects of their first- and second-order statistics. In this subsection, their randomness will be approached from a different perspective. We will assume that I (m, n) is a real nondiscrete random variable, and we will try to model its underlying generation mechanism by adopting an appropriate parametric model. The parameters of the resulting models encode useful information and
427
“09-Ch07-SA272” 18/9/2008 page 428
428
CHAPTER 7 Feature Generation II
lend themselves as powerful feature candidates for a number of pattern recognition tasks. We will move in two directions. One is to treat an image as a successive sequence of rows or columns. That is, our random variables will be considered as successive realization samples from a one-dimensional random process I (n). The alternative looks at the image as a two-dimensional random process I (m, n), also known as random field.
One-Dimensional Parametric Models Let I (n) denote the random sequence. We will assume that it is stationary in the wide sense. This means that its autocorrelation sequence r(k) exists and is of the form r(k) ⫽ E[I (n)I (n ⫺ k)]
and the Fourier transform of r(k) also exists and is a positive function (power spectral density) I () ⫽
⫹⬁
r(k) exp(⫺jk)
k⫽⫺⬁
Under certain assumptions, which are met in practice most of the time [Papo 91, Theo 93], it can be shown that such a random sequence can be generated at the output of a linear, causal, stable, time-invariant system with impulse response h(n), whose input is excited by a white noise sequence, as shown in Figure 7.6. In simple terms, this means that we can write I (n) ⫽
⬁
h(k)(n ⫺ k)
k⫽0
where h(n) satisfies the stability condition n |h(n) | ⬍ ⬁. The sequence (n) is a white noise sequence, that is, E[(n)] ⫽ 0 and E[(n)(n ⫺ l)] ⫽ 2 ␦(l) : ␦(l) ⫽ 1 for l ⫽ 0 and zero otherwise. Such processes of a special type are the so-called autoregressive processes (AR), which are generated by systems of the form I (n) ⫽
p
a(k)I (n ⫺ k) ⫹ (n)
(7.26)
k⫽1
(n)
h(n)
I(n)
FIGURE 7.6 Generation model of a stationary random process at the output of a stable, linear, time-invariant system excited by a white noise sequence.
“09-Ch07-SA272” 18/9/2008 page 429
7.2 Regional Features
(n)
I(n)
⫹
⌺a(k)I(n⫺k)
FIGURE 7.7 Generation model of an AR stationary random process.
In words, the random sequence I (n) is given as a linear combination of previous samples I (n ⫺ k) and the current input sample (n) (Figure 7.7). Here p is the order of the AR model, and we write AR(p). The coefficients a(k), k ⫽ 1, 2, . . . , p are the AR model parameters. AR models are a special case of a more general class of models, known as autoregressive–moving average (ARMA(p,m)), for which I (n) ⫽
p
a(k)I (n ⫺ k) ⫹
k⫽1
m
b(l)(n ⫺ l)
(7.27)
l⫽0
That is, the model is regressive with respect to both input and output sequences. The major advantage of the AR models, compared with their ARMA relatives, is that the former lead to linear systems of equations for the estimation of the model parameters.
Estimation of the AR Parameters Another way to look at (7.26) is to interpret the coefficients a(k), k ⫽ 1, . . . , p, as the predictor parameters of the sequence I (n). That is, the parameters weigh previous samples, I (n ⫺ 1), . . . , I (n ⫺ p), in order to predict the value of the current sample I (n), and (n) is the prediction error, Iˆn ⫽
p
a(k)I (n ⫺ k) ≡ a Tp I p (n ⫺ 1)
(7.28)
k⫽1
where I Tp (n ⫺ 1) ≡ [I (n ⫺ 1), . . . , I (n ⫺ p)]. The unknown parameter vector a Tp ⫽ [a(1), a(2), . . . , a(p)] is optimally estimated, for example, by minimizing the mean square prediction error, E[2 (n)] ⫽ E[(I (n) ⫺ Iˆ (n))2 ] ⫽ E[(I (n) ⫺ a Tp I p (n ⫺ 1))2 ]
(7.29)
The problem is exactly the same as that of the mean square linear classifier estimation of Chapter 3, and the unknown parameters result from the solution of E[I p (n ⫺ 1)I Tp (n ⫺ 1)]a p ⫽ E[I (n)I p (n ⫺ 1)]
(7.30)
429
“09-Ch07-SA272” 18/9/2008 page 430
430
CHAPTER 7 Feature Generation II
or ⎡
r(0)
r(⫺1)
...
⎢ ⎢ r(1) ⎢ ⎢ .. ⎢ ⎢ . ⎢ ⎢r(p ⫺ 2) ⎣
r(0) .. .
... .. .
r(p ⫺ 3)
...
r(p ⫺ 1)
r(p ⫺ 2)
...
⎤⎡ ⎤ ⎡ ⎤ r(⫺p ⫹ 1) a(1) r(1) ⎥⎢ ⎥ ⎢ ⎥ r(⫺p ⫹ 2)⎥ ⎢ a(2) ⎥ ⎢ r(2) ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ .. .. .. ⎥⎢ ⎥⫽⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ . . . ⎥⎢ ⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ r(⫺1) ⎦ ⎣a(p ⫺ 1)⎦ ⎣r(p ⫺ 1)⎥ ⎦ r(0)
a(p)
r(p)
or Ra p ⫽ r p
(7.31)
with r p ≡ [r(1), . . . , r(p)]T . The relation of the optimal parameters a(k) with the mean square error (variance of generating noise) is obtained from (7.29) and (7.31) and is given by 2 ⫽ E[2 (n)] ⫽ r(0) ⫺
p
a(k)r(k)
(7.32)
k⫽1
The autocorrelation matrix has a computationally rich structure. It is symmetric (r(k) ⫽ r(⫺k)) and Toeplitz—that is, all the elements across any of its diagonals are the same. Exploitation of these properties leads to the development of a computationally efficient scheme for the solution of (7.31). This is Levinson’s algorithm, which solves the linear system of equations in O(p2 ) multiplications and additions, as opposed to O(p3 ) required by more classical algorithmic schemes [Theo 93, Hayk 96]. In Chapter 3, we saw that when the autocorrelation sequence is not known, it is often preferable to adopt the least sum of squares instead of the mean square criterion. Then the AR parameters are still provided by a linear system of equations, but the associated matrix is no longer Toeplitz. However, it is still computationally rich, and Levinson-type O(p2 ) algorithms for the efficient solution of such systems have also been derived [Theo 93]. Besides images,AR (ARMA) models have been used extensively to model other type of random sequences,such as those resulting from digitizing speech signals and electroencephalographic signals. For all these cases the resulting AR parameters can be used as features to classify one type of signal from another. Example 7.4 Let the AR random sequence of order p ⫽ 2 be I (n) ⫽
2
a(k)I (n ⫺ k) ⫹ (k)
k⫽1
with r(0) ⫽ 1, r(1) ⫽ 0.5, r(2) ⫽ 0.85. Computing the mean square estimates of a(k), k ⫽ 1, 2, we obtain
“09-Ch07-SA272” 18/9/2008 page 431
7.2 Regional Features
1 0.5
0.5 1
a(1) 0.5 ⫽ a(2) 0.85
and its solution gives a(1) ⫽ 0.1, a(2) ⫽ 0.8.
Two-Dimensional AR Models A two-dimensional AR random sequence I (m, n) is defined as Iˆ (m, n) ⫽
k
a(k, l)I (m ⫺ k, n ⫺ l), (k, l) ∈ W
(7.33)
l
I (m, n) ⫽ Iˆ (m, n) ⫹ (m, n)
(7.34)
Figure 7.8 shows the region W of the pixels that contribute to the prediction of Iˆ (m, n), for a number of possible choices. The case in Figure 7.8a corresponds to what is known as a strongly causal predictor model. This is because all pixels in the contributing area have coordinates smaller than the coordinates m, n of the predicted pixel, which is represented by the unshaded node in the figure. The corresponding window is W1 ⫽ {0 ⱕ k ⱕ p, 0 ⱕ l ⱕ q, (k, l) ⫽ (0, 0)}. However, the notions of past and present have no real meaning for an image, and alternative windows can also be used. A noncausal predictor is defined as I (m, n) ⫽
q p
a(k, l)I (m ⫺ k, n ⫺ l) ⫹ (m, n)
k⫽⫺p l⫽⫺q
In Figure 7.8d the corresponding window is shown for the case of p ⫽ q ⫽ 2. Figure 7.8c shows a third possibility, which is known as a semicausal predictor, and Figure 7.8b shows the case of a causal predictor. Next we summarize the last three cases, which are the most common in practice: Causal : W2 ⫽ {(⫺p ⱕ k ⱕ p, 1 ⱕ l ⱕ q) ∪ (1 ⱕ k ⱕ p, l ⫽ 0)} Semicausal : W3 ⫽ {⫺p ⱕ k ⱕ p, 0 ⱕ l ⱕ q , (k, l) ⫽ (0, 0)} Noncausal : W4 ⫽ {⫺p ⱕ k ⱕ p, ⫺q ⱕ l ⱕ q , (k, l) ⫽ (0, 0)}
AR Parameter Estimation We have Iˆ (m, n) ⫽
k
a(k, l)I (m ⫺ k, n ⫺ l)
l
Recalling the orthogonality condition from Chapter 3, in its two-dimensional generalization, we obtain that the minimum mean square error solution satisfies
E I (m ⫺ i, n ⫺ j) I (m, n) ⫺
k
l
a(k, l)I (m ⫺ k, n ⫺ l)
⫽ 0,
(i, j) ∈ W
(7.35)
431
“09-Ch07-SA272” 18/9/2008 page 432
432
CHAPTER 7 Feature Generation II
m
n
m
n (a)
(b) m
n
m
n (c)
(d)
FIGURE 7.8 Different types of two-dimensional prediction models. The predicted pixel is represented by the unshaded node. The red pixels are those that take part in the prediction and the corresponding window W is the area enclosed by the dotted line. (a) Strictly causal, (b) causal, (c) semicausal, and (d) noncausal.
or r(i, j) ⫽
k
a(k, l)r(i ⫺ k, j ⫺ l),
(i, j) ∈ W
(7.36)
l
where r(i, j) ≡ E[I (m, n)I (m ⫺ i, n ⫺ j)] is the two-dimensional autocorrelation sequence of the random field I (m, n). The set of equations in (7.36) constitutes a linear system of equations leading to the estimates of a(k, l). The associated matrix also has a computationally rich structure, which can be exploited to develop efficient schemes to compute the solution. Let us take, for example, the noncausal window for p ⫽ q. This is a symmetric window, in the sense that for each index pair (i, j), the (⫺i, ⫺j) is also present. Combining (7.36) with the equation of the
“09-Ch07-SA272” 18/9/2008 page 433
7.2 Regional Features
variance of the minimum error, which is given by (Problem 7.11) 2 ⫽ r(0, 0) ⫺
k
a(k, l)r(k, l)
(7.37)
l
the following system results: ⎤ 0 ⎢ ⎥ Ra ⫽ ⫺ ⎣n2 ⎦ 0 ⎡
(7.38)
where 0 is the zero vector of appropriate dimension and a T ⫽ [a(p, p), . . . , a(p, ⫺p), . . . , a(0, 0), . . . , a(⫺p, p), . . . , a(⫺p, ⫺p)]
where a(0, 0) ≡ 1 and R is the corresponding autocorrelation matrix. The dimension of a is (2p⫹1)2 . The correlation of a homogeneous (i.e.,E[I (m, n)I (m ⫺ k, n⫺l)] ⫽ r(k, l)) and isotropic (no direction dependence) image depends only on the relative distance between pixels, r(k, l) ⫽ r
√ k2 ⫹ l 2
and the resulting autocorrelation matrix is easily shown to be symmetric and block Toeplitz with each block being itself a Toeplitz matrix, ⎡
R0 ⎢R ⎢ 1 R⫽⎢ ⎢ .. ⎣ . R2p
where
⎡
R1 R0 .. . R2p⫺1
r(i, 0) ⎢ . ⎢ Ri ⫽ ⎣ .. r(i, 2p)
... ... .. . ...
... .. . ...
⎤ R2p R2p⫺1 ⎥ ⎥ .. ⎥ ⎥ . ⎦ R0
⎤ r(i, 2p) .. ⎥ ⎥ . ⎦ r(i, 0)
(7.39)
(7.40)
For homogeneous images and symmetric windows it is easy to show that the AR parameters are symmetric a(k, l) ⫽ a(⫺k, ⫺l) and the system can be solved efficiently by a Levinson-type algorithm [Kalo 89]. If the image is homogeneous but anisotropic, the resulting system’s associated matrix is block Toeplitz, but the elements are no longer Toeplitz. Furthermore, more general windows than the ones introduced in this section have also been suggested and used. Efficient Levinsontype algorithms for such cases have also been developed (e.g., [Glen 94]). Finally, besides the squared error criteria,maximum likelihood techniques can be employed for the estimation of the unknown parameters,which can lead to more accurate estimates. Of course, in such cases assumptions about the underlying statistics have to be adopted (e.g., [Kash 82]).
433
“09-Ch07-SA272” 18/9/2008 page 434
434
CHAPTER 7 Feature Generation II
Remarks ■
The AR modeling of images has been used in the classification context in a number of cases [Chel 85, Cros 83, Kash 82, Sark 97]. In [Kash 86, Mao 92] extensions have been proposed for rotation-invariant models.
■
The AR random field models are related to a class of models known as Markov random fields. The essence of these fields is that for each pixel (m, n) the image is divided into three areas: ⍀⫹ (“future”), ⍀ (“present”), and ⍀⫺ (“past”). It is then assumed that the random variable I (m, n), (m, n) ∈ ⍀⫹ , is independent of its values in ⍀⫺ and depends only on the values in ⍀; thus, the conditional density function satisfies p I (m, n), (m, n) ∈ ⍀⫹ |I (m, n), (m, n) ∈ ⍀⫺ ∪ ⍀ ⫹ ⫽ p I (m, n), (m, n) ∈ ⍀ |I (m, n), (m, n) ∈ ⍀
In words, the“future”depends only on the“present”and not on the“past”; that is, the value of the random variable at a pixel depends on the values that the random variable takes in a specific (neighboring) area only, and it does not depend on the values in the remaining regions of the image. ■
It can be shown that every Gaussian AR model is a Markov random field. [Wood 72, Chel 85].
Example 7.5 For an image whose autocorrelation sequence obeys r(k, l) ⫽ 0.8
√
k2 ⫹l 2
estimate the AR parameters for a noncausal p ⫽ q ⫽ 1 window. From the definition we have Iˆ (m, n) ⫽ a(1, 1)I (m ⫺ 1, n ⫺ 1) ⫹ a(1, 0)I (m ⫺ 1, n) ⫹ a(1, ⫺1)I (m ⫺ 1, n ⫹ 1) ⫹ a(0, 1)I (m, n ⫺ 1) ⫹ a(0, ⫺1)I (m, n ⫹ 1) ⫹ a(⫺1, 1)I (m ⫹ 1, n ⫺ 1) ⫹ a(⫺1, 0)I (m ⫹ 1, n) ⫹ a(⫺1, ⫺1)I (m ⫹ 1, n ⫹ 1) The resulting matrix R is a block (2p ⫹ 1) ⫻ (2p 3 ⫻ 3 matrices ⎡ R0 R1 ⎢ R ⫽ ⎣ R1 R0 R2 R1 where
⎡
r(0, 0) ⎢ R0 ⫽ ⎣r(0, 1) r(0, 2)
⫹ 1) ⫽ 3 ⫻ 3 matrix with elements the
r(0, 1) r(0, 0) r(0, 1)
⎤ R2 ⎥ R1 ⎦ R0 ⎤ r(0, 2) ⎥ r(0, 1)⎦ r(0, 0)
“09-Ch07-SA272” 18/9/2008 page 435
7.3 Features for Shape and Size Characterization
⎡
r(1, 0) ⎢ R1 ⫽ ⎣r(1, 1) r(1, 2) ⎡ r(2, 0) ⎢ R2 ⫽ ⎣r(2, 1) r(2, 2)
r(1, 1) r(1, 0) r(1, 1) r(2, 1) r(2, 0) r(2, 1)
⎤ r(1, 2) ⎥ r(1, 1)⎦ r(1, 0) ⎤ r(2, 2) ⎥ r(2, 1)⎦ r(2, 0)
For this specific model the linear system in (7.38) has nine unknowns and the solution gives a(1, 1) ⫽ a(⫺1, ⫺1) ⫽ ⫺0.011,
a(1, 0) ⫽ a(⫺1, 0) ⫽ ⫺0.25
a(1, ⫺1) ⫽ a(⫺1, 1) ⫽ ⫺0.011,
a(0, 1) ⫽ a(0, ⫺1) ⫽ ⫺0.25
2 ⫽ 0.17
7.3 FEATURES FOR SHAPE AND SIZE CHARACTERIZATION In a number of image analysis applications, an important piece of information is the shape and size of an object of interest within the image. For example, in medical applications the shape and size of nodules are crucial in classifying them as malignant, or benign. Nodules with an irregular boundary have a high probability of being malignant, and those with a more regular boundary are usually benign. Also, it has been observed that in certain cases nodules with a perimeter of more than 3 cm are usually malignant [Cavo 92]. Another example in which the shape of the object is of major importance is the automatic character recognition in an optical character recognition (OCR) system [Mori 92, Plam 00, Vinc 02]. Although OCR systems employing our already familiar regional features, there is a large class of techniques that use the shape information residing in the boundary curve of the characters. Figure 7.9a shows the character “5” as seen from the scanner of an OCR system. An appropriate image segmentation algorithm (e.g., [Pita 94]) has first been applied to separate the character from the rest of the image. The character in Figure 7.9b is in binary form. This is the result of the binarization phase, in which all gray
(a)
(b)
(c)
FIGURE 7.9 The character “5” after (a) the segmentation of the scanned image and then (b) the application of a binarization algorithm and (c) its boundary after the application of a boundary extraction algorithm in the binarized version.
435
“09-Ch07-SA272” 18/9/2008 page 436
436
CHAPTER 7 Feature Generation II
levels of the character region below a certain threshold become 0 and all above it become 1 [Trie 95]. Figure 7.9c shows the resulting boundary, after the application of a boundary extraction algorithm (e.g., [Pita 94]) on the binary version. Thus, in the last version there is no texture of interest inside the character. What is of paramount importance in such systems is feature invariance in geometric transformations. The recognition of the character must be insensitive to its position, size, and orientation. A review of various methodologies for invariant pattern recognition techniques can be found in [Wood 96]. The shape characterization of a region or an object can be achieved in various ways. Two are the major directions along which we will proceed. One is to develop techniques that provide a full description of the boundary of the object in a regenerative manner. In words, the boundary can be reobtained from the description coefficients, such as by using a Fourier expansion of the boundary, which in turn can be reconstructed from its Fourier coefficients. The other direction is to use features that are descriptive of the characteristics of the shape of the region but are not regenerative. Examples of such features are the number of corners in the boundary and the perimeter. They provide useful information about the boundary, but they are not sufficient to reproduce it. In the following we will focus on some basic techniques, which have in turn given birth to a large number of variants shaped to fit specific application requirements (for example, see [Trie 96] for a review).
7.3.1 Fourier Features Let (xk , yk ), k ⫽ 0, 1, . . . , N ⫺ 1, be the coordinates of N samples on the boundary of an image region, Figure 7.10a. For each pair (xk , yk ) we define the complex variable uk ⫽ xk ⫹ jyk
For the N uk points we obtain the DFT fl fl ⫽
N ⫺1 k⫽0
2 lk , uk exp ⫺j N
y
l ⫽ 0, 1, . . . , N ⫺ 1
y Rmax
Rmin
yk
t
y0 x0
o
xk (a)
x
FIGURE 7.10 Boundary of an image region (a) and associated parameters (b).
(b)
x
“09-Ch07-SA272” 18/9/2008 page 437
7.3 Features for Shape and Size Characterization
The coefficients fl are also known as the Fourier descriptors of the boundary. Once the fl are available, the uk can be recovered and the boundary can be reconstructed. However, our goal in pattern recognition is not to reconstruct the boundary. Thus, a smaller number of coefficients (or descriptors) is usually used, enough to include sufficient discriminatory information. In the sequel, we will investigate how the Fourier descriptors are affected by the actions of translation, rotation, and scaling. For translation we have x⬘k ⫽ xk ⫹ ⌬x y⬘k ⫽ yk ⫹ ⌬y
Then u⬘k ⫽ uk ⫹ (⌬x ⫹ j⌬y) ≡ uk ⫹ ⌬u⬘
For rotation, it is not difficult to verify that in rotating all points of the region by , with respect to the origin, the rotated coordinates correspond to (Problem 7.13) u⬘k ⫽ uk exp j
If fl , fl⬘ are the DFTs of uk , u⬘k , respectively, then from the DFT definition we get Translation : u⬘k ⫽ uk ⫹ ⌬u⬘ Rotation : u⬘k ⫽ uk exp j Scaling : u⬘k ⫽ auk
⇒ fl⬘ ⫽ fl ⫹ ⌬u⬘␦(l) ⇒ fl⬘ ⫽ fl exp j ⇒ fl⬘ ⫽ afl
Translation of the sampling origin : u⬘k ⫽ uk⫺k0
⇒
l fl⬘ ⫽ fl exp ⫺j2k0 N
In words, translation affects only the f0⬘ coefficient. Rotation affects the phase of all the coefficients by the same factor, and it has no effect on their magnitude. Scaling affects all coefficients in the same way, and thus it has no effect on the ratios ffij . The sampling point origin, within the boundary, affects the phase but leaves invariant the magnitude |fl |. This deterministic manner, in which the three geometric transformations affect the Fourier coefficients, allows the development of appropriate normalized versions that are invariant to these actions [Crim 82, Arbt 90, Gran 72]. Let us demonstrate the rationale of such approaches via an example, by considering the boundary of an object. The first decision to be taken, prior to the computation of the Fourier coefficients, is to define the first sampling point (x0 , y0 ) on the boundary. In practice, the choice of this point for each character has a degree of randomness. The choice of a different sampling origin corresponds to a relative translation of, say, k0 ⬍ N samples (since the boundary is a closed curve, the relative
437
“09-Ch07-SA272” 18/9/2008 page 438
438
CHAPTER 7 Feature Generation II
translation will always be (k ⫺ k0 ) modulo N ⬍ N ). As we have seen earlier, this affects the Fourier descriptors u⬘k ⫽ uk⫺k0
hence
k0 f1⬘ ⫽ f1 exp ⫺j2 N
l ⇒ fl⬘ ⫽ fl exp ⫺j2k0 N
(7.41)
k0 ⇒ f1⬘ ⫽ | f1 | exp(⫺j1 ) exp ⫺j2 N
where |f1 |, 1 are the magnitude and phase of f1 , respectively. Hence, the phase of f1 ⬘ is 1 ⬘ ⫽ 1 ⫹ 2 kN0 . In the sequel we define the following normalized Fourier coefficients: fˆl ⫽ fl exp( jl1 )
(7.42)
The corresponding normalized coefficient with shifted origin will be fˆ ⬘l ⫽ fl ⬘ exp( jl⬘1 ) ⫽ fl ⬘ exp
jl1 ⫹ j2k0
l N
(7.43)
Taking into account (7.41), we obtain fˆ ⬘l ⫽ fˆl
Thus,the preceding normalization generates features that are invariant to the choice of the sampling origin (x0 , y0 ). This method of exploiting the power of the Fourier transform as a tool for boundary description is not the only possibility. An alternative is to express the coordinates of the boundary contour points as functions of the boundary length t, measured from an origin within the boundary, that is, (x(t), y(t)). Since the boundary is a closed curve, these are periodic functions and they can be expanded in their Fourier series. Invariant versions of the Fourier coefficients can then be computed and used as features for pattern recognition [Kuhl 82, Lin 87]. Comparative performance studies of a number of invariant Fourier-based features, in the context of handwritten character recognition, can be found in [Pers 77, Taxt 90]. Another way is to generate Fourier descriptors from the curvature k(t) function of the boundary, defined as k(t) ⫽
d(t) dt
where (t) is the tangent angle (Figure 7.10b) at a point a distance t from the origin, which is marked “o” in the figure. Such a description is justified by Gauss’s theorem, stating that every curvature function corresponds to one and only one curve in space (with the exception of its position in space). The advantage of such a description stems from its obvious scale invariance property. If we measure the length of the boundary at a point by the number of pixels n
“09-Ch07-SA272” 18/9/2008 page 439
7.3 Features for Shape and Size Characterization
between this point and the origin of the curve, the curvature of the boundary is approximated by n ⫽ tan⫺1
yn⫹1 ⫺ yn , n ⫽ 0, 1, . . . , N ⫺ 1 xn⫹1 ⫺ xn
kn ⫽ n⫹1 ⫺ n ,
(7.44)
n ⫽ 0, 1, . . . , N ⫺ 1
In the previous chapter we have seen that an alternative to Fourier descriptors is to use wavelet coefficients. However, as we pointed out there, the definition of invariant wavelet descriptors is not a straightforward task, and invariance is attempted via indirect methods.
7.3.2 Chain Codes Chain coding is among the most widely used techniques for boundary shape description. In [Free 61],the boundary curve is approximated via a sequence of connected straight line segments of preselected direction and length. Every line segment is coded with a specific coding number depending on its direction. In Figure 7.11 two possible choices, usually encountered in practice, are shown. In this way a chain code [di ] is created, where di is the coding number of the direction of the line segment that connects boundary pixel (xi , yi ) with the next one (xi⫹1 , yi⫹1 ), sweeping the boundary in, say, the clockwise sense. A disadvantage of this description is that the resulting chain codes are usually long and at the same time are very sensitive in the presence of noise. This leads to chain codes with variations due to noise and not necessarily to the boundary curve. A way out is to resample the boundary curve by selecting a grid of larger dimensions. For each of the boxes of the grid all points inside a box are assigned the value of the respective box center. In Figure 7.12a the original samples are shown alongside the larger sampling grid. Figure 7.12b is the resulting resampled version. The chain
1
2
3
0
3 (a)
2
0
4
5
1
6
7
(b)
FIGURE 7.11 Directions for a (a) four-directional chain code and (b) an eight-directional chain code.
439
“09-Ch07-SA272” 18/9/2008 page 440
440
CHAPTER 7 Feature Generation II
1 2 2 2 2 3
4 4 4 4 4 5 6 6 0 0 0 0 7 7 4 4 4 4 3 1
1 3
(a)
(b)
7
0 0 0 0 0 0
0 0 0 0 4 4 4 4 4
6 6 6
2 5 5
(c)
FIGURE 7.12 The character “5” and (a) its original sampled image, (b) its resampled version on a coarser grid, and (c) the resulting chain code.
code is formed from the sequential connection of these pixels, Figure 7.12c. If we consider the length of the grid side as the basic measurement unit, then for evencoded directions, 0, 2, 4, 6, the length of the corresponding straight line segment √ is 1, and for the odd-coded directions, 1, 3, 5, 7, it is 2 (from the Pythagoras theorem). For the case of Figure 7.12b and for a coding with eight possible directions, the resulting chain code is shown in the Figure 7.12c. This sequence of numbers constitutes the spine on which a number of shape-related features are built. Two possibilities, for example, are the following [Lai 81, Mahm 94].
Direction and Direction Length Features For each direction we count the number of times a specific chain code number appears in the chain. Then this number is divided by the total number of chain codes that appear in the chain description of the boundary. For an eight-code scheme this procedure gives rise to eight features (one for each direction). Another way that also provides eight features is to divide the total length of the line segments in each direction by the total length of the boundary line.
Curvature Features These features quantify concave (smaller than 180◦ ) and convex (larger than 180◦ ) external angles between adjacent edges at the corners of the polygon that is formed by the line segments when the boundary curve is scanned in the clockwise sense. Figure 7.13 shows the possible cases. For example, successive directions 01, 02, 23, 71 correspond to concave external angles, and the pairs 06, 41, 64, 75 to convex angles. The occurrence percentage of each of these cases in the chain code defines a respective feature. Sometimes chain code pairs are grouped according to whether the first chain code is even or odd. Thus, a total of 16 features are generated, 8 for the convex and 8 for the concave case.
“09-Ch07-SA272” 18/9/2008 page 441
7.3 Features for Shape and Size Characterization
01
02
03
45
46
47
12
13
14
50
56
57
23
24
25
60
61
67
34
35
36
70
71
72
(a) Concave features
05
06
07
41
42
43
10
16
17
52
53
54
20
21
27
63
64
65
30
31
32
74
75
76
(b) Convex features
FIGURE 7.13 Curvature features characterizing the boundary polygon that results from an eight-directional chain code description of the boundary.
7.3.3 Moment-Based Features In (7.21) and (7.25) the geometric moments and central moments were defined. If in the place of I (i, j) we consider the sequence I (i, j) ⫽
1
(i, j) ∈ C
0
(i, j) otherwise
where C is the set of points (i, j) inside the object of interest, then we obtain a way to describe the shape of the object through the moments. Indeed, in such a case only the limits in the summations (hence the object’s shape) are taken into account, whereas the details inside the object (i.e., texture) do not participate. Hence mpq ⫽
i
ip j q ,
(i, j) ∈ C
j
with m00 ⫽ N , the total number of pixels inside the region. The features x¯ ⫽
m10 m00
and
y¯ ⫽
m01 m00
441
“09-Ch07-SA272” 18/9/2008 page 442
442
CHAPTER 7 Feature Generation II
define the center of mass ( x, ¯ y¯ ). The respective central moments become pq ⫽
i
(i ⫺ x) ¯ p ( j ⫺ y¯ )q ,
(i, j) ∈ C
j
The invariant moments can in turn be computed and used, whenever appropriate. Two useful quantities that are related to these moments and provide useful discriminatory information are: 1. Orientation ⫽
211 1 tan⫺1 2 20 ⫺ 02
which is the angle between the axis with the least moment of inertia and the x-coordinate axis (Problem 7.18). 2. Eccentricity ⑀⫽
(20 ⫺ 02 )2 ⫹ 411 area
Another representation of the eccentricity is via the ratio RRmax of the maximin mum to the minimum distance of the center of mass ( x, ¯ y¯ ) from the object’s boundary (Figure 7.10b).
7.3.4 Geometric Features The features of this subsection are derived directly from the geometry of the object’s shape. The perimeter P of the object and its area A are two widely used features. If x i , i ⫽ 1, 2, . . . , N , are the samples of the boundary, then the perimeter is given by P⫽
N ⫺1
x i⫹1 ⫺ x i ⫹ x N ⫺ x 1
i⫽1
If we consider the area of a pixel as the measuring unit, a straightforward way to compute the area enclosed by a boundary is by counting the number of pixels inside the region of the object. The roundness ratio is a third quantity, defined as ␥⫽
P2 4A
A useful feature that is related to the curvature of the boundary, as defined in (7.44), is the so-called bending energy at a point n, given by E(n) ⫽
n⫺1 1 |ki |2 P i⫽0
“09-Ch07-SA272” 18/9/2008 page 443
7.3 Features for Shape and Size Characterization
Another popular feature is the number of corners in the boundary contour. These correspond to points where the curvature ki takes large values (infinity in theory). In [Ghos 97] corners as well as other topological features are detected via the use of Zernike moments and appropriate parametric modeling of the respective topological image intensity profile. The number of holes inside the region of an object is another useful quantity. For example, a large error percentage in handwriting character recognition tasks is related to the difficulty of the classifiers in distinguishing “8” from “0,” because their boundaries look alike. The detection of the presence of holes inside the object, using appropriate algorithms, is extra information that can be beneficially used for recognition [Lai 81, Mahm 94]. In our study so far, we have demonstrated how to derive geometric features from the boundary curve. However, this is not the only possibility. For example, in [Wang 98] geometric features are extracted directly from the gray-level variation within the image region. In this way, the binarization phase is avoided, which in some cases can become tricky. Another direction that has been used extensively in OCR is to work on the thinned version of the binarized character. Figure 7.14 illustrates the procedure via an example. Figure 7.14b is the result of the application of a thinning algorithm (e.g., [Pita 94]) on the binary version of the character “5” of Figure 7.14a. Also in Figure 7.14b the so-called key points are denoted. These can be node points where one or more lines (strokes) of the character are crossed or corner points or end points. These can be computed by processing neighboring pixels. For example, in order to identify an end point, we look at its eight neighboring pixels. An end point has only one neighbor at gray-level 1, and the rest are 0. In the sequel, the thinned version of the character is simplified as a set of line segments (edges) connecting the key points, Figure 7.14c. Each edge can then be characterized by its direction, for example, using the chain code; its length, for example, long or short; and its relation to its neighboring edges. In the sequel each character is described by an array providing this information in a coded form. Classification is then based on these coded matrices by defining appropriate costs. The interested reader may consult for example [Lu 91, Alem 90] for more details.
(a)
(b)
(c)
FIGURE 7.14 (a) Binarized version of 5 (b) the thinned version with the key points and (c) version with edges connecting key points.
443
“09-Ch07-SA272” 18/9/2008 page 444
444
CHAPTER 7 Feature Generation II
7.4 A GLIMPSE AT FRACTALS We have already seen that the 1980s was the decade in which two major tools were introduced into the realm of pattern recognition applications (among others): neural networks and wavelets. The same decade was also the time when another tool was adopted in many application areas to offer its potential power. Fractals and fractal dimension have become the focus of considerable research effort. This section aims at giving the basic definitions and outlining the basic concepts behind the use of fractals in pattern recognition. A deeper study of the area is beyond the goals of a short section, and the interested reader may refer to a number of books and articles available [Mand 77, Tson 92, Falc 90].
7.4.1 Self-Similarity and Fractal Dimension Let us consider the straight-line segment of length L in Figure 7.15a. Divide L into N (two for the example of the figure) equal parts of length l. Each of the resulting parts is still a straight-line segment, and its length has been scaled down by a factor m ⫽ Ll ⫽ N1 . Magnification of any of these parts by the same factor will reproduce the original line segment. We refer to such types of structures as self-similar. If instead of a straight-line segment we had a square of side L (Figure 7.15b), then scaling
L/2 L
L/2
L
(b)
(a)
(c)
(d)
(e)
FIGURE 7.15 Self similar structures (a) line segment, (b) square, (c)–(e) three stages in the generation of Koch’s curve.
“09-Ch07-SA272” 18/9/2008 page 445
7.4 A Glimpse at Fractals
down the side by m ⫽ N 11/2 would result into N square parts. Each part looks like the original square,which can be reobtained from the parts after magnification. The same is true for all dimensions, where N similar parts result after scaling the sides 1 , D ⫽ 1, 2, . . .. That is, the Euclidean of the D-dimensional (hyper)cube by m ⫽ N 1/D dimension D is directly related to the scaling and the number N of the resulting self-similar parts. We can write N⫽
D L ≡ m⫺D l
(7.45)
If we now want to measure the length (area,volume) of the original segment (hypercube) using as a measurement unit a scaled element of length l (l 2 , l 3 , etc.), then the result is independent of the size l of the measuring unit. Indeed, the resulting metric property (length, area, etc.) is given by M ⫽ N (l)l D
(7.46)
where N (l) is the number of parts that cover the curve (area, etc.) to be measured and l is the size of the measuring unit. Combination of (7.45) and (7.46) leads to a metric M that is always constant (LD ) for the same structure, and it is independent of the size l of the chosen unit, as expected. Let us now turn our attention to some more interesting structures, such as the one in Figure 7.15d. The curve in Figure 7.15d results from the straight-line segment of Figure 7.15c,known as the initiator,by the following strategy:(a) divide the segment into three equal parts and (b) replace the central one by the two sides of an equilateral triangle, with sides of size equal to the size of the scaled parts. The procedure is then repeated for each of the line segments in Figure 7.15d, and this results in the structure of Figure 7.15e. This process can go on indefinitely, and the limit curve is the so-called Koch curve [Mand 77]. Such a curve is everywhere continuous but nowhere differentiable. It is readily observed that at each step of the iteration, the resulting structure is part of the structure that will result in the next iteration, after a scaling by 3. The curve therefore has a self-similar structure. In the sequel we will try to measure the length of the curve. Using as a (measuring) unit a segment of length l ⫽ 3L (i.e., Figure 7.15d), the resulting length is 4. For a unit segment l ⫽ 3L2 (i.e., Figure 7.15e), the measured length is 42 . It is not difficult to see that the length keeps increasing with decreasing unit size and tends to infinity as the size of the measuring unit tends to zero! That is, the length of the curve depends not only on the curve itself but also on the adopted measurement unit! This strange result is the outcome of an “unfair” measurement process. Indeed, in the case of the Koch curve, scaling by 3 results in four similar parts. In contrast, in the case of a straight-line segment, scaling by m ⫽ N1 results in the same number N of similar parts. In higher dimensional Euclidean space, scaling by m ⫽ N ⫺1/D results in N parts. The measurement process involves this number N , the scaled side length l, and the Euclidean dimension D, as (7.46) suggests. From this discussion, the Euclidean dimension can also be seen as the ratio ⫺lnlnNm ⫽ D. Starting from this observation, let us now define the similarity dimension of a
445
“09-Ch07-SA272” 18/9/2008 page 446
446
CHAPTER 7 Feature Generation II
general self-similar structure as D⫽
ln N ⫺ ln m
(7.47)
where N is the number of the resulting similar parts, when scaling by a factor m. For hypercube structures the similarity dimension is the respective Euclidean dimension, which is an integer. In contrast, the corresponding similarity dimension 4 of the Koch curve D ⫽ ⫺ ln is a fraction and not an integer. Such structures ln( 1 ) 3
are called fractals, and the corresponding similarity dimension is called a fractal dimension. Measuring a fractal structure, we can adopt (7.46) with the corresponding fractal dimension in the place of D. The result of the measurement process now becomes independent of the measuring tool l. Indeed, it is easy to see that using the definition in (7.47), (7.46) results in a constant M ⫽ LD for m ⫽ Ll . The use of similarity dimension, therefore, results in a consistent description of the metric properties of such self-similar structures. For a deeper treatment and other definitions of the dimension the interested reader may consult more specialized texts (e.g., [Falc 90]).
7.4.2 Fractional Brownian Motion A major part of our effort in this chapter was dedicated to the description of statistical properties of signals and images and to the ways these can be exploited to extract information-rich features for classification (e.g.,co-occurrence matrices,AR models). In this section we will focus our attention on whether the notion of self-similarity is extendable to stochastic processes and, if it is, how useful it can be for our interests. In the previous section “similarity” referred to the shape of a curve. For statistics such a view would be of no interest. From a statistical point of view it would be more reasonable and justifiable to interpret similarity from the perspective of “similar statistical properties,”that is, mean, standard deviation, and so forth. Indeed, it can be shown that stochastic processes that are self-similar under scaling do exist. Furthermore, such processes can model adequately a number of processes met in practice. Let (n) be a white (Gaussian) noise sequence with variance 2 ⫽ 1. The process defined as x(n) ⫽
n
(i)
i⫽1
is known as a random walk sequence, and it belongs to a more general class of stochastic processes known as Brownian motion processes [Papo 91, p. 350]. It is straightforward to see that E[x(n)] ⫽ 0
“09-Ch07-SA272” 18/9/2008 page 447
7.4 A Glimpse at Fractals
and that its variance is given by E[x 2 (n)] ⫽ n2
Thus, the process is a nonstationary one because its variance is time dependent. A direct generalization of the previous result is E[⌬2 x(n)] ≡ E[(x(n ⫹ n0 ) ⫺ x(n0 ))2 ] ⫽ n2
(7.48)
where by definition ⌬x(n) is the sequence of increments. Scaling the time axis by m results in E[⌬2 x(mn)] ≡ E[(x(mn ⫹ n0 ) ⫺ x(n0 ))2 ] ⫽ mn2
(7.49)
Hence, if the sequence √ of increments is to retain the same variance after scaling, it should be scaled by m. Furthermore, it is easy to see that the sequence of increments, as well as the scaled versions, follow a Gaussian distribution (e.g., [Falc 90]). Recalling that Gaussian processes are completely specified by their mean and variance, we conclude that the increments ⌬x(n) of x(n) are statistically selfsimilar in the sense that ⌬x(n)
and
1 √ ⌬x(mn) m
(7.50)
are described by the same probability density functions, for any n0 and m. Figure 7.16 shows three curves of the scaled random walk increments for m ⫽ 1, 3, 6. It is readily observed that they indeed “look” alike. Such curves, for which different scaling has been used for the coordinates (⌬x, n), are also known as statistically self-affine. The random walk Brownian motion is a special case of a more general class of processes known as fractional Brownian motion sequences (fBm), introduced in [Mand 68]. The increments of this type of processes have variance of the general form E[⌬2 x]⬀(⌬n)2H
(7.51)
with 0 ⬍ H ⬍ 1,⌬n ≡ n⫺n0 and ⬀ denoting proportionality. The parameter H is also known as the Hurst parameter. As in the case of Brownian motion, the increments of such processes are statistically self-affine in the sense that the processes ⌬x(n)
and
1 ⌬x(mn) mH
are described by the same probability density functions. The parameter H relates to the visual smoothness or coarseness of the respective graph of the increments versus time. This is an implication of (7.51). Let us start from a maximum interval ⌬n, corresponding to an incremental variance 2 . In the sequel we halve the interval to ⌬n/2. The respective variance will be reduced by the factor (1/2)2H .
447
“09-Ch07-SA272” 18/9/2008 page 448
448
CHAPTER 7 Feature Generation II
40
m⫽1
0 ⫺40 m⫽3
40 0 ⫺40
m⫽6
40 0 ⫺40 0
250
500
FIGURE 7.16 Time evolution of the random walk (m ⫽ 1) sequence and two of its self-affine versions. They all do look alike.
This process can go on. Each time we reduce the interval ⌬n by half, we look at increments between points located closer in time. The higher the value of H, the greater the reduction of the variance of the increments between these points, indicating smoother curves. For H ⫽ 0, the variance of the increments remains constant and independent of ⌬n. This process is not an fBm, and it corresponds to a white noise stationary process, with no dependence between adjacent time instants. Hence, it exhibits the most erratic behavior, and its graph has the most coarse appearance. This observation indicates that the parameter H could be used as a measure of the “smoothness” of such curves. By varying H one can get curves of varying degree of smoothness [Saup 91]. Figure 7.17 indeed verifies that the curve for H ⫽ 0.8 is smoother than the one for H ⫽ 0.2, and both are smoother than the top one, which corresponds to a white noise sequence. As was the case with the fractal curves of the previous subsection, a dimension can also be defined for curves resulting from fBm processes. It can be shown [Falc 90, p. 246] that an fBm process with parameter H corresponds to a curve with fractal dimension 2 ⫺ H. In general, if l is the number of free parameters of the graph, the corresponding fractal dimension is l ⫹ 1 ⫺ H. For a graph as in Figure 7.17, l ⫽ 1 and for an image l ⫽ 2. The question now is how all these no doubt mind-stimulating points can be of use to us in the context of pattern recognition. The terms smoothness and coarseness have been used in association with the parameter H and equivalently with the dimension D of an fBm process. On the other hand, the terms smoothness
“09-Ch07-SA272” 18/9/2008 page 449
7.4 A Glimpse at Fractals
4
0 ⫺4 1
H ⫽ 0.2
⫺0.5 ⫺2 1
H ⫽ 0.8
0.4 ⫺0.2 0
250
500
FIGURE 7.17 Time evolution of a white noise sequence (top) and two fBm processes of different Hurst parameters H. The lower the value of H, the coarser the appearance of the graph.
and roughness, were in a central position when dealing with feature generation for texture classification. We have now reached our crucial point. Provided that we can describe the sequence of gray levels of an image as an fBm process, the corresponding fractal dimension can be used as a potential feature for texture classification. In [Pent 84] it is reported that this is indeed true for a number of cases. Using a number of textured images from [Brod 66], as well as images from natural scenes, it was found that a large percentage of them exhibited fBm behavior. This was easily verified by constructing the histogram of differences (increments) of the gray-level intensities for various relative pixel distances ⌬n. It turned out that for each value of ⌬n the corresponding histogram was close to a Gaussian pdf centered at zero. Furthermore, the widths of the Gaussian-like histograms were different for the different relative pixel distances ⌬n. The larger the ⌬n, the wider the resulting histogram. However, we know that the width of a Gaussian is directly related to its variance. The plot of the variance as a function of relative pixel distance revealed an underlying fBm nature of the intensity processes, at least over a 10:1 range of relative distances measured. The parameter H, or equivalently the fractal dimension D, was then used successfully to distinguish a number of different textured regions in an image. The estimation of the H can take place via its definition in (7.51). Taking the logarithm, we get ln E[⌬2 x] ⫽ ln c ⫹ 2H ln ⌬n
449
“09-Ch07-SA272” 18/9/2008 page 450
450
CHAPTER 7 Feature Generation II
where c is the proportionality constant and ⌬x is now the difference in the gray-level intensities between pixels at relative distance ⌬n, that is, ⌬n ⫽ 1, 2, and so on. Obviously, c ⫽ E[⌬2 x] for ⌬n ⫽ 1. For each pixel distance ⌬n the corresponding average ⌬2 x is computed over the image window of interest. The resulting points (E[⌬2 x], ⌬n) are plotted in a two-dimensional logarithmic plot. A straight line is then fitted through the points using a least squares linear regression technique. The parameter H is provided by the slope of the line. This is also a test of the fractal nature of the underlying process. If the resulting points do not lie approximately on a straight line, the fractal model assumption will not be valid. Figure 7.18 demonstrates the procedure for two images. The one on the right is an artificially produced fractal image with H ⫽ 0.76. The least squares fit in the logarithmic plot of the standard deviation against ⌬n results in a straight line of slope 0.76. The image on the left is from [Brod 66]. We observe that the resulting least squares fit is reasonable, suggesting that the image is approximately fractal in nature. The slope is now H ⫽ 0.27. The lower value of H reflects the fact that the latter image is coarser than the former.
4.7
3 2.6
4.5
2.2 4.3 1.8 4.1
3.9
1.4
0
0.5
1
1.5
2
2.5
1
0
0.5
1
1.5
2
2.5
FIGURE 7.18 Examples of images with corresponding logarithmic plots of the standard deviation of increments (vertical axis) versus relative distance (horizontal axis).
“09-Ch07-SA272” 18/9/2008 page 451
7.5 Typical Features for Speech and Audio Classification
The method presented previously for the computation of the Hurst parameter is not the only one, and a number of other techniques have been suggested. A popular alternative is based on the wavelet analysis of the underlying fBm process. The basis sequences (functions) used in the wavelet analysis are scaled and translated versions of a mother sequence (function). This underlying notion of scale invariance is shown to relate to fBm processes, whose statistical properties are scale invariant [Flan 92]. It turns out that the wavelet coefficients of an fBm process,at a given resolution level, i, form a stationary sequence, with a variance proportional to 2⫺i(2H⫹1) , see [Worn 96]. This leads to a simple method for estimating the associated Hurst parameter. Other methods for estimating the fractal dimension include the boxcounting and variation method [Huan 94, Kell 87];maximum likelihood estimates,as in [Lund 86, Deri 93, Fieg 96];morphological covers [Mara 93];methods in the spectral domain, as in [Gewe 83]; and fractal interpolation function models [Penn 97]. Fractional modeling and the use of fractal dimension D as a feature for classification have been demonstrated in a number of applications [Chen 89, Lund 86, Rich 95]. However, the method is not without drawbacks. Indeed, it may happen that different textures result in the same fractal dimension, thus limiting the classification potential of the method. Another shortcoming is that in practice physical processes retain their fractal characteristics over a range of distances but not over all ranges. Thus, the fractal dimension may change as we pass from one range of scales to another [Pent 84, Pele 84], and the use of a single Hurst parameter may not lead to sufficient modeling. To overcome these drawbacks, a number of possible solutions have been suggested. The multifractional Brownian motion (mBm) is an extension of an fBm process with a parameter H, which is allowed to vary, as in [Ayac 00]. Extended self similar (ESS) processes allow in (7.51) for a more general dependence on ⌬n, via a so-called structure function [Kapl 94]. For more on these issues the interested reader is referred, for example, to [Bass 92, Ardu 92, Kapl 95, Kapl 99, Pesq 02]. A comparative study of various textural features, including fractal modeling techniques, can be found in [Ohan 92, Ojal 96].
7.5 TYPICAL FEATURES FOR SPEECH AND AUDIO CLASSIFICATION As we have already commented in the Preface, speech recognition is a major application area of pattern recognition, and a number of speech-recognizing systems are already available in the market. Audio classification and recognition have also received a lot of attention in recent years. A great number of commercial applications are envisaged for the future in the field of multimedia databases. Techniques for automatic indexing tools, intelligent browsers, and search engines with content-based retrieval capabilities are currently the focus of a major research effort. In this context, audiovisual data segmentation and indexing, based not only on visual information but on the accompanying audio signal, greatly enhance the performance. For instance, classifying video scenes of gun fights using the audio
451
“09-Ch07-SA272” 18/9/2008 page 452
452
CHAPTER 7 Feature Generation II
information related to shooting and explosions will, no doubt, enhance the performance of the classifier compared to a system that is based on the corresponding visual information only. Content-based retrieval from music databases is another application that attracts the interest of current research. It is very likely that not far in the future a large corpus of the recorded music in human history will be available on the Web. Automatic music analysis is envisaged to be one of the main services to facilitate content distribution. Automatic music genre classification, querying music databases by humming the tune of a song or querying by example (i.e., providing a music extract of short duration in order to locate and retrieve the complete recording) are examples of services that vendors would very much like to offer in such systems. More on these issues can be found in [Wold 96, Wang 00, Zhan 01, Pikr 03, Pikr 06, Pikr 08, Frag 01, Clau 04]. This section focuses on the generation of some typical and commonly used features for speech recognition and audio classification/recognition. However, as has already been stated elsewhere in the book one must keep in mind that feature generation is very much a problem-dependent task. Thus, the combination of the designer’s imagination with his or her good knowledge of the peculiarities of the specific task can only benefit the generation of informative features.
7.5.1 Short Time Processing of Signals Speech and audio signals are statistically nonstationary; that is, their statistical properties vary with time. A way to circumvent this problem and be able to use analysis tools that have been developed and make sense for stationary signals only, such as the Fourier transform, is to divide the time signal into a series of successive frames. Each frame consists of a finite number, N , of samples. During the time interval of a frame, the signal is assumed to be “reasonably stationary.” Such signals are also known as quasistationary. Figure 7.19 shows a signal segment and three successive frames, each consisting of N ⫽ 20 samples, with an overlap among neighboring frames of 5 samples. Choosing the length, N , of the frames is a “bit of an art” and is a problem-dependent task. First, each frame must be long enough for an analysis method to have enough “data resources” to build up the required information. For example, if we are interested in estimating the period of a periodic signal the number of samples in each frame must be large enough to allow the signal periodicity to be revealed. Of course, this will depend on the value of the period. For short periods, a few samples can be sufficient. On the other hand, for long periods more samples will be necessary. Second, N must be short enough to guarantee the approximate stationarity of the signal within the time scale of each frame in order for the results to be meaningful. For speech signals sampled at a frequency of fs ⫽ 10 KHz, reasonable frame sizes range from 100 to 200 samples, corresponding to 10–20 msecs time duration. For music signals sampled at 44.1 KHz,reasonable frame sizes range from 2048 to 4096 samples,corresponding to 45–95 msecs, approximately.
“09-Ch07-SA272” 18/9/2008 page 453
7.5 Typical Features for Speech and Audio Classification
overlap x(n) 5
1st frame
2nd frame
3rd frame
4.5 4 3.5 3 2.5 2 1.5 1 0.5 0
0
5
10
15
20
25
30
35
40
45
50
n
FIGURE 7.19 Three successive frames, each of length N ⫽ 20 samples. The overlap between successive frames is 5 samples.
From a mathematical formulation point of view, dividing the time signal in a sequence of successive frames is equivalent to multiplying the signal segment by a window sequence, w(n), of finite duration N . The simplest window sequence is the rectangular window, defined as w(n) ⫽
1
0ⱕnⱕN ⫺1
0
elsewhere
For different frames, the window is shifted to different points, mi , in the time axis. Hence, if x(n) denotes the signal sequence, the samples of the ith frame can be written as xi (n) ⫽ x(n ⫹ mi )w(n)
where mi is the corresponding window shift associated with the ith frame. This implies that all samples in the ith frame are identically zero, except for the time instants n ⫽ 0, . . . , N ⫺ 1, which correspond to the original signal samples, x(n), with n ∈ [mi , mi ⫹N ⫺1]. The procedure is illustrated in Figure 7.20. As it is known from the Fourier transform theory basics, multiplying a sequence by a window in the time domain smooths out its Fourier transform by convolving it with the Fourier transform of the window sequence. Some of the effects of this smoothing action can be minimized by using a different window sequence, with a smoother decay to
453
“09-Ch07-SA272” 18/9/2008 page 454
454
CHAPTER 7 Feature Generation II
x(n) 5
xi(n) 5
4
4
3
3
2
2
1
1
0 mi
n mi ⫹ N ⫺1
0
N ⫺1
(a)
n (b)
FIGURE 7.20 A signal segment (a) and the resulting frame (b) after the application of a rectangular window sequence of duration equal to 14 samples and shifted at mi .
zero. A popular choice is the Hamming window, defined as
0.54 ⫺ 0.46 cos
w(n) ⫽
2n N ⫺1
0
0ⱕnⱕN ⫺1 elsewhere
More on these issues can be found in [Rabi 78, Dell 00]. As an example, let us assume that we have divided a speech segment into a sequence of I frames, each of length N . Then, for each frame, i ⫽ 1, 2, . . . , I , we can compute the discrete Fourier transform (DFT) as Xi (m) ⫽
N ⫺1 n⫽0
2 mn , xi (n) exp ⫺j N
m ⫽ 0, 1, . . . , N ⫺ 1
It is common to refer to this DFT as the short-time DFT. It must be pointed out that this definition implies much more theory and interesting implementation issues (which, however, are not of interest to us and we will not delve into this topic any deeper). Selecting l ⱕ N DFT coefficients from each frame, we can construct a sequence of feature vectors ⎡ ⎢ ⎢ xi ⫽ ⎢ ⎢ ⎣
Xi (0) Xi (1) .. .
⎤ ⎥ ⎥ ⎥, ⎥ ⎦
i ⫽ 1, 2, . . . , I
(7.52)
Xi (l)
Thus, the pattern of interest (i.e., the speech segment) is not represented by a single feature vector but by a sequence of feature vectors. We will see how to attack such problems in Chapters 8 and 9. The autocorrelation sequence is another very important statistical quantity, which is also defined for stationary processes. Recall from Section 7.2.4 that, if x(n) is a stationary process, the autocorrelation sequence is defined as r(k) ⫽ E[x(n)x(n ⫺ k)] ⫽ E[x(n)x(n ⫹ k)] ⫽ r(⫺k)
(7.53)
“09-Ch07-SA272” 18/9/2008 page 455
7.5 Typical Features for Speech and Audio Classification
In other words, it is the expectation of the product of x(n) with its shifted version x(n ⫾ k). In practice, the expectation can be obtained as r(k) ⫽
N 1 x(n)x(n ⫹ k) 2N ⫹ 1 n⫽⫺N
which under mild assumptions (i.e., ergodicity) tends to the true value of r(k) as N tends to infinity. In the case of a quasistationary process, the short-time autocorrelation sequence, ri (k), is defined for each of the frames as ri (k) ⫽
1 N
N ⫺1⫺|k|
xi (n)xi (n ⫹ |k|)
(7.54)
n⫽0
where | · | denotes the absolute value operator. The limits in the sum indicate that outside the interval [0, N ⫺ 1 ⫺ |k|] the product xi (n)xi (n ⫹ |k|) is identically zero, due to the finite duration, N , of the frame. This definition complies with the definition in (7.53) in the sense that for stationary processes ri (k) is an asymptotically unbiased estimate of the autocorrelation as the length of the frame N ⇒ ⬁. Indeed, viewing ri (k) as an estimate of r(k), its mean value for different realizations is easily shown to be E[ri (k)] ⫽
N ⫺ |k| r(k) N
(7.55)
Thus, for finite frame length, N , Eq. (7.54) results in a biased estimate of r(k). However, for small values of the lag k, with respect to N , the bias is small. On the other hand, for values of k close to N we expect ri (k) to get values close to zero, something that is verified in practice. To remedy this drawback, other definitions of the short-time autocorrelation have been proposed. See, for example, [Rabi 78]. Another important property of the definition in (7.54), from the computational point of view, is that the corresponding (short-time) autocorrelation matrix retains the computationally elegant properties of being symmetric and having a Toeplitz structure (Section 7.2.4).
7.5.2 Cepstrum Let x(0), x(1), . . . , x(N ⫺ 1) be the samples from the current data frame (the index i has been dropped for notational simplicity). The Fourier transform (FT) of this finite-length sequence of data is defined as the periodic complex function X() ⫽
N ⫺1
x(n) exp ⫺jTn
(7.56)
n⫽0
with period (in the frequency domain) 2/T , where T is the sampling period. It is well known from the basic theory of signal processing (e.g., [Proa 92] and Chapter 6), that the coefficients of the DFT transform X(m) ⫽
N ⫺1 n⫽0
2 mn , x(n) exp ⫺j N
m ⫽ 0, 1, . . . , N ⫺ 1
(7.57)
455
“09-Ch07-SA272” 18/9/2008 page 456
456
CHAPTER 7 Feature Generation II
2 2 are the samples of the FT taken at the frequency points 0, NT , . . . , NT (N ⫺ 1). Assuming, without loss of generality, T ⫽ 1, the inverse FT is defined as
x(n) ⫽
1 2
⫺
X() exp jn d,
n ⫽ 0, 1, . . . , N ⫺ 1
(7.58)
That is, the resulting samples are equal to the samples of the original sequence and identical to what is obtained by the inverse DFT. That is, x(n) ⫽
N ⫺1 1 2 mn , X(m) exp j N N
n ⫽ 0, 1, . . . , N ⫺ 1
(7.59)
m⫽0
The cepstrum, c(n), of a sequence, x(n), is the sequence resulting from the inverse FT of the logarithm of the magnitude of its FT. That is, c(n) ⫽
1 2
⫺
log10 |X()| exp jn d
(7.60)
Although the base 10 logarithm has been used, the logarithm of any base can be adopted. Another way of looking at the cepstral coefficients, c(n), is the following. Since the FT function X() is a periodic function of , with period 2, the function log10 |X()| is also periodic with the same period. Therefore, it can be written in terms of its Fourier series expansion log10 |X()| ⫽
⬁ 2 n ⫽ c(n) exp ⫺j c(n) exp ⫺jn 2 n⫽⫺⬁ n⫽⫺⬁ ⬁
(7.61)
Hence, Eq. (7.60) is the formula that provides the coefficients of the Fourier series expansion in (7.61). However, the function log10 |X()| is defined in the frequency and not in the time domain. Its Fourier transform domain is known as the quefrency domain, and the respective Fourier series coefficients, c(n), are known as cepstral coefficients. This is only to remind us that the original transformed function is in the frequency domain. Otherwise, all Fourier transform/series properties are still valid. Thus, since log10 |X()| is a real and even function, (|X()| is even for a real sequence x(n)), the cepstral coefficients are real and even. That is, c ∗ (n) ⫽ c(n) ⫽ c(⫺n)
Cepstral coefficients have very good information-packing properties, from the class discrimination point of view, and are very popular candidates as features, both for speech recognition and audio classification tasks [Rabi 93, Tzan 02]. Computation of the cepstral coefficients is achieved via the DFT (using the FFT) of X(). However, this computation is not as innocent as it is for the case of X(). As we have already seen, the inverse transforms of X() and X(m) coincide [Eqs. (7.58) and (7.59)]. This is because the input sequence is of finite length, N , and the sampling period s , in the frequency domain, is chosen to obey the Nyquist criterion; that is, s ⫽ 2 N . This is not the case with the cepstral coefficients. Using log10 |X(m)|, m ⫽ 0, 1, . . . , N ⫺ 1, and taking the inverse DFT results in cˆ (n) ⫽
N ⫺1 1 2 log10 |X(m)| exp j mn , N N m⫽0
n ⫽ 0, 1, . . . , N ⫺ 1
(7.62)
“09-Ch07-SA272” 18/9/2008 page 457
7.5 Typical Features for Speech and Audio Classification
where cˆ (n) and c(n) are related as cˆ (n) ⫽
⬁
c(n ⫹ rN )
(7.63)
r⫽⫺⬁
For those readers familiar with basic digital signal processing this is most natural. The sequence c(n) is not of finite duration. Thus, sampling its FT (C() ≡ log10 |X()|) with a sampling period of 2 N does not satisfy Nyquist’s criterion. Hence, taking the inverse DFT [Eq. (7.62)] will result in aliasing with a periodic repetition of the aliased sequence (c(n)) every N samples, which is expressed via (7.63). See, for example, [Proa 92]. In practice, the effects of this aliasing are minimized if one extends the length of the frame from N to M by appending M ⫺ N zeros at the end of it. That is, x(n) : x(0), . . . , x(N ⫺ 1), x(N ) ⫽ 0, . . . , x(M ⫺ 1) ⫽ 0
(7.64)
These zeros (as it can easily be checked out) have no effect on the FT X(). However, the DFT is now of length M (corresponding to sampling the FT every 2 M frequency points), which makes the implicit repetition period in (7.63) equal to M. Assuming that the cepstral coefficients decay fast enough,with respect to the repetition period M,we can assume that cˆ (n) ≈ c(n), n ⫽ 0, 1, . . . , N ⫺1,since successive repetitions in (7.63) now have little overlap. In practice, a number of 512 or 1024 zeros may be necessary. For further information related to cepstrum, the interested reader may refer to [Rabi 78, Dell 00]. In summary, the computational steps to obtain the cepstral coefficients of a frame, xi (n), n ⫽ 0, 1, . . . , N ⫺ 1, are the following: ■
Extend the length of the frame by appending M ⫺ N zeros at the end of the frame.
■
Obtain the DFT of length M of the extended frame.
■
Compute the logarithm of the magnitude of the DFT coefficients.
■
Compute the inverse DFT of length M.
The obtained coefficients are the (approximate) cepstral coefficients of the sequence xi (n).
7.5.3 The Mel-Cepstrum Human perception of audio has often been studied from a psychophysical point of view. Experiments have suggested that perception of the frequency content of pure tones does not follow a linear scale. This led to the idea of mapping acoustic frequency content to a linear“perceptual”frequency scale. A popular approximation to this type of mapping is known as the mel scale [Pico 93, Rabi 93]: fmel ⫽ 2595 log10 10(1 ⫹ f /700.0)
(7.65)
Equation (7.65) suggests that an actual frequency of 1 KHz is mapped to 1000 mel units. The plot of (7.65) is shown in Figure 7.21, and as can be seen the mapping
457
“09-Ch07-SA272” 18/9/2008 page 458
CHAPTER 7 Feature Generation II
3000 2500 Mel units
458
2000 1500 1000 500 0
0 1000 2000
4000 6000 8000 Frequency (Hz)
10000
FIGURE 7.21 Subjectively perceived pitch, in mel units, as a function of the measured frequency.
is approximately linear from 0 Hz up to 1000 Hz, and its logarithmic nature prevails for frequencies above 1000 Hz. As an example, Eq. (7.65) suggests that the perceived frequency of a pure tone at the frequency of 10 KHz will be approximately 3000 mel units (mels). That is, a tenfold increase in the frequency will be subjectively perceived as a threefold increase. Another psychoacoustics phenomenon is related to the way our auditory system perceives the differences in frequency among different tones that contribute to the formation of a more complex sound. It turns out that the tones cannot be individually distinguished if they fall within a certain bandwidth around the center frequency of the sound. We refer to this bandwidth as the critical bandwidth [Pico 93, Rabi 93]. Furthermore, if the bandwidth of a complex sound is less than the critical bandwidth around its center frequency the ear would perceive it as a single tone at the center of the critical band and with its loudness being equal to a weighted average of the loudness of each one of the contributing tones. The critical bandwidth around a frequency f can approximately be given by BW critical ⫽ 25 ⫹ 75[1 ⫹ 1.4( f /1000)2 ]0.69
(7.66)
A plot of this equation reveals that the critical bandwidth is approximately linear below 1 KHz and increases logarithmically for frequencies above 1 KHz. In an effort to generate features that are rich in information, we will try to “manipulate” the frequency content of a sound segment by imitating nature, in the way our auditory system perceives and recognizes sounds. To this end the following steps are adopted:
“09-Ch07-SA272” 18/9/2008 page 459
7.5 Typical Features for Speech and Audio Classification
A sound segment, of length N , is analyzed via the DFT transform. As we have already observed, it is useful to append a number of zeros at the end and let M denote the number of samples after the extension. If fs ⫽ 1/T is the sampling s frequency, each DFT coefficient X(m) corresponds to a real frequency of mf M .
■
In the sequel, a number, L, of critical bands are “spread” over the frequency range up to fs /2. Figure 7.22 is an example, where only the first 17 of L ⫽ 35 such bands are shown (for illustration simplicity) to occupy the frequency range from 0 Hz up to approximately 3700 Hz. The shape of each frequency band is a graphical representation of the weighting imposed on the corresponding frequency sample (frequency bin) within the bandwidth of the band. The sampling frequency is fs ⫽ 44.1 KHz. These bands are uniformly distributed in a mel scale, and their bandwidth has been chosen to be approximately equal to 110 mels. In the frequency scale, these bands are almost uniformly spaced below 1 KHz and logarithmically above it. The shape of the bands has been chosen to be triangular. In general, the shape, bandwidth, and number of bands are critical design issues, and several approaches have been suggested throughout the years, depending on the application domain (see, for example, [Pico 93, Rabi 93, Davi 80]). In our example, we chose nonoverlapping bands, although in some cases an overlap between successive bands is allowed to exist.
■
Since, in general, the center frequencies of these frequency bands do not coincide with the frequency quantization performed by the DFT, each band is moved so that its center frequency coincides with the nearest DFT frequency
Frequency Response Magnitude
■
1 0.8 0.6
... 0.4 0.2 0
0
500
1000 1500 2000 2500 3000 3500 Frequency (Hz)
FIGURE 7.22 A critical-band filter bank consisting of nonoverlapping triangular bandpass filters.
459
“09-Ch07-SA272” 18/9/2008 page 460
460
CHAPTER 7 Feature Generation II
s bin ( mf M ). Denote by mi this (center) frequency bin of the ith band in the bank, i ⫽ 1, 2, . . . , L. ■
For each of the bands in the bank, compute the weighted average of the log-magnitude of the DFT coefficients that fall within the frequency band. That is, Y (mi ) ≡
log10 |X(m)|Hi
m
m M
fs ,
i ⫽ 1, 2, . . . , M
(7.67)
where Hi (·) is the corresponding weighting value. Note that since the width of each band is finite,this summation,over m,is restricted to the DFT coefficients located within the bandwidth of the respective critical band. ■
Define the sequence Y (m) ⫽
Y (mi ) 0
if m ⫽ mi , i ⫽ 1, 2, . . . , L otherwise
(7.68)
In other words, this sequence is zero everywhere except at the frequency bins corresponding to the centers of the bands in the bank, where the value is equal to the weighted average of the log-magnitude of the DFT coefficients at the frequency bins within the bandwidth of the respective band. We can think of this new sequence as the psychologically perceived log-magnitude spectrum equivalent to the physically measured one. ■
Take the inverse DFT M⫺1 1 2 mn , cmel (n) ⫽ Y (m) exp j M M
n ⫽ 0, 1, 2, . . . , N ⫺ 1
(7.69)
m⫽0
These are known as the mel-cepstral coefficients and are among the most powerful features in speech and audio recognition/classification. Note that since the log-magnitude DFT coefficients are real and symmetrical, the previous inverse DFT can also be efficiently computed via a cosine transform, as in [Proa 92]. The reader should note that the previous method of defining mel-cepstral coefficients is just one of many variants that have been proposed over the years. For a more extensive treatment the reader is referred to more specialized texts and articles, such as [Pico 93, Rabi 93, Dell 00].
7.5.4 Spectral Features Let xi (n), n ⫽ 0, 1, . . . , N ⫺ 1 be the samples of the ith frame and Xi (m), m ⫽ 0, 1, . . . , N ⫺ 1, the corresponding DFT coefficients. The following features are quite common in speech recognition and audio classification/recognition, each providing information for different acoustic qualities.
“09-Ch07-SA272” 18/9/2008 page 461
7.5 Typical Features for Speech and Audio Classification
Spectral Centroid N ⫺1 m|Xi (m)| C(i) ⫽ m⫽0 N ⫺1 m⫽0 |Xi (m)|
The centroid is a measure of the spectral shape. High values of the centroid correspond to “brighter”acoustic structures with more energy in the high frequencies.
Spectral Roll-off The spectral roll-off is the frequency sample, mRc (i), below which the c% (e.g., c ⫽ 85 or 90) of the magnitude distribution of the DFT coefficients is concentrated. That is, for this frequency sample the following is true: mR c (i)
|Xi (m)| ⫽
m⫽0
N ⫺1 c |Xi (m)| 100 m⫽0
This is another measure indicating where most of the spectral energy is concentrated. It is a measure of skewness of the spectral shape, with right-skewed shapes (brighter sounds) resulting in higher values.
Spectral Flux F (i) ⫽
N ⫺1
(Ni (m) ⫺ Ni⫺1 (m))2
m⫽0
Here, Ni (m) is the normalized (by its maximum value) magnitude of the respective DFT coefficient of the ith frame and is a measure of the local spectral change between successive frames.
Fundamental Frequency Speech and audio signals can be either noise-like in nature,such as unvoiced speech segments or audio segments corresponding to an applause or footstep recordings, or can exhibit a periodic nature. In the latter case, we talk about harmonic signals to distinguish them from their noise-like inharmonic counterparts. Audio signals produced from musical instruments and voiced speech segments are two examples of harmonic signals. A distinct characteristic of a harmonic sound signal is its fundamental frequency. In the case of voiced speech signals,this is the frequency of successive vocal fold openings and is also known as the pitch of the signal. For men this lies in the range of 80 to 200 Hz and for women in the range of 150 Hz to 350 Hz. For musical instruments,the fundamental frequency may vary a lot,and in some cases the fundamental frequency may not be present in the frequency spectrum, although the ear can have the ability to perceive it, by processing the information provided by the higher harmonics. This is a psychoacoustics phenomenon. Psychoacousticians as well as musicologists use the term pitch to define the frequency perceived by the ear,which in some cases may even be different from the fundamental.
461
“09-Ch07-SA272” 18/9/2008 page 462
CHAPTER 7 Feature Generation II
1 0.9 Normalized Fourier coefficients
462
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0
500
1000
1500 2000 2500 Frequency (Hz)
3000
3500
FIGURE 7.23 Normalized DFT coefficients of a clarinet sound, whose fundamental frequency is absent.
The fundamental frequency estimation is not an easy task, and a number of techniques have been proposed in the published literature, including [Schr 68, Brow 91, Wu 03, Tolo 00, Klap 03, Goto 04] and the references therein. Figure 7.23 is an example of the (normalized) amplitude of the DFT of a signal segment corresponding to a harmonic sound produced by a clarinet. The length of the frame is 4096 samples long. The spectrum consists of the regularly spaced harmonics of the fundamental frequency, which is equal to 230 Hz but is missing from the spectrum. Note that, listening to this sound, a trained ear will perceive that the pitch of this signal is indeed 230 Hz. It can be observed that odd and even multiples of the fundamental frequency are present as peaks. ( For this frame, it turned out that the amplitudes of the odd multiples are considerably smaller than the amplitudes of the even multiples of the fundamental frequency.) Application of the algorithm given in [Schr 68], for the estimation of the fundamental frequency, returns the true value of 230 Hz. The method builds on the idea of exploiting the greatest common divisor of all peaks present in the spectrum.
7.5.5 Time Domain Features Zero-Crossing Rate The zero-crossing rate is defined as Z(i) ⫽
N ⫺1 1 |sgn[xi (n)] ⫺ sgn[xi (n ⫺ 1)]| 2N n⫽0
“09-Ch07-SA272” 18/9/2008 page 463
7.5 Typical Features for Speech and Audio Classification
where
sgn[xi (n)] ⫽
1
xi (n) ⱖ 0
⫺1
xi (n) ⬍ 0
This is a measure of the noisiness of the signal. Thus, unvoiced speech signals have higher zero-crossing values compared to the voiced ones. Temporal curves of variation of the zero-crossing rate from frame to frame may also be informative of the type of signal.
Energy This is a very simple feature, defined as E(i) ⫽
N ⫺1 1 |xi (n)|2 N n⫽0
and it can be used to discriminate voiced from unvoiced speech signals, since the latter tend to have much lower energy values than the former. It is also useful to discriminate silent periods in a recording and can be useful during the segmentation process. All features discussed in this section are also known as frame-level features. They provide local information with respect to individual frames, and their goal is to capture short-term characteristics. However, to extract semantic content information, one needs to follow how the previously cited features change from frame to frame over a longer time scale. To this end, one can develop various methods to quantify this variation. In [Tzan 02], the mean and variances of the frame-level features have been used as features for music genre classification. Besides the above features, other features—such as wavelet coefficients, fractal dimension, AR modeling, and independent components ( ICA), presented in the previous and the present chapter—are also popular candidates. For music signals, based on early studies on the human perception of pitch, it has been proposed to use a 12-element representation of the spectral energy of a music signal, known as the chroma vector [Bart 05]. Each element of the vector corresponds to one of the 12 traditional pitch classes (i.e., 12 notes) of the equal-tempered scale of Western music. The chroma vector can encode and represent harmonic relationships within a particular music signal.
7.5.6 An Example To demonstrate the classification power of two of the previously discussed features, let us take a simple example. Figure 7.24 shows the variation of the zero-crossing rate from frame to frame as time evolves, for a clapping sound. The sampling frequency was 44.1 KHz, and the length of each frame was equal to 1024 samples, with a successive frame overlap of 512 samples. For each frame the Hamming window was used. One can observe the noisy look of the resulting graph, with a large change of the feature values from frame to frame. This noisy nature of the clapping sound can also be revealed if the fundamental frequency is adopted as a
463
“09-Ch07-SA272” 18/9/2008 page 464
CHAPTER 7 Feature Generation II
120
Zero Crossing Rate
100 80 60 40 20 0
0
0.5
1
1.5 2 Time (secs)
2.5
3
3.5
FIGURE 7.24 Zero-crossing rate results for a clapping sound recording, using a Hamming moving window technique.
700
Fundamental Frequency (Hz)
464
600 500 400 300 200 100
0
0.5
1
1.5 2 Time (secs)
2.5
3
3.5
FIGURE 7.25 Fundamental frequency tracking results for the same clapping sound recording as in Figure 7.24.
“09-Ch07-SA272” 18/9/2008 page 465
7.5 Typical Features for Speech and Audio Classification
feature. Figure 7.25 shows the change of the fundamental frequency from frame to frame. The algorithm used for the fundamental frequency tracking was that proposed in [Brow 91]. In contrast to the noisy nature of the previous audio recording, Figures 7.26 and 7.27 show the graphs of the change, from frame to frame, of the
55 50
Zero Crossing Rate
45 40 35 30 25 20 15
0
0.5
1
1.5 2 2.5 Time (secs)
3
3.5
4
FIGURE 7.26 Zero-crossing rate results for the piano music recording, using a Hamming moving window technique.
Fundamental Frequency (Hz)
1200 1000 800 600 400 200 0
0
0.5
1
1.5 2 2.5 Time (secs)
3
3.5
4
FIGURE 7.27 Fundamental frequency tracking results for the same piano music recording as in Figure 7.26.
465
“09-Ch07-SA272” 18/9/2008 page 466
466
CHAPTER 7 Feature Generation II
zero-crossing rate and fundamental frequency, respectively, for a piano recording (from Bach’s English suite in A major). The frame parameters used for the analysis were the same as before. One can observe that both graphs are now much less noisy. The variation from frame to frame is much smaller, and at some points of the graph the values of both features remain almost constant for relatively long periods of time, revealing the “structured” harmonic nature of the analyzed sound.
7.6 PROBLEMS 7.1 Consider the image array ⎡
0 ⎢1 ⎢ ⎢ I ⫽ ⎢3 ⎢ ⎣1 0
1 2 0 2 0
2 0 3 2 1
2 2 2 2 1
⎤ 3 0⎥ ⎥ ⎥ 1⎥ ⎥ 3⎦ 2
Compute the co-occurrence matrix for d ⫽ 1 and the four directions. Then compute the ASM and CON features. 7.2 For the image array of the previous problem compute the run length matrix for the four directions and then the features SRE, LRE. 7.3 Construct a 4 ⫻ 4 array that will have a high contrast (CON) value in the 0◦ direction and low CON value in the 45◦ direction. 7.4 For two of the test images provided in the Web site of the book, develop a program that computes the co-occurrence and run length matrices for the four directions and d ⫽ 1. Then compute the ASM, CON , IDF , Hxy , SRE, LRE, GLNU , RLN features and average their respective values over the four directions. Justify your findings. 7.5 Show that if N
Pi ⫽ 1
i⫽1
then S⫽
N
Pi2
i⫽1
becomes minimum if Pi ⫽
1 N,i
⫽ 1, 2, . . . , N .
7.6 Show that the central moments defined in (7.22) are invariant to translations and the normalized central moments in (7.23) are invariant to both translation and scaling.
“09-Ch07-SA272” 18/9/2008 page 467
7.6 Problems
7.7 Show that the Zernike polynomials are orthogonal on the unit circle disk, that is, x 2 ⫹y2 ⱕ1
∗ Vnm (x, y)Vpq (x, y) dx dy ⫽
␦np ␦mq n⫹1
7.8 Show that if a region in an image is rotated by an angle with respect to the origin, the Zernike moments of the rotated region are related to the unrotated ones by A⬘pq ⫽ Apq exp(⫺jq)
7.9 Write a program to compute the moments of Hu. Then apply it to two of the test images involving objects, available from the web, and compute the respective moments. 7.10 Repeat Problem 7.9 for the Zernike moments of order A11 , A20 , A02 . 7.11 Show Eq. ( 7.37 ). 7.12 Write a program that computes the AR parameters for a noncausal prediction model. Apply it to a homogeneous and isotropic image whose autocorrelation sequence is given by √ r(k, l) ⫽ exp(⫺ k2 ⫹ l 2 )
for a window W of order p ⫽ q ⫽ 1. 7.13 Let uk ⫽ xk ⫹ jyk , where (xk , yk ) are the coordinates of the points on the boundary of an object in an image. Show that if the object is rotated by an angle with respect to the origin of the axis, the new complex sequence is u⬘k ⫽ uk exp( j)
7.14 Let t denote the length along a closed boundary curve measured from an origin within the curve, and x(t), y(t) the coordinates as functions of t. If T is the total length of the curve, then the following Fourier series expansion holds. x(t) ⫽ a0 ⫹
an cos
2nt 2nt ⫹ bn sin T T
cn cos
2nt 2nt ⫹ dn sin T T
⬁ n⫽1
y(t) ⫽ c0 ⫹
⬁ n⫽1
467
“09-Ch07-SA272” 18/9/2008 page 468
468
CHAPTER 7 Feature Generation II
Prove that if x(t), y(t) are piecewise linear functions between the sampled points (x(t), y(t)), t ⫽ 0, 1, . . . , m ⫺ 1, the Fourier coefficients an , bn , cn , dn are given by an ⫽
m T ⌬xi [cos i ⫺ cos i⫺1 ] 2 2 2 n ⌬ti i⫽1
bn ⫽
m T ⌬xi [sin i ⫺ sin i⫺1 ] 2 2 n2 ⌬ti i⫽1
cn ⫽
m T ⌬yi [cos i ⫺ cos i⫺1 ] 2 2 n2 ⌬ti i⫽1
dn ⫽
m T ⌬yi [sin i ⫺ sin i⫺1 ] 2 2 n2 ⌬ti i⫽1
where ⌬xi ⫽ xi ⫺ xi⫺1 , ⌬yi ⫽ yi ⫺ yi⫺1 ⌬ti ⫽
i ⌬xi2 ⫹ ⌬yi2 , ti ⫽ ⌬tj j⫽1
T ⫽ tm , i ⫽
2nti T
7.15 For the Fourier coefficients in Problem 7.14, prove that the following parameters are rotation invariant: In ⫽ a2n ⫹ b2n ⫹ cn2 ⫹ dn2 Jn ⫽ an dn ⫺ bn cn K1,n ⫽ (a21 ⫹ b21 )(a2n ⫹ b2n ) ⫹ (c12 ⫹ d12 )(cn2 ⫹ dn2 ) ⫹ 2(a1 c1 ⫹ b1 d1 )(an cn ⫹ bn dn )
7.16 If (x(t), y(t)) are defined as in Problem 7.14 and z(t) ⫽ x(t) ⫹ jy(t)
the respective complex exponential Fourier series is given as z(t) ⫽
⬁
an exp( j2nt/T )
n⫽⫺⬁
an ⫽
1 T
T
z(t) exp(⫺j2nt/T ) dt 0
“09-Ch07-SA272” 18/9/2008 page 469
7.6 Problems
Prove that the following parameters are scale and rotation invariant [Gran 72]: bn ⫽
a1 ⫹ n a1⫺n , a21
dmn ⫽
an1 ⫹ m am 1⫺n a(m⫹n) 1
where n ⫽ 1. 7.17 Prove that if x(t), y(t) of the previous problem are piecewise linear functions between the points (x(t), y(t)), t ⫽ 0, 1, . . . , m ⫺ 1, then the Fourier coefficients an are given by [Lai 81] an ⫽
m T (bi⫺1 ⫺ bi ) exp(⫺jn2ti /T ) 2 (2n) i⫽1
where bi ⫽
Vi⫹1 ⫺ Vi , |Vi⫹1 ⫺ Vi |
ti ⫽
i
|Vk ⫺ Vk⫺1 |,
i ⬎ 0, t0 ⫽ 0
k⫽1
and Vi , i ⫽ 1, 2, . . . , m, the phasors at the respective points. 7.18 Show that the orientation in Section 7.3.3 results from minimizing I () ⫽
[(i ⫺ x) ¯ cos ⫺ ( j ⫺ y¯ ) sin ]2 i
j
7.19 Show that the power spectrum of an fdm process with Hurst parameter H is given by S( f )⬀
1 f (2H⫹1)
7.20 Show that the definition of M in ( 7.46 ) results in a consistent metric for the Koch curve. 7.21 Assuming that x(0) ⫽ 0, show that E[x(n)(x(n ⫹ n0 ) ⫺ x(n))] ⫽
" 1! (n ⫹ n0 )2H ⫺ n2H ⫺ n2H 0 2
For the case of a Brownian motion (H ⫽ 12 ) this suggests that x(n) is uncorrelated to the increment. This is not true for H ⫽ 12 ,where a nonzero correlation exists, positive for H ⬎ 1/2 and negative for H ⬍ 1/2. To prove this, use, the definition in ( 7.51 ). This can be generalized. That is, if n1 ⱕ n2 ⱕ n3 ⱕ n4 and the process is Brownian then E[(x(n2 ) ⫺ x(n1 ))(x(n4 ) ⫺ x(n3 ))] ⫽ 0
469
“09-Ch07-SA272” 18/9/2008 page 470
470
CHAPTER 7 Feature Generation II
MATLAB PROGRAMS AND EXERCISES Computer Exercises 7.1 First-order image statistics. Write a MATLAB function named first_order_ stats that computes the first-order statistics of a set of images. Specifically, the function takes as inputs (a) a num_in ⫻ q dimensional array name_images, whose ith row contains the name of the ith image file, (b) the number N _gray specifying the range [0, N _gray ⫺ 1] in which the intensity of the pixels will be scaled. It returns a num_im ⫻ 4 dimensional matrix features, the ith row of which contains the mean,the standard deviation,the skewness and the kurtosis of the intensity of the pixels of the ith image.
Solution In practice, it is more convenient to work with images where the intensity of the pixels lie in a rather small range of values (e.g. [0, 31]), in order to produce smoother histograms and smaller (nonsparse and easier to handle) co-occurence matrices (see next program). This is the reason for the N _gray input argument in the functions of this section. function features=first_order_stats(name_images,N_gray) [num_im,q]=size(name_images); features=zeros(num_im,4); for i=1:num_im A=imread(name_images(i,:)); A=double(A); %Normalization of the pixels intensity in [0,N_gray-1] A=round((N_gray-1)*((A-min(A(:)))/(max(A(:))-min(A(:))))); features(i,1)=mean2(A); features(i,2)=std2(A); features(i,3)=skewness(A(:)); features(i,4)=kurtosis(A(:)); end
7.2 Second-order image statistics. Write a MATLAB function named second_ order_stats that computes second-order statistics of a set of images. Specifically, the function takes as inputs: (a) a num_in⫻q dimensional array name_images, whose ith row contains the name of the ith image file, (b) the number N _gray specifying the range [0, N _gray ⫺ 1] in which the intensity of the pixels will be scaled. For each image, four co-occurence matrices determined by the pixel pairs that are at relative positions (1, 0)◦ , (1, 45◦ ), (1, 90◦ ), (1, 135◦ ) should be computed. The function returns a num_im ⫻ 8 dimensional matrix features, the ith row of which contains (a) in its first four entries: the means of the contrast, the correlation, the angular second moment (in MATLAB termed “Energy”) and the inverse difference moment (in MATLAB termed
“09-Ch07-SA272” 18/9/2008 page 471
MATLAB Programs and Exercises
“Homogeneity”) computed from the four co-occurence matrices of the ith image and (b) in its last four entries: the ranges of the values of the previous features of the ith image.
Solution In the following implementation, we make use of the graycomatrix and graycoprops built-in MATLAB functions. The first computes the co-occurence matrices of an image, while the second is applied on co-occurence matrices and computes the features ‘Contrast’,‘Correlation’,‘Energy’,‘Homogeneity’. function features=second_order_stats(name_images,N_gray) [num_im,q]=size(name_images); features=zeros(num_im,8); for i=1:num_im A=imread(name_images(i,:)); A=double(A); %Normalization of the pixels intensity in [0, N_gray-1] A=round((N_gray-1)*((A-min(A(:)))/(max(A(:))-min(A(:))))); [glcm,SI]=graycomatrix(A,'GrayLimits',[0,N_gray-1],... 'NumLevels',... N_gray,'Offset',[0 1;-1 0;-1 1; -1 -1],'Symmetric',true); stats=graycoprops(glcm,{'Contrast','Correlation',... 'Energy','Homogeneity'}); features(i,1)=mean(stats.Contrast); features(i,2)=mean(stats.Correlation); features(i,3)=mean(stats.Energy); features(i,4)=mean(stats.Homogeneity); features(i,5)=range(stats.Contrast); features(i,6)=range(stats.Correlation); features(i,7)=range(stats.Energy); features(i,8)=range(stats.Homogeneity); end
7.3 Second-order image statistics (masks). Write a MATLAB function named mask_order_stats that takes as input a set of original images. Each one of them is convolved with nine masks, and for each one of the nine resulting images first order statistics are computed. Specifically,the function takes as inputs: (a) a num_in ⫻ q dimensional array name_images, whose ith row contains the name of the ith original image file,(b) the number N _gray specifying the range [0, N _gray ⫺ 1] in which the intensity of the pixels will be scaled. The function should convolve each one of the original images with each one of the nine masks, defined in Section 7.2.2, producing nine (convolved) images for each original image. Then, the first-order statistics (mean, standard deviation, skewness, kurtosis) for each one of the convolved images is computed. The function returns a num_in ⫻ 4 ⫻ 9 three-dimensional matrix features, where
471
“09-Ch07-SA272” 18/9/2008 page 472
472
CHAPTER 7 Feature Generation II
its ith num_im ⫻ 4 two-dimensional component corresponds to the results produced when the ith mask is applied to each one of the original images. Each one of the two-dimensional components is defined as in the first_order_stats function.
Solution function features=mask_stats(name_images,N_gray) [num_im,q]=size(name_images); features=zeros(num_im,4,9); %Definition of the masks mask(:,:,1)=[1 2 1; 2 4 2; 1 2 1]; mask(:,:,2)=[-1 0 1; -2 0 2; -1 0 1]; mask(:,:,3)=[-1 2 -1; -2 4 -2; -1 2 -1]; mask(:,:,4)=[-1 -2 -1; 0 0 0; 1 2 1]; mask(:,:,5)=[1 0 -1; 0 0 0; -1 0 1]; mask(:,:,6)=[1 -2 1; 0 0 0; -1 2 -1]; mask(:,:,7)=[-1 -2 -1; 2 4 2; -1 -2 -1]; mask(:,:,8)=[1 0 -1; -2 0 2; 1 0 -1]; mask(:,:,9)=[1 -2 1; -2 4 -2; 1 -2 1]; % The following is useful in normalizing the convolution result sum_mask=sum(sum(mask))+(sum(sum(mask))==0); for i=1:num_im A=imread(name_images(i,:)); A=double(A); %Normalization of the pixels intensity in [0, N_gray-1] A=round((N_gray-1)*((A-min(A(:)))/(max(A(:))-min(A(:))))); for j=1:9 B=conv2(A,mask(:,:,j),'same')/sum_mask(j); features(i,1,j)=mean2(B); features(i,2,j)=std2(B); features(i,3,j)=skewness(B(:)); features(i,4,j)=kurtosis(B(:)); end end
Computer Experiments Notes: ■
All filenames included in the rows of the name_images array should have the same number of characters.
■
Test images on which the above programs can be applied can be found in www.elsevierdirect.com/9781597492720 (‘ROI _01_seeds.bmp’, ‘ROI _02_seeds.bmp’, . . ., ‘ROI _10_seeds.bmp’). In the sequel, this set of images is called “set of seeds”.
“09-Ch07-SA272” 18/9/2008 page 473
References
7.1 Compute the first-order statistics for the set of seeds, using N _gray ⫽ 32 and comment on the results. 7.2 Compute the second-order statistics for the set of seeds, using N _gray ⫽ 32 and comment on the results. 7.3 Compute the first-order statistics of the nine images produced by each image of the set of seeds, after its convolution with each one of the nine masks, given in Section 7.2.2 and comment on the results. Use N _gray ⫽ 32.
REFERENCES [Alem 90] Al-Emami S., Usher M. “On-line recognition of handwritten Arabic characters,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 12(7), pp. 704–710, 1990. [Arbt 89] Arbter K. “Affine-invariant Fourier descriptors,” in From Pixel to Features (Simon J.C., ed.), pp. 153–164, Elsevier, 1989. [Arbt 90] Arbter K., Snyder W.E., Burkhardt H., Hirzinger G. “Application of affine-invariant Fourier descriptors to recognition of 3-D objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 12, pp. 640–647, 1990. [Ardu 92] Ardunini F., Fioravanti S., Giusto D.D., Inzirillo F.“Multifractals and texture classification,” IEEE International Conference on Image Processing, pp. 454–457, 1992. [Ayac 00] Ayache A., Véhel J.L. “The generalized multifractional Brownian motion,” Statistical Inference for Stochastic Processes,Vol. 3, pp. 7–18, 2000. [Bart 05] Bartch M., Wakefield G.H. “Audio thumbnailing of popular music using chroma-based representations,” IEEE Transactions on Multimedia,Vol. 7(1), pp. 96–104, February 2005. [Bass 92] Bassevile M., Benveniste A., Chou K., Golden S.A., Nikoukhah R.,Willsky A.S. “Modeling and estimation of multiresolution stochastic processes,” IEEE Transactions on Information Theory,Vol. 38, pp. 766–784, 1992. [Brod 66] Brodatz P. Textures—A Photographic Album for Artists and Designers, Dover, 1966. [Brow 91] Brown J.C., Zhang B. “Musical frequency tracking using the methods of conventional and narrowed autocorrelation,” Journal of the Acoustical Society of America,Vol. 89(5), 1991. [Cavo 92] Cavouras D., Prassopoulos P., Pantelidis N. “Image analysis methods for solitary pulmonary nodule characterization by CT,” European Journal of Radiology,Vol. 14, pp. 169–172, 1992. [Chel 85] Chellapa R. “Two dimensional discrete Gaussian Markov random field models for image processing,” in Progress in Pattern Recognition (Kanal L.N., Rosenfeld A., eds.), Vol. 2, pp. 79–112, North Holland, 1985. [Chen 89] Chen C.C., Daponee J.S., Fox M.D. “Fractal feature analysis and classification in medical imaging,” IEEE Transactions on Medical Imaging,Vol. 8, pp. 133–142, 1989. [Chon 03] Chong C.-W., Raveendran P., Mukundan R. “ Translation invariants of Zernike moments,” Pattern Recognition,Vol. 36, pp. 1765–1773, 2003. [Chon 04] Chong C.-W., Raveendran P., Mukundan R. “Translation and scale invariants of Legendre moments,” Pattern Recognition,Vol. 37, pp. 119–129, 2004. [Clau 04] Clausen M., Kurth F. “A unified approach to content-based and fault-tolerant music recognition,” IEEE Transactions on Multimedia,Vol. 6(5), pp. 717–731, October 2004.
473
“09-Ch07-SA272” 18/9/2008 page 474
474
CHAPTER 7 Feature Generation II
[Crim 82] Crimmins T.R. “A complete set of Fourier descriptors,” IEEE Transactions on Systems Man and Cybernetics,Vol. 12, pp. 236–258, 1982. [Cros 83] Cross G.R.,JainA.K.“Markov random field texture models,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 5(1), pp. 25–39, 1983. [Davi 79] Davis L., Johns S.,Aggrawal J.K. “Texture analysis using generalized co-occurrence matrices,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 1(3), pp. 251–259, 1979. [Davi 80] Davis S.B., Mermelstein P. “Comparison of parametric representations of monosyllabic word recognition in continuously spoken sentences,” IEEE Transactions on Acoustics, Speech, and Signal Processing,Vol. 28(4), pp. 357–366, 1980. [Dell 00] Deller J.R., Hansen J.H.L., Proakis J.G. Discrete Processing of Speech Signals, John Wiley & Sons, New York, 2000. [Deri 93] Deriche M., Tewfik A.H. “Signal modeling with filtered discrete fractional noise processes,” IEEE Transactions on Signal Processing,Vol. 41, pp. 2839–2850, 1993. [Falc 90] Falconer K. Fractal Geometry:Mathematical Foundations and Applications,JohnWiley & Sons, 1990. [Fieg 96] Fieguth P.W., Willsky A.S. “Fractal estimation using models on multiscale trees,” IEEE Transactions on Signal Processing,Vol. 41, pp. 1297–1300, 1996. [Flan 92] Flandrin P. “Wavelet analysis and synthesis of fractional Brownian motion,” IEEE Transactions on Information Theory,Vol. 38, pp. 910–917, 1992. [Flus 93] Flusser J., Suk T.“Pattern recognition by affine moment invariants,”Pattern Recognition, Vol. 26(1), pp. 167–174, 1993. [Flus 94] Flusser J.,SukT.“Affine moment invariants: A new tool for character recognition,”Pattern Recognition,Vol. 15(4), pp. 433–436, 1994. [Frag 01] Fragoulis D., Rousopoulos G., Panagopoulos T., Alexiou C., Papaodysseus C. “On the automated recognition of seriously distorted musical recordings,” IEEE Transactions on Signal Processing,Vol. 49(4), pp. 898–908, 2001. [Free 61] Freeman H. “On the encoding of arbitrary geometric configurations,” IRE Transactions on Electronic Computers,Vol. 10(2), pp. 260–268, 1961. [Gall 75] Galloway M. “Texture analysis using gray-level run lengths,” Computer Graphics and Image Processing,Vol. 4, pp. 172–179, 1975. [Gewe 83] Geweke J., Porter-Hudak S. “The estimation and application of long memory time series,” Journal of Time Series Analysis,Vol. 4, pp. 221–237, 1983. [Ghos 97] Ghosal S., Mehrotra R. “A moment based unified approach to image feature detection,” IEEE Transactions on Image Processing,Vol. 6(6), pp. 781–794, 1997. [Glen 94] Glentis G., Slump C., Herrmann O. “An efficient algorithm for two-dimensional FIR filtering and system identification,” SPIE Proceedings,VCIP, pp. 220–232, Chicago, 1994. [Goto 04] Goto M. “A real-time music-scene-description system: Predominant - F0 estimation for detecting melody and bass lines in real-world audio signals,” Speech Communication (ISCA) Journal,Vol. 43(4), pp. 311–329, 2004. [Gran 72] Granlund G.H. “Fourier preprocessing for hand print character recognition,” IEEE Transactions on Computers,Vol. 21, pp. 195–201, 1972. [Hara 73] Haralick R., Shanmugam K., Distein I. “Textural features for image classification,” IEEE Transactions on Systems Man and Cybernetics,Vol. 3(6), pp. 610–621, 1973.
“09-Ch07-SA272” 18/9/2008 page 475
References
[Hayk 96] Haykin S. Adaptive Filter Theory, 3rd ed., Prentice Hall, 1996. [Heyw 95] Heywodd M.I.,Noakes P.D.“Fractional central moment method for movement-invariant object classification,” IEE Proceedings Vision, Image and Signal Processing, Vol. 142 (4), pp. 213–219, 1995. [Hu 62] Hu M.K. “Visual pattern recognition by moment invariants,” IRE Transactions on Information Theory,Vol. 8(2), pp. 179–187, 1962. [Huan 94] Huang Q., Lorch J.R., Dubes R.C. “Can the fractal dimension of images be measured?” Pattern Recognition,Vol. 27(3), pp. 339–349, 1994. [Huan 06] Huang S.-K., Kim W.-Y. “A novel approach to the fast computation of Zernike moments,” Pattern Recognition,Vol. 39(11), pp. 2065–2076, 2006. [Kalo 89] Kalouptsidis N., Theodoridis S. “Concurrent algorithms for a class of 1-D and 2-D Wiener filters with symmetric impulse response,” IEEE Transactions on Signal Processing,Vol. ASSP-37, pp. 1780–1782, 1989. [Kan 02] Kan C., Srinath M.D. “Invariant character recognition with Zernike moments and orthogonal Fourier-Mellin moments,” Pattern Recognition,Vol. 35, pp. 143–154, 2003. [Kapl 99] Kaplan L.M. “Extended fractal analysis for the texture classification and segmentation,” IEEE Transactions on Image Processing,Vol. 8(11), pp. 1572–1585, 1999. [Kapl 94] Kaplan L.M., Kuo C.-C.J. “Extending self similarity for fractional Brownian motion,”IEEE Transactions on Signal Processing,Vol. 42(12), pp. 3526–3530, 1994. [Kapl 95] Kaplan L.M., Kuo C.C.J. “Texture roughness analysis and synthesis via extended selfsimilar model,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 17(11), pp. 1043–1056, 1995. [Kara 95] Karayannis Y.A., Stouraitis T. “Texture classification using fractal dimension as computed in a wavelet decomposed image,” IEEE Workshop on Nonlinear Signal and Image Processing, pp. 186–189, Neos Marmaras, Halkioliki, June 95. [Kash 82] Kashyap R.L., Chellapa R., Khotanzad A. “Texture classification using features derived from random field models,” Pattern Recognition Letters,Vol. 1, pp. 43–50, 1982. [Kash 86] Kashyap R.L., Khotanzad A. “A model based method for rotation invariant texture classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 8(4), pp. 472–481, 1986. [Kell 87] Keller J.M., Crownover R., Chen R.Y. “Characteristics of natural scenes related to fractal dimension,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, pp. 621–627, 1987. [Khot 90a] Khotanzad A., Hong Y.H. “Invariant image recognition by Zernike moments,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 12(5), pp. 489–497, 1990. [Khot 90b] Khotanzad A., Lu J.H. “Classification of invariant image representations using a neural network,” IEEE Transactions on Acoustics Speech and Signal Processing, Vol. 38(6), pp. 1028–1038, 1990. [Klap 03] Klapuri A. “Multiple fundamental frequency estimation by harmonicity and spectral smoothness,” IEEE Transactions on Speech and Audio Processing, Vol. 11(6), pp. 804–816, 2003. [Kuhl 82] Kuhl F.P., Giardina C.R. “Elliptic Fourier features of a closed contour,” Comput. Vis. Graphics Image Processing,Vol. 18, pp. 236–258, 1982. [Lai 81] Lai M.,SuenY.C.“Automatic recognition of characters by Fourier descriptors and boundary line encoding,” Pattern Recognition,Vol. 14, pp. 383–393, 1981.
475
“09-Ch07-SA272” 18/9/2008 page 476
476
CHAPTER 7 Feature Generation II
[Laws 80] Laws K.I. “Texture image segmentation”Ph.D. Thesis, University of Southern California, 1980. [Liao 96] Liao S., Pawlak M. “On image analysis by moments,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 18(3), pp. 254–266, March 1996. [Lin 87] Lin C.S., Hwang C.L. “New forms of shape invariants from elliptic Fourier descriptors,” Pattern Recognition,Vol. 20(5), pp. 535–545, 1987. [Lu 91] Lu S.Y., Ren Y., Suen C.Y. “Hierarchical attributed graph representation and recognition of handwritten Chinese characters,” Pattern Recognition,Vol. 24( 7 ), pp. 617–632, 1991. [Lund 86] Lundahl T., Ohley W.J., Kay S.M., Siffer R. “Fractional Brownian motion: A maximum likelihood estimator and its application to image texture,” IEEE Transactions on Medical Imaging, Vol. 5, pp. 152–161, 1986. [Mahm 94] Mahmoud S. “Arabic character recognition using Fourier descriptors and character contour encoding,” Pattern Recognition,Vol. 27(6), pp. 815–824, 1994. [Mami 98] Mamistvalov A.G. “n-Dimensional moment invariants and the conceptual mathematical theory of recognition n-dimensional objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 20(8), pp. 819–831, 1998. [Mand 68] Mandelbrot B.B, Van Ness J.W. “Fractional Brownian motion, fractional noises and applications,” SIAM Review,Vol. 10, pp. 422–437, 1968. [Mand 77] Manderbrot B.B. The Fractal Geometry of Nature,W.H. Freeman, New York, 1982. [Mao 92] Mao J., Jain A.K. “Texture classification and segmentation using multiresolution simultaneous autoregressive models,” Pattern Recognition,Vol. 25(2), pp. 173–188, 1992. [Mara 93] Maragos P., Sun F.K. “Measuring the fractal dimension of signals: Morphological covers and iterative optimization,”IEEE Transactions on Signal Processing,Vol. 41,pp. 108–121,1993. [Mori 92] Mori S., Suen C. “Historical review of OCR research and development,” Proceedings of IEEE,Vol. 80( 7 ), pp. 1029–1057, 1992. [Muku 95] Mukundan R., Ramakrshnan J. “Fast computation of Legendre and Zernike moments,” Pattern Recognition,Vol. 28(9), pp. 1433–1442, 1995. [Muku 98] Mukundan R., Ramakrshnan J. Moment Functions in Image Analysis–Theory and Applications,World Scientific, Singapore, 1998. [Ohan 92] Ohanian P., Dubes R. “Performance evaluation for four classes of textural features,” Pattern Recognition,Vol. 25(8), pp. 819–833, 1992. [Ojal 96] Ojala T., Pietikainen M., Harwood D. “A comparative study of texture measures with classification based on feature distributions,” Pattern Recognition,Vol. 29(1), pp. 51–59, 1996. [Papo 91] Papoulis A. Probability, Random Variables, and Stochastic Processes,3rd ed.,McGrawHill, 1991. [Pele 84] Peleg S., Naor J., Hartley R.,Anvir D. “Multiple resolution texture analysis and classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 6, pp. 818–523, 1984. [Penn 97] Penn A.I., Loew M.H. “Estimating fractal dimension with fractal interpolation function models,” IEEE Transcations on Medical Imaging,Vol. 16, pp. 930–937, 1997. [Pent 84] Pentland A. “Fractal based decomposition of natural scenes,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 6(6), pp. 661–674, 1984.
“09-Ch07-SA272” 18/9/2008 page 477
References
[Pers 77] Persoon E., Fu K.S. “Shape discrimination using Fourier descriptors,” IEEE Transactions on Systems Man and Cybernetics,Vol. 7, pp. 170–179, 1977. [Petr 06] Petrou M.,Sevilla P.G. Image Processing: Dealing with Texture,John Wiley & Sons,2006. [Pesq 02] Pesquet-Popescu B., Vehel J.L. “Stochastic fractal models for image processing,” IEEE Signal Processing Magazine,Vol. 19(5), pp. 48–62, 2002. [Pico 93] Picone J. “Signal modeling techniques in speech recognition,” Proceedings of the IEEE, Vol. 81(9), pp. 1215–1247, 1993. [Pikr 03] Pikrakis A.,Theodoridis S., Kamarotos D. “Recognition of isolated musical patterns using context dependent dynamic time warping,”IEEE Transactions on Speech and Audio Processing, Vol. 11(3), pp. 175–183, 2003. [Pikr 06] Pikrakis A., Theodoridis S., Kamarotos D. “Classification of musical patterns using variable duration hidden Markov models,” IEEE Transactions on Speech and Audio Processing, to appear in 2006. [Pikr 08] Pikrakis A., Gannakopoulos T., Theodoridis S. “A speech-music discriminator of radio recordings based on dynamic programming and Bayesian networks,” IEEE Transactions on Multimedia,Vol. 10(5), pp. 846–856, 2008. [Pita 94] Pitas I. Image Processing Algorithms, Prentice Hall, 1994. [Plam 00] Plamondon R., Srihari S.N. “On-line and off-line handwriting recognition: A comprehensive survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 22(1), pp. 63–84, 2000. [Proa 92] Proakis J., Manolakis D. Digital Signal Processing: Principles, Algorithms, and Applications, 2nd ed., Macmillan, 1992. [Rabi 93] Rabiner L., Juang B.H. Fundamentals of Speech Recognition, Prentice Hall, Englewood Cliffs, NJ, 1993. [Rabi 78] Rabiner L.R., Schafer R.W. Digital Processing of Speech Signals, Prentice Hall, 1978. [Rand 99] Randen T., Husoy H.H. “Filtering for texture classification: A comparative study,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 21(4), pp. 291–310, 1999. [Reis 91] Reiss T.H. “The revised fundamental theorem of moment invariants,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 13, pp. 830–834, 1991. [Rich 95] RichardsonW.“Applying wavelets to mammograms,”IEEE Engineering in Medicine and Biology,Vol. 14, pp. 551–560, 1995. [Sark 97] Sarkar A., Sharma K.M.S., Sonak R.V. “A new approach for subset 2-D AR model identification for describing textures,”IEEE Transactions on Image Processing,Vol. 6(3), pp. 407–414, 1997. [Saup 91] Saupe D. “Random fractals in image processing,” in Fractals and Chaos (Crilly A.J., Earnshaw R.A., Jones H., eds.), pp. 89–118, Springer-Verlag, 1991. [Schr 68] Schroeder M.R.“Period histogram and product spectrum: New methods for fundamental frequency measurement,” Journal of Acoustical Society of America, Vol. 43(4), pp. 829–834, 1968. [Sing 06] Singh C. “Improved quality of reconstructed images using floating point arithmetic for moment calculation,” Pattern Recognition,Vol. 39(11), pp. 2047–2064, 2006. [Tamu 78] Tamura H., Mori S.,Yamawaki T. “Textural features corresponding to visual Perception,” IEEE Transactions on Systems, Man, and Cybernetics,Vol. 8(6), pp. 460–473, 1978.
477
“09-Ch07-SA272” 18/9/2008 page 478
478
CHAPTER 7 Feature Generation II
[Tang 98] Tang X. “Texture information in run-length matrices,” IEEE Transactions on Image Processing,Vol. 7(11), pp. 1602–1609, 1998. [Taxt 90] Taxt T., Olafsdottir J.B., Daechlen M. “Recognition of hand written symbols,” Pattern Recognition,Vol. 23(11), pp. 1155–1166, 1990. [Teag 80] Teague M.“Image analysis via the general theory of moments,”Journal of Optical Society of America,Vol. 70(8), pp. 920–930, 1980. [Teh 88] Teh C.H., Chin R.T. “On image analysis by the method of moments,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 10(4), pp. 496–512, 1988. [Theo 93] Theodoridis S., Kalouptsidis N. “Spectral analysis,” in Adaptive System Identification and Signal Processing Algorithms (Kalouptsidis N.,Theodoridis S., eds.), Prentice Hall, 1993. [Tolo 00] Tolonen T., Karjalainen M. “A computationally efficient multipitch analysis model,” IEEE Transactions on Speech and Audio Processing,Vol. 8(6), pp. 708–716, November 2000. [Trie 95] Trier O.D.,JainA.K.“Goal-directed evaluation of binarization methods,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 17(12), pp. 1191–1201, 1995. [Trie 96] Trier O.D., Jain A.K., Taxt T. “Feature extraction methods for character recognition—A survey,” Pattern Recognition,Vol. 29(4), pp. 641–661, 1996. [Tson 92] Tsonis A. Chaos: From Theory to Applications, Plenum Press, 1992. [Tzan 02] Tzanetakis G., Cook P. “Musical genre classification of audio signals,” IEEE Transactions on Speech and Audio Processing,Vol. 10(5), pp. 293–302, 2002. [Unse 86] Unser M.“Local linear transforms for texture measurements,”Signal Processing,Vol. 11, pp. 61–79, 1986. [Unse 89] Unser M., Eden M. “Multiresolution feature extraction and selection for texture segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11(7), pp. 717–728, 1989. [Vinc 02] Vinciarelli A. “A survey on off-line cursive word recognition,” Pattern Recognition, Vol. 35, pp. 1433–1446, 2002. [Wang 98] Wang L.,Healey G.“Using Zernike moments for the illumination and geometry invariant classification of multispectral textures,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 7(2), pp. 196–203, 1998. [Wang 93] Wang L., Pavlidis T. “Direct gray-scale extraction of features for character recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 15(10), pp. 1053–1067, 1993. [Wang 00] Wang Y., Huang J.C. “Multimedia content analysis,” IEEE Signal Processing Magazine, Vol. 17(6), pp. 12–36, 2000. [Wee 06] Wee C.-Y., Paramesran R. “Efficient computation of radial moment functions using symmetrical property,” Pattern Recognition,Vol. 39(11), pp. 2036–2046, 2006. [Wold 96] Wold E.,BlumT.,Keislar D.,Wheaton J.“Content-based classification,search,and retrieval of audio,” IEEE Multimedia Magazine,Vol. 22, pp. 27–36, 1996. [Wood 72] Woods J. “Markov image modeling,” IEEE Transactions on Information Theory, Vol. 18(3), pp. 232–240, 1972. [Wood 96] Wood J.“Invariant pattern recognition,”Pattern Recognition,Vol. 29(1),pp. 1–17,1996.
“09-Ch07-SA272” 18/9/2008 page 479
References
[Worn 96] WornellW.G. Signal Processing with Fractals.AWavelet Based Approach,Prentice Hall, 1996. [Wu 03] Wu M., Wang D., Brown G.J. “A multipitch tracking algorithm for noisy speech,” IEEE Transactions on Speech and Audio Processing,Vol. 11(3), pp. 229–241, May 2003. [Zhan 01] Zhang T., Kuo C.C.J. “Audio content analysis for online audiovisual data segmentation and classification,” IEEE Transactions on Speech and Audio Processing, Vol. 9(4), pp. 441–458, 2001.
479
“10-Ch08-SA272” 18/9/2008 page 481
CHAPTER
Template Matching
8
8.1 INTRODUCTION In all previous chapters,the major concern was to assign an unknown pattern to one of the possible classes. The problem that will accompany us throughout this chapter is of a slightly different nature. We will assume that a set of reference patterns (templates) are available to us, and we have to decide which one of these reference patterns an unknown one (the test pattern) matches best. These templates may be certain objects in a scene or can be strings of patterns,such as letters forming words in a written text or words or phrases in a spoken text. Typically, such problems arise in speech recognition, in automation using robot vision, in motion estimation for video coding, and in image database retrieval systems, to name but a few. A reasonable first step to approaching such a task is to define a measure or a cost measuring the“distance”or the“similarity”between the (known) reference patterns and the (unknown) test pattern, in order to perform the matching operation known as template matching. We know by now that each pattern is expressed in terms of a vector or a matrix with elements the set of the selected features. Then why not use one of the already known distance measures, that is, Euclidean or Frobenius norms, and perform the matching operation based on the minimum distance? A little more thinking reveals that such a straightforward approach is not enough and something more is needed. This is the crucial point that makes template matching different and at the same time interesting. To understand this issue better, let us consider a written text matching problem, that is, to identify which one from a set of written words is the word, say,“beauty.” However, because of errors in the reading sensors, the specific test pattern may appear, for example, as “beety” or “beaut.” In a speech recognition task, if a specific word is spoken by the same speaker a number of times, it will be spoken differently every time. Sometimes it will be spoken quickly, and the resulting pattern will be of short duration in time, sometimes slowly, and the pattern will be longer. Yet in all cases it is the “same” word spoken by the same person. In a scene analysis application, the object to be identified may be present in an image, but its location within the image is not known. In content-based image database retrieval systems,
481
“10-Ch08-SA272” 18/9/2008 page 482
482
CHAPTER 8 Template Matching
queries often include the shape of an object. However, the shape provided by the user, most often, does not match exactly the shape of the object residing in the database images. The major goal of this chapter is to define measures that accommodate the distinct characteristics for each category of these problems. As is always the case with a textbook, only general directions and typical cases will be treated. We will begin with the problem of string pattern matching and will deal with the scene analysis and shape recognition problems later on. The tasks, although they share the same goal, require different tools because of their different nature.
8.2 MEASURES BASED ON OPTIMAL PATH SEARCHING TECHNIQUES We will first focus on a category of template matching, where the involved patterns consist of strings of identified symbols or feature vectors (string patterns). That is, each of the reference and test patterns is represented as a sequence (string) of measured parameters, and one has to decide which reference sequence the test-pattern matches best. Let r(i), i ⫽1, 2, . . . , I , and t( j), j ⫽1, 2, . . . , J , be the respective feature vector sequences for a specific pair of reference and test patterns, where in general I ⫽ J . The objective is to develop an appropriate distance measure between the two sequences. To this end, we form a two-dimensional grid with the elements of the two sequences as points on the respective axes, that is, the reference string at the abscissa (i-axis) and the test one at the ordinate ( j-axis). Figure 8.1 is an example for I ⫽ 6, J ⫽ 5. Each point of the grid (node) marks a correspondence between the respective elements of the two sequences. For example, node (3, 2) maps the element r(3) to t(2). Each node (i, j) of the grid is associated with a cost, which is an appropriately defined function d(i, j) measuring the “distance” between the respective elements of the strings, t( j) and r(i). A path through the grid from an initial node (i0 , j0 ) to a final one (if , jf ) is an ordered set of nodes of the form (i0 , j0 ), (i1 , j1 ), (i2 , j2 ), . . . , (if , jf )
Each path is associated with an overall cost D defined as D⫽
K ⫺1
d(ik , jk )
k⫽0
where K is the number of nodes along the path. For the example of Figure 8.1, K ⫽ 8. The overall cost up to node (ik , jk ) will be denoted by D(ik , jk ), and by convention we assume D(0, 0) ⫽ 0 and also d(0, 0) ⫽ 0. The path is said to be complete if (i0 , j0 ) ⫽ (0, 0), (if , jf ) ⫽ (I , J )
“10-Ch08-SA272” 18/9/2008 page 483
8.2 Optimal Path Searching Techniques
j 5
4
3
2
1
0
0
1
2
3
4
5
6
i
FIGURE 8.1 Each point along the path marks a correspondence between the respective elements of the test and reference patterns.
The distance1 between the two sequences is defined as the minimum D over all possible paths. At the same time, the minimum cost path unravels the optimal pairwise correspondence between the elements of the two sequences, which is the crucial part, since the two sequences are of different lengths. In other words, the optimal path procedure makes the alignment or warping of the elements of the test string to the elements of the reference string, corresponding to the best matching score. Before we talk about the optimization procedure, we must point out that there are a number of variations of this scheme. For example, one may not impose the constraint of having necessarily a complete path but may adopt more relaxed constraints known as end point constraints. Furthermore, one could associate a cost not only with each node but also with each transition between nodes, making certain transitions more costly than others. In such cases, the cost at a node (ik , jk ) also depends on the specific transition, that is, from which node (ik⫺1 , jk⫺1 ) the (ik , jk ) node was reached. Thus, the cost d is now of the form d(ik , jk |ik⫺1 , jk⫺1 ) and the overall path cost is D⫽
d(ik , jk |ik⫺1 , jk⫺1 )
k
In some cases the overall path cost is defined as the product D⫽
d(ik , jk |ik⫺1 , jk⫺1 )
k
1 The
term distance here must not be interpreted with its strict mathematical definition.
483
“10-Ch08-SA272” 18/9/2008 page 484
484
CHAPTER 8 Template Matching
Finally, there are cases where d is chosen so that maximization instead of minimization is required. Obviously, in all these variations appropriate initial conditions have to be adopted. Let us now come back to the optimization problem itself. To obtain the best path, one has to search all possible combinations of paths. However, this is a computationally costly procedure. Dynamic programming algorithms based on Bellman’s principle are powerful tools that we will adopt to reduce the computational complexity.
8.2.1 Bellman’s Optimality Principle and Dynamic Programming Let the optimal path between an initial node (i0 , j0 ) and a final one (if , jf ) be denoted as opt
(i0 , j0 ) ⫺→ (if , jf )
If (i, j) is an intermediate node between (i0 , j0 ) and (if , jf ), we will denote the optimal path constrained to pass through (i, j) as opt
(i0 , j0 ) ⫺→ (if , jf ) (i, j)
Bellman’s principle states that [Bell 57] opt
opt
opt
(i0 , j0 ) ⫺→ (if , jf ) ⫽ (i0 , j0 ) ⫺→ (i, j) ⊕ (i, j) ⫺→ (if , jf ) (i, j)
where ⊕ denotes concatenation of paths. In other words,Bellman’s principle states that the overall optimal path from (i0 , j0 ) to (if , jf ) through (i, j) is the concatenation of the optimal path from (i0 , j0 ) to (i, j) and the optimal path from (i, j) to (if , jf ). The consequence of this principle is that once we are at (i, j) through the optimal path, then to reach (if , jf ) optimally we need to search only for the optimal path from (i, j) to (if , jf ). Let us now express this in a way that will be useful to us later on. Assume that we have departed from (i0 , j0 ) and let the kth node of the path be (ik , jk ). The goal is to compute the minimum cost required to reach the latter node. The transition to (ik , jk ) has to take place from one of the possible nodes that are allowed to be in the (k ⫺ 1)th position of the path (that is, the (ik⫺1 , jk⫺1 ) node). This is important. For each node of the grid we assume that there is a set of allowed predecessors, defining the so-called local constraints. Bellman’s principle readily leads to Dmin (ik , jk ) ⫽
min [Dmin (ik⫺1 , jk⫺1 ) ⫹ d(ik , jk |ik⫺1 , jk⫺1 )]
ik⫺1 ,jk⫺1
(8.1)
Indeed, the overall minimum cost to reach node (ik , jk ) is the minimum cost up to node (ik⫺1 , jk⫺1 ) plus the extra cost of the transition from (ik⫺1 , jk⫺1 ) to (ik , jk ). Furthermore, the search for the minimum is constrained only within the set of allowable predecessors for the (ik , jk ) node. This procedure is carried out for all the nodes of the grid. However, in many cases not all nodes of the grid are involved,
“10-Ch08-SA272” 18/9/2008 page 485
8.2 Optimal Path Searching Techniques
and the optimal path searching takes place among a subset of the nodes, which are defined via the so-called global constraints. The resulting algorithm is known as dynamic programming. Equation (8.1) has to be modified accordingly if the cost D is given in its multiplicative form and/or if maximization is required. Let us now focus on our string pattern matching task and see how the recursive equation (8.1) is used to construct the optimal complete path. Figure 8.2 illustrates the procedure. The set of nodes involved in the optimization (global constraints) is denoted as dark dots, and the local constraints, defining the allowable transitions among these nodes, are shown in the figure by the black lines. Having decided to search for the complete path and assuming D(0, 0) ⫽ 0, the respective costs D(i1 , j1 ) for all the allowed nodes involved in step k ⫽ 1 are computed, via (8.1) (in our case there are only two allowable nodes, ((1, 1) and (1, 2)). Subsequently, the costs of the (three) nodes at step k ⫽ 2 are computed, and the procedure is repeated until we arrive at the final node (I , J ). The sequence of transitions leading to the minimum D(I , J ) of the final node defines the minimum cost path, denoted by the red line. The optimal node correspondence, between the test and reference patterns, can then be unraveled by backtracking the optimal path. In the example of Figure 8.2, each step k of the recursion involves only nodes with the same abscissa coordinate, which reflects the local constraints
j
1
2
3
4
5
step k
1
2
3
4
5
i
5
4
3
2
1
0
0
FIGURE 8.2 The optimal path (red line) is constructed by searching among all allowable paths, as defined by the global and local constraints. The optimal node correspondence, between the test and reference patterns, is unraveled by backtracking the optimal path.
485
“10-Ch08-SA272” 18/9/2008 page 486
486
CHAPTER 8 Template Matching
adopted. In general,this is not necessary,and more involved topologies may be used. However, the philosophy of the search for the minimum remains the same. In the following subsections, we will apply the procedure in two different popular application areas. Example 8.1 Figure 8.3 shows the optimal paths (black lines) to reach the nodes at step k ⫽ 3 starting from the nodes at step k ⫽ 0. The grid contains three nodes per step. Only the optimal paths, up to step k ⫽ 3, have been drawn. The goal of this example is to extend the previous paths to the next step and compute the optimal paths terminating at the three nodes at step k ⫽ 4. Bellman’s principle will be employed. Assume that the accumulated costs of the optimal paths Dmin (3, j3 ), j3 ⫽ 0, 1, 2 at the respective nodes are: Dmin (3, 0) ⫽ 0.8, Dmin (3, 1) ⫽ 1.2, Dmin (3, 2) ⫽ 1.0
(8.2)
We are also given the transition costs d(4, j4 |3, j3 ), j3 ⫽ 0, 1, 2, j4 ⫽ 0, 1, 2, in the form of a transition matrix in Table 8.1. In other words, the transition cost, for example, from node (3, 1) to node (4, 2) is equal to 0.2. To obtain the optimal path to node (4, 0) one has to combine the values given in (8.2) and the corresponding transition costs provided in Table 8.1. Thus Total cost for the transition from (3, 0) to (4, 0) is equal to 0.8 ⫹ 0.8 ⫽ 1.6. Total cost for the transition from (3, 1) to (4, 0) is equal to 1.2 ⫹ 0.2 ⫽ 1.4. Total cost for the transition from (3, 2) to (4, 0) is equal to 1.0 ⫹ 0.7 ⫽ 1.7. Applying Eq. (8.1) shows that the optimal path, with the minimum accumulated cost, to reach node (4, 0) is obtained via the transition from node (3, 1). The reader can verify that the optimal paths to the nodes at step k ⫽ 4 are the ones shown in Figure 8.3. The optimal costs associated with nodes (4, 1) and (4, 2) are equal to 1.2 and 1.3, respectively. Transitions from step k ⫽ 3 to k ⫽ 4 are indicated by red lines. Note that, for the case of our example,
j
2
1
0 0
1
2
3
4
i
FIGURE 8.3 Optimal paths for the grid of example 8.1. Red lines correspond to the extensions of the optimal paths from step k ⫽ 3 to step k ⫽ 4.
“10-Ch08-SA272” 18/9/2008 page 487
8.2 Optimal Path Searching Techniques
Table 8.1 Transition Costs Between Nodes for the Example 8.1 Nodes
(4, 0)
(4, 1)
(4, 2)
(3, 0)
0.8
0.6
0.8
(3, 1)
0.2
0.3
0.2
(3, 2)
0.7
0.2
0.3
the path originating from node (0,1) will not take place in the computations if more steps are added, that is, k ⫽ 5,6, . . . . As we say, this path does not survive beyond step k ⫽ 3.
8.2.2 The Edit Distance In this section, we will be concerned with patterns that consist of sets of ordered symbols. For example,if these symbols are letters,then the patterns are words from a written text. Such problems arise in automatic editing and text retrieval applications. Other examples of symbol strings occur in structural pattern recognition. Once the symbols of a (test) pattern have been identified, for example, via a reading device, the task is to recognize the pattern, searching for the best match of it against a set of reference patterns. The measure to be adopted for the matching procedure should take into account the following errors, which may occur during the symbol identification phase.
■
Wrongly identified symbol (e.g.,“befuty” instead of “beauty”) Insertion error (e.g.,“bearuty”)
■
Deletion error (e.g.,“beuty”)
■
Obviously, a combination of these errors may also occur. For the matching procedure we will adopt the philosophy behind the so-called variational similarity. In other words, the similarity between two patterns is based on the “cost” associated with converting one pattern to the other. If the patterns are of the same length,then the cost is directly related to the number of symbols that have to be changed in one of them so that the other pattern results. More interest arises when the two patterns are not of equal length. In such cases symbols have to be either deleted or inserted at certain places of the test string. The location where deletions or insertions are to be made presupposes an optimal alignment (warping) among the symbols of the two patterns. The Edit distance [Dame 64, Leven 66] between two string patterns A and B, denoted D(A, B), is defined as the minimum total number of changes C, insertions I , and deletions R required to change pattern A into pattern B, D(A, B) ⫽ min[C( j) ⫹ I ( j) ⫹ R( j)] j
(8.3)
487
“10-Ch08-SA272” 18/9/2008 page 488
488
CHAPTER 8 Template Matching
where j runs over all possible combinations of symbol variations in order to obtain B from A. To elaborate a bit, note that there is more than one way to change, say, “beuty” to “beauty.” For example, one can either insert “a” after “e” or change “u” to “a” and then insert “u.” We will employ the dynamic programming methodology to compute the required minimum in (8.3). To this end, we form the grid by placing the symbols of the reference pattern in the abscissa axis and the test pattern in the ordinate one. Figure 8.4 demonstrates the procedure via four examples. As already pointed out, the first step in an optimal path searching procedure via dynamic programming
j
j D⫽1
m
m
u
e
e
r
r
d
d d
r
e
a
m
D⫽1
d
i
r
(a)
e
a
m
i
(b)
j m
j
a
D⫽1
m
a
a
e
e
r
r
d
d d
r
e (c)
a
m
i
D⫽0
d
r
e
a
m
i
(d) i ⫺ 1, j i ⫺ 1, j ⫺ 1
i, j
i, j ⫺ 1
FIGURE 8.4 Computation of the Edit distance with (a) an insertion, (b) a change, (c) a deletion, and (d) an equality. The local constraints are shown at the bottom right corner.
“10-Ch08-SA272” 18/9/2008 page 489
8.2 Optimal Path Searching Techniques
techniques is to state the node transition constraints imposed by the problem. For our case of interest, the following constraints are adopted. ■
The cost D(0, 0) of the (0, 0) node is zero.
■
A complete path is searched.
■
Each node (i, j) can be reached only through three allowable predecessors, that is, (i ⫺ 1, j),
(i ⫺ 1, j ⫺ 1),
(i, j ⫺ 1)
as indicated at the bottom of Figure 8.4. The costs associated with the above three transitions are: 1. Diagonal transitions: d(i, j|i ⫺ 1, j ⫺ 1) ⫽
0
if r(i) ⫽ t( j )
1
if r(i) ⫽ t( j )
That is, the cost of a transition is zero if the symbols corresponding to the (i, j) node are the same and unity if they are different; hence a symbol change has to take place. 2. Horizontal and vertical transitions: d(i, j|i ⫺ 1, j) ⫽ d(i, j|i, j ⫺ 1) ⫽ 1
The meaning of horizontal transitions is that they attempt alignment of the two strings by insertion of a symbol; see Figure 8.4a. Thus, they add to the cost, because they imply local mismatch. Similarly, vertical transitions add to the cost because they imply deletions, Figure 8.4c. Incorporating these constraints and the distance (8.3) in a dynamic programming procedure, the following algorithm results. Algorithm for Computing the Edit Distance ■
D(0, 0) ⫽ 0
■
For i ⫽ 1 to I • D(i, 0) ⫽ D(i ⫺ 1, 0) ⫹ 1
■
End { For }
■
For j ⫽ 1 to J • D(0, j) ⫽ D(0, j ⫺ 1) ⫹ 1
■
End { For }
489
“10-Ch08-SA272” 18/9/2008 page 490
490
CHAPTER 8 Template Matching
■
For i ⫽ 1 to I • For j ⫽ 1 to J
c1 ⫽ D(i ⫺ 1, j ⫺ 1) ⫹ d(i, j|i ⫺ 1, j ⫺ 1)
c2 ⫽ D(i ⫺ 1, j ) ⫹ 1
c3 ⫽ D(i, j ⫺ 1) ⫹ 1
D(i, j) ⫽ min(c1, c2, c3)
• End { For } ■
End { For }
■
D(A, B) ⫽ D(I , J )
In other words,we first compute the minimum cost for reaching each node of the grid, starting at (0, 0), and the optimal (complete) path is subsequently constructed. Figure 8.4 shows the respective minimum cost paths and the resulting Edit distances for each of the cases. Verify that any other path for the examples of Figure 8.4 results in a higher cost. The Edit distance is also known as Levenstein distance. Over the years a number of variants of the previous basic Edit distance scheme have been suggested to better address problems rising in various applications. In [Ocud 76] the cost for a change of one symbol to another is allowed to take values different to one, depending on the dependence between different symbols in different applications. For example, for the spelling correction task, it is reasonable to assume that changing an “a” to a “q” results in lower cost than changing an “a” to a “b.” This is because in touch typing the letters “a” and “q” are typed using the same finger whereas “a” and “b” are not. Another generalization as suggested in [Seni 96] allows for merges, splits, and two-letter substitutions in the context of handwriting recognition. A drawback of the basic Edit distance scheme is that it takes no account of the length of the string sequences that are compared. Thus, for example, if two string sequences differ in one symbol, their Edit distance will be equal to one regardless of their length being, say, equal to two or fifty. However, common sense leads us to assume that in the latter case the two strings are more similar than in the former, for which the two sequences share only one out of two symbols. In [Marz 93] the normalization by the length of the corresponding optimal path in the grid is proposed to account for the length of the involved sequences. In [Mei 04] a variant called Markov Edit distance is defined that accounts for the local interactions among adjacent symbols. For example, this modified Edit distance assigns a lower cost to symbol changes in the test pattern if these are reshuffles of the corresponding subpattern in the reference pattern. This is natural since in practice it is not uncommon for one to mess up locally in typing. Taking this into consideration, comparing the reference pattern “beauty” with the test pattern “beuaty”will result in lower Markov Edit distance than comparing the same reference pattern with “besrty,” in contrast to the basic Edit distance that is equal to two for both cases.
“10-Ch08-SA272” 18/9/2008 page 491
8.2 Optimal Path Searching Techniques
The Edit distance and its variants have been used in a number of applications, where the problem can be posed as a string matching, such as polygon matching ([Koch 89]), OCR ([Tsay 93, Seni 96]), stereo vision ([Wang 90]), computational biology, and genome sequence matching ([Durb 97]).
8.2.3 Dynamic Time Warping in Speech Recognition In this section, we highlight the application of dynamic programming techniques in speech recognition. We will focus on the simpler form of the task,known as discrete or isolated word recognition (IWR). That is, we will assume that the spoken text consists of discrete words,well isolated with sufficient silent periods between them. In tasks of this type, it is fairly straightforward to decide where, in time, one word finishes and another one starts. This is not, however, the case in the more complex continuous speech recognition (CSR) systems,where the speaker speaks in a natural way and temporal boundaries between words are not well defined. In the latter case, more elaborate schemes are required (e.g., [Silv 90, Desh 99, Neg 99]). When words are spoken by a single speaker and the purpose of the recognition system is to recognize words spoken by this person, then the recognition task is known as speaker-dependent recognition. A more complex task is speaker-independent recognition. In the latter case, the system must be trained using a number of speakers and the system must be able to generalize and recognize words spoken by people outside the training population. At the heart of any IWR system are a set of known reference patterns and a distance measure (recall the footnote 1 in Section 8.2). Recognition of an unknown test pattern is achieved by searching for the best match between the test and each of the reference patterns, on the basis of the adopted measure. Figures 8.5a and 8.6a show the plots of two time sequences resulting from the sampling of the word “love,” spoken twice by the same speaker. The samples were taken at the output of a microphone at a sampling rate of 22,050 Hz. Although it is difficult to describe the differences, these are readily noticeable. Moreover, the two spoken words are of different duration. The arrows indicate (approximately) the intervals in which the spoken segments lie. The intervals outside the arrows correspond to silent periods. Specifically, the sequence in Figure 8.6a is 0.4 second long, and the sequence in Figure 8.5a is 0.45 second long. Furthermore, it is important to say that this is not the result of a simple linear time scaling. On the contrary, a highly nonlinear mapping is required to obtain a match between these two “same”words spoken by the same person. For comparison, Figure 8.6b shows the plot of the time sequence corresponding to another word,“kiss,” spoken by the same speaker. We will resort to dynamic programming techniques to unravel the nonlinear mapping (warping) required to achieve the optimal match between a test and a reference pattern. To this end, we must first express the spoken words as sequences (strings) of appropriate feature vectors, r(i) i ⫽ 1, . . . , I , for the reference pattern and t( j), j ⫽ 1, . . . , J , for the test one. Obviously, there is more than one way to choose the feature vectors. We will focus on Fourier transform
491
“10-Ch08-SA272” 18/9/2008 page 492
492
CHAPTER 8 Template Matching
s(n)
0.3 0.1
⫺0.1 ⫺0.3 ⫺0.5 ⫺0.7
0
0.4
0.8
1.2
1.6
Number of samples (⫻10
4)
(a) DFT (db)
5
⫺5 ⫺15 ⫺25 ⫺35
0
10
20
30
40
50
Fourier coefficients (b)
FIGURE 8.5 Plots of (a) the time sequence corresponding to the word “love” and (b) the magnitude of the DFT, in dB, for one of its frames.
features. Each of the time sequences involved is divided into successive overlapped time frames. In our case, each frame is chosen to be tf ⫽ 512 samples long and the overlap between successive frames is t0 ⫽100 samples, as shown in Figure 8.7. The resulting number of frames for the speech segment shown in Figure 8.5a is I ⫽ 24, and for the other two they are J ⫽21 (Figure 8.6a) and J ⫽ 23 (Figure 8.6b), respectively. We assume that the former is the reference pattern and the latter two the test patterns. Let xi (n), n ⫽ 0, . . . , 511, be the samples for the ith frame of the reference pattern, with i ⫽ 1, . . . , I . The corresponding DFT is given as n⫽511 1 2 mn , Xi (m) ⫽ √ xi (n) exp ⫺j 512 512 n⫽0
m ⫽ 0, . . . , 511
“10-Ch08-SA272” 18/9/2008 page 493
8.2 Optimal Path Searching Techniques
s(n) 0.3 0.1 20.1 20.3 20.5 20.7
0
0.5
1
1.5 Number of samples (310
2 4)
(a) s(n) 0.5
0.2
20.1
20.4
0
0.6
1.2
1.8
Number of samples (310 4 ) (b)
FIGURE 8.6 Plots of the time sequences resulting from the words (a) “love” and (b) “kiss,” spoken by the same speaker. t0 frame 1
frame 2
frame 3
tf
FIGURE 8.7 Successive overlapping frames for computation of the DFT feature vectors.
Figure 8.5b shows the magnitude of the DFT coefficients for one of the I frames of the reference pattern. The plot is a typical one for speech segments. The magnitude of the higher DFT coefficients is very small, with little contribution to the signal. This justifies use of the first l DFT coefficients as features, where usually
493
“10-Ch08-SA272” 18/9/2008 page 494
494
CHAPTER 8 Template Matching
l ⬍⬍ tf . In our case l ⫽ 50 was considered to be sufficient. Thus,the vector sequence becomes ⎡
⎢ ⎢ r(i) ⫽ ⎢ ⎢ ⎣
Xi (0) Xi (1) .. .
⎤
⎥ ⎥ ⎥, ⎥ ⎦
i ⫽ 1, . . . , I
(8.4)
Xi (l ⫺ 1)
The feature vectors t( j) for each of the test patterns are formed in a similar way. The choice of the DFT coefficients as features is just one of various possibilities that have been suggested and used over the years. Other popular alternatives include the parameters from an AR modeling of the speech segment, the ceptral coefficients (inverse DFT of the logarithm of the magnitude of the DFT coefficients), and so on (e.g.,[Davi 80, Dell 93]). Having completed the preprocessing and feature selection, the reference and test patterns are expressed as (ordered) sequences of feature vectors r(i) and t( j). Our goal becomes to compute the best match among the frames of the test and reference patterns. That is,the test pattern will be stretched in time (one test frame corresponds to more than one frame of the reference patterns) or compressed in time (more than one test frame corresponds to one frame of the reference pattern). This optimal alignment of the vectors in the two string patterns will take place via the dynamic programming procedure. To this end, we first locate the vectors of the reference string along the abscissa and those of the test pattern along the ordinate. Then, the following need to be determined: ■
Global constraints
■
Local constraints
■
End-point constraints
■
The cost d for the transitions
Various assumptions can be adopted for each of these, leading to different results with relative merits. In the sequel, we will focus on some widely used cases.
End-Point Constraints In our example, we will look for the optimal complete path that starts at (0, 0) and ends at (I , J ) and whose first transition is to the node (1, 1). Thus, it is implicitly assumed that the end points of the speech segments (i.e., r(1), t(1) and r(I ), t(J )) match to a fair degree. These can be the vectors resulting from the silent periods just before and just after the speech segments, respectively. A simple variation of the complete path constraints results if the end points of the path are not specified a priori and are assumed to be located within a distance ⑀ from points (1, 1) and (I , J ). It is left to the optimizing algorithm to locate them.
“10-Ch08-SA272” 18/9/2008 page 495
8.2 Optimal Path Searching Techniques
j J j⫽ J⫺
1 1 i⫹ (J ⫺ I) 2 2
j ⫽ 2i⫺ 1 j ⫽ 2i ⫹ ( J ⫺ 2I )
1⫹ j⫽
1 1 i⫹ 2 2
1 1
1⫹
I⫺
I
i
FIGURE 8.8 Itakura global constraints. The maximum compression/expansion factor is 2, and it determines the slope of the boundary line segments. The red lines correspond to the same global constraints when the relaxed end-point constraints are adopted.
Global Constraints The global constraints define the region of nodes that are searched for the optimal path. Nodes outside this region are not searched. Basically, the global constraints define the overall stretching or compression allowed for the matching procedure. An example is shown in Figure 8.8. They are known as Itakura constraints and impose a maximum factor of 2 for any expansion or compression of the test with respect to the reference pattern. The allowable nodes are then located within the parallelogram shown in Figure 8.8 by the black line. The red lines correspond to the same global constraints when the relaxed end-point constraints, mentioned before, are adopted. Observe from the figure that paths across the sides of the parallelogram compress or expand corresponding frame intervals by a factor of 2, and this is the maximum possible factor attained. This constraint is usually reasonable and at the same time it reduces the number of nodes to be searched for the optimal path substantially. If I ≈ J , then it is not difficult to show that the number of grid points to be searched is reduced by approximately one-third.
Local Constraints These constraints define the set of predecessors and the allowable transitions to a given node of the grid. Basically, they impose limits for the maximum expansion/
495
“10-Ch08-SA272” 18/9/2008 page 496
496
CHAPTER 8 Template Matching
compression rates that successive transitions can achieve. A property that any local constraint must satisfy is monotonicity. That is, ik⫺1 ⱕ ik
and
jk⫺1 ⱕ jk
In other words, all predecessors of a node are located to its left and south. This guarantees that the matching operation follows the natural time evolution and avoids confusing, for example, the word “from” with the word “form.” Two examples of nonmonotonic paths are shown in Figure 8.9. A popular set of local constraints,known as the Itakura constraints [Itak 75],is shown in Figure 8.10. The maximum achievable expansion (compression) rate over a local path is measured by the associated slope, which is defined as the maximum ratio of the total change ⌬i, in the ith direction, to the total change ⌬j, in the jth direction, over the local path. The slope for the Itakura constraints is 2, and it is the result of a (repetitive) transition of the type (i ⫺ 1, j ⫺ 2) to (i, j). Another notable characteristic of the Itakura constraints is that horizontal transitions are allowed, but not successively, and this is indicated by the cross over the arrow. Thus, Itakura constraints do not allow long horizontal paths, corresponding to ⬁ slopes. Finally, these constraints allow the path to skip at most one feature vector in the test pattern string, that is, the one at the j ⫺ 1 position of the ordinate axis, and the path jumps from (i ⫺ 1, j ⫺ 2) to (i, j). In contrast, feature vectors in the reference string are not skipped, and all take part in the optimal path. Such constraints are known as asymmetrical. j 5
4
3
2
1
0
0
1
2
3
4
5
6
i
FIGURE 8.9 Examples of nonmonotonic paths. Such paths are not allowed and are not considered in the search for the optimum.
“10-Ch08-SA272” 18/9/2008 page 497
8.2 Optimal Path Searching Techniques
i ⫺ 2, j
i, j
i ⫺ 1, j
i ⫺ 2, j ⫺ 1
i ⫺ 1, j ⫺ 2
FIGURE 8.10 The Itakura local constraints. Two successive horizontal transitions are not allowed.
(a)
(b)
(c)
(d)
FIGURE 8.11 The Sakoe and Chiba local constraints.
A number of alternative local constraints have also been suggested and used in practice. Figure 8.11 shows four different types of constraints considered by Sakoe and Chiba [Sako 78]. For the type in Figure 8.11a, there is no limit in the rate of expansion/compression, since successive horizontal or vertical transitions can take place, until of course one falls outside the region defined by the global constraints. In contrast, in Figure 8.11b horizontal (vertical) transitions are allowed only after a diagonal transition and in Figure 8.11d after two successive diagonal transitions. In Figure 8.11c, at most two successive horizontal (vertical) transitions are allowed only after a diagonal one. The slopes for each of the constraints in Figures 8.11a, b, c and d are ⬁, 2, 3, and 3/2, respectively (Problem 8.2).
497
“10-Ch08-SA272” 18/9/2008 page 498
498
CHAPTER 8 Template Matching
For a more detailed treatment of the topic, the interested reader may consult [Dell 93, Silv 90, Myer 80].
The Cost A commonly used cost, which we will also adopt here, is the Euclidean distance between r(ik ) and t(jk ), corresponding to node (ik , jk ), that is, d(ik , jk |ik⫺1 , jk⫺1 ) ⫽ r(ik ) ⫺ t(jk ) ≡ d(ik , jk )
In this, we assume that no cost is associated with the transitions to a specific node, and the cost depends entirely on the feature vectors corresponding to the respective node. Other costs have also been suggested and used [Gray 76, Gray 80, Rabi 93]. More recently ([Pikr 03]) used a cost that accounts for the most commonly encountered errors (e.g.,different players style) in the context of music recognition. Finally, it must be stated that often a normalization of the overall cost D is carried out. This is to compensate for the difference in the path lengths, offering “equal” opportunities to all of them. A logical choice is to divide the overall cost D by the length of each path [Myer 80]. The resulting overall costs for the two test patterns of Figure 8.6, against the reference pattern of Figure 8.5,using the Itakura constraints,were D ⫽ 11.473, D ⫽ 25.155, respectively. Thus, the overall cost for the word “love” is lower than the overall cost for the word “kiss,” and our procedure has recognized the spoken word correctly. The resulting normalized overall costs after dividing by the number of nodes along each path were 0.221 and 0.559, respectively.
8.3 MEASURES BASED ON CORRELATIONS The major task to be addressed in this section can be summarized as follows: “Given a block of recorded data, find whether a specific known (reference) pattern is contained within the block and where it is located.”A typical application of this is found in scene analysis, when we want to search for specific objects within the image. Such problems arise in many applications, such as target detection, robot vision, and video coding. For example, in video coding a major step is that of motion estimation—that is, the process of locating corresponding pixels (of the same moved object) among successive image frames at different time instants. This step is then followed by the motion compensation stage, which compensates for the displacement of moving objects from one frame to another. One then codes the frame difference e(i, j, t) ⫽ r(i, j, t) ⫺ r(i ⫺ m, j ⫺ n, t ⫺ 1)
where r(i, j, t) are the pixel gray levels of the image frame at time t and r(i ⫺ m, j ⫺ n, t ⫺ 1) the corresponding pixel values at spatial locations i ⫺ m,
“10-Ch08-SA272” 18/9/2008 page 499
8.3 Measures Based on Correlations
j ⫺ n of the previous frame at time t ⫺ 1. In this way, we code only the new information contained at the latest frame, avoiding redundancies. Let us assume that we are given a reference pattern expressed as an M ⫻ N image array r(i, j), i ⫽ 0, . . . , M ⫺ 1, j ⫽ 0, . . . , N ⫺ 1, and an I ⫻ J image array t(i, j), i ⫽ 0, . . . , I ⫺ 1, j ⫽ 0, . . . , J ⫺ 1, where M ⱕ I , N ⱕ J . The goal is to develop a measure for detecting an M ⫻ N subimage within t(i, j) that matches best the reference pattern r(i, j). To this end, the reference image r(i, j) is superimposed on the test image, and it is translated to all possible positions (m, n) within it. For each of the points (m, n), the mismatch between r(i, j) and the M ⫻ N subimage of t(i, j) is computed according to D(m, n) ⫽
m⫹M⫺1 n⫹N ⫺1 i⫽m
|t(i, j) ⫺ r(i ⫺ m, j ⫺ n)|2
(8.5)
j⫽n
Template matching is conducted by searching for the location (m, n) for which D(m, n) is minimum. Let us now give this a computationally more attractive form. Equation (8.5) is equivalent to D(m, n) ⫽
i
|t(i, j)|2 ⫹
j
⫺2
i
i
|r(i, j)|2
j
t(i, j)r(i ⫺ m, j ⫺ n)
(8.6)
j
The second summand is constant for a given reference pattern. Assuming that the first one does not change much across the image,that is,there is not much variation of the pixel gray levels over the test image, the minimum of D(m, n) is achieved when c(m, n) ⫽
i
t(i, j)r(i ⫺ m, j ⫺ n)
(8.7)
j
is maximum for all possible locations (m, n). The quantity c(m, n) is nothing other than a cross-correlation sequence between t(i, j) and r(i, j) computed at the point (m, n). In cases for which the assumption of little gray-level variation is not valid, this measure is very sensitive to gray-level variations within t(i, j). In such cases the cross-correlation coefficient, defined as cN (m, n) ⫽ i
c(m, n) 2 2 j |t(i, j)| i j |r(i, j)|
(8.8)
is a more appropriate measure. Here, cN (m, n) is a normalized version of c(m, n), and variations in t(i, j) tend to cancel out. Recall now the Cauchy–Schwarz inequality ⱕ t(i, j)r(i ⫺ m, j ⫺ n) |t(i, j)|2 |r(i, j)|2 i
j
i
j
i
j
499
“10-Ch08-SA272” 18/9/2008 page 500
500
CHAPTER 8 Template Matching
Equality holds if and only if t(i, j) ⫽ ␣r(i ⫺ m, j ⫺ n),
i ⫽ m, . . . , m ⫹ M ⫺ 1 and
j ⫽ n, . . . , n ⫹ N ⫺ 1
with ␣ being an arbitrary constant. Hence, cN (m, n) is always less than unity and achieves its maximum value of one only if the (test) subimage is the same (within a scaling factor) as the reference pattern. In our discussion so far,we have assumed that the reference pattern has only been translated within t(i, j) and no rotation or scaling has been involved. In applications such as video coding, this is a valid assumption and it has been adopted in the video coding standards [Bhas 95]. However, this is not always the case and the technique has to be modified. One way is to describe the reference and test subimages in terms of invariant moments and measure the similarity using correlations involving these moments [Hall 79] (Problem 8.4). Another rotation- and scale-invariant technique, using a combination of the Fourier and Mellin transforms, is described in [Scha 89]. This technique tries to exploit the translation invariance of the magnitude of the Fourier transform (already discussed in Chapter 7) and the scale invariance of the Mellin transform ([Ravi 95], Problem 8.5). Another path, which of course demands high computational resources, is to have a set of distorted (e.g., rotated and scaled) reference templates to cover all possibilities. Correlation matching will then reveal the best match between a test pattern and one of the reference templates. A computationally more attractive technique is to employ the Karhunen–Loève transform [Ueno 97]. The main idea is that rotated templates are highly correlated, and each of them can be approximated by its projection onto a lower dimension eigenspace, using the most significant eigenvectors of their correlation matrix. Matching of an unknown pattern with the template of the right orientation is performed in the lower dimensional space, leading to substantial computational savings. Example 8.2 The image t(i, j) in Figure 8.12a contains two objects, a screwdriver and a hammer. The latter is the object that we want to search for in the image. The reference image is shown at the top right corner of Figure 8.12a. The dotted area represents the general (m, n) position of the reference image when it is superimposed on the test one. Figure 8.12b shows the cross-correlation c(m, n) between the two images. We readily observe that the maximum (black) occurs at the position (13, 66), that is, where the hammer is located in t(i, j).
Computational Considerations ■
In some cases, it is more efficient to compute the cross-correlation via its Fourier transform. Recall that in the frequency domain (8.7) is written as C(k, l) ⫽ T (k, l)R∗ (k, l)
(8.9)
“10-Ch08-SA272” 18/9/2008 page 501
8.3 Measures Based on Correlations
m
I
I
M N
n
66
J
J 13 (a)
(b)
FIGURE 8.12 Example of (a) reference and test images and (b) their respective correlation.
where T (k, l), R(k, l) are the DFT transforms of t(i, j) and r(i, j),respectively, with “∗” denoting complex conjugation. Of course, in order to write (8.9), both images must be of the same size. If they are not, which is usually the case, a number of zeros must be appended to extend the smaller sized image. c(m, n) is obtained via the inverse DFT of C(k, l). Taking into account the computational efficiency of the FFT, this procedure may lead to savings depending, of course, on the relative size of M, N and I , J . ■
A major computational load in correlation-based template matching is searching over the pixels of t(i, j) in order to locate the maximum correlation. Usually, the search is restricted within a rectangle [⫺p, p] ⫻ [⫺p, p] centered at a point (x, y) in t(i, j). For example, in video coding, if the M ⫻ N block is centered at a position (x, y) in the frame at time t ⫺ 1, then the current frame is searched within (x ⫾ p, y ⫾ p). The value of p depends on the application. For broadcast TV p ⫽ 15 is sufficient. For sporting events (high motion) p ⫽ 63 is more appropriate. Thus, an exhaustive search for the maximum of c(m, n), defined in (8.7), will require a number proportional to (2p ⫹ 1)2 MN additions and multiplications. This leads to a huge number of operations indeed (Problem 8.6). Thus, in practice, suboptimal heuristic searching techniques are usually adopted, which, although they do not guarantee locating the maximum, reduce the required number of operations substantially. There are two major directions. One is to reduce the search points and the other is to reduce the size of the involved images.
Two-Dimensional Logarithmic Search Logarithmic Search Figure 8.13 shows the rectangular [⫺p, p] ⫻ [⫺p, p] searching area for the case of p ⫽ 7. The center of the rectangle is assumed to be the point (0, 0). The cross-correlation computation is first performed at the center as well
501
“10-Ch08-SA272” 18/9/2008 page 502
502
CHAPTER 8 Template Matching
(⫺7,7)
(0,7)
(7,7)
(⫺7,0)
(7,0)
d1
(⫺7,⫺7)
(0,⫺7)
(7,⫺7)
FIGURE 8.13 Logarithmic search to find the point of maximum cross-correlation.
as the eight points located on the perimeter of the inner [⫺p/2, p/2] ⫻ [⫺p/2, p/2] area (p/2 rounded to an integer). These points are denoted by a square. The spacing between these points is d1 ⫽ 4 pixels, that is, d1 ⫽ 2k⫺1 and k ⫽ log2 p , where · denotes rounding to the first larger integer. For p ⫽ 7 we get k ⫽ 3 and d1 ⫽ 4. We will demonstrate the procedure via an example. Let us assume that the largest cross-correlation value resulted at the position (⫺4, 0) (shaded square). Then we consider this point as the center of a rectangle of size [⫺p/4, p/4] ⫻ [⫺p/4, p/4] ([⫺2, 2] ⫻ [⫺2, 2] in our case) and compute the correlation at the eight points of its perimeter. These points are denoted by a circle, and the spacing between them is now d2 ⫽ d1 /2 (2). The process is repeated, and finally the computation is performed on the eight (diamond) points on the perimeter of the rectangle of size [⫺1, 1] ⫻ [⫺1, 1], which is centered at the optimum (of the previous step) shaded circle point. The spacing between the diamond points is d3 ⫽ 1. The shaded diamond corresponds to the point with the maximum cross-correlation, and the process is terminated. The number of computations has now been reduced to MN (8k ⫹ 1) operations, which is a substantial saving compared with the exhaustive search. A variant of the two-dimensional logarithmic search is to search the i and j directions independently. The point whose coordinates are the resulting best values of i and j becomes the new origin of the coordinate system, and the search is repeated in the new i, j directions, with smaller spacing d. The process is repeated until the spacing becomes unity.
Hierarchical Search The hierarchical search technique springs from the multiresolution concept considered in Chapter 6. Let us again consider an example.
“10-Ch08-SA272” 18/9/2008 page 503
8.3 Measures Based on Correlations
■
■
Step 1: A reference block of,say,16 ⫻ 16 is given,and the search area is assumed to be the rectangle [⫺p, p] ⫻ [⫺p, p], centered at the point (x, y) in the test image. We refer to level 0 versions of the images. Both the reference block and the test image are low-pass filtered and subsampled by 2, resulting in their level 1 versions. The total number of pixels in the level 1 versions has been reduced by 4. Level 1 versions are in turn low-pass filtered and subsampled, resulting in level 2 versions. In general, this process can continue. Step 2: At level 2 the search for the maximum takes place with the 4 ⫻ 4 low-pass version of the reference block. The search area in the level 2 low-pass version of the test image is the rectangle [⫺p/4, p/4] ⫻ [⫺p/4, p/4] centered at (x/4, y/4). Either a full or a logarithmic search can be employed. Let (x1 , y1 ) be the coordinates of the optimum, with respect to (x/4, y/4).
■
Step 3: At level 1, the search for the maximum is performed using the corresponding 8 ⫻ 8 version of the reference block. The search area, within the level 1 version of the test image, is the rectangle of size [⫺1, 1] ⫻ [⫺1, 1] centered at (x/2 ⫹ 2x1 , y/2 ⫹ 2y1 ), that is, nine pixels in total. This is because the eight pixels at the perimeter of this area were not involved at level 2, due to the subsampling (see Figure 6.23 of Chapter 6). The center point must also be included in order to have a fair comparison at this level for the search for the maximum. Let the maximum occur at (x2 , y2 ) with respect (x/2, y/2).
■
Step 4: At level 0 the search is performed using the original reference template, within the area of size [⫺1, 1] ⫻ [⫺1, 1] centered at (x ⫹ 2x2 , y ⫹ 2y2 ) in the test image. The location of the maximum is the final one and the process terminates.
The computational saving with this method depends on the number of levels, as well as the type of search adopted at the highest level (Problem 8.5). In general, hierarchical methods are very efficient from a computational point of view. This is gained at the expense of increased memory requirements, due to the various image versions that must be kept. A disadvantage of the method is that if small objects are present in the templates, they may disappear at the lowest resolution images due to the subsampling. Furthermore, the method cannot guarantee to find the global best match. In [Alkh 01] an alternative philosophy is suggested that results in the global best match. Computational savings are achieved by pruning the number of candidates for the best match in a level, using the results in a higher level and an appropriately chosen threshold value.
Sequential Method A number of other techniques have also been suggested. For example,the sequential search method computes a variant of (8.5) directly. Specifically, define Dpq (m, n) ⫽
p⫺1 q⫺1 i⫽0 j⫽0
|t(i ⫹ m, j ⫹ n) ⫺ r(i, j)|
(8.10)
503
“10-Ch08-SA272” 18/9/2008 page 504
504
CHAPTER 8 Template Matching
Thus, the error is computed in a smaller and sequentially increasing window area, for p, q ⫽ 1, 2, . . . and p ⱕ M, q ⱕ N . The computations stop when Dpq (m, n) becomes larger than a predetermined threshold. Then computations start in a different direction (m, n). Hence, saving is achieved, because for bad positions only a small number of computations need to be performed.
8.4 DEFORMABLE TEMPLATE MODELS In the previous section, we considered the problem of searching for a known reference pattern (template) within a test image. We assumed that the template and the object, residing in the image, were identical. The only differences that were allowed to enter into our discussion were those imposed by a different orientation and/or scaling. However,there are many problems where we know a priori that the available template and the object we search for in the image may not look exactly the same. This may be due to varying imaging conditions, occlusion, and imperfect image segmentation. Furthermore, in a content-based image database retrieval system, the user may provide the system with a sketch of the shape of the object to be retrieved. Obviously, the sketch will not match exactly the corresponding object in the database images. Our goal in this section is to allow the template matching procedure to account for deviations between the reference template and the corresponding test pattern in the image. In our discussion, we will assume that the reference template is available in the form of an image array containing the object’s boundary information (contour). That is, we will focus on shape information only. Extensions incorporating more information, for example, texture, are also possible. Let us denote the reference template image array as r(i, j). This is also known as prototype. The basic idea behind the deformable template matching procedure is simple: Deform the prototype and produce deformed variants of it. From a mathematical point of view a deformation consists of the application of a parametric transform T on r(i, j) to produce a deformed version T [r(i, j)]. Different values of the vector parameter lead to different versions. From the set of the deformed prototype variants that can be generated, there will be one that “best” matches the test pattern. The goodness of fit is measured via a cost, which we will call the matching energy Em (). Obviously, the goal is to choose so that Em () is minimum. However, this is not enough. If, for example, the optimal set of parameters is such that the corresponding deformed template has been deformed to such an extent that it bears little resemblance to the original prototype, the method will be meaningless. Thus, one more term has to be taken into account in the optimization process. This is the cost measuring the “deformation,” which the prototype needs to undergo in order to fit the test pattern. We will call this term of the cost deformation energy Ed (). Then the optimal vector parameter is computed so that : min{Em () ⫹ Ed ()}
(8.11)
“10-Ch08-SA272” 18/9/2008 page 505
8.4 Deformable Template Models
In words,one could think of the boundary curve of the prototype as made by rubber. Then with the help of a pencil we deform the shape of the rubber curve to match the test pattern. The more we deform the shape of the prototype, the higher the energy we have to spend for it. This energy, quantified by Ed (), depends on the shape of the prototype. That is, it is an intrinsic property of the prototype, and this is the reason that it is also known as internal energy. The other energy term, Em (), depends on the input data (test image), and we usually refer to it as external energy. The optimal vector parameter,,is chosen so that the best trade-off between these two energy terms is achieved. Sometimes, a weighting factor C is used to give preference to one of the two terms, and is computed so that : min{Em () ⫹ CEd ()}
(8.12)
Hence, in order to apply the above procedure in practice, one must have at one’s disposal the following ingredients: ■ ■ ■
A prototype A transformation procedure to deform the prototype The two energy function terms
Choice of the Prototype This should be carefully chosen so that it is a (typical) representative of the various instances in which this object is expected to appear in practice. In a way, the prototype should encode the “mean shape” characteristics of the corresponding “shape class.”
Deformation Transformation This consists of a set of parametric operations. Let (x, y) be the (continuous) coordinates of a point in a two-dimensional image. Without loss of generality, assume that the image is defined in a square [0, 1] ⫻ [0, 1]. Then each point (x, y) is mapped using a continuous mapping function, as (x, y) ⫺→(x, y) ⫹ (D x (x, y), D y (x, y))
(8.13)
For discrete image arrays a quantization step is necessary after the transformation. Different functions can be used to perform the above mapping. A set that has successfully been used in practice is ([Amit 91]) D x (x, y) ⫽
M N
x x mn emn (x, y)
(8.14)
m⫽1 n⫽1
D y (x, y) ⫽
M N
y
y
mn emn (x, y)
(8.15)
m⫽1 n⫽1 x emn (x, y) ⫽ ␣mn sin nx cos my y emn (x, y)
⫽ ␣mn cos mx sin ny
(8.16) (8.17)
505
“10-Ch08-SA272” 18/9/2008 page 506
506
CHAPTER 8 Template Matching
for appropriately chosen values of M, N . The normalizing constants ␣mn can be taken as ␣mn ⫽
1 2 (n2 ⫹ m2 )
Other basis functions can also be used, such as splines or wavelets. Figure 8.14 shows a prototype for an object and three deformed versions obtained for the simplest case of the transformation model in (8.14)–(8.17), that is, M ⫽ N ⫽ 1.
Internal Energy This should be minimum for no deformation, that is, for ⫽ 0. A reasonable choice, associated with the transformation functions considered above, is Ed () ⫽
y x 2 ((mn ) ⫹ (mn )2 ) m
(8.18)
n
External Energy Here again a number of choices are possible, measuring the goodness of fit between the test pattern and each one of the deformed template variants. For example, for a specific position,orientation,and scale of a deformed template,this energy term can be measured in terms of the distance of each point in the contour of the deformed
(c)
(b)
(a)
(d)
(e)
FIGURE 8.14 (a) A reference pattern, (b) its contour used as prototype, and (c), (d), (e) three of its deformed variants.
“10-Ch08-SA272” 18/9/2008 page 507
8.4 Deformable Template Models
template from the nearest point in the contour of the test image, I . One way to achieve this is via the following energy function: Em (, , I ) ⫽
1 (1 ⫹ ⌽(i, j)) Nd
(8.19)
i,j
where is the vector of the parameters defining the position,orientation,and scaling and Nd the number of contour pixels of the corresponding deformed template and ⌽(i, j) ⫽ ⫺ exp ⫺(␦2i ⫹ ␦2j )1/2
(8.20)
where a constant and (␦i , ␦j ) is the displacement of the (i, j) pixel of the deformed template from the nearest point in the test image. In [Jain 96] directional information is also incorporated into the cost. Remarks ■
One can arrive at (8.11) in a more systematic way via probabilistic arguments, that is, by seeking to maximize the a posteriori probability density of (, ) given the test image array, that is, p(, |I ) ⫽
p(, )p(I |, ) p(I )
(8.21)
where the Bayes rule has been employed. In this framework, (8.18) results x , y are statistically indeif one assumes that the various parameters mn mn pendent and are normally distributed, for example, p() ⫽ N (0, 2 ). The higher the variance 2 , the wider the range of the values that occur with high probability. To obtain (8.19)–(8.20) the model p(I |, ) ⫽ ␣ exp ⫺Em (, , I )
(8.22)
is adopted, where ␣ is a normalizing constant [Jain 96]. ■
The cost in (8.11) is a nonlinear function,and its minimization can be achieved via any nonlinear optimization scheme. Besides complexity, the major drawback is the omnipresent danger that the algorithm will be trapped in a local minimum. In [Jain 96] a procedure has been suggested that builds around the gradient descent methodology (Appendix C). The idea is to adopt a multiresolution iterative approach and use larger values of in (8.20) for the coarser levels and smaller values for the finer ones. This procedure seems to help the algorithm to avoid local minima, at an affordable computing cost.
■
The methodology we described in this section belongs to a more general class of deformable template matching techniques,which builds around an available prototype. This is not the only available path. Another class of deformable models stems from an analytic description of the prototype shape, using a set of parameterized geometrical shapes,for example,ellipses or parabolic curves.
507
“10-Ch08-SA272” 18/9/2008 page 508
508
CHAPTER 8 Template Matching
To delve deeper into these issues the reader may refer to the review articles [Jain 98, McIn 96, Cheu 02] and the references therein. [Widr 73, Fisc 73] seem to be the first attempts to introduce the concept of deformable models in computer vision. ■
In the pattern recognition context, given an unknown test pattern, we seek to see to which one from a known set of different prototypes this matches best. For each prototype, the best deformed variant is selected, based on the minimum energy cost. The prototype that wins is the one whose best deformed variant results in the overall minimum energy cost.
8.5 CONTENT-BASED INFORMATION RETRIEVAL: RELEVANCE FEEDBACK With the rapid development and spread of the Internet,a large corpus of information is stored and distributed over the Web. Search engines have become indispensable tools for searching and retrieving information in all possible forms, including text, images, audio and more recently video. The more traditional way of information retrieval is text-based; stored information is manually annotated by text descriptors, which are in turn used by a distributed database system to perform the information retrieval task. Such a procedure has the obvious drawback of requiring manual annotation, which, besides being time consuming and costly, is vulnerable to annotation inaccuracies and also to the subjectivity of the human perception. Due to the advances in pattern recognition, an alternative search procedure, known as content-based retrieval, is gaining in importance, and it is becoming more and more popular. Stored information is now indexed and searched based on its content. For example, in image retrieval, images can be indexed automatically by using features “describing” image content qualities such as color, texture and shapes. A music or speech segment could be represented in terms of a number of features such as those described in Section 7.5. Content-based information retrieval is similar in concept to template matching, as introduced in this chapter. The goal is to search for and retrieve stored pieces of information, that is, patterns/templates that are most “similar” to the pattern presented as input to the search engine system. Similarity is quantified in terms of a similarity measure defined in the feature space. The similarity measures described at the beginning of this chapter may be possible candidates for some content-based retrieval tasks. A popular metric that has extensively been used is the weighted lp metric between two feature vectors x, y, given by: d(x, y) ⫽
l i⫽1
p1 wi |xi ⫺ yi |p
“10-Ch08-SA272” 18/9/2008 page 509
8.5 Content-Based Information Retrieval: Relevance Feedback
Obviously, for p ⫽ 2 and wi ⫽ 1, i ⫽ 1, 2, . . . , l, this becomes the Euclidean distance and for p ⫽ 1 the so called weighted l1 (Manhattan) norm. A more detailed treatment of similarity/dissimilarity measures is given in Chapter 11. A major disadvantage of such a contend-based approach is that search and subsequent retrieval is based solely on the derived features, which are usually referred to as low-level features. Humans, being much more intelligent than the machines they themselves develop, utilize a number of so-called high-level concepts when they come to recognize objects (patterns). The notion of semantic gap is usually adopted to express this discrepancy between the low-level features, derived from and describing the patterns, and the high-level descriptions that are meaningful to a human. Artificial Intelligence is the discipline that focuses on developing methodologies and techniques for machine high-level reasoning in association with the low-level derived features. However, so far, such techniques are feasible and applicable only to restricted domains and applications. The goal of the current section is a more humble one, compared to the goals originally set in the field of artificial intelligence, and yet very interesting. Since learning machines cannot compete with the high-level concepts and reasoning of a human being,let the user be involved and become part of the learning“game”. Such a methodology offers the system the advantage of exploiting the user’ s own way of conceptualizing the patterns, which are experienced through his or her senses. To this end, the search/retrieval session is divided into a number of consecutive loops. At every loop, the user provides feedback regarding the results by characterizing the retrieved patterns as either relevant or irrelevant. Relevance is usually defined by a characteristic that is shared by some of the patterns. It can be a perceptual characteristic or a more semantic one [Cruc 04]. Such a methodology is known as relevance feedback (RF). Since the user is directly involved, for the learning process to be useful in practice, convergence should be achieved within a few iteration steps and the search engine must operate in real time. In turn, this imposes the constraint that the selected (low-level) features must be as informative as possible. Thus the feature selection techniques, as exposed in Chapter 5, are of significance here too. One of the most successful paradigms for RF is the case of content-based image retrieval (CBIR), for which commercial products are already available; see, for example, [Liu 07]. A typical scenario, met in a number of RF tasks, is given below and it is schematically presented in Figure 8.15. 1. The system provides an initial set of patterns“similar”to the pattern presented by the user to the search engine (e.g., an image, a Web page, or an audio segment from a piece of music). 2. The user marks a number from the returned patterns as “relevant” or “irrelevant.” 3. A classification procedure is used to “learn” the user’ s feedback.
509
“10-Ch08-SA272” 18/9/2008 page 510
510
CHAPTER 8 Template Matching
Initial user query
RF loop
Retrieved results
Labeling by the user
Training of the classifier
Final retrieved results
FIGURE 8.15 Block diagram illustrating the basic steps involved in a relevance feedback task.
Steps 2–3 are repeated untill the RF algorithm converges to a level that satisfies the user; that is, enough of truly relevant patterns have been retrieved by the search engine. Obviously, different techniques have been suggested and used for all three steps in the loop. For Step 1, one can initialize the system either randomly or, preferably, by retrieving an initial set of “similar” patterns based on a similarity metric, such as the lp metric. For Step 3, a popular choice is to employ a binary classifier that is trained to classify the marked patterns, in Step 2, either to the “relevant” or the “irrelevant” class, according to their label as judged by the user. Support vector machines have enjoyed a high popularity among the researchers, although other classifiers can also be used, see, [Druc 01, Cruc 04, Liu 07]. Interestingly enough, Step 2 is of critical importance. A selection strategy is first adopted concerning the type of the patterns that the system returns and on which the user must apply the query concept and label these patterns accordingly. Obviously, the patterns that the search engine returns in each round depend on the current knowledge of the “learner” in Step 3. In each iteration step, the system performs a ranking of the retrieved patterns, according to a confidence measure associated with the classifier’ s decisions. For example, in the case an SVM has been adopted, this confidence measure can be the distance of the feature vector,
“10-Ch08-SA272” 18/9/2008 page 511
8.5 Content-Based Information Retrieval: Relevance Feedback
representing the pattern, from the decision hyperplane in the RKHS feature space, which is proportional to the value of |g(x)| in Eq. (4.72) (Section 4.18). In [Druc 01], the remotest instance to the positive (relevant) side is ranked at the top of the list, and the remotest one to the negative side (irrelevant) is ranked at the bottom. The user selects a number of patterns (say, 10 to 20) among the top ranked in the list. That is, selection is done among patterns that have been classified (by the current classifier) as relevant and with high confidence. Obviously, if the system has not yet converged, some of the returned patterns will not satisfy the user and will be judged as being irrelevant. This strategy seems to be the most popular one. The strong point of such a selection procedure is that the user gets a few good relevant patterns quite early in the iteration process. On the other hand, such a philosophy turns out to lead to a relatively slow convergence of the method. Another point of view has been adopted in the strategy proposed in [Tong 01]. This is inspired by the notion of active learning used in pattern recognition (see, [Lewi 94, Scho 00]). Active learning is an approach that trains the classifier by using a subset of the data, that are considered the most informative ones. Hence, one can achieve better performance with less training data. The most informative data points are taken to be the most uncertain instances. Thus, in such a RF setting, the user is asked to label a number of pool patterns that lie closest to the classifier’ s boundary decision. In [Tong 01] a strong theoretical justification is also provided for such a scenario. In other words,such a selection criterion forces the system to elicit from the user a crucial part of information: what makes distinct the “relevant” from the “irrelevant.” This is because, once the user labels as relevant or irrelevant such “uncertain” patterns, a significant part of uncertainty is removed from the system. The advantage of this selection criterion is that it speeds up the convergence of the RF scheme. Figure 8.16 compares the convergence performance of an RF system for the two previous strategies and for two different users; a “lazy” user, who only marks up to two relevant and two irrelevant patterns (if a single relevant pattern is returned by the system, the user marks only one) and a “patient”user who marks all the relevant and all the irrelevant patterns, among the patterns that the engine returns at each iteration step. The horizontal axis corresponds to the number of iteration steps and the vertical one to the precision, denoted as Pr. As precision, we define the ratio of the relevant patterns to the total number of returned (at each iteration round) patterns. In all cases, the curves start from the same point. This is the precision value corresponding to the initialization step, where a simple similarity measure was used. Also, all curves tend to Pr ⫽ 1; that is, as the learning process advances, more and more of the returned patterns are judged by the user as being relevant. For both users, the curve corresponding to the active learning strategy tends to Pr ⫽ 1 faster. As it is natural, the system tends to Pr ⫽ 1 much faster for the cases of the patient user, since, at each iteration step, more patterns are available for training the classifier in Step 3. The curves in the figure have been obtained using the Wang image database [Wang 01] and an SVM classifier. More details about the features and
511
“10-Ch08-SA272” 18/9/2008 page 512
CHAPTER 8 Template Matching
1 active SVM-all
0.95 simple SVM-2pos-2neg
0.9 Precision
512
active SVM-2pos-2neg
0.85
0.8 simple SVM-all
0.75 0.7 0
1
2
3 4 5 6 7 Relevance Feedback Step
8
9
10
FIGURE 8.16 Dotted lines correspond to the active learning scenario. Learning curves for the “lazy”user, who, in each iteration round, marks up to two positive (relevant) and two negative (irrelevant) images, converge slower compared to those associated with the user who labels all the returned images as either positive or negative. For both users, active learning leads to faster convergence. We have used the terms active and simple to annotate the curves corresponding to the two strategies.
the parameters used are available in the book’s Web site, and the interested reader can also perform a set of related experiments. For the experienced researcher, it will not come as a surprise to say that a third route to the selection strategy has been proposed by combining the previous two “extremes.” In [Xu 03], a hybrid approach has been suggested consisting of the following steps: 1. An SVM is trained by the user’ s initial query. 2. The system returns a set of M patterns consisting of the K ⱕ M remotest ones, from the decision hyperplane in the RKHS and on the positive side, and the M–K closest ones to it. 3. The user labels the patterns as relevant or irrelevant. 4. The SVM classifier is retrained using all the patterns labeled,so far,by the user. The algorithm is either terminated, if the user is satisfied, or it is redirected to Step 2. In [Xu 03], it is claimed that such a scheme shares the advantages of both previous selection criteria. It speeds up convergence and at the same time presents to the user some truly relevant patterns early enough in the iteration process.
“10-Ch08-SA272” 18/9/2008 page 513
8.6 Problems
Although in this section we have focused on the use of a classifier in Step 3,other alternatives are also possible. For example, in the so-called query point method (QPM) the existence of an ideal query point in the feature space is assumed, which, if found, would provide the appropriate answer to the user’s query. Each feature in the feature vector is weighted and the task of the learner is to adjust the weights so that to move the point in the feature space appropriately. Learning can be based either on the positive examples (e.g., [Scla 97]) or on both positive and negative ones (e.g.,[Rui 98]). For a more detailed reference survey on the topic the interested reader may consult [Liu 07, Long 03, Cruc 04].
8.6 PROBLEMS 8.1 Find the Edit distance between the word “poem” and its misspelled version “poten.” Draw the optimal path. 8.2 Derive the slopes for the Sakoe–Chiba constraints and draw paths that achieve the maximum expansion/compression rates as well as paths corresponding to intermediate rates. 8.3 Develop a computer program for dynamic time warping, for a complete optimum path, and for the Itakura constraints. Verify the algorithm, using speech segments available from the book’s web site. 8.4 Let the seven Hu moments of the reference M ⫻ N image block be i , i ⫽ 1, 2, . . . , 7. Also, denote by i (m, n), i ⫽ 1, 2, . . . , 7, the respective moments resulting from the test subimage located at (m, n). Explain why 7
M(m, n) ⫽ 7
i⫽1 i i (m, n)
2 7 2 (m, n) i⫽1 i i⫽1 i
1/2
is a reasonable measure of similarity. 8.5 Show that the Mellin transform M(u, v) of a function f (x, y), defined as
M(u, v) ⫽
f (x, y)x ⫺ju⫺1 y⫺jv⫺1 dx dy
is scale invariant. 8.6 For the motion compensation step in a video coding application, the image frame of size I ⫻ J is divided into subblocks of size M ⫻ N . For each subblock the search for its corresponding one, in the current frame, is restricted in an area of size [⫺p, p] ⫻ [⫺p, p]. Find the required number of operations per second for the computation of the maximum cross-correlation for a full search, a two-dimensional and a one-dimensional (independent i, j) logarithmic search, and a hierarchical search of three levels. Typical values for broadcast TV are M ⫽ N ⫽ 16, I ⫽ 720, J ⫽ 480, and p ⫽ 15. The number of frames transmitted per second is f ⫽ 30.
513
“10-Ch08-SA272” 18/9/2008 page 514
514
CHAPTER 8 Template Matching
MATLAB PROGRAMS AND EXERCISES Computer Programs 8.1 Edit distance. Write a MATLAB function named edit_distance that computes the Edit distance between two strings. This function takes as inputs (a) the reference string ref _str and (b) the test string test_str. It returns (a) the matching cost edit_cost and (b) the node predecessors matrix pred. Each element of the last matrix stores the coordinates of the predecessor of a node as a complex number, the real part of which stands for the row index and the imaginary part for the column index.
Solution In the following implementation it is assumed that the reference pattern is associated with the horizontal axis. function [edit_cost,pred]=edit_distance(ref_str,test_str) I = length(ref_str); J = length(test_str); D = zeros(J,I); %Initialization D(1,1) = ~(ref_str(1) == test_str(1)); pred(1,1) = 0; for j = 2:J D(j,1) = D(j-1,1)+1; pred(j,1) = (j-1) + sqrt(-1)*1; end for i = 2:I D(1,i) = D(1,i-1)+1; pred(1,i) = 1+ sqrt(-1)*(i-1); end %Main Loop for i = 2:I for j = 2:J if(ref_str(i) == test_str(j)) d(j,i) = 0; else d(j,i) = 1; end c1 = D(j-1,i-1)+d(j,i); c2 = D(j,i-1)+1; c3 = D(j-1,i)+1; [D(j,i),ind] = min([c1 c2 c3]); if(ind == 1) pred(j,i) = (j-1)+sqrt(-1)*(i-1); elseif(ind == 2)
“10-Ch08-SA272” 18/9/2008 page 515
MATLAB Programs and Exercises
pred(j,i) = j+sqrt(-1)*(i-1); else pred(j,i) = (j-1)+sqrt(-1)*i; end end end edit_cost = D(J,I);
8.2 Backtracking. Write a MATLAB function named back_tracking, which takes as inputs (a) a node predecessors matrix pred, (b) the coordinates k and l of the node from which the backtracking will start. It returns the best path on the cost grid, best_path, and also plots the best path. It is assumed that backtracking always terminates when a node whose predecessor is (0, 0) is reached.
Solution function best_path=back_tracking(pred,k,l) Node = k+sqrt(-1)*l; best_path = [Node]; while (pred(real(Node),imag(Node)) = 0) Node = pred(real(Node),imag(Node)); best_path = [Node;best_path]; end %Plot the best path [I,J] = size(pred); clf; hold for j = 1:J for i = 1:I plot(j,i,'r.') end end plot(imag(best_path),real(best_path),'g') axis off
8.3 Dynamic time warping with Sakoe–Chiba local path constraints. Write a MATLAB function named Dtw_Sakoe that implements a dynamic timewarping scheme where the Sakoe–Chiba local path constraints are adopted. More specifically, the function takes as input (a) a row vector ref that corresponds to the reference sequence and (b) a row vector test that corresponds to the test sequence. It returns (a) the time-warping matching cost, matching_cost and (b) the best path best_ path. It is assumed that (i) the cost assigned to each node in the grid upon initialization is equal to the Euclidean distance of the respective pattern elements and (ii) the cost of a transition
515
“10-Ch08-SA272” 18/9/2008 page 516
516
CHAPTER 8 Template Matching
depends only on the cost that has been assigned to the node at the end of the transition.
Solution In the following implementation, function back_tracking introduced before is utilized to determine the best path. function [matching_cost,best_path]=Dtw_Sakoe(ref,test) I = length(ref); J = length(test); for i = 1:I for j = 1:J %Euclidean distance node_cost(i,j) = sqrt(sum((ref(:,i)-test(:,j)).^2)); end end %Initialization D(1,1) = node_cost(1,1); pred(1,1) = 0; for i = 2:I D(i,1) = D(i-1,1)+node_cost(i,1); pred(i,1) = i-1 + sqrt(-1)*1; end for j = 2:J D(1,j) = D(1,j-1)+node_cost(1,j); pred(1,j) = 1 + sqrt(-1)*(j-1); end %Main Loop for i = 2:I for j = 2:J [D(i,j),ind] = min([D(i-1,j-1) D(i-1,j) ... D(i,j-1)]+node_cost(i,j)); if (ind == 1) pred(i,j) = (i-1)+sqrt(-1)*(j-1); elseif (ind == 2) pred(i,j) = (i-1)+sqrt(-1)*(j); else pred(i,j) = (i)+sqrt(-1)*(j-1); end end %for j end %for i %End of Main Loop matching_cost = D(I,J); best_path = back_tracking(pred,I,J);
“10-Ch08-SA272” 18/9/2008 page 517
References
Computer Experiments 8.1 a. Compute the Edit distance between the following pairs of strings: (i) (beauty, beaty), (ii) (beauty, biauty), (iii) (beauty, betty), using the first element of each pair as the reference string. b. Plot the respective matching paths using the function back_tracking. 8.2 Use the Dtw_Sakoe function to compute the time-warping cost between the sequences {1, 2, 3} and {1, 1, 2, 2, 2, 3, 3, 3}, using the former as reference sequence. Comment on the results. 8.3 Let r1 ⫽ [1, 0]T , r2 ⫽ [0, 1]T and ref ⫽ [r1, r2]. Generate a sequence of 10 two-dimensional vectors with the following MATLAB command test ⫽ [1 ⫹ rand(1, 4)/2 rand(1, 6)/3; rand(1, 4)/2 1 ⫹ rand(1, 6)/3]. Use the Dtw_Sakoe function to compute the time-warping cost and the respective best path between ref and test, taking the former as the reference sequence. Comment on the results.
REFERENCES [Alkh 01] Ghavari-Alkhansavi M. “A fast globally optimal algorithm for template matching using low resolution pruning,” IEEE Transactions on Image Processing, Vol. 10(4), pp. 526–533, 2001. [Amit 91] Amit Y., Grenander U., Piccioni M. “Structural image restoration through deformable template,” J.Amer. Statist.Association,Vol. 86(414), pp. 376–387, 1991. [Bell 57] Bellman R.E. Dynamic Programming, Princeton University Press, 1957. [Bhas 95] Bhaskaran V., Konstantinides K. Image and Video Compression Standards, Kluwer Academic Publishers, 1995. [Cheu 02] Cheung K.-W.,Yeung D.-Y., Chin R.T. “On deformable models for visual pattern recognition,” Pattern Recognition,Vol. 35, pp. 1507–1526, 2002. [Cruc 04] Crucianu M., Ferecatu M., Boujemaa N. “Relevance feedback for image retrieval: a short survey,” in Audiovisual Content-based Retrieval, Information Universal Access and Interaction, Including Datamodels and Languages, Report of the DELOS2 European Network of Excellence, FP6, 2004. [Dame 64] Damerau F.J. “A technique for computer detection and correction of spelling errors,” Commun.ACM,Vol. 7(3), pp. 171–176, 1964. [Davi 80] Davis S.B., Mermelstein P. “Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 28(4), pp. 357–366, 1980. [Dell 93] Deller J.,Proakis J.,Hansen J.H.L. Discrete-Time Processing of Speech Signals,Macmillan, 1993. [Desh 99] Deshmukh N., Ganapathirajn A., Picone J. “Hierarchical search for large vocabulary conversational speech recognition,”IEEE Signal Processing Magazine,Vol. 16(5), pp. 84–107, 1999.
517
“10-Ch08-SA272” 18/9/2008 page 518
518
CHAPTER 8 Template Matching
[Druc 01] Drucker H., Shahraray B., Gibbon D. “Relevance feedback using support vector machines,” Proceedings of th 18th International Conference on Machine Learning, pp. 122–129, 2001. [Durb 97] Durbin K., Eddy S., Krogh A., Mitchison G. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids, Cambridge University Press, Cambridge, MA, 1997. [Fisc 73] Fischler M., Elschlager R. “The representation and matching of pictorial structures,”IEEE Transations on Computers,Vol. 22(1), pp. 67–92, 1973. [Gray 76] Gray A.H., Markel J.D.“Distance measures for speech processing,”IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 24(5), pp. 380–391, 1976. [Gray 80] Gray R.M.,Buzo A.,Gray A.H.,MatsuyamaY.“Distortion measures for speech processing,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 28(4), pp. 367–376, 1980. [Hall 79] Hall E. Computer Image Processing and Recognition,Academic Press, 1979. [Itak 75] Itakura F. “Minimum prediction residual principle applied to speech recognition,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 23(2), pp. 67–72, 1975. [Jain 98] Jain A.K., Zhong Y., Dubuisson-Jolly M.P. “Deformable template models: A review,” Signal Processing,Vol. 71, pp. 109–129, 1998. [Jain 96] Jain A.K., Zhong Y., Lakshmanan S. “Object matching using deformable templates,” IEEE Transactions on Pattern Analysis and Machine Itelligence,Vol. 18(3), pp. 267–277, 1996. [Koch 89] Koch M.W., Kashyap R.L. “Matching polygon fragments,” Pattern Recognition Letters, Vol. 10, pp. 297–308, 1989. [Leven 66] LevenshteinV.I.“Binary codes capable of correcting deletions,insertions and reversals,” Soviet phys. Dokl.,Vol. 10(8), pp. 707–710, 1966. [Lewi 94] Lewis D., Gale W. “A sequential algorithm for training text classifiers,” Proceedings of the 11th International Conference on Machine Learning, pp. 148–156, Morgan Kaufmann, 1994. [Liu 07] Liu Y., Zhang D., Lu G., Ma W.-Y. “A survey of content-based image retrieval with high-level semantics,” Pattern Recognition,Vol. 40, pp. 262–282, 2007. [Long 03] Long F., Zang H.J., Feng D.D. “Fundamentals of content-based image retrieval,” in Multimedia Information Retrieval and Management (Feng D. ed.), Springer, Berlin, 2003. [Marz 93] Marzal A., Vidal E. “Computation of normalized edit distance and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 15(9), 1993. [McIn 96] McInerney T.,Terzopoulos D. “Deformable models in medical image analysis: A survey,” Med. Image Anal., Vol. 1(2), pp. 91–108, 1996. [Mei 04] Mei J. “Markov edit distance,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 6(3), pp. 311–320, 2004. [Myer 80] Myers C.S., Rabiner L.R., Rosenberg A.E. “Performance tradeoffs in dynamic time warping algorithms for isolated word recognition,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 28(6), pp. 622–635, 1980. [Neg 99] Neg H., Ortmanns S.“Dynamic programming search for continuous speech recognition,” IEEE Signal Processing Magazine,Vol. 16(5), pp. 64–83, 1999. [Ocud 76] Ocuda T.,Tanaka E., Kasai T. “A method for correction of garbled words based on the Levenstein metric,” IEEE Transactions on Computers, pp. 172–177, 1976.
“10-Ch08-SA272” 18/9/2008 page 519
References
[Pikr 03] Pikrakis A., Theodoridis S., Kamarotos D. “Recognition of isolated musical patterns using context dependent dynamic time warping,” IEEE Transactions on Speech and Audio Processing,Vol 11(3), pp. 175–183, 2003. [Rabi 93] Rabiner L., Juang B.H. Fundamentals of Speech Recognition, Prentice Hall, 1993. [Ravi 95] Ravichandran G.,Trivedi M.M. “Circular-Mellin features for texture segmentation,” IEEE Transactions on Image Processing,Vol. 2(12), pp. 1629–1641, 1995. [Rui 98] Rui Y., Huang T.S., Ortega M., Mehrotra S. “Relevance feedback: power tool in interactive content-based image retrieval,” IEEE Transactions on Circuits and Systems for Video Technology,Vol. 8(5), pp. 644–655, 1998. [Sako 78] Sakoe H., Chiba S. “Dynamic programming algorithm optimization for spoken word recognition,”IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 26(2),pp. 43– 49, 1978. [Scha 89] Schalkoff R. Digital Image Processing and Computer Vision, John Wiley & Sons, 1989. [Scho 00] Schohn G., Cohn D. “Less is more: Active learning with support vector machines,” Proceedings of the 17th International Conference on Machine Learning, pp. 839–846, Morgan Kaufmann, 2000. [Scla 97] Sclaroff S.,Taycher L. Cascia M“Imagerover: A content-based image browser for the world wide web,” Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL’97), pp. 2–9, IEEE Computer Society, 1997. [Seni 96] Seni G., Kripasundar V., Srihari R. “Generalizing Edit distance to incorporate domain information: Handwritten text recognition as a case study,” Pattern Recognition, Vol. 29(3), pp. 405–414, 1996. [Silv 90] Silverman H., Morgan D.P. “The application of the dynamic programming to connected speech recognition,” IEEE Signal Processing Magazine,Vol. 7(3), pp. 7–25, 1990. [Tong 01] Tong S., Chang E. “Support vector machine active learning for image retrieval,” Proceedings of the 9th ACM International Conference on Multimedia, pp. 107–118,ACM Press, 2001. [ Tsay 93] Tsay Y.T.,Tsai W.H. “Attributed string matching split and merge for on-line Chinese character recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15(2), pp. 180–185, 1993. [Ueno 97] Uenohara M., Kanade T. “Use of the Fourier and Karhunen–Loève decomposition for fast pattern matching with a large set of templates,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 19(8), pp. 891–899, 1997. [Wang 01] Wang J.Z., Li J.,Wiederhold G. “SIMPLIcity: Semantics-sensitive Integrated Matching for Picture LIbraries,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 23, no.9, pp. 947–963, 2001. [Wang 90] Wang Y.P., Pavlidis T. “Optimal correspondence of string subsequences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12(11), pp. 1080–1086, 1990. [Widr 73] Widrow B. “The rubber mask technique, Parts I and II,” Pattern Recognition, Vol. 5, pp. 175–211, 1973. [Xu 03] Xu Z., Xu X., Yu K., Tresp V. “A hybrid relevance-feedback approach to text retrieval,” Proceedings of the 25th European Conference on Information Retrieval Research, Lecture Notes in Computer Science,Vol. 2633, Springer Verlag, 2003.
519
“11-Ch09-SA272” 18/9/2008 page 521
CHAPTER
9
Context-Dependent Classification
9.1 INTRODUCTION The classification tasks considered so far have assumed that no relation exists among the various classes. In other words, having obtained a feature vector x from a class i , the next feature vector could belong to any other class. In this chapter we will remove this assumption, and we will assume that the various classes are closely related. That is, successive feature vectors are not independent. Under such an assumption, classifying each feature vector separately from the others obviously has no meaning. The class to which a feature vector is assigned depends (a) on its own value, (b) on the values of the other feature vectors, and (c) on the existing relation among the various classes. Such problems appear in various applications such as communications, speech recognition, and image processing. In the context-free classification, our starting point was the Bayesian classifier. In other words, a feature vector was classified to a class i if P(i |x) P(j |x),
j i
The Bayesian point of view will also be adopted here. However, the dependence among the various classes sets demands for a more general formulation of the problem. The mutual information that resides within the feature vectors requires the classification to be performed using all feature vectors simultaneously and also to be arranged in the same sequence in which they occurred from the experiments. For this reason, in this chapter we will refer to the feature vectors as observations occurring in sequence, one after the other, with x 1 being the first and x N the last from a set of N observations.
9.2 THE BAYES CLASSIFIER Let X : x 1 , x 2 , . . . , x N be a sequence of N (feature vectors) observations and i , i 1, 2, . . . , M, the classes in which these vectors must be classified. Let
521
“11-Ch09-SA272” 18/9/2008 page 522
522
CHAPTER 9 Context-Dependent Classification
i : i1 , i2 , . . . , iN be one of the possible sequences of these classes corresponding to the observation sequence, with ik ∈ {1, 2, . . . , M} for k 1, 2, . . . , N . The total number of these class sequences i is M N , that is, the number of possible ordered combinations of M distinct objects taken in groups of N . Our classification task is to decide to which class sequence i a sequence of observations X corresponds. This is equivalent to appending x 1 to class i1 , x 2 to i2 , and so on. A way to approach the problem is to view each specific sequence X as an (extended) feature vector and i , i 1, 2, . . . , M N , as the available classes. Having observed a specific X, the Bayes rule assigns it to i for which P(i |X) P(j |X),
i j
(9.1)
and following our already familiar arguments, this is equivalent to P(i )p(X|i ) P(j )p(X|j ),
i j
(9.2)
In the following we will investigate the specific form that Eq. (9.2) takes for some typical class dependence models.
9.3 MARKOV CHAIN MODELS One of the most widely used models describing the underlying class dependence is the Markov chain rule. If i1 , i2 , . . . is a sequence of classes, then the Markov model assumes that P(ik |ik1 , ik2 , . . . , i1 ) P(ik |ik1 )
(9.3)
The meaning of this is that the class dependence is limited only within two successive classes. This type of model is also called a first-order Markov model, to distinguish it from obvious generalizations (second,third,etc.). In other words,given that the observations x k1 , x k2 , . . . , x 1 belong to classes ik1 , ik2 , . . . , i1 , respectively,the probability of the observation x k ,at stage k,belonging to class ik depends only on the class from which observation x k1 , at stage k1, has occurred. Now combining (9.3) with the probability chain rule [Papo 91, p. 192], P(i ) ≡ P(i1 , i2 , . . . , iN ) P(iN |iN 1 , . . . , i1 )P(iN 1 |iN 2 , . . . , i1 ) . . . P(i1 )
we obtain P(i ) P(i1 )
N
P(ik |ik1 )
(9.4)
k2
where P(i1 ) is the prior probability for class i1 , i1 ∈ {1, 2, . . . , M}, to occur. Furthermore, two commonly adopted assumptions are that (a) given the sequence of classes, the observations are statistically independent, and (b) the probability density function in one class does not depend on the other classes. That is,
“11-Ch09-SA272” 18/9/2008 page 523
9.4 The Viterbi Algorithm
dependence exists only on the sequence in which classes occur, but within a class observations “obey” the class’ own rules. This assumption implies that p(X|i )
N
p(x k |ik )
(9.5)
k1
Combining Eqs. (9.4) and (9.5), the Bayes rule for Markovian models becomes equivalent to the statement:
Statement Having observed the sequence of feature vectors X : x 1 , . . . , x N , classify them in the respective sequence of classes i : i1 , i2 , . . . , iN , so that the quantity p(X|i )P(i ) P(i1 )p(x 1 |i1 )
N
P(ik |ik1 )p(x k |ik )
(9.6)
k2
becomes maximum. As we have already stated, searching for this maximum requires the computation of Eq. (9.6) for each of the i , i 1, 2, . . . , M N . This amounts to a total of O(NM N ) multiplications, which is usually a large number indeed. However, this direct computation is a brute-force task. Let us take for example two sequences i and j , which we assume differ only in the last class, that is, ik jk , k 1, 2, . . . , N 1 and iN jN . It does not take much “scientific thought” to realize that the computation of (9.6) for these two sequences shares all multiplications (which need not be repeated) but one. Furthermore, closer observation of (9.6) reveals that it has a rich computational structure, which can be exploited in order to maximize it in a much more efficient way. This now becomes our next concern.
9.4 THE VITERBI ALGORITHM Figure 9.1 shows a diagram with N dot columns, where each dot in a column represents one of the M possible classes,1 , 2 , . . . , M . Successive columns correspond to successive observations x k , k 1, 2, . . . , N . The arrows determine transitions from one class to another, as observation vectors are obtained in sequence. Thus, each of the class sequences i corresponds to a specific path of successive transitions. Each transition from one class i to another j is characterized by a fixed probability P(j |i ), which is assumed to be known for the adopted model of class dependence. Furthermore, we assume that these probabilities are the same for all successive stages k. That is, the probabilities depend only on the respective class transitions and not on the stage at which they occur. We will further assume that the conditional probability densities p(x|i ), i 1, 2, . . . , M, are also known to the model. The task of maximizing (9.6) can now be stated as follows. Given a sequence of observations x 1 , x 2 , . . . , x N , find the path of successive (class) transitions that
523
“11-Ch09-SA272” 18/9/2008 page 524
524
CHAPTER 9 Context-Dependent Classification
M
M1
2 |
)
1
2 P( 1 k1 x1
2 x2
N1 xN1
3 x3
N xN
FIGURE 9.1 Trellis diagram for the Viterbi algorithm. The red line denotes the optimal path. The classes along this optimal path will be the ones in which the respective observations are classified.
maximizes (9.6) (e.g., red line in the figure). The classes along this optimal path will be the ones in which the respective observations are classified. To search for the optimal path we have to associate a cost with each of the transitions, in agreement with the cost function given in (9.6). A careful look at (9.6) suggests adopting ˆ ik , ik1 ) P(ik |ik1 )p(x k |ik ) d(
(9.7)
as the cost associated with a transition of a path i from node (class) ik1 , at stage k 1, to node ik , at stage k and at the same time observation x k occurring. The initial condition for k 1 is given by ˆ i1 , i0 ) P(i1 )p(x 1 |i1 ) d(
Using this notation, the overall cost to be optimized becomes ˆ ≡ D
N
ˆ ik , ik1 ) d(
(9.8)
k1
It will not come as a surprise to us if instead of (9.8) one chooses to maximize ˆ ln( D)
N
ˆ ik , ik1 ) ln d(
k1
≡
N k1
d(ik , ik1 ) ≡ D
(9.9)
“11-Ch09-SA272” 18/9/2008 page 525
9.4 The Viterbi Algorithm
ˆ ·). Looking carefully at (9.9) or (9.8), it will not take us where d(·, ·) ≡ ln d(·, long to realize that Bellman’s principle can again offer us the means for efficient optimization. Indeed, let us define, in accordance with D, the cost for reaching class ik at stage k via a path i as D(ik )
k
d(ir , ir1 )
(9.10)
r1
Then, Bellman’s principle states that Dmax (ik ) max[Dmax (ik1 ) d(ik , ik1 )], ik1
ik , ik1 1, 2, . . . , M
(9.11)
with Dmax (i0 ) 0
(9.12)
It is now straightforward to see that the optimal path, which leads to the maximum D in (9.9), is the one that ends at the final stage N in the class ∗iN for which ∗iN arg max Dmax (iN ) iN
(9.13)
Going back to Figure 9.1, we see that at each stage k, k 1, 2, . . . , N , there are M possible transitions to each of the nodes ik . The recursive relation in (9.11) suggests that in searching for the maximum we need only keep one of these transitions for every node, the one that leads to the maximum respective cost Dmax (ik ) (red lines in the figure). Hence, at each stage there are only M surviving paths. Therefore, the number of operations is M for each node, thus M 2 for all the nodes at each stage, and NM 2 in total. The latter number compares very favorably with the NM N of the “brute-force task.” The resulting algorithm is known as the Viterbi algorithm. This dynamic programming algorithm was first suggested for optimal decoding in convolutional codes in communications [Vite 67]. In previous chapters, we stated that for a number of reasons an alternative to the optimal Bayes classifier can be used. No doubt context-dependent classification can also be emancipated from its Bayesian root. This can easily be achieved by adopting different transition costs d(ik , ik1 ), which are not necessarily related to probability densities. In the following sections we will present two typical application areas of the Viterbi algorithm. Example 9.1 In this example we apply the Viterbi algorithm to compute the optimal paths up to stage k 4, once observation x4 has been received. Assume that x4 1.2 and that the observations reside in the one-dimensional space. Let the task involve three classes, namely, 1 , 2 , 3 . We will further assume that the optimal paths up to stage k 3 have been computed and
525
“11-Ch09-SA272” 18/9/2008 page 526
526
CHAPTER 9 Context-Dependent Classification
3 2 1
0
1
2
3
4
k
FIGURE 9.2 Optimal paths for the grid corresponding to Example 9.1. The red lines correspond to the extensions of the optimal paths from stage k 3 to stage k 4.
Table 9.1 Transition Costs Between Nodes for Example 9.1 Classes
i k 1
i k 2
i k 3
ik1 1
0.1
0.7
0.2
ik1 2
0.4
0.3
0.3
ik1 3
0.3
0.1
0.6
are shown in black lines in Figure 9.2. Let the optimal costs associated with each class at stage k 3 be equal to D(1 ) 0.5,
D(2 ) 0.6,
D(3 ) 0.2
(9.14)
All the values are negative, since, as Eqs. (9.9) and (9.10) suggest, costs result by summing up logarithms of probability products. The previous costs are assumed known and they have been computed based on the initial class probability values and (a) the transition probability costs among the three classes, which are given in Table 9.1 and (b) the values of the received observations, x0 , x1 , x2 , x3 . We have further assumed that the probability density function describing the emission of an observation from each one of the classes follows a Gaussian distribution 1 (x i )2 p(x|i ) √ exp 2i2 2i where 1 1.0 and 12 0.03, 2 1.5 and 22 0.02, 3 0.5 and 32 0.01. We will first compute the optimal path reaching class 1 at stage k 4. According to Eqs. (9.11), (9.10), and (9.7), the following calculations are in order: ln p(x4 1.2|i4 1 ) 0.1578 Total cost for the transition from i3 1 to i4 1 is equal to 0.5 ln(0.1) 0.1578 2.9604.
“11-Ch09-SA272” 18/9/2008 page 527
9.5 Channel Equalization
Total cost for the transition from i3 2 to i4 1 is equal to 0.6 ln(0.4) 0.1578 1.6741. Total cost for the transition from i3 3 to i4 1 is equal to 0.2 ln(0.3) 0.1578 1.5617. Hence, the optimal path reaching class 1 at stage k 4 is through 3 at stage k 3. For the transitions to 2 at k 4, we have ln p(x4 1.2|i4 2 ) 0.2591 and the respective values for the paths reaching class 2 from classes 1 , 2 and 3 at k 3 are 1.1158, 2.0631, 2.7617. Thus the optimal path reaching 2 at k 4 is through 1 at k 3. Finally, the respective values for the paths reaching 3 at k 4 are ln p(x4 1.2|i4 3 ) 2.2176 and 4.3271, 4.0216, 2.9285. As a result, the best path reaching node 3 at stage k 4 goes through class 3 at stage k 3 (self-transition). If k 4 is the final stage, that is, only four observations are available, then the optimal path, denoted by a bold line in Figure 9.2, is the one ending at stage 2 with an overall cost equal to 1.1158. Going backwards along the optimal path (backtracking), we assign: x4 to 2 , x3 to 1 , x2 to 2 , x1 to 1 and x0 to 2 .
9.5 CHANNEL EQUALIZATION Channel equalization is the task of recovering a transmitted sequence of information bits Ik (e.g., 1 or 0) after they have been corrupted by the transmission channel and noise sources. The samples received at the receiver end are, thus, given by xk f (Ik , Ik1 , . . . , Ikn1 ) k
(9.15)
where the function f (·) represents the action of the channel and k the noise sequence. The channel contribution to the overall corruption is called the intersymbol interference ( ISI ) and it spans n successive information bits. The equalizer is the inverse system whose task is to provide decisions Iˆk about the transmitted information bits Ik , based on l successively received samples [xk , xk1 , . . . , xkl1 ] ≡ x Tk . Usually, a delay r must be used in order to accommodate the (possible) noncausal nature of the inverse system. In such cases the decisions made at time k correspond to the Ikr transmitted information bit ( Figure 9.3). A simple example will reveal to us how the equalization problem comes under the umbrella of a Markovian context-dependent classification task. Assume a simple linear channel xk 0.5Ik Ik1 k
(9.16)
527
“11-Ch09-SA272” 18/9/2008 page 528
528
CHAPTER 9 Context-Dependent Classification
xk
Equalizer Ik⫺r
FIGURE 9.3 Block diagram of an equalizer.
For l 2, successively received samples are combined in vectors in the twodimensional space, that is, x Tk [xk , xk1 ]
Let us further assume that there are N such observation vectors available. From (9.16) it is readily seen that each x k , k 1, 2, . . . , N , depends on the values of three successive information bits, namely Ik , Ik1 , Ik2 . Neglecting the effects of noise, the possible values of the received samples xk are given in Table 9.1, together with the respective information bits. Figure 9.4a shows the geometry in the two-dimensional space with (xk , xk1 ) on its axis. When the effect of noise is taken into account, the received vectors are clustered around these points (for example, shaded area). For the specific channel of (9.16) there are eight possible clusters, i , i 1, 2, . . . , 8 (Table 9.1). Clusters (red) around “” correspond to Ik 1 and those (gray) around “o” to Ik 0. In general, the total number of clusters for a binary information sequence is m 2nl1 . On the reception of each x k [xk , xk1 ]T ,the equalizer has to decide whether the corresponding transmitted information bit Ik was either a “1” (i.e., class “A”) or a “0” (class “B”). In other words, this is nothing other than a two-class classification problem (M class for the M-ary case), and each class consists of a union of clusters. Thus, various techniques, from those we have already studied in previous chapters, can be used. A simple way, which was followed in [Theo 95], consists of two steps. During the training period a sequence of known information bits is transmitted, and the representative center i for each of the clusters is computed by a simple averaging of all the vectors x k belonging to the respective cluster. This is possible during the training period, since the transmitted information bits are known, and thus we know to which of the clusters each received x k belongs. For example, in the case of Table 9.1 if the sequence of transmitted bits is (Ik 1, Ik1 0, Ik2 1), then the received x k belongs to 6 . At the same time, the clusters are labeled as“1”or“0” depending on the value of the Ik bit. At the so-called decision directed mode the transmitted information bits are unknown, and the decision about the transmitted Ik is based on which cluster (“1”or“0”label) the received vector x k is closest to. For this purpose a metric is adopted to define distance. The Euclidean distance of the received vector x k from the representatives i of the clusters is an obvious candidate. Although such an equalizer can result in reasonable performance, measured in bit error rate (BER) (the percentage of information bits wrongly identified), there is still a great deal of information that has not been exploited. Let us search for it!
“11-Ch09-SA272” 18/9/2008 page 529
9.5 Channel Equalization
Table 9.2 Received Samples and Respective Information Bits for the Channel of Eq. (9.16) Ik
Ik1
Ik2
xk
xk1
0
0
0
0
0
1
0
0
1
0
1
2
0
1
0
1
0.5
3
0
1
1
1
1.5
4
1
0
0
0.5
0
5
1
0
1
0.5
1
6
1
1
0
1.5
0.5
7
1
1
1
1.5
1.5
8
xk
1.5
1 0.5
0 0.5 0.5
0
Cluster
0.5
1
(a)
1.5
xk 1
8
8
7
7
6
6
5
5
4
4
3
3
2
2
1
1 (b)
FIGURE 9.4 Plot (a) of the eight possible clusters associated with the channel of Eq. (9.16) and (b) the allowable transitions among them.
From the definition of ISI in (9.15) we know that the channel spans a number of successive information bits. Thus, only certain transitions among clusters are possible. Indeed, let us assume, for example, that at time k the transmitted information bits (Ik , Ik1 , Ik2 ) were (0, 0, 1); hence, the corresponding observation vector x k belongs to cluster 2 . The next received observation vector x k1 will depend on the triple (Ik1 , Ik , Ik1 ) bits,which can be either (1, 0, 0) or (0, 0, 0). Thus,x k1
529
“11-Ch09-SA272” 18/9/2008 page 530
530
CHAPTER 9 Context-Dependent Classification
will belong either to 5 or to 1 . That is,there are only two possible transitions from 2 . Figure 9.4b shows the possible transitions among the various clusters. Assuming equiprobable information bits, then from Figure 9.4b we can easily conclude that all allowable transitions have probability 0.5, that is, P(1 |1 ) 0.5 P(5 |1 )
and the rest are zero (not allowable). We now have all the ingredients to define the equalization problem as a context-dependent classification task. Given N observation vectors x k , k 1, 2, . . . , N , classify them in a sequence of clusters i1 , i2 , . . . , iN . This automatically classifies each x k in one of the two classes “A” and “B,” which is equivalent to deciding that Ik is 1 or 0. For this goal a cost function has to be adopted. In [Theo 95, Geor 97] the cost d(ik , ik1 ) in (9.9) used in the Viterbi algorithm, for the allowable transitions, was taken to be d(ik , ik1 ) dik (x k )
(9.17)
where dik (x k ) is the distance of x k from the representative of the ik cluster. This can be either the Euclidean dik (x k ) x k ik
(9.18)
1/2 dik (x k ) (x k ik )T 1 (x ) k ik ik
(9.19)
or the Mahalanobis distance
The covariance matrices ik , describing the distribution of the observation vectors around the respective cluster representatives, are learned during the training period together with the cluster representatives i . The Viterbi algorithm is then used to obtain the optimal overall minimum distance sequence,and recursion (9.11) is modified for the search of the minimum. An alternative way to define the cost function is to consider observations as scalars xk , that is, the received samples (l 1), and make the following assumptions: ■
Successive noise samples are statistically independent and Gaussian.
■
The channel impulse response is known or can be estimated. In practice, a specific channel model is assumed,that is, fˆ (·),and its parameters are estimated via an optimization method, for example, least squares [Proa 89].
Under the preceding assumptions, the cost for the allowable state transitions in (9.9) becomes d(ik , ik1 ) ln p(xk |ik ) ≡ ln( p(k )) (xk fˆ (Ik , . . . , Ikn1 ))2
(9.20)
where k is the respective Gaussian distributed noise sample. Obviously, in (9.20) the constants in the Gaussian density function have been omitted. If the Gaussian and independence assumptions are valid, this is obviously a Bayesian optimal
“11-Ch09-SA272” 18/9/2008 page 531
9.5 Channel Equalization
classification to the clusters (from which the “0” and “1” classes result). However, if this is not true, the cost in (9.20) is no longer the optimal choice. This is, for example, the case when the so-called cochannel (nonwhite) interference is present. In such cases, the cluster-based approach is justifiable, and indeed it leads to equalizers with more robust performance [Geor 97]. Furthermore, the fact that in the clustering approach no channel estimation is required can be very attractive in a number of cases, where nonlinear channels are involved and their estimation is not a straightforward task [Theo 95]. In [Kops 03] equalization is performed in the onedimensional space, that is, l 1. Although this increases the probability of having clusters with different labels to overlap, this is not crucial to the performance, since the Viterbi algorithm has the power to detect the correct label, by exploiting the history in the path. Furthermore, it is pointed out that one needs not determine directly all the 2n cluster representatives; it suffices to learn, during the training phase, only n of the clusters, and the rest can be obtained by simple arithmetic operations. This is achieved by exploiting the mechanism underlying the cluster formation and the associated symmetries. Both of these observations have a substantial impact on reducing the computational complexity as well as the required length of the training sequence. The discussion so far has been based on the development of a trellis diagram associated with the transitions among clusters, and the goal has been to unravel the optimal path, using the Viterbi algorithm. However, although this came as a natural consequence of the context-dependent Bayesian classifier, it turns out this is not the most efficient way from a computational point of view. From Figure 9.4b one can easily observe that pairs of clusters jump to the same clusters after transition. For example, the allowable transitions from 1 and 2 are the same and lead to either 1 or 5 . The same is true for 3 and 4 , and so on. This is because the allowable transitions are determined by the n l 2 most recent bits. For the example of Figure 9.4, transitions are determined by the pair (Ik , Ik1 ), which, however, is shared by two clusters, that is, (Ik , Ik1 , Ik2 ), depending on the value of Ik2 if it is 0 or 1. The pair (Ik , Ik1 ) is known as the state at time k. This is because, knowing the state at time k and the transmitted bit Ik1 at time k 1, we can determine the next state (Ik1 , Ik ) at time k 1, as is the case in the finite state machines. Since transitions are determined by the states, one can construct a trellis diagram based on the states instead of on clusters. For the example of Figure 9.5, where eight clusters are present, there is a total of four states, that is, s1 : (0, 0), s2 : (0, 1), s3 : (1, 0), s4 : (1, 1). Figure 9.5 shows these states and the allowable transitions among them. Each transition is associated with one bit, which is the current transmitted bit. Obviously, there is a close relationship between states and clusters. If we know the state transition, that is, (Ik , Ik1 ) → (Ik1 , Ik ), then the current cluster at time k 1 will be determined by the corresponding values of (Ik1 , Ik , Ik1 ), and this automatically determines the cost of the respective transition, for example, (9.18) or (9.19). Hence the estimates of the transmitted bits are obtained from the sequence of bits along the optimal path, that is, the one with the minimum total cost, in the state trellis diagram instead of the larger cluster trellis diagram.
531
“11-Ch09-SA272” 18/9/2008 page 532
532
CHAPTER 9 Context-Dependent Classification
s4
1
1 s3
s4
0
s3
0 1 s2
s2 1 s1
0 0
s1
FIGURE 9.5 Plot of the four states associated with the channel of Eq. (9.16) and a two-dimensional equalizer showing the allowable transitions among them.
9.6 HIDDEN MARKOV MODELS In the channel equalization application of the previous section, the states of the Markov chain were observable. That is, given the l n 1 most recent information bits (i.e., Ik , Ik1 , Ik2 , in the given example), the state to which the corresponding observation vector x k belongs is readily known. Thus, during the training period these states can be “labeled,” and we can estimate their associated parameters (i.e., the related clusters). In this section, we will be concerned with systems where the states cannot be directly observed. The observations will be considered as the result of an action associated with each state and which is described by a set of probabilistic functions. Moreover, the sequence in which the different states are visited by successive observations is itself the result of another stochastic process, which is hidden to us, and the associated parameters describing it can only be inferred by the set of the received observations. These types of Markov models are known as hidden Markov models ( HMMs). Let us consider some simple examples of such processes inspired by the wellknown coin-tossing problem. We will assume that in all experiments the coin tossing takes place behind a curtain, and all that is disclosed to us is the outcome of each experiment. That is, each time an experiment is performed we cannot know the specific coin (in the case of multiple coins) whose tossing resulted in the current observation ( heads or tails). Thus, a crucial part of the probabilistic process is hidden to us. In the first experiment,a single coin is tossed to produce a sequence of heads (H ) and tails (T ). This experiment is characterized by a single parameter indicating the propensity of the coin for landing heads. This is quantified by the probability P(H) of H (P(T ) 1 P(H)). A straightforward modeling of this statistical process is to associate one state with the outcome H and one with the outcome T . Hence, this is another example of a process with observable states, since states coincide with the
“11-Ch09-SA272” 18/9/2008 page 533
9.6 Hidden Markov Models
P(1|1) 5 P(H)
P(2|2) 5 1 2 P(H) P(2|1) 5 1 2 P(H)
1
P(2|2)
P(1|1) P(2|1) 5 1 2 P(1|1)
2 P(1|2) 5 P(H)
P(H)
1 2 P(H)
P(1|2) 5 P(2|2) P1(H)
P2(H)
P1(T) 5 1 2 P1(H) (a)
P2(T) 5 1 2 P2(H) (b)
FIGURE 9.6 Markov models for hidden coin-tossing experiments: (a) single coin and (b) two coins.
observations. Figure 9.6a illustrates the underlying generation mechanism of the sequence of observations. P(i|j) denotes the transition probability from state sj to state si , once the coin has been tossed and an observation has been made available to us. For simplicity,states are shown simply as j and i,respectively. For this specific example, state i 1 represents H and state j 2 represents T . Also, for this case, all transition probabilities are expressed in terms of one parameter; that is, P(H). For example, assume that the coin is in state i 1 (H ). Tossing the coin again it can either result in H (the coin stays in the same state and P(1|1) P(H)) or it can jump into the other state resulting in T (P(2|1) P(T ) 1 P(H)). In the second experiment,we assume that two coins are used behind the curtain. Although, again, the sequence of observations consists of a random succession of heads or tails, it is apparent that this process cannot be sufficiently modeled by a single parameter. To model the process we assume two states, corresponding to the two coins. The model is shown in Figure 9.6b. Two sets of parameters are involved. One of them consists of the probabilities P1 (H) and P2 (H); that is, the probabilities of H for coins 1 and 2, respectively. The other set of parameters are the transition probabilities, P(i|j), i, j 1, 2. For example, P(1|2) is the probability that the current observation (which can be either H or T ) is the outcome of an experiment performed using coin 1 (state i 1) and that the previous observation was the result of tossing coin 2 (state j 2). Self-transition probabilities,for example, P(1|1), mean that the same coin (1) is tossed twice and the process remains in the same state (i 1) for two successive times. Taking into account that probabilities of an event add to one,two of the transition parameters are sufficient,and this amounts to a total of four parameters (i.e., P1 (H), P2 (H), P(1|1), P(2|2)). It is important to point out that the states of this process are not observable, since we have no access to the information related to which coin is tossed each time. Figure 9.7 shows the Markov model for the case of tossing three coins behind the curtain. Nine parameters are now required to describe the process; that is, the probabilities Pi (H), i 1, 2, 3, one for each coin and six transition probabilities (the number of transition probabilities is nine but there are three constraints;
533
“11-Ch09-SA272” 18/9/2008 page 534
534
CHAPTER 9 Context-Dependent Classification
P(2|2)
P(1|1) P(2|1)
P(1|2) P(3|1)
P(2|3) P(3|2)
P(1|3)
P(3|3)
P1(H)
P2(H)
P3(H)
P1(T) 1 P1(H)
P2(T) 1 P2(H )
P3(T) 1 P3(H)
FIGURE 9.7 Markov model for three hidden coins.
i.e., 3i1 P(i|j) 1, j 1, 2, 3). The case of three coins is another example of a probabilistic process with hidden states. As we will soon see, a major task associated with HMM is first to adopt a model for the underlying process that produces the sequence of observations and then to estimate the unknown set of parameters based on these observations. For the tossing coins examples, these parameters are the transition probabilities as well as the head or tail probabilities for each of the coins. No doubt, adopting the right model is crucial. If, for example, the head or tail observations were produced by tossing two coins and we selected, wrongly, a three-coin model, the resulting estimates of the parameters would lead to an overall poor modeling of the received observations. In general, an HMM is a type of stochastic modeling appropriate for nonstationary stochastic sequences, with statistical properties that undergo distinct random transitions among a set of different stationary processes. In other words, an HMM models a sequence of observations as a piecewise stationary process. Such models have been used extensively in speech recognition to model speech utterances [Bake 75, Jeli 76]. An utterance may be a spoken word, part of a word, or even a complete sentence or a paragraph. The statistical properties of the speech signal within an utterance undergo a series of transitions. For example, a word consists of
“11-Ch09-SA272” 18/9/2008 page 535
9.6 Hidden Markov Models
subword portions of voiced (vowels) and unvoiced (consonants) sounds. These are characterized by distinctly different statistical properties,which are in turn reflected in transitions of the speech signal from one statistic to another. Handwriting recognition [Chen 95, Vlon 92, Agaz 93, ElYa 99, Aric 02, Ramd 03], texture classification [Chen 95a, Wu 96], blind equalization [Anto 97, Kale 94, Geor 98], musical pattern recognition [Pikr 06] are some other example applications in which the power of HMM modeling has been successfully exploited. An HMM model is basically a stochastic finite state automaton, which generates an observation string, that is, the sequence of observation vectors, x 1 , x 2 , . . . , x N . Thus,an HMM model consists of a number of,say K ,states and the observation string is produced as a result of successive transitions from one state i to another j. We will adopt the so-called Moore machine model, according to which observations are produced as emissions from the states upon arrival (of the transition) at each state. Figure 9.8 shows a typical diagram of an HMM of three states, where arrows indicate transitions. Such a model could correspond to a short word with three different stationary parts. The model provides information about the successive transitions between the states (P(i| j), i, j 1, 2, 3—temporal modeling of the spoken word) and also about the stationary statistics underlying each state ( p(x|i), i 1, 2, 3). This type of HMM model is also known as left to right, because transitions to states with a smaller index are not allowed. Other models also do exist [Rabi 89]. In practice, the states correspond to certain physical characteristics, such as distinct sounds. In speech recognition, the number of states depends on the expected number of such sound phenomena (phonemes)1 within one word. Actually, a number of states (typically three or four) are used for each phoneme. The average number of observations, resulting from various versions of a spoken word,
P(1|1)
P(2|2)
P(2|1)
p(x|1)
P(3|3)
P(3|2)
p(x|2)
x
p(x|3)
x
x
FIGURE 9.8 Model parameters describing a three-state hidden Markov model. 1 A phoneme is a basic sound unit corresponding to a unique set of articulatory gestures,characterizing
the vocal tract articulators for speech sound production [Dell 93].
535
“11-Ch09-SA272” 18/9/2008 page 536
536
CHAPTER 9 Context-Dependent Classification
can also be used as an indication of the number of required states. However, the exact number of states is often the result of experimentation and cannot be determined accurately a priori. In blind equalization, the states are associated with the number of clusters formed by the received data [Geor 98]. In character recognition tasks, the states may correspond to line or arc segments in the character [Vlon 92]. In the sequel we will assume that we are given a set of M known reference patterns, and our goal is to recognize which one of them an unknown test pattern matches best. This problem was studied in the previous chapter from a template matching (deterministic) perspective. A different (stochastic) path will be taken here. Specifically, we will assume that each known (reference) pattern is described via an HMM model. That is, each of the M patterns is characterized by the following set of parameters: ■ ■
■
■
The number Ks of the states, s 1, 2, . . . , M. The probability densities p(x| j), j 1, 2, . . . , Ks , describing the distribution of the observations emitted from state j. The transition probabilities P(i| j), i, j 1, 2, . . . , Ks , among the various states. Some of them can be zero. The probabilities P(i), i 1, 2, . . . , Ks , of the initial state.
Although this is quite a general description of an HMM model, it is worth pointing out that variations are also possible. For example, sometimes the place of the selftransition probability (P(i|i)) is taken by the state duration probability distribution, which describes the number of successive stages for which the model stays in state i (Section 9.7 ). In some other cases, a model for the generation mechanism of the observations is adopted—for example, an autoregressive model [Pori 82] or even a time-varying one that models nonstationary state statistics [Deng 94]. In the sequel we will adhere to the foregoing model. Our problem now consists of two major tasks. One is the training for each of the HMM models, that is, the computation of the parameters just listed. The other is the task of recognition. That is,once the HMM parameters of the reference models are known, how do we decide which reference model the unknown pattern matches best? We will start with the latter task.
Recognition Any path method To start with, we will treat each of the M reference HMM models as a distinct class. The sequence of observations X is the result of emissions, due to transitions among the different states of the respective model. The problem then becomes a typical classification task. Given the sequence of N observations X : x 1 , x 2 , . . . , x N , resulting from the unknown pattern, decide to which class it belongs. The Bayesian classifier decides in favor of pattern S ∗ for which S ∗ arg max P(S|X), S
that is, over all the models
(9.21)
“11-Ch09-SA272” 18/9/2008 page 537
9.6 Hidden Markov Models
and for equiprobable reference models (classes) this is equivalent to S ∗ arg max p(X|S)
(9.22)
S
where for convenience we have used S to denote the set of parameters describing each HMM model, that is, S {P(i|j), p(x|i), P(i), Ks }
For each model S there is more than one possible set of successive state transitions i , each having probability of occurrence P(i |S). Thus, recalling the known rule for probabilities, we can write [Papo 91] p(X|S)
i
p(X, i |S)
p(X|i , S)P(i |S)
(9.23)
i
In order to find the maximum p(X|S) over all possible models, the quantity in (9.23) has to be computed for each of the M reference models. Its efficient computation can be achieved via an approach similar to the Viterbi algorithm. Indeed, the only difference between (9.23) and (9.6) is that, instead of simply searching for the maximum over all possible state sequences i , (9.23) requires summing up the respective values for each of them. To this end, let us define as ␣(ik ) the probability density of the joint event: (a) a path is at state ik (ik ∈ {1, 2, . . . , Ks }) at stage k and (b) observations x 1 , x 2 , . . . , x k1 have been emitted at the previous stages and (c) observation x k is emitted from the state ik at stage k. From the definition of ␣(ik ) the following recursive relation is easily understood: ␣(ik1 ) ≡ p(x 1 , . . . , x k1 , ik1 |S) ␣(ik )P(ik1 |ik )p(x k1 |ik1 ),
k 1, 2, . . . , N 1
(9.24)
ik
with ␣(i1 ) P(i1 )p(x 1 |i1 )
The product P(ik1 |ik )p(x k1 |ik1 ) in (9.24) provides the local information for the last transition, and ␣(ik ) is the information accumulated from the path history up to stage k. As is apparent from its definition, ␣(ik ) does not depend on the subsequent observations x k1 , . . . , x N . The definition of a joint probability density function, which depends on all available observations, is also possible and will be used later on. To this end, let us define (ik ) as the probability density function of the event: observations x k1 , . . . , x N occur at stages k1, . . . , N ,given that at stage k the path is at state ik . Then after a little thought we conclude that (ik ) obeys the following recursion: (ik ) ≡ p(x k1 , x k2 , . . . , x N |ik , S) (ik1 )P(ik1 |ik )p(x k1 |ik1 ), ik1
k N 1, . . . , 1
(9.25)
537
“11-Ch09-SA272” 18/9/2008 page 538
538
CHAPTER 9 Context-Dependent Classification
where by definition (iN ) 1, iN ∈ {1, 2, . . . , Ks }
(9.26)
Thus, the probability density of the joint event: (a) a path is at state ik at stage k and ( b) x 1 , . . . , x N have been observed, is given by ␥(ik ) ≡ p(x 1 , . . . , x N , ik |S) p(x 1 , . . . , x k , ik |S)p(x k1 , . . . , x N |ik , S) ␣(ik )(ik )
(9.27)
where the assumption about the observations’ independence has been employed. Equation (9.27) also justifies the choice (iN ) 1, iN ∈ {1, 2, . . . , Ks }. Let us now return to our original goal of computing the maximum p(X|S). Equation (9.24) suggests that we can write Eq. (9.23) as p(X|S)
Ks
␣(iN )
(9.28)
iN 1
For the computation of (9.28), we need to compute all ␣(ik ), for k 1, 2, . . . , N . This is efficiently achieved using the diagram of Figure 9.9. Each node corresponds to a stage k and a state ik , ik 1, 2, . . . , Ks . For each of the nodes the density i1 5 Ks
i 2 5 Ks
ik 5 Ks
ik 1 1 5 Ks
iN 5 Ks
P(ik 11 ik) a(ik)
a(ik 1 1) p(xk 1 1 ik 1 1)
i1 5 1
i2 5 1
ik 5 1
ik 1 1 5 1
iN 5 1
x1
x2
xk
xk 1 1
xN
FIGURE 9.9 Diagram showing the computational flow for the any path method.
“11-Ch09-SA272” 18/9/2008 page 539
9.6 Hidden Markov Models
␣(ik ) from Eq. (9.24) is computed. Thus, the number of computations is of the order of NKs2 . The resulting algorithm is known as the any path method, since all paths participate in the final cost. The computation of (9.28) is performed for each of the M models, and the unknown string pattern of the observations x 1 , x 2 , . . . , x N is classified to the reference model S for which p(X|S) becomes maximum.
Best Path Method An alternative, suboptimal, approach is the so-called best path method. According to this method, for a given observation sequence X, we compute the most probable (best) path of states sequence for each of the reference models. The task now becomes that of (9.1), and the search for each of the optima can be achieved efficiently via the Viterbi algorithm, with the cost D given as in (9.9), D
N
d(ik , ik1 )
(9.29)
k1
d(ik , ik1 ) ln P(ik |ik1 ) ln p(x k |ik )
In other words, for each of the models we compute the maximum of P(i ) p(X|i ) p(i , X). Hence, the summation in (9.23) is replaced by a maximum operation. The unknown pattern is classified to that reference model S for which the resulting optimal cost D is maximum.
Training Training is a more difficult task. The states now are not observable, and a direct training approach, such as the one used in Section 9.5, cannot be adopted. The parameters that define each HMM model S can only be inferred from the available observations. One way to achieve this goal is to estimate the unknown parameters, so that the output for each model, (9.29) or (9.28), becomes maximum for a training set of observations known to belong to the model. This is an optimization task with a nonlinear cost function, and it is carried out iteratively. To this end, assumptions about the probability density functions p(x|i) are required. The procedure can be simplified if one assumes that the observations x k can take only discrete values. In such cases probability density functions p(x|i) become probabilities, P(x|i).
Discrete Observation HMM Models We will assume that the training observation string x k , k 1, 2, . . . , N , consists of quantized vectors. In practice,this is achieved via vector quantization techniques,to be discussed later in Chapter 14. Hence, each observation vector can take one only out of L possible distinct values in the l-dimensional space. Thus, observations
539
“11-Ch09-SA272” 18/9/2008 page 540
540
CHAPTER 9 Context-Dependent Classification
can be described as integers r, r 1, 2, . . . , L. The steps for each of the two methods, that is, any path and best path, are as following:
Baum–Welch Reestimation The “output” quantity in the any path procedure is p(X|S). Thus, estimating the parameters of the model S so that p(X|S) is a maximum is nothing but a maximum likelihood parameter estimation procedure. Before going into the discussion of the iteration steps some definitions are needed. Definitions ■
■
k (i, j, X|S) ≡ the probability of the joint event: (a) a path passes through state i at stage k and ( b) through state j at the next stage k 1 and (c) the model generates the available sequence of observations X, given the parameters of the model S. ␥k (i|X, S) ≡ the probability of the event: a path passes through state i at stage k given the model and the available observation sequence.
From these definitions, it is not difficult to show that k (i, j) ≡ k (i, j|X, S)
k (i, j, X|S) P(X|S)
(9.30)
Mobilizing the definitions in Eqs. (9.24) and (9.25), Eq. (9.30) becomes k (i, j)
␣(ik i)P( j|i)P(x k1 | j)(ik1 j) P(X|S)
(9.31)
where ␣(ik i) accounts for the path history terminating at stage k and state i. (ik1 j) accounts for the future of the path, which at stage k 1 is at state j and then evolves unconstrained until the end. The product P( j|i)P(x k1 | j) accounts for the local activity at stage k. The other quantity of interest is given by ␥k (i) ≡ ␥k (i|X, S)
␣(ik i)(ik i) P(X|S)
(9.32)
From the foregoing it is not difficult to see that N ■ k1 ␥k (i) can be regarded as the expected (over the number of stages) number of times state i occurs, given the model S and the observation sequence X. When the upper index in the summation is N 1, this quantity is the expected number of transitions from state i. N 1 ■ k1 k (i, j) can be regarded as the expected number of transitions from state i to state j, given the model and the observation sequence.
“11-Ch09-SA272” 18/9/2008 page 541
9.6 Hidden Markov Models
The preceding definitions lead us to adopt the following (re)estimation formulas as reasonable estimates of the unknown model parameters. N 1 (i, j) ¯ j|i) k1 k P( N 1 k1 ␥k (i) N
P¯ x (r|i)
(k1 and x →r) ␥k (i) N k1 ␥k (i)
¯ P(i) ␥1 (i)
(9.33)
(9.34) (9.35)
The numerator in (9.34) sums only those of ␥k (i) for which the corresponding observation x k takes the rth discrete value. The iterative algorithm can now be expressed in terms of the following steps: Iterations ■
Initial conditions: Assume initial conditions for the unknown quantities. Compute P(X|S).
■
Step 1: From the current estimates of the model parameters reestimate the new model S¯ via Eqs. (9.33) to (9.35).
■
¯ If P(X | S) ¯ P(X | S) ⑀ set S S¯ and go to Step 2: Compute P(X | S). step 1. Otherwise stop.
Remarks ■
¯ that is, it is true that P(X|S) ¯ P(X|S). Each iteration improves the model S;
■
The algorithm may lead to local maxima; see,for example,[Baum 67, Baum 68, Baum 70]. This is why the algorithm in practice runs a number of times, starting from different initial conditions, in order to find a favorable local maximum for P(X|S). Other computational issues, including parallelism and memory requirements, are treated in [Turi 98].
■
The Baum–Welch algorithm is basically an implementation of the EM algorithm,which was introduced in Chapter 2. Indeed,a little thought reveals that the ML estimation of the HMM parameters is a typical ML problem with an incomplete data set, that is, the unobserved states (e.g., [Moon 96, Diga 93]). A generalization of the method that allows multiple observation training sequences is given in [Li 00]. Other, gradient-based, optimizing techniques for the estimation of the unknown parameters have also been suggested and used [Levi 83].
■
Practical implementation issues: 1. Scaling: The probabilities ␣(ik ), (ik ) are obviously less than one, and their values tend to zero very fast as the number of terms in the products (9.24) and (9.25) increases. In practice, the dynamic range of their computed values may exceed the precision range of the computer, so
541
“11-Ch09-SA272” 18/9/2008 page 542
542
CHAPTER 9 Context-Dependent Classification
appropriate scaling is required. A basic procedure is to scale ␣(ik ) in proportion to the number of stages. If the same scaling factor is used for the (ik ), then on taking their product in the recursions the effect of the scaling cancels out [Levi 83, Rabi 89] ( Problem 9.4). 2. Initial conditions: This is an omnipresent problem in all iterative optimization algorithms. Usually, the unknown parameters are initialized randomly, subject, of course, to the constraints of the problem. That is, if some transitions are not allowed, the corresponding probabilities are set to zero, and also probabilities must add to one. 3. Insufficient training data: Generally, a large amount of training data is necessary to learn the HMM parameters. The observation sequence must be sufficiently long with respect to the number of states of the HMM model. This will guarantee that all state transitions will appear a sufficient number of times,so that the reestimation algorithm learns their respective parameters. If this not the case,a number of techniques have been devised to cope with the issue. For a more detailed treatment the reader may consult [Rabi 89, Dell 93] and the references therein.
Viterbi Reestimation In the speech literature the algorithm is also known as the segmental k-means training algorithm [Rabi 89]. It is related to the best path method. Definition 1. ■
ni|j ≡ number of transitions from state j to state i.
■
n|j ≡ number of transitions originated from state j.
■
ni| ≡ number of transitions terminated at state i.
■
n(r|i) ≡ number of times observation r ∈ {1, 2, . . . , L} occurs jointly with state i.
Iterations ■
Initial conditions: Assume the initial estimates of the unknown parameters. Obtain the best path and compute D.
■
Step 1: From the available best path, reestimate the new model parameters as: ¯ j) P(i| ¯ Px(r|i)
■
ni| j n| j n(r|i) ni|
Step 2: For the new model parameters obtain the best path and compute the ¯ Compare it with the cost D of the previous corresponding overall cost D. ¯ D ⑀ set D D ¯ and go to step 1. Otherwise stop. iteration. If D
“11-Ch09-SA272” 18/9/2008 page 543
9.6 Hidden Markov Models
¯ Symbol Px(r|i) is the current iteration estimate of the probability of emitting from state i the rth value from the available palette of the L possible vectors. The preceding algorithm has assumed that the initial state is known; thus no estimation of the respective probabilities is needed. This is, for example, true for left-to-right models, such as the one shown in Figure 9.8. The Viterbi reestimation algorithm can be shown to converge to a proper characterization of the underlying observations [Fu 82, Lee 72].
Continuous Observation HMM The discrete observation modeling of originally continuous variables suffers from a serious drawback. During the (vector) quantization stage of the signal (e.g., speech segment),a severe loss of information about the original waveform may occur,which can seriously degrade the performance of the recognizer. The alternative is to work with continuous observation modeling, albeit at the expense of higher complexity. This approach requires modeling of the probability densities p(x|i), prior to estimation. Once these have been estimated, the recognition problem evolves along the same lines as with the discrete observation case. The difference exists only in the training task. One way to approach the problem is to assume a parametric model for the probability density function and then use reestimation procedures to compute the unknown model parameters. As we have already discussed in Chapter 2, a very general parameterization of the probability density function is via mixture modeling, that is, p(x|i)
L
cim F (x, im , im )
(9.36)
m1
where F (·, ·, ·) is a density function and im , im are the mean vector and the covariance matrix of the mth mixture. We will adhere to Gaussian functions, which are usually employed in practice. The mixture coefficients cim have to satisfy the constraint L
cim 1,
1 i Ks
m1
so that
p(x|i) dx 1,
1 i Ks
Following arguments similar to those used to reestimate the parameters in the discrete observation HMM case, the following reestimation formulas are obtained [Lipo 82, Juan 85, Juan 86]. N
k1 ␥k (i, m) L k1 r1 ␥k (i, r)
c¯im N
(9.37)
N ¯ im
k1 ␥k (i, m)x k
N
k1 ␥k (i, m)
(9.38)
543
“11-Ch09-SA272” 18/9/2008 page 544
544
CHAPTER 9 Context-Dependent Classification
¯ im
N
¯ im )(x k k1 ␥k (i, m)(x k N k1 ␥k (i, m)
¯ im )T
(9.39)
The term ␥k (i, m) is the probability density of being at state i and stage k with the mth mixture component accounting for x k , that is, ␣(ik i)(ik i) cim F (x k , im , im ) ␥k (i, m) K L s ␣(i )(i ) k k r1 cir F (x k , ir , ir ) ik 1
(9.40)
where cim is the ratio of the expected number of times the system is at state i using the mth mixture component to the overall expected number of times the system is at state i. Similar interpretations can be made for the other formulas too. When the Viterbi method is employed, reestimation of the parameters is based on averages computed across the best path. For example, for mixture modeling using a single Gaussian (L 1) we get i
N 1 x k ␦ik Ni k1
i
N 1 (x k i )(x k i )T ␦ik Ni k1
where ␦ik 1 for the stages where the path goes through state i and is zero otherwise, and Ni is the respective number of times the path passes through state i. Remarks ■
The algorithms just described estimate the unknown parameters using all the available observations simultaneously. An alternative path, of major practical importance, is to employ adaptive techniques in which new information can be incorporated to adapt an already trained model, without it being necessary to retrain it with all previously used data. It is generally accepted that speaker-dependent recognizers outperform speaker-independent systems, as long as sufficient training data are available. Thus, a long-standing idea is to use speaker-independent recognizers, trained with enough data on a multispeaker platform,and then adapt the model parameters to fit a specific speaker (and/or acoustic environment). This can be achieved by using the minimum number of data from the new speaker. Both batch and sequential schemes have been suggested. Some examples of such learning procedures are given in [Lee 91, Diga 95, Legg 95, Huo 95, Huo 97, Diga 93, Wang 01].
■
A drawback of the modeling in (9.36) is that a mixture model is adopted for each of the states. This makes the number of parameters to be estimated rather high. Thus, for a given size of training data, it affects the robustness of the parameter estimation. To alleviate such problems and decrease the number of unknown parameters, so-called tied-mixture densities modeling has been
“11-Ch09-SA272” 18/9/2008 page 545
9.7 HMM with State Duration Modeling
suggested, where the same Gaussian densities are shared across the mixtures of all the states [Bell 90] or groups of states [Diga 96, Kim 95, Gale 99, Gu 02]. ■
The Baum–Welch algorithm is an iterative procedure to maximize the likelihood function with respect to the unknown parameters. MAP procedures incorporating prior statistical information have also been proposed, and enhanced performance has been reported [Gauv 94]. An alternative is to optimize with respect to all the unknown parameters, instead of optimizing each HMM model separately, as was the case with ML earlier. The goal of such an optimization approach is to enhance the discrimination capabilities of the models, see, for example, [He 08]. Maximizing the mutual information [Bahl 86] or minimizing the cross-entropy [Ephr 89] and, more recently, the classification error rate by using either a smooth version of it [ Juan 92, Juan 97] or the deterministic annealing technique [Rao 01] or controlling the influence of the outliers [Arsl 99] are examples of approaches that enhance performance at the expense of complexity. More recently, [Li 04] suggested the use of a deterministic annealing technique that allows one to adapt the number of states during training. This can offer some advantages for those cases where the number of states cannot be accurately predetermined.
■
HMM are graphical models and they belong to a class of Bayesian networks (discussed in Chapter 2) known as dynamic Bayesian networks [Neap 04].
9.7 HMM WITH STATE DURATION MODELING Hidden Markov modeling,as we have approached it so far,falls short of expectations in many cases in practice. Experimental evidence has identified a serious shortcoming associated with the use of the self-transition probabilities, P(i|i), as parameters in the standard HMMs. This is related to the exponential modeling of the state duration probability, Pi (d), that such a modeling implies, where d is the successive number of times the model remains in state i. Indeed, given P(i|i) the probability of a path leaving current state i is equal to 1 P(i|i). Hence, the probability of being in state i for d successive instants (i.e., d 1 self-transitions, and emission of d consecutive observations from state i) is given by Pi (d) (P(i|i))d1 (1 P(i|i))
(9.41)
For many cases (e.g., for a number of audio signals), such an exponential state duration modeling is inappropriate. To alleviate this drawback, it has been suggested to substitute the self-transition probability, P(i|i), by an explicit variable state duration
545
“11-Ch09-SA272” 18/9/2008 page 546
546
CHAPTER 9 Context-Dependent Classification
probability Pi (d) in the set of unknown HMM parameters [Ferg 80]. Thus, under this new setting, the set of the unknown parameters defining an HMM consists of ■
The number Ks of the states.
■
The probability densities p(x|j) (they become probabilities for the case of discrete observation models, i.e., x ∈ {1, 2, . . . , L}).
■
The state transition probabilities P(i|j), i, j 1, 2, . . . , Ks .
■
The state duration probabilities, Pi (d), i 1, 2, . . . , Ks , 1 d D.
■
The probabilities P(i), i 1, 2, . . . , Ks , of the initial state.
Observe that a maximum allowable state duration D has been adopted. Thus, the model S can now be written as S {P(i| j), Pi (d), p(x|i), P(i), Ks }
Our goal remains the computation of the maximum of P(X|S) in (9.23). To achieve this in an efficient way, we have to modify the set of auxiliary variables used with the standard HMM so that we can adapt to the needs of the new parameterization. To this end, define ␣k (i) to be the probability density of the joint event: (a) a stay at state i ends at stage k and (b) observations x 1 , x 2 , . . . , x k have been emitted up to stage k. That is, ␣k (i) p(x 1 , x 2 , . . . , x k , state i ends at stage k|S)
(9.42)
Note the slightly different notation used here, compared to (9.24), to emphasize the different meaning of the involved variables. Since the stay at state i ends at stage k, the next state, at stage k 1, can take any value except i; that is, ik1 i. Furthermore, looking at the path history up to stage k, there are various ways to reach state ik i. One is to jump to ik i from an ik1 i, and this suggests that only one symbol is emitted from state i. The probability of this event depends on the value of Pi (d 1), since we know that the path will depart from state i at k 1. The second possibility is the path to be at state i at stage k 1 and remain there for two successive stages (i.e., ik1 ik i). The probability of such an event is equal to Pi (d 2). This rationale can be pushed backward D 1 steps prior to k (i.e., ikD1 · · · ik i), and the probability of this event is given by Pi (D). This, of course, can be applied to all stages prior to k (i.e., 1, 2, . . . , k 1). From this discussion, it is not difficult to write the counterpart of recursion (9.24) as ␣k (i)
Ks
D
( j1, j i) d1
k
␣kd ( j)P(i| j)Pi (d)
p(x m |i)
(9.43)
mkd1
where, once more, statistical independence between observations has been assumed. Initialization of (9.43) requires the following computations as it can easily be understood from the respective definitions: ␣1 ( j) P( j)Pj (1)p(x 1 | j)
“11-Ch09-SA272” 18/9/2008 page 547
9.7 HMM with State Duration Modeling
For k 2, 3, . . . , D and j 1, 2, . . . , Ks , ␣k (j) P( j)Pj (k)
k
p(x m | j)
m1
Ks
k1
k
␣kd (r)P( j|r)Pj (d)
(r1, r j) d1
p(x m | j)
mkd1
As was the case with the standard form of HMMs during the recognition phase, the desired quantity p(X|S) can now be obtained from p(X|S)
Ks
␣N (i)
(9.44)
i1
which is easily understood from the respective definitions and can efficiently be obtained by the repeated computation of Eq. (9.43) over all ks and is. For the training phase, in order to derive reestimation formulas for the set of the unknown parameters (P(i| j), P(i), Pi (d)) and for a given number of states Ks the following auxiliary variables need to be defined [Rabi 93]. Variable ␣∗k (i) is the probability density ( probability for the discrete observations case) of the joint event: (a) a path starts its stay at state i at stage k 1 and ( b) observations x 1 , x 2 , . . . , x k have been emitted. That is, ␣∗k (i) p(x 1 , x 2 , . . . , x k , state i starts at stage k 1|S)
From the respective definitions, the following are easily established. ␣∗k (i)
Ks
␣k ( j)P(i|j)
(9.45)
j1, j i
␣k (i)
D
␣∗kd (i)Pi (d)
d1
k
p(x m |i)
(9.46)
mkd1
In addition, let k (i) be the conditional probability density of the event: observations x k1 , x k2 , . . . , x N have been observed, given that the path ends at state i and at stage k. That is, k (i) p(x k1 , x k2 , . . . , x N |the path ends its stay at state i at stage k, S )
Also, ∗k (i) is the conditional probability density of the event: observations x k1 , x k2 , . . . , x N have been observed, given that the path starts its stay at state i at stage k 1. That is, ∗k (i) p(x k1 , x k2 , . . . , x N |the path starts its stay at state i and stage k 1, S)
The previous definitions allow us to write k (i)
Ks j1, ji
∗k ( j)P( j|i)
(9.47)
547
“11-Ch09-SA272” 18/9/2008 page 548
548
CHAPTER 9 Context-Dependent Classification
∗k (i)
D
kd
kd (i)Pi (d)
d1
p(x m |i)
(9.48)
mk1
Here, via the definitions, the following initial conditions hold (combined with (9.47)): N (i) 1,
i 1, 2, . . . , Ks
(9.49)
and ∗k (i)
N k
kd (i)Pi (d)
d1
kd
p(x m |i),
k N 1, . . . , N D
(9.50)
k1
Based on the previous auxiliary variables and the derived relationships, the following reestimation formulas for the set of unknown parameters are obtained for the discrete observations case (x k → r ∈ {1, 2, . . . , L}). ¯ P(i)
P(i)∗0 (i) P(X|S)
N 1 ␣k (i)P( j|i)∗k ( j) ¯ j|i) k1 P( Ks N 1 ∗ k1 ␣k (i)P( j|i)k ( j) j1
N ∗ ∗ m k ␣m (i)m (i) m k ␣m (i)m (i) k1,x →r P¯ x (r|i) L N k
∗ ∗ m k ␣m (i)m (i) m k ␣m (i)m (i) r1 k1,x k →r N d ∗ ␣k (i)Pi (d)kd (i) kd mk1 P(x m |i) k1 kd N d ∗ ␣ (i)P (d) (i) P(x |i) i kd m d1 k1 k mk1
P¯ i (d) D
(9.51)
(9.52)
(9.53)
(9.54)
Equation (9.51) is straightforward from the definitions and is an implication of the Bayes theorem. Equation (9.52) is the total number of path transitions from state i to state j along all the stages, divided by the total number of transitions that occur from state i. Equation (9.54) is the ratio of the number of times the path starts its stay at state i with duration d, divided by the number of times state i occurs with any duration. Equation (9.53) needs a bit more elaboration. The numerator is the number of times observation x k ∈ {1, 2, . . . , L} is emitted from state i. To be simultaneously at state i and stage k means that a path can either start its stay at state i at stage k or may have started to be at this state at a previous stage (i.e., at k 1, k 2, . . .) and remain there for a corresponding number of successive instants. However, there is a finite probability for a path to start being at state i at a stage earlier than k but not to survive long enough at this state for us to have the chance to “meet” it there at
“11-Ch09-SA272” 18/9/2008 page 549
9.7 HMM with State Duration Modeling
stage k. The term ␣∗m (i)∗m (i) is the probability that a path starts its stay at state i at some stage m 1, and the term ␣m1 (i)m1 (i) is the probability that a path ends its stay at state i at stage m 1. The first summation is the total probability that a path starts its stay at state i at any stage up to k. The second summation is the total probability that a path ends its stay at i at any stage prior to k. Hence, the subtraction of the two summation terms gives the probability of having a path through i at stage k, for the given observation sequence. The denominator is the same quantity, but the summation is over all times that state i is visited by a path, regardless of the emitted observation. Adopting HMM with an explicit state duration probability modeling improves the performance in many recognition tasks compared to the standard HMM. The cost one pays for such an improvement is the increased computational complexity. The storage requirements are increased by an order of D and the computational cost by an order of D 2 . Besides it, the state duration model requires D more parameters to be estimated, in addition to those in the standard HMM.This, unavoidably, leads to a demand for longer training sequences to safeguard enough data for the accurate estimation of all unknown parameters. Some of these problems can be minimized by adopting a parametric model for Pi (d), so that the number of unknown parameters is reduced to the number of parameters describing the parametric probability function. To this end, Gaussian, Poisson, and Gamma distribution models have been used [Levi 86, Russ 85]. In [Chie 03] the case of adapting the parameters of the adopted parametric state duration model is treated, to fit nonstationary speech variations for large vocabulary continuous speech recognition.
Best Path Method For the recognition phase, given a trained HMM and an observation sequence X, the goal now becomes to compute the probability of the most probable path of states sequence. However, this can no longer be achieved by employing the Viterbi algorithm in the form given in Section 9.4. To adapt to the needs of the new parameterization, imposed by the explicit time duration modeling, we have to attack the problem in a slightly different way. For the computation of the best path, up to a node corresponding to stage k and state i, there are now two types of competing paths: (a) paths that end their stay in a state j different to i at stage k 1 and jump to the node (k, i) and (b) paths that end their stay in a j i state at previous stages, k 2, . . . , k D, and then jump to state i and remain there for 2, . . . , D, time instants, respectively. Let ak (i) be the optimal cost up to stage k and state i. According to Bellman’s principle, for the computation of ak (i) the following are valid: ■
For paths through (k 1, j), j 1, 2, . . . , Ks , j i and then jumping to i, ak (i) ak1 ( j)P(i| j)p(x k |i)Pi (1),
j i
(9.55)
549
“11-Ch09-SA272” 18/9/2008 page 550
550
CHAPTER 9 Context-Dependent Classification
■
For paths through nodes (k d, j), j i, d 2, . . . , D, that jump to i and remain there for d successive instants, k
ak (i) akd ( j)P(i| j)Pi (d)
p(x m |i)
(9.56)
mkd1
Thus, the optimal cost associated with node (k, i) results from ak (i)
max
1 d D,1 j Ks ,j i
[␦k ( j, d, i)] k
␦k ( j, d, i) akd ( j)P(i| j)Pi (d)
(9.57)
p(x m |i)
(9.58)
mkd1
Equations (9.57) and (9.58) hold for k D. For k D, initialization of (9.57) and (9.58) requires the following computations: a1 (i) P(i)Pi (1)p(x 1 |i),
i 1, . . . , Ks
For k 2, 3, . . . , D,
ak (i) max P(i)Pi (k)
k
p(x m |i), ␦k ( j, d, i) ,
1 d k, 1 j Ks , j i
m1
According to the previous definitions, it turns out that the optimal path is the one ending at state i, where aN (i) arg max aN ( j) 1 j Ks
Equations (9.57) and (9.58) suggest that for k D there exist (Ks D D) candidate arguments, ␦k ( j, d, i) for the maximization of each quantity ak (i).
Reestimation Equations for the Best Path Method The reestimation formulas for the best path method require maintaining counters to track state transitions, symbol emissions from the individual states, and state durations. For example, the state transition probability P(i|j) is reestimated by counting the number of times the transition from state j to state i appears along the optimal path (computed at the current iteration) and dividing it by the total number of times transitions from state j to any other state are detected. This approach results in the following formulas: ¯ P(i|j) P¯ x (r|i)
number of transitions from state j to state i , total number of transitions from state j
number of times x → r was emitted from state i , total number of observations at state i
i j
r 1, 2, . . . , L
“11-Ch09-SA272” 18/9/2008 page 551
9.7 HMM with State Duration Modeling
P¯ i (d)
number of times d successive observations are emitted from state i total time spent at state i
A variant of the best path state duration HMM modeling is proposed in [Pikr 06], which has been developed to fit the needs of the music recognition/classification task. To this end,the cost function is modified to account for possible errors that are usually encountered in practice—that is,errors in fundamental frequency estimation or variations among recordings of the same music item due to different instrument players.
Segment Modeling Although HMM modeling is one of the most powerful and widely used techniques in recognition, it is not without its shortcomings. One of its principal limitations is the required assumption of independence among the observations (conditioned on the state sequence). In fact, this is not true for most of the cases. Another limitation is the rather weak state duration modeling achieved by standard HMM modeling. To overcome these limitations, a number of schemes have been suggested. Such an example is the state duration HMM modeling. More recently, an effort was made to present a variety of such schemes in a unified framework, under the notion of segment modeling. Here only the basic definitions will be reviewed. In HMM modeling on the arrival of a transition at a state, a single observation (corresponding to a single frame of the original speech samples) is assumed to be emitted and the fundamental observation distribution is on the frame level, that is, p(x|i). In contrast, in segment modeling a segment Xrd consisting of d frames, Xrd [x r , . . . , x rd1 ], is assumed to be emitted upon the arrival at a state. Here d is a random variable itself. The fundamental distribution is now at the segment level,that is,p(Xrd |i, d). A schematic representation is given in Figure 9.10. The parameters describing a segment model are (a) the number of states; (b) their transition modeling parameters; (c) the joint probability density function for the segment distribution, given the duration d; and (d) the duration probability P(d|i). Training of these parameters follows generalizations of the Baum–Welch andViterbi schemes. A more detailed treatment of the topic is beyond our scope, and the interested reader may consult, for example, [Oste 96, Russ 97, Gold 99] and the references therein. xr, xr 1, ..., xr d 1
xr
HMM
SM
FIGURE 9.10 HMM and segment modeling (SM) for the emission of observations upon arrival of a transition at a state.
551
“11-Ch09-SA272” 18/9/2008 page 552
552
CHAPTER 9 Context-Dependent Classification
9.8 TRAINING MARKOV MODELS VIA NEURAL NETWORKS The training phase of an HMM-based recognition system is entirely dedicated to the learning of probabilities (and densities). In Chapter 3 we have seen that a supervised classifier optimized via certain criteria, such as least squares, can approximate posterior class probabilities. This is the kickoff point for our current concern. States can be treated as classes, and a multilayer perceptron can be used as a nonlinear classifier. The observations feed the input nodes, and the network has as many outputs as the states. Training can be done via the backpropagation algorithm, and the desired responses will be 1 at the true state output and 0 at the others (see Chapter 4). It is now straightforward to see that the outputs of the NN will sufficiently approximate the posterior probabilities P(i|x). These can then be changed to p(x|i), as required in the recognition phase of a Markov model–based recognizer, via the Bayes rule p(x|i)
P(i|x)p(x) P(i)
where the state priors P(i) are determined from their relative occurrence frequencies and p(x) is constant for all states during recognition. In following this procedure, we have made a crucial assumption. That is, the states are treated here as being observable,and during training we know the specific state from which each observation originates. For example, in speech recognition this is possible by associating each phoneme in a spoken word with a state. Here lies a disadvantage of this approach, since accurate segmentation of the speech signal is required and the boundaries are not always well defined. This is not the case in the HMM approach, where it is left to the algorithm to decide optimally about the state boundaries. A scheme for unified training of HMM/MLP that avoids the segmentation problem has been suggested [Koni 96]. Let us now turn to the benefits of bringing neural networks into the scene. We have already mentioned that a major disadvantage of the standard HMM is the assumption of independence among the observations. Using a multilayer perceptron,the underlying statistical dependence can easily be accommodated. Figure 9.11 shows a possible way. Together with the “current” observation vector x k , p “past” as well as p “future” ones appear simultaneously at the input nodes. Thus, the input nodes amount to (2p 1)l, with l being the dimension of the observation vectors. During training, the desired response will be 1 at the output node, corresponding to the state that “gives birth”to the “current”vector. We say that the network is trained with contextual input information. Further data dependence can also be accommodated by providing the input with information about the previous state in the sequence. During recognition,this is provided via an output feedback,shown in the figure by the dotted lines [Bourl 90]. Obviously, in such a configuration the output kp nodes of the network compute the conditional state probabilities P(ik |Xkp , ik1 ), kp
where Xkp denotes the contextual input information ranging from x kp to x kp .
“11-Ch09-SA272” 18/9/2008 page 553
9.8 Training Markov Models via Neural Networks
Output Layer
2nd Layer
1st Layer
xkp ... xk1
xk
xk 1 ... xkp
FIGURE 9.11 A multilayer perceptron architecture for training the parameters of a Markov model.
Having these probabilities at our disposal, a number of new “opportunities”open to us. Let us, for example, return to our original goal in (9.1) and treat states as classes. By the chain rule we have P(i |X) ≡ P(i1 , i2 , . . . , iN |X) P(iN |iN 1 , . . . , i1 , X) . . . P(iN 1 |iN 2 , . . . , i1 , X)P(i1 |X)
(9.59)
Taking into account the Markovian property of the state dependence and relaxing a bit the conditional constraint on the observations, this can be written as P(i |X)
kp
P(ik |Xkp , ik1 )
(9.60)
k
with some appropriate initial conditions. Computing its maximum can easily be done via dynamic programming arguments. A number of other alternatives are also possible; see for example [Bourl 90, Bourl 94, Morg 95] for a more detailed discussion of the topic. In [Pikr 06a] the use of Bayesian networks is suggested in place of neural networks. Finally, it must be emphasized that in order to obtain good probability estimates the size of the multilayer perceptron must be large enough to have good approximating capabilities. This, of course, requires increased computational resources for the training. Furthermore, the incorporation of the contextual information imposes its own demands on large networks. Another point is that the approximation of probabilities by the network is valid at the global minimum of the minimized cost function, at least in theory. Practical issues affecting the overall performance of such an approach are reported in [Spec 94].
553
“11-Ch09-SA272” 18/9/2008 page 554
554
CHAPTER 9 Context-Dependent Classification
9.9 A DISCUSSION OF MARKOV RANDOM FIELDS So far, our concern with context-dependent classification has been limited to the one-dimensional case. The current subsection is focused on the related twodimensional generalizations. That is,observations will be treated as two-dimensional sequences X(i, j). Such problems result in image processing, and observations can be, for example, the gray levels of the image array pixels. No doubt, complications arise, and our aim here is to provide the basic definitions and directions and not a detailed treatment of the topic. Let us assume that we are given an array of observations X:X(i, j), i 0, 1, . . . , Nx 1, j 0, 1, . . . , Ny 1,and a corresponding array of classes/states : ij ,where each ij can take one of M values. Once more our objective is, given the array of the observations, to estimate the corresponding values of the state array so that p(X|)P() is maximum
(9.61)
Within the scope of context-dependent classification the values of the elements of will be assumed to be mutually dependent. Furthermore, we will assume that the range of this dependence is limited within a neighborhood. This brings us to the notion of Markov random fields (MRFs) defined in Chapter 7. Thus, for each (i, j) element of the array a respective neighborhood Nij is defined so that ■
ij ∈ Nij
■
ij ∈ Nkl ⇐⇒ kl ∈ Nij
In words, the (i, j) element does not belong to its own set of neighbors, and if ij is a neighbor of kl , then kl is also a neighbor of ij . The Markov property is then defined as ¯ ij ) P(ij |Nij ) P(ij |
(9.62)
¯ ij includes all the elements of except the (i, j) one. Figure 9.12 gives a where typical example of a neighborhood with eight neighbor pixels. Equation (9.62) is a generalization of (9.3). In the one-dimensional case the ordering of the sequence led to the relation (9.4). Unfortunately, this sequence ordering does not generalize in a natural way to the two-dimensional case and imposes limitations on the involvement of the computationally elegant dynamic programming techniques [Hans 82]. A seminal paper that had an impact on the use of MRF modeling in image processing and analysis was that of Geman and Geman [Gema 84]. They built upon the important Hammersley–Clifford theorem, which establishes the equivalence between Markov random fields and Gibbs distributions [Besa 74]. Thus, we can talk of Gibbs random fields (GRFs). A Gibbs conditional probability is of the form P(ij |Nij )
1 1 exp Fk (Ck (i, j)) Z T k
(9.63)
“11-Ch09-SA272” 18/9/2008 page 555
9.9 A discussion of Markov Random Fields
j
i
FIGURE 9.12 Example of a neighborhood involving eight neighbors of the (i, j) element (red).
where Z is a normalizing constant so that probabilities sum up to 1,T is a parameter, and Fk (·) are functions of the states of the pixels in the cliques Ck (i, j). A clique consists of either a single pixel or a set of pixels, which are neighbors of each other, with respect to the type of the chosen neighborhood. Figure 9.13 shows two cases of neighborhoods and the corresponding sets of cliques. A typical example of the exponent function in (9.63) for the four neighbors case is
1 ij ␣1 ␣2 (i1,j i1,j ) ␣2 (i,j1 i,j1 ) T
where the ai ’s are constants. It turns out that the joint probability P() for the Gibbsian model is U () P() exp T
(9.64)
where U ()
i,j
Fk (Ck (i, j))
(9.65)
k
that is, the sum of the functions over all possible cliques associated with the neighborhood. In many cases, the posterior probability P(|X), which is to be maximized (i.e., (9.61)) also turns out to be Gibbsian. Such cases result, for example, if the regions in the image are themselves generated by Markov (e.g., Gaussian twodimensional AR) processes [Deri 86, Chel 85]. Simulating annealing techniques can then be employed to obtain the required maximum [Gema 84].
555
“11-Ch09-SA272” 18/9/2008 page 556
556
CHAPTER 9 Context-Dependent Classification
(a)
(b)
FIGURE 9.13 Two examples of neighborhoods with the corresponding cliques.
Hidden Markov generalizations to the two-dimensional plane have also been considered [Povl 95]. The idea here is to build a pseudolikelihood function starting from the local state transition probabilities, using Besag’s method for coding the image in mutually independent pixel sets [Besa 74]. An alternative EM formulation of the problem was given in [Zhan 94]. Finally,another direction is the combination of Markov random fields and multiresolution analysis. At the subsampling stage, the Markov property is lost and suitable models are derived for the coarser resolutions. For more details the reader may consult, for example, [Laks 93, Bell 94, Kris 97] and the references therein.
9.10 PROBLEMS 9.1 Assume an HMM model with K states and an observation string X of N continuous observations. Assume that the pdf in each state is described by a Gaussian with known diagonal covariance matrix and unknown mean values. Using the EM algorithm, derive the reestimation recursions. Hint: Form the complete data set as Y (X, ), where is the set of the states. 9.2 If the self-transition probability of a state is P(i|i), then the probability of the model being at state i for d successive stages is given by Pi (d) (P(i|i))d1 (1 P(i|i)). Show that the average duration for staying in state i is equal to d¯ 1 1P(i|i)) . 9.3 In practice, a number Q of different versions of the spoken word are used for training, each resulting in a sequence of observations Xm of length Nm ,
“11-Ch09-SA272” 18/9/2008 page 557
MATLAB Programs and Exercises
m 1, 2, . . . , Q. Then comment on the following reestimation formulas: Q
¯ j|i) P(
Nm 1 1 m1 P(Xm |S) k1 k (i, j, Xm |S) Q Nm 1 m 1 m m1 P(Xm |S) k1 ␣ (ik i) (ik Q
P¯ x (r|i)
i)
N m
1 m m m1 P(Xm |S) (k1 and x →r) ␣ (ik i) (ik Q Nm 1 m m m1 P(Xm |S) k1 ␣ (ik i) (ik i)
i)
where superscript m refers to the mth observation sequence, and (i, j, X|S) is defined in (9.30). 9.4 Rederive recursions (9.33) and (9.34) in terms of the scaled versions of ␣,  ␣(i ˆ k i)
where ck
Ks
1 ␣(ik i), ck
ˆ k1 i) ck (ik1 i) (i
ik 1 ␣(ik ).
9.5 Assume that the HMM models are not equiprobable and let be the set of all the unknown parameters for the M available models. Assume now that a training string X is known to correspond to the model Sr . However, during training, maximization of P(Sr |X, ) is done with respect to all the parameters and not only those of the specific model. Show that this optimization leads to a maximum ratio between the contribution p(X|Sr , )P(Sr ) of the correct model and sr p(X|Ss , )P(Ss ) of the incorrect models. That is, optimization with respect to all the parameters offers maximum discrimination power.
MATLAB PROGRAMS AND EXERCISES Computer Programs 9.1 Recognition score for HMMs using the Baum-Welch method. Write a MATLAB function named Baum_Welch_Do_HMM that takes as input: (a) a column vector of initial state probabilities pi_init, (b) the transition matrix A, whose (i, j) element is the probability of transition from state i to state j, (c) the matrix of the emission probabilities B, whose (i, j) element is the probability to emit the ith alphabet symbol from state j, and (d) a row vector O, which contains a sequence of the code numbers of discrete symbols. It returns the score produced when the HMM, defined by pi_init, A, B, is applied to the sequence of symbols contained in O. Assume that if the alphabet symbols are, say, s1 , s2 , . . . , sq , the corresponding code numbers are 1, 2, . . . , q.
Solution In order to avoid underflow problems, in the following implementation the score is computed as the log product of scaling factors.
557
“11-Ch09-SA272” 18/9/2008 page 558
558
CHAPTER 9 Context-Dependent Classification
function matching_prob=Baum_Welch_Do_HMM(pi_init,A,B,O) %Initialization T=length(O); [M,N]=size(B); alpha(:,1)=pi_init .* B(O(1),:)'; c(1)=1/(sum(alpha(:,1))); alpha(:,1)=c(1)*alpha(:,1); for t=2:T for i=1:N alpha(i,t)=sum((alpha(:,t-1).* A(:,i)) * B(O(t),i)); end c(t)=1/(sum(alpha(:,t))); alpha(:,t)=c(t)*alpha(:,t); end matching_prob=-sum(log10(c));
9.2 Viterbi method for Discrete Observation HMMs. Write a MATLAB function named Viterbi_Do_HMM that takes the same inputs as Baum_Welch_ Do_HMM and returns (a) the best-state sequence and (b) the respective probability produced when the HMM, defined by pi_init, A and B, is applied to O, using the Viterbi method.
Solution In the following implementation, the trellis diagram is constructed first, and then the best state sequence is extracted using the back_traking function defined in the computer programs’ section of Chapter 8. function [matching_prob,best_path]=Viterbi_Do_HMM... (pi_init,A,B,O) %Initialization T=length(O); [M,N]=size(B); pi_init(find(pi_init==0))=-inf; pi_init(find(pi_init>0))=log10(pi_init(find(pi_init>0))); A(find(A==0))=-inf; A(find(A>0))=log10(A(find(A>0))); B(find(B==0))=-inf; B(find(B>0))=log10(B(find(B>0))); % First observation alpha(:,1)=pi_init + B(O(1),:)'; pred(:,1)=zeros(N,1); % Construct the trellis diagram for t=2:T
“11-Ch09-SA272” 18/9/2008 page 559
MATLAB Programs and Exercises
for i=1:N temp=alpha(:,t-1)+A(:,i)+B(O(t),i); [alpha(i,t),ind]=max(temp); pred(i,t)=ind+sqrt(-1)*(t-1); end end [matching_prob,winner_ind]=max(alpha(:,T)); best_path=back_tracking(pred,winner_ind,T);
9.3 Viterbi method for Continuous Observation HMMs. Under the hypothesis that the emission pdfs p(x|i) are Gaussians, write a MATLAB function named Viterbi_Co_HMM, which takes as inputs: (a) a column vector of initial state probabilities pi_init, (b) the transition matrix A, (c) a row vector, m, whose ith element is the mean of the ith Gaussian emission pdf, (d) a row vector, sigma, containing the variances of the previous pdfs, and (e) a row vector O, which contains a feature sequence. It returns, (a) the best-state sequence and (b) the respective probability, produced when the HMM, defined by pi_init, A, m, sigma, is applied to O, using the Viterbi method.
Solution function [matching_prob,best_path]=Viterbi_Co_HMM... (pi_init,A,m,sigma,O) %Initialization T=length(O); [N,N]=size(A); pi_init(find(pi_init==0))=-inf; pi_init(find(pi_init>0))=log10(pi_init(find(pi_init>0))); A(find(A==0))=-inf; A(find(A>0))=log10(A(find(A>0))); for i=1:N alpha(i,1)=pi_init(i)+log10(normpdf(O(1),m(i),sigma(1))); end pred(:,1)=zeros(N,1); % Construction of the trellis diagram for t=2:T for i=1:N temp=alpha(:,t-1)+A(:,i)+log10(normpdf(O(t),m(i),... sigma(i))); [alpha(i,t),ind]=max(temp); pred(i,t)=ind+sqrt(-1)*(t-1); end end [matching_prob,winner_ind]=max(alpha(:,T)); best_path=back_tracking(pred,winner_ind,T);
559
“11-Ch09-SA272” 18/9/2008 page 560
560
CHAPTER 9 Context-Dependent Classification
Computer Experiments 9.1 Two coins are used for a coin-tossing experiment, that is, coin A and coin B.The probability that coin A returns heads is 0.6, and the respective probability for coin B is 0.4. An individual standing behind a curtain decides which coin to toss as follows: the first coin to be tossed is always coin A, the probability that coin A is re-tossed is 0.4, and similarly, the probability that coin B is re-tossed is 0.6. An observer can only have access to the outcome of the experiment, that is, the sequence of heads and tails that is produced. (a) Model the experiment by means of a HMM (i.e., define the vector of the initial state probabilities, the transition matrix and the matrix of the emission probabilities) and (b) use the Baum_Welch_Do_HMM function to compute the HMM score for the sequence of observations {H, H, T , H, T , T } where H stands for heads and T stands for tails. Hint: In defining the input sequence of symbols O for Baum_Welch_Do_ HMM function, use “1” for “H” and “2” for “T”. 9.2 For the HMM of the previous experiment, use the Viterbi_Do_HMM function to find the best state sequence and respective path probability,for the following observation sequences: {H, T , T , T , H} and {T , T , T , H, H, H, H}. Hint: In defining the input sequence of symbols O for Viterbi_Do_HMM function, use “1” for “H” and “2” for “T”. 9.3 Assume that two number generators, Gaussian in nature, operate with mean values 0 and 5, respectively. The values for the respective standard deviations are 1 and 2. The following experiment is carried out behind a curtain: a person tosses a coin to decide which generator will be the first to emit a number. Heads has a probability of 0.4 and stands for generator A. Then the coin is tossed 8 times, and each time the coin decides which generator will emit the next number. An observer has only access to the outcome of the experiment, i.e., to the following sequence of numbers: {0.3, 0.4, 0.2, 2.1, 3.2, 5, 5.1, 5.2, 4.9}. (a) Model the experiment by means of a HMM that emits continuous observations and (b) use the Viterbi_Co_HMM function to compute the best-state sequence and the corresponding probability for the given sequence of numbers.
REFERENCES [Agaz 93] Agazi O.E., Kuo S.S. “Hidden Markov model based optical character recognition in the presence of deterministic transformations,” Pattern Recognition,Vol. 26, pp. 1813–1826, 1993. [Anto 97] Anton-Haro C., Fonollosa J.A.R., Fonollosa J.R. “Blind channel estimation and data detection using HMM,” IEEE Transactions on Signal Processing, Vol. 45(1), pp. 241–247, 1997.
“11-Ch09-SA272” 18/9/2008 page 561
References
[Aric 02] Arica N.,Yarman-Vural F.T. “Optical character recognition for cursive handwriting,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 24(6), pp. 801–813, 2003 [Arsl 99] Arslan L., Hansen J.H.L. “Selective training for hidden Markov models with applications to speech classification,” IEEE Transactions on Speech and Audio Processing,Vol. 7(1), pp. 46–54, 1999. [Bahl 86] Bahl L.R., Brown B.F., Desouza P.V. “Maximum mutual information estimation of hidden Markov model parameters for speech recognition,” Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing,Vol. 1, pp. 872–875, Japan, 1986. [Bake 75] Baker J. “The DRAGON system—an overview,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 23(1), pp. 24–29, 1975. [Baum 67] Baum L.E., Eagon J.A. “An inequality with applications to statistical prediction for functions of Markov processes and to a model for ecology,”Bulletin of the American Mathematical Society,Vol. 73, pp. 360–362, 1967. [Baum 70] Baum L.E., Petrie T., Soules G.,Weiss N. “A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains,”Annals of Mathematical Statistics, Vol. 41, pp. 164–171, 1970. [Baum 68] Baum L.E.,Sell G.R.“Growth functions for transformations of manifolds,”Pacific Journal of Mathematics,Vol. 27, pp. 211–227, 1968. [Bell 90] Bellegarda J.R., Nahamoo D. “Tied mixture continuous parameter modeling for speech recognition,” IEEE Transactions on Acoustics Speech and Signal Processing, Vol. 38(12), pp. 2033–2045, 1990. [Bell 94] Bello M.G. “A combined Markov random field and wave-packet approach to image segmentation,” IEEE Transactions on Image Processing,Vol. 3(6), pp. 834–847, 1994. [Besa 74] Besag J. “Spatial interaction and the statistical analysis of lattice systems,” J. Royal Stat. Soc. B,Vol. 36(2), pp. 192–236, 1974. [Bourl 94] Bourland H., Morgan N. Connectionist Speech Recognition. Kluwer Academic Publishers, 1994. [Bourl 90] Bourland H., Wellekens C.J. “Links between Markov models and the multilayer perceptrons,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12(12), pp. 1167–1178, 1990. [Chel 85] Chellapa R., Kashyap R.L. “Texture synthesis using 2-D noncausal autoregressive models,” IEEE Transactions on Acoustics Speech and Signal Processing,Vol. 33(1), pp. 194–203, 1985. [Chen 95a] Chen J.-L., Kundu A. “Unsupervised texture segmentation using multichannel decomposition and hidden Markov models,” IEEE Transactions on Image Processing, Vol. 4(5), pp. 603–620, 1995. [Chen 95] Chen M.Y., Kundu A., Srihari S.N. “Variable duration HMM and morphological segmentation for handwritten word recognition,” IEEE Transactions on Image Processing,Vol. 4(12), pp. 1675–1689, 1995. [Chie 03] Chien J.-T., Huang C.-H. “Bayesian learning of speech duration models,” IEEE Transactions on Speech and Audio Processing,Vol. 11(6), pp. 558–567, 2003. [Dell 93] Deller J., Proakis J., Hansen J. Discrete Time Processing of Speech Signals. Macmillan, 1993.
561
“11-Ch09-SA272” 18/9/2008 page 562
562
CHAPTER 9 Context-Dependent Classification
[Deng 94] Deng L.,Aksmanovic M.“Speaker-independent phonetic classification using HMM with mixtures of trend functions,” IEEE Transactions on Speech and Audio Processing, Vol. 5(4), pp. 319–324, 1997. [Deri 86] Derin H.“Segmentation of textured images using Gibb’s random fields,”ComputerVision, Graphics, and Image Processing,Vol. 35, pp. 72–98, 1986. [Diga 99] Digalakis V. “Online adaptation of hidden Markov models using incremental estimation algorithms,” IEEE Transactions on Speech and Audio Processing,Vol. 7(3), pp. 253–261, 1999. [Diga 95] Digalakis V., Rtischef D., Neumeyer L.G. “Speaker adaptation using constrained estimation of Gaussian mixtures,” IEEE Transaction on Speech and Audio Processing, Vol. 3(5), pp. 357–366, 1995. [Diga 96] Digalakis V., Monaco P., Murveit H. “Genones: Generalized mixture tying in continuous HMM model-based speech recognizers,” IEEE Transactions on Speech and Audio Processing, Vol. 4(4), pp. 281–289, 1996. [Diga 93] DigalakisV.,Rohlicek J.R.,Ostendorf M.“ML estimation of a stochastic linear system with the EM algorithm and its application to speech recognition,”IEEE Transactions on Speech and Audio Processing,Vol. 1(4), pp. 431–441, 1993. [ElYa 99] El-Yacoubi A., Gilloux M., Sabourin R., Suen C.Y. “An HHM-based approach for off-line unconstrained handwritten word modeling and recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 21(8), pp. 752–760, 1999. [Ephr 89] EphraimY.,DemboA.,Rabiner L.R.“A minimum discrimination information approach to hidden Markov modelling,”IEEE Transactions on Information Theory,Vol. 35,pp. 1001–1023, September 1989. [Ferg 80] Ferguson J. D. “Hiden Markov analysis: An introduction,” in Hidden Markov Models for Speech, Institute for Defence Analysis, Princeton university, 1980. [Fu 82] Fu K.S. Syntactic Pattern Recognition and Applications, Prentice Hall, 1982. [Gale 99] Gales M.J.F. “Semitied covariance matrices for hidden Markov models,” IEEE Transactions on Speech and Audio Processing,Vol. 7(3), pp. 272–281, 1999. [Gauv 94] Gauvain J.L., Lee C.H. “Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains,” IEEE Transactions on Speech and Audio Processing, Vol. 2(2), pp. 291–299, 1994. [Gema 84] Geman S., Geman D. “Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 6(6), pp. 721–741, 1984. [Geor 97] Georgoulakis C.,Theodoridis S.“Efficient clustering techniques for channel equalization in hostile environments,” Signal Processing,Vol. 58, pp. 153–164, 1997. [Geor 98] Georgoulakis C.,Theodoridis S. “Blind equalization for nonlinear channels via hidden Markov modeling,” Proceedings EUSIPCO-98, Rhodes, Greece, 1998. [Gold 99] Goldberger J., Burshtein D., Franco H. “Segmental modeling using a continuous mixture of noparametric models,” IEEE Transactions on Speech and Audio Processing, Vol. 7(3), pp. 262–271, 1999. [Gu 02] Gu L., Rose K. “Substate tying with combined parameter training and reduction in tied-mixture HMM design,” IEEE Transactions on Speech and Audio Processing, Vol. 10(3), 2002.
“11-Ch09-SA272” 18/9/2008 page 563
References
[Hans 82] Hansen F.R.,Elliot H.“Image segmentation using simple Markov field models,”Computer Graphics and Image Processing,Vol. 20, pp. 101–132, 1982. [He 08] He X., Deng L., Chou W. “Discriminative Learning in Sequential Pattern Recognition—A Unifying Review for Optimization-Oriented Speech Recognition,” to appear IEEE Signal Processing Magazine, september 2008. [Huo 95] Huo Q., Chan C., Lee C.H. “Bayesian adaptive learning of the parameters of hidden Markov model for speech recognition,” IEEE Transactions on Speech and Audio Processing, Vol. 3(5), pp. 334–345, 1995. [Huo 97] Huo Q., Lee C.H. “On-line adaptive learning of the continuous density HMM based on approximate recursive Bayes estimate,” IEEE Transactions on Speech and Audio Processing, Vol. 5(2), pp. 161–173, 1997. [ Jeli 76] Jelinek F. “Continuous speech recognition by statistical methods,” Proceedings of the IEEE,Vol. 64(4), pp. 532–555, 1976. [ Juan 85] Juang B.H. “Maximum likelihood estimation for mixture multivariate stochastic observations of Markov chains,” AT&T System Technical Journal, Vol. 64, pp. 1235–1249, July–August 1985. [ Juan 97] Juang B.H., Chou W., Lee C.H. “Minimum classification error rate methods for speech recognition,” IEEE Transactions on Speech and Audio Processing, Vol. 5(3), pp. 257–266, 1997. [ Juan 92] Juang B.H., Katagiri S. “Discriminative learning for minimum error classification,” IEEE Transactions on Signal Processing,Vol. 40(12), pp. 3043–3054, 1992. [ Juan 86] Juang B.H., Levinson S.E., Sondhi M.M. “Maximum likelihood estimation for multivariate mixture observations of Markov chains,” IEEE Transactions on Information Theory, Vol. IT-32, pp. 307–309, March 1986. [Kale 94] Kaleh G.K., Vallet R. “Joint parameter estimation and symbol detection for linear and nonlinear unknown channels,” IEEE Transactions on Communications, Vol. 42(7), pp. 2406–2414, 1994. [Kim 95] Kim N.S., Un C.K. “On estimating robust probability distribution in HMM-based speech recognition,”IEEE Transactions on Speech and Audio Processing,Vol. 3(4),pp. 279–286,1995. [Koni 96] Konig Y. “REMAP: Recursive estimation and maximization of a-posteriori probabilities in transition-based speech recognition,”Ph.D. thesis, University of California at Berkeley, 1996. [Kops 03] Kopsinis Y.,Theodoridis S. “An efficient low-complexity technique for MLSE equalizers for linear and nonlinear channels,” IEEE Transactions on Signal Processing, Vol. 51(12), pp. 3236–3249, 2003. [Kris 97] Krishnamachari S.,Chellappa R.“Multiresolution Gauss–Markov random field models for texture segmentation,” IEEE Transactions on Image Processing,Vol. 6(2), pp. 251–268, 1997. [Laks 93] Lakshmanan S., Derin H. “Gaussian Markov random fields at multiple resolutions,” in Markov Random Fields: Theory and Applications (R. Chellappa, ed.), Academic Press, 1993. [Lee 72] Lee C.H., Fu K.S. “A stochastic syntax analysis procedure and its application to pattern recognition,” IEEE Transactions on Computers,Vol. 21, pp. 660–666, 1972. [Lee 91] Lee C.H., Lin C.H., Juang B.H. “A study on speaker adaptation of the parameters of continuous density hidden Markov models,” IEEE Transactions on Signal Processing, Vol. 39(4), pp. 806–815, 1991.
563
“11-Ch09-SA272” 18/9/2008 page 564
564
CHAPTER 9 Context-Dependent Classification
[Legg 95] Leggetter C.J., Woodland P.C. “Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models,” Comput. Speech Lang., Vol. 9, pp. 171–185, 1995. [Levi 86] Levinson S.E. “Continuously variable duration HMMs for automatic speech recognition,” Computer Speech and Language,Vol. 1, pp. 29–45, March 1986. [Levi 83] Levinson S.E., Rabiner L.R., Sondhi M.M. “An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition,” Bell System Technical Journal,Vol. 62(4), pp. 1035–1074,April 1983. [Li 04] Li J., Wang J., Zhao Y., Yang Z. “Self adaptive design of hidden Markov models,” Pattern Recognition Letters,Vol. 25, pp. 197–210, 2004. [Li 00] Li X., Parizeau M., Plamondon R. “Training hidden Markov models with multiple observations-A combinatorial method,” IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 22(4), pp. 371–377, 2000. [Lipo 82] Liporace L.A. “Maximum likelihood estimation for multivariate observations of Markov sources,” IEEE Transactions on Information Theory,Vol. IT-28(5), pp. 729–734, 1982. [Moon 96] Moon T. “The expectation maximization algorithm,” Signal Processing Magazine, Vol. 13(6), pp. 47–60, 1996. [Morg 95] Morgan N. Boulard H. “Continuous speech recognition,” Signal Processing Magazine, Vol. 12(3), pp. 25–42, 1995. [Neap 04] Neapolitan R.D. Learning Bayesian Networks, Prentice Hall, Gliffs, N.J. 2004. [Oste 96] Ostendorf M., Digalakis V., Kimball O. “From HMM’s to segment models: A unified view of stochastic modeling for speech,” IEEE Transactions on Audio and Speech Processing, Vol. 4(5), pp. 360–378, 1996. [Papo 91] Papoulis A. Probability Random Variables and Stochastic Processes, 3rd ed., McGraw-Hill 1991. [Pikr 06] Pikrakis A., Theodoridis S., Kamarotos D. “Classification of musical patterns using variable duration hidden Markov models,” IEEE Transactions on Speech and Audio Processing, Vol. 14(5), pp. 1795–1807, 2006. [Pikr06a] Pikrakis A., Gaunakopoulos T., Theodoridis S. “Speech/music discrimination for radio broadcasts using a hybrid HMM–Bayesiay network architecture,” Proceedings, EUSIPCOFlorence, 2006. [Pori 82] Poritz A.B. “Linear predictive HMM and the speech signal,” Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pp. 1291–1294, Paris, 1982. [Povl 95] Povlow B., Dunn S.“Texture classification using noncausal hidden Markov models,”IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 17(10),pp. 1010–1014,1995. [Proa 89] Proakis J. Digital Communications, 2nd ed., McGraw-Hill, 1989. [Rabi 89] Rabiner L. “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of IEEE,Vol. 77, pp. 257–285, February, 1989. [Rabi 93] Rabiner L., Juang B.H. Fundamentals of Speech Recognition, Prentice Hall, 1993. [Ramd 03] Ramdane S.,Taconet B., Zahour A. “Classification of forms with handwritten fields by planar Markov models,” Pattern Recognition,Vol. 36, pp. 1045–1060, 2003. [Rao 01] Rao A.V., Rose K. “Deterministically annealed design of hidden Markov Model speech recognizers,”IEEE Transactions on Speech and Audio Precessing,Vol. 9(2),pp. 111–127,2001.
“11-Ch09-SA272” 18/9/2008 page 565
References
[Russ 97] Russell M., Holmes W. “Linear trajectory segmental HMM’s,” IEEE Signal Processing Letters,Vol. 4(3), pp. 72–75, 1997. [Russ 85] Russell M.J., Moore R.K. “Explicit modeling of state occupancy in HMMs for automatic speech recognition,” Proceedings of the International Conference on Acoustics, Speech and Signal Processing,Vol. 1, pp. 5–8, 1985. [Spec 94] Special issue on neural networks for speech in IEEE Transactions on Speech and Audio Processing,Vol. 2(1), 1994. [Theo 95] Theodoridis S., Cowan C.F.N., See C.M.S. “Schemes for equalization of communication channels with nonlinear impairments,” IEE Proceedings on Communications, Vol. 142(3), pp. 165–171, 1995. [Turi 98] Turin W. “Unidirectional and parallel Baum-Welch algorithms,” IEEE Transactions on Speech and Audio Processing,Vol. 6(6), pp. 516–523, 1998. [Vite 67] Viterbi A.J. “Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,” IEEE Transactions on Information Theory,Vol. 13, pp. 260–269, 1967. [Vlon 92] Vlontzos J.A., Kung S.Y. “Hidden Markov models for character recognition,” IEEE Transactions on Image Processing,Vol. 1(4), pp. 539–543, 1992. [Wang 01] Wang S., Zhao Y. “Online Bayesian tree-structured transformation of HMM’s with optimal model selection for speaker adaptation,” IEEE Transactions on Speech and Audio Processing,Vol. 9(6), pp. 663–677, 2001. [Wu 96] WuW.R.,Wei S.C.“Rotational and gray scale transform invariant texture classification using spiral resampling, subband decomposition, and hidden Markov model,” IEEE Transactions on Image Processing,Vol. 5(10), pp. 1423–1435, 1996. [Zhan 94] Zhang J., Modestino J.W., Langan D.A. “Maximum likelihood parameter estimation for unsupervised stochastic model based image segmentation,” IEEE Transactions on Image Processing,Vol. 3(4), pp. 404–421, 1994.
565
“12-Ch10-SA272” 18/9/2008 page 567
CHAPTER
Supervised Learning: The Epilogue
10
10.1 INTRODUCTION This chapter is the last one related to supervised learning, and it is intended to serve three purposes. The first sections focus on the last stage of the design procedure of a classification system. In other words, we assume that an optimal classifier has been designed, based on a selected set of training feature vectors. Our goal now is to evaluate its performance with respect to the probability of classification error associated with the designed system. To this end, methodologies will be developed for the estimation of the classification error probability, using the available, hence finite, set of data. Once the estimated error is considered satisfactory, full evaluation of the system performance is carried out in the real environment for which the system has been designed, such as a hospital for a medical diagnosis system or a factory for an industrial production–oriented system. It is important to note that the evaluation stage is not cut off from the previous stages of the design procedure. On the contrary, it is an integral part of the procedure. The evaluation of the system’s performance will determine whether the designed system complies with the requirements imposed by the specific application and intended use of the system. If this is not the case, the designer may have to reconsider and redesign parts of the system. Furthermore, the misclassification probability can also be used as a performance index, in the feature selection stage, to choose the best features associated with a specific classifier. The second goal of this chapter is to tie together the various design stages that have been considered separately, so far, in the context of a case study coming from medical ultrasound imaging. Our purpose is to help the reader to get a better feeling, via an example, on how a classification system is built by combining the various design stages. Techniques for feature generation, feature selection, classifier design and system evaluation will be mobilized in order to develop a realistic computeraided diagnosis medical system to assist a doctor reaching a decision. In the final sections of the chapter, we will move away from the fully supervised nature of the problem that we have considered so far in the book, and we will allow unlabeled data to enter the scene. As we will see, in certain cases, unlabeled
567
“12-Ch10-SA272” 18/9/2008 page 568
568
CHAPTER 10 Supervised Learning: The Epilogue
data can offer additional information that can help the designer in cases where the number of labeled data is limited. Semi-supervised learning is gaining in importance in recent years, and it is currently among the hottest research areas. The aim of this chapter is to introduce the reader to the semi-supervised learning basics and to indicate the possible performance improvement that unlabeled data may offer if used properly.
10.2 ERROR-COUNTING APPROACH Let us consider an M-class classification task. Our objective is to estimate the classification error probability by testing the“correct/false”response of an independently designed classifier using a finite set of N test feature vectors. Let Ni be the vectors in each class, with M i⫽1 Ni ⫽ N and Pi the corresponding error probability for class i . Assuming independence among the feature vectors, the probability of ki vectors from class i being misclassified is given by the binomial distribution Ni Piki (1 ⫺ Pi )Ni ⫺ki prob{ki misclassified } ⫽ ki
(10.1)
In our case the probabilities Pi are not known. An estimate Pˆ i results if we maximize (10.1) with respect to Pi . Differentiating and equating to zero result in our familiar estimate Pˆ i ⫽
ki Ni
(10.2)
Thus, the total error probability estimate is given by Pˆ ⫽
M i⫽1
P(i )
ki Ni
(10.3)
where P(i ) is the occurrence probability of class i . We will now show that Pˆ is an unbiased estimate of the true error probability. Indeed, from the properties of the binomial distribution (Problem 10.1) we have E[ki ] ⫽ Ni Pi
(10.4)
which leads to ˆ ⫽ E[P]
M
P(i )Pi ≡ P
(10.5)
i⫽1
that is, the true error probability. To compute the respective variance of the estimator, we recall from Problem 10.1 that k2i ⫽ Ni Pi (1 ⫺ Pi )
(10.6)
“12-Ch10-SA272” 18/9/2008 page 569
10.3 Exploiting the Finite Size of the Data Set
leading to P2ˆ ⫽
M
P 2 (i )
i⫽1
Pi (1 ⫺ Pi ) Ni
(10.7)
Thus, the error probability estimator in (10.3), which results from simply counting the errors, is unbiased but only asymptotically consistent as Ni → ⬁. Thus, if small data sets are used for testing the performance of a classifier, the resulting estimate may not be reliable. In [Guyo 98] the minimum size of the test data set, N , is derived in terms of the true error probability P of the already designed classifier. The goal is to estimate N so that to guarantee, with probability 1 ⫺ a, 0 ⱕ a ⱕ 1, that P does not exceed the ˆ by an amount larger than ⑀(N , a), that is estimated from the test set, P, prob{P ≥ Pˆ ⫹ ⑀(N , a)} ⱕ a
(10.8)
Let ⑀(N , a) be expressed as a function of P, that is, ⑀(N , a) ⫽ P. An analytical solution for Eq. (10.8) with respect to N is not possible. However, after some approximations certain bounds can be derived. For our purposes, it suffices to consider a simplified formula, which is valid for typical values of a and  (a ⫽ 0.05,  ⫽ 0.2), N≈
100 P
(10.9)
In words, if we want to guarantee, with a risk a of being wrong, that the error Pˆ probability P will not exceed 1⫺ , then N must be of the order given in Eq. (10.9). For P ⫽ 0.01, N ⫽ 10,000 and for P ⫽ 0.03, N ⫽ 3000. Note that this result is independent of the number of classes. Furthermore, if the samples in the test data set are not independent, this number must be further increased. Such bounds are also of particular importance, if the objective is to determine the size N of the test data set that provides good confidence in the results, when comparing different classification systems with relatively small differences in their error probabilities. Although the error-counting approach is by far the most popular one, other techniques have also been suggested in the literature. These techniques estimate the error probability by using smoother versions of the discriminant function(s) realized by the classifier. The error-counting approach can be thought of as an extreme case of a hard limiter,where a 1 or 0 is produced and counted,depending on the discriminant function’s response, that is, whether it is false or true, respectively. See, for example, [Raud 91, Brag 04].
10.3 EXPLOITING THE FINITE SIZE OF THE DATA SET The estimation of the classification error probability presupposes that one has decided upon the data set to which the error counting will be applied. This is
569
“12-Ch10-SA272” 18/9/2008 page 570
570
CHAPTER 10 Supervised Learning: The Epilogue
not a straightforward task. The set of samples that we have at our disposal is finite, and it has to be utilized for both training and testing. Can we use the same samples for training and testing? If not, what are the alternatives? Depending on the answer to the question, the following methods have been suggested: ■
Resubstitution Method: The same data set is used, first for training and then for testing. One need not go into mathematical details in order to see that such a procedure is not very fair. Indeed, this is justified by the mathematical analysis. In [Fole 72] the performance of this method was analyzed using normal distributions. The analysis results show that this method provides an optimistic estimate of the true error probability. The amount of bias of the resubstitution estimate is a function of the ratio N/l,that is,the data set size and the dimension of the feature space. Furthermore, the variance of the estimate is inversely proportional to the data set size N . In words, in order to obtain a reasonably good estimate, N as well as the ratio N/l must be large enough. The results from the analysis and the related simulations show that N/l should be at least three and that an upper bound of the variance is 1/8N . Of course,if this technique is to be used in practice, where the assumptions of the analysis are not valid, experience suggests that the suggested ratio must be even larger [Kana 74]. Once more, the larger the ratio N/l, the more comfortable one feels.
■
Holdout Method: The available data set is divided into two subsets, one for training and one for testing. The major drawback of this technique is that it reduces the size for both the training and the testing data. Another problem is to decide how many of the N available data will be allocated to the training set and how many to the test set. This is an important issue. In Section 3.5.3 of Chapter 3, we saw that designing a classifier using a finite data set introduces an excess mean error and a variance around it, as different data sets, of the same size, are used for the design. Both of these quantities depend on the size of the training set. In [Raud 91], it is shown that the classification error probability of a classifier, designed using a finite training data set, N , is always higher than the corresponding asymptotic error probability (N → ⬁). This excess error decreases as N increases. On the other hand, in our discussion in the previous section we saw that the variance of the error counting depends on the size of the test set, and for small test data sets the estimates can be unreliable. Efforts made to optimize the respective sizes of the two sets have not yet led to practical results.
■
Leave-One-Out Method: This method [Lach 68] alleviates the lack of independence between the training and test sets in the resubstitution method and at the same time frees itself from the dilemma associated with the holdout method. The training is performed using N ⫺1 samples, and the test is carried out using the excluded sample. If this is misclassified, an error is counted. This is repeated N times, each time excluding a different sample. The total
“12-Ch10-SA272” 18/9/2008 page 571
10.3 Exploiting the Finite Size of the Data Set
number of errors leads to the estimation of the classification error probability. Thus, training is achieved using, basically, all samples, and at the same time independence between training and test sets is maintained. The major disadvantage of the technique is its high computational complexity. For certain types of classifiers (i.e., linear or quadratic) it turns out that a simple relation exists between the leave-one-out and the resubstitution method ([Fuku 90], Problem 10.2). Thus, in such cases the former estimate is obtained using the latter method with some computationally simple modifications. The estimates resulting from the holdout and leave-one-out methods turn out to be very similar, for comparable sizes of the test and training sets. Furthermore, it can be shown (Problem 10.3, [Fuku 90]) that the holdout error estimate, for a Bayesian classifier, is an upper bound of the true Bayesian error. In contrast, the resubstitution error estimate is a lower bound of the Bayesian error, confirming our previous comment that it is an optimistic estimate. To gain further insight into these estimates and their relation, let us make the following definitions: ■
PeN denotes the classification error probability for a classifier designed using a finite set of N training samples.
■
P¯ eN denotes the average E[PeN ] over all possible training sets of size N .
■
Pe is the average asymptotic error as N →⬁.
It turns out that the holdout and leave-one-out methods (for statistically independent samples) provide an unbiased estimate of P¯ eN . In contrast, the resubstitution method provides a biased (underestimated) estimate of P¯ eN . Figure 10.1 shows the trend of a typical plot of P¯ eN and the average (over all possible sets of size N ) resubstitution error as functions of N [Fole 72, Raud 91]. It is readily observed that as the data size N increases, both curves tend to approach the asymptotic Pe .
P Leave-one-out
N
Pe Resubstitution
FIGURE 10.1 Plots indicating the general trend of the average resubstitution and leave-one-out error probabilities as functions of the number of training points.
571
“12-Ch10-SA272” 18/9/2008 page 572
572
CHAPTER 10 Supervised Learning: The Epilogue
A number of variations and combinations of these basic schemes have also been suggested in the literature. For example, a variation of the leave-one-out method is to leave k ⬎ 1, instead of one, samples out. The design and test process is repeated for all distinct choices of k samples. References [Kana 74, Raud 91] are two good examples of works discussing various aspects of the topic. In [Leis 98] a method called cross-validation with active pattern selection is proposed, with the goal of reducing the high computational burden required by the leave-one-out method. It is suggested not to leave out (one at a time) all N feature vectors, but only k ⬍ N . To this end the “good” points of the data set (expected to contribute a 0 to the error) are not tested. Only the k “worst” points are considered. The choice between “good” and “bad” is based on the respective values of the cost function after an initial training. This method exploits the fact that the outputs of the classifier, trained according to the least squares cost function, approximate posterior probabilities, as discussed in Chapter 3. Thus, those feature vectors whose outputs have a large deviation from the desired value (for the true class) are expected to be the ones that contribute to the classification error. Another set of techniques have been developed around the bootstrap method [Efro 79, Hand 86, Jain 87]. A major incentive for the development of these techniques is the variance of the leave-one-out method estimate for small data sets [Efro 83]. According to the “bootstrap” philosophy, new data sets are artificially generated. This is a way to overcome the limited number of available data and create more data in order to better assess the statistical properties of an estimator. Let X be the set of the available data of size N . A bootstrap design sample set of size N, X ∗ , is formed by random sampling with replacement of the set X. Replacement means that when a sample, say x i , is “copied” to the set X ∗ , it is not removed from X but is reconsidered in the next sampling. A number of variants have been built upon the bootstrap method. A straightforward one is to design the classifier using a bootstrap sample set and count the errors using the samples from X that do not appear in this bootstrap sample set. This is repeated for different bootstrap sample sets. The error rate estimate, e0 , is computed by counting all the errors and dividing the sum by the total number of test samples used. However, in [Raud 91] it is pointed out that the bootstrap techniques improve on the leave-one-out method only when the classification error is large. Another direction is to combine estimates from different estimators. For example, in the so-called 0.632 estimator ([Efro 83]), the error estimate is taken as a convex combination of the resubstitution error, eres , and the bootstrap error e0 , e0.632 ⫽ 0.368eres ⫹ 0.632e0
It has been reported that the 0.632 estimator is particularly effective in cases of small size data sets [Brag 04]. An extension of the 0.632 rule is discussed in [Sima 06] where convex combinations of different estimators are considered and the combining weights are computed via an optimization process.
“12-Ch10-SA272” 18/9/2008 page 573
10.4 A Case Study from Medical Imaging
Confusion Matrix, Recall and Precision In evaluating the performance of a classification system, the probability of error is sometimes not the only quantity that assesses its performance sufficiently. Let us take for example, an M-class classification task. An important issue is to know whether there are classes that exhibit a higher tendency for confusion. The confusion matrix A ⫽ [A(i, j)] is defined so that its element A(i, j) is the number of data points whose true class label was i and were classified to class j. From A, one can directly extract the recall and precision values for each class, along with the overall accuracy: ■
Recall (Ri ). Ri is the percentage of data points with true class label i, which were correctly classified in that class. For example, for a two-class problem, A(1,1) . the recall of the first class is calculated as R1 ⫽ A(1,1)⫹A(1,2)
■
Precision (Pi ). Pi is the percentage of data points classified as class i, whose true class label is indeed i. Therefore, for the first class in a two-class problem, A(1,1) . P1 ⫽ A(1,1)⫹A(2,1)
■
Overall Accuracy (Ac). The overall accuracy, Ac, is the percentage of data that has been correctly classified. Given an M-class problem, Acis computed from the confusion matrix according to the equation Ac ⫽ N1 M i⫽1 A(i, i), where N is the total number of points in the test set.
Take as an example a two-class problem where the test set consists of 130 points from class 1 and 150 points from class 2 . The designed classifier classifies 110 points from 1 correctly and 20 points to class 2 . Also, it classifies 120 points from class 2 correctly and 30 points to class 1 . The confusion matrix for this case is A⫽
110 30
20 120
110 The recall for the first class is R1 ⫽ 110 130 and the precision P1 ⫽ 140 . The respective values for the second class are similarly computed. The accuracy is Ac ⫽ 110⫹120 130⫹150 .
10.4 A CASE STUDY FROM MEDICAL IMAGING Our goal in this section is to demonstrate the various design stages, discussed in the previous chapters, via a case study borrowed from a real application. It will not come as a surprise to say that focusing on a single example cannot cover all possible design approaches that are followed in practice. However, our aim is to provide a flavor for the newcomer. After all,“perfection is the enemy of the good.” Our chosen application comes from the medical imaging discipline. Our task is to develop a pattern recognition system for the diagnosis of certain liver diseases. Specifically, the system will be presented with ultrasound images of the
573
“12-Ch10-SA272” 18/9/2008 page 574
574
CHAPTER 10 Supervised Learning: The Epilogue
(a)
(b)
(c)
FIGURE 10.2 Ultrasound images corresponding to (a) normal liver, (b) liver with fatty infiltration, and (c) liver with cirrhosis. The square shows the image area on which the analysis was carried out.
liver, and it must be able to recognize normal from abnormal cases. Abnormal cases correspond to two types of liver diseases, namely, cirrhosis and fatty liver infiltration. For each case, two different gratings must be recognized, depending on the degree of the disease development [Cavo 97]. Figure 10.2 shows three examples of ultrasound images corresponding to (a) a normal liver, (b) an abnormal liver with fatty infiltration, and (c) an abnormal liver with cirrhosis. It is readily realized that the visual differences between the images are not great. This makes the clinical diagnosis and the diagnostic accuracy very much dependent on the skill of the doctor. Thus, the development of a pattern recognition system can assist the doctor in assessing the case and, together with other clinical findings, reduce the need for invasive techniques (biopsy). The first stage in the design process involves the close cooperation of the system designer with the specialist, that is, the doctor, in order to establish a “common language” and have the designer understand the task and define, in common with the doctor, the goals and requirements of the pattern recognition system. Besides the acceptable error rate, other performance issues come into play, such as complexity, computational time, and cost of the system. The next stage involves various image processing steps, such as image enhancement, in order to assist the system by presenting it only useful information as much as possible. Then things are ripe to begin with the design of the pattern recognition system. Figure 10.3 outlines the task. There are five possible classes. The pattern recognition system can be designed either around a single classifier, which assigns an unknown image directly to one of the five classes, or around a number of classifiers built on a tree structure philosophy. The latter approach was adopted here. Figure 10.4 illustrates the procedure. A separate classifier was used at each node, and each of them performs a two-class decision. At the first node, the respective classifier decides between normal and abnormal cases. At the second node, images, classified as abnormal, are tested and classified in either the cirrhosis or the fatty liver infiltration class, and so on. The advantage of such a procedure is that we break the problem into a number of simpler ones. It must be stressed, however, that in
“12-Ch10-SA272” 18/9/2008 page 575
10.4 A Case Study from Medical Imaging
liver
fatty liver infiltration
normal
grating A
cirrhosis
grating B
grating A
grating B
FIGURE 10.3 The classification task.
liver 1
normal
abnormal 2
fatty liver infiltration
cirrhosis 4
3
A
B
A
B
FIGURE 10.4 A tree-structured hierarchy of classifiers.
other applications such a procedure may not be applicable. For the design of the classification system,150 ultrasound liver images were obtained from a medical center. Fifty of them correspond to normal cases, 55 of them to patients suffering from cirrhosis, and 45 of them to patients suffering from fatty liver infiltration. Three classifiers were adopted for comparison, namely the least squares linear classifier, the minimum Euclidean distance classifier, and the kNN for different values of k. Each time, the same type of classifier was used for all nodes. From the discussions
575
“12-Ch10-SA272” 18/9/2008 page 576
576
CHAPTER 10 Supervised Learning: The Epilogue
with the specialists, we concluded that what was of interest here was the texture of the respective images. The methods described in Section 7.2.1 of Chapter 7 were used, and a total of 38 features were generated for each of the images. This is a large number,and a feature selection procedure was“mobilized”to reduce this number. Let us first concentrate on the first node classification task and the LS linear classifier. ■
For each of the 38 features the t-test was applied, and only 19 of them passed the test at a significance level of 0.001. The latter is chosen so that “enough” features pass the test. Taking into account the size of the problem, enough was considered to be around 15 for our problem. However, 19 is still a large number, and a further reduction was considered necessary. For example, 19 is of the same order as 50 (the number of normal patterns), which would lead to poor generalization.
■
The 19 features were considered in pairs, in triples, up to groups of seven, in all possible combinations. For each combination, the optimal LS classifier was designed and each time the corresponding classification error rate was estimated, using the leave-one-out method. It turned out that taking the features in groups larger than two did not improve the error rate significantly. Thus, it was decided that l ⫽ 2 was satisfactory and that the best combination consisted of the kurtosis and theASM.The percentage of correct classification with this combination was 92.5%.
For the design of the linear classifier of “node 2” the same procedure was followed, using, of course, only the images originating from abnormal cases. Of the 38 originally produced features, only 15 passed the t-test. The optimal combination of features turned out to be the mean,the variance,and the correlation. It may be worth pointing out that the variance was rejected from the t-test during the design of the “node 1” classifier. The percentage of correct classification for node 2 was 90.1%. The optimal combination for the “node 3” LS classifier was the variance, entropy, the sum entropy, and the difference entropy corresponding to a correct classification rate of 92.2%. Finally, the optimization procedure for the “node 4” classifier resulted in the mean value, the ASM, and the contrast with a correct classification rate estimate of 83.8%. Having completed the design with the LS linear classifiers, the same procedure was followed for the Euclidean minimum distance classifier and the kNN classifier. However,in both of these cases the resulting error rate estimates were always higher than the ones obtained with the LS classifier. Thus,the latter one was finally adopted. Once more, it must be stated that this case study does not and cannot represent the wealth of classification tasks encountered in practice, each with its own specific requirements. We could state, with a touch of exaggeration, of course, that each classification task is like a human being. Each one has its own personality! For example, the dimension of our problem was such that it was computationally feasible, with today’s technology to follow the procedure described. The feature
“12-Ch10-SA272” 18/9/2008 page 577
10.5 Semi-Supervised Learning
selection, classifier design, and classification error stages were combined to compute the best combination. This was also a motivation for choosing the specific case study, that is, to demonstrate that the various stages in the design of a classification system are not independent but they can be closely interdependent. However, this may not be possible for a large number of tasks, as, for example, the case of a large multilayer neural network in a high-dimensional feature space. Then the feature selection stage cannot be easily integrated with that of classifier design, and techniques such as those presented in Chapter 5 must be employed. Ideally, what one should aim at is to have a procedure to design the classifiers by minimizing the error probability directly (not the LS, etc.), and at the same time this procedure should be computationally simple (!) to allow also for a search for the optimal feature combination. However, this “utopia” is still quite distant.
10.5 SEMI-SUPERVISED LEARNING All the methods that we have considered in the book so far have relied on using a set of labeled data for the training of an adopted model structure (classifier). The final goal was, always, to design a “machine,” which, after the training phase, can predict reliably the labels of unseen points. In other words, the scope was to develop a general rule based on the inductive inference rationale. In such a perspective, the generalization performance of the designed classifier was a key issue that has “haunted” every design methodology. In this section, we are going to relax the design procedure from both “pillars” on which all our methods were so far built; (a) the labeled data set used for the training and (b) our concern about the generalization performance of the developed classifier. Initially, unlabeled data will be brought into the game, and we will investigate the possibility of whether this extra information, in conjunction with the labeled data, can offer performance improvement. Moving on, in a later stage, we will consider cases where the classifier design is not focused on predicting the labels of “future”unseen data points. In contrast, the optimization of a loss function will entirely rely on best serving the needs of a given set of unlabeled data, which are at the designer’s disposal “now”, that is, at the time of the design. The latter concept of designing a classifier is known as transductive inference to contrast it to the inductive inference mentioned earlier. Designing classifiers by exploiting information that resides in both labeled and unlabeled data springs from a fact of life; that is, in many real applications collecting unlabeled data is much easier than the task of labeling them. In a number of cases, the task of labeling is time consuming, and it requires annotation by an expert. Bioinformatics is a field in which unlabeled data is abundant, yet only a relatively small percentage is labeled, as, for example in protein classification tasks. Text classification is another area where unlabeled data is fairly easy to collect while the labeling task requires the involvement of an expert. Annotating music is also a very
577
“12-Ch10-SA272” 18/9/2008 page 578
578
CHAPTER 10 Supervised Learning: The Epilogue
demanding task, which, in addition, involves a high degree of subjectivity, as, for example, in deciding the genre of a music piece. On the other hand, it is very easy to obtain unlabeled data. Figures 10.5 and 10.6,inspired by [Sind 06],present two simple examples raising expectations that performance may be boosted by exploiting additional information that resides in an unlabeled data set. In Figure 10.5a we are given two labeled points and one, denoted by “?”, whose class is unknown. Based on this limited information, one will readily think that the most sensible decision is to classify the unknown point to the “*” class. In Figure 10.5b, in addition to the previous three points, a set of unlabeled points is shown. Having this more complete picture, one will definitely be tempted to reconsider the previous decision. In this case,
+
+ ?
*
?
(a)
*
(b)
FIGURE 10.5 (a) The unknown point, denoted by “?”, is classified in the same class as point “∗”. (b) The setup after a number of unlabeled data have been provided, which leads us to reconsider our previous classification decision.
?
+
*
?
*
+
(a)
(b)
FIGURE 10.6 (a) The unknown point, denoted by “?”, is classified in the same class as point “∗”. (b) The setup after a number of unlabeled data have been provided. The latter forces us, again, to reconsider our previous classification decision.
“12-Ch10-SA272” 18/9/2008 page 579
10.5 Semi-Supervised Learning
the extra information unveiled by the unlabeled data, and used by our perceptive mechanism, is the clustered structure of the data set. Figure 10.6 provides us with a slightly different viewpoint. Once more, we are given the three points, and the same decision as before is drawn (Figure 10.6a). In Figure 10.6b, the unlabeled data reveal a manifold structure on which the points reside (see also Section 6.6). The extra piece of information, which is disclosed to us now, is that the unknown point is closer to the “⫹” than to the “∗” point, if the geodesic, instead of the Euclidean, distance is used. Of course, in both cases, reconsideration of our initial decision is justified only under the following assumptions: ■
Cluster assumption: If two points are in the same cluster, they are likely to originate from the same class.
■
Manifold assumption: If the marginal probability distribution, p(x), is supported on a manifold, then points lying close to each other on the manifold are more likely to be in the same class. Another way to express this is that the conditional probability, P(y|x), of the class label y, is a smooth function of x, with respect to the underlying structure of the manifold.
Figure 10.5 illustrates the cluster assumption, and Figure 10.6 the manifold one. Both assumptions can be seen as particular instances of a more general assumption that covers both classification and regression: ■
Semi-supervised assumption: If two points are close in a high-density region, then their corresponding outputs should have close values.
In other words, closeness between points is not a decisive factor, if, considered by itself. It has to be considered in the context of the underlying data distribution. This is apparent from the previous two figures. According to the semi-supervised smoothness assumption, if two points are close and linked by a path through a highdensity area, they are likely to give closely located outputs. On the other hand, if the path that links them goes through a low-density region, there is no need for the corresponding outputs to be close ([Chap 06a, p. 5]). Although semi-supervised learning has attracted a lot of interest recently, it is not new as a problem. Semi-supervised learning has been addressed in the statistics community as early as the mid-1960s, for example, [Scud 65]. Transductive learning was introduced by Vapnik and Chervonenkis in the mid-1970s, [Vapn 74]. Over the years, a large number of approaches and algorithms have been proposed, and it is beyond the scope of the present section to cover the area in depth and in its entire breadth. Our goal is to present some of the basic directions that are currently popular and at the same time are based on methods previously addressed in this book.
10.5.1 Generative Models Generative models are perhaps the oldest semi-supervised methods and they have been used in statistics for many years. The essence behind these methods is to
579
“12-Ch10-SA272” 18/9/2008 page 580
580
CHAPTER 10 Supervised Learning: The Epilogue
model the class-conditional densities p(x|y), using information provided by both labeled and unlabeled data. Once such a model is available, one can compute the marginal distribution p(x) p(x) ⫽
P( y)p(x| y)
(10.10)
p( y, x) ⫽ P( y)p(x| y)
(10.11)
y
the joint distribution
and finally the quantity that is required by the Bayesian classifier P( y)p(x| y) P( y|x) ⫽ y P( y)p(x| y)
(10.12)
The class label is an integer, y ∈ {1, 2, . . . , M}, where M is the number of classes. If P(y) is not known then an estimate of it is used. The above formulas are familiar to us from Chapter 2, but they are repeated here for the sake of completeness. In the sequel, we adopt a parametric model for the class conditional densities, that is, p(x| y; ). Let also Py , y ⫽ 1, 2, . . . , M, denote the respective estimates of the class priors P(y). Assume that we are given two types of data sets: ■
■
Unlabeled data: This data set consists of Nu samples x i ∈ Rl , i ⫽ 1, 2, . . . , Nu , which are assumed to be independently and identically distributed random vectors drawn from the marginal distribution p(x; , P ), which is also parameterized in terms of , and P ⫽ [P1 , P2 , . . . , PM ]T . The corresponding set is denoted by Du . Labeled data: We assume that Nl samples are randomly and independently generated and they are subsequently labeled by an expert. Let Ny of them be associated with class y ⫽ 1, 2, . . . , M, where Nl ⫽ y Ny . We adopt the notation z iy , i ⫽ 1, 2, . . . , Ny , y ⫽ 1, 2, . . . , M, to represent the ith sample assigned in the yth class. The set of labeled samples is denoted as Dl ⫽{z iy , i ⫽ 1, 2, . . . , Ny , y ⫽ 1, 2, . . . , M}. This type of labeling data matches best a number of practical applications. For example, in medical imaging an expert is given a set of images,which have been previously produced,and labels them accordingly. Other “mechanisms” of generating labeled data are also possible, by adopting different assumptions, see [Redn 84, Shas 94, Mill 97, Mill 03].
Our task now is to estimate the set of the unknown parameters, that is, Q ≡ [T , P T ]T in the mixture model (see Section 2.5.5 of Chapter 2.) p(x; Q) ⫽
M
Py p(x|y; )
(10.13)
y⫽1
using the observations in Du and Dl . For simplicity, in the previous mixture model we have assumed one mixture component per class. This can be relaxed,
“12-Ch10-SA272” 18/9/2008 page 581
10.5 Semi-Supervised Learning
for example, [Mill 97]. If only Du was available, then the task would reduce to the mixture modeling task with hidden class (mixture) labels, as discussed in Section 2.5.5. It is known from statistics and it is readily deduced from the definition of the log-likelihood in Section 2.5, that if the set of observations is the union of two independent sets then the log-likelihood is the sum of the log-likelihoods of the respective sets. In our case the following are valid (see also [Redn 84]): Du :
Nu
Lu ( Q) ⫽
ln p(x i ; Q) ⫽
i⫽1
Dl :
Ll ( Q) ⫽
Nu
ln
ln p(y, z iy ; Q) ⫹ ln
y⫽1 i⫽1
⫽
Ny M
ln Py p(z iy |y; ) ⫹ ln
y⫽1 i⫽1
Py p(x i |y; )
(10.14)
y⫽1
i⫽1
Ny M
M
Nl ! N1 !N2 ! . . . NM !
Nl ! N1 !N2 ! . . . NM !
(10.15)
Note that in the case of labeled data the“full”observations of the joint events (y, z iy ) are made available to us. The second term in the log-likelihood function results from the generalized Bernoulli theorem [Papo 02, p.110]. This is a consequence of the way labeled samples were assumed to occur. Basically, we are given Nl random samples and after the labeling N1 of them are assigned to class y ⫽ 1, N2 of them to class y ⫽ 2 and so on. However, this term is independent of and P and, in practice, is neglected. The unknown set of parameters , P can now be obtained by maximizing the sum Lu (Q) ⫹ Ll (Q) with respect to and P . Due to the nature of Lu (Q) optimization has to be carried out in the framework discussed in Section 2.5.5. The EM algorithm is the most popular alternative toward this end. In order to get a feeling on how the presence of labeled data affects the results of the EM algorithm, when compared with the case where only unlabeled data are used, let us consider the example of Section 2.5.5, where the conditional densities were assumed to be Gaussians, that is, 1
p(x| y; ) ⫽
2y2
l/2 exp ⫺
x ⫺ y 2
2y2
(10.16)
2 ]T . The E-step now becomes: where ⫽ [T1 , . . . , TM , 12 , . . . , M
E-step: Q( Q; Q(t)) ⫽
Nu M
P(y|x i ; Q(t)) ln p(x i |y; y , y2 )Py
i⫽1 y⫽1
⫹
Nl i⫽1
ln p(z iy |y; y , y2 )Py
(10.17)
581
“12-Ch10-SA272” 18/9/2008 page 582
582
CHAPTER 10 Supervised Learning: The Epilogue
In words,the expectation operation is required for the unlabeled samples only,since for the rest the corresponding labels are known. Using similar steps as in Problem 2.31 and considering both log-likelihood terms,recursions (2.98),(2.99) and (2.100) are modified as follows: M-step: Ny i⫽1 P( y|x i ; Q(t))x i ⫹ i⫽1 z iy N u P( y|x ; Q (t)) ⫹ N i y i⫽1
Nu y (t ⫹ 1) ⫽
Nu y2 (t
⫹ 1) ⫽
i⫽1 P( y|x i ; Q(t))x i ⫺ y (t ⫹ 1)
Nu l i⫽1 P( y|x i ; Q(t)) ⫹ Ny
Ny 2 i⫽1 z iy ⫺ y (t ⫹ 1) ⫹ Nu l i⫽1 P( y|x i ; Q(t)) ⫹ Ny 1 Py (t ⫹ 1) ⫽ Nu ⫹ N l
N u
(10.18) 2
(10.19)
P( y|x i ; Q(t)) ⫹ Ny
(10.20)
i⫽1
Remarks ■
■
Provided that the adopted mixture model for the marginal density is correct, the use of unlabeled data is guaranteed to improve performance, for example, [Cast 96]. However, if this is not the case, and the adopted model does not match the characteristics of the true distribution that generates the data, incorporating unlabeled data may actually degrade the performance. This is a very important issue, since in practice it may not be an easy task to have good knowledge about the exact nature of the underlying distribution. This claim has been supported by a number of researchers, and a theoretical justification is provided in [Cohe 04]. Looking at Eqs. (10.14) and (10.15), we observe that if Nu ⬎⬎Nl , which is usually the case in practice, the unlabeled data term is the dominant one. This is also clear by inspecting the recursion in (10.18)–(10.20). To overcome this, an appropriate weighting of the two log-likelihood terms may be used, for example, [Cord 02, Niga 00]. Another problem associated with the EM algorithm is, as we already know, that it can be trapped in a local maximum. This can also be a source for performance degradation when using unlabeled data. This is treated in [Niga 00].
10.5.2 Graph-Based Methods In any classification task, the ultimate goal is to predict the class label given the observation x. In the generative modeling, the philosophy is to model the “generation”mechanism of the data and also to adopt a model for p(x|y),which then implies
“12-Ch10-SA272” 18/9/2008 page 583
10.5 Semi-Supervised Learning
all the required information, that is, p(x), p( y, x), P( y|x). However, throughout this book,the majority of the methods we dealt with were developed on a different rationale. If all we need is to infer the class labels, let us model the required information directly. As Vapnik stated: When solving a given problem, try to avoid solving a more general one as an intermediate step.
For example, if the densities underlying the classes are Gaussians with the same covariance matrix, one need not bother to estimate the covariance parameters; exploiting the fact that the optimum discriminant function is linear can be sufficient to design a good classifier [Vapn 99]. Such techniques are known as diagnostic or discriminative methods. Linear classifiers, backpropagation neural networks, and support vector machines are typical examples that fall under the diagnostic design methodology. In all these methods, the marginal probability density, p(x), was not considered explicitly for the estimation of the corresponding optimal parameters. The obvious question that now arises is whether and how such techniques can benefit from the existence of unlabeled data. The latter “express” themselves via p(x). On the other hand, the marginal probability density does not enter into the discriminative models explicitly. The way out comes through penalization, where one forces the solution to respect certain general characteristics of p(x). Typical such characteristics, which have been exploited in semi-supervised learning, are: (a) the clustering structure that may underlie the data distribution and (b) the manifold geometry on which the data might lie. This information can be embedded in the form of regularization in the optimization of a loss function associated with the classification task (see Section 4.19). Graph methods fall under the diagnostic design approach, and a number of techniques have been proposed to exploit classification-related information that resides in the data distribution. In order to present the basic rationale behind graph methods, we will focus on a technique that builds around the manifold assumption. This technique also fits nicely with a number of concepts discussed in previous chapters in the book. As we have already seen in Section 6.7.2, graph methods start with the construction of an undirected graph G(V , E). Each node, vi , of the graph corresponds to a data point, x i , and the edges connecting nodes, e.g., vi , vj , are weighted by a weight W (i, j) that quantifies similarity between the corresponding points, x i , x j . There was discussed how these weight values can be used to provide information related to the local structure of the underlying manifold—that is, the intrinsic geometry associated with p(x). Assume that we are given a set of Nl labeled points x i , i ⫽ 1, 2 . . . , Nl , and a set of Nu unlabeled points x i , i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu . Our kickoff point is Eq. (4.79) Nl i⫽1
L g(x i ), yi ⫹ ||g||2H
(10.21)
583
“12-Ch10-SA272” 18/9/2008 page 584
584
CHAPTER 10 Supervised Learning: The Epilogue
where H is used explicitly to denote that the norm of the regularizer is taken in the RKHS space, and we have assumed ⍀(·) to be a square one. The loss function is considered over the labeled data only. In [Belk 04, Sind 06],it is suggested to adding an extra regularization term that reflects the intrinsic structure of p(x). Using some differential geometry arguments,which are not of interest to us here,it turns out that a quantity that approximately reflects the underlying manifold structure is related to the Laplacian matrix of the graph (see also Section 6.7.2). The proposed in [Belk 04] optimization task is: arg min g∈H
Nl 1 L g(x i ), yi ⫹ ␥H ||g||2H ⫹ Nl i⫽1
N l ⫹Nu 2 ␥I g(x i ) ⫺ g(x j ) W (i, j) 2 (Nl ⫹ Nu )
(10.22)
i, j⫽1
Observe that two normalizing constants are present in the denominators and account for the number of points contributing to each of the two data terms. The parameters ␥H , ␥I control the relative significance of the two terms in the objective function. Also note that in the last term all points, labeled as well as unlabeled, are considered. For those who do not have “theoretical anxieties,” it suffices to understand that the last term in the cost accounts for the local geometry structure. If two points are far apart, the respective W (i, j) is small so their contribution to the cost is negligible. On the other hand,if the points are closely located,W (i, j) is large, and these points have an important “say” in the optimization process. This means that the demand (through the minimization task) of nearby points to be mapped to similar values (i.e., g(x i ) ⫺ g(x j ) to be small) will be seriously taken into account. This is basically a smoothness constraint, which is in line with the essence of the manifold assumption stated before. Using similar arguments as in Section 6.7.2, we end up with the following optimization task arg min g∈H
Nl 1 ␥I L g(x i ), yi ⫹ ␥H ||g||2H ⫹ g T Lg Nl (Nl ⫹ Nu )2
(10.23)
i⫽1
where g ⫽ [g(x 1 ), g(x 2 ), . . . , g(x Nl ⫹Nu )]T . Recall that the Laplacian matrix is defined as L⫽D⫺W
Nl ⫹Nu where D is the diagonal matrix with elements Dii ⫽ W (i, j) and W ⫽ j [W (i, j)], i, j ⫽ 1, 2, . . . , Nl ⫹ Nu . A most welcome characteristic of this procedure is that the Representer Theorem, discussed in Section 4.19.1, is still valid and the minimizer of (10.23) admits an expansion g(x) ⫽
N l ⫹Nu j⫽1
aj K (x, x j )
(10.24)
“12-Ch10-SA272” 18/9/2008 page 585
10.5 Semi-Supervised Learning
where K (·, ·) is the adopted kernel function. Observe that the summation is taken over labeled as well as unlabeled points. In Section 4.19.1, it was demonstrated how use of the Representer Theorem can facilitate the way the optimal solution is sought. This is also true here. Take asan example the case where the loss function is the least squares one that is, L g(x i ), yi ⫽ (yi ⫺ g(x i ))2 . Then it is easy to show (Problem 10.4) that the coefficients in the expansion (10.24) are given by [a1 , a2 , . . . aNl ⫹Nu ]T ≡ a ⫽ ( J K ⫹ ␥H Nl I ⫹
␥I Nl LK)⫺1 y (Nl ⫹ Nu )2
(10.25)
where I is the identity matrix, y ⫽ [y1 , y2 , . . . , yNl , 0, . . . , 0]T , J the (Nl ⫹ Nu ) ⫻ (Nl ⫹ Nu ) diagonal matrix with Nl entries as 1 and the rest 0, i.e., J ⫽ diag(1, 1, . . . , 1, 0, . . . , 0) and K ⫽ [K (i, j)] is the (Nl ⫹Nu )⫻(Nl ⫹Nu ) Gram matrix. Combining (10.25) and (10.24) results in the optimum classifier, employing labeled as well as unlabeled data, given by g(x) ⫽ y T ( J K ⫹ ␥H Nl I ⫹
␥I Nl LK)⫺1 p (Nl ⫹ Nu )2
(10.26)
where p ⫽ [K (x, x 1 ), . . . , K (x, x Nl ⫹Nu )]T (see also Section 4.19.1). The resulting minimizer is known as the Laplacian regularized kernel least squares (LRKLS) solution and it can be seen as a generalization of the kernel ridge regressor given in (4.100). Indeed, if ␥I ⫽ 0, unlabeled data do not enter into the game and the last term in the parenthesis becomes zero. Then for C ⫽ ␥H Nl we obtain the kernel ridge regressor form (Eq. (4.110)). Note that Eq. (10.26) is the result of a scientific evolution process that spans a period spreading over three centuries. It was Gauss in the nineteenth century who solved for the first term in the parenthesis. The second term was added in the mid-1960s, due to the introduction of the notion of regularized optimization ([Tiho 63, Ivan 62, Phil 62]). To our knowledge, ridge regression was introduced in statistics in [Hoer 70]. The kernelized version was developed in mid-1990s following the work of Vapnik and his coworkers, and the Laplacian “edition” was added in the beginning of this century! We can now summarize the basic steps in computing the LRKLS algorithm: Laplacian regularized kernel least squares classifier ■
Construct a graph using both labeled and unlabeled points. Choose weights W (i, j) as described in Section 6.7.2.
■
Choose a kernel function K (·, ·) and compute the Gram matrix K(i, j).
■
Compute the Laplacian matrix L ⫽ D ⫺ W .
■
Choose ␥H , ␥I .
■
Compute a1 , a2 , . . . , aNl ⫹Nu from Eq. (10.25).
585
“12-Ch10-SA272” 18/9/2008 page 586
586
CHAPTER 10 Supervised Learning: The Epilogue
l ⫹Nu Given an unknown x, compute g(x) ⫽ N aj K (x, x j ). For the two-class j⫽1 classification case the class label, y ∈ [⫹1, ⫺1], is obtained by y ⫽ sign{g(x)}. By changing the cost function and/or the regularization term different algorithms result with different performance trade-offs,for example,[Wu 07, Dela 05, Zhou 04]. Another direction that has been followed within the graph theory framework is what is called label propagation, for example, [Zhu 02]. Given a graph, nodes corresponding to the labeled points are assigned their respective class label (e.g., ⫾1 for the two-class case) and the unlabeled ones are labeled with a zero. Labels are then propagated in an iterative way through the data set along high-density areas defined by the unlabeled data, until convergence. In [Szum 02] label propagation is achieved by considering Markov random walks on the graph. An interesting point is that the previously discussed two directions to semi-supervised learning, which build on graph theoretic arguments, turn out to be equivalent or, at least, very similar, see, for example, [Beng 06]. Once more, the world is small!
10.5.3 Transductive Support Vector Machines According to the inductive inference knowledge (training set using labeled fier or regressor) is derived, which is specific points comprising the test set.
philosophy, one starts from the particular data) and then the general rule (the classisubsequently used to predict the labels of In other words, one follows a path
particular ⫺→ general ⫺→ particular Vapnik and Chervonenkis, pushing the frontiers, questioned whether this is indeed the best path to follow in practice. In cases where the training data set is limited in size, deriving a good general rule becomes a hard task. For such cases, they proposed the transductive inference approach, where one follows a “direct” path particular ⫺→ particular In such a way, one may be able to exploit information residing in a given test set and obtain improved results. Transductive learning is a special type of semi-supervised learning and the goal is to predict the labels of the points in a specific test set by embedding the points of the set, explicitly, in the optimization task. From this perspective, label propagation techniques, discussed before, are also transductive in nature. For a more theoretical treatment of transductive learning, the reader is referred to, for example, [Vapn 06, Derb 03, Joac 02]. In the framework of support vectors machines (see Section 3.7), and for the two class problem, transductive learning is cast as follows. Given the set Dl ⫽ {x i , i ⫽ 1, 2, . . . , Nl } of labeled points and the set Du ⫽ {x i , i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu } compute the labels yNl ⫹1 , . . . , yNl ⫹Nu of the points in Du so that the hyperplane that separates the two classes, by taking into consideration both labeled and unlabeled
“12-Ch10-SA272” 18/9/2008 page 587
10.5 Semi-Supervised Learning
points, has maximum margin. The corresponding optimization tasks for the two versions (hard margin and soft margin) become: Hard margin TSVM minimize
1 J (yNl ⫹1 , . . . , yNl ⫹Nu , w, w0 ) ⫽ ||w||2 2
(10.27)
subject to
yi (w T x i ⫹ w0 ) ⱖ 1, i ⫽ 1, 2, . . . , Nl
(10.28)
yi (w T x i ⫹ w0 ) ⱖ 1, i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu
(10.29)
yi ∈ {⫹1, ⫺1}, i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu
(10.30)
Soft margin TSVM minimize
1 J (yNl ⫹1 , . . . , yNl ⫹Nu , w, w0 , ) ⫽ ||w||2 ⫹ 2 Cl
Nl i⫽1
subject to
i ⫹ Cu
N l ⫹Nu
i
(10.31)
i⫽Nl ⫹1
yi (w T x i ⫹ w0 ) ⱖ 1 ⫺ i , i ⫽ 1, 2, . . . , Nl
(10.32)
yi (w T x i ⫹ w0 ) ⱖ 1 ⫺ i , i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu
(10.33)
yi ∈ {⫹1, ⫺1}, i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu
(10.34)
i ⱖ 0, i ⫽ 1, 2, . . . , Nl ⫹ Nu
(10.35)
where Cl , Cu are user-defined parameters that control the importance of the respective terms in the cost. In [Joac 99,Chap 05] an extra constraint is used that forces the solution to assign unlabeled data to the two classes in roughly the same proportion as that of the labeled ones. Figure 10.7 shows a simplified example,for the hard margin case,illustrating that the optimal hyperplane, which results if only labeled examples are used (SVM), is different from the one obtained when both labeled and unlabeled data are employed (TSVM). Performing labeling (of the unlabeled samples),so that the margin between the resulting classes is maximized,pushes the decision hyperplane in sparse regions and it is in line with the clustering assumption stated in the beginning of this section. A major difficulty associated with TSVM is that, in contrast to the convex nature of the standard SVM problem, the optimization is over the labels yi , i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu , which are integers in {⫹1, ⫺1} and it is an NP-hard task. To this end, a number of techniques have been suggested. For example, in [DeBi 04] the task is relaxed, and it is solved in the semidefinite programming framework. Algorithms based on coordinate descent searching have been proposed in [Joac 02, Demi 00, Fung 01]. A slight reformulation of the problem is proposed
587
“12-Ch10-SA272” 18/9/2008 page 588
588
CHAPTER 10 Supervised Learning: The Epilogue
x2
x1
FIGURE 10.7 Red lines correspond to the SVM classifier when only labeled “⫺” and “⫹” points are available. The black lines result when the unlabeled data have been considered and have “pushed” the decision hyperplane to an area which is sparse in data.
in [Chap 06]. The constraints associated with the unlabeled data are removed and replaced by |w T x i ⫹ w0 | ⱖ 1 ⫺ i , i ⫽ Nl ⫹ 1, . . . , Nl ⫹ Nu . Such constraints push the hyperplane away from the unlabeled data, since penalization occurs each time the absolute value becomes less than one. Hence, they are in line with the cluster assumption. This problem formulation has the advantage of removing the combinatorial nature of the problem; yet it remains nonconvex. Strictly speaking, the problem solved in [Chap 06] is not transductive in nature, since one does not try to assign labels to the unlabeled points. All one tries to do is to locate the hyperplane in sparse regions and do not “cut” clusters. This is the reason that such techniques are known as semi-supervised SVM or S3VM (see, e.g., [Benn 98, Sind 06a]). At this point it is interesting to note that the borderline between transductive learning and semi-supervised learning that is inductive in nature is not clearly marked, and it is a topic of ongoing discussion. Take, for example, TSVM. After training, the resulting decision hyperplane can be used for induction to predict the label of an unseen point. We are not going to delve into such issues. A very interesting and quite enlightening discussion on the topic,in a“Platonic dialogue”style,is provided in [Chap 06a,Chapter 25]. Remarks ■
Besides the previous semi-supervised methodologies that we have presented, a number of other techniques have been suggested. For example, self-training is simple in concept and a commonly used approach. A classifier is first trained with the labeled data. Then this is applied to the unlabeled data to perform label predictions. Based on a confidence criterion,those of the data that result
“12-Ch10-SA272” 18/9/2008 page 589
10.5 Semi-Supervised Learning
in confident predictions are added in the labeled training set. The classifier is retrained, and the process is repeated, for example, [Culp 07]. The procedure is similar with that used in the decision feedback equalization (DFE) ([Proa 89]) in communications for more than three decades, in the sense that the classifier uses its own predictions to train itself. Similar in concept is the co-training ([Mitc 99, Balc 06, Zhou 07] procedure. However, in this case, the feature set is split into a number of subsets, for example, two, and for each subset a separate classifier is trained using labeled data. The trained classifiers are then used to predict (in their respective feature subspace) labels for the unlabeled points. Each one of the classifiers passes to the other the most confident of the predictions,together with the respective points. The classifiers are then retrained using this new extra information. Splitting the feature set and training the classifiers in different subspaces provides a different complementary “view” of the data points. This reminds us of the classifier combination method where classifiers are trained in different subspaces (Section 4.21). As it was the case there, independence of the two sets is a required assumption. ■
A major issue concerning semi-supervised techniques is whether and under what conditions one can obtain enhanced performance compared to training the classifier using labeled data only. A number of papers report that enhanced performance has been obtained. For example, in [Niga 00] a generative approach has been applied to the problem of text classification. It is reported that, using a number of 10,000 unlabeled articles, substantial improvement gains have been attained when the number of labeled documents is small. As the number of labeled data increases from a few tens to a few thousands, the classification accuracies (corresponding to semi-supervised and supervised training) start to converge. In [Chap 06a] a number of semi-supervised techniques were compared using eight benchmark data sets. Some general conclusions are: (a) One must not always expect to obtain improved performance when using unlabeled data. (b) Moreover, the choice of the type of the semi-supervised technique is a crucial issue. The algorithm should “match” the nature of the data set; algorithms that implement the clustering assumption (e.g., TSVM) must be used with data exhibiting a cluster structure, and algorithms that implement the manifold assumption (e.g., Laplacian LS) must be used with data residing on a manifold. So, prior to using a semi-supervised technique, one must have a good “feeling” about the data at hand. This point was also stressed in the remarks given previously, when dealing with the generative methods. It was stated that if the adopted model for the class conditional densities is not correct,then the performance,using unlabeled data,may degrade. Besides the cases already mentioned before in this section, the performance of semi-supervised techniques has also been tested in the context of other real applications. In [Kasa 04] transductive SVMs were used to recognize promoter sequences in genes. Their results show that TSVM achieve
589
“12-Ch10-SA272” 18/9/2008 page 590
590
CHAPTER 10 Supervised Learning: The Epilogue
enhanced performance compared to the (inductive) SVM. The news coming from [Krog 04], however, is not that encouraging. In the task of predicting functional properties of proteins, the SVM approach resulted in much better performance compared to TSVM. These results are in line with the comments made before; there is no guarantee for performance improvement when using semi-supervised techniques. Some more samples of applications of semi-supervised learning techniques include [Wang 03] for relevance feedback in image retrieval, [Kock 03] for mail classification, and [Blum 98] for Web mining. ■
As it was stated in the beginning of this section, our goal was to present to the reader the main concepts behind semi-supervised learning and in particular to see how techniques that have been used in this book in earlier chapters can be extended to this case. A large number of algorithms and methods are around, and the area is, at the time of writing, the focus of an intense research effort by a number of serious groups worldwide. For deeper and broader information, the interested reader may consult, [Chap 06a, Zhu 07].
■
Another type of classification framework was proposed (once more) by Vapnik [Chap 06a, Chapter 24]. It is proposed that an additional data set is used, which is not from the same distribution as the labeled data. In other words, the points of this data set do not belong to either of the classes of interest and are called the Universum. This data set is a form of data-dependent regularization and it encodes prior knowledge related to the problem at hand. The classifier must be trained so that the decision function to result in small values for the points in the Universum; that is, these points are forced to lie close to the decision surface. In [West 06] it is shown that different choices of Universa and loss functions result in some known types of regularizers. Early results reported in [West 06, Sinz 06] indicate that the obtained performance depends on the quality of Universum set. The choice of an appropriate Universum is still an open issue. The results obtained in [Sinz 06] suggest that must be carefully chosen and must contain invariant directions and to be positioned “in between” the two classes.
10.6 PROBLEMS 10.1 Let P be the probability that event A occurs. The probability that event A occurs k times in a sequence of N independent experiments is given by the binomial distribution N k P (1 ⫺ P)N ⫺k k
Show that E[k] ⫽ NP and k2 ⫽ NP(1 ⫺ P).
“12-Ch10-SA272” 18/9/2008 page 591
References
10.2 In a two-class problem the classifier to be used is the minimum Euclidean distance one. Assume N1 samples from class 1 and N2 from class 2 . Show that the leave-one-out method estimate can be obtained from the resubstitution method, if the distance of x from the class means di (x), i ⫽ 1, 2, are modified i )2 di (x), if x belongs to class i. Furthermore, show that in as di⬘ (x) ⫽ ( NN i ⫺1 this case the leave-one-out method always results in larger error estimates than the resubstitution method. 10.3 Show that for the Bayesian classifier, the estimate provided by the resubstitution method is a lower bound of the true error and that computed from the holdout method is an upper bound. 10.4 Show Eq. (10.25).
REFERENCES [Balc 06] Balcan M. F., Blum A. “An augmented PAC model for semi-supervised learning,” in SemiSupervised Learning (Chapelle O., Sch¨olkopf B., Zien A., eds.), MIT Press, 2006. [Belk 04] Belkin V., Niyogi P., Sindhwani V. “Manifold regularization: A geometric framework for learning from examples,” Technical Report, TR:2004-06, Department of Computer Science, University of Chicago, 2004. [Benn 98] Bennett K., Demiriz A. “Semi-supervised support vector machines,” in Advances in Neural Information Processing Systems,Vol. 12, 1998. [Beng 06] Bengio Y., Delalleau O., Le Roux N. “Label propagation and quadratic criterion,”in SemiSupervised Learning (Chapelle O., Sch¨olkopf B., Zien A., eds.), MIT Press, 2006. [Blum 98] Blum A., Mitchell T. “Combining labeled and unlabeled data with co-training,” Proceedings of the 11th Annual Conference on Computational Learning Theory, pp. 92–100, 1998. [Brag 04] Braga-Neto U., Dougherty E. “Bolstereol error estimation,” Pattern Recognition,Vol. 37, pp. 1267–1281, 2004. [Cast 96] Castelli V., Cover T. “The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter,” IEEE Transactions on Information Theory, Vol. 42, pp. 2101–2117, 1996. [Chap 05] Chapelle O., Zien A. “Semi-supervised classification by low density separation,” Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, pp. 57–64, 2005. [Chap 06] Chapelle O., Chi M., Zien A. “A continuation method for semi-supervised SVMs,” Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA. 2006. [Chap 06a] Chapelle O., Scho¨ lkopf B., Zien A. Semi-Supervised Learning, MIT Press, 2006. [Cavo 97] Cavouras D., et al. “Computer image analy
