A 3rd Edition
r y d
D
o o <.
0
f
Pattern Recognition and Computer Vision
This page is intentionally left blank
A 3rd Edition
r J i b o o <
of
Pattern Recognition and Computer Vision editors
C H Chen University of Massachusetts Dartmouth, USA
P S P Wang Northeastern University, USA
\[p World Scientific NEW JERSEY • LONDON • SINGAPORE •
BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI
Published by World Scientific Publishing Co. Pte. Ltd. 5 Tori Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
First published 2005 Reprinted 2006
HANDBOOK OF PATTERN RECOGNITION & COMPUTER VISION (3rd Edition) Copyright © 2005 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN 981-256-105-6
Printed in Singapore by Mainland Press
Preface to the Third Edition
Dedicated to the memory of the late Professor King Sun Fu (1930-1985), the handbook series, with first edition (1993), second edition (1999) and third edition (2005), provides a comprehensive, concise and balanced coverage of the progress and achievements in the field of pattern recognition and computer vision in the last twenty years. This is a highly dynamic field which has been expanding greatly over the last thirty years. No handbook can cover the essence of all aspects of the field and we have not attempted to do that. The carefully selected 33 chapters in the current edition were written by leaders in the field and we believe that the book and its sister volumes, the first and second editions, will provide the growing pattern recognition and computer vision community a set of valuable resource books that can last for a long time. Each chapter will speak for itself the importance of the subject area covered. The book continues to contain five parts. Part 1 is on the basic methods of pattern recognition. Though there are only five chapters, the readers may find other coverage of basic methods in the first and second editions. Part 2 is on basic methods in computer vision. Again readers may find that Part 2 complements well what were offered in the first and second editions. Part 3 on recognition applications continues to emphasize on character recognition and document processing. It also presents new applications in digital mammograms, remote sensing images and functional magnetic resonance imaging data. Currently one intensively explored area of pattern recognition applications is the personal identification problem, also called biometrics, though the problem has been around for a number of years. Part 4 is especially devoted to this topic area. Indeed chapters in both Part 3 and Part 4 represent the growing importance of applications in pattern recognition. In fact Prof. Fu had envisioned the growth of pattern recognition applications in the early 60's He and his group at Purdue had worked on the character recognition, speech recognition, fingerprint recognition, seismic pattern recognition, biomedical and remote sensing recognition problems, etc. Part 5 on system and technology presents other important aspects of pattern recognition and computer vision. Our sincere thanks go to all contributors of this volume for their outstanding technical contributions. We would like to mention specially Dr. Quang-Tuan Luong, Dr. Giovanni Garibotto and Prof. Ching Y. Suen for their original contributions to all three volumes. Other authors who have contributed to all three volumes are: Prof. Thomas S. Huang, Prof. J.K. Aggarwal, Prof. Yun Y. Tang, Prof. C.C. Li, Prof. R. Chellappa and Prof. P.S.P. Wang. We are pleased to mention that Prof. Thomas Huang and Prof. Jake Aggarwal are the recipients respectively in 2002 and 2004, of the prestigious K.S. Fu Prize sponsored by the International Association of Pattern Recognition (IAPR). Among Prof. Fu's Ph.D. graduates at Purdue who have contributed to the handbook series are: C.H. Chen (1965), M.H. Loew (1972), S.M. Hsu (1975), S.Y. Lu (1977), K.Y. Huang (1983) and H.D. Cheng (1985). Finally we would like to pay tribute to the late Prof. Azirel Rosenfeld (1931-2004) who, as one IAPR member put it, is a true scientist and a great giant in the field. He was awarded the K.S. Fu Prize by IAPR in 1988. Readers are reminded to read Prof. Rosenfeld's inspirational article on "Vision - Some Speculations" that appeared as Foreword of the second edition of the handbook series. Prof. Rosenfeld's profound influence in the field will be felt in the many years to come. v
VI
The camera ready manuscript production requires certain amount of additional efforts, as compared to typeset printing, on the part of editors and authors. We like to thank all contributors for their patience in making the necessary revisions to comply with the format requirements during this long process of manuscript preparation. Our special thanks go to Steven Patt, in-house editor of World Scientific Publishing, for his efficient effort to make a timely publication of the book possible.
September, 2004
The Editors
Contents
Preface to the Third Edition
v
Contents
vii
Part 1. Basic Methods in Pattern Recognition
1
Chapter 1.1
Statistical Pattern Recognition R.P.W. Duin and D.M.J. Tax
3
Chapter 1.2
Hidden Markov Models for Spatio-Temporal Pattern Recognition Brian C. Lovell and Terry Caelli
25
A New Kernel-Based Formalization of Minimum Error Pattern Recognition Erik McDermott andShigeru Katagiri
41
Chapter 1.3
Chapter 1.4
Parallel Contextual Array Grammars with Trajectories P. Helen Chandra, C. Martin-Vide, K.G. Subramanian, D.L. Van and P. S. P. Wang
55
Chapter 1.5
Pattern Recognition with Local Invariant Features C. Schmid, G. Dorko, S. Lazebnik, K. Mikolajczyk and J. Ponce
71
Part 2. Basic Methods in Computer Vision
93
Chapter 2.1
Case-Based Reasoning for Image Analysis and Interpretation Petra Perner
95
Chapter 2.2
Multiple Image Geometry - A Projective Viewpoint Quang-Tuan Luong
115
Chapter 2.3
Skeletonization in 3D Discrete Binary Images Gabrielle Sanniti di Baja and Injela Nystrom
137
Chapter 2.4
Digital Distance Transforms in 2D, 3D, and 4D Gunilla Borgefors
157
Chapter 2.5
Computing Global Shape Measures Paul L. Rosin
177
VII
VIII
Chapter 2.6
Texture Analysis with Local Binary Patterns Topi Mdenpdd and Matti Pietikdinen
197
Part 3. Recognition Applications
217
Chapter 3.1
Document Analysis and Understanding Yuan Yan Tang
219
Chapter 3.2
Chinese Character Recognition Xiaoqing Ding
241
Chapter 3.3
Extraction of Words from Handwritten Legal Amounts on Bank Cheques In Cheol Kim and Ching Y. Suen
259
OCR Assessment of Printed-Fonts for Enhancing Human Vision Ching Y. Suen, Qizhi Xu and Cedric Devoghelaere
273
Chapter 3.4
Chapter 3.5
Clustering and Classification of Web Documents Using a Graph Model Adam Schenker, Horst Bunke, Mark Last and Abraham Kandel
287
Chapter 3.6
Automated Detection of Masses in Mammograms H.D. Cheng, X.J. Shi, R. Min, X.P. Cai and H.N. Du
Chapter 3.7
Wavelet-Based Kalman Filtering in Scale Space for Image Fusion Hsi-Chin Hsin and Ching-Chung Li
325
Multisensor Fusion with Hyperspectral Imaging Data: Detection and Classification Su May Hsu and Hsiao-hua Burke
347
Independent Component Analysis of Functional Magnetic Resonance Imaging Data V.D. Calhoun andB. Hong
365
Chapter 3.8
Chapter 3.9
303
Part 4. Human Identification
385
Chapter 4.1
Multimodal Emotion Recognition Nicu Sebe, Ira Cohen and Thomas S. Huang
387
Chapter 4.2
Gait-Based Human Identification from a Monocular Video Sequence Amit Kale, AravindSundaresan, AmitK. RoyChowdhury and Rama Chelappa
411
IX
Chapter 4.3
Palmprint Authentication System David Zhang
Chapter 4.4
Reconstruction of High-Resolution Facial Images for Visual Surveillance Jeong-Seon Park and Seong Whan Lee
445
Object Recognition with Deformable Feature Graphs: Faces, Hands, and Cluttered Scenes Jochen Triesch and Christian Eckes
461
Chapter 4.5
Chapter 4.6
Hierarchical Classification and Feature Reduction for Fast Face Detection Bernd Heisele, Thomas Serre, Sam Prentice and Tomaso Poggio
Part 5. System and Technology Chapter 5.1
Tracking and Classifying Moving Objects Using Single or Multiple Cameras Quming Zhou andJ.K. Aggarwal
431
481
497 499
Chapter 5.2
Performance Evaluation of Image Segmentation Algorithms Xiaoyi Jiang
525
Chapter 5.3
Contents-Based Video Analysis for Knowledge Discovery Chia-Hung Yeh, Shih-Hung Lee andC.-C. Jay Kuo
543
Chapter 5.4
Object-Process Methodology and Its Applications to Image Processing and Pattern Recognition Dov Dori
559
Chapter 5.5
Musical Style Recognition — AQuantative Approach Peter van Kranenburg and Eric Backer
583
Chapter 5.6
Auto-Detector: Mobile Automatic Number Plate Recognition Giovanni B. Garribotto
601
Chapter 5.7
Omnidirectional Vision Hiroshi Ishiguro
619
Index
629
C H A P T E R 1.1 STATISTICAL PATTERN
RECOGNITION
R.P.W. Duin, D.M.J. Tax Information and Communication Theory Group Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology P.O.Box 5031, 2600 GA, Delft, The Netherlands E-mail: {R.P. W.Duin,D. M.J. Tax} @ ewi.tudelft.nl A review is given of the area of statistical pattern recognition: the representation of objects and the design and evaluation of trainable systems for generalization. Traditional as well as more recently studied procedures are reviewed like the classical Bayes classifiers, neural networks, support vector machines, one-class classifiers and combining classifiers. Further we introduce methods for feature reduction and error evaluation. New developments in statistical pattern recognition are briefly discussed.
1. I n t r o d u c t i o n Statistical p a t t e r n recognition is the research area t h a t studies statistical tools for the generalization of sets of real world objects or phenomena. It thereby aims to find procedures t h a t answer questions like: does this new object fit into the p a t t e r n of a given set of objects, or: to which of the p a t t e r n s defined in a given set does it fit best? T h e first question is related to cluster analysis, b u t is also discussed from some perspective in this chapter. T h e second question is on p a t t e r n classification and t h a t is what will be the main concern here. T h e overall structure of a p a t t e r n recognition system may be summarized as in Figure 1. Objects have first to be appropriately represented before a generalization can be derived. Depending on the demands of the procedures used for this the representation has to be adapted, e.g. transformed, scaled or simplified. T h e procedures discussed in this chapter are partially also studied in areas like statistical learning theory 3 2 , machine learning 2 5 and neural networks 1 4 . As the emphasis in p a t t e r n recognition is close to application areas, questions related to the representation of the objects are important here: how are objects described (e.g. features, distances to prototypes), how extensive may this description be, what are the ways to incorporate knowledge from the application domain? Representations have to be adapted to fit the tools t h a t are used later. Simplifications of representations 3
4
like feature reduction and prototype selection should thereby be considered. In order to derive, from a training set, a classifier that is valid for new objects (i.e. that it is able to generalize) the representation should fulfill an important condition: representations of similar real world objects have to be similar as well. The representations should be close. This is the so-called compactness hypothesis 2 on which the generalization from examples to new, unseen objects is built. It enables the estimation of their class labels on the basis of distances to examples or on class densities derived from examples. Objects are traditionally represented by vectors in a feature space. An important recent development to incorporate domain knowledge is the representation of objects by their relation to other objects. This may be done by a so called kernel method 29 , derived from features, or directly on dissimilarities computed from the raw data 2 6 . We will assume that, after processing the raw measurements, objects are given in a p-dimensional vector space Cl. Traditionally this space is spanned by p features, but also the dissimilarities with p prototype objects may be used. To simplify the discussion we will use the term feature space for both. If K is the number of classes to be distinguished, a pattern classification system, or shortly classifier C{x) is a function or a procedure that assigns to each object x in fi a class u>c, with c = 1,...,K. Such a classifier has to be derived from a set of examples Xtv = {xi,i = 1...N} of known classes y,. Xtr will be called the training set and yi £ wc, c = 1...K a label. Unless otherwise stated it is assumed that yt is unique (objects belong to just a single class) and is known for all objects in Xtl. In section 2 training procedures will be discussed to derive classifiers C(x) from training sets. The performance of these classifiers is usually not just related the quality of the features (their ability to show class differences) but also to their number, i.e. the dimensionality of the feature space. A growing number of features
charactt rization 4
evaluation
update
•s
generalization
update
adaptation
a update
representation
class labels,
classifiers, class models
feature extraction prototype selection
features, dissimilarities class models, object models
larger tra ning sets
^ obj ects better sensors or measurement conditions
Fig. 1.
confidences
The pattern recognition system
5
may increase the class separability, but, may also decrease the statistical accuracy of the training procedure. It is thereby important to have a small number of good features. In section 3 a review is given of ways to reduce the number of features by selection or by combination (so called feature extraction). The evaluation of classifiers, discussed in section 4, is an important topic. As the characteristics of new applications are often unknown before, the best algorithms for feature reduction and classification have to be found iteratively on the basis of unbiased and accurate testing procedures. This chapter builds further on earlier reviews of the area of statistical pattern recognition by Fukunaga 12 and by Jain et al 16 . It is inevitable to repeat and summarize them partly. We will, however, also discuss some new directions like oneclass classifiers, combining classifiers, dissimilarity representations and techniques for building good classifiers and reducing the feature space simultaneously. In the last section of this chapter, the discussion, we will return to these new developments. 2. Classifiers For the development of classifiers, we have to consider two main aspects: the basic assumptions that the classifier makes about the data (which results in a functional form of the classifier), and the optimization procedure to fit the model to the training data. It is possible to consider very complex classifiers, but without efficient methods to fit these classifiers to the data, they are not useful. Therefore, in many cases the functional form of the classifier is restricted by the available optimization routines. We will start discussing the two-class classification problem. In the first three sections, 2.1, 2.2 and 2.3, the three basic approaches with their assumptions are given: first, modeling the class posteriors, second, modeling class conditional probabilities and finally modeling the classification boundary. In section 2.4 we discuss how these approaches can be extended to work for more than two classes. In the next section, the special case is considered where just one of the classes is reliably sampled. The last section, 2.6, discusses the possibilities to combine several (non-optimal) classifiers. 2.1. Bayes
classifiers
and
approximations
A classifier should assign a new object x to the most likely class. In a probabilistic setting this means that the label of the class with the highest posterior probability should be chosen. This class can be found when p(wi|x) and p(w2|x) (for a two class classification problem) are known. The classifier becomes: if p(wi|x) > p(w2|x)
assign object x to u>\, otherwise to W2-
(1)
When we assume that p(ui\x) and p(w2|x) are known, and further assume that misclassifying an object originating from o>i to W2 is as costly as vise versa, classifier (1) is the theoretical optimal classifier and will make the minimum error. This classifier is called the Bayes optimal classifier.
6
In practice p(wi|x) and p(w2|x) are not known, only samples x, are available, and the misclassification costs might be only known in approximation. Therefore approximations to the Bayes optimal classifier have to be made. This classifier can be approximated in several different ways, depending on knowledge of the classification problem. The first way is to approximate the class posterior probabilities p(w c |x). The logistic classifier assumes a particular model for the class posterior probabilities: P(wi|x) =
, ( , T ^, 1 + exp(—w T x)
1
P(w 2 |x) = 1 - p ( w 2 | x ) ,
(2)
where w is a p-dimensional weight vector. This basically implements a linear classifier in the feature space. An approach to fit this logistic classifier (2) to training data Xu, is to maximize the data likelihood L: N
L = np(wi|xO n i ( x ) p(w2|xi) n a ( x ) >
(3)
i
where n c (x) is 1 if object x belongs to class o>c, and 0 otherwise. This can be done by, for instance, an iterative gradient ascent method. Weights are iteratively updated using: Wnew = W 0 l d + ? 7 - — ,
(4)
where rj is a suitably chosen learning rate parameter. In Ref. 1 the first (and second) derivative of L with respect to w are derived for this and can be plugged into (4). 2.2.
Class densities
and Bayes
rule
Assumptions on p(w|x) are often difficult to make. Sometimes it is more convenient to make assumptions on the class conditional probability densities p(x|w): they indicate the distribution of the objects which are drawn from one of the classes. When assumptions on these distributions can be made, classifier (1) can be derived using Bayes' decision rule: PH*)
=? » ! .
(5)
This rule basically rewrites the class posterior probabilities in terms of the class conditional probabilities and the class priors p(w). This result can be substituted into (1), resulting in the following form: if P(X\LOI)P(U>I) > p(x|w2)p(w2)
assign x to wi, otherwise to o;2-
(6)
The term p(x) is ignored because this is constant for a given x. Any monotonically increasing function can be applied to both sides without changing the final decision. In some cases, a suitable choice will simplify the notation significantly. In particular,
7
using a logarithmic transformation can simplify the classifier when functions from the exponential family are used. For the special case of a two-class problem the classifiers can be rewritten in terms of a single discriminant function / ( x ) which is the difference between the left hand side and the right hand side. A few possibilities are: / ( x ) =p(wi|x) - p ( w 2 | x ) ,
(7)
/ ( x ) = p(x|on)p(wi) - p(x|w 2 )p(w 2 ),
(8)
/(x)=ln^+ln^ll.
(9)
p(x|w 2 )
P{LU2)
The classifier becomes: if
/(x) > 0
assign x t o w j , otherwise to o>2.
(10)
In many cases fitting p(x|a>) on training data is relatively straightforward. It is the standard density estimation problem: fit a density on a data sample. To estimate each p(x|w) the objects from just one of the classes UJ is used. Depending on the functional form of the class densities, different classifiers are constructed. One of the most common approaches is to assume a Gaussian density for each of the classes: p(x|w) = J V ( X ; / I , E ) =
( 2 7 r ) P /2| S |i/2
ex
P (-^(x-MfS-^x-M)) -
(n)
where /x is the (p-dimensional) mean of the class u>, and £ is the covariance matrix. Further, |£| indicates the determinant of £ and £ - 1 its inverse. For the explicit values of the parameters /x and E usually the maximum likelihood estimates are plugged in, therefore this classifier is called the plug-in Bayes classifier. Extra complications occur when the sample size TV is insufficient to (in particular) compute £ _ 1 . In these cases a standard solution is to regularize the covariance matrix such that the inverse can be computed: EA = £ + AJ,
(12)
where X is the p x p identity matrix, and A is the regularization parameter to set the trade off between the estimated covariance matrix and the regularizer I. Substituting (11) for each of the classes u>\ and w2 (with their estimated /Xj, /x2 and Ei, £ 2 ) into (9) results in: /(x) = i x W
- E r > + ^(MiSr 1 - M 2 S 2 - 1 ) T x
- ^ E r V i + ^ S j V s - i l n l E t l + ilnlE.I+ln^ij.
(13)
This classifier rule is quadratic in terms of x, and it is therefore called the normalbased quadratic classifier. For the quadratic classifier a full covariance matrix has to be estimated for each of the classes. In high dimensional feature spaces it can happen that insufficient
8
data is available to estimate these covariance matrices reliably. By restricting the covariance matrices to have less free variables, estimations can become more reliable. One approach the reduce the number of parameters, is to assume that both classes have an identical covariance structure: Ei = £2 = E. The classifier simplifies to: / ( x ) = \{^
- . / S - ' x - i / i f E - V i + \l%*-
V 2 + In ^
(14)
Because this classifier is linear in terms of x, this classifier is called the normal-based linear classifier. For the linear and the quadratic classifier, strong class distributional assumptions are made: each class has a Gaussian distribution. In many applications this cannot be assumed, and more flexible class models have to be used. One possibility is to use a 'non-parametric' model. An example is the Parzen density model. Here the density is estimated by summing local kernels with a fixed size h which are centered on each of the training objects: 1 N p(x|w) = - ^ J V f c x i . f t l ) ,
(15)
i—l
where X is the identity matrix and h is the width parameter which has to be optimized. By substituting (15) into (6), the Parzen classifier is defined. The only free parameter in this classifier is the size (or width) h of the kernel. Optimizing this parameter by maximizing the likelihood on the training data, will result in the solution h = 0. To avoid this, a leave-one-out procedure can be used 9 . 2.3. Boundary
methods
Density estimation in high dimensional spaces is difficult. In order to have a reliable estimate, large amounts of training data should be available. Unfortunately, in many cases the number of training objects is limited. Therefore it is not always wise to estimate the class distributions completely. Looking at (1), (6) and (10), it is only of interest which class is to be preferred over the other. This problem is simpler than estimating p(x\u>). For a two-class problem, we just a function / ( x ) is needed which is positive for objects of UJI and negative otherwise. In this section we will list some classifiers which avoid estimating p(x|w) but try to obtain a suitable / ( x ) . The Fisher classifier searches to find a direction w in the feature space, such that the two classes are separated as well as possible. The degree in which the two classes are separated, is measured by the so-called Fisher ratio, or Fisher criterion:
j = \™\~mf.
(16)
y sl + s22 ' Here mi and mo, are the means of the two classes, projected onto the direction w: mi = ~wTHi and rri2 = w T /x 2 . The si and s% are the variances of the two classes projected onto w. The criterion therefore favors directions in which the means are far apart and the variances are small.
9
This Fisher ratio can be explicitly rewritten in terms of w. First we rewrite s
l = E x 6 a , c ( w T x - w 7 > c ) 2 = Exea, c w T ( x - Mc)(x - ^ c ) T w = w r S c w . Second we write (mi - m 2 ) 2 = (w T /x 1 — w T /z 2 ) 2 = w T (/x 1 — /x2)(Ati _ i u 2 ) T w = w T SflW. The term 5 s is also called the between scatter matrix. J becomes: _ \rn-j - m 2 | 2 _ wr5Bw _ wTSBw T T , T sf + s?, w 5 i w + w S 2W w 5iyw' where Sw = S\ + S2 is also called the within scatter matrix. In order to optimize (17), we set the derivative of (17) to zero and obtain: (w T 5.Bw)5iyw = (wrS,^vw)S'BW.
(18)
We are interested in the direction of w and not in the length, so we drop the scalar terms between brackets. Further, from the definition of SB it follows that SBW is always in the direction fj,x — fj,2- Multiplying both sides of (18) by Sw gives: w~S^(Mi-M2)-
(19)
This classifier is known as the Fisher classifier. Note that the threshold b is not defined for this classifier. It is also linear and requires the inversion of the withinscatter Sw- This formulation yields an identical shape of w as the expression in (14), although the classifiers use very different starting assumptions! Most classifiers which have been discussed so far, have a very restricted form of their decision boundary. In many cases these boundaries are not flexible enough to follow the true decision boundaries. A flexible method is the k-nearest neighbor rule. This classifier looks locally which labels are most dominant in the training set. First it finds the k nearest objects in the training set AW(x), and then counts the number of these neighbors are from class u>i or u>2if
ni > 712 assign x t o u i , otherwise to cj2-
(20)
Although the training of the fc-nearest neighbor classifier is trivial (it only has to store all training objects, k can simply be optimized by a leave-one-out estimation), it may become expensive to classify a new object x. For this the distances to all training objects have to be computed, which may be prohibitive for large training sets and high dimensional feature spaces. Another classifier which is flexible but does not require the storage of the full training set is the multi-layered feed-forward neural network4. A neural network is a collection of small processing units, called the neurons, which are interconnected by weights w and v to form a network. A schematic picture is shown in Figure 2. An input object x is processed through different layers of neurons, through the hidden layer to the output layer. The output of the j - t h output neuron becomes: °j(*) = hj nT.vMvrTy)]
(21)
(see Figure 2 for the meaning of the variables). The object x is now assigned to the class j for which the corresponding output neuron has the highest output Oj.
10
X\ • l+exp(-vTh)
X2%
• Xp # •—-
<*n
Fig. 2. Schematic picture of a neural network.
To optimize this neural network, the squared error between the network output and the desired class label is defined: N
£
K
= EE(ni^)-°iW)2-
(22)
where rij (x) is 1 if object x belongs to class w,-, and 0 otherwise. To simplify the notation, we will combine all the weights w, and v into one weight vector w. This error E is a continuous function of the weights w, and the derivative of E with respect to these weights can easily be calculated. The weights of the neural network can therefore be optimized to minimize the error by gradient descent, analogous to (4): dE W n e w = Wold - T y — ,
(23)
where 77 is the learning parameter. After expanding this learning rule (23), it appears that the weight updates for each layer of neurons can be computed by back-propagating the error which is computed at the output of the network (nj(xj) — Oj(xi)). This is therefore called the back-propagation update rule. The advantage of this type of neural networks is that they are flexible, and that they can be trained using these update rules. The disadvantages are, that there are many important parameters to be chosen beforehand (the number of layers, the number of neurons per layer, the learning rate, the number of training updates, etc.), and that the optimization can be extremely slow. To increase the training speed, several additions and extensions are proposed, for instance the inclusion of momentum terms in (23), or the use of second order moments. Neural networks can be easily overtrained. Many heuristic techniques have been developed to decrease the chance of overtraining. One of the methods is to use weight decay, in which an extra regularization term is added to equation (22). This regularization term, often something of the form E = E + \\\w\\2,
(24)
tries to reduce the size of the individual weights in the network. By restricting the size of the weights, the network will adjust less to the noise in the data sample
11
and become less complex. The regularization parameter A regulates the trade-off between the classification error E and the classifier complexity. When the size of the network (in terms of the number of neurons) is also chosen carefully, good performances can be achieved by the neural network. A similar approach is chosen for the support vector classifier32. The most basic version is just a linear classifier as in Eq. (10) with / ( x ) = w T x + 6.
(25)
The minimum distance from the training objects to the classifier is thereby maximized. This gives the classifier some robustness against noise in the data, such that it will generalize well for new data. It, appears that this maximum margin p is inversely related to ||w|| 2 , such that maximizing this margin means minimizing ||w|| 2 (taking the constraints into account that all the objects are correctly classified).
wrx + 6 = 0
1 1
o
o o
Fig. 3.
1
1
/*;
o 4
/
,P
f
i
o
f
1
/
i 1
o
1
|
I
o
1 f
1
• ' t
1 i
1 / 1
i i
•
• cti = 0
•
o
Schematic picture of a support vector classifier
Given linearly separable data, the linear classifier is found which has the largest margin p to each of the classes. To allow for some errors in the classification, some slack variables are introduced to weaken the hard constraints. The error to minimize for the support vector classifier therefore consists of two parts: the complexity of the classifiers in terms of w T w , and the number of classification errors, measured by X^jCi- The optimization can be stated by the following mathematical formulation: min w T w + C y " & , w
(26)
'-—' t
such that (w ( Xi + b > 1 - &
if
1 w ' x , + b < - l + ?i
otherwise.
xewi,
12 Parameter C determines the trade-off between the complexity of the classifier, as measured by w T w , and the number of classification errors. Although the basic version of the support vector classifier is a linear classifier, it can be made much more powerful by the introduction of kernels. When the constraints (27) are incorporated into (26) by the use of Lagrange multipliers a, this error can be rewritten in the so-called dual form. For this, we define the labels y, where y» = 1 when x$ £ ui\ and j/j = — 1 otherwise. The optimization becomes: maxaTxxTa - lTa,
s.t.
y T a = 0,
0 < at < C, Vi
(28)
with w = J2i aiVixi- Due to the constraints in (28) the optimization is not trivial, but standard software packages exist which can solve this quadratic programming problem. It appears that in the optimal solution of (28) many of the Qj become 0. Therefore only a few Qj ^ 0 determine the w. The corresponding objects Xi are called the support vectors. All other objects in the training set can be ignored. The special feature of this formulation is that both the classifier / ( x ) and the error (28) are completely stated in terms of inner products between objects xfXj. This means that the classifier does not explicitly depend on the features of the objects. It depends on the similarity between the object x and the support vectors Xj, measured by the inner product x T Xj. By replacing the inner product by another similarity, defined by the kernel function isT(x, x,), other non-linear classifiers are obtained. One of the most popular kernel functions is the Gaussian kernel: ir(x,x,) = e x p ( -
l | x
^
1 1 2
),
(29)
where a is still a free parameter. The drawback of the support vector classifier is that it requires the solution of a large quadratic programming problem (28), and that suitable settings for the parameters C and a have to be found. On the other hand, when C and a are optimized, the performance of this classifier is often very competitive. Another advantage of this classifier is, that it offers the possibility to encode problem specific knowledge in the kernel function K. In particular for problems where a good feature representation is hard to derive (for instance in the classification of shapes or text documents) this can be important. 2.4. Multi-class
classifiers
In the previous section we focused on the two-class classification problems. This simplifies the formulation and notation of the classifiers. Many classifiers can trivially be extended to multi-class problems. For instance the Bayes classifier (1) becomes: assign x to
w c .,
c* = argmaxp(w c |x)
(30)
c
Most of the classifiers directly follow from this. Only the boundary methods which were constructed to explicitly distinguish between two classes, for instance the
13 Fisher classifier or the support vector classifier, cannot be trivially extended. For these classifiers several combining techniques are available. The two main approaches to decompose a multi-class problem into a set of two-class problems are: (1) one-against-all: train K classifiers between one of the classes and all others, (2) one-against-one: train K(K — l ) / 2 classifiers to distinguish all pairwise classes. Afterward classifiers have to be combined using classification confidences (posterior probabilities) or by majority voting. A more advanced approach is to use ErrorCorrecting Output Codes (ECOC), where classifiers are trained to distinguish specific combinations of classes, but are allowed to ignore others 7 . The classes are chosen such that a redundant output labeling appears, and possible classification errors can be fixed. 2.5. One-class
classifiers
A fundamental assumption in all previous discussions, is that a representative training set XtT is available. That means that examples from both classes are present, sampled according to their class priors. In some applications one of the classes might contain diverse objects, or its objects are difficult or expensive to measure. This happens for instance in machine diagnostics or in medical applications. A sufficient number of representative examples from the class of ill patients or the class of faulty machines are sometimes hard to collect. In these cases one cannot rely on a representative dataset to train a classifier, and a so-called one-class classifier30 may be used.
Fig. 4.
One-class classifier example.
In one-class classifiers, it is assumed that we have examples from just one of the classes, called the target class. From all other possible objects, per definition the outlier objects, no examples are available during training. When it is assumed that the outliers are uniformly distributed around the target class, the classifier should
14 circumscribe the target object as tight as possible in order to minimize the chance of accepting outlier objects. In general, the problem of one-class classification is harder than the problem of conventional two-class classification. In conventional classification problems the decision boundary is supported from both sides by examples of both classes. Because in the case of one-class classification only one set of data is available, only one side of the boundary is supported. It is therefore hard to decide, on the basis of just one class, how strictly the boundary should fit around the data in each of the feature directions. In order to have a good distinction between the target objects and the outliers, good representation of the data is essential. Approaches similar to standard two-class classification can be used here. Using the uniform outlier distribution assumption, the class posteriors can be estimated and the class conditional distributions or direct boundary methods can be constructed. For high dimensional spaces the density estimators suffer and often boundary methods are to be preferred. 2.6. Combining
classifiers
In practice it is hard to find (and train) a classifier which fits the data distribution sufficiently well. The model can be difficult to construct (by the user), too hard to optimize, or insufficient training data is available to train. In these cases it can be very beneficial to combine several "weak" classifiers in order to boost the classification performance 21 . It is hoped that each individual classifier will focus on different aspects of the data and err on different objects. Combining the set of so-called base classifiers will then complement their weak areas.
y, base classifier 1 / feature/ space \
classification base classifier 2
base classifier 3 /
Fig. 5.
combining classifier
base classifier outputs (e.g. confidences)
Combining classifier
The most basic combining approach is to train several different types of classifiers on the same dataset and combine their outputs. One has to realize that classifiers can only correct each other when their outputs vary, i.e. the set of classifiers is diverse 22 . It appears therefore to be more advantageous to combine classifiers which were trained on objects represented by different features. Another approach to force classifiers to become diverse is to artificially change the training set by resampling (resulting in a bagging6 or a boosting 8 approach).
15 The outputs of the classifier can be combined using several combining rules 18 , depending on the type of classifier outputs. If the classifiers provide crisp output labels, a voting combining rule has to be used. When the real valued outputs are available, they can be averaged, weighted averaged or multiplied, the maximum or minimum output can be taken or even an output classifier can be trained. If fixed (i.e. not trained) rules are used, it is important that the output of a classifier is properly scaled. Using a trainable combining rule, this constraint can be elevated but clearly training data is required to optimize this combining rule 10 . 3. Feature reduction In many classification problems it is unclear what features have to be taken into account. Often a large set of k potentially useful features is collected, and by feature reduction the k most suitable features are chosen. Often the distinction between feature selection and feature extraction is made. In selection, only a subset of the original features is chosen. The advantage is that in the final application just a few features have to be measured. The disadvantage is that the selection of the appropriate subset is an expensive search. In extraction new features are derived from the original features. Often all original features are used, and no reduction is obtained in the number of measurement. Butt in many cases the optimization is easier. In Section 3.1 we will discuss several evaluation criteria, then in Section 3.2 feature selection and finally in 3.3 feature extraction. 3.1. Feature set evaluation
criteria
In order to evaluate a feature set, a criterion J has to be defined. Because feature reduction is often applied in classification, the most obvious criterion is thus the performance of the classifier. Unfortunately, the optimization of a classifier is often hard. Other evaluation criteria might be a cheaper approximation to this classification performance. Therefore approximate criteria are used, measuring the distance or dissimilarity between distributions, or even ignoring the class labels, but just focusing on unsupervised characteristics. Some typical evaluation criteria are listed in Table 1. The most simple ones use the scatter matrices characterizing the scatter within classes (showing how samples scatter around their class mean vector, called Sw, the within scatter) and the the scatter between the clusters (showing how the means of the clusters scatter, SB, the between scatter matrix, see also the discussion of the Fisher ratio in section 2.3). These scatter matrices can be combined using several functions, listed in the first part of Table 1. Often Si = SB is used, and 52 = Sw or 5 2 = Sw + SBThe measures between distributions involve the class distributions p(x\u>i), and in practice often single Gaussian distributions for each of the classes are chosen. The reconstruction errors still contain free parameters in the form of a matrix of basis vectors W or a set of prototypes fj,k. These are optimized in their respective
16 procedures, like the Principal Component Analysis or Self-Organizing Maps. These scatter criteria and the supervised measures between distributions are mainly used in the feature selection, Section 3.2. T h e unsupervised reconstruction errors are used in feature extraction 3.3. Table 1. Feature selection criteria for measuring the difference between two distributions or for measuring a reconstruction error. Measures using scatter matrices For explanation J = tr(5^" 1 5i) of Si and S2 J = ln|5^15i| see text. J = gf2Measures between distributions Kolmogorov J = / |p(wi|x) — p(cj 2 |x)|p(x)cbc Average separation J = ^ x j e w i Sx^ewi Divergence Chernoff
J = / (p(x|wi)p(x|w 2 )) ( p ( * ] ^ j ) dx J = - log/p s (x|wi)p 1 _ s (x|u;2)dx
Fisher
J = y / ( p ( x | w i ) p ( w i ) -p(x|w 2 )p(w 2 )) 2 dx
PCA SOM
Reconstruction errors E=\\x(W(W'i'W)_1W'r)xf B = min f c ||x-Ai f c || 2
3 . 2 . Feature
selection
In feature selection a subset of the original features is chosen. A feature reduction procedure consist of two ingredients: the first is the evaluation criterion to evaluate a given set of features, the second is a search strategy to search over all possible feature subsets 1 6 . Exhaustive search is in many applications not feasible. W h e n we start with k = 250 features, and we want to select k = 10, we have to consider in principle ( 250 ) — 2 • 10 1 7 different subsets, which is clearly too much. Instead of exhaustive search, a forward selection can be applied. It starts with the single best feature (according to the evaluation criterion) and adds the feature which gives the biggest improvement in performance. This is repeated till the requested number of features k is reached. Instead of forward selection, the opposite approach can be used: backward selection. This starts with the complete set of features and removes t h a t feature such t h a t the performance increase is the largest. These approaches have the significant drawback t h a t they might miss the optimal subset. These are the subsets for which the individual features have poor discriminability but combined give a very good performance. In order to find these subsets, a more advanced search strategy is required. It can be a floating search where adding
17 and removing features is alternated. Another approach is the branch-and-bound algorithm 12 , where all the subsets of features is arranged in a search tree. This tree is traversed in such order that as soon as possible large sub branches can be disregarded, and the search process is shortened significantly. This strategy will yield the optimal subset when the evaluation criterion J is monotone, that means that when for a certain feature set a value of Jk is obtained, a subset of the features cannot have higher value for Jj,. Criteria like the Bayes error, the Chernoff distance or the functions on the scatter matrices fulfill this. Currently, other approaches appear which combine the traditional feature selection and subsequent training of a classifier. One example is a linear classifier (with the functional form of (25)) called LASSO, Least Absolute Shrinkage and Selection Operator 3 1 . The classification problem is approached as a regression problem with an additional regularization. A linear function is fitted to the data by minimizing the following error: n
m i n ^ ( y i - w T x i - 6 ) 2 + C||w||.
(31)
i
The first part defines the deviation of the linear function w T Xj + 6 from the expected label yi. The second part shrinks the weights w, such that many of them become zero. By choosing a suitable value for C, the number of retained features can be changed. This kind of regularization appears to be very effective when the number of feature is huge (in the thousands) and the training size is small (in the tens). A similar solution can be obtained when the term w T w in (26) is replace by |w| 3 . 3.3. Feature
extraction
Instead of using a subset of the given features, a smaller set of new features may be derived from the old ones. This can be done by linear or nonlinear feature extraction. For the computation of new features usually all original features are used. Feature extraction will therefore almost never reduce the amount of measurements. The optimization criteria are often based on reconstruction errors as in Table 1. The most well-known linear extraction method is the Principal Component Analysis (PCA) 17 . Each new feature i is a linear combination of the original features: x\ = WjX. The new features are optimized to minimize the PCA mean squared error reconstruction error, Table 1. It basically extracts the directions W , in which the data set shows the highest variance. These directions appear to be equivalent to the eigenvectors of the (estimated) covariance matrix E with the largest eigenvalues. For the i-th principal component W j therefore holds: E W i = XiWi,
Xi > Xj, iff < j .
(32)
An extension of the (linear) PCA is the kernelized version, kernel-PCA 24 . Here the standard covariance matrix E is replaced by a covariance matrix in a feature space. After rewriting, the eigenvalue problem in the feature space reduces to the
18 following eigenvalue problem: Koti = AjCKj. Here K is a N x N kernel matrix (like for instance (29)). An object x is mapped onto the i-th principal component by:
^ = J>}if(x, X j ).
(33)
j
Although this feature extraction is linear in the kernel space, in the feature space it will obtain non-linear combinations of features. There are many other methods for extracting nonlinear features, for instance the Self-Organizing Map (SOM) 20 . The SOM is an unsupervised clustering and feature extraction method in which the cluster centers are constrained in their placing. The construction of the SOM is such that all objects in the input space retain as much as possible their distance and neighborhood relations in the mapped space. In other words, the topology is preserved in the mapped space. The mapping is performed by a specific type of neural network, equipped with a special learning rule. Assume that we want to map an /c-dimensional measurement space to a fc'-dimensional feature space, where k' < k. In fact, often k' = 1 or k' = 2. In the feature space, we define a finite orthogonal grid with grid points. At each grid point we place a neuron or prototype . Each neuron stores an fc-dimensional vector /ifc that serves as a cluster center. By defining a grid for the neurons, each neuron does not only have a neighboring neuron in the measurement space, it also has a neighboring neuron in the grid. During the learning phase, neighboring neurons in the grid are enforced to also be neighbors in the measurement space. By doing so, the local topology will be preserved. Unfortunately, training a SOM involves the setting of many unintuitive parameters and heuristics (similar to many neural network approaches). A more principled approach to the SOM is the Generative Topographic Mapping, GTM 5 . The idea is to find a representation of the original p-dimensional data x in terms of //-dimensional latent variables z. For this a mapping function y(z|W) has to be defined. In the GTM it is assumed that the distribution of z in the latent variable space is a grid of delta functions z*: 1
K
W = ^ £ <*(*-*).
(34)
i=l
For the mapping function a generalized linear regression model is used: y(z;W)=W#(z),
(35)
where <&(z) consist of M fixed basis functions (in many cases Gaussian functions) and W is a p x M weight matrix. Because in reality the data will never fit the lowdimensional manifold perfectly, a noise model is introduced: a Gaussian distribution with variance a2: p(x|z,W,<7)=JV(x;y(z|W),a).
(36)
19 The distribution p(x) can then be obtained by integration over the z distribution: p(x|W,(T)= /"p(x|z,W,
(37)
The advantage is that the model is a full probability model. This model can be fitted by optimizing the log likelihood of the training data (In J^^ p(x|W,«r)) using an Expectation-Maximization algorithm. When the user supplies the dimensionality of the latent variable space L, the number of grid points M in this space and the basis functions 3>(z), then the parameters W and a can be optimized. An even simpler model to optimize is the Local Linear Embedding, LLE 28 . Here also the goal is to find a low dimensional representation of the training set Xu'. But unlike the GTM, where an explicit manifold is fitted, here the low dimensional representation is optimized such that the objects can be reconstructed from their neighbors in the training set in the same manner in the low dimensional representation as in the high dimensional one. First, the weights Wij for reconstructing each object Xj from its neighbors Xj are optimized (minimizing the LLE reconstruction error, Table 1, under the constraint that V • Wij = 1). Given the weights, the location of low-dimensional feature vectors Zj,i = 1,...,N is optimized, using the same LLE reconstruction error, but where x, is replaced by z*. This can be minimized by solving a eigenvalue problem (similar to finding the principal components). The feature extraction methods presented above, are all unsupervised, i.e. other information like class labels is not used. This can be a significant drawback when the feature reduction is applied as a preprocessing for solving a classification problem. It might actually happen that all informative features are removed. To avoid this, supervised feature extraction has to be used. Very well known is Linear Discriminant Analysis (LDA) 27 , which is using the weight vector w from the Fisher classifier (see section 2.3) as feature direction. A multi-class extension is presented in 27 but it assumes equal covariance matrices for all classes and the number of features is restricted to if — 1. The LDA can be extended to include the difference in covariance matrix by using the Chernoff criterion instead of the Fisher criterion 23 . 4. Error estimation At various stages in the design of a pattern classification system an estimation of the performance of a procedure, or the separability of a set of classes is needed. Examples are the selection the 'best' feature during feature selection, the feature subspace to be used when several feature extraction schemes are investigated, the performance of the base classifiers in order to find a good set of classifiers to be combined, the optimization of various parameters in classification schemes like the smoothing parameter in the Parzen classifier and the number of hidden units used in a neural network classifier, and the final selection of the overall classification procedure if various competing schemes are followed consecutively. Moreover, at the end an estimate of the performance of the selected classifier is desired.
20
In order to find an unbiased error estimate, a set of test objects with known labels is desirable. This set should be representative for the circumstances expected during the practical use of the procedures under study. Usually this implies that the test set has to be randomly drawn from the future objects to be classified. As their labels should be known for proper testing, these objects are suitable for training as well. Once an object is used for training, however, the resulting classifier is expected to be good for this object. Consequently, if it is also used for testing it generates an optimistic bias in the error estimate. Below two techniques will be discussed to solve this problem. The first is cross-validation, which aims at circumventing the bias. The second is a bootstrap technique by which the bias is estimated and corrected. 4.1.
Cross-validation
Assume that a design set Xd is available for the development of a pattern recognition system, or one of its subsystems, and that in addition to the classifier itself an unbiased estimate of its performance is needed. If Xd is split (e.g. at random) into a training set Xtv and a test set Xte then we want Xtr to be as large as possible to train a good classifier, but simultaneously Xte has to be sufficiently large for an accurate error estimate. The standard deviation of this estimate is sqrt(e * (1 — e) Nte) (e.g. 0.003 for e = 0.01, Nte = 1000 and 0.03 for e = 0.1 and Nte = 100). When the design set is not sufficiently large to split it into a test set and a training set of appropriate sizes, a cross validation procedure might be used in which the design set is split into B ( B > = 2) subsets of about the same size. In total B different classifiers are trained, each by a different group of B — 1 of these subsets. Each classifier is tested by the single subset not used for its training. Finally the B test results are averaged. Consequently all objects are used for testing once. The classifiers they are testing are all based on an (B — 1)/B part of the training set. For larger B these classifiers are expected to be similar and they will be just slightly worse than the classifier based on all objects. A good choice seems to be a 10-fold stratified cross-validation, see 19 , i.e. N = 10, and objects are selected evenly from the classes, i.e. in agreement with their prior probabilities.
4.2. Bootstrap
procedures
Instead of following a procedure that tries to minimize the bias in the error estimate, one may try to estimate the bias 13 ' 15 . A general procedure (independent of the used classifier) can be based on a comparison of the expected apparent error E^pp of a classifier trained by bootstrapping the design set with its error E\ estimated by the entire design set. The difference can be used as an estimate for the bias in the apparent error: E\,ias = E^—E^, which can be used as a correction for the apparent error E% of the classifier based on the design set: Eboot = E%pp + Ebd — Ehapp. A second estimator based on bootstrapping is the so called Ees2 error n ' 1 3 ' 1 5 . It is based on a weighted average of the apparent error of the classifier based on
21 the design set E% and an error estimate EQ for the bootstrap classifier based on the out-of-bootstrap part of the design set. The first is optimistically biased (an apparent error) and the second is an unbiased error estimate (tested by independent samples) of a classifier that is somewhat worse (based on just a bootstrap sample) than the target classifier based on the design set. The weights are given by the asymptotic probability that a sample will be included in a bootstrap sample: 0.632. The £ 6 3 2 error estimate thereby is given by: E632 = 0.368 * E%pp + 0.632 * £#. 4.3. Error
curves
The graphical representation of the classification error is an important tool to study, compare and understand the behavior of classification systems. Some examples of such error curves are: Learning curve : the error as a function of the number of training samples. Simple classifiers decrease faster, but have often a higher asymptotic value than more complex ones. Complexity curve : the error as a function of the complexity of the classifier, e.g. the feature size or the number of hidden units. Such a curve often shows an increasing error after an optimal feature size or complexity. Parameter curve : the error as a function of a parameter in the training procedure, e.g. the smoothing parameter in the Parzen classifier. The optimum that may be observed in such curves is related to the best fit of the underlying model in the classification system w.r.t. the data. Error-reject trade off : the error as function of the reject probability. If a classifier output (e.g. a confidence estimate) is thresholded to reject unreliably classified objects, then this curve shows the gain in error reduction. ROC curves : the trade-off between two types of errors, e.g. the two types of error in a 2-class problem. These Receiver Operator Curves were first studied in communication theory and are useful to select a classifier if the point of operation may vary, e.g. due to unknown classification costs or prior probabilities. 5. Discussion In the previous sections an overview is given of well established techniques for statistical pattern recognition with a few excursion to more recent developments. Modern scientific and industrial developments, the use of computers and internet in daily life and the fast growing sensor technology raise new problems as well as they enable new solutions. We will summarize some new developments in statistical pattern recognition, partially introduced above, partially not yet discussed. Other types of representation than the traditional features enable other ways to incorporate expert knowledge. The dissimilarity representation is an example for this, as it offers the possibility to express knowledge in the definition of the dissimilarity measure, but it opens also other options. Instead of being based on the
22
raw data like spectra, images, or time signals it may be defined on models of objects, like graphs. In such cases structural knowledge is used for the object descriptions. In addition to the nearest neighbor rule, dissimilarity based classifiers offer a richer set of tools with more possibilities to learn from examples, thereby bridging the gap between structural and statistical pattern recognition. Several problems, however, have still to be solved, like the selection of a representation set, optimal modifications of a given dissimilarity measure and the construction of dedicated classifiers. More complicated pattern recognition problems may not be solved by a single off-the-shelf classifier. By the combining classifier technique a number of partial solutions can be combined. Several questions are still open here, like the selection or generation of the base classifiers, the choice of the combiner, the use of a finite training set. Moreover, an overall mathematical foundation is still not available. One-class classifiers are a good way to handle ill sampled problems, or to build classifiers when some of the classes are undersampled. This is important for applications like man or machine monitoring when one of the classes, e.g. normal behavior, is very well defined. Such classifiers may also be used when it is not possible to select a representative training set by an appropriate sampling of the domain of objects. In such cases a domain based class description may be found, locating the class boundary in the representation, without building a probability density function. The well spread availability of computers and sensors, and the costs of labeling objects by human experts, may sometimes result in large databases in which just a small fraction of the objects is labeled. Techniques for training classifiers by partially labeled datasets are still in their early years. This may also be considered as combining clustering and classification. For such problems in which the costs of expert labeling are high, one may also try to optimize the set of objects to be labeled. This technique is called active learning. Several competing strategies exist, e.g. sampling close to an initial decision boundary, or retrieving objects in the modes of the class density distributions. Another variant is online learning, in which the order of the objects to be presented to a decision function is determined by the application, e.g. by a production line in a factory. It has now to be decided whether objects can be safely classified, or whether a human expert has to be solicited, not only to reduce the risk of misclassification, but also to optimally improve the available classification function. An often returning question in dynamic environments is whether a trained classification function is still valid, or whether it should be retrained due to new circumstances. In such problems 'learning' and 'forgetting' are directly related. If a new situation demands retraining, old objects may not be representative anymore and should be forgotten. (They may still be stored in case the old situation appears to return after some time). Many techniques are proposed and many more are to come for solving problems as the above. A difficulty that cannot be easily handled is that they are often ill defined. Consequently, generally valid benchmarks are not available, by which it
23 is not straightforward to detect the good procedures t h a t may work well over a series of applications. As good and bad procedures cannot easily be distinguished, it is to be expected t h a t the set of tools used in statistical p a t t e r n recognition will significantly grow in the near future.
References 1. J.A. Anderson. Logistic discrimination. In P.R. Kirshnaiah and L.N. Kanal, editors, Classification, Pattern Recognition and Reduction of Dimensionality, volume 2 of Handbook of Statistics, pages 169-191. North Holland, Amsterdam, 1982. 2. A.G. Arkadev and E.M. Braverman. Computers and Pattern Recognition. Thompson, Washington, D.C., 1966. 3. C. Bhattacharyya, L.R. Grate, A. Rizki, D. Radisky, F.J. Molina, M. I. Jordan, M.J. Bissell, and I.S. Mian. Simultaneous relevant feature identification and classification in high-dimensional spaces: Application to molecular profiling data. Signal Processing, 83:729-743, 2003. 4. C M . Bishop. Neural Networks for Pattern Recognition. Oxford University Press, Walton Street, Oxford 0X2 6DP, 1995. 5. C M . Bishop, M. Svensen, and C.K.I. Williams. The generative topographic mapping. Neural Computation, 10(l):215-234, 1998. 6. L. Breiman. Bagging predictors. Machine Learning, 26(2):123-140, 1996. 7. T.G. Dietterich and G. Bakiri. Solving multiclass learning problems via errorcorrecting output codes. Journal of Artificial Intelligence Research, 2:263-286, 1995. 8. H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun, and V. Vapnik. Boosting and other ensemble methods. Neural Computation, 6, 1994. 9. R.P.W. Duin. On the choice of the smoothing parameters for Parzen estimators of probability density functions. IEEE Trans, on Computers, C-25(ll):1175-1179, 1976. 10. R.P.W. Duin. The combining classifier: To train or not to train? In International Conference on Pattern Recognition, volume II, pages 765-770, Quebec, Canada, 2002. 11. B. Efron and R.J. Tibshirani. Improvements on cross-validation: the .632+ bootstrap method. J. Amer. Statist. Assoc, 92:548-560, 1997. 12. K. Fukanaga. Introduction to Statistical pattern recognition. Academic press, San Diego, 2nd edition, 1990. 13. D.J. Hand. Recent advances in error rate estimation. Pattern Recognition Letters, 4(5):335-346, 1986. 14. S.S. Haykin. Neural Networks, a comprehensive foundation. Prentice-Hall, 1999. 15. A.K. Jain, R . C Dubes, and Chen, C.-C Bootstrap techniques for error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(5):628-633, September 1987. 16. A.K. Jain, R.P.W. Duin, and J. Mao. Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(l):4-37, 2000. 17. I.T. Jolliffe. Principal Component Analysis. Springer-Verlag, New York, 1986. 18. J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(4):226-239, 1998. 19. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proc. of the 15th Int. Joint Conference on Artificial Intelligence, pages 1137-1143, 1995. 20. T. Kohonen. Self-organizing maps. Springer-Verlag, Heidelberg, Germany, 1995. 21. L.I. Kuncheva. Combining Pattern Classifiers: Methods and Algorithms. Wiley, 2004.
24 22. L.I. Kuncheva and C.J. Whitaker. Measures of diversity in classifier ensembles. Machine Learning, 51:181-207, 2003. 23. M. Loog, R.P.W. Duin, and R. Haeb-Umbach. Multiclass linear dimension reduction by weighted pairwise fisher criteria. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(7):762-766, 2001. 24. S. Mika, B. Scholkopf, A.J. Smola, K.-R. Miiller, M. Scholz, and G. Ratsch. Kernel PCA and de-noising in feature spaces. In M.S. Kearns, S.A. Solla, and D.A. Cohn, editors, Advances in Neural Information Processing Systems, volume 11, pages 536542. MIT Press, 1999. 25. T. M. Mitchell. Machine Learning. Mc Graw-Hill, New York, 1997. 26. E. Pekalska and R.P.W. Duin. Dissimilarity representations allow for building good classifiers. Pattern Recognition Letters, 23(8):943-956, 2002. 27. C.R. Rao. The utilization of multiple measurements in problems of biological classification (with discussion). Journal of the Royal Statistical Society, B, 10:159-203, 1948. 28. S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323-2326, 2000. 29. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004. 30. D.M.J. Tax. One-class classification. PhD thesis, Delft University of Technology, http://www.ph.tn.tudelft.nl/~davidt/thesis.pdf, June 2001. 31. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B, 58(l):267-288, 1996. 32. V.N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.
CHAPTER 1.2 HIDDEN MARKOV MODELS FOR SPATIO-TEMPORAL PATTERN RECOGNITION
Brian C. Lovell" and Terry Caellib The Intelligent Real-Time Imaging and Sensing (IRIS) Group The School of Information Technology and Electrical Engineering The University of Queensland, Australia QLD 4072 E-mail:lovell@ itee. uq. edu.au a
National Information and Communications Technology Australia (NICTA) Research School of Information Sciences and Engineering Australian National University, Australia email:
[email protected] The success of many real-world applications demonstrates that hidden Markov models (HMMs) are highly effective in one-dimensional pattern recognition problems such as speech recognition. Research is now focussed on extending HMMs to 2-D and possibly 3-D applications which arise in gesture, face, and handwriting recognition. Although the HMM has become a major workhorse of the pattern recognition community, there are few analytical results which can explain its remarkably good pattern recognition performance. There are also only a few theoretical principles for guiding researchers in selecting topologies or understanding how the model parameters contribute to performance. In this chapter, we deal with these issues and use simulated data to evaluate the performance of a number of alternatives to the traditional Baum-Welch algorithm for learning HMM parameters. We then compare the best of these strategies to Baum-Welch on a real hand gesture recognition system in an attempt to develop insights into these fundamental aspects of learning. 1. Introduction There is an enormous volume of literature on the application of hidden Markov Models (HMMs) to a broad range of pattern recognition tasks. In the case of speech recognition, the patterns we wish to recognise are spoken words which are audio signals against time. Indeed, the value of Markov models to model speech was recognised by Shannon 26 as early as 1948. In the case of hand gesture recognition, the patterns are hand movements in both space and time — we call this a spatio-temporal pattern recognition problem. The suitability and efficacy of HMMs to such problems is undeniable and they are now established as one of the major tools of the pattern recognition community. Yet, when one looks for research which address fundamental problems such as efficient learning strategies for HMMs or perhaps analytically determining the most suitable architectures for a given problem, die number of papers is greatly diminished. So despite the enormous uptake of HMMs since 25
26
their introduction in the 1960's, we believe that there is still a great deal of unexplored territory. Much of the application of HMMs in the literature is based firmly on the methodology popularised by Rabiner et al. (1983) 25>16-24 for speech recognition and these studies are the primary reference for many HMM researchers resulting in two common practices. One, to use the forward algorithm to determine the MAP(maximum posterior probability) of the model, given an observation sequence, as a classification metric. Two, to use the Baum-Welch as a model estimation/update procedure. We will see how these are not ideal strategies to use as, in the former case, classification is reduced to a single number without directly using the model (data summary) parameters, attributes, per se. As for the latter, the Baum-Welch 4 algorithm (a version of the famous Expectation-Maximisation algorithm14,1'21) is, in the words of Stolke and Omohundro 28 ,"... far from foolproof since it uses what amounts to a hill-climbing procedure that is only guaranteed to find a local likelihood maximum." Moreover, as observed by Rabiner 24 , results can be very dependent on the initial values chosen for the HMM parameters. The problem of finding local rather that global maxima is encountered in many other areas of learning theory and optimisation. These problems are familiar territory to researchers in the artificial neural network community and many techniques have been proposed to counter them. Moreover genetic and evolutionary algorithmic techniques specialise in solving such problems — albeit often very slowly, especially in the case of biological evolution11. With this in mind, we use simulated data to investigate other approaches to learning HMMs from observation sequences in an attempt to find superior alternatives to the traditional Baum-Welch Algorithm. Then we compare and test the best of the alternate strategies on real data from a hand gesture recognition system to see if the real data trials corroborate the conclusions drawn from simulated trials. 1.1. Background and Notation In this study, we focus on the discrete HMM as popularised by Rabiner24. Using the familiar notation from his tutorial paper, a hidden Markov model consists of a set of N nodes, each of which is associated with a set of M possible observations. The parameters of the model include an initial state vector 7T=
\Pl,P2,P3,-,PN}T
with elements pn, n £ [1, AT] which describes the distribution over the initial node set, a transition matrix / a n a12 ... aiw \ 021
a22
•••
02JV
\ajvi ajv2 • • •
OJVJV/
with elements atj with i,j £ [l,N] for the transition probability from node i to node j
27 b) Left-Right
Fig. 1. Cyclic and Left-Right structures. Bold arrows indicate high probability transistions. No arrow between vertices indicates a forbidden (zero-probability) transition.
conditional on node i, and an observation matrix /bn 621
B
6 i2 ... bXM \ b22
•••
b2M
\ &W1 bN2 • • • bNM
J
with elements bim for the probability of observing symbol m G [1,M] given that the system is in state i G [1, N]. We denote the HMM model parameter set by A = (A, B, 7r). The model order pair (N, M) together with additional restrictions on allowed transitions and emissions defines the topology or structure of the model (see figure 1 for an illustration of two different transition structures). One commonly used topology is called Fully-Connected (FC) or Ergodic. In the FC HMM there is not necessarily a defined starting state and all state transitions are possible such that a^ ^ 0 V i, j £ [1, N]. Another topology, especially popular in speech recognition applications, is called Left-Right. In an LR HMM there is a defined starting state (usually state 1) and only state transitions to higher-index states are allowed such that a^• = 0 V i > j where i, j G [1, N]. Rabiner24 defines the three basic problems of HMMs by: Problem 1 Given the observation sequence O = 0\02 • • -OT, and a model A = (A, B, IT), how do we efficiently compute P(0\X), the probability of the observation sequence given the model? Problem 2 Given the observation sequence O = O1O2 • • • Or, and the model A, how do we choose a corresponding state sequence Q = q\q2 ... qr which is optimal in some meaningful sense {i.e., best "explains" the observations)? Problem 3 How do we adjust the model parameters A = (A, B, n) to maximize P((9|A)? Problems 1 and 2 are elegantly and efficiently solved by the forward and Viterbi29'12 algorithms respectively as described by Rabiner in his tutorial. The forward algorithm is used to recognise matching HMMs (i.e., highest probability models, MAP) from the observation sequences. Note, again, that this is not a typical approach to pattern classification as
28
it does not involve matching model with observation attributes. That would involve comparing the model parameters and estimated observation model parameters. MAP does not perform this and so it cannot be as sensitive a measure as exact parameter comparisons. Indeed, a number of reports have already shown quite different HMMs can have identical emissions( observation sequences) 18 ' 3 . The Viterbi algorithm is used less frequently as we are normally more interested in finding the matching model than in finding the state sequence. However, this algorithm is critical in evaluating the precision of the HMM; in other words, how well the model can reconstruct (predict) the observations. Rabiner proposes solving Problem 3 via the Baum-Welch algorithm which is, in essence, a gradient ascent algorithm — a method which is guaranteed to find local maxima only. Solving Problem 3 is effectively the problem of learning to recognise new patterns, so it is really the fundamental problem of HMM learning theory; a significant improvement here could boost the performance of all HMM based pattern recognition systems. Therefore it is somewhat surprising that there appear to be relatively few papers devoted to this topic — the vast majority are devoted to applications of the HMM. In the next section we compare a number of alternatives to and variations of Baum-Welch in an attempt to find superior learning strategies. 2. Comparison of Methods for Robust HMM Parameter Estimation We focus on the problem of reliably learning HMMs from a small set of short observation sequences. The need to learn rapidly from small sets arises quite often in practice. In our case, we are interested in learning hand gestures which are limited to just 25 observations. The limitation arises because we record each video at 25 frames per second and each of our gestures takes less than one second to complete. Moreover, we wish to obtain good recognition performance from small training sets to ensure that new gestures can be rapidly recognised by the system. Four HMM parameter estimation methods are evaluated and compared by using a train and test classification methodology. For these binary classification tests we create two random HMMs and then use each of these to generate test and training data sequences. For normalization, we ensure that each test sequence can be correctly recognized by its true model; thus the true models obtain 100% classification accuracy on the test data by construction. The various learning methods are then used to estimate the two HMMs from their respective training sets and then the recognition performance of the pair of estimated HMMs is evaluated on the unseen test data sets. This random model generation and evaluation process is repeated 16 times for each data sample to provide meaningful statistical results. Before parameter re-estimation, we initialize with two random HMMs which should yield 50% recognition performance on average. So an average recognition performance above 50% after re-estimation shows that some degree of learning must have taken place. Clearly if the learning strategy can perfectly determine both of the HMMs which generated the training data sets, we would have 100% recognition performance on the test sets. We compare four learning methods 1) traditional Baum-Welch, 2) ensemble averaging
29 Classification Performance Averaged Over 16 Experiments True model 100% by construction.
0.95
3
0.9
H
i!
•i= rt
0.85
S 0)
0.8
§ £
0.75
o
t:
0 !
on
0-
0.65
rt!
sit
o
O)
0.6
o
0.55 0.5
Baum-Welch - 0 - Viterbi Path Counting - * - Entropic MAP -A- Ensemble Averaging 5 10 Number of training sequences
15
Fig. 2. Relative performance of the HMM parameter estimation methods as a function of the number of training sequences. Viterbi Path Counting produces the best quality models with a much smaller number of training iterations.
introduced by Davis and Lovell9 based on ideas presented by Mackay19, 3) Entropic MAP introduced by Brand6, and 4) Viterbi Path Counting10 which is a special case of Stolke and Omhundro's Best-First algorithm28. The results in figure 2 indicate that these alternate HMM learning methods all classify significantly better than the well-known Baum-Welch algorithm and also require less training data. The Entropic MAP estimator performs well but surprisingly the performance is much the same as simple ensemble averaging. Ensemble averaging involves training multiple models using the Baum-Welch algorithm and then simply averaging the model parameters without regard to structure. Note that for a single sequence, ensemble averaging is identical to the traditional usage of the Baum-Welch algorithm. Overall, the stand-out performer was the VPC algorithm. In these and other trials, this method converges to good models very rapidly and has performed better than the other methods in virtually all of our simulated HMM studies.
3. Video Gesture Recognition In an attempt to corroborate the strong performance of VPC compared to Baum-Welch on a real-world application, we test various learning techniques on a system for real-time video gesture recognition as shown in figure 3.
30 In earlier related work, Starner and Pentland 27 developed a HMM-based system to recognise gesture phrases in American Sign Language. Later, Lee and Kim 1 5 used HMMbased hand gesture recognition to control viewgraph presentation in data projected seminars. Our system recognizes gestures based on the letters of the alphabet traced in space in front of a video camera. The motivation for this application is to produce a way of typing messages into a camera-equipped mobile phone or PDA using video gestures instead of the keypad or pen interface. We use single stroke letter gestures similar to those already widely used for pen data entry in PDAs. For example, figure 3 shows the hand gestures for the letters "Z" and "W." The complete gesture set is shown in figure 6.
Fig. 3. "Fingerwriting:" Single stroke video gesture for letters "W" and "Z." Each video sequence comprises 25 frames corresponding to one second of video. Skin colour segmentation in YUV colour space is applied to locate the hand. Pre-processing (morphological) operations smooth the image and remove noise before tracking the hand with a modified Camshift algorithm 5 . After segmenting the hand, we calculate image moments to find the centroid in each frame. Along the trajectory, the direction (angle) of motion of each of the 25 hand movements is calculated and quantized to one of 18 discrete symbols. The resultant discrete angular observation sequence is input to the HMM classification module for training and recognition. We compare traditional Baum-Welch with the most promising alternative from the stimulated study, VPC. We evaluate recognition performance over all 26 character gestures using fully connected (FC), left-right (LR), and left-right banded (LRB) model topologies with the number of states ranging from 1 to 14. A LRB model is an LR model which has a transition structure containing self-transitions and next state transitions only (i.e., states cannot be skipped) as shown in figure 5. More formally, a^ ^ 0 V j = i or j = * + 1, and 0 otherwise, i, j e [1, N]. Our video gesture database contains 780 video gestures with 30 examples of each gesture. Recognition accuracy is evaluated using threefold cross-validation where 20 gestures are used for training and 10 for testing in each partition. These HMMs are initialized with random HMM parameters before using either Baum-Welch or VPC for learning. From figure 4 the best average recognition accuracy achieved is 97.31% when VPC is used for training, topology is LRB, and the number of states is 13. Although this corrobo-
31
Number of States 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mean Max
Baum-Welch FC LR LRB 80.00 80.00 80.00 72.69 94.23 93.85 66.54 92.31 96.15 80.00 84.80 85.38 75.20 81.20 90.77 75.60 84.80 85.77 77.60 86.40 89.62 76.80 86.00 89.62 77.60 85.60 90.00 76.00 81.60 88.46 65.20 86.80 89.23 74.80 86.80 88.08 84.80 84.00 90.00 72.80 81.60 88.46 75.40 85.44 88.96 84.80 92.31 96.15
FC 80.38 71.15 63.85 53.20 59.60 55.20 45.60 44.40 49.20 43.20 42.80 40.80 39.60 38.80 51.98 63.85
VPC LR 80.38 91.92 91.15 91.20 91.20 90.40 91.20 90.40 90.40 90.00 90.00 90.00 90.00 90.40 89.90 91.20
LRB 80.38 90.77 93.08 90.38 95.00 93.85 94.23 94.23 94.62 95.00 95.00 95.77 97.31 93.46 93.08 97.31
Fig. 4. Average percent correct recognition for all 26 video letter gestures against topology and training method.
0808 Fig. 5. Left-Right banded topology.
rates the stronger VPC performance exhibited in our simulated data performance trials, a closer investigation of Table 4 raises some doubts about this conjecture through the following observations. • The Baum Welch algorithm did almost as well as VPC with a best performance of 96.15% correct recognition with only 3 states. Moreover we achieve a very surprising 80% correct recognition with just a single state. • Topology (i.e., constraints on the initial value of the A matrix) has more impact on performance than the choice of learning algorithm. • Good recognition performance can be obtained over a very broad range of N, the number of states. 3.1. Comments on Learning Algorithm Performance We do not suggest that the above observations can be generalized to other real-world application domains but anecdotal evidence from other researchers suggests that similar be-
32
c 0 r* Rh J K L /N 0 p q s u \J 0) xf I
A6
im^""''€ '
Fig. 6. The alphabet of single-stroke letter hand gestures.
haviour is often encountered. When we designed this gesture system, we thought that this pattern recognition problem was quite challenging and would significantly differentiate learning strategies. Yet the surprisingly good performance over a number of learning algorithms, topologies, and a broad range N suggests that the problem is significantly easier than we suspected. Our intuition suggests that 3 states is far too small a number to adequately model all of these complex letter gestures, but results show that it is indeed possible to find a three state HMM which yields very good recognition performance. We conjecture that the observation matrix B seems to provide most of the recognition performance and that recognition may be only weakly affected by good estimation of the transition matrix A. In support of this idea, we may consider the following interpretation of the HMM. Consider each row of the B matrix as the probability mass function of the observation symbols emitted in a given state. In the limiting case of a single state HMM, the B matrix becomes a vector of source symbol probabilities and application of the forward algorithm for recognition is thus equivalent to the well-known and powerful MAP classifier. Indeed from figure 4, we see that this single state degenerate HMM can achieve 80% recognition performance. So sometimes even if the state transitions are poorly modelled, it is quite possible to find good classifiers based on source statistics. Now clearly if three states can yield strong performance, good HMMs with more than three states must also exist — a simple way to prove this is to note that we can always add additional states which are unreachable {i.e., transition probability of zero) without affecting recognition performance. This may help explain why performance stays much the same over a broad range of N as we increase N beyond three. The question that arises is, "Why does the Baum-Welch algorithm perform so well on real-world data despite its theoretical flaws and rather poor performance on the simulated HMM data?" Once again, a possible explanation is that this particular spatio-temporal recognition task is relatively easy, so all methods can do quite well. This conjecture may be
33
true for other common HMM applications and is a focus of current research. Unfortunately, unlike simulated data, the effort of gathering very large and diverse databases of real-world pattern recognition problems to evaluate the performance of different training algorithms is immense, so progress is slow. 3.2. Comments on Topology The FC topology allows transitions from any state to any other state and does not have a defined starting state. Being the most general topology, it is hardly surprising that performance is relatively poor. In this case we are required to search for a good solution in a parameter space of much higher dimensionality than for LR — so it would be much harder to locate globally optimal solutions. Gestures have a natural start and finish and thus it is reasonable to adopt the LR model as used in speech recognition to great effect. An even simpler topology is the LRB HMM where only self-transitions and next state transitions are allowed. Thus the A matrix is of the form: /oiil-onO ...0 0 a22 1 - a22 . . . 0
\ (1)
\0
0
0
...aNNJ
In this case the expected number of observations, n, (i.e., duration) in state i is simply given by24 n= — ^ . (2) 1 -an One can interpret the LRB HMM as being an adjustable clock that ideally synchronizes the changes in observation statistics with the changes in state to produce a time-variant MAP classifier. In many ways this HMM topology may be considered as a form of dynamic time-warping22 — a earlier technique used in speech recognition that has fallen out of favour since the advent of HMMs. The good performance of LR and LRB topologies on the gesture data set help make the point that simple HMM topologies often work best on real data. 4. Direct Calculation of HMM Parameters from Video Gestures A major dissatisfaction with the foregoing treatment of HMM learning strategies on real data is that the learning procedure is be treated like a black-box with little real insight into the learning process. In contrast to simulated data, we are unlikely to know the true HMM that originally generated the data — indeed real data is very rarely generated by a process that even remotely resembles a HMM. Thus we usually don't really know the best topology or number of states a priori — practitioners just try ranges of values and pick the one that yields good classification performance. Sometimes, as in the above example, this utilitarian method can find HMMs that are far too simple to accurately characterize the
34
state transitions of the underlying pattern. Alternatively, we may find HMMs that are far more complicated than is necessary. In either case this may lead to poorer generalization ability than might otherwise have been the case. To investigate this topic further, we devised two simple video gestures as shown in figure 7 where it is possible to approximately determine the form of the true HMM directly from the gestures themselves. In the case of the triangle gesture, this can be modelled by a 3-state HMM — each state corresponding to one side of the triangle. There is no need for skipped states to model this gesture, so the LRB topology is the most appropriate.
4.1. Analytic Calculation of A and B The triangle gesture is intended to be an equilateral triangle, so the expected number of observations in each state is equal and corresponds to n = 24/3 = 8. Thus the expected values of the transition matrix {A) parameters can be calculated from the duration equation (2). The triangle gesture consists of a horizontal movement at angle 0 degrees, followed by movement of 120 degrees and then -120 degrees for the other two sides of the triangle. To account for variations in the real gestures, one could use an error distribution (eg., Gaussian) centred on each of these angles as the expected value of each corresponding row of the B matrix. Similar analysis can be used to derive the expected form of the A and B for the square gesture.
35 4.2. Re-Estimation of A Matrix Using Baum-Welch After Correct Initialization Figure 8 show the calculated values for the A matrix for both gestures as well as the average value over 20 trials after Baum-Welch re-estimation using the calculated value as the initial value for A and a random matrix as the initial value for B. Clearly, the application of Baum-Welch re-estimation did not significantly alter the A matrix which implies that a local maximum was reached near the calculated value of A as expected. Nevertheless, when we used the traditional approach of initial random values for both A and B, we achieved values close to the calculated values only about 50% of the time. Thus even for these simple patterns, it seems that traditional application of the Baum-Welch algorithm with random initial starts has difficulty in finding the transition structure.
Triangle
A Matrix
Calculated
0.87 0 0
0.13 0.87 0
0 0.13 1
Average after BW re-estimation
0.87 0 0
0.13 0.87 0
0 0.13 1
0.83 0 0 0 0.85 0 0 0
Square 0.17 0 0.83 0.17 0 0.83 0 0 0.15 0 0.83 0.17 0 0.85 0 0
0 0 0.17 1 0 0 0.15 1
Fig. 8. Calculated A matrix and average values after 20 trials of Baum-Welch re-estimation for triangle and square gestures.
Figure 9 shows the performance of a number of alternate strategies to learning HMMs. We measure the sum of the squares of the differences of the elements between the A and B matrices obtained from the Baum-Welch algorithm and our reference matrices whcih are our best estimate of the correct HMM. In the case of the A matrix we use the calculated value from subsection 4.1 as the reference. For the B matrix, we could determine the reference B matrix in a number of ways (eg., histogram of observation symbols in each state, fitted Gaussian distribution, fitted Von Mises Distribution). Here we choose to use the histogram method. In methods 1 and 2, we keep A fixed at the calculated reference value and allow B to be re-estimated by Baum-Welch. In method 1, B is initialized to a random value, but in method 2 it is initialized to the reference value. Method 3 is traditional Baum-Welch where both A and B are re-estimated form random initial values. Finally, in method 4 we use Baum-Welch with both matrices initialized to the reference values. From this table it is quite clear that Baum-Welch is not very good at finding the true HMM parameters even if it is given the topology (A) through a calculated A matrix as in method 1. However when provided with good estimates for both A and B as initial values as in method 4, Baum-Welch converges to the given solution indicating that a local maximum has indeed been reached.
36
Method: Initialization 1. A fixed: A calc, B random 2. A fixed: A calc, B calc 3. A re-est: A random B random 4. A re-est: A calc B calc
A triangle 0 0 0.0708 2.13e-5
B triangle 0.4654 0.0066 0.8632 0.0059
A square 0 0 0.1314 8.19e-4
B square 0.7284 0.0698 1.437 0.0676
Fig. 9. Average squared error of matrices over 20 trials using different Baum-Welch re-estimation techniques and initial values.
4.3. Comments on HMM Learning The foregoing indicates that the Baum-Welch algorithm may have great difficulty in finding solutions that closely match the transition structure of the physical systems even for very simple examples. Nevertheless, it is clear that the resulting HMMs may still be extremely useful for pattern recognition. We conjecture that this may be because many real-world problems are adequately distinguished by their source statistics so the recognition problem is simpler that it would first appear. 5. Tools for Investigating HMM Parameters and Performance Such observed anomalies in HMM learning behaviour led us propose a methodology and tools to diagnose and analyze HMM pattern recognition systems. Too often researchers simply report "a HMM with x-states and y-symbols successfully classified the observation sequences". What does this mean? Is this really the case? Could it have been simply due to excellent discriminating observation attributes (the B-matrix) or a very strong memory model (A-matrix)? Since these matrices are typically not reported very little can be concluded on such issues. Further, a set of HMM models may well be sufficient for classifying a subset of observation sequences but if the A and B matrix probabilities demonstrate high uncertainty the classification performance can easily degrade as many different sets of observations can produce the same, often low, MAP scores. In order to be able to reproduce past results, examine model generalisations, and update model topology the model parametric values require interpretation. We7,20 have made some initial studies into such issues and, in particular, have developed some measures of performance of a given HMM that allows for model refinement and some objective measures of how the observation (B) and state (A) models contribute to performance. Consider the row-augmented matrix: C = A\B.
(3)
Each row represents all the information about a given state at time t in terms of expected state transitions and observation likelihoods. Consequently the rank of this matrix is critical in understanding the uniqueness or redundancy of states and observations. However, some properties of these states can already be deduced from the model components, per se. First, the Markov chain component. For a Markov process, a well known way of defining the distance between the current state transition probabilities and the steady state density function
37
(invariant pdj) is from the total of the left-handed "residual eigenvalues" of A (fires(A)) — the total of all the sub-dominant eigenvalues23. When all rows of the Markov Chain are identical the Markov Chain condition breaks down in so far as: P[St+1 \St] = P[St+1] = P[St] = 7T.
(4)
Consequently fj,res(A) = J2i=2 MiC^) provides a measure of how ergodic the process is and consequently the potential for generating variable state sequences as a function of the observation. Secondly, as the rows of the state dependent observation matrix, B, become more correlated, the evidence for a specific state from observations decreases. Similar to the A matrix steady state condition, observations do not evidence any state when the B matrix has only one non-zero eigenvalue, leading to: P[Ok\Si] = P[Ok}. In all then, using the singular values of C, a, the Inverse Condition Number (ICN) of C is:
(5) 13
where crmax is the largest singular value of C and amin is the smallest, is an appropriate normalized measure of the "HMM bandwidth" in so far as 7 _ 1 = 1.0 indicates that all states can be realized within the limits of the steady state probabilities and state dependent observations. However, 7 - 1 = 0 results in a "zero-bandwidth" HMM in so far as the process, on any experiment, does not provide any predictive information about state sequences except those provided by the prior or steady state conditions. By projecting individual rows of C into it's eigenspace we can then observe the redundancy of given states (also evident in the row correlation matrix) and consequently split and merge states to update the model topology to minimize model redundancy with respect to both state and observation parameters. This we found to be successful in the case of gesture and speech recognition experiments (see [20] for details). Given that these above techniques can be used to analyse and refine HMM model topologies — independent of any observation sequence — we now consider how the A and B parameters contribute to the prediction of state sequences given a model and observations and we show how Conditional Information2 can be used for this purpose. If the B matrix is unambiguous (for example, orthogonal with an ICN of 1.0) then a direct use of either Maximum Likelihood (ML: maxs{P[0(t)\S}}) or maximum posterior probability (MAP: maxs{P[S\6(t)] = P[0{t)\S]P[S}} would suffice to predict the most likely state at time, t. This condition would eliminate the need for the Markov component (A matrix) by use of a simple Bayesian (ML or MAP) classifier. Conversely, if the model B matrix is ambiguous (ICN of 0), we may as well dispense with the observation part and simply use the Markov component of the HMM to determine the most likely state sequence given the Markov model. Accordingly, we show how Conditional Information can be used to tease out the contributions of each component to the solutions for optimal state sequences.
38
Given a model and an input observation sequence two state sequences can be generated, one using the Viterbi algorithm with the entire HMM, Sv , and the other with a Bayesian classifier using only the B matrix (ML classifier) resulting in Sb. This latter condition assumes that the predictions at each time period are independent of all others — a condition consistent with the independence of observations over time for regular HMMs. Given the resultant two state sequences, Sv and Sb, respectively, we can calculate the following quantities: H(v\b)=H(v,b)-H(b)
(7)
where H(v, b) = - J2(p(sv
=i,Sb=
j)logP(Sv =i,Sb=
j))
(8)
and H(P) = - J2(P(sb
= J)logP(Sb = j))-
(9)
3
H(v\b) is the conditional entropy, and P(SV = i,Sb = j) is computed from the joint frequencies of the two state sequences. This measures the amount of information about the Viterbi solution given the Bayesian classifier solution. The residual information R(v\b) = H(v) - H(y\b)
(10)
provides a measure of how much information the A matrix, and the associated Viterbi algorithm, add to the complete optimal state sequence prediction. These measures provide a clear measure of the degree of information contributed by the observation and state models in the (MAP-based) performance of any HMM on a given data set. 6. Conclusions HMMs are an immensely powerful tool for solving pattern recognition and classification problems. Many studies demonstrate that it is a powerful technique, but few studies give any insight into why the performance is so good. It is well-known that the Baum-Welch algorithm is a hill-climbing technique that is generally unable to find global maxima. Yet the performance of pattern recognition systems based on Baum-Welch is often very good. Although we can find HMM training algorithms that appear to perform much better than Baum-Welch on simulated HMM classification data, these results do not necessarily translate into greatly improved performance in real-world applications. We conjecture that that is possibly because many important real-world problems have little need for highly complex HMMs. Also it appears that embedding knowledge of the topology of the system can often improve recognition performance more significantly than changes in learning algorithm. Finally we offer some tools and methodologies to analytically investigate and diagnose these problems.
39
7. Acknowledgements This paper covers a number of research themes currently being studied within our research groups and collaborators. I would like to acknowledge the contributions of Brendan McCabe, Peter Kootsookos, Richard Davis, Nianjun Liu, and Christian Walder. References 1. A A. Dempster, N. Laird, and D. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39(l):l-38, 1977. 2. R. Ash. Information Theory. Interscience Publishers, 1995. 3. V. Balasubramanian. Equivalence and reduction of hidden markov models. Technical Report A.I. Technical Report No. 1370, MIT Artificial Intelligence Laboratory, 1993. 4. L. Baum, T. Petrie, T. Soules, and N. Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov cahins. The Annals of Mathematical Statistics, 41:164-171,1970. 5. G. R. Bradski. Computer video face tracking for use in a perceptual user interface. Intel Technology Journal, Q2, 1998. last visited May 2004. 6. M. Brand. An entropic estimator for structure discovery. Advances in Neural Info. Proc. Systems, 11:723-729,1999. 7. T. Caeli and B. McCane. Components analysis of hidden markov models in computer vision. In Proc. of 12th International Conference on Image Analysis and Processing, pages 510-515. 8. Terry Caelli, Andrew McCabe, and Garry Briscoe. Shape tracking and production using hidden markov models. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), 15(1):197-221, February 2001. 9. R. I. A. Davis, B. C. Lovell, and T Caelli. Improved estimation of hidden markov model parameters from multiple observation sequences. In Proceedings of the International Conference on Pattern Recognition (ICPR2002), volume 2, pages 168-171, Quebec City, August 2002. IEEE. 10. Richard I. A. Davis and Brian C. Lovell. Comparing and evaluating hmm ensemble training algorithms using train and test and condition number criteria. Pattern Analysis and Applications, 6(4):327-336, February 2003. 11. Richard Dawkins. The Blind Watchmaker: Why the Evidence of Evolution Reveals a Universe Without Design. W. W. Norton, New York, 1996. 12. G. D. Forney. The viterbi algorithm. Proc. IEEE, 61:268-278, March 1973. 13. G. Golub and C. Van Loan. Matrix Computations, chapter 2.7, pages 79-81. The Johns Hopkins University Press, 2nd edition, 1989. 14. H. Hartley. Maximum likelihood estimation from incomplete data. Biometrics, 14:174-194, 1958. 15. H.-K. Lee and J. H. Kim. An hmm-based threshold model approach for gesture recognition. IEEE Trans on Pattern Analysis and Machine Intelligence, 21(10):961-973, October 1999. 16. S. E. Levinson, L. R. Rabiner, and M. M. Sondhi. An introduction to the application of the theory of probabilistic functions of a markov process to automatic speech recognition. The Bell System Technical Journal, 62(4): 1035-1074, 1983. 17. Nianjun Liu, R. I. A. Davis, B. C. Lovell, and P. J. Kootsookos. Effect of initial hmm choices in multiple sequence training for gesture recognition. In Proceedings of the International Conference on Information Technology, volume 1, pages 608 - 613, Las Vegas, Nevada USA, April 5-7 2004. 18. R.B. Lyngso, C.N. Pedersen, and H. Nielsen. Metrics and similarity measures for hidden markov models. In International Conference on Intelligent Systems for Molecular Biology, pages 178186, 1999.
40 19. D. J. C. Mackay. Ensemble learning for hidden markov models. Technical report, University of Cambridge, 1997. 20. B. McCane and T. Caelli. Diagnostic tools for evaluating hmm components. Technical report, Department of Computer Science, University of Otago, 2001. 21. G. McLachlan and T. Krishnan. The EM algorithm and extensions. Wiley series in probability and statistics. John Wiley and Sons, 1997. 22. C. S. Myers and L. R. Rabiner. A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical Journal, 60(7): 1389-1409, September 1981. 23. J. Norris. Markov Chains. Cambridge University Press, Cambridge, England, 1997. 24. L. R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-286, February 1989. 25. L.R. Rabiner and B.H. Juang. Fundamentals of Speech Recognition. Prentice Hall, New Jersey, 1993. 26. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379-423, July 1948. 27. T. Starner and A. Pentland. Real-time american sign language recognition. IEEE Trans, On Pattern Analysis and Machine Intelligence, 20:1371-1375, December 1998. 28. A. Stolke and S. M. Omohundro. Best-first model merging for hidden markov model induction. Technical report, International Computer Science Institute, April 1994. TR-94-003. 29. A. J. Viterbi. Error bounds for convolutional codes and an asymtotically optimal decoding algorithm. IEEE Transactions on Information Theory, IT-13:260-269, April 1967.
C H A P T E R 1.3 A N E W KERNEL-BASED FORMALIZATION OF MINIMUM ERROR PATTERN RECOGNITION
Erik McDermott & Shigeru Katagiri NTT Communication Science Laboratories, NTT Corporation, Kyoto 619-0237, Japan E-mail:
[email protected]
Previous expositions of the Minimum Classification Error framework for discriminative training of pattern recognition systems describe the use of a smoothed version of the error count as the criterion function for classifier design, but do not specify the origin and nature of the smoothing used. In this chapter we show that the same optimization criterion can be derived from the classic Parzen window approach to smoothing in the context of non-parametric density estimation. The density estimated is not that of the category pattern distributions - as performed in conventional non-discriminative methods such as maximum likelihood estimation - but rather that of a transformational variable comparing correct and best incorrect categories. The density estimate can easily be integrated over the domain corresponding to classification mistakes, yielding a cost function that is closely related to the original MCE cost function. The risk estimate formulation presented here provides a new link, Parzen estimation, between the empirical cost function calculated on a finite training set and the true theoretical classification risk. The classic Parzen concepts of linking the kernel width to the amount of training data can be used in the context of discriminative training to express the intuitive concept of using a smaller margin as the amount of training data increases.
1. I n t r o d u c t i o n 1.1. Towards
minimizing
classification
error
T h e Minimum Classification Error (MCE) framework is an approach to discriminative training for p a t t e r n classification t h a t explicitly incorporates classification performance into t h e optimization criterion 1 , 2 ' 3 ' 4 . Given discriminant functions for each category, M C E defines a loss function t h a t is a smoothed approximation of the recognition error rate and uses this function as t h e criterion function for optimization. T h r o u g h minimization of this criterion function, M C E is aimed directly at minimizing classification error rather t h a n at learning t h e t r u e d a t a probability distributions, the target of Maximum Likelihood Estimation (MLE). M C E has been 41
42
used successfully to train hidden Markov models (HMMs) for large-scale speech recognition tasks, as well as handwriting recognition and document classification tasks, yielding significant improvements over conventional MLE-based training and Support Vector Machines 4 ' 5,6 ' 7 ' 8 . Here we show that the continuous, 0-1 MCE loss function can be derived from an estimate of the theoretical classification risk, using Parzen estimation of the density of a suitably defined transformational variable. In this analysis, the specific kernel type used for Parzen estimation leads to a specific type of MCE loss function, and vice versa; the width of the Parzen kernel directly corresponds to the steepness of the MCE loss function, and vice versa. Minimization of the MCE loss function therefore corresponds to the minimization of a Parzen window based estimate of the theoretical classification risk. The well known convergence properties of Parzen estimates to the true densities, as the training set increases and the kernel width is reduced, can now be applied directly to the MCE framework. The reader is referred to a considerable literature on alternative approaches to discriminative training 9 ' 10 ' 11,12,13 , and in particular to the Maximum Mutual Information (MMI) approach that has been used extensively in speech recognition 14,15,16,17,18,19,20,21 . MMI and MCE, though they derive from fundamentally different backgrounds, are closely related, as discussed in several previous studies 3,22 . 1.2. Useful estimates
of classification
risk
The concept of estimating classification risk is not new 23 . However, in general, evaluating classification risk analytically is a difficult task. Classification systems typically create complex, highly non-linear decision boundaries in the pattern space; furthermore, the pattern space is usually high dimensional and often variable-length. Evaluating the risk integral can usually be performed only for low-dimensional spaces and simple classifier structures. As an alternative to attempting to evaluate the risk integral in the pattern space, smoothed versions of the misclassification count have been proposed by several authors 24,25 , but these proposals may have lacked practical applicability to general classifier design. The risk estimate formulated here can be seen as a smoothed version of the misclassification count, but it is explicitly linked to the original integral of classification risk in the pattern space. Furthermore, the risk estimate is a tractable function that can be used in practical gradient-based optimization techniques to find model parameters that minimize the risk estimate for general classifier structures such a HMMs or Multi-Layer Perceptrons 26,27,28 . Though this approach makes use of the "memory-based" Parzen window method to estimate classification risk, after the loss derivatives have been calculated for each training token, the kernel for that training token is no longer needed and can be discarded. In this sense the approach is very different from memory-based approaches to incorporating Parzen window estimation into machine learning architectures, such as the Probabilistic Neural Networks 29 , which in principle stores all the train-
43
ing tokens as kernel centers. 2. The Minimum Classification Error framework The MCE framework has been described in several publications 3,4 ' 30 . MCE uses a three-step definition, mapping each training pattern token x and the system parameters A (e.g., all the HMM means and covariances) to a continuous 0-1 loss function reflecting classification error. The pattern x could be a single pattern vector or a sequence of, e.g., speech-derived feature vectors, x = x^ = (x 1 , ...,x*, ...,x T ). Not limited to speech recognition or HMMs, the formalism simply assumes that discriminant functions
W*'A))=l
+ e -L i( x,A).
(!)
but can be any of a variety of continuous 0-1 functions. The total loss or empirical cost is the local loss summed over the training data X: ,
M
Nj
L(X,A) = -1£Yit(dj(Xij,A)), j
(2)
i=i
where xy denotes the i-th training token in category Cj, M the number of categories, and Nj the number of training tokens for category Cj. The overall loss function can be minimized using several different optimization methods 4 . 3. M C E for speech recognition In the context of speech recognition, the discriminant function typically used for MCE is defined in terms of a speech utterance x f and the corresponding best Viterbi state sequence 6 l = {8\, ...,6\, ...,8lT) for word string category Cj 5 : T gi(xJ,A)
T
i
= logPA(xf,e ) = 5>g°fli_ie{ + 5 > g & 8 j ( x t ) , t=i
(3)
t=i
where auv denotes a transition probability from state u to state v and bs(-) denotes a Gaussian mixture used to model the observation probability of a feature vector in state s. The effectiveness of MCE has been demonstrated on many speech recognition tasks. On the TIMIT task, MCE-trained HMMs yielded a phoneme classification accuracy of 77.6%, which is identical to the highest classification results reported for Support Vector Machines (SVM) 31 . In order to enable the use of SVMs on this
44
task, variable-length speech segments were converted to fixed-dimensional vector patterns 31 . The need for this conversion makes it more difficult to apply the SVM approach to general speech recognition problems, beyond the classification of short speech segments. In contrast, since the MCE loss function is defined using discriminant functions that embed Viterbi decoding applied to HMMs and arbitrary-length speech input, MCE can directly be applied at the utterance level for a variety of recognition tasks. Among many such results, the utility of MCE for practical speech recognition was illustrated by results for a "real-world", large-vocabulary, telephone-based task, where MCE was shown to yield striking improvements in recognition accuracy and system compactness compared to conventional MLE / Viterbi Training 6 . 4. A novel analysis of the smoothness of the M C E loss function The smoothness of the 0-1 MCE loss function has two important roles. First, it enables the use of gradient based optimization techniques. Second, it has a strong effect on generalization 32 . In this chapter we clarify the nature and meaning of the MCE loss function with regard to this second aspect. In particular, we show that the smoothness of the MCE loss function can be viewed as the direct consequence of using a well-understood type of smoothing, Parzen estimation, to estimate the theoretical risk. The width of the Parzen kernel used in this estimation is inversely related to the steepness of the MCE loss function. The minimization of the overall MCE loss function, summed over all the training tokens, is shown to be equivalent to the minimization of an estimate of the theoretical classification risk. The benefit for generalization of using a smooth MCE loss function is directly related to the benefit of using kernels to smooth over finite training data in Parzen estimation. 4 . 1 . Classification
risk & Bayes
decision
rule
Essentially, we show that the smooth MCE loss function can be derived from a particular approach to modeling - with a view to then minimizing - the true theoretical classification risk 33 : R = I R{a(x)\x)p(x)dx.
(4)
Jx Here a(x) represents the classification decision (the choice of one out of M categories) made for the input pattern x; R(a(x)\x) represents the risk involved in making specific classifications: M
i2(a i |x) = ^ A ( a i | C , ) P ( C i | x ) ,
(5)
where \(ai\Cj) denotes the cost of mis-classifying a member of category Cj as category d. Typically this cost is 1 whenever i and j are different - in other words, classifying a member of a given category as any other category incurs the same cost
45
of 1. It is easy to show 33 that the overall risk (Equ. (4)) is minimized by the Bayes decision rule: decide d if P(d |x) > P{Cj |x) for all j ^ i.
(6)
An important point is that achieving the optimal Bayes risk does not require explicit knowledge of the posterior probabilities P(Cj|x): knowledge of the category with the maximal P(Cj|x) is sufficient. Consistent with this observation, the MCE approach is to focus directly on designing a pattern classification system so that its decisions minimize the overall classification error. 4.2. Linking
the classification
risk to discriminant
functions
In the following, we start with the above definitions of risk and arrive at the MCE approach to classifier design. This derivation is different from previous expositions of MCE and links the empirical error measured on a given, finite training set, and the theoretical risk given above. The first step is to link Equ. (4) to an actual pattern recognition system. Using the expression for the risk incurred by an individual classification decision, given by Equ. (5), we can rewrite Equ. (4) as M
.
R = J2
A(a(x)|C J -)P(C J -|x)p(x)dx;
3= 1
(7)
J X
which in turn can be written as M
R = Y,
.
/ A(a(x)|C,-)p(C,-,x)dx.
(8)
The link between this expression of risk and an actual classification system is embodied by the decisions a(x) that the system takes. In a system where the classification decision is to choose the category with the largest discriminant function value <7j(x, A), we can express the categorical cost \(a(x)\Cj) as A(a(x)|C j ) = l((-
0).
(9)
Here we again assume that \(a>i\Cj) is 0 when i and j are the same, and 1 when they are different. The indicator function 1((— gj(x) + m a x ^ <7i(x)) > 0) compares the discriminant function value for category Cj with all other categories' discriminant functions; if Cj has the largest discriminant function value, the system will choose Cj and the indicator function 1((—^(x, A) + m a x ^ j ^ ( x , A)) > 0) is 0; otherwise the system does not choose Cj and the indicator function is 1. The discriminant functions are defined in a system specific way, that we leave open but that will in general depend on a number of modifiable parameters, A. We can now rewrite the overall risk in terms of the indicator function just introduced: M R
= Y,
r
l ( ( - 0 i ( x , A) +max 0)p(C,-,x)dx.
(10)
46
P(C)p(xlC)
a(x,A)=C2
a(x,A)=Ci
Fig. 1. Defining classification risk in a new domain. For each category, pattern vectors on the wrong side of the decision boundaries (shaded portions) are mapped to positive values in the one-dimensional transformed space.
4.3. Defining
classification
risk in a new, simpler
domain
The misclassification measure used in the following is: rrij = 4J(X, A) = — <7j(x, A) +max
(11)
The overall risk can then be written as M
R
JT f l(di(x,A)>0)p(CJ-,x)dx.
(12)
In turn, this is equivalent to integrating over a reduced part of the pattern space X: M
(13) where Xj is given by Xj = {x€X\
dj(x,A)>0}.
(14)
If X and gi(X,A),...,gi(X,A),...,gM{X,A) are continuous random variables, Mj = dj (X, A) is also a continuous random variable. We know that the cumulative distribution of a function of a random variable, Y — g(X), can be found by
47
integrating the density fx(x)
over the appropriate region, i.e.
fy(y)dy
P[Y
= P[g(X) <
J —(
fx(x)dx.
y}= I
(15)
Jx:g(x)
We can therefore express the integral for each category Cj over the reduced space Xj with an integral over the positive domain of the misclassification measure ray. P(Cj) [ p(x\Cj)dx. = P[dj(X,A)
> 0,Cj] = P(Cj) [
JXj
p{mj\Cj)dmj.
(16)
J0
Herep(x.\Cj) andp(m,j\Cj) could be written / x ( x | C j ) and fM(iTij\Cj) to emphasize that they operate in different spaces. Note that each nij corresponds to a distinct set of points x in Xj. For clarity, the dependence of rrij on the system parameters A is not explicit in the notation. We can now express the overall risk expressed in Equ. (13) as: R = Y.P(Cj) j=l
p(mj\Cj)dmj.
(17)
J
<>
This is a new expression for the classification risk that is equivalent to the original expression given in Equ. (4), given only the assumption that X and the gi(X,A) are continuous random variables. Rather than integrate over a part of the pattern space from which the "input" random variable X takes its values, this expression integrates over the corresponding part of the space for the transformed random variable Mj - a much easier task. Fig. 1 illustrates the new definition of risk. Estimate of classification risk: sum integrals (m>0) for each Parzen kernel
Parzen estimate of p(mlC): Sum kernels centered on (transformed) data points
kernel center c: transformed data point d(x,A)
x\ m = d(x,A) Fig. 2. Estimate of classification risk in a new domain. Parzen kernels used to estimate the density of the misclassification measure are integrated in the positive domain corresponding to classification errors.
48
4.4. Parzen
estimate
of risk
In order to explicitly relate the empirical cost to the theoretical cost, we use the following Parzen estimate of the density p(rrij\Cj): N •
PN^mjlCj)
= — ^
-^H—
Y~*—-)•
(18)
Here 4>{{mj — dj(xij,A))/h) is a window or kernel centered on the data point dj(xij,A) and with width h. The dependence on category Cj is reflected in the fact that the sum is only over data points x^- from category Cj. We know from Parzen window estimation theory 33 that the estimate given by Equ. (18) converges to the true density p(rrij\Cj) as the number of data points approaches infinity, reducing the window width h to zero as we increase the data set size. Furthermore, this property holds for a wide variety of window/kernel types, including Gaussian kernels and simple uniform kernels. With the estimate p ^ (rrij\Cj) defined for any value of rrij, we can define an estimate of the true overall risk expressed in Equ. (17): R
N = YJP{Cj) j=i
Expanding this using the estimate
pNj{mj\Cj)dmj.
PN^TUJICJ)
given by Equ. (18) yields
RN = J2 P{CJ) j f ^ £ ^ " ' f ^ W Rearranging, and using Nj/N
(19)
Jo
(20)
(where N = Y.f=i Nj) for P(Cj), gives:
Equ. (21) is an estimate of classification risk, defined in terms of Parzen estimates of the densities of category-dependent misclassification measures, and expressible as a sum of integrals of Parzen kernels over positive values of the misclassification measure. The estimate is anchored on actual (transformed) data points. The properties of Parzen estimation 33 entail that this estimate converges to the theoretical risk as the number of data points approaches infinity and the kernel width is reduced. Furthermore, we have some control over the value of RN via the system parameters A that are implicit in the definition of the misclassification measure, and that therefore affect the specific values of the window anchor points dj(xij,A). 4.5. Estimate
of risk for specific
kernels
We now show the utility of the Parzen window based estimate of risk (Equ. (21)) by considering simple Parzen kernel types. For these kernels, we can easily find the closed form of the integral in Equ. (21).
49 4.5.1. Uniform Parzen kernel leads to piece-wise linear MCE loss
PW=1HT)
P(n 0)
1/h
0.0 C
Fig. 3. Integrating Parzen estimates of misclassification measure density over the positive domain (corresponding to the part of the pattern space that is misclassified) yields, for each data point, 0-1 functions of the Parzen kernel center that are strikingly similar to typical MCE loss functions. A uniform Parzen kernel leads to a piece-wise linear estimate of classification risk.
W e first c o n s i d e r t h e o n e - d i m e n s i o n a l u n i f o r m k e r n e l ,
4>{u)
r i i f «| < 1/2, \ 0 ootherwise. t
(22)
(/>({rrij — dj (XJJ , A))//i) will be one if mj falls within the segment of length h centered at dj(x.ij,A), and zero otherwise. We can now integrate Equ. (21) for this specific choice of kernel. Focusing first on a single data point x ^ , and abbreviating d = dj(xij,A), we have
i r
(23)
hJo
The integral over m,j yields a piece-wise linear function that rises monotonically from zero to one. This corresponds to the classification risk for a single token x^-, given a Parzen kernel centered on the transformed data point dj(xij,A). Writing this expression as £i(), we can express the estimate of classification loss (21) as M
Nj
*" = ^ £ X > (<*;(*«.A))j=i.
(24)
50
Strikingly, this is identical to the empirical MCE cost defined earlier in this chapter (Equ. (2)), assuming the piece-wise linear MCE loss used in several MCE-based studies 4 . As the training set grows larger, the kernel width h should be decreased as per Parzen estimation theory, resulting in an increasingly steep ^i(-). As the training set size approaches infinity, h approaches zero, and t\{-) approaches the binary loss function. The transition from the integral of a single Parzen kernel over positive values of rrij to a function of the kernel center c = dj(xij,A) is illustrated in Figure 3. 4.5.2. Sigmoid MCE loss function corresponds to simple Parzen kernel Having shown how a simple uniform Parzen kernel leads to a simple piece-wise linear MCE loss function, we can now go the other way, start with an MCE loss function and ask what Parzen kernel it corresponds to. The most commonly used MCE loss function is the simple sigmoid function (Equ. (1)), with steepness controlled by a. Here we show that using this loss function for MCE corresponds to using, in Equ. (21), a specific Parzen kernel whose width is inversely related to a. Since we're looking for a kernel which integrates to Equ. (1), it will clearly have the form of the derivative of Equ. (1). One can verify that using the kernel *(«) = ( T ^
(25)
in the integral in Equ. (21) yields back the sigmoid loss. As above, the kernel is centered on the given data points dj (XJJ , A) and its width and height are controlled by a scalar h. Again abbreviating d = dj(xij,A), the integral for a single data point X7; o IS
1 f°°
xfrrij-d.
,
e^mi-V
f°° 1
drrij,
(26)
which yields, with h = 1/a, the MCE sigmoid loss function (Equ. (1)). The corresponding overall estimate of risk (Equ. (21)) then becomes identical to the empirical MCE cost (Equ. (2)) assuming, this time, the sigmoid loss function that has also been used in previous MCE-based studies. Figure 4 illustrates the relation between the sigmoid loss function and the corresponding Parzen kernel. 4.5.3. Gaussian kernel leads to classic error junction A common choice in Parzen estimation is the Gaussian kernel, 1
/
1 2
Using this in the integral in Equ. (21) yields a loss function based on the classic error function. Again abbreviating d = dj(xij,A), the integral for a single data
51 p(m) kernel:
2 4h P(m)=^(^)
(1+eu
P(m>0) =
Fig. 4. The commonly used MCE sigmoid loss function can be viewed as the probability that the misclassification measure is greater than zero, given a data point Xy mapped to a kernel center c = dj (XJJ•, A), for a particular type of Parzen kernel used in estimating the density of the misclassification measure.
point Xij is 1
frrij
—d
h
~h
)drrijJ
'~""
1 /m,\2
can
: exp
h J_d/h V2i
drrin
(28)
\ h
Given the definitions of erf(-) and its complement erfc(-), 2
erfc(x) = 1 — erf(x)
f00
= —= /
exp (—t2) dt,
(29)
we can see that Equ. (28) can be written as 1 f°° ,,mi-ds )d
h L ^-h- ^
,
1
„ / -d\
1
=2
[h^)
=2
GrfC
erf
/ d
[17=2
(30)
In this context, this result is very natural, since the classic error function is itself defined as the integral of a Gaussian. 4.6. Minimizing
the risk
estimate
Through the use of Parzen estimates of overall risk, and kernel types that allow us to integrate directly the general expression for risk estimate given by Equ. (21), we have arrived at expressions of empirical MCE cost that we are already familiar with, and that can be optimized in exactly the same way as the MCE cost. The MCE
52
derivatives described in previous studies 4 apply unchanged to the Parzen-based risk estimate. Though the derivative of Equ. (11) has discontinuities, these are not a significant obstacle to gradient-based optimization. Probabilistic Descent1, modified Newton's methods 4 and approximated re-estimation approaches 34 are among the optimization methods used so far for MCE. From the point of view of the Parzen analysis provided here, the task of any optimization procedure applied to the risk estimate (Equ. (21)) is to choose the system parameters A so as to position the kernel centers in a way that minimizes the estimated risk. Whether the resulting minimization corresponds to the theoretically optimal Bayes risk rests on the effectiveness of the optimization procedure, and of course on the model structure the number of parameters and other structural aspects relating to discriminative ability. 5. S u m m a r y The MCE formalism, as presented so far2A4,5,30; d e s c ribes the use for pattern recognition system design of a smoothed 0-1 loss function that approximates the ideal binary classification error. The fact that the MCE loss function is a smooth approximation of the ideal binary loss has been justified in previous work as 1) allowing the use of gradient-based optimization methods, and 2) performing an unspecified type of smoothing over available training data. Here we have shown that the MCE loss function can be seen as the direct result of a particular type of smoothing, the classic Parzen window estimation method, and the corresponding estimate of classification risk, in the domain of a suitably defined random variable. This variable, Mj = dj(X,A), embeds the potentially variable duration input pattern X, the system parameters A, and a comparison between correct and incorrect category matches. Parzen estimation, typically limited to modeling fixed-dimensional pattern vectors, can be applied in this domain to provide a simple expression of classification risk. Typical MCE loss functions £(dj(xij,A)) can be interpreted as the integral of a Parzen kernel of width h centered on dj(xij,A), the transformed version of the training token x ^ :
/(d,(x«,A)) = ^ £ % ( m j ~ ^ ' A W
(31)
The specific form of the kernel corresponds to the form of an MCE loss function, and vice versa. This analysis broadens the theoretical understanding of the MCE framework, and provides a novel link between the MCE empirical cost measured on a finite training set and the theoretical classification risk. References 1. S. Katagiri, C-H. Lee, and B.-H. Juang. New discriminative training algorithms based on the generalized descent method. In Proc. IEEE Worshop on Neural Networks for Signal Processing, pages 299-308, 1991.
53 2. S. Katagiri, C-H. Lee, and B.-H. Juang. Discriminative Multilayer Feed-Forward Networks. In Proc. IEEE Worshop on Neural Networks for Signal Processing, pages 309318, 1991. 3. S. Katagiri, B.-H. Juang, and C.-H. Lee. Pattern recognition using a family of design algorithms based upon the Generalized Probabilistic Descent method. Proceedings of the IEEE, 86(ll):2345-2373, 1998. 4. E. McDermott. Discriminative Training for Speech Recognition. PhD thesis, Waseda University, School of Engineering, March 1997. 5. W. Chou, B.-H. Juang, and C.-H. Lee. Segmental GPD training of HMM based speech recognizer. In Proc. IEEE ICASSP, volume 1, pages 473-476, March 1992. 6. E. McDermott, A. Biem, S. Tenpaku, and S. Katagiri. Discriminative training for large vocabulary telephone-based name recognition. In Proc. IEEE ICASSP, volume 6, pages 3739-3742, 2000. 7. A. Biem. Minimum Classification Error Training of Hidden Markov Models for Handwriting Recognition. In Proc. IEEE ICASSP, volume 3, 2001. 8. S. Gao, W. Wu, C.-H. Lee, and T.-S. Chua. A Maximal Figure-of-Merit Learning Approach to Text Categorization. In Proc. of ACM SIGIR Conference on Research and Development in Information Retrieval, pages 174-181, 2003. 9. T. Applebaum and B B. Hanson. Enhancing the discrimination of speaker independent hidden Markov models with corrective training. In Proc. IEEE ICASSP, volume 1, pages 302-305, 1989. 10. H. Franco and A. Serralheiro. Training HMMs using a Minimum Error approach. In Proc. IEEE ICASSP, volume 1, pages 357-360, 1990. 11. Y. Ephraim A. Ljolje and L. Rabiner. Estimation of hidden Markov model parameters by minimizing empirical error rate. In Proc. IEEE ICASSP, volume 2, pages 709-712, 1990. 12. H. Gish. A Minimum Classification Error, Maximum Likelihood, Neural Network. In Proc. IEEE ICASSP, volume 2, pages 289-292, 1992. 13. J. Hampshire. A differential theory of learning for efficient statistical pattern recognition. PhD thesis, Carnegie Mellon University, Department of Electrical and Computer Engineering, 1991. 14. L. Bahl, P.F. Brown, P. V. de Souza, and K. L. Mercer. Maximum Mutual Information Estimation of hidden Markov parameters for speech recognition. In Proc. IEEE ICASSP, volume 1, pages 49-52, 1986. 15. P. Brown. The acoustic-modeling problem in automatic speech recognition. PhD thesis, Carnegie Mellon University, Department of Computer Science, 1987. 16. D. Nahamoo A. Nadas and M. Picheny. On a model-robust training method for speech recognition. IEEE Transactions on Signal Processing, 36(9): 1432-1435, 1988. 17. P. S. Gopalakrishnan, D. Kanevsky, A. Nadas, D. Nahamoo, and M. Picheny. Decoder selection based on cross-entropies. In Proc. IEEE ICASSP, volume 1, pages 20-23, 1988. 18. A. Nadas P. S. Gopalakrishnan, D. Kanevsky and D. Nahamoo. An inequality for rational functions with applications to some statistical estimation problems. IEEE Transactions on Information Theory, 37(1):107-113, 1991. 19. Y. Normandin. Hidden Markov Models, Maximum Mutual Information Estimation, and the Speech Recognition Problem. PhD thesis, McGill University, Montreal, Department of Electrical Engineering, 1991. 20. V. Valtchev, J. J. Odell, P. C. Woodland, and S. J. Young. Lattice-based discriminative training for large vocabulary speech recognition. In International Conference on Spoken Language Processing, volume 2, pages 605-609, 1996.
54 21. P.C. Woodland and D. Povey. Large scale discriminative training of hidden Markov models for speech recognition. Computer Speech and Language, 16:25-47, 2002. 22. R. Schluter, W. Macherey, S. Kanthak, H. Ney, and L. Welling. Comparison of Discriminative Training Criteria. In Proc. ICASSP '98, pages 493-496, 1998. 23. G. J. McLachlan. Discriminant Analysis and Statistical Pattern Recognition. Wiley, 1992. 24. S. Amari. A Theory of Adaptive Pattern Classifiers. IEEE Trans, on Electronic Computers, EC-16(3):299-307, 1967. 25. L. Devroye, L. Gyorfi, and Gabor Lugosi. A Probabilistic Theory of Pattern Recognition. Springer Verlag, 1996. 26. E. McDermott and S. Katagiri. A Parzen window based derivation of Minimum Classification Error from the theoretical Bayes classification risk. In International Conference on Spoken Language Processing, pages 2465-2468, 2002. 27. E. McDermott and S. Katagiri. A new formalization of Minimum Classification Error using a Parzen estimate of classification chance. In Proc. IEEE ICASSP, volume 2, pages 713-716, 2003. 28. E. McDermott and S. Katagiri. A derivation of Minimum Classification Error from the theoretical classification risk using Parzen estimation. In Computer Speech and Language, 2004. 29. D.F. Specht. Probabilistic Neural Networks. Neural Networks, (3):109-118, 1990. 30. B.-H. Juang and S. Katagiri. Discriminative Learning for Minimum Error Classification. IEEE Trans, on Acoustics, Speech, and Signal Processing, 40(12):3043-3054, Dec. 1992. 31. P. Clarkson and P. Moreno. On the use of support vector machines for phonetic classification. In Proc. IEEE ICASSP, volume 2, pages 585-588, 1999. 32. B.-H. Juang and S. Katagiri. Discriminative training. Journal of the Acoustical Society of Japan, 13(6):333-339, 1992. 33. R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley Interscience Publications, 1973. 34. Q. Li and B.-H. Juang. A new algorithm for fast discriminative training. In Proc. IEEE ICASSP, volume 1, pages 97-100, 2002.
C H A P T E R 1.4 PARALLEL CONTEXTUAL ARRAY G R A M M A R S TRAJECTORIES
WITH
P. Helen Chandra Department of Mathematics,
J. A. College for Women, Periyakulam,
Theni, India
C. Martin-Vide Research Group on Mathematical Linguistics, Rovira i Virgili University, PL Imperial Tarraco 1, 43005 Tarragona, Spain
K.G. Subramanian Department of Mathematics,
Madras Christian College, Chennai, India
D.L. Van Department of Mathematics of Computer Sciences, Institute of Mathematics , NCST, 18 Hoang Quoc Viet Road,10307, Hanoi, Vietnam P.S.P. Wang College of Computer Science, Northeastern University, 360 Huntington Avenue CN, Boston MA 02115
#221
A simple and effective syntactic method of description of picture patterns of rectangular arrays of symbols is proposed. The method is based on contextual grammars of Marcus(1969) guided by trajectories in the generation process.The resulting grammar is called Parallel contextual array grammar with contexts shuffled on trajectories. This grammar is capable of describing pictorial patterns that cannot be handled by certain other contextual array grammars introduced in the literature.
1. I n t r o d u c t i o n Syntactic approaches, on account of their structure-handling ability, have played a significant role in t h e problem of description of picture p a t t e r n s considered as connected digitized, finite arrays of symbols 1 5 . Adapting t h e techniques of formal string language theory, various types of picture or array grammars have been introduced and investigated 1 , 2 ' 4 ' 5 ' 1 3 ' 1 4 . Most of t h e array grammars developed t o handle pic55
56
ture languages, are based on Chomskian string grammars. Another well-investigated class of grammars, called Contextual grammars, was proposed by Marcus 8 . A contextual grammar defines a string language by starting from a given finite set of strings and adjoining iteratively pairs of strings (called contexts) associated to sets of words (called selectors), to the strings already obtained. These contextual grammars 3 ' u are known to offer new approaches for a number of basic problems in formal language theory. But they have not received much attention regarding their suitability to describe picture patterns. Recently, extension of string contextual grammars to array structures has been attempted 7 ' 12 . In 6 , based on contextual grammars, a new method of description of pictures of digitized rectangular arrays, called a Parallel Contextual Array Grammar, is introduced. This model is different from 7 ' 12 . Several properties of these array grammars have been studied in 5 . On the other hand Martin-Vide et al 9 introduced the notion of adjoining contexts to words by shuffling them on 'trajectories' giving rise to a very general class of contextual string grammars. The notion of shuffle on trajectories was introduced in 10 , while studying the problem of parallel composition of words and languages. Motivated by these investigations, description of rectangular arrays by Contextual array grammars guided by trajectories, is considered here. Parallel Contextual array grammars with contexts shuffled on trajectories are introduced . These grammars are simple yet powerful in describing array structures. Comparison with certain other array generating models is done and closure properties under certain geometric operations are obtained.
2. Preliminaries Let V be a finite alphabet. V* is the set of all words over V including the empty word A. A picture A over V is a rectangular m x n array of elements of V of the form an
• • • a-\r.
A ^ml * ' "
^jnn
The set of all pictures or arrays over V is denoted by V**. A picture or an array language over V is a subset of V**. Definition 1: Let A = \an] „ , The column concatenation
B = \aa] „
A<&B of A and B is defined only when m = n and is
57
given by
A$B
an
aip b 1 1
Ami
*mp
• •
nq
= Ki
u
nq
Similarly, the row concatenation AQB of A and B is defined only when p = q. The empty array is also denoted by A, and A$P = P$A = P and AGP = POA = P for any P G V**. 3. Parallel Contextual Array Grammars We now recall the notions of array contexts and Parallel Contextual Array Gram-
Definition 2: Let V be a finite alphabet. A column array context over V is an element of V**$cV**of the form Ml
V\
(1)
$c
u2
_v2_
where u\, u2 are of size 1 x p and V\,v2 are of size 1 x q, for some p, q > 1 and is a special symbol not in V. If ui = U2 = A, then we write c = $ c
and if ^2
w
i = u2 = A, then c =
Ml
$c. A
TXJIO
array context over V is an element of
«2
V**$rV** of the form r = [ui u2] $ r [vi v2]
(2)
where ui,u% are of size p x l and v\, V2 are of size q x 1, for some p, g > 1 and $ r is a special symbol not in V. If u\ = U2 = A, then we write r = $ r [v\ v2] and if vi — v2 = A, then r = [ui u2\ %r • Definition 3: A parallel internal contextual array grammar is a system G = (V,B,Ca,Ra,(pc,ipr) where V is an alphabet, B is a finite subset of V** called the base of G, Ca is a finite subset of V**%CV** consisting of column array contexts, Ra is a finite subset of V**$rV** consisting of row array contexts, ipc : V** —* 2Ca and <pr : V** —> 2Ra are the choice mappings. When in Y, where X, Y G ^** if and only if X = XX$>X2§X3, Y = XX^W^X2^Z^X3 o r I = XiGXzGXs, Y = X1eWGX2ezeX3 for some Xi, X2, X3 G V**, and W, Z are contexts 'built' from the column/row array contexts
58
according to the choice mappings. We denote by =>*n, the reflexive and transitive closure of the relation =>»„ . Informally speaking, while 'building' W, Z from column array contexts, we imagine that a window is moved down the columns of the array X2 and W, Z are constructed from the column contexts as 'dictated' by the choice mappings. Likewise for W, Z 'built' from row array contexts. The language generated by G, denoted by L(G) is defined as L(G) = {Y € V**/ there exists X £ B such that X =^*n Y}. We write X =>cin Y (respectively X = ^ n Y) when column (respectively row) operations are performed on X to yield Y. The family of all parallel internal contextual array languages is denoted by PI AC. Example 4: Let G = (V,B,Ca,Ra,
be a parallel internal contextual array
aba B = < b bb > . Let B\, Bi, • • • , BQ be respectively
grammar where V = {a, b},
aba aba
b bb
aba
bbb , aba,
aba
ca = 1 Ra
a b
a b
ab
ba
bb
bb
bb
a a , a b, b a
a $c
= \ ab
fc [Si] =
aa
b •b>r
a $c
<j)r
b
b 5
b
a ab
5
$c
a
a
aa
a
q>r
aa
b~ , fc [B2] =
a $c
a
ba
i
I
$ r 1 b a 11 ,
V
a $c a , fc[B3} =
a a ^ V\{i B 5 ] =
ab
$r
aI
,
5
ipr
a a
a $c
[B3 ] =
a
ba
ba
The picture language L generated by G has pictures of the following forms
•
where 'a' stands for black square and 'b' stands for white square. Definition 5: A parallel external contextual array grammar is a system G = (V,B,Ca,Ra,ipc,<pr) where the components are as in Definition 3.
59 The direct derivation with respect to G is a binary relation =>ex on V** defined as X =>ex Y, where X,Y £ V** if and only if Y = W$X$Z or Y = WQXQZ, and W, Z are contexts as explained in Definition 3. We denote by =>*x the reflexive and transitive closure of the relation =>ex . The language generated by G, denoted by L(G) is defined as L(G) = {Y G V**/ there exists X € B such that X ^-*ex Y}. The family of all parallel external contextual array languages is denoted by PEAC.
Example 6: Let G = (V, B, Ca,Ra, ipc, ipr) be a parallel external contextual array bab grammar where V = {a, b}. B = <
r-
-i
aa
-1
$c
ab
b aJ ab aa
aa
<pr
aa ab
for all n > 1, for all m > 1, x We obtain the arrays of the picture language L generated by G which is the set of floor designs of the following forms
represented by
60
baba bab aaaaaaa bab
bababab
b ab a b a b
aaa
aaaaaaa
aaaaaaa
bab , bababab,
b ab ab ab aaaaaaa
bababab where 'a' stands for the primitive 'circle' and 'b' stands for blank. 3.1.
Necessary
conditions
Necessary conditions for a language to be of a certain type of parallel contextual array language is now considered. Notations: \X\C is the number of columns in array X. \X\r is the number of rows in array X. Definition 7: A language M C V** has the External Bounded Step (EBS) property if there is a constant p such that for each X e M, with \X\C > p, or \X\r > p, either of the following properties i),ii) holds: i)there is a Y e M such that X = LQY&R, and \LR\C < p (External Column Bounded Step property), ii) there is a Y € M such that X = UQYGD, and
U D
< p (External Row Bounded
Step property). Definition 8: A language M C V** has the Internal Bounded Step (IBS) property if there is a constant p such that for each X € M, with |X| c > p, or \X\r > p, either of the following properties i), ii) holds: i) there i s a F e M such that X = X1$L$X23>R$X3, Y = X1$X2$X3, and 0 < |LJJ|C < p. (Internal Column Bounded Step property) (ii) there is a Y € M such that X = X1QU9X2ODOX3,
Y = x1ex2ex3,&ndo< D
< p (Internal Row Bounded Step property)
Theorem 9: Every PEAC language has the EBS property. Proof: Let M € PEAC and G = (V,B,Ca,Ra,ipc,(pr) be a parallel external contextual array grammar generating M. Let pi = max {\X\C : X e B} , p2 = max {\X\r : X G B} , P3 = max {\uiVi\c} , where Ci
Ui
Vi
e
Ca,
Vi+l
P4 = max <
"
> where r,
Uj Uj +
i
Vj
Vj+i
eRa.
61
Note that IL.RI = \uivA„ if L =
u2
V2
,R
Ul
for some I and
Vl
if U = u\ U2 • • • Uk,D = v\V2 • • • Vk for some k. et p = max{pi,p2,p3,Pi}. If Z = [ p or \Z\r > p then Z £ B. Hence either Z — L$Z'$R for some L and R where Ui
L$cReipc(Z'),
Vi v
Ui+1_
_ i+l_
1 and Z' G M or Z = UQZ"GD Uj u
fc
$c
Ci
Vj- Vj
J + 1
Qi,l
, 1 < i < m-
Oi+1,1 • • • ®i+l,n
for some U and D where U$rD
G y?r(Z"),
1 < j < n - 1 and Z" G M.
G Vr
+ 1
^i,n
'''
Hence there is a constant p such that for each Z £ M, either for \Z\C > p, there exists Z' G M such that Z = L$Z'$R and 0 < \LR\C < p or for | Z | r > p, there exists Z" G M such that Z = UQZ"QD and 0 <
D
< p. Hence M has the EBS
a
property. Theorem 10: Every PI AC language has the IBS property. The proof is similar Remark 11: The converse of the theorem 9 is not true.
We notice this by the following counter example. Consider the language L consisting of the following arrays
{
~b b
aa
aaaa
aa
bb
> bbbb aaaa
5
bb
aa
aaaaaa ) bbbbbb aaaaaa
Obviously L satisfies the EBS property with p = 3. Let G = {B, Cc, Cr, (fic, (pr} be the PEAC grammar associated with L. Obviously B
bb
aa
aa
bb
bb
aa
Cr.=
a b
a $c
b
b
b 1
a
$c
a
62
Since
(Pc(Z2) =
a
aa
a
b
bb
b
a ~b~
aa
a
a 5c
y a
G L we have <^c(Zi) =
a a
whe
and Z 2 =
bb
bb aa
6 & It follows Yi
G L(G). But F x £ L.
a a 66
Definition 12: (i) A language M C V** has the Bounded Array Length Increasing {BALI) property if there is a constant p such that for each X G M, there is F € M with either -p < \X\c - \Y\C < p or -p < \X\r - \Y\r < p. Remark 13: Every PI AC language and PEAC language has the BALI
property.
Theorem 14: If L C V**,L G PI AC, then there are two constants p, q such that every Z £ L, \Z\c > p or \Z\r > p can be written in one of the following forms (i), (ii) : (i) Z = U§V$W§X§Y with U,V,W,X,Y G F**, 0 < \VX\c < q and U$Vi$W$Xi$Y G L for all i > 0; v (ii) z = UQvewexQY with [/,y,w,x,y e r < q and 0• < X UQV^WeX^Y G L for all i > 0. Proof: In order to prove (i) for a parallel internal contextual array grammar G = (V, B, Ca, Ra, (pc, <pr), we define p = max{\W\c, \W\r /WeB}, Ui
where cs —
Vi $c
Ui+1
. Vj Vj + i \ G
.
_Vi+l_
G Ca
Ra
If X = [aij]mXn G L(G) such that \X\C > p, or \X\r > p, then X £ B. When |X| C > p, or \X\r > p, there is a F g £(in X with Y = Xi$X2$X3, X = X1$L§X2i5>R$X3 or a Y G L(G) such that 7 =» in X with Y" = X x G ^ G X a , X = X i e L 0 X 2 0 i ? e X 3 . In the former case this means that there exist column array contexts c, G COJ 1 < i < m — 1, such that Ui
Vi
G
$c
3+1_
_vi+l_
*i,3 a
i+l,j
a.i,k
1 < j < k < n,
' ' ' 0,i+l,k
to form L and R with L$CR G in X i $ L i + 1 $ X 2 * i i i + 1 $ X 3 , for i > 0. All these arrays are in L(G). The argument is similar for (ii). •
63
Corollary 15: {anbncn)r L = < (bncnan)m
I n > 1, m > 1 } $ PI AC.
n n n
(c a b )r The fact that L is not in PI AC is a consequence of Theorem 14Theorem 16: If L C V**,L G PEAC, then there are two constants p,q such that every Z G L, \Z\C > p oi \Z\r > p can be written in one of the following forms (*).(") : (i) Z = U$Y$V with U,Y,V e V**,Q < \UV\C < q and U'QYQV* G L for all i >0 U (ii) Z = UQYQV with U,Y,V €V**,0 < < q and WGYQV1 G L for all V i >0.
The proof is similar Corollary 17: L = {{anbanban)m 3.2.
Closure
j n,m > 1,} £ P E A C .
properties
In this section, we state closure results for the classes PEAC and PI AC.
Theorem 18: (i) The families PEAC with or without choice and PI AC with or without choice are not closed under column and row concatenations. (ii)The families PI AC with or without choice and PEAC with or without choice are not closed under union. (iii) The families PI AC and PEAC are closed under reflection on the base and right leg and rotations by 90°, 180° and 270°. 3.3.
Comparisons
We now state results that compare the class PEAC with the classes LOC5 of local array languages and £(2RLG) of picture languages of 2-dimensional right linear grammars 5 . Theorem 19: The family PEAC and the classes LOC of local array languages are incomparable but not disjoint. Theorem 20: The family PEAC intersects with £(2RLG) generated by two dimensional right linear grammar.
5
of picture languages
64
Theorem 21: The family PEAC without choice and the family EAC of languages generated by external array column contextual grammar without choice 12 are incomparable but not disjoint. Theorem 22: (i) The family PEAC with choice intersects with the family EACC 12 . (ii) The family PI AC intersects with the family I AC 7 . (iii)The family PI AC with choice intersects with the family IACC 7 .(iv) The family PEAC intersects with the family EAC 7 .(v)The family PEAC with choice intersects with the family EACC 7 . Theorem 23: The emptiness problem and the finiteness problem are decidable for all classes of parallel contextual array grammars 4.
Parallel Contextual Array Grammars with Contexts Shuffled on Trajectories
We now introduce the notions of parallel shuffled contexts. Definition 24: A column shuffle array context over V is an element of V** of the form Ml
c =
where u\,U2 are of size 1 x k, k > 1 .
(3)
U2
A row shuffle array context over V is an element of V** of the form r = [u\ U2\ where U\,xi2 are of size k x 1, k > 1 .
(4)
Definition 25: Parallel Shuffled Contexts Let V be an alphabet. Let Ca be a finite subset of V** whose elements are the column shuffle array contexts of the form (3) and 2Ca- be a choice mapping. Given an array X = [a^] of size m x. n,
X =
flu
•• •
:
' •.
a
in
:
, aij
eV,
Ami * ' * Q,mn
we define ipc : V*
V** such that U e
U
u2 if Ci
Ui Ui+l
e ¥>c flj+1,1 ' ' '
a
i+l,n
Ci £ Ca, 1 < i < m — 1, not all c* need be distinct. U is called a parallel column context.
shuffled
65
Similarly we can define a parallel shuffled row context. Let Ra be a finite subset of V** whose elements are the row shuffle array contexts of the form (4) and <pr : V** —• 2Ra be a choice mapping. Given an array X = [ajj] of size m x n, we define <pr : V** -> V** such that U G y v W , ^
Ui Ui+i
G ¥V
=
ixi U2 • • • w n
if
, r; e i? a , 1 < i < n — 1, not all r» need be
distinct. 1/ is called a parallel shuffled row
context.
Definition 26: A (simple) parallel shuffled contextual array grammar is an ordered system G = (V,B,Ca,Ra,ipc,ipr) where V is an alphabet, B is a finite subset of V** called the base of G, Ca is a finite subset of column shuffle array contexts, Ra is a finite subset of row shuffle array contexts, ipc : V** —> 2Ca and ipr : V** —> 2Ra are the choice mappings. When tpc and <pr are omitted, we call G a parallel shuffled contextual array grammar without choice. The direct derivation with respect to G is a binary relation =4>UJ on V** and is defined as X =>LU Y, where X,Y e V** iSY G X LUC U or Y € X LLir U where U is a parallel shuffled column or row context. We denote by = ^ ^ the reflexive and transitive closure of the relation =>LUThe language generated by G, denoted by L(G) is defined as follows: L{G) = {Y e V**/ there exists X &B such that X = ^ Y}. The family of all parallel shuffled contextual array languages is denoted by VACS. Now we introduce the notions of trajectory on arrays, shuffle on trajectories and parallel contextual array grammars with contexts shuffled on trajectories. Definition 27: Let V\ = {I, r} , V2 = {u, d} be the sets of versors in the plane. The symbols l,r, u and d stand for the left, right, up and down directions respectively. A trajectory is an element t £ (Vj* U V2*). Notation: | X | c ( | X | r ) respectively denotes the number of columns (rows) in an array X. \t\x denotes the number of occurrences of x in t. Definition 28: Let V be a finite alphabet, t a trajectory, v a versor , v G Vj. and P,Q G V**. The column shuffle of P with Q on the trajectory vt, denoted by P LLl„t Q, is recursively defined as follows. If P = A$X and Q = B$Y where A,B,X,Y G V**, A and B are the first columns of P and Q respectively, t £ V{ then P L±JvtQ = {A$X) mvt(B$Y)
j A(X LL\t(B$Y)) if v = l = j ^ , k ^ 2 . , ; : .„ . B((A$X) UJtY) if v = r
(5)
66
If
If
P = X,
Q = \,
and
if
\l±Avt(B$Y)
A$X
B(\LUtY) '
LUvt\
A(X\JJtX)
Al_LJtA
(6)
if if
v =l
if
v=r
if
t =X
otherwise
(7)
(8)
The symbol 0 denotes the empty set. The row shuffle of P with Q on the trajectory vt, v G V2, t G V2*1S defined in a similar way except that I, r are replaced by u, d and $ by 9 . Also if \P\C j= \t\i or \Q\C ^ \t\r then P LUt Q = 0. Similarly, if \P\r + \t\u or \Q\r ^ |t| d then P LUt Q = 0. Definition 29: A parallel contextual array grammar with contexts shuffled on trajectories (PACST) is a construct G = (V,B,Ca,Ra,ipc,(pr,Tc,Tr) where the components are as in Definition 7, Tc is a family of sets T\ C V* of trajectories indexed over the subsets of column array contexts and Tr is a family of sets T2 C V% of trajectories indexed over the subsets of row array contexts. When ipc and ipr are omitted, we call G as a parallel contextual array grammar with contexts shuffled on trajectories without choice. When Tc or Tr consists of exactly one set Ti or T2 respectively corresponding to Ca or Ra, we write Tc = T\ or Tr = TiThe direct derivation with respect to G is a binary relation => on V** and is denned as follows: X =>LUT Y, where X, Y £ V** iff Y = X LUt U , t G Tx for some Ti € T c , where t/ is a parallel shuffled column context. X ^U-J-ZV y, where X, y G V** iff F = X LUt C/ , i G T2 for some T2 G T r , where C/ is a parallel shuffled row context. X ^ U J T Y iff y = X LUTo C7 or y = X LUTr. J7. We denote by =»^j T the reflexive and transitive closure of the relation =^LUT • The language generated by G, denoted by L(G) is defined as follows: L(G) = {Y G F** / there exists X G P such that X = ^ ^ j r y }. The family of all languages generated by PACST is denoted by VACT. We illustrate with an example. (bnanbn)m Example 30: Let V = {a, b} , and L — < (6 3 ") m
/m,n>l
^ (6"a"6") m One can easily see that L is generated by the following bab
PACST,
G — (V, B, Ca,Ra,
are
67
b ab
bb b
b ab
bbb
hb b,
b ab,
b ab,
bbb.
where r\,r2,r3, r^ are Ra — {ri,r2,r3,rA} n n ba ab bb a a bb bb bb bb b a,
ab,
bb,
a a.
n n n
bab
bnbnbn
=
ci,
=
c3,
fc
bnanbn
C2,
= C4, for all n > 1.
bnbnbn (ab),
"(6a))m = rx,
tpr
(bb),
_(ba))m
(ab)
'(bb))m
(aa) m
(bb) ')m Tc
=
bnanbn bnbnbn
bnanbn (bb), )m
bnbnbn
(bb) o))m m\ 2n 2 n = {rl r l
^3,
V
(bb)m
\_(aa)m 771 I n > 1} and
=
r2,
= r4, for all m > 1. T r = {du 2 m d 2 u m / m > 1}
Figure 1 Tokens H where black square is 'a' and white square is 'b'. 5. VACT and some families of Parallel Contextual Array Languages The parallel contextual array grammars with contexts shuffled on trajectories introduced here have more generative power than the parallel external and internal contextual array grammars considered in 6 . Theorem 31: (i) A language L is a parallel external contextual array language if and only if it is generated by a PACST, G = (V, B, Ca,Ra,ipc, <pr,Tc, Tr) such that rml*rn £ Tc for some m, n > 0 and dpu*d9 £ Tr for some p, q > 0. Hence the family of parallel external contextual array languages 6 is strictly contained in the family
68
PACT, the trajectories being regular languages. (ii) A language I is a parallel internal contextual array language if and only if it is generated by a PACST grammar G = (V,B,Ca,Ra,(pc,(pr,Tc,Tr) such that m n p q l*r l*r l* G Tc for some m,n > 0 and u*d u*d u* G Tr for some p, q > 0. Hence the family of parallel internal contextual array languages 6 is strictly contained in the family PACT, the trajectories being regular languages. (Hi) A language L is a parallel (simple) shuffled contextual array language if and only if it is generated by a PACST grammar G = (V,B,Ca,Ra,ipc,(pr,Tc,Tr) such that Tc contains {l,r}* and Tr contains {u,d}*. Hence the family PACS of all parallel shuffled contextual array languages is strictly contained in the family PACT, the trajectories being regular languages. Proof: We prove (%). The other statements are similar. Assume that L is generated by the PEAC grammar G' = (V, B, C, R', (p'c, (p'r). Construct the PACST grammar G = (V, B, Ca,Ra, r,Tc, Tr) as follows: Vi
For each c'- =
G C, the set Ca contains the element
2/i+i Vi _xi+l
Vi+1 _
If
Vi
G V'c
$c
3+i_ c^eC, l
Vi
a
e
i,l
fc
•'
Xi
Hence for each L $r R =
-
a
i,k
2/1 2/2
X2 $c
xi
G
Vh
Xh
column context L $ R =
O-iM
O-i+1,1 • • • Oi+l.fc
j/i
X2 2/2
e tpc{X).
Xh Vh
Let rml*rn G T c , where \L\C = \x, ich r[ — Xi
Xi^,\
Vi 3/t+i
^i *^i+l
%r
and |i?| c = |j/i|c = n. [Vi 2/i+i G R', the set i? a contains the element
If 2R' is defined by a
a
l,i
X{ ^i-j-1
l,»+l
e v»;
Vi Vi+i
a
/i,i a/i,t+i
r< G R', 1 < i < k - 1, for any X = [ay] h x f c G V*; define
a
l,t+l
G VV
_2/i 2/i+i J Hence for each U $ r D = j j i 2 • • • i t yiV2 there exists a parallel shuffled column context
•••
Vk
G *?/(*),
Vi 2/2 ••• 2/fc J Let dpu*dq G T r , where |f/|r = |a;,|r = p and \D\r = |j/j| r = Tr — dpu*dq, where p, q > 0, then we define a PEAC grammar G' = (V,B,C',R',
Xj
Vi 1
Vi+i_
. *+ . X%
Vi
G Ca such that \xi\c = m, \yi\c = n. .
Ci =
_%i+i
Vi+i
Similarly R' contains the element r\ X{
where
$c x
Xi-^-i
Xi Xi+i
$r
G -Ra such that \xi\ = p and |j/i| = q.
Vi Vi+i
If
Oi,fc
Vi G
oc>+1 2/i+i_
i+1,1
C j € C 0 ) 1 < i < h - 1, for any X = [a ij ] /iXfc G V**, define <^ : V** -> 2 C ' such that Xj
9>c _Xi+i_
Vi Vi+l
a
y.
e ^
j/i+i_
i,l
_ai+l,l
If <pr : 7** -> V** is denned by " &l,i Ol.i+1 G (fir
•'
-
a
i,k
• • • Oj+l.A:
where
70 ri £ Ra,
l < i < k - l , for any X = [a»j] hxfc € V**,
define ip'r : V** - * 2R> such t h a t a
l,i
%i ^ i + 1
a
l,»+l
3>r 2/< J/i+i _0*h,i ah,i+l_
Again it is easy t o see t h a t L(G')
= L(G).
•
T h e o r e m 3 2 : T h e family VACT is closed under reflection on t h e base a n d right leg and rotations by 90°, 180°, 270°. 6.
Conclusion
T h e extension of trajectory guided contextual string g r a m m a r s t o picture generation considered here is a first step in this direction. Other properties of this array g r a m m a r model remain t o be examined. T h e power of trajectories, for example, is one such problem worth examining. 3D picture p a t t e r n s have also been investigated in the literature 16 > 17 . Extension of this study t o 3D pictures is also worth examining. References 1. H. Bunke and A.Sanfeliu,(World Scientific, 1990). 2. C. Cook and P.S.P.Wang, Computer Graphics and Image Processing 8(1978) 3. A. Ehrenfeucht, Gh.Paun and G. Rozenberg, In Handbook of Formal Languages, Vol.2. Eds. G. Rozenberg and A. Salomaa, Springer-Verlag, (1997). 4. K. S. Fu, Prentice-Hall, (1982). 5. D. Giammarresi and A. Restivo, In Handbook of Formal Languages Vol.3. Eds. G. Rozenberg and A. Salomaa, Springer Verlag, (1997). 6. P. Helen Chandra, K.G. Subramanian and D.G. Thomas, presented at the Nineth International Workshop on Combinatorial Image Analysis(IWCIA), Palermo, Italy, May (2003). 7. K. Krithivasan, M.S. Balan and R. Rama, In Recent Topics in Mathematical and Computational Linguistics, Eds. C. Martin-Vide and Gh. Paun, (2000). 8. S. Marcus, Rev. Roum. Math. Pures et Appl. 14 , 10 (1969) 9. C. Martin-Vide, A. Mateescu, G. Rozenberg and A. Salomaa , Int. J. Computer Math. 73 (1999) 10. A. Mateescu, G. Rozenberg and A. Salomaa, Theoretical Computer Science, 197 ,1-2 (1998) 11. Gh. Paun , (Kluwer Academic Publishers, Dordrecht, Boston, London, 1997). 12. R. Rama and T A . Smitha, Int. J. Pattern Recognition and Artificial Intelligence 14 (2000) 13. A. Rosenfeld, ( Academic Press, 1979). 14. A. Rosenfeld and R. Siromoney, Languages of design, 1 (1993) 15. P.S.P.Wang, IEEE-PAMI 5,1 (1983) 16. P.S.P. Wang, " IJPRAI8, 2 (1994) 17. P.S.P. Wang, In Handbook of Pattern Recognition and Computer Vision, ed by C.H.Chen, L.F.Pau, and P.S.Wang, (WSP 1999) .
C H A P T E R 1.5
PATTERN RECOGNITION WITH LOCAL INVARIANT FEATURES
C. Schmid , G. Dorko , S. Lazebnik , K. Mikolajczyk
and J. Ponce
1
INRIA Rhone-Alpes, GRAVIR-CNRS 655, av. de I'Europe, 38330 Montbonnot, France Dept. of Computer Science and Beckman Institute University of Illinois, Urbana, IL 61801, USA
Local invariant features have shown to be very successful for recognition. They are robust to occlusion and clutter, distinctive as well as invariant to image transformations. In this chapter recent progress on local invariant features is summarized. It is explained how to extract scale and affine-invariant regions and how to obtain discriminant descriptors for these regions. It is then demonstrated that combining local features with pattern classification techniques allows for texture and category-level object recognition in the presence of varying viewpoints and background clutter.
1. I n t r o d u c t i o n Local photometric invariants have become more and more popular over the past years. This is due to (i) their locality which permits recognition in the presence of occlusion and clutter, (ii) their distinctiveness due to the use of photometric information and (iii) their invariance which makes t h e m stable under image transformations and illumination changes. This is in contrast to classical recognition approaches which were mostly based on global photometric information or geometric features. Color histograms 3 7 and eigenimages 3 8 are examples of recognition methods based on global photometric information. T h e y require segmentation in the presence of clutter and are not robust to occlusions. Furthermore, they are not invariant to image transformations. However, these methods can discriminate due to the use of photometric information and they perform very well in constrained settings. Recognition based on geometric features, on the other hand, does not require segmentation and is invariant to image transformations 3 0 . However, the features are not very discriminating and are sensitive to noise and errors in the extraction process. T h e idea of combining the distinctiveness of photometric information with the locality and invariance of geometric features has led t o t h e development of local pho-
71
72
tometric invariants. An initial solution was presented by Schmid and Mohr 35 . They describe the image with a set of rotation-invariant descriptors computed at automatically extracted interest points. A multi-scale framework makes the description invariant to similarity transformations. This description combined with a voting scheme and neighbourhood constraints allows for excellent recognition results. The approach is, however, limited to similarity transformations and only images of the same object or scene can be recognized. A solution is possible due to the following extensions: (i) the extraction of affine-invariant photometric descriptors and (ii) the use of pattern classification techniques. Section 2 describes the extraction of local invariant features and their application to recognition in the presence of viewpoint changes. Section 3 and Section 4 demonstrate that combining local features with pattern classification techniques allows for texture and category-level object recognition in the presence of varying viewpoints and background clutter. 2. Extraction of local invariant features This section presents the extraction of local invariant features. Sections 2.1 and 2.2 present scale and affine-invariant detectors and in Section 2.3 different descriptors are compared. An application to recognition is presented in Section 2.4. 2.1. Scale-invariant
regions
Most scale-invariant detectors search for maxima in the 3D scale-space representation of an image (x, y and scale). They differ mainly in the differential expression used to build the scale-space representation. Crowley 6 detects local features in an image pyramid. Lindeberg 16 searches for 3D maxima of the scalenormalized Laplacian-of-Gaussian (LoG). The LoG operator detects blob-like structures. Lowe 19 uses the difference-of-Gaussian (DoG) to approximate the LoG. This results in an efficient algorithm for extracting local 3D extrema in scale-space. Kadir and Brady n use local complexity as saliency measure. They compute the entropy of greylevel intensities over a neighborhood and then search for entropy maxima in scale and location. Our approach, the Harris-Laplace detector 21>24, selects a complementary type of regions: corners and regions of "high information content" (cf. Figure 9). The detector first extracts interest points at multiple scale levels which are the local spatial maxima of the scale-adapted Harris function. It then uses the Laplacian for scale selection 16 . Scale selection determines the characteristic scale of a local structure, i.e. the scale with maximum similarity between the feature detection operator and the local image structures. This scale is determined as the extremum over scale of a given function (see Figure 1) and is independent of the image resolution. Note that there might be several extrema, that is several characteristic scales corresponding to different local structures centered at one point. The Laplacian operator is used for scale selection since it gave the best results in the experimental comparison in 21 .
73
(")
(b)
(c)
(d)
Fig. 1. Example of characteristic scales. (a),(c) images with different resolutions; the radius of displayed regions is equal to 3 times the characteristic scale. (b),(d) responses of the normalized LoG over scales for (a),(e). The characteristic scales are 10.1 and 3.89 for (a) and (c), respectively. The ratio of scales corresponds to the scale factor (2.5) between the two images.
Initial multi-scale interest points are rejected if the Laplacian attains no extremum at the scale of extraction or if the Laplacian response is below a given threshold. We then obtain a set of scale-invariant points with associated scales. Figure 2 presents an example of points detected with Harris-Laplace. Images (a) and (b) show points detected with the multi-scale Harris detector; the radius of displayed regions equals 3 times the detection scale. Points corresponding to the same local structure are selected manually. Note that interest points, detected for the same image structure, change their location relative to the detection scale in the gradient direction. Images (c) and (d) show the points selected by the Laplacian. Note that two or more points can be selected, as several local maxima might exist in scale-space. We can see that the location and the scale of the points correspond to the transformation between the images.
(a)
(b)
(c)
(d)
Fig. 2. Scale-invariant interest point detection, (a), (b) Initial multi-scale Harris points corresponding to one local structure (selected manually), (c), (d) Scale-invariant interest points selected with the Harris-Laplace detector.
An experimental evaluation of the Harris-Laplace detector for images of real scenes 2 1 , 2 4 has shown a very good performance up to a scale factor of 4. The performance was measured by the repeatability rate, that is the percentage of points detected at the same relative position and with corresponding regions, as well as by the performance in the context of recognition. 2.2. Affine-invariant
regions
The Harris-Laplace detector is not invariant to significant affine transformations. Figure 3 shows Harris-Laplace regions in black and the corresponding regions pro-
74
jected with the affine transformation to the other image in white. The regions detected with Harris-Laplace do not cover the same part of the affine transformed image. Affine-invariant region extraction is therefore required.
Fig. 3. Scale-invariant interest point detection for an image pair related by an affine transformation. The regions detected with Harris-Laplace in black and the corresponding regions projected with t h e affine transformation to the other image in white.
Alvarez and Morales 2 proposed an affine-invariant algorithm for corner detection which uses affine morphological multi-scale analysis. Their approach can only be applied to perfect corners. Lindeberg and Garding 17 presented a method for finding blob-like affine features. Their approach first extracts maxima of the normalized Laplacian in scale-space and then modifies iteratively the scale and local affine shape of the regions. Local affine shape is determined with the second moment matrix. Baumberg 4 as well as Schaffalitzky and Zisserman 3 3 use the local affine shape estimation to adapt the point neighbourhood of multi-scale Harris points and Harris-Laplace points respectively. Tuytelaars and Van Gool 39 combine Harris points with nearby edges. An affine-invariant parallelogram region is determined by a Harris point and one point on each of two nearby edges. They also proposed a purely intensity-based method. It is initialized with local intensity extrema. For each extremum the algorithm finds significant changes in the intensity profiles along rays going out from the extremum. An ellipse is fitted to the region defined by the locations of these changes. Matas et al. 20 introduced maximally stable extremal regions (MSERs). An extremal region is a connected component of pixels which are all brighter or darker than all pixels on the region's boundary. These regions are invariant to affine transformations as well as to monotonic intensity transformations. Our approach, the Harris-Affine detector 2 2 ' 2 4 extends the Harris-Laplace detector by estimating the local affine shape based on the second moment matrix 17 . The second moment matrix describes the local image structure and is defined by the gradient distribution in a point neighbourhood: Mx ? <xj5 aD) = al g(aM) * [ / j ( ? * " > , L*%<* *"> } [LxLy(x,(TD) L2(X,CJD) J The local derivatives are computed by convolution with Gaussian derivatives of scale |
75
eigenvalues of the second moment matrix determine the affine shape of the point neighbourhood. Affine normalization projects the affine pattern to the one with equal eigenvalues, i.e. uses as transformation the square root of the second moment matrix. See Figure 4 for illustration. The normalized regions are isotropic in terms of the second moment matrix and are related by a simple rotation. Rotation preserves the eigenvalue ratio for an image patch and the affine deformation can therefore be determined up to a rotation factor.
(a)
(b)
(c)
(d)
Fig. 4. Affine normalization with the second moment matrix: (a), (c) initial images with the affine shape matrices, (b), (d) normalized images. The normalized images are related by a rotation.
In practice the second moment matrix as well as the characteristic scale change if the patch is transformed and therefore need to be re-estimated iteratively. Initial points are obtained by the multi-scale Harris detector. For each point we iteratively estimates location, scale and local shape. Figure 5 shows points and regions detected in consecutive steps of the iterative procedure. For this example, the location, scale and shape of the point do not change after four iterations. We stop iterating when the second moment matrix fj, (of the transformed patch) is sufficiently close to a pure rotation. This is measured by the similarity of the eigenvalues Amoa.(/x) and Amiw(p)? i-e. A™*"|»\ > 0-95 for our experiments. Another important point is to stop in the case of divergence. In theory there is a singular case when the eigenvalue ratio tends to infinity i.e. on a step-edge. Therefore, the point should be rejected if the ratio of the eigenvalues is too large (i.e. bigger than 6), otherwise it leads to unstable elongated structures. The convergence properties of the shape adaptation algorithm are extensively studied in 17 . It is shown that except for the singular case the point of convergence is always unique. In general the procedure converges to the correct solution provided that the initial estimate of the affine deformation is sufficiently close to the true deformation, and the scale is correctly selected with respect to the size of the local image structure. Figure 6 presents an example of points detected with Harris-Affine. Images (a) and (b) show the multi-scale Harris points used for initialization, in black the region selected by the Harris-Laplace detector. Images (c) and (d) show the points and regions after an iterative estimation for each point in (a) and (b). Note that the points in (a) and (b) which correspond to the same physical structure, but are detected at different locations due to scale, converge to the same point location and
76
Initial
1
2
3
4
Fig. 5. Iterative detection of an affine-invariant interest point. T h e left image shows the point used for initialization. The consecutive images show the points and regions after iterations 1, 2, 3 and 4.
region. We can see that the method converges correctly even if the location and the scale of the initial point is relatively far from the point of convergence. It is straightforward to identify these points by comparing their location, scale and local shape and to select one of them.
(a)
(b)
(c)
(d)
Fig. 6. Affine-invariant interest point detection : (a), (b) Multi-scale Harris points/regions (in black - Harris-Laplace). The radius of the circles is three times the detection scale. (c),(d) Points and affine regions obtained with the iterative algorithm applied to points in (a) and (b). Note that points representing the same structure converge to the same solution.
An experimental evaluation of the Harris-Affine detector for images of real scenes 2 2 , 2 4 has shown a very good performance up to a viewpoint change of 70 degrees. Harris-Laplace shows a similar performance up to 30 degrees of viewpoint change. For larger viewpoint changes the performance of Harris-Laplace decreases rapidly. 2.3.
Descriptors
To obtain local invariant features, the extracted and normalized regions have to be described. Many different image descriptors have been developed. They should be distinctive and at the same time robust to viewpoint changes as well as to errors of the detector. In the following we have selected a few promising descriptors and compare them in the context of recognition. Note that the normalized regions are not rotation-invariant. To eliminate rotation our comparison uses rotation-invariant descriptors. A simple way to obtain rotation invariance is to orientate the patch in the direction of the dominant gradient.
77
The compared descriptors are presented in the following. SIFT descriptors describe the gradient distribution in a region by a 3D histogram of location and gradient orientation. The quantization of location and gradient orientation makes the descriptor robust to small geometric distortions and small errors in the region extraction. Steerable filters 10 steer derivatives in a particular direction. Steering in direction of the gradient orientation makes them invariant to rotation. Differential invariants 9 are combinations of image derivatives which are rotation-invariant. Complex filters 3 3 differ from the Gaussian derivatives by a linear coordinates change in filter response space. Moment invariants 4 0 characterize the shape and the intensity distribution in a region. Cross-correlation for a sub-sampled image patch is used as a baseline descriptor. Our evaluation criterion is the ROC (receiver operating characteristics) of the detection rate for the query image with respect to the false positive rate in a database of images. Detection rate is the number of correctly matched points with respect to the number of possible matches. False positive rate is the probability of a false match in the database of descriptors, that is the total number of false matches with respect to the product of the number of database points and the number of query image points. The similarity threshold between descriptors is varied to obtain the ROC curves. The results are displayed for a false positive rate up to 0.012, that is each point from the query image matches at most with 1.2% of points in the database. The threshold is usually set below this value; otherwise the number of false matches is too high to permit reliable recognition. Figure 7 compares the performance in the presence of an affine transformation between image pairs for which the viewpoint of the camera is changed by 60 degrees. This introduces a perspective transformation which can be locally approximated by an affine transformation. There are also some scale and brightness changes in the test images. To eliminate the effects of the affine transformation, we use the HarrisAffine detector. The descriptors are computed on point neighborhoods normalized with the locally estimated affine transformations. SIFT descriptors perform better that the other ones and steerable filters come second, but they perform significantly worse than SIFT. Note that SIFT descriptors computed on Harris-Laplace regions perform worse than any of the other descriptors (see HL s i f t ) , as these regions and therefore the descriptors are only scale and not affine-invariant. Cross-correlation obtains the lowest score. This can be explained by the sensitivity of cross-correlation to errors in the point and region extraction. A comparison for image rotation, scale and illumination changes 23 shows similar results as in the case of affine viewpoint changes. In conclusion we observe that the performance varies significantly for different descriptors. SIFT descriptors perform best. This shows the robustness and the distinctiveness of the region-based SIFT descriptor. Second best are steerable filters; they can be considered a good choice given their low dimensionality. Cross-correlation gives unstable results. The performance depends on the accuracy of interest point and region detection, which
78
0
2
4
6
false positive rate
8
to
12
xio"3
Fig. 7. Evaluation for a 60° viewpoint change of the camera. Descriptors are computed for HarrisAffine regions. HL s i f t is the SIFT descriptor computed for Harris-Laplace regions.
decreases for significant geometric transformations. The differential invariants give significantly worse results than the steerable filters, which is surprising as they are based on the same basic components (Gaussian derivatives). The multiplication of derivatives necessary to obtain the rotation invariance increases the instability of the descriptors. Overall, the comparison shows that a robust region-based descriptor performs better than point-wise descriptors.
2.4.
Recognition
Local invariant photometric descriptors are suitable for recognition of the same object or scene in the presence of large viewpoint changes. Several approaches based on local invariants have been developed 19.22>29.33 a nd have shown an excellent performance. They all combine similarity measures of local descriptors with semilocal and global consistency constraints. Our approach 22 extracts affine-invariant regions with the Harris-Affine detector and describes each of them with a set of Gaussian derivatives invariant to rotation and affine illumination changes. The similarity of descriptors is measured by the Mahalanobis distance where the covariance matrix is estimated statistically over a large set of images. A voting algorithm is used to select the most similar images in the database. For each interest point of a query image, its descriptor is compared to the descriptors in the database. If the distance is less than a fixed threshold, a vote is added for the corresponding database image. Note that a point cannot vote several times for the same database image. Votes are then verified by robustly estimating the geometric transformation between image pairs based on RANdom SAmple Consensus (RANSAC) and rejecting inconsistent matches. For our experimental results the transformation is either a homography or a fundamental matrix. A model selection algorithm 12 can be used to automatically decide which transformation is the most appropriate one.
79 Figure 8 illustrates retrieval results from a database with more than 5000 images. The top row displays query images for which the corresponding image in the database (second row) was correctly retrieved. Note the significant transformations between the query images and the images in the database. There is a scale change of a factor of 3 between images of pair (a). Image pairs (b) and (c) show large viewpoint changes. The displayed matches are the inliers to a robustly estimated fundamental matrix or homography between the query image and the database image.
Pig. 8. The top row shows the query images and the bottom row shows the most similar images in the database. The displayed matches are the inliers to a robustly estimated fundamental matrix or homography between the query image and the database image. There are (a) 22 matches, (b) 34 matches and (c) 22 matches. All of them are correct.
3. Recognizing t e x t u r e s We address in this section the problem of representing and recognizing non-rigid textures observed from arbitrary viewpoints. Recent approaches to texture recognition 18,42 perform impressively well on datasets as challenging as the Brodatz database 5 . Unfortunately, these schemes rely on restrictive assumptions about their input (e.g., the texture must be stationary) and are not generally invariant under 2D similarity and affine transformations, much less 3D transformations caused by camera motions and non-rigid deformations of textured surfaces. In addition, most existing approaches to texture analysis use a dense representation where some local image descriptor is computed over a fixed neighborhood of each pixel. Affineinvariant patches can be used to address the issues of spatial selection—Inding a sparse set of texture descriptors at "interesting" image locations—and shape
80
selection—computing shape and scale characteristics of the descriptors—(see 3 2 for related work). In addition, they afford a texture representation that is invariant under any geometric transformation that can be locally approximated by an affine model: Local affine invariants are capable of modeling not only global affine transformations of the image, but also perspective distortions and non-rigid deformations that preserve the locally flat structure of the surface (e.g., the bending of paper or cloth). In this context, it is appropriate to combine the Harris-Affine interest point detector 2 2 with the affine-adapted Laplacian blob detector proposed by Lindeberg and Garding 1T . The two feature detectors are dubbed H (for Harris) and L (for Laplacian), and they provide two description "channels" for local image patterns. Their output on two sample images is shown in Figure 9. Intuitively, the two detectors provide complementary kinds of information: H responds to corners and other regions of "high information content55, while L produces a perceptually plausible decomposition of the image into a set of blob-like primitives.
Fig. 9. Prom left to right: H-detector and L-deteetor output for a building and flower image.
3.1. Image
signatures
This section demonstrates the power of affine-invariant features as image descriptors in texture classification tasks. Our approach applies the following process to each image in the database using both the H and L feature detectors: (1) find the affine-invariant patches; (2) construct an affine-invariant description of these patches; (3) find the most significant clusters of similar descriptions and use them to construct the signature of the image; and (4) compare all pairs of signatures using the Earth Mover's Distance (EMD). Like most approaches to texture analysis, ours relies on clustering to discover a small set of basic primitives in the initial collection of candidate texture elements. We use a standard agglomerative clustering algorithm. The final representation for the image is a signature of the form {(mi, t^i), (m2,ti2) 5 . • •, (rrifc, «*)}, where mi is the medoid (the most centrally located element of the ith cluster) and Ui is the relative weight of the cluster (the size of the cluster divided by the total number of descriptors extracted from the image). Signatures have been introduced by Rubner et al. 31 as representations suitable for matching using the Earth Mover's Distance (EMD). For our application, the signature/EMD framework offers several important
81
advantages. A signature is more descriptive than a histogram, and it does not require global clustering of the descriptors found in all images. In addition, EMD can match signatures of different sizes, and it is not very sensitive to the number of clusters— that is, if one cluster is split into several clusters with similar medoids, the magnitude of the EMD is not greatly affected. This is a very important property, since the automatic selection of the number of clusters remains a largely unsolved problem. Finally, recall that the proposed texture representation is designed to work with multiple channels corresponding to different affine-invariant region detectors (here, the H and L operators). Each channel generates its own signature representation for each image in the database, and therefore its own EMD value for any pair of images. We have experimented with several methods of combining the EMD matrices of the separate channels to arrive at a final estimate of the distance between each pair of images. Empirically, simply adding the distances produces the best results.
\
trrr-- ~-: ~ ~ ' ~ ^ ^
1 3
Y'--:A • 8 0.7
8 0.7
CD
CD
O) 0.6
CD
--- H I .... 1 H+L I
0.5 0.4]
y^^h:
CD
D) 0.6
^
L^-zj^zZr:
0-»
2 | 0.8 'c
Number of retrievals
(a) Spin images
l
/
/
/
•
\
--- H
- | 0.5
1
H + LI
Number of retrievals
(b) Gabor-like filters
II 11 + L Class L 0.89 0.98 0.98 TO 0.71 Tl 0.83 0.76 T2 0.95 0.90 0.87 0.93 1.00 1.00 T3 0.52 0.47 0.43 T4 1.00 0.93 1.00 T5 0.84 0.95 0.97 T6 0.84 0.74 0.71 T7 0.85 0.63 0.89 T8 0.99 0.91 0.98 T9 0.89 0.80 Mean 0.86 (e) Classification rates
Fig. 10. Top: Samples of the ten texture classes used in our experiments. Bottom: Retrieval and classification results 1 4 .
We have implemented the proposed approach and conducted experiments with a dataset consisting of 200 images—20 samples each of ten different textured surfaces. Figure 10(top) shows two sample images of each texture. Significant viewpoint changes and scale differences are featured within each class. Several of the classes include additional sources of variability: inhomogeneities in the texture patterns, nonrigid transformations, illumination changes, and unmodeled viewpoint-dependent appearance changes. Figure 10(bottom, left) shows retrieval results using intensitydomain spin images 14 as local image descriptors. Spin images are two-dimensional
82
histograms encoding the distribution of brightness values. The dimensions are the distance from the center or the origin of the normalized coordinate system of the patch, and the intensity value. Spin images are similar to the SIFT descriptors introduced in the previous section, but avoid estimation of the gradient orientation. Notice that for this dataset, the H channel is more discriminative than the L channel. Adding the EMD estimates provided by the two channels results in improved performance. Figure 10(bottom, center) shows the results obtained using the Gaborlike filters commonly used as image descriptors in texture analysis 3 4 , 4 1 instead of intensity-domain spin images. Figure 10(bottom, right) summarizes the classification results obtained by using five samples from each class as training images. The classification rate for each class provides an indication of the "difficulty" of this class for our representation. The mean classification rate is 89% with two classes achieving 100%, showing the robustness of our system against a large amount of intra-class variability. Performance is very good for the rather inhomogeneous textures T5 and T6, but class T4 is not recognized very well, which is probably explained by the lack of an explicit model for viewpoint-dependent appearance changes caused by non-Lambertian reflectance and fine-scale 3D structure.
3.2. Generative
models
The previous section demonstrated the adequacy of our image descriptors in simple texture classification tasks. Here we go further and introduce generative models for the distribution of these descriptors, along with co-occurrence statistics for nearby patches. At recognition time, initial probabilities computed from the generative model are refined using a relaxation step that incorporates co-occurrence statistics learned at modeling time. In the supervised framework, the training data consists of single-texture sample images from classes with labels C(. The class-conditional densities p(x\C() can be estimated using all the feature vectors extracted from the images belonging to class C(. We model class-conditional densities as p{x\Ci) = Y^m=i v{x\cim) p(ctm), where the components cem are thought of as sub-classes and each p(x\cem) is assumed to be a Gaussian. The EM algorithm is used to estimate the parameters of the mixture model and is initialized with the output of the K-means algorithm. We limit the number of free parameters and control numerical behavior by using spherical Gaussians with covariance matrices of the form S^TO = o\mI. In situations where it is difficult to obtain large amounts of fully labeled examples, training on incompletely labeled or unlabeled data helps to improve classification performance 26 . The EM framework provides a natural way of incorporating unsegmented multi-texture images into the training set. Suppose we are given a multi-texture image annotated with the set C of class indices that it contains— that is, each feature vector x extracted from this image has a label set of the form Cc = {Ce\i € £}• To accommodate label sets, we now use all the data simultaneously to estimate a single mixture model with L x M components. The
83
estimation process starts by selecting some initial values for model parameters. During the expectation or E-step, we use the parameters to compute probabilistic sub-class membership weights given the feature vectors x and the label sets Cc- p(ctm\x,Cc) ex p(x\clm)p(cim\Cc), where p{ctm\Cc) = 0 for all ( g C and ^2iiec li2m=iP(ce™-\C'c) = 1- During the maximization or M-step, we use the computed weights to re-estimate the parameters by maximizing the expected likelihood of the data. At this stage, each region in the training image is assigned the sub-class label that maximizes the posterior probability p(cem\x,Cc)- Next, we need to define a neighborhood for a given region which depends on the size and shape of the region. Here we "grow" the ellipse by adding a constant absolute amount (15 pixels in the implementation) to the major and minor axes. We can now effectively turn the image into a directed graph with arcs emanating from the center of each region to other centers that fall within its neighborhood. The existence of an arc from a region with sub-class label c to another region with label c' is a joint event (c, c') (note that the order is important since the neighborhood relation is not symmetric). For each possible pair of labels, we estimate p(c, c') from the relative frequency of its occurrence, and also find the marginal probabilities p(c) = ^ c , p(c, c') and p(c') = ^ C P ( C ' C ' ) - Finally, we compute the values r(c,c') =
p{c,c')-p{c)p{c') [{p{c)-p{c))(p{c>)-p>{c>))}^
representing the correlations between the events that the labels c and c', respectively, belong to the source and destination nodes of the same arc. The values of r(c, c') must lie between —1 and 1; negative values indicate that c and c' rarely co-occur as labels at endpoints of the same edge, while positive values indicate that they co-occur often. In our experiments, we have found that the values of r(c, c') are reliable only in cases when c and c' are sub-class labels of the same class C. We set r(c, c') to a constant negative value that serves as a "smoothness constraint" in the relaxation algorithm described next, whenever c and c' belong to different classes. We have implemented the probability-based iterative relaxation algorithm described in the classic paper by Rosenfeld et al. 28 to enforce spatial consistency. The initial estimate of the probability that the ith region has label c, denoted p\ '(c), is obtained from the learned Gaussian mixture model as the posterior probability p(c\xi). Note that since we run relaxation on unlabeled test data, these probabilities must be computed for all L x M sub-class labels corresponding to all possible classes. At each iteration, new probability estimates p\ ' (c) are obtained by updating the current values p\ ' (c) using the equation Pi
(c)
pf\c)[l + q^(c)}
,(*>
w=E
Wi
Y,r{c,c')pf{c') . c'
The scalars Wij are weights that indicate how much influence region j exerts on
84 region i. We treat i% as a binary indicator variable that is nonzero if and only if the j t h region belongs to the ith neighborhood. Note that the weights are required to be normalized so that ]P Wij = 1. Excellent results for classification/segmentation are obtained for images of an indoor scene and pictures of wild animals. Our first data set contains seven different textures present in a single indoor scene, see Figure 11. Class labels are assigned automatically to each region. Regions of a class are shown in the corresponding column. The second and third rows of Figure 11 show the improvement between the initial labeling and final labeling after relaxation which takes into account the spatial layout. Our second data set consists of unsegmented images of three kinds of animals: cheetahs, giraffes, and zebras. The training set contains 10 images from each class. To account for the lack of segmentation, we introduce an additional "background" class, and each training image is labeled as containing the appropriate animal and the background. Typical classification examples are shown in Figure 12.
Original image
Brick
Carpet
Chair
Floor 1
Floor 2
Marble
Wood
Fig. 11. Segmentation/classification results. From top to bottom: a sample image of each texture class from an indoor scene; initial labeling for an indoor image vs. the final labeling after relaxationnote the significant improvement-; another successful indoor image segmentation (final labeling). See 1 3 for additional results.
Original image
Cheetah
j^tej
Zebra
P1R
Giraffe
Background
1 &*., .^51$
Fig. 12. Segmentation/classification results (final results after relaxation). Animal image classification examples.
85
4. Recognizing object classes Learning and recognizing object class models from images is one of the most difficult problems in computer vision. The combination of image description and machine learning/pattern classification techniques has recently led to significant progress. Several recent approaches to category-level object recognition use classification techniques to acquire part models from training images, and then train a classifier to recognize objects. This paradigm has been successfully applied to the recognition of cars *, faces 3 6 and human beings 2 5 in complex imagery. However, the image descriptors used in these methods enjoy very limited invariance properties (mostly translational invariance), which severely limits the range of admissible viewpoints that they can handle. This can be avoided by using local invariants as image descriptors 7>8>15«27. The approach is section 4.1 constructs parts from individual local invariants, whereas the one presented in section 4.2 uses set of local invariant features described by their appearance and neighbourhood relations to learn parts. 4.1. Discriminative
local
parts
In this section we demonstrate the power of using invariant local features for building discriminative local parts. The idea is to find groups (clusters) of similar local features, i.e. local parts, and to select among these parts the ones which best discriminate between positive and negative images. Figure 13 shows two discriminative parts for the categories airplane, motorbike, wild cat and person.
Fig. 13. Examples of discriminative parts (clusters) for the categories airplane, motorbike, wild cat and person. Parts are chosen from the 10 most discriminative ones. Each cluster is represented by five of its members. The circles represent the regions extracted by the scale-invariant detector.
The approach is weakly supervised, that is the training images are labeled as positive and negative, but the objects are not labeled in the positive images. The training set is split in a "clustering" set and a "validation" set. Parts are learned based on the "clustering" set and the significance of each part is determined with
86
the "validation" set. The first step of our approach is to extract local invariant features. Here we use Harris-Laplace and Harris-Affine as well as the entropy detector by Kadir and Brady n and describe the regions by the SIFT descriptor. The descriptors of the "clustering" set are then used to learn the individual parts. We estimate the Gaussian mixture model of their distribution and each component of the mixture represents a part (cluster). The EM algorithm is used to estimate the parameters of the mixture model and is initialized with the output of the if-means algorithm. Descriptors are assigned to the components with the maximum posteriori probability. We then select parts with the "validation" set. We first compute probabilities p(Ci\x) for each descriptor of this set, i.e. determine the probability for each part C, (component of the Gaussian mixture model). Each part d is then ranked by the likelihood ratio between the descriptors of the positive images {xj} -note that the individual descriptors are unlabeled- and the negative images {#£}: R = J2jP(^i\x^)IY^kP(Ci\x%). Other criteria can be used, such as mutual information 7 . The final classifier then sets the n highest rank parts as positive and the others as negative. A descriptor is classified as an object descriptor if it is labeled as belonging to one of the top n parts (maximum posteriori probability for that class) where n is a parameter of our approach. Our approach is evaluated in two different ways. We first verify that the positive descriptors lie mostly on the object. Figure 14 shows the results for a few test images. Note that the results are very good. Only a few points are incorrectly classified and they could easily be eliminated by any simple coherence criterion. We have also verified that (as in Section 2.3) SIFT features again give significantly better results than steerable filters. We then evaluate the performance by image classification, that is if the image contains the object or not. This is a standard criterion which allows comparison with existing methods. We report and compare image classification results in table 1. Training and test images are the same as in Fergus et al. 8 and in Opelt et al. 27 . We measure performance with the Receiver Operating Characteristic (ROC) equal error rate. It is defined to be the point on the ROC curve-obtained by varying the number of parts n- where the proportion of true positives is equal to the proportion of true negatives. Classification requires an additional parameter, namely the minimum number p of positive descriptors in the image for which the image is classified as positive. The "best" p is estimated from the validation set. Note that this value doesn't necessary lead to the best classification performance on the test set. Table 1 shows that our model in combination with the two scale-invariant detectors HarrisLaplace and the entropy detector outperforms the other methods.
4.2. Affine-invariant
semi-local
parts
In this section we use characteristic patterns formed by affine-invariant patches and semi-local spatial relations to describe salient object parts. Figure 15 illustrates this
87 n = 10
n = 20
n = 30
Fig. 14. Positive detections with increasing n for different categories and detectors. First and second row: entropy detector. Third row: Harris-Aifine detector.
airplanes faces motorbikes wildcats bikes people
Our model 0.985 0.991 0.995 0.87 0.88 0.88
Fergus et al. 0.902 0.964 0.925 0.900 — —
8
Opelt et al. 0.889 0.935 0.922 — 0.865 0.808
27
Table 1. Image classification performance measured with the equal error rate.
idea with an affine-invariant semi-local part found between face images using the output of the affine-invariant Laplacian and a variant of affine alignment 3 . Note that the part is stable despite large viewpoint variations and appearance changes. Affine-invariant semi-local parts are geometrically stable configurations of multiple affine-invariant regions, found by the affine Laplace detector. These parts are approximately affinely rigid by construction, i.e. the mapping between instances of the same part in two images can be well represented by a 2D affine transformation. Combined with the locality of the parts, this property makes our method suitable for modeling a wide range of 3D transformations, including viewpoint changes and nonrigid deformations. Furthermore, they are more distinctive than individual features
88
Fig. 15.
Matching faces: an affine-invariant semi-local part.
used in the previous section. The mechanism for learning affine-invariant semi-local parts is based on the idea that a direct search for visual correspondence is key to successful recognition. Thus, at training time we seek to identify groups of neighboring affine regions whose appearance and spatial configuration remains stable across multiple instances. To avoid the prohibitive complexity of establishing simultaneous correspondence across the whole training set, we separate the problem into two stages: Parts are initialized by matching pairs of images and then matched against a larger validation set. Even though finding optimal correspondence between features in two images is still intractable, effective sub-optimal solutions can be found using non-exhaustive constrained search. Admiral Swallowtail Machaon Monarch 1 Monarch 2
Peacock
Zebra
Fig. 16. The butterfly dataset. Two samples of each class are shown in each column. We have implemented the approach 15 and conducted experiments for the challenging application to the automated acquisition and recognition of butterfly models in heavily cluttered natural images. Figure 16 shows two samples for the seven classes in our dataset which is composed of 619 butterfly images. Note that we don't use any negative images for modeling. The pictures, which are collected from the Internet, are extremely diverse in terms of size and quality. Motion blur, lack of focus, resampling and compression artifacts are common. This dataset is appropriate for exercising the descriptive power of semi-local affine parts, since the geometry of a butterfly is locally planar for each wing (though not globally planar). In addition, the species identity of a butterfly is determined by a basically stable geometric wing
09
pattern, though appearance can be significantly affected by variations between individuals, lighting, and imaging conditions. It is crucial to point out that butterfly recognition is beyond the capabilities of many current state-of-the-art recognition systems 1 , s .
Fig. 17. Butterfly modeling and detection examples, (a) T h e part with the highest validation score for a class. The part size is listed below each modeling pair, (b) Example of detecting the part from (a) in a single test image. Detected regions are shown in yellow and occluded ones are reprojected from the model in blue. The total number of detected regions (absolute repeatability) and the corresponding repeatability ratio are shown below each image.
The recognition framework is straightforward. Matching and validation are used to identify a fixed-size collection of parts for representing the classes. Candidate parts are formed by matching between eight randomly chosen pairs of training images. Ten verification images per class are used to rank candidate parts according to their repeatability score, and the top ten parts per class are retained for recognition. Figure 17(a) shows the part having the highest repeatability for each of the classes. At testing time, the parts for all classes are detected in each training image. Though multiple instances of the same part may be found, we retain only the single instance with highest number of detected regions. Figure 17 (b) shows examples of part detections in individual test images. The score for a class is defined as the cumulative relative repeatability of all its parts, or the total number of regions detected in all parts of the classes divided by the sum of part sizes. For multi-class classification, each image is assigned to the class having the maximum score. Figure 18(a) shows classification results obtained using the above approach (the average rate is 90.4%), and Figure 18(b) shows how performance is improved by using multiple parts.
90
(a)
Class Admiral Swallowtail Machaon Monarch 1 Monarch 2 Peacock Zebra Total
Test images Correct (rate) P a r t size (0.871) 74 85 179 (12/28) (0.750) 12 16 252 (18/29) (0.965) 55 57 148 (12/21) (0.729) 35 48 289 (14/67) (0.914) 53 58 275 (19/36) (1.000) 108 102 (8/14) 108 (0.892) 58 65 209 (16/31) (0.904) 395 437
0.9 0.88
1086
(b) J I Io.8 Number of parts
Fig. 18. Classification results for the butterflies, (a) The second column shows the size of the model (top 10 parts) for each class (the sum of sizes of individual parts), and the size of the smallest and the largest parts are listed in parentheses, (b) Classification rate vs. number of parts. 5. C o n c l u s i o n a n d d i s c u s s i o n In this chapter we have presented a state of the art on local invariant photometric regions and descriptors. We have shown t h a t the resulting local photometric invariants are very well a d a p t e d to object recognition. Research on invariant regions and their description is now well advanced and these invariant features are building blocks for general recognition systems. Of course, improvements of detectors and descriptors are still possible, for example by developing different types of detectors and different region-based descriptors. T h e remaining open problems are the recognition of a large number of objects and the recognition of object categories. We think t h a t b o t h problems require the use of machine learning, i.e. p a t t e r n classification, techniques. To handle a large number of objects we have to structure the d a t a based on d a t a reduction, clustering and d a t a mining techniques. Recognizing object categories requires the description of intra-class variation, feature selection, a flexible description of t h e spatial structure as well as a hierarchical organization of the categories. Acknowledgments This research was supported by t h e E u r o p e a n Projects VIBES a n d LAVA, by a UIUC-CNRS collaboration agreement, by the UIUC Campus Research Board and by the National Science Foundation grants IRI-990709 and IIS-0308087. We are also grateful to D. Lowe, T. Kadir, F . Schaffalitzky and A. Zisserman for providing the code for their detectors a n d descriptors. Bike and people images have been provided by A. Opelt 2 7 , the airplane, face, motorbike and wild cat images by R. Fergus 8 and the Valbonne images by the INRIA group Robot Vis. References 1. S. Agarwal and D. Roth. Learning a sparse representation for object detection. In European Conference on Computer Vision, vol. IV, pp. 113-127, 2002. 2. L. Alvarez and F. Morales. Affine morphological multiscale analysis of corners and multiple junctions. International Journal of Computer Vision, 2(25):95-107, 1997. 3. R. Basri and S. Ullman. The alignment of objects with smooth surfaces. In International Conference on Computer Vision, pp. 482-488, 1988.
91 4. A. Baumberg. Reliable feature matching across widely separated views. In Computer Vision and Pattern Recognition, vol. I, pp. 774-781, 2000. 5. P. Brodatz. Textures: A Photographic Album for Artists and Designers. Dover Publications, New York, 1966. 6. J.L. Crowley and A.C. Parker. A representation for shape based on peaks and ridges in the difference of low pass transform. Transactions on Pattern Analysis and Machine Intelligence, 6(2):156-170, 1984. 7. G. Dorko and C. Schmid. Selection of scale-invariant parts for object class recognition. In International Conference on Computer Vision, vol. 1, pp. 634-640, 2003. 8. R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning. In Computer Vision and Pattern Recognition, vol. II, pp. 264271, 2003. 9. L.M.T. Florack, B. ter Haar Romeny, J.J Koenderink, and M.A. Viergever. General intensity transformation and differential invariants. Journal of Mathematical Imaging and Vision, 4(2):171-187, 1994. 10. W.T. Freeman and E.H. Adelson. The design and use of steerable filters. Transactions on Pattern Analysis and Machine Intelligence, 13(9):891-906, 1991. 11. T. Kadir and M. Brady. Scale, saliency and image description. International Journal of Computer Vision, 45(2):83-105, 2001. 12. K. Kanatani. Geometric information criterion for model selection. International Journal of Computer Vision, 26(3):171-189, 1998. 13. S. Lazebnik, C. Schmid, and J. Ponce. AfRne-invariant local descriptors and neighborhood statistics for texture recognition. In International Conference on Computer Vision, vol. 1, pp. 649-655, 2003. 14. S. Lazebnik, C. Schmid, and J. Ponce. Sparse texture representation using affineinvariant neighborhoods. In Computer Vision and Pattern Recognition, vol. 2, pp. 319324, 2003. 15. S. Lazebnik, C. Schmid, and J. Ponce. Local affine parts for object recognition. In British Machine Vision Conference, 2004. 16. T. Lindeberg. Feature detection with automatic scale selection. International Journal of Computer Vision, 30(2): 79-116, 1998. 17. T. Lindeberg and J. Garding. Shape-adapted smoothing in estimation of 3D shape cues from affine deformations of local 2D brightness structure. Image and Vision Computing, 15(6):415-434, 1997. 18. F. Liu and W. Picard. Periodicity, directionality, and randomness: Wold features for image modeling and retrieval. Transactions on Pattern Analysis and Machine Intelligence, 18(7):722-733, 1996. 19. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-110, 2004. 20. J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. In British Machine Vision Conference, pp. 384393, 2002. 21. K. Mikolajczyk and C. Schmid. Indexing based on scale invariant interest points. In International Conference on Computer Vision, vol. 1, pp. 525-531, 2001. 22. K. Mikolajczyk and C. Schmid. An affine invariant interest point detector. In European Conference on Computer Vision, vol. I, pp. 128-142, 2002. 23. K. Mikolajczyk and C. Schmid. A performance evaluation of local descriptors. In Computer Vision and Pattern Recognition, vol. 2, pp. 257-263, 2003. 24. K. Mikolajczyk and C. Schmid. Scale and affine invariant interest point detectors. International Journal of Computer Vision, 60(l):63-86, 2004.
92 25. K. Mikolajczyk, C. Schmid, and A. Zisserman. Human detection based on a probabilistic assembly of robust part detectors. In European Conference on Computer Vision, vol. I, pp. 69-81, 2004. 26. K. Nigam, A. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2-3):103-134, 2000. 27. A. Opelt, M. Fussenegger, A. Pinz, and P. Auer. Weak hypotheses and boosting for generic object detection and recognition. In European Conference on Computer Vision, vol. II, pp. 71-84, 2004. 28. A. Rosenfeld, R. A. Hummel, and S.W. Zucker. Scene labeling by relaxation operations. Transactions on Systems, Man and Cybernetics, 6:420-433, 1976. 29. F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce. 3D object modeling and recognition using affine-invariant patches and multi-view spatial constraints. In Computer Vision and Pattern Recognition, vol. 2, pp. 272-277, 2003. 30. C. Rothwell, A. Zisserman, D. Forsyth, and J. Mundy. Canonical frames for planar object recognition. In European Conference on Computer Vision, pp. 757-772, 1992. 31. Y. Rubner, C. Tomasi, and L.J. Guibas. A metric for distributions with applications to image databases. In International Conference on Computer Vision, pp. 59-66, 1998. 32. F. Schaffalitzky and A. Zisserman. Viewpoint invariant texture matching and wide baseline stereo. In International Conference on Computer Vision, vol. II, pp. 636-643, 2001. 33. F. Schaffalitzky and A. Zisserman. Multi-view matching for unordered image sets. In European Conference on Computer Vision, vol. I, pp. 414-431, 2002. 34. C. Schmid. Weakly supervised learning of visual models and its application to contentbased retrieval. International Journal of Computer Vision, 56(1):7-16, 2003. 35. C. Schmid and R. Mohr. Local grayvalue invariants for image retrieval. Transactions on Pattern Analysis and Machine Intelligence, 19(5):530-534, 1997. 36. H. Schneiderman and T. Kanade. A statistical method for 3D object detection applied to faces and cars. In Computer Vision and Pattern Recognition, vol. I, pp. 746-751, 2000. 37. M.J. Swain and D.H. Ballard. Color indexing. International Journal of Computer Vision, 7(l):ll-32, 1991. 38. M. Turk and A. Pentland. Face recognition using eigenfaces. In Computer Vision and Pattern Recognition, pp. 586-591, 1991. 39. T. Tuytelaars and L. Van Gool. Matching widely separated views based on affine invariant regions. International Journal of Computer Vision, 59(l):61-85, 2004. 40. L. Van Gool, T. Moons, and D. Ungureanu. Affine / photometric invariants for planar intensity patterns. In European Conference on Computer Vision, pp. 642-651, 1996. 41. M. Varma and A. Zisserman. Classifying images of materials: Achieving viewpoint and illumination indepence. In European Conference on Computer Vision, vol. Ill, pp. 255-271, 2002. 42. K. Xu, B. Georgescu, D. Comaniciu, and P. Meer. Performance analysis in contentbased retrieval with textures. In International Conference on Pattern Recognition, vol. IV, pp. 275-278, 2000.
CHAPTER 2.1 CASE-BASED REASONING FOR IMAGE ANALYSIS AND INTERPRETATION
Petra Perner Institute of Computer Vision and Applied Computer Science, Kornerstr. 10, 04107 Leipzig, Germany, [email protected] http://www.ibai-research.de The development of image interpretation systems is concerned with tricky problems such as a limited number of observations, environmental influence, and noise. Recent systems lack robustness, accuracy, and flexibility. The introduction of case-based reasoning (CBR) strategies can help to overcome these drawbacks. The special type of information {i.e. images) and the problems mentioned above provide special requirements for CBR strategies. In this chapter we review what has been achieved so far and research topics concerned with case-based image analysis and interpretation.
1.
Introduction
Image interpretation systems are becoming increasingly popular in medical and industrial applications. The existing statistical and knowledge-based techniques lack robustness, accuracy, and flexibility. New strategies are necessary that can adapt to changing environmental conditions, user needs and process requirements. Introducing case-based reasoning (CBR) strategies into image interpretation systems can satisfy these requirements. CBR provides a flexible and powerful method for controlling the image processing process in all phases of an image interpretation system to derive information of the highest possible quality. Beyond this CBR offers different learning capabilities, for all phases of an image interpretation system, that satisfy different needs during the development process of an image interpretation system. Therefore, they are especially appropriate for image interpretation. Although all this has been demonstrated in various applications1"6, case-based image interpretation systems are still not well established in the computer vision community. One reason might be that CBR is not very well known within this community. In this paper, we will show that a CBR framework can be used to overcome the modeling 95
96
burden usually associated with the development of image interpretation systems. We will review current activities on image interpretation. In Sec. 2, we will introduce the tasks involved when interpreting an image. In Sec. 3, we will describe the special needs of an image interpretation system and how they are related to CBR topics. Then, we will review in Sec. 4 activities for image analysis and interpretation in the CBR community. In Sec. 5, we describe the case-based reasoning process. The case description is given in Sec. 6. We will overview what kinds of similarity measures are useful and what are the open research topics in Sec. 7. In Sec. 8 we will show how a case base can be organized to speed up the retrieval process. The different learning strategies of a CBR system are described in Sec. 9. The problems concerned with the competences of case bases is discussed in Sec. 10. How cases can be acquired is presented in Sec. 11. Finally, we offer conclusions in Sec. 12 based on our CBR systems working in real-world environments.
2.
Tasks an Image Interpretation System Must Solve
Image interpretation is the process of mapping the numerical representation of an image into a logical representation such as suitable for scene description. An image interpretation system must be able to extract symbolic features from the pixels of an image (e.g., irregular structure inside the nodule, area of calcification, and sharp margin). This is a complex process; the image passes through several general processing steps until the final symbolic description is obtained. These include image preprocessing, image segmentation, image analysis, and image interpretation (see Fig. 1). Interdisciplinary knowledge from image processing, syntactical and statistical pattern recognition, and artificial intelligence is required to build such systems. The primitive (low-level) image features will be extracted at the lowest level of an image interpretation system. Therefore, the image matrix acquired by the image acquisition component must first undergo image pre-processing to remove noise, restore distortions, undergo smoothing, and sharpen object contours. In the next step, objects of interest are distinguished from background and uninteresting objects, which are removed from the image matrix. In the x-ray computed tomography (CT) image shown in Fig. 1, the skull and the head shell is removed from the image in a preprocessing step. Afterwards, the resulting image is partitioned into objects such as brain and liquor. After having found the objects of interest in an image, we can then describe the objects using primitive image features. Depending on the particular objects and focus of interest, these features can be lines, edges, ribbon, etc. A geometric object such as a block will be described, for example, by lines and edges. The objects in the ultrasonic image shown in Fig. 1 are described by regions and their spatial relation to each other. The region's features could include size,
97
shape, or the gray level. Typically, these low-level features have to be mapped to highlevel features. A symbolic feature such as fuzzy margin will be a function of several lowlevel features. Lines and edges will be grouped together by perceptual criteria such as collinearity and continuity in order to describe a block.
Defect is a Crack of Type A on the bottom of the industrial part Image Interpretation
Class is a Crack
Object Class
r
Image Classification
J
Object 1 (Greylevel=black, Size=little, sfiape=round), Left_of(Object1, Object2)
Symbolic Features Grouping or Mapping to High- J level Features
Object 1 (Greylevel=0, Size=2000, shape=0,9) ? Extraction of Low-level Feature mature I
Binary Image Image Segmentation
2D image
*&&&&,$&& lmQ#
Image Pre-Processing
2D Image
:
;
Image Acquisition
i
i#Bi* Sami mage
Fig. 1. Architecture of an Image Interpretation System Image classification is usually referred to as the mapping of numeric features to predefined classes. Sometimes image interpretation requires only image classification. However, image classification is frequently only a first step of image interpretation. Lowlevel features or part of the object description are used to classify the object into different object classes in order to reduce the complexity of the search space. The image
98
interpretation component identifies an object by finding the object that it belongs to (among the models of the object class). This is done by matching the symbolic description of the object in the scene to the model of the object stored in the knowledge base. When processing an image using an image interpretation system, an image's content is transformed into multiple representations that reflect different abstraction levels. This incrementally removes unnecessary detail from the image. The highest abstraction level will be reached after grouping the image's features. It is a product of mapping the image pixels contained in the image matrix into a logical structure. This higher level representation ensures that the image interpretation process will not be affected by noise appearing during image acquisition, and it also provides an understanding of the image's content. A bottom-up control structure is shown for the generic system in Fig. 1. This control structure allows no feedback to preceding processing components if the result of the outcome of the current component is unsatisfactory. A mixture of bottom-up and top-down control would allow the outcome of a component to be refined by returning to previous component.
3.
Development Concerns of an Image Interpretation System
Several factors influence the quality of the final result of an image interpretation system, including environmental conditions, the selected imaging device, noise, the number of observations from the task domain, and the chosen part of the task domain. These cannot often all be accounted for during system development, and many of them will only be discovered during system execution. Furthermore, the task domain cannot even be guaranteed to be limited. For example, in defect classification for industrial tasks, new defects may occur because the manufacturing tool that had been used for a long period suddenly causes scratches on the surface of the manufactured part. In optical character recognition, imaging defects (e.g., heavy print, light print, or stray marks) can occur and influence the recognition results. It is not yet possible to observe all real-world influences, nor provide a sufficiently large enough sample set for system development and testing. A robust image interpretation system must be able to deal with such influences. It must have intelligent strategies on all levels of an image interpretation system that can adapt the processing components to these new requirements. A strategy that seems to satisfy these requirements could be case-based reasoning. CBR does not rely on a wellformulated domain theory, which is, as we have seen, often difficult to acquire. This suggests that we must consider different aspects during system development that are frequently studied CBR issues. Because we expect users will discover new aspects of the environment and the objects during system usage, an automatic image interpretation system should be able to incrementally update the system's model, as illustrated in Fig. 2.
99 This requires knowledge maintenance and learning. The designated lifetime of a case also plays an important role. Other aspects are concerned with system competence. The range of target problems that a given system or algorithm can solve are often not quite clear to the developer of the image interpretation system. Often researchers present to the community a new algorithm that can, for example, recognize the shape of an object in a particular image and then claim that they have developed a model. Unfortunately, all too often another researcher inputs a different image to the same algorithm and finds that it fails. Did the first researcher develop a model or did they instead develop a function? Testing and evaluation of algorithms and systems is an important problem in computer vision, as is designing the algorithm's control structure so that it fits best to the current problem. CBR strategies can help to solve this problem in computer vision.
Heal world-environment
Imaging Device Observations 2D/3D Image Data
Model A ^ Refinement
Model Development Model B
And so on
Fig. 2. Model Development Process
4.
Image Analysis and Interpretation in the Case-Based Reasoning
Image processing is a challenging field. The unique data (images) and the necessary computation techniques require extraordinary case-representations, similarity measures and CBR strategies to be utilised. Perner7 proposes a system that uses CBR to optimize image segmentation at the low level unit according to changing image acquisition conditions and image quality. The intermediate-level unit extracts the case representation used by the high-level unit
100 employed to dynamically adapt image interpretation. The system works on different case representations such as a graph-based representation for the cases of the high-level image description and the raw image matrix for the low-level image representation. Therefore the system uses different CBR strategies for the reasoning and learning part one is based on structural similarity and the other is based on the digital image distance. Grimnes and Aamodt2 present a system that integrates CBR into a task-oriented model-based system for the interpretation of abdominal CT images. A case-based reasoner working on a segment case-base contains the individual image segments. These cases with labels are considered indexes for another case-based reasoner working on an organ interpretation case-base. There system is based on a propose-critique-modify learning cycle. Jarmulak' presents a system for ultra-sonic B-scans that are one-dimensional signals. He presents a tree-based retrieval strategy. Micarelli et al4 applies CBR to scene recognition. They calculated image properties from images and stored them into a case base. They used the Wavelet transform because it is scale-independent, but this limits their similarity measure to consider only object rotation. The application of CBR to image segmentation for CT images from the brain is described in Ref. 3. Different learning strategies in a hierarchy of structural cases are presented in Ref. 7 and Ref. 8. Learning case representation and improving the system performance by controlling the similarity measure is described in Ref. 9. The application of CBR image interpretation to health monitoring and biotechnology is described in Ref. 10. A new challenging application field are geographical information systems. This kind of application requires special spatial problem solving techniques and spatial similarity measures. The first application of CBR to geographic information systems was reported by Holt and Benwell". Their approach consists of combining case-based reasoning with geographical information systems to form a hybrid system to solve spatial problems in soil classification. Carswell et al. '2 described in their paper a spatial similarity measure for comparing the location objects in different images. A completely different application of CBR to image processing has been described by Ficet-Cauchard et a/.3. They apply CBR for the development of the image processing steps of formerly unknown image processing problems by using past experiences and plan adaption. Case-based object recognition for fungi identification and the developed similarity measures are described in Ref. 13. A system for case acquisition of visual forms and case mining has been developed in Ref. 14. Finally, in Perner'3 is built a bridge between the work in CBR and the one in dissimilarity classification which became recently important in pattern recognition.
101
5.
Case-Based Reasoning
Rule-based systems, model-based systems or decision trees are difficult to utilize in domains where generalized knowledge is lacking. However, often there is a need for an interpretation system, even though there is not enough generalized knowledge. Such a system should a) solve problems using the already stored knowledge and b) capture new knowledge making it immediately available to solve the next problem. To accomplish these tasks, case based reasoning is useful. Case-based reasoning explicitly uses past cases from the domain expert's successful or failing experiences. Therefore case-based reasoning can be seen as a method for problem solving as well as a method to capture new experiences. It can be seen as a learning and knowledgediscovery approach since it can capture from new experiences some general knowledge, such as case classes, prototypes and some higher level concepts. To point out the differences between a CBR learning system and a symbolic learning system, which represents a learned concept explicitly, e.g. by formulas, rules or decision trees, we follow the notion of Wess et al.I6: A case-based reasoning system describes a concept C implicitly by a pair (CB, sim). The relationship between the case base CB and the measure sim used for classification may be characterized by the equation: C = CB + sim.
(1)
This equation indicates in analogy to arithmetic that it is possible to represent a given concept C in multiple ways, i.e. there exist many pairs C = (CBt, sirrij). [CB,, sim,), .... [CBn simj for the same concept C . Furthermore the equation gives a hint how a case-based learner can improve its classification ability. There are three possibilities to improve a case-based system. The system can: • • •
store new cases in the case base CB, change the measure of similarity sim, or change CB and sim.
During the learning phase a case-based system gets a sequence of cases X ,.X2 X, with X,• = (x,, class(x,)) and builds a sequence of pairs C = {CBr sim/), [CB,, sim2), .... [CB,, simt) with CB, c{X,.X, X,}. The aim is to get in the limit a pair C = (CBn. simn), that needs no further change, i. e. 3« V/w > n (CBn. simn) = (CBm. simm), because it is a correct classifier for the target concept C . 5.1. The case-based reasoning
process
At the highest level of generality a general CBR cycle may be described by four tasks: Retrieve the most similar case or cases, reuse the information and knowledge in that case
102
to solve the problem, revise the proposed solution, and retain the parts of this experience likely to be useful for future problem solving. The process is graphically represented in Fig. 3. The current problem is described by some keywords, attributes or any abstraction that allows describing the basic properties of a case. In pattern recognition and image interpretation this can be attributes describing the basic properties of an object or an attributed graph describing a scene. Based on this case description a set of close cases is retrieved from the case base (retrieve task). The closest case is evaluated and the solutions associated to the closest case are given as output to the user. The problem solution associated to the current case is applied to the current problem and the result of the test is observed and evaluated by a teacher (revise task). When the user is not satisfied with the result, or no similar case could be found in the case base, then case-base maintenance (retain tasks) starts. The revise and retain task can be viewed as a supervised learning task which is necessary to improve the performance of the system.
(Case\—•
Failure/Success istration
Problem Description
Case Base Management
Case Evaluation
Case Base
Fig. 3: Case-Based Reasoning Process Retrieval can be based on an index. The index can be a structure, such as for example a classifier or any hierarchical organization of the case base. This index is built and updated during the retain task. 5.2. CBR
maintenance
CBR maintenance will operate on new cases as well as on cases already stored in the case base. If a new case has to be stored into the case base, it means that there is no similar case in the case base. The system has recognized a gap in the case base. A new case has
103 to be incorporated into the case base in order to close this gap. From the new case a predetermined case description has to be extracted, which should be formatted into the predefined case format. Afterwards the case can be stored into the case base. Selective case registration means that no redundant cases will be stored into the case base and that the case will be stored at the right place depending on the chosen organization of the case base. Similar cases will be grouped together or generalized by a case that applies to a wider range of problems. Generalization and selective case registration ensure that the case base will not grow too large and that the system can find similar cases fast. It might also happen that too many cases would be retrieved from the case base that are not applicable to the current problem. Then it might be wise to rethink the case description, or to adapt the similarity measure. For the case description, more distinguishing attributes should be found that allow sorting out cases that do not apply to the current problem. The weights in the similarity measure might be updated in order to retrieve only a small set of similar cases. CBR maintenance is a complex process and works over all knowledge containers (vocabulary, similarity, retrieval, case base) of a CBR system. Consequently, architectures and systems have been developed which support this process8 17 ' l8 . 5.3. Design
consideration
The main problems concerned with the development of a CBR system are: • • • • •
What is the right case description? What is an appropriate similarity measure for the problem? How to organize a large number of cases for efficient retrieval? How to acquire and refine a new case for entry in the case base? How to generalize specific cases to a case that is applicable to a wide range of situations?
6.
Case Description
There are different opinions about the formal description of a case. Each system utilizes a different representation of a case. Formally we like to understand the following definition for a case: Definition 1: A case F is a triple [P.E.L) with a problem description P, an explanation of the solution E and a problem solutions L.
104 For image related tasks we have two main different types of information that make up a case consisting of image-related information and non-image-related information. Image-related information could be the ID, 2D or 3D images of the desired application. Non-image-related information could be information about the image acquisition, such as the type and parameters of the sensor, and information about the objects or the illumination of the scene. It depends on the type of application what type of information should be taken into consideration for the interpretation of the image. In case of the CBR based image segmentation system for medical CT image segmentation described in Ref. 3 we used patient-specific parameters, such as age and sex, slice thickness and number of slices. Jarmulak1 took into consideration the type of sensor for the railway inspection application. Based on this information the system controls the type of case base that the system is using during reasoning. How the 2D or 3D image matrix is represented depends on the purpose and not seldom on the developer's point of view. In principle it is possible to represent an image by one of various abstraction levels. An image may be described by the pixel matrix itself or by parts of this matrix (pixel-based representation). It may be described by the objects contained in the image and their features (feature-based representation). Furthermore it may be described by a more complex model of the image scene comprised of objects and their features as well as the spatial relation between the objects (attributed graph representation or semantic networks). Grimnes and Aamodt" developed a model-based image interpretation system for the interpretation of abdominal CT images. The image content is represented by a semantic network where concepts can be general, special cases or heuristic rules. Not well understood parts of the model are expressed by cases and can be revised during the usage of the system by the learning component. The combination of the partially wellunderstood model with cases helps them to overcome the usual burden of modeling. The learning component is based on failure driven learning and case integration. Non-image information is also stored such as sex, age, earlier diagnosis, social condition etc. Micarelli et al4 have also calculated image properties from their images and stored them into the case base. They use the Wavelet transform, since it is scale-independent. By doing so they only take into consideration the rotation of the objects in their similarity measure. In all this work, CBR is only used for the high-level unit. We have studied different approaches for the different processing stages of an image interpretation system. For the image-segmentation unitJ we studied two approaches: 1. a pixel-based approach and 2. a feature-based approach that described the statistical properties of an image. Our results show that the pixel-based approach can give better results for the purpose of image segmentation. For the high-level approach of an ultra-sonic image interpretation system, we used an attributed graph-based representation .
105 However, if we do not store the image matrix itself as a case, but store the representation of a higher-level abstraction instead, we will lose some information. An abstraction means that we have to make a decision between necessary and unnecessary details of an image. It might happen that not having seen all objects at the same time, we might think that one detail is not of interest, since our decision is only based on a limited number of objects. This can cause problems later on. Therefore, to keep the images themselves is always preferable, but needs a lot of storage capacity. The different possible types of representation require different types of similarity measures. 7.
Similarity
An important point in case-based reasoning is the determination of similarity between a case A and a case B. We need an evaluation function that gives us a measure for similarity between two cases. This evaluation function reduces each case from its case description to a numerical similarity measure sim. These similarity measures show the relation to other cases in the case base. The concept of similarity is important for many tasks, therefore we will briefly review the basic properties of similarity and the design criteria for similarity. 7.1. Formalization of similarity The problem with similarity is that it has no meaning unless one specifies the kind of similarity. Smith19 distinguishes five different kinds of similarity: • • •
Overall similarity Similarity Identity
• •
Partial similarity and Partial identity
Overall similarity is a global relation that includes all other similarity relations. All colloquial similarity statements are subsumed here. Similarity and identity are relations that consider all properties of objects at once, no single part is left unconsidered. A red ball and a red ball are similar, a red ball and a red car might be dissimilar. The holistic relation's similarity and identity are different in the degree of the similarity. Identity describes objects that are not significantly different. All red balls are identical and, therefore, they are also similar. Similarity contains identity and is more general concept. Partial similarity and partial identity compare the significant parts of objects. One aspect or attribute can be marked. Partial similarity and partial identity are different with
106
respect to the degree of similarity. A red ball and a pink cube are partially similar, but a red ball and a red cube are partially identical. The described similarity relations are in connection with many respects. Identity and similarity are unspecified relations between whole objects. Partial identity and similarity are relations between single properties of objects. Identity and similarity are equivalence relations. That means they are reflexive, symmetrical, and transitive. For partial identity and similarity these relations do not hold. From identity follows similarity and partial identity. From that follows partial similarity and general similarity. It seems advisable to require from a similarity measure the reflexivity, which means an object is similar to itself. Symmetry should be another property of similarity. Let us consider the statements "A is similar to 2?" or "A is the same as B". We notice that these statements are directed and that the roles of A and B can not be exchanged. People say: "A circle is like an ellipse." but not "An ellipse is like a circle." or "The son looks like the father." but not "The father looks like to the son.". Therefore, symmetry is not necessarily a basic property of similarity. However, in the above examples it can be useful to define the similarity relation to be symmetrical. The transitivity relation also must not necessarily hold. Let us consider the block world: a red ball and a red cube might be similar; a red cube and a blue square are similar; but a red ball and a blue square are dissimilar. However, a concrete similarity relation might be transitive. Similarity and identity are two concepts that strongly depend on the context. The context defines the essential attributes of the objects that are taken into consideration when similarity is determined. An object "red ball" may be similar to an object "red chair" because of the color "red". However the objects "ball" and "chair" are dissimilar. These attributes may be relevant, depending on whether they are given priority or saliency in the considered problem. This little example shows that the calculation of similarity between the attributes must be meaningful. It makes no sense to compare two attributes that do not make a contribution to the considered similarity. Since attributes can be numerical and categorical or a combination of both, we need to pay attention to this by the selection of the similarity measure. Not all similarity measures can be used for categorical attributes or can deal at the same time with numerical and categorical attributes. 7.2. Similarity measures for images Images can be rotated, translated, different in scale, or may have different contrast and energy, but they might be considered as similar. In contrast to that two images may be dissimilar since the object in one image is rotated by 180 degrees. The concept of
107
invariance in image interpretation is closely related to that of similarity. A good similarity measure should take this into consideration. The classical similarity measures do not allow this. Usually the images or the features have to be pre-processed in order to be adapted to the scale, orientation or shift. This process is a further processing step which is expensive and needs some a priori information which is not always given. Filters such as matched filters, linear filters, Fourier or Wavelet filters are especially useful for invariance under translation and rotation which has also been shown by Micarelli et al*. There has been a lot of work done to develop such filters for image interpretation in the past. The best way to achieve scale invariance from an image is by means of invariant moments, which can also be invariant under rotation and other distortions. Some additional invariance can be obtained by normalization (reduces the influence of energy). Depending on the image representation we can divide similarity measures into: • • •
pixel (Iconic)-matrix based similarity measures, feature-based similarity measures, (numerical or symbolical or mixed type) and, structural similarity measures.
Since a CBR image interpretation system has also to take into account non-image information, such as about the environment or the objects etc, we need similarity measures which can combine non-image and image information. We have shown a first approach to this in Ref. 3. To better understand the concept of similarity, systematic studies on the different kinds of image similarity have to be done. Zamperoni et. al20 studied how pixel-matrix based similarity measures behave under different real world influences such as translation, noise (spikes, salt and pepper noise), different contrast and so on. Image feature-based similarity measures have been studied from a broader perspective by Santini and Jain21. We have studied structural similarity in our work in Ref. 8. A study on matched filters and similarity has been done in Ref. 13. Otherwise, every new conference on pattern recognition contains proposals for new similarity measures for specific purposes and different kinds of image representation.
8.
Organization of a Case Base
Cases can be organized into a flat case base or in a hierarchical fashion. In a flat organization we have to calculate the similarity between the problem case and each case in memory. It is clear that this will take time, even if the case base is very large. Systems with a flat case base organization usually run on a parallel machine to perform retrieval in a reasonable time and do not allow the case base to grow over a predefined limit.
108 Maintenance is done by partitioning the case base into case clusters and by controlling the number and size of these clusters22. To speed up the retrieval process, a more sophisticated organization of the case base is necessary. This organization should allow separating the set of similar cases from those cases not similar to the recent problem at the earliest stage of the retrieval process. Therefore we need to find a relation p that allows us to order our case base. Definition 2: A binary relation p on a set CB is called a partial order on CB if it is reflexive, antisymmetric, and transitive. In this case the pair (CB, p) is called a partial ordered set or poset. The relation can be chosen depending on the application. One common approach is to order the case base based on the similarity value. The set of case can be reduced by the similarity measure to a set of similarity values. The relation < over these similarity values gives us a partial order over these cases. The derived hierarchy consists of nodes and edges. Each node in this hierarchy contains a set of cases that do not exceed a specified similarity value. The edges show the similarity relation between the nodes. The relation between two successor nodes can be expressed as follows: Lemma 1: Let z be a node and x and y are two successor nodes of z, then x subsumes z and y subsumes z. By tracing down the hierarchy, the space gets smaller and smaller, until finally a node will not have any successor. This node will contain a set of close cases. Among these cases is to be found the closest case to the query case. Although we still have to carry out matching, the number of matches will have decreased through the hierarchical ordering. The nodes can be represented by the prototypes of the set of cases assigned to the node. When classifying a query through the hierarchy the query is only matched with the prototype. Depending on the outcome of the matching process, the query branches right or left of the node. Such kind of hierarchy can be created by hierarchical or conceptual clustering8, k-d trees2"1 and decision trees'. There are also set-membership-based organizations known, such as semantic nets2 and object-oriented representations24.
9.
Learning in a CBR System
CBR management is closely related to learning. It aims to improve the performance of the system. Let X be a set of cases collected in a case base CB. The relation between
109 each case in the case base can be expressed by the similarity value sim. The case base can be partitioned into n case classes: n
C:CB = \JC, /=/
(2)
such that the intra-case class similarity is high and the inter-case class similarity is low. The set of cases in each class C can be represented by a representative who generally describes the cluster. This representative can be the prototype, the mediod, or an a priori selected case. Whereas the prototype implies that the representative is the mean of the cluster which can easily be calculated from numerical data. The mediod is the case whose sum of all distances to all other cases in a cluster is minimal. The relation between the different case classes C can be expressed by higher-order constructs expressed e.g. as super classes which give us a hierarchical structure over the case base. There are different learning strategies that can take place in a CBR system: •
• • •
Learning takes place when a new case x has to be stored into the case base such that: CBn+i = CBn u {x}. That means that the case base is incrementally updated according to the new case. It may incrementally learn the case classes and/or the prototypes representing the class.8 The relationship between the different cases or case classes may be updated according the new case classes.7 The system may learn the similarity measure. ' '
9.1. Learning new cases and forgetting old cases Learning new cases means just adding cases into the case base upon some notification. Closely related to case adding is case deletion or forgetting cases which have shown low utility. This should control the size of the case base. There are approaches that keep the size of the case base constant and delete cases that have not shown good utility within a fixed time window'. The failure rate is used as utility criterion. Given a period of observation of N cases, if the CBR component exhibits M failures in such a period, we define the failure rate as fr=M/N. Other approaches try to estimate the "coverage" of each case in memory and by using this estimate to guide the case memory revision process26. The adaptability to the dynamic of the changing environment that requires storing new cases in spite of the case base limit is addressed in Ref. 22. Based on intra class similarity it is decided whether a case is to be removed from or to be stored in a cluster. Case deletion in a pre-determined time window based on failure results-' might not be
110 appropriate for image interpretation Because a failure might mean that, instead of the retrieved case being erroneous, there is some relevant knowledge that we could not describe using features. Also, cases that occur infrequently (i.e., that have not been used recently) should be recognized by the system. The causes for case deletion or addition might differ from other CBR applications since images may be distorted and very noisy it might not be useful to store distorted representations. Determining which representation is distorted is sometimes not easy even if you have seen only a few images, and it is usually necessary to have domain knowledge that must also be built up over time. Imprecise or noisy measurements can be caused by some defects of illumination, the image acquisition device, or the object itself. If the image analysis cannot adapt to these measurements, or the reasoning process cannot handle it, then this might cause failure results. However, if this is a systematic event then it might be worthwhile to store the recent case in the case base. The last fact comes from the real world environment. It is not possible to determine all real world influences a priori. Thus, developers prefer to incorporate cases into a case base instead of forgetting them. Although case bases can grow very large, instead of forgetting cases, we would rather subdivide the case base into frequently vs. rarely used cases. This requires addressing the issue of how should the addition of cases into one of these two case bases be controlled, as well as their respective reasoning processes. 9.2. Learning of prototypes Learning of prototypes has been described in Ref. 8 for a flat organization of a case base and for a hierarchical representation of a case base in Ref. 7. The prototype or the representative of a case class is the most general representation of a case class. A class of cases is a set of cases sharing similar properties. The set of cases does not exceed a boundary for the intra class dissimilarity. Cases that are on the boundary of this hyperball have a maximal dissimilarity value. A prototype can be selected a priori by the domain user. This approach is preferable when the domain expert knows for sure the properties of the prototype. The prototype can be calculated by averaging over all cases in a case class or the median of the cases is chosen. If only a few cases are available in a class and subsequently new cases are stored in the class, then it is preferable to incrementally update the prototype according to the new cases.
111 9.3. Learning of higher-order
constructs
The ordering of the different case classes gives an understanding of how these case classes are related to each other. For two case classes which are connected by an edge similarity relation holds. Case classes that are located at a higher position in the hierarchy apply to a wider range of problems than those located near the leaves of the hierarchy. By learning how these case classes are related to each other, higher-order constructs are learnt.8 9.4. Learning of similarity By introducing feature weights we can put special emphasis on some features for the similarity calculation. It is possible to introduce local and global feature weights. A feature weight for a specific attribute is called local feature weight. A feature weight that averages over all local feature weights for a case is called global feature weight. This can improve the accuracy of the CBR system. By updating these feature weights we can learn similarity.27
10. Competence of Case Bases An important problem in image interpretation concerns system competence. We follow the definition in Ref. 26 and define the competence of a system as the range of target problems the system can solve. As we have already pointed out in Sec. 3 it is often not clear to the system developer what problems the desired algorithm can solve. So we have to find a way to describe the competence of a system. This differs from what is usually understood about this problem in CBR. Competence is described based on statistical properties such as case-base size, density and distribution, or group coverage and group density. But what if some groups overlap? Smyth and McKenna26 argue that these groups have shared competence and can be linked together in some way. However, we can also view it as having a poor description of the target problem. Based on this description we may retrieve a similar case but its solution application to the query image may be low in quality. By investigating the failure we may learn that we did not consider a property of the environment or maybe we could not specify it because it was not contained in the description of the target problem. Therefore, the system performance decreases. The measures described in Ref. 26 and Ref. 28 only view competence based on the coverage of the problem space. How do we know that cases in group 1 and group 2 belong to the same target problem group? Proximity in problem space does not imply that they belong to the same problem group; misclassifications can
112 occur because the patterns overlap. We argue that system competence must also account for the misclassification of the target problem based on the problem description. 11. Case Acquisition A CBR system for image classification needs to have some particular features with respect to images. These features result from: special requirements of visual knowledge acquisition (image-language problem) and the need to transform an image's numerical data into a symbolic description. The main problem with images and their translation into a language is that the knowledge about an image is usually tacit. To make this knowledge explicit is often hard. Sometimes the meaning of a word does not correspond to the correct meaning of the image. Therefore, it is necessary to support the operator in an efficient way. Most case-based image interpretation systems do not pay attention to this problem. The only functionality these systems provide is visualization of the image or the processed image. Usually, new case knowledge is obtained via manual acquisition with the expert. This is a time-consuming and sometimes boring process for both the system developer and the expert. A CBR system for image interpretation should have a special case acquisition tool. By using a questioning strategy and evaluating the answers given by the expert, the expert or operator is forced to specify the right knowledge for image interpretation. The questioning strategy is designed to force an expert to explain what distinguishes one object from another and to specify the right property for the object. A special case acquisition tool for image segmentation was described in Ref. 3. With the help of this tool the user can control the parameters of the image segmentation algorithm. Simultaneously, he can view the segmented image and, if he is satisfied with the segmentation quality, he can store the parameters of the image segmentation algorithm together with the case description in the case base. Acquiring cases by tracing image objects and learning more compact cases from these examples by case mining is described in Ref. 14.
12. Conclusions We surveyed special topics associated with a case-based image interpretation system. From our point of view case-based image interpretation differs in many aspects from other CBR applications that require further investigation. First, more systematic work on special image similarity measures is needed that investigates the measures under different influences that may occur in an image. Next, case representations are required for all the different abstraction levels of an image. Finally, the maintenance and learning strategies
113 must be defined so that they can help to improve the system performance and discover the range of target problems that the system can solve. We have deployed two CBR image interpretation systems. One is installed at the university hospital in Halle; it is used for image segmentation to determine the brain/liquor ratio of the head in a CT image. The second system is used to interpret ultrasonic SAFT images. In both applications the CBR strategies we used achieved good system performance that satisfied the users and outperformed other systems. The learning and maintenance facilities installed to date have been particularly well-received. We are recently going on to develop a new CBR based architecture for fungi identification.' 3 In summary, we believe that investigations of case-based image interpretation systems can reveal special challenges to both the CBR and computer vision communities, and encourage more people to work on this topic.
References 1.
J. Jarmulak. Case-based classification of ultrasonic b-scans: case-base organisation and case retrieval. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, pages 100-111. Springer Verlag. Berlin. 1998. 2. M. Grimnes and A. Aamodt. A two layer case-based reasoning architecture for medical image understanding. In I. Smith and B. Faltings (Eds.). Advances in Case-Based Reasoning, pages 164-178. Springer Verlag. Berlin. 1996. 3. P. Perner. An architecture for a CBR image segmentation system. In Journal of Engineering Application in Artificial Intelligence, vol. 12 (6). pages 749-759. 1999. 4. A. Micarelli. A. Neri and G. Sansonetti. A case-based approach to image recognition. In E. Blanzieri and L. Portinale (Eds.). Advances in Case-Based Reasoning, pages 443-454. Springer Verlag. Berlin, 2000. 5. V. Ficet-Cauchard. C. Porquet. and M. Revenu. CBR for the reuse of image processing knowledge: A recursive retrieval/adaption strategy. In K.-D. Althoff. R. Bergmann. and L.K. Branting (Eds.). Case-Based Reasoning Research and Development, pages 438-453. Springer Verlag, ^Berlin. 1999. 6. W. Cheetham and J. Graf. Case-based reasoning in color matching. In D.B. Leake and E. Plaza (Eds.). Case-Based Reasoning Research and Development, pages 1-12. Springer Verlag. Berlin. 1997. 7. P. Perner. Using CBR learning for the low-level and high-level unit of a image interpretation system. In Sameer Singh (Eds.). Advances in Pattern Recognition, pages 45-54. Springer Verlag. Berlin. 1998. 8. P. Perner. Incremental learning of retrieval knowledge in a case-based reasoning system. In K.D. Ashley and D.G. Bridge (Eds.). Case-Based Reasoning - Research and Development. Inai 2689. pages 422-436. Springer. Verlag Berlin. 2003. 9. P. Perner. H. Perner and B. Miiller. Similarity guided learning of the case description and improvement of the system performance in an image classification system. In S. Craw and A.Preece (Eds.). Advances in Case-Based Reasoning. Inai 2416. pages 604-612. Springer Verlag. Berlin. 2002. 10. P. Perner. Th. Guenther and H. Perner. Airborne fungi identification by case-based reasoning. In L. McGinty (Eds.). 5th Intern. Conference on Case-Based Reasoning, Workshop Proceedings. Case-Based Reasoning in Health Science, pages 73-79. 2003.
114 11. A. Holt and G.L. Benwell. Applying case-based reasoning techniques in GIS. In Journal of Geograpical Information Science 13(1), pages 9-25. 1999. 12. J.D. Carswell. D.C. Wilson and M. Bertolotto. Digital image similarity for geo-spatial knowledge management. In Proceedings of 6th European Conference on Case-Based Reasoning. Lecture Notes in Artificial Intelligence (ECCBR2002). Springer Verlag, Berlin 2002. 13. P. Perner and A. Buhring. Case-based object recognition. European Conference on CaseBased Reasoning ECCBR 2004. Springer Verlag. to be published. 14. P. Perner and S. Janichen. Learning of form models from exemplars. SSPR 2004. Springer Verlag 2004. to be published. 15. P. Perner. Are case-based reasoning and dissimilarity-based classification two sides of the same coin? In Journal Engineering Applications of Artificial Intelligence 15 (3). pages 205216. 16. S. Wess and C. Globig. Case-based and symbolic classification. In S. Wess. K.-D. Althoff and M.M. Richter M.M. (Eds.). Topics in Case-Based Reasoning, pages 77-91. Springer Verlag. Berlin. 1994. 17. F. Heister and W. Wilke. An architecture for maintaining case-based reasoning systems. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 221232. Springer Verlag. Berlin. 1998. 18. J.L. Arcos and E. Plaza. A reflective architecture for integrated memory-based learning and reasoning. In S. Wess. K.-D. Althoff and M.M. Richter (Eds.), Topics in Case-based Reasoning, pages 289-300. Springer Verlag. Berlin. 1993. 19. L.B. Smith. From global similarities to kinds of similarities: the construction of dimensions in development. In S. Vosniadou and A. Ortony (Eds.). Similarity and Analogical Reasoning. Cambridge University Press. 1989. 20. P. Zamperoni and V. Starovoitov. How dissimilar are two gray-scale images. In Proceedings of 17. DAGM Symposium, pages 448-455. Springer Verlag. Berlin. 1995. 21. S. Santini and R. Jain. Similarity measures. In IEEE Trans, on Pattern Analysis and Machine Intelligence 21(9). pages 871-883. 1999. 22. J. Surma and J. Tyburcy. A study on competence-preserving case replacing strategies in casebased reasoning. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning. lnai 1488. pages 233-238. Springer Verlag. Berlin. 1998. 23. S. Wess. K.-D. Althoff. and G. Derwand. Using k-d trees to improve the retrieval step in casebased reasoning. In S. Wess. K.-D. Althoff. and M.M. Richter (Eds.). Topics in Case-based Reasoning, pages 167-182. Springer Verlag. Berlin. 1993. 24. R. Bergmann and A. Stahl. Similarity measures for object-oriented case representations. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 2536. Springer Verlag. Berlin. 1998. 25. L. Portinale. P. Torasso. and P. Tavano. Speed-Up. quality and competence in multi-modal case-based reasoning. In: K.-D. Althoff. R. Bergmann and L. K. Branting (Eds.). Case-Based Reasoning Research and Development, lnai 1650. pages 303-317. Springer Verlag. Berlin. 1999. 26. B. Smyth and E. McKenna. Modelling the competence of case-bases. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 208-220. Springer Verlag. "Berlin. 1998. 27. D. Wettscherek. D.W. Aha and T. Mohri. A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms. In Artifical Intelligence Review (also available on the Web from http://www.aic.nrl.navy.mil/~aha). 28. D.B. Leake and D.C. Wilson. Remembering why to remember: performance-guided case-base maintenance. In E. Blanzieri and L. Portinale (Eds.). Advances in Case-Based Reasoning, pages 161-172. Springer Verlag. Berlin. 2000.
C H A P T E R 2.2 MULTIPLE IMAGE GEOMETRY - A PROJECTIVE
VIEWPOINT
Quang-Tuan Luong AIC - SRI International 333 Ravenswood Ave, Menlo Park, CA 94025, USA A central problem in computer vision is to reconstruct a 3D scene given multiple images of it. In this chapter, we describe a general geometric framework based on projective geometry. Prom a theoretical point of view, it leads to a simpler and more unified description of the problem than the traditional Euclidean framework. Prom a practical point of view, it makes it possible to use cameras without knowing beforehand their internal parameters. After a brief overview of projective geometry and its application to camera modeling, we examine how to compute the epipolar geometry and planar correspondences of a pair of uncalibrated views from point matches (Fundamental matrix and homographies). We then move to the problem of computing the 3D positions of the points that generate these matches, as well as the cameras ( stratification of reconstruction in projective, affine and Euclidean stages). We eventually tackle the problem of computing "on the fly" the internal calibration of a camera from a sequence of ordinary images (self-calibration). 1. P r o j e c t i v e g e o m e t r y Projective geometry is an extension of Euclidean geometry, which describes a larger class of transformations t h a n just rotations and translations, including in particular the perspective projection performed by a camera. This makes it the n a t u r a l framework to study the relationship between 3D points and their projections in images. 1.1. Projective
coordinates
T h e space of (n + l)-tuples of coordinates, with t h e rule t h a t proportional (n + 1)tuples represent t h e same point, is called t h e projective space of dimension n and denoted P n . This rule is introduced so t h a t there is a one-to-one correspondence between "usual" n Euclidean coordinates and n + 1 projective coordinates. T h e object space is P 3 and t h e image space is P 2 , called the projective plane. Given coordinates in IR" we build projective coordinates by: [ E l , . . . ,Xnf
-)• [X!,...
115
,Xn,l]T.
116
To transform a point in the projective coordinates back into Euclidean coordinates, we just divide by the last coordinate and then drop it: [xi,...
,xn,xn+1]
-] T
-> [Xn+l
%n+l
Projective coordinates represent naturally the operation performed by a camera. The projective space contains more points than the Euclidean space. Points with coordinates [xi,... ,xn,xn+i]T with xn+1 ^ 0 can be viewed as the usual points, whereas the points with coordinates [xi,... ,a; n ,0] T have no Euclidean equivalent. If we consider them as the limit of [2:1,... , xn, X]T, when X —>• 0 i.e. the limit of [xi/X,... ,xn/X, 1] T , then we see that they are the limit of a point of M" going to infinity in the direction [xi,... ,xn]T, hence the appellation point at infinity. The projective space P n can be viewed as the union of the usual space K" (points [xi,... , xn, 1]T) and the set of points at infinity [x\,... , xn, 0] T . 1.2. The projective
plane
The equation of a line I, au + bv + c = 0 can be written using projective coordinates m = [x, y, z]T of the point m: l T m = m T l