A 3rd Edition
r y d
D
o o <.
0
f
Pattern Recognition and Computer Vision
This page is intentionally left blank
A 3rd Edition
r J i b o o <
of
Pattern Recognition and Computer Vision editors
C H Chen University of Massachusetts Dartmouth, USA
P S P Wang Northeastern University, USA
\[p World Scientific NEW JERSEY • LONDON • SINGAPORE •
BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI
Published by World Scientific Publishing Co. Pte. Ltd. 5 Tori Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
First published 2005 Reprinted 2006
HANDBOOK OF PATTERN RECOGNITION & COMPUTER VISION (3rd Edition) Copyright © 2005 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN 981-256-105-6
Printed in Singapore by Mainland Press
Preface to the Third Edition
Dedicated to the memory of the late Professor King Sun Fu (1930-1985), the handbook series, with first edition (1993), second edition (1999) and third edition (2005), provides a comprehensive, concise and balanced coverage of the progress and achievements in the field of pattern recognition and computer vision in the last twenty years. This is a highly dynamic field which has been expanding greatly over the last thirty years. No handbook can cover the essence of all aspects of the field and we have not attempted to do that. The carefully selected 33 chapters in the current edition were written by leaders in the field and we believe that the book and its sister volumes, the first and second editions, will provide the growing pattern recognition and computer vision community a set of valuable resource books that can last for a long time. Each chapter will speak for itself the importance of the subject area covered. The book continues to contain five parts. Part 1 is on the basic methods of pattern recognition. Though there are only five chapters, the readers may find other coverage of basic methods in the first and second editions. Part 2 is on basic methods in computer vision. Again readers may find that Part 2 complements well what were offered in the first and second editions. Part 3 on recognition applications continues to emphasize on character recognition and document processing. It also presents new applications in digital mammograms, remote sensing images and functional magnetic resonance imaging data. Currently one intensively explored area of pattern recognition applications is the personal identification problem, also called biometrics, though the problem has been around for a number of years. Part 4 is especially devoted to this topic area. Indeed chapters in both Part 3 and Part 4 represent the growing importance of applications in pattern recognition. In fact Prof. Fu had envisioned the growth of pattern recognition applications in the early 60's He and his group at Purdue had worked on the character recognition, speech recognition, fingerprint recognition, seismic pattern recognition, biomedical and remote sensing recognition problems, etc. Part 5 on system and technology presents other important aspects of pattern recognition and computer vision. Our sincere thanks go to all contributors of this volume for their outstanding technical contributions. We would like to mention specially Dr. Quang-Tuan Luong, Dr. Giovanni Garibotto and Prof. Ching Y. Suen for their original contributions to all three volumes. Other authors who have contributed to all three volumes are: Prof. Thomas S. Huang, Prof. J.K. Aggarwal, Prof. Yun Y. Tang, Prof. C.C. Li, Prof. R. Chellappa and Prof. P.S.P. Wang. We are pleased to mention that Prof. Thomas Huang and Prof. Jake Aggarwal are the recipients respectively in 2002 and 2004, of the prestigious K.S. Fu Prize sponsored by the International Association of Pattern Recognition (IAPR). Among Prof. Fu's Ph.D. graduates at Purdue who have contributed to the handbook series are: C.H. Chen (1965), M.H. Loew (1972), S.M. Hsu (1975), S.Y. Lu (1977), K.Y. Huang (1983) and H.D. Cheng (1985). Finally we would like to pay tribute to the late Prof. Azirel Rosenfeld (1931-2004) who, as one IAPR member put it, is a true scientist and a great giant in the field. He was awarded the K.S. Fu Prize by IAPR in 1988. Readers are reminded to read Prof. Rosenfeld's inspirational article on "Vision - Some Speculations" that appeared as Foreword of the second edition of the handbook series. Prof. Rosenfeld's profound influence in the field will be felt in the many years to come. v
VI
The camera ready manuscript production requires certain amount of additional efforts, as compared to typeset printing, on the part of editors and authors. We like to thank all contributors for their patience in making the necessary revisions to comply with the format requirements during this long process of manuscript preparation. Our special thanks go to Steven Patt, in-house editor of World Scientific Publishing, for his efficient effort to make a timely publication of the book possible.
September, 2004
The Editors
Contents
Preface to the Third Edition
v
Contents
vii
Part 1. Basic Methods in Pattern Recognition
1
Chapter 1.1
Statistical Pattern Recognition R.P.W. Duin and D.M.J. Tax
3
Chapter 1.2
Hidden Markov Models for Spatio-Temporal Pattern Recognition Brian C. Lovell and Terry Caelli
25
A New Kernel-Based Formalization of Minimum Error Pattern Recognition Erik McDermott andShigeru Katagiri
41
Chapter 1.3
Chapter 1.4
Parallel Contextual Array Grammars with Trajectories P. Helen Chandra, C. Martin-Vide, K.G. Subramanian, D.L. Van and P. S. P. Wang
55
Chapter 1.5
Pattern Recognition with Local Invariant Features C. Schmid, G. Dorko, S. Lazebnik, K. Mikolajczyk and J. Ponce
71
Part 2. Basic Methods in Computer Vision
93
Chapter 2.1
Case-Based Reasoning for Image Analysis and Interpretation Petra Perner
95
Chapter 2.2
Multiple Image Geometry - A Projective Viewpoint Quang-Tuan Luong
115
Chapter 2.3
Skeletonization in 3D Discrete Binary Images Gabrielle Sanniti di Baja and Injela Nystrom
137
Chapter 2.4
Digital Distance Transforms in 2D, 3D, and 4D Gunilla Borgefors
157
Chapter 2.5
Computing Global Shape Measures Paul L. Rosin
177
VII
VIII
Chapter 2.6
Texture Analysis with Local Binary Patterns Topi Mdenpdd and Matti Pietikdinen
197
Part 3. Recognition Applications
217
Chapter 3.1
Document Analysis and Understanding Yuan Yan Tang
219
Chapter 3.2
Chinese Character Recognition Xiaoqing Ding
241
Chapter 3.3
Extraction of Words from Handwritten Legal Amounts on Bank Cheques In Cheol Kim and Ching Y. Suen
259
OCR Assessment of Printed-Fonts for Enhancing Human Vision Ching Y. Suen, Qizhi Xu and Cedric Devoghelaere
273
Chapter 3.4
Chapter 3.5
Clustering and Classification of Web Documents Using a Graph Model Adam Schenker, Horst Bunke, Mark Last and Abraham Kandel
287
Chapter 3.6
Automated Detection of Masses in Mammograms H.D. Cheng, X.J. Shi, R. Min, X.P. Cai and H.N. Du
Chapter 3.7
Wavelet-Based Kalman Filtering in Scale Space for Image Fusion Hsi-Chin Hsin and Ching-Chung Li
325
Multisensor Fusion with Hyperspectral Imaging Data: Detection and Classification Su May Hsu and Hsiao-hua Burke
347
Independent Component Analysis of Functional Magnetic Resonance Imaging Data V.D. Calhoun andB. Hong
365
Chapter 3.8
Chapter 3.9
303
Part 4. Human Identification
385
Chapter 4.1
Multimodal Emotion Recognition Nicu Sebe, Ira Cohen and Thomas S. Huang
387
Chapter 4.2
Gait-Based Human Identification from a Monocular Video Sequence Amit Kale, AravindSundaresan, AmitK. RoyChowdhury and Rama Chelappa
411
IX
Chapter 4.3
Palmprint Authentication System David Zhang
Chapter 4.4
Reconstruction of High-Resolution Facial Images for Visual Surveillance Jeong-Seon Park and Seong Whan Lee
445
Object Recognition with Deformable Feature Graphs: Faces, Hands, and Cluttered Scenes Jochen Triesch and Christian Eckes
461
Chapter 4.5
Chapter 4.6
Hierarchical Classification and Feature Reduction for Fast Face Detection Bernd Heisele, Thomas Serre, Sam Prentice and Tomaso Poggio
Part 5. System and Technology Chapter 5.1
Tracking and Classifying Moving Objects Using Single or Multiple Cameras Quming Zhou andJ.K. Aggarwal
431
481
497 499
Chapter 5.2
Performance Evaluation of Image Segmentation Algorithms Xiaoyi Jiang
525
Chapter 5.3
Contents-Based Video Analysis for Knowledge Discovery Chia-Hung Yeh, Shih-Hung Lee andC.-C. Jay Kuo
543
Chapter 5.4
Object-Process Methodology and Its Applications to Image Processing and Pattern Recognition Dov Dori
559
Chapter 5.5
Musical Style Recognition — AQuantative Approach Peter van Kranenburg and Eric Backer
583
Chapter 5.6
Auto-Detector: Mobile Automatic Number Plate Recognition Giovanni B. Garribotto
601
Chapter 5.7
Omnidirectional Vision Hiroshi Ishiguro
619
Index
629
C H A P T E R 1.1 STATISTICAL PATTERN
RECOGNITION
R.P.W. Duin, D.M.J. Tax Information and Communication Theory Group Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology P.O.Box 5031, 2600 GA, Delft, The Netherlands E-mail: {R.P. W.Duin,D. M.J. Tax} @ ewi.tudelft.nl A review is given of the area of statistical pattern recognition: the representation of objects and the design and evaluation of trainable systems for generalization. Traditional as well as more recently studied procedures are reviewed like the classical Bayes classifiers, neural networks, support vector machines, one-class classifiers and combining classifiers. Further we introduce methods for feature reduction and error evaluation. New developments in statistical pattern recognition are briefly discussed.
1. I n t r o d u c t i o n Statistical p a t t e r n recognition is the research area t h a t studies statistical tools for the generalization of sets of real world objects or phenomena. It thereby aims to find procedures t h a t answer questions like: does this new object fit into the p a t t e r n of a given set of objects, or: to which of the p a t t e r n s defined in a given set does it fit best? T h e first question is related to cluster analysis, b u t is also discussed from some perspective in this chapter. T h e second question is on p a t t e r n classification and t h a t is what will be the main concern here. T h e overall structure of a p a t t e r n recognition system may be summarized as in Figure 1. Objects have first to be appropriately represented before a generalization can be derived. Depending on the demands of the procedures used for this the representation has to be adapted, e.g. transformed, scaled or simplified. T h e procedures discussed in this chapter are partially also studied in areas like statistical learning theory 3 2 , machine learning 2 5 and neural networks 1 4 . As the emphasis in p a t t e r n recognition is close to application areas, questions related to the representation of the objects are important here: how are objects described (e.g. features, distances to prototypes), how extensive may this description be, what are the ways to incorporate knowledge from the application domain? Representations have to be adapted to fit the tools t h a t are used later. Simplifications of representations 3
4
like feature reduction and prototype selection should thereby be considered. In order to derive, from a training set, a classifier that is valid for new objects (i.e. that it is able to generalize) the representation should fulfill an important condition: representations of similar real world objects have to be similar as well. The representations should be close. This is the so-called compactness hypothesis 2 on which the generalization from examples to new, unseen objects is built. It enables the estimation of their class labels on the basis of distances to examples or on class densities derived from examples. Objects are traditionally represented by vectors in a feature space. An important recent development to incorporate domain knowledge is the representation of objects by their relation to other objects. This may be done by a so called kernel method 29 , derived from features, or directly on dissimilarities computed from the raw data 2 6 . We will assume that, after processing the raw measurements, objects are given in a p-dimensional vector space Cl. Traditionally this space is spanned by p features, but also the dissimilarities with p prototype objects may be used. To simplify the discussion we will use the term feature space for both. If K is the number of classes to be distinguished, a pattern classification system, or shortly classifier C{x) is a function or a procedure that assigns to each object x in fi a class u>c, with c = 1,...,K. Such a classifier has to be derived from a set of examples Xtv = {xi,i = 1...N} of known classes y,. Xtr will be called the training set and yi £ wc, c = 1...K a label. Unless otherwise stated it is assumed that yt is unique (objects belong to just a single class) and is known for all objects in Xtl. In section 2 training procedures will be discussed to derive classifiers C(x) from training sets. The performance of these classifiers is usually not just related the quality of the features (their ability to show class differences) but also to their number, i.e. the dimensionality of the feature space. A growing number of features
charactt rization 4
evaluation
update
•s
generalization
update
adaptation
a update
representation
class labels,
classifiers, class models
feature extraction prototype selection
features, dissimilarities class models, object models
larger tra ning sets
^ obj ects better sensors or measurement conditions
Fig. 1.
confidences
The pattern recognition system
5
may increase the class separability, but, may also decrease the statistical accuracy of the training procedure. It is thereby important to have a small number of good features. In section 3 a review is given of ways to reduce the number of features by selection or by combination (so called feature extraction). The evaluation of classifiers, discussed in section 4, is an important topic. As the characteristics of new applications are often unknown before, the best algorithms for feature reduction and classification have to be found iteratively on the basis of unbiased and accurate testing procedures. This chapter builds further on earlier reviews of the area of statistical pattern recognition by Fukunaga 12 and by Jain et al 16 . It is inevitable to repeat and summarize them partly. We will, however, also discuss some new directions like oneclass classifiers, combining classifiers, dissimilarity representations and techniques for building good classifiers and reducing the feature space simultaneously. In the last section of this chapter, the discussion, we will return to these new developments. 2. Classifiers For the development of classifiers, we have to consider two main aspects: the basic assumptions that the classifier makes about the data (which results in a functional form of the classifier), and the optimization procedure to fit the model to the training data. It is possible to consider very complex classifiers, but without efficient methods to fit these classifiers to the data, they are not useful. Therefore, in many cases the functional form of the classifier is restricted by the available optimization routines. We will start discussing the two-class classification problem. In the first three sections, 2.1, 2.2 and 2.3, the three basic approaches with their assumptions are given: first, modeling the class posteriors, second, modeling class conditional probabilities and finally modeling the classification boundary. In section 2.4 we discuss how these approaches can be extended to work for more than two classes. In the next section, the special case is considered where just one of the classes is reliably sampled. The last section, 2.6, discusses the possibilities to combine several (non-optimal) classifiers. 2.1. Bayes
classifiers
and
approximations
A classifier should assign a new object x to the most likely class. In a probabilistic setting this means that the label of the class with the highest posterior probability should be chosen. This class can be found when p(wi|x) and p(w2|x) (for a two class classification problem) are known. The classifier becomes: if p(wi|x) > p(w2|x)
assign object x to u>\, otherwise to W2-
(1)
When we assume that p(ui\x) and p(w2|x) are known, and further assume that misclassifying an object originating from o>i to W2 is as costly as vise versa, classifier (1) is the theoretical optimal classifier and will make the minimum error. This classifier is called the Bayes optimal classifier.
6
In practice p(wi|x) and p(w2|x) are not known, only samples x, are available, and the misclassification costs might be only known in approximation. Therefore approximations to the Bayes optimal classifier have to be made. This classifier can be approximated in several different ways, depending on knowledge of the classification problem. The first way is to approximate the class posterior probabilities p(w c |x). The logistic classifier assumes a particular model for the class posterior probabilities: P(wi|x) =
, ( , T ^, 1 + exp(—w T x)
1
P(w 2 |x) = 1 - p ( w 2 | x ) ,
(2)
where w is a p-dimensional weight vector. This basically implements a linear classifier in the feature space. An approach to fit this logistic classifier (2) to training data Xu, is to maximize the data likelihood L: N
L = np(wi|xO n i ( x ) p(w2|xi) n a ( x ) >
(3)
i
where n c (x) is 1 if object x belongs to class o>c, and 0 otherwise. This can be done by, for instance, an iterative gradient ascent method. Weights are iteratively updated using: Wnew = W 0 l d + ? 7 - — ,
(4)
where rj is a suitably chosen learning rate parameter. In Ref. 1 the first (and second) derivative of L with respect to w are derived for this and can be plugged into (4). 2.2.
Class densities
and Bayes
rule
Assumptions on p(w|x) are often difficult to make. Sometimes it is more convenient to make assumptions on the class conditional probability densities p(x|w): they indicate the distribution of the objects which are drawn from one of the classes. When assumptions on these distributions can be made, classifier (1) can be derived using Bayes' decision rule: PH*)
=? » ! .
(5)
This rule basically rewrites the class posterior probabilities in terms of the class conditional probabilities and the class priors p(w). This result can be substituted into (1), resulting in the following form: if P(X\LOI)P(U>I) > p(x|w2)p(w2)
assign x to wi, otherwise to o;2-
(6)
The term p(x) is ignored because this is constant for a given x. Any monotonically increasing function can be applied to both sides without changing the final decision. In some cases, a suitable choice will simplify the notation significantly. In particular,
7
using a logarithmic transformation can simplify the classifier when functions from the exponential family are used. For the special case of a two-class problem the classifiers can be rewritten in terms of a single discriminant function / ( x ) which is the difference between the left hand side and the right hand side. A few possibilities are: / ( x ) =p(wi|x) - p ( w 2 | x ) ,
(7)
/ ( x ) = p(x|on)p(wi) - p(x|w 2 )p(w 2 ),
(8)
/(x)=ln^+ln^ll.
(9)
p(x|w 2 )
P{LU2)
The classifier becomes: if
/(x) > 0
assign x t o w j , otherwise to o>2.
(10)
In many cases fitting p(x|a>) on training data is relatively straightforward. It is the standard density estimation problem: fit a density on a data sample. To estimate each p(x|w) the objects from just one of the classes UJ is used. Depending on the functional form of the class densities, different classifiers are constructed. One of the most common approaches is to assume a Gaussian density for each of the classes: p(x|w) = J V ( X ; / I , E ) =
( 2 7 r ) P /2| S |i/2
ex
P (-^(x-MfS-^x-M)) -
(n)
where /x is the (p-dimensional) mean of the class u>, and £ is the covariance matrix. Further, |£| indicates the determinant of £ and £ - 1 its inverse. For the explicit values of the parameters /x and E usually the maximum likelihood estimates are plugged in, therefore this classifier is called the plug-in Bayes classifier. Extra complications occur when the sample size TV is insufficient to (in particular) compute £ _ 1 . In these cases a standard solution is to regularize the covariance matrix such that the inverse can be computed: EA = £ + AJ,
(12)
where X is the p x p identity matrix, and A is the regularization parameter to set the trade off between the estimated covariance matrix and the regularizer I. Substituting (11) for each of the classes u>\ and w2 (with their estimated /Xj, /x2 and Ei, £ 2 ) into (9) results in: /(x) = i x W
- E r > + ^(MiSr 1 - M 2 S 2 - 1 ) T x
- ^ E r V i + ^ S j V s - i l n l E t l + ilnlE.I+ln^ij.
(13)
This classifier rule is quadratic in terms of x, and it is therefore called the normalbased quadratic classifier. For the quadratic classifier a full covariance matrix has to be estimated for each of the classes. In high dimensional feature spaces it can happen that insufficient
8
data is available to estimate these covariance matrices reliably. By restricting the covariance matrices to have less free variables, estimations can become more reliable. One approach the reduce the number of parameters, is to assume that both classes have an identical covariance structure: Ei = £2 = E. The classifier simplifies to: / ( x ) = \{^
- . / S - ' x - i / i f E - V i + \l%*-
V 2 + In ^
(14)
Because this classifier is linear in terms of x, this classifier is called the normal-based linear classifier. For the linear and the quadratic classifier, strong class distributional assumptions are made: each class has a Gaussian distribution. In many applications this cannot be assumed, and more flexible class models have to be used. One possibility is to use a 'non-parametric' model. An example is the Parzen density model. Here the density is estimated by summing local kernels with a fixed size h which are centered on each of the training objects: 1 N p(x|w) = - ^ J V f c x i . f t l ) ,
(15)
i—l
where X is the identity matrix and h is the width parameter which has to be optimized. By substituting (15) into (6), the Parzen classifier is defined. The only free parameter in this classifier is the size (or width) h of the kernel. Optimizing this parameter by maximizing the likelihood on the training data, will result in the solution h = 0. To avoid this, a leave-one-out procedure can be used 9 . 2.3. Boundary
methods
Density estimation in high dimensional spaces is difficult. In order to have a reliable estimate, large amounts of training data should be available. Unfortunately, in many cases the number of training objects is limited. Therefore it is not always wise to estimate the class distributions completely. Looking at (1), (6) and (10), it is only of interest which class is to be preferred over the other. This problem is simpler than estimating p(x\u>). For a two-class problem, we just a function / ( x ) is needed which is positive for objects of UJI and negative otherwise. In this section we will list some classifiers which avoid estimating p(x|w) but try to obtain a suitable / ( x ) . The Fisher classifier searches to find a direction w in the feature space, such that the two classes are separated as well as possible. The degree in which the two classes are separated, is measured by the so-called Fisher ratio, or Fisher criterion:
j = \™\~mf.
(16)
y sl + s22 ' Here mi and mo, are the means of the two classes, projected onto the direction w: mi = ~wTHi and rri2 = w T /x 2 . The si and s% are the variances of the two classes projected onto w. The criterion therefore favors directions in which the means are far apart and the variances are small.
9
This Fisher ratio can be explicitly rewritten in terms of w. First we rewrite s
l = E x 6 a , c ( w T x - w 7 > c ) 2 = Exea, c w T ( x - Mc)(x - ^ c ) T w = w r S c w . Second we write (mi - m 2 ) 2 = (w T /x 1 — w T /z 2 ) 2 = w T (/x 1 — /x2)(Ati _ i u 2 ) T w = w T SflW. The term 5 s is also called the between scatter matrix. J becomes: _ \rn-j - m 2 | 2 _ wr5Bw _ wTSBw T T , T sf + s?, w 5 i w + w S 2W w 5iyw' where Sw = S\ + S2 is also called the within scatter matrix. In order to optimize (17), we set the derivative of (17) to zero and obtain: (w T 5.Bw)5iyw = (wrS,^vw)S'BW.
(18)
We are interested in the direction of w and not in the length, so we drop the scalar terms between brackets. Further, from the definition of SB it follows that SBW is always in the direction fj,x — fj,2- Multiplying both sides of (18) by Sw gives: w~S^(Mi-M2)-
(19)
This classifier is known as the Fisher classifier. Note that the threshold b is not defined for this classifier. It is also linear and requires the inversion of the withinscatter Sw- This formulation yields an identical shape of w as the expression in (14), although the classifiers use very different starting assumptions! Most classifiers which have been discussed so far, have a very restricted form of their decision boundary. In many cases these boundaries are not flexible enough to follow the true decision boundaries. A flexible method is the k-nearest neighbor rule. This classifier looks locally which labels are most dominant in the training set. First it finds the k nearest objects in the training set AW(x), and then counts the number of these neighbors are from class u>i or u>2if
ni > 712 assign x t o u i , otherwise to cj2-
(20)
Although the training of the fc-nearest neighbor classifier is trivial (it only has to store all training objects, k can simply be optimized by a leave-one-out estimation), it may become expensive to classify a new object x. For this the distances to all training objects have to be computed, which may be prohibitive for large training sets and high dimensional feature spaces. Another classifier which is flexible but does not require the storage of the full training set is the multi-layered feed-forward neural network4. A neural network is a collection of small processing units, called the neurons, which are interconnected by weights w and v to form a network. A schematic picture is shown in Figure 2. An input object x is processed through different layers of neurons, through the hidden layer to the output layer. The output of the j - t h output neuron becomes: °j(*) = hj nT.vMvrTy)]
(21)
(see Figure 2 for the meaning of the variables). The object x is now assigned to the class j for which the corresponding output neuron has the highest output Oj.
10
X\ • l+exp(-vTh)
X2%
• Xp # •—-
<*n
Fig. 2. Schematic picture of a neural network.
To optimize this neural network, the squared error between the network output and the desired class label is defined: N
£
K
= EE(ni^)-°iW)2-
(22)
where rij (x) is 1 if object x belongs to class w,-, and 0 otherwise. To simplify the notation, we will combine all the weights w, and v into one weight vector w. This error E is a continuous function of the weights w, and the derivative of E with respect to these weights can easily be calculated. The weights of the neural network can therefore be optimized to minimize the error by gradient descent, analogous to (4): dE W n e w = Wold - T y — ,
(23)
where 77 is the learning parameter. After expanding this learning rule (23), it appears that the weight updates for each layer of neurons can be computed by back-propagating the error which is computed at the output of the network (nj(xj) — Oj(xi)). This is therefore called the back-propagation update rule. The advantage of this type of neural networks is that they are flexible, and that they can be trained using these update rules. The disadvantages are, that there are many important parameters to be chosen beforehand (the number of layers, the number of neurons per layer, the learning rate, the number of training updates, etc.), and that the optimization can be extremely slow. To increase the training speed, several additions and extensions are proposed, for instance the inclusion of momentum terms in (23), or the use of second order moments. Neural networks can be easily overtrained. Many heuristic techniques have been developed to decrease the chance of overtraining. One of the methods is to use weight decay, in which an extra regularization term is added to equation (22). This regularization term, often something of the form E = E + \\\w\\2,
(24)
tries to reduce the size of the individual weights in the network. By restricting the size of the weights, the network will adjust less to the noise in the data sample
11
and become less complex. The regularization parameter A regulates the trade-off between the classification error E and the classifier complexity. When the size of the network (in terms of the number of neurons) is also chosen carefully, good performances can be achieved by the neural network. A similar approach is chosen for the support vector classifier32. The most basic version is just a linear classifier as in Eq. (10) with / ( x ) = w T x + 6.
(25)
The minimum distance from the training objects to the classifier is thereby maximized. This gives the classifier some robustness against noise in the data, such that it will generalize well for new data. It, appears that this maximum margin p is inversely related to ||w|| 2 , such that maximizing this margin means minimizing ||w|| 2 (taking the constraints into account that all the objects are correctly classified).
wrx + 6 = 0
1 1
o
o o
Fig. 3.
1
1
/*;
o 4
/
,P
f
i
o
f
1
/
i 1
o
1
|
I
o
1 f
1
• ' t
1 i
1 / 1
i i
•
• cti = 0
•
o
Schematic picture of a support vector classifier
Given linearly separable data, the linear classifier is found which has the largest margin p to each of the classes. To allow for some errors in the classification, some slack variables are introduced to weaken the hard constraints. The error to minimize for the support vector classifier therefore consists of two parts: the complexity of the classifiers in terms of w T w , and the number of classification errors, measured by X^jCi- The optimization can be stated by the following mathematical formulation: min w T w + C y " & , w
(26)
'-—' t
such that (w ( Xi + b > 1 - &
if
1 w ' x , + b < - l + ?i
otherwise.
xewi,
12 Parameter C determines the trade-off between the complexity of the classifier, as measured by w T w , and the number of classification errors. Although the basic version of the support vector classifier is a linear classifier, it can be made much more powerful by the introduction of kernels. When the constraints (27) are incorporated into (26) by the use of Lagrange multipliers a, this error can be rewritten in the so-called dual form. For this, we define the labels y, where y» = 1 when x$ £ ui\ and j/j = — 1 otherwise. The optimization becomes: maxaTxxTa - lTa,
s.t.
y T a = 0,
0 < at < C, Vi
(28)
with w = J2i aiVixi- Due to the constraints in (28) the optimization is not trivial, but standard software packages exist which can solve this quadratic programming problem. It appears that in the optimal solution of (28) many of the Qj become 0. Therefore only a few Qj ^ 0 determine the w. The corresponding objects Xi are called the support vectors. All other objects in the training set can be ignored. The special feature of this formulation is that both the classifier / ( x ) and the error (28) are completely stated in terms of inner products between objects xfXj. This means that the classifier does not explicitly depend on the features of the objects. It depends on the similarity between the object x and the support vectors Xj, measured by the inner product x T Xj. By replacing the inner product by another similarity, defined by the kernel function isT(x, x,), other non-linear classifiers are obtained. One of the most popular kernel functions is the Gaussian kernel: ir(x,x,) = e x p ( -
l | x
^
1 1 2
),
(29)
where a is still a free parameter. The drawback of the support vector classifier is that it requires the solution of a large quadratic programming problem (28), and that suitable settings for the parameters C and a have to be found. On the other hand, when C and a are optimized, the performance of this classifier is often very competitive. Another advantage of this classifier is, that it offers the possibility to encode problem specific knowledge in the kernel function K. In particular for problems where a good feature representation is hard to derive (for instance in the classification of shapes or text documents) this can be important. 2.4. Multi-class
classifiers
In the previous section we focused on the two-class classification problems. This simplifies the formulation and notation of the classifiers. Many classifiers can trivially be extended to multi-class problems. For instance the Bayes classifier (1) becomes: assign x to
w c .,
c* = argmaxp(w c |x)
(30)
c
Most of the classifiers directly follow from this. Only the boundary methods which were constructed to explicitly distinguish between two classes, for instance the
13 Fisher classifier or the support vector classifier, cannot be trivially extended. For these classifiers several combining techniques are available. The two main approaches to decompose a multi-class problem into a set of two-class problems are: (1) one-against-all: train K classifiers between one of the classes and all others, (2) one-against-one: train K(K — l ) / 2 classifiers to distinguish all pairwise classes. Afterward classifiers have to be combined using classification confidences (posterior probabilities) or by majority voting. A more advanced approach is to use ErrorCorrecting Output Codes (ECOC), where classifiers are trained to distinguish specific combinations of classes, but are allowed to ignore others 7 . The classes are chosen such that a redundant output labeling appears, and possible classification errors can be fixed. 2.5. One-class
classifiers
A fundamental assumption in all previous discussions, is that a representative training set XtT is available. That means that examples from both classes are present, sampled according to their class priors. In some applications one of the classes might contain diverse objects, or its objects are difficult or expensive to measure. This happens for instance in machine diagnostics or in medical applications. A sufficient number of representative examples from the class of ill patients or the class of faulty machines are sometimes hard to collect. In these cases one cannot rely on a representative dataset to train a classifier, and a so-called one-class classifier30 may be used.
Fig. 4.
One-class classifier example.
In one-class classifiers, it is assumed that we have examples from just one of the classes, called the target class. From all other possible objects, per definition the outlier objects, no examples are available during training. When it is assumed that the outliers are uniformly distributed around the target class, the classifier should
14 circumscribe the target object as tight as possible in order to minimize the chance of accepting outlier objects. In general, the problem of one-class classification is harder than the problem of conventional two-class classification. In conventional classification problems the decision boundary is supported from both sides by examples of both classes. Because in the case of one-class classification only one set of data is available, only one side of the boundary is supported. It is therefore hard to decide, on the basis of just one class, how strictly the boundary should fit around the data in each of the feature directions. In order to have a good distinction between the target objects and the outliers, good representation of the data is essential. Approaches similar to standard two-class classification can be used here. Using the uniform outlier distribution assumption, the class posteriors can be estimated and the class conditional distributions or direct boundary methods can be constructed. For high dimensional spaces the density estimators suffer and often boundary methods are to be preferred. 2.6. Combining
classifiers
In practice it is hard to find (and train) a classifier which fits the data distribution sufficiently well. The model can be difficult to construct (by the user), too hard to optimize, or insufficient training data is available to train. In these cases it can be very beneficial to combine several "weak" classifiers in order to boost the classification performance 21 . It is hoped that each individual classifier will focus on different aspects of the data and err on different objects. Combining the set of so-called base classifiers will then complement their weak areas.
y, base classifier 1 / feature/ space \
classification base classifier 2
base classifier 3 /
Fig. 5.
combining classifier
base classifier outputs (e.g. confidences)
Combining classifier
The most basic combining approach is to train several different types of classifiers on the same dataset and combine their outputs. One has to realize that classifiers can only correct each other when their outputs vary, i.e. the set of classifiers is diverse 22 . It appears therefore to be more advantageous to combine classifiers which were trained on objects represented by different features. Another approach to force classifiers to become diverse is to artificially change the training set by resampling (resulting in a bagging6 or a boosting 8 approach).
15 The outputs of the classifier can be combined using several combining rules 18 , depending on the type of classifier outputs. If the classifiers provide crisp output labels, a voting combining rule has to be used. When the real valued outputs are available, they can be averaged, weighted averaged or multiplied, the maximum or minimum output can be taken or even an output classifier can be trained. If fixed (i.e. not trained) rules are used, it is important that the output of a classifier is properly scaled. Using a trainable combining rule, this constraint can be elevated but clearly training data is required to optimize this combining rule 10 . 3. Feature reduction In many classification problems it is unclear what features have to be taken into account. Often a large set of k potentially useful features is collected, and by feature reduction the k most suitable features are chosen. Often the distinction between feature selection and feature extraction is made. In selection, only a subset of the original features is chosen. The advantage is that in the final application just a few features have to be measured. The disadvantage is that the selection of the appropriate subset is an expensive search. In extraction new features are derived from the original features. Often all original features are used, and no reduction is obtained in the number of measurement. Butt in many cases the optimization is easier. In Section 3.1 we will discuss several evaluation criteria, then in Section 3.2 feature selection and finally in 3.3 feature extraction. 3.1. Feature set evaluation
criteria
In order to evaluate a feature set, a criterion J has to be defined. Because feature reduction is often applied in classification, the most obvious criterion is thus the performance of the classifier. Unfortunately, the optimization of a classifier is often hard. Other evaluation criteria might be a cheaper approximation to this classification performance. Therefore approximate criteria are used, measuring the distance or dissimilarity between distributions, or even ignoring the class labels, but just focusing on unsupervised characteristics. Some typical evaluation criteria are listed in Table 1. The most simple ones use the scatter matrices characterizing the scatter within classes (showing how samples scatter around their class mean vector, called Sw, the within scatter) and the the scatter between the clusters (showing how the means of the clusters scatter, SB, the between scatter matrix, see also the discussion of the Fisher ratio in section 2.3). These scatter matrices can be combined using several functions, listed in the first part of Table 1. Often Si = SB is used, and 52 = Sw or 5 2 = Sw + SBThe measures between distributions involve the class distributions p(x\u>i), and in practice often single Gaussian distributions for each of the classes are chosen. The reconstruction errors still contain free parameters in the form of a matrix of basis vectors W or a set of prototypes fj,k. These are optimized in their respective
16 procedures, like the Principal Component Analysis or Self-Organizing Maps. These scatter criteria and the supervised measures between distributions are mainly used in the feature selection, Section 3.2. T h e unsupervised reconstruction errors are used in feature extraction 3.3. Table 1. Feature selection criteria for measuring the difference between two distributions or for measuring a reconstruction error. Measures using scatter matrices For explanation J = tr(5^" 1 5i) of Si and S2 J = ln|5^15i| see text. J = gf2Measures between distributions Kolmogorov J = / |p(wi|x) — p(cj 2 |x)|p(x)cbc Average separation J = ^ x j e w i Sx^ewi Divergence Chernoff
J = / (p(x|wi)p(x|w 2 )) ( p ( * ] ^ j ) dx J = - log/p s (x|wi)p 1 _ s (x|u;2)dx
Fisher
J = y / ( p ( x | w i ) p ( w i ) -p(x|w 2 )p(w 2 )) 2 dx
PCA SOM
Reconstruction errors E=\\x(W(W'i'W)_1W'r)xf B = min f c ||x-Ai f c || 2
3 . 2 . Feature
selection
In feature selection a subset of the original features is chosen. A feature reduction procedure consist of two ingredients: the first is the evaluation criterion to evaluate a given set of features, the second is a search strategy to search over all possible feature subsets 1 6 . Exhaustive search is in many applications not feasible. W h e n we start with k = 250 features, and we want to select k = 10, we have to consider in principle ( 250 ) — 2 • 10 1 7 different subsets, which is clearly too much. Instead of exhaustive search, a forward selection can be applied. It starts with the single best feature (according to the evaluation criterion) and adds the feature which gives the biggest improvement in performance. This is repeated till the requested number of features k is reached. Instead of forward selection, the opposite approach can be used: backward selection. This starts with the complete set of features and removes t h a t feature such t h a t the performance increase is the largest. These approaches have the significant drawback t h a t they might miss the optimal subset. These are the subsets for which the individual features have poor discriminability but combined give a very good performance. In order to find these subsets, a more advanced search strategy is required. It can be a floating search where adding
17 and removing features is alternated. Another approach is the branch-and-bound algorithm 12 , where all the subsets of features is arranged in a search tree. This tree is traversed in such order that as soon as possible large sub branches can be disregarded, and the search process is shortened significantly. This strategy will yield the optimal subset when the evaluation criterion J is monotone, that means that when for a certain feature set a value of Jk is obtained, a subset of the features cannot have higher value for Jj,. Criteria like the Bayes error, the Chernoff distance or the functions on the scatter matrices fulfill this. Currently, other approaches appear which combine the traditional feature selection and subsequent training of a classifier. One example is a linear classifier (with the functional form of (25)) called LASSO, Least Absolute Shrinkage and Selection Operator 3 1 . The classification problem is approached as a regression problem with an additional regularization. A linear function is fitted to the data by minimizing the following error: n
m i n ^ ( y i - w T x i - 6 ) 2 + C||w||.
(31)
i
The first part defines the deviation of the linear function w T Xj + 6 from the expected label yi. The second part shrinks the weights w, such that many of them become zero. By choosing a suitable value for C, the number of retained features can be changed. This kind of regularization appears to be very effective when the number of feature is huge (in the thousands) and the training size is small (in the tens). A similar solution can be obtained when the term w T w in (26) is replace by |w| 3 . 3.3. Feature
extraction
Instead of using a subset of the given features, a smaller set of new features may be derived from the old ones. This can be done by linear or nonlinear feature extraction. For the computation of new features usually all original features are used. Feature extraction will therefore almost never reduce the amount of measurements. The optimization criteria are often based on reconstruction errors as in Table 1. The most well-known linear extraction method is the Principal Component Analysis (PCA) 17 . Each new feature i is a linear combination of the original features: x\ = WjX. The new features are optimized to minimize the PCA mean squared error reconstruction error, Table 1. It basically extracts the directions W , in which the data set shows the highest variance. These directions appear to be equivalent to the eigenvectors of the (estimated) covariance matrix E with the largest eigenvalues. For the i-th principal component W j therefore holds: E W i = XiWi,
Xi > Xj, iff < j .
(32)
An extension of the (linear) PCA is the kernelized version, kernel-PCA 24 . Here the standard covariance matrix E is replaced by a covariance matrix in a feature space. After rewriting, the eigenvalue problem in the feature space reduces to the
18 following eigenvalue problem: Koti = AjCKj. Here K is a N x N kernel matrix (like for instance (29)). An object x is mapped onto the i-th principal component by:
^ = J>}if(x, X j ).
(33)
j
Although this feature extraction is linear in the kernel space, in the feature space it will obtain non-linear combinations of features. There are many other methods for extracting nonlinear features, for instance the Self-Organizing Map (SOM) 20 . The SOM is an unsupervised clustering and feature extraction method in which the cluster centers are constrained in their placing. The construction of the SOM is such that all objects in the input space retain as much as possible their distance and neighborhood relations in the mapped space. In other words, the topology is preserved in the mapped space. The mapping is performed by a specific type of neural network, equipped with a special learning rule. Assume that we want to map an /c-dimensional measurement space to a fc'-dimensional feature space, where k' < k. In fact, often k' = 1 or k' = 2. In the feature space, we define a finite orthogonal grid with grid points. At each grid point we place a neuron or prototype . Each neuron stores an fc-dimensional vector /ifc that serves as a cluster center. By defining a grid for the neurons, each neuron does not only have a neighboring neuron in the measurement space, it also has a neighboring neuron in the grid. During the learning phase, neighboring neurons in the grid are enforced to also be neighbors in the measurement space. By doing so, the local topology will be preserved. Unfortunately, training a SOM involves the setting of many unintuitive parameters and heuristics (similar to many neural network approaches). A more principled approach to the SOM is the Generative Topographic Mapping, GTM 5 . The idea is to find a representation of the original p-dimensional data x in terms of //-dimensional latent variables z. For this a mapping function y(z|W) has to be defined. In the GTM it is assumed that the distribution of z in the latent variable space is a grid of delta functions z*: 1
K
W = ^ £ <*(*-*).
(34)
i=l
For the mapping function a generalized linear regression model is used: y(z;W)=W#(z),
(35)
where <&(z) consist of M fixed basis functions (in many cases Gaussian functions) and W is a p x M weight matrix. Because in reality the data will never fit the lowdimensional manifold perfectly, a noise model is introduced: a Gaussian distribution with variance a2: p(x|z,W,<7)=JV(x;y(z|W),a).
(36)
19 The distribution p(x) can then be obtained by integration over the z distribution: p(x|W,(T)= /"p(x|z,W,
(37)
The advantage is that the model is a full probability model. This model can be fitted by optimizing the log likelihood of the training data (In J^^ p(x|W,«r)) using an Expectation-Maximization algorithm. When the user supplies the dimensionality of the latent variable space L, the number of grid points M in this space and the basis functions 3>(z), then the parameters W and a can be optimized. An even simpler model to optimize is the Local Linear Embedding, LLE 28 . Here also the goal is to find a low dimensional representation of the training set Xu'. But unlike the GTM, where an explicit manifold is fitted, here the low dimensional representation is optimized such that the objects can be reconstructed from their neighbors in the training set in the same manner in the low dimensional representation as in the high dimensional one. First, the weights Wij for reconstructing each object Xj from its neighbors Xj are optimized (minimizing the LLE reconstruction error, Table 1, under the constraint that V • Wij = 1). Given the weights, the location of low-dimensional feature vectors Zj,i = 1,...,N is optimized, using the same LLE reconstruction error, but where x, is replaced by z*. This can be minimized by solving a eigenvalue problem (similar to finding the principal components). The feature extraction methods presented above, are all unsupervised, i.e. other information like class labels is not used. This can be a significant drawback when the feature reduction is applied as a preprocessing for solving a classification problem. It might actually happen that all informative features are removed. To avoid this, supervised feature extraction has to be used. Very well known is Linear Discriminant Analysis (LDA) 27 , which is using the weight vector w from the Fisher classifier (see section 2.3) as feature direction. A multi-class extension is presented in 27 but it assumes equal covariance matrices for all classes and the number of features is restricted to if — 1. The LDA can be extended to include the difference in covariance matrix by using the Chernoff criterion instead of the Fisher criterion 23 . 4. Error estimation At various stages in the design of a pattern classification system an estimation of the performance of a procedure, or the separability of a set of classes is needed. Examples are the selection the 'best' feature during feature selection, the feature subspace to be used when several feature extraction schemes are investigated, the performance of the base classifiers in order to find a good set of classifiers to be combined, the optimization of various parameters in classification schemes like the smoothing parameter in the Parzen classifier and the number of hidden units used in a neural network classifier, and the final selection of the overall classification procedure if various competing schemes are followed consecutively. Moreover, at the end an estimate of the performance of the selected classifier is desired.
20
In order to find an unbiased error estimate, a set of test objects with known labels is desirable. This set should be representative for the circumstances expected during the practical use of the procedures under study. Usually this implies that the test set has to be randomly drawn from the future objects to be classified. As their labels should be known for proper testing, these objects are suitable for training as well. Once an object is used for training, however, the resulting classifier is expected to be good for this object. Consequently, if it is also used for testing it generates an optimistic bias in the error estimate. Below two techniques will be discussed to solve this problem. The first is cross-validation, which aims at circumventing the bias. The second is a bootstrap technique by which the bias is estimated and corrected. 4.1.
Cross-validation
Assume that a design set Xd is available for the development of a pattern recognition system, or one of its subsystems, and that in addition to the classifier itself an unbiased estimate of its performance is needed. If Xd is split (e.g. at random) into a training set Xtv and a test set Xte then we want Xtr to be as large as possible to train a good classifier, but simultaneously Xte has to be sufficiently large for an accurate error estimate. The standard deviation of this estimate is sqrt(e * (1 — e) Nte) (e.g. 0.003 for e = 0.01, Nte = 1000 and 0.03 for e = 0.1 and Nte = 100). When the design set is not sufficiently large to split it into a test set and a training set of appropriate sizes, a cross validation procedure might be used in which the design set is split into B ( B > = 2) subsets of about the same size. In total B different classifiers are trained, each by a different group of B — 1 of these subsets. Each classifier is tested by the single subset not used for its training. Finally the B test results are averaged. Consequently all objects are used for testing once. The classifiers they are testing are all based on an (B — 1)/B part of the training set. For larger B these classifiers are expected to be similar and they will be just slightly worse than the classifier based on all objects. A good choice seems to be a 10-fold stratified cross-validation, see 19 , i.e. N = 10, and objects are selected evenly from the classes, i.e. in agreement with their prior probabilities.
4.2. Bootstrap
procedures
Instead of following a procedure that tries to minimize the bias in the error estimate, one may try to estimate the bias 13 ' 15 . A general procedure (independent of the used classifier) can be based on a comparison of the expected apparent error E^pp of a classifier trained by bootstrapping the design set with its error E\ estimated by the entire design set. The difference can be used as an estimate for the bias in the apparent error: E\,ias = E^—E^, which can be used as a correction for the apparent error E% of the classifier based on the design set: Eboot = E%pp + Ebd — Ehapp. A second estimator based on bootstrapping is the so called Ees2 error n ' 1 3 ' 1 5 . It is based on a weighted average of the apparent error of the classifier based on
21 the design set E% and an error estimate EQ for the bootstrap classifier based on the out-of-bootstrap part of the design set. The first is optimistically biased (an apparent error) and the second is an unbiased error estimate (tested by independent samples) of a classifier that is somewhat worse (based on just a bootstrap sample) than the target classifier based on the design set. The weights are given by the asymptotic probability that a sample will be included in a bootstrap sample: 0.632. The £ 6 3 2 error estimate thereby is given by: E632 = 0.368 * E%pp + 0.632 * £#. 4.3. Error
curves
The graphical representation of the classification error is an important tool to study, compare and understand the behavior of classification systems. Some examples of such error curves are: Learning curve : the error as a function of the number of training samples. Simple classifiers decrease faster, but have often a higher asymptotic value than more complex ones. Complexity curve : the error as a function of the complexity of the classifier, e.g. the feature size or the number of hidden units. Such a curve often shows an increasing error after an optimal feature size or complexity. Parameter curve : the error as a function of a parameter in the training procedure, e.g. the smoothing parameter in the Parzen classifier. The optimum that may be observed in such curves is related to the best fit of the underlying model in the classification system w.r.t. the data. Error-reject trade off : the error as function of the reject probability. If a classifier output (e.g. a confidence estimate) is thresholded to reject unreliably classified objects, then this curve shows the gain in error reduction. ROC curves : the trade-off between two types of errors, e.g. the two types of error in a 2-class problem. These Receiver Operator Curves were first studied in communication theory and are useful to select a classifier if the point of operation may vary, e.g. due to unknown classification costs or prior probabilities. 5. Discussion In the previous sections an overview is given of well established techniques for statistical pattern recognition with a few excursion to more recent developments. Modern scientific and industrial developments, the use of computers and internet in daily life and the fast growing sensor technology raise new problems as well as they enable new solutions. We will summarize some new developments in statistical pattern recognition, partially introduced above, partially not yet discussed. Other types of representation than the traditional features enable other ways to incorporate expert knowledge. The dissimilarity representation is an example for this, as it offers the possibility to express knowledge in the definition of the dissimilarity measure, but it opens also other options. Instead of being based on the
22
raw data like spectra, images, or time signals it may be defined on models of objects, like graphs. In such cases structural knowledge is used for the object descriptions. In addition to the nearest neighbor rule, dissimilarity based classifiers offer a richer set of tools with more possibilities to learn from examples, thereby bridging the gap between structural and statistical pattern recognition. Several problems, however, have still to be solved, like the selection of a representation set, optimal modifications of a given dissimilarity measure and the construction of dedicated classifiers. More complicated pattern recognition problems may not be solved by a single off-the-shelf classifier. By the combining classifier technique a number of partial solutions can be combined. Several questions are still open here, like the selection or generation of the base classifiers, the choice of the combiner, the use of a finite training set. Moreover, an overall mathematical foundation is still not available. One-class classifiers are a good way to handle ill sampled problems, or to build classifiers when some of the classes are undersampled. This is important for applications like man or machine monitoring when one of the classes, e.g. normal behavior, is very well defined. Such classifiers may also be used when it is not possible to select a representative training set by an appropriate sampling of the domain of objects. In such cases a domain based class description may be found, locating the class boundary in the representation, without building a probability density function. The well spread availability of computers and sensors, and the costs of labeling objects by human experts, may sometimes result in large databases in which just a small fraction of the objects is labeled. Techniques for training classifiers by partially labeled datasets are still in their early years. This may also be considered as combining clustering and classification. For such problems in which the costs of expert labeling are high, one may also try to optimize the set of objects to be labeled. This technique is called active learning. Several competing strategies exist, e.g. sampling close to an initial decision boundary, or retrieving objects in the modes of the class density distributions. Another variant is online learning, in which the order of the objects to be presented to a decision function is determined by the application, e.g. by a production line in a factory. It has now to be decided whether objects can be safely classified, or whether a human expert has to be solicited, not only to reduce the risk of misclassification, but also to optimally improve the available classification function. An often returning question in dynamic environments is whether a trained classification function is still valid, or whether it should be retrained due to new circumstances. In such problems 'learning' and 'forgetting' are directly related. If a new situation demands retraining, old objects may not be representative anymore and should be forgotten. (They may still be stored in case the old situation appears to return after some time). Many techniques are proposed and many more are to come for solving problems as the above. A difficulty that cannot be easily handled is that they are often ill defined. Consequently, generally valid benchmarks are not available, by which it
23 is not straightforward to detect the good procedures t h a t may work well over a series of applications. As good and bad procedures cannot easily be distinguished, it is to be expected t h a t the set of tools used in statistical p a t t e r n recognition will significantly grow in the near future.
References 1. J.A. Anderson. Logistic discrimination. In P.R. Kirshnaiah and L.N. Kanal, editors, Classification, Pattern Recognition and Reduction of Dimensionality, volume 2 of Handbook of Statistics, pages 169-191. North Holland, Amsterdam, 1982. 2. A.G. Arkadev and E.M. Braverman. Computers and Pattern Recognition. Thompson, Washington, D.C., 1966. 3. C. Bhattacharyya, L.R. Grate, A. Rizki, D. Radisky, F.J. Molina, M. I. Jordan, M.J. Bissell, and I.S. Mian. Simultaneous relevant feature identification and classification in high-dimensional spaces: Application to molecular profiling data. Signal Processing, 83:729-743, 2003. 4. C M . Bishop. Neural Networks for Pattern Recognition. Oxford University Press, Walton Street, Oxford 0X2 6DP, 1995. 5. C M . Bishop, M. Svensen, and C.K.I. Williams. The generative topographic mapping. Neural Computation, 10(l):215-234, 1998. 6. L. Breiman. Bagging predictors. Machine Learning, 26(2):123-140, 1996. 7. T.G. Dietterich and G. Bakiri. Solving multiclass learning problems via errorcorrecting output codes. Journal of Artificial Intelligence Research, 2:263-286, 1995. 8. H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun, and V. Vapnik. Boosting and other ensemble methods. Neural Computation, 6, 1994. 9. R.P.W. Duin. On the choice of the smoothing parameters for Parzen estimators of probability density functions. IEEE Trans, on Computers, C-25(ll):1175-1179, 1976. 10. R.P.W. Duin. The combining classifier: To train or not to train? In International Conference on Pattern Recognition, volume II, pages 765-770, Quebec, Canada, 2002. 11. B. Efron and R.J. Tibshirani. Improvements on cross-validation: the .632+ bootstrap method. J. Amer. Statist. Assoc, 92:548-560, 1997. 12. K. Fukanaga. Introduction to Statistical pattern recognition. Academic press, San Diego, 2nd edition, 1990. 13. D.J. Hand. Recent advances in error rate estimation. Pattern Recognition Letters, 4(5):335-346, 1986. 14. S.S. Haykin. Neural Networks, a comprehensive foundation. Prentice-Hall, 1999. 15. A.K. Jain, R . C Dubes, and Chen, C.-C Bootstrap techniques for error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(5):628-633, September 1987. 16. A.K. Jain, R.P.W. Duin, and J. Mao. Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(l):4-37, 2000. 17. I.T. Jolliffe. Principal Component Analysis. Springer-Verlag, New York, 1986. 18. J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(4):226-239, 1998. 19. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proc. of the 15th Int. Joint Conference on Artificial Intelligence, pages 1137-1143, 1995. 20. T. Kohonen. Self-organizing maps. Springer-Verlag, Heidelberg, Germany, 1995. 21. L.I. Kuncheva. Combining Pattern Classifiers: Methods and Algorithms. Wiley, 2004.
24 22. L.I. Kuncheva and C.J. Whitaker. Measures of diversity in classifier ensembles. Machine Learning, 51:181-207, 2003. 23. M. Loog, R.P.W. Duin, and R. Haeb-Umbach. Multiclass linear dimension reduction by weighted pairwise fisher criteria. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(7):762-766, 2001. 24. S. Mika, B. Scholkopf, A.J. Smola, K.-R. Miiller, M. Scholz, and G. Ratsch. Kernel PCA and de-noising in feature spaces. In M.S. Kearns, S.A. Solla, and D.A. Cohn, editors, Advances in Neural Information Processing Systems, volume 11, pages 536542. MIT Press, 1999. 25. T. M. Mitchell. Machine Learning. Mc Graw-Hill, New York, 1997. 26. E. Pekalska and R.P.W. Duin. Dissimilarity representations allow for building good classifiers. Pattern Recognition Letters, 23(8):943-956, 2002. 27. C.R. Rao. The utilization of multiple measurements in problems of biological classification (with discussion). Journal of the Royal Statistical Society, B, 10:159-203, 1948. 28. S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323-2326, 2000. 29. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004. 30. D.M.J. Tax. One-class classification. PhD thesis, Delft University of Technology, http://www.ph.tn.tudelft.nl/~davidt/thesis.pdf, June 2001. 31. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B, 58(l):267-288, 1996. 32. V.N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.
CHAPTER 1.2 HIDDEN MARKOV MODELS FOR SPATIO-TEMPORAL PATTERN RECOGNITION
Brian C. Lovell" and Terry Caellib The Intelligent Real-Time Imaging and Sensing (IRIS) Group The School of Information Technology and Electrical Engineering The University of Queensland, Australia QLD 4072 E-mail:lovell@ itee. uq. edu.au a
National Information and Communications Technology Australia (NICTA) Research School of Information Sciences and Engineering Australian National University, Australia email:
[email protected] The success of many real-world applications demonstrates that hidden Markov models (HMMs) are highly effective in one-dimensional pattern recognition problems such as speech recognition. Research is now focussed on extending HMMs to 2-D and possibly 3-D applications which arise in gesture, face, and handwriting recognition. Although the HMM has become a major workhorse of the pattern recognition community, there are few analytical results which can explain its remarkably good pattern recognition performance. There are also only a few theoretical principles for guiding researchers in selecting topologies or understanding how the model parameters contribute to performance. In this chapter, we deal with these issues and use simulated data to evaluate the performance of a number of alternatives to the traditional Baum-Welch algorithm for learning HMM parameters. We then compare the best of these strategies to Baum-Welch on a real hand gesture recognition system in an attempt to develop insights into these fundamental aspects of learning. 1. Introduction There is an enormous volume of literature on the application of hidden Markov Models (HMMs) to a broad range of pattern recognition tasks. In the case of speech recognition, the patterns we wish to recognise are spoken words which are audio signals against time. Indeed, the value of Markov models to model speech was recognised by Shannon 26 as early as 1948. In the case of hand gesture recognition, the patterns are hand movements in both space and time — we call this a spatio-temporal pattern recognition problem. The suitability and efficacy of HMMs to such problems is undeniable and they are now established as one of the major tools of the pattern recognition community. Yet, when one looks for research which address fundamental problems such as efficient learning strategies for HMMs or perhaps analytically determining the most suitable architectures for a given problem, die number of papers is greatly diminished. So despite the enormous uptake of HMMs since 25
26
their introduction in the 1960's, we believe that there is still a great deal of unexplored territory. Much of the application of HMMs in the literature is based firmly on the methodology popularised by Rabiner et al. (1983) 25>16-24 for speech recognition and these studies are the primary reference for many HMM researchers resulting in two common practices. One, to use the forward algorithm to determine the MAP(maximum posterior probability) of the model, given an observation sequence, as a classification metric. Two, to use the Baum-Welch as a model estimation/update procedure. We will see how these are not ideal strategies to use as, in the former case, classification is reduced to a single number without directly using the model (data summary) parameters, attributes, per se. As for the latter, the Baum-Welch 4 algorithm (a version of the famous Expectation-Maximisation algorithm14,1'21) is, in the words of Stolke and Omohundro 28 ,"... far from foolproof since it uses what amounts to a hill-climbing procedure that is only guaranteed to find a local likelihood maximum." Moreover, as observed by Rabiner 24 , results can be very dependent on the initial values chosen for the HMM parameters. The problem of finding local rather that global maxima is encountered in many other areas of learning theory and optimisation. These problems are familiar territory to researchers in the artificial neural network community and many techniques have been proposed to counter them. Moreover genetic and evolutionary algorithmic techniques specialise in solving such problems — albeit often very slowly, especially in the case of biological evolution11. With this in mind, we use simulated data to investigate other approaches to learning HMMs from observation sequences in an attempt to find superior alternatives to the traditional Baum-Welch Algorithm. Then we compare and test the best of the alternate strategies on real data from a hand gesture recognition system to see if the real data trials corroborate the conclusions drawn from simulated trials. 1.1. Background and Notation In this study, we focus on the discrete HMM as popularised by Rabiner24. Using the familiar notation from his tutorial paper, a hidden Markov model consists of a set of N nodes, each of which is associated with a set of M possible observations. The parameters of the model include an initial state vector 7T=
\Pl,P2,P3,-,PN}T
with elements pn, n £ [1, AT] which describes the distribution over the initial node set, a transition matrix / a n a12 ... aiw \ 021
a22
•••
02JV
\ajvi ajv2 • • •
OJVJV/
with elements atj with i,j £ [l,N] for the transition probability from node i to node j
27 b) Left-Right
Fig. 1. Cyclic and Left-Right structures. Bold arrows indicate high probability transistions. No arrow between vertices indicates a forbidden (zero-probability) transition.
conditional on node i, and an observation matrix /bn 621
B
6 i2 ... bXM \ b22
•••
b2M
\ &W1 bN2 • • • bNM
J
with elements bim for the probability of observing symbol m G [1,M] given that the system is in state i G [1, N]. We denote the HMM model parameter set by A = (A, B, 7r). The model order pair (N, M) together with additional restrictions on allowed transitions and emissions defines the topology or structure of the model (see figure 1 for an illustration of two different transition structures). One commonly used topology is called Fully-Connected (FC) or Ergodic. In the FC HMM there is not necessarily a defined starting state and all state transitions are possible such that a^ ^ 0 V i, j £ [1, N]. Another topology, especially popular in speech recognition applications, is called Left-Right. In an LR HMM there is a defined starting state (usually state 1) and only state transitions to higher-index states are allowed such that a^• = 0 V i > j where i, j G [1, N]. Rabiner24 defines the three basic problems of HMMs by: Problem 1 Given the observation sequence O = 0\02 • • -OT, and a model A = (A, B, IT), how do we efficiently compute P(0\X), the probability of the observation sequence given the model? Problem 2 Given the observation sequence O = O1O2 • • • Or, and the model A, how do we choose a corresponding state sequence Q = q\q2 ... qr which is optimal in some meaningful sense {i.e., best "explains" the observations)? Problem 3 How do we adjust the model parameters A = (A, B, n) to maximize P((9|A)? Problems 1 and 2 are elegantly and efficiently solved by the forward and Viterbi29'12 algorithms respectively as described by Rabiner in his tutorial. The forward algorithm is used to recognise matching HMMs (i.e., highest probability models, MAP) from the observation sequences. Note, again, that this is not a typical approach to pattern classification as
28
it does not involve matching model with observation attributes. That would involve comparing the model parameters and estimated observation model parameters. MAP does not perform this and so it cannot be as sensitive a measure as exact parameter comparisons. Indeed, a number of reports have already shown quite different HMMs can have identical emissions( observation sequences) 18 ' 3 . The Viterbi algorithm is used less frequently as we are normally more interested in finding the matching model than in finding the state sequence. However, this algorithm is critical in evaluating the precision of the HMM; in other words, how well the model can reconstruct (predict) the observations. Rabiner proposes solving Problem 3 via the Baum-Welch algorithm which is, in essence, a gradient ascent algorithm — a method which is guaranteed to find local maxima only. Solving Problem 3 is effectively the problem of learning to recognise new patterns, so it is really the fundamental problem of HMM learning theory; a significant improvement here could boost the performance of all HMM based pattern recognition systems. Therefore it is somewhat surprising that there appear to be relatively few papers devoted to this topic — the vast majority are devoted to applications of the HMM. In the next section we compare a number of alternatives to and variations of Baum-Welch in an attempt to find superior learning strategies. 2. Comparison of Methods for Robust HMM Parameter Estimation We focus on the problem of reliably learning HMMs from a small set of short observation sequences. The need to learn rapidly from small sets arises quite often in practice. In our case, we are interested in learning hand gestures which are limited to just 25 observations. The limitation arises because we record each video at 25 frames per second and each of our gestures takes less than one second to complete. Moreover, we wish to obtain good recognition performance from small training sets to ensure that new gestures can be rapidly recognised by the system. Four HMM parameter estimation methods are evaluated and compared by using a train and test classification methodology. For these binary classification tests we create two random HMMs and then use each of these to generate test and training data sequences. For normalization, we ensure that each test sequence can be correctly recognized by its true model; thus the true models obtain 100% classification accuracy on the test data by construction. The various learning methods are then used to estimate the two HMMs from their respective training sets and then the recognition performance of the pair of estimated HMMs is evaluated on the unseen test data sets. This random model generation and evaluation process is repeated 16 times for each data sample to provide meaningful statistical results. Before parameter re-estimation, we initialize with two random HMMs which should yield 50% recognition performance on average. So an average recognition performance above 50% after re-estimation shows that some degree of learning must have taken place. Clearly if the learning strategy can perfectly determine both of the HMMs which generated the training data sets, we would have 100% recognition performance on the test sets. We compare four learning methods 1) traditional Baum-Welch, 2) ensemble averaging
29 Classification Performance Averaged Over 16 Experiments True model 100% by construction.
0.95
3
0.9
H
i!
•i= rt
0.85
S 0)
0.8
§ £
0.75
o
t:
0 !
on
0-
0.65
rt!
sit
o
O)
0.6
o
0.55 0.5
Baum-Welch - 0 - Viterbi Path Counting - * - Entropic MAP -A- Ensemble Averaging 5 10 Number of training sequences
15
Fig. 2. Relative performance of the HMM parameter estimation methods as a function of the number of training sequences. Viterbi Path Counting produces the best quality models with a much smaller number of training iterations.
introduced by Davis and Lovell9 based on ideas presented by Mackay19, 3) Entropic MAP introduced by Brand6, and 4) Viterbi Path Counting10 which is a special case of Stolke and Omhundro's Best-First algorithm28. The results in figure 2 indicate that these alternate HMM learning methods all classify significantly better than the well-known Baum-Welch algorithm and also require less training data. The Entropic MAP estimator performs well but surprisingly the performance is much the same as simple ensemble averaging. Ensemble averaging involves training multiple models using the Baum-Welch algorithm and then simply averaging the model parameters without regard to structure. Note that for a single sequence, ensemble averaging is identical to the traditional usage of the Baum-Welch algorithm. Overall, the stand-out performer was the VPC algorithm. In these and other trials, this method converges to good models very rapidly and has performed better than the other methods in virtually all of our simulated HMM studies.
3. Video Gesture Recognition In an attempt to corroborate the strong performance of VPC compared to Baum-Welch on a real-world application, we test various learning techniques on a system for real-time video gesture recognition as shown in figure 3.
30 In earlier related work, Starner and Pentland 27 developed a HMM-based system to recognise gesture phrases in American Sign Language. Later, Lee and Kim 1 5 used HMMbased hand gesture recognition to control viewgraph presentation in data projected seminars. Our system recognizes gestures based on the letters of the alphabet traced in space in front of a video camera. The motivation for this application is to produce a way of typing messages into a camera-equipped mobile phone or PDA using video gestures instead of the keypad or pen interface. We use single stroke letter gestures similar to those already widely used for pen data entry in PDAs. For example, figure 3 shows the hand gestures for the letters "Z" and "W." The complete gesture set is shown in figure 6.
Fig. 3. "Fingerwriting:" Single stroke video gesture for letters "W" and "Z." Each video sequence comprises 25 frames corresponding to one second of video. Skin colour segmentation in YUV colour space is applied to locate the hand. Pre-processing (morphological) operations smooth the image and remove noise before tracking the hand with a modified Camshift algorithm 5 . After segmenting the hand, we calculate image moments to find the centroid in each frame. Along the trajectory, the direction (angle) of motion of each of the 25 hand movements is calculated and quantized to one of 18 discrete symbols. The resultant discrete angular observation sequence is input to the HMM classification module for training and recognition. We compare traditional Baum-Welch with the most promising alternative from the stimulated study, VPC. We evaluate recognition performance over all 26 character gestures using fully connected (FC), left-right (LR), and left-right banded (LRB) model topologies with the number of states ranging from 1 to 14. A LRB model is an LR model which has a transition structure containing self-transitions and next state transitions only (i.e., states cannot be skipped) as shown in figure 5. More formally, a^ ^ 0 V j = i or j = * + 1, and 0 otherwise, i, j e [1, N]. Our video gesture database contains 780 video gestures with 30 examples of each gesture. Recognition accuracy is evaluated using threefold cross-validation where 20 gestures are used for training and 10 for testing in each partition. These HMMs are initialized with random HMM parameters before using either Baum-Welch or VPC for learning. From figure 4 the best average recognition accuracy achieved is 97.31% when VPC is used for training, topology is LRB, and the number of states is 13. Although this corrobo-
31
Number of States 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mean Max
Baum-Welch FC LR LRB 80.00 80.00 80.00 72.69 94.23 93.85 66.54 92.31 96.15 80.00 84.80 85.38 75.20 81.20 90.77 75.60 84.80 85.77 77.60 86.40 89.62 76.80 86.00 89.62 77.60 85.60 90.00 76.00 81.60 88.46 65.20 86.80 89.23 74.80 86.80 88.08 84.80 84.00 90.00 72.80 81.60 88.46 75.40 85.44 88.96 84.80 92.31 96.15
FC 80.38 71.15 63.85 53.20 59.60 55.20 45.60 44.40 49.20 43.20 42.80 40.80 39.60 38.80 51.98 63.85
VPC LR 80.38 91.92 91.15 91.20 91.20 90.40 91.20 90.40 90.40 90.00 90.00 90.00 90.00 90.40 89.90 91.20
LRB 80.38 90.77 93.08 90.38 95.00 93.85 94.23 94.23 94.62 95.00 95.00 95.77 97.31 93.46 93.08 97.31
Fig. 4. Average percent correct recognition for all 26 video letter gestures against topology and training method.
0808 Fig. 5. Left-Right banded topology.
rates the stronger VPC performance exhibited in our simulated data performance trials, a closer investigation of Table 4 raises some doubts about this conjecture through the following observations. • The Baum Welch algorithm did almost as well as VPC with a best performance of 96.15% correct recognition with only 3 states. Moreover we achieve a very surprising 80% correct recognition with just a single state. • Topology (i.e., constraints on the initial value of the A matrix) has more impact on performance than the choice of learning algorithm. • Good recognition performance can be obtained over a very broad range of N, the number of states. 3.1. Comments on Learning Algorithm Performance We do not suggest that the above observations can be generalized to other real-world application domains but anecdotal evidence from other researchers suggests that similar be-
32
c 0 r* Rh J K L /N 0 p q s u \J 0) xf I
A6
im^""''€ '
Fig. 6. The alphabet of single-stroke letter hand gestures.
haviour is often encountered. When we designed this gesture system, we thought that this pattern recognition problem was quite challenging and would significantly differentiate learning strategies. Yet the surprisingly good performance over a number of learning algorithms, topologies, and a broad range N suggests that the problem is significantly easier than we suspected. Our intuition suggests that 3 states is far too small a number to adequately model all of these complex letter gestures, but results show that it is indeed possible to find a three state HMM which yields very good recognition performance. We conjecture that the observation matrix B seems to provide most of the recognition performance and that recognition may be only weakly affected by good estimation of the transition matrix A. In support of this idea, we may consider the following interpretation of the HMM. Consider each row of the B matrix as the probability mass function of the observation symbols emitted in a given state. In the limiting case of a single state HMM, the B matrix becomes a vector of source symbol probabilities and application of the forward algorithm for recognition is thus equivalent to the well-known and powerful MAP classifier. Indeed from figure 4, we see that this single state degenerate HMM can achieve 80% recognition performance. So sometimes even if the state transitions are poorly modelled, it is quite possible to find good classifiers based on source statistics. Now clearly if three states can yield strong performance, good HMMs with more than three states must also exist — a simple way to prove this is to note that we can always add additional states which are unreachable {i.e., transition probability of zero) without affecting recognition performance. This may help explain why performance stays much the same over a broad range of N as we increase N beyond three. The question that arises is, "Why does the Baum-Welch algorithm perform so well on real-world data despite its theoretical flaws and rather poor performance on the simulated HMM data?" Once again, a possible explanation is that this particular spatio-temporal recognition task is relatively easy, so all methods can do quite well. This conjecture may be
33
true for other common HMM applications and is a focus of current research. Unfortunately, unlike simulated data, the effort of gathering very large and diverse databases of real-world pattern recognition problems to evaluate the performance of different training algorithms is immense, so progress is slow. 3.2. Comments on Topology The FC topology allows transitions from any state to any other state and does not have a defined starting state. Being the most general topology, it is hardly surprising that performance is relatively poor. In this case we are required to search for a good solution in a parameter space of much higher dimensionality than for LR — so it would be much harder to locate globally optimal solutions. Gestures have a natural start and finish and thus it is reasonable to adopt the LR model as used in speech recognition to great effect. An even simpler topology is the LRB HMM where only self-transitions and next state transitions are allowed. Thus the A matrix is of the form: /oiil-onO ...0 0 a22 1 - a22 . . . 0
\ (1)
\0
0
0
...aNNJ
In this case the expected number of observations, n, (i.e., duration) in state i is simply given by24 n= — ^ . (2) 1 -an One can interpret the LRB HMM as being an adjustable clock that ideally synchronizes the changes in observation statistics with the changes in state to produce a time-variant MAP classifier. In many ways this HMM topology may be considered as a form of dynamic time-warping22 — a earlier technique used in speech recognition that has fallen out of favour since the advent of HMMs. The good performance of LR and LRB topologies on the gesture data set help make the point that simple HMM topologies often work best on real data. 4. Direct Calculation of HMM Parameters from Video Gestures A major dissatisfaction with the foregoing treatment of HMM learning strategies on real data is that the learning procedure is be treated like a black-box with little real insight into the learning process. In contrast to simulated data, we are unlikely to know the true HMM that originally generated the data — indeed real data is very rarely generated by a process that even remotely resembles a HMM. Thus we usually don't really know the best topology or number of states a priori — practitioners just try ranges of values and pick the one that yields good classification performance. Sometimes, as in the above example, this utilitarian method can find HMMs that are far too simple to accurately characterize the
34
state transitions of the underlying pattern. Alternatively, we may find HMMs that are far more complicated than is necessary. In either case this may lead to poorer generalization ability than might otherwise have been the case. To investigate this topic further, we devised two simple video gestures as shown in figure 7 where it is possible to approximately determine the form of the true HMM directly from the gestures themselves. In the case of the triangle gesture, this can be modelled by a 3-state HMM — each state corresponding to one side of the triangle. There is no need for skipped states to model this gesture, so the LRB topology is the most appropriate.
4.1. Analytic Calculation of A and B The triangle gesture is intended to be an equilateral triangle, so the expected number of observations in each state is equal and corresponds to n = 24/3 = 8. Thus the expected values of the transition matrix {A) parameters can be calculated from the duration equation (2). The triangle gesture consists of a horizontal movement at angle 0 degrees, followed by movement of 120 degrees and then -120 degrees for the other two sides of the triangle. To account for variations in the real gestures, one could use an error distribution (eg., Gaussian) centred on each of these angles as the expected value of each corresponding row of the B matrix. Similar analysis can be used to derive the expected form of the A and B for the square gesture.
35 4.2. Re-Estimation of A Matrix Using Baum-Welch After Correct Initialization Figure 8 show the calculated values for the A matrix for both gestures as well as the average value over 20 trials after Baum-Welch re-estimation using the calculated value as the initial value for A and a random matrix as the initial value for B. Clearly, the application of Baum-Welch re-estimation did not significantly alter the A matrix which implies that a local maximum was reached near the calculated value of A as expected. Nevertheless, when we used the traditional approach of initial random values for both A and B, we achieved values close to the calculated values only about 50% of the time. Thus even for these simple patterns, it seems that traditional application of the Baum-Welch algorithm with random initial starts has difficulty in finding the transition structure.
Triangle
A Matrix
Calculated
0.87 0 0
0.13 0.87 0
0 0.13 1
Average after BW re-estimation
0.87 0 0
0.13 0.87 0
0 0.13 1
0.83 0 0 0 0.85 0 0 0
Square 0.17 0 0.83 0.17 0 0.83 0 0 0.15 0 0.83 0.17 0 0.85 0 0
0 0 0.17 1 0 0 0.15 1
Fig. 8. Calculated A matrix and average values after 20 trials of Baum-Welch re-estimation for triangle and square gestures.
Figure 9 shows the performance of a number of alternate strategies to learning HMMs. We measure the sum of the squares of the differences of the elements between the A and B matrices obtained from the Baum-Welch algorithm and our reference matrices whcih are our best estimate of the correct HMM. In the case of the A matrix we use the calculated value from subsection 4.1 as the reference. For the B matrix, we could determine the reference B matrix in a number of ways (eg., histogram of observation symbols in each state, fitted Gaussian distribution, fitted Von Mises Distribution). Here we choose to use the histogram method. In methods 1 and 2, we keep A fixed at the calculated reference value and allow B to be re-estimated by Baum-Welch. In method 1, B is initialized to a random value, but in method 2 it is initialized to the reference value. Method 3 is traditional Baum-Welch where both A and B are re-estimated form random initial values. Finally, in method 4 we use Baum-Welch with both matrices initialized to the reference values. From this table it is quite clear that Baum-Welch is not very good at finding the true HMM parameters even if it is given the topology (A) through a calculated A matrix as in method 1. However when provided with good estimates for both A and B as initial values as in method 4, Baum-Welch converges to the given solution indicating that a local maximum has indeed been reached.
36
Method: Initialization 1. A fixed: A calc, B random 2. A fixed: A calc, B calc 3. A re-est: A random B random 4. A re-est: A calc B calc
A triangle 0 0 0.0708 2.13e-5
B triangle 0.4654 0.0066 0.8632 0.0059
A square 0 0 0.1314 8.19e-4
B square 0.7284 0.0698 1.437 0.0676
Fig. 9. Average squared error of matrices over 20 trials using different Baum-Welch re-estimation techniques and initial values.
4.3. Comments on HMM Learning The foregoing indicates that the Baum-Welch algorithm may have great difficulty in finding solutions that closely match the transition structure of the physical systems even for very simple examples. Nevertheless, it is clear that the resulting HMMs may still be extremely useful for pattern recognition. We conjecture that this may be because many real-world problems are adequately distinguished by their source statistics so the recognition problem is simpler that it would first appear. 5. Tools for Investigating HMM Parameters and Performance Such observed anomalies in HMM learning behaviour led us propose a methodology and tools to diagnose and analyze HMM pattern recognition systems. Too often researchers simply report "a HMM with x-states and y-symbols successfully classified the observation sequences". What does this mean? Is this really the case? Could it have been simply due to excellent discriminating observation attributes (the B-matrix) or a very strong memory model (A-matrix)? Since these matrices are typically not reported very little can be concluded on such issues. Further, a set of HMM models may well be sufficient for classifying a subset of observation sequences but if the A and B matrix probabilities demonstrate high uncertainty the classification performance can easily degrade as many different sets of observations can produce the same, often low, MAP scores. In order to be able to reproduce past results, examine model generalisations, and update model topology the model parametric values require interpretation. We7,20 have made some initial studies into such issues and, in particular, have developed some measures of performance of a given HMM that allows for model refinement and some objective measures of how the observation (B) and state (A) models contribute to performance. Consider the row-augmented matrix: C = A\B.
(3)
Each row represents all the information about a given state at time t in terms of expected state transitions and observation likelihoods. Consequently the rank of this matrix is critical in understanding the uniqueness or redundancy of states and observations. However, some properties of these states can already be deduced from the model components, per se. First, the Markov chain component. For a Markov process, a well known way of defining the distance between the current state transition probabilities and the steady state density function
37
(invariant pdj) is from the total of the left-handed "residual eigenvalues" of A (fires(A)) — the total of all the sub-dominant eigenvalues23. When all rows of the Markov Chain are identical the Markov Chain condition breaks down in so far as: P[St+1 \St] = P[St+1] = P[St] = 7T.
(4)
Consequently fj,res(A) = J2i=2 MiC^) provides a measure of how ergodic the process is and consequently the potential for generating variable state sequences as a function of the observation. Secondly, as the rows of the state dependent observation matrix, B, become more correlated, the evidence for a specific state from observations decreases. Similar to the A matrix steady state condition, observations do not evidence any state when the B matrix has only one non-zero eigenvalue, leading to: P[Ok\Si] = P[Ok}. In all then, using the singular values of C, a, the Inverse Condition Number (ICN) of C is:
(5) 13
where crmax is the largest singular value of C and amin is the smallest, is an appropriate normalized measure of the "HMM bandwidth" in so far as 7 _ 1 = 1.0 indicates that all states can be realized within the limits of the steady state probabilities and state dependent observations. However, 7 - 1 = 0 results in a "zero-bandwidth" HMM in so far as the process, on any experiment, does not provide any predictive information about state sequences except those provided by the prior or steady state conditions. By projecting individual rows of C into it's eigenspace we can then observe the redundancy of given states (also evident in the row correlation matrix) and consequently split and merge states to update the model topology to minimize model redundancy with respect to both state and observation parameters. This we found to be successful in the case of gesture and speech recognition experiments (see [20] for details). Given that these above techniques can be used to analyse and refine HMM model topologies — independent of any observation sequence — we now consider how the A and B parameters contribute to the prediction of state sequences given a model and observations and we show how Conditional Information2 can be used for this purpose. If the B matrix is unambiguous (for example, orthogonal with an ICN of 1.0) then a direct use of either Maximum Likelihood (ML: maxs{P[0(t)\S}}) or maximum posterior probability (MAP: maxs{P[S\6(t)] = P[0{t)\S]P[S}} would suffice to predict the most likely state at time, t. This condition would eliminate the need for the Markov component (A matrix) by use of a simple Bayesian (ML or MAP) classifier. Conversely, if the model B matrix is ambiguous (ICN of 0), we may as well dispense with the observation part and simply use the Markov component of the HMM to determine the most likely state sequence given the Markov model. Accordingly, we show how Conditional Information can be used to tease out the contributions of each component to the solutions for optimal state sequences.
38
Given a model and an input observation sequence two state sequences can be generated, one using the Viterbi algorithm with the entire HMM, Sv , and the other with a Bayesian classifier using only the B matrix (ML classifier) resulting in Sb. This latter condition assumes that the predictions at each time period are independent of all others — a condition consistent with the independence of observations over time for regular HMMs. Given the resultant two state sequences, Sv and Sb, respectively, we can calculate the following quantities: H(v\b)=H(v,b)-H(b)
(7)
where H(v, b) = - J2(p(sv
=i,Sb=
j)logP(Sv =i,Sb=
j))
(8)
and H(P) = - J2(P(sb
= J)logP(Sb = j))-
(9)
3
H(v\b) is the conditional entropy, and P(SV = i,Sb = j) is computed from the joint frequencies of the two state sequences. This measures the amount of information about the Viterbi solution given the Bayesian classifier solution. The residual information R(v\b) = H(v) - H(y\b)
(10)
provides a measure of how much information the A matrix, and the associated Viterbi algorithm, add to the complete optimal state sequence prediction. These measures provide a clear measure of the degree of information contributed by the observation and state models in the (MAP-based) performance of any HMM on a given data set. 6. Conclusions HMMs are an immensely powerful tool for solving pattern recognition and classification problems. Many studies demonstrate that it is a powerful technique, but few studies give any insight into why the performance is so good. It is well-known that the Baum-Welch algorithm is a hill-climbing technique that is generally unable to find global maxima. Yet the performance of pattern recognition systems based on Baum-Welch is often very good. Although we can find HMM training algorithms that appear to perform much better than Baum-Welch on simulated HMM classification data, these results do not necessarily translate into greatly improved performance in real-world applications. We conjecture that that is possibly because many important real-world problems have little need for highly complex HMMs. Also it appears that embedding knowledge of the topology of the system can often improve recognition performance more significantly than changes in learning algorithm. Finally we offer some tools and methodologies to analytically investigate and diagnose these problems.
39
7. Acknowledgements This paper covers a number of research themes currently being studied within our research groups and collaborators. I would like to acknowledge the contributions of Brendan McCabe, Peter Kootsookos, Richard Davis, Nianjun Liu, and Christian Walder. References 1. A A. Dempster, N. Laird, and D. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39(l):l-38, 1977. 2. R. Ash. Information Theory. Interscience Publishers, 1995. 3. V. Balasubramanian. Equivalence and reduction of hidden markov models. Technical Report A.I. Technical Report No. 1370, MIT Artificial Intelligence Laboratory, 1993. 4. L. Baum, T. Petrie, T. Soules, and N. Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov cahins. The Annals of Mathematical Statistics, 41:164-171,1970. 5. G. R. Bradski. Computer video face tracking for use in a perceptual user interface. Intel Technology Journal, Q2, 1998. last visited May 2004. 6. M. Brand. An entropic estimator for structure discovery. Advances in Neural Info. Proc. Systems, 11:723-729,1999. 7. T. Caeli and B. McCane. Components analysis of hidden markov models in computer vision. In Proc. of 12th International Conference on Image Analysis and Processing, pages 510-515. 8. Terry Caelli, Andrew McCabe, and Garry Briscoe. Shape tracking and production using hidden markov models. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), 15(1):197-221, February 2001. 9. R. I. A. Davis, B. C. Lovell, and T Caelli. Improved estimation of hidden markov model parameters from multiple observation sequences. In Proceedings of the International Conference on Pattern Recognition (ICPR2002), volume 2, pages 168-171, Quebec City, August 2002. IEEE. 10. Richard I. A. Davis and Brian C. Lovell. Comparing and evaluating hmm ensemble training algorithms using train and test and condition number criteria. Pattern Analysis and Applications, 6(4):327-336, February 2003. 11. Richard Dawkins. The Blind Watchmaker: Why the Evidence of Evolution Reveals a Universe Without Design. W. W. Norton, New York, 1996. 12. G. D. Forney. The viterbi algorithm. Proc. IEEE, 61:268-278, March 1973. 13. G. Golub and C. Van Loan. Matrix Computations, chapter 2.7, pages 79-81. The Johns Hopkins University Press, 2nd edition, 1989. 14. H. Hartley. Maximum likelihood estimation from incomplete data. Biometrics, 14:174-194, 1958. 15. H.-K. Lee and J. H. Kim. An hmm-based threshold model approach for gesture recognition. IEEE Trans on Pattern Analysis and Machine Intelligence, 21(10):961-973, October 1999. 16. S. E. Levinson, L. R. Rabiner, and M. M. Sondhi. An introduction to the application of the theory of probabilistic functions of a markov process to automatic speech recognition. The Bell System Technical Journal, 62(4): 1035-1074, 1983. 17. Nianjun Liu, R. I. A. Davis, B. C. Lovell, and P. J. Kootsookos. Effect of initial hmm choices in multiple sequence training for gesture recognition. In Proceedings of the International Conference on Information Technology, volume 1, pages 608 - 613, Las Vegas, Nevada USA, April 5-7 2004. 18. R.B. Lyngso, C.N. Pedersen, and H. Nielsen. Metrics and similarity measures for hidden markov models. In International Conference on Intelligent Systems for Molecular Biology, pages 178186, 1999.
40 19. D. J. C. Mackay. Ensemble learning for hidden markov models. Technical report, University of Cambridge, 1997. 20. B. McCane and T. Caelli. Diagnostic tools for evaluating hmm components. Technical report, Department of Computer Science, University of Otago, 2001. 21. G. McLachlan and T. Krishnan. The EM algorithm and extensions. Wiley series in probability and statistics. John Wiley and Sons, 1997. 22. C. S. Myers and L. R. Rabiner. A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical Journal, 60(7): 1389-1409, September 1981. 23. J. Norris. Markov Chains. Cambridge University Press, Cambridge, England, 1997. 24. L. R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-286, February 1989. 25. L.R. Rabiner and B.H. Juang. Fundamentals of Speech Recognition. Prentice Hall, New Jersey, 1993. 26. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379-423, July 1948. 27. T. Starner and A. Pentland. Real-time american sign language recognition. IEEE Trans, On Pattern Analysis and Machine Intelligence, 20:1371-1375, December 1998. 28. A. Stolke and S. M. Omohundro. Best-first model merging for hidden markov model induction. Technical report, International Computer Science Institute, April 1994. TR-94-003. 29. A. J. Viterbi. Error bounds for convolutional codes and an asymtotically optimal decoding algorithm. IEEE Transactions on Information Theory, IT-13:260-269, April 1967.
C H A P T E R 1.3 A N E W KERNEL-BASED FORMALIZATION OF MINIMUM ERROR PATTERN RECOGNITION
Erik McDermott & Shigeru Katagiri NTT Communication Science Laboratories, NTT Corporation, Kyoto 619-0237, Japan E-mail:
[email protected]
Previous expositions of the Minimum Classification Error framework for discriminative training of pattern recognition systems describe the use of a smoothed version of the error count as the criterion function for classifier design, but do not specify the origin and nature of the smoothing used. In this chapter we show that the same optimization criterion can be derived from the classic Parzen window approach to smoothing in the context of non-parametric density estimation. The density estimated is not that of the category pattern distributions - as performed in conventional non-discriminative methods such as maximum likelihood estimation - but rather that of a transformational variable comparing correct and best incorrect categories. The density estimate can easily be integrated over the domain corresponding to classification mistakes, yielding a cost function that is closely related to the original MCE cost function. The risk estimate formulation presented here provides a new link, Parzen estimation, between the empirical cost function calculated on a finite training set and the true theoretical classification risk. The classic Parzen concepts of linking the kernel width to the amount of training data can be used in the context of discriminative training to express the intuitive concept of using a smaller margin as the amount of training data increases.
1. I n t r o d u c t i o n 1.1. Towards
minimizing
classification
error
T h e Minimum Classification Error (MCE) framework is an approach to discriminative training for p a t t e r n classification t h a t explicitly incorporates classification performance into t h e optimization criterion 1 , 2 ' 3 ' 4 . Given discriminant functions for each category, M C E defines a loss function t h a t is a smoothed approximation of the recognition error rate and uses this function as t h e criterion function for optimization. T h r o u g h minimization of this criterion function, M C E is aimed directly at minimizing classification error rather t h a n at learning t h e t r u e d a t a probability distributions, the target of Maximum Likelihood Estimation (MLE). M C E has been 41
42
used successfully to train hidden Markov models (HMMs) for large-scale speech recognition tasks, as well as handwriting recognition and document classification tasks, yielding significant improvements over conventional MLE-based training and Support Vector Machines 4 ' 5,6 ' 7 ' 8 . Here we show that the continuous, 0-1 MCE loss function can be derived from an estimate of the theoretical classification risk, using Parzen estimation of the density of a suitably defined transformational variable. In this analysis, the specific kernel type used for Parzen estimation leads to a specific type of MCE loss function, and vice versa; the width of the Parzen kernel directly corresponds to the steepness of the MCE loss function, and vice versa. Minimization of the MCE loss function therefore corresponds to the minimization of a Parzen window based estimate of the theoretical classification risk. The well known convergence properties of Parzen estimates to the true densities, as the training set increases and the kernel width is reduced, can now be applied directly to the MCE framework. The reader is referred to a considerable literature on alternative approaches to discriminative training 9 ' 10 ' 11,12,13 , and in particular to the Maximum Mutual Information (MMI) approach that has been used extensively in speech recognition 14,15,16,17,18,19,20,21 . MMI and MCE, though they derive from fundamentally different backgrounds, are closely related, as discussed in several previous studies 3,22 . 1.2. Useful estimates
of classification
risk
The concept of estimating classification risk is not new 23 . However, in general, evaluating classification risk analytically is a difficult task. Classification systems typically create complex, highly non-linear decision boundaries in the pattern space; furthermore, the pattern space is usually high dimensional and often variable-length. Evaluating the risk integral can usually be performed only for low-dimensional spaces and simple classifier structures. As an alternative to attempting to evaluate the risk integral in the pattern space, smoothed versions of the misclassification count have been proposed by several authors 24,25 , but these proposals may have lacked practical applicability to general classifier design. The risk estimate formulated here can be seen as a smoothed version of the misclassification count, but it is explicitly linked to the original integral of classification risk in the pattern space. Furthermore, the risk estimate is a tractable function that can be used in practical gradient-based optimization techniques to find model parameters that minimize the risk estimate for general classifier structures such a HMMs or Multi-Layer Perceptrons 26,27,28 . Though this approach makes use of the "memory-based" Parzen window method to estimate classification risk, after the loss derivatives have been calculated for each training token, the kernel for that training token is no longer needed and can be discarded. In this sense the approach is very different from memory-based approaches to incorporating Parzen window estimation into machine learning architectures, such as the Probabilistic Neural Networks 29 , which in principle stores all the train-
43
ing tokens as kernel centers. 2. The Minimum Classification Error framework The MCE framework has been described in several publications 3,4 ' 30 . MCE uses a three-step definition, mapping each training pattern token x and the system parameters A (e.g., all the HMM means and covariances) to a continuous 0-1 loss function reflecting classification error. The pattern x could be a single pattern vector or a sequence of, e.g., speech-derived feature vectors, x = x^ = (x 1 , ...,x*, ...,x T ). Not limited to speech recognition or HMMs, the formalism simply assumes that discriminant functions
W*'A))=l
+ e -L i( x,A).
(!)
but can be any of a variety of continuous 0-1 functions. The total loss or empirical cost is the local loss summed over the training data X: ,
M
Nj
L(X,A) = -1£Yit(dj(Xij,A)), j
(2)
i=i
where xy denotes the i-th training token in category Cj, M the number of categories, and Nj the number of training tokens for category Cj. The overall loss function can be minimized using several different optimization methods 4 . 3. M C E for speech recognition In the context of speech recognition, the discriminant function typically used for MCE is defined in terms of a speech utterance x f and the corresponding best Viterbi state sequence 6 l = {8\, ...,6\, ...,8lT) for word string category Cj 5 : T gi(xJ,A)
T
i
= logPA(xf,e ) = 5>g°fli_ie{ + 5 > g & 8 j ( x t ) , t=i
(3)
t=i
where auv denotes a transition probability from state u to state v and bs(-) denotes a Gaussian mixture used to model the observation probability of a feature vector in state s. The effectiveness of MCE has been demonstrated on many speech recognition tasks. On the TIMIT task, MCE-trained HMMs yielded a phoneme classification accuracy of 77.6%, which is identical to the highest classification results reported for Support Vector Machines (SVM) 31 . In order to enable the use of SVMs on this
44
task, variable-length speech segments were converted to fixed-dimensional vector patterns 31 . The need for this conversion makes it more difficult to apply the SVM approach to general speech recognition problems, beyond the classification of short speech segments. In contrast, since the MCE loss function is defined using discriminant functions that embed Viterbi decoding applied to HMMs and arbitrary-length speech input, MCE can directly be applied at the utterance level for a variety of recognition tasks. Among many such results, the utility of MCE for practical speech recognition was illustrated by results for a "real-world", large-vocabulary, telephone-based task, where MCE was shown to yield striking improvements in recognition accuracy and system compactness compared to conventional MLE / Viterbi Training 6 . 4. A novel analysis of the smoothness of the M C E loss function The smoothness of the 0-1 MCE loss function has two important roles. First, it enables the use of gradient based optimization techniques. Second, it has a strong effect on generalization 32 . In this chapter we clarify the nature and meaning of the MCE loss function with regard to this second aspect. In particular, we show that the smoothness of the MCE loss function can be viewed as the direct consequence of using a well-understood type of smoothing, Parzen estimation, to estimate the theoretical risk. The width of the Parzen kernel used in this estimation is inversely related to the steepness of the MCE loss function. The minimization of the overall MCE loss function, summed over all the training tokens, is shown to be equivalent to the minimization of an estimate of the theoretical classification risk. The benefit for generalization of using a smooth MCE loss function is directly related to the benefit of using kernels to smooth over finite training data in Parzen estimation. 4 . 1 . Classification
risk & Bayes
decision
rule
Essentially, we show that the smooth MCE loss function can be derived from a particular approach to modeling - with a view to then minimizing - the true theoretical classification risk 33 : R = I R{a(x)\x)p(x)dx.
(4)
Jx Here a(x) represents the classification decision (the choice of one out of M categories) made for the input pattern x; R(a(x)\x) represents the risk involved in making specific classifications: M
i2(a i |x) = ^ A ( a i | C , ) P ( C i | x ) ,
(5)
where \(ai\Cj) denotes the cost of mis-classifying a member of category Cj as category d. Typically this cost is 1 whenever i and j are different - in other words, classifying a member of a given category as any other category incurs the same cost
45
of 1. It is easy to show 33 that the overall risk (Equ. (4)) is minimized by the Bayes decision rule: decide d if P(d |x) > P{Cj |x) for all j ^ i.
(6)
An important point is that achieving the optimal Bayes risk does not require explicit knowledge of the posterior probabilities P(Cj|x): knowledge of the category with the maximal P(Cj|x) is sufficient. Consistent with this observation, the MCE approach is to focus directly on designing a pattern classification system so that its decisions minimize the overall classification error. 4.2. Linking
the classification
risk to discriminant
functions
In the following, we start with the above definitions of risk and arrive at the MCE approach to classifier design. This derivation is different from previous expositions of MCE and links the empirical error measured on a given, finite training set, and the theoretical risk given above. The first step is to link Equ. (4) to an actual pattern recognition system. Using the expression for the risk incurred by an individual classification decision, given by Equ. (5), we can rewrite Equ. (4) as M
.
R = J2
A(a(x)|C J -)P(C J -|x)p(x)dx;
3= 1
(7)
J X
which in turn can be written as M
R = Y,
.
/ A(a(x)|C,-)p(C,-,x)dx.
(8)
The link between this expression of risk and an actual classification system is embodied by the decisions a(x) that the system takes. In a system where the classification decision is to choose the category with the largest discriminant function value <7j(x, A), we can express the categorical cost \(a(x)\Cj) as A(a(x)|C j ) = l((-
0).
(9)
Here we again assume that \(a>i\Cj) is 0 when i and j are the same, and 1 when they are different. The indicator function 1((— gj(x) + m a x ^ <7i(x)) > 0) compares the discriminant function value for category Cj with all other categories' discriminant functions; if Cj has the largest discriminant function value, the system will choose Cj and the indicator function 1((—^(x, A) + m a x ^ j ^ ( x , A)) > 0) is 0; otherwise the system does not choose Cj and the indicator function is 1. The discriminant functions are defined in a system specific way, that we leave open but that will in general depend on a number of modifiable parameters, A. We can now rewrite the overall risk in terms of the indicator function just introduced: M R
= Y,
r
l ( ( - 0 i ( x , A) +max 0)p(C,-,x)dx.
(10)
46
P(C)p(xlC)
a(x,A)=C2
a(x,A)=Ci
Fig. 1. Defining classification risk in a new domain. For each category, pattern vectors on the wrong side of the decision boundaries (shaded portions) are mapped to positive values in the one-dimensional transformed space.
4.3. Defining
classification
risk in a new, simpler
domain
The misclassification measure used in the following is: rrij = 4J(X, A) = — <7j(x, A) +max
(11)
The overall risk can then be written as M
R
JT f l(di(x,A)>0)p(CJ-,x)dx.
(12)
In turn, this is equivalent to integrating over a reduced part of the pattern space X: M
(13) where Xj is given by Xj = {x€X\
dj(x,A)>0}.
(14)
If X and gi(X,A),...,gi(X,A),...,gM{X,A) are continuous random variables, Mj = dj (X, A) is also a continuous random variable. We know that the cumulative distribution of a function of a random variable, Y — g(X), can be found by
47
integrating the density fx(x)
over the appropriate region, i.e.
fy(y)dy
P[Y
= P[g(X) <
J —(
fx(x)dx.
y}= I
(15)
Jx:g(x)
We can therefore express the integral for each category Cj over the reduced space Xj with an integral over the positive domain of the misclassification measure ray. P(Cj) [ p(x\Cj)dx. = P[dj(X,A)
> 0,Cj] = P(Cj) [
JXj
p{mj\Cj)dmj.
(16)
J0
Herep(x.\Cj) andp(m,j\Cj) could be written / x ( x | C j ) and fM(iTij\Cj) to emphasize that they operate in different spaces. Note that each nij corresponds to a distinct set of points x in Xj. For clarity, the dependence of rrij on the system parameters A is not explicit in the notation. We can now express the overall risk expressed in Equ. (13) as: R = Y.P(Cj) j=l
p(mj\Cj)dmj.
(17)
J
<>
This is a new expression for the classification risk that is equivalent to the original expression given in Equ. (4), given only the assumption that X and the gi(X,A) are continuous random variables. Rather than integrate over a part of the pattern space from which the "input" random variable X takes its values, this expression integrates over the corresponding part of the space for the transformed random variable Mj - a much easier task. Fig. 1 illustrates the new definition of risk. Estimate of classification risk: sum integrals (m>0) for each Parzen kernel
Parzen estimate of p(mlC): Sum kernels centered on (transformed) data points
kernel center c: transformed data point d(x,A)
x\ m = d(x,A) Fig. 2. Estimate of classification risk in a new domain. Parzen kernels used to estimate the density of the misclassification measure are integrated in the positive domain corresponding to classification errors.
48
4.4. Parzen
estimate
of risk
In order to explicitly relate the empirical cost to the theoretical cost, we use the following Parzen estimate of the density p(rrij\Cj): N •
PN^mjlCj)
= — ^
-^H—
Y~*—-)•
(18)
Here 4>{{mj — dj(xij,A))/h) is a window or kernel centered on the data point dj(xij,A) and with width h. The dependence on category Cj is reflected in the fact that the sum is only over data points x^- from category Cj. We know from Parzen window estimation theory 33 that the estimate given by Equ. (18) converges to the true density p(rrij\Cj) as the number of data points approaches infinity, reducing the window width h to zero as we increase the data set size. Furthermore, this property holds for a wide variety of window/kernel types, including Gaussian kernels and simple uniform kernels. With the estimate p ^ (rrij\Cj) defined for any value of rrij, we can define an estimate of the true overall risk expressed in Equ. (17): R
N = YJP{Cj) j=i
Expanding this using the estimate
pNj{mj\Cj)dmj.
PN^TUJICJ)
given by Equ. (18) yields
RN = J2 P{CJ) j f ^ £ ^ " ' f ^ W Rearranging, and using Nj/N
(19)
Jo
(20)
(where N = Y.f=i Nj) for P(Cj), gives:
Equ. (21) is an estimate of classification risk, defined in terms of Parzen estimates of the densities of category-dependent misclassification measures, and expressible as a sum of integrals of Parzen kernels over positive values of the misclassification measure. The estimate is anchored on actual (transformed) data points. The properties of Parzen estimation 33 entail that this estimate converges to the theoretical risk as the number of data points approaches infinity and the kernel width is reduced. Furthermore, we have some control over the value of RN via the system parameters A that are implicit in the definition of the misclassification measure, and that therefore affect the specific values of the window anchor points dj(xij,A). 4.5. Estimate
of risk for specific
kernels
We now show the utility of the Parzen window based estimate of risk (Equ. (21)) by considering simple Parzen kernel types. For these kernels, we can easily find the closed form of the integral in Equ. (21).
49 4.5.1. Uniform Parzen kernel leads to piece-wise linear MCE loss
PW=1HT)
P(n 0)
1/h
0.0 C
Fig. 3. Integrating Parzen estimates of misclassification measure density over the positive domain (corresponding to the part of the pattern space that is misclassified) yields, for each data point, 0-1 functions of the Parzen kernel center that are strikingly similar to typical MCE loss functions. A uniform Parzen kernel leads to a piece-wise linear estimate of classification risk.
W e first c o n s i d e r t h e o n e - d i m e n s i o n a l u n i f o r m k e r n e l ,
4>{u)
r i i f «| < 1/2, \ 0 ootherwise. t
(22)
(/>({rrij — dj (XJJ , A))//i) will be one if mj falls within the segment of length h centered at dj(x.ij,A), and zero otherwise. We can now integrate Equ. (21) for this specific choice of kernel. Focusing first on a single data point x ^ , and abbreviating d = dj(xij,A), we have
i r
(23)
hJo
The integral over m,j yields a piece-wise linear function that rises monotonically from zero to one. This corresponds to the classification risk for a single token x^-, given a Parzen kernel centered on the transformed data point dj(xij,A). Writing this expression as £i(), we can express the estimate of classification loss (21) as M
Nj
*" = ^ £ X > (<*;(*«.A))j=i.
(24)
50
Strikingly, this is identical to the empirical MCE cost defined earlier in this chapter (Equ. (2)), assuming the piece-wise linear MCE loss used in several MCE-based studies 4 . As the training set grows larger, the kernel width h should be decreased as per Parzen estimation theory, resulting in an increasingly steep ^i(-). As the training set size approaches infinity, h approaches zero, and t\{-) approaches the binary loss function. The transition from the integral of a single Parzen kernel over positive values of rrij to a function of the kernel center c = dj(xij,A) is illustrated in Figure 3. 4.5.2. Sigmoid MCE loss function corresponds to simple Parzen kernel Having shown how a simple uniform Parzen kernel leads to a simple piece-wise linear MCE loss function, we can now go the other way, start with an MCE loss function and ask what Parzen kernel it corresponds to. The most commonly used MCE loss function is the simple sigmoid function (Equ. (1)), with steepness controlled by a. Here we show that using this loss function for MCE corresponds to using, in Equ. (21), a specific Parzen kernel whose width is inversely related to a. Since we're looking for a kernel which integrates to Equ. (1), it will clearly have the form of the derivative of Equ. (1). One can verify that using the kernel *(«) = ( T ^
(25)
in the integral in Equ. (21) yields back the sigmoid loss. As above, the kernel is centered on the given data points dj (XJJ , A) and its width and height are controlled by a scalar h. Again abbreviating d = dj(xij,A), the integral for a single data point X7; o IS
1 f°°
xfrrij-d.
,
e^mi-V
f°° 1
drrij,
(26)
which yields, with h = 1/a, the MCE sigmoid loss function (Equ. (1)). The corresponding overall estimate of risk (Equ. (21)) then becomes identical to the empirical MCE cost (Equ. (2)) assuming, this time, the sigmoid loss function that has also been used in previous MCE-based studies. Figure 4 illustrates the relation between the sigmoid loss function and the corresponding Parzen kernel. 4.5.3. Gaussian kernel leads to classic error junction A common choice in Parzen estimation is the Gaussian kernel, 1
/
1 2
Using this in the integral in Equ. (21) yields a loss function based on the classic error function. Again abbreviating d = dj(xij,A), the integral for a single data
51 p(m) kernel:
2 4h P(m)=^(^)
(1+eu
P(m>0) =
Fig. 4. The commonly used MCE sigmoid loss function can be viewed as the probability that the misclassification measure is greater than zero, given a data point Xy mapped to a kernel center c = dj (XJJ•, A), for a particular type of Parzen kernel used in estimating the density of the misclassification measure.
point Xij is 1
frrij
—d
h
~h
)drrijJ
'~""
1 /m,\2
can
: exp
h J_d/h V2i
drrin
(28)
\ h
Given the definitions of erf(-) and its complement erfc(-), 2
erfc(x) = 1 — erf(x)
f00
= —= /
exp (—t2) dt,
(29)
we can see that Equ. (28) can be written as 1 f°° ,,mi-ds )d
h L ^-h- ^
,
1
„ / -d\
1
=2
[h^)
=2
GrfC
erf
/ d
[17=2
(30)
In this context, this result is very natural, since the classic error function is itself defined as the integral of a Gaussian. 4.6. Minimizing
the risk
estimate
Through the use of Parzen estimates of overall risk, and kernel types that allow us to integrate directly the general expression for risk estimate given by Equ. (21), we have arrived at expressions of empirical MCE cost that we are already familiar with, and that can be optimized in exactly the same way as the MCE cost. The MCE
52
derivatives described in previous studies 4 apply unchanged to the Parzen-based risk estimate. Though the derivative of Equ. (11) has discontinuities, these are not a significant obstacle to gradient-based optimization. Probabilistic Descent1, modified Newton's methods 4 and approximated re-estimation approaches 34 are among the optimization methods used so far for MCE. From the point of view of the Parzen analysis provided here, the task of any optimization procedure applied to the risk estimate (Equ. (21)) is to choose the system parameters A so as to position the kernel centers in a way that minimizes the estimated risk. Whether the resulting minimization corresponds to the theoretically optimal Bayes risk rests on the effectiveness of the optimization procedure, and of course on the model structure the number of parameters and other structural aspects relating to discriminative ability. 5. S u m m a r y The MCE formalism, as presented so far2A4,5,30; d e s c ribes the use for pattern recognition system design of a smoothed 0-1 loss function that approximates the ideal binary classification error. The fact that the MCE loss function is a smooth approximation of the ideal binary loss has been justified in previous work as 1) allowing the use of gradient-based optimization methods, and 2) performing an unspecified type of smoothing over available training data. Here we have shown that the MCE loss function can be seen as the direct result of a particular type of smoothing, the classic Parzen window estimation method, and the corresponding estimate of classification risk, in the domain of a suitably defined random variable. This variable, Mj = dj(X,A), embeds the potentially variable duration input pattern X, the system parameters A, and a comparison between correct and incorrect category matches. Parzen estimation, typically limited to modeling fixed-dimensional pattern vectors, can be applied in this domain to provide a simple expression of classification risk. Typical MCE loss functions £(dj(xij,A)) can be interpreted as the integral of a Parzen kernel of width h centered on dj(xij,A), the transformed version of the training token x ^ :
/(d,(x«,A)) = ^ £ % ( m j ~ ^ ' A W
(31)
The specific form of the kernel corresponds to the form of an MCE loss function, and vice versa. This analysis broadens the theoretical understanding of the MCE framework, and provides a novel link between the MCE empirical cost measured on a finite training set and the theoretical classification risk. References 1. S. Katagiri, C-H. Lee, and B.-H. Juang. New discriminative training algorithms based on the generalized descent method. In Proc. IEEE Worshop on Neural Networks for Signal Processing, pages 299-308, 1991.
53 2. S. Katagiri, C-H. Lee, and B.-H. Juang. Discriminative Multilayer Feed-Forward Networks. In Proc. IEEE Worshop on Neural Networks for Signal Processing, pages 309318, 1991. 3. S. Katagiri, B.-H. Juang, and C.-H. Lee. Pattern recognition using a family of design algorithms based upon the Generalized Probabilistic Descent method. Proceedings of the IEEE, 86(ll):2345-2373, 1998. 4. E. McDermott. Discriminative Training for Speech Recognition. PhD thesis, Waseda University, School of Engineering, March 1997. 5. W. Chou, B.-H. Juang, and C.-H. Lee. Segmental GPD training of HMM based speech recognizer. In Proc. IEEE ICASSP, volume 1, pages 473-476, March 1992. 6. E. McDermott, A. Biem, S. Tenpaku, and S. Katagiri. Discriminative training for large vocabulary telephone-based name recognition. In Proc. IEEE ICASSP, volume 6, pages 3739-3742, 2000. 7. A. Biem. Minimum Classification Error Training of Hidden Markov Models for Handwriting Recognition. In Proc. IEEE ICASSP, volume 3, 2001. 8. S. Gao, W. Wu, C.-H. Lee, and T.-S. Chua. A Maximal Figure-of-Merit Learning Approach to Text Categorization. In Proc. of ACM SIGIR Conference on Research and Development in Information Retrieval, pages 174-181, 2003. 9. T. Applebaum and B B. Hanson. Enhancing the discrimination of speaker independent hidden Markov models with corrective training. In Proc. IEEE ICASSP, volume 1, pages 302-305, 1989. 10. H. Franco and A. Serralheiro. Training HMMs using a Minimum Error approach. In Proc. IEEE ICASSP, volume 1, pages 357-360, 1990. 11. Y. Ephraim A. Ljolje and L. Rabiner. Estimation of hidden Markov model parameters by minimizing empirical error rate. In Proc. IEEE ICASSP, volume 2, pages 709-712, 1990. 12. H. Gish. A Minimum Classification Error, Maximum Likelihood, Neural Network. In Proc. IEEE ICASSP, volume 2, pages 289-292, 1992. 13. J. Hampshire. A differential theory of learning for efficient statistical pattern recognition. PhD thesis, Carnegie Mellon University, Department of Electrical and Computer Engineering, 1991. 14. L. Bahl, P.F. Brown, P. V. de Souza, and K. L. Mercer. Maximum Mutual Information Estimation of hidden Markov parameters for speech recognition. In Proc. IEEE ICASSP, volume 1, pages 49-52, 1986. 15. P. Brown. The acoustic-modeling problem in automatic speech recognition. PhD thesis, Carnegie Mellon University, Department of Computer Science, 1987. 16. D. Nahamoo A. Nadas and M. Picheny. On a model-robust training method for speech recognition. IEEE Transactions on Signal Processing, 36(9): 1432-1435, 1988. 17. P. S. Gopalakrishnan, D. Kanevsky, A. Nadas, D. Nahamoo, and M. Picheny. Decoder selection based on cross-entropies. In Proc. IEEE ICASSP, volume 1, pages 20-23, 1988. 18. A. Nadas P. S. Gopalakrishnan, D. Kanevsky and D. Nahamoo. An inequality for rational functions with applications to some statistical estimation problems. IEEE Transactions on Information Theory, 37(1):107-113, 1991. 19. Y. Normandin. Hidden Markov Models, Maximum Mutual Information Estimation, and the Speech Recognition Problem. PhD thesis, McGill University, Montreal, Department of Electrical Engineering, 1991. 20. V. Valtchev, J. J. Odell, P. C. Woodland, and S. J. Young. Lattice-based discriminative training for large vocabulary speech recognition. In International Conference on Spoken Language Processing, volume 2, pages 605-609, 1996.
54 21. P.C. Woodland and D. Povey. Large scale discriminative training of hidden Markov models for speech recognition. Computer Speech and Language, 16:25-47, 2002. 22. R. Schluter, W. Macherey, S. Kanthak, H. Ney, and L. Welling. Comparison of Discriminative Training Criteria. In Proc. ICASSP '98, pages 493-496, 1998. 23. G. J. McLachlan. Discriminant Analysis and Statistical Pattern Recognition. Wiley, 1992. 24. S. Amari. A Theory of Adaptive Pattern Classifiers. IEEE Trans, on Electronic Computers, EC-16(3):299-307, 1967. 25. L. Devroye, L. Gyorfi, and Gabor Lugosi. A Probabilistic Theory of Pattern Recognition. Springer Verlag, 1996. 26. E. McDermott and S. Katagiri. A Parzen window based derivation of Minimum Classification Error from the theoretical Bayes classification risk. In International Conference on Spoken Language Processing, pages 2465-2468, 2002. 27. E. McDermott and S. Katagiri. A new formalization of Minimum Classification Error using a Parzen estimate of classification chance. In Proc. IEEE ICASSP, volume 2, pages 713-716, 2003. 28. E. McDermott and S. Katagiri. A derivation of Minimum Classification Error from the theoretical classification risk using Parzen estimation. In Computer Speech and Language, 2004. 29. D.F. Specht. Probabilistic Neural Networks. Neural Networks, (3):109-118, 1990. 30. B.-H. Juang and S. Katagiri. Discriminative Learning for Minimum Error Classification. IEEE Trans, on Acoustics, Speech, and Signal Processing, 40(12):3043-3054, Dec. 1992. 31. P. Clarkson and P. Moreno. On the use of support vector machines for phonetic classification. In Proc. IEEE ICASSP, volume 2, pages 585-588, 1999. 32. B.-H. Juang and S. Katagiri. Discriminative training. Journal of the Acoustical Society of Japan, 13(6):333-339, 1992. 33. R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley Interscience Publications, 1973. 34. Q. Li and B.-H. Juang. A new algorithm for fast discriminative training. In Proc. IEEE ICASSP, volume 1, pages 97-100, 2002.
C H A P T E R 1.4 PARALLEL CONTEXTUAL ARRAY G R A M M A R S TRAJECTORIES
WITH
P. Helen Chandra Department of Mathematics,
J. A. College for Women, Periyakulam,
Theni, India
C. Martin-Vide Research Group on Mathematical Linguistics, Rovira i Virgili University, PL Imperial Tarraco 1, 43005 Tarragona, Spain
K.G. Subramanian Department of Mathematics,
Madras Christian College, Chennai, India
D.L. Van Department of Mathematics of Computer Sciences, Institute of Mathematics , NCST, 18 Hoang Quoc Viet Road,10307, Hanoi, Vietnam P.S.P. Wang College of Computer Science, Northeastern University, 360 Huntington Avenue CN, Boston MA 02115
#221
A simple and effective syntactic method of description of picture patterns of rectangular arrays of symbols is proposed. The method is based on contextual grammars of Marcus(1969) guided by trajectories in the generation process.The resulting grammar is called Parallel contextual array grammar with contexts shuffled on trajectories. This grammar is capable of describing pictorial patterns that cannot be handled by certain other contextual array grammars introduced in the literature.
1. I n t r o d u c t i o n Syntactic approaches, on account of their structure-handling ability, have played a significant role in t h e problem of description of picture p a t t e r n s considered as connected digitized, finite arrays of symbols 1 5 . Adapting t h e techniques of formal string language theory, various types of picture or array grammars have been introduced and investigated 1 , 2 ' 4 ' 5 ' 1 3 ' 1 4 . Most of t h e array grammars developed t o handle pic55
56
ture languages, are based on Chomskian string grammars. Another well-investigated class of grammars, called Contextual grammars, was proposed by Marcus 8 . A contextual grammar defines a string language by starting from a given finite set of strings and adjoining iteratively pairs of strings (called contexts) associated to sets of words (called selectors), to the strings already obtained. These contextual grammars 3 ' u are known to offer new approaches for a number of basic problems in formal language theory. But they have not received much attention regarding their suitability to describe picture patterns. Recently, extension of string contextual grammars to array structures has been attempted 7 ' 12 . In 6 , based on contextual grammars, a new method of description of pictures of digitized rectangular arrays, called a Parallel Contextual Array Grammar, is introduced. This model is different from 7 ' 12 . Several properties of these array grammars have been studied in 5 . On the other hand Martin-Vide et al 9 introduced the notion of adjoining contexts to words by shuffling them on 'trajectories' giving rise to a very general class of contextual string grammars. The notion of shuffle on trajectories was introduced in 10 , while studying the problem of parallel composition of words and languages. Motivated by these investigations, description of rectangular arrays by Contextual array grammars guided by trajectories, is considered here. Parallel Contextual array grammars with contexts shuffled on trajectories are introduced . These grammars are simple yet powerful in describing array structures. Comparison with certain other array generating models is done and closure properties under certain geometric operations are obtained.
2. Preliminaries Let V be a finite alphabet. V* is the set of all words over V including the empty word A. A picture A over V is a rectangular m x n array of elements of V of the form an
• • • a-\r.
A ^ml * ' "
^jnn
The set of all pictures or arrays over V is denoted by V**. A picture or an array language over V is a subset of V**. Definition 1: Let A = \an] „ , The column concatenation
B = \aa] „
A<&B of A and B is defined only when m = n and is
57
given by
A$B
an
aip b 1 1
Ami
*mp
• •
nq
= Ki
u
nq
Similarly, the row concatenation AQB of A and B is defined only when p = q. The empty array is also denoted by A, and A$P = P$A = P and AGP = POA = P for any P G V**. 3. Parallel Contextual Array Grammars We now recall the notions of array contexts and Parallel Contextual Array Gram-
Definition 2: Let V be a finite alphabet. A column array context over V is an element of V**$cV**of the form Ml
V\
(1)
$c
u2
_v2_
where u\, u2 are of size 1 x p and V\,v2 are of size 1 x q, for some p, q > 1 and is a special symbol not in V. If ui = U2 = A, then we write c = $ c
and if ^2
w
i = u2 = A, then c =
Ml
$c. A
TXJIO
array context over V is an element of
«2
V**$rV** of the form r = [ui u2] $ r [vi v2]
(2)
where ui,u% are of size p x l and v\, V2 are of size q x 1, for some p, g > 1 and $ r is a special symbol not in V. If u\ = U2 = A, then we write r = $ r [v\ v2] and if vi — v2 = A, then r = [ui u2\ %r • Definition 3: A parallel internal contextual array grammar is a system G = (V,B,Ca,Ra,(pc,ipr) where V is an alphabet, B is a finite subset of V** called the base of G, Ca is a finite subset of V**%CV** consisting of column array contexts, Ra is a finite subset of V**$rV** consisting of row array contexts, ipc : V** —* 2Ca and <pr : V** —> 2Ra are the choice mappings. When in Y, where X, Y G ^** if and only if X = XX$>X2§X3, Y = XX^W^X2^Z^X3 o r I = XiGXzGXs, Y = X1eWGX2ezeX3 for some Xi, X2, X3 G V**, and W, Z are contexts 'built' from the column/row array contexts
58
according to the choice mappings. We denote by =>*n, the reflexive and transitive closure of the relation =>»„ . Informally speaking, while 'building' W, Z from column array contexts, we imagine that a window is moved down the columns of the array X2 and W, Z are constructed from the column contexts as 'dictated' by the choice mappings. Likewise for W, Z 'built' from row array contexts. The language generated by G, denoted by L(G) is defined as L(G) = {Y € V**/ there exists X £ B such that X =^*n Y}. We write X =>cin Y (respectively X = ^ n Y) when column (respectively row) operations are performed on X to yield Y. The family of all parallel internal contextual array languages is denoted by PI AC. Example 4: Let G = (V,B,Ca,Ra,
be a parallel internal contextual array
aba B = < b bb > . Let B\, Bi, • • • , BQ be respectively
grammar where V = {a, b},
aba aba
b bb
aba
bbb , aba,
aba
ca = 1 Ra
a b
a b
ab
ba
bb
bb
bb
a a , a b, b a
a $c
= \ ab
fc [Si] =
aa
b •b>r
a $c
<j)r
b
b 5
b
a ab
5
$c
a
a
aa
a
q>r
aa
b~ , fc [B2] =
a $c
a
ba
i
I
$ r 1 b a 11 ,
V
a $c a , fc[B3} =
a a ^ V\{i B 5 ] =
ab
$r
aI
,
5
ipr
a a
a $c
[B3 ] =
a
ba
ba
The picture language L generated by G has pictures of the following forms
•
where 'a' stands for black square and 'b' stands for white square. Definition 5: A parallel external contextual array grammar is a system G = (V,B,Ca,Ra,ipc,<pr) where the components are as in Definition 3.
59 The direct derivation with respect to G is a binary relation =>ex on V** defined as X =>ex Y, where X,Y £ V** if and only if Y = W$X$Z or Y = WQXQZ, and W, Z are contexts as explained in Definition 3. We denote by =>*x the reflexive and transitive closure of the relation =>ex . The language generated by G, denoted by L(G) is defined as L(G) = {Y G V**/ there exists X € B such that X ^-*ex Y}. The family of all parallel external contextual array languages is denoted by PEAC.
Example 6: Let G = (V, B, Ca,Ra, ipc, ipr) be a parallel external contextual array bab grammar where V = {a, b}. B = <
r-
-i
aa
-1
$c
ab
b aJ ab aa
aa
<pr
aa ab
for all n > 1, for all m > 1, x We obtain the arrays of the picture language L generated by G which is the set of floor designs of the following forms
represented by
60
baba bab aaaaaaa bab
bababab
b ab a b a b
aaa
aaaaaaa
aaaaaaa
bab , bababab,
b ab ab ab aaaaaaa
bababab where 'a' stands for the primitive 'circle' and 'b' stands for blank. 3.1.
Necessary
conditions
Necessary conditions for a language to be of a certain type of parallel contextual array language is now considered. Notations: \X\C is the number of columns in array X. \X\r is the number of rows in array X. Definition 7: A language M C V** has the External Bounded Step (EBS) property if there is a constant p such that for each X e M, with \X\C > p, or \X\r > p, either of the following properties i),ii) holds: i)there is a Y e M such that X = LQY&R, and \LR\C < p (External Column Bounded Step property), ii) there is a Y € M such that X = UQYGD, and
U D
< p (External Row Bounded
Step property). Definition 8: A language M C V** has the Internal Bounded Step (IBS) property if there is a constant p such that for each X € M, with |X| c > p, or \X\r > p, either of the following properties i), ii) holds: i) there i s a F e M such that X = X1$L$X23>R$X3, Y = X1$X2$X3, and 0 < |LJJ|C < p. (Internal Column Bounded Step property) (ii) there is a Y € M such that X = X1QU9X2ODOX3,
Y = x1ex2ex3,&ndo< D
< p (Internal Row Bounded Step property)
Theorem 9: Every PEAC language has the EBS property. Proof: Let M € PEAC and G = (V,B,Ca,Ra,ipc,(pr) be a parallel external contextual array grammar generating M. Let pi = max {\X\C : X e B} , p2 = max {\X\r : X G B} , P3 = max {\uiVi\c} , where Ci
Ui
Vi
e
Ca,
Vi+l
P4 = max <
"
> where r,
Uj Uj +
i
Vj
Vj+i
eRa.
61
Note that IL.RI = \uivA„ if L =
u2
V2
,R
Ul
for some I and
Vl
if U = u\ U2 • • • Uk,D = v\V2 • • • Vk for some k. et p = max{pi,p2,p3,Pi}. If Z = [ p or \Z\r > p then Z £ B. Hence either Z — L$Z'$R for some L and R where Ui
L$cReipc(Z'),
Vi v
Ui+1_
_ i+l_
1 and Z' G M or Z = UQZ"GD Uj u
fc
$c
Ci
Vj- Vj
J + 1
Qi,l
, 1 < i < m-
Oi+1,1 • • • ®i+l,n
for some U and D where U$rD
G y?r(Z"),
1 < j < n - 1 and Z" G M.
G Vr
+ 1
^i,n
'''
Hence there is a constant p such that for each Z £ M, either for \Z\C > p, there exists Z' G M such that Z = L$Z'$R and 0 < \LR\C < p or for | Z | r > p, there exists Z" G M such that Z = UQZ"QD and 0 <
D
< p. Hence M has the EBS
a
property. Theorem 10: Every PI AC language has the IBS property. The proof is similar Remark 11: The converse of the theorem 9 is not true.
We notice this by the following counter example. Consider the language L consisting of the following arrays
{
~b b
aa
aaaa
aa
bb
> bbbb aaaa
5
bb
aa
aaaaaa ) bbbbbb aaaaaa
Obviously L satisfies the EBS property with p = 3. Let G = {B, Cc, Cr, (fic, (pr} be the PEAC grammar associated with L. Obviously B
bb
aa
aa
bb
bb
aa
Cr.=
a b
a $c
b
b
b 1
a
$c
a
62
Since
(Pc(Z2) =
a
aa
a
b
bb
b
a ~b~
aa
a
a 5c
y a
G L we have <^c(Zi) =
a a
whe
and Z 2 =
bb
bb aa
6 & It follows Yi
G L(G). But F x £ L.
a a 66
Definition 12: (i) A language M C V** has the Bounded Array Length Increasing {BALI) property if there is a constant p such that for each X G M, there is F € M with either -p < \X\c - \Y\C < p or -p < \X\r - \Y\r < p. Remark 13: Every PI AC language and PEAC language has the BALI
property.
Theorem 14: If L C V**,L G PI AC, then there are two constants p, q such that every Z £ L, \Z\c > p or \Z\r > p can be written in one of the following forms (i), (ii) : (i) Z = U§V$W§X§Y with U,V,W,X,Y G F**, 0 < \VX\c < q and U$Vi$W$Xi$Y G L for all i > 0; v (ii) z = UQvewexQY with [/,y,w,x,y e r < q and 0• < X UQV^WeX^Y G L for all i > 0. Proof: In order to prove (i) for a parallel internal contextual array grammar G = (V, B, Ca, Ra, (pc, <pr), we define p = max{\W\c, \W\r /WeB}, Ui
where cs —
Vi $c
Ui+1
. Vj Vj + i \ G
.
_Vi+l_
G Ca
Ra
If X = [aij]mXn G L(G) such that \X\C > p, or \X\r > p, then X £ B. When |X| C > p, or \X\r > p, there is a F g £(in X with Y = Xi$X2$X3, X = X1$L§X2i5>R$X3 or a Y G L(G) such that 7 =» in X with Y" = X x G ^ G X a , X = X i e L 0 X 2 0 i ? e X 3 . In the former case this means that there exist column array contexts c, G COJ 1 < i < m — 1, such that Ui
Vi
G
$c
3+1_
_vi+l_
*i,3 a
i+l,j
a.i,k
1 < j < k < n,
' ' ' 0,i+l,k
to form L and R with L$CR G in X i $ L i + 1 $ X 2 * i i i + 1 $ X 3 , for i > 0. All these arrays are in L(G). The argument is similar for (ii). •
63
Corollary 15: {anbncn)r L = < (bncnan)m
I n > 1, m > 1 } $ PI AC.
n n n
(c a b )r The fact that L is not in PI AC is a consequence of Theorem 14Theorem 16: If L C V**,L G PEAC, then there are two constants p,q such that every Z G L, \Z\C > p oi \Z\r > p can be written in one of the following forms (*).(") : (i) Z = U$Y$V with U,Y,V e V**,Q < \UV\C < q and U'QYQV* G L for all i >0 U (ii) Z = UQYQV with U,Y,V €V**,0 < < q and WGYQV1 G L for all V i >0.
The proof is similar Corollary 17: L = {{anbanban)m 3.2.
Closure
j n,m > 1,} £ P E A C .
properties
In this section, we state closure results for the classes PEAC and PI AC.
Theorem 18: (i) The families PEAC with or without choice and PI AC with or without choice are not closed under column and row concatenations. (ii)The families PI AC with or without choice and PEAC with or without choice are not closed under union. (iii) The families PI AC and PEAC are closed under reflection on the base and right leg and rotations by 90°, 180° and 270°. 3.3.
Comparisons
We now state results that compare the class PEAC with the classes LOC5 of local array languages and £(2RLG) of picture languages of 2-dimensional right linear grammars 5 . Theorem 19: The family PEAC and the classes LOC of local array languages are incomparable but not disjoint. Theorem 20: The family PEAC intersects with £(2RLG) generated by two dimensional right linear grammar.
5
of picture languages
64
Theorem 21: The family PEAC without choice and the family EAC of languages generated by external array column contextual grammar without choice 12 are incomparable but not disjoint. Theorem 22: (i) The family PEAC with choice intersects with the family EACC 12 . (ii) The family PI AC intersects with the family I AC 7 . (iii)The family PI AC with choice intersects with the family IACC 7 .(iv) The family PEAC intersects with the family EAC 7 .(v)The family PEAC with choice intersects with the family EACC 7 . Theorem 23: The emptiness problem and the finiteness problem are decidable for all classes of parallel contextual array grammars 4.
Parallel Contextual Array Grammars with Contexts Shuffled on Trajectories
We now introduce the notions of parallel shuffled contexts. Definition 24: A column shuffle array context over V is an element of V** of the form Ml
c =
where u\,U2 are of size 1 x k, k > 1 .
(3)
U2
A row shuffle array context over V is an element of V** of the form r = [u\ U2\ where U\,xi2 are of size k x 1, k > 1 .
(4)
Definition 25: Parallel Shuffled Contexts Let V be an alphabet. Let Ca be a finite subset of V** whose elements are the column shuffle array contexts of the form (3) and 2Ca- be a choice mapping. Given an array X = [a^] of size m x. n,
X =
flu
•• •
:
' •.
a
in
:
, aij
eV,
Ami * ' * Q,mn
we define ipc : V*
V** such that U e
U
u2 if Ci
Ui Ui+l
e ¥>c flj+1,1 ' ' '
a
i+l,n
Ci £ Ca, 1 < i < m — 1, not all c* need be distinct. U is called a parallel column context.
shuffled
65
Similarly we can define a parallel shuffled row context. Let Ra be a finite subset of V** whose elements are the row shuffle array contexts of the form (4) and <pr : V** —• 2Ra be a choice mapping. Given an array X = [ajj] of size m x n, we define <pr : V** -> V** such that U G y v W , ^
Ui Ui+i
G ¥V
=
ixi U2 • • • w n
if
, r; e i? a , 1 < i < n — 1, not all r» need be
distinct. 1/ is called a parallel shuffled row
context.
Definition 26: A (simple) parallel shuffled contextual array grammar is an ordered system G = (V,B,Ca,Ra,ipc,ipr) where V is an alphabet, B is a finite subset of V** called the base of G, Ca is a finite subset of column shuffle array contexts, Ra is a finite subset of row shuffle array contexts, ipc : V** —> 2Ca and ipr : V** —> 2Ra are the choice mappings. When tpc and <pr are omitted, we call G a parallel shuffled contextual array grammar without choice. The direct derivation with respect to G is a binary relation =4>UJ on V** and is defined as X =>LU Y, where X,Y e V** iSY G X LUC U or Y € X LLir U where U is a parallel shuffled column or row context. We denote by = ^ ^ the reflexive and transitive closure of the relation =>LUThe language generated by G, denoted by L(G) is defined as follows: L{G) = {Y e V**/ there exists X &B such that X = ^ Y}. The family of all parallel shuffled contextual array languages is denoted by VACS. Now we introduce the notions of trajectory on arrays, shuffle on trajectories and parallel contextual array grammars with contexts shuffled on trajectories. Definition 27: Let V\ = {I, r} , V2 = {u, d} be the sets of versors in the plane. The symbols l,r, u and d stand for the left, right, up and down directions respectively. A trajectory is an element t £ (Vj* U V2*). Notation: | X | c ( | X | r ) respectively denotes the number of columns (rows) in an array X. \t\x denotes the number of occurrences of x in t. Definition 28: Let V be a finite alphabet, t a trajectory, v a versor , v G Vj. and P,Q G V**. The column shuffle of P with Q on the trajectory vt, denoted by P LLl„t Q, is recursively defined as follows. If P = A$X and Q = B$Y where A,B,X,Y G V**, A and B are the first columns of P and Q respectively, t £ V{ then P L±JvtQ = {A$X) mvt(B$Y)
j A(X LL\t(B$Y)) if v = l = j ^ , k ^ 2 . , ; : .„ . B((A$X) UJtY) if v = r
(5)
66
If
If
P = X,
Q = \,
and
if
\l±Avt(B$Y)
A$X
B(\LUtY) '
LUvt\
A(X\JJtX)
Al_LJtA
(6)
if if
v =l
if
v=r
if
t =X
otherwise
(7)
(8)
The symbol 0 denotes the empty set. The row shuffle of P with Q on the trajectory vt, v G V2, t G V2*1S defined in a similar way except that I, r are replaced by u, d and $ by 9 . Also if \P\C j= \t\i or \Q\C ^ \t\r then P LUt Q = 0. Similarly, if \P\r + \t\u or \Q\r ^ |t| d then P LUt Q = 0. Definition 29: A parallel contextual array grammar with contexts shuffled on trajectories (PACST) is a construct G = (V,B,Ca,Ra,ipc,(pr,Tc,Tr) where the components are as in Definition 7, Tc is a family of sets T\ C V* of trajectories indexed over the subsets of column array contexts and Tr is a family of sets T2 C V% of trajectories indexed over the subsets of row array contexts. When ipc and ipr are omitted, we call G as a parallel contextual array grammar with contexts shuffled on trajectories without choice. When Tc or Tr consists of exactly one set Ti or T2 respectively corresponding to Ca or Ra, we write Tc = T\ or Tr = TiThe direct derivation with respect to G is a binary relation => on V** and is denned as follows: X =>LUT Y, where X, Y £ V** iff Y = X LUt U , t G Tx for some Ti € T c , where t/ is a parallel shuffled column context. X ^U-J-ZV y, where X, y G V** iff F = X LUt C/ , i G T2 for some T2 G T r , where C/ is a parallel shuffled row context. X ^ U J T Y iff y = X LUTo C7 or y = X LUTr. J7. We denote by =»^j T the reflexive and transitive closure of the relation =^LUT • The language generated by G, denoted by L(G) is defined as follows: L(G) = {Y G F** / there exists X G P such that X = ^ ^ j r y }. The family of all languages generated by PACST is denoted by VACT. We illustrate with an example. (bnanbn)m Example 30: Let V = {a, b} , and L — < (6 3 ") m
/m,n>l
^ (6"a"6") m One can easily see that L is generated by the following bab
PACST,
G — (V, B, Ca,Ra,
are
67
b ab
bb b
b ab
bbb
hb b,
b ab,
b ab,
bbb.
where r\,r2,r3, r^ are Ra — {ri,r2,r3,rA} n n ba ab bb a a bb bb bb bb b a,
ab,
bb,
a a.
n n n
bab
bnbnbn
=
ci,
=
c3,
fc
bnanbn
C2,
= C4, for all n > 1.
bnbnbn (ab),
"(6a))m = rx,
tpr
(bb),
_(ba))m
(ab)
'(bb))m
(aa) m
(bb) ')m Tc
=
bnanbn bnbnbn
bnanbn (bb), )m
bnbnbn
(bb) o))m m\ 2n 2 n = {rl r l
^3,
V
(bb)m
\_(aa)m 771 I n > 1} and
=
r2,
= r4, for all m > 1. T r = {du 2 m d 2 u m / m > 1}
Figure 1 Tokens H where black square is 'a' and white square is 'b'. 5. VACT and some families of Parallel Contextual Array Languages The parallel contextual array grammars with contexts shuffled on trajectories introduced here have more generative power than the parallel external and internal contextual array grammars considered in 6 . Theorem 31: (i) A language L is a parallel external contextual array language if and only if it is generated by a PACST, G = (V, B, Ca,Ra,ipc, <pr,Tc, Tr) such that rml*rn £ Tc for some m, n > 0 and dpu*d9 £ Tr for some p, q > 0. Hence the family of parallel external contextual array languages 6 is strictly contained in the family
68
PACT, the trajectories being regular languages. (ii) A language I is a parallel internal contextual array language if and only if it is generated by a PACST grammar G = (V,B,Ca,Ra,(pc,(pr,Tc,Tr) such that m n p q l*r l*r l* G Tc for some m,n > 0 and u*d u*d u* G Tr for some p, q > 0. Hence the family of parallel internal contextual array languages 6 is strictly contained in the family PACT, the trajectories being regular languages. (Hi) A language L is a parallel (simple) shuffled contextual array language if and only if it is generated by a PACST grammar G = (V,B,Ca,Ra,ipc,(pr,Tc,Tr) such that Tc contains {l,r}* and Tr contains {u,d}*. Hence the family PACS of all parallel shuffled contextual array languages is strictly contained in the family PACT, the trajectories being regular languages. Proof: We prove (%). The other statements are similar. Assume that L is generated by the PEAC grammar G' = (V, B, C, R', (p'c, (p'r). Construct the PACST grammar G = (V, B, Ca,Ra, r,Tc, Tr) as follows: Vi
For each c'- =
G C, the set Ca contains the element
2/i+i Vi _xi+l
Vi+1 _
If
Vi
G V'c
$c
3+i_ c^eC, l
Vi
a
e
i,l
fc
•'
Xi
Hence for each L $r R =
-
a
i,k
2/1 2/2
X2 $c
xi
G
Vh
Xh
column context L $ R =
O-iM
O-i+1,1 • • • Oi+l.fc
j/i
X2 2/2
e tpc{X).
Xh Vh
Let rml*rn G T c , where \L\C = \x, ich r[ — Xi
Xi^,\
Vi 3/t+i
^i *^i+l
%r
and |i?| c = |j/i|c = n. [Vi 2/i+i G R', the set i? a contains the element
If 2R' is defined by a
a
l,i
X{ ^i-j-1
l,»+l
e v»;
Vi Vi+i
a
/i,i a/i,t+i
r< G R', 1 < i < k - 1, for any X = [ay] h x f c G V*; define
a
l,t+l
G VV
_2/i 2/i+i J Hence for each U $ r D = j j i 2 • • • i t yiV2 there exists a parallel shuffled column context
•••
Vk
G *?/(*),
Vi 2/2 ••• 2/fc J Let dpu*dq G T r , where |f/|r = |a;,|r = p and \D\r = |j/j| r = Tr — dpu*dq, where p, q > 0, then we define a PEAC grammar G' = (V,B,C',R',
Xj
Vi 1
Vi+i_
. *+ . X%
Vi
G Ca such that \xi\c = m, \yi\c = n. .
Ci =
_%i+i
Vi+i
Similarly R' contains the element r\ X{
where
$c x
Xi-^-i
Xi Xi+i
$r
G -Ra such that \xi\ = p and |j/i| = q.
Vi Vi+i
If
Oi,fc
Vi G
oc>+1 2/i+i_
i+1,1
C j € C 0 ) 1 < i < h - 1, for any X = [a ij ] /iXfc G V**, define <^ : V** -> 2 C ' such that Xj
9>c _Xi+i_
Vi Vi+l
a
y.
e ^
j/i+i_
i,l
_ai+l,l
If <pr : 7** -> V** is denned by " &l,i Ol.i+1 G (fir
•'
-
a
i,k
• • • Oj+l.A:
where
70 ri £ Ra,
l < i < k - l , for any X = [a»j] hxfc € V**,
define ip'r : V** - * 2R> such t h a t a
l,i
%i ^ i + 1
a
l,»+l
3>r 2/< J/i+i _0*h,i ah,i+l_
Again it is easy t o see t h a t L(G')
= L(G).
•
T h e o r e m 3 2 : T h e family VACT is closed under reflection on t h e base a n d right leg and rotations by 90°, 180°, 270°. 6.
Conclusion
T h e extension of trajectory guided contextual string g r a m m a r s t o picture generation considered here is a first step in this direction. Other properties of this array g r a m m a r model remain t o be examined. T h e power of trajectories, for example, is one such problem worth examining. 3D picture p a t t e r n s have also been investigated in the literature 16 > 17 . Extension of this study t o 3D pictures is also worth examining. References 1. H. Bunke and A.Sanfeliu,(World Scientific, 1990). 2. C. Cook and P.S.P.Wang, Computer Graphics and Image Processing 8(1978) 3. A. Ehrenfeucht, Gh.Paun and G. Rozenberg, In Handbook of Formal Languages, Vol.2. Eds. G. Rozenberg and A. Salomaa, Springer-Verlag, (1997). 4. K. S. Fu, Prentice-Hall, (1982). 5. D. Giammarresi and A. Restivo, In Handbook of Formal Languages Vol.3. Eds. G. Rozenberg and A. Salomaa, Springer Verlag, (1997). 6. P. Helen Chandra, K.G. Subramanian and D.G. Thomas, presented at the Nineth International Workshop on Combinatorial Image Analysis(IWCIA), Palermo, Italy, May (2003). 7. K. Krithivasan, M.S. Balan and R. Rama, In Recent Topics in Mathematical and Computational Linguistics, Eds. C. Martin-Vide and Gh. Paun, (2000). 8. S. Marcus, Rev. Roum. Math. Pures et Appl. 14 , 10 (1969) 9. C. Martin-Vide, A. Mateescu, G. Rozenberg and A. Salomaa , Int. J. Computer Math. 73 (1999) 10. A. Mateescu, G. Rozenberg and A. Salomaa, Theoretical Computer Science, 197 ,1-2 (1998) 11. Gh. Paun , (Kluwer Academic Publishers, Dordrecht, Boston, London, 1997). 12. R. Rama and T A . Smitha, Int. J. Pattern Recognition and Artificial Intelligence 14 (2000) 13. A. Rosenfeld, ( Academic Press, 1979). 14. A. Rosenfeld and R. Siromoney, Languages of design, 1 (1993) 15. P.S.P.Wang, IEEE-PAMI 5,1 (1983) 16. P.S.P. Wang, " IJPRAI8, 2 (1994) 17. P.S.P. Wang, In Handbook of Pattern Recognition and Computer Vision, ed by C.H.Chen, L.F.Pau, and P.S.Wang, (WSP 1999) .
C H A P T E R 1.5
PATTERN RECOGNITION WITH LOCAL INVARIANT FEATURES
C. Schmid , G. Dorko , S. Lazebnik , K. Mikolajczyk
and J. Ponce
1
INRIA Rhone-Alpes, GRAVIR-CNRS 655, av. de I'Europe, 38330 Montbonnot, France Dept. of Computer Science and Beckman Institute University of Illinois, Urbana, IL 61801, USA
Local invariant features have shown to be very successful for recognition. They are robust to occlusion and clutter, distinctive as well as invariant to image transformations. In this chapter recent progress on local invariant features is summarized. It is explained how to extract scale and affine-invariant regions and how to obtain discriminant descriptors for these regions. It is then demonstrated that combining local features with pattern classification techniques allows for texture and category-level object recognition in the presence of varying viewpoints and background clutter.
1. I n t r o d u c t i o n Local photometric invariants have become more and more popular over the past years. This is due to (i) their locality which permits recognition in the presence of occlusion and clutter, (ii) their distinctiveness due to the use of photometric information and (iii) their invariance which makes t h e m stable under image transformations and illumination changes. This is in contrast to classical recognition approaches which were mostly based on global photometric information or geometric features. Color histograms 3 7 and eigenimages 3 8 are examples of recognition methods based on global photometric information. T h e y require segmentation in the presence of clutter and are not robust to occlusions. Furthermore, they are not invariant to image transformations. However, these methods can discriminate due to the use of photometric information and they perform very well in constrained settings. Recognition based on geometric features, on the other hand, does not require segmentation and is invariant to image transformations 3 0 . However, the features are not very discriminating and are sensitive to noise and errors in the extraction process. T h e idea of combining the distinctiveness of photometric information with the locality and invariance of geometric features has led t o t h e development of local pho-
71
72
tometric invariants. An initial solution was presented by Schmid and Mohr 35 . They describe the image with a set of rotation-invariant descriptors computed at automatically extracted interest points. A multi-scale framework makes the description invariant to similarity transformations. This description combined with a voting scheme and neighbourhood constraints allows for excellent recognition results. The approach is, however, limited to similarity transformations and only images of the same object or scene can be recognized. A solution is possible due to the following extensions: (i) the extraction of affine-invariant photometric descriptors and (ii) the use of pattern classification techniques. Section 2 describes the extraction of local invariant features and their application to recognition in the presence of viewpoint changes. Section 3 and Section 4 demonstrate that combining local features with pattern classification techniques allows for texture and category-level object recognition in the presence of varying viewpoints and background clutter. 2. Extraction of local invariant features This section presents the extraction of local invariant features. Sections 2.1 and 2.2 present scale and affine-invariant detectors and in Section 2.3 different descriptors are compared. An application to recognition is presented in Section 2.4. 2.1. Scale-invariant
regions
Most scale-invariant detectors search for maxima in the 3D scale-space representation of an image (x, y and scale). They differ mainly in the differential expression used to build the scale-space representation. Crowley 6 detects local features in an image pyramid. Lindeberg 16 searches for 3D maxima of the scalenormalized Laplacian-of-Gaussian (LoG). The LoG operator detects blob-like structures. Lowe 19 uses the difference-of-Gaussian (DoG) to approximate the LoG. This results in an efficient algorithm for extracting local 3D extrema in scale-space. Kadir and Brady n use local complexity as saliency measure. They compute the entropy of greylevel intensities over a neighborhood and then search for entropy maxima in scale and location. Our approach, the Harris-Laplace detector 21>24, selects a complementary type of regions: corners and regions of "high information content" (cf. Figure 9). The detector first extracts interest points at multiple scale levels which are the local spatial maxima of the scale-adapted Harris function. It then uses the Laplacian for scale selection 16 . Scale selection determines the characteristic scale of a local structure, i.e. the scale with maximum similarity between the feature detection operator and the local image structures. This scale is determined as the extremum over scale of a given function (see Figure 1) and is independent of the image resolution. Note that there might be several extrema, that is several characteristic scales corresponding to different local structures centered at one point. The Laplacian operator is used for scale selection since it gave the best results in the experimental comparison in 21 .
73
(")
(b)
(c)
(d)
Fig. 1. Example of characteristic scales. (a),(c) images with different resolutions; the radius of displayed regions is equal to 3 times the characteristic scale. (b),(d) responses of the normalized LoG over scales for (a),(e). The characteristic scales are 10.1 and 3.89 for (a) and (c), respectively. The ratio of scales corresponds to the scale factor (2.5) between the two images.
Initial multi-scale interest points are rejected if the Laplacian attains no extremum at the scale of extraction or if the Laplacian response is below a given threshold. We then obtain a set of scale-invariant points with associated scales. Figure 2 presents an example of points detected with Harris-Laplace. Images (a) and (b) show points detected with the multi-scale Harris detector; the radius of displayed regions equals 3 times the detection scale. Points corresponding to the same local structure are selected manually. Note that interest points, detected for the same image structure, change their location relative to the detection scale in the gradient direction. Images (c) and (d) show the points selected by the Laplacian. Note that two or more points can be selected, as several local maxima might exist in scale-space. We can see that the location and the scale of the points correspond to the transformation between the images.
(a)
(b)
(c)
(d)
Fig. 2. Scale-invariant interest point detection, (a), (b) Initial multi-scale Harris points corresponding to one local structure (selected manually), (c), (d) Scale-invariant interest points selected with the Harris-Laplace detector.
An experimental evaluation of the Harris-Laplace detector for images of real scenes 2 1 , 2 4 has shown a very good performance up to a scale factor of 4. The performance was measured by the repeatability rate, that is the percentage of points detected at the same relative position and with corresponding regions, as well as by the performance in the context of recognition. 2.2. Affine-invariant
regions
The Harris-Laplace detector is not invariant to significant affine transformations. Figure 3 shows Harris-Laplace regions in black and the corresponding regions pro-
74
jected with the affine transformation to the other image in white. The regions detected with Harris-Laplace do not cover the same part of the affine transformed image. Affine-invariant region extraction is therefore required.
Fig. 3. Scale-invariant interest point detection for an image pair related by an affine transformation. The regions detected with Harris-Laplace in black and the corresponding regions projected with t h e affine transformation to the other image in white.
Alvarez and Morales 2 proposed an affine-invariant algorithm for corner detection which uses affine morphological multi-scale analysis. Their approach can only be applied to perfect corners. Lindeberg and Garding 17 presented a method for finding blob-like affine features. Their approach first extracts maxima of the normalized Laplacian in scale-space and then modifies iteratively the scale and local affine shape of the regions. Local affine shape is determined with the second moment matrix. Baumberg 4 as well as Schaffalitzky and Zisserman 3 3 use the local affine shape estimation to adapt the point neighbourhood of multi-scale Harris points and Harris-Laplace points respectively. Tuytelaars and Van Gool 39 combine Harris points with nearby edges. An affine-invariant parallelogram region is determined by a Harris point and one point on each of two nearby edges. They also proposed a purely intensity-based method. It is initialized with local intensity extrema. For each extremum the algorithm finds significant changes in the intensity profiles along rays going out from the extremum. An ellipse is fitted to the region defined by the locations of these changes. Matas et al. 20 introduced maximally stable extremal regions (MSERs). An extremal region is a connected component of pixels which are all brighter or darker than all pixels on the region's boundary. These regions are invariant to affine transformations as well as to monotonic intensity transformations. Our approach, the Harris-Affine detector 2 2 ' 2 4 extends the Harris-Laplace detector by estimating the local affine shape based on the second moment matrix 17 . The second moment matrix describes the local image structure and is defined by the gradient distribution in a point neighbourhood: Mx ? <xj5 aD) = al g(aM) * [ / j ( ? * " > , L*%<* *"> } [LxLy(x,(TD) L2(X,CJD) J The local derivatives are computed by convolution with Gaussian derivatives of scale |
75
eigenvalues of the second moment matrix determine the affine shape of the point neighbourhood. Affine normalization projects the affine pattern to the one with equal eigenvalues, i.e. uses as transformation the square root of the second moment matrix. See Figure 4 for illustration. The normalized regions are isotropic in terms of the second moment matrix and are related by a simple rotation. Rotation preserves the eigenvalue ratio for an image patch and the affine deformation can therefore be determined up to a rotation factor.
(a)
(b)
(c)
(d)
Fig. 4. Affine normalization with the second moment matrix: (a), (c) initial images with the affine shape matrices, (b), (d) normalized images. The normalized images are related by a rotation.
In practice the second moment matrix as well as the characteristic scale change if the patch is transformed and therefore need to be re-estimated iteratively. Initial points are obtained by the multi-scale Harris detector. For each point we iteratively estimates location, scale and local shape. Figure 5 shows points and regions detected in consecutive steps of the iterative procedure. For this example, the location, scale and shape of the point do not change after four iterations. We stop iterating when the second moment matrix fj, (of the transformed patch) is sufficiently close to a pure rotation. This is measured by the similarity of the eigenvalues Amoa.(/x) and Amiw(p)? i-e. A™*"|»\ > 0-95 for our experiments. Another important point is to stop in the case of divergence. In theory there is a singular case when the eigenvalue ratio tends to infinity i.e. on a step-edge. Therefore, the point should be rejected if the ratio of the eigenvalues is too large (i.e. bigger than 6), otherwise it leads to unstable elongated structures. The convergence properties of the shape adaptation algorithm are extensively studied in 17 . It is shown that except for the singular case the point of convergence is always unique. In general the procedure converges to the correct solution provided that the initial estimate of the affine deformation is sufficiently close to the true deformation, and the scale is correctly selected with respect to the size of the local image structure. Figure 6 presents an example of points detected with Harris-Affine. Images (a) and (b) show the multi-scale Harris points used for initialization, in black the region selected by the Harris-Laplace detector. Images (c) and (d) show the points and regions after an iterative estimation for each point in (a) and (b). Note that the points in (a) and (b) which correspond to the same physical structure, but are detected at different locations due to scale, converge to the same point location and
76
Initial
1
2
3
4
Fig. 5. Iterative detection of an affine-invariant interest point. T h e left image shows the point used for initialization. The consecutive images show the points and regions after iterations 1, 2, 3 and 4.
region. We can see that the method converges correctly even if the location and the scale of the initial point is relatively far from the point of convergence. It is straightforward to identify these points by comparing their location, scale and local shape and to select one of them.
(a)
(b)
(c)
(d)
Fig. 6. Affine-invariant interest point detection : (a), (b) Multi-scale Harris points/regions (in black - Harris-Laplace). The radius of the circles is three times the detection scale. (c),(d) Points and affine regions obtained with the iterative algorithm applied to points in (a) and (b). Note that points representing the same structure converge to the same solution.
An experimental evaluation of the Harris-Affine detector for images of real scenes 2 2 , 2 4 has shown a very good performance up to a viewpoint change of 70 degrees. Harris-Laplace shows a similar performance up to 30 degrees of viewpoint change. For larger viewpoint changes the performance of Harris-Laplace decreases rapidly. 2.3.
Descriptors
To obtain local invariant features, the extracted and normalized regions have to be described. Many different image descriptors have been developed. They should be distinctive and at the same time robust to viewpoint changes as well as to errors of the detector. In the following we have selected a few promising descriptors and compare them in the context of recognition. Note that the normalized regions are not rotation-invariant. To eliminate rotation our comparison uses rotation-invariant descriptors. A simple way to obtain rotation invariance is to orientate the patch in the direction of the dominant gradient.
77
The compared descriptors are presented in the following. SIFT descriptors describe the gradient distribution in a region by a 3D histogram of location and gradient orientation. The quantization of location and gradient orientation makes the descriptor robust to small geometric distortions and small errors in the region extraction. Steerable filters 10 steer derivatives in a particular direction. Steering in direction of the gradient orientation makes them invariant to rotation. Differential invariants 9 are combinations of image derivatives which are rotation-invariant. Complex filters 3 3 differ from the Gaussian derivatives by a linear coordinates change in filter response space. Moment invariants 4 0 characterize the shape and the intensity distribution in a region. Cross-correlation for a sub-sampled image patch is used as a baseline descriptor. Our evaluation criterion is the ROC (receiver operating characteristics) of the detection rate for the query image with respect to the false positive rate in a database of images. Detection rate is the number of correctly matched points with respect to the number of possible matches. False positive rate is the probability of a false match in the database of descriptors, that is the total number of false matches with respect to the product of the number of database points and the number of query image points. The similarity threshold between descriptors is varied to obtain the ROC curves. The results are displayed for a false positive rate up to 0.012, that is each point from the query image matches at most with 1.2% of points in the database. The threshold is usually set below this value; otherwise the number of false matches is too high to permit reliable recognition. Figure 7 compares the performance in the presence of an affine transformation between image pairs for which the viewpoint of the camera is changed by 60 degrees. This introduces a perspective transformation which can be locally approximated by an affine transformation. There are also some scale and brightness changes in the test images. To eliminate the effects of the affine transformation, we use the HarrisAffine detector. The descriptors are computed on point neighborhoods normalized with the locally estimated affine transformations. SIFT descriptors perform better that the other ones and steerable filters come second, but they perform significantly worse than SIFT. Note that SIFT descriptors computed on Harris-Laplace regions perform worse than any of the other descriptors (see HL s i f t ) , as these regions and therefore the descriptors are only scale and not affine-invariant. Cross-correlation obtains the lowest score. This can be explained by the sensitivity of cross-correlation to errors in the point and region extraction. A comparison for image rotation, scale and illumination changes 23 shows similar results as in the case of affine viewpoint changes. In conclusion we observe that the performance varies significantly for different descriptors. SIFT descriptors perform best. This shows the robustness and the distinctiveness of the region-based SIFT descriptor. Second best are steerable filters; they can be considered a good choice given their low dimensionality. Cross-correlation gives unstable results. The performance depends on the accuracy of interest point and region detection, which
78
0
2
4
6
false positive rate
8
to
12
xio"3
Fig. 7. Evaluation for a 60° viewpoint change of the camera. Descriptors are computed for HarrisAffine regions. HL s i f t is the SIFT descriptor computed for Harris-Laplace regions.
decreases for significant geometric transformations. The differential invariants give significantly worse results than the steerable filters, which is surprising as they are based on the same basic components (Gaussian derivatives). The multiplication of derivatives necessary to obtain the rotation invariance increases the instability of the descriptors. Overall, the comparison shows that a robust region-based descriptor performs better than point-wise descriptors.
2.4.
Recognition
Local invariant photometric descriptors are suitable for recognition of the same object or scene in the presence of large viewpoint changes. Several approaches based on local invariants have been developed 19.22>29.33 a nd have shown an excellent performance. They all combine similarity measures of local descriptors with semilocal and global consistency constraints. Our approach 22 extracts affine-invariant regions with the Harris-Affine detector and describes each of them with a set of Gaussian derivatives invariant to rotation and affine illumination changes. The similarity of descriptors is measured by the Mahalanobis distance where the covariance matrix is estimated statistically over a large set of images. A voting algorithm is used to select the most similar images in the database. For each interest point of a query image, its descriptor is compared to the descriptors in the database. If the distance is less than a fixed threshold, a vote is added for the corresponding database image. Note that a point cannot vote several times for the same database image. Votes are then verified by robustly estimating the geometric transformation between image pairs based on RANdom SAmple Consensus (RANSAC) and rejecting inconsistent matches. For our experimental results the transformation is either a homography or a fundamental matrix. A model selection algorithm 12 can be used to automatically decide which transformation is the most appropriate one.
79 Figure 8 illustrates retrieval results from a database with more than 5000 images. The top row displays query images for which the corresponding image in the database (second row) was correctly retrieved. Note the significant transformations between the query images and the images in the database. There is a scale change of a factor of 3 between images of pair (a). Image pairs (b) and (c) show large viewpoint changes. The displayed matches are the inliers to a robustly estimated fundamental matrix or homography between the query image and the database image.
Pig. 8. The top row shows the query images and the bottom row shows the most similar images in the database. The displayed matches are the inliers to a robustly estimated fundamental matrix or homography between the query image and the database image. There are (a) 22 matches, (b) 34 matches and (c) 22 matches. All of them are correct.
3. Recognizing t e x t u r e s We address in this section the problem of representing and recognizing non-rigid textures observed from arbitrary viewpoints. Recent approaches to texture recognition 18,42 perform impressively well on datasets as challenging as the Brodatz database 5 . Unfortunately, these schemes rely on restrictive assumptions about their input (e.g., the texture must be stationary) and are not generally invariant under 2D similarity and affine transformations, much less 3D transformations caused by camera motions and non-rigid deformations of textured surfaces. In addition, most existing approaches to texture analysis use a dense representation where some local image descriptor is computed over a fixed neighborhood of each pixel. Affineinvariant patches can be used to address the issues of spatial selection—Inding a sparse set of texture descriptors at "interesting" image locations—and shape
80
selection—computing shape and scale characteristics of the descriptors—(see 3 2 for related work). In addition, they afford a texture representation that is invariant under any geometric transformation that can be locally approximated by an affine model: Local affine invariants are capable of modeling not only global affine transformations of the image, but also perspective distortions and non-rigid deformations that preserve the locally flat structure of the surface (e.g., the bending of paper or cloth). In this context, it is appropriate to combine the Harris-Affine interest point detector 2 2 with the affine-adapted Laplacian blob detector proposed by Lindeberg and Garding 1T . The two feature detectors are dubbed H (for Harris) and L (for Laplacian), and they provide two description "channels" for local image patterns. Their output on two sample images is shown in Figure 9. Intuitively, the two detectors provide complementary kinds of information: H responds to corners and other regions of "high information content55, while L produces a perceptually plausible decomposition of the image into a set of blob-like primitives.
Fig. 9. Prom left to right: H-detector and L-deteetor output for a building and flower image.
3.1. Image
signatures
This section demonstrates the power of affine-invariant features as image descriptors in texture classification tasks. Our approach applies the following process to each image in the database using both the H and L feature detectors: (1) find the affine-invariant patches; (2) construct an affine-invariant description of these patches; (3) find the most significant clusters of similar descriptions and use them to construct the signature of the image; and (4) compare all pairs of signatures using the Earth Mover's Distance (EMD). Like most approaches to texture analysis, ours relies on clustering to discover a small set of basic primitives in the initial collection of candidate texture elements. We use a standard agglomerative clustering algorithm. The final representation for the image is a signature of the form {(mi, t^i), (m2,ti2) 5 . • •, (rrifc, «*)}, where mi is the medoid (the most centrally located element of the ith cluster) and Ui is the relative weight of the cluster (the size of the cluster divided by the total number of descriptors extracted from the image). Signatures have been introduced by Rubner et al. 31 as representations suitable for matching using the Earth Mover's Distance (EMD). For our application, the signature/EMD framework offers several important
81
advantages. A signature is more descriptive than a histogram, and it does not require global clustering of the descriptors found in all images. In addition, EMD can match signatures of different sizes, and it is not very sensitive to the number of clusters— that is, if one cluster is split into several clusters with similar medoids, the magnitude of the EMD is not greatly affected. This is a very important property, since the automatic selection of the number of clusters remains a largely unsolved problem. Finally, recall that the proposed texture representation is designed to work with multiple channels corresponding to different affine-invariant region detectors (here, the H and L operators). Each channel generates its own signature representation for each image in the database, and therefore its own EMD value for any pair of images. We have experimented with several methods of combining the EMD matrices of the separate channels to arrive at a final estimate of the distance between each pair of images. Empirically, simply adding the distances produces the best results.
\
trrr-- ~-: ~ ~ ' ~ ^ ^
1 3
Y'--:A • 8 0.7
8 0.7
CD
CD
O) 0.6
CD
--- H I .... 1 H+L I
0.5 0.4]
y^^h:
CD
D) 0.6
^
L^-zj^zZr:
0-»
2 | 0.8 'c
Number of retrievals
(a) Spin images
l
/
/
/
•
\
--- H
- | 0.5
1
H + LI
Number of retrievals
(b) Gabor-like filters
II 11 + L Class L 0.89 0.98 0.98 TO 0.71 Tl 0.83 0.76 T2 0.95 0.90 0.87 0.93 1.00 1.00 T3 0.52 0.47 0.43 T4 1.00 0.93 1.00 T5 0.84 0.95 0.97 T6 0.84 0.74 0.71 T7 0.85 0.63 0.89 T8 0.99 0.91 0.98 T9 0.89 0.80 Mean 0.86 (e) Classification rates
Fig. 10. Top: Samples of the ten texture classes used in our experiments. Bottom: Retrieval and classification results 1 4 .
We have implemented the proposed approach and conducted experiments with a dataset consisting of 200 images—20 samples each of ten different textured surfaces. Figure 10(top) shows two sample images of each texture. Significant viewpoint changes and scale differences are featured within each class. Several of the classes include additional sources of variability: inhomogeneities in the texture patterns, nonrigid transformations, illumination changes, and unmodeled viewpoint-dependent appearance changes. Figure 10(bottom, left) shows retrieval results using intensitydomain spin images 14 as local image descriptors. Spin images are two-dimensional
82
histograms encoding the distribution of brightness values. The dimensions are the distance from the center or the origin of the normalized coordinate system of the patch, and the intensity value. Spin images are similar to the SIFT descriptors introduced in the previous section, but avoid estimation of the gradient orientation. Notice that for this dataset, the H channel is more discriminative than the L channel. Adding the EMD estimates provided by the two channels results in improved performance. Figure 10(bottom, center) shows the results obtained using the Gaborlike filters commonly used as image descriptors in texture analysis 3 4 , 4 1 instead of intensity-domain spin images. Figure 10(bottom, right) summarizes the classification results obtained by using five samples from each class as training images. The classification rate for each class provides an indication of the "difficulty" of this class for our representation. The mean classification rate is 89% with two classes achieving 100%, showing the robustness of our system against a large amount of intra-class variability. Performance is very good for the rather inhomogeneous textures T5 and T6, but class T4 is not recognized very well, which is probably explained by the lack of an explicit model for viewpoint-dependent appearance changes caused by non-Lambertian reflectance and fine-scale 3D structure.
3.2. Generative
models
The previous section demonstrated the adequacy of our image descriptors in simple texture classification tasks. Here we go further and introduce generative models for the distribution of these descriptors, along with co-occurrence statistics for nearby patches. At recognition time, initial probabilities computed from the generative model are refined using a relaxation step that incorporates co-occurrence statistics learned at modeling time. In the supervised framework, the training data consists of single-texture sample images from classes with labels C(. The class-conditional densities p(x\C() can be estimated using all the feature vectors extracted from the images belonging to class C(. We model class-conditional densities as p{x\Ci) = Y^m=i v{x\cim) p(ctm), where the components cem are thought of as sub-classes and each p(x\cem) is assumed to be a Gaussian. The EM algorithm is used to estimate the parameters of the mixture model and is initialized with the output of the K-means algorithm. We limit the number of free parameters and control numerical behavior by using spherical Gaussians with covariance matrices of the form S^TO = o\mI. In situations where it is difficult to obtain large amounts of fully labeled examples, training on incompletely labeled or unlabeled data helps to improve classification performance 26 . The EM framework provides a natural way of incorporating unsegmented multi-texture images into the training set. Suppose we are given a multi-texture image annotated with the set C of class indices that it contains— that is, each feature vector x extracted from this image has a label set of the form Cc = {Ce\i € £}• To accommodate label sets, we now use all the data simultaneously to estimate a single mixture model with L x M components. The
83
estimation process starts by selecting some initial values for model parameters. During the expectation or E-step, we use the parameters to compute probabilistic sub-class membership weights given the feature vectors x and the label sets Cc- p(ctm\x,Cc) ex p(x\clm)p(cim\Cc), where p{ctm\Cc) = 0 for all ( g C and ^2iiec li2m=iP(ce™-\C'c) = 1- During the maximization or M-step, we use the computed weights to re-estimate the parameters by maximizing the expected likelihood of the data. At this stage, each region in the training image is assigned the sub-class label that maximizes the posterior probability p(cem\x,Cc)- Next, we need to define a neighborhood for a given region which depends on the size and shape of the region. Here we "grow" the ellipse by adding a constant absolute amount (15 pixels in the implementation) to the major and minor axes. We can now effectively turn the image into a directed graph with arcs emanating from the center of each region to other centers that fall within its neighborhood. The existence of an arc from a region with sub-class label c to another region with label c' is a joint event (c, c') (note that the order is important since the neighborhood relation is not symmetric). For each possible pair of labels, we estimate p(c, c') from the relative frequency of its occurrence, and also find the marginal probabilities p(c) = ^ c , p(c, c') and p(c') = ^ C P ( C ' C ' ) - Finally, we compute the values r(c,c') =
p{c,c')-p{c)p{c') [{p{c)-p{c))(p{c>)-p>{c>))}^
representing the correlations between the events that the labels c and c', respectively, belong to the source and destination nodes of the same arc. The values of r(c, c') must lie between —1 and 1; negative values indicate that c and c' rarely co-occur as labels at endpoints of the same edge, while positive values indicate that they co-occur often. In our experiments, we have found that the values of r(c, c') are reliable only in cases when c and c' are sub-class labels of the same class C. We set r(c, c') to a constant negative value that serves as a "smoothness constraint" in the relaxation algorithm described next, whenever c and c' belong to different classes. We have implemented the probability-based iterative relaxation algorithm described in the classic paper by Rosenfeld et al. 28 to enforce spatial consistency. The initial estimate of the probability that the ith region has label c, denoted p\ '(c), is obtained from the learned Gaussian mixture model as the posterior probability p(c\xi). Note that since we run relaxation on unlabeled test data, these probabilities must be computed for all L x M sub-class labels corresponding to all possible classes. At each iteration, new probability estimates p\ ' (c) are obtained by updating the current values p\ ' (c) using the equation Pi
(c)
pf\c)[l + q^(c)}
,(*>
w=E
Wi
Y,r{c,c')pf{c') . c'
The scalars Wij are weights that indicate how much influence region j exerts on
84 region i. We treat i% as a binary indicator variable that is nonzero if and only if the j t h region belongs to the ith neighborhood. Note that the weights are required to be normalized so that ]P Wij = 1. Excellent results for classification/segmentation are obtained for images of an indoor scene and pictures of wild animals. Our first data set contains seven different textures present in a single indoor scene, see Figure 11. Class labels are assigned automatically to each region. Regions of a class are shown in the corresponding column. The second and third rows of Figure 11 show the improvement between the initial labeling and final labeling after relaxation which takes into account the spatial layout. Our second data set consists of unsegmented images of three kinds of animals: cheetahs, giraffes, and zebras. The training set contains 10 images from each class. To account for the lack of segmentation, we introduce an additional "background" class, and each training image is labeled as containing the appropriate animal and the background. Typical classification examples are shown in Figure 12.
Original image
Brick
Carpet
Chair
Floor 1
Floor 2
Marble
Wood
Fig. 11. Segmentation/classification results. From top to bottom: a sample image of each texture class from an indoor scene; initial labeling for an indoor image vs. the final labeling after relaxationnote the significant improvement-; another successful indoor image segmentation (final labeling). See 1 3 for additional results.
Original image
Cheetah
j^tej
Zebra
P1R
Giraffe
Background
1 &*., .^51$
Fig. 12. Segmentation/classification results (final results after relaxation). Animal image classification examples.
85
4. Recognizing object classes Learning and recognizing object class models from images is one of the most difficult problems in computer vision. The combination of image description and machine learning/pattern classification techniques has recently led to significant progress. Several recent approaches to category-level object recognition use classification techniques to acquire part models from training images, and then train a classifier to recognize objects. This paradigm has been successfully applied to the recognition of cars *, faces 3 6 and human beings 2 5 in complex imagery. However, the image descriptors used in these methods enjoy very limited invariance properties (mostly translational invariance), which severely limits the range of admissible viewpoints that they can handle. This can be avoided by using local invariants as image descriptors 7>8>15«27. The approach is section 4.1 constructs parts from individual local invariants, whereas the one presented in section 4.2 uses set of local invariant features described by their appearance and neighbourhood relations to learn parts. 4.1. Discriminative
local
parts
In this section we demonstrate the power of using invariant local features for building discriminative local parts. The idea is to find groups (clusters) of similar local features, i.e. local parts, and to select among these parts the ones which best discriminate between positive and negative images. Figure 13 shows two discriminative parts for the categories airplane, motorbike, wild cat and person.
Fig. 13. Examples of discriminative parts (clusters) for the categories airplane, motorbike, wild cat and person. Parts are chosen from the 10 most discriminative ones. Each cluster is represented by five of its members. The circles represent the regions extracted by the scale-invariant detector.
The approach is weakly supervised, that is the training images are labeled as positive and negative, but the objects are not labeled in the positive images. The training set is split in a "clustering" set and a "validation" set. Parts are learned based on the "clustering" set and the significance of each part is determined with
86
the "validation" set. The first step of our approach is to extract local invariant features. Here we use Harris-Laplace and Harris-Affine as well as the entropy detector by Kadir and Brady n and describe the regions by the SIFT descriptor. The descriptors of the "clustering" set are then used to learn the individual parts. We estimate the Gaussian mixture model of their distribution and each component of the mixture represents a part (cluster). The EM algorithm is used to estimate the parameters of the mixture model and is initialized with the output of the if-means algorithm. Descriptors are assigned to the components with the maximum posteriori probability. We then select parts with the "validation" set. We first compute probabilities p(Ci\x) for each descriptor of this set, i.e. determine the probability for each part C, (component of the Gaussian mixture model). Each part d is then ranked by the likelihood ratio between the descriptors of the positive images {xj} -note that the individual descriptors are unlabeled- and the negative images {#£}: R = J2jP(^i\x^)IY^kP(Ci\x%). Other criteria can be used, such as mutual information 7 . The final classifier then sets the n highest rank parts as positive and the others as negative. A descriptor is classified as an object descriptor if it is labeled as belonging to one of the top n parts (maximum posteriori probability for that class) where n is a parameter of our approach. Our approach is evaluated in two different ways. We first verify that the positive descriptors lie mostly on the object. Figure 14 shows the results for a few test images. Note that the results are very good. Only a few points are incorrectly classified and they could easily be eliminated by any simple coherence criterion. We have also verified that (as in Section 2.3) SIFT features again give significantly better results than steerable filters. We then evaluate the performance by image classification, that is if the image contains the object or not. This is a standard criterion which allows comparison with existing methods. We report and compare image classification results in table 1. Training and test images are the same as in Fergus et al. 8 and in Opelt et al. 27 . We measure performance with the Receiver Operating Characteristic (ROC) equal error rate. It is defined to be the point on the ROC curve-obtained by varying the number of parts n- where the proportion of true positives is equal to the proportion of true negatives. Classification requires an additional parameter, namely the minimum number p of positive descriptors in the image for which the image is classified as positive. The "best" p is estimated from the validation set. Note that this value doesn't necessary lead to the best classification performance on the test set. Table 1 shows that our model in combination with the two scale-invariant detectors HarrisLaplace and the entropy detector outperforms the other methods.
4.2. Affine-invariant
semi-local
parts
In this section we use characteristic patterns formed by affine-invariant patches and semi-local spatial relations to describe salient object parts. Figure 15 illustrates this
87 n = 10
n = 20
n = 30
Fig. 14. Positive detections with increasing n for different categories and detectors. First and second row: entropy detector. Third row: Harris-Aifine detector.
airplanes faces motorbikes wildcats bikes people
Our model 0.985 0.991 0.995 0.87 0.88 0.88
Fergus et al. 0.902 0.964 0.925 0.900 — —
8
Opelt et al. 0.889 0.935 0.922 — 0.865 0.808
27
Table 1. Image classification performance measured with the equal error rate.
idea with an affine-invariant semi-local part found between face images using the output of the affine-invariant Laplacian and a variant of affine alignment 3 . Note that the part is stable despite large viewpoint variations and appearance changes. Affine-invariant semi-local parts are geometrically stable configurations of multiple affine-invariant regions, found by the affine Laplace detector. These parts are approximately affinely rigid by construction, i.e. the mapping between instances of the same part in two images can be well represented by a 2D affine transformation. Combined with the locality of the parts, this property makes our method suitable for modeling a wide range of 3D transformations, including viewpoint changes and nonrigid deformations. Furthermore, they are more distinctive than individual features
88
Fig. 15.
Matching faces: an affine-invariant semi-local part.
used in the previous section. The mechanism for learning affine-invariant semi-local parts is based on the idea that a direct search for visual correspondence is key to successful recognition. Thus, at training time we seek to identify groups of neighboring affine regions whose appearance and spatial configuration remains stable across multiple instances. To avoid the prohibitive complexity of establishing simultaneous correspondence across the whole training set, we separate the problem into two stages: Parts are initialized by matching pairs of images and then matched against a larger validation set. Even though finding optimal correspondence between features in two images is still intractable, effective sub-optimal solutions can be found using non-exhaustive constrained search. Admiral Swallowtail Machaon Monarch 1 Monarch 2
Peacock
Zebra
Fig. 16. The butterfly dataset. Two samples of each class are shown in each column. We have implemented the approach 15 and conducted experiments for the challenging application to the automated acquisition and recognition of butterfly models in heavily cluttered natural images. Figure 16 shows two samples for the seven classes in our dataset which is composed of 619 butterfly images. Note that we don't use any negative images for modeling. The pictures, which are collected from the Internet, are extremely diverse in terms of size and quality. Motion blur, lack of focus, resampling and compression artifacts are common. This dataset is appropriate for exercising the descriptive power of semi-local affine parts, since the geometry of a butterfly is locally planar for each wing (though not globally planar). In addition, the species identity of a butterfly is determined by a basically stable geometric wing
09
pattern, though appearance can be significantly affected by variations between individuals, lighting, and imaging conditions. It is crucial to point out that butterfly recognition is beyond the capabilities of many current state-of-the-art recognition systems 1 , s .
Fig. 17. Butterfly modeling and detection examples, (a) T h e part with the highest validation score for a class. The part size is listed below each modeling pair, (b) Example of detecting the part from (a) in a single test image. Detected regions are shown in yellow and occluded ones are reprojected from the model in blue. The total number of detected regions (absolute repeatability) and the corresponding repeatability ratio are shown below each image.
The recognition framework is straightforward. Matching and validation are used to identify a fixed-size collection of parts for representing the classes. Candidate parts are formed by matching between eight randomly chosen pairs of training images. Ten verification images per class are used to rank candidate parts according to their repeatability score, and the top ten parts per class are retained for recognition. Figure 17(a) shows the part having the highest repeatability for each of the classes. At testing time, the parts for all classes are detected in each training image. Though multiple instances of the same part may be found, we retain only the single instance with highest number of detected regions. Figure 17 (b) shows examples of part detections in individual test images. The score for a class is defined as the cumulative relative repeatability of all its parts, or the total number of regions detected in all parts of the classes divided by the sum of part sizes. For multi-class classification, each image is assigned to the class having the maximum score. Figure 18(a) shows classification results obtained using the above approach (the average rate is 90.4%), and Figure 18(b) shows how performance is improved by using multiple parts.
90
(a)
Class Admiral Swallowtail Machaon Monarch 1 Monarch 2 Peacock Zebra Total
Test images Correct (rate) P a r t size (0.871) 74 85 179 (12/28) (0.750) 12 16 252 (18/29) (0.965) 55 57 148 (12/21) (0.729) 35 48 289 (14/67) (0.914) 53 58 275 (19/36) (1.000) 108 102 (8/14) 108 (0.892) 58 65 209 (16/31) (0.904) 395 437
0.9 0.88
1086
(b) J I Io.8 Number of parts
Fig. 18. Classification results for the butterflies, (a) The second column shows the size of the model (top 10 parts) for each class (the sum of sizes of individual parts), and the size of the smallest and the largest parts are listed in parentheses, (b) Classification rate vs. number of parts. 5. C o n c l u s i o n a n d d i s c u s s i o n In this chapter we have presented a state of the art on local invariant photometric regions and descriptors. We have shown t h a t the resulting local photometric invariants are very well a d a p t e d to object recognition. Research on invariant regions and their description is now well advanced and these invariant features are building blocks for general recognition systems. Of course, improvements of detectors and descriptors are still possible, for example by developing different types of detectors and different region-based descriptors. T h e remaining open problems are the recognition of a large number of objects and the recognition of object categories. We think t h a t b o t h problems require the use of machine learning, i.e. p a t t e r n classification, techniques. To handle a large number of objects we have to structure the d a t a based on d a t a reduction, clustering and d a t a mining techniques. Recognizing object categories requires the description of intra-class variation, feature selection, a flexible description of t h e spatial structure as well as a hierarchical organization of the categories. Acknowledgments This research was supported by t h e E u r o p e a n Projects VIBES a n d LAVA, by a UIUC-CNRS collaboration agreement, by the UIUC Campus Research Board and by the National Science Foundation grants IRI-990709 and IIS-0308087. We are also grateful to D. Lowe, T. Kadir, F . Schaffalitzky and A. Zisserman for providing the code for their detectors a n d descriptors. Bike and people images have been provided by A. Opelt 2 7 , the airplane, face, motorbike and wild cat images by R. Fergus 8 and the Valbonne images by the INRIA group Robot Vis. References 1. S. Agarwal and D. Roth. Learning a sparse representation for object detection. In European Conference on Computer Vision, vol. IV, pp. 113-127, 2002. 2. L. Alvarez and F. Morales. Affine morphological multiscale analysis of corners and multiple junctions. International Journal of Computer Vision, 2(25):95-107, 1997. 3. R. Basri and S. Ullman. The alignment of objects with smooth surfaces. In International Conference on Computer Vision, pp. 482-488, 1988.
91 4. A. Baumberg. Reliable feature matching across widely separated views. In Computer Vision and Pattern Recognition, vol. I, pp. 774-781, 2000. 5. P. Brodatz. Textures: A Photographic Album for Artists and Designers. Dover Publications, New York, 1966. 6. J.L. Crowley and A.C. Parker. A representation for shape based on peaks and ridges in the difference of low pass transform. Transactions on Pattern Analysis and Machine Intelligence, 6(2):156-170, 1984. 7. G. Dorko and C. Schmid. Selection of scale-invariant parts for object class recognition. In International Conference on Computer Vision, vol. 1, pp. 634-640, 2003. 8. R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning. In Computer Vision and Pattern Recognition, vol. II, pp. 264271, 2003. 9. L.M.T. Florack, B. ter Haar Romeny, J.J Koenderink, and M.A. Viergever. General intensity transformation and differential invariants. Journal of Mathematical Imaging and Vision, 4(2):171-187, 1994. 10. W.T. Freeman and E.H. Adelson. The design and use of steerable filters. Transactions on Pattern Analysis and Machine Intelligence, 13(9):891-906, 1991. 11. T. Kadir and M. Brady. Scale, saliency and image description. International Journal of Computer Vision, 45(2):83-105, 2001. 12. K. Kanatani. Geometric information criterion for model selection. International Journal of Computer Vision, 26(3):171-189, 1998. 13. S. Lazebnik, C. Schmid, and J. Ponce. AfRne-invariant local descriptors and neighborhood statistics for texture recognition. In International Conference on Computer Vision, vol. 1, pp. 649-655, 2003. 14. S. Lazebnik, C. Schmid, and J. Ponce. Sparse texture representation using affineinvariant neighborhoods. In Computer Vision and Pattern Recognition, vol. 2, pp. 319324, 2003. 15. S. Lazebnik, C. Schmid, and J. Ponce. Local affine parts for object recognition. In British Machine Vision Conference, 2004. 16. T. Lindeberg. Feature detection with automatic scale selection. International Journal of Computer Vision, 30(2): 79-116, 1998. 17. T. Lindeberg and J. Garding. Shape-adapted smoothing in estimation of 3D shape cues from affine deformations of local 2D brightness structure. Image and Vision Computing, 15(6):415-434, 1997. 18. F. Liu and W. Picard. Periodicity, directionality, and randomness: Wold features for image modeling and retrieval. Transactions on Pattern Analysis and Machine Intelligence, 18(7):722-733, 1996. 19. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-110, 2004. 20. J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. In British Machine Vision Conference, pp. 384393, 2002. 21. K. Mikolajczyk and C. Schmid. Indexing based on scale invariant interest points. In International Conference on Computer Vision, vol. 1, pp. 525-531, 2001. 22. K. Mikolajczyk and C. Schmid. An affine invariant interest point detector. In European Conference on Computer Vision, vol. I, pp. 128-142, 2002. 23. K. Mikolajczyk and C. Schmid. A performance evaluation of local descriptors. In Computer Vision and Pattern Recognition, vol. 2, pp. 257-263, 2003. 24. K. Mikolajczyk and C. Schmid. Scale and affine invariant interest point detectors. International Journal of Computer Vision, 60(l):63-86, 2004.
92 25. K. Mikolajczyk, C. Schmid, and A. Zisserman. Human detection based on a probabilistic assembly of robust part detectors. In European Conference on Computer Vision, vol. I, pp. 69-81, 2004. 26. K. Nigam, A. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2-3):103-134, 2000. 27. A. Opelt, M. Fussenegger, A. Pinz, and P. Auer. Weak hypotheses and boosting for generic object detection and recognition. In European Conference on Computer Vision, vol. II, pp. 71-84, 2004. 28. A. Rosenfeld, R. A. Hummel, and S.W. Zucker. Scene labeling by relaxation operations. Transactions on Systems, Man and Cybernetics, 6:420-433, 1976. 29. F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce. 3D object modeling and recognition using affine-invariant patches and multi-view spatial constraints. In Computer Vision and Pattern Recognition, vol. 2, pp. 272-277, 2003. 30. C. Rothwell, A. Zisserman, D. Forsyth, and J. Mundy. Canonical frames for planar object recognition. In European Conference on Computer Vision, pp. 757-772, 1992. 31. Y. Rubner, C. Tomasi, and L.J. Guibas. A metric for distributions with applications to image databases. In International Conference on Computer Vision, pp. 59-66, 1998. 32. F. Schaffalitzky and A. Zisserman. Viewpoint invariant texture matching and wide baseline stereo. In International Conference on Computer Vision, vol. II, pp. 636-643, 2001. 33. F. Schaffalitzky and A. Zisserman. Multi-view matching for unordered image sets. In European Conference on Computer Vision, vol. I, pp. 414-431, 2002. 34. C. Schmid. Weakly supervised learning of visual models and its application to contentbased retrieval. International Journal of Computer Vision, 56(1):7-16, 2003. 35. C. Schmid and R. Mohr. Local grayvalue invariants for image retrieval. Transactions on Pattern Analysis and Machine Intelligence, 19(5):530-534, 1997. 36. H. Schneiderman and T. Kanade. A statistical method for 3D object detection applied to faces and cars. In Computer Vision and Pattern Recognition, vol. I, pp. 746-751, 2000. 37. M.J. Swain and D.H. Ballard. Color indexing. International Journal of Computer Vision, 7(l):ll-32, 1991. 38. M. Turk and A. Pentland. Face recognition using eigenfaces. In Computer Vision and Pattern Recognition, pp. 586-591, 1991. 39. T. Tuytelaars and L. Van Gool. Matching widely separated views based on affine invariant regions. International Journal of Computer Vision, 59(l):61-85, 2004. 40. L. Van Gool, T. Moons, and D. Ungureanu. Affine / photometric invariants for planar intensity patterns. In European Conference on Computer Vision, pp. 642-651, 1996. 41. M. Varma and A. Zisserman. Classifying images of materials: Achieving viewpoint and illumination indepence. In European Conference on Computer Vision, vol. Ill, pp. 255-271, 2002. 42. K. Xu, B. Georgescu, D. Comaniciu, and P. Meer. Performance analysis in contentbased retrieval with textures. In International Conference on Pattern Recognition, vol. IV, pp. 275-278, 2000.
CHAPTER 2.1 CASE-BASED REASONING FOR IMAGE ANALYSIS AND INTERPRETATION
Petra Perner Institute of Computer Vision and Applied Computer Science, Kornerstr. 10, 04107 Leipzig, Germany, [email protected] http://www.ibai-research.de The development of image interpretation systems is concerned with tricky problems such as a limited number of observations, environmental influence, and noise. Recent systems lack robustness, accuracy, and flexibility. The introduction of case-based reasoning (CBR) strategies can help to overcome these drawbacks. The special type of information {i.e. images) and the problems mentioned above provide special requirements for CBR strategies. In this chapter we review what has been achieved so far and research topics concerned with case-based image analysis and interpretation.
1.
Introduction
Image interpretation systems are becoming increasingly popular in medical and industrial applications. The existing statistical and knowledge-based techniques lack robustness, accuracy, and flexibility. New strategies are necessary that can adapt to changing environmental conditions, user needs and process requirements. Introducing case-based reasoning (CBR) strategies into image interpretation systems can satisfy these requirements. CBR provides a flexible and powerful method for controlling the image processing process in all phases of an image interpretation system to derive information of the highest possible quality. Beyond this CBR offers different learning capabilities, for all phases of an image interpretation system, that satisfy different needs during the development process of an image interpretation system. Therefore, they are especially appropriate for image interpretation. Although all this has been demonstrated in various applications1"6, case-based image interpretation systems are still not well established in the computer vision community. One reason might be that CBR is not very well known within this community. In this paper, we will show that a CBR framework can be used to overcome the modeling 95
96
burden usually associated with the development of image interpretation systems. We will review current activities on image interpretation. In Sec. 2, we will introduce the tasks involved when interpreting an image. In Sec. 3, we will describe the special needs of an image interpretation system and how they are related to CBR topics. Then, we will review in Sec. 4 activities for image analysis and interpretation in the CBR community. In Sec. 5, we describe the case-based reasoning process. The case description is given in Sec. 6. We will overview what kinds of similarity measures are useful and what are the open research topics in Sec. 7. In Sec. 8 we will show how a case base can be organized to speed up the retrieval process. The different learning strategies of a CBR system are described in Sec. 9. The problems concerned with the competences of case bases is discussed in Sec. 10. How cases can be acquired is presented in Sec. 11. Finally, we offer conclusions in Sec. 12 based on our CBR systems working in real-world environments.
2.
Tasks an Image Interpretation System Must Solve
Image interpretation is the process of mapping the numerical representation of an image into a logical representation such as suitable for scene description. An image interpretation system must be able to extract symbolic features from the pixels of an image (e.g., irregular structure inside the nodule, area of calcification, and sharp margin). This is a complex process; the image passes through several general processing steps until the final symbolic description is obtained. These include image preprocessing, image segmentation, image analysis, and image interpretation (see Fig. 1). Interdisciplinary knowledge from image processing, syntactical and statistical pattern recognition, and artificial intelligence is required to build such systems. The primitive (low-level) image features will be extracted at the lowest level of an image interpretation system. Therefore, the image matrix acquired by the image acquisition component must first undergo image pre-processing to remove noise, restore distortions, undergo smoothing, and sharpen object contours. In the next step, objects of interest are distinguished from background and uninteresting objects, which are removed from the image matrix. In the x-ray computed tomography (CT) image shown in Fig. 1, the skull and the head shell is removed from the image in a preprocessing step. Afterwards, the resulting image is partitioned into objects such as brain and liquor. After having found the objects of interest in an image, we can then describe the objects using primitive image features. Depending on the particular objects and focus of interest, these features can be lines, edges, ribbon, etc. A geometric object such as a block will be described, for example, by lines and edges. The objects in the ultrasonic image shown in Fig. 1 are described by regions and their spatial relation to each other. The region's features could include size,
97
shape, or the gray level. Typically, these low-level features have to be mapped to highlevel features. A symbolic feature such as fuzzy margin will be a function of several lowlevel features. Lines and edges will be grouped together by perceptual criteria such as collinearity and continuity in order to describe a block.
Defect is a Crack of Type A on the bottom of the industrial part Image Interpretation
Class is a Crack
Object Class
r
Image Classification
J
Object 1 (Greylevel=black, Size=little, sfiape=round), Left_of(Object1, Object2)
Symbolic Features Grouping or Mapping to High- J level Features
Object 1 (Greylevel=0, Size=2000, shape=0,9) ? Extraction of Low-level Feature mature I
Binary Image Image Segmentation
2D image
*&&&&,$&& lmQ#
Image Pre-Processing
2D Image
:
;
Image Acquisition
i
i#Bi* Sami mage
Fig. 1. Architecture of an Image Interpretation System Image classification is usually referred to as the mapping of numeric features to predefined classes. Sometimes image interpretation requires only image classification. However, image classification is frequently only a first step of image interpretation. Lowlevel features or part of the object description are used to classify the object into different object classes in order to reduce the complexity of the search space. The image
98
interpretation component identifies an object by finding the object that it belongs to (among the models of the object class). This is done by matching the symbolic description of the object in the scene to the model of the object stored in the knowledge base. When processing an image using an image interpretation system, an image's content is transformed into multiple representations that reflect different abstraction levels. This incrementally removes unnecessary detail from the image. The highest abstraction level will be reached after grouping the image's features. It is a product of mapping the image pixels contained in the image matrix into a logical structure. This higher level representation ensures that the image interpretation process will not be affected by noise appearing during image acquisition, and it also provides an understanding of the image's content. A bottom-up control structure is shown for the generic system in Fig. 1. This control structure allows no feedback to preceding processing components if the result of the outcome of the current component is unsatisfactory. A mixture of bottom-up and top-down control would allow the outcome of a component to be refined by returning to previous component.
3.
Development Concerns of an Image Interpretation System
Several factors influence the quality of the final result of an image interpretation system, including environmental conditions, the selected imaging device, noise, the number of observations from the task domain, and the chosen part of the task domain. These cannot often all be accounted for during system development, and many of them will only be discovered during system execution. Furthermore, the task domain cannot even be guaranteed to be limited. For example, in defect classification for industrial tasks, new defects may occur because the manufacturing tool that had been used for a long period suddenly causes scratches on the surface of the manufactured part. In optical character recognition, imaging defects (e.g., heavy print, light print, or stray marks) can occur and influence the recognition results. It is not yet possible to observe all real-world influences, nor provide a sufficiently large enough sample set for system development and testing. A robust image interpretation system must be able to deal with such influences. It must have intelligent strategies on all levels of an image interpretation system that can adapt the processing components to these new requirements. A strategy that seems to satisfy these requirements could be case-based reasoning. CBR does not rely on a wellformulated domain theory, which is, as we have seen, often difficult to acquire. This suggests that we must consider different aspects during system development that are frequently studied CBR issues. Because we expect users will discover new aspects of the environment and the objects during system usage, an automatic image interpretation system should be able to incrementally update the system's model, as illustrated in Fig. 2.
99 This requires knowledge maintenance and learning. The designated lifetime of a case also plays an important role. Other aspects are concerned with system competence. The range of target problems that a given system or algorithm can solve are often not quite clear to the developer of the image interpretation system. Often researchers present to the community a new algorithm that can, for example, recognize the shape of an object in a particular image and then claim that they have developed a model. Unfortunately, all too often another researcher inputs a different image to the same algorithm and finds that it fails. Did the first researcher develop a model or did they instead develop a function? Testing and evaluation of algorithms and systems is an important problem in computer vision, as is designing the algorithm's control structure so that it fits best to the current problem. CBR strategies can help to solve this problem in computer vision.
Heal world-environment
Imaging Device Observations 2D/3D Image Data
Model A ^ Refinement
Model Development Model B
And so on
Fig. 2. Model Development Process
4.
Image Analysis and Interpretation in the Case-Based Reasoning
Image processing is a challenging field. The unique data (images) and the necessary computation techniques require extraordinary case-representations, similarity measures and CBR strategies to be utilised. Perner7 proposes a system that uses CBR to optimize image segmentation at the low level unit according to changing image acquisition conditions and image quality. The intermediate-level unit extracts the case representation used by the high-level unit
100 employed to dynamically adapt image interpretation. The system works on different case representations such as a graph-based representation for the cases of the high-level image description and the raw image matrix for the low-level image representation. Therefore the system uses different CBR strategies for the reasoning and learning part one is based on structural similarity and the other is based on the digital image distance. Grimnes and Aamodt2 present a system that integrates CBR into a task-oriented model-based system for the interpretation of abdominal CT images. A case-based reasoner working on a segment case-base contains the individual image segments. These cases with labels are considered indexes for another case-based reasoner working on an organ interpretation case-base. There system is based on a propose-critique-modify learning cycle. Jarmulak' presents a system for ultra-sonic B-scans that are one-dimensional signals. He presents a tree-based retrieval strategy. Micarelli et al4 applies CBR to scene recognition. They calculated image properties from images and stored them into a case base. They used the Wavelet transform because it is scale-independent, but this limits their similarity measure to consider only object rotation. The application of CBR to image segmentation for CT images from the brain is described in Ref. 3. Different learning strategies in a hierarchy of structural cases are presented in Ref. 7 and Ref. 8. Learning case representation and improving the system performance by controlling the similarity measure is described in Ref. 9. The application of CBR image interpretation to health monitoring and biotechnology is described in Ref. 10. A new challenging application field are geographical information systems. This kind of application requires special spatial problem solving techniques and spatial similarity measures. The first application of CBR to geographic information systems was reported by Holt and Benwell". Their approach consists of combining case-based reasoning with geographical information systems to form a hybrid system to solve spatial problems in soil classification. Carswell et al. '2 described in their paper a spatial similarity measure for comparing the location objects in different images. A completely different application of CBR to image processing has been described by Ficet-Cauchard et a/.3. They apply CBR for the development of the image processing steps of formerly unknown image processing problems by using past experiences and plan adaption. Case-based object recognition for fungi identification and the developed similarity measures are described in Ref. 13. A system for case acquisition of visual forms and case mining has been developed in Ref. 14. Finally, in Perner'3 is built a bridge between the work in CBR and the one in dissimilarity classification which became recently important in pattern recognition.
101
5.
Case-Based Reasoning
Rule-based systems, model-based systems or decision trees are difficult to utilize in domains where generalized knowledge is lacking. However, often there is a need for an interpretation system, even though there is not enough generalized knowledge. Such a system should a) solve problems using the already stored knowledge and b) capture new knowledge making it immediately available to solve the next problem. To accomplish these tasks, case based reasoning is useful. Case-based reasoning explicitly uses past cases from the domain expert's successful or failing experiences. Therefore case-based reasoning can be seen as a method for problem solving as well as a method to capture new experiences. It can be seen as a learning and knowledgediscovery approach since it can capture from new experiences some general knowledge, such as case classes, prototypes and some higher level concepts. To point out the differences between a CBR learning system and a symbolic learning system, which represents a learned concept explicitly, e.g. by formulas, rules or decision trees, we follow the notion of Wess et al.I6: A case-based reasoning system describes a concept C implicitly by a pair (CB, sim). The relationship between the case base CB and the measure sim used for classification may be characterized by the equation: C = CB + sim.
(1)
This equation indicates in analogy to arithmetic that it is possible to represent a given concept C in multiple ways, i.e. there exist many pairs C = (CBt, sirrij). [CB,, sim,), .... [CBn simj for the same concept C . Furthermore the equation gives a hint how a case-based learner can improve its classification ability. There are three possibilities to improve a case-based system. The system can: • • •
store new cases in the case base CB, change the measure of similarity sim, or change CB and sim.
During the learning phase a case-based system gets a sequence of cases X ,.X2 X, with X,• = (x,, class(x,)) and builds a sequence of pairs C = {CBr sim/), [CB,, sim2), .... [CB,, simt) with CB, c{X,.X, X,}. The aim is to get in the limit a pair C = (CBn. simn), that needs no further change, i. e. 3« V/w > n (CBn. simn) = (CBm. simm), because it is a correct classifier for the target concept C . 5.1. The case-based reasoning
process
At the highest level of generality a general CBR cycle may be described by four tasks: Retrieve the most similar case or cases, reuse the information and knowledge in that case
102
to solve the problem, revise the proposed solution, and retain the parts of this experience likely to be useful for future problem solving. The process is graphically represented in Fig. 3. The current problem is described by some keywords, attributes or any abstraction that allows describing the basic properties of a case. In pattern recognition and image interpretation this can be attributes describing the basic properties of an object or an attributed graph describing a scene. Based on this case description a set of close cases is retrieved from the case base (retrieve task). The closest case is evaluated and the solutions associated to the closest case are given as output to the user. The problem solution associated to the current case is applied to the current problem and the result of the test is observed and evaluated by a teacher (revise task). When the user is not satisfied with the result, or no similar case could be found in the case base, then case-base maintenance (retain tasks) starts. The revise and retain task can be viewed as a supervised learning task which is necessary to improve the performance of the system.
(Case\—•
Failure/Success istration
Problem Description
Case Base Management
Case Evaluation
Case Base
Fig. 3: Case-Based Reasoning Process Retrieval can be based on an index. The index can be a structure, such as for example a classifier or any hierarchical organization of the case base. This index is built and updated during the retain task. 5.2. CBR
maintenance
CBR maintenance will operate on new cases as well as on cases already stored in the case base. If a new case has to be stored into the case base, it means that there is no similar case in the case base. The system has recognized a gap in the case base. A new case has
103 to be incorporated into the case base in order to close this gap. From the new case a predetermined case description has to be extracted, which should be formatted into the predefined case format. Afterwards the case can be stored into the case base. Selective case registration means that no redundant cases will be stored into the case base and that the case will be stored at the right place depending on the chosen organization of the case base. Similar cases will be grouped together or generalized by a case that applies to a wider range of problems. Generalization and selective case registration ensure that the case base will not grow too large and that the system can find similar cases fast. It might also happen that too many cases would be retrieved from the case base that are not applicable to the current problem. Then it might be wise to rethink the case description, or to adapt the similarity measure. For the case description, more distinguishing attributes should be found that allow sorting out cases that do not apply to the current problem. The weights in the similarity measure might be updated in order to retrieve only a small set of similar cases. CBR maintenance is a complex process and works over all knowledge containers (vocabulary, similarity, retrieval, case base) of a CBR system. Consequently, architectures and systems have been developed which support this process8 17 ' l8 . 5.3. Design
consideration
The main problems concerned with the development of a CBR system are: • • • • •
What is the right case description? What is an appropriate similarity measure for the problem? How to organize a large number of cases for efficient retrieval? How to acquire and refine a new case for entry in the case base? How to generalize specific cases to a case that is applicable to a wide range of situations?
6.
Case Description
There are different opinions about the formal description of a case. Each system utilizes a different representation of a case. Formally we like to understand the following definition for a case: Definition 1: A case F is a triple [P.E.L) with a problem description P, an explanation of the solution E and a problem solutions L.
104 For image related tasks we have two main different types of information that make up a case consisting of image-related information and non-image-related information. Image-related information could be the ID, 2D or 3D images of the desired application. Non-image-related information could be information about the image acquisition, such as the type and parameters of the sensor, and information about the objects or the illumination of the scene. It depends on the type of application what type of information should be taken into consideration for the interpretation of the image. In case of the CBR based image segmentation system for medical CT image segmentation described in Ref. 3 we used patient-specific parameters, such as age and sex, slice thickness and number of slices. Jarmulak1 took into consideration the type of sensor for the railway inspection application. Based on this information the system controls the type of case base that the system is using during reasoning. How the 2D or 3D image matrix is represented depends on the purpose and not seldom on the developer's point of view. In principle it is possible to represent an image by one of various abstraction levels. An image may be described by the pixel matrix itself or by parts of this matrix (pixel-based representation). It may be described by the objects contained in the image and their features (feature-based representation). Furthermore it may be described by a more complex model of the image scene comprised of objects and their features as well as the spatial relation between the objects (attributed graph representation or semantic networks). Grimnes and Aamodt" developed a model-based image interpretation system for the interpretation of abdominal CT images. The image content is represented by a semantic network where concepts can be general, special cases or heuristic rules. Not well understood parts of the model are expressed by cases and can be revised during the usage of the system by the learning component. The combination of the partially wellunderstood model with cases helps them to overcome the usual burden of modeling. The learning component is based on failure driven learning and case integration. Non-image information is also stored such as sex, age, earlier diagnosis, social condition etc. Micarelli et al4 have also calculated image properties from their images and stored them into the case base. They use the Wavelet transform, since it is scale-independent. By doing so they only take into consideration the rotation of the objects in their similarity measure. In all this work, CBR is only used for the high-level unit. We have studied different approaches for the different processing stages of an image interpretation system. For the image-segmentation unitJ we studied two approaches: 1. a pixel-based approach and 2. a feature-based approach that described the statistical properties of an image. Our results show that the pixel-based approach can give better results for the purpose of image segmentation. For the high-level approach of an ultra-sonic image interpretation system, we used an attributed graph-based representation .
105 However, if we do not store the image matrix itself as a case, but store the representation of a higher-level abstraction instead, we will lose some information. An abstraction means that we have to make a decision between necessary and unnecessary details of an image. It might happen that not having seen all objects at the same time, we might think that one detail is not of interest, since our decision is only based on a limited number of objects. This can cause problems later on. Therefore, to keep the images themselves is always preferable, but needs a lot of storage capacity. The different possible types of representation require different types of similarity measures. 7.
Similarity
An important point in case-based reasoning is the determination of similarity between a case A and a case B. We need an evaluation function that gives us a measure for similarity between two cases. This evaluation function reduces each case from its case description to a numerical similarity measure sim. These similarity measures show the relation to other cases in the case base. The concept of similarity is important for many tasks, therefore we will briefly review the basic properties of similarity and the design criteria for similarity. 7.1. Formalization of similarity The problem with similarity is that it has no meaning unless one specifies the kind of similarity. Smith19 distinguishes five different kinds of similarity: • • •
Overall similarity Similarity Identity
• •
Partial similarity and Partial identity
Overall similarity is a global relation that includes all other similarity relations. All colloquial similarity statements are subsumed here. Similarity and identity are relations that consider all properties of objects at once, no single part is left unconsidered. A red ball and a red ball are similar, a red ball and a red car might be dissimilar. The holistic relation's similarity and identity are different in the degree of the similarity. Identity describes objects that are not significantly different. All red balls are identical and, therefore, they are also similar. Similarity contains identity and is more general concept. Partial similarity and partial identity compare the significant parts of objects. One aspect or attribute can be marked. Partial similarity and partial identity are different with
106
respect to the degree of similarity. A red ball and a pink cube are partially similar, but a red ball and a red cube are partially identical. The described similarity relations are in connection with many respects. Identity and similarity are unspecified relations between whole objects. Partial identity and similarity are relations between single properties of objects. Identity and similarity are equivalence relations. That means they are reflexive, symmetrical, and transitive. For partial identity and similarity these relations do not hold. From identity follows similarity and partial identity. From that follows partial similarity and general similarity. It seems advisable to require from a similarity measure the reflexivity, which means an object is similar to itself. Symmetry should be another property of similarity. Let us consider the statements "A is similar to 2?" or "A is the same as B". We notice that these statements are directed and that the roles of A and B can not be exchanged. People say: "A circle is like an ellipse." but not "An ellipse is like a circle." or "The son looks like the father." but not "The father looks like to the son.". Therefore, symmetry is not necessarily a basic property of similarity. However, in the above examples it can be useful to define the similarity relation to be symmetrical. The transitivity relation also must not necessarily hold. Let us consider the block world: a red ball and a red cube might be similar; a red cube and a blue square are similar; but a red ball and a blue square are dissimilar. However, a concrete similarity relation might be transitive. Similarity and identity are two concepts that strongly depend on the context. The context defines the essential attributes of the objects that are taken into consideration when similarity is determined. An object "red ball" may be similar to an object "red chair" because of the color "red". However the objects "ball" and "chair" are dissimilar. These attributes may be relevant, depending on whether they are given priority or saliency in the considered problem. This little example shows that the calculation of similarity between the attributes must be meaningful. It makes no sense to compare two attributes that do not make a contribution to the considered similarity. Since attributes can be numerical and categorical or a combination of both, we need to pay attention to this by the selection of the similarity measure. Not all similarity measures can be used for categorical attributes or can deal at the same time with numerical and categorical attributes. 7.2. Similarity measures for images Images can be rotated, translated, different in scale, or may have different contrast and energy, but they might be considered as similar. In contrast to that two images may be dissimilar since the object in one image is rotated by 180 degrees. The concept of
107
invariance in image interpretation is closely related to that of similarity. A good similarity measure should take this into consideration. The classical similarity measures do not allow this. Usually the images or the features have to be pre-processed in order to be adapted to the scale, orientation or shift. This process is a further processing step which is expensive and needs some a priori information which is not always given. Filters such as matched filters, linear filters, Fourier or Wavelet filters are especially useful for invariance under translation and rotation which has also been shown by Micarelli et al*. There has been a lot of work done to develop such filters for image interpretation in the past. The best way to achieve scale invariance from an image is by means of invariant moments, which can also be invariant under rotation and other distortions. Some additional invariance can be obtained by normalization (reduces the influence of energy). Depending on the image representation we can divide similarity measures into: • • •
pixel (Iconic)-matrix based similarity measures, feature-based similarity measures, (numerical or symbolical or mixed type) and, structural similarity measures.
Since a CBR image interpretation system has also to take into account non-image information, such as about the environment or the objects etc, we need similarity measures which can combine non-image and image information. We have shown a first approach to this in Ref. 3. To better understand the concept of similarity, systematic studies on the different kinds of image similarity have to be done. Zamperoni et. al20 studied how pixel-matrix based similarity measures behave under different real world influences such as translation, noise (spikes, salt and pepper noise), different contrast and so on. Image feature-based similarity measures have been studied from a broader perspective by Santini and Jain21. We have studied structural similarity in our work in Ref. 8. A study on matched filters and similarity has been done in Ref. 13. Otherwise, every new conference on pattern recognition contains proposals for new similarity measures for specific purposes and different kinds of image representation.
8.
Organization of a Case Base
Cases can be organized into a flat case base or in a hierarchical fashion. In a flat organization we have to calculate the similarity between the problem case and each case in memory. It is clear that this will take time, even if the case base is very large. Systems with a flat case base organization usually run on a parallel machine to perform retrieval in a reasonable time and do not allow the case base to grow over a predefined limit.
108 Maintenance is done by partitioning the case base into case clusters and by controlling the number and size of these clusters22. To speed up the retrieval process, a more sophisticated organization of the case base is necessary. This organization should allow separating the set of similar cases from those cases not similar to the recent problem at the earliest stage of the retrieval process. Therefore we need to find a relation p that allows us to order our case base. Definition 2: A binary relation p on a set CB is called a partial order on CB if it is reflexive, antisymmetric, and transitive. In this case the pair (CB, p) is called a partial ordered set or poset. The relation can be chosen depending on the application. One common approach is to order the case base based on the similarity value. The set of case can be reduced by the similarity measure to a set of similarity values. The relation < over these similarity values gives us a partial order over these cases. The derived hierarchy consists of nodes and edges. Each node in this hierarchy contains a set of cases that do not exceed a specified similarity value. The edges show the similarity relation between the nodes. The relation between two successor nodes can be expressed as follows: Lemma 1: Let z be a node and x and y are two successor nodes of z, then x subsumes z and y subsumes z. By tracing down the hierarchy, the space gets smaller and smaller, until finally a node will not have any successor. This node will contain a set of close cases. Among these cases is to be found the closest case to the query case. Although we still have to carry out matching, the number of matches will have decreased through the hierarchical ordering. The nodes can be represented by the prototypes of the set of cases assigned to the node. When classifying a query through the hierarchy the query is only matched with the prototype. Depending on the outcome of the matching process, the query branches right or left of the node. Such kind of hierarchy can be created by hierarchical or conceptual clustering8, k-d trees2"1 and decision trees'. There are also set-membership-based organizations known, such as semantic nets2 and object-oriented representations24.
9.
Learning in a CBR System
CBR management is closely related to learning. It aims to improve the performance of the system. Let X be a set of cases collected in a case base CB. The relation between
109 each case in the case base can be expressed by the similarity value sim. The case base can be partitioned into n case classes: n
C:CB = \JC, /=/
(2)
such that the intra-case class similarity is high and the inter-case class similarity is low. The set of cases in each class C can be represented by a representative who generally describes the cluster. This representative can be the prototype, the mediod, or an a priori selected case. Whereas the prototype implies that the representative is the mean of the cluster which can easily be calculated from numerical data. The mediod is the case whose sum of all distances to all other cases in a cluster is minimal. The relation between the different case classes C can be expressed by higher-order constructs expressed e.g. as super classes which give us a hierarchical structure over the case base. There are different learning strategies that can take place in a CBR system: •
• • •
Learning takes place when a new case x has to be stored into the case base such that: CBn+i = CBn u {x}. That means that the case base is incrementally updated according to the new case. It may incrementally learn the case classes and/or the prototypes representing the class.8 The relationship between the different cases or case classes may be updated according the new case classes.7 The system may learn the similarity measure. ' '
9.1. Learning new cases and forgetting old cases Learning new cases means just adding cases into the case base upon some notification. Closely related to case adding is case deletion or forgetting cases which have shown low utility. This should control the size of the case base. There are approaches that keep the size of the case base constant and delete cases that have not shown good utility within a fixed time window'. The failure rate is used as utility criterion. Given a period of observation of N cases, if the CBR component exhibits M failures in such a period, we define the failure rate as fr=M/N. Other approaches try to estimate the "coverage" of each case in memory and by using this estimate to guide the case memory revision process26. The adaptability to the dynamic of the changing environment that requires storing new cases in spite of the case base limit is addressed in Ref. 22. Based on intra class similarity it is decided whether a case is to be removed from or to be stored in a cluster. Case deletion in a pre-determined time window based on failure results-' might not be
110 appropriate for image interpretation Because a failure might mean that, instead of the retrieved case being erroneous, there is some relevant knowledge that we could not describe using features. Also, cases that occur infrequently (i.e., that have not been used recently) should be recognized by the system. The causes for case deletion or addition might differ from other CBR applications since images may be distorted and very noisy it might not be useful to store distorted representations. Determining which representation is distorted is sometimes not easy even if you have seen only a few images, and it is usually necessary to have domain knowledge that must also be built up over time. Imprecise or noisy measurements can be caused by some defects of illumination, the image acquisition device, or the object itself. If the image analysis cannot adapt to these measurements, or the reasoning process cannot handle it, then this might cause failure results. However, if this is a systematic event then it might be worthwhile to store the recent case in the case base. The last fact comes from the real world environment. It is not possible to determine all real world influences a priori. Thus, developers prefer to incorporate cases into a case base instead of forgetting them. Although case bases can grow very large, instead of forgetting cases, we would rather subdivide the case base into frequently vs. rarely used cases. This requires addressing the issue of how should the addition of cases into one of these two case bases be controlled, as well as their respective reasoning processes. 9.2. Learning of prototypes Learning of prototypes has been described in Ref. 8 for a flat organization of a case base and for a hierarchical representation of a case base in Ref. 7. The prototype or the representative of a case class is the most general representation of a case class. A class of cases is a set of cases sharing similar properties. The set of cases does not exceed a boundary for the intra class dissimilarity. Cases that are on the boundary of this hyperball have a maximal dissimilarity value. A prototype can be selected a priori by the domain user. This approach is preferable when the domain expert knows for sure the properties of the prototype. The prototype can be calculated by averaging over all cases in a case class or the median of the cases is chosen. If only a few cases are available in a class and subsequently new cases are stored in the class, then it is preferable to incrementally update the prototype according to the new cases.
111 9.3. Learning of higher-order
constructs
The ordering of the different case classes gives an understanding of how these case classes are related to each other. For two case classes which are connected by an edge similarity relation holds. Case classes that are located at a higher position in the hierarchy apply to a wider range of problems than those located near the leaves of the hierarchy. By learning how these case classes are related to each other, higher-order constructs are learnt.8 9.4. Learning of similarity By introducing feature weights we can put special emphasis on some features for the similarity calculation. It is possible to introduce local and global feature weights. A feature weight for a specific attribute is called local feature weight. A feature weight that averages over all local feature weights for a case is called global feature weight. This can improve the accuracy of the CBR system. By updating these feature weights we can learn similarity.27
10. Competence of Case Bases An important problem in image interpretation concerns system competence. We follow the definition in Ref. 26 and define the competence of a system as the range of target problems the system can solve. As we have already pointed out in Sec. 3 it is often not clear to the system developer what problems the desired algorithm can solve. So we have to find a way to describe the competence of a system. This differs from what is usually understood about this problem in CBR. Competence is described based on statistical properties such as case-base size, density and distribution, or group coverage and group density. But what if some groups overlap? Smyth and McKenna26 argue that these groups have shared competence and can be linked together in some way. However, we can also view it as having a poor description of the target problem. Based on this description we may retrieve a similar case but its solution application to the query image may be low in quality. By investigating the failure we may learn that we did not consider a property of the environment or maybe we could not specify it because it was not contained in the description of the target problem. Therefore, the system performance decreases. The measures described in Ref. 26 and Ref. 28 only view competence based on the coverage of the problem space. How do we know that cases in group 1 and group 2 belong to the same target problem group? Proximity in problem space does not imply that they belong to the same problem group; misclassifications can
112 occur because the patterns overlap. We argue that system competence must also account for the misclassification of the target problem based on the problem description. 11. Case Acquisition A CBR system for image classification needs to have some particular features with respect to images. These features result from: special requirements of visual knowledge acquisition (image-language problem) and the need to transform an image's numerical data into a symbolic description. The main problem with images and their translation into a language is that the knowledge about an image is usually tacit. To make this knowledge explicit is often hard. Sometimes the meaning of a word does not correspond to the correct meaning of the image. Therefore, it is necessary to support the operator in an efficient way. Most case-based image interpretation systems do not pay attention to this problem. The only functionality these systems provide is visualization of the image or the processed image. Usually, new case knowledge is obtained via manual acquisition with the expert. This is a time-consuming and sometimes boring process for both the system developer and the expert. A CBR system for image interpretation should have a special case acquisition tool. By using a questioning strategy and evaluating the answers given by the expert, the expert or operator is forced to specify the right knowledge for image interpretation. The questioning strategy is designed to force an expert to explain what distinguishes one object from another and to specify the right property for the object. A special case acquisition tool for image segmentation was described in Ref. 3. With the help of this tool the user can control the parameters of the image segmentation algorithm. Simultaneously, he can view the segmented image and, if he is satisfied with the segmentation quality, he can store the parameters of the image segmentation algorithm together with the case description in the case base. Acquiring cases by tracing image objects and learning more compact cases from these examples by case mining is described in Ref. 14.
12. Conclusions We surveyed special topics associated with a case-based image interpretation system. From our point of view case-based image interpretation differs in many aspects from other CBR applications that require further investigation. First, more systematic work on special image similarity measures is needed that investigates the measures under different influences that may occur in an image. Next, case representations are required for all the different abstraction levels of an image. Finally, the maintenance and learning strategies
113 must be defined so that they can help to improve the system performance and discover the range of target problems that the system can solve. We have deployed two CBR image interpretation systems. One is installed at the university hospital in Halle; it is used for image segmentation to determine the brain/liquor ratio of the head in a CT image. The second system is used to interpret ultrasonic SAFT images. In both applications the CBR strategies we used achieved good system performance that satisfied the users and outperformed other systems. The learning and maintenance facilities installed to date have been particularly well-received. We are recently going on to develop a new CBR based architecture for fungi identification.' 3 In summary, we believe that investigations of case-based image interpretation systems can reveal special challenges to both the CBR and computer vision communities, and encourage more people to work on this topic.
References 1.
J. Jarmulak. Case-based classification of ultrasonic b-scans: case-base organisation and case retrieval. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, pages 100-111. Springer Verlag. Berlin. 1998. 2. M. Grimnes and A. Aamodt. A two layer case-based reasoning architecture for medical image understanding. In I. Smith and B. Faltings (Eds.). Advances in Case-Based Reasoning, pages 164-178. Springer Verlag. Berlin. 1996. 3. P. Perner. An architecture for a CBR image segmentation system. In Journal of Engineering Application in Artificial Intelligence, vol. 12 (6). pages 749-759. 1999. 4. A. Micarelli. A. Neri and G. Sansonetti. A case-based approach to image recognition. In E. Blanzieri and L. Portinale (Eds.). Advances in Case-Based Reasoning, pages 443-454. Springer Verlag. Berlin, 2000. 5. V. Ficet-Cauchard. C. Porquet. and M. Revenu. CBR for the reuse of image processing knowledge: A recursive retrieval/adaption strategy. In K.-D. Althoff. R. Bergmann. and L.K. Branting (Eds.). Case-Based Reasoning Research and Development, pages 438-453. Springer Verlag, ^Berlin. 1999. 6. W. Cheetham and J. Graf. Case-based reasoning in color matching. In D.B. Leake and E. Plaza (Eds.). Case-Based Reasoning Research and Development, pages 1-12. Springer Verlag. Berlin. 1997. 7. P. Perner. Using CBR learning for the low-level and high-level unit of a image interpretation system. In Sameer Singh (Eds.). Advances in Pattern Recognition, pages 45-54. Springer Verlag. Berlin. 1998. 8. P. Perner. Incremental learning of retrieval knowledge in a case-based reasoning system. In K.D. Ashley and D.G. Bridge (Eds.). Case-Based Reasoning - Research and Development. Inai 2689. pages 422-436. Springer. Verlag Berlin. 2003. 9. P. Perner. H. Perner and B. Miiller. Similarity guided learning of the case description and improvement of the system performance in an image classification system. In S. Craw and A.Preece (Eds.). Advances in Case-Based Reasoning. Inai 2416. pages 604-612. Springer Verlag. Berlin. 2002. 10. P. Perner. Th. Guenther and H. Perner. Airborne fungi identification by case-based reasoning. In L. McGinty (Eds.). 5th Intern. Conference on Case-Based Reasoning, Workshop Proceedings. Case-Based Reasoning in Health Science, pages 73-79. 2003.
114 11. A. Holt and G.L. Benwell. Applying case-based reasoning techniques in GIS. In Journal of Geograpical Information Science 13(1), pages 9-25. 1999. 12. J.D. Carswell. D.C. Wilson and M. Bertolotto. Digital image similarity for geo-spatial knowledge management. In Proceedings of 6th European Conference on Case-Based Reasoning. Lecture Notes in Artificial Intelligence (ECCBR2002). Springer Verlag, Berlin 2002. 13. P. Perner and A. Buhring. Case-based object recognition. European Conference on CaseBased Reasoning ECCBR 2004. Springer Verlag. to be published. 14. P. Perner and S. Janichen. Learning of form models from exemplars. SSPR 2004. Springer Verlag 2004. to be published. 15. P. Perner. Are case-based reasoning and dissimilarity-based classification two sides of the same coin? In Journal Engineering Applications of Artificial Intelligence 15 (3). pages 205216. 16. S. Wess and C. Globig. Case-based and symbolic classification. In S. Wess. K.-D. Althoff and M.M. Richter M.M. (Eds.). Topics in Case-Based Reasoning, pages 77-91. Springer Verlag. Berlin. 1994. 17. F. Heister and W. Wilke. An architecture for maintaining case-based reasoning systems. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 221232. Springer Verlag. Berlin. 1998. 18. J.L. Arcos and E. Plaza. A reflective architecture for integrated memory-based learning and reasoning. In S. Wess. K.-D. Althoff and M.M. Richter (Eds.), Topics in Case-based Reasoning, pages 289-300. Springer Verlag. Berlin. 1993. 19. L.B. Smith. From global similarities to kinds of similarities: the construction of dimensions in development. In S. Vosniadou and A. Ortony (Eds.). Similarity and Analogical Reasoning. Cambridge University Press. 1989. 20. P. Zamperoni and V. Starovoitov. How dissimilar are two gray-scale images. In Proceedings of 17. DAGM Symposium, pages 448-455. Springer Verlag. Berlin. 1995. 21. S. Santini and R. Jain. Similarity measures. In IEEE Trans, on Pattern Analysis and Machine Intelligence 21(9). pages 871-883. 1999. 22. J. Surma and J. Tyburcy. A study on competence-preserving case replacing strategies in casebased reasoning. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning. lnai 1488. pages 233-238. Springer Verlag. Berlin. 1998. 23. S. Wess. K.-D. Althoff. and G. Derwand. Using k-d trees to improve the retrieval step in casebased reasoning. In S. Wess. K.-D. Althoff. and M.M. Richter (Eds.). Topics in Case-based Reasoning, pages 167-182. Springer Verlag. Berlin. 1993. 24. R. Bergmann and A. Stahl. Similarity measures for object-oriented case representations. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 2536. Springer Verlag. Berlin. 1998. 25. L. Portinale. P. Torasso. and P. Tavano. Speed-Up. quality and competence in multi-modal case-based reasoning. In: K.-D. Althoff. R. Bergmann and L. K. Branting (Eds.). Case-Based Reasoning Research and Development, lnai 1650. pages 303-317. Springer Verlag. Berlin. 1999. 26. B. Smyth and E. McKenna. Modelling the competence of case-bases. In B. Smyth and P. Cunningham (Eds.). Advances in Case-Based Reasoning, lnai 1488. pages 208-220. Springer Verlag. "Berlin. 1998. 27. D. Wettscherek. D.W. Aha and T. Mohri. A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms. In Artifical Intelligence Review (also available on the Web from http://www.aic.nrl.navy.mil/~aha). 28. D.B. Leake and D.C. Wilson. Remembering why to remember: performance-guided case-base maintenance. In E. Blanzieri and L. Portinale (Eds.). Advances in Case-Based Reasoning, pages 161-172. Springer Verlag. Berlin. 2000.
C H A P T E R 2.2 MULTIPLE IMAGE GEOMETRY - A PROJECTIVE
VIEWPOINT
Quang-Tuan Luong AIC - SRI International 333 Ravenswood Ave, Menlo Park, CA 94025, USA A central problem in computer vision is to reconstruct a 3D scene given multiple images of it. In this chapter, we describe a general geometric framework based on projective geometry. Prom a theoretical point of view, it leads to a simpler and more unified description of the problem than the traditional Euclidean framework. Prom a practical point of view, it makes it possible to use cameras without knowing beforehand their internal parameters. After a brief overview of projective geometry and its application to camera modeling, we examine how to compute the epipolar geometry and planar correspondences of a pair of uncalibrated views from point matches (Fundamental matrix and homographies). We then move to the problem of computing the 3D positions of the points that generate these matches, as well as the cameras ( stratification of reconstruction in projective, affine and Euclidean stages). We eventually tackle the problem of computing "on the fly" the internal calibration of a camera from a sequence of ordinary images (self-calibration). 1. P r o j e c t i v e g e o m e t r y Projective geometry is an extension of Euclidean geometry, which describes a larger class of transformations t h a n just rotations and translations, including in particular the perspective projection performed by a camera. This makes it the n a t u r a l framework to study the relationship between 3D points and their projections in images. 1.1. Projective
coordinates
T h e space of (n + l)-tuples of coordinates, with t h e rule t h a t proportional (n + 1)tuples represent t h e same point, is called t h e projective space of dimension n and denoted P n . This rule is introduced so t h a t there is a one-to-one correspondence between "usual" n Euclidean coordinates and n + 1 projective coordinates. T h e object space is P 3 and t h e image space is P 2 , called the projective plane. Given coordinates in IR" we build projective coordinates by: [ E l , . . . ,Xnf
-)• [X!,...
115
,Xn,l]T.
116
To transform a point in the projective coordinates back into Euclidean coordinates, we just divide by the last coordinate and then drop it: [xi,...
,xn,xn+1]
-] T
-> [Xn+l
%n+l
Projective coordinates represent naturally the operation performed by a camera. The projective space contains more points than the Euclidean space. Points with coordinates [xi,... ,xn,xn+i]T with xn+1 ^ 0 can be viewed as the usual points, whereas the points with coordinates [xi,... ,a; n ,0] T have no Euclidean equivalent. If we consider them as the limit of [2:1,... , xn, X]T, when X —>• 0 i.e. the limit of [xi/X,... ,xn/X, 1] T , then we see that they are the limit of a point of M" going to infinity in the direction [xi,... ,xn]T, hence the appellation point at infinity. The projective space P n can be viewed as the union of the usual space K" (points [xi,... , xn, 1]T) and the set of points at infinity [x\,... , xn, 0] T . 1.2. The projective
plane
The equation of a line I, au + bv + c = 0 can be written using projective coordinates m = [x, y, z]T of the point m: l T m = m T l = ax + by + cz = 0. The projective representation of I is a vector with three coordinates defined up to a scale factor, exactly like a 2-D point: 1 = [a, b, c]T. The line containing the two points m and m' (their join) is expressed as the cross-product of their representations: yz' - zy' 1 ~ m x m' = _xy'
-yx'
Other properties of lines in 2-D projective space are summarized in the table below: points coordinates of m m= [x,y,z] incidence m £ I m l = 0 line obtained by join m x m' points at infinity moo = [a^J/iO]
lines coordinates of I incidence I 3 m point obtained by meet line at infinity
l=[a,6,c]T lTm = 0 lxl' loo = [ 0 , 0 , l ] T
Points and lines have the same representation (results concerning points can be transposed to lines by duality), and the cross-product describes both meet and join. Being a linear operator, the cross-product can be written as a matrix product v x x = [v] x x where [v] x is the skew-symmetric matrix whose left and right nullspaces are the vector v: 0 —V3
[v] x =
vz~
«3 0 - « i • _ — v2 vi 0 _
(1)
If the lines / and /' are parallel, then the previous formula is still valid and gives a point whose coordinates are found to be proportional to [6, —a, 0] T . This is
117 a point at infinity which represents the direction of /; there is exactly one point at infinity for each direction. We note that all the points at infinity belong to the line of equation [0,0,1] T , which is called the line at infinity of P 2 , and denoted l^. 1.3. The projective
space
A plane of P 3 is represented by a vector with four coordinates defined up to a scale factor, exactly like a 3-D point: II = [7ri,7T2,7T3,7r4]T represents the projective equation of the plane -K\X + -K-^Y + ir3Z + m = 0, which means that a point M = [X, Y, Z, 1]T belongs to a plane II if and only if I I T M = 0. In P 3 , the points [X, Y, Z, 0] T therefore form a plane of equation [0,0,0,1] T , called the plane at infinity, and denoted IIoo. This plane represents the directions of the all the other planes. When a point in ffi3 tends to the point at infinity M<x,, for example a point on a line receding towards the horizon, its projection tends to the projection of Moo which is usually a finite point of P 2 called a vanishing point. 1.4.
Transformations
Between projective geometry and Euclidean geometry there are two other geometries, similarity and affine. Each geometry is a characterized by the objects that are left invariant by its transformations. Each geometry is a subset of the next, so they form a hierarchy where more general transformations mean weaker invariants. See also the table at the end of Sec. 4.3.
X
projective
X
X X X X
X
affine X X X
similarity
X X X
Euclidean
X X X X
Transformations rotation, translation isotropic scaling scaling along axes, shear perspective projections Geometric Invariants distance angles, ratios of distances parallelism, center of mass incidence, cross-ratio
X X
X
P r o j e c t i v e t r a n s f o r m a t i o n s . A homography, or projective transformation, is any transformation H of P™ which is linear in projective coordinates and invertible (thus it conserves the space globally, a fact we denote: i?(P") = P"). These properties are equivalent to the fact that collinearity is preserved and subspaces are mapped into subspaces of the same dimension. A homography can be described by an (n + 1) x (n 4-1) non-singular matrix H, such that the image of x is x': x' ~ Hx. n
In P we need two sets of n -1- 2 points such that no n + 1 of them are linearly dependent to define a homography. Each such set is called a projective basis, and it
118
corresponds to the choice of a projective coordinate system. In this chapter, we will consider only P 2 (see Sec. 2.2 for an example discussed in detail) and P 3 . Afflne transformations and the plane at infinity. An affine transformation of P 3 is a particular projective transformation P 3 which preserves the plane at infinity IIoo. A transformation A conserves IIoo if, and only if the last row of the matrix of A is of the form [0,0,0, [i], with /j, ^ 0. Since this matrix is defined only up to a scale factor, it is fully described by its first 3 x 3 sub-matrix M and the 3 first coordinates of the last column vector b:
A=
Mb Ojl
which yields the classical description of a transformation of the affine space R3: x' = M x -I- b . We have seen that points at infinity represent directions. An affine transformation therefore preserves parallelism: parallel subspaces are transformed into subspaces which are still parallel. Other properties are that depth ordering and the ratios of distances of three aligned points are also preserved. Similarity transformations and the absolute conic. In projective geometry all the second order loci (ellipses, parabolas, hyperbolas) lose their distinction and are called conies, with an equation of the form m T Q m = 0, where Q is a symmetric matrix. Using the duality of points and lines, a conic can be considered not only as a locus of points, but also as a locus of lines, the set of lines which are tangent to the conic. For a given conic of matrix Q, its dual conic (the set of its tangents) has matrix the adjoint matrix of Q: Q* = det(Q)CT 1
(2)
In the projective plane P 2 , just like two lines always intersect provided that we add to the affine points the line at infinity, so do two circles, provided that we add to the affine points two complex points in the line at infinity. Indeed, we encourage the reader to verify the surprising fact that all the circles contain the two particular points I — [1,2,0] and J — [1,— i,0], called circular points, which satisfy x2 + y2 = z = 0. Similarly, the Euclidean space P 3 is characterized by the absolute conic ft, which is the set of points [X, Y, Z, T]T satisfying X2 + Y2 + Z2 = 0 and T = 0. In other words Q is the conic in IIoo of matrix I 3 . It has no real points. Like the plane at infinity characterizes affine transformations, the absolute conic characterizes similarity transformations: a projective transformation is a similarity transformation if and only if it leaves the absolute conic Cl invariant. Just like affine transformations are particular projective transformations, similarity transformations are particular affine transformations, since the absolute conic is a subset of the plane at infinity. Similarity transformations are affine transformations for which the first 3 x 3 sub-matrix satisfies M M T — SI3.
119 2. Modeling a single c a m e r a 2.1. The projection
matrix
The transformation from the three-dimensional space to the two-dimensional plane performed by a camera is described using the pinhole model (Fig. 2.1). The projection m of a point of the space M is the intersection of the optical ray (C, M) with the image plane H. The optical axis is the line going through the optical center C and perpendicular to the image plane. It pierces that plane at the principal point c. If we consider an orthonormal system of coordinates in the image plane, centered at c, we can define a three-dimensional orthonormal system of coordinates, called the camera coordinate system, centered at the optical center C with two axes of coordinates parallel to the image ones and the third one parallel to the optical axis. The focal length is the distance between the point C and the plane 11. We choose here the focal length as the unit in the world coordinate system.
(b) Fig. 1. (a) The pinhole model expressed in the camera coordinate system where the world coordinate system is aligned with the camera and the image coordinate system, (b) The way the five intrinsic parameters of Equation 3 affect the pixel image coordinate system and the image of the absolute conic in the imaginary plane.
In these two systems of coordinates, the relationship between t h e coordinates of M, [X, Y,Z]T, and those of its projection m , [u,v] T , is given by: u = ^,v = ^ . Using projective coordinates, this can be written linearly instead:
ri oo oi
~ x~
y _z .
=
0100 .0 0 1 0 .
'X
y
z
= 7>oM
•<
To
The usual coordinates are related to projective coordinates by: u — x/z,v = y/z and X = X/T, Y = y/T, Z = Z/T. The points T = 0 are points at infinity (in space), obtained by the intersection of parallel lines. They are mapped correctly by the projection matrix producing, in general, a real vanishing point. In general, the image coordinate system is defined by the pixels, and the world coordinate system is not aligned with the camera: A general projection matrix is
120 uniquely decomposed as:
T—
au 7 wo 0 av vo To . 0 0 1.
R t
oi"i
= A[Rt],
(3)
where • A describes the characteristics of the camera. As a 3 x 3 matrix it represents a change of image coordinate system. Its five entries are called the camera intrinsic parameters. au and av describe the total magnification of the imaging system resulting from both optics and image sampling. Their ratio, called the aspect ratio, is usually fixed, but is not always equal to 1 due to the digitalization phase. (uo,vo) represents the coordinates of the principal point, which usually are not (0,0) because we count pixels from a corner. The parameter 7, called the skew, is zero except for some very particular imaging situations. • 2? describes the location and orientation of the camera with respect to the world coordinate system. It is a 4 x 4 displacement matrix describing the change of world coordinate system as a rotation R and a translation t (the pose of the camera), called the extrinsic parameters. A general projection matrix, being 3 x 4 , depends on eleven parameters (twelve minus a scale factor), which is the number of the intrinsic and extrinsic parameters combined. In the camera coordinate system (the particular coordinate system defined at the beginning of Sec. 2), the projective coordinates of a pixel represent a 3D point on its optical ray, and therefore give us the position of this optical ray in space in the coordinate system of the camera. In general it is not sufficient to measure pixels in order to infer from a pixel m the position of the optical ray in space. The matrix A is used to transform pixel coordinates into camera coordinates. A camera for which A is known is said to be calibrated. It then acts as a metric measurement device, able to measure the angle between optical rays. The classical way to calibrate a camera is by determining its projection matrix using known control points in 3D, however model-based calibration is not always possible.
2.2. The plane-to-image transformation
homography
as a
projective
We introduce an important example of projective transformation, the homography between a plane II in space and the image plane. If we choose the world coordinate system so that the first two axes define the plane ( Fig. 2.2a), the projection of points of II can be viewed as a transformation
121
Fig. 2. (a) The homography between a plane in space and the image plane. The world coordinate system is aligned with plane II. (b) The first submatrix P of the projection matrix represents a homography between the plane at infinity and the image plane.
between two spaces P 2 , since for those points
' x' y ~v _z _
'X
Vu'Pu'Pi*']
y
V21 V22 V24
o .r
V31V32V3i\
rxi
y LTJ
H
where the Vij are the entries of the 3 x 4 projection matrix V. Each point correspondence (m,p) yields two independent proportionality equations that we can write as a function of entries hij of the 3 x 3 matrix H: hnX + h12y + h13T x h3iX + h32y + h33T z which can be linearized in the entries of f huzX + h12zy + hi3zT \ h2izX + h22zy + h23zT -
y ' z H: h3ixX h31yX
h21X + h22y + h23T h31X + h32y + /i 3 3 T" - ^ z ^ - h33xT = 0, - h32yy - h33yT = 0.
H has eight entries (nine minus a scale factor), therefore from four correspondences m, p in general position, H is determined uniquely by solving a linear system of equations. Once H is computed, we can use it to determine positions of points on II from a single image. This simple example illustrates the power of projective geometry: the mapping between the two planes is done using just linear operations and four reference points, without the need to refer to more complicated representations like rotations, translations and camera parameters. 2.3. Affine
description
of the
projection
Let us partition the projection matrix "p into the concatenation of a 3 x 3 submatrix P and a 3 x 1 vector p . The origin of the world coordinate system, the point [0,0,0,1], is projected onto p .
122 The optical center is also decomposed by separating its last coordinate from the first three:
C
P = [Pp],
Since the optical center is the only point for which the projection is not defined, it is determined by the equation "PC = 0. Using the decomposition just introduced, P C = —cp. Therefore, if det(P) ^ 0, then the solution is given by a point in the affine space
-p~V
(4)
Any projection matrix arising from a physical system has to satisfy det(P) ^ 0, since the optical center has to lie in the affine space (we refer to that as perspective projection, as opposed to parallel projection, obtained for det(P) = 0 for which the optical center lies in the plane at infinity, i.e. c = 0). This is what we will assume in the rest of this chapter. At this affine level of description, we can introduce directions of optical rays. Since the projection of each point at infinity [d T , 0] T is the vanishing point v = P d , P can be considered as the homography between the plane at infinity Iloo and the image plane 1Z. Note that parallel lines have the same direction, hence the same point at infinity, thus their projection is a set of lines of 72. which contains the vanishing point projection of this point at infinity. The optical ray corresponding to the pixel m thus has the direction P _ 1 m (Fig. 2.2b). The table below summarizes the descriptions of the projection matrix in the perspective projection case: level projective
decomposition V
interpretation projection from object space P to image plane V:
n. affine
Euclidean
[PP]
A[Rt]
P:
homography between plane at infinity Iloo and
P:
1Z. projection of the origin of the world coordinate system.
A: (R,t):
change of coordinates in H (5 intrinsic parameters). camera pose in world coordinates.
3. P r o j e c t i v e g e o m e t r y of two views In this section, we describe the geometry of a pair of uncalibrated cameras.
123
3.1. The homography
between two images
of a plane
We first examine the particular situation when the 3-D points lie on a plane II. Planes are important entities: in practice because they appear naturally in many scenes and in theory because they are subspaces which have the same dimension as the images. As we have seen in Sec. 2.2, there is a homography between the image plane of the first camera and the plane II and also a homography between the image plane of the second camera and the plane II; therefore by composition there is a homography H between the two image planes called a planar homography (Fig. 3.1a) because it is induced by the plane II and described by a 3 x 3 matrix H. If m and m' are projections of a point M which belongs to II, then m' ~ Hm. Reversing the roles of the two images transforms H into its inverse. As in Sec. 2.2, H can be determined in general from 4 correspondences. Once H is known, for any projection m of a point of II, it is possible to predict the position of its correspondent in the other image.
(a)
(b)
Fig. 3. (a) The planar homography between two images is obtained by composition of single view nomographics, (b) Epipolar geometry. Given m, the point ml has to lie on its the epipolar line l'm. Given a valid correspondence m -B- m ' , the intersection of the optical rays Lm and Lmi is not empty, and they are coplanar with the baseline CC •
An important special case occurs when II is the plane at infinity n,^. Then, its homography, Hoc (called the infinity homography) has a particularly simple expression in terms of the two projection matrices "P and V1, obtained using the decomposition in Sec. 2.3: H^-P'P-1.
(5)
When the two optical centers are identical, i.e. when the two images are taken from the same viewpoint with the camera rotated, the correspondence between any points (not just points at infinity) is described by HQO-
124 8.2. The epipolar
constraint
between corresponding
points
When the points in space and the two cameras are in general position, it is not possible to predict the position of the correspondent m' of a point m, because this position depends on the depth of the 3-D point M along the optical ray. However, geometrically, this position is not arbitrary: M has to lie along the optical ray of m, and therefore m1 is necessarily located on the projection of that optical ray in the second camera. This line is called the epipolar line of the point m in the second image (Fig. 3.1b). If we are able to compute this line, then when looking for the correspondent of m, we need not search the whole second image, but only this line, hence reducing the search space from 2-D to 1-D. There is another way to view the same construction, by considering it as a way to constrain the cameras rather than the correspondence. Assuming that we know a valid correspondence, m mf depends on a total of four parameters, there must be an algebraic constraint linking the coordinates of m and m'. 3.3. The Fundamental
matrix
The relationship between the point m and its epipolar line l'm in the second image is projective linear, since the optical ray of m is a linear function of m, and projection is also linear. Therefore, there is a 3 x 3 matrix which describes this correspondence, called the Fundamental matrix, giving the epipolar line of the point m: Vm = F m . If two points m and ml are in correspondence, then the point m1 belongs to the epipolar line of m, therefore they satisfy the epipolar constraint: m / T F m = 0,
(6)
which is bilinear in the coordinates of the image points. Reversing the roles of the two images transforms F into its transpose. Fig. 3.3 shows two images and a few epipolar lines.
Fig. 4. Two images with a few corresponding points and epipolar lines.
The Fundamental matrix depends only on the configuration of the cameras (in-
125 trinsic parameters, position and orientation) and not on the 3-D points in the scene. In the generic case where we do not assume any spatial relationship between the points in space, the only information available comes from projective correspondences, the correspondence of points undergoing linear projection. The epipolar constraint fully describes the correspondence of a pair of generic corresponding points in each image, the ambiguity along the epipolar line being caused by the ambiguity along the optical ray of the projection operation. The fundamental matrix describes all the epipolar constraints, so it encodes all the information available from projective correspondences. Since all the optical rays contain the optical center C of the first camera, all the epipolar lines contain the projection of C in the second image (the point where the first camera is seen by the second camera), called the epipole (Fig. 3.1b): e T F m = 0 for any m, and therefore e T F = 0, or equivalently, F T e ' = 0. By reversing the role of the two images, Fe = 0. We conclude that F is a matrix of rank two. Since it satisfies the algebraic constraint det(F) = 0 and is only defined up to a scale factor (like all projective quantities), F depends on seven parameters.
3.4. Computing
the Fundamental
matrix
Each point correspondence (m,m') yields one equation (Eq. 6), therefore with a sufficient number of point correspondences we can determine F . No knowledge about the cameras or scene structure is necessary. The first step for all the algorithms that we discuss in this chapter is the computation of the Fundamental matrix. Eq. 6 is linear in the entries of F. It can be rewritten as U T f = 0, where m = [u, v, 1]T and m ' = [u',v',l]T, and U = [uu1, vu', u', uv',vv', v1', u, v, 1] T , f = [Fll,Fi2,Fi3,F21,F22,F23,F3i,F32,F33]
.
Combining the rows U for each correspondence provides a linear system of the form Uf = 0. Using seven points, it is possible to compute F using the rank constraint det(F) = 0, however because this constraint is cubic there can be three solutions. With eight correspondences in general position, there is a unique solution which is obtained linearly. In practice, we have more than eight correspondences, but they are not exact, so we can seek a least-squares solution: min||Uf||
subject to ||f || = 1.
(7)
The constraint ||f|| = 1 is necessary because F is defined up to a scale factor. In general, more sophisticated methods based on non-linear optimization will yield results that are more precise, however, the simple approach just described can give acceptable results if care is taken in renormalizing the pixel coordinates to the interval [—1,1] to improve the numerical conditioning of matrix U.
126
3.5. Planar
homographies
and the Fundamental
matrix
There is an important relation between the planar homography and the Fundamental matrix. If m is a point of the first image, then its optical ray intersects the plane IT at Mn- H m represents the projection of Mu into the second image. Since by construction the point Mn belongs to the optical ray of m, the point H m belongs to the epipolar line of m. Therefore, given H, it is sufficient to know the epipole e' to determine the Fundamental matrix: the epipolar line l'm contains H m and the epipole e', therefore, l'm — e' x H m . Since by definition of F , Vm = F m , using the operator in Eq. 1, we conclude that F~[e']xH
(8)
Applying both sides of Equation 8 to the vector e, and using the fact that Fe = 0 and [e'] x e' = 0 shows that H e ~ e'; therefore, once the Fundamental matrix is known, the correspondence of three points is sufficient to define a planar homography. The decomposition (8) of the Fundamental matrix is not unique, since it is obtained for any planar homography. Considering two planar homographies H i and H 2 , Eq. 8 implies that there exist scalars Ai and A2 such that [e'] x (A1H1+ A 2 H 2 ) = 0. If a matrix H is such that [e'] x H = 0, then there exists a vector r such that H = e'r T . We therefore conclude that H 2 ~ H i + e'r T .
(9)
This equation can be understood geometrically by applying both of its terms to the point m of the first image. Because r T m is a scalar, it says that the point of coordinates H 2 m in the second image belongs to the line defined by e' and the point of coordinates H i m . This is true because, as discussed in the derivation of Equation 8, it is the epipolar line of m. In fact, the direction of the vector r represents the projection in the first image of the intersection of the planes corresponding respectively to H i and to H 2 . To see that, note that a point belongs to both planes if and only if H 2 m ~ H i m , and therefore r T m = 0. The consequence of Eq. 9 is that the family of planar homographies is parameterized by a vector of dimension three (which is expected since the family of planes of P 3 has this dimension). Knowing any of these homographies make it possible to generate all of them. At this point, one could wonder how to obtain one planar transformation without relying on some prior knowledge of the scene to identify a plane. The trick is that from just F (remember that e' is given by F T e ' = 0), we can always obtain one by using the special matrix defined as: S = [e'] x F.
(10) T
It can be verified that S satisfies F ~ [e'] x S, using the identity I ~ v v - [v] x [v] x ; therefore it is equivalent to know F or S. The small price to pay for this "free"
127 planar transformation is that it is singular since the plane generating S is through the optical center C" (hence the singularity). We end this section by giving an expression of the Fundamental matrix as a function of the projection matrices, which can can be used on a calibrated stereo rig to guide the correspondence process by limiting it to epipolar lines. We write the two projection matrices "P = [P p] and "P' = [P' p']. The epipole e' is the projection T'C of the optical center C given by Eq. 4; therefore e' ~ T'C ~ [P' p']
--P-ip] 1
p' - P ' P ^ p .
Now that we know the epipole, to apply Eq. 8 in order to determine F, we need only a planar homography between the two images. One such homography is the one induced by the plane at infinity, given in Eq. 5, therefore: F = [p'-P;P1p]xP'p-1 4. A stratified approach to reconstruction The reconstruction problem can be stated as that of determining the projection matrices V and V', as well as the 3-D points M,, given a set of N correspondences (mj,mj) such that
The solution is not unique because it depends on the choice of a coordinate system, expressed by the 4 x 4 matrix H. If (V,P','Mi,--,MJV) is a solution to the reconstruction problem, then (7>JH"1,P'yH~1,'HM.i,-•• , ? £ M J V ) is also a solution, since m ' ~ V'M =
(V'-W^CHM).
(12)
In other words, all the pairs of projection matrices of the form {V'H,'P''H), where "H is an arbitrary projective transformation, are potentially equivalent. However, if we have some constraints about the correspondences, then we can hope to limit the ambiguity "H by enforcing that these constraints have to be satisfied by the pair (VH,V'U). We will see in Sec. 4.1 that from uncalibrated images, there are no further restrictions. We can recover only a projective reconstruction, which means reconstruction of points up to a general projective transformation *H of space. To obtain an Euclidean reconstruction (i.e. up to a Euclidean transformation plus scale), we need to use either some a priori information about the world, which makes it possible to determine in succession the plane at infinity (Sec. 4.2) and the intrinsic parameters of a camera (Sec. 4.3), or either some a priori information about the camera, which makes it possible to perform self-calibration (Sec. 5). The flow chart
128 of the approach to recover Euclidean reconstruction from uncalibrated images is summarized in Fig. 5.
correspondences "
Estimate Projection matrices
Projective reconstruct ion
ARine information about tlie scene: 3D parallel lines ios of lengths of parallel segments
Estimate the plane at infinitJ
Afflne reconstruction Euclidean information about the scene: 3D angles ratios of lengths of 3D segments
information about the intrinsic parameters Estimate the intrinsic parameters of a camera
constant parameters known parameters Euclidean reconstruction (up to scale factor)
Fig. 5. Euclidean reconstruction from images can be achieved either using information about the world or about the cameras.
4 . 1 . Projective
reconstruction
Point correspondences (m^, mj) are the only information that we have in this section. The projective correspondence information can be summarized by the Fundamental matrix F of the pairs of images, that we compute from the correspondences. Any pair of projection matrices (V, V1) is a valid solution to the reconstruction problem if and only if its Fundamental matrix is compatible with the point correspondences (77^, mj), or in other words if its Fundamental matrix is F . It can be shown that any pair ( P , P ' ) has Fundamental matrix F if, and only if it is of the form
with n =
V
(13)
where V is an arbitrary projection matrix (11 parameters) I I is the projective equation of an arbitrary plane (3 parameters), fi is an arbitrary constant (1
129 parameter). Together, these 15 parameters represent the projective ambiguity in reconstruction: the arbitrary choice of the projective basis in 3-D, or, equivalently, of the matrix /H. • The remaining elements in V' are: the epipole e' of F in the second image and the homography H, compatible with F and generated by the plane II. These entities are uniquely determined by F and II. This shows that given the Fundamental matrix F , once a projective basis in P 3 is chosen by fixing the 15 previous parameters, T>' is uniquely determined. We have partitioned the 22 = 11x2 parameters of a pair of projection matrices into two types of parameters: the projective correspondence of the pair of cameras embedded in the Fundamental matrix (7 parameters), and a projective transformation (15 parameters), which represents the ambiguity in reconstruction. The Fundamental matrix is invariant to the choice of the projective basis in P 3 . From Eq. 13: CPi,"P'i) and {V2,V2)
have the same Fundamental matrix
Vi = V-iH. and V'x = T^fi, where 'H is a projective transformation of P 3 which shows that when the only constraints on the matches come from the Fundamental matrix, all of the reconstructions up to a projective transformations are acceptable. The basic steps of the projective reconstruction are as follows: • • • •
Obtain pairs of correspondences rrii^m^. Solve for the Fundamental matrix with Eq. 7. Compute the special matrix S from Eq. 10. Choose a particular pair of projection matrices, called the projective canonical representation, obtained by choosing the "simplest" pairs among those in Eq. 13 using S as a particular instance of H:
r^psos],
(14)
This pair is an invariant representation in the sense that its elements do not depend on the choice of projective coordinates in P 3 . • Solve for M; with Eq. 11. Alternatively, if a planar homography matrix is available (after identifying the image of a plane in the two views), it could be used instead of S. The reconstruction will often have a very significant distortion because a projective transformation does not even preserve depth ordering, let alone relative distances. However, incidence (coplanarity, alignment, and intersections) is correct (Fig. 4.1a). 4.2. Affine
reconstruction
We have just seen that in the general case, from two images we are able to reconstruct points and projection matrices which are obtained from the "true" points by
130
(a)
(b)
(c)
Fig. 6. Projective (a), Affine (b), and Euclidean (b) 3D reconstructions of the scene in Pig. 3.3. Courtesy S. Bougnoux/INRIA
a projective transformation. If, in addition, we have some affine information, we can reduce the ambiguity in reconstruction from a general projective transformation of P 3 to an affine transformation of P 3 , which means that we are able to reconstruct points and projection matrices which are obtained from the "true" points by a transformation which is more constrained, and therefore induces less deformations and preserves more properties. However, the affine information has to come from some additional knowledge about the world or the system of cameras. Correspondences alone cannot provide affine information. The affine space is characterized by the plane at infinity IIoo in P 3 , which has three parameters. Affine information between two images is encoded as the correspondence of projections of points of IIoo. The correspondence of points of IIoo, is described by a particular planar homography matrix called the infinity homogmphy Hoo ( Eq. 5). Once F is known, three correspondences of points at infinity are necessary to determine Hoo. One way to obtain them is to consider three corresponding vanishing points. Since parallel lines in P 3 intersect on the plane at infinity, a vanishing point, which is the intersection of projections of parallel lines, is the projection of one point of IIoo. It is also often possible in practice to use the correspondence of points that are the farthest from the camera to compute an approximation to Hoo. Ideally those points would be on the horizon, with no translational parallax, however even if the chosen points do have parallax, this approach will still yield a valid projective reconstruction with less distortions than one using an arbitrary homography. Knowledge of the ratio of distances for three aligned points also provides a constraint on Hoo. When affine correspondence is known, we can restrict further the pairs (T^T1) of possible projection matrices. We require, in addition to the fact that it has F as its Fundamental matrix, the fact that Hoo is its infinity homography. The pairs of projection matrices which have infinity homography Ho© and epipole e ; are of the form
MaM, [Hoo A*e']«4,
with A, =
0g*lj
(15)
131 This decomposition is a particular case of Eq. 14, obtained with II — [0,0,0,1] T . The crucial remark is that the transformation of space is an affine transformation rather than a projective one. This decomposition separates the total 22 parameters into two types of parameters: • 12 correspond to the affine ambiguity in reconstruction: the arbitrary choice of the affine basis (11 obtained by fixing T>, 1 is fx) • 10 describe the affine correspondence: 8 as the infinity homography Hoc and 2 as the epipole e'. That is, given affine correspondence as an infinity homography and an epipole, once an affine basis is chosen by fixing the 12 previous parameters, T1 is uniquely defined. From the decomposition in Eq. 15 it is easy to verify that ("Pi, "Pi) and {'Pii'P^) have the same infinity homography and Fundamental matrix V\ — V2A and V'x = V'2A, A being an affine transformation of P 3 . To summarize, when we have identified the plane at infinity IIoo, a pair of images with epipole e' and infinity homography H ^ determines a reconstruction up to an affine transformation of P 3 . The reconstruction can be performed using one particular pair of projection matrices, the affine canonical representation, whose elements do not depend on the choice of the affine basis in P 3 . We remark that it can be obtained from the projective representation described in Eq. 14: T=
[I3O3]
.[IsOalQl1
,7" = [IWe'] "[S/jeTO1
w i t h Q
WlthQ
^
-i_
-
I3 0 3 T
In this equation, r^> is the 3-parameter vector of Eq. 9 that gives Hoc once F (and therefore S) is known H ^ - S + e'r^. According to Eq. 12, to upgrade the projective reconstruction of the points to an affine reconstruction, we need only to apply the transformation Q ^ . Because affine transformations include shear and different scalings along axes, the relative distances of points are not preserved (Fig. 4.1b). However, the relative distances of aligned points are preserved, so we can make some quantitative measurements, such as locating the middle of a segment. 4.3. Euclidean
reconstruction
We now go one step further and reach more familiar ground by examining the case when in addition to the affine correspondence, we have Euclidean information. This information makes it possible to reduce the ambiguity to a similarity transformation (displacement plus scale) and obtain the reconstruction illustrated in Fig. 4.1c.
132 Euclidean information in an image is encoded as as the matrix of intrinsic parameters of the camera which we will call A. In general, this matrix represents five parameters which are known when the camera is calibrated. They can be determined from a combination of knowledge about the camera (for example most real cameras have a zero skew and known aspect ratio, which reduces the number of parameters to three) and of knowledge about the world (such as angles and ratios of distances). When A is known, we can restrict further the pairs ('P,7>') of admissible reconstructions. We first note that if we decompose each projection matrix into intrinsic and extrinsic parameters, we have the classical decomposition for any pair of projection matrices: • 7 > ~ A [ R i t 1 ] = [A0 3 ]5 . V ~ A'[R 2 1 2 ] = A ' [ R / i t ] 5
Ri ti OJ 1/yuJ '
where R — R 2 R f and t = t 2 — R 2 R f t i represents the relative displacement between the two camera coordinate systems. Let's count again: of the total 22 parameters: • 7 correspond to a similarity transformation representing the arbitrary choice of the Euclidean basis (6 obtained by fixing the coordinate system of the first camera through R i and t i , and 1 being // which represents the scale), • 15 describe the intrinsic parameters (5 for each camera) and the relative Euclidean transformation R, t (position and orientation) of the two cameras. The direction of the translation is determined, but its norm is not because of the depth-speed ambiguity: one cannot distinguish between a close point moving slowly and a distant point moving proportionally faster. Computing from the projection matrix, we obtain with easy algebra e' = At , Hoc = A ' R A 1 . We can therefore characterize the Euclidean correspondence by either one of the two sets of fifteen parameters: • the affine correspondence plus intrinsic parameters of one camera: H e' A • the intrinsic parameters of both cameras and the displacement between two cameras: A, A', R, t. OOJ
c
) •**•
Similarly to the previous situations: ('Pii'P'i) and (V2,T'2)
have the same Euclidean correspondence
Vx = V2S and V\ = T'2S, where S is a Euclidean (similarity) transformation of P 3 .
133
We can now obtain a Euclidean canonic representation as a specialization of affine and projective strata. By using as 3-D coordinate system the first camera's coordinate system: V ~ [A0 3 ] _ [ I s O s l Q ^ Q , " P ' ~ A'[R/it] [S/ze'lQ^Qi 1
(16)
with Q/
=
i 3 o3
and
QE1
=
MJ
A 03 0T 1
(17)
Starting from a projective reconstruction, which requires only point correspondences, we can upgrade to an affine reconstruction when r^, is known (3 degrees of freedom) by applying Q^ to the points Mi, and to a Euclidean reconstruction when A is known (5 DOF) by applying Q E . QA is a projective transformation which moves the plane at infinity, and Q s is an affine transformation which moves the absolute conic in the plane at infinity. Each upgrade reduces the reconstruction ambiguity, first from a general homography (15 DOF) to affine (12 DOF), then from affine to similarity (7 DOF). The representations and ambiguities in reconstruction are summarized in the following table: PROJECTIVE reconstruction ambiguity invariant description canonical form AFFINE reconstruction ambiguity invariant description canonical form EUCLIDEAN reconstruction ambiguity invariant description canonical form
homography (incidence, P ) V U fi non-singular 1
n
S : special matrix e': epipole V * [I3 03]U V' =: [S + e'rg fie']-H affine transformation (parallelism, IIoo) A = P P P non-singular O^l/p Hoo : infinity homography e ' : epipole V ~ [I3 03]A VI ~ [Hoo l*e!]A similarity (angles, fi) Ri ti S =
oi i/ M
15
22
12
22
Ri orthogonal
A,A' : intrinsic parameters R : rotation between cameras t : direction of translation between cameras V s [A0 3 ]5 V' ^ A'[Rjut]5
5+5 3 2 22
134 5. Self-calibration of a moving c a m e r a Using some knowledge about the world made it possible to upgrade a projective reconstruction to affine and Euclidean. This knowledge has to be supplied to the system interactively by the user. An alternative, and more automatic, way to supply this information is to use constraints about the camera's intrinsic parameters, such that the fact that they are partially known (for instance for most cameras the skew is zero since the pixel grid is orthogonal, and the aspect ratio is known) or constant across images.
5.1. Euclidean
structure
and the absolute
conic
In order to express these constraints for this purpose, we use the absolute conic fi as a projective encoding of Euclidean structure. Because the change of camera pose corresponds to a Euclidean transformation, which, as a particular case of a similarity transformation, leaves the absolute conic invariant, we conclude that its projection w which is also a conic with only complex points, does not depend on the pose of the camera. Therefore, its equation in the image coordinate system does not depend on the extrinsic parameters and depends only on the intrinsic parameters. Computing in the camera coordinate system, it is easy to see that its matrix is B = A ~ T A - 1 , whereas its dual conic has matrix, using Eq. 2, K = B* = d e t ( B ) B - 1 ~ A A T . When the camera is calibrated, in the camera coordinate system, the projection of the absolute conic is just an imaginary circle of radius one, while in general it is an ellipse (Fig.2.1b). The principle of self-calibration is to use the absolute conic as a calibration object. It has the advantage of being always available for free, however since it is a rather inaccessible object, all we can do with it is to write constraints across images. The knowledge of the infinity homography or of the Fundamental matrix makes it possible to write equations relating the intrinsic parameters of the two images. When we move a camera, we can obtain several such equations. Combining enough of them with the constraints on intrinsic parameters make it possible to recover the intrinsic parameters. Therefore just by moving a camera, observing its environment, and establishing point correspondences, we are able to calibrate the camera and eventually perform a Euclidean reconstruction of the environment, without ever needing a calibration object.
5.2. From affine to
Euclidean
We start with the simpler case when the infinity homography is known. A practically important situation when this occurs is the case of a camera that rotates and zooms while staying at a fixed position.
135 In HQO — A ' R A - 1 , Hoc depends on eight parameters, and the rotation on three parameters, so we should be able to write five constraints on the intrinsic parameters. We eliminate the motion using the fact that a rotation matrix is orthogonal: R R T = I3. This step yields a linear relation between the infinity homography and the intrinsic parameters: H o c K H ^ ~ K',
(18)
where K and K' are the two Kruppa matrices. One could think that this is sufficient to solve for the five intrinsic parameters when they are constant (K — K'), but it turns out that one of the equations in this case is redundant because the infinity homography satisfies the additional constraint det(Hoo) = 1. More precisely, it can be verified that there is a two dimensional space of solutions, spanned by the expected solution K = A A T and the spurious solution K = A U ( A U ) T , where U is the axis of the rotation. Two rotations along different axis are therefore necessary to solve for all of the intrinsic parameters. We can remark that self-calibration depends only on the rotational component of the motion: it relies on the absolute conic, which being an object at infinity, has an projection not affected by translations but only by rotations. In particular, if there is no rotation, then Eq. 18 becomes a tautology. In order to recover camera calibration from H^ (and therefore from F as seen next) it is therefore necessary to have a motion with a non-null rotational component. 5.3. From, projective
to
Euclidean
We now consider the general case when only a projective representation is available. We note that the representation of Euclidean correspondence consisting of A, A', F is redundant since it contains seventeen parameters, while a minimal representation has only fifteen parameters, so there must be two constraints between the Fundamental matrix and the intrinsic parameters. Another way to see it is to remark that the Fundamental matrix can be expressed as F = A'-T[t]xRA-1. F depends on seven parameters and the motion on five parameters (the scale of the translation is not determined), so there must be two constraints, obtained again by eliminating the displacement. Multiplying Eq. 18 by [e'] x left and right: [e'] x K'[e']x * [ e ' ] x H M K C [ e ' ] x , and using Eq. 8: FKFT~[e']xK'[e']x, which is equivalent to two polynomial equations of degree 2 in the coefficients of K and K', called the Kruppa equations. Examples of applications of these equations are:
136 • Computing the focal lengths of the two cameras from the Fundamental matrix of a pair of images, assuming that the other intrinsic parameters are known. • Computing all of the intrinsic parameters of a moving camera with constant intrinsic parameters from three images. The Kruppa equations make it possible to perform self-calibration from the Fundamental matrices by solving only for the intrinsic parameters using polynomial methods. In practice, it is advantageous to start from the projective canonical representation, because being global, it is numerically a more stable representation. We recover affine and Euclidean information at the same time by solving for the eight parameters of the projective transformation of Eq. 17: ^ =
QA Q E
=
T A ,,
.rooA M.
According to Eq. 16, this transformation maps the set of canonical perspective projection matrices "P^ into canonical Euclidean projection matrices: ViU ~ AjfRiti],
2
We use the same trick as we did in the affine case: we multiply the first 3 x 3 submatrix (remember that only the rotational motion is relevant for self-calibration) by its transpose, eliminating the unknown rotation matrices R J : K Kr0 rT K rT Kr
V\ ~ Ki,
2 < i < n.
These equations generalize self-calibration to the case of variable intrinsic parameters. Any constraints on the intrinsic parameters Aj translates into constraints on the Kruppa matrices K;. For instance, when the pixels are orthogonal (7 = 0) K13K23 — ^33^12 = 0. If enough constraints are available, we can solve for K and r ^ by nonlinear minimization. Since there are 8 unknowns, it is crucial to have a good starting point, which can be obtained with the Kruppa equations. 6. Further reading During the last dozen of years, there has been considerable interest in the topics surveyed in this chapter. Instead of attempting to list a bibliography that could be only very partial, we will refer the reader to three books 1 2 3 . References 1. O. Faugeras and Q.-T. Luong. The geometry of multiple images. MIT Press, 2001. 2. R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004. 3. Yi Ma, Stefano Soatto, Jana Kosecka, and Shankar Sastry. An Invitation to 3-D Vision. Springer-Verlag, 2003.
CHAPTER 2.3 SKELETONIZATION IN 3D DISCRETE BINARY IMAGES
Gabriella Sanniti di Baja1 and Ingela Nystrom2 'institute of Cybernetics "E. Caianiello", CNR, IT- 80078 Pozzuoli (Naples), Italy, E-mail: g. sannitidibaja @ cib. na. cnr. it 2 Centre for Image Analysis, Uppsala University, SE-752 37 Uppsala, Sweden, E-mail: [email protected] Skeletonization is a way to reduce dimensionality of digital objects. Here, we present in detail an algorithm that computes the curve skeleton of a solid object, i.e., an object without cavities, in a 3D binary image. The algorithm consists of three main steps. During the first step, the surface skeleton is detected, by directly marking in the distance transform of the object the voxels that should be assigned to the surface skeleton. The curve skeleton is then computed by iteratively thinning the surface skeleton, during the second step. Finally, the third step is performed to reduce the curve skeleton to unit width and to prune, in a controlled manner, some of its peripheral branches.
1. Introduction The skeleton is a medial representation of an object with lower dimension. It reflects a number of geometrical features of the object, is thin and symmetrically placed within the object, and has the same topology. The object can be recovered starting from the skeleton, if this includes all the centres of maximal discs/balls of the object. With the above features, the skeleton is a promising tool for an increasing number of possible applications, especially in biomedical imagery. Skeletonization has initially been suggested to reduce a 2D discrete object to a linear representation. Given a binary 2D image, skeletonization changes object pixels to background pixels until a subset of the object is obtained, which is a union of arcs and curves placed symmetrically with respect to the border of the object. Reducing discrete objects to lower dimensions is even more desirable when dealing with 3D volume images. The 3D skeleton is either a set of 3D surfaces and curves (called surface skeleton, SS) or, if reversibility is not requested and the object is solid, consists of 3D curves (called curve skeleton, 137
138
CS). The general strategy for 3D skeletonization does not differ from the 2D case: object voxels are changed to background voxels, provided that topology and geometry are not altered, until the skeleton is obtained. A commonly followed approach to skeletonization is iterative thinning, which is based on the repeated use of topology preserving removal operations. To have a skeleton centred within the object, and hence reflecting its geometrical features, removal operations have to be applied border after border. This generally implies long computation time, proportional to object thickness, which prevents the actual use of the skeleton for real time applications. An alternative approach is based on the use of the distance transform of the object. For example, in [1], the city-block distance transform of a 2D object was considered. In the distance transform, the layers, i.e., the sets of pixels having the same distance value, were interpreted as the successive borders that would have characterised the object, if this was undergoing iterated pixel removal. Being all layers available at the same time, the skeletal pixels could be directly identified and marked in a small and fixed number of inspections of the distance transform. This method has been extended in [2] to the computation of the surface skeleton in the 3D case. While the literature on 2D skeletonization is very rich, not so many efficient algorithms are available for 3D skeletonization. 3D objects can be directly reduced to ID curve skeletons, e.g., [3-5], or they can be first reduced to 2D surface skeletons, e.g., [2, 6, 7], and then furthermore compressed to ID curve skeletons, e.g., [8, 9]. All above papers regard discrete approaches to skeletonization. Alternative continuous approaches can be found, e.g., [10-11]. In this Chapter, we follow the discrete approach and describe a computationally efficient algorithm to skeletonize 3D solid objects in binary images. The algorithm consists of three main steps. The first step is devoted to the computation of SS by directly identifying in the distance transform of the object, the voxels that should be assigned to the surface skeleton. These voxels include all centres of maximal balls (necessary to guarantee reversibility of surface skeletonization), voxels placed in saddle configurations, and other voxels necessary to have SS homotopic to the original object. Computation of SS follows the procedure introduced in [2]; SS is at most two-voxel thick (except for some cases of complex surface intersections), centred within the object and, due to the inclusion in SS of all the centres of maximal balls, the object can be recovered starting from SS (i.e., surface skeletonization is reversible). An important advantage of our surface skeletonization is its reduced computational cost, compared to iterative thinning performed on conventional sequential computers. In fact, SS is obtained in a small number of scans, independently of object thickness.
139
The second step extracts CS from SS by means of iterative thinning. We remark that curve skeletonization is not reversible. In fact, a number of centres of maximal balls are unavoidably removed from SS to obtain the curve skeleton with lower dimensionality. However, CS is still a useful representation, if it maintains shape information. Our curve skeletonization is based on the work presented in [9]. The voxels of SS are classified as internal, edge, curve, and junction voxels. Only edge voxels are considered for removal at each iteration. The basic idea behind this curve skeletonization algorithm is to always prevent removal of curve voxels and to postpone as much as possible removal of the voxels initially classified as junction voxels, since curves and junctions between different surfaces in SS retain the most significant shape information. Curve skeletonization is performed in two phases. During the first phase, removal of voxels classified as junction voxels in the initial SS is prevented, even if junction voxels are transformed into edge voxels at some iteration. During the second phase, all current edge voxels are candidate to removal. The timing of the two phases forces junctions among peripheral surfaces to be kept in the skeleton, independently of the shape of peripheral surfaces. Moreover, any non-peripheral surface, i.e., a surface completely bordered by voxels initially classified as junction voxels, is isotropically compressed. An important advantage of preservation of the curve voxels is the fact that unwanted shortening of skeleton branches is avoided without need of resorting to an end point detection criterion, which often fails to act isotropically. The obtained CS is at most two-voxel thick, is centred within the surface skeleton, and has the same topology as SS and, hence, as the original object. The third step of the algorithm is performed to reduce the at most two-voxel thick CS to unit width (final thinning), and to delete, in a controlled manner, some of its peripheral branches (pruning). This step is accomplished by using the method introduced in [12]. Final thinning makes use of different strategies, depending on whether two-voxel thickness is due to pairs of skeletal voxels that are face- or edge-neighbours of each other. Pruning is performed by tracing any peripheral branch and by removing its voxels, provided that the branch includes a small percentage of significant voxels. We regard as significant in CS the voxels that were classified in SS as curve or junction voxels. We show results from the various steps mainly on synthetic images. A real example, blood vessels in magnetic resonance angiography (MRA) images, is given in the Conclusions.
140
2. Definitions and Notions The images considered here are 3D bi-level images consisting of object, F, and background, F. Any voxel v has in the 3x3x3 neighbourhood centred on v six face-neighbours, twelve edge-neighbours, and eight vertex-neighbours, sharing with v a face, an edge and a vertex, respectively. The set including all the above 26 neighbours of v is called the 26-neighbourhood of v. The set including only the face-neighbours and the edge-neighbours of v is called the 18-neighbourhood of v. To deal with two-voxel thick surfaces and curves, we introduce also the 4x3x3, 3x4x3, and 3x3x4 neighbourhoods of v and refer to them as to the neighbourhoods extended in the x-, y-, and z-direction, respectively. In particular, if the eight voxels in the corners of the 4x3x3 (3x4x3, 3x3x4) neighbourhood are disregarded, the obtained set includes, besides v, 27 elements and is called the 27-neighbourhood_x (27-neighbourhood_y, 27neighbourhood_z). See Fig.l, where u is face-neighbour of v in the z-direction and the 27-neighbourhood_z of v is shown. / / / / u / V /
y _V
Fig. 1. The 27-neighbourhood_z of v.
To avoid topological paradoxes, two different connectedness types are necessary for F and F. We use the 26-connectedness for F, and the 6connectedness for F. Let N26, Nfi8, and Nf27, respectively denote the number of 26-connected object components in the 26-neighbourhood of v, the number of 6-connected background components in the 18-neighbourhood that have v as a face-neighbour, and the number of 6-connected background components in the 27-neighbourhood _{x,y,z}, having v or u as a face neighbour. N 26 and Nf18 can be computed by means of the computationally convenient algorithm introduced in [13]. The algorithm can be extended to deal also with the 27-neighbourhood _{x,y,z}, which is necessary for the computation of Nf27 in presence of twovoxel thick sets. An object has a tunnel if there exists a closed 26-connected path in the object that cannot be deformed to a single voxel (for details, see [14]), and a cavity if a background component is fully enclosed in the object. An object that does not include any cavity is a solid object.
141
A voxel v can be safely removed if its removal does not change the number of object components, the number of cavities, and the number of tunnels in the 3x3x3 neighbourhood of v, [15-16]. In other words, object voxels with N 26 =l and Nfi8=l can be safely removed. In turn, if N26 >1, removal of v would break the connectivity, i.e., a component would break into two or more components. If N26 <1, removal of v would cause vanishing of an object component consisting of one voxel. If Nfi8l removal of v would create a cavity or a tunnel, respectively. An at most two-voxel thick set is said to be nearly thin. The distance transform of F is a replica of F, where each object voxel is assigned a value equal to its distance to the closest voxel of F. Two often used metrics in 3D are D6 and D26, for which the distance between any two voxels is equal to the number of steps connecting the two voxels in a minimal 6-connected and 26-connected path, respectively. Better approximations of the Euclidean distance, hence resulting in a distance transform more stable under object rotation, are obtained by using suitable weights wfi we, and wv to measure the steps towards face-, edge-, and vertex-neighbours along the path. Good values for the weights have been shown in [17] to be w/=3, we=4, and wv=5. In this paper, we will use the D6 distance, i.e., the 3D equivalent of the cityblock distance, and will call the corresponding distance transform DT. For D6, w p l , we=®°, and nv=°°. For simplicity, the same letter will denote both a voxel and its associated distance value in DT. The distance value v of a voxel v of DT can be interpreted as the radius of a D6 ball centred on v and included in the object. The shape of a D6 ball is an octahedron, see Fig. 2. An object voxel v in DT is centre of a maximal ball (CMB) if the D6 ball centred on v is not completely covered by any other single ball in the object.
Fig. 2. A D6 ball.
In DT, a voxel is a CMB if none of its face-neighbours has a higher distance value. The set of CMBs is nearly thin in presence of parts of F with thickness given by an even number of voxels. F can be recovered, e.g., by applying the reverse distance transformation to the set of its CMBs, [18].
142
In DT, the saddle voxels are the voxels located along ridges of DT connecting areas of DT with higher distance values. Many saddle voxels are also CMBs. However, those delimiting the ridge are not necessarily CMBs. The surface skeleton SS of F is a subset of F consisting of surfaces and curves, centred within F, with the same topology and including all CMBs. Unfortunately, one-voxel thickness and inclusion of all CMBs are seldom compatible, due to the discrete nature of the digital space. Inclusion of all CMBs is necessary for a faithful recovery of F by the union of the maximal balls. This property is often referred to as reversibility of skeletonization. In this Chapter, we compute a nearly thin reversible skeleton. If desired, the nearly thin SS can be reduced to one-voxel thickness. Here, we do not illustrate how to perform reduction to one-voxel thickness, since our final goal is to compute the curve skeleton of F. Hence, 55 is seen as an intermediate result, rather than as a representation of F itself. For a description of thinning methods adequate to compress the nearly thin 55 to one-voxel thickness, see, for example, [8]. To compute SS with the above features, we directly identify in DT the voxels that should be assigned to the surface skeleton of F. CMBs and saddle voxels constitute the kernel of the surface skeleton. These voxels are called intrinsic skeletal voxels and guarantee that 55 is nearly thin, symmetrically placed within F and reversible. The remaining skeletal voxels, called induced skeletal voxels and tunnel voxels are necessary to make SS homotopic to F. A marker is assigned to the skeletal voxels in DT, to distinguish them from all other voxels. It can be proved that a voxel v is an intrinsic or an induced skeletal voxel if any of the following Conditions A1-A3, involving the voxels shown in Fig. 3, is satisfied: Condition Al: No pair of opposite face-neighbours of v exists such that one has a value smaller than v, and the other has a value larger than v. Condition A2: There exists an edge neighbour e of v, with e=v and such that the face-neighbours of both e and v have values smaller than v. Condition A3: There exists a vertex-neighbour x of v, with x=v and such that the six common neighbours of both x and v have values smaller than v.
Fig. 3. Voxels involved in Condition Al, left, A2, middle, A3, right. White, grey and black denote voxels with distance value smaller, equal and larger with respect to the voxel at hand.
143
The presence of marked voxels in the neighbourhood of v requires a suitable interpretation so that the above conditions allow detection of the induced skeletal voxels. In detail, marked neighbours with distance value equal to or larger than v have to be seen as unmarked, all other marked neighbours have to be seen as having value v. A deeper explanation is given in Section 3.1. For a solid F, the curve skeleton CS is a one-voxel thick subset of F consisting of curves symmetrically placed within F and homotopic to F. Curve skeletonization cannot be reversible, since some CMBs will be removed from F to obtain CS. This notwithstanding, the curve skeleton can still be a useful object representation since it reflects the main shape features of F. In the curve skeleton, voxels with only one neighbour and with more than two neighbours in CS are called end points and branch points, respectively. To compute the curve skeleton from the surface skeleton, we need to classify the voxels in the surface skeleton as internal, curve, edge, and junction voxels. (The border of a surface is the set including curve and edge voxels.) In [19, 20], the voxels of a surface were classified, by means of 3x3x3 operations. That classification works correctly only for "ideal" surfaces, i.e., surfaces that are onevoxel thick everywhere, but fails whenever surfaces cross each other in such a way that they produce two-voxel thick junctions. As an example, refer to Fig. 4, where the dotted dark grey voxels would not be correctly classified as junction voxels. Therefore, we need a classification able to deal with two-voxel thick surfaces. In a nearly thin surface/curve, a set of four voxels aligned along the x-, y-, or z-direction, where the two internal voxels are object voxels and the two external voxels are background voxels is called quadruplet_{x,y,z} .
Fig. 4. Two-voxel thick junctions (dotted dark grey voxels).
1. 2. 3. 4.
A surface voxel v is classified according to the following criteria: N26>2and N f i8 >l, curve voxel. N26=l and Nf18>2, junction voxel. N26=l and Nf18=0, junction voxel. N26=l and Nf18=2, one -voxel thick internal voxel, or two-voxel thick
144
junction voxel. 5. N26=l and N f i 8 =l, border voxel, two-voxel thick internal voxel, or two-voxel thick junction voxel. For any voxel v satisfying condition 4 (condition 5) further processing is necessary. If v is in a quadruplet_{x,y,z} with a voxel, u, also satisfying condition 4 (condition 5), then the 27-neighbourhood_{x,y,z} of v is examined and v is classified as follows: 6. Nf27=2, internal voxel. 7. Nf27 >2, junction voxel. 8. Nf27=l, border voxel (edge voxels or curve voxel). Any voxel v satisfying condition 4 and that is not in a quadruplet_{x,y,z}, is classified as internal voxel. Any voxel v satisfying condition 5 and that is not in a quadruplet^ x,y,z}, is classified as border voxel. Finally, for all border voxels, i.e., voxels satisfying conditions 5 or 8, a decision can be taken on whether they are edge or curve voxels. A border voxel v is classified as a curve voxel if 9. it has all its neighbours in the surface classified as border voxels, or 10. it has a neighbouring curve voxel w, such that u does not have a neighbouring internal voxel.
\
junction
internal
edge
Fig. 5. A surface, top, and its associated classification, bottom.
The remaining border voxels are classified as edge voxels. For the sake of completeness, we remark that a few special cases, occurring in presence of junctions thicker than two voxels in the x-, y-, or z-direction, have to
145
be treated separately. These cases are exhaustively considered in [21]. In Fig. 5 a nearly thin surface and the associated classification are shown. 3. Surface Skeleton Computation Surface skeletonization can be performed by iteratively applying to F, border after border, topology preserving removal operations, e.g., based on the computation of N26 and Nfi8. At each iteration, the current border of F is parallelwise identified as the set of voxels of F having a face-neighbour in the current F (all other voxels of F are internal voxels). Then, border voxels sequentially undergo topology-preserving removal. The surface skeleton obtained in this way is generally not symmetric. In fact, only voxels indispensable for topology preservation are kept and the voxels that are not deleted depend on the order in which the border voxels are examined. This is a drawback of iterative skeletonization, since symmetry is an important feature for shape analysis. Moreover, a large number of inspections, dependent on object thickness, are generally necessary, which makes the computation cost high, when conventional sequential computers are used. Our surface skeletonization algorithm follows a different strategy that originates a surface skeleton SS symmetrically placed within F at a limited computation cost. In particular, rather than "removing" suitable voxels border after border, we directly identify and "mark" as skeletal voxels in DT, the voxels of F that have to be certainly assigned to its surface skeleton. In fact, in DT all borders, that would successively characterise F if skeletonization was performed by means of iterated removal, are simultaneously available. A voxel v is interpreted as belonging to the border that would be identified at the v-th iteration, its neighbours with value smaller than v as belonging to the v-th background, its neighbours with value v as belonging to the v-th border, and its neighbours with value larger than v as belonging to the v-th set of internal voxels.
Fig. 6. A digital Euclidean ball, left, and the centres of maximal balls, right.
In Section 3.1, we use the digital Euclidean ball, shown in Fig. 6, left, as a
146
running example. This is a difficult object because the skeleton that we compute is based on D6. The set of the CMBs of the object is shown to the right. 3.1. Marking the Skeletal Voxels in the Distance Transform We distinguish three types of skeletal voxels: intrinsic, induced, and tunnel voxels. Intrinsic voxels include CMBs and saddle voxels, and are identified parallelwise in DT. Induced voxels become skeletal voxels only after their neighbourhood has been modified by the marker assigned to other (intrinsic or induced) skeletal voxels. Intrinsic and induced voxels together guarantee that SS has the same number of components and cavities as F. Tunnel voxels are necessary to guarantee that SS has also the same number of tunnels as F. Surface skeletonization is performed in four steps. Intrinsic voxels are found in the first step. Induced voxels are found in the second and the third steps. Tunnel voxels are found in the fourth step. Conditions A1-A3 are used in DT tojdentify and mark intrinsic and induced voxels. Conditions based on Nf18 and N 27 are used to detect tunnel voxels. Step 1. This step is performed to detect and mark the intrinsic voxels in one scan of DT. The CMB detection criterion, based on the comparison of the distance values of the voxel at hand and its four face-neighbours, and Conditions A1-A3 are checked parallelwise. Note that even though Condition Al would be enough to automatically detect all CMBs, we prefer to resort to the CMB detection criterion specifically, in order to distinguish the CMBs from the remaining intrinsic voxels. Step 2. This step identifies the voxels that are induced to become skeletal voxels by the CMBs. Any of these induced voxels is characterised by a distance value equal to or larger than the distance value of the inducing CMB, and is identified and marked parallelwise during one scan of DT. For any CMB, say w, encountered in DT, more than one neighbour could be induced to become a skeletal voxel. To avoid unnecessary thickening of the set of the skeletal voxels, for each 26-connected component of neighbours of w, consisting of voxels with distance value > w, only the voxel with maximal distance value, say v, is checked against Conditions A1-A3. Neighbours of v that are already marked as skeletal voxels and have distance value less than v are seen as having value v. (For all other marked neighbours, distance values are seen as they are.) The reason for this interpretation can easily be understood by referring to skeletonization accomplished by iterative thinning: once a voxel is ascribed to the skeleton at a given iteration of skeletonization, that voxel will belong to all successive borders of the object in all successive iterations. Marked voxels with distance value less
147
than v are, hence, interpreted as belonging to the border, when v is examined. During this step, some of the saddle voxels found in Step 1 become superfluous for connectedness preservation. This is the case when a saddle voxel s is neighbouring both a CMB w and the voxel(s) induced by w. When this is the case, the marker is removed from s to avoid unnecessary thickening of SS. Step 3. This step is aimed at the detection of any voxel v induced to become skeletal voxel by a skeletal voxel z that is not a CMB (i.e., z is a saddle voxel or, an already detected, induced voxel) and is such that v>z. To keep the symmetry of the skeleton, Conditions A1-A3 are only checked for v, in the directions of the already visited neighbours of v, passing through the inducing voxel z. When v is inspected, one of its vertex-neighbours, three of its edge-neighbours and three of its face-neighbours have already been visited. This implies that for each inspection, Condition A3 is considered in one of eight possible directions, Condition A2 in three of twelve possible directions, and Condition Al in three of six possible directions. Thus, we need eight inspections, so that Condition A3 is checked in all directions. These eight inspections can be arranged in such a way that after four inspections Condition A2 and Al result to have been checked in all possible directions. We propose the inspection order shown in Table 1. The order implies that after four inspections, Condition A2 is investigated once in every direction and Condition Al twice. Different ordering of the inspections will produce the same result, but the number of inspections could be larger. The set of eight scans of DT is performed twice, so as to find and mark all the induced voxels, even for very complex and thick objects. Table 1. Proposed inspection order for the third step.
Condition Al prevents creation of spurious cavities. Moreover, merging of true cavities does not occur since marking v by means of Conditions A1-A3 does not change the number of components in the 3x3x3 neighbourhood of v.
148
However, the set of skeletal voxels obtained after Steps 1-3, shown in Fig. 7, left, is not homotopic to F, since intrinsic and induced skeletal voxels may group together creating tunnels that do not correspond to true tunnels of F. To have a homotopic surface skeleton, tunnel voxels should also be detected and added to the set of skeletal voxels. Step 4. This step is aimed at filling tunnels in a sequential fashion. Marking all DT voxels for which Nfig>l is not enough, due to the discrete nature of the digital space. In fact, tunnels can be also created in correspondence with the two internal voxels of a quadruplet, both having Nfis=l, when all four voxels in the quadruplet are not marked, the two internal voxels have the same value, and the two external voxels have smaller value. To fill these tunnels, also voxels in any such a quadruplet and for which Nf2?>l are marked. Tunnel filling is iterated as long as tunnel voxels are found. Due to its sequential nature, tunnel filling is likely to produce slightly asymmetric results. On the other hand, to fill the tunnels symmetrically, a parallel (iterative) procedure should be used, which would require a large number of iterations related to the size of the tunnels. To reduce computation time sequential tunnel filling can start already during Step 3. (Note that tunnel voxels do not induce any voxel to become a skeletal voxel.) To limit the asymmetries, tunnel detection should start only from the fifth scan of Step 3, since most of the skeletal voxels have already been marked by then. The resulting SS is shown in Fig. 7, middle. SS is centred within the original object, homotopic to the object, at most two-voxel thick and includes all CMBs.
Fig.7. Skeletal voxels found after Steps 1-3, left, and after Step 4, middle, for the ball shown in Fig.6. Surface skeleton resulting after pruning short branches, right.
Often, a number of short noisy curves are present along the border of SS. They should be pruned to simplify the structure of SS and facilitate the successive curve skeletonization. An efficient algorithm to prune the surface skeleton by removing peripheral short branches has been introduced in [22]. The surface skeleton of the Euclidean ball as resulting after pruning the short branches is shown in Fig. 7, right.
149
A few examples showing the performance of the DT based surface skeletonization are given in Fig. 8. We remark that surface skeletonization is not invariant under rotation, since the used D6 is a rough approximation of the Euclidean distance. To have stability under rotation, weighted distances, better approximating the Euclidean distance, should be used. However, surface skeletonization would become more complex. Moreover, in case of elongated objects, D6 can be used to provide skeletons whose rotation dependency can be seen as disregardable.
Fig.8. A cube, a cylinder and a pyramid, ton, and their surface skeletons, bottom.
4. Curve Skeleton Computation The curve skeleton CS of F can be computed, if F is a solid object. CS is requested to be centred within F, homotopic to F and one-voxel thick. Of course, F cannot be recovered starting from CS, because only a subset of the CMBs of F can be retained in the curve skeleton. However, even without the reversibility property, curve skeletonization is useful for shape analysis tasks, provided that the curve skeleton is somehow "similar" to F and reflects its main shape features. This is particularly the case when F is constituted by the superposition of elongated parts, each of which represented in CS by a branch. Curve skeletonization is performed starting from the nearly thin SS$ without its preliminary compression to one-voxel thickness. This is a positive aspect of the algorithm for at least two different reasons: the resulting CS is more symmetric, which increases its usefulness; computation time is reduced, because final thinning is performed only once, i.e., after curve skeletonization. In fact, the
150
obtained CS is likely to be two-voxel thick at parts independently of whether SS was one- or two-voxel thick. The algorithm for curve skeletonization is based on the classification of voxels in SS and on iterated voxel removal. Once the nearly thin CS is obtained, final thinning will be performed to reduce it to one-voxel thickness. We will reach this aim by means of a sequence of directional processes compressing the nearly thin CS from different directions. The obtained CS is finally subjected to pruning, to remove less significant peripheral branches and obtain a linear representation more easy to handle. 4.1. From the Nearly Thin Surface Skeleton to the Nearly Thin Curve Skeleton Curve skeletonization is performed by iterative thinning. Using the classification described in Section 2, the border of SS is parallelwise identified; border voxels are sequentially removed, provided that topology is maintained and that branch shortening is prevented. A new border is then detected and the process is iterated as far as voxel removal is active. Topology preservation is guaranteed by removing only border voxels for which N26=l and N f 18 =l. The main problem is, indeed, to prevent unwanted branch shortening. The most commonly adopted strategy is to prevent removal of voxels which can be interpreted as end points, i.e., the voxels delimiting skeleton branches. Unfortunately, the property characterizing an end point, i.e., to have exactly one neighbour in the skeleton, cannot be used as a mean to prevent from removal during skeletonization voxels that will become end points. In fact, a voxel that will become an end point in CS is likely to be surrounded by more than one neighbour in SS, when it is inspected and checked for removal. Even if the end point condition is partially relaxed, e.g., by preventing removal of voxels with no more than two or three neighbours, the final result is a curve skeleton whose branches are strongly conditioned by the order in which removal operations are applied. Here, we use an alternative method, [9], to prevent unwanted shortening of skeleton branches, by exploiting the classification of the voxels of SS given in Section 2 and necessary to identify the border of SS. Indeed, that classification also allows us to identify internal, curve and junction voxels. In particular, curve voxels belong to the border of SS, together with edge voxels, but they already constitute linear subsets of SS and, as such, are certainly part of skeleton branches. If we do not apply removal operations to curve voxels, but only to edge voxels, we automatically prevent unwanted branch shortening without need of an end point detection criterion. Removal of edge voxels obviously causes some voxels of SS, classified as internal or junction voxels in the original surface skeleton, to become border voxels at a successive iteration. Again, border voxels
151
can be distinguished in curve and edge voxels and only the latter undergo topology-preserving removal. This process can be iterated as far as new edge voxels are identified and voxel removal is active. The resulting CS would be homotopic to SS (and, hence, to F), nearly thin, and characterised by branches corresponding to protrusions and convexities of SS. However, CS would not be guaranteed to account for the shape of F. To this purpose, besides curve voxels, we should also prevent from removal voxels classified as junction voxels in the initial surface skeleton, as they account for relevant shape information. On the other hand, if voxels initially classified in SS as junction voxels are never considered for removal, we would not be able to compute the curve skeleton whenever non-peripheral surfaces exist in SS, i.e., surfaces completely bordered by voxels initially classified as junction voxels. See Fig. 9, bottom left. There, all voxels bordering the central square region were classified as junction voxels in SS (Fig. 9, top). Thus, if voxels classified as junction voxels in the initial SS are always prevented from removal, skeletonization could not generate the desired curve skeleton (Fig. 9, bottom right).
Fig. 9. The surface skeleton of a brick, top, and the results of curve skeletonization after the first phase, bottom left, and after the second phase, bottom right.
To prevent from removal as long as possible voxels classified as junction voxels in the initial SS, iterated skeletonization is performed in two phases. During the first phase both the initial and the current classification are taken into account, iteration after iteration. At each iteration, only current edge voxels that were not classified as junction voxels in the initial classification undergo topology-preserving removal. The first phase of the process terminates, when no more voxels are removed. Regions delimited by loops of initial junction voxels
152
can finally be isotropically compressed. In fact, in the second phase, only the current classification is used and all current edge voxels undergo topologypreserving removal. The process terminates when no more voxels are removed. The set resulting at the end of the first and of the second phase of curve skeletonization applied to the surface skeleton of a brick (Fig. 9, top) are shown in Fig. 9, bottom left and bottom right, respectively. The performance of curve skeletonization can be seen in Fig. 10 for a few more examples corresponding to the cube, the cylinder and the pyramid of Fig. 8.
V Fig. 10. Curve skeletons of a cube, a cylinder and a pyramid.
4.2 From the Nearly Thin to the One-voxel Thick Curve Skeleton To extract the nearly thin CS from SS we did not introduce any end point detection criterion. This is however necessary to reduce the nearly thin CS to the desired one-voxel thick set. We note that, since the nearly thin CS is at most twovoxel thick, end point detection criteria can be designed which cause a limited unwanted shortening effect on skeleton branches. Final thinning is performed by resorting to a series of directional processes, [12], since tips of branches are differently structured in different orientations and, accordingly, different end point detection criteria should be used. Generally, a tip of a branch includes from one to four voxels, each of which candidate to be the end point of a resulting thin branch of CS. Of course, whichever directional process is used, voxels with only one neighbour in CS are certainly end points and, as such, are prevented from removal. Two-voxel thickness affects skeleton branches consisting of pairs of faceneighbours or of pairs of edge-neighbours. Moreover, thickening also occurs in presence of complex crossings of skeleton branches, when branch points are grouped into components including more than one branch point {cluster). Thinning of CS along the three principal directions is performed to get rid of thickening due to pairs of face-neighbours. No end point criterion is used (except
153
for the obvious one corresponding to the voxels with exactly one neighbour in CS), but three directional processes are accomplished one after the other in forward fashion and, then, repeated in backward fashion. During each process, only voxels that have more than one neighbour in CS and belong to quadruplets, oriented in the scanning direction, are sequentially removed by means of topology preserving removal operations. Of course, which voxel remains as endpoint depends on the scanning order. To reduce thickening due to pairs of edge-neighbours, end points and candidate end points are identified and marked. The candidate end points are the voxels with one 26-connected component of skeletal voxels (N26=l) and at most three neighbours in CS. They may be grouped in components including more than one voxel. In turn, each group can actually be placed on a tip of a branch, or can be in correspondence with T-junctions or 90° corners. Topology preserving removal operations are then applied sequentially to all unmarked voxels, without any risk of causing unwanted shortening. Once all removable unmarked voxels have been removed, the marked voxels are sequentially visited and removed if they have one 26-connected component of neighbours in CS (marked or unmarked) and this component includes more than one element. This process will be effective both on tips of branches and on T-junctions and 90° corners. 4.3. Pruning and Jaggedness Removal Pruning CS is performed to remove non-significant branches and to make the skeleton less rotation dependent. To decide on skeleton branch significance we use the classification of the voxels of SS. In particular, a skeleton branch is more significant the larger is the number of voxels along the branch that were classified in SS as curve voxels or junction voxels. Besides significance-based pruning, brute force pruning, based on the total number of voxels in a skeleton branch, is preliminarily performed to remove very short branches (i.e., branches consisting of one or two voxels). Presence in the skeleton of short branches, coming into a branch point from which a longer non-significant branch originates, may prevent removal of a non-significant branch. In fact, to preserve skeleton connectedness, only peripheral branches (i.e., branches delimited by an end point) can be pruned. A non-significant branch can become peripheral, only after removal of the short branches, coming into one of the branch points delimiting it. The general strategy for both brute force and significance-based pruning involves tracing skeleton branches, starting from the end points, and counting voxels along the branches (with significant and non-significant voxels individually counted only for significance-based pruning). When the second
154 voxel delimiting the branch is encountered, a decision on pruning can be taken based on the length of the branch (brute force pruning) or on the ratio between the number of significant voxels and the total number of voxels (significancebased pruning). Brute force and significance-based pruning are performed only if the second extreme of the peripheral branch at hand is a branch point and, of course, if the condition on branch length or significance is satisfied.
Fig. 11. A handlebar, top left; its curve skeleton before pruning, top right; the curve skeleton after brute force pruning, bottom left, and after significance-based pruning, bottom right.
If all branches coming into a branch point are removed, a new peripheral branch is originated which, in turn, can be traced and possibly pruned. Of course, to avoid excessive pruning, information regarding the significance of the already pruned branches is taken into account during iterations of significance-based pruning. The degree of pruning that should be performed is very much dependent on the application. This means that both the value of the pruning thresholds and the number of times significance-based pruning should be iterated cannot be decided independently of the class of images. For an example of the performance of pruning, see the handlebar shown in Fig. 11, top left, its curve skeleton before pruning, top right, after brute force pruning, bottom left, and after one iteration of significance-based pruning, bottom right. When skeleton branches meeting in a cluster of branch points are pruned, the cluster is processed to remove voxels that have become unnecessary for connectedness preservation or cause thickening of newly created branch tips.
Fig. 12. The three basic skeleton configurations for jaggedness reduction.
155
A process aimed at reducing the jaggedness caused in CS by final thinning is also accomplished. Three basic configurations (shown in Fig. 12), each in twelve rotations, are taken into account; whenever one of these configurations occurs, the middle voxel (gray) is shifted into alignment with the other two voxels. 5. Conclusions In this Chapter, we have presented a skeletonization algorithm for 3D discrete objects in binary images. The nearly thin surface skeleton SS is first computed; then, the nearly thin curve skeleton CS is extracted from it; finally, CS is reduced to one-voxel thickness and a post processing aimed at pruning the skeleton to improve its manageability and stability under rotation is accomplished. SS has been computed by using D6. If a different distance is used, a different SS is obtained and, hence, a different CS. The differences are disregardable when working with elongated objects. The curve skeleton is a representation of the object with reduced dimensionality useful in a number of applications, especially when dealing with elongated objects, e.g., blood vessels. An example is shown in Fig. 13, where an MRA image is shown after binarization, left. The resulting curve skeleton before and after pruning is shown in the middle and to the right, respectively.
Fig. 13. Segmented blood vessels from MR angiography, left [by courtesy of Dr. Saha, MIPG, Dept. of Radiology, Univ. of Pennsylvania, Philadelphia, USA], the curve skeleton before and after pruning, middle and right, respectively.
Acknowledgements Thanks to Dr. Stina Svensson who co-authored many of the papers (and implemented the corresponding algorithms) on which this Chapter is based.
156
References [I] C. Arcelli, G. Sanniti di Baja. A one-pass two-operation process to detect the skeletal pixels on the 4-distance transform. IEEE Trans on PAMI, 11(4):411-414,. 1989. [2] G. Sanniti di Baja, S. Svensson. Surface skeletons detected on the D6 distance transform, in F.J. Ferri et al. eds., Advances in Pattern Recognition, LNCS 1121, Springer-Verlag, 387-396, 2000. [3] G. Bertrand, Z. Aktouf. A three-dimensional thinning algorithm using subfields. In: R.A. Melter and AY. Wu eds.,Vision Geometry III. Proceedings of SPIE 2356,1994. [4] K. Palagyi, A. Kuba. A parallel 3D 12-subiteration thinning algorithm. Graphical Models and Image Processing 61, 199-221,1999. [5] P.K. Saha, B.B. Chaudhuri, D.D. Majumder. A new shape preserving parallel thinning algorithm for 3D digital images. Pattern Recognition 30 (12), 1939-1955, 1997. [6] G. Borgefors, I. Nystrom, G. Sanniti di Baja. Surface skeletonization of volume objects. In P. Perner et al. eds., Advances in Structural and Syntactical Pattern Recognition, LNCS 1121, Springer-Verlag, 251-259, 1996. [7] G. Bertrand. A parallel thinning algorithm for medial surfaces. Pattern Recognition Letters 16, 979-986,1995. [8] G. Borgefors, I. Nystrom, G. Sanniti di Baja. Computing skeletons in three dimensions. Pattern Recognition, 32(7): 1225-1236,1999. [9] S. Svensson , I. Nystrom, G. Sanniti di Baja, Curve skeletonization of surface-like objects in 3D images guided by voxel classification, Pattern Recognition Letters, 23/12, 1419-1426, 2002. [10] D. Attali, A. Montanvert. Computing and simplifying 2D and 3D continuous skeletons. Computer Vision and Image Understanding 67 (3), 261-273, 1997. [II] F.F. Leymarie, B.B. Kimia. The shock scaffold for representing 3D shape. In: C. Arcelli et al. eds., Visual Form 2001, LNCS 2059. Springer-Verlag, 216-228, 2001. [12] S. Svensson, G. Sanniti di Baja, Simplifying curve skeletons in volume images, Computer Vision and Image Understanding, 90, 242-257, 2003. [13] G. Borgefors, I. Nystrom, G. Sanniti di Baja. Connected components in 3D neighborhoods. In M. Frydrych et al. eds., Proc. SCIA97, Lappeeranta, Finland, 567-572, 1997. [14] T. Y. Kong. A digital fundamental group. Computers & Graphics, 13(2): 159-166, 1989. [15] G. Bertrand, G. Malandain. A new characterization of three-dimensional simple points. Pattern Recognition Letters, 15:169-175, Feb. 1994. [16] P. K. Saha, B.B. Chaudhuri. 3D digital topology under binary transformation with applications. Computer Vision and Image Understanding, 63(3):418-429, 1996. [17] G. Borgefors. On digital distance transform in three dimensions. Computer Vision and Image Understanding, 64(3):368-376, 1996. [18] I. Nystrom, G. Borgefors. Synthesising objects and scenes using the reverse distance transformation in 2D and 3D. In C. Braccini, et al. eds., Image Analysis and Processing, LNCS 974, Springer-Verlag, 441-446, 1995. [19] G. Malandain, G. Bertrand, N. Ayache. Topological segmentation of discrete surfaces. International Journal of Computer Vision, 10(2): 183-197, 1993. [20] P.K. Saha, B.B. Chaudhuri. Detection of 3D simple points for topology preserving transformation with application to thinning. IEEE Trans, on PAMI, 16(10):1028-1032, 1994. [21] G. Sanniti di Baja, S. Svensson. Classification of two-voxel thick surfaces:a first approach. Internal Report 19, Centre for Image Analysis, Uppsala, Sweden, 2000. (available from the authors). [22] G. Borgefors, I. Nystrom, S. Svensson, G. Sanniti di Baja. Simplification of 3D skeletons using distance information. In L.J. Latecki et al. eds.,Vision Geometry IX, Proc. SPIE 4117, 300309, 2000.
C H A P T E R 2.4 DIGITAL DISTANCE TRANSFORMS IN 2D, 3D, AND 4D Gunilla Borgefors Centre for Image Analysis, Swedish University of Agricultural Lagerhyddsvagen 3, SE-752 37 Uppsala, Sweden e-mail: gunillaScb .uu.se
Sciences
Digital distance transforms have been used in image processing and analysis since the 1960s. Distance transforms are excellent tools for all applications regarding shape. They are, in fact, extensively used, especially in industrial and medical applications. At the same time, from the mid 1980s until today, there has been a rich literature that investigates distance transforms theoretically, constructs new ones, and improves computation algorithms. Despite this, distance transforms have not really been incorporated into the general image analysis toolbox. They are usually not mentioned at all - or the oldest ones (e.g., City block and Chessboard) are mentioned very briefly - in the basic books on image analysis used in education. One reason for the under-use of distance transforms could be that the oldest distance transforms are very rotation dependent, giving quite different results depending of the position of an object. The Euclidean distance transform is rotation independent up to digitisation effects, but often leads to complex algorithms where it is used. The compromise is the integer weighted distance transforms, that combines the simplicity of the old distance transforms with a reasonable rotation independence. Here, a large number of distance transforms will be described,with some of their properties and the simplest computation algorithms. 1 Introduction A digital distance transform (DT) is a way of structuring t h e interior of objects a n d / o r background of a binary image. Each element gets a value t h a t is equal t o t h e distance between itself and the nearest element in a set of pixels, i.e., the object elements get the distance t o t h e background. 157
158
One way, but not the only one, of looking at the DT value is that it tells when the element would be reached in an iterative erosion process. This imposed structure can then be used in many image analysis algorithms, either instead of iterative processes or on its own merit. The most used applications are probably skeletonizing and watershed algorithms, but also morphological operations, other shape manipulation, compression, template matching, path-finding and Voronoi-tessellation should be mentioned.Reverse DTs (rDT)are less common, but also useful for, e.g., recovering a shape from its skeleton. The DT concept was introduced already in the mid 1960s, [19, 20], but for a long time only the simple type of distance transforms represented by the City block and Chessboard distances were used. In fact, these are still often the only ones mentioned in general books on image processing and analysis. These distance are easy and efficient to compute, but have the disadvantage of being very direction dependent. Thus, results for a specific shape will be very different for different orientations. In the mid 1980s weighted distance transforms, which are much less direction dependent,were introduced. At about the same time, people began investigating and using the Euclidean distance transform. Distance transforms are easy to extend to higher dimensions. The computations to determine optimality does become more complex, but not the use of the DT itself, once it has been discovered and its properties have been determined. Today we have a good selection of distance transforms available. Which one to choose is of course dependent on the application, but it should be noted already here, that the more "exact" a DT is, the more complex the algorithm that uses it necessarily becomes. The simplest possible DT should always be chosen. This Chapter starts with the Basic concepts of DTs, and then presents a number of different DTs in 2D, 3D, and 4D. DT computation is left to a chapter of its own, as almost all DTs can be computed in similar ways. Other information can be spread over the image together with the DT, which if briefly mentioned. Finally, the choice of DT is discussed. 2 Basic C o n c e p t s Denote a digital shape S, and the complement of the shape S, where the sets S and S are not necessarily connected. A distance transformation converts the binary image to a distance image, or Distance Transform (DT). The image is a discrete structure and when considering digital distances it is best viewed as a graph structure, where the elements are the nodes and the neighbouring relation defines the arcs. A good underlying concept
159
for all DTs is the one proposed in [28]: Definition 1 The distance between two elements x and y is the length of the shortest path connecting x to y in an appropriate graph. Any digital distance is definable in the above manner. What should be determined is the neighbourhood relation, i.e., the arcs in the graph, and the lengths of these arcs. These lengths are called local distances. Note that this definition is not dependent on dimension or on which discrete grid is used. In 2D, if each pixel is connected only to its edge-neighbours and all arc lengths are set to 1, we get the City block distance. If each element is connected to every other element and the arc lengths are the Euclidean lengths of the corresponding vectors, we get the Euclidean DT. The DT is defined from the minimal paths in the graph as follows. Definition 2 Let x G S and Dp(x,y) between x and y. Then DT(x) = miu_ (y)es
be the length of the shortest path
Dp(x,y).
If the arc neighbourhoods of all elements are equal and symmetric, it is easy to prove that all DTs defined in this way are metrics. Symmetry means that all local distances between neighbours in the same relation should be equal, independent of rotation. Only symmetric DTs are treated here. To get "well-behaved" DTs, some constraints on the local distances are necessary. One useful property is semi-regularity. Definition 3 Consider two elements that can be connected by a straight path, i.e., by the homologue arc in each connection. If that path is minimal, then the resulting DT is semi-regular. If there are no other minimal paths, the DT is regular. Due to symmetry, when computing minimal path lengths in ZN between two elements, we can consider one element to be the origin and the other x, where xi > X2 > . . . > XN > 0. This convention will be used everywhere, unless otherwise stated. DTs can be divided into three classes, of increasing complexity. Path-generated, PGDT. Here, only the number of steps (i.e. arcs) in the minimal path is counted, so all arc lengths are set to 1. We get different DTs by using different neighbourhoods. A special case of PGDTs are the Octagonal DTs, where the neighbourhood size used differs for different steps in the path. Weighted, WDT. Here, the arcs in the neighbourhood get different lengths, depending on the neighbourhood relation. In practice, neighbourhoods larger than 5 ^ , where N is the dimension, are seldom used, even
160
if theoretical results do exist. The lengths can, and have been, optimised according to a number of different criteria, but the aim is to make a DT that is as rotation independent as possible. (These DTs are often called Chamfer DTs, from the chamfering algorithm that can be used to compute them.) Euclidean, EDT. Here, only one arc connects any two elements, and its length is set to its corresponding Euclidean length. The simplest algorithms that compute the EDT give minor errors, as only small neighbourhoods are used to propagate distance information. These are usually insignificant. However, the resulting DT is not a metric. A good way to characterise a DT is to look at its ball, i.e., all elements x with DT{x) < r, where r is the distance to a single, central, element. "Ball" should be understood as "disc" in 2D and as "hyper-ball" in 4D. The shape of the ball is a good indication both of the direction dependence (how "round" is it?) and of the approximation to the Euclidean distance values (what size is it?). All DTs can be used to define reverse DTs, rDTs. Though much less used than the DTs, rDTs were defined from the beginning, [19]. Definition 4 The reverse distance transform (rDT) is computed from a set of seed elements with DT values Si. Each element in the rDT has the value maxj[0, Si — Dp], where Dp is the length of the minimal path between the closest seed element and the element itself. Probably the most important type of elements to be identified in a DT are the "maxima," or centres of maximal balls. Definition 5 A maximal ball is a ball in S not completely covered by any other ball in S. A centre of maximal ball (CMB) is the centre element in the ball, labeled with its radius. Note that the definition is valid for any metric, even though the algorithms for identifying them differs. Note also that the union of the maximal balls is equivalent to the original shape. This means that if the seed elements in rDT consist of the set of CMBs, then S is the set of rDT elements x with DT(x) > 0. The set of CMBs is the only property of the DTs that will be covered here, both because it is very important and because it characterises the DTs. 3 Distance Transforms in 2 D In 2D, we will consider neighbourhoods up to 5 x 5. As the DT should be isotropic we have three different steps, a, b, c, in a total of 16 directions in the full 5x5 neighbourhood. Example steps from the origin for the different step types are: a = (1,0), b = (1,1), and c = (2,1), i.e. steps between
161
edge-neighbours, vertex-neighbours, and knight-neighbours, respectively. ("Knight-neighbour" because this is how the knight moves in chess.) As remarked before, all combinations of local distances a, b, and c result in metrics, but not all choices of a, 6, and c will give semi-regular DTs. Semiregular DTs are desirable for several reasons, not the least because general expressions for the minimal path lengths can be found for these DTs. 3.1 Path-generated
Distance
Transforms
The first DTs to be described where PGDTs, [19, 20], and the very first was the City block DT, presented 1966. Here, only edge-neighbours are used, so a = 1 (and b = c = oo). It is often denoted D4 (or d*), because it uses 4-connected paths between pixels. The City block DT is semi-regular and the length of the minimal path (between the origin (0,0) and (xi, x2), is
DJ = Xl+x2.
(1)
The maximum absolute difference, L°°, between the City block and the EDT in an M x M image is L°° = (2 - y/2)M « 0.5858M. In the Chessboard DT, pixels are linked to edge-neighbours and vertexneighbours, so a = b — 1. It is often denoted D8 (or ds), because it uses 8-connected paths between pixels. The Chessboard DT is semi-regular and the length of the minimal path is D* = Xl.
(2)
(Remember xi < x2 < 0.) For the Chessboard DT, L°° = (y/2 - l)M « 0.4142M. To illustrate some different 2D DTs, a running example in the shape of a pear will be used. The City block and Chessboard DTs of the pear are found in Fig. 1. In the DT, all CMBs are marked and the largest maximal balls in the pear body and in the leaf are encircled. The large ball gives a good indication of the characteristic ball of the DT, while the small ball shows that for short distances, all DTs are rather similar. For both the City block and the Chessboard DTs, the balls are squares, although differently oriented. The City block DT gives too large distance values and the Chessboard DT gives too small distance values compared to the Euclidean distance. They are also very rotation dependent. Using them alternatively results in a much less rotation dependent DT. Alternating them means that in the minimal path, vertex-neighbours are allowed only at some steps. The resulting DT, called an Octagonal DT, was first presented in 1968, [20]. In the simplest octagonal DT, vertex-neighbours are allowed only for even steps. This DT was proved to be a metric in [20]. It is also semi-regular.
162
Figure 1: City block, (l,oo) (left) and Chessboard, (1,1) (right) distance transforms. All CMBs are boxed. The largest maximal ball in the pear body and in the leaf are encircled by thick lines.
The length of the minimal path is D°ct = max{ xi,
-(xi +x2 + l)
(3)
where \_z\ denotes the greatest integer not exceeding z. For the Octagonal DT, L°° » 0.1180M for M > 5 and the same as for the Chessboard DT for smaller M. This Octagonal DT is found in Fig. 2. Not surprisingly, its balls are octagons. Note that Octagonal DTs are not necessary metrics, as this depends on the sequence of neighbourhoods. For example, if the neighbourhood order is reversed, thus allowing vertex-neighbours for odd steps, the resulting DT is not a metric. Other sequences of neighbourhoods can give better Octagonal DTs, see [12] for a complete investigation, but these become increasingly difficult to compute with sequential algorithms. It was suggested in [28], that the path generated only by the knight's move, c, might define an interesting and peculiar distance. This challenge resulted in an investigation into this perplexing PGDT, [13]. All pixels can be reached by knight paths, but it takes, e.g., three steps to move from a pixel to its edge-neighbour. The Knight DT balls are not simply connected. Even so, the Knight DT is both metric and regular, which should be a warning that more is needed for a practically useful DT. The Knight DT of the Pear example is found in Fig. 2.
163 J l 1 1|11 [TIITJ
m
IT
12 12 12 lj2 2 2
1 1 12 2 2 2 2 2 3 | 2 22 2 3QJ3 2 3 HH3 2 3 4 4 4 3
4[H 4
12 1 2 ^]2 ill
2 1 2 1
2 ^ 2 1 ^ ^ 2 [2] 2 if if
1 1 2 2
1 1 1
lQJ 1 1 1 4 , , ,1 1 1 _ 111 1 1 i l l lfTI I l I i\i\l 1 1 i l l ill 1 1 1 2 2 1 1 1 2 2 2 1i 1 112 2 2 2 11 112 2 3 2 11 inn 2i~si 3f¥ i 2|_2J 3|_2 2 2 2 3 3 2 2 2 2 3 3 3 2 2 1 i 1 2 2[2\ 3 3 3 [ U 2 2 1 "~ T\2 2 3 3 J? 3 3 2 2\l 3 3 2 2 1 1 2 2 3 3 Tj 2 2 3 3 3 3 3 3 2\2_ ; 2 2 ~2\3 3 3 [ U 2 2 2 1 1 2 2 2 2 2 2 2 2 2 Til 2 2 2_2_ 2_2 l l jfTI 1 1 1 1 1 1 1| 1 1 1 i i
a
1 4 5[S 1 5 6] 6 60 6 6 5 4 3 2 1
_Tpn
5 6 6 6
1 11 1
Figure 2: Octagonal (left) and Knight (right) distance transforms. All CMBs are boxed on the Octagonal DT, but only two on the Knight DT. The largest maximal balls in the pear body and in the leaf are encircled by thick lines. In the Knight DT the figures in the balls are in italic. Note that these balls are not simply connected.
3.2 Weighted
Distance
Transforms
It was soon discovered that weighting the neighbourhood arcs with suitable values make distance transforms much less rotation dependent, but no serious investigation into optimising these weights, or local distances, was done until the mid 1980s, [2, 3]. Since then, articles on WDTs are steadily produced, extending them in various directions. To have well-behaved and computable DTs, they should be semi-regular. In the 3 x 3 neighbourhood, this means that a straight horizontal/vertical path is shorter than a path between the same pixels using diagonal steps, and that a straight diagonal path is shorter than a path using horizontal/vertical steps. Thus two 6-steps (e.g. North-West + South-West) should be longer than two a-steps (West), 26 > 2a; and two a-steps (e.g. North + West) should be longer than one 6-step (North-West), 2a > 6. Similar reasoning in the 5 x 5 neighbourhood gives the regularity conditions there. WDTs are semi-regular iff a < 6 < 2a 2a < c,
36 < 2c,
for 3 x 3, c < a + 6 for 5 x 5,
and regular if the inequalities are strict.
(4)
164
For semi-regular DTs lengths of the minimal the paths can be computed. For the 3 x 3 neighbourhood: Dpix3 = (x1-x2)a
+ x2b.
(5)
For the 5 x 5 neighbourhood: r>5x5 = f x2(c-2a) + xia for 0 < x2 < x1/2 P \ x2(2b - c) + Xl(c - b) for x^2 < x2 < xx
.. [ )
The local distances a, b, c can now be optimised. The desired rotation independence is expressed as approximating the digital Euclidean distance as well as possible. The two most used optimisation criteria are: minimising the absolute difference from the Euclidean in an M x M image (L°° norm); and minimising the squared difference around a digital Euclidean circle of radius M {I? norm). The results are quite similar. Here, the L°° norm is used. The optimal local distance for the 3 x 3 neighbourhood becomes aopt = \ + | ( \ / 2 v / 2 - 2 ) « 0.9551, bopt = \/2-\ L
+ \{\/2yf2-2
- 1) « 1.3693, with
(7)
°° = \ - s ( \ / 2 > / 2 - 2 ) ~ 0.0449M.
In the 5 x 5 neighbourhood the expressions become much more complex, so only approximations are given here. Note that bopt can vary within a small interval without affecting L°°. aopt = 0.9859, bopt 6 [1.4002,1.4218], L°° = 0.0196M.
copt = 2.2078, with
, . '
{
Using real valued local distances is generally neither computationally nor algorithmically desirable. Instead, integer approximations of the optimal local distances are used. Candidate integer approximations are found by multiplying the optimal local distances by a scale factor and rounding to the nearest integer. Then the L°° norm is computed using the expressions used when finding the optimal values. In the 3 x 3 case, a = 2, b = 3, with L°° « 0.1349M, and a = 3, b = 4, with L°° w 0.0809M, are good approximations. The resulting DTs are denoted (2,3) and (3,4), respectively. These DTs of the pear example are found in Fig. 3. Their balls are octagons, which is natural, as distance information can spread in eight directions. The next better approximation is (8,11) with L°° w 0.0730M. The naive, but surprisingly often seen (1, y/2) DT has L°° « 0.0898M, so it is worse than (3,4), even though it uses real numbers.
165
0 0 4 03 02 [02 02 03 02 02 |O2O4O5O40; 02I02 02 02I [o203 05 0 06 04 02| 02 03 05 06 0 06 04 02| 02 04 06 0 0 0 7 05 03 02 02 04 06 08 0 0 8 06 04 02 02 04 06|08 |lo!bg|07 05 03 02 02 03J05 07 09 Q B 08 06|04 02 02[o406 08 10 0 10 08 0604|o2 070911 02 03 05 07 09 11 BJ 11 09 07 05 030 2 02 04 06 08 10 12 5H 12 10 08 06 04 02 02 04 06 08 10 1 2 0 1 2 1008 06 04 02 02 04 06 08 10 1 2 0 1 2 10 08 06 04 02 02 04 06 08 1 0 0 1 2 0 1 0 08 06 04 02 O2I04 06 08 0 10 10 1 0 0 0 9 07 O5I03 02 02|o4 0 0 7 08 08 08 08 08 0 0 0 6 ) o 4 02 02 O3I04 05 06 06 06 06 06 06 06|o5 03 02
03 03 04 03 03 06 07 06
|o3O4O700 04 07 08 0 0 06 0 8 0 0 1 0
1
03 04 04 04 04 04 04|04 03 02) 02 02[02 02 02|02 02 02 02|
06 09 0 0 1 1 06J09 12ll5ll2 03J04 07 10 1 3 0 0 1108 06|03 03 06 08 11 1 4 0 0 12 09 06 03 03 04 07 10 12 1 5 0 0 13 100704 0 3 03 06 08 11 14 1 6 0 0 14 11 08 06 03 03 06 09 12 1 5 0 0 0 15 12 09 06 03 03 06 09 12 15 1 8 0 1 8 15 12 09 06 03 03 06 09 12 1 5 0 1 8 0 15 12 09 06 03 03 06 09 0 1 4 1 5 1 5 15 0 0 1 0 07 04 0 3 06 08 10 11 12 12 12 1 2 0 0 0 8 06 03 04 06 07 08 09 09 09 09 09 08 07 04 03 04 06 06 06 06 06 06 06|o4 [o3|o3 03 03 03 03|03 03 03|
Figure 3: The (2,3) (left) and (3,4) (right) distance transforms. All CMBs are boxed. The largest maximal ball in the pear body and in the leaf are encircled by thick lines.
In the 5 x 5 case the integer approximation o = 5,6 = 7, c = 11, L°° « 0.0202M, is so good that no other integer values are worthy of consideration. The (5,7,11) DT of the pear is found in Fig. 4. The ball of this DT is a hexadecagon, (16 sides), as the distance information can spread in 16 directions, but this is not visible for the small example in the Figure. In [27, 4] a method was suggested for getting even better approximations using the integer WDTs. The final DT values are rescaled using a real valued scale factor, / . Rescaling does not affect the shape of the WDT balls, only their sizes, so rotation dependence is not changed by rescaling. However, if the Euclidean distance values are important, rescaling is useful. The best example is (2,3), where rescaling by / ~ 0.9412 gives L°° « 0.0588M. Note that 5 x 5 (5,7,11) is so near the optimal solution that rescaling has no significant effect. 3.3 Euclidean
Distance
Transform
Even though the PGDTs and WDTs are suitable for most purposes, some researchers do not accept any other DT than the Euclidean one. Decidedly, there are applications where only the EDT will do, so it should always be an option. The DTs presented so far can be computed and stored in the original image. This is not true of the EDT. In the final DT, each pixel (usually)
166
16 11|0705 18 14 lo|o5 20 15 10 05 |05 10 14 18 21 161107 05 05 0711 1621 22 18 14 10 05 05 10 14 18 22 25 20 15 10 05 05 10 15 20 25 25 20 15 10 05 05 10 15 20 25 05 10 15 20 25 0 3 0 0 25 20 15 10 05 05 1 0 1 5 0 2 2 25 25 25 2 5 0 1 6 1 1 0 7 05 05 10 14 16 18 20 20 20 2 0 0 0 1 4 1 0 0 5 05]o71011 14 15 15 15 15 15 14 1l[o7 05 05|05 0710 10 10 10 10 10 10)0705) |05 05 05 05 05 05 05K)5 05 05)1015
05 07 11 16
]0ll 0102 01 |0104 05 04 01 02 0 5 0 0 02 05 0 8 0 0 04 0 8 0 0 1 0 04 0 9 0 0 1 3 04 09 lsbdllT) 10 05|02 01 05 10 1 7 0 0 13 08 04]oi Ol 01 04 08 13 2 0 0 0 16 09 04 01 01 02 05 10 1 7 2 5 0 0 171005 0201 01 04 08 13 20 2 9 0 2 9 20 13 08 04 01 0104 09 16 25 3 4 0 3 4 25 1609 0401 01 04 09 1625 3 6 0 3 6 25 1609 0401 01 04 09 1 6 2 5 0 3 6 0 25 1609 0401 _ 0104 09 0 2 0 25 25 25 2 5 0 1 0 0 5 02C 01 04 08 10 13 16 16 16 1 6 0 0 0 8 04C Ollo2 04 05 08 09 09 09 09 09 08 05jo2 01 02 04 04 04 04 04 04 04J02 Olf 01 01 0 1 0 1 0 1 0 1 0 1 OlOlloiOll
Figure 4: The (5,7,11) and squared Euclidean distance transforms. All CMBs are boxed. The largest MBs in the pear body and in the leaf are encircled by thick lines.
has the squared EDT value, as this obviates the necessity for taking square roots and results in an integer DT. However, when computing the DT, at least when computing it in any reasonably efficient way, a vector v is stored in each pixel, where v\ is the horizontal distance in the minimal path and V2 the corresponding vertical distance. If v\ and V2 are always positive, we get the (standard) EDT, [11]. If they are signed, thus pointing directly at the starting pixel in the minimal path, we get the signed EDT, [29]. The squared EDT of the pear is found in Fig. 4. The ball of this DT is of course as circular as possible in the digital plane. It should be noted that it is not straightforward to compute the true EDT. If small neighbourhoods are used, and they usually are for efficiency reasons, small errors can occur, [11, 14]. Simply put, the reason is that distance information cannot be distributed in more than a limited number of directions even though the minimal path can have any direction. This happens in configurations where more than two "wave- fronts" meet. As the errors are very small, they might seem insignificant, and in many applications they are. However, the resulting DT is not a metric and it has been pointed out that for reconstruction based on labeled skeletons this non-metric property may actually produce wrong results, [14]. Much effort has been spent on constructing algorithms that give the true EDT.
167
3.4 Distance
Transforms
in Rectangular
Grids
Some imaging devices produce non-isotropic images, so that pixels are rectangles rather than squares. These images are often interpolated into a square grid before analysis. This creates new data that does not include any new information. It also makes the image larger and, hence, more memory and computation demanding. It would be better to accept the rectangular grid and instead use measurement and analysis methods that are optimised for it. Of the DTs, the PGDTs ones are clearly unsuitable, but WDTs can be optimised for the rectangular grid, [23]. Consider a rectangular grid, where each pixel has height 1 and width A > 1. There are now three different arcs in the 3 x 3 neighbourhood, the short edge-neighbour arc with local distance as, the long edge-neighbour arc, a,L, and the diagonal neighbour arc b. As before, L°° is optimised, but in a square image of size M x AM. The rectangular WDT is semi-regular if b < as + aL,
as
aL
(9)
The optimal local distances can be computed as functions of A. The results are: ,Ax 3A2 - A - (A - 1 ) V W + 1 , + asoPM) = 5A'-2A + 1 2Ay / (A + l ) ( v / A 2 T T - l ) 5A2 - 2A + 1
'
(10)
aLcpt(A) = Aa s (A), bopt (A) = Aas (A) + VA 2 + 1 - A,
with
L°°(A) = ( A - A a s ( A ) ) M . These solutions are valid for all A > 1. With A = 1, the equations reduce to the solutions for the square grid in Section 3.2. If A = 4/3 (a format occurring in cathode ray tubes), a good integer DT is as = 3, ai, = 4, b = 5; with scaling factor / = 1.056 we get L°° w 0.088. Unfortunately, L°° increases rapidly with M. For A = 3, a reasonable integer DT is as = 5,ai, = 15, b = 16; with scaling factor / = 1.134 we get L°° w 0.356. The EDT computation can be altered to take the elongatedness into account, which is easy in some computation algorithms and trickier in others. 3.6 Identifying
Centres
of Maximal
Balls
In the pear examples, Figs. 1-4, the CMBs have been marked in the DTs (except the Knight DT). These are identified by investigating the neighbourhood of a pixel, but the investigations are different for different DTs.
168
Path generated DTs. For PGDTs the CMBs are simply the locally maximal pixels, i.e., x is a CMB if DT(x)>DT(y), yeN(x)
(11)
where N(x) is the appropriate neighbourhood of x, i.e., the same as that used when computing the DT. Thus, in the Octagonal DT, the vertexneighbours are checked only if DT{x) is even. Weighted DTs. For integer WDTs in the 3 x 3 neighbourhood, the smallest distance between neighbours is larger than 1, so a pixel can be a CMD even if it has neighbours with higher values, [1]. By adding the local distances, this can be taken into account. For sufficiently large DT values, DT(x) >T,xisa CMB if DT(x) + a > DT(if), y G Ne(x) and DT(x) + b > DT(y), y G Nv(£),
^
}
where Ne{x) are the edge-neighbours and Nv(x) the vertex-neighbours of x. The larger a is, the larger T becomes. Condition (12) can be used also for DT(x) < T, if some DT values are replaced by "equivalent labels." For (2,3),T = 3 and all DT{x) = 2 should be changed to DT{x) = 1. For (3,4), T = 7 and values 3 and 6 should be changed to 1 and 5, respectively. For (8,11), T is quite high, T — 71, so many replacements are necessary. For WDTs in the 5 x 5 neighbourhood, the situation is more complex, [5]. For high DT values Condition (12) can be used if a check of knightneighbours is added, DT(x) + c > DT(y),
y G Nk(x),
(13)
where Ne(x) are the knight-neighbours of x. However, for values below the threshold T, simple replacement rules are not sufficient, so look-up tables are necessary. For the (5,7,11) DT, we have T = 61. See [5] for the complete algorithm. Euclidean DTs. For EDTs, CMB computation can only be accomplished by look-up tables, as there are no useful regularities in the DT. Pre-computed look-up tables is thus the only reasonable solution. A further complication is that checking neighbours in the 3 x 3 neighbourhood is enough only for small DT values. The larger the DT values are, the larger the neighbourhood that must be considered to identify CMBs becomes, as proved in [18], where Tables for quite high values are found. 4 Distance Transforms in 3D The three types of DTs in 3D have the same basic properties as the corresponding ones in 2D. Therefore, they will be only briefly covered here. As in the 2D case, neighbourhoods up to 5 x 5 x 5 will be considered.
169
4-1 Path-generated
Distance
Transforms
The three simplest PGDTs are those defined by the 6 closest neighbours (face-neighbours); the 18 closest neighbours (face- + edge-neighbours); and the 26 closest neighbours (face- + edge- + vertex-neighbours). These DTs are denoted D6, D18, and D26. D6 is the 3D equivalent of the city block distance and D26 the 3D equivalent of the chessboard distance. The lengths of the minimal paths of the three DTs are, from [15], D6
=
X!+X2
+ X3,
D18 = x1 + max{0,[1-xi\x* D26 =
(14) + x
xi.
*\},
(15) (16)
D6 and D26 are semi-regular but D18 is not, [6]. Some D 1 8 minimal paths contain edge-neighbour steps in two different directions, so there may be up to four different steps in a path. In fact, the D18 has some similarities with the Knight DT in 2D, so even if it may seem to be a good choice at first, it should be avoided. The balls of the three DTs are octagons, cuboctagons, and cubes, respectively. The maximum differences from the EDT in an M x M x M image are: Lg° = 1.2679, Lf§ = 0.4142, and L°° = 0.7321. Of course, these three DTs can be combined into octagonal DTs as in 2D, see [15, 2]. CMBs for PGDTs are the local maxima in the appropriate neighbourhoods. 4-2 Weighted
Distance
Transforms
Computations of optimal WDTs in 3D are found in [6, 25]. The optimality criterion is minimising the difference from the EDT in an M x M x M image. In a full 5 x 5 x 5 neighbourhood we have six different steps, a, 6, c, d, e, / , in a total of 98 directions. Example steps from the origin for the different step types are: a = (1,0,0), a = (1,1,0), c = (1,1,1), d = (2,1,0), e = (2,1,1), and / = (2,2,1). In the 3 x 3 x 3 neighbourhood, WDTs are semi-regular iff a < b < la 26 < 2c < 36.
(17)
These inequalities are not enough to determine expressions for the minimal paths, as they are different for a + c < 26 and a + c > 26. For larger neighbourhoods, using more steps, the hyper-volumes of semi-regularity are found in [25]. They are split into many parts, generating many different expressions for minimal path lengths. Here, only the recommended integer DTs for the 5 x 5 x 5 neighbourhood are given. In the 3 x 3 x 3 case, a = 3, 6 = 4, c = 5 with L°° = 0.1181M
170
is a good choice. This is, in fact, the most commonly used 3D WDT. Its ball is a polyhedron delimited by 24 equal, kite-shaped tetragons. (The next better combination is a — 8, b = 11, c = 13 with L°° = 0.1073M.) Adding the e-step gives a = 3, 6 = 4, c = 5, e = 7 with L°° = 0.0809M; adding d gives a = 8, b = 11, c = 14, d = 18, e = 20 with L°° = 0.0653M; and finally, adding / gives a = 10, b = 14, c = 17, d = 22, e = 24, / = 30 with L°° = 0.0408M. As in 2D, the integer DTs in 3D can also be rescaled to make the absolute L°° smaller, but again, this will not affect rotation dependence. CMBs can be computed using the same principles as for WDTs in 2D. 4-3 Euclidean
Transforms
The EDT is, in principle, easily extendible to 3D. Three values have to be stored in each voxel, but the final DT is again the squared distance values. Computational complexity goes up much more that for the other DTs, and the number of configurations that can cause small errors increases. Computing CMBs requires pre-computing look-up tables. 4-4 Non-isotropic
grids
The question of handling non-isotropic grids is more relevant in 3D than in 2D. 3D Imaging systems often generate images with one resolution in the "slices" and another, lower, between slices. Examples are tomographic and many microscopic images. This results in elongated voxels, where two sides are equal, length 1, and the third is longer, length A. Interpolation to cubic voxels creates much more data, without adding information. To make it possible to work directly in the elongated grid, optimal DTs have been computed for such images, [24]. There are five different arcs in the 3 x 3 x 3 neighbourhood: the short and the long face arcs as, ai, the short and the long edge arcs bs, bi, and the diagonal arc c. Expressions for the optimal local distances are frankly horrible - the reader is referred to [24]. The rotation dependence of the DTs increases rapidly with A; already for A = 1.57, L°° is doubled compared to the cubic grid. However, for small A, or if the DT is used for ordering the voxels, rather than for getting measurements to compare with real values, this may work very well. One example: for A = 1.5, as = 4, a^ = 7, bs = 6, &L = 8, c = 9 gives L°° = 0.169; after rescaling. 5 Distance Transforms in 4 D Results regarding data sets in 4D digital space, Z 4 , are being found more and more in the literature, both regarding theory and emerging applications. Therefore, DTs in 4D are becoming useful, [7]. In 4D each hyxel
171
(hyper-volume element) has neighbours in the 3 x 3 x 3 x 3 neighbourhood: 8 volume neighbours, step a, 24 face- neighbours b, 32 edge-neighbours c, and 16 vertex-neighbours d. The PGDTs corresponding to the City Block and Chessboard DTs are the Ds and D^o DTs. Their balls are hyper-octahedra and hypercubes, respectively, and the maximal difference from the Euclidean in an M xM x M x M image are L|° = 2.000 and L§§ = 1.000. A WDT in 4D is semi-regular iff a < b, b<2a,
b < c, 2c < 3b, c< d, 3d < Ac.
(18)
As in 3D, these conditions are not enough to determine the expressions for the minimal path - in fact, there are now eight different cases. Optimal local distances have been computed, [7]. The recommended integer WDT in 4D is a = 3, b = 4, c = 5, and d = 6, with Lf = 0.1835. The EDT can of course also be extended to 4D. 6 Computation There are essentially three approaches to computing DTs. The first is chamfering where an additive mask is passed over the image in two raster scans, thus "chamfering" the DT from the original binary image. The second is variations on wave-front propagation, using bucket sort techniques, where only elements at the "front" are considered for computation. The third is parallel algorithms, using special computer architectures. Many of the latter type of algorithms were published a decade ago, but with the efficiency of today's standard computers they are now seldom seen. They will not be discussed here. 6.1
Chamfering
All WDTs and all PGDTs that use a single neighbourhood can be computed using two raster scans through the image, independently of dimension or neighbourhood size. Initialise the binary image by setting elements in the shape S to oo and the elements in the background S to 0. The easiest way to illustrate the algorithm is by its additive masks, see Fig. 5 for the 5 x 5 2D case. The two masks are simply the 5 x 5 neighbourhood split in two parts, where the split occurs between visited and non-visited pixels in the raster scan. After initialisation, the image is scanned by the first, forward, mask from left to right (long arrow) and from top to bottom (short arrow), putting the mask centre , marked "+0," over each pixel. The values in the mask are added to the corresponding values in the image, and the centre element is set to the minimum of the five sums. In the second scan, from right to left and from bottom to top, the backward
172
+c +c +c +b +a +b+c +a +0
+0 +a +c +b +a +b +c +c +c "*
Figure 5: Masks for 2D chamfering. neighbourhood as in 2D.
For higher dimensions, split the
mask is used. The only difference is that now the centre pixel must be treated as the other pixels in the mask.The algorithm can be made more efficient by applying the mask only to pixels with values > 0 (as they will not change values) and skipping the central pixel in the minimisation using the forward mask (it is then valued 0 or oo). If a smaller neighbourhood than 5 x 5 is used, of course only the relevant parts of the masks are used. In higher dimensions, it is still enough to use two raster scans in opposite directions through the image, where the masks consist of the already visited elements in the neighbourhood. The Reverse DTs, rDTs, are easily computed using a similar scheme. The image is initialised with seed elements with positive values (radii of maximal balls) and the rest of the elements are set to 0. The same masks scan the image in the same way as for the DT, but mask values are subtracted from the pixels below them and the maximum of the values is taken as the new central pixel value. Octagonal DTs and their rDTs can also be computed using raster scans and masks that are subsets of the 3 x 3 neighbourhood, but more scans are needed. For the 2D octagonal DT that uses two neighbourhoods alternately, four scans are necessary. In 3D, octagonal DTs that uses two neighbourhoods alternatively need eight scans. See [11, 2] for details. The almost correct EDT can also be computed using raster scans, but also here, more scans are necessary. The reason is as follows: Using a 3 x 3 mask, information can be spread in eight directions. This is enough for 3 x 3 WDTs as, there are only eight possible steps from each pixel. But in the EDT, the direction from a pixel is not limited. The 3 x 3 forward mask from Fig. 5 covers directions in the interval [7r/4, TT], while the backward mask covers directions [57r/4, 27T], leaving the intervals [0,7r/4] and [-7T, 57r/4] inaccessible. Enlarging the masks helps, but does not solve the problem. To cover all directions, at least three masks are necessary, see Fig. 6, from [17]. After initialisation to (oo, oo) and (0,0) of pixels in S and S, respectively, the image is scanned using the three masks, and setting the central pixel to the vector with the shortest length. This will
173
+(1,1)
+(1.1) +(0,1) +(1.1) + (1,0) +(0,0)
+(1,0) + (0,0) +(1.1) + (0,1) +(1,1)
+ (0,0) + (1-0) +(1.1)
+"
Figure 6: Masks for sequential computation of the 2D EDT. For higher dimensions, more masks have to be added.
give a good approximation of the EDT. Approximation, because the errors mentioned in Section 3.3 will be present. If 5 x 5 masks are used, the errors will be smaller, but unless the masks' size is equal to the size of the image itself, errors can occur. In 3D, the EDT needs at least four masks, covering all directions, [17]. Generally, in dimension N, the upper limit of the number of masks is 2N, [17] again. For EDTs, computing the rDT is not straightforward, due to the unlimited lengths of the steps in minimal paths. It is not enough to use "reverse" masks. A recent algorithm for the rEDT is found in [9]. 6.2 Wave-front
propagation
For the WDTs, chamfering is so efficient that it may at first seem unnecessary to create other schemes for sequential computers. In fact, it was not until constrained DTs began to be used that this was done, [16]. A constrained DT is an image with three types of elements, object S and background S as before, but also obstacle elements, than cannot be passed. Distance information has to travel around these obstacles. Constrained DTs can be computed using the masks in Fig. 5, but now the number of necessary scans is unknown, as it is necessary to keep scanning until no pixel values change. With spiral channels facing the "wrong" way, this might mean many scans. A 2D wave-front algorithm is found in [16]. If this approach is taken, the integer DTs can be computed without arithmetic - it is enough to use a counter and check if neighbouring elements have a specific value, see [8]. This wave-front approach is of course also ideal for octagonal DTs, where it is no longer necessary to restrict the complexity of the sequence of neighbourhoods used, see [20, 12] for optimal sequences. For the EDTs, there is a wealth of papers on different algorithms for computing it efficiently and correctly. Many of the early ones use special architectures or pseudo-parallel algorithms, neither of which is desirable. Efficiency is not really a problem today (except perhaps in high dimensions), but correctness is still an issue. Using wave-front propagation is
174
usually the route taken for computing the true EDT, see [10] for a good algorithm. When propagating distance information, it is possible to also propagate other image information. An early example is Voronoi tessellation from a set of seeds (not necessarily isolated points), where the identity of the seed is propagated [2]. Grey-level information can also be propagated, [26]. Generally the term salience DTs is used when propagating additional information, [21]. In that paper there are many examples of kinds of information that can be spread, e.g., edge strength and curvature. If using chamfering, spreading such information needs many raster scans, just like the constrained DT, but when using wave front propagation there is no extra effort. Lately, these types of techniques of wave-front propagation of various properties have been given much prominence, but now under the name Fast Marching. All theory is there done in continuous space, and in the end the expressions are digitised and applied in ZN. When effort is spent on this discretisation, the results are good. It is thus a pity that the final result often is equivalent to a salience DT where the rotation dependent City Block DT is used for spreading some property that could have been derived in a much simpler way. 7 Comparing Distance Transforms A large number of possible DTs have been presented. Which one should be chosen for a specific application? The computation of CMBs indicates a general property of DTs - the more "exact" they are, the more complex the algorithms using them usually becomes. Therefore, one should always balance the simplicity of the DT with the necessary accuracy. The PGDTs using only the closest neighbours, i.e. City Block, D6, and Ds in 2D, 3D, and 4D, respectively, are useful in many applications, specially if the objects are thin, so only short distances occur. Using larger neighbourhood PGDTs is seldom a good idea, as too many elements get the same DT value, which can create problems. If using wave-front propagation, the octagonal DTs can be considered, but if using chamfering the WDTs are better. The size of the neighbourhood used determines both complexity and accuracy. In 3D (3,4) and (5,7,11) are good, in 3D a = 3, b = 4, c = 5; a = 3, b = 4, c = 5, e = 7; and a = 10, b = 14, c = 17, d = 22, e = 24, / = 30. In some cases, the EDT is necessary. If so, do not forget to consider the signed EDT, where the vector in each element points directly at the source of its distance value.
175
References Arcelli C. and Sanniti di Baja, G. (1988). Finding Local Maxima in a Pseudo- Euclidean Distance Transform, Computer Vision, Graphics and Image Processing 4 3 , pp. 361—367. Borgefors, G. (1984). Distance Transformations in Arbitrary Dimensions, Computer Vision, Graphics and Image Processing 27, pp. 321-345. Borgefors, G. (1986). Distance Transformations in Digital Images, Computer Vision, Graphics and Image Processing 34, pp. 344-371. Borgefors, G. (1991). Another Comment on 'A Note on "Distance Transformations in Digital Images'", Computer Vision, Graphics and Image Processing 54, pp. 301-306. Borgefors, G. (1993). Centres of Maximal Discs in the 5-7-11 Distance Transform, In: Proc. 8th Scandinavian Conference on Image Analysis, Troms0, Norway, pp. 105-111. Borgefors, G. (1996). On Digital Distance Transforms in Three Dimensions, Computer Vision and Image Understanding 64 pp. 368-376. Borgefors, G. (2003). Weighted Digital Distance Transforms in Four Dimensions. Discrete Applied Mathematics 125, pp. 161-176. Borgefors, G. Hartmann, T., and Tanimoto, S.L. (1990). Parallel Distance Transforms on Pyramid Machines: Theory and Implementation, Signal Processing 2 1 , pp. 61-86. Coeurjolly, D. (2003).
Vision,
Das, P.P.(1992). Best Simple Octagonal Distances in Digital Geometry, Journal of Approximation Theory 68, pp. 155-174. Das, P.P. and Chatterji, B.N. (1988). Knight's Distance in Digital Geometry, Pattern Recognition Letters 7, pp. 215-226. Forchhammer, S. (1989). Euclidean Distances from Chamfer Distances for Limited Distances, In: Proc. 6th Scand. Conf. on Image Analysis, Oulu, Finland, pp. 393-400. Okabe N., Toriwaki J., and Fukumura, T. (1983). Paths and Distance Functions on Three-Dimensional Digitized Pictures, Pattern Recognition Letters 1, pp. 205-212.
176 [16] Piper, J. and Granum, E. (1987). Computing Distance Transformations in Convex and Non-Convex Domains, Pattern Recognition 20, pp. 599-615. [17] Ragnemalm, I. (1993). The Euclidean Distance Transform in Abritrary Dimensions, Pattern Recognition Letters 14, pp. 883-888. [18] Remy, E. and Thiel, E. (2003). Look-Up Tables for Medial Axis on Squared Euclidean Distance Transform, In: Discrete Geometry for Computer Imagery, Eds. Nystrom, Sanniti di Baja, and Svensson, Lecture Notes in Computer Science 2886, Springer-Verlag, pp. 224-235. [19] Rosenfeld, A. and Pfaltz, J.L. (1966). Sequential Operations in Digital Picture Processing, Journal of the ACM 13, pp. 4 7 1 - 494. [20] Rosenfeld, A. and Pfaltz, J.L. (1968). Distance Functions on Digital Pictures, Pattern Recognition 1, pp. 33-61. [21] Rosin, P.R. and West, G.A.W. (1995). Salience Distance Transforms, Graphical Models and Image Processing 57, pp. 483-521. [22] Sethian, J.A. (1999). Level Set Methods and Fast Marching Methods, Cambridge University Press, Cambridge, U.K. [23] Sintorn, I.M. and Borgefors, G. (2001). Weighted Distance Transforms in Rectangular Grids, In: Proc. 11th International Conference on Image Analysis and Processing, Palermo, Italy, Eds. Ardizzone and Di Gesu, pp. 322326. [24] Sintorn, I.M. and Borgefors, G. (2004). Weighted Distance Transforms for Volume Images Digitized in Elongated Voxel Grids, Pattern Recognition Letters 25, pp. 571—580. [25] Svensson, S. and Borgefors, G. (2002). Digital Distance Transforms in 3D Images Using Information From Neighbourhoods up to 5 x 5 x 5, Computer Vision and Image Understanding 88 pp. 24-53. [26] Toivanen, P. (1996). New Geodesic Distance Transforms for Grey-Scale Images, Pattern Recognition Lettersl7, pp. 437-450. [27] Vossepoel, A.M. (1988). A Note on: 'Distance Transformations in Digital Images', Computer Vision, Graphics and Image Processing 4 3 , pp. 88-97. [28] Yamashita, M. and Ibaraki, T. (1986). Distance Denned by Neighborhood Sequences, Pattern Recognition 19, pp. 237-246. [29] Ye, Q.Z. (1988). The Signed Euclidean Distance Transform and Its Applications, In: Proc. 9th International Conference on Pattern Recognition, Rome, Italy, pp. 495-499.
CHAPTER 2.5 COMPUTING GLOBAL SHAPE MEASURES
Paul L. Rosin School of Computer Science, Cardiff University P.O. Box 916, Cardiff, CF24 3XF, UK E-mail: [email protected] Global shape measures are a convenient way to describe regions. They are generally simple and efficient to extract, and provide an easy means for high level tasks such as classification as well as helping direct low-level computer vision processes such as segmentation. In this chapter a large selection of global shape measures (some from the standard literature as well as other newer methods) are described and demonstrated. 1. Introduction Formed by the eye and therefore, like the eye, Full of strange shapes, of habits and of forms, Varying in subjects as the eye doth roll To every varied object in his glance Love's Labour's Lost, Shakespeare Shape is an important aspect of visual interpretation, and over the years many schemes have been proposed for representing the shape of objects (see reviews in Refs. 1, 2, 3, 4). Sometimes the goal is to recover three dimensional information from shape content present in the two dimensional scene (see examples in both computer vision5'6 and human vision7), but in many instances full 3D recovery is not essential for interpretation. In fact, as Biederman and Ju showed8, the human visual system performs very well even in the absence of many cues such as intensity, colour, texture, etc. In computer vision there are many uses for 2D shape analysis. These range from high level applications such as content based image retrieval; even the earliest systems such as QBIC9 used circularity, eccentricity, moment invariants, etc., while more recent systems (particularly those aimed for trademarks and logos) use many more shape measures10. At the other end of the spectrum, shape analysis can be incorporated as a component in low level computer vision processes to direct image segmentation11, line grouping12, etc. While some techniques focus on the interior components (e.g. pixels) of a region, and others prefer to analyse the boundary (see Pavlidis1), both provide essentially the same in177
178 formation if only binary regions are considered (i.e. interior colour and texture are ignored), and we will not distinguish between the two. 2. Shape Decomposition One approach to analyse shape is to break up the region into simpler parts such as triangles, convex polygons, etc. An example of partitioning is shown in Fig. 1 where the global shape measure of convexity has provided the criterion for selecting appropriate dividing cuts. The overall convexity of the decomposition (into a specif! ed number of parts) is maximised, although individual parts need not be perfectly convex.
Fig. 1. The gray lines indicate the cuts used to partition the shapes into a pre-specifi ed number of roughly convex parts.
Fig. 2. The original curve is approximated in turn by straight line segments, circular arcs, and parabolic arcs. Breakpoints (i.e. the ends of primitives) are marked by crosses.
More often, segmentation is applied to the boundary which is broken up into a sequence of basic curve primitives. A popular scheme is polygonal approximation, which is straightforward and effi cient, but other primitives are possible. Examples are shown in Fig. 2 of approximation by straight lines, circular arcs 13 , and parabolic arcs 14 . All approximations are found by recursively splitting the curve at the point of maximum deviation from the fi tted primitive (no parameters are required). It can be useful to represent the boundary by multiple primitive types. For example, roughly straight sections are represented by straight line segments, and curved sections by arcs. This results in a more compact representation which may also be more effective for matching. Since the different representations generally yield different selected breakpoints
179
Fig. 3. The polygonal approximation in Fig. 2 is post-processed to replace appropriate sequences of lines by circular arcs, and by elliptic arcs.
the results in Fig. 2 cannot be easily merged. An approach that circumvents this diffi culty is to sequentially fi nd each representation, starting with approximating the pixel data by the lowest level primitive type, and then replacing primitive sequences by higher level representations. The results of this are shown in Fig. 3 where circular or elliptical arcs have been substituted for sequences of lines if they provide a better fi t.
Fig. 4. The curvature extrema (marked by crosses) and point of inflection (boxes) are detected in the input data (top row). Each curve section is fi tted by straight line segments, elliptical arcs, and superelliptical arcs (second row). The curvature extrema can also be used to determine codon labels (bottom row). The results of adaptive smoothing are also shown in the bottom row; line thickness is proportional to degree of smoothing.
180 An alternative approach is to determine the breakpoints by a fi xed rule such as points of inflection as shown in Fig. 4. This enables all sections of curve to be easily represented by all the curve primitives (in this case straight line segments, elliptical arcs, and superelliptical arcs)15. The curvature extrema can be used to decompose the curve. One example is Asada and Brady's curvature primal sketch16 which analyses the scale space tree of curvature extrema of a region's boundary to extract the following curve primitives: corner, smooth join, crank, end, and bump. Another well known scheme is the set of codons which were defi ned by Hoffman and Richards17 as sections terminating in curvature minima, and divided into fi ve types depending on the curvature extrema within the section. Their catalogue of codon types was extended by Rosin18 to incorporate straight sections and endings, and an example of the application of this extended codon set is given in Fig. 4.
Fig. 5. The global natural scales of the original pixel data (left). The second natural scale was used for processing in Fig. 4.
Detecting curvature extrema requires some fi ltering to eliminate responses due to noise. One possibility is to determine adaptively an appropriate degree of smoothing for local sections of curve. This can be done in a similar manner to fi tting curve primitives as described above14, and results are shown in Fig. 4. Alternatively, rather than represent each part of the curve at a single scale multiple scales can be employed. The results of representing the hand data at its "natural scales"19 are presented in Fig. 5. The curve has been smoothed over a range of scales, and those scales maximising a signifi cance measure are considered to contain the most useful information, and are retained. 3. Global Shape Measures Although the multiple component shape representations described above provide a detailed shape description they have the disadvantage that matching such descriptions is complicated. The same applies to other representations such as medial axes, shock graphs 20 , etc. Algorithms such as dynamic programming, interpretation trees, multiple cliques in association graphs, etc. are often used to fi nd the correspondences between subparts, based on their multiple properties (e.g. orientation, position, curvature) and the spatial inter-relationships between groups of subparts. In contrast, describing regions by global shape measures enables much simpler match-
181
ing to be used (e.g. a Euclidean distance classifi er), and the many standard property based classifi ers in the literature can be directly used without modifi cation (e.g. support vector machines, neural networks, etc.). Global shape measures (sometimes called global scalar transforms) just return a single value to describe a region, and therefore they obviously do not provide a one-to-one mapping between shapes and values; for example, there are infi nitely many perfectly convex shapes, all of which should yield the same value for any given convexity measure. Nevertheless, if a region is described by a combination of shape measures this should be suffi cient to provide discrimination between different categories of shapes. The main disadvantage of global shape measures is that they do not cope well with occlusion, since this can change the global nature of a region's shape to an arbitrary degree. There are many standard global shape measures in the literature, e.g. moments, Fourier descriptors, auto-regressive models. These types of approaches have been proven to be effective, although a downside is that their values are often not easily understandable. In many applications it is preferable if the region descriptors can be analysed by the domain experts, as this aids validation and development of the computer vision system. In this chapter we shall concentrate on descriptors that have direct intuitive meanings, e.g. convexity. A wide variety of global shape measures are described in the following sections, and a variety of methods have been used in their construction. They are generally based on the deviation (in some sense) of a region from the best matching perfect instance of the shape. However, while it is usually straightforward to determine whether a region satisfi es a shape type or not (e.g. is a region convex, or is a region a rectangle?) there are no universally accepted standard methods for measuring the discrepancy between the region and the canonical form. Nevertheless, since real data tends to be noisy and distorted it is necessary to be able to cope with errors, and so continuous shape measures are required rather than binary shape tests. Some of the approaches use a combination of primary measurements of the region such as perimeter, area, and diameter (for which there are accepted precise defi nitions although there may still be diffi culty in estimating the values in the presence of noise). From these primary measurements the shape measures can be built, e.g. by defi ning them as algebraic combinations of the primary measurements. Other approaches explicitly fi t a model to the data (using least squares, moments, Fourier transform, etc.) and measure the discrepancy of the fi t using the geometric distance between the points and the model or between parameters (e.g. Fourier descriptors) of the fi tted model and the region. In the extreme case, similarity can even be measured by determining how many pixels need to be moved, and how far, to change one shape into another21. One of the goals of the shape measures is to be able to capture a class of forms which may be fairly general, but also to be specifi c enough to discriminate against other shapes. A shape measure needs to be invariant under certain transformations, but the particular type of invariance will depend on the application. For instance, a convexity measure should ideally be invariant under any perspective projection. On the other hand, a shape measure to enable digits "6" to be distinguished from "9" in an OCR application should not be rotationally
182 invariant. A further requirement is that the shape measures should be robust in the presence of noise. In other words, small changes to a region's form should have small effects on the values computed by the shape measures. As we shall see, several of the standard shape measures fail on this count. 3.1. Eccentricity and Elongatedness There are various formulations which measure some notion of the aspect ratio of a region, and they go under several names such as eccentricity, elongatedness, etc. A common and convenient measure uses the central moments fipq, and is computed as the ratio of the lengths of the axes of the image ellipse ^ ° + ^ + V ^ ° r ^ ) 2 + 4 ^ i , w h i c h c a n b e s i m p l i . A120+M02 - V (A«20 -M02 ) 2 + 4 M I !
2
fi ed and reformulated as \/(/x2o - /xo2) + 4/z2^/ (/x2o + M02) to provide a measure in the range [0,1]. Alternatively, an example of the boundary based techniques is to compute the ratio of the region's width and diameter. This requires determining the diameter (i.e. the longest chord between pairs of boundary points), which can be done in linear time 22 . The width which is the longest chord perpendicular to the diameter can then be easily found. Alternatives such as the ratio of the shortest Feret diameter to the longest chord perpendicular to it are also used.
Fig. 6. The spike in the rectangular region on the left makes it appear less elongated than the shell on the right according to the aspect ratio measure, but not according to the moments based measure.
Because the aspect ratio is based on maximum distances it is more sensitive to narrow area spikes than the moments based measure, as is demonstrated in Fig 6. 3.2. Circularity The most common approach to measuring circularity (also called compactness, shape facp2
tor, etc.) is to compute ^ - where P and A are the perimeter and area of the region. Low values correspond to circular shapes (47t in the limit). Although this is a convenient formulation, there are problems due to the artifacts caused by digitisation23'24, and moreover, since perimeter estimates are extremely sensitive to noise (prone to overestimation), then in such cases the value of the measure becomes inflated, incorrectly assigning non-circular values to noisy circles. An alternative that avoids this problem was suggested by Haralick25. Given the distance R between the centre and any point on the perimeter his measure is ^ , where high values
183 correspond to circular shapes. Note that the centre should be calculated as the centroid of the polygon and not the average boundary coordinate values, as otherwise it will be biased in the event of noise being unevenly 2distributed along the boundary. Figure 7 demonstrates how the ^p - measure is confused by the circle's boundary texture, and incorrectly estimates the pear as the more circular of the two regions, Although the undulation superimposed on the circle causes its value of ^ to decrease compared to a perfect circle it is still higher than the pear's value, which is therefore correctly estimated as the less circular of the regions.
Fig. 7. The undulations on the circle cause it to be measured as less circular than the pear according to ^ - , but not according to £&.
3.3. Squareness Squareness is not such a common shape measure in computer vision, but it is straightforward to defi ne such a measure in various ways. One approach is to use complex Fourier descriptors26. If the boundary of the region is given by the points (xj,yj) then the coordinates are represented by complex numbers Zj = Xj + iyj and applying the discrete Fourier transform leads to the complex coeffi cients comprising the descriptors q. = a\. + ibk = ~k X)m=i Zm e x P {—ilirmk/N). Just the magnitude will be used rfc = \/a\ + bf,, and it can be shown that r 0 represents the region's centroid, and rj indicates the size of the region. To make the descriptors scale invariant they are normalised as Wk = rk/r\. Bowman et al.27 use w_3 to measure squareness. Here we modify this approach slightly to include the harmonics and also divide to take into account the remaining harmonics which do not contribute to squareness: (w_3 + w_ 7 + w _ n + •••)/ Z)vtg{-i,o,i} wi- Computing the FFT takes 0{ N log N) time. 3.4. Ellipticity Voss and SiiBe describe a method for fitting geometric primitives by the method of moments28. The data is normalised into a canonical frame (which for an ellipse they take as the unit circle), by applying an affi ne transformation. Applying the inverse transformation to the primitive (i.e. the circle) produces the fi tted ellipse. An ellipticity measure can be calculated as the RMS of the differences between the normalised moments of the data
184 (m'ij) and the moments of the canonical primitive (ra^) where only the moments not used to determine the normalisation are included: 1/ (1 + Yli+j where di are the conic distance approximations of the N boundary points to the ellipse to the ellipse, need to be made scale invariant. This is done by weighting by the square root of the region's area A, giving the ellipticity measure32: 1/ (l + J^TJE) .
QQ Fig. 8. The region is shown with thefitted ellipse using the LMedS method (inliers plotted bold) and Voss and SiiBe's moments method.
Fig. 9. The two regions show discrepancies between the two ellipticity measures. The LMedS method rates die cat as more elliptical while Voss and SiiBe's method considers the zigzag region to be more elliptical.
185
As shown in Figs. 8-9 the two approaches fi t quite different ellipses. In the fi rst example (Fig. 8) this does not substantially affect the measured ellipticity values - the overall errors of fi t are similar. Figure 9 shows more difference where the LMedS fi ts most of the cat fairly closely, and therefore assigns it a higher ellipticity value than the zigzag region, whereas Voss and SuBe's method ranks the zigzag region as more elliptical than the cat. 3.5. Triangularity Triangles remain triangular under affi ne transformations, and this naturally leads to affi ne moment invariants being employed to measure triangularity. We will describe the methods for computing moments based on the region's interior pixels33, but similar methods are available that operate directly on the region boundary (line moments).34 To avoid using higher order moments (which are less reliable) we use the simplest affi ne moment invariant35 of the triangle to characterise it: I\ =
Wo, 0
2
* | ^ " , where aPQ are the Moo
central moments. For all triangles it can be shown that h = j^g- The triangularity measure defi ned in Ref. 32 is normalised to [0,1] as 108ii if I± < -^ and 10 g 7 otherwise. A second approach fi ts a geometric triangle model to the data, and measures the error between the model and data. Fitting is performed by fi nding a three line polygonal approximation of the boundary. The summed error E of points along the boundary is computed, and the fi nal triangularity measure is defi ned in the same manner as the previous ellipticity measure: 1/ ( l + KjEj. The computation complexity of this measure depends on which polygonal approximation algorithm is used; complexities range from linear to
0(N3).
Fig. 10. The moment based triangularity measure incorrectly ranks the bird between the two triangles, but the polygonal approximation algorithm does not.
A weakness of the moment based approach is that other shapes can yield the same moment invariant (since an infi nite number of moments are required to encode all the information in a region). This can leads to errors in the shape measure as shown in Fig. 10. 3.6. Rectangularity The standard approach to measuring rectangularity is to use the ratio of the area of the region X to the area of its minimum bounding rectangle (MBR), i.e. B r " m B L • The MBR of a convex polygon can be calculated in linear time by Toussaint's36 "rotating caliper" method. Since the convex hull of a simple polygon can be found in linear time the overall algorithm remains linear.
186 A weakness of the MBR is its sensitivity to narrow protrusions (while it is relatively insensitive to intrusions). If a modifi ed MBR is found that need contain only most of the region then it should be more robust in the presence of small area deviations in the boundary 32 ' 37 . This robustifi ed MBR is fitted by iteratively minimising the functional area(MBRJX)+ar^(X/ MBR) _ ^
^
o p e r a t i o n s c a n b e d o n e [n
O(NlogN)
time38. The
functional provides a trade-off between forcing the rectangle to contain most of the data while keeping the rectangle as small as possible. The resulting functional is subtracted from one and provides a rectangularity measure in [0,1].
< Fig. 11. Two regions with their bounding boxes, both the standard MBR and the robustifi ed MBR.
Figure 11 demonstrates how the MBR fi tted by the robust method is a more appropriate description of the gross shape of the regions, and consequently leads to more intuitively correct results. The robustifi ed MBR measure ranks the cat as more rectangular while the standard MBR measure ranks the zigzag region are more rectangular since the cat's standard MBR needs to be very large (compared to the size of the cat) to include the protruding limbs. 3.7. Rectilinearity
Fig. 12. Although all the polygon's angles are very close to ±90° the polygon should not be considered as rectilinear.
A relatively new shape measure is rectilinearity - which is the degree to which a polygon's angles conform to ±90°. However, directly measuring angles can lead to counter-
187 intuitive results (see Fig. 12), and so Zunic and Rosin39 defi ned the following measure. Let us denote the perimeter of X under the L\ and L2 norms as Vi(X) and V2{X). That is, for each straight line segment e = [(xi,yi), (0:2,2/2)] making up the boundary of X its length is calculated as h(e) = \x\ - x2\ + |j/i - 3/21 for the L\ norm and h{e) = \/{xi ~ x2)2 + (yi - yi)2 for the L2 norm. Now let Vi(X,a) be the L\ perimeter obtained by first rotating X by the angle a with the origin as the centre of rotation. It can be shown that X is rectilinear if and only if Vi(X,a) = \flV2{X) for some a. This leads to a measure of rectilinearity max ^ ^ , ' ° l which is normalised to obtain a value in the range (0,1]: ae[0,2-7r]
V2V2(X)
7
—7-7= ( max ^ i X ' " \ - 2^L I. While it is possible to solve for a symbolically it is K *-2v/2 ^Qe[0,27r] V^MX) * J J J more straightforward to numerically maximise the measure over [0,2n] which can be performed effi ciently by multi-resolution sampling of a. 3.8. Sigmoidality A sigmoidality measure determines how much a region is S-shaped. The methods described here preprocess the region to extract a single centreline curve which is the focus of all further analysis. This is easily performed by smoothing the region until the skeleton (obtained by any thinning algorithm) is non-branching. A modifi ed version40 of Fischer and Bunke's 41 technique is to fi t the cubic y = aa 3 + bx + c after the centreline is fi rst rotated so that its principal axis lies along the X axis. The correlation coeffi cient p is used to measure the quality of fi t. Inverse correlation is not expected, and so the value is truncated at zero. We now describe an alternative method for computing sigmoidality based on a less restrictive characteristic of the sigmoid, namely its single, central point of inflection. For perfect data its presence would be easy to test, but in practise, with real data the sensitivity of curvature estimation to noise makes this diffi cult. Therefore, instead of identifying zero crossings of curvature the overall distribution of curvature values along the curve is analysed. The curvature at each point is estimated as Kj using kernels of the Gaussian function and its fi rst two derivatives. Separating the positive and negative curvature values as if K >0 - _ / ° \ — K, otherwise othe
f 0 if Ki < 0 \ Ki otherwise
l
then the positive and negative curvature values are summed over the curve to the left and right respectively of the midpoint y . In addition the total curvature is computed for normalisation purposes: N
A+ = Yi"t
K
A- = Y. i
N
s= x >
j=i
The sections of positive and negative curvature should be restricted to either side of the inflection point and so the quantity A+ + A~ should be large. Also, the amount of positive
188
curvature on the left should equal the (absolute) value of the negative curvature on the right, and the discrepancy is measured by \A+ — A~\. These values are scaled and combined to obtain the following measure [2 (A+ + A") /S - 1] [1 - \A+ - A~\ /S]. To cope with the curve bending in either direction it is analysed both for Kj and —«;,, and the larger of the two values returned.
Fig. 13. The regions are shown with their extracted centrelines. Next to each region is its centreline (aligned with the Y axis) and the fi tted cubic. The cubic fi t measure rates the fi sh more sigmoidal than the '5".
The weakness of the cubic fi t is demonstrated in Fig. 13 in which it rates the fi sh as more sigmoidal than the digit "5". This is because the fi sh's centreline is well fi t by a cubic, even though it is almost straight, while the shape of the digit does not conform to the smooth, symmetric cubic. The curvature method correctly assigns the digit a higher sigmoidality value since its model is more general.
3.9. Convexity In a similar approach to the MBR method for rectangularity, convexity can be measured with respect to its convex hull. If we denote the convex hull of region X by CH(X) then the standard convexity measure is defined as area(cH(X)) • ^ n e c o n v e x n u ^ °f a simple polygon can be found in linear time42 and so the overall computational complexity of the measure is linear. Again, like the MBR, the convex hull is sensitive to protrusions but not indentations, and so would estimate the left region in Fig. 14 as more convex then the right region even though the former has greater deviations from the underlying rectangle.
Fig. 14. The convex hull based convexity measure considers the left shape to be the more convex, while 2uni£ and Rosin's measure more correctly determines the right shape to be the more convex.
189 Zunic and Rosin43 proposed a scheme based on perimeter rather than area measurements that responds more equally to intrusions and protrusions. Instead of being based on the convex hull it utilises the minimal rectangle with edges parallel to the coordinate axes which includes X, denoted by R ( X ) . Also, let R(X, a) be the minimal rectangle with edges parallel to the coordinate axes which includes X rotated by an angle a around the origin. X is convex if and only for any choice of the coordinate system the Li perimeter of X equals the perimeter of the minimal rectangle whose edges are parallel to the coordinate axes and which includes X, i.e., Vi(P,a) = ^(R-C-P, <*)) for any a e [0,27r]. This leads to the following convexity measure: min 7? (x a) whose values are in the range [0,1]. There is a 0(N2) procedure for the analytic computation of the measure, but like the rectilinearity measure it is convenient to estimate it numerically. 3.10. Symmetry There are many schemes for detecting and quantifying symmetry 44,45 . A simple scheme for measuring n-fold rotation symmetric is to measure the amount of overlap between the region and rotated versions of itself. This can be averaged over n—1 rotations by increments of 2£ radians. If the result of rotating X by a is T(X, a) then the symmetry measure is n-arla(X) ^i=i o,rea(X n T (X, ^ p ) ) . The easiest approach is to use the centroid of X as centre of rotation, but of course for imperfectly symmetric regions this may not be the best centre. An alternative is to optimise the symmetry measure as a function of the centre of rotation, although this is considerably more computationally expensive. Even without the optimisation stage this approach to symmetry detection is more expensive than the previous measures encountered so far since the polygon intersection stage takes 0(N2) time 38 . The two measures of symmetry (with and without optimisation of the centre of rotation) are demonstrated in Fig. 15. Both successfully detect the leftmost region as perfectly symmetric. However, the occlusion in the middle region causes the centroid (shown as the white circle) to be a poor estimate of the centre of rotation and therefore rates this region as less symmetric than the rightmost region. The iterative fi tting method correctly determines the centre of rotation (shown as the black circle) and consequently rates this region as more symmetric than the rightmost region.
Fig. 15. Both measures of symmetry assign the leftmost region maximal values. For the middle region the centroid (white circle) is a poor estimate of the centre of rotation (black circle) and therefore its use leads to this region being rated as less symmetric than the rightmost region. The iterative fi tting method correctly determines the centre of rotation and determines the middle region to be more symmetric than the rightmost region.
190 3.11. Chirality Chirality is the degree of left or right handedness of a region, and is a useful measure in many disciplines46 apart from computer vision47. A chiral region cannot be superimposed on its mirror image, and this directly leads to a measure of chirality similar to the symmetry measure above. Since the axis of mirror symmetry is not known the overlap is maximised over all possible orientations through the centre, which as above can also be optimised. If the mirror image of X with the line of reflection through (xc,yc) at angle a is M.(X, a,xc,yc) then the chirality measure is m a x a i I n 9 c {1 — area(X n M.(X, a, xc, yc))} since chiral shapes have minimal overlap. Like symmetry detection this measure is computationally expensive.
Fig. 16. The left hand region is chiral while the other two regions are not.
Figure 16 shows three regions whose chirality was measured with and without optimisation of the centre of line of reflection. The leftmost region gets a high score from both since there is no mirror reflection that produces any signifi cant overlap with the region. Although the centroid of the middle fi gure is displaced due to occlusion both measures give zero chirality values. Reflecting about the horizontal line through the centroid is suffi cient to get perfect overlap. Finally, the noisy boundaries in the rightmost region cause a fairly small chirality value (0.027) to be initially estimated using the centroid. The chirality value is subsequently refi ned to zero (to six decimal places) by optimising the centre of line of reflection. 4. Conclusions Many shape measures have been described in this chapter, but this is not an exhaustive list. Many alternative measures for these shapes exist, as well as measures for other, specialised shapes. The majority of the measures have linear computational complexity in the number of (boundary or interior) points. Some are sensitive to noise or small area protrusions; while more robust methods are available (and were described) they are often more computationally expensive. As can be seen, not all measures are specifi c, but can also produce their peak response for shapes other than their target shape (e.g. compare the two triangularity measures). Deciding on the most appropriate measures depends on their suitability for particular applications. To conclude, Figs. 17-27 demonstrate applying all the shape measures described in
191
this chapter to a variety of regions. The regions are presented2 ranked into descending order p according to each measure, except for width/diameter and ^ - which are put into ascending order for easier interpretation. These results help provide a qualitative understanding of the performances of the measures; more quantitative comparisons are given in the context of using the shape measures for classifi cation in Refs 10, 32, 37, 39,40,43,48.
sj(M20 - M02)2 + 4/ifj/ (M20 + M02)
width/diameter Fig. 17. Eccentricity and Elongatedness
P2/A
Haralick's HR/CTR measure Fig. 18. Circularity
References 1. T. Pavlidis. A review of algorithms for shape analysis. Computer Graphics and Image Processing, 7(2):243-258, 1978.
192
normalised w-3 and harmonics Fig. 19. Squareness
•tl*MI-lt*»W*t* Voss and SuBe's 1/(1 + 5Zi+7<4(mi,-
—m
ij)2)
measure
Fig. 20. Ellipticity Rosin's LMedS fit measure
measure based on first affine invariant
measure based on error of three line polygonal fit Fig. 21. Triangularity 2. S. Marshall. Review of shape coding techniques. Image and Vision Computing, 7(4):281-294., 1989. 3. S. Loncaric. A survey of shape analysis techniques. Pattern Recognition, 31(8):983—1001,1998. 4. D. Zhang and G. Lu. Review of shape representation and description techniques. Pattern Recog-
193
area(X)/area(MBR)
, _ greg(MBR / X ) + g r e g ( X / MBR) orea(XnMBR)
Fig. 22.
Rectangulanty
P i (X,a)
vr-2V2 V«e[0,2^] V^f 2 (X) max ———— Fig. 23.
2v/2
± LJL
^
Rectilinearity
Cubic fit
Curvature distribution
Fig. 24. Sigmoidality mrion, 37(1): 1-19, 2004. 5. H.G. Barrow and J.M. Tenenbaum. Retrospective on interpreting line drawings as threedimensional surfaces. Artificial Intelligence, 59(l-2):71-80, 1993. 6. D.P. Mukheijee, A. Zisserman, and J.M. Brady. Shape from symmetry - detecting and exploiting
194
area(X)/area(CH(X))
min
P 2 (R(X,a))/-Pi(A',a)
ae[0,27r]
Fig. 25. Convexity
2-fold rotational symmetry - not optimising over centre of rotation
2-fold rotational symmetry - optimising over centre of rotation
3-fold rotational symmetry - optimising over centre of rotation Fig. 26. Rotational Symmetry symmetry in affine images. Phil. Trans. Royal Soc. London, series A, 351(1695):77-106, 1995. 7. Norman J.F. and Raines S.R. The perception and discrimination of local 3-d surface structure from deforming and disparate boundary contours. Perception & Psychophysics, 64(7): 11451159,2002. 8. I. Biederman and G. Ju. Surface versus edge-based determinants of visual recognition. Cognitive Psychology, 20:38-64, 1988.
195
optimising over rotation and centre of rotation Fig. 27.
Chirality
9. M. Flickner et al. Image and video content: The QBIC system. IEEE Computer, 28(9):23-32, 1995. 10. J.P. Eakins, J.D. Edwards, J. Riley, and PL. Rosin. A comparison of the effectiveness of alternative feature sets in shape retrieval of multi-component images. In SPIE Conf. Storage and Retrieval for Media Databases, pages 196-207, 2001. 11. M.A. Shahin and S.J. Symons. Lentil type identification using machine vision. Canadian Biosystems Engineering, 45:3.5-3.11, 2003. 12. D.W. Jacobs. Robust and efficient detection of salient convex groups. IEEE Trans, on Patt. Anal, and Mack Intell, 18(l):23-37, 1996. 13. PL. Rosin and G.A.W. West. Non-parametric segmentation of curves into various representations. IEEETrans. on Patt. Anal, and Mack Intell, 17:1140-1153, 1995. 14. P. L. Rosin. Non-parametric multi-scale curve smoothing. Int. J. Pattern Recognition and Al, 8:1381-1406, 1994. 15. P.L. Rosin. A hierarchy of representations for curves. In Arcelli et al., editor, Aspects of Visual Form Processing, pages 465-474, 1994. 16. H. Asada and M. Brady. The curvature primal sketch. IEEE Trans, on Patt. Anal, and Mack Intell, 8(1):2-14,1986. 17. D. Hoffman and W. Richards. Parts of recognition. Cognition, 18:65-96, 1984. 18. P. L. Rosin. Multi-scale representation and matching of curves using codons. CVGIP: Graphical Models and Image Processing, 55:286-310, 1993. 19. P. L. Rosin. Representing curves at their natural scales. Pattern Recognition, 25:1315-1325, 1992. 20. P.J. Giblin and B.B. Kimia. On the local form and transitions of symmetry sets, medial axes, and shocks. Int. Journal of Computer Vision, 54(1): 143-156, 2003. 21. H. Sanchez-Cruz and E. Bribiesca. A method of optimum transformation of 3d objects used as a measure of shape dissimilarity. Image and Vision Computing, 21(12):1027-1036, 2003. 22. F.P. Preparata and M.I. Shamos. Computational Geometry. Springer-Verlag, 1985. 23. A. Rosenfeld. Compact figures in digital pictures. IEEE Trans, on Systems, Man and Cybernetics, 4:221-223, 1974. 24. Bogaert J., Rousseau R., Van Hecke P., and Impens I. Alternative area-perimeter ratios for measurement of 2D-shape compactness of habitats. Applied Mathematics and Computation, 111:7185, 2000. 25. R.M. Haralick. A measure for circularity of digital figures. IEEE Trans, on Systems, Man and Cybernetics, 4:394-396, 1974. 26. G.H. Granlund. Fourier preprocessing for hand print character recognition. IEEE Trans, on Computers, 21:195-201, 1972. 27. E.T. Bowman, K. Soga, and T. Drummond. Particle shape characterization using Fourier analysis. Geotechnique, 51(6):545-554, 2001. 28. K. Voss and H. SuBe. Invariant fitting of planar objects by primitives. IEEE Trans, on Patt. Anal, and Mack Intell, 19(l):80-84, 1997.
196 29. P.L. Rosin. Further five-point fit ellipse fitting. CVGIP: Graphical Models and Image Processing, 61(5):245-259,1999. 30. P.L. Rosin. Ellipse fitting using orthogonal hyperbolae and Stirling's oval. CVGIP: Graphical Models and Image Processing, 60(3):209-213, 1998. 31. A.W. Fitzgibbon, M. Pilu, and R.B. Fisher. Direct least square fitting of ellipses. IEEE Trans, on Pan. Anal, and Mack Intell, 21(5):476-480, 1999. 32. P.L. Rosin. Measuring shape: EUipticity, rectangularity, and triangularity. Machine Vision and Applic, 14(3):172-184, 2003. 33. S. Maitra. Moment invariants. Proc. IEEE, 67:697-699, 1979. 34. M.H. Singer. A general approach to moment calculation for polygons and line segments. Pattern Recognition, 26(7):1019-1028, 1993. 35. J. Flusser and T. Suk. Pattern recognition by affine moment invariants. Pattern Recognition, 26:167-174, 1993. 36. G.T. Toussaint. Solving geometric problems with the rotating calipers. In Proc. IEEE MELECON •83, pages A10.02/1-4,1983. 37. P.L. Rosin. Measuring rectangularity. Machine Vision and Applic, 11:191-196, 1999. 38. M. de Berg, M. van Kreveld, M. Overmars, and O. Schwarzkopf. Computational Geometry: Algorithms and Applications. Springer-Verlag, 2nd edition, 2000. 39. J. Zunic and P.L. Rosin. A rectilinearity measurement for polygons. IEEE Trans, on Patt. Anal. andMach. Intell., 25(9):1193-1200, 2003. 40. P.L. Rosin. Measuring sigmoidality. In Computer Analysis of Images and Patterns, volume 2756 of LNCS, pages 410-417, 2003. 41. S. Fischer and H. Bunke. Identification using classical and new features in combination with decision tree ensembles. In J.M.H. du Buf and M.M. Bayer, editors, Automatic Diatom Identification, pages 109-140. World Scientific, 2002. 42. D. McCallum and D. Avis. A linear algorithm for finding the convex hull of a simple polygon. Inform. Process. Lett., 9:201-206, 1979. 43. J. Zunic and P.L. Rosin. A new convexity measurement for polygons. IEEE Trans, on Patt. Anal. andMach. Intell., 26(7):923-934, 2004. 44. J.J. Leou and W.H. Tsai. Automatic rotational symmetry determination for shape analysis. Pattern Recognition, 20(6):571-582, 1987. 45. G. Marola. On the detection of the axes of symmetry of symmetric and almost symmetric planar images. IEEE Trans, on Patt. Anal. andMach. Intell., 11(1):104-108, 1989. 46. M. Petitjean. Chirality and symmetry measures: A transdisciplinary review. Entropy, 5:271-312, 2003. 47. Y. Hel-Or, S. Peleg, and D. Avnir. Characterization of right handed and left handed shapes. Computer Vision, Graphics and Image Processing, 53(2):297-302, 1991. 48. M.L. Hentschel and N.W. Page. Selection of descriptors for particle shape characterization. Part. Part. Syst. Charact., 20:25-38, 2003.
C H A P T E R 2.6
T E X T U R E ANALYSIS W I T H LOCAL B I N A R Y PATTERNS
Topi Maenpaa & Matti Pietikainen Department of Electrical and Information Engineering, Infotech Oulu, University of Oulu P.O. Box 4500, 90014 University of Oulu, Finland E-mail: [email protected], [email protected]
The LBP operator is a theoretically simple yet very powerful method of analyzing textures. Through its recent extensions, it has been made into a really powerful measure of image texture, showing excellent results in terms of accuracy and computational complexity in many empirical studies. The LBP operator can be seen as a unifying approach to the traditionally divergent statistical and structural models of texture analysis. Texture is described in terms of micro-primitives (textons) and their statistical placement rules. Optionally, the primitives may be coupled with a complementary measure of local image contrast, which measures the strength of the primitives. In this chapter the relation of the LBP operator to other texture analysis methods is explained. The chapter shows how the LBP combines aspects of statistical and structural texture analysis, and why it is called a "unifying approach". The theoretical foundation of the operator is explained starting from a definition of texture in a local neighborhood. A number of extensions to the basic operator are also introduced. The extensions include three different multi-scale models, an opponent color operator and a rotation invariant version. Finally, a number of successful applications of the operator are highlighted. 1. I n t r o d u c t i o n T h e local binary p a t t e r n (LBP) texture operator was first introduced as a complementary measure for local image c o n t r a s t 1 . T h e first incarnation of the operator worked with the eight-neighbors of a pixel, using t h e value of t h e center pixel as a threshold. An L B P code for a neighborhood was produced by multiplying t h e thresholded values with weights given t o the corresponding pixels, and summing u p t h e result (Fig. 1). Since t h e L B P was, by definition, invariant t o monotonic changes in gray scale, it was supplemented by an independent measure of local contrast. Fig. 1 shows how t h e contrast measure (C) was derived. T h e average of the gray levels below t h e center pixel is subtracted from t h a t of t h e gray levels above (or equal to) the center pixel. Two-dimensional distributions of t h e L B P and local contrast measures were used as 197
198 Multiply
Threshold
" " 5
4
3
1
4
3
1
1
2
0
3
0
1
0
1
1
0
8
1
32
2
64
4
1
16
8
128
0
2
4 0
0
128
L B P = 1+2+4+8+128 = 143 C = (5+4+3+4+3)/5 - (l+2+0)/3 = 2.8 Fig. 1.
Calculating the original LBP code and a contrast measure.
features. The operator was called LBP/C, and very good discrimination rates were reported with textures selected from the photographic album of Brodatz 2 1. The current form of the LBP operator, described in Sec. 3, is quite different from this basic version: the original definition is extended to arbitrary circular neighborhoods, and a number of extensions have been developed. The basic idea is however the same: a binary code that describes the local texture pattern is built by thresholding a neighborhood by the gray value of its center. The operator is related to many well-known texture analysis methods. The relations are summarized in Fig. 2. N pixels instead of pixel pairs
G r a y level co-occurrences
N-tuples
Multiple displacements, dimensionality reduction
Global binarization,/ oriented tuples /
Multidimensional co-occurrence distributions Differences of absolute gray values
Signed g r a y level d i f f e r e n c e distributions
\
Binary texture co-occurrence spectrum
Fig. 2.
Eight-neighbors, \ local quantization ^ into three levels Texture unit texture spectrum
Local binarization A arbitrary circular \ neighbor sets \
Signs of the differences. arbitrary circular neighbor sets
Textons
z
Generalization to gray-scale
G a b o r filtering
.ocal quantization Into two levels, arbitrary circular neighbor sets
LBP
Micro-textons, universal texton vocabulary
Clustering of fitter responses, statistics of best matches Gray-scale (macro-)texton distributions
LBP in the field of texture analysis operators.
2. A Unifying Approach to Texture Analysis Texture analysis methods have been traditionally divided into two categories. The first one, called the statistical or stochastic approach, treats textures as statistical
199 phenomena. The formation of a texture is described with the statistical properties of the intensities and positions of pixels. Co-occurrence statistics and difference histograms, researched by Haralick et al.3, Weszka et al.4, and Unser 5 , can serve as simple examples of statistical texture measures. These kinds of texture models work best with stochastic microtextures. As shown in Fig. 2, the LBP can be treated as a special case of a multi-dimensional co-occurrence statistic. The stochastic formulation of texture is based on a model in which a texture is viewed as a sample of a two-dimensional stochastic process describable by its statistical parameters 6 . Usually, as in MRF models, the interactions between pixels are assumed to be mainly of a local type 7 . Cross and Jain 8 write: "The brightness level at a point in an image is highly dependent on the brightness levels of neighboring points unless the image is simply random noise". Jain and Karu 9 note that texture is characterized not only by the gray levels of pixels, but also by local gray value "patterns". The second category, called the structural approach, introduces the concept of texture primitives, often called texels or textons. To describe a texture, a vocabulary of texels and a description of their relationships is needed. The goal is to describe complex structures with simpler primitives, for example via graphs. Structural texture models work well with macrotextures with clear constructions. Through the pioneering work of Julesz 10 and Beck et al.11, primitive-based models have been widely used in explaining human perception of textures. Until recently, this paradigm was not, however, extended to gray-scale textures. Malik et al.12 reintroduced the term "texton" in a study of texture segmentation. In their model, multi-dimensional features are created by filtering with a Gabor filter bank, and representative vectors (i.e. textons) are found with fc-means clustering. The texton vocabulary is created once for a set of textures, and used in describing all textures in it. Distributions of textons are used for texture description. Later, a number of people, for example Varma and Zisserman 13 , have used similar models for recognizing textures in the CUReT 1 4 database. It will later become apparent how the LBP can be seen as an instance of the structural texture analysis model. In the structural approach, the spatial structure of a texture is emphasized. According to Faugeras and Pratt 6 , the deterministic approach considers texture a basic local pattern that is periodically or quasi-periodically repeated over a region. Cross and Jain 8 assume these primitives to be of some deterministic shape with varying sizes. In their definition of the structural approach, placement rules specify the positions of the primitives with respect to each other and on the image. The problem with two distinct approaches has been recognized, most recently by Sanchez-Yanez et al.15: "Any texture contains both regular and statistical characteristics. In practice one can find textures between two extremes, completely periodic and completely random ones. That is why it is difficult to classify texture by one single method." That is also why unified models, in which the deterministic and non-deterministic parts of a texture field are separated, have been proposed, even
200
though Tomita and Tsuji 16 , p. 100 suspect: "There is no unique way to analyze every texture." Francos et al.17 use a Wold-like decomposition to accomplish this task. This method does, however, have a number of shortcomings. First, Wold-like decomposition requires textures to be realizations of regular homogeneous random fields. Second, the decomposition requires the extraction of a harmonic component which is prominent only in highly regular textures, and badly affected by 3-D distortions 18 . Third, the method does not actually unify the stochastic and structural approaches, as the concept of a texton is neglected. Rather, it provides a means of analyzing the stochastic and deterministic parts of a texture separately. The LBP method can be regarded as a truly unifying approach. Instead of trying to explain texture formation on a pixel level, local patterns are formed. Each pixel is labeled with the code of the texture primitive that best matches the local neighborhood. Thus each LBP code can be regarded as a micro-texton. Local primitives detected by the LBP include spots, flat areas, edges, edge ends, curves and so on. Some examples are shown in Fig. 3 with the LBPS,.R operator. In the figure, ones are represented as white circles, and zeros are black.
V ^ V Spot
cx^jo
Q-o-*
zx^jd
CLQJ»
Spot/flat
Line end
Edge
Corner
Fig. 3. Different texture primitives detected by the LBP.
The combination of the structural and stochastic approaches stems from the fact that the distribution of micro-textons can be seen as statistical placement rules. The LBP distribution therefore has both of the properties of a structural analysis method: texture primitives and placement rules. On the other hand, the distribution is just a statistic of a non-linearly filtered image, clearly making the method a statistical one. For these reasons, it is to be assumed that the LBP distribution can be successfully used in recognizing a wide variety of texture types, to which statistical and structural methods have conventionally been applied separately. 3. LBP a n d its Extensions In this section, the LBP operator is first derived from a general definition of texture in a local neighborhood. The non-parametric classification principle used in conjunction with the operator is explained in Sec. 3.2. A number of extensions to the LBP texture operator are also presented. In Sec. 3.3, a rotation invariant version is derived, and the number of rotation invariant codes is reduced by introducing the concept of uniformity. A discussion about the complementing roles of contrast and texture patterns is presented in Sec. 3.4. In Sec. 3.5, three ways of extending
201 the LBP to multiple resolutions are presented. Finally, Sec. 3.6 describes how an opponent color version of the LBP operator can be constructed. 3.1.
Derivation
The derivation of the LBP follows that represented by Ojala et al.19. Due to the lack of a universally accepted definition of texture, the derivation must start with a custom one. Let us therefore define texture T in a local neighborhood of a gray-scale image as the joint distribution of the gray levels of P + 1 (P > 0) image pixels: T =
t(gc,g0,...,gp~i),
(1)
where gc corresponds to the gray value of the center pixel of a local neighborhood. gp(p = 0 , . . . , P — 1) correspond to the gray values of P equally spaced pixels on a circle of radius R (R > 0) that form a circularly symmetric set of neighbors. This set of P+1 pixels is later denoted by Gp. In a digital image domain, the coordinates of the neighbors gp are given by (xc + Rcos{2irp/P),yc-Rsm(2irp/P)), where (xc,yc) are the coordinates of the center pixel. Fig. 4 illustrates three circularly symmetric neighbor sets for different values of P and R. The values of neighbors that do not fall exactly on pixels are estimated by bilinear interpolation. Since correlation between pixels decreases with distance, much of the textural information in an image can be obtained from local neighborhoods.
-P-—
p=8,
R=1.0
P=12,
R=2.5
p—lQI-il—
p=16,
R=4.0
Fig. 4. Circularly symmetric neighbor sets. Samples that do not exactly match the pixel grid are obtained via interpolation.
If the value of the center pixel is subtracted from the values of the neighbors, the local texture can be represented — without losing information — as a joint distribution of the value of the center pixel and the differences: T = t(gc,go-gc,---,9p-i-gc)-
(2)
Assuming that the differences are independent of gc, the distribution can be factorized: T ttt{gc)t(g0-gc,...,gp-i-gc). (3)
202
In practice, the independence assumption may not always hold true. Due to the limited nature of the values in digital images, very high or very low values of gc will obviously narrow down the range of possible differences. However, accepting the possible small loss of information allows one to achieve invariance with respect to shifts in the gray scale. Since t{gc) describes the overall luminance of an image, which is unrelated to local image texture, it does not provide useful information for texture analysis. Therefore, much of the information about the textural characteristics in the original joint distribution (Eq. 2) is preserved in the joint difference distribution 20 : Tttt(g0-gc,...,gP-i-gc).
(4)
The P—dimensional difference distribution records the occurrences of different texture patterns in the neighborhood of each pixel. For constant or slowly varying regions, the differences cluster near zero. On a spot, all differences are relatively large. On an edge, differences in some directions are larger than in others. Although invariant against gray scale shifts, the differences are affected by scaling. To achieve invariance with respect to any monotonic transformation of the gray scale, only the signs of the differences are considered: T w t(s(g0 - gc),...,
s(firp_i - gc)),
(5)
where S{X)
=
\0X<0.
<6>
Now, a binomial weight 2 P is assigned to each sign s(gp — gc), transforming the differences in a neighborhood into a unique LBP code. The code characterizes the local image texture around [xc,yc)\ p-i
LBPp,R(:rc, yc) = Y, s(gP - gc)2p.
(7)
p=0
In practice, Eq. 7 means that the signs of the differences in a neighborhood are interpreted as a P-bit binary number, resulting in 2P distinct values for the LBP code. The local gray-scale distribution, i.e. texture, can thus be approximately described with a 2 p -bin discrete distribution of LBP codes: Tt*t(LBPPtR(xc,yc)).
(8)
Let us assume we are given a n J V x M image sample (xc € { 0 , . . . , N — 1}, yc € {0, ...,M — 1}). In calculating the L B P p ^ distribution (feature vector) for this image, the central part is only considered because a sufficiently large neighborhood cannot be used on the borders. The LBP code is calculated for each pixel in the cropped portion of the image, and the distribution of the codes is used as a feature vector, denoted by S:
203
S = t(LBPP,R(x,y)), xe{\R\,...,N-l-\K\},y€{\R],...,M-l-lR\}.
[
> '
The original LBP (Sec. 1) is very similar to LBP 8 ) i, with two differences. First, the neighborhood in the general definition is indexed circularly, making it easier to derive rotation invariant texture descriptors. Second, the diagonal pixels in the 3x3 neighborhood are interpolated in LBPs,i3.2. Non-parametric
Classification
Principle
In classification, the dissimilarity between a sample and a model LBP distribution is measured with a non-parametric statistical test. This approach has the advantage that no assumptions about the feature distributions need to be made. Originally, the statistical test chosen for this purpose was the cross-entropy principle introduced by Kullback 2 1 1 . Later, Sokal and Rohlf22 have called this measure the G statistic: <-,
B
B
= 2 J2 lSb log Sb - Sb log Mb],
G(S, M) = 2 Y, Sb log j± b
6=1
(10)
6=1
where S and M denote (discrete) sample and model distributions, respectively. Sb and Mb correspond to the probability of bin b in the sample and model distributions. B is the number of bins in the distributions. For classification purposes, this measure can be simplified. First, the constant scaling factor 2 has no effect on the classification result. Furthermore, the term J2b=i [Sb log Sb] is constant for a given S, rendering it useless too. Thus the G statistic can be used in classification in a modified form: B
L(S,M)
= -YSb\ogMb.
(11)
6=1
Model textures can be treated as random processes whose properties are captured by their LBP distributions. In a simple classification setting, each class is represented with a single model distribution Ml. Similarly, an unidentified sample texture can be described by the distribution S. L is a pseudo-metric that measures the likelihood that the sample S is from class i. The most likely class C of an unknown sample can thus be described by a simple nearest-neighbor rule: c=
argmin
^
i
Apart from a log-likelihood statistic, L can also be seen as a dissimilarity measure. Therefore, it can be used in conjunction with many classifiers, like the fc-NN classifier or the self-organizing map (SOM). The log-likelihood measure works well in many situations, but may be unstable with small sample sizes. The reason is that with small samples, the historam is likely to contain many zeros, for which the
204
logarithm is undefined. With small samples, the Chi square distance usually works better 2 3 :
rtw-tl^Jg. 0=1
(13)
Almost equivalent accuracy can be achieved with the histogram intersection, with a significantly smaller computational overhead: B
H(S, M) = J2
min 5
( °> Mb)-
(14)
6=1
3.3. Rotation
Invariance
Due to the circular sampling of neighborhoods, making LBP codes invariant with respect to rotation of the image domain is fairly straightforward. "Rotation invariance" here does not however account for textural differences caused by changes in the relative positions of a light source and the target object. It also does not consider artifacts that are caused by digitizing effects. Each pixel is considered a rotation center, which seems to be the convention in deriving rotation invariant operators. With the assumptions stated above, the rotation invariant LBP can be derived as follows. When an image is rotated, the gray values gp in a circular neighbor set move along the perimeter of a circle centered at gc. Since the neighborhood is always indexed counter-clockwise, starting in the direction of the positive x axis, the rotation of the image naturally results in a different LBPp,^ value. This does not, however, apply to patterns comprising of only zeros or ones which remain constant at all rotation angles. To remove the effect of rotation, each LBP code must be rotated back to a reference position, effectively making all rotated versions of a binary code the same. This transformation can be defined as follows: L B P £ f i = min{ROR(LBP P , fi , i) \ i = 0 , 1 , . . . , P - 1}
(15)
where the superscript ri stands for "rotation invariant". The function ROR(a:,i) circularly shifts the P-bit binary number x i times to the right (|z| < P). In short, the rotation invariant code is produced by circularly rotating the original code until its minimum value is attained. The LBPROT operator introduced by Pietikainen et al.24 is equivalent to LBPg^. In total, there are 36 different 8-bit rotation invariant codes. Therefore, LBPg' fi produces 36-bin histograms. In practice, images are seldom rotated in 45° steps. This makes the crude quantization of the angular space in the LBPs.jj operator quite inefficient. Furthermore, the occurrence frequencies of the 36 different patterns vary a lot. Some of them are encountered only rarely, making them statistically unstable. The concept of "uniform" patterns was introduced in Maenpaa et al.25. It was observed that certain patterns seem to be fundamental properties of texture, providing the vast majority of patterns, sometimes over 90%. This proposion was further
205
confirmed with a large amount of data by Ojala et al.19. These patterns are called "uniform" because they have one thing in common: at most two one-to-zero or zeroto-one transitions in the circular binary code. The LBP codes shown in Fig. 3 are all uniform. To formally define the "uniformity" of a neighborhood G, a uniformity measure U is needed: P-\
U{GP) = \s(gP-i
- gc) - s(g0 - gc)\ + J2 \s(gP ~ 9c^ ~
S
(5P-I
~ 9c)\-
(16)
P=I
Patterns with a U value of less than or equal to two are designated as "uniform". For a P-bit binary number, the U value can be calculated efficiently with binary arithmetic as follows: P-i
U(x) = ^2
F x
(
xor
R-OR(z, l),p),
(17)
where a; is a binary number. The function F(x, i) extracts the ith bit from a binary number x: F(x, i) = ROR(x, i) and 1.
(18)
The total number of patterns with U(GP) < 2 is P(P - 1 ) + 2. When "uniform" codes are rotated to their minimum values, the total number of patterns becomes P+l. The rotation invariant uniform (riu2) pattern code for any "uniform" pattern is calculated by simply counting ones in the binary number. All other patterns are labeled "miscellaneous" and collapsed into one value:
LBP™£ = / £ P = O
P,R
I p +1
S{9p
~
5c)
'
U{Gp)
-
2
otherwise .
(19) V
'
In practice, LBPp 1 ^ is best implemented by creating a look-up table that converts the "basic" LBP codes into their LBP™"2 correspondents. A straightforward fix to the problem of the crude 45° quantization of LBPg^ is to employ a larger number of samples in the circular neighborhood. The value of P cannot however be increased carelessly. First, the number of possible pattern codes that need to be mapped to their rotation invariant counterparts is 2 P . Obviously, the size of the look-up table will put a practical upper bound for the range of usable P values. It would be possible to calculate the mapping on-line, without a look-up table, but then the computational simplicity of the method would suffer significantly. Another issue that limits the range of usable values for P is the nature of digital images. For example, there are only nine pixels in a 3x3 neighborhood. If a P greater than eight is used in conjunction with R = 1, the samples contain a lot of redundant information.
206
3.4. Contrast
and Texture
Patterns
Contrast is a property of texture usually regarded as a very important cue for our vision system, but the LBP operator by itself totally ignores the magnitude of gray level differences. In many applications, especially in industrial visual inspection, illumination can be accurately controlled. In such a situation, a purely gray-scale invariant texture operator may waste useful information, and adding gray-scale dependent information may enhance the accuracy of the method. Furthermore, in applications such as image segmentation, gradual changes in illumination may not require the use of a gray-scale invariant method 26 . In a more general view, texture is distinguished not only by texture patterns but also the strength of the patterns. Texture can thus be regarded as a two-dimensional phenomenon characterized by two orthogonal properties: spatial structure (patterns) and contrast (the strength of the patterns). Pattern information is independent of the gray scale, whereas contrast is not. On the other hand, contrast is not affected by rotation, but patterns are, by default. These two measures supplement each other in a very useful way. The LBP operator was originally designed just for this purpose: to complement a gray-scale dependent measure of the "amount" of texture. Ojala et al.l used the joint distribution of LBP codes and a local contrast measure (LBP/C, see Fig. 1) as a texture descriptor. Rotation invariant local contrast can be measured in a circularly symmetric neighbor set just like the LBP: 1 P~l VARp,fl = -p^2{9pp=0
2
AO , where
1 P 9_ 1 u = -pYl P
"
( 20 )
p=0
is, by definition, invariant against shifts in the gray scale. Since contrast is measured locally, the measure can resist even intra-image illumination variation as long as the absolute gray value differences are not badly affected. A rotation invariant description of texture in terms of texture patterns and their strength is obtained with the joint distribution of LBP and local variance, denoted as LBPp u |j /VARp2>j?2. Typically, the neighborhood parameters are chosen so that Pi = P2 and R\ = R2, although nothing prevents one from choosing different values. VARP^R
3.5. Multi-Resolution
LBP
The most prominent limitation of the original LBP operator is its small spatial support area. Features calculated in a local 3x3 neighborhood cannot capture largescale structures that may be the dominant features of some textures. However, adjacent LBP codes are not totally independent of each other. Fig. 5 displays three adjacent four-bit LBP codes. Assuming that the first bit in the leftmost code is zero, the third bit in the code to the right of it must be one. Similarly, the first bit in the code in the center and the third bit of the rightmost one must be either different
207 or both equal to one. The right half of the figure shows an impossible combination of the codes. Each LBP code thus limits the set of possible codes adjacent to it, making the "effective area" of a single code actually slightly larger than 3x3 pixels. Nevertheless, the operator is not very robust against local changes in the texture, caused, for example, by varying viewpoints or illumination directions. An operator with a larger spatial support area is therefore needed.
Fig. 5. Three adjacent LBP^JJ neighborhoods and an impossible combination of codes. A black disk means the gray level of a sample is lower than that of the center.
3.5.1. Combining Operators A straightforward way of enlarging the spatial support area is to combine the information provided by N LBP operators with varying P and R values. This way, each pixel in an image gets N different LBP codes. The most accurate information would be obtained by using the joint distribution of these codes. However, such a distribution would be overwhelmingly sparse with any reasonable image size. For example, the joint distribution of LBPs.i, LBP"f 3> and LBP24 5 would contain 256 x 243 x 555 « 3.5 x 107 bins. Therefore, only the marginal distributions of the different operators are considered even though the statistical independence of the outputs of the different LBP operators at a pixel cannot be warranted. The aggregate dissimilarity between a sample and a model can be calculated as a sum of the dissimilarities between the marginal distributions: N
LN = -YJL{Sn,Mn),
(21)
n=l
where Sn and M " correspond to the sample and model distributions extracted by the nth operator 19 . Of course, the Chi square distance or histogram intersection can also be used instead of the log-likelihood measure. With most applications, this straightforward way of building a multi-scale LBP operator results in very good accuracy. The advantage obtained with filtering and cellular automata, introduced in the next section, is usually marginal. The cellular automaton version is, however, promising because it allows the use of an arbitrary number of LBP scales with a fixed feature vector length.
208
3.5.2. Filtering and Cellular Automata Prom a signal processing point of view, the sparse sampling exploited by LBP operators with large neighborhood radii may not result in an adequate representation of the two-dimensional image signal. Aliasing effects are an obvious problem. So might be noise sensitivity as sampling is made at single pixel positions without low-pass filtering. One might argue that collecting information from a larger area would thus make the operator more robust. From a statistical point of view, however, even sparse sampling is acceptable provided that the number of samples is large enough. The sparse sampling approach is also commonly used with co-occurrence methods. The LBP operator can be combined with multi-scale filtering in a straightforward way. Using Gaussian low-pass filters, each sample in the neighborhood can be made to collect intensity information from an area larger than the original single pixel. The utility obtained with filtering may, however, not be significant enough compared to the increased computational burden. 27,28 The length of the final feature vector soon becomes a limiting factor in using a large number of different scales. Considering only "uniform" patterns helps to some extent, but not enough. Memory consumption and computational burden in classification still make the use of many different scales impractical. The ability to encode a large neighborhood compactly while preserving important information is a crucial issue in building a fast multi-scale extension of the LBP operator. Cellular automata may provide a solution to this problem. The concept of a cellular automaton was first introduced by John von Neumann in the 1940's. In his model, the state of a system was presented with a twodimensional grid of ones and zeros — just like a digital binary image. The value of a slot in the state grid at time instant t + 1 was determined by the values of its nearest neighbors at time instant t. Wolfram29 later considered a one-dimensional case in which the value of a slot at time instant t +1 is dependent only on the value of the slot itself and its two neighbors at time instant t. If the one-dimensional state is binary, there is a total of 2 3 = 8 different neighborhoods. The rules determine whether a certain neighborhood produces one or zero at the next iteration. There is therefore a total number of 2 8 = 256 different rules. The relation between multi-scale LBP and cellular automata may not be quite obvious. It becomes so if we think of the thresholded circular neighborhoods as one-dimensional circular binary signals. In Fig. 6, two LBPS,.R codes are dressed to form the two topmost rows of a two-dimensional pattern. The notation of "time" in the evolution of the cellular automaton pattern is thus replaced by the radius of the circular neighborhood. Since the rows are treated circularly, the breakpoint (dashed line) plays no role. This property has an important consequence: the cellular automata are always invariant with respect to rotation, irrespective of the LBP version used as the input signal. Any number of LBP scales can be added to the pattern. The problem of encoding a neighborhood now becomes one of finding the rule that most probably produced the observed pattern, i.e. inversion.
209
Fig. 6.
Turning a multi-resolution LBP into a two-dimensional pattern.
The inversion problem can be solved with a simple algorithm based on majority voting 28 . Arbitrarily large binary neighborhoods can thus be encoded with the input signal (P bits) and a cellular automaton rule (eight bits). The marginal distributions of the input signal and the cellular automaton rule code can be used as a texture descriptor. The L B P ^ i concatenated with the marginal distribution of the cellular automaton codes deduced from S co-centric LBP^/j codes is denoted by LBPCAp^. The computational overhead of the LBPCAp^s is over S times that of L B P p ^ . Therefore, the operator may not be well suited for applications where fast feature extraction is needed. Its ability to describe large neighborhoods with a relatively short feature vector is, however, a very important feature in image retrieval, for example. 3.6. Opponent
Color
LBP
The opponent color LBP (OCLBP) operator was developed as a joint color-texture operator for comparing gray-scale and color texture features 30 . The mechanism with which the OCLBP feature is extracted is motivated by Jain and Healey 31 . Also the use of the term "opponent color" here follows the convention adopted by them: all pairs of color channels are called "opponent colors". More precisely, opponent colors are colors perceived as opposing pairs by humans: red-green and yellow-blue. The "generalization" of the term is however justified because with computer vision there is no need to restrict oneself to the color processing model of the human eye. In the opponent color LBP, the LBP operator is applied on each color channel separately. In addition, each pair of color channels is used in collecting opponent color patterns so that the center pixel for a neighborhood and the neighborhood itself are taken from different color channels. Fig. 7 illustrates the three situations in which the center pixel is taken from the red channel. In total, three inter-channel LBP histograms and six intra-channel histograms are extracted and concatenated into a single distribution. Since opposing pairs, like R-G and G-R, are highly redundant, either will suffice for analysis. Consequently, three of the six inter-channel
210
Fig. 7. Creating opponent color LBP codes with a red center. The three planes illustrate color channels.
histograms can be discarded. Even then, the resulting texture descriptor is six times longer than the gray-scale version, which may make it unusable in some applications. Even though the opponent color LBP has been shown to perform well in comparison to other color texture measures, it is probably better not to mix color and texture. It has been demostrated with a large number of natural textures that joint color texture descriptors and combined color and texture features are outperformed by either color or gray-scale texture alone. The utility attained with a joint description is at most minimal. 30 4. Applications As a very simple texture operator, the LBP is ideally suited for applications requiring fast feature extraction. Due to its simplicity and performance, many people have applied it to a number of different applications. These are outlined in the following sections. 4.1. Industrial
Visual
Inspection
Although many potential application areas for texture analysis exist in industry, only a limited number of successful exploitations have been reported 32 ' 33 . One of the major problems is the non-uniformness of real-world textures, which is caused by changes in orientation, illumination, scale, etc. In addition, the computational requirements of many proposed texture features are high. 34 One of the first industrial applications of the LBP was metal surface inspection 35 37 " . Later, Marzabal et al.38 introduced a system in which a subset of the LBP codes was used in characterizing defects in automobile engine valves. Yet another industrial application was reported by Paclfk et al.3g. They compared different feature extraction schemes in a segmentation process needed for the analysis of laundry detergents. One of the most commonly reported applications of the LBP is wood inspection, with which the operator has been used with color measures 40 . A common approach has been to use color and texture features with non-supervised clustering methods like the SOM 4 1 . Color and texture features, including the LBP, have been used in
211
a demanding real-time parquet inspection problem 42 . Paper inspection has been a successful application area with the LBP. Iivarinen used the LBP and GLCM texture features with a SOM-based, non-supervised segmentation method in detecting defects in surface images. Paper samples were used in experiments 43 . The performance of the LBP was found to be equivalent to that of the GLCM methods, but its computational simplicity was seen as a clear advantage. Since then, the LBP has been applied to the measurement of overall paper quality by Turtinen et al.44. Non-supervised learning was used in classifying paper samples to four quality classes. With the LBP, excellent results were reported. Maenpaa et al.45 report a study in which the same problem is solved in real time, utilizing a highly optimized software implementation of the LBP operator, feature reduction, and fast classification. 4.2. Image
Retrieval
A number of researchers have used LBP as part of their image retrieval systems. Lew 46 presented a system called "Imagescape", which used distribution-based texture features as texture models. These features included the LBP, LBP/C, and "Trigrams", which was a variant of the LBP calculated from an edge image. Also Berman and Shapiro 47 have used LBP as a texture feature in their "Flexible Image Database System". Schaefer48 in turn applied the operator to JPEG-compressed images for image retrieval. He found that with compressed images, sub-sampling does not decrease indexing accuracy. LBP has also appeared as a texture feature in studies by Tao and Dickinson 49 , Huijsmans et al.50, and Kauniskangas et al.51. Liao and Chen 52 and later Yao and Chen 5 3 reported studies in which retrieval experiments were performed with distorted color textures. A new texture feature, called "local edge patterns", was derived from the LBP. A large-scale empirical study of image retrieval in which the LBP was pitted against "standardized" texture features in the MPEG-7 standard was reported by Ojala et al.54. A large number of natural textures from the Outex 5 5 database were used in experiments. 4.3. Scene
Analysis
The interpretation of natural scenes has been the ultimate goal of computer vision since its early days 5 6 . The problem has however proved to be very difficult, and successful demonstrations have been presented only in strictly limited applications 57
Recent findings from human psychophysics, neurophysiology and computer vision provide evidence for a framework in which objects and scenes are represented as collections of viewpoint-specific features rather than 2D templates or 3D models 58 . This approach has been utilized in recognizing real-world textures with the LBP by Turtinen and Pietikainen 59 and Pietikainen et al.60. In these works, representative models for textures were found with feature self-organization. It was shown that
212
it is possible to learn a small subset of texture samples that can reliably work as representatives in classifying textures. Promising results were reported in classifying CUReT 1 4 textures and in the segmentation of outdoor scene images.
4.4. Face
Analysis
Face analysis is among the most intriguing, and perhaps surprising, applications of the LBP. Ahonen et al.23 reported a study in which both shape and texture information is used in representing face images. The face area is first divided into small regions from which LBP histograms are extracted. The histograms are concatenated into a single, spatially enhanced feature histogram efficiently representing the face image. Experiments clearly show that the proposed method is superior to all the state-of-the art methods it was compared to (PCA, Bayesian Intra/extrapersonal Classifier and Elastic Bunch Graph Matching). The evaluation was performed with the FERET database, which includes different facial expressions, lighting and aging of the subjects. Hadid et al.61 introduced the LBP as a powerful feature for face detection and recognition. The same facial representation is used in both tasks, which is seen as an important property: a unified feature space makes it possible to combine detection and recognition into one framework in which both tasks support each other. In experiments, the LBP performed equally well or better than any reference method in face detection. In recognizing faces from low-resolution images, the LBP was found to be superior to the reference methods. The study by Feng et al.62 indicates that LBP-based methodology can be used in recognizing facial expressions. In experiments, LBP features combined with the linear programming classification technique yielded better results than other methods with the JAFFE 6 3 database.
5. Other Applications Many people have chosen the LBP for applications requiring image segmentation. The LBP/C and LBP/VAR operators in particular have been popular 26,43,64-67 One of the most exotic uses of the LBP is the hair-based person detection system of Cohen et al.68. Garcia et al.69 have developed a system for the creation of visual mosaics of the seabed, in which LBP/C was used as a texture feature. The results were utilized in estimating the motion of an underwater robot 6 9 . Soriano et al.70 report an attempt to combine color and texture (LBP) in classifying images of coral reef. Finally, Heikkila et al.71 report a study in which the LBP was successfully applied to the detection of moving objects via background subtraction. The proposed algorithm models the background by its texture content. An adapting texture model is maintained in overlapping image blocks, and movement is detected by changes in the texture model. Very good results are achieved in real time.
213
6. Conclusions The LBP operator can be seen as a truly unifying approach to the traditionally divergent statistical and structural models of texture analysis. Texture is described in terms of micro-primitives (textons) and their statistical placement rules. Optionally, the primitives may be coupled with a complementary measure of local image contrast, which measures the strength of the primitives. The LBP operator is relatively invariant with respect to changes in illumination and image rotation. It can even resist changes in texture scale. Still more work needs to be done to make the operator invariant against 3-D distortions, often present in natural scenes, for example. The promising results achieved in viewbased recognition and in detecting moving objects indicate that this problem can be solved with the LBP 6 0 ' 7 1 . Furthermore, the success in detecting and recognizing faces suggests that the LBP may be useful in many object recognition tasks that have not previously been considered texture analysis problems. Perhaps the most important property of the LBP operator in real-world applications is its tolerance against illumination changes. Equally important is its computational simplicity, which makes it possible to analyze images in challenging real time settings. To help researchers and application developers, C++ and Matlab implementations of the operator are provided free of charge 72 ' 73 . References 1. T. Ojala, M. Pietikainen and D. Harwood. Pattern Recognition 29, 51 (1996). 2. P. Brodatz. Textures: a Photographic Album for Artists and Designers (Dover Publications, New York, 1966). 3. R. Haralick, K. Shanmugam and I. Dinstein. IEEE Transactions on Systems, Man, and Cybernetics 3, 610 (1973). 4. J. Weszka, C. Dyer and A. Rosenfeld. IEEE Transactions on Systems, Man, and Cybernetics 6, 269 (1976). 5. M. Unser. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 118 (1986). 6. O. Faugeras and W. Pratt. IEEE Transactions on Pattern Analysis and Machine Intelligence 2, 323 (1980). 7. J. Woods. IEEE Transactions on Information Theory 18, 232 (1972). 8. G. Cross and A. Jain. IEEE Transactions on Pattern Analysis and Machine Intelligence 5, 25 (1983). 9. A. Jain and K. Karu. IEEE Transactions on Pattern Analysis and Machine Intelligence 18, 195 (1996). 10. B. Julesz. Nature 290, 91 (1981). 11. J. Beck, S. Prazdny and A. Rosenfeld. In Human and Machine Vision, eds. J. Beck, B. Hope and A. Rosenfeld (Academic Press, New York, 1983). 12. J. Malik, S. Belongie, J. Shi and T. Leung. In 7th International Conference on Computer Vision (Kerkyra, Greece, 1999), vol. 2, pp. 918-925.
214 13. M. Varma and A. Zisserman. In 7th European Conference on Computer Vision (Copenhagen, Denmark, 2002), vol. 3, pp. 255-271. 14. K. Dana, B. van Ginnegen, S. Nayar and J. Koenderink. ACM Transactions on Graphics 18, 1 (1999). 15. R. Sanchez-Yafiez, E. Kurmyshev and F. Cuevas. Pattern Recognition Letters 24, 21 (2003). 16. F. Tomita and S. Tsuji. Computer Analysis of Visual Textures (Kluwer, 1990). 17. J. Francos, A. Meiri and B. Porat. IEEE Transactions on Signal Processing 4 1 , 2665 (1993). 18. T. Maenpaa. Texture Analysis Using Fourier Spectra and Local Binary Patterns. Diploma thesis, University of Oulu (1999). 19. T. Ojala, M. Pietikainen and T. Maenpaa. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 971 (2002). 20. T. Ojala, K. Valkealahti, E. Oja and M. Pietikainen. Pattern Recognition 34, 727 (2001). 21. S. Kullback. Information Theory and Statistics (Dover Publications, New York, 1968). 22. R. Sokal and F. Rohlf. Biometry (W.H. Freeman, 1969). 23. T. Ahonen, A. Hadid and M. Pietikainen. In 8th European Conference on Computer Vision (Prague, Czech Republic, 2004). In press. 24. M. Pietikainen, T. Ojala and Z. Xu. Pattern Recognition 33, 43 (2000). 25. T. Maenpaa, T. Ojala, M. Pietikainen and M. Soriano. In 15th International Conference on Pattern Recognition (Barcelona, Spain, 2000), vol. 3, pp. 947950. 26. T. Ojala and M. Pietikainen. Pattern Recognition 32, 400 (1999). 27. T. Maenpaa and M. Pietikainen. In 13th Scandinavian Conference on Image Analysis (Halmstad, Sweden, 2003), pp. 885-892. 28. T. Maenpaa. The Local Binary Pattern Approach to Texture Analysis — Extensions and Applications. Dr. tech. dissertation, University of Oulu (2003). http://herkules.oulu.fi/isbn9514270762/. 29. S. Wolfram. Reviews of Modern Physics 55, 601 (1983). 30. T. Maenpaa and M. Pietikainen. Pattern Recognition 37 (2004). 31. A. Jain and G. Healey. IEEE Transactions on Image Processing 7, 124 (1998). 32. K. Song, M. Petrou and J. Kittler. In SPIE Vol. 1708 Applications of Artificial Intelligence X: Machine Vision and Robotics (1992), pp. 99-106. 33. M. Pietikainen and T. Ojala. In Image Technology - Advances in Image Processing, Multimedia and Machine Vision, ed. J. Sanz (Springer-Verlag, 1996), pp. 337-359. 34. O. Silven. In Texture Analysis in Machine Vision, ed. M. Pietikainen (World Scientific, 2000), pp. 207-217. 35. M. Pietikainen, T. Ojala, J. Nisula and J. Heikkinen. In SPIE Vol. 2354 intelligent Robots and Computer Vision XII: 3D Vision, Product Inspection, and
215 Active Vision (1994), pp. 197-204. 36. T. Ojala. Nonparametric Texture Analysis Using Spatial Operators, with Applications in Visual Inspection. Dr. tech. dissertation, University of Oulu (1997). http://herkules.oulu.fi/isbn9514246187/. 37. Z. Xu, M. Pietikainen and T. Ojala. In International Conference on Quality Control by Artificial Vision (Le Creusot, Burgundy, France, 1997), pp. 179-184. 38. A. Marzabal, C. Torrens and A. Grau. In IX Symposium Nacional de Reconicimiento de Formas y Analisis de Imagenes (2001). 39. P. Paclik, R. Duin, G. van Kempen and R. Kohlus. In 16th International Conference on Pattern Recognition (Quebec, Canada, 2002), vol. 2, pp. 490493. 40. J. Kyllonen and M. Pietikainen. In IAPR Workshop on Machine Vision Applications (Tokyo, Japan, 2000), pp. 187-192. 41. O. Silven, M. Niskanen and H. Kauppinen. Machine Vision and Applications 3, 275 (2003). 42. T. Maenpaa, J. Viertola and M. Pietikainen. Pattern Analysis and Applications 6, 169 (2003). 43. J. Iivarinen. In Machine Perception and Artificial Intelligence, ed. M. Pietikainen (World Scientific, 2000), vol. 40, pp. 231-238. 44. M. Turtinen, M. Pietikainen, O. Silven, T. Maenpaa and M. Niskanen. International Journal of Advanced Manufacturing Technology 22, 890 (2003). 45. T. Maenpaa, M. Turtinen and M. Pietikainen. Real-Time Imaging 9, 289 (2003). 46. M. Lew. IEEE Computer 33, 46 (2000). 47. A. Berman and L. Shapiro. Computer Vision and Image Understanding 75, 175 (1999). 48. G. Schaefer. In Data Compression Conference DCC '01 (Snowbird, USA, 2001). 49. B. Tao and B. Dickinson. In SPIE Multimedia Storage and Archiving Systems (1996), vol. 2916, pp. 130-139. 50. D. Huijsmans, M. Leq and D. Denteener. In 9th International Conference on Image Analysis and Processing (Florence, Italy, 1997), vol. 2, pp. 22-29. 51. H. Kauniskangas, J. Sauvola, M. Pietikainen and D. Doermann. In 10th Scandinavian Conference on Image Analysis (Lappeenranta, Finland, 1997), pp. 35-42. 52. C.-J. Liao and S.-Y. Chen. Pattern Recognition 35, 1705 (2002). 53. C.-H. Yao and S.-Y. Chen. Pattern Recognition 36, 913 (2003). 54. T. Ojala, T. Maenpaa, J. Viertola, J. Kyllonen and M. Pietikainen. In The 2nd International Workshop on Texture Analysis and Synthesis (Copenhagen, Denmark, 2002), pp. 99-102. 55. T. Ojala, T. Maenpaa, M. Pietikainen, J. Viertola, J. Kyllonen and S. Huovinen. In 16th International Conference on Pattern Recognition (Quebec, Canada, 2002), vol. 1, pp. 701-706. Http://www.outex.oulu.fi/. 56. M. Baird and M. Kelly. Pattern Recognition 6, 61 (1974).
216 57. Y. Aloimonos, C. Fermiiller and A. Rosenfeld. ACM Computing Surveys 27, 307 (1995). 58. H. Bulthoff, C. Wallraven and A. Graf. In 16th International Conference on Pattern Recognition (Quebec, Canada, 2002), vol. 3, pp. 768-776. 59. M. Turtinen and M. Pietikainen. In The 3rd International Workshop on Texture Analysis and Synthesis (Nice, France, 2003), pp. 101-106. 60. M. Pietikainen, T. Nurmela, T. Maenpaa and M. Turtinen. Pattern Recognition 37, 313 (2004). 61. A. Hadid, M. Pietikainen and T. Ahonen. In Computer Vision and Pattern Recognition (2004). In press. 62. X. Feng, M. Pietikainen and A. Hadid. In British Machine Vision Conference (London, UK, 2004). Submitted. 63. M. Lyons, J. Budynek and S. Akamatsu. IEEE Transactions on Pattern Analysis and Machine Intelligence 2 1 , 1357 (1999). 64. P. Wang, S. Krishnan, C. Kugean and M. Tjoa. In 23rd Annual EMBS International Conference (Istanbul, Turkey, 2001), pp. 3691-3695. 65. K.-M. Chen and S.-Y. Chen. Pattern Recognition Letters 23, 755 (2002). 66. T. Rubio, A. Bandera, C. Urdiales and F. Sandoval. In 2nd International Workshop on Texture Analysis and Synthesis (Copenhagen, Denmark, 2002), pp. 117-121. 67. A. Lucieer, P. Fisher and A. Stein. In Second International Symposium on Spatial Data Quality (Hong Kong, 2003). 68. I. Cohen, A. Garg and T. Huang. In 15th International Conference on Pattern Recognition (Barcelona, Spain, 2000), vol. 1, pp. 1119-1124. 69. R. Garcia, X. Cufi and M. Carreras. In IEEE/RSJ International Conference on Intelligent Robots and Systems (Maui, Hawaii, 2001), pp. 1682-1687. 70. M. Soriano, S. Marcos, M. Quibilan, P. Alino and C. Saloma. In OCEANS 2001 (Honolulu, Hawaii, 2001), pp. 1008-1013. 71. M. Heikkila, M. Pietikainen and J. Heikkila. In British Machine Vision Conference (London, UK, 2004). Submitted. 72. Pattern Recognition C + + Library, http://www.ee.oulu.fi/~topiolli/cpplibs/ (2000). 73. LBP Implementation in Matlab. http://www.ee.oulu.fi/research/imag/texture /image_data/matlab.html (2000).
C H A P T E R 3.1 D O C U M E N T ANALYSIS A N D U N D E R S T A N D I N G
YUAN YAN TANG Department
of Computer Science, Hong Kong Baptist
E-mail: [email protected],
University,
Hong Kong
tangQconcour.es.concordia.ca.
The basic concepts and underlying techniques of document processing are presented in this chapter. A basic model for document processing is described. In this model, document processing can be divided into two phases: document analysis and document understanding. A document has two structures: geometric (layout) structure and logical structure. Extraction of the geometric structure from a document refers to document analysis; mapping the geometric structure into the logical structure deals with document understanding. Both types of document structures and the two areas of document processing are discussed in this chapter. Two categories of methods have been used in document analysis, namely, (1) hierarchical methods including top-down and bottom-up approaches, (2) no-hierarchical methods including modified fractal signature and order stochastic filtering. Tree transform, formatting knowledge and description language approaches have been used in document understanding. All the above approaches are presented in this chapter. A particular case - form document processing is discussed. Form description and form registration approaches are presented. A form processing system is also introduced. Finally, many techniques, such as skew detection, Hough transform, Gabor filters, projection, crossing counts, form definition language, wavelet transform, etc. which have been used in these approaches are discussed in this chapter. Keywords: Document processing, Document analysis and understanding, Geometric and Logical structures, Hierarchical and no-hierarchical methods, Tree transform, Formatting knowledge, Description languages, Texture analysis, Wavelet transform.
1. Introduction Documents contain knowledge. Precisely, they are medium for transferring knowledge. In fact, much knowledge is acquired from documents such as technical reports, government files, newspapers, books, journals, magazines, letters, bank cheques, to name a few. The acquisition of knowledge from such documents by an information system can involve an extensive amount of hand-crafting. Such hand-crafting is time-consuming and can severely limit the application of information systems. Actually, it is a bottleneck of information systems. Thus, automatic knowledge acquisition from documents has become an important subject. Since the 1960's, 219
220
much research on document processing has been done based on Optical Character Recognition (OCR) [40]. Some OCR machines which are used in specific domains have appeared in the commercial market. Surveys of the underlying techniques are made by several researchers [17]. The study of automatic text segmentation and discrimination started about two decades ago [40]. With rapid development of modern computers and the increasing need to acquire large volumes of data, automatic text segmentation and discrimination have been widely studied since the early 1980's [1]. To date, a lot of methods have been proposed, and many document processing systems have been described [7]. More than 1000 papers have been presented in the International Conferences on Document Analysis and Recognition ICDAR'91, ICDAR'93, ICDAR'95, ICDAR'97, ICDAR'99, ICDAR'01 and ICDAR'03 [22, 23, 24, 25, 26, 20, 21]. This chapter is organized into the following sections: (i) Basic Model for Document Processing (ii) Document Structure: (a) Strength of structure, (b) Geometric and logical structures (iii) Document Analysis: (a) Hierarchical Methods (Top-down and bottom-up approaches) , (b) No-hierarchical Methods (Modified fractal signature, and' order stochastic filtering), (c) Web Document Analysis (iv) Document Understanding: (a) Tree transform, (b) Formatting knowledge, (c) Description language (v) Form Document Processing: (a) Characteristics of form documents, (b) Wavelet transform approach, (c) Form description language approach, (d) Form registration, (e) A form document processing system (vi) Major Techniques: Hough transform, Skew detection, Projection profile cuts, Run-Length smoothing algorithm (RLSA), Neighborhood line density (NLD), Connected components algorithm, Crossing counting, Form description language (FDL), Texture analysis, Wavelet transform, Other segmentation techniques. 2. Basic Model of Document Processing A basic model of processing the concrete document was first proposed in our early work, which was presented at the First International Conference on Document Analysis and Recognition [53]. Its graphic illustration can be found in Fig. 1, where the relationships among the geometric structure, logical structure, document analysis and document understanding are depicted. The following principal concepts were proposed in this model: • A concrete document is considered to have two structures: the geometric (or layout) structure and the logical structure. • Document processing is divided into two phases: document analysis and document understanding.
221
Figure 1: Basic model of document processing • Extraction of the geometric structure from a document is defined as document analysis; mapping the geometric structure into the logical structure is defined as document understanding. Once the logical structure has been captured, its meaning can be decoded by AI or other techniques. • But in some cases, the boundary between the two phases just described is not clear. For example, the logical structure of bank cheques may also be found during an analysis by knowledge rules. The basic model of document processing can be formally described below: Definition 1. A document Q, is specified by a quintuple
fi
-
(X,*,5,a,0)
3 = {0\e 2 ,...,e\...,e m }
(i) (2)
{
S a
3 x $ -> 2 s 1
(3)
2
{a ,a ,...,a*} C 3
1 2 0 = {/3 ,/? ,...,/?*}
C3CQ
where 9 is a finite set of document objects which are sets of blocks 0* (i = 1,2,..., m). 0* denotes repeated sub-division. $ is a finite set of linking factors, ipi and <pr stand for leading linking and repetition linking respectively. 8 is a finite set of logical linking functions, which indicate logical linking of the document objects, a is a finite set of heading objects, /? is a finite set of ending objects. Definition 2. Document processing is a process to construct the quintuple represented by Eqs. (1) - (3). Document analysis refers to extracting elements 3 , 0* and Qlj in Eq. (2), i.e. extraction of the geometric structure of fi. Document understanding deals with finding $, 6, a, and /3 in Eq. (3), considering the logical structure of fl.
222
Example - A simple example is illustrated in Fig. 2, we have
3 = {e\02,e3,e4,e5} 0* = {0j}* = {04,04} 0^ = {©5}* = { 0 5 , 0 5 , 0 5 } {Q\Q2}
a =
/? = { 0 4 , 0 5 } ( 3?X$
e'
L ©2_J
e3
r? ej
©?
(&,
\
(0 3 ,w) (04,^) V (© 5 ,¥v) J
(
03 04 05 0|*
\
k •
Ft ^
©2
©2 5
©3
B5
Figure 2: A simple example of document processing described by the basic model From the above definition, it is obvious that there is a nondeterministic mapping from the geometric structure into the logical structure. However, as the geometric structure is extracted, a deterministic mapping can also be achieved. It can be formally described below: Theorem 1. Let fi be a document defined by a quintuple (9, $,<S, ct,0) having nondeterministic mapping from geometric structure into logical structure, then there exists a quintuple ($>',$',5',a 1 , /?') which contains a deterministic mapping from the geometric structure of $1 into the logical structure. 3. Document Structures The key concept in document processing is that of structure. Document structure is the division and repeated subdivision of the content of a document into increasingly smaller parts, which are called objects. An object, which can not be subdivided into smaller objects, is called a basic object. All other objects are called composite objects. Structure can be realized as a geometric (layout) structure in terms of its geometric characteristics, and a logical structure due to its semantic properties.
223
3.1. Strength of Structure To measure a document structure, a Strength of Structure Ss has been introduced. Definition 3. Suppose a document is divided into n objects associated with n variables. Hi stands for the partial entropy of the i-th variable, and H for the entropy of the whole document. The strength of structure is n
Ss = J2Hi-H.
(4)
»=i
For instance, if the entire document consists of four composite objects associated with the variables x\ -14, the strength will be 4
Ss = ~^2 i=l
n
n x
^2 Pj( i)
lo
9 Pj{xi) + '^2 Pj(xi,x2,x3,xA)
j=l
log pj(x1,x2,x3,x4).
(5)
j=l
3.2. Geometric Structure Geometric structure represents the objects of a document based on the presentation, and connection among these objects. According to the International Standard ISO 8613-1:1989(E) [30], the geometric or layout structure can be defined below: Definition 4. Geometric or layout structure is the result of dividing and subdividing the content of a document into increasingly smaller parts, on the basis of the presentation. Geometric (Layout) Object is an element of the specific geometric structure of a document. The following types of geometric objects are defined: (1) Block is a basic geometric object corresponding to a rectangular area on the presentation medium containing a portion of the document content; (2) Frame is a composite geometric object corresponding to a rectangular area on the presentation medium containing either one or more blocks or other frames; (3) Page is a basic or composite geometric object corresponding to a rectangular area, if it is a composite object, containing either one or more frames or one or more blocks; (4) Page Set is a set of one or more pages; (5) Document Geometric (Layout) Root is the object at the highest level in the hierarchy of the specific geometric structure. The root node in the above example represents a page. The geometric structure can be formally described by the following definition according to the basic model given by Eqs. (1) and (2). Definition 5. The geometric structure is described by the element Q in the document space ft, = ( 3 , $, S, a, /?) shown in Eqs. (1 - 2) and flu which is a set of operations performed on 9 such that 3 = {3B,3c},
/?£/ = {U,n}, V ^ - ^ U ^ J C f l ) ,
V w ( 9 i n 9 , - ) = ^)
(6)
where 3jg represents a set of Basic objects, and 3 c stands for a set of Composite objects.
3 C = {0\ e2,..., em}, 3 B = {ejie* e 3C}
224
CWnese script being computeiCsd
T, T3
T2
Gl
H2 T4
T6 T5 H3 T9 T8 TIO
17 G2
Figure 3: Geometric and logical structures of a page in a newspaper This is a general definition of the geometric structure. Different types of specific documents have their specific forms. For example, for a specific document shown in Fig. 3(a) which is a page extracted from newspaper, it is divided into several blocks illustrated in Fig. 3(b). According to the above general model, its specific document geometric structure can be presented graphically in Fig. 3(c). In this page, a document is divided into several composite objects - text areas and graphic areas, which are broken into headline blocks, text line blocks, graphic blocks, etc. 3.2.1. Geometric Complexity The geometric complexity of a document can be measured by a complexity function n which is defined below: Definition 6. Let |9?T| and |3ta| be the number of elements in sets G T and 3 G respectively. Complexity function fi can be presented as M = |9T| + |3G|.
In terms of the complexity, documents can be classified into four categories: • Documents without graphics (e.g. editorials): 3 G =
(7)
225
• Document forms (e.g. bank cheques and other business forms): QG = {©?}; • Documents with graphics (e.g. general newspaper articles): 3 G ^ <j>\ • Documents with graphics as the main elements (e.g. advertisements, front page of magazine): \^ST\ < | ^ G | 3.3. Logical Structure Document understanding emphasizes the finding of logical relations between the objects of a document. To facilitate this process, a logical structure and its model have been developed in our early work [53] which can be summarized below. Logical structure represents the objects of a document based on the humanperceptible meaning, and connection among these objects. According to the International Standard ISO 8613-1:1989(E), the logical structure can be defined as follows [30]: Definition 7. Logical structure is the result of dividing and subdividing the content of a document into increasingly smaller parts, on the basis of the humanperceptible meaning of the content, for example, into chapters, sections, subsections, and paragraphs. Logical Object is an element of the specific logical structure of a document. For logical object, no classification other than Basic logical object, Composite logical object and Document logical root is defined. Logical object categories such as Chapter, Section and Paragraph are application-dependent and can be defined using the Object class mechanism [30]. The document understanding process finds the logical relations between the objects of a document. According to the basic model represented by Eqs. (1) and (2), a formal description of the logical structure is presented as follows: Definition 8. The logical structure is described by the elements $, S, a, and /3 in the document space fi = ( 3 , $, S, a, (5) in Eqs. (1 - 2), such that " $ a P _ 5
{a\a2,...,aP}
{/3\/?v..,/?n s
9 x $ -> 2
(8)
9
For a specific document shown in Fig. 3, its logical structure can be represented graphically by Fig. 3(d). 4. Document Analysis Document analysis is defined as the extraction of geometric structure of a document. In this way, a document image is broken down into several blocks, which represent coherent components of a document, such as text lines, headlines, graphics, etc. with or without the knowledge regarding the specific format [5, 47, 53, 57]. This structure can be represented as a geometric tree shown in Fig. 3(c). To build a such tree, there are many methods which can be classified into two categories:
226
• Hierarchical Methods: When we break a page of document into blocks, we conceder the geometric relationship among the blocks. In this way, we have two approaches, i.e. top-down and bottom-up approaches. • No-hierarchical Methods: When we are break a page of document into blocks, we do not consider the geometric relationship among the blocks. In this way, we have two approaches, i.e. modified fractal signature and order stochastic filtering. 4-1. Hierarchical Methods In the hierarchical Methods, we have two ways: (1) from parents to children, or (2) from children to parents. Corresponding to these two ways, there are two approaches: top-down and bottom-up approaches. These two approaches have been used in document analysis. Each has its advantages and disadvantages. The top-down approach is fast and very effective for processing documents that have a specific format. On the other hand, the bottomup approach is time consuming. But it is possible to develop algorithms which are applicable to a variety of documents. A better result may be achieved by combining the two approaches [41]. 4-1.1. Top-Down Approach Top-down (knowledge based) approach proceeds with an expectation of the nature of the document. It divides the document into major regions which are further divided into sub-regions, etc. [6, 15, 28].
Figure 4: The i-th and (i+l)-th level of a structure tree The geometric structure of a document can be represented by a tree. Suppose this tree contains K levels. Fig. 4 indicates the i-th and (i + l)-th levels. Suppose the upper layer has nodes N(, Nl, •••, N^; and the lower layer has nodes N{+1, A^2+1, ..., N^1. The relations between these two layers are expressed by edges
227
between the nodes. They can also be represented in the form of 1 0
1 .. 0 ..
1 0
0 0 .. 1 1 ..
0 1
0 0
0 0
.. ..
0 0 (9)
Nlm \
_| ° ° - ° ° ° - °
l 1
-
1
Values l's in Eq. (9) correspond to the edges in Fig. 4 meaning that
iq
<=>
(Ni+\Nt2+\...,Nlr+1)
(KtlKll,...,Ni+1)
m <=3>
Wp+1 +i
ATi+l p+2'
JV
..,N?>).
Eq. (9) gives two ways, "=>-" from left to right corresponding to "from top to bottom" in the tree structure (Fig. 4), and "<==" from right to left corresponding to "from bottom to top" in the same structure. In the top-down approach, the former way is used, and a document is divided into several regions each of which can be further divided into smaller sub-regions. 4-1.2. Bottom-Up Approach Bottom-up (data-driven) approach progressively refines the data by layered grouping operations. The bottom-up approach is time consuming. But it is possible to develop algorithms which can be applied to a variety of documents [4, 43]. Bottom-up approach corresponds to the direction of "•$=" in Eq. (9). In this way, basic geometric components are extracted and connected into different groups in terms of their characteristics, then the groups are combined into larger groups, etc. There are two practical bottom-up methods: (1) neighborhood line density (NLD) indicating the complexity of characters and graphics [31]; and (2) connected components analysis indicating the component properties of the document blocks [43]. 4.2. Non-hierarchical Methods Traditionally, two approaches have been used in document analysis, namely, topdown and bottom-up approaches [51]. Both approaches have their weaknesses: • They are not effective for processing documents with high geometrical complexity. Specifically, the top-down approach can process only the simple documents which have specific format or contain some a priori information. It fails to process the documents which have complicated geometric structures as shown in Fig. 5. • To extract the geometric (layout) structure of a document, the top-down approach needs iterative operations to break the document into several blocks
228
while the bottom-up approach needs to merge small components into large ones iteratively. Consequently, both approaches are time consuming.
0=)
Figure 5: A document with complicated geometric structure
4-2.1. Modified Fractal Signature Tang et al. [49] presented a new approach based on modified fractal signature for document analysis. It does not need iterative breaking or merging, and can divide a document into blocks in only one step. This approach can be widely used to process various types of documents including even some with high geometrical complexity. An algorithm has been developed in [49]. 4-2.2. Order Stochastic Filtering In this section, we present a new approach called Order Stochastic Filtering (OSF) [37]. It consists of two parts: (1) Median Order Statistic Filter (MedOSF), and (2) Maximum Order Statistic Filter (MaxOSF). This approach is more direct and much simpler. We use the MedOSF to remove the salt-pepper noise of the document and use the MaxOSF to do the page segmentation. In practice, they not only can adaptively process the documents with high geometrical complexity, but also save a lot of computing time. Definition 9. The order statistic of a sample (X\, • • •, Xn) are the sample values placed in ascending order. They are denoted by (X^,- • •,-X'(n)) which satisfy X(i)
< X(2)
<•••
<
X(n) is the sample maximum of (-X"(i), • • • ,^(n))
X(n).
and we denote it as
X(„) = max(X 1 ,---,X„). The sample median, which we denote by X me d, is a number such that approximately one-half of the observations are less than Xmed and one-half are greater. In terms
229
of the order statistics, X me d is defined by f -X"f2±i>
Xmed = |
{x^
+ X(f+i))/2
*/ if
n
„ is
* s odd, even.
4-3. Web Document Analysis Web images play a crucial role in bringing visual impact to an otherwise plain text medium. Wed page designers regularly create page headers and titles as well as other semantically important textual entities in image form. However, using current technology it is not possible to analyse the text embedded in images on Web pages. This is a significant problem for a number of reasons [34]. There is a number of approaches solving the problem of text segmentation and extraction. [34] proposed two approaches: (1) Split-and-Merge approach, and (2) Fuzzy approach. They enable the extraction of text in complex situations such as in the presence of varying colour and texture. XML (extensible Markup Language) is a standard language for defining structured documents of text-bades data. [29] proposed a technique for document transformation using OCR to generate various XML documents from printed documents. [39] presented a method for automatic discovery of semantic structures in HTML document. [45] proposed an architecture for ink annotation on Web documents. [19] presented an algorithm for automatic identification of story and preview images in news Web pages. [33] developed a string matching-based method for indexing and retrieval of on-line handwritten documents on Web pages. 5. Document Understanding As document analysis extracts geometric structures from a document image by using the knowledge about the general document and/or the specific document format, document understanding maps the geometric structures into logical structures considering the logical relationship between the objects in specific documents. There are several kinds of mapping methods in document understanding: [57] proposed a tree transformation method for understanding multi-article documents. [56] discussed the extraction of Japanese newspaper articles using a domain specific knowledge. [27] constructed a special purpose machine for understanding Japanese documents. [16] proposed a flexible format understanding method, using a form definition language. Our research [50, 52] has led to the development of a form description language for understanding financial documents. These mapping methods are based on specific rules applied to different documents with different formats. A series of document formatting rules are explicitly or implicitly used in all these understanding techniques. In this section, document understanding based on tree transformation, document formatting knowledge and document description language will be discussed. 5.1. Document Understanding Based on Tree Transformation
230
This method defines document understanding as the transformation of a geometric structure tree into a logical structure tree [57].
4B ^eMHIe) S ^ ~
HW {Mfe-Sile} -~-
V u . H l H L . U U I U . I U
I I.
MJMHUje.MB
III.IIII.L1.L..LI..I
d
Figure 6: Document understanding based on tree transformation A document has an obvious hierarchical geometric structure, represented by a tree as shown in Fig. 6 And the logical structure of a document is also represented by a tree which is illustrated in Fig. 6(c). In this example, three kinds of blocks are defined: H (head), B (body) and S (either body or head). During the transformation, a label is attached to each node. Labels include title, abstract, sub-title, paragraph, header, foot note, page number, and caption. The transformation, which moves the nodes in the tree, is based on four transformation rules. These rules are created according to a layout designed according to the manner in which humans read. Rules 1 and 2 are based on the observation that a title should have a single set of paragraphs as a child in the logical structure. The paragraph body in another node is moved to the node under the body title by these rules. Rule 3 is mainly for the extraction of characters or sections headed by a sub-title. By rule 4, a unique class is attached to each node. This method was implemented on a SUN-3 workstation. Pilot experiments were carried out using 106 documents taken from magazines, journals, newspapers, books, manuals, letters, scientific papers, and so on. The results show that only 12 out of 106 tested documents were not interpreted correctly. 5.2. Document Understanding Based on Formatting Knowledge Since a logical structure can correspond to a variety of geometric structures, the generation of logical structure from the geometric structure is difficult. One of the promising solutions to this problem is use of formatting knowledge. The formatting rules may differ from each other because of the type of document and language to be used in it. However, for a specific kind of documents, once the formatting knowledge is acquired, its logical structure can be deduced. An example can be found in [56] where a method of extracting articles from Japanese newspapers has
231
been proposed. In this method, six formatting rules of Japanese newspaper layout are summarized. An algorithm for extracting articles from Japanese newspaper has been designed based on the formatting knowledge. Another example can be found in [10] where a business letter processing approach has been developed. Because business letters are normally established in a single-column representation, letter understanding is mainly the identification of the logical objects, like sender, receiver, date, etc. In this approach, the logical objects of the letter are identified according to a Statistical Database (SDB). As the author reported, the SDB consists of about 71 rule packages derived from the statistical evaluation of a few hundred business letters. Other knowledge, like the shape, size and pixel density etc. of the image block can also be used for document understanding. Reference [11] uses statistical features of connected components to identify the address blocks on envelopes. 5.3. Document Understanding Based on Description Language One of the most effective ways to describe the structures of a document is the use of a description language. [16] detects the logical structure of a document and makes use of the knowledge rules represented by a form definition language (FDL). The basic concept of the form definition language is that both the geometric and logical structures of a document can be described in terms of a set of rectangular regions. For example, a part of a program in form definition language coded for the United Nations' (UN) documents is listed below: (defform UN-DOC# (width 210) (height 297) (if (box ( ? ? ? ? ) (mode IN Y LESS) (area (0 210 60 100)) (include (160 210 1 5))) (form UN-DOC-A (0 210 0 297)) (form UN-DOC-B (0 210 0 297)))) (defform UN-DOC-A ...) (defform UN-DOC-B ...) It means that the UN documents have a width of 210 mm and a height of 297 mm. The if predicate is one of the control structures. If the box predicate succeeds, the document named UN-DOC# is compared with UN-DOC-A and UN-DOC-B, and analyzed as UN-DOC-A. Otherwise, it is analyzed as UN-DOC-B. The box states that a rule line should exist inside the region (0 210 60 100) and satisfy the conditions that the width of the ruled line is between 160 mm and 210 mm and the height is between 1 mm and 5 mm. (defform UN-DOC-A ...) and (defform UN-DOC-B ...) will give the definition of the UN documents with and without a ruled line with these properties stated above. According to the definition, a form dividing engine will analyze the document
232
and produce the images of some logical objects, such as the organization which issued the document, document number, and section, etc. More details about this method can be found in [16]. 6. Form Document Processing Form document is a type of special-purpose documents commonly used in our daily life. For example, millions of financial transactions take place every day. Associated with them are form documents such as bank cheques, payment slips and bills. For this specific type of documents, according to their specific characteristics, it is possible to use specific method to acquire knowledge from it. 6.1. Characteristics of Form Documents Specific characteristics of form documents have been identified and analyzed in our early work [50] which are listed below: • In general, form document may consist of straight lines which are oriented mostly in horizontal and vertical directions. • The information that should be acquired from a form is usually the filled data. The filling positions can be determined by the above lines as references. • Texts in form documents often contain a small set of known machine-printed, hand-printed and handwritten characters, such as legal and numeric amounts. They can be recognized with current character recognition techniques. 6.2. Wavelet Transform Approach According to the above analysis, extraction of horizontal and vertical lines, which construct a form, plays a very important role. [48] has applied wavelet transform to do so. A document image can be transformed into four sub-images, by the wavelet transform, namely:
(a)
(b)
Figure 7: A square image and its sub-images produced by wavelet transform • LL sub-image: both horizontal and vertical directions have low-frequencies.
233 • LH sub-image: the horizontal direction has low-frequencies, and the vertical one has high-frequencies. • HL sub-image: the horizontal direction has high-frequencies, and the vertical one has low-frequencies. • HH sub-image: both horizontal and vertical directions have high-frequencies. We are interested in the LH and HL sub-images (Fig. 7) The LH sub-image results from a filter which allows lower frequency components to pass through along the horizontal direction, and higher frequencies along the vertical direction. That is an "enhancing" effect on the vertical, and "smoothing" effect on the horizontal. The result of the HL sub-image is opposite to that of the LH one. The higher frequencies can pass through along the horizontal direction, and lower frequency components along the vertical direction. That is an "enhancing" effect on the horizontal, and "smoothing" effect on the vertical. A practical example can be found in Fig. 8.
>.™
„~W„&i£&
(a)
(b)
Figure 8: Portion of USA federal tax return form and its wavelet LH sub-image
6.3. Approach based on Form Description Language A form document processing method based on form description language (FDL) has been proposed in [50, 54]. The FDL consists of two parts: (1) FSD which describes the structure of a document and (2) IDP which describes the items in a document. A block diagram of the FSD is illustrated in Fig. 9, and a block diagram of the IDP is presented in Fig. 10. The goal of this method is to extract the information of a form called items from the form document. Suppose there exists a finite set of relations F = { r l 3 T 2? ..., Tk} between the finite set of items a = {au a 2 , - , ®m} and the finite set of graphs E = {Si, S 2 ? ..., £ „ } , and it can be represented by 0-Fi matrix. We call it an Item Description Matrix: MID, such that MlD
-\o
f r* if («,,£,) er
ifc^s^r
satisfying the following condition: Vj(r, = (a»E)),
3? = {R, L, A, B},
234
Dv PH
DH
PL
Figure 9: Syntax diagram of the form structure description (FSD)
R
H L I—
-*• L
A B
Hi^ r<£h w
KEH Figure 10: Syntax diagram of the item description (IDP) where R, L, A and B represent Right, Left, Above and Below respectively. For example, the finite set of items and the finite set of graphs are given by a = {ai, OL2-, OLZ, ai) and S = {Li, L>2, L3, L4, L$, L6} respectively. Let T = {R, L, A, B}. MID is represented by the following matrix:
MID
=
Oil Ct!2
&3
OlA
A 0 0 L 0 0 B A 0 R L 0 0 0 B 0 R L 0 0 B 0 0 R
Eq. (10) means that 1) a.\ is located above line Li and also on the left of line L4; 2) c*2 is located below line L\ and above line L2 and also on the right of line Li and left of line L$; 3) Q3 is located below line L 3 and also on the right of line L$ and left of line Le; 4) ai is located below line L3 and also on the right of line L 6 . 6.4- Form Document Processing based on Form Registration
(10)
235
A form document processing system based on the pre-registered empty forms has been developed in [42]. The process includes two steps: (1) empty form registration, and (2) data filled form recognition. During the registration step, a form sample without any data is first scanned and registered with the computer. Through line enhancement, contour extraction and square detection, both the label and data fields are extracted. The relationships among these fields are then determined. Man-machine conversation is required during this registration process. The result of registration is stored as the format data of the form sample. During the recognition step, only the data fields are extracted according to the locations indicated by the format data. 6.5. Form Document Processing System An automatic form-document processing system (FPS) has been described in [7], It provides capabilities for automatically indexing form-document for storage/retrieval to/from a document library, and for capturing information from scanned form images using OCR. The FPS also provides capabilities for efficiently storing form images. The overall organization of FPS is shown in Fig. 11, which contains two parallel paths, one for image applications such as retrieval, display and printing of a form-document, the other for data processing applications that deal with information contained on a form.
Figure 11: An automatic form-document processing system FPS consists of 6 major processing components: (1) Defining form model; (2) Storing the form model in a form library; (3) Matching input form against the model stored in the form library; (4) Registering the selected model to the input form; (5) Converting the extracted image data to symbol code for input to data base; (6) Removing the fixed part of a form, and retaining only the data filled in for storage. 7. Major Techniques To implement the above approaches, a lot of practical techniques have been used. The major techniques are listed below: • Hough Transform [18],
236
• • • • • • • • • •
Skew Detection [44, 60, 61], Projection Profile Cuts [4, 58], Run-Length Smoothing Algorithm (RLSA) [12, 14, 15]., Neighborhood Line Density (NLD) [36], Connected Components Algorithm [9, 11, 59], Crossing Counting [4], Form Description Language (FDL) [16], Texture Analysis [32], Wavelet Transform [55], Other Segmentation Techniques [2, 3, 8, 13, 35, 38, 46].
8. Conclusions Every day, millions of documents including technical reports, government files, newspapers, books, magazines, letters, bank cheques, etc. have to be processed. A great deal of time, effort and money will be saved if it can be executed automatically. However, in spite of major advances in computer technology, the degree of automation in acquiring data from such documents is very limited and a great deal of manual labour is still needed in this area. Thus, any method which can speed up this process will make a significant contribution. This chapter deals with the essential concepts of document analysis and understanding. It begins with a key concept, document structure. The importance of this concept can be seen in the whole chapter: constructing a geometric structure model and a logical structure model; considering document analysis as a technique of extracting the geometric structure; regarding document understanding as a mapping from the geometric structure into logical structure; etc. This chapter attempts to theoretically analyze document structure and top-down, bottom-up approaches which are commonly used in document analysis in terms of entropy function. Some open questions and problems still exist, especially in document understanding. Any practical document can be viewed differently depending on its geometric structure space and logical structure space. Because there is no one-to-one mapping between these two spaces, it is difficult to find a correct mapping to transform a geometric structure into a logical one. For example, rules based on knowledge may vary in different documents, how to find the correct rules is a profound subject for future research. References 1. L. Abele, F. Wahl, and W. Scheri. "Procedures for an automatic segmentation of text graphic and halftone regions in document," Proc. 2nd Scandinavian Conf. on Image Analysis, pp. 177-182, 1981. 2. P. Ahmed and C. Y. Suen. "Computer recognition of totally unconstrained handwritten Zipcodes," Int. Journal of Pattern Recognition and Artificial Intelligence, Vol. 1, No. 1, pp. 1-15, 1987. 3. P. Ahmed and C. Y. Suen. "Segmentation of unconstrained handwritten postal
237
zipcodes," Proc. 6th Int. Conf. on Pattern Recognition, 1982, pp. 545-547. 4. T. Akiyama and N. Hagita. "Automated entry system for printed documents," Pattern Recognition, Vol. 23, No. 11, pp. 1141-1154, 1990. 5. A. Bagdanov and M. Worring. "Granulometric Analysis of Document Images," Proc. of of The 16th Int. Conf. on Pattern Recognition (ICPR'02), pp. 468471, 2002. ^ 6. F. L. Bourgeois, H. Emptoz, E. Trinh, and J. Duong. "Networking Digital Document Image," Proc. of of The Sixth Int. Conf. on Document Analysis and Recognition (ICDAR'01), pp. 379-383, 2001. 7. R. G. Casey, D. R. Ferguson, K. M. Mohiuddin, and E. Walach. "An intelligent forms processing system," Machine Vision and Applications, Vol. 5, No. 3, pp. 143-155, 1992. 8. M. Cesar and R. Shinghal. "An algorithm for segmentation handwritten postal codes," Man Machine Studies, Vol. 33, pp. 63-80, 1990. 9. V. Demjanenko, Y. C. Shin, R. Sridhar, P. Palumbo, and S. Srihari. "Real-time connected component analysis for address block location," Proc. J^th USPS Advanced Technology Conf, pp. 1059-1071, 1990. 10. A. Dengel. "Document image analysis - expectation-driven text recognition," Proc. of Syntactic and Structural Pattern Recognition (SSPR90), pp. 78-87, 1990. 11. A. C. Downton and C. G. Leedham. "Preprocessing and presorting of envelope images for automatic sorting using OCR," Pattern Recognition, Vol. 23, No. 3/4, pp. 347-362, 1990. 12. F. Esposito, D. Malerba, G. Semeraro, E. Annese, and G. Scafuro. "An experimental page layout recognition system for office document automatic classification: an integrated approach for inductive generalization," Proc. 10th Int. Conf. on Pattern Recognition, pp. 557-562, 1990. 13. R. Fenrich. "Segmentation of automatically located handwritten words," Proc. 3rd International Workshop on Frontiers in Handwriting Recognition, Chateau de Bonas, France, pp. 33-44, 1991. 14. J. L. Fisher, S. C. Hinds, and D. P. D'Amato. "A rule-based system for document image segmentation," Proc. 10th Int. Conf. on Pattern Recognition, pp. 567572, 1990. 15. H. Fujisawa and Y. Nakano. "A top-down approach for the analysis of document images," Proc. SSPR90, pp. 113-122, 1990. 16. J. Higashino, H. Fujisawa, Y. Nakano, and M. Ejiri. "A knowledge-based segmentation method for document understanding," Proc. 8th Int. Conf. on Pattern Recognition, pp. 745-748, 1986. 17. T. H. Hilderbrandt and W. Liu. "Optical recognition of handwritten Chinese characters: advances since 1980," Pattern Recognition, Vol. 26, No. 2, pp. 205225, 1993.
238
18. S. C. Hinds, J. L. Fisher, and D. P. D'Amato. "A document skew detection method using run-length encoding and the Hough transform," Proc. J Oth Int. Conf. on Pattern Recognition, pp. 464-468, 1990. 19. J. Y. Hu and A. Bagga. "Identifying Story and Preview Images in News Web Pages," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 640-644, 2003. 20. ICDAR'01. Proc. Sixth Int. Conf. on Document Analysis and Recognition, Seattle, USA, September 10-13, 2001. 21. ICDAR'03. Proc. Seventh Int. Conf. on Document Analysis and Recognition, Edinburgh, Scotland, August 3-6, 2003. 22. ICDAR'91. Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30 - Oct. 2, 1991. 23. ICDAR'93. Proc. Second Int. Conf. on Document Analysis and Recognition, Tsukuba Science City, Japan, Oct. 20-22, 1993. 24. ICDAR'95. Proc. Third Int. Conf. on Document Analysis and Recognition, Montreal, Canada, August 14-16, 1995. 25. ICDAR'97. Proc. Fourth Int. Conf. on Document Analysis and Recognition, Ulm, Germany, August 18-20, 1997. 26. ICDAR'99. Proc. Fifth Int. Conf. on Document Analysis and Recognition, Bangalore, India, September 20-22, 1999. 27. K. Inagaki, T. Kato, T. hiroshima, and T. Sakai. "MACSYM: A hierarchical parallel image processing system for event-driven pattern understanding of documents," Pattern Recognition, Vol. 17, No. 1, pp. 85-108, 1984. 28. R. Ingold and D. Armangil. "A top-down document analysis method for logical structure recognition," Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, pp. 41-49, 1991. 29. Y. Ishitani. "Document Transformation System from Papers to XML Data Based on Pivot XML Document Method," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 250-255, 2003. 30. ISO. 8613: Information Processing-Text and Office Systems-Office, Document Architecture (ODA) and Interchange Format, International Organization for Standardization, 1989. 31. O. Iwaki, H. Kida, and H. Arakawa. "A Segmentation method based on office document hierarchical structure," Proc. IEEE Int. Conf. Syst. Man. Cybern., Alexandria, VA, pp. 759-763, 1987. 32. A. K. Jain and S. K. Bhattacharjee. "Text segmentation using Gabor niters for automatic document processing," Machine Vision and Applications, Vol. 5, No. 3, pp. 169-184, 1992. 33. A. K. Jain and A. M. Namboodiri. "Indexing and Retrieval of On-line Handwritten Documents," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 655-659, 2003.
239 34. D. Karatzas and A. Antonacopoulos. "Two Approaches for Text Segmentation in Web Images," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 131-136, 2003. 35. F. Kimura and M. Shridhar. "Recognition of connected numerals," 1st Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, pp. 731739, 1991. 36. K. Kubota, O. Iwaki, and H. Arakawa. "Image segmentation techniques for document processing," Proc. 1983 Int. Conf. on Text Processing with a Large Character Set, pp. 73-78, 1983. 37. H. Ma, J. Zhou, and Y. Y. Tang. "Order Statistic Filter (OSF): A Novel Approach to Document Analysis," International Journal of Pattern Recognition and Artificial Intelligence, Vol. 16, No. 5, pp. 551-571, 2002. 38. B. T. Mitchell and A. M. Gillies. "A Model-based computer vision system for recognizing handwritten ZIP codes," Machine Vision and Applications, Vol. 2, pp. 231-243, 1989. 39. S. Mukherjee, G. Z. Yang, W. F. Tan, and I. V. Ramakrishnan. "Automatic Discovery of Semantic Structures in HTML Document," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 245-249, 2003. 40. G. Nagy. "A preliminary investigation of techniques for the automated reading of unformatted text," Comm, ACM, Vol. 11, No. 7, pp. 480-487, 1968. 41. G. Nagy, J. Kanai, and M. Krishnamoorthy. "Two complementary techniques for digitized document analysis," Proc. ACM Conf. on Document Processing Systems, pp. 169-176, 1988. 42. Y. Nakano, H. Fujisawa, O. Kunisaki, K. Okada, and T. Hananoi. "A document understanding system incorporating with character recognition," Proc. 8th Int. Conf. on Pattern Recognition, pp. 801-803, 1986. 43. L. O'Gorman. "The document spectrum for structural page layout analysis," IEEE Trans, on Pattern Analysis and Machine Intelligence, Vol. 15, No. 11, pp. 1162-1173, 1993. 44. L. O'Gorman and R. Kasturi. (Eds.). Document Image Analysis, New York: IEEE Computer Society Press, 1995. 45. S. Ramachandran and R. Kashi. "An Architecture for Ink Annotation on Web Documents," Proc. of of The Seventh Int. Conf. on Document Analysis and Recognition (ICDAR'03), pp. 256-260, 2003. 46. M. Shridhar and A. Badreldin. "Recognition of isolated and simply connected handwritten numerals," Pattern Recognition, Vol. 19, No. 1, pp. 1-12, 1986. 47. A. Spitz. "Progress in Document Reconstruction," Proc. of of The 16th Int. Conf. on Pattern Recognition (ICPR'02), pp. 464-467, 2002. 48. Y. Y. Tang, Hong Ma, Jiming Liu, Bingfa Li, and Dihua Xi. " Multiresolution analysis in extraction of reference lines from documents with graylevel back-
240
ground," IEEE Trans, on Pattern Analysis and Machine Intelligence, Vol. 19, No. 8, pp. 921-926, 1997. 49. Y. Y. Tang, Hong Ma, Dihua Xi, Xiaogang Mao, and C. Y. Suen. "Modified fractal signature (MFS): A new approach to document analysis for automatic knowledge acquisition," IEEE Trans, on Knowledge and Data Engineering, Vol. 9, No. 5, pp. 747-762, 1997. 50. Y. Y. Tang, C. Y. Suen, and C. D. Yan. "Chinese form pre-processing for automatic data entry," Proc. Int. Conf. on Computer Processing of Chinese and Oriental Languages, pp. 313-318, 1991. 51. Y. Y. Tang, C. Y. Suen, and C. D. Yan. "Document processing for automatic knowledge acquisition," IEEE Trans, on Knowledge and Data Engineering, Vol. 6, No. 1, pp. 3-21, 1994. 52. Y. Y. Tang, C. D. Yan, M. Cheriet, and C. Y. Suen. "Automatic analysis and understanding of documents," Handbook of Pattern Recognition and Computer Vision, pp. 625-654, edited by Patrick S.P. Wang, C.H. Chen and L.F. Pau, Singapore: World Scientific Publishing Co. Pte, Ltd., 1993. 53. Y. Y. Tang, C. D. Yan, M. Cheriet, and C. Y. Suen. "Document analysis and understanding: a brief survey," Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, pp. 17-31, 1991. 54. Y. Y. Tang, C. D. Yan, M. Cheriet, and C. Y. Suen. "Financial document processing based on staff line and description language," IEEE Trans, on Systems, Man, and Cybernetics, Vol. 25, No. 5, pp. 738-754, 1995. 55. Y. Y. Tang, L. H. Yang, J. M. Liu, and H. Ma. Wavelets Theory and Its Application to Pattern Recognition Singapore: World Scientific, 2000. 56. J. Toyoda, Y. Noguchi, and Y. Nishimura. "Study of extracting Japanese newspaper article," Proc. 6th Int. Conf. on Pattern Recognition, pp. 1113-1115, 1982. 57. S. Tsujimoto and H. Asada. "Understanding multi-articled documents," Proc. 10th Int. Conf. on Pattern Recognition, pp. 551-556, 1990. 58. M. Viswanathan. "Analysis of scanned documents - a syntactic approach," Proc. SSPR90, pp. 450-459, 1990. 59. P. S. Yeh, S. Antoy, A. Litcher, and A. Rosenfeld. "Address location on envelopes," Pattern Recognition, Vol. 20, No. 2, pp. 213-227, 1987. 60. C.L. Yu, Y.Y. Tang, and C.Y. Suen. "Document skew detection based on the fractal and least squares method," Proc. 3-nd Int. Conf. on Document Analysis and Recognition, Montreal, Canada, pp. 1149-1152, 1995. 61. X. Y. Zhu and X. X. Yin. "A New Textual/Non-textual Classifier for Document Skew Correction," Proc. of of The 16th Int. Conf. on Pattern Recognition (ICPR'02), pp. 480-482, 2002.
CHAPTER 3.2 CHINESE CHARACTER RECOGNITION Xiaoqing Ding Department of Electronic Engineering, Tsinghua University Beijing 100084, P. R. China dingxq @ tsinghua. edu.cn
Chinese character recognition is a very difficult and important pattern recognition problem, as well as a computer vision problem. It has enjoyed great success in recent decade. This chapter introduces the key technology progress of Chinese character recognition and its related research topics and the future prospect for Chinese character recognition.
1. Introduction In the last decade, the character recognition technology has made significant advances with applications throughout the world. At the same time, much progress has been made of the very difficult Chinese character recognition with the largest set of Chinese characters up to 4000/20000 and complex character structures. The entire area of Chinese character recognition involves a large set of multifonts of printed Chinese character recognition, the large set of on-line handwritten Chinese character recognition and naturally handwritten Chinese character recognition. These progresses have reported a robust high performance of recognition for practical documents and in fact for all digitized documents, such as the electronic publication for newspapers, magazines and books, digital library, form inputting, license recognition, etc. More concretely Chinese character OCRs have been marketed in China and also world wide. Chinese characters have recorded for more than five thousand years, the longest Chinese histories and cultures of China, and are used in people's daily life today. Because inputting more than 4000 to 20000 Chinese characters into computer is so difficult, Chinese character recognitions for automatically inputting Chinese characters are very important for modernization of Chinese culture and economic development in the information age. This chapter will introduce the key technology progresses of Chinese character recognition and its related research topics. It also introduces the future prospect for Chinese character recognition. 241
242
2. Principle of Chinese Characters Recognition Chinese character is a glorious and brilliant prestige of Chinese culture. It is the oldest ideographical character sets in the world and the only one still widely used today. Therefore, the Chinese characters are closely related to the development of Chinese culture. Chinese character recognition problem has often been challenging for researchers. The main difficulties encountered are: 1. Huge number of characters, of more than 4000, are used daily; more than 20000 are used (Chinese GB18030 includes 27484 Chinese characters); 2. More complex Chinese character pattern structure: average stroke number is 16, maximum stroke number is 41; 3. Existence of mutually similar character patterns; 4. Large variety of character shapes and size: Different fonts in printed documents: such as Song, Fang, He, Kai, Yuan, Lishu, Weibei, etc.; Wide variety in handwritten characters by different people in different writing style; 5. More complex document layout structure: vertical or horizontal text lines, complex table forms and image/graphics mixed in a page; 6. Poor quality of documents in actual application; Since a large set of Chinese characters as well as complex constructed Chinese characters should be recognized, the methods and technologies for Chinese character recognition are much different with the character recognition of western languages with small set of characters. Also it is impossible using stroke structure analysis methods to solve the Chinese character recognition problems, because stokes are so difficult to be perfectly extracted, as was proved by many Chinese character recognition experiments and information entropy theory, although it has been introduced by many papers published. Other methods have been found and have been proved by many experiments and practical applications to successfully solve Chinese character recognition problems against a large set of characters and more complex character structures. In contrast to the recognition methods based on the stroke structure analysis of Chinese character, currently the successful methods are based on the whole character image or based on the microstructures on the whole character image, in a way more similar to have human being read and identify the characters. For this reason, the statistical feature classification algorithms of the whole character images are more often used in the Chinese character recognition. Information Entropy Theory on Pattern Recognition[l] and the research experiments on Chinese character recognition prove that the methods based on the statistical feature classification on the whole character images have made very successful progresses and achieved high performance. More robust Chinese character recognition systems which are capable to recognize low quality characters, even with broken/connected strokes have been developed. 2.1 Three types of Chinese character recognition Character recognition techniques mainly have three types. They are on-line handwritten character recognition, machine printed character recognition, and off-line handwritten character recognition with increasing difficulties.
243
One may note that there are more differences in three types of Chinese character recognition: machine printed Chinese characters are more regular than hand written Chinese characters, but it is off-line recognition; in on-line handwritten Chinese character recognition the extra stroke track information could be recorded to help recognition. The off-line handwritten Chinese character recognition is the most difficult because the structure and shape vary in handwritten character and there is no writing stroke information to be used in recognition. One may also note that the same Chinese characters in general have more similar visual patterns whether they are on-line or off-line hand written or even printed. In general Chinese characters have more strokes in each character. Therefore, more complex stroke construction and stronger relation constraint among strokes remain in the same character. Also Chinese character, especially more complex Chinese character, has more stable shape structure than Western language characters with simple structure of characters. These give us more heuristic ideas for three types of Chinese character recognition. Further research reveals that the same Chinese characters in different type have nearly the same character shape. We can see the differences only on the degree of the shape or feature variances of the same characters. For example for the printed Chinese characters, character shapes are more stable, especially for the same font characters. That means, for the printed Chinese characters, the character shape or character features are closely clustered each other. In this case the feature probability distribution could be represented by Gaussian distribution with small variances, even approximately by equal covariance Gaussian distribution. Using the minimum Euclidean distance classifier with category feature centers can achieve good recognition performance, even for the characters from different fonts. For the handwritten Chinese characters the character differences come from different writers or from different writing situations. There are more shape dispersion and larger feature scatter variances than printed Chinese characters. Even so, a little stable shape and feature still remain in the same characters. Based on the analysis above, we can understand that the handwritten Chinese character recognition methods are nearly the same as printed Chinese character recognition but much larger shape dispersion and feature scatter variance of character should be considered. At the same time the feature probability distribution could be represented by a Gaussian distribution with larger covariance, even by a mixture Gaussian distribution. Feature probability distribution determines the optimum Bayes classifier design. We need design more complex than Minimum Euclidean Distance classifiers to obtain better recognition performance.
co(x) = a>i, i - argmax p(con/x) where (0\)
(l)
denotes a classification operation.
More differences between the on-line handwritten Chinese character recognition and off-line character recognition exist, because extra recorded person's handwriting time and space information, including stroke orders and
stroke trajectories could be used directly. Using this person's handwriting information and adding the character shape information at the same time, the performance of on-line handwritten character recognition could be improved significantly. 3. Feature Extraction and Compression for Chinese Character Recognition The information entropy theory (IET) [1] analyses the relationship between the feature-space F ={X,p(X) I XsPS1} and the category-space W={a>j, P(cOj)\cOie£2c, 1< i< C) in the recognition process, where p(X) denotes the observation probability of the J-dimensional feature vector X, P(a>i) is a priori probability of category co„ and Qc is the category set of C categories. A basic result is that the recognition error Pe of a Bayes classifier is upper-bounded by the a posteriori conditional entropy H(W\F), Pe<-H(W\F)=-[H(W)-I(F,W)]
(2)
This bound shows the restriction imposed on the best performance of an ideal Bayes classifier by feature extraction. As the System Entropy H(W) is constant for fixed category-space, the only way to obtain lower Pe bound is to increase I(F, W), the mutual information between the feature-space and the category-space. Feature extraction combines feature detection and compression into an integrated optimization process, generating more efficient feature-space based on some optimization criterion J. An optimal feature-space is extracted by linear mapping from each eigen-space subject to J. The best one is then obtained with maximum criterion value over the scale interval. A good optimization criterion indicating the efficiency of feature-space is quite important for feature extraction, which is used in both feature compression and the final feature-space selection. Hence it is reasonable to choose J = I(F,W) as the optimization criterion. In order to construct a robust Chinese character recognition system, the first important step is to extract effective and efficient statistical features from the entire character images, which should bring enough mutual information for Chinese category classification. For the Chinese character recognition system working with more than 4000 to 20000 characters, the System Entropy H(W) which represents the system uncertainty will be: H(W) = -fjP(coi)LogP(coi)
(3)
i=\
When a priori probability of categoryft),P(cc>i), i=l..c are all equal, then the System Entropy H(W) will be 12db to 14 db Therefore, the Mutual Information I{W, F) required also should be as near as possible to the System Entropy H(W), that is, I(W , F) = H(W) = 12 ~\4db For the Gaussian feature distribution, the mutual information is: /0v ,F)=_L_ ln ||il V
'
2 In 2
I .
(4)
245
where £ is the total feature covariance matrix, £
is the category covariance
matrix. When the feature covariance matrices are diagonal, then, l I(W,F)=— — V
inJILiL—!_V in£«_
(5)
' 2 In 2 | I „ I 2 In 2 ft <x lB where CT is the total feature variance of the i-th feature, a is the category variance of the i-th feature. We can see that the dimensions of features are required higher enough to get enough mutual information for character recognition. Much more kinds of features are extracted and have been used in Chinese character recognition, such as Peripheral features, Special Key-point features, Gross meshed features, Stroke density features, etc. But at last, the original Directional Elementary Features are extracted and have been proved suitable for Chinese characters mainly consisting of straight lines, and are effective and efficient for the robust Chinese character recognition. Even when such salient visual primitives have been selected, there still have much room left for its better utilization. Dimensionality reduction can be applied after feature detection to avoid the "curse of dimensionality," and get compact low dimensional representation. Though linear discriminant analysis (LDA) [6] is an attractive compression method in large-set problems, creating optimal feature-space subject to some criterion, but its quality has been largely limited by the information preserved in the fixed detection outputs. Assuming identical Gaussian feature distribution, LDA can be proved to apply optimal linear mapping to the fixed feature detection outputs that maximizes I(F,W) [1]. It obtains the d-dimensional feature-space F by applying a kxd linear transform *F on the k-dimensional eigen-space(dje £2c, 1^ & C}, if we have full knowledge of the prior probabilities p (
246
o)(x) = coi, i = argmax.p(a>n)p(x/con)
(6)
In character recognition, the prior probabilities />(&>,) can usually be assumed equal, but neither the function form nor the parameters of
p^x/co^
are known. Generally we would like to assume that /?(jc/ft)(.) takes a simple Gaussian form, and then the parameters are estimated by the training samples under the maximum likelihood framework (ML). The resulting classifier derived by (6) takes the form of quadratic discriminant functions (QDF) for each class, so it is called QDF classifier. It is also called Mahalanobis distance classifier, which is given by co(X) = argrmn{gi(X), l
= fj±[<J>rj(X-fii)f /=i
(8)
\
where fit denotes the mean vector, A,y and <E>y, are they'-th (\<jy, \<j
247
performance on the test data is considered as a good estimation of the machine's generalization ability, provided that the test samples are typical and sufficient. So we follow the simple rule to take the best design which shows the minimal recognition error on the test data. Since the training sample number is always finite in practical application, QDF is sensitive to the estimation error of the covariance matrix Zn,
especially
on the smaller eigen-values. In order to achieve more robustness the modified quadratic discriminant function (MQDF) was proposed to against the parameter estimation error under the practical situation of limited training samples. Compared with QDF, the modified quadratic discriminant function can be represented as: M
(nT
/=i
v
//
V
d
(
OT
v
//
°M
V
(
M
)
V
'=1
\
(12) 2
where d is the dimensionality of pattern X . oni is the j-th eigenvalue of the within-class covariance matrix of the n-th class Xn (o n X > anl >•••> ond ), and Qni
is the corresponding eigenvector. junj is the expectation of the
projection OniX . MQDF classifier achieves higher recognition accuracy and at the same time takes less computational cost Thus MQDF classifier has been widely used in handwritten character recognition, especially in handwritten Chinese character recognition. The classifier has also been used in printed Chinese character recognition, because it is robust for various noise and interfere conditions. Minimum distance classifier is a common selection in Chinese character recognition for its simplicity and for its good generalization ability. To achieve higher accuracy, more sophisticated classifier designs are used, which engenders the problem of generalization, especially for high dimensionality and comparatively deficient training samples. Hence some researches on the balance between machine capacity and the performance on the training data are going on based on the ideas of structural risk minimization (Vapnik, 1979) [4]. Fig. 1 and Fig. 2 show example images of machine-printed and handwritten characters. The recognition rates that have been reached for omni-font machineprinted Chinese characters of more than 10,000 categories are over 99.7%. For major fonts, the rates even reach 99.97% As for handwritten characters, the offline recognition rate of the 1st and 2nd level simplified characters is 94%, and the online recognition rate is over 98%. The recognition speed of such commercial OCR's whose software recognition engine runs on Pentium IV is more than 200 characters per second.
Fig. 1. Examples of different fonts of printed Chinese characters
Fig. 2. Examples of handwritten cursive Chinese characters
5. High Accuracy Handwritten Character Recognition Natural offline handwritten Chinese character recognition has long been regarded an extremely difficult problem in pattern recognition community. However, current OCR systems are prone to awesome performance deterioration for natural writing with severe structural distortions and stroke connections. Hence further improvement of feature matching technique, of which efficient representation and robust classification are two essential problems for a larger character set with weaker constraints, is desired. The real problem happened in handwritten Chinese character recognition, however, is hardly to gather a sufficient number of samples for classifier training that need training sample number near 10 times of feature dimension for overcome "curse of dimensionality" problem and for the MLE of the highdimensional feature distributions of various character shapes. It is obvious that the more samples, the better performances could be gotten, although dimensionality reduction has been processed after feature detection to avoid the "curse of dimensionality," and get compact low dimensional feature representation. Until now much effort has been put on the efficient feature representation and robust classification for weak constrains and natural offline handwritten Chinese character recognition to obtain better classification performance, which include Mixed Gaussian models for no-Gaussian sample distribution, Cox-Box transformation to get more Gaussian-like feature probability distributions, discrimiannt analysis, and Minimum Error Discriminant Learning. For freely written characters, HMM (Hidden Markov Models), SVM (Support Vector Machines) and some other nonlinear classification algorithms are studied [8] [14]. (1). Global Box-Cox transformation with applications to handwritten character recognition A variety of statistical classification methods depend on or benefit from the a priori normal assumption for the form of class-conditional probability density.
However, in many practical handwritten character recognition systems, multivariate normal distributions with symmetric shapes do not give good approximation to the densities of the features directly extracted A global Box-Cox transformation to the original feature space is adopted for the selection of proper data transforms to improve the normality of a positive random variable x: xx-\ A* 0 (13) A y = i ln(x) A=0 where y is a transformed random variable with more Gaussian-like feature probability distributions than x, different selections of A. result in different transforms. Experiments show that these methods are quite effective in accommodating the density departure of the high-dimensional features directly extracted from raw images in many character recognition tasks. For examples, namely the verylarge-set handwritten Chinese character recognition, and the small set digit recognition. Supervised parameter selection methods based on maximum likelihood principle are adopted. The top choice classification errors decrease by over 20% in both tasks, and a 55% drop in the first ten candidates error for the large-set problem is observed. (2). Gaussian Mixture Models for sample distribution representation are more adapted for the real handwritten sample probability distribution (Table 1) Because the conditional probability densities p(x/coi) are assumed Gaussian function which parameters could be estimated based on the Maximum Likelihood (ML) method. Under this assumption, Modified Quadratic Discriminative Function (MQDF) has been adopted and an excellent performance was achieved in offline handwritten Chinese and numeral recognition. In fact, the Gaussian distribution is not the true distribution form of p(x/coi) , so it is hard to represent p(x/coj) precisely and the rigorous execution of Bayesian decision rule could not lead to the optimal status. The Gaussian mixture model (GMM) is a linearly weighted sum of many Gaussian distributions, and in general it can reach any distribution. So using GMM to represent the conditional probability density p(x/coi) of samples could be benefit. The GMM of class conditional density for pattern i is a linearly weighted sum of some Gaussian components: M
p(*/0=5>***(*)
d4)
where, x is a J-dimensional feature vector of pattern i. M is the number of the Gaussian components. pk,k = 1,-M are the weights of each Gaussian M
component, satisfying ^ pk = 1, pk > 0 . And,
Table 1. A comparisons of different methods on handwritten Chinese character recognition Recognition Accuracy (%) Training Set Testing Set
MQDF
Regular Cursive Regular
99.10 93.18 97.49
MQDF + MCE 99.53 97.12 97.93
Cursive
88.76
91.19
gk(x), k = 1,••• Af are the Gaussian distribution functions, ex
3A ]
P[ - 2 ( x - Pxfc )T 1 2 ( x " R*))
* = wv
All parameters of the GMM is denoted by
Some research works and the experiments show that Gaussian Mixed Models GMM could improve the recognition performance of handwritten character recognition, but more computer costs for estimating the parameters. (3). Minimum Error Discriminant Training Using minimum error discriminant training is the other way to improve the recognition performanceWhile straightforward, ML training itself cannot guarantee the minimization of classification error on MQDF classifier, and MQDF takes the assumption that pyx/Cn)
is a Gaussian distribution, which deviates from the
true distribution of samples .For further improving the performance of MQDF classifier, discriminative training is required, which minimizes a criterion directly corresponding to the classification error. The MCE is a minimum classification error based discriminative training algorithm that minimizes the overall classification errors by a gradient descent procedure. First, a continuous misclassification measure is defined by:
dn(x,0) = hn(x,0n)-hn(x,0,ri) where hn(x,0n) 0
n = l,---,N
(16)
is the discrimiant function, hn (x,6n)>0
;
is the classifier parameter of the n-th class, all the parameters are represented
by0 = {0l,02,---,0N}-
Also, hn (x,0,T])
is its collective representation of
all the other competing classes. Then, plugging the misclassification measure dyX,0) function, a continuous loss function is obtained:
into a sigmoid
251
S(X,0)
=
—-r
(17)
MCE training is a procedure to find the optimal value of 0 , which is through an iterative generalized probability descent algorithm (GPD) to reach the minimum of the general loss function S ( X , 0) . Based on the minimum classification error (MCE) criterion and generalized probability descent (GPD) algorithm to design MQDF classifier, and get high recognition accuracy recognizer for handwritten character recognition. The experiments on handwritten numeral and handwritten Chinese recognition have shown the effectiveness of this method. 6. Gray Scale Image Based Character Recognition Character recognition had achieved great success in the past decades, especially in the field of document image processing [12]. But now image capturing is becoming much easier than scanner by using small, less expensive digital video recorders, digital cameras, camera-equipped PDA and mobile phones etc. Therefore, video/camera based character recognition, called Camera OCR/Video OCR/Smart Camera used in a lot of applications. Such techniques as perspective distortion correction, text detection from the background, character segmentation and recognition, possibly on the color or grayscale images are being studied. But the most challenging problem is the Character recognition in low quality images; in the lower resolutions, the illumination variances, noises, dirty backgrounds and so on. It is extremely difficult to obtain "clean" binary character images, which can be processed by the traditional binary image based recognition methods. For the problem of character recognition in low quality images, several methods have been developed which can be classified into two categories: (1) Binary image based methods, in which the binarization process will lose a lot of information in character image and will bring great amount of stroke break, connection or noises into the binarized images. (2) Greyscale image based methods, in which extracting recognition features directly from greyscale images [12] [15]. Without going through the binarization, greyscale image based recognition methods could use the entire information in the greyscale image as humans do. Gabor transformation based greyscale image character recognition methods have been developed, which extracts local features directly from greyscale images to fulfill all information extraction on the character image. Because the information loss from the binarization step is avoided, this method gives much better recognition performance, especially in low resolution and low quality images, than the methods based on the traditional binary images. These methods also set up a solid foundation for camera OCR or Video OCR. Using multi-directional Gabor filter set to extract the basic stroke structure information in character images, the robust local features extracted are powerful to discriminate different directional character strokes from the noised backgrounds, and get better performance for low quality images.
Gabor filters provide optimum joint spatial/spatial-frequency localization and have ability to simulate the receptive fields of simple cells in the visual cortex. 2D Gabor Filter is a complex sinusoidal-modulated Gaussian function with the response in spatial (Eq.18) and spatial-frequency domain (Eq.19) as follows (Figs. 3,4). (18) 2ft/?! h{x,y,k,tj>,ax,ay) = e\p\-exp where #, = * • cos0 + y • sin 0, , R2 =-jtsin0 + ycos0 (19)
H(u,r,W,(T„Oy) = C-cxp -2tf2- ^ ( V } ] +03y-(Fi)2 where F> = u ' c o s < t > + vsmt> , C=const. F2 = —u • sin <j> + v • cos0
R2
V
~ABK /
Rl F2
sl r
\"
;\/ v
\
V ' | \ ^ ' Ax /
X/ (b)
(a)
Fig. 3. Top-views of a Gabor filter in spatial domain (a) and spatial-frequency domain (b).
AX
D( Fig. 4. The relationsip of spatial sampling intervals and effective spatial widths of Gabor filters.
F1
253
Experiments show that the Gabor filter based gray scale character recognition methods (Fig. 5) perform excellently in low quality and lowresolution images, which provide the powerful capability for many OCR applications.
60%
40%
20%
0%
16
20
22
24
28
(a) SongTi characters
31
16
20
22
24
(b) KaiTi characters
Fig. 5. Recognition rates of printed Chinese characters at different resolutions (Abscissa: Chinese character size in pixels) 7. Some Related Research Topics for Chinese Character Recognition 7.1 Document layout analysis Document layout analysis is another area of research. Document layout analysis combined with document recognition is successfully applied to document digitization, electronic publishing, etc. A lot of research works have been done, from top-down or bottom to up strategy. Until now some new ideas for Document Layout Analysis are appeared, such as researches based on Visual Perception and Statistical Model, or HMM based document layout analysis, etc. Document layout analysis is also the key step for automatically Electronic Document layout reconstruction as the same as original layout. Content independent font identification is very useful in document reconstruction, and recent work has achieved about 95% font identification accuracy using only onecharacter information by using Wavelet transformation. Based on these techniques, some of the historical literature of China has been recognized by OCR to produce e-Book. Document layout analysis is still a very difficult and open problem, however, because there are so many layout patterns as in books, magazines, reports, forms, letters, and newspapers. The progress on document layout analysis will bring the blind OCR reader and full-automatically e-book publishing in the future, to produce the final output, human cooperation is needed today. Newspaper pages are scanned and recognized to convert into the digital archive as shown in Fig.6. The layout is analyzed so that the output looks like the original. In this way, the printed documents can then be converted into such formats as PDF, RTF and HTML and used as electronic archives.
254
7.2 Character segmentation for touching characters Segmenting touching characters in documents, which are mainly, composed of oriental glyph characters, such as Chinese, Japanese and Korean characters, as well as Latin characters. The samples we encounter are images scanned casually from newspaper, magazine and document etc., for daily office usage. Therefore the system should be robust enough to tolerate both degradation and multilingual turbulence. Character segmentation, especially for degraded and touching multilingual characters, becomes a bottleneck of the system performance.
UJJM
MMMMmmm
mmnaan
Fig. 6. Automatically layout analyzed newspaper.
Image degradation
Narrowly distributed or kerned font
mm &m <J# cm Pepper noise or blurring ink
Italic font
Hybrid samples
Fig. 7. Touching samples in Asian documents.
Segmenting touching characters in western documents (which are composed of pure Latin characters) has drawn significant attention before. The main idea to solve this problem attributes to two key sub-problems: dissection and combination. The same basic steps from dissection to combination are adapted in the context of Asian multilingual documents, that is a progressive searching process with multiple features to break the touching characters and a constrained dynamic programming scheme to combine valid characters. An algorithm takes into account occurrence of multilingual characters and special features for Asian characters and brings obvious improvement as compared with simple general algorithm. It has been applied in a practical OCR system to analyze Asian documents in office. Experimental results show that the algorithm is competent simultaneously in Chinese, Japanese and Korean documents (Fig. 7 and Table 2). Language Simplified Chinese Traditional Chinese Japanese Korean English
Inter Code
Character set
Text image samples
GB2312
6763+ASCII+
GIB5
13053+ASCII+
SJIS
6524+ASCU+
tpJct^fa tpjcmm ^~rmm
KSC5601
7238+ASCII+
ASCII
94
r
•>
j-.r.
t
>
7\%UM Az09,.!?
Table 2. Example images of Asian characters 7.3 Asian and minority language character recognition More attentions have been paid to the research on multilingual character recognition recently. Recognition of such Oriental characters as Tibetan and Uighur, etc. are being studied in addition to Japanese and Korean. All of these characters are more or less block type characters and similar character recognition methods can be applied. Basic Character Text image samples Language letter set Conson Tibetan: Tibetan ant: 30 592 Vowel: 4 Sanskrit: 3,700+
^ 15%^1
Uighur
32
108+
66 7 04*U> ™ r ^ , i /SJj) ^"
Table 3. Example images of Tibetan and Uighur characters
8. Conclusion Characters are important forever to convey information among the human society. China has one of the longest histories and cultures in the world. Chinese character recognition is believed to be very important in China. This chapter has focused on the main principles and methods of Chinese character recognition, and some related research topics, such as unconstrained handwritten Chinese character recognition, grayscale image based character recognition etc., which are pertinent to the practical application of Chinese character recognition. Although much progress has been made in Chinese character recognition in past decades and a lot of products have been in the markets and used in people's daily life, there still have more problems in character recognition with more robust and more flexible high performance, especially comparing with human being's reading ability. We should simulate the human visualization to further research on the character recognition and as well as computer vision problems. Through the research on character recognition, especially recognition of Chinese characters, a deeper understanding of image pattern recognition should be reached. References 1. X.Q. Ding, "Information entropy theory in pattern recogni-tion", Acta Electronica Sinica, 21(8), 1992, pp. 2-8. 2. F. Kimura, et al., "Improvement of handwritten Japanese character recognition using weighted direction code histogram", Pattern Recognition, 30(8), 1997, pp. 13291337 3. N. Kato, et al., "A handwritten character recognition system using directional element feature and asymmetric Mahalanobis distance", IEEE Trans, on PAMI, 21(3), 1999, pp.258-262. 4. V. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, New York, 1995. 5. C.L. Liu, et al., "Multiresolution statistical and structural feature extraction for handwritten numeral recognition", Proc. IWFHR5, Colchester, England, 1996, pp. 61-66. 6. K. Fukunaga, Introduction to Statistical Pattern Recognition (2nd Edn), Academic Press, New York, 1990. 7. J.. Zhang, X. Ding, C. Liu, Multi-Scale Feature Extraction and Nested-Subset Classifier Design for High Accuracy Handwritten Character Recognition, ICPR2000, Vol.2, pp581-584 8. X. Lin, X. Ding, Y. Wu, "Handwritten numeral recognition using MFNN based multi-expert combination strategy," Proc. 4th ICDAR, 1997. 9. J. Zheng, X. Ding, Y. Wu, "Spatio-temporal unified model for on-line handwritten Chinese character recognition," Proc. 5th ICDAR, 1999, pp. 649-652. 10. J. Gao, X. Ding, "On improvement of feature extraction algorithms for discriminative pattern classification," Proc. 15th ICPR, 2000, Vol. II, pp. 101-104. 11. R. Zhang and X. Ding, "Offline handwritten Chinese character recognition using optimal sampling features," Lecture Notes in Computer Science 1948, T. Tan, et al. Eds. Advances in Multimodal Interfaces - ICMI 2000, Springer, 2000, pp. 458-465. 12. C. Li, X. Ding, Y. Wu, Automatic text location in natural scene images, Proc. 6th ICDAR, 2001, pp. 1069-1073.
257 13. X. Ding, "The research of Chinese character recognition," Character Recognition and Intelligent Information Processing, Vol. 7, Tsinghua University, Feb. 2002, pp.14-18. 14. B. Feng, X. Ding, Y. Wu, "Chinese handwriting recognition using Hidden Markov Models," Proc. 16th ICPR, 2002. 15 X. Wang, X. Ding, C. Liu, "Optimized Gabor filter based feature extraction for character ecognition," Proc. 16th ICPR, 2002. 16 R.Zhang, X. Ding, Minimum Classification Error Training for Handwritten Character Recognition, ICPR2002,.Vol.l, pp. 580-583. 17 S. Katagiri, B.H. Juang, "Pattern recognition Using a Family of Design Algorithms Based upon Generalized Probabilistic Descent Method", IEEE Proceedings, vol. 86, no. 10, pp. 2345-2373,1998. 18 X. Wang, X. Ding, Gray-scale image-based character recognition algorithm to lowquality and low-resulution image, Document Recognition and Retrieval VIII, Proceeding of SPIE Vol.4307(2001) p.315-322. 19 R. Zhang, X. Ding, Offline handwritten numeral recognition using orthogonal Gaussian mixture model, ICIP-2001, pp.1126-1129. 20. R. Zhang, X. Ding, Minimum Classification Error Training for Handwritten Character Recognition ICPR2002, VOL I, PROCEEDINGS,580-583, 2002. 21. B. Feng, X. Ding,: Chinese handwriting recognition using Hidden Markov Models,. ICPR2002, VOL m , PROCEEDINGS, 212-215, 2002 22. M. Chen, X. Ding, Y Wu, Unified HMM-based layout analysis framework and algorithm, SCI CHINA SER F,46 (6): 401-408 DEC 2003, 23. Fujisawa, X. Ding, Character recognition in China: its trend and future perspectives, The Journal of the Institute of Electronics, Information and Communication Engineers, Vol.86, 2003-9, pp.679-682XXX 24. L. Chen, X. Ding, A universal method for single character type recognition, 23-26, UK, ICPR2004
CHAPTER 3.3 EXTRACTION OF WORDS FROM HANDWRITTEN LEGAL AMOUNTS ON BANK CHEQUES
In Cheol Kim and Ching Y. Suen Centre for Pattern Recognition and Machine Intelligence (CENPARMI), Concordia University 1550 de Maisonneuve Blvd. West, Suite GM606, Montreal, H3G 1M8, Canada E-mail: {kiminc, suen}@cenparmi.concordia.ca
We present an efficient method of extracting words from handwritten legal amounts on bank cheques based on the spatial gaps between connected components. As typical existing distance measures for gap estimation suffer from underestimation and overestimation errors, a modified version of each distance measure is explored. Moreover, an efficient method of combining three different types of distance measures based on 4-class clustering technique is proposed to compensate the errors in each measure. Lastly, we introduce a heuristic re-segmentation procedure to deal with the under-segmented errors due to gap irregularity. In experiments on real bank cheque database, the modified distance measures show about 3% of better separation rate than their original counterparts. Also, the combining method shows a much higher performance than those of individual distance measures. In addition, by applying the re-separating procedure, an 81.7% of notable correct separation rate was finally achieved.
1. Introduction As a representative application of automatic handwritten text processing, recognition of legal amounts on bank cheques has been intensively studied for several years ' ' . However, it still remains as a challenging task due to unconstrained writing styles, the great variability of handwriting instruments, etc. Separating a sentence into words and recognizing them is a typical approach to analyze a handwritten text 4 ' 5 ' 6 ' 7 . In such approach, precise extraction of the individual words is an essential preprocessing step for achieving a reliable recognition performance. In many related studies, extracting words in a handwritten text has been conducted by measuring the spatial gaps between connected components using a specific distance measure and then categorizing those gaps as the inter-word or inter-character gaps based on a given threshold8'9' 10. Unlike machineprinted text where gaps between words are almost constant and larger than those between characters, handwritten text has totally different shape and size of spatial gaps due to the 259
260
great shape variability of unconstrained writing styles, thereby causing substantial difficulties in computing the gap distance accurately. Thus quite elaborate distance measures are required to locate words effectively based on spatial gaps. Seni and Cohen proposed eight distance measures for extracting words from a line of handwritten text. Among them, bounding box (BB) method that computes the horizontal distance between the bounding boxes of two adjacent connected components, and runlength/Euclidean with heuristics (RLEH) method that estimates the gap between two connected components using either the minimum run-length or the minimum Euclidean distance, have shown superior performance. Mahadevan and Nagabushnam10 proposed a convex hull (CH) method that approximates the gap between components by the distance between the convex hulls surrounding each component. The above three distance measures, BB, RLEH, and CH methods, are simple and quite efficient. Thus they have been widely used by many researchers for solving their problems 5 '"' 12 . However these distance measures all suffer from the inherent problem of underestimation or overestimation. In order to overcome such problems, thereby improving the performance of word segmentation, other methodologies have been reported. Zhou et al.1 proposed a feedback-based re-segmentation approach that utilizes the confidence value of a recognition result to remedy underestimation errors resulting from the initial segmentation stage. Similarly, Kim et al.14 used the confidence value derived from the HMM-MLP hybrid system to verify all possible combinations of segmentation hypotheses, thereby separating words from a legal amount. Other recognition-based segmentation strategy can be found in several related studies15'16. Kim et al.17 introduced another word segmentation method that employs some of the cues that humans use and author's writing style in terms of spacing captured by characterizing the variation of spacing between adjacent characters as a function of the corresponding characters themselves. Park et al.18 also presented an adaptive segmentation method considering the writing styles of individuals. The parameters for classifying a given gap as an inter-word gap or an inter-character gap are tuned to the distribution of the gaps within the instance of the phrase image being examined. In addition, scale space technique, statistical approach, etc19'20 have been reported for word segmentation. In this chapter, we first introduce the modified versions of the existing distance measures, BB, RLEH, and CH methods, to reduce their estimation errors and to extract words effectively from handwritten legal amounts based on spatial gaps between connected components. In the case of BB method, the left and right boundaries of bounding boxes are adjusted appropriately to reduce the underestimation errors. Such concept of adjusting boundaries is also applied to the RLEH method to reduce its oversensitivity to component shape caused by dealing with component contours directly. In CH method, a pair of convex hulls each of which respectively encloses the left and right parts of a given connected component is newly introduced to avoid an overestimation problem. Next, a new method of integrating these three distance measures based on 4class clustering technique is proposed to compensate effectively the errors introduced by each measure, whereby further improvement in performance of word separation is expected. However, using a spatial distance cue these extraction methods still have a limitation when two or more words overlap horizontally or when the inter-word gap is
261 smaller than the inter-character gap. Such cases occur frequently in the legal amounts due to the lack of writing space in bank cheques. In order to remove these obstacles, we finally introduce a heuristic re-segmentation procedure based on the possible maximum word length in the lexicon and the writer's typical writing style in a limited writing space. In word extraction experiments on a legal amount database extracted from real-life bank cheques, we demonstrate the effectiveness of the proposed three extraction methods by comparing their extraction rates with those of the original distance measures. 2. Estimation of Spatial Gap Distance In this paper, our focus is to split a legal amount into the individual words based on the physical gaps between the connected components. We first employ three well-known distance measures, BB, RLEH, and CH method, to compute the distance of all gaps in a given legal amount image. Next, their modified versions are introduced to reduce the errors due to overestimation and underestimation. 2.1 Distance Measures The BB method simply computes the length of a horizontal line between the smallest rectangles respectively enclosing two adjacent connected components, for gap estimation, as shown in Fig. 1 (a). If the bounding boxes overlap horizontally, i.e. the right boundary of the left box extends to the left boundary of the right box, the distance is considered as zero. The RLEH method, shown in Fig. 1 (b), uses the minimum run-length or the minimum Euclidean distance with some heuristics to estimate the gap between a given pair of connected components. If two adjacent connected components overlap vertically by more than a threshold determined heuristically, the minimum run-length is used, and the minimum Euclidean distance, otherwise. In the CH method, the smallest convex polygon, called convex hull, enclosing each connected component should be estimated first. Next, we obtain the particular points where two adjacent convex hulls are intersected by the line linking their centroids. Finally, the gap between two connected components is defined as the Euclidean distance between these intersection points as shown in Fig. 1 (c). A more detailed description of these distance measures can be found in Refs. 9 and 10.
^UrfflfHl«S-lLi^%^BBgg2 (a)
(b)
262
Fig. 1. Gap estimation using (a) BB, (b) RLEH, and (c) CH method These distance measures provide a simple way for gap estimation but frequently they also produce serious estimation errors like underestimation or overestimation. Owing to the rectangular nature of suppressing a component shape to a box, the BB method suffers from underestimation errors. As can be seen from Fig. 2 (a), the gap between the connected components, at least one of which has horizontally a protruded long 'head' or 'tail' stroke (see gaps a, b), is usually underestimated. The RLEH method estimates the gap distance directly from component contours. Thus the estimation results are very sensitive to component shape and vertical positioning of components, as shown in Fig. 2 (b) (see gaps a, b, c). The CH method also suffers from an overestimation problem when an included ascender or a descender is located at either the beginning or the end part of a connected component, as shown in Fig. 2 (c) (see gaps a, b).
(a)
(b)
Fig. 2. Examples of misestimation by (a) BB, (b) RLEH, and (c) CH methods
2.2 Modified Distance Measures Next, we present the modified versions of the above-mentioned distance measures to minimize their estimation errors. In the case of BB method, we adjust the left and right boundaries of the bounding box of the component containing horizontally protruded 'head' or 'tail' part. Considering that such protruded 'head' and 'tail' parts usually consist
263
of a single horizontal stroke as shown Fig. 3 (a), the 'head' part is defined as the region from the leftmost position to a node point (denoted by 'N') where the value of horizontal histogram is larger than §W . Here, Wis the average stroke thickness in the entire image calculated using a simple mathematical method proposed in Ref. 21, and the parameter p" is empirically determined, considering some possible noise or distortion on a stroke. Similarly, the 'tail' part is defined as the region from the rightmost position to the node point (denoted by 'M'). Then, the new left and right sides of the bounding box are defined as follows: L"ew = L°ld +aH wd
(i)
/?"ew = /?°ld - aT,wd
(2)
Here, 7/wd and rwd denote the widths of the 'head' and 'tail' parts, respectively. And a is 0.35 if these parts are located within the body area and 0.25, otherwise. As can be seen from Fig. 3 (a), the gap between two connected components with horizontally protruded 'head' or 'tail' part is quite well estimated by employing the modified bounding box (MBB). The RLEH method is also modified (MRLEH) slightly based on such concept of adjusting the bounding box to reduce its over-sensitivity to 'head' and 'tail' parts; the component contour located within the adjusted bounding box is only considered for gap estimation. Additionally, one more heuristic is introduced to deal with some special components like the dot in characters 'i' and ' j ' , and broken part of a character. If two connected components do not overlap vertically, the gap between them is estimated by the bounding box distance instead of the run-length or Euclidean distance. B B d ista n ce
Head
Fig. 3. Gap estimation using (a) modified BB distance and (b) modified CH distance
Lastly, in order to deal with the overestimation problem in the CH method, a pair of
264
convex hulls for each connected component is introduced. We divide a connected component vertically into two equal parts and estimate two convex hulls enclosing each divided part. The gap between two adjacent components is then computed by considering two convex hulls enclosing the right part of the left component and the left part of the right component, as can be seen from Fig. 3 (b). This modified CH (MCH) method is quite attractive because it produces a reasonable gap distance even though an ascender or a descender is located at either the beginning or the end part of component whereby the gap is usually overestimated in the original CH method (compare gap A with B). Additionally, like the MRLEH method, a heuristic condition for dot or broken part is also added to the MCH method; if two connected components do not overlap vertically, the gap is defined as the horizontal distance between the intersection points of two convex hulls, instead of the Euclidean distance. 3. Word Extraction Experiments In our experiments, we applied the aforementioned distance measures and their modified versions to gap estimation and word extraction task using a set of 1030 legal amount image samples included in CENPARMI database called IRIS. The IRIS database was obtained from real-life bank cheques. Thus considerable noise, shape distortion, and space irregularity were present in the images. However, using these images enables a word extraction module to analyze more accurately the real handwriting style on bank cheques and increase its performance and robustness. 3.1 Word Extraction by 2-Class Clustering In order to extract words from a legal amount based on the spatial gaps, we first compute the distances in all the gaps in the image using a given distance measure. Using a clustering procedure based on the LBG algorithm22, those gaps are then divided into two classes: inter-character gap (ICG) and inter-word gap (IWG). Before clustering for word extraction, some special images containing only ICGs or only IWGs are sorted out first using a heuristic procedure that considers the maximum, minimum, and average gap distances, and the ratio of the width from the leftmost component to the rightmost one to the image width to determine whether all gaps belong to ICGs or IWGs. Figure 4 shows the examples of word extraction from such image samples; the first and second images contain only ICGs, i.e. only one word appears to represent legal amount, and other images contain only IWGs, i.e. all characters within any word in the image are connected to each other.
265
KCXteeyJ
\pa*- C C r Vt^e^\J^f^j^
\^ix^\~^^
KfS^S
B5R*«I
^ ^
Fig. 4. Extracting words from image samples containing only inter-character gaps (ICG) or only interword gaps (rWG). After extracting such special image samples, the 2-class clustering process based on LBG algorithm is then applied to the remaining samples. The experimental results in Table 1 show that the RLEH method performs best with its correct separation rate of 70.1% among the three original distance measures. Here, it should be noted that correct extraction means that all words in a given legal amount image are perfectly isolated through the clustering procedure. Such extraction results differ somewhat from those of other comparative studies'0'". The main reason for that difference is that other studies adopted a different method to evaluate the performance of distance measures: the percentage of correctly ordered text lines (i.e., the sorted list of gaps for line contains all IWGs before any ICGs). In addition, the RLEH method has been analyzed and found to be more suitable for handling legal amounts for which only a specific small lexicon (32 specific words included for English legal amounts) is used, while the CH method has been reported to show better performance for a handwritten text with a large or an unlimited lexicon. Also, it can be found that all modified distance measures produced about 3% of better correct extraction rate, when compared to their corresponding original distance measures. Table 1. Experimental results of word separation
Distance Measures
BB
RLEH
CH
MBB
MRLEH
MCH
Correct Extraction (%)
69.0
70.1
68.3
71.8
72.7
71.7
266
^flAR* f/u*j
IAMSY^L*JLJI^~P<*4SVJJL2
hMsYJUcajf]
"pU^Jzj
\tZvA
(a)
LW
TV^WVlrX^^-^
~"TUH
ik^tA^v^A. -B-«L|
Vl^yM^X^
(b)
DrvJ (c) Fig. 5. Examples showing word separation error made by the original distance measures (upper part of each figure) and correction by their modified versions: (a) BB and MBB, (b) RLEH and MRLEH, and (c) CH and MCH
Figure 5 shows the examples of word separation illustrating the effectiveness of the modified distance measures. The words "hundred twenty" and "Thousand Two" shown in Fig.5 (a) and (b) are misrecognized as one word (upper part of each figure) due to the underestimation by BB method and shape sensitivity by RLEH method, respectively. However, these parts are successfully divided into two words by using the modified measures, as shown in the lower part of each figure. In Fig. 5 (c), the word "hundred" is divided into three parts incorrectly due to overestimation by the original CH distance but successfully merged by the modified method. 3.2 Combining Three Distance Measures Based on 4-Class Clustering Next, we introduce a new method of integrating the three aforementioned types of distance measures to reduce effectively the errors in each measure, thereby achieving further improvement in word extraction. The Venn diagrams shown in Fig. 6 represent the distribution of word extraction errors due to the original BB, RLEH, and CH methods, and their modified versions, respectively. From these diagrams, it can be found that many errors are commonly produced by two or all three distance measures even modified methods are applied. Thus a simple combining scheme such as the conventional majority voting is not expected to be effective in reducing such common errors.
267
X /
\
17
\
MBB 30
/
/ 211
\ 33 /
32 ^ L _ _ ^Y
27
MRLEH\ 2 1 / MCH (b)
Fig. 6. Distribution of word separation errors in (a) BB, RLEH, CH methods and (b) their modified versions
In order to compensate effectively those errors commonly produced by two or all of distance measures as well as the errors caused by one measure only, we propose a combining method based on a 4-class clustering technique as follows: (1) Estimate all gaps in a given image using a distance measure and divide them into 4 classes using a clustering algorithm. The partitions are sorted in order of magnitude of their centroids. (2) Assign an integer value, a e {2,1, - 1 , - 2} to all the gaps according to the class to which they belong. oc,
2 1 -1 - 2
for for for for
£,•£ class 1 gfe class 2 g, e class 3 gt e class 4
(3)
where, gt denotes the j-th gap in a given legal amount image. Thus, a larger positive value ( a = 2 ) is assigned to the gaps belonging to the class with a smaller centroid (class 1) to accentuate their possibility of ICG, and vice versa. (3) For the j-th gap, three different integer values, afB, a!lLE, and afH are assigned according to the distance measures: MBB, MRLEH, and MCH. (4) Define the i-th gap as ICG if a?B +afLE +afH is positive, and IWG, otherwise. In the second part of word extraction experiments on the proposed combining method, we achieved more than 75% of correct separation rate. This performance is much higher than those of any types of individual distance measures so far used in our experiment as well as those of other studies that used a similar database for evaluating their approaches, as shown in Fig. 7.
268 !
Integration Kim [14]
—
|
—
! ^
^
Zhou [13]
wam^mKfmmmmm^m,71.7 MRLEH wmmmmmmimmmmm MBB wmmmmgi^mg^mmmm 71.8 MCH
CH RLEH BB
1 65
1— 70 S e p a r a t i o n r a t e (%)
75
Fig. 7. Comparison of word extraction rate according to methodologies
The Venn diagram in Fig. 8 clearly shows that the errors generated from only one of three distance measures are almost perfectly removed. Moreover, it can be found that the errors common to two of three distance measures are also reduced effectively. However, the number of errors common to all of three distance measures is hardly reduced even when the combining method based on the 4-class clustering is applied.
Fig. 8. Error distribution by combining method using 4-class clustering A primary assumption to use the spatial gaps between connected components for separating a handwritten sentence into words is that the inter-word gaps are usually larger than inter-character gaps. However, such assumption does not match well with the problem of extracting words from a legal amount on bank cheque due to limited writing space. Figure 9 shows some representative examples of word extraction error caused by such gap irregularity. Accordingly, our analyses indicate that employing other information such as the maximum word length in the lexicon or the number of possible words in a legal amount is needed to remove such troublesome errors.
269
ft^k^J
W^W^fM^A/cK^: fej^J
. . .
»—_«_
—
\J
Fig. 9. Examples of word extraction error due to gap irregularity
3.3 Heuristic Re-separating Procedure for Under-Segmentation Errors Finally, we introduce a heuristic re-separating procedure considering the maximum possible word length in the lexicon and the writer's typical writing style in a limited writing space to deal with the under-segmentation errors (two or more words are misrecognized as one word). Through a series of word extraction experiments, we found that most errors belong to the case of under-segmentation; less than 16% of errors are identified as over-segmented when the combining method is applied. Actually, the case that two adjacent words overlap horizontally, and the case that some inter-word gaps are similar or slightly smaller than inter-character gaps in a given legal amount image cause the error of under-segmentation rather than over-segmentation. Therefore, we now focus on reducing such under-segmentation errors to improve effectively the overall performance of word extraction. The second step of word extraction, re-separating procedure starts by exactly sorting out the under-segmented parts from all the segmented parts resulting from the first extraction step using a distance cue. Such under-segmented parts are detected based on the assumption that the length of under-segmented part consisting of two or more words is usually longer than that of correctly segmented word. Such assumption is quite reasonable because any possible combination of two words is longer than or at least similar to the longest word (e.g. 'seventeen') in the lexicon of legal amount. The writer's typical writing style in a limited space is also considered as another important factor for determining if a given part is under-segmented or not. In many cases characters are written in a normal size at the beginning of a given writing space. But writing becomes compressed if the remaining space is estimated not enough to complete the legal amount. The procedure of the proposed heuritic approach for detecting and re-separating the under-segmented parts can be summarized as follows.
270
(1) For a given segmented part, calculate the length, / and position of left boundary, px. (2) Consider that part as a candidate for re-separation, if / is larger than the threshold, Th, defined as ocL. Here, L denotes the length of the entire image. The value of parameter a is varied according to px to reflect the writing style in a limited space. a=\
[ 0.3 0.25 [ 0.2
for px<03L for 03L0.1L
(4)
(3) Calculate three types of maximum gap distance within the candidate part using the MBB, MRLEH, and MCH methods, respectively. (4) Determine the candidate as under-segmented and divide it into two parts, if two of three maximum gaps are larger than a given threshold. Here, the threshold is empirically determined according to the type of distance measure and the left position of the given part. (4) Repeat the above steps for the re-separated parts to deal with the under-segmented part consisting of more than two words. By applying the above re-separating procedure, an 81.7% of correct separation rate was finally achieved in word extraction experiments on IRIS dataset. This performance is about 6% better than that of our combining method based on 4-class clustering technique. Thus we can conclude that the burden of under-segmentation was lowered remarkably through the re-separation procedure. However since the spatial gaps still play an important role in sorting out the under-segmented parts, many under-segmented errors due to the horizontal overlapping of words still remain unsolved. Accordingly, employing totally different types of methodologies such as an implicit segmentation scheme or a recognition-based segmentation strategy are required to overcome such limitation and moreover to reduce errors caused by touching between words and over-segmented errors. 4. Conclusions Various methods of separating a legal amount on bank cheque into words based on the spatial gaps between connected components have been presented in this chapter. The typical existing distance measures for gap estimation: BB, RLEH, and CH, are quite simple but all suffer from the inherent problem of underestimation and overestimation, thereby causing considerable errors of word extraction. To alleviate such problem, we have introduced a modified version of each distance measure; the left and right boundaries of the bounding box were adjusted for the BB and RLEH methods, and a pair of convex hulls enclosing equally divided connected components was introduced to the CH method. Moreover, an efficient method of integrating three different types of distance measures based on 4-class clustering technique has been newly presented to compensate effectively the errors in each measure whereby further improvement in performance of word extraction is expected. Lastly, we have introduced a heuristic re-segmentation procedure based on the possible maximum word length in lexicon and the writer's typical writing style in a limited writing space to deal with the under-segmented errors due to gap
271 irregularity referring that some inter-word gaps are similar or slightly smaller than intercharacter gaps. Through a series of word extraction experiments on CENPARMI IRIS database, we found that the modified measures show a better performance in terms of their word separation rates compared with their corresponding original distance measures. Also, the proposed combining method shows much higher performance than those of any types of individual distance measures used in our experiment. In addition, by applying the reseparating procedure, an 81.7% of notable correct separation rate was finally achieved. As for future work, employing other methodologies such as implicit segmentation or recognition based segmentation approach and combining with our extraction method based on the physical distance cue will be studied to solve the problems of undersegmentation by overlapping or touching of words and over-segmentation. References 1. D. Guillevic and C.Y. Suen. Recognition of legal amounts on bank cheques. Pattern Analysis and Applications, 1(1): 28-41, 1998. 2. G. Kaufmann and H. Bunke. Automated reading of cheque amounts. Pattern Analysis and Applications, 3(2): 132-141, 2000. 3. C.O.A. Freitas, A. Yacoubi, F. Bortolozzi, and R. Sabourin. Brazilian bank check handwritten legal amount recognition, In Proc. 13th Brazilian Symposium on Computer Graphics and Image Processing, pages 97-104, 2000. 4. R.G. Casey and E. Lecolinet. A survey of methods and strategies in character segmentation. IEEE Trans, on Patt. Anal, and Mack Intell, 18(7): 690-706, 1996. 5. A.W. Senior and A.J. Robinson. An off-line cursive handwriting recognition system. IEEE Trans, on Patt. Anal, andMach. Intell, 20(3): 309-321, 1998. 6. G. Kim, V. Govindaraju, and S.N. Srihari. An architecture for handwritten text recognition systems. Int. J. Document Analysis and Recognition, 2(1): 37-44, 1999. 7. M. Feldbach and K.D. Tonnies. Word segmentation of handwritten dates in historical documents by combining semantic a-priori-knowledge with local features. In Proc. Int. Conf. Document Analysis and Recognition, 1: 333-337, 2003. 8. B. Yanikoglu and RA. Sandon. Segmentation of off-line cursive handwriting using linear programming. Pattern Recognition, 31(12): 1825-1833, 1998. 9. G. Seni and E. Cohen. External word segmentation of off-line handwritten text lines. Pattern Recognition, 27(1): 41-52, 1994. 10. U. Mahadevan and R.C. Nagabushnam. Gap metrics for word separation in handwritten lines, la. Proc. Int. Conf. Document Analysis and Recognition, 1: 124-127, 1995. 11. S.H. Kim, S. Jeong, G.S. Lee, and C.Y. Suen. Gap metrics for handwritten Korean word segmentation. Electronics Letters, 37(14): 892-893, 2001. 12. U.V. Marti and H. Bunke. Text line segmentation and word recognition in a system for general writer independent handwriting recognition. In Proc. 6th Int. Conf. Document Analysis and Recognition, pages 159-163, 2001. 13. J. Zhou, C.Y. Suen, and K. Liu, A feedback-based approach for segmenting
272
handwritten legal amounts on bank cheques. In Proc. Int. Conf. Document Analysis and Recognition, pages 887-891, 2001. 14. K.K. Kim, J.H. Kim, Y.K. Chung, and C.Y. Suen. Legal amount recognition based on the segmentation hypotheses for bank check processing. In Proc. Int. Conf. Document Analysis and Recognition, pages 964-967, 2001. 15. M. Zimmermann and H. Bunke. Automatic segmentation of the LAM off-line database for handwritten English text," In Proc. 1& Int. Conf. Pattern Recognition, 4: 35-39, 2002. 16. K. Kimura, M. Shridhar, and N. Narasimhamurthi. Lexicon directed segmentationrecognition procedure for unconstrained handwritten words. In Proc. Int. Workshop on Frontiers in Handwriting Recognition, pages 122-131,1993. 17. G. Kim and V. Govindaraju. Handwritten phrase recognition as applied to street name images. Pattern Recognition, 31(1): 41-51, 1998. 18. J. Park and V. Govindaraju. Use of adaptive segmentation in handwritten phrase recognition. Pattern Recognition, 35: 245-252, 2002. 19. R. Manmatha and N. Srimal. Scale space technique for word segmentation in handwritten manuscript. In Proc. 2nd Int. Conf. Scale-Space Theories in Computer Vision, pages 22-23, 1999. 20.Y Wang, I.T. Phillips, and R. Haralick. Statistical approach to word segmentation. In Proc. 15th Int. Conf. Pattern Recognition, 4: 555-558, 2002. 21. J. Schurmann. Document analysis - from pixels to contents. Proc. IEEE, 80(7): 11011119,1992. 22. Y Linde, A. Buzo, and R.M. Gray. An algorithm for vector quantizer design. IEEE Trans, on Communications, COM-28(l): 84-95, 1980.
CHAPTER 3.4
OCR ASSESSMENT OF PRINTED-FONTS FOR ENHANCING HUMAN VISION
Ching Y. Suen, Qizhi Xu, and Cedric Devoghelaere Centre for Pattern Recognition and Machine Intelligence (CENPARMI), Concordia University 1550 de Maisonneuve Blvd. West, Suite GM606, Montreal, H3G 1M8, Canada E-mail: {suen, qxu}@cenparmi.concordia.ca
The ways humans and computers read texts printed in different fonts are described from a pattern recognition point of view. The results of several experiments involving OCR technology are presented. The characteristics of some fonts have been analyzed, and factors affecting the recognition rate have been identified. Suggestions are made on the desirable features of print-fonts which could enhance human reading.
1. Introduction Eyes bring us visual information, giving us the power of observing, judging, estimating, and discriminating things around us. Indeed eyesight is of paramount importance to our lives. With the great progress of civilization and human evolution, and advances in art, science, and technology, humans have to do more and more readings to keep up with our daily lives in an active society. As a result, we have been using our eyes more and more in acquiring information. Unfortunately such activities have also brought unprecedented demand on our eyes. We can easily see that many children and adults in the modern society have damaged their eyes. Wearing eyeglasses has become a very common phenomenon these days. Therefore it is very important to carry out research related to legibility of type-fonts and to find ways of enhancing legibility and maintenance of good eyesight. The long-term goal of this research is to identify print fonts or invent some new ones that will soothe our eyes and prevent their decline due to too much reading. 2. Legibility of Printed-Fonts: Based on the OCR Technologies of Computers Simply by looking at the various printed-fonts, one can easily deduce that they are not equally legible, i.e. some fonts are more legible than others. As a result, more processing is required for those fonts which are less legible, and we hypothesize that fonts, which produce a higher legibility, are better for our eyes from both the viewpoints of eyestrain and processing time. 273
274
Since the OCR systems are strongly inspired by human reader recognition methods, the legibility of fonts is discussed in this section by using the OCR technologies of computers. We will present several interesting experiments about the kinds of printedtexts which are more legible to the computer, which will help us to identify the most legible printed-fonts and lead us to discover their properties responsible for legibility. 2.1 Experiment 1: Recognition of Texts Printed in Different Fonts In this experiment, a commercial OCR system, which is supposed to be an omni font text recognizer, is used to recognize a text paragraph printed in different fonts. Based on the recognition results and analyses some interesting conclusions have been obtained. Throughout the history of printing, hundreds of font types have been designed for typewriters, typesetters, and printers. In this experiment, we base our investigation on five different fonts: Garamond, Baskerville, Bodoni, Century Expanded and Helvetica representing different periods of the history of typography. Some discussions about these five fonts are given below, and more descriptions can be found in1. •
Garamond: Garamond is a typical Old Style face having very little contrast between the thicks and the thins, heavily bracketed serifs, and oblique stress. The letterforms are open and round. The capital letters are shorter than the ascenders of the lowercase letters. An example is given below:
Garamond •
Bakerville: Baskerville is an excellent example of a Transitional typeface. Transitional typefaces are so called because they form a bridge between the Old Style and the Modern faces. Compared to the Old Style, Baskerville shows greater contrast between the thicks and thins, serifs are less heavily bracketed, and the stress is almost vertical. The letters are very wide for their x-height, are closely fitted, and are of excellent proportions. An example is given below:
Baskerville •
Bodoni: Bodoni is a Modern typeface which introduces the fashion for faces with a stronger contrast between the thicks and the thins, unbracketed serifs and a strong vertical stress. Although Bodoni has a small x-height it appears very wide and black. Because of strong vertical stress, accentuated by its very thicks and hairline thins, the horizontal flow necessary for comfortable reading is impaired. Bodoni therefore must be well-leaded. An example is given below:
Bodoni •
Century Expanded: Century Expanded is an example of a refined Egyptian typeface. It is based on a type called Century. The Egyptian style appeared from this wish, characterised by thick slab serifs and thick main strokes with little contrasts between
275
the thicks and thins. Century Expanded has a large X-height and should be leaded. The large letters and simple letterforms combine to make it very legible and especially popular for children's books. Like most members of the Egyptian family of typefaces, Century Expanded makes a good display type because of its boldness.
Century Expanded Helvetica: Helvetica is a contemporary typeface, although it has a large X-height and narrow letters, its clean design makes it very readable. Sans serif types in general have relatively little stress and the strokes are optically equal. Because there is no serif to aid the horizontal flow that we have seen is so necessary to comfortable reading, sans serif type should always be leaded.
Helvetica The above five fonts can be grouped into three principal categories in style1: - Roman (Garamond, Baskerville, Bodoni) - Egyptian (Century) - Sans Serifs (Helvetica)
|-
Serif Types
For the test, images of five paragraphs obtained by scanning printed documents with same content but printed in the five different fonts (same 10 points size) are recognized by a commercial OCR system. The recognition results are given below:
Garamond Baskerville Bodoni Century Expanded Helvetica
Misrecognized characters 0% 0% 0% 0% 0%
Uncertain characters 1.55% 1.36% 4.08% 1.74% 0.19%
Average
0%
1.77%
From the above table, we observe that the recognition rate reaches 100% with an average certainty rate of 98.23%. Here the commercial OCR gives the uncertain characters based on the confidence value of the recognition. If the three principal styles are considered, the following results can be obtained:
Roman Egyptian
Recognition rate 100% 100%
Certainty Rate 97.67% 98.26%
Serifs Sans Serifs
100% 100%
97.82% 99.81%
Here Roman and Egyptian are Serif types. In this experiment, although the test is based only on a simple text sample, we can infer the following: •
It seems that sans serif types are recognized with a greater certainty than serifs type. Therefore, we wonder: Why are Sans Serif types more easily recognised than Serif ones ? To answer this question, first based on the following simple image we can see the different designs of the Serifs and the Sans Serif types:
Serifs
Sans Serifs
m
m
We know that Serifs help the reader by reinforcing the horizontal flow of reading, they are often horizontal bars that accompany the reader's movement of the eyes when it goes along the line. But the main drawback in the use of Serifs is the increased complexity in the design of characters. In addition, by recognizing a text with an OCR system, we cannot really talk about a horizontal flow of reading because usually the system makes use of integrated segmentation and recognition algorithms to recognize individual characters and words. Therefore serifs are not so important in this situation, and they can even make it more difficult in segmentation (often serifs generate more touching characters). Compared with Serif texts, Sans Serif texts are more easily recognized by the OCR system. The possible reason is that Sans Serif types have a clearer and lighter design for characters. From the recognition results we find that the uncertainty rate for Bodoni is really well above the mean. So Why Bodoni has such a high uncertainty rate? According to the property of font, Bodoni has a very strong contrast between thicks and thins. This strong contrast modifies lightly the general aspect of some characters, highlighting some parts at the expense of others. This contrast factor perhaps makes some characters really difficult to be recognised by an OCR system. Since the "s" represents 52.30% of the uncertainties for the Bodoni type, more discussions about this character is given below: Design of the "s" in Bodoni:
277
thin upper part strong central part thin lower part
We can see that the "s" is unbalanced in different parts, which may create problem in several ways. If the recognizer uses features such as the repartition of black pixels, there will be a strong concentration of pixels in the middle which is not really characteristic of an "s". Moreover it can also add problems in the segmentation phase, because the thin strokes might be seen as insignificant, and the character might be segmented through these strokes. Therefore we can infer that very strong contrast between thicks and thins is not good for OCR recognition. 2.2 Experiment 2: Influence of Leading In typography, the leading, which is the spaces between lines, is a variable and can be adjusted properly for comfortable reading. It is important to note that adding spaces between lines does not affect the type size or the length of the line. It merely moves the lines further apart which has an influence on our reading facility. If the lines are set solid (with no linespacing), it is sometimes (depending on the type) difficult to read, and when the horizontal flow of reading is important, the horizontal flow of reading can be broken and it becomes more difficult for the reader. As the OCR systems are often inspired by the human way of reading, we will examine how much the leading influences the recognition rate. For this purpose, same texts written with different leadings have been recognized, and based on the results we have inferred some aspects about the influence of leading. For this test, by using the commercial OCR system, we try to recognize text samples with the same content but printed in the Garamond font with various linespacings: solid, 1 point, 2 points and 3 points linespacing:
solid lpt linespacing 2pt linespacing 3pt linespacing
Uncertain characters 2 3 2 1
Errors 0 0 0 0
Missing Characters 0 0 0 0
Extra Characters 0 1 0 0
From the above table, we can see that the results with the various linespacings do not really differ. Even if the linespacing has a great importance for the comfort of reading of a human reader, its influence on the recognition rate of a text by an OCR is very small.
278
2.3 Experiment 3: Recognition Without Context In our experiments, we find that the recognition and certainty rates of a certain typeface increase if we recognize a complete text instead of isolated characters. This is because that the commercial OCR system makes use of some lexical knowledge in its mechanism of error-correction. A dictionary helps it to resolve uncertainties when it hesitates on certain characters, that is, the OCR recognizes the characters in their surroundings (their neighbors) and has also access to context knowledge (dictionary) to resolve the major uncertainties and mistakes. In order to assess the real recognition ability of the OCR system for different fonts, we try to reduce the influence of context processing used in the commercial OCR. An experiment is designed here to recognize bigram text files generated by grouping all possible bigrams as follows:
12 points: recognition of bigrams| aaabacadae af agahaia; akalamanaoapaqaras atauavawaxayazbabbbc bdbebf bg bh bi bj bk bl bm bn bo bp bq br bs bt bu bv bx by bz ca cb cc cd ce cf eg ch ci cj ck cl cm en co cp cq cr cs ct cu cv cw ex cy cz da db dc dd de df dg dh di dj dk dl dm dn do dp dq dr ds dt du dv dw dx dy dz ea eb ec ed ef eg eh ei ej ek el em en eo ep eq er es et eu ev ew ex ey ez fa fb fc fd fe ff fg fhfifj fkflfm fn fo fp fq fr fs ft fu fv fw fx fy fz ga gb gc gd ge gf gg g n g? g) gk g1 g™ gn g° gP g
Bigram text files with the same character size but different fonts (five fonts) are recognized by the commercial OCR system, and the results are tabulated below:
279
Garamond Baskerville Bodoni Bookman Helvetica
Uncertain characters 35.9% 24.4% 38.1% 26.1% 9.3%
errors 6.0% 4.7% 5.5% 6.5% 1.6%
From the results we can see that the text file printed in Helvetica is much better recognized. Since the Helvetica font is the only sans serif type font in this test, we believe that its simpler design makes the recognition of bigrams easier. Based on more detailed analyses on the errors and uncertain characters, no possible confusion between "i", "1" and " 1 " is found for sans serifs type (Helvetica) (these confusions represent half the mistakes on the otiier type). We also found that for all types, there is a difficulty in differentiating "e" from "c". In addition, some common errors caused by two connected characters, e.g. be/be, fl/fi, U/ll, 11/11 ...highlight some difficulties in printed document recognition by using an OCR system. 2.4 Experiment 4: Recognition of Roman Type Fonts In this experiment, twenty-one Roman type fonts have been investigated, and some interesting results have been obtained. In the following, two examples are given. 2.4.1 Bodoni Ultra Italic (a) Main characteristics of the character set Italic form, but with only a slightly leaned angle Really strong contrast between thicks and thins, the thicks are really impressive and the thins are hairlinelike. The characters are really dark f and j have a very special design (b) Original set
ABCDEFGHMJKLMNOPQRSTUVWX YZ abedefghijktmnopqrstuvwxyz 12345G7890 BODONI, ULTRA ITALIC
(c) Output of the OCR system In the following recognition result, both errors and uncertain characters are highlighted:
ABCDEFGHIJKLMNOPQRSTUVWX
\Z
280
ubcdefghijhltnnop
qrstuvwxyz
1234567890
(d) Statistical results • •
62 characters uppercase, lowercase, digits Uncertain characters Details Number Percentage
B,H,J,K,L,M,N,0,P,Q,R,S, U,V,W,Z,d,e,i,j,n,o,u,v,w,y 29 46.77%
Misrecognized charatcers k (h), m(tn) a (u), Y (\) 4 6.45%
Extra characters
Missing characters
0 0
0 0
(e) Discussions of the results This font type is error prone. The error rate reaches 6.45% which is really high for an OCR system on recognizing such restricted samples. Some of the errors are due to the particular design of this type, especially the rather strong contrast between thicks and thins:
/
.
Because of a particularly strong contrast between thicks and thins, the highlight part is likely not recognized as being part of the character k (but rather as noise), and therefore an "h" is produced by the OCR system.
m a
/
.
rn
Similarly because of this emphasized contrast, the "m" is wrongly segmented. Usually strokes which are too thin are difficult to be detected. Here the letter "m" is segmented into two parts, which are recognized as "t" plus "n". One more example of the error caused by the strong contrast between thicks and thins. The top stroke of "a" does not seem to have been detected. The OCR system recognizes the character represented on the right as letter "u".
M
2.4.2 Caledonia (a) Main characteristics of the character set In this typeface, the design is classical, the contrast between thicks and thins is week,
281
serifs are straight and of medium size and the characters are well balanced. But as in several other Roman types, the letter "J" has its tail under the baseline. (b) Original set
ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 1234567890 CALEDONIA
(c) Output of the OCR system In the following recognition result, both errors and uncertain characters are highlighted:
ABCDEFGHUKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 1234567890 (d) Statistical results • •
62 characters uppercase, lowercase, digits Uncertain characters Details Number Percentage
H, I, J, Q, h, j 6 9.7%
Misrecognized characters
Extra characters
Missing characters
0 0%
0 0%
0 0%
(e) Discussions of the results Caledonia is a perfectly recognized typeface with a high confidence factor (>90%). The uncertain characters: H, I, J, Q, h, j are uncertain in numerous Roman typefaces. The possible reasons for these uncertainties are wrong segmentation and insufficient intercharacter spacing, etc. In addition, the "J" has a particular design, which is different from "J" of common font types. This special "J" may also introduce some uncertainty. With these good results, Caledonia shows its high legibility and being so readable, and it is natural that Caledonia is used as main types for books. 2.5 Experiment 5: Characteristic Analyses of 15 Fonts with Different Recognition Performances In order to obtain more comprehensive and reliable statistical results for studying the legibility of different fonts by considering the performances of OCR systems, we have designed a new experiment recently. In this experiment, 15 different fonts in size 8 and
282
size 10, seven different contents of documents, and 4 popular commercial OCR systems have been selected for the study: (a) Fonts: 15 different fonts are selected and studied in this experiment, which are shown below. T i m e s N e w Roman New C e n t u r y School Book Albertus Batang
Courier
Palatino
Century Microsoft sans Serif
Letter Gothic
Helvetica Garamond
Impact
Arial
Georgia lasttenschweler
We can see from the above samples that some fonts are Sans Serifs and some are Serifs. In addition, these 15 fonts consist of various x-highs, and other different properties. (b) Documents: In order to generate more general and reasonable recognition performance, seven different contents of documents have been tested, which come from the following different sources: > > > > > > >
Documents from Devoghelaere's Report2 110 High Frequency Words3 126 Low Frequency Words3 Documents from Elementary Book Non-Sense Words by using Chinese pinyin syllables N-Gram words: Top 50 2-Gram, 3-Gram, 4-Gram and 5-Gram words4 Documents from Better Type5
(c) OCR systems: In this experiment, the recognition performance of each font has been tested by four existing popular commercial OCR systems Based on the above test set, a simple average rule has been used to obtain an average recognition rate and a corresponding rank for each font, as shown in the following table6:
283
K.illr
f-..,v III-.
Arial
99.202
^
Microsoft Sans serif
99.151
Batang
98.673
Courier New
98.663
Letter Gothic
98.655
Century
98.571
Palatino
98.534
New Century School Book
98.270
Georgia
98.240
Time New Roman
98.231
Albertus
97.604
Garamond
96.669
Impact
69.918
Haettenschweiler
29.372
i,,
' •»
i •
II, i
-
-
i
!i]iiiiii L'
From the results above, we can observe the best 4 fonts and 4 poorest fonts based on the recognition rates: 4 good legible fonts: Helvetica, Arial, Microsoft Sans serif, and Batang 4 poor legible fonts: Albertus, Garamond, Impact, and Haettenschweiler In order to analyze the properties of good legible fonts and poor legible fonts, several measurement factors have been selected or defined as follows: (a)
Contrast: Typographic contrast is the ratio of the weights of vertical to horizontal letter elements. It can be expressed as:
C T =T V /T h
(1)
Where Tv is the vertical stem thickness and Th is the horizontal stroke thickness. (b)
Evenness of the stroke: The evenness of the stroke is the degree of contrast between the thickest and thinnest parts of the letter. A font with an even stroke weight will produce a smooth, while a face with a strong thick/thin contrast will produce a texture analogous to the rough tweed fabric [5]. The evenness of the stroke can be
284
expressed as follows:
E = Thicknest / Thinnest (c)
(2)
Weight: The weight of a font is a measure of its heaviness or boldness. It can be expressed as:
W = T V /X
(3)
Where Tv is the vertical stem thickness and X is x-height. Apparently, the larger the W is, the darker the typeface appears. It is also true that narrow fonts appear darker than wide fonts. (d)
Aspect ratios among x-height, ascender, descender and body size: There are four measurement factors formed from x-height, ascender, descender and body size. They are:
XA = X - height / Ascender XD = X - height / Descender AD = Ascender / Descender XB = X - height / Bodysize (e)
(4) (5) (6) (7)
Inter-letter space: The Inter-letter space can be treated as a legible measurement factor, because very small inter-letter spaces will affect the recognition performance significantly. For example, for fonts Impact and Haettenschweiler, the small interletter space is the main factor that reduces the performance significantly. But, if the inter-letter space is not very small, which can not cause the touching problem seriously, this factor is not important in OCR recognition systems.
In our experiment, from the analyses of the relatively good and poor legible fonts based on the measurement factors, the following phenomenon has been observed: •
•
•
Among our test fonts, the first three good legible fonts are all sans serif, but the two most illegible poor legible fonts are sans serifs too. By considering the errors, we can find that this Serif or Sans Serif factor should be investigated together with other factors such as inter-letter space. The Inter-letter space can be considered as a legibility measurement factor, because very small inter-letter spaces will affect the recognition performance significantly, such as the fonts Impact and Haettenschweiler. Weight is an important factor to measure the darkness of a font. In our experiment, the very dark fonts Impact and Haettenschweiler have poor legibility, but for the light fonts Batang and Courier, the legibility is not bad.
285
• •
Our experiments show that the fonts with even contrast of horizontal stroke/ vertical stem and thickest/thinnest have better legibility. The relationships between x-height and ascender, descender, body size are important, which need further analyses.
From the above conclusions, we can find that some similar conclusions have been drawn in the previous experiments (e. g. Experiments 1 and 4) which are based on smaller test sets. 3. Concluding Remarks Our experiments have revealed that not all printed-fonts produce the same recognition by OCR technologies. It was found that legibility is affected by several factors. We tried to apply some OCR algorithms to analyze the properties that make a text legible and discover some problems associated with reading and legibility. In a separate study by Suen and Komoda7, it was revealed that font-styles played an important role in legibility, ambiguities between letters, and comprehension in reading. As the incidence rate of myopia of school children becoming more acute in recent years, coupled by the deterioration of the eyesight of adults, it is suggested that more extensive investigation on this research topic be carried out to stop the growth of myopia in humankind. In our experiments, we found that sometimes the commercial OCR system could not be fully controlled so that some properties of fonts cannot be investigated well. Therefore, we are building an OCR system in our center in order to find more detailed information related to font legibility from the OCR's point of view. References 1. James Craig. Designing with type. Watson-Guptill Publications, New York, 1992. 2. Cedric Devoghelaere. Analysis of font recognition. Technical Report, CENPARMI, Concordia University, Montreal, Quebec, Canada, August 2003. 3. Henry Kucera and W. Nelson Francis. Computation analysis of present-day American English. Brown University Press, Providence, Rhode island, 1970. 4. Ching Y. Suen. N-gram statistics for natural language understanding and text processing. IEEE Trans, on Pat. Anal, and Mach. Intel., Vol. PAMI-1, No. 2, pp. 164172, April 1979. 5. Betty Binns. Better type. Waton-Guptill Publications, New York, 1989 6. Yan Zhang. Legibility of type-fonts. Technical Report, CENPARMI, Concordia University, Montreal, Quebec, Canada, April 2004. 7. Ching Y Suen and Melvin k. Komoda. Legibility of digital type-fonts and comprehension in reading. In Proc. of the Int. Conf. on Test Processing and Document manipulation, pp. 178-187, University of Nottingham, U.K., April 1986
CHAPTER 3.5 CLUSTERING AND CLASSIFICATION OF WEB DOCUMENTS USING A GRAPH MODEL
Adam Schenker University of South Florida 4202 E. Fowler Avenue ENB 118, Tampa, FL 33620 E-mail: [email protected]
Horst Bunke University of Bern CH-3012 Bern, Switzerland E-mail: [email protected]
Mark Last Ben-Gurion University of the Negev Beer-Sheva 84105, Israel E-mail: [email protected]
Abraham Kandel University of South Florida 4202 E. Fowler Avenue ENB 118, Tampa, FL 33620 and Faculty of Engineering, Tel-Aviv University Tel-Aviv 69978, Israel E-mail: [email protected]
In this chapter we provide a summary of our previous work concerning the application of traditional machine learning techniques to data represented by graphs. We show how the fc-means clustering algorithm and the A;-nearest neighbors classification algorithm can easily and intuitively be extended from dealing with vector representations to graph representations. We present some of our experimental results, which confirm that the addition of structural information, not present in vector representations, improves both clustering and classification performance when dealing with web documents.
287
288
1. Introduction In the fields of pattern recognition and machine learning, two popular tasks are clustering and classification. With clustering, the goal is to separate a collection of objects into groups called "clusters." Objects in the same cluster should be similar to each other, but dissimilar to objects in other clusters. Classification algorithms assign a label to each object in a collection, based on some previously seen training examples. One practical application of these methods is the clustering and classification of web documents. Clustering of web documents is done to organize the documents into related groups. This has benefits when the classes are not known a priori, such as in web search engines,1 since it allows systems to display results grouped by cluster (topic), in comparison to the usual "endless" ranked list, making browsing easier for the user. Content-based classification of web documents is useful because it allows users to more easily navigate and browse collections of documents. 2 ' 3 Such classifications are often costly to perform manually, as it requires a human expert to examine the content of each web document. Due to the large number of documents available on the Internet in general, or even when we consider smaller collections of web documents, such as those associated with corporate or university web sites, an automated system which performs web document classification is desirable in order to reduce costs and increase the speed with which new documents are classified. The usual method of representing documents in traditional information retrieval systems is as a series of m numeric values, which is usually taken to be a vector in the Euclidean space 5Rm. Each value is associated with a specific term (word) that may appear on a document, and the set of possible terms is shared across all documents; this is called the vector-space model of information retrieval. 4 The values associated with each vector may be binary, indicating the presence or absence of the corresponding term, or they may be non-negative integers, representing the number of times a term appears on a document (i.e. term frequency). Non-negative real numbers can also be used, in this case indicating the importance or weight of each term. These values are derived through a method such as the popular inverse document frequency model (tf • idf)4, which reduces the importance of terms that appear on many documents. Regardless of the method used, each series of values represents a document and corresponds to a point (i.e. vector) in a Euclidean feature space. This model is often used when applying machine learning techniques to documents, as there is a strong mathematical foundation for performing distance measure and centroid calculations using vectors. However, this method of document representation does not capture important structural information, such as the order and proximity of term occurrence, or the location of terms within the document; more importantly, it does not take into account any web-related information. Only recently have a few papers appeared in the literature that deal with graph representations of documents. Lopresti and Wilfong compare web documents using a graph representation that primarily utilizes HTML parse information, in addition to
289 hyperlink and content order information. 5 In their approach they use graph probing, which extracts numerical feature information from the graphs, such as node degrees or edge label frequencies, rather than comparing the graphs themselves. In contrast, our representation uses graphs created solely from the content, and we use the graphs themselves rather than a set of extracted features. Liang and Doermann represent the physical layout of document images as graphs. 6 In their layout graphs nodes represent elements on the page of a document, such as columns of text or headings, while edges indicate how these elements appear together on the page {i.e. spatial relationships). This method is based on the formatting and appearance of the documents when rendered, not the textual content (words) of a document as in our approach. In this chapter we describe methods of extending classical machine learning algorithms, the fc-means clustering algorithm and the fc-nearest neighbors (/c-NN) classification algorithm, to work with graph representations of data rather than vector representations. The benefit of this method, as mentioned above, is that it allows us to take advantage of the additional structural information that is present in the graph representations, while still using well-known, proven algorithms. The chapter is organized as follows. In Sec. 2 we give the mathematical notation and equations needed to describe our methods. We explain how the fc-NN algorithm can be extended to work with graph-based data in Sec. 3; similarly, in Sec. 4, we describe the graph-theoretic /c-means algorithm. We explain how web documents can be represented by graphs in Sec. 5. In Sec. 6 we give some experimental results which compare the performance of the graph-based approaches to the vector-based methods. Conclusions are presented in Sec. 7. 2. Formal Notation In this section we will give the formal mathematical notation and equations which pertain to the remainder of the chapter. Section 2.1 describes some fundamental definitions regarding graph theory. In Sec. 2.2 we give some of the distance measures typically used with vector representations, which are what are traditionally used for machine learning algorithms. To allow these algorithms to work with graphs we need methods of measuring distances between graphs rather than vectors; these are described in Sec. 2.3. In Sec. 2.4, we introduce the last graph-theoretic concept relevant to the current work, the median graph. 2.1. Definitions
relating
to graphs
Graphs are a mathematical formalism for dealing with structured entities and systems. In basic terms a graph consists of vertices (or nodes), which correspond to some objects or components. Graphs also contain edges, which indicate relationships between the vertices. The first definition we have is that of the graph itself. Each object (web document, in the context of this chapter) in the collection will be represented by such a graph:
290
Definition 1: A graph G is defined by a 4-tuple (quadruple): G = (V,E,a,/3), where V is a set of vertices (also called nodes), E C V x V is a set of edges connecting the vertices, a : V —> S y is a function labeling the vertices, and /?:.£—> T,E is a function labeling the edges (Ey and E ^ being the sets of labels that can appear on the nodes and edges, respectively). The graphs we will use in this chapter are directed graphs with node and edge labels. The next definition we have is that of a subgraph. One graph is a subgraph of another graph if it exists as part of the larger graph: Definition 2: A graph G± = (Vi,Ei,ai,/3i) is a subgraph of a graph G2 = (V2.JE2.a2,ft), denoted Gx C G 2 , if Vi C V2, E1CE2n (Vi x Vi), ax{x) = a 2 (x) Vx e Vi, and (3i((x,y)) = /32{{x,y)) V(x,y) € £ 1 . Conversely, graph G2 is also called a supergraph of Gi. Next we have the important concept of the maximum common subgraph, or mcs for short, which is the largest subgraph a pair of graphs have in common: Definition 3: A graph G is a maximum common subgraph {mcs) of graphs G\ and G 2 , denoted mcs(G1,G2), if: (1) G C Gx (2) G C G 2 and (3) there is no other subgraph G' (G' C Gu G' C G 2 ) such that \G'\ > \G\. (Here \G\ is intended to convey the "size" of the graph G; we define it to be the sum of the number of nodes and edges in a graph, |V| + \E\.) Complementary to Definition 3, we also have the concept of minimum supergraph:
common
Definition 4: A graph G is a minimum common supergraph7 (MCS) of graphs Gi and G 2 , denoted MCS{Gl,G2), if: (1) Gi C G (2) G 2 C G and (3) there is no other supergraph G' (Gi C G', G 2 C G') such that \G'\ < \G\. 2.2. Vector-based
distance
measures
When representing data by vectors, the distances between two objects can be computed using the Euclidean distance in m dimensions: distEUCL(x,y)
=
S2(Xi-yiy,
(1)
where Xi and y, are the ith components of vectors x = [xi,x2,. •• ,xm] and y = [y 1) S/2> - - - ,2/m], respectively. However, for applications in web document clustering and classification, which is what this chapter is concerned with, the cosine similarity measure 4 is often used due to its length invariance property. We can convert this to a distance measure by the following: distcos(x,y)
= 1- ,
\\x\\ ' 112/11
•
(2)
291
Here • indicates the dot product operation and || • • • || indicates the magnitude (length) of a vector. Another popular distance measure for determining document similarity is the extended Jaccard similarity 4 , which is converted to a distance measure as follows: m distJAC(x,
2.3. Graph-based
y) = l -
distance
x
2
i=l1 'm
x
iVi „ „, 2 ' ^ m Z^i=l i + 1^»=1 Vi ~ A^i=l Xiy*
^rri
•
(3)
measures
We have examined several distance measures for measuring the similarity of graphs. The first distance measure (MCS) is a classic graph distance measure based on the maximum common subgraph 8 : Ai«t (r r \ i distMcs{Gi,G2) = 1
\mcs{G1,G2)\ . . ., , . , (4) max(|Gi|,|G 2 |) where G\ and G 2 are the graphs to compare, mcs(Gi,G2) is their maximum common subgraph, |---| is the size of a graph, and max(- • •) is the usual maximum operation. The concept behind this distance measure is that as the size of the maximum common subgraph of a pair of graphs becomes larger, the more similar the two graphs are (i.e. they have more in common). The larger the maximum common subgraph, the smaller distMCs(Gi,G2) becomes, indicating more similarity and less distance. If the two graphs are in fact identical, their maximum common subgraph is the same as the graphs themselves and thus the size of all three graphs is equal: |C?i| = |G 2 | = \mcs(Gi,G2)\. This leads to the distance, distMCs(Gi,G2), becoming 0. Conversely, if no maximum common subgraph exists, then |mcs(Gi, G 2 )| = 0 and distMCs{G\,G2) = 1. This distance measure has been shown to be a metric, and produces a value in [0, l]. 8 A second distance measure (WGU) which has been proposed by Wallis et al.9 is: 7,^ ^ N l m cs(Gi,Go)| , , v u 2 distWGU Gj, G a ) = 1 /' . (5 |Gi| + |G 2 | - |mcs(Gi,G 2 )| This distance measure behaves similarly to MCS. If the maximum common subgraph does not exist (i.e. \mcs(Gi, G2)\ — 0), then distwGu(G\,G^) = 1. If the maximum common subgraph is identical to the original graphs, |Gi| = |G 2 | = |mcs(Gi,G 2 )|, then the graphs Gi and G 2 are identical and thus dis£wGf/(Gi,G 2 ) = 0. The denominator used in this method is based on the idea of "graph union": it represents the size of the union of the two graphs in the set theoretic sense; specifically, adding the size of each graph (\Gi\ + |G 2 |) then subtracting the size of their intersection (|mcs(Gi,G 2 )[) leads to the size of the union (the reader may easily verify this using a Venn diagram). The motivation for doing this is to allow for changes in the smaller graph to exert some influence over the distance measure, which does not happen with MCS. This measure was also demonstrated to be a metric, and creates distance values in [0, l]. 9
292
The third distance measure (MMCS), proposed by Fernandez and Valiente, is based on both the maximum common subgraph and the minimum common supergraph 7 : distMMcs(G1,G2)
= \MCS{GUG2)\
- \mcs(Gi,G2)\,
(6)
where MCS(G\,G2) is the minimum common supergraph of graphs Gi and G2. The concept that drives this distance measure is that the maximum common subgraph provides a "lower bound" on the similarity of two graphs, while the minimum common supergraph is an "upper bound". If two graphs are identical, then both their maximum common subgraph and minimum common supergraph are the same as the original graphs and |Gi| = \G2\ = \MCS(Gi,G2)\ = \mcs(Gi,G2)\, which leads to distMMCs(Gi,G2) = 0. As the graphs become more dissimilar, the size of the maximum common subgraph decreases, while the size of the minimum common supergraph increases. This in turn leads to increasing values of distMMCs{G\,G2). For two graphs with no maximum common subgraph, the distance will become \MCS(Gi,G2)\ = \GX\ + \G2\. MMCS has also been shown to be a metric, but it does not produce values normalized to the interval [0,1], unlike the previously described distance measures. 7 Note that if it holds that \MCS{Gi,G2)\ = |Gi| + \G2\ - \mcs{Gl,G2)\ VGi,G 2 , we can compute distMMCs(G\,G2) as \G\\ + \G2\ — 2 • \mcs(Gi,G2)\. This is much less computationally intensive than computing the minimum common supergraph.
2.4. Median
of a set of graphs
Given one of the graph-theoretic distance measures from Sec. 2.3 we can define the median graph,10 which is the graph which has the minimum average distance to all graphs in some set of graphs: G = argmin(-y]dist(s,GO).
(7)
Here 5 is a set of n graphs (S = {G\,G2,..., G n } and thus | 5 | = n) for which we wish to compute the median and G is the median graph. The median is defined to be a graph in set S. The distance dist(- • •) is computed using one of Eqs. (4-6) above. There is also the concept of the generalized median and weighted mean, 10 where we don't require that G be a member of S, but we will not consider them here because they are quite expensive to compute. 3. fc-Nearest Neighbors with Graphs In this section we describe the fc-nearest neighbors (fc-NN) classification algorithm and how we can easily extend it to work with the graph-based representations of data. The basic fc-NN algorithm is given as follows.11 First, we have a set of training examples; in the traditional fc-NN approach these are numerical feature vectors.
293
Inputs: the set of n data items and a parameter, k, defining the number of clusters to create Outputs: the centroids of the clusters and for each data item the cluster (an integer in [l,t]) it belongs to Step 1. Step 2. Step 3. Step 4.
Assign each data item randomly to a cluster (from 1 to k). Using the initial assignment, determine the centroids of each cluster. Given the new centroids, assign each data item to be in the cluster of its closest centroid. Re-compute the centroids as in Step 2. Repeat Steps 3 and 4 until the centroids do not change. Fig. 1.
The fc-means clustering algorithm.
Each of these training instances is associated with a label which indicates to what class the instance belongs. Given a new, previously unseen instance, called an input instance, we attempt to estimate which class it belongs to. Under the fc-NN method this is accomplished by looking at the fc training instances closest (i.e. with least distance) to the input instance. Here fc is a user provided parameter. Once we have found the k nearest training instances using some vector distance measure, such as one of those defined above in Eqs. (1-3), we estimate the class by the majority among the k instances. This class is then assigned as the predicted class for the input instance. If there are ties due to more than one class having equal numbers of representatives amongst the nearest neighbors we can either choose one class randomly or we can break the tie with some other method, such as selecting the tied class which has the minimum distance neighbor. Notice that if methods of computing distance between graphs are available, it is possible to directly extend the fc-NN algorithm to work with data represented by graphs simply by: 1. representing the objects to be classified as graphs and 2. using a graph distance measure, such as those described in Sec. 2.3, instead of the typical vector distance measure. 4. fc-means Clustering with Graphs The fc-means clustering algorithm is a simple and straightforward method for clustering data. 1 1 The basic algorithm is given in Fig. 1. Like the fc-NN algorithm, we can easily extend fc-means to work with graph representations of data. As before, we substitute a graph-theoretic distance measure (Sec. 2.3) for a traditional vectorbased distance measure (Sec. 2.2). However, with the fc-means algorithm there is a further complication: we also need a graph-theoretic substitute for the centroid (which acts as a cluster representative). For this, we will use the median graph (Sec. 2.4); instead of calculating a centroid, we will calculate the median graph of the set of graphs in the cluster. This will be used as the cluster representative, and since the median itself is a graph, distance can easily be computed between medians and any data graphs using a graph distance measure. At this point, we wish to address a point that may cause some confusion. Clustering with graphs is well established in the literature. However, with those methods
294
the entire clustering problem is treated as a graph, where nodes represent the items to be clustered and weights on edges connecting nodes indicate the distance between the objects the nodes represent. The goal is to partition this graph, breaking it up into several connected components which represent clusters. The usual procedure is to create a minimal spanning tree of the graph and then remove the remaining edges with the largest weight until the number of desired clusters is achieved.12 This is very different than the technique we described in this section, since it is the data itself which are represented by graphs, not the overall clustering problem. 5. Graph Representations of Web Documents In the previous two sections, we showed how theoretically the fc-means clustering algorithm and the fc-NN classification algorithm can be extended to work with graphs instead of vectors. What we have not yet provided is a method by which the original data will be represented by graphs. In this section we describe methods for representing web documents using graphs instead of the traditional vector representations. All representations presented here are based on the adjacency of terms in a web document. These representations are named: standard, simple, n-distance, n-simple distance, raw frequency and normalized frequency. Note that the graphbased extensions of Sees. 3 and 4 are applicable to any domain where the data is representable by graphs, not just web documents. Under the standard method each unique term (word) appearing in the web document, except for stop words such as "the", "of", and "and" which convey little information, becomes a node in the graph representing that web document. Each node is labeled with the term it represents. Note that we create only a single node for each word even if a word appears more than once in the text. Second, if word a immediately precedes word b somewhere in a "section" s of the document, then there is a directed edge from the node corresponding to term a to the node corresponding to term b with an edge label s. We take into account certain punctuation (such as periods) and do not create an edge when these are present between two words. Sections we have defined for web documents are: title, which contains the text related to the document's title and any provided keywords (meta-data); link, which is text that appears in hyper-links on the document; and text, which comprises any of the readable text in the document (this includes link text but not title and keyword text). Next we remove the most infrequently occurring words on each document, leaving at most m nodes per graph (m being a user provided parameter). This is similar to the dimensionality reduction process for vector representations. 4 Finally we perform a simple stemming method and conflate terms to the most frequently occurring form by re-labeling nodes and updating edges as needed. An example of this type of graph representation is given in Fig. 2. The ovals indicate nodes and their corresponding term labels. The edges are labeled according to title (TI), link (L), or text (TX). The document represented by the example has the title "YAHOO NEWS", a link whose text reads "MORE NEWS", and text containing "REUTERS
295
NEWS SERVICE REPORTS". Note there is no restriction on the form of the graph and that cycles are allowed. While this approach to document representation appears superficially similar to the bigram, trigram, or N-gram methods, those are statistically-oriented approaches based on word occurrence probability models. 13 The methods presented here, with the exception of the frequency representations described below, do not require or use the computation of term probability relationships.
Fig. 2.
Example of a standard graph representation of a web document.
The second type of graph representation we will look at is what we call the simple representation. It is basically the same as the standard representation, except that we look at only the visible text on the page (no title or meta-data information is examined) and we do not label the edges between nodes. Thus we ignore the information about the "section" where the two respective words appear together. An example of this type of representation is given in Fig. 3.
Fig. 3.
Example of a simple graph representation of a web document.
The third type of representation is called the n-distance representation. Under this model, there is a user-provided parameter, n. Instead of considering only terms immediately following a given term in a web document, we look up to n terms ahead and connect the succeeding terms with an edge that is labeled with the distance between them (unless the words are separated by certain punctuation marks). For example, if we had the following text on a web page, "AAA BBB CCC DDD", then we would have an edge from term AAA to term BBB labeled with a 1, an edge from term AAA to term CCC labeled 2, and so on. The complete graph for this example is shown in Fig. 4. Similar to n-distance, we also have the fourth graph representation, n-simple distance. This is identical to n-distance, but the edges are not labeled, which means
296
we only know that the distance between two connected terms is not more than n.
Fig. 4.
Example of an n-distance
graph representation.
The fifth graph representation is what we call the raw frequency representation. This is similar to the simple representation (adjacent words, no section-related information) but each node and edge is labeled with an additional frequency measure. For nodes this indicates how many times the associated term appeared in the web document; for edges, this indicates the number of times the two connected terms appeared adjacent to each other in the specified order. The raw frequency representation uses the total number of term occurrences (on the nodes) and co-occurrences (edges). A problem with this representation is that large differences in document size could lead to skewed comparisons, similar to the problem encountered when using Euclidean distance with vector representations of documents. Under the normalized frequency representation, instead of associating with each node the total number of times the corresponding term appears in the document, a normalized value in [0,1] is assigned by dividing each node frequency value by the maximum node frequency value that occurs in the graph; a similar procedure is performed for the edges. Thus each node and edge has a value in [0,1] associated with it, which indicates the normalized frequency of the term (for nodes) or co-occurrence of terms (for edges). For the two frequency representations we modify the definition of graph size (Definition 3). Instead the graph size for these representations is defined as the total of the node frequencies added to the total of the edge frequencies. Further, when we compute the maximum common subgraph we take the minimum frequency element (either node or edge) as the value for the maximum common subgraph. For example, if we had two graphs that each had a node A, one with frequency of 10 and the other with frequency of 20, then node A in the maximum common subgraph would have frequency of 10. We wish to mention here an interesting feature the graph representations described here have on the time complexity of determining the distance using Eqs. (46). For general graphs the computation of the maximum common subgraph is NPcomplete. However, for the graph representations of web documents presented in this section, the computation of the maximum common subgraph is 0(n2), with n being the number of nodes, due to the restriction of using unique node labels in the graph representations [i.e. we need only examine the intersection of the nodes,
297
since each node has a unique label). 14 6. Samples of Experimental Results To test the performance of our graph-based methods as compared to the traditional vector-based approaches, we performed experiments with three web document collections, called the F-series, the J-series and the K-series.3, These data sets were selected because of two major reasons. First, all of the original HTML documents are available, which is necessary if we are to represent the documents as graphs; many other document collections only provide a pre-processed vector representation, which is unsuitable for use with our method. Second, ground truth assignments are provided for each data set, and there are several classes representing easily understandable groupings that relate to the content of the documents. Some web document collections are not labeled or are presented with some other task in mind than content-related clustering {e.g. building a predictive model based on user preferences). Pre-created term-document matrices are also available for these data sets, and we use them in our vector-based experiments. In this section, we give a sample of the experimental results we have obtained. Detailed and more systematic results are reported in Refs. 15, 16, 17, and 18. In Table 1 we show the accuracy of our graph-based fc-NN approach as compared to the traditional vector-based approach for the K-series data set. Accuracy is measured by the leave-one-out approach, with higher accuracy being desirable. The first column is the number of nearest neighbors (k). The next four columns are for our graphbased approach, with the associated number indicating the maximum number of nodes used per graph (parameter m, see Sec. 5). The final two columns are the traditional vector approach with distance based on cosine similarity (Eq. 2) or Jaccard similarity (Eq. 3). Further results comparing the performance of fc-NN for all three web document data sets can be found in Ref. 16. Table 1. Classification accuracy for the K-series data set using fc-NN for different distance measures when using the standard representation. k
G (40)
G (70)
G (100)
G (150)
Cos
Jac
1 3 5 10
71.32% 73.85% 77.35% 78.21%
73.50% 77.69% 80.56% 81.97%
76.15% 80.34% 83.08% 84.74%
76.28% 80.90% 84.27% 85.56%
74.10% 76.62% 77.74% 76.03%
77.31% 79.40% 81.24% 79.83%
In Table 2 we show the clustering performance of the fc-means algorithm for both vector and graph methods, using mutual information 19 to measure how close the created clusters match ground truth (higher mutual information is better). Due a
T h e data sets are available under these names at: ftp://ftp.cs.umn.edu/dept/users/boley/PDDPdata/
298
to the random initialization of the A;-means algorithm, we repeated each experiment ten times and report the average in the table. The columns of the table are, from left to right: graph size (number of nodes per graph), standard, simple, 5-distance (i.e. n-distance with n = 5), 5-simple distance, raw frequency, normalized frequency, cosine, and Euclidean (the last two being vector-based results). More experimental results which compare the performance of the different graph representations when clustering with fc-means are given in Ref. 18. Table 2. Mutual information measures for the F-series data set after fc-means clustering for different representations when using MCS distance. GS
Std
Sim
5-D
5-SD
RF
NF
Cos
Euc
10 20 30 40 50
0.1613 0.1365 0.1604 0.1408 0.2138
0.1110 0.1431 0.1323 0.1312 0.1314
0.1273 0.1361 0.1371 0.1014 0.1044
0.1137 0.1140 0.1457 0.1221 0.1615
0.1327 0.1454 0.1404 0.1137 0.1374
0.1263 0.1702 0.1319 0.1728 0.1525
0.1101 0.1101 0.1101 0.1101 0.1101
0.0467 0.0467 0.0467 0.0467 0.0467
In Figs. 5 and 6, we show examples of performing multidimensional scaling 20 with the data set of Table 2 when using a graph-based representation of the data. Multidimensional scaling is a technique for transforming a set of data into a lower dimensional space; usually this is either done to reduce time requirements (by reducing dimensionality) or to visualize the set of data, by projecting it into two dimensions. The latter is the motivation in this CELS6, £LS it allows us to visualize a set of data that is otherwise difficult to represent graphically. To apply multidimensional scaling to graphs, we need only implement a graph distance measure (Sec. 2.3) and measure the distance between each pair of graphs. For the figures we used the MCS distance measure and the standard representation. The symbols indicate to which cluster each document was assigned by the clustering algorithm; the ellipses indicate the general shape and size of clusters. Figure 5 shows a clustering produced when using 10 nodes per graph maximum; the mutual information measured for this clustering is 0.1242. Similarly in Fig. 6 we show the clustering created when using 50 nodes per graph maximum; the mutual information of the clustering depicted in this figure is 0.1981. Note that the increase in mutual information, which indicates a better match with the ground truth clustering, is reflected in the plot as better organized, more compact, clusters. Similar to Table 2, in Table 3 we give the results of fc-means clustering when using different graph-theoretic distance measures (Sec. 2.3). We see that the graphbased methods that use normalized distance measures (MCS and WGU) generally performed as well or better than vector-based methods. Graph distance measures that were not normalized to the interval [0,1] (namely, MMCS) performed poorly. To see why this occurs, we have provided the following example. Let |Gi| = 10, \G2\ = 10, \mcs(GuG2)\ = 0, \MCS(GUG2)\ = 20, \G3\ = 20, |G 4 | - 20,
299
0.4
F-series Data Set (10 nodes/graph, MCS distance)
X
0.2
/ /
p .
*sOo
/
^
A
L ^
f
A / &yM J Ji& X*^«* + ^r
^^^
I /
&
——«^-^i__EI!3v \J A +^+
/
U*
/
XK A -0.2
/ / /
/
/
/
x* /
/
S**x "^"^ L
1 X
/ /
-0.4
X
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Component 1
Fig. 5. Multidimensional scaling performed on the F-series d a t a set with 10 nodes per graph maximum. The mutual information for this clustering was 0.1242.
F-series Data Set (50 nodes/graph, MCS distance)
s
x
1
x
/
f
0 o o
4
+
o
oa o
'A
y
) j\
+
V 1 ^ *i 4
\A -1 -0.2
r -0.1
0.0
0.1
" 1 0.2
Fig. 6. Multidimensional scaling performed on the F-series d a t a set with 50 nodes per graph maximum. The mutual information for this clustering was 0.1981.
\mcs{G3,Gi)\ — 5, and \MCS(G-\,,G2)\ = 35. Clearly graphs G 3 and G 4 are more similar to each other than graphs G\ and G2 since G\ and Gi have no common
300
subgraph whereas G3 and G4 do. However, the distances computed for these graphs are distMcs(Gi,G2) = 1.0, distMcs{G3,G4) = 0.75, distMMCs{Gi,G2) = 20 and distMMCs{Gz,G4) = 30. So we have the case that the un-normalized distance is actually greater for the pair of graphs that are more similar. This is both counterintuitive and the opposite of what happens in the cases of the normalized distance measures. Thus this phenomenon leads to the poor performance; similarly poor performance also occurs for fc-NN classification when using MMCS. 15 Additional comparisons of the performance of graph-theoretic distance measures when used with fc-means are reported in Ref. 17. Table 3. Mutual information measures for the J-series data set after fc-means clustering for different distance measures when using the standard representation. GS
MCS
WGU
MMCS
Cos
Euc
10 20 30 40 50
0.2384 0.2636 0.2487 0.2498 0.2240
0.2444 0.2210 0.2563 0.2669 0.2598
0.1757 0.1824 0.1553 0.0388 0.0273
0.2205 0.2205 0.2205 0.2205 0.2205
0.0674 0.0674 0.0674 0.0674 0.0674
7. Conclusions In this chapter we described methods of easily extending machine learning algorithms, specifically the fc-nearest neighbors classification and A;-means clustering algorithms, to work with documents that are represented by graphs instead of vectors. The benefit of this approach is that it allows us to retain additional structural information which is usually discarded when using a simpler vector representation. Experimental results show an improvement in both clustering and classification performance when using graph-based representations over the case of vector-based representations. Acknowledgments This work was supported in part by the National Institute for Systems Test and Productivity at the University of South Florida under the U.S. Space and Naval Warfare Systems Command Contract No. N00039-02-C-3244 and by the Fulbright Foundation that has granted Professor Kandel the Fulbright Research Award at Tel-Aviv University, College of Engineering during the academic year 2003-2004. References 1. O. Zamir and O. Etzioni. Web document clustering: A feasibility demonstration. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 46-54, 1998.
301 2. S. Dumais and H. Chen. Hierarchical classification of web content. In Proceedings of SIGIR-00, 23rd ACM International Conference on Research and Development in Information Retrieval, pages 256-263, 2000. 3. C. Apte, F. Damerau, and S. M. Weiss. Automated learning of decision rules for text categorization. ACM Transactions on Information Systems, 12:233-251, 1994. 4. G. Salton. Automatic Text Processing: the Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley, Reading, 1989. 5. D. Lopresti and G. Wilfong. Applications of graph probing to web document analysis. In Proceedings of the 1st International Workshop on Web Document Analysis, pages 51-54, 2001. 6. J. Liang and D. Doermann. Logical labeling of document images using layout graph matching with adaptive learning. In D. Lopresti, J. Hu, and R. Kashi, editors, Document Analysis Systems V, volume 2423 of Lecture Notes in Computer Science, pages 224-235. SpringerVerlag, 2002. 7. M.-L. Fernandez and G. Valiente. A graph distance metric combining maximum common subgraph and minimum common supergraph. Pattern Recognition Letters, 22:753-758, 2001. 8. H. Bunke and K. Shearer. A graph distance metric based on the maximal common subgraph. Pattern Recognition Letters, 19:225-259, 1998. 9. W. D. Wallis, P. Shoubridge, M. Kraetz, and D. Ray. Graph distances using graph union. Pattern Recognition Letters, 22:701-704, 2001. 10. X. Jiang, A. Muenger, and H. Bunke. On median graphs: properties, algorithms, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10):1144-1151, 2001. 11. T. M. Mitchell. Machine Learning. McGraw-Hill, Boston, 1997. 12. C. T. Zahn. Graph-theoretical methods for detecting and describing gestalt structures. IEEE Transactions on Computers, C-20:68-86, 1971. 13. C.-M. Tan, Y.-F. Wang, and C.-D. Lee. The use of bigrams to enhance text categorization. Information Processing and Management, 38:529-546, 2002. 14. P. Dickinson, H. Bunke, A. Dadej, and M. Kretzl. On graphs with unique node labels. In Proceedings of the 4th International Workshop on Graph Based Representations in Pattern Recognition, volume 2726 of Lecture Notes in Computer Science, pages 13-23. Springer, 2003. 15. A. Schenker. Graph-Theoretic Techniques for Web Content Mining. PhD thesis, University of South Florida, 2003. 16. A. Schenker, M. Last, H. Bunke, and A. Kandel. Classification of web documents using a graph model. In Proceedings of the 7th International Conference on Document Analysis and Recognition, pages 240-244, 2003. 17. A. Schenker, M. Last, H. Bunke, and A. Kandel. Comparison of distance measures for graph-based clustering of documents. In Proceedings of the J^th IAPR-TC15 International Workshop on Graph-Based Representations in Pattern Recognition, volume 2726 of Lecture Notes in Computer Science, pages 202-213. Springer-Verlag, 2003. 18. A. Schenker, M. Last, H. Bunke, and A. Kandel. Graph representations for web document clustering. In Proceedings of the 1st Iberian Conference on Pattern Recognition and Image Analysis, volume 2652 of Lecture Notes in Computer Science, pages 935942. Springer-Verlag, 2003. 19. T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, 1991. 20. T. F. Cox and M. A. A. Cox. Multidimensional Scaling. Chapman and Hall, 1994.
CHAPTER 3.6 AUTOMATED DETECTION OF MASSES IN MAMMOGRAMS
H. D. Cheng, X. J. Shi, R. Min, X. P. Cai and H. N. Du Department of Computer Science Utah State University Logan, UT 84322-4205 E-mail: [email protected]
Breast cancer continues to be a significant public health problem in the world. Early detection is the key for improving breast cancer prognosis. Mammography has been one of the most reliable methods for early detection of breast carcinomas. However, the estimated sensitivity of radiologists in breast cancer screening is only about 75%, and the performance would be improved if they were prompted with the possible locations of abnormalities. Breast cancer Computer Aid Diagnosis (CAD) systems can provide such help and they are important and necessary for breast cancer control. Microcalcifications and masses are the two most important indicators of malignancy, and their automated detection is very valuable for early breast cancer diagnosis. Automated detection and classification of masses is even more challenging than of microcalcification. This chapter discusses the methods for mass detection and classification, and microcalcification detection and compares their advantages and drawbacks.
1. Introduction Breast cancer is the leading cause of death of women in US [1]. Currently the most effective method for early detection and screening of breast cancers is mammography [2]. Microcalcifications and masses are two important early signs of the diseases [3]. It is more difficult to detect masses than microcalcifications because their features can be obscured by or similar to normal breast parenchyma. Masses are quite subtle, and often occurred in the dense areas of the breast tissue, have smoother boundaries than microcalcifications, and have many shapes such as circumscribed, speculated (or stellate), lobulated or ill-defined [4]. 303
304
Masses must be classified as benign and malignant in order to improve the biopsy yield ratio. Generally speaking, masses with radiopaque and more irregular shapes are usually malignant, and those combined with radiolucent are benign [5]. A mammogram is basically distinct with four levels of the intensities: background, fat tissue, breast parenchyma and calcifications with increasing intensity. Several studies have revealed a positive association of tissue type with breast cancer risks [6]. Women who have breast cancers can easily get contralateral cancers in the other side breast [7]. Asymmetry of breast parenchyma between the two sides has been one of the most useful signs for detecting primary breast cancer [8]. Reading mammograms is a demanding job for radiologists. Their judgments depend on training, experience, and subjective criteria. Even well-trained experts may have an inter-observation variation rate of 65-75% [9]. Computer aided diagnosis (CAD) systems may help radiologists in interpreting mammograms for mass detection and classification. Since 65-90% of the biopsy of suspected cancers turned out to be benign, it is very important to develop CADs that can distinguish benign and malignant lesions for the diagnosis. The combination of the CAD scheme and experts' knowledge would greatly improve the detection accuracy of the abnormality detection [10]. Most of the mass detection CAD scheme involves the phases described in Fig. 1. Although the CAD schemes were independently developed using different data sets of limited size, most of the schemes yielded similar performance. In this chapter, the methods of five major research areas: preprocessing, image segmentation, feature extraction and selection, mass detection/classification, and performance evaluation will be studied. Digitizing Mammogram Image Preprocessing Image Segmentation Feature Analysis and Extraction Classification
t
Evaluation Fig. 1. A CAD mass classification system
305
2. Preprocessing of Mammograms Preprocessing of mammograms is to produce reliable representations of breast tissue structures. The effective method for mammogram enhancement must aim to enhance texture and features of masses. The reasons are: (1) low-contrast of mammographic images; (2) hard to read masses in mammogram; (3) generally, variation of the intensities of the masses such that radiopaque mass with highdensity and radiolucent mass with low-density in comparison with the background. The enhancement methods are grouped as: (1) global histogram modification approach; (2) local-processing approach; and (3) multi-scale processing approach. The ideal contrast enhancement approach should have no over-enhancement and under-enhancement. 2.1 Global Histogram Modification Approach A commonly used global histogram modification approach is histogram equalization (HE) [11]. The HE technique is simple and effective in enhancing the entire image with low contrast, only if a) it contains single object or b) there is no apparent contrast change between object and background. To improve HE method, Multi-peak HE method [12] has been developed. Another global histogram modification is histogram stretching [11]. It uses a linear transfer function to stretch intensities. The global histogram modification has done almost nothing for texture enhancement since it cannot change the order of gray levels of the original image, and it is not suitable for enhancing mammograms. 2.2 Local Processing Approach Local-processing approaches are also studied for image contrast enhancement. There are many methods to implement contrast enhancement by changing pixel intensities. One way is based on nonlinear mapping methods (bi-linear, sigmoid, non-continuous, etc.) [11, 13]. The implementation can be feature-based, and the local features may be gained by edge detection, or by using local statistic information such as local mean, standard deviation, etc. [13-15, 17-19]. Another feature-based method is to define the contrast ratio first, then to enhance image contrast by increasing the contrast ratio [15, 20]. [21] proposed a multistage treestructured filter to enhance the digital mammogram. Each stage is based on central weighted median filter. The local-processing methods are quite effective in local texture enhancement. However, most of local methods have little contribution to enhancing the contrast among the objects. The compositive technique combining the above methods is proposed in [22]. 2.3 Multiscale Processing Some methods for feature enhancement are based on wavelet transformation [2328]. The general stages can be described as: (1) the digitized mammogram is transformed using wavelets. (2) The coefficients are modified to enhance the
306
mass features. (3) The enhanced mammogram is obtained using inverse wavelet transformation. The differences of implementing this kind of methods are in the basis functions (mother wavelets) and the ways to modify coefficients [23-27]. It can detect the directional features and remove unwanted perturbations. More orientational information is obtained using hexagonal sampling than rectangular sampling. However, it is difficult to decide what mother wavelet for transformation and what method for modification to enhance all kinds of masses. 3. Image Segmentation The second stage of mass detection CAD schemes is to separate the suspicious regions that may contain masses from the background parenchyma, i.e., to partition the mammogram into several non-intersecting regions, and then extract regions of interests (ROIs) and suspicious mass candidates from ROIs. The suspicious area is one that is brighter than its surroundings, has almost same density, has a regular shape with varying size, and has fuzzy boundaries [16]. This is a very essential and important step that determines the sensitivity of the entire system. Segmentation methods do not need to be excruciating in finding mass locations but the result for segmentation is supposed to include regions containing all masses even with some FPs. FPs will be removed at later stage. However, the result of segmentation depends on the suitable algorithm for specific features, and if the algorithm is fixed, the result can be improved by enhancement techniques [14]. According to their natures, there are four kinds of segmentation techniques: classical techniques, fuzzy techniques, bilateral image subtraction and multiscale technique. 3.1 Classical Approaches 3.1.1 Global thresholding Global thresholding has been widely used for segmentation [29, 30]. It is based on global information, such as histogram. Since masses are brighter than the surrounding tissues, it makes thresholding a useful method for segmentation. Methods depending only on global thresholding are not good to identify ROIs since mammograms are 2-dimension projections of 3-dimension breasts, the regions of overlapping tissues including three kinds of tissues: a fat region, a fatty and glandular region and a dense region, may be brighter than masses. 3.1.2 Local thresholding Local thresholding (LT) can refine the results of global thresholding or identify suspicious. LT is better for mass detection than global thresholding. Two variables of the local thresholding should be considered, the window size and thresholding value [33]. LT is a pixel-based operation and cannot accurately separate pixels into suitable sets, therefore, an adaptive clustering process is used
307
to refine the result from localized adaptive thresholding [31]. LT is also used as pre-processing for other algorithms, such as Markov Random Field [32]. 3.1.3 Iterative pixel classification (1) MRF/GRF (Markov Random Field /Gibbs Random Field) The MRF/GRF algorithms for segmentation have been studied [32, 34, 35]. They are considered as powerful modeling tools [32, 34]. A common criterion for MRF is to estimate a function of maximum a posteriori (MAP). There are three well-known algorithms to estimate MAP function, i.e., Simulated Annealing (SA), Maximiser of Posterior Marginals (MPM), and Iterated Conditional Modes (ICM) [32]. A modified Markov Random Field (MRF) model-based method was employed for segmenting images [32, 35]. The algorithm uses the statistical properties of the pixel and its neighbors. The probability mass function px(x) is defined as [35] /?.v(*) = - e x p ( - X fi t(x„x,)Z
|r..v!tl'
X
Yx)
Met'
where the texture class label X is an MRF with a four nearest neighborhood system, z is a normalized constant, C is the collection of cliques, p and yx are parameters of MRF model, and t{xr,xs) = •
1
xr = xs
0
otherwise
A method based on the discrete wavelet transform (DWT) and multiresolution Markov random field (MMRF) to segment the suspicious regions is studied [3638]. They employed different wavelet mother functions. The expectationmaximization (EM) algorithm was used to evaluate the segmentation result [38, 39]. A comparison of DWT with MRF methods was given [40] by using the free-response receiver operating characteristic curves. It claimed that better results were obtained using DWT method than using MRF method. (2) Region Growing Region growing is one of the popular techniques for segmenting masses. The basic idea of the algorithm is to find a set of seed pixels, and then to grow iteratively and aggregate with pixels that have similar properties. If the region is not growing any more, two regions, grown region and surrounding region are obtained. The advantages and disadvantages of various methods based on region growing technique are summarized in Table 1. (3) Region Clustering Region clustering and region growing are very similar. As described in Table 2, region clustering searches the region directly without any prior information. The K-means algorithm is a well-known clustering procedure. An adaptive clustering algorithm for segmentation was introduced [49] to overcome two
308
problems of the common K-means algorithm: the lack of spatial constraints and the assumption of constant intensity in each cluster. The performance is better than K-means algorithm with/without spatial constraints, and better than region growing techniques. The adaptive clustering to refine the segmentation was also applied in [31]. A clustering algorithm was used for fully automated segmentation [50]. Similar to region growing technique, [51] used a pixel-bypixel K-means clustering methods [52-54] for initial mass segmentation. The clustering process separates one or more disjoint objects within the ROI, which were filled, grown in a local neighborhood, and eroded and dilated by morphological operators. Table 1. Comparison of region growing techniques Preprocessing 1. Contrast Stretching Simple 2. Histogram Equalization graphical 3. Fixed-Neighborhood Statistical seed-fill 4. Convolution Mask Enhancement ANCE 1. Region Partition Adaptive Thresholding 2. Automatic Seed Selection A Gaussian function h(x, y) = f(x, y)N(x, y; jlx,Hy, o] )*
Probabilistic method
to do lesion segmentation Radial gradient index (RGI)based method Adaptive region growing *
h(x, y)
A Gaussian function Kx, y) = fix, y)N(x,
y\nx,iiy,o2c)'
Advantage Simple and easy to implement
Disadvantage Does not work well if no obvious peaks
Able to remove speckle noise Work well for masses fitting Gaussian distribution
Very time- [43] consuming Discrete contour model is better than this method Sensitive to noise
Ref. [18, 41, 42]
[4445]
[45, 46]
to do lesion segmentation
time- [45, Reduce some Very 1. Select an initial seed point 47, kinds of consuming 2. Define cutoff factor, mean and standard 48] noise deviation 3. Define local window size is the multiplication of the original ROI with the constraint function, 2
N(x,y;/1
,fi ,ol)
is a circular normal distribution centered at (^ , p ) with a variance Gc .
Table 2. Comparison of region growing and region clustering Need prior Need a Is an iterative information Algorithms Start point process Region Growing Yes Yes Yes Region Clustering No No Yes
Require update/ Stop function Yes Yes
3.1.4 Edge detection Edge detection is a traditional method for image segmentation. There are a lot of operators: Roberts gradient, Sobel gradient [55], Prewitt gradient, Laplacian operator, etc. The combined edge detection method was developed in [56]. DWCE: Density-Weighted Contrast Enhancement (DWCE) filter in conjunction with Laplacian-Gaussian (LoG) filter for segmenting suspicious mass regions was studied [40]. DWCE was employed in two stages, one uses it globally to
309
enhance the contrast and uses LoG to isolate objects, and the other applies it locally to each of the segmented objects to refine the segmentation. [57] evaluated a modified DWCE in combination with a texture classification scheme. Logic Filter: A nonlinear filter, logic filter, was introduced [58, 59]. The logic operators of AND, OR and XOR can be used. Label the window structure as below, , a b c [D] Here a, b, c, and D are four pixels and [ ] is the center of the structure. The logic filter is defined as: D = (D XOR b) OR (D XOR c) A modified logic filter [60] detects the edges in all possible directions: D = (D XOR a) OR (D XOR b) OR (D XOR c) It successfully detected a test set of 25 sample mammograms [60]. However, the quality of the mammograms directly affects the results. Iris Filter: An adaptive filter, iris filter, to extract ROIs in digital mammograms was studied [61]. After using the iris filter, the area of a tumor candidate is estimated by a simple thresholding. Then the iris filter and a SNAKES algorithm are employed locally to find the approximate boundary of the tumor candidate. Gaussian Filter: A model-based vision (MBV) algorithm was used to obtain ROIs and classify the masses [62]. The targets of the algorithm are to reduce the rate of FPs, to extract features from ROIs, and to match these features with those truth models. ROIs were highlighted by a difference of Gaussian (DoG) filter. DoG comes from Lapiacian of the Gaussian (LoG), and because of computational reasons, LoG is implemented as DoG [63]. The DoG mask is as the following, h(x,y) = -^e-ix2+>1),2ai lK(Tx
1
eH.r^V2a}
17T<J2
Applying Fourier transform to the above formula, A good edge detector should satisfy the following conditions [64]: (1) low error probability of marking non-edge pixels and losing edge pixels; (2) edge pixels should be as near as possible to the real edge; (3) the boundary width should be one pixel. Contour model: The active contour models, known as "snakes", are good ways to find the contours of interested regions. The algorithm to improve the detection quality of closed edges was employed [65]. A fast algorithm for finding active contours was developed [66] that improved the active contours algorithm and saved the computational time. The discrete contour algorithm was studies, which
310
is fast and robust to detect the boundaries [67]. A survey on deformable models in medical image analysis was published [68]. One of the deformable models, a discrete contour model, was discussed [46, 67], and it was implemented to segment mammograms [44]. The discrete dynamic contour model to discriminate malignant masses was discussed [69]. 3.1.5 Template matching Template matching method uses the prior information of mammograms, segment possible masses from the background using prototypes. The prototypes of possible masses are created based on the characteristics or physical features of the targeted masses [16], or based on the two-dimensional search function [70]. When the a priori information about the size of the masses on a mammogram is not available, a range of sizes for the templates is used [1]. The matching criterion is measured by the least square technique [70] or by a cross correlation coefficient of the templates [16, 52]. Incorporating additional pre-processing methods can make the process more efficient [53]. Template matching results in a large subset of possible masses, a majority of which are false positives. The adaptive thresholding technique may fail to find suspicious masses with a partial loss, while [71] proposed a template matching algorithm that can solve the problem by using the similarity. 3.1.6 Stochastic relaxation An unsupervised segmentation method with an evidential constrained optimization method was reported [72]. It aimed to detect all different lesions. The method performs unsupervised partitioning to segment the image into homogenous regions by using 'generic labels'. It uses a constrained stochastic relaxation algorithm for building an optimal label map to separate tissue and suspicious areas. An evidential disparity function estimates the feature similarity of two blocks of pixels and realizes the partitioning. A modified contextual Bayesian relaxation labeling (CBRL) algorithm was developed to segment possible masses [73]. The method creates a finite generalized Gaussian mixture (FGGM) probability density function using statistical modeling, and FGGM is a good model when image features are unknown [14]. The Expectation Maximization (EM) algorithm is employed to estimate the FGGM model parameters. It performed pixel labeling using CBRL algorithm. A statistical model based on a finite generalized Gaussian mixture was used to localize the possible mass areas using stochastic relaxation labeling scheme. Abnormalities in the mammograms may be considered as disturbance or noise to the background and their probability can be estimated.
311
3.2 Fuzzy Technique Fuzzy logic has been introduced for segmenting suspicious masses [74]. It has been proved that fuzzy set theory coupled with texture-based algorithms was very useful for the classification of masses [75]. 3.2.1 Fuzzy thresholding Classical (global/local) thresholding techniques try to segment ROIs, but the techniques are only effective for the objects with well-defined boundaries. A few methods with fuzzy thresholding are proposed for segmenting objects with illdefined boundaries. A membership value is assigned for each pixel in an image by a fuzzy membership function, and then an iterative process is operated, an error value is calculated and the fuzzy membership values are updated by an update rule [74]. 3.2.2 Fuzzy region growing Fuzzy approach is studied for handling the malignant tumors with fuzzy boundaries extend from a dense core region to the surrounding tissues. It proposed a new concept of acceptance with restriction, and the algorithm is more stable than the classic region growing. Control parameters Aumax, ACVmax and P were em ployed [76]. Here coefficient of variation (CoV) is defined as the standard deviation divided by the mean, i.e., V = a/\i. The experiments show that fuzzy region growing can attain very good results. 3.3 Bilateral Image Subtraction The bilateral image subtraction technique, also called asymmetry approach, is used to determine suspicious regions [77, 78]. It is based on the normal symmetry between left and right breasts. The algorithm consists of two steps: 1) Alignment of the left and right breast images. 2) Asymmetry detection. Many suspicious regions can be identified using bilateral image subtraction technique. Some of them may not be true masses. To reduce the number of FPs, the featureanalysis techniques are needed. [77] illustrates the results using bilateral image subtraction technique. However, the source of the database was not given. This technique can be used in an automated mass detection, and can reduce the suspicious regions while true positive regions are surely detected. But, there are two main disadvantages. First, the left and right breast mammograms are not always symmetric because of different image acquisition, orientation, and compression. Second, asymmetry method cannot remove the FPs and classify the true positive regions into benign and malignant masses. It only provides clues to extract features for further processing. 3.4 Multiscale Technique Multiscale techniques were applied to segment suspicious areas, and they can improve the detection rate. Tumors with radii between 2 to 30 mm can be detected in different scales [63, 77]. Discrete wavelet transform (DWT) is a
312
powerful mathematical tool for image analysis, and DWT is one of multiscale techniques. Adaptive and multi-scale processing for improving segmentation was studied [31]. It used DWT to decompose features and multi-scale representation to process mammograms, and then segment ROIs with an adaptive method. [79] introduced a method based on wavelets for extracting suspicious areas according to their shapes. [80] considered the shapes of suspicious areas were not enough for classification, and proposed a DWT method to analyze the contributing factors of scale in discriminating area shapes after these areas are extracted. 4. Feature Analysis and Extraction The features can be calculated from the characteristics of suspicious area such as size, shape, density, and smoothness of borders, etc. [81]. The feature space is very large and complex due to the wide diversity of normal tissues and variety of abnormalities. Only some of them are significant. Some redundant features should be removed to improve the performance of the classifier. [82] made an investigation of feature-analysis techniques for extracting features from mammograms. [5] demonstrated that the performance of ANN (Artificial Neural Network) and BBN (Bayesian Belief Network) can come to the same level in detecting masses with the same features from the same mammographic database. The feature analysis and extraction is a key step in mass detection since the performance of CAD may depend more on the optimization of feature selection than the classification method. Feature selection is the process of selecting an optimum subset of features from the enormous set of potentially available features in a given problem domain after the image segmentation [83]. The feature space can be divided into three sub-spaces: intensity features, geometric features, and texture features. The typical features sorted by sub-spaces are summarized in Table 3. The general guidelines to select significant features mainly include four considerations [84]: Discrimination, Reliability, Independence, and Optimality. The feature extraction and selection processes for mass detection can base on principle component analysis [2], linear discriminate analysis [51], and GA algorithm [85]. [3] proposed recursive functions to calculate the features and significantly reduced the complexity of the feature extraction. In addition, some features that are not listed in the Table 3 were discussed in [86]. 4.1 Intensity Features This kind of features is the simplest among three sub-spaces, and most of them are the simple statistics. The feature FI1 is the contrast measure of suspicious region. Generally, it is the difference between the average gray level of the ROI
313 Table 3. The features Intensity Features FI1: contrast measure of ROIs [48. 84]; FI2: Average grey level of ROIs (Mean) [4. 99]: FI3: standard derivation inside ROIs or variance [4. 84]: FI4: skewness of ROIs [4]: FI5:kurtosisofROIs[4]: FI6: A set of features composed of third-order normalized Zernike moments [87]: Shape Features FG1: Margin spiculation [88. 99]: FG2: Margin sharpness [99]: FG3: Area measure [48. 84. 91]: FG4: Circularity measure [48. 84. 91]: FG5: Convexity [48. 91]: FG6: Rectangularity [48. 91]: FG7: perimeter [48. 91]: FG8: Perimeter-to-area ratio [48. 90. 91]: FG9: Acutance measure [133. 141]: FG10: A shape factor MF,.3 [90]: FG11: A set of seven low-order, central invariant moments [90]: FG12: Fourier descriptor [90. 91]; FG13: NRL boundary roughness [84]: FG14: NRL mean [48. 84." 91]; FG15: NRL entropy [48. 91]: FG16: NRL area ratio [48. 91]: FG17: NRL standard deviation [48. 84. 91]: FG18: NRL zero crossing count [48. 91]: FG19: NCL mean [90]: " FG20: NCL variance [90]:
FG21: NCL skewness [90]: FG22: NCL kurtosis [90]: Texture Features FT1: energy measure (OR angular second moment) [48. 84. 85]; FT2: correlation of co-occurrence matrix [48. 84. 85]: FT3: inertia of co-occurrence matrix [48. 84. 85. 91]; FT4: entropy of co-occurrence matrix [48, 85]; FT5: difference moment [84. 85]: FT6: inverse difference moment [48. 84. 85. 91]: FT7: sum average [48. 84. 85. 91]: FT8: sum entropy [48. 84. 85. 91]: FT9: difference entropy [48. 84. 85. 91]: FT 10: sum variance [48, 91]; FT11: difference variance [48. 91]: FT12: difference average [48. 91]; FT13: information measure of correlation 1 [48. 91]: FT14: information measure of correlation 2 [48. 91]; FT15: contrast [92]: FT16: Angular second moment [92]; FT 17: entropy [92]; FT 18: mean [92]: FT19: short runs emphasis [85. 91]; FT20: long runs emphasis [85. 91]; FT21: grey-level non-uniformity [85. 91]; FT22: run length non-uniformity [85. 91]; FT23: run percentage [85. 91]
and the average gray level of the surrounding region. The features FI2 - FI5 are the statistics pertinent to the moments. FI6 is a set of features consisted of the third-order normalized Zernike moments [87]. 4.2 Shape Features The shape features are also called morphological or geometric features. Almost twenty significant shape features are extracted in a variety of classifiers. The first eleven features are directly calculated from the boundaries and areas of ROIs. Six of them are the statistics of normalized radial length (NRL). The last four features are statistics based on the distribution of normalized chord length (NCL). FG1 is obtained by applying the radial edge- gradient analysis technique within various neighborhoods about the grown regions to quantify the margin speculation of a mass [88]. FG2 is the magnitude of the average gradient along the margin of the mass [89]. It can be used to evaluate the degree of spiculation of mass. The method to calculate the features FG3-FG8 is described [48]. FG9
314
has the ability to measure density variations across the boundaries of ROIs and can help to decide whether the tumor is benign or malignant [90]. FG10 is a shape factor independent of pixel intensity that is described in [90]. FG11 is a set of seven features that are pertinent to the second-order and third-order central invariant moments [90]. FG12 is based on the Fourier transform of object boundary sequence [91]. The features FG13-FG18 are the statistics based on the normalized radial length (NRL) distribution. The radial length of a point on the tumor boundary is the Euclidean distance from this point to the mass centroid, whose co-ordinates are the average of the co-ordinates of all the points on the mass boundary. The NRL distribution is a set of data, each of which is normalized by dividing the maximum radial length. The mathematical definitions of these six features can be found in [48]. The features FG19-FG22 are the statistics based on the normalized chord length (NCL) distribution [90]. The definition of the NCL is similar to the NRL. The difference between them is the definition of the length. The chord length is defined as the Euclidean distance of a pair of points on the tumor boundary. 4.3 Textural Features The texture features can be grouped into three classes based on what they are derived from: SGLD-based features, GLDS-based features, and RLS-based features. FT1-FT14 are the SGLD-based features. They are based on the spatial gray level dependence (SGLD) matrices, or gray-level co-occurrence (GCM) matrices. SGLD matrices are used to measure the texture-context information. It is a 2-D histogram. An element of the SGLD matrix P(i,j,d,0) is defined as the joint probability that the gray levels /' and j occur, when the two pixels are separated by distance d and along direction 0 of the image. In order to simplify the computational complexity of the algorithm, the 0 is often given as 0°, 45°, 90°, and 135°, and the distance d is often defined as the Manhattan or city block distance. The element P(i,j,d,6) of the SGLD matrix can be expressed: P(/,y,£/,0°) = ||{((^,^ l ),(x : ,^)),|x 2 -x 1 | = £ /,y 2 -y l =0}|j f((.x . y , ) . ( x , . y , ) ) , ( . T , -x s = d,y, - y = -d) P(i J. d. 45°) • [
or (.*, - xt = -d. y, - y i = d)
A/,7W,90 o ) = |{((x 1 ,y 1 ),(x 2 ,>; 2 )),x : -x 1 =0,|>' 2 -j l | = C f}| P(i,j,d,U50)-
[((*• '•>".)' (x2' ^ )M*2-*> =d^y2-yl =d) or (x2 - x, = -d, y2-yt=-d)
where l(x, y) is the intensity value of the pixel at the position (x, y), l(xl,yl)
= i,l(x2,y2)
= j , and | | s | | the number of the elements in the set S.
Features FT 1-FT 14 can be extracted from SGLD matrices with different distance
315 d and direction 0. The image on which the SGLD matrices are calculated can be ROIs, or the rubber-band straightening transform (RBST) image in which the ROIs are transformed by the RBST [85]. Based on RBST image, two kinds of texture features were obtained [85, 91, 92]. The GLDS-based features FT15FT18 [92] are extracted from the gray level difference statistics (GLDS) vector of an image [93]. The GLDS vector is the histogram of the absolute difference of pixel pairs separated by a given displacement vector S=(Ax, Ay), where Is(x, y) = \I(x+Ax)-I(y+ Ay) |, and I(x,y) is the intensity value of the pixel at the position (x, y), Ax and Ay are integers. An element of GLDS vector ps(i) can be computed by counting the number of times each value of Is(x, y) occurs. In practice, the displacement vector S=(Ax, Ay) is usually selected to have a phase of value as 0°, 45°, 90°, or 135° to obtain the oriented texture features. The run length statistics (RLS) features FT19-FT22 of the RBST image are the inputs to the classifiers in [85, 91]. For a given image, it can compute a RLS matrix R0 in a given direction 6 [93]. A gray level run is a set of consecutive, collinear pixels with the same gray level value. The run length is the number of pixels in a given direction. The matrix element RrfiJ) represents the frequencies of the run length 7 of gray level i in the direction 6. 4.4 Feature selection Features extracted from gray level characteristics, shape, and texture of the lesion and the surrounding tissue can usually be expressed as mathematical description, and are helpful for a classifier to distinguish masses as malignant or benign. But, it is very difficult to predict which feature or feature combinations will achieve better classification rate. Generally, different feature combinations will result in different classification performance. In addition, experience shows that the relatively few features used in a classifier can keep the classification performance robust [94], Therefore, one often faces with the task of selecting an optimized subset of features from a large number of available features. Two major methods to optimize the feature selection have been employed for CAD in mammography. 4.4.1 Stepwise feature selection The common method to reduce the number of the features and obtain the best feature set is known as feature selection with stepwise linear discriminant analysis, or stepwise feature selection [53, 54, 85, 95, 96]. Stepwise feature selection is a heuristic procedure using statistical techniques based on Fisher's linear discriminant. At the beginning, the selected feature pool is empty. At each step followed, one available feature is input into or removed from the selected feature pool by analyzing its effect on a selection criterion [95, 96]. Most studies in mass detection [54, 85, 96] employed the minimization of Wilks' lambda as the selection criterion, which is defined as the ratio of within-group sum of
316
squares to the total sum of squares [97]. [95] test all available selection criteria. A set of 340 features is reduced to 41 features with stepwise feature selection [85]. 4.4.2 Genetic algorithm (GA) Another commonly method to select an optimized subset of features uses Genetic Algorithms (GAs) that are adaptive heuristic search algorithms based on the principles of Darwinian evolution. In particular, GAs work very well on mixed combinatorial problems, however, they might be computationally expensive. The application of GA-based feature selection to mass detection has been studied [54, 85, 96, 98]. The GA method with different fitness functions can reduce a set of 340 features to 39-62 features [85]. 5. Classification and Evaluation of CAD Results Automatic classifiers will classify the detected suspicious areas into normal tissues, benign masses, or malignant masses. Classifiers such as linear
discriminants (LDA) and artificial neural network (ANN) have performed well in mass classification [5, 53, 54, 57, 84, 92]. Table 4 shows the major classifiers for mass classification. It is important to notice that an objective comparison of the performance of different methods in mammography is very difficult and even impossible due to the use of different databases. In Table 5 we list several available mammogram databases. Next, we will summarize various performance indices for the evaluation of CAD systems. Table 4. Major Classifier Linear Discriminant Analysis (LDA) Artificial Neural Networks (ANNs) Bayesian Network
Binary Decision Tree
classifiers for mass classification. Description Construct linear decision boundaries by optimizing certain error criterion to classify cases into one of several mutually exclusive classes Construct non-linear mapping function as a decision boundary. Two ANNs were used: the three-layer backpropagation neural network and the Radial Basis Function (RBF) network A probabilistic approach to estimate the class conditional probability density functions of the two features for background and tumor A binary decision tree recursively subdivides regions of feature space into two subspaces by using a threshold to separate input mammogram data into two classes each time.
Feature Used Texture features, shape features, morphological, and spiculation features
Refs. [51.57. 91. 96]
Texture features, shape features, wavelet-based features, peak-related and contour-related features
[2. 4. 50. 69. 80. 99]
50 local and four global features
[5]
Intensity features, shape features, texture features
[29.31. 32. 36. 37]
317 A receiver operating characteristic (ROC) curve is a plot of true positive as a function of false positive. A higher ROC, approaching the perfection at the upper left hand corner, would indicate greater discrimination capacity. The CLABROC program tests the statistical significance of the difference between two ROC curves [101]. For evaluating true-positive detection, sometimes it is required not only the existence but also the localization of the tumor. A better method for this purpose is free-response receiver operating characteristic (FROC) analysis which is a plot of operating points showing the tradeoff between the TP rate versus the average number of false positives per image [102]. The difference between FROC and ROC methods can be notified by the abscissa of the FROC plot that begins at zero and no upper limit, while the abscissa of the ROC plot is from zero to one. Both FROC and ROC analyses suffer from their limitations. For instance, they do not address the complexity of images and are difficult to transform the subjective measurements (radiologist's observations) to the objective FROC/ROC curves. Table 5. several available mammogram databases [100] Description Database Name The Digital Database for Screening Mammography (DDSM) was created by DDSM Massachusetts General Hospital, the University of South Florida, and Sandia National Laboratories. Source: http://marathon.csee.usf.edu/Ivlammoeraphv/Database.htnil This database was created by Lawrence Livermore National Laboratories LLNL/UCSF (LLNL) and the Radiology Department at the University of California at San Francisco (UCSF) Source: [email protected] This database was created by the Mammographic Image Analysis Society MIAS (MIAS). United Kingdom. Source: http://www.wiau.tnan.ac.uk/services/MIAS/MIASweb.html [email protected] The Nijmegen Digital Mammogram database was created by the National Nijmegen Expert and Training Centre for Breast Cancer Screening and the Department of Radiology at the University of Nijmegen. the Netherlands. Source:nico@mbfys. kun.nl
The area under the ROC curve or the FROC curve is an important criterion for evaluating diagnostic performance [102]. Usually, it is referred as the Az index. The value of Az is 1.0 when the diagnostic detection has perfect performance which means that TP rate is 100% and FP rate is 0%. Fig. 2 shows ROC curves of an experienced mammographer, average performance of five radiologists, and the computerized scheme with ANN alone, and computerized the hybrid system [89]. Azs were also calculated to evaluate the ability of different classifiers. The performance of hybrid system with Az =0.94 is better than all the others'. The experienced mammographer had &nAz of 0.91, whereas the average of the five
318
radiologists yielded an Az of 0.81. The four-input ANN yielded an Az value of 0.90. An example of the use of FROC curves for mass detection on two different datasets is given in Fig. 3 [69]. To measure the performance of the final result, an index Af similar to index Az was defined as the area under the logarithmically plotted FROC curves between 0.05 and 4 FPs per image. i.o,
Hybrid ayrtm • ,-0 •»«> E>p*n mammograplwr
E
u.
. ANN (4-2-1K*,«0.80)
> 0.5
I
rfomntftiicv of > (A.-0.80)
0.5 False-Positive Fraction
l.o
Fig.2. ROC curves illustrate the performances of different systems to differentiate between benign and malignant masses [89]
Find oaves slal* M l N J W J M I
Fhal ewes data i * ! DOSM
1
o.s 0-8 0 u
0?
m u
/
.'V
,r'
J
I: I:
1 0.6 •1 ft.5
y FislsSnj*
*
J'
j? 0.3
•y
#.2
s
/
y/
•
S*:trri Mage. o s * iw W
s
.j
first sfctga SMXKHJstage • Second stage, case tasted
y
0-1
0,1
L_ 0,01
0
0.01
0,1
i
Fsli*pos*fMSperima§i
1*
0.1
1
False pwlww per mage
Fig. 3. FROC curves show the test results for the datasets from Nijmegen (left figure) and from the DDSM (right figure) [69].
10
319 References 1.
A. S. Constantinidis. M. C Fairhurst. F. Deravi. M. Hanson. C. P. Wells. C. Chapman-Jones. Image Processing and its Applications, 435( 1999).
2.
K. Bovis, S. Singh. J. Fieldsend. C. Pinder. IJCNN 2000: Proceedings of the IEEE-INNSENNS International Joint Conference on Neural Networks, 342(2000).
3.
H. D. Cheng. X. Cai. X. Chen. L. Hu. and X. Lou. Pattern Recognition, vol 36. 2967(2003).
4.
I. Christoyianni, E. Dermatas. and G. Kokkinakis. IEEE Signal Processing Magazine. 54(2000).
5.
B. Zheng. Y. H. Chang. Xiao-Hui Wang. Walter F. Good. IEEE Iternational Conference on Neural Networks. 4181(1999).
6.
R. L. Egan and R. C. Mosteller. Cancer 40. 2087( 1977).
7.
H. H. Storm and O. M. Jensen. Br J Cancer 54. 483( 1986).
8.
T. J. Rissanen. H. P. Makarainen. M. A. Apaja-Sarkkinen and E. L. Lindholm. Acta Radiologica 36. 358(1995).
9.
P. Skaane. K. Engedal. and A. Skjennald. Acta Radiologica 38. 497( 1997).
10. K. Doi. Proceedings of National Forum '95, 9(1996). 11. S. H. Nam. and J. Y Choi. Proceedings of The 20th Annual International Conference of The IEEE Engineering in Medicine and Biology Society 20. v 20. n 2. 1009(1998). 12. H. D. Cheng and X. J. Shi. Digital Signal Processing, vol 14 no 2. 158(2004). 13. J. K. Kim. J. M. Park. K. S. Song, and H. W. Park. IEEE Transactions on Medical Imaging, v 16. n 5. 495(1997). 14. H. Li. K. Liu. Y. Wang, and S. C. Lo. The IEEE 18th Annual International Conference on Bridging Disciplines for BioMedicine. v 3. 1045(1997). 15. D. Braccialarghw. G. H. Kaufmann. Optical Engineering, v 35. n 1. 76(1996). 16. S.-M. Lai. X. Li. W.F. Biscof. IEEE Transactions on Medical Imaging, v 18. Issue: 4. 377(1989). 17. P. G. Tahoces. J. Correa. M. Souto. C. Gonzalez. L. Gonez. and J. J. Vidal. IEEE Transactions on Medical Imaging. MI-10. 330(1991). 18. S. Singh and R. Al-Mansoori. Journal of Intelligent Systems, v 10. N 2. 183(2000). 19. E. D. Pisano. S. Zong. B. M. Hemminger. M. DeLuca. R. E. Johnston. K. Muller. M. P. Braeuning. and S. M. Pizer. Journal of Digital Imaging, v 11. n 4. 193(1998). 20. J. F. Sosa Urias. Appl. Opt., v 30. 4598(1991). 21. W. qian. L. P. Clarke. M. Kallergi. and R.A. Clark. IEEE Transactions on Medical Imaging, v 13. n 1.25(1994). 22. M. Wilson. R. Hargrave. S. Mitra. Y. Y. Shieh. and G. H. Roberson. Proceedings of The IEEE Symposium on Computer-Based Medical Systems, 270(1998). 23. S. Mallat and S. Zhong. IEEE Transactions on Pattern Analysis and Machine Intelligence, v 14. n 7. 710(1992). 24. J. Lu. D. M. Healy, and J. B. Weaver. Optical Engineering, v 33. n 7. 2151 (1994).
320 25. C. Chang and A. Laine. IEEE Transactions on Information Technology in BioMedicine, v 3 Issue: 1.32(1999). 26. R. Pfisterer. and F. Aghdasi. Optical Engineering, v 40. n 2. 209(2001). 27. R. Pfisterer and F. Aghdasi. 1999 Africon, ISBN: 0-7803-5546-6. 383(1999). 28. L. Li, W. Qian, and L. P. Clarke. Acad. Radiol. Vol. 4 No. 11.724(1997) 29. D. Brzakovic. X. M. Luo. and P. Brzakovic. IEEE Transactions on Medical Imaging. Vol. 9. No. 3. 233(1990). 30. T. Matsubara. H. Fujita. S. Kasai. M. Goto. Y. Tani. T. Hara. and T. Endo. Intelligent Information Systems. 1997. IIS '97. 63(1997). 31. L. H. Li. W. Qian. L. P. Clarke. R. A. Clark, and J. Thomas. Proceedings of SPIE - The International Society for Optical Engineering, v 3661. n I. 490(1999). 32. H. D. Li. M. Kallergi. L. P. Clarke. V. K. Jain, and R.A. Clark. IEEE Transactions on Medical Imaging. Vol. 14 Issue 3. 565(1995). 33. R. C. Gonzalez and P. Wintz. Digital Image Processing. New York: Addison-Wesley Publishing Company. (1992). 34. H. Derin and H. Elliott. IEEE Trans. Pattern Anal. Machine lntell. v PAMI-9. n 1. 39(1987). 35. M. L. Comer. S. Liu. and E.J. Delp. Digital Mammography, K. Doi. MIL. Giger. R.M. Nishikavva. and R. A. Schmidt. Eds.. Elsevier. Amsterdam. 475(1996). 36. L. Zhen and A. K. Chan. IEEE Transactions on Medical Imaging, v 20. n 7. 559(2001). 37. L. Zheng. A. K. Chan. G. McCord. S. Wu. and J. S. Liu. Journal of Digital Imaging, v 12. n 2 SUPPL. 1. 18(1999). 38. C. H. Chen and G. G. Lee. Int. Journal of Imaging Systems and Technology, v 8. 491(1997). 39. C. H. Chen and G. G. Lee. Graphical Models and Image Processing, v 59. n 5. 349(1997). 40. L. H. Li. W. Qian. and L. P. Clarke. Acad Radiol, v 4. 724(1997). 41. W. M. Morrow and R. B. Paranjape. IEEE Transactions on Medical Imaging, v 11. n 3. 392(1992). 42. R. M. Rangayyan. L. Shen. Y. Shen. J. E. L. Desautels. H. Bryant. T. J. Terry. N. Horeczko. and M. S. Rose. IEEE Tran. on Information Technology in BioMedicine. v 1. n 3. 161(1997). 43. S. A. Hojjatoleslami and J. Kittler. IEEE Transaction on Image Processing, v 7. 1079(1998). 44. G..M. te Brake. M. J. Stoutjesdijk. and N. Karssemeijer. Proceedings of SPIE Conference on image processing, San Diego, CA, v 3661, n II, 911(1999). 45. M. A. Kupinski and M. L. Giger. IEEE Tran. on Med. Imag.. v 17. Issue 4. pp. 510(1998). 46. G. M. te Brake and N. Karssemeijer. Medical Physics, v 28. n 2. 258(2001). 47. S. Pohlman. K.A. Powell. N.A. Obuchowski. W.A. Chilcote. and S. Grundfest-Broniatowski, Med. Phys. 23. 1337(1996). 48. N. Petrick. H. P. Chan. B. Sahiner. and M. A. Helvie. Med. Phys. 26. 1642(1999). 49. T. N. Pappas. IEEE Transaction on Signal Processing, v 40. n 2. 901(1992). 50. B. Sahiner. H. P. Chan. N. Petrick. M. A. Helvie. and M. M Goodsitt. Acad. Radiol., v 7. 1077(2000).
321 51. B. Sahiner. N. Petrick. and H. P. Chan. IEEE Transactions on Medical Imaging, v 20. 1275(2001). 52. S. L. Ng. and W. F. Bischof. Computers and Biomedical Research 25. 218(1992). 53. F. N. Che. M. C. Fairhurst. C. P. Wells, and M. Hanson. Proceedings of the 1996 IEE Colloquium on Digital Mammography, Mar 27 1996, London, UK. IEE Colloquium (Digest). n072. 4p. (1996) 54. B. Sahiner. H. P. Chan. N. Petrick. D. Wei. M. A. Helvie. D. D. Adler. and M. M. Goodsitt. MedPhys\23.
1671(1996).
55. J. L. Viton. M. Rasigni. G. Rasigni. and A. Liebaria. Society Instrumentation,
of
Photo-Optical
v. 35. n. 12. 3453(1996).
56. M. Abdel-Mottaleb. C. S. Carman. C. R. Hill. S. Vafai. Digital Mammography, K. Doi. M. L. Giger. R. M. Nishikawa and R. A. Schmidt, ed.. 467(1996). 57. N. Petrick. H.P. Chan. D. Wei. B. Sahiner. MA. Helvie. and D.D. Adler. Med. Phys.. v 23. n 10. 1685(1996). 58. K. Tsirikolias. B. G. Mertzios. In Proceedings of The International Conference on Digital Signal Processing, Florence, Italy. Sept. 4-6. 285(1991). 59. K. Tsirikolias and B. G. Mertzios. Image Processing: Theory and Applications. 251(1993). 60. A. Rocha. F. Tong. and Z. Z. Yan. Journal of Computer Science and Technology, vI5, N6, Nov. 2000, Allerton Press Inc. New York, NY, USA. ISSN: 1000-9000. 629(2000). 61. H. Kobatake. M. Murakami. H. Takeo. and S. Nawano. IEEE Transactions on Medical Imaging, v 18. n 5. 369(1999). 62. W. E. Polakowski. D. A. Cournoyer. S. K. Rogers. M. P. DeSimio. D. W. Ruck. J. W. Hoffmeister. and R. A. Raines. IEEE Tran. on Med. Imag.. v 16. Issue: 6. 811(1997). 63. G.M. te Brake and N. Karssemeijer. IEEE Tran. on Med. Imag.. v 18. n 7. 628(1999). 64. J. F. Canny. IEEE Transactions on Pattern Analysis March Intell. 8. 679( 1986). 65. L. D. Cohen. CVG1P: Image Understanding, v 53. n 2. 211(1991). 66. D. J. Williams and M. Shah. Cl'GIP: Image Understanding, v 55. n 1. 14(1992). 67.
S. Lobregt and M. A. Viergever. IEEE Transactions on Medical Imaging 14. 12(1995).
68. T. Mclnerney and D. Terzopoulos. Medical Image Analysis 1(2). 91(1986). 69. G. M. Brake. N. Karssemeijer and J. H. C. L. Hendriks. Physics in Medicine and Biology, v 45. n 10.2843(2000). 70. S. Morrison and L.M. Linnett. 1999 International
Conference on Image
Processing:
Icip99c, v2. 182(1999) 71. Y. Hatanaka. T. Hara. H. Fujita. S. Kasai. T. Endo. and T. Iwase. IEEE Transactions on Medical Imaging, v 20. n 12. 1209(2001). 72. G. L. Rogova. C. Ke. R. Acharya. and P. Stomper. SPIE Conference on Image Processing, vol.3661, 1440(1999). 73. H. Li. Y. Wang. K. J. R. Liu. S. B. Lo. and M. T. Freedman. IEEE Transactions on Medical Images, v 20. n 4. 289(2001).
322 74. M Sameti, R. K. Ward. Digital Mammography, K. Doi. M. L. Giger. R. M. Nishikawa and R. A. Schmidt. Ed.. Elsevier Science B. V.. 471(1996). 75. M. Sameti. R. K. Ward. B. Palcic. and J. Morgan-Parkes. 1997 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing. 831(1997). 76. D. Guliato. R. M. Rangayyan. W. A. Carnielli. J. A. Zuffo. and J. E. L. Desautels. Proc. 4th Int. Workshop Digital Mammography, The Netherlands. 253(1998). 77. B. Zheng. Y. H. Chang, and D. Gur. Association of University Radiologists, v 2. n 12. 1056(1995). 78. T. K. Lau and W. F. Bischof. Computers and Biomedical Research, v 24. 273 (1991). 79. L. M. Bruce and R. R. Adhami. SPIE's Int. Symp. Medl Imag, v 3304, (1997). 80. L. M. Bruce and N. Shanmugam. Proceedings of The 22nd Annual EMBS International Conference. V 2. pp. 985(2000). 81. P. Undrill. R. Gupta. S. Henry and M Downing. Proceedings Colloquium on Digital Mammography,
of the 1996
IEE
5/1(1996).
82. F. F. Yin. M. L. Giger. K. Doi. C. J. Vyborny. and R. A. Schmidt. Journal of Digital Imaging. v7. 18(1994) 83. M. Sameti. R. K. Ward. J. Morgan-Parkes, and B. Palcic. Proceedings of 19th Internation Conference IEEE, EMBS Oct. 30-Nov.2, 1997.513(2000). 84. H. Li. Y. Wang. K. J. R. Liu. S. B. Lo. and M. T. Freedman. IEEE Tran. on Medl lmag.. v 20. n 4. 302(2001). 85. B. Sahiner. H. P. Chan. N. Petrick. M. A. Helvie and M. M Goodsitt. Phys. Med. Biol, v 43. 2853(1998). 86. D. J. Marchette. R. A. Lorey and C. E. Priebe. Pattern Recognition, v 30. n 9. 1547(1997). 87. S. O. Belkasim. M. Shridhar. and M. Ahmadi. Pattern recognition, v 24. n 12. 1117(1991) 88. Z. Huo. M. L. Giger. C. J. Vyborny. U. Bick. P. Lu. D. E. Wolverton. and R. A. Schmidt. Med. Phys.. v 22. 1569(1995) 89. Z. M. Huo. M. L. Giger. C. J. Vyborny. D. E. Wolverton. R. A. Schmidt, and K. Doi. Acad. Radio., v 5. 155(1998). 90. R. M. Rangayyan. N. M. El-Faramawy. J. E. Desautels and O. A. Alim. IEEE Transactions on Medical Imaging, v 16. n 6. (1997). 91. B. Sahiner. H. P. Chan. N. Petrick. M. A. Helvie. and L. M. Hadjiiski. Med. Phys. v 28. n 7. 1455(2001). 92. B. Sahiner. H. P. Chan. N. Petrick. D. Wei. M. A. Helvie. D. D. Adler. and M. M. Goodsitt. IEEE Transactions on Medical Imaging, v 15. n 5. Oct. (1996). 93. M. M. Galloway. Computer graphics and image processing. Vol. 4. 172. (1975) 94. M. L. Giger. Z. Huo. M. A. Kupinski. and C. J. Vyborny. Handbook of Medical Imaging, Vol. 2. Milan Sonka and J. M. Fitzpatrick. Ed.. 915(2000) 95. Datong Wei. H. P. Chan. M. A. Helvie. B. Sahiner. N. Petrick. D. D. Adler. and M. M. Goodsitt. Phys. Med. v 22. n 9. 1501(1995)
323 96. H. P. Chan. D. Wei. M. A. Helvie. B. Sahiner. D. D. Adler. M. M. Goodsitt. and N. Petrick. Phys. Med. Biol, v 40. 857(1995) 97. M. M. Tatsuoka. Multivariate Analysis. Techniques for Educational and Psychological Research Macmillan, New York. (1988). 98. B. Zheng. Y. H. Chang. X. H. Wang. W. F. Good. D. Gur. Acad. Radiol., v 6. 327(1999) 99. Z. Huo. M. L. Giger. and C. J. Vyborny. IEEE Tran. on Med. Imag.. v 20. n 12. (2001). 100. R. M. Mishikara. "Mapmmographic databases." Breast Disease, v 10. 137(1998). 101. C. E. Metz. P. L. Wang, and H. B. Kronman. Proc. Info. Proc. Med. Imaging 8th conf. Boston. MA. 432(1984). 102. K. Woods. M. Y. Sallam and K. W. Bowver. In N.Karssemeijer, M.Thijssen, J.Hendriks, L van Erning, ed. Digital Mammography Nijmegen 98. Amsterdam: Kluwer Academic Publisher. 19(1998).
C H A P T E R 3.7 WAVELET-BASED K A L M A N FILTERING I N SCALE SPACE FOR IMAGE FUSION
Hsi-Chin Hsin Department of Electrical Engineering, Chung-Hwa University 30 Tung Shiang, Hsin Chu 30067, Taiwan, ROC E-mail: [email protected] Ching-Chung Li Department of Electrical and Computer Engineering, University of Pittsburgh Pittsburgh, PA 15261, USA E-mail: [email protected] In this chapter, a 2-dimensional (2D) dynamic system of images is formulated in the scale space based on the 2D wavelet transform. The original image is assigned to the highest index (the finest scale) and the decomposed approximations at successively coarser scales (lower indices) are interpreted as the state variables at the corresponding scales. The state transition takes place from a coarse scale to a fine scale. Through the use of the Kalman filtering, the optimal estimation of the finest scale original image from a set of multiscale noisy measurements can be obtained in a scale recursive form. This can be applied to multiresolution image fusion. An efficient filtering algorithm has been developed by using the orthogonal wavelet packet transform that greatly reduces its computational complexity. This methodology is illustrated by experiments including one performed on the fusion of a Landsat TM band 1 image and a SPOT image. 1. Introduction Image fusion is to merge complimentary information from multiple images of a scene into a single image for the purpose of enhancing computer vision or human visual perception. It is widely used in diverse fields, from remote sensing 31,34 ' 35 , intelligent robots 29 , to medical imaging 33 . The demand of its capability and performance has induced its rapid development in recent years. In a suite of multi-sensor images, each gives an observation with particular emphasis on certain characteristics in the scene at either the same or different spatial resolutions; together they provide more detail and accurate information as a whole. Image fusion may then be considered as an estimation of the source image based upon a set of multi-sensor measurements 14 . In the statistical estimation problem, the Kalman filtering has shown its great success in various applications 1 except the issue of excessive computation required 325
326
in the case of filtering two-dimensional images 2 . Wavelet theory 3 ' 4,5 ' 25 ' 26 ' 27 provides the multiresolution representation of images that leads to a natural filtering direction in the scale space. In terms of stochastic signals, a strong decorrelation property of the wavelet transform has been noted. Flandrin analyzed fractional Brownian motions (fBms), which are nonstationary processes with stationary increments, through the use of continuous orthonormal wavelet transform and showed that the spectra of fBms are inversely proportional to | / | 7 which implies that fBms can be used to model 1 / / fractal processes 6 ' 7 . Wornell decomposed the nearly 1 / / fractal processes into a set of uncorrelated white sequences using the orthonormal wavelet transform 8 9 ' , which suggests that the wavelet transform acts as a whitening filter for such nonstationary processes. Dijkerman studied the orthonormal wavelet transform of stationary processes and fBms and found that the correlation of wavelet coefficients decays exponentially fast across scales 10 ' 11 . Zhang and Walter used a set of biorthogonal wavelets to decorrelate the wavelet coefficients of stationary processes and of nonstationary processes with stationary increments 12 leading to a Karhunen-LoeVelike expansion. Due to the strong decorrelation property of wavelet transforms of stationary processes and of some nonstationary processes, it is much easier to analyze and process the wavelet coefficients instead of the random signal itself. This may offer substantial computational advantages in dealing with images. The wavelet synthesis equation provides a way to model random processes at different scales leading to a scale dynamics structure in which the detail (wavelet) coefficients may be interpreted as the driving signal, scaling coefficients as state variables and state transition takes place on the scale index. With an orthonormal wavelet, Chou and Willsky formulated a scale recursive Kalman filtering for signals and images 13 ' 14 , and developed an estimation and data fusion algorithm under appropriate conditions. Miller and Willsky developed a multiscale statistical inversion algorithm for sensor fusion15. Hsin and Li developed a wavelet-packet based 2D Kalman filtering algorithm for image fusion 16 ' 17 ' 18 . Ni, Ho and Tse developed their multirate Kalman filtering for image reconstruction from noisy subband systems 19 . The work on scale recursive data fusion was also reported by Wen and Zhou 20 . Different methods of wavelet-based image fusion have been developed, to mention but a few, by Li, Manjunath and Mitra 21 , Zhang and Blum 22 , Ranchin et al. 23 , Garguet et al 2 4 , and Liu et al 2 8 . The development in satellite image fusion using wavelet analysis is particularly active 35 . This chapter will present the wavelet-packet based Kalman filtering for multiresolution image fusion. 2. B a c k g r o u n d Wavelet transform provides a multiresolution analysis that gives an efficient way to represent signals at multiple scales and to extract the difference in information between two successive scales. Let L2{R) be a set of finite energy ID signals, / p
L2(R) = f {x(t) : I / x(t)x(t)dt)
\ 1/2
< oo}
327
where x(t) denotes the complex conjugate of x(t). Mathematically, L2(R) is a Hilbert space with the inner product; < x(t),y(t) >— J x(t)y(t)dt, where x(t) and y(t) are in L2(R). There exists a function ip(t) G L2(R), called a mother wavelet, which can be used to generate a basis for L2(R) by simply dilating and translating this single function ip{t). For the discrete wavelet transform (DWT), the dilation factor is set to be a power of 2. The function expansion of x(t) in L2(R) with respect to a wavelet basis is then given by
.(t)^^W(m;n)CW TO
(1)
n
where w{m; n) denotes the wavelet coefficient of x(t) at the scale level m and time n (n G Z, Z is the set of all integers) and il>£(t) = 2m/2V»(2mi - n) is a scaled and translated wavelet. In this definition, a large scale index m means a fine resolution. Let us consider an orthonormal wavelet ip(t), < ip™(t),ifrj.it) >= SmjSnk, where 5mj is the Kronecker delta. The wavelet coefficients w(m;n)'s of a given signal x(t) are determined by the inner product of x(t) and tp™(t), w(m;n)=<x(t),ip™(t)>
(2)
Let us define a subset Wj, called the wavelet space of L2(R) at the scale level j , Wj d= Wn(t)
;neZ},
jeZ
(3)
we then have an orthonormal basis of Wj with L2(R) = ejL-aoWi
(4)
where © denotes the direct sum and ®'£L_0OWj is dense in L2(R). This means that an orthonormal basis of L2(R) can be constructed by dilating and translating the mother wavelet ip(t). Daubechies 3 has constructed the famous compactly supported orthonormal wavelets. A subset Vj;j e Z, called the scaling space of L2(R) at the scale level j , is
defined by Vj = ©kLooWm
(5)
One may find an orthonormal basis, {<j>>n{t);n e Z}, of Vj by dilating and translating a function <j>(t) e L2(R), called the scaling function, corresponding to a given orthonormal mother wavelet i()(t), and <j>%(t) = 2 m / 2 0(2 m i - n) is a scaled and translated scaling function. The sequence of subspaces {Vj} then forms a multiresolution analysis (MRA), which has the following properties: •••CK.iCV&cVi---,
Vj+i = Vj + Wj,
and
®%-ooVj = L2(R),
x(t) £ Vj
implies
n ^ =
x{2t) G Vj+i,
0
'
j e Z
328
The scaling function <j>(t) 6 Vb C V\, where the set {y/2(j>(2t -n);n€ of Vi, one has the following two-scale dilation equation <j>(t) = ] T V^2 h(k) <j>(2t - k),
^2h(k)
k
= V2.
Z} is a basis
(6)
k
Since the wavelet ip(t) G Wo C Vi, we have
m = Y,^s(k)(2t-k).
(7)
For orthogonal wavelet and orthogonal scaling function, the relationship between the coefficients h(k)'s and
>
(8)
The orthogonal wavelet decomposition and reconstruction are given by the following equations: wavelet analysis equations (wavelet transform) x(m;n) = /J/i(fc — 2n)x(m + l;k) = (Hm
x(m-(-l))(n)
(9)
w(m;n) — S2g(k — 2n)x(m + l;k) = (Gm x(m-|-l))(n)
(10)
k
yfc
and wavelet synthesis equation (inverse wavelet transform) x{m + 1; n) = Y J h{n — 2k)x(m; k) + 2_. g(n ~ 2k)w(m; k) k
k
= {H*m x(m))(n) + (G*m w(m))(n)
(11)
where Hm, —
.. A(0) h{\) h{2) h(3) .. • • h(0) h(l)
;h(n)=<4>(t),4,1n(t)>
(12)
;g(n)=(t),n(t)>
(13)
...g(0)g(l)g(2)g(3) Gm —
•••
•
• 9(0) 9(1)
x(m) is the scaling coefficient vector which approximates the signal x(t) at the scale level m, x(m;n) is the nth element of x(m), w(m) is the wavelet coefficient vector which represents the detail information of the signal x(t) between the scale level m and the finer scale level m + 1, Hm and Gm are the coarsening operator
329
and differencing operator at the scale level m respectively, and the superscript * denotes an adjoint operator, " .
.
.'
MO) • • Mi) • • > h(2) hQO) • h(S) h(l) •
5(0) • • 5(1) " • G*m — 5(2) 0(0) • 5(3) 5 (1) •
Due to the orthogonality of the wavelets ip™(t), we have the following identities (from Eqs. (9), (10) and (11)): HmHm
=
GmG*m = J, HmG*m = GmH^ = 0, H^Hm
2.1. Scale-Recursive
Kalman
Filtering
for ID
+ G*mGm = I.
(14)
Signals
In the wavelet synthesis equation x(m + 1) = Hm x(m) + G*m w(m), the first term H^ x{m) is the prediction of the signal x(m + 1) at the scale level m + 1 based on x(m) at the coarser scale level m and the second term G*m w(m) is the added detail information. Suppose that a signal with zero mean is measured at several scales (m = L, L + 1 , . . . , M) as given by y(m) = C(m) x(m) + v(m)
(15)
where y(m), C(m) and v(m) denote the measurement of signal x(m), the measurement matrix and the additive noise at the scale level m, respectively; L is the coarsest measurement scale level and M is the finest measurement scale level. Based on these measurements, i/(m)'s, (m = L,..., M), one would like to estimate the original signal x(M) at the finest scale such that the mean square error is minimized, that is, to fuse all these multiresolution measurements together to obtain the least squares estimation of the original signal x(M). Prom the system theory point of view, we may interpret the scaling coefficient vector x(m) as a state vector, the operator H^ as the transition matrix, and the wavelet coefficient vector w(m) as the driving signal in the wavelet synthesis equation which then becomes a state equation evolving with the scale. In that sense, a linear dynamic system on the scale index m is formulated for ID signals based on the ID orthonormal wavelet transform. With this scale dynamic system 13,14 , the Kalman filtering can be used to optimally estimate the signal x(m) recursively with the scale index m. Specifically,
330
forTO= £ , . . . , M, we have x{m\m) — x(m\m — 1) + K{m)
[y(m) — C{m)x{m\m — 1)]
(16) 1
K{m) = P(m\m - l)C*(m)[C(m)P(m|m - l)C*(m) + Rim)]P(m\m - 1) = H^Pim
- l|m - l)Hm^
+ G'^Qim
- l)Gm-i
l
P - H m l m ) = P - ^ m l m - 1) + C*{m)R- {m)C(m) X(TO|TO
- 1) = H^_!
(17) (18) (19)
x{m-l\m-\)
(20)
with the initial conditions: x{L\L-\)
= 0\
P{L\L-l)
= Px{L)
(21)
where Q(m) d= E[w(m)w(m)T],
E[x(L)x(L)T] (22) the component state variables in the state vector a; (TO) denote the sequence of signal values at scale level m, (TO = L,...,M), PX{L) denotes the covariance matrix of signal x(L) at the coarsest scale level L, X(TO|TO — 1) denotes the one step prediction of x(m), x(TO|m) denotes the filtering estimate of x(m), and P(TO|TO—1) and P(m|m) denote their corresponding error covariance matrices. This scale recursive filtering estimates the entire signal x(m) at each scale level m. The computation of the inverse of large matrices involved in the associated Riccati equations would impose a heavy burden on the filtering process. 2.2. Wavelet
Packets
R(m) = f E[t>(jn)u(m) T ],
in Signal
P , ( i ) =f
Filtering
Wavelet packets {/x„(£); n € Z+} relative to the orthonormal scaling function 4>(t) is defined by 25 ' 26
Mt) = Ht) ; /*i(0 = ^(*) m(t) = J2^2 M*) W(2* - *)
(23)
k
»2i+i{t) = J ] v ^ g{k) m(2t - k) k
where Z+ denotes the set of non-negative integers. The orthogonality properties of the orthonormal scaling function and wavelet are preserved in the wavelet packets:
= 8jk;
=0;
j,k€Z;neZ+ j,keZ;lGZ+
(24) (25)
Let us define a family of subspaces £/" as follows: , def-
tf" %* {2H*nn{2H -k);kEZ};
jeZ;n€Z+
(26)
which are generated by {/zn} where n is the packet index and j is the scale index. Note that Uf = Vy, Uj = Wj, {Vj}j&z is a multiresolution analysis and {Wj}j£z
331
^
r Fig. 1. Eight packets of the Daubechies orthogonal wavelet D4, re-ordered according to their frequency content. {/*£(i);n = 0,1,2,3,4,5,6, 7} correspond to {jun(t);rc = 0,1,3,2,6,7,5,4}.
is an orthonormal basis of L2(R). Since Vj+\ = Vj © Wj, we have [/9+1 = [7° Uj. This can be extended to any wavelet packet index n, Uj+x = U]n © Ufn+1;
jeZ;nEZ+.
(27)
It is noted that Vj = U$ = Vf_! © Uj_x = U?_2 © Uj_2 © I7?_2 © U]_2 — t/j_ f c © Uj_k © f/j_fc ©
®^V
1
= US®U£®---@U?-
(28a)
Wj = Uj = U]_x © t/?_! = Uf_2 © J7|_ 2 © Uf_2 © C/J_2
= D& © Uf_tl = U*
l
®U*+ ®
••©&?-* 1 TT2«+ )_1 ©C/Q
(28b)
For a given scale index j 6 Z+ and for decomposition to a scale level I (I < j,l € Z+), the functions {21/2 fj,n(2lt — k); 0 < n < 2j~l — 1} form an orthonormal basis of Vj and {2l>2 /j,n(2lt - k); 2j~l
332
Wj. While we interpret the wavelet subspace as containing certain high frequency components in the signal x(t), the use of wavelet packets is to further partition the high frequency octaves to provide finer frequency localization. Consider the Daubechies compactly supported orthogonal wavelet D4 (designated as db2 in MATLAB software) that we will use in the examples discussed later, a set of eight wavelet packets, for decomposition from the finest resolution space U§ (i.e. V3) at the scale level 3 down to the scale level 0, are shown in Fig. 1. The coefficients {h(k)} and {g(k)} are given by {-0.12941, 0.22414, 0.83652, 0.48296} and {-0.48296, 0.83652, -0.22414, -0.12941} respectively. Note that the natural order of n of the above-derived packets does not correspond to the order of the increasing frequency content. The packets (/J* (£); n = 0 , 1 , . . . , 7) shown in Fig. 1 reflect the reordering of /J,n(t) accordingly. In what follows, let us denote the packet index by k instead of n. The orthogonal projection of a signal x(t) onto the subspace 17* may be written in terms of the orthonormal basis functions {2->Hk{2H — q); q G Z}, with the set of coefficients {Cjq}q£z- It can be shown that the decomposition relationships between the coefficients c*, r „'s at a fine scale level j + 1 and the coefficients c?*„'s and c?* +1 's at the coarser scale level j are given by
C
)T
= Y,9V-
1
2<
?) CJ+U-
(29)
1
Note that [7° = Vj\ Uj = Wj, which implies that c® q = x(j; q), c^q = w(j;q), where x(j; q) and w(j; q) are respectively the scaling coefficient and the wavelet coefficient of the signal x(t) at scale j level and time q. Refer to Eqs. (28a) and (29), the approximation xm(t) of a signal x(t), in the resolution space Vm = U^ is represented by the scaling coefficient vector x(m) or the zeroth packet coefficient vector [c^ ]. The approximation #(771+ 1) at the scale level m + 1 may be expressed, from Eq. (11), as x(m + 1) = Po(m) x(m) + P?(m) w{m) where we let Po(m) = Hm and Pi(m) = Gm, and w{m) is the wavelet coefficient vector at the scale level m. Because of Eq. (28a), xm(t) may also be represented in terms of the orthonormal basis functions of subspaces £/? ,Uj,Uj,--and £/? ~x, xm{t)
= £ w j ( w ; g ) • 2 ^ o ( 2 ^ - q) + Ylu((m;q) g
• ^^{2H
- q)
1
+ ... + Y,ui(rn_j)_1(m;q)
• 2 = /z 2Cm -„_ 1 (2^ - q)
(30)
1
where v?h{m;q) is the qXh. element of u3k(m), uJk(m) is the fcth wavelet packet coefficient vector of the approximation xm(t) decomposed down to the scale level j (j < m), and {22^,k{2H — q);q £ Z} are the orthonormal basis functions of 17*. Obviously, x(m) is equivalent to {ujk(m); k = 0 , 1 , . . . , 2 ( m _ j ) - 1}. For finite length
333
signals, the scaling coefficient vector x(m) has only finite length; consequently, the coefficient vectors of all wavelet packets {u3k(m); k = 0 , 1 , . . . , 2(-m~^ — 1} are all of finite lengths, their lengths are much shorter and are all equal. 3. Wavelet-Based Scale-Recursive Image Filtering The 2D wavelet transform can be obtained by the tensor product of two ID wavelet transforms. In this case, we have one scaling function (j>(x,y) and three wavelets ipk(x,y), k — 1,2,3, which are given by tpl(x,y)
(x) cj>(y), 2
= ip(x) (j)(y),
3
ip (x, y) = (j>{x) ip(y),
rp (x, y) = ip(x) ip{y)
where the Fourier transform of (p(x, y) has the low-pass characteristic, and Fourier transforms of ipl{x,y), ip2(x,y), and ip3{x,y) have the horizontal, vertical, and diagonal high-pass characteristics respectively. 3.1. Scale Dynamic
System
of 2D Images
and Kalman
Filtering
Let X(m + 1) be a N(m + 1) x N(m + 1) matrix which represents an image at the scale level m +1. Its coarser version X(m) and detail images Di(m), Z?2(m), Ds(m) (which represent the difference in information between X(m + 1) and X{m) in the horizontal, vertical and diagonal directions) at the next scale level m are given by X(m) = HmX(m D2(m)=GmX(m
+ l)H*m,
D1(m) = HmX(m
+ l)G*m
+ l)H*m,
D3(m) = GmX(m
+ l)G*m
where Hm and Gm, as defined before, are respectively the coarsening operator and the differencing operator at the scale level m, the superscript * denotes an adjoint operator, and all the matrices X(m), £>i(m), £>2(m) and Dz(m) are of the size N{m) x N(m). The above equations are the 2D wavelet analysis equations which are obtained by the tensor product of two ID wavelet analyses, i.e., the row-processing followed by the column-processing 13,16 . In this case, X{m), Di(m), D2{m) and D3(m) are respectively the scaling coefficient matrix, the horizontalhigh-frequency, vertical-high-frequency and diagonal-high-frequency wavelet coefficient matrices at the scale level m. For an orthonormal wavelet under consideration, the image X(m +1) at the fine scale level m + 1 can be reconstructed from its coarse version X(m) at the scale level m, by adding the detail information Di(m), D2(m) and D3(m) as follows: X(m + 1) = H*mX(m)Hm
+ H^lD1(m)Gm
+ G*mD2{m)Hm + G*mD3(m)Gm
(33)
which is the 2D wavelet synthesis equation. We may rewrite it as X(m + 1) = H*mX(m)Hm
+ LmD(m)Rm
(34)
where Lm = [HZn;G*m\G*Tn],
RTn = [G*m;H'}n\G*m\*,
(35)
334 £»i(m) 0 0 0 D2(m) 0 0 0 D3{m)_
D(m) =
(36)
This can be interpreted as a state equation in which X{m) is composed of state variables, D(m) is the driving signal, and the state transition takes place from the coarse scale level m to the fine scale level m + 1 . For the sake of convenience in analysis, let us again rewrite the 2D wavelet synthesis equation Eq. (34) represented in matrix form into its equivalent vector form. Specifically, we may find an isomorphism (transformation) T of a finite matrix space onto a finite vector space and define the following equivalent variables: x(m)=T w2(m)=T
X(m),
wi(m) = T
Dx{m),
D2(m),
w3(m)=T
D3(m)
(37)
where x(m),wi(m), w2(m) a n d u ^ m ) areiV 2 (m)xl vectors because X(m), £>i(m), D2(m) and D3(m) are N(m) x N(m) matrices. The isomorphism may be given by T
:
X(m;i,j)-+x{m;(i-l)-N(m)+j) th
where X(m;i,j) is the (i,j) entry of the matrix X(m), element of the vector x{m). In that case, x(m + 1) = H^*x(m)
+ G£
(38)
and x(m;k)
dx{m) + G^*d 2 (m) + G£
is the kth
d3{m)
(39)
where Hm and G^ , (k = 1,2,3), are the augmented operators such that Hm x(m), G 1 , di(m), G^ d2{m) and G^ d3{rn) are respectively equivalent t o H*mX{m)Hm, H^lD1(m)Gm, G*mD2(m)Hm and G*mD3(m)Gm in the following sense: Hmx(m)=T
{H*mX{m)Hm),
G)ndl{m)=T
{H*mDx{m)Gm)
G3m d3(m) = T
G*m d2{m) = T (G*mD2(m)Hm),
(40)
(G*mD3(m)Gm).
Due to the orthonormality properties of Hm and Gm (refer to Eq. (14)), the orthonormality of Hm and (?*, are preserved, as shown by the following equations 16 : HmHm
= GkmGkm = J :
for k = 1,2,3
JCGi* = G^lC* = 0 for G^G^*
= 0
k = 1,2,3
for k # I
Hm Hm + G]n Gl + Gl Gl + Gl G*m = I.
(41) (42) (43) (44)
Let us define CZ* = [GC;Gl;GC]
,
d(m) = [d 1 (m) T ;d 2 (m) T ;d 3 (m) T ] :
(45)
The synthesis equation, Eq. (39), then becomes x(m + 1) = Hm x(m) + Gm
d(m).
(46)
335
From the orthonormal property, we have the following identities similar to Eq. (14): HmHm
= GmGm
= J, HmGm
= GmHm
= 0, i / m .ffro + Gm Gm — I. (47)
Suppose that an image is measured at several scales, from the coarsest scale level L to the finest scale level M, and is represented in its equivalent vector form: y(m) = C{m) x(m) + v(m)
(48)
where x(m) = T X(m) and y(m) = T Y{m) represent respectively the original image and its noisy measurement (represented in the vector forms), C{m) is the measurement matrix, and v(m) is the additive noise. From Eqs. (46) and (48), the wavelet-based scale dynamic system of 2D images (in vector form) is formulated, in which x(m) is the state vector, d(m) is the driving signal, and the aug—* mented operator Hm is the state transition matrix. Similar t o Eqs. (16)-(22), the scale recursive Kalman filtering can be applied to this scale dynamic system (The corresponding Eqs. (17) and (20) will be referred to later in Propositions 3 and 4). The least-square estimate X(M\M) of the image X{M) at the finest scale level M is given by the inverse transformation of x(M\M), = T-1
X{M\M)
x(M\M).
(49)
It is noted that the above scale-recursive Kalman filtering is a single filtering process. The inverse of huge N(m)2 x N(m)2 correlation matrices P _ 1 ( m | m — 1), etc., which is required in the Kalman gain equation and the associated Riccati equation at each scale level m, is very difficult to compute. The following section presents an efficient algorithm based on the wavelet packet transform 16,17 . 3.2. Wavelet
Packets
in Kalman
Filtering
Algorithm
for
Images
The 2D wavelet packet transform can be obtained by using the tensor product of two ID wavelet packet transforms. Let us rename the 2D scaling function and three wavelets given in Eq. (31) as Ho(x,y) = (p{x) <j>(y), Mx,y)
= {x) V>(y),
Hi(x,y) = ip(x) (y) Hz{x,y) = ip{x) ip(y)
(50)
and define the wavelet packet basis functions (for n € Z+) as -q,y-i) /x4n+i = 2 ^2^2g(q) q
h(l) /j,n{x -q,y-i)
I
tHn+2 = 2 X]5Z h^ 9W Vn(x -q,y-i) 1
I
»4n+3 = 2 ^2 ] T s ( g ) g(l) fin(x -q,y-i)q
i
(51)
336
Subspaces U"'s spanned by each basis function (in(x, y) can be similarly denned as in Eq. (26) U ? = f {2JHn{Vx - q,2iy -l);k<E Z};
j E Z;n € Z+.
(52)
However, the 2D wavelet packets have a quad-tree structure
u? = u£ x © u^j 1 e u^t 2 © u^+3.
(53)
The approximation of a given image at the scale level m is represented by its scaling coefficient matrix X(m) of the translated basis functions in subspace U ^ . It can be decomposed into four packets (coefficient matrices) in four subspaces TJjrj-i (k = 0,1,2,3) at the next scale level m — 1, and each of which can again be decomposed into four packets in the next coarser scale level. Let Uk (m) denote the fcth packet coefficient matrix of X{m) decomposed down to the scale level j , Uk (m) is in the subspace U*. Rewrite Eq. (33) as follows to synthesize the approximate image X(m+1) at the fine scale level ra+1 in terms of the scaling image X{m) € U ^ and detail images Di(m) in \5%m, (i = 1,2,3), at the scale level m, X(m + 1) = P 0 *(m)X(m)P 0 (m) + P0* (m)£>i(m)Pi(m) + P?(m)D2(m)P0{m)
+ P]*(m)Z>3(m)Pi(m)
(54)
where Po(m) = Hm and Pi(m) = Gm. With an isomorphism T, defined in Eq. (38), which bijectively transforms a finite matrix space onto a finite vector space, we can rewrite the above equation into its equivalent vector form x(m + 1) = T X(m + 1) = T (PZ(m)X(m)P0(m))+T T
(P^^Dr^P^m))
(P{(m)D2(m)P0(m))+T
= Po*x{m) + P7dx{m) + p7d2(m)
+
(P^m^M^iM) + F?'d3(m)
(55)
where Pv{m)=H^
; PvW^Gl
; Pi{m) = 0%, ; H ( m ) = G\
that have been given in Eq. (40). To compute the fcth wavelet packet coefficient matrix Uk (m) (or its equivalent vector u£(m)) at the scale level j , from a scaling image X(m) (or its equivalent vector x(m)) at the scale level m, let us introduce an operator (vk(m))* on x(m) where w^(m) is defined as 771 — 1
= ^ ( r a - i ) ( n » - l ) o P ; f c ( r a _ 2 ) ( m - 2 ) o . . . p ; i k ( . ) ( j ) ; ek(i) G {0,1,2,3}
337
which denotes a sequence of operations from P* tj)(j) to P* („,_!)(?« — 1), and the packet index fc is expressed in terms of £fc(i)'s: m—1
ek (») • 4<*-J>; fc = 0 , . . . , 4 ( m - ^ - 1
k = £
(57)
i=i
so that, for a given k, Sk (i) 's can be determined and the operators PSk (») (i) involved can be identified. For example, for two-level decomposition j — J — m — 2, fc runs from 0 to 15; and for fc = 6 = 2-4° + l-4 1 , we have v^(m) = pT(m-l)oF^(m-2). In particular, let us consider the decomposition down to the coarsest level j = J; for simplicity in notation, let us drop the supscript j on the operator vjt(m), vector Uk(m), and matrix Uk(m), so the wavelet packet decomposition is given by Uk(m) = (vk(m))* oX(m)
(58)
where Uk(m) = T Uk(m) is the equivalent vector representation of the wavelet packet coefficient matrix Uk(m). The feth wavelet packet Uk{m) (at the coarse scale level J, J <m) of X{m) (at the scale level m) is given by Uk(m)=T-1o(Vk(m))*oT
X(m).
(59)
T
Let Rxx(m) = E[x{m)x(m) ] be the autocorrelation matrix of x{m) which is the vector equivalent of the scaling coefficient matrix at the scale level m. The correlation matrix between its fcth and Zth wavelet packets at the scale level J is given by E [uk(m) ui(m)T]
= E [(vk{m))*
x(m)x(m)T
= (fjt (m)) * Rxx(m)
vi(m).
vi(m)] (60)
We know that the wavelet coefficients (at all scales) of wide sense stationary signals or nonstationary signals with stationary increments are also stationary and their correlations decay exponentially fast both across scales and along the time index 10,11 . So, the wavelet packet coefficients Wfc(m)'s are stationary and their packet correlations decay fast across the packet index 16 . In fact, it has been observed that for orthogonal wavelet transforms, the transformed packets are nearly uncorrelated, i.e., the correlation matrix of the wavelet packet transform is almost block-diagonal. This strong decorrelation phenomenon of the wavelet packet transform enables us to model images by using the wavelet packet synthesis equation x m
( ) =
5Z
v
k(m)
u
k{m)
(61)
if fc £ I.
(62)
it=o
with the following constraint (for J < m): E [uk(m) ui{m)T] = 0;
This will be the wavelet-packet-based dynamic s y s t e m model to approximate images, yielding the Kalman filtering algorithm with correlation matrices Q, R and P all block-diagonalized.
338
Due to the orthogonality properties of {Pk(m); k = 0,1,2,3} (see Eqs. (41)(44)), and the definition of Vk{m), it can be shown that the following identities hold true: (vk(m))*vi{m)
= ISki
(63)
vk(m)(vk(m)y
=I
(64)
4(">-J)_l
£ fc=0
where the identity matrix in Eq. (64) has the size of x(m) while the identity matrix in Eq. (63) has the size of the packets. The following four propositions have been shown to be valid 16 . Proposition 1: If a matrix A can be bloch-diagonalized by the operators {vk(m)'s ;fc = 0 , . . . , 4 ( m - J ) - l } , i.e., {vk{m)Y Avi{m) = Ak8kl
(65)
then its inverse A~l can also be block-diagonalized (vkim^A^Viim)
= A^Ski
(66)
and vice versa. We will choose an appropriate orthonormal wavelet such that the wavelet packets C/fc(M)'s of an image matrix X(M) are decorrelated (or nearly decorrelated), i.e. E [uk(M)u,(M)T]
=0;
if k ? I.
(67)
The above equation means that (vk(m))* Qa(m) vi(m) = Qa,k(m)
6
ki ;
a = 1,2,3
(68)
where Qa(m) = E[da(m)da(m)T],
da(m) = T Da(m)
(69)
Qa(jn) is the augmented autocorrelation matrix of the wavelet coefficient matrix Da(m) (at the scale m < M) of X(M). Similarly, we may have (vk(m))* R(m) vi(m) = Rk(m) Ski
(70)
where R(m) is the augmented autocorrelation matrix of the additive noise, represented in vector form, at the scale level m. The measurement matrix C(m) may also satisfy the following equation, (vk(m)y
C(m) v,(m) - Ck(m) 8kl.
(71)
Now, we can transform all the large dimensional matrices and vectors in the original 2D Kalman filtering process (similar to Eq. (16) to (22)) into a set of small size
339
uncorrelated variables as follows: uk{m\m — 1) = (vk(m))*x(m\m uk(m\m)
— 1)
(72)
= (vk(m))*x(m\m)
Pk{m\m - 1) d= {vk{m))*P{m\m
(73) - l)vk{m)
(74)
d
P fc (m|m) = («fc(»n))*P(m|m)t;fc(m)
(75)
f
J/fc(m)H K(m))*y(m)
(76)
-J _1
for k = 0 , 1 , . . . , 4("* ) . The following propositions are essential to develop an efficient 2D Kalman filtering algorithm using orthonormal wavelet packets (for proof, see Ref. 16). Proposition 2: The augmented error covariance matrices P(m\m) and P(m|m—1) can be block-diagonalized by {vk(m); k = 0 , . . . ; 4 ( m - J ) — 1}. That is, (vk(m))* P(m\m - 1) v,(m) = Pk(m\m - 1) Su
(77)
and hence, by Proposition 1, {vk{m)Y p - ^ m l r o ) vi(m) = P ^ M m ) SH.
(78)
Proposition 3: Prom Eq. (20) for 2D images (Section 3.1), we have Uk{m\m — 1) = (vk(m))*
x(m\m — 1) •
—
•
*
= («*(m))* o P 0 ( m - 1 ) _ ( uk(m - l\m - 1) ;
x(m-l\m-l) k €[0,4{m~J~1]) otherwise
(79)
Proposition 4: Tfte Kalman gain matrix K(m) in Eq. (17) for 2D images (Section 3.1) can be block-diagonalized by {vk(in); k = 0 , . . . ; 4 ( m - J ) — l } ; i.e., (vk(m))* K(m) vi(m) = Kk(m) Ski
(80)
where Kk(m) = Pk(m\m - 1) C*k(m) [Ck(m) Pk(m\m - 1) C*k(m) + P f c ( m ) ] _ 1 . 3.3. Wavelet-Packet-Based Kalman Filtering Multiresolution Image Fusion
Algorithm
(81)
for
Based on the above discussions, we can summerize the wavelet-packet-based Kalman filtering algorithm in the following. Given a set of noisy measurements Y(m) of an image X at scales m = L,L + l,...,M, the least squares estimate X(M\M) of X at the finest scale level M can be obtained, in a scale recursive form, from the coarsest scale level L to the finest scale level M. Choose an appropriate orthonormal analyzing wavelet such that the wavelet packets of the image X are nearly uncorrelated. We can take the 2D wavelet packet transform of each measurement
340 Y(m), (m = L,..., M), down to the packets of small size at a coarse scale J (J < L in general), and apply the Kalman filtering separately to each wavelet packet of F(m) from the coarse scale level m = L to the fine scale level m = M recursively. Specifically, for packet index k = 0 , . . . , 4 ( M ~ J ) - 1, and scale level m = L,...,M, yk(m) uk(m\m)
= (vk(m))*oT
Y{m)
(82)
= uk(m\m - 1) + Kk{m) [yk{m) - Ck{m)uk(m\m
Kk{m) = Pk(m\m - l)C*k(m)[Ck(m)Pk(m\m ( Pk{m - l\m - 1) Pk{m\m-
1) = <
Qi,fc-4("— j-i)(rn—
1)
Q2,fc-2-4<"— J-l)(m
— 1)
> <33,fc-3-4C"— J-i){m
- 1)
ke ke fcG ke
- 1)]
(83)
- l)C*k{m) + ifc(m)]- 1 (84)
[o^™-- 7 - 1 )) [4(ro-J-1),2-4(m_J~1)) [2-4(m-J-1),3-4(m-J-1)) [3-4(m-J-1),4(m"J))
(85)
l) + Ct{m)Rk\m)Ck{m)
(86)
uk(m\m - 1) = { "fc(m " 1 | m ~ 1; 5 ^ 6 [°' 4(m ~' 7 " 1) )
(87)
Pk-\m\m)=Pk-\m\m
\ 0
;
otherwise
with the initial conditions (i = 0 , . . . ,4^ L J ) — 1): ut(L\L - 1) = 0 ;
Pi(L\L-l)
Finally, the least squares estimate X(M\M) by
= (Vi(L))* Px(L)Vi(L).
(88)
of X at the finest scale level M is given
^4(M-J)_1
X(M|M) = r -
1
I
^
t; fc (M)u fc (M|M)
(89)
fc=0
where T~x is the inverse transformation of the isomorphism T. Let the size of image X(m) at the scale level m be N(m) x N(m). The standard Kalman filtering for 2D images (Section 3.1) needs to compute the inverse of a huge N(m)2 x N(m)2 matrix in the matrix Riccati equation. Since the computation complexity of the 2D wavelet packet transform is only 0(N(m) log2 N(m)), we can decompose with ease each measurement Y(m) down to the wavelet packets of small size N(J) x N(J) at a coarse scale level J , say, N(J) = 4. The packet-based 2D filtering algorithm transforms each 4 x 4 small size matrix into a 16 x 1 vector equivalent for performing the scale-recursive Kalman filtering, which needs the inverse matrix computation of only a small 16 x 16 matrix. The filtering takes place on each of the (j^fjf)2 small size matrices, which can be implemented in parallel processing. The storage complexity can also be significantly reduced. The filtered small matrices are obtained by transforming the filtered vector equivalents back to the matrix form, from which the reconstruction (inverse transform) is applied to obtain the least squares estimate of the N(m) x N(m) image.
341
Fig. 2. Fusion of two registered, noisy measurements of a pebble image: (a) at 0 dB, (b) at 5 dB but with coarser spatial resolution, and (c) the fused image at 19.4 dB.
4. E x p e r i m e n t s of t h e F i l t e r i n g A l g o r i t h m for I m a g e Fusion We show two examples on image fusion by applying the wavelet-packet-based Kalman filtering. The first example is on a pebble image, and the second example is on fusion of a Landsat TM image and a SPOT image. 4 . 1 . A Simple
Experiment
on Image
Fusion
Consider two noisy measurements of a pebble image as shown in Fig. 2(a) and 2(b): one with signal-to-noise-ratio (SNR) of 0 dB at the same spatial resolution as that of the original image (m = 1), and the other with SNR of 5 dB at the dyadically next coarser resolution (m = 0). The fused image obtained by applying the wavelet-packet-based Kalman filtering algorithm is shown in Fig. 2(c), giving a SNR of 19.4 dB. Daubechies' D4 orthonormal wavelet was used in this experiment. Its eight wavelet packets of three levels are shown earlier in Fig. 1. 4.2. Fusion
of TM and SPOT
Images
In this experiment, we considered fusion of two remote sensing satellite images covering the same region, one image is a segment of a SPOT panchromatic image, and the other is a corresponding Landsat Thematic Mapper (TM) band 1 image. Thematic Mapper produces images in seven spectral bands with high spectral resolution, each contains special textures for discriminating natural objects, but the spatial resolution is low. SPOT images have higher spatial resolution but lower spectral resolution 23,24,30,32 . Before processing the fusion algorithm, the following calibrations were done first: (1) A geometrical transform was performed to register the TM image with respect to the SPOT image. The spatial resolution of the SPOT image is about three times finer than that of the TM image, but we transformed it to be only finer by two. A coarser version, by a factor of 2, was generated as the reference image against
342
(c)
(d)
Fig. 3. Fusion of a TM band 1 image and a SPOT image: (a) the SPOT image (256 x 256); (b-left) the coarsened SPOT image (128 x 128); (b-right) the registered TM band 1 (128 x 128); (c) the amplitude matched SPOT image; and (d) the fused image.
which the TM image was registered but retaining its original spatial resolution. (2) A look-up-table (LUT) between intensity values of the registered TM band 1 image and those of the spatially coarsened SPOT image, which have the same spatial resolution, was constructed such that a high spatial resolution estimate of the TM band 1 image was obtained by transforming the pixel intensity of the original SPOT image into the corresponding TM band 1 amplitude through this look-up-table30. After the above calibrations, we had the registered original TM band 1 image and the amplitude matched (transformed) SPOT image which were used as inputs to our wavelet-packet-based Kalman filtering algorithm for multiresolution image fusion. Fig. 3(a) shows a segment of the SPOT image (256 x 256). Fig. 3(b-left) is a coarsened SPOT image (128 x 128), which was obtained byfilteringthe SPOT image with the Daubechies D4?s low-pass filter (ft-filter) and then down-sampling by 2 in both directions. Fig. 3(b-right) is the TM band 1 image (128 x 128) registered with respect to the coarsened SPOT image. Based on these two images, a look-up-table was constructed through which the original SPOT image was transformed into the amplitude matched SPOT image shown in Fig. 3(c). The latter was regarded as
343 (Solid: TM1. Dulled: TnoSpot)
50
100
150
(a)
200
Hutognun (Solid: TMI, Dtsbed Bucd method)
250
300
50
100
150
200
250
300
(b)
Fig. 4. Histograms of the TM band 1 image (solid) and the transformed S P O T image (dashed) in (a), and of the fused image (dotted) in (b).
equivalent to a high spatial resolution estimate of the TM band 1 image. We assumed that the TM band 1 image and the SPOT image were corrupted by white noises whose respective variances are estimated by computing the variances within the corresponding small regions in the two images, covering a river area with smooth reflections. The estimated signal-to-noise ratios were 8.7 dB and 10.2 dB in the TM band 1 image and the SPOT image respectively. Fig. 3(d) shows the fused image of the registered TM band 1 image (Fig. 3(b-right)) and the amplitude matched SPOT image (Fig. 3(c)), obtained by applying the wavelet-packet-based Kalman filtering algorithm with D4 orthonormal wavelet in 5-level decompositions. The spectral information consistency between the fused image and the registered TM band 1 image was examined by comparing their grey level distributions. Fig. 4(a) shows the grey level distribution of the TM band 1 image (solid) and of the transformed SPOT image (dashed). Fig. 4(b) shows the comparison of the grey level distribution of the fused image (dotted) with that of the TM band 1 image (solid). The fused image, while bringing in information from the high spatial resolution SPOT image, has essentially the same spectral information as that of the original TM band 1 image. 5. Conclusions In a multiresolution setting, the fine scale representation of images can be produced by adding detail informations into a coarse scale representation, leading to a scale dynamic system for images and making the scale recursive Kalman filtering in the wavelet domain applicable for multiresolution image fusion. Choose an orthogonal analyzing wavelet such that the wavelet packet transform of an image is greatly decorrelated across the packet index; this will make autocorrelation matrices to be block-diagonalizable. The packet-based Kalman filtering for multisensory image fusion involves computation of inverse matrices of small sizes, and thus it is computationally efficient and robust, and may be implemented for parallel processing.
344 Acknowledgment T h e a u t h o r s would like t o t h a n k Mr. Jennting Hsu for his help in t h e final phase of t h e manuscript preparation.
References 1. M.S. Grewal. Ralman Filtering : Theory and Practice. Prentice-Hall, 1993. 2. J.W. Woods. Two-Dimensional Kalman Filtering, In T. S. Huang, editor, Two Dimensional Digital Signal Processing, pp. 155-205, New York, Springer-Verlag, 1981. 3. I. Daubechies. Orthonormal Bases of Compactly Supported Wavelets. Comm. in Pure Appl. Math., 41:909-996, 1988. 4. S. Mallat. A Theory for Multiresolution Signal Decomposition : The Wavelet Representation. IEEE Trans, on Patt. Anal, and Mach. Intell, ll(7):674-693, 1989. 5. S. Mallat and W. Hwang. Singularity Detection and Processing with Wavelets. IEEE Trans, on Information Theory, 38(2):617-643, 1992. 6. P. Flandrin. On the Spectrum of Fractional Brownian Motions. IEEE Trans, on Information Theory, 35(1):197-199, 1989. 7. P. Flandrin. Wavelet Analysis and Synthesis of Fractional Brownian Motion. IEEE Trans, on Information Theory, 38:910-917, 1992. 8. G.W. Wornell. A Karhunen-Loeve-Like Expansion for 1/f Processes via Wavelets. IEEE Trans, on Information Theory, 36(9):859-861, 1990. 9. G.W. Wornell and A.V. Oppenheim. Estimation of Fractal Signals from Noisy Measurements Using Wavelets. IEEE Trans, on Signal Processing, 40(3):611-623, 1992. 10. R.W. Dijkerman and R.R. Mazumdar. On the Correlation Structure of the Wavelet Coefficients of Fractional Brownian Motion. IEEE Trans, on Information Theory, 40(5):1609-1612, 1994. 11. R.W. Dijkerman and R.R. Mazumdar. Wavelet Representations of Stochastic Processes and Multiresolution Stochastic Models. IEEE Trans, on Signal Processing, 42(7):1640-1652, 1994. 12. Jun Zhang and Gilbert Walter. A Wavelet-Based KL-Like Expansion for Wide-Sense Stationary Random Processes. IEEE Trans, on Signal Processing, 42(7):1737-1745, 1994. 13. K.C. Chou. A Stochastic Modeling Approach to Multiscale Signal Processing. Ph. D. Diss., Dept. of Elec. Engrg. and Computer Sci., MIT, Cambridge, MA, May 1991. 14. K.C. Chou, A.S. Willsky and A. Benveniste. Multiscale Recursive Estimation, Data Fusion, And Regularization. IEEE Trans, on Automatic Control, 39(3):464-478, 1994. 15. E. Miller and A.S. Willsky. A Multiscale Approach to Sensor Fusion and the Solution of Linear Inverse Problems. Applied and Computational Harmonic Analysis, 2:127-147, 1995. 16. Hsi-Chin Hsin. Wavelet-Based Filtering in Scale Space and Image Fusion. Ph.D. dissertation, Dept. of Elec. Engrg., University of Pittsburgh, Pittsburgh, PA, 1995. 17. H.C. Hsin and C.C. Li. Multiscale 2D Kalman Filtering Based on Wavelet Transform. Proc. IEEE Int. Conf. on Image Processing, Vol. 1, pp. 80-84, 1994. 18. H.C. Hsin and C.C. Li. Wavelet-Based Filtering in Scale Space for Data Fusion. In Proc. SPIE, Vol. 2569, Wavelet Applications in Signal and Image Processing, III, pp. 701-712, 1995. 19. J.Q. Ni, K.L. Ho and K.W. Tse. Model-based Multirate Kalman Filtering Approach for Optimal Two-dimensional Signal Reconstruction from Noisy Subband Systems. Optical Engineering, 37(8):2376-2386, 1998.
345 20. C. Wen and D. Zhou. Multiscale Stochastic System Modeling and Recursive Data Fusion Estimation. Chinese Journal of Electronics, 11(2):192-195, 2002. 21. H. Li, B.S. Manjunath and S.K. Mitra. Multiscale Image Fusion Using the Wavelet Transform. Graphical Models And Image Processing, 57(3):235-245, 1995. 22. Z. Zhang and R.S. Blum. A Categorization of Multiscale-Decomposition-Based Image Fusion Schemes with a Performance Study for a Digital Camera Application. Proceedings of the IEEE, 87(8):1315-1326, 1999. 23. T. Ranchin, L. Wald and M. Mangolini. Efficient Data Fusion using Wavelet Transform: The Case for SPOT Satellite Images. Proc. SPIE, Vol. 2034, Wavelet Applications in Signal and Image Processing, pp. 171-178, 1993. 24. B. Garguet, J. Girel, J.-M. Chassery and G. Pautou. The Use of Multiresolution Analysis and Wavelet Transform for Merging SPOT Panchromatic and Multispectral Image Data. Photogrammetric Engrg. & Remote Sensing, 62(9):1057-1066, 1996. 25. C. K. Chui, An Introduction to Wavelets. Academic Press, Inc., San Diego, CA, 1992. 26. M.V. Wickerhauser. Adapted Wavelet Analysis from Theory to Software. A. K. Peters, Wellesley, MA, 1995. 27. G. Strang and T. Nguyen. Wavelets and Filter Banks, revised edition. WellsleyCambridge Press, Wellsley, MA, 1997. 28. W. Liu and L. Zhou. Image Fusion Using Wavelet Packet Transform. P r o c , 3rd Intern. Conf. on Wavelet Anal. & Appl., Chongqing, China, Vol. 1, pp. 275-280, 2003. 29. M.A. Abidi and R.C. Gonzalez. Data Fusion in Robotics and Machine Intelligence. Academic Press, Inc., San Diego, CA, 1992. 30. P.S. Chavez, S.C. Sides and J. A. Anderson. Gomparison of Three Different Methods to Merge Multiresolution and Multispectral Data: Landsat TM and SPOT Panchromatic. Photogrammetric Engineering & Remote Sensing, 57(3):295-303, 1991. 31. T.A. Wilson, S.K. Rogers and M. Kabrisky. Perceptual-Based Image Fusion for Hyperspectral Data. IEEE Trans, on Geoscience and Remote Sensing, 35(4):1007-1017, 1997. 32. C. Pohl and J.L. Van Genderen. Multisensor Image Fusion in Remote Sensing: Concepts, Methods and Applications. International J. Remote Sensing, 19(5):823-854, 1998. 33. W.P. Segars, B.M.W. Tsui, A.J. Da Silva and L. Shao. CT-PET Image Fusion using the 4D NCAT Phantom with the Purpose of Attenuation Correction. IEEE Nuclear Science Symposium Conference Record, Vol. 3, pp. 1775-1779, 2002. 34. Q. Wang, Y. Shen, Y. Zhang and J.Q. Zhang. A Quantitative Method for Evaluating the Performances of Hyperspectral Image Fusion. IEEE Trans, on Instrumentation and Measurement, 52(4):1041-1047, 2003. 35. Y. Du, P.W. Vachon and J.J. van der Sanden. Satellite Image Fusion with Multiscale Wavelet Analysis for Marine Applications: Preserving Spatial Information and Minizing Artifacts (PSIMA). Can. J. Remote Sensing, 29(l):14-23, 2003.
CHAPTER 3.8 MULTISENSOR FUSION WITH HYPERSPECTRAL IMAGING DATA: DETECTION AND CLASSIFICATION Su May Hsu and Hsiao-hua Burke Massachusetts Institute of Technology Lincoln Laboratory 244 Wood Street, Lexington, Massachusetts 02420-9185 We present two examples that show how fusing data from hyperspectral imaging (HSI) sensors with data from other sensors can enhance overall detection and classification performance. The first example involves fusing HSI data with foliage-penetration synthetic aperture radar (FOPEN SAR) data; the second example involves fusing HSI data with high-resolution panchromatic imaging (HRI) data. The fusion of HSI and SAR data exploits different phenomenology from the two different sensors. The fusion of HSI and HRI combines their superior respective spectral and spatial information. Fusion of HSI and SAR data is accomplished at the feature level. HSI data provide background characterization and material identification; HSI-SAR fusion allows us to reduce false detections and confirm target detection in the SAR image. Fusion of HSI and HRI data is implemented at both data and feature levels, resulting in a combined spatial-spectral analysis that enhances target identification. 1
INTRODUCTION
Hyperspectral imaging sensors collect data that can be represented by a threedimensional image cube. The hyperspectral data cube is resolved in along-track, cross-track and spectral dimensions. The HSI sensor affords fine spectral resolutions (AX, - 1 0 nm), typically in visible to shortwave infrared (SWIR) wavelength region (0.4 - 2.5 um). For each pixel within a hyperspectral image, a continuous spectrum is sampled and can be used to identify materials by their reflectance. One shortcoming of HSI is that it provides no surface penetration. Another shortcoming is that HSI spatial resolutions are usually coarser than those from panchromatic imagery. To overcome these limitations and enhance HSI system performanc, we fuse HSI data with other sensor data. For example, in counter camouflage, concealment and deception (CC&D) applications, HSI data can be used to identify ground cover and surface material and a low-frequency foliage penetration synthetic aperture radar (FOPEN SAR) [1] can determine if any threat objects are under concealment. 347
348
Because FOPEN SAR and HSI sensors exploit different phenomenology, their detection capabilities complement each other. FOPEN SAR typically operates at 20-700 MHz. It penetrates foliage and detects targets under tree canopy, but has significant clutter returns from trees. HSI, on the other hand, is capable of subpixel detection and material identification. Both SAR and HSI systems may suffer substantial false-alarm and missed detection rates because of their respective background clutter but we expect that combining SAR and HSI data will greatly enhance detection and identification performance. Another opportunity for HSI data fusion occurs in surface surveillance of exposed targets. Unlike conventional single-band or multispectral sensors, HSI sensors collect image data in hundreds of contiguous narrow spectral bands with only moderate spatial resolutions. By spatially sharpening a hyperspectral imge with a panchromatic high-resolution image, we can enhance image visualization for the analyst. Such a combination of sensors can be found, for example, on the National Aeronautic and Space Administration's Earth Observing (EO)-l satellite, which was launched in November 2000. This satellite includes an HSI sensor, Hyperion, and a high-resolution imaging (HRI) sensor as part of the Advanced Land Imager. The HRI sensor spatial resolution of 10 m is three times better than that of the HSI sensor. In the next section, we describe an example of HSI-SAR fusion. Because HSI and SAR sensors are distinct and exploit different phenomenology, fusion of HSI and SAR data is established at the level of features such as material composition and terrain type. We then explore HSI-HRI data fusion. HSI sharpening with HRI data is first investigated. A combined spatial-spectral analysis is then discussed to illustrate enhanced target detection and identification. 2
HSI AND FOPEN SAR DATA FUSION
In May 1997 a P-3 ultrawideband (UWB) radar operating from 215 to 730 MHz, and a Hyperspectral Digital Image Collection Experiment (HYDICE) sensor, operating from 0.4 to 2.5 urn in 210 bands [2], collected data at a site in Vicksburg, Mississippi. The SAR data were collected with a 32° depression angle and a ground sample distance (GSD) of 0.23 m x 0.4 m. The HSI data were collected at a 1.5-km altitude with a nadir viewing geometry and GSD of 0.76 m x 1.1 m. These measurements formed part of the Dixie-97 data set that we use here to demonstrate the framework of HSI-SAR data fusion. Figure 1 shows a sketch of the target site and the composite data set, including a hyperspectral data cube and a SAR graysclae image. Targets in the forest background included fabric nets and vehicles. Several fabric nets are placed along the tree line around an open area. One fabric net at the tree line covers a vehicle; all other nets are empty or cover non-radar reflecting decoys.
349
One vehicle is obscured at the lower right corner of the sketch and another vehicle is partially exposed near the top left corner. For the Dixie-97 data, an HSI-SAR fusion strategy was established on the basis of each sensor's detection characteristics. HSI and SAR data were first processed separately for detection and terrain classification, respectively. Then co-registration was performed to allow overlay of the images. Terrain mapping reduced SAR false alarm from trees. Detection of concealed targets under nets was verified and detection of partially exposed targets was further confirmed with material identification by HSI. We provide an analysis of the HSI data first, followed by illustration of SAR-HSI image co-registration and the fusion results. HYDICE HSI
P-3UWBSARImage
Site Sketch ,r\~
&~ 10nm 200 bands
Figure 1. Target site sketch (right), hyperspectral data cube (left), and synthetic aperture radar (SAR) data (middle), part of the Dixie-97 data set. The site sketch shows targets in a forest background of fabric nets and vehicles. Several fabric nets are placed along the tree line around an open area. One fabric net at the tree line covers a vehicle. The hyperspectral data arc displayed as a cube with a red-green-blue composite image on the front face. The SAR data are displayed as a radar-cross-section image in gray scale.
2.1
HSI Data Analysis
Sample spectra of backgrounds of road, grass, trees, and nets are plotted as a function of wavelength in Figure 2. The road spectrum is significantly higher than other spectra in the visible wavelength region from 0.4 to 0.7 pm. The spectra for grass and trees each exhibit a decrease of reflectance at 0.68 pm, followed by a large increase at near infrared, characteristic of vegetation, because the chlorophyll in green plants absorbs the visible light from the sun, and reflects the infrared radiation. In the SWIR wavelength region, spectral signatures of road and nets also differ from the vegetation signatures. Reduction in spectral dimensionality is first applied to the HSI data cube to extract the spectral features that then lead to further analysis. Principal
350
component analysis (PCA) is used to de-correlate data and maximize the information content in a reduced number of features [3]. Figure 3 shows sample principal components calculated from the Dixie97 HSI data. Background classes of open area, trees and roads are apparent in the first and third principal components. Fabric nets appear in strong contrast to the backgrounds in the seventh principal component. We constructed a matched filter from the mean of several pixels extracted from the nets. A matched-filtering algorithm with thresholding was then applied to the HSI data to detect all pixels of fabric nets. Figure 4 shows the fabric-net detection and a color-coded terrain classification map for the Dixie-97 HSI data. Superimposed on the terrain classification map is the HSI fabric-net detection. The map shows background classes for roads, grass, trees and shadow regions; these classes result from an unsupervised data-clustering operation that uses the first five principal components. r—i—'—•—•—•—•—i—;—'—'—'—•—r
Wavelength C"'")
Figure 2. Sample spectra from a forest scene. The road spectrum is significantly higher than other spectra in the visible wavelength region of 0.4 to 0.7 u,m. The spectra for grass and trees each exhibit decreasde of reflectance at 0.68 u.m followed by a large increase at near infrared because green plants use chlorophyll to absorb the visible light from the sun and reflect the infrared radiation. Background (1st component)
Road (3rd component)
Fabric Net (7th component)
Figure 3. Principal components calculated from the Dixie-97 HSI data. Background classes of open area, trees and roads are apparent in the first and third principal components. Fabric nets appear in strong contrast to the backgrounds in the seventh component.
351 Fabric net detection
Background characterization
Figure 4. HSI fabric net detection with a matched-filtering algorithm (left) and terrain classification map (right). The HSI fabric-net detection has been superimposed on the terrain classification map. Roads, grass, trees and shadow regions are shown as separate terrain classes. The background classes result from an unsupervised data-clustering operation that uses the first five principal components.
2.2
Combined FOPEN SAR-HSI Analysis and Fusion
To combine HSI and SAR detection results, we performed co-registration with reference to terrain classes, such as open areas and trees by using scaling, rotation and translation operations. We then fused the data by using the coregistered images according to the process illustrated in Figure 5. The SAR data were first processed with pixel grouping and thresholding. The sample SAR detection at a 6-dBsm threshold is shown in the top left image of Figure 5. The detection of a near vertical line in the top right area of this SAR image corresponds to a power line. The terrain map derived from the HYDICE HSI data with open-area and fabric-net detections is the top middle image of Figure 5. In combining these analyses, we retained SAR detections only from open areas or around fabric nets indicated in HSI data. Detections that coincided with HSI identifications of trees, far-from-open areas, and nets were considered false alarms. In the combined detection result shown in the top right image of Figure 5, a SAR detection of a vehicle under a net appears; other nets are empty with no SAR detections. There are several strong SAR detections at the left side of the open area. A spectral angle mapper (SAM) algorithm was applied to match HSI data in the open area to paints in our spectral library. Three pixels matched well with military gray-tan paint. This match indicated the presence of a vehicle, possibly military, in the open area and thus confirms the SAR detection.
352 P-3 UHF SAR 6 dBsm Thresholded
HYDICE HSl Open area/Fabric net Detection
Combined SAR/HSI Data Vehicle under Net Identified
SAR Chip SAR detection confirmed and material identified
using HSl
Figure 5. SAR detection (top left), a terrain map derived from the Hyperspecctral Digital Imagery Collection Experiment (HYDICE) data with net detections (top middle), and the combined detection result (top right). The SAR data were first processed with pixel grouping and thresholding. The sample SAR detection shows the detection of a vertical line in the top right area that corresponds to a power line. When we combined the analysises, only SAR detections from either open areas or around fabric nets indicated in the HSl data were retained. SAR detections that corresponded to identifications of trees, far-from-open areas or nets on the hyperspectral image were considered false alarms. In the combined SAR-HSI data, a SAR vehicle detection appears under a net at the top-right corner. Other nets detected in the HSl data are considered empty because they have no corresponding SAR detections. There are several strong SAR detections on the left side of the open area. A spectral angle mapper (SAM) algorithm was applied to match HSl data in the area to paints in our spectral library. Three pixels match well with military gray-tan paint, indicating the presence of a vehicle, possibly military; this match confirms the SAR detection.
HSl and HRI DATA Fusion To assess the fusion of HSl and HRI data, we generated a set of co-registered HSl and HRI data with known ground truth. We used the first frame of HYDICE data from Forest Radiance I Run 05 as the "truth" in both spatial and spectral domains; it has 320x320 pixels and 0.8-m pixel resolution. The truth data were processed to generate simulated HSl (4 m/pixel, AX ~ 10 nm) and HRI (0.8 m /pixel, panchromatic) data. The simulated HSl data were generated by a 5x5
353 spatial averaging of the input data followed by a 5x5-to-l under-sampling. The resulting HSI is 64 x 64 pixels in size and 4-m pixel resolution. The simulated HRI data were generated with a 25-band integration, from 0.63 urn to 0.9 um, and Ak ~ 10 nm, resulting in a 320 x 320 single-band image with 0.8-m pixel resolution. Figure 6 shows the flow of data generation for HSI and HRI fusion. In the following section, we first apply sharpening to the HSI data and then conduct a combined spatial-spectral analysis.
Figure 6. Flow of simulated data generation from the HSI truth data for HSI and High Resolution Imaging (HRI) data fusion. The first frame of HYDICE data from Forest Radiance I Run 05 is used as the truth in both spatial and spectral domains; it has 320x320 pixels, 0.8 m per pixel. The truth data are processed to generate the simulated HSI (4m/pixel, AX ~ 10 nm) and HRI (0.8m/pixel, panchromatic) data. The simulated HSI data are generated by a 5x5 spatial averaging of the input data followed by a 5x5-to-l undersampling. The resulting hyperspectral data cube has 25 bands; each band is 64x64 pixels in size and 4 m per pixel. The HRI data are generated with a 25-band integration from 0.63 um to 0.9 urn, AX ~ 10 nm. The result is a 320x320 single-band image, 0.8 m per pixel.
3.1
Reviews and Demonstration of Sharpening Algorithms
A number of image sharpening techniques are often applied to multispectral images for visualization enhancement. This section reviews three of the commonly used sharpening algorithms—pseudo-inverse, color normalization and spatial frequency correction—that are adapted for implementation on HSI data. We show the results of applying these algorithms to the generated HSI and HRI data for a sharpened HSI image. Spectral fidelity is evaluated by comparing the original and sharpened data.
354
3.1.1
Pseudo-inverse Technique [4,5]
Given a high-resolution image co-registered with a hyperspectral image, a system of equations can be established for the reconstruction of a sharpened, or highspatial-resolution, hyperspectral image. The value at a pixel in the highresolution image is the spectral average at the same pixel in the sharpened hyperspectral image. The spectral value at a pixel in the hyperspectral image is the pixel sum of of the sharpened hyperspectral image within the GSD of the hyperspectral image in the same spectral band. If the sharpening ratio—the ratio of GSDs between unsharpened and sharpened images—is integer r and the number of spectral bands is K, then the number of equations for each hyperspectral image pixel is r2 + K, and the number of unknowns for the sharpened hyperspectral image reconstruction is r2 x K. As the sharpening ratio and the number of spectral bands increase, the number of unknowns increases faster than the number of additional equations. The system of equations is generally underdetermined for a unique solution. Pseudo matrix-inversion algorithms with least mean squared (LMS) estimations are applied to obtain a solution for the fusion problem. This method is described in the following equations: E = AT (A A1)"1 w , and AE = w , where E is a (K r2) X 1 array containing K bands of r X r sharpened hyperspectral image pixels, A is a (K + r2) X (K r2) matrix of the system equation, AT (A A1)"1 is the pseudo inverse of A and w is a (K + r2) X 1 array of the HSI and HRI input values. In this approach the high-resolution image needs to be in perfect registration with the hyperspectral image to establish the system of equations. The pseudo-inverse technique uses singular value decomposition to obtain an LMS solution for the system. Performed on a pixel-by-pixel basis, the pseudoinverse technique is very time comsuming. Also, the LMS estimations do not necessarily attain spectral information beyond that originally contained in the HSI data. 3.1.2
Sharpening Color Normalization (SCN) [6]
The color normalization algorithm conventionally used in multispectral imaging is modified for HSI sharpening. The HSI data are first oversampled to the same pixel size as the HRI data. The algorithm multiplies each of the HSI bands by the HRI data and the resulting values are each normalized by the averaged HSI data
355
over the spectral bands covered in the panchromatic range of HRI. This process is defined by the following equation: _ _ . , _ (HSI, x HRI)
where HSI is an HSI band, SCNt is the HSI sharpened by color normalization, and (HSI)p is band-averaged HSI data over the HRI panchromatic wavelength range. This approach is straightforward in merging the spatial contrast of HRI into spectral bands of HSI. The method also requires good HSI-HRI registration. The sharpened HSI data appear visually sharper but do not contain more spectral information than the original HSI. 3.1.3
Spatial-Frequency Correction (SFC)
Some sharpening approaches use wavelet transformations [7] that decompose images into different spatial-frequency scales. In spatial-frequency correction, the HSI data are first oversampled to the same pixel size as the HRI data. For each HSI band, the high-frequency components are replaced with components from the HRI data. The sharpened image is then obtained via inverse transformation of the modified spectra. In practice, two-dimensional (2-D) Fourier transformation can be applied for image spatial-frequency analysis. The sharpening process is described as below: SFd = FFT 1 {FFT (HSI,), FFT(HRI)hi} where SFCj is the sharpened HSI, FFT(HSI,) represents the 2-D spatial frequency components of HSI, and FFT(HRT)hi represents the high-frequency components in the 2-D spatial Fourier transform of the HRI data. The sharpened HSI is the inverse 2-D Fourier transform of FFT(HRI) with the central (low-frequency) portion replaced by FFT(HSI,). In this approach, the spatial shift caussed by imperfect HSI-HRI registration shows up as a phase shift in the frequency spectrum. Some loss of spectral quality is expected in the sharpened data. 3.1.4 Discussion of Performance In the three sharpening approaches reviewed, we emphasize that good spatial registration between HSI and HRI data is required. For our purpose of spectral fidelity evaluation, we have perfect co-registration through the method used to generate the HSI and HRI data. Two measures, spectral angle and spectral
356
distance (Euclidean distance), are used to evaluate the sharpening algorithms. These quantities are calculated as
spectral angle(x!, x , ) = cos" and
x
I
i * * V
2
V
spectral distance(x,, x 2 ) = |x, - x. where xi and x2 are two multiband spectra. A zero angle or zero distance represents a perfect match of the two spectra. The sharpened images are compared to the truth data in terms of spectral angle and spectral distance. The spectral distance normalized by the pixel amplitude in the truth image is calculated for the fraction of spectral difference. The frame-averaged differences for the three sharpening algorithms are listed in Table 1. For comparison, the difference measurements from the unsharpened HSI data are also included in the table. The sharpened 4-m resolution image data are five times oversampled to compare with the truth image data at 0.8-m resolution. The sharpened images improve significantly from the unsharpened HSI data in spectral distance, but show no obvious improvement in spectral angle. Among the sharpened images, color normalization is closer to the truth in spectral angle than the results of pseudo-inverse and frequency-correction. On the other hand, the spatial-frequency correction method achieves results that are closest to the truth in spectral distance. Although the sharpened images generally appear sharper, the impact of this effect is small on target detection and identification. Because the combined HSI-HRI data are severely underdetermined for the reconstruction of sharpened HSI data, spectral features of objects smaller than the original HSI resolution cannot be fully resolved in the the sharpened HSI. Additional spatial and spectral analysis at the feature level is necessary for enhanced target detection and identification. In the subsequent spatial-spectral analysis discussion that follows, HSI data sharpened by color normalization will be used. Table 1. Sharpening-Algorithm Comparison with Respect to the Truth Data Performance Metrics Spectral Angle Spectral Distance
Unsharpened Pseudo Inverse HSI 4.9° 21.6%
8.3° 13.8%
Sharpened Frequency Correction 5.8° 8.2%
Color Normalization 3.8° 10.0%
357
3.2
Spatial-spectral Analysis
Various combinations of spatial and spectral analysis of HSI and HRI images have been attempted [8,9,10]. To demonstrate HSI and HRI fusion for enhanced background characterization as well as target detection and identification, we employ a similar combined spatial-spectral analysis. Figure 7 shows our analysis approach. We first obtain background classification and anomaly detection from HSI data. Applying these results to the sharpened HSI data provides enhanced background classification and target detection, while the HRI data provides target and background boundaries with spatial edge detection. These boundaries, combined with results from the sharpened HSI data, spatially enhance the definition of targets and backgrounds. After we apply spectral-matched filtering, the HSI data further reveal the background and target materials.
HSI
Figure 7. Diagram of a spatial-spectral analysis approach. Background classification and anomaly detection are first obtained from HSI data. Applying the results to the sharpened HSI data provides enhanced background classification and target detection, while the HRI data provide target and background boundaries with spatial edge detection. These edges, combined with results from the sharpened HSI data, spatially enhance the definition of targets and backgrounds. Finally, spetralmatched filtering for target detection is applied to the sharpened HSI data.
358
To illustrate this analysis approach, we used an expanded data set consisting of three major frames of HYDICE data from Forest Radiance I Run 05. The original truth image of 300x960 pixels, 0.8-m pixel resolution and 210 bands was used as input. The degraded low-resolution hyperspectral spectral image is 60x192 pixels in size, 4-m pixel resolution, and 210 bands. The corresponding HRI image was generated with band integration from 0.4 to 0.8 urn, 300x960 pixels, with 0.8-m pixel resolution. A sharpened hyperspectral data cube is obtained from the combined HSI and HRI data by using the colornormalization method. Figure 8 shows a reference HSI image, as well as a background classification and anomaly-detection map derived from the unsharpened (4-m GSD) HSI data. The background classification is the result of an unsupervised data-clustering operation on the first five principal components [11]. Road, ground, vegetation, and shade are delineated. The background map and class statistics are employed in all subsequent processing. Areas in the hyperspectral image that do not match the background classification are considered anomalies; these anaomalies are further processed with spatial and spectral analysis for target detection and identification. Edges detected from the HRI image with the Sobel operator [12] are shown in Figure 9. Overlay of the edges with background classification from sharpened HSI data is also shown in the figure. Background edges in the sharpened HSI data are enhanced with the edges derived from the HRI image. The image on the right of Figure 9 depicts the combined results of spectralmatched filtering and spatial edges over regions containing anomaly detections from HSI. In the far right grayscale image, red and green represent different types of vehicle paints, and the detections are bounded by the edges shown in blue. Two regions of detection located near the top of the scene, however, are not well defined in shape and appear to be large in size. These are inconsistent with vehicle features. Further testing with spectral matched filtering confirms pieces of fabric in these regions, as shown in Figure 10, which contains an enlarged view of the vehicle detections. The vehicle size and orientation can be determined from the bounding edges. Objects are classified as large vehicles (4 m x 8 m) if they are 4 to 7 pixels wide and 8 to 11 pixels long. Likewise, objects are small vehicles (3 m x 6 m), if they are 3 to 5 pixels wide and 6 to 7 pixels long. The colored bar next to each vehicle in the enlarged image depicts the vehicle's size, orientation and paint type.
359
RGB Image
Background classification
Anomaly detection • • "| •
Road Ground Brt.Veg. I Dk.Veg. I Shade
Figure 8. Reference HSl image (left), background classification (middle), and anomaly detection (right) derived from the unsharpened HSl data. The background classification is the result of an unsupervised data-clustering operation on the first five principal components. Road, ground, vegetation, and shade are delineated. The background map and class statistics are employed in all subsequent processing. Areas in the HSl reference image that do not match the background classification are considered anomalies as shown on the right.
360
HRl with edges
Detection in regions cued by HSI
Background &HRI edges • • 11 •
Road Ground Brt.Veg. Dk.Veg.
•
Shade
• Paint 1 • Paint 2 • "Edges Figure 9. HRl image with detected edges (left); background classification from sharpened HSI data overlaid with HRI-derived edges (middle); and target detection on sharpened HSI data (right). Background boundaries in the sharpened HSI are enhanced with the edges derived from the HRl image. The image on the right depicts the combined results of spectral matched filtering and spatial edges over regions with anomaly detections from HSI. Red and green colors each represent different types of vehicle paints; the detections are bounded by the edges shown in blue.
361
. «•—Net 1
et2
•
P aint 1
•
P aint 2
• • Edges | Large (4 mx 8 m) ;
Small(3mx6m)
Figure 10. Fabric and additional object identification. An enlarged view of the vehicle detections is on the right. The vehicle size and orientation can be determined from the bounding edges. Objects are classified as large vehicles (4 m x 8 m) if they are 4 to 7 pixels wide and 8 to 11 pixels long. Likewise, objects are small vehicles (3 m x 6 m), if they are 3 to 5 pixels wide and 6 to 7 pixels long. The colored bar next to each vehicle in the enlarged image depicts the vehicle's size, orientation and paint type.
SUMMARY AND DISCUSSION Relative to previous multispectral sensors, HSI sensors offer superior spectral resolution that allows detailed background characterization and material identification-related applications. These capabilities employed in conjunction with other remote sensing modalities can further enhance the combined system performance. The potential payoff of fusing HSI data with data from other sensors has been illustrated in this chapter. FOPEN SAR's capability of surface penetration compensats HSI's passive surface sensing function. The fact that SAR and HSI sense different phenomena helps in reducing false detections. HSI's other shortcoming is suboptimal spatial resolution due to trade-off with
362
fine spectral resolution. A panchromatic HSI image, typically with much better spatial resolution, provides additional spatial enhancement on target detection and background delineation derived from HSI data. Coordinated data collection is required to demonstrate multisensor fusion performance. The platforms should be relatively stable and capable of georeferencing to allow multisensor co-registration for data fusion. For the purpose of algorithm development and assessment, spatial and spectral ground truth supporting all participating sensors is also required to fully realize the system performance. While there have been some multisensor data collections to date, they were typically designed for a single sensor rather than designed synergistically for multisensors. In general, these data are not optimally suited for data/image fusion demonstration. For the SAR-HSI fusion example we provided, co-registration was accomplished by manual selection of tie points over a small common area of SAR and HSI. Simulated data with perfect co-registration were used for the HSI and HRI fusion example. In this chapter, we demonstrated the framework of HSI fusion with FOPEN SAR and HRI data by using various sources of data. P-3 UWB SAR and HYDICE HSI data from the Dixie-97 data collection over Vicksburg, Mississippi, were used for SAR-HSI data fusion. Targets in the forest background of this data set included fabric nets and vehicles. Principal component analysis on HSI data was shown to allow effective spectral dimension reduction and feature extraction for terrain characterization and fabric net detection. SAR-HSI feature fusion was accomplished with a co-registration of the images by using references to terrain features. The fusion results showed detection of a vehicle under a fabric net and reduction of SAR false alarms due to trees. A case of SAR detection was also confirmed by HSI material matching with military vehicle paint. The Dixie-97 example demonstrates the UHF SAR's capability of penertration through nets. However, the detections come with significant false alarms from trees. Background classification and anomaly detection from HSI data reduce SAR false alarms in target detection, as illustrated in the Dixie-97 example. Additional target processing of UHF SAR data with template matching and HSI material identification will both enhance target detection and reduce false alarms. For HSI-HRI data fusion, sharpening approaches were investigated and implemented on a combined HSI and HRI data set. The sharpened hyperspectral image retained the spectral signatures of the extended area and in general appeared spatially sharper. Spectral features of objects smaller than the original HSI resolution, however, were not fully resolved through sharpening, due to the underdetermined nature of the reconstruction of the sharpened HSI. Thus the utility of sharpening alone is limited. Further analysis was conducted to combine respective high-spectral and high-spatial resolution information from HSI and HRI data; this HSI-HRI fusion demonstrated enhanced background
363
characterization as well as target detection and identification performance. Anomalies detected from HSI data were used to cue for target detection and identification in the combined analysis. Spatial image processing was applied to HRI for edge delineation. As the result of combined analysis, target size, shape and material were determined. REFERENCES 1. T. G. Bryant, G. B. Morse, L. M. Novak, and J. C. Henry, "Tactical Radars for Ground Surveillance," Lincoln Laboratory Journal, vol. 12, no. 2, pp. 351-353. 2. L. J. Rickard, et al; "HYDICE: An Airborne System for Hyperspectral Imaging", SPIE Proceedings, Imaging Spectrometry of the Terrestrial Environment, Vol. 1937, p. 173, 1993 3. Schott, J. R., Remote Sensing, Oxford University Press, 1997 4. Tim J. Patterson, Michael E. Bullock, "Radiometrically Correct Sharpening of Multispectral Images using a Panchromatic Image", p. 252-264, SPIE Vol.2234 5. Tim J. Patterson, Lester Gong, Robert Haxton, Troy Chinen, "Extension of Sharpening Techniques to Hyperspectral Data", P. 114-122, SPIE Vol.3372, April 1998 6. Jim Vrabel, "Advanced Band Sharpening Study", p. 73-84, SPIE Vol.3071, April, 1997 7. Mark A. Goforth, "Multispectral Image Sharpening with Multiresolution Analysis and the MTF", P. 123-131, SPIE Vol.3372, April 1998 8. Harry N. Gross, John R. Schott, "Application of Spatial Resolution Enhancement and Spectral Mixture Analysis to Hyperspectral Images", P. 30-41, SPIE Vol.2821, 1996 9. Harry N. Gross, John R. Schott, "Application of Spectral Mixture Analysis and Image Fusion Techniques for Image Sharpening", P.85-94, Remote Sensing and Environment 63, 1998 10. Bing Zhang, Jiangui Liu, Xiangjun Wang, Changshan Wu, "Study on the Classification of Hyperspectral Data in Urban Area", P. 169-172, SPIE Vol.3502 11. S. M. Hsu, H. K. Burke, et al, "An Efficient Approach to Background Characterization of Hyperspectral Images and its Applications", Project Report HTAP-5, MIT Lincoln Laboratory, 20 October, 2000 12. R. C. Gonzalez, P. Wintz, Digital Image Processing, Addison-Wesley, 1987
364
Acknowledgements The authors are grateful for SITAC for the HYDICE data. This work was conducted under ODUSD S&T Hyperspectral Technology Assessment Program (HTAP) at Lincoln Laboratory. This work was sponsored by the Department of Defense under contract F19628-00-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and not recessarily endorsed by the United State Government. This article is reprinted from MIT LINCOLN LABORATORY Journal, Volume 14, Number 1, 2003, Special Issue on Spectral Imaging, pp. 145-159, with permission from MIT LINCOLN LABORATORY Journal.
CHAPTER 3.9 INDEPENDENT COMPONENT ANALYSIS OF FUNCTIONAL MAGNETIC RESONANCE IMAGING DATA V. D. Calhoun§Vo, B. Hong§ ^Olin Neuropsychiatry Research Center, Institute of Living, Hartford, CT 06106. v Dept. of Psychiatry, Yale University, New Haven, CT 06520. "Dept. of Psychiatry, Johns Hopkins University, Baltimore, MD 21205. E-mail: [email protected], [email protected] Independent component analysis (ICA) has recently demonstrated considerable promise in characterizing fMRI data, primarily due to its intuitive nature and ability for flexible characterization of brain function. As typically applied, spatial brain networks are assumed to be systematically non-overlapping. Often temporal coherence of brain networks is also assumed, although convolutive and other models can be utilized to relax this assumption. ICA has been successfully utilized in a number of exciting fMRI applications including the identification of various signal-types (e.g. task and transiently task-related, and physiology-related signals) in the spatial or temporal domain, the analysis of multi-subject fMRI data, the incorporation of a priori information, and for the analysis of complex-valued fMRI data (which has proved challenging for standard approaches). In this chapter, we 1) introduce ICA and review current algorithms, 2) relate ICA to several well-known pattern recognition techniques, 3) introduce fMRI data and its properties, 4) review the basic motivation for using ICA on fMRI data, and 5) review the current work on ICA of fMRI and the incorporation of prior information. 1
Introduction What an antithetical mind! - tenderness, roughness - delicacy, coarseness sentiment, sensuality - soaring and groveling, dirt and deity - all mixed up in that one compound of inspired clay! -Lord Byron
Independent component analysis (ICA) is a statistical method that recovers a set of linearly mixed hidden independent factors (sources or features) from a set of measurements or observed data. Unlike principal component analysis (PCA) which seeks for an orthogonal transformation to remove 2nd-order statistical dependency, ICA not only decorrelates the 2nd-order signal statistical moments, but also reduces higher-order statistical dependencies. ICA is a way of finding a linear non-orthogonal transformation which makes the resulting signals or factors as statistically independent from each other as possible. 365
366 An intuitive example of ICA can be given by a scatter-plot of two independent signals x and x2. Figure 1.1a shows a plot of the two independent signals (*,,* 2 ) in a scatter-plot. Figure 1.1b and c show the projections for PCA and ICA, respectively, for a linear mixture of x, and x2. PCA finds the orthogonal vectors ux,u2, but does not find independent vectors. In contrast, ICA is able to find the independent vectors ax,a2 of the linear mixed signals (x{,x2), and is thus able to restore the original sources.
X .<
X Figure 1.1: (a) The joint density of two independent signals, (b) PCA projection (w,, u2), (c) ICA projection (a,, a 2 ) (adoptedfrom[1]) A typical ICA model assumes that the source signals are not observable, statistically independent and non-Gaussian, with an unknown, but linear, mixing process. Consider an observed M - dimensional random vector is denoted by x = (x,, • • • xM )T which is generated by the ICA model: x = As (1) where s = [j,,5 2 ...sNf is an N-dimensional vector whose elements are assumed independent sources and AMxJV is an unknown mixing matrix. Typically M >= N, so that A is of full rank. Mathematically, since AMxN is an unknown mixing matrix. The goal of ICA is to estimate an unmixing matrix Ww„w by an iteration approach such that y (defined in equation (2)) is a good approximation to the 'true' sources: s . y = Wx (2) ICA is a type of blind source separation, meaning that it can separate independent signals from linear mixtures with virtually no prior knowledge on the signals. A typical example of blind source separation is the cocktail party problem in which several people are speaking simultaneously in the same room. Then the problem is to separate the voices of the different speakers, using recordings of several microphones in the room [2]. The basic ICA model for blind source separation is shown in Figure 1.2.
Unknown Figure 1.2: ICA basic model for blind source separation
367
ICA is area of active research in fields such as neural networks, advanced statistics, and signal processing. So far, there have been a number of different methods for deriving ICA algorithms. Popular methods include those using the Infomax principle [3], cumulant-based methods [4,5], maximal likelihood estimation [6], mutual information minimization [7,8]. Among these methods, the most typical ICA algorithms are Infomax [3], FastICA [9] and JADE [10]. The original infomax algorithm for blind separation by [3] is suitable for super-Gaussian sources. To overcome this limitation, techniques have been developed for simultaneously separating sub- and super-Gaussian sources [11]. A flexible independent component analysis approach using generalized Gaussian density model method was introduced in [12]. These algorithms typically work well only for symmetric distributions and are less accurate for skewed distributions. Recently, some researchers have made contributions for the extension of ICA to more general distributions including introducing non-parametric ICA [13] and kernel independent component analysis [14]. Other ICA models which adaptively vary the nonlinear functions (or activation functions) to better fit the underlying sources have also been proposed [15,16]. The variety of recent approaches to ICA algorithms demonstrates the vitality of current research in this area. 2
ICA and Pattern Recognition
Pattern recognition has a wide array of applications in such diverse areas as face and character recognition, data mining, medical diagnosis, image processing, computer vision, bioinformatics, speech recognition, fraud detection, and stock market prediction, etc. ICA is an unsupervised statistical approach and is used for both feature extraction or classification/clustering problems in pattern recognition. 2.1
ICA Feature extraction
Feature extraction seeks a transformation mapping from an original data space to a lower dimensional feature subspace. PCA seeks a set of orthogonal basis vectors in the data space with respect to which the transformed (projected) data are uncorrelated. ICA goes farther by seeking a basis (not necessarily orthogonal) such that the corresponding transformed data are statistically independent. The use of ICA for feature extraction is motivated by the representation of components (features) as independent from each other as possible. The feature extraction procedure for ICA involves two standard preprocessing steps. First, the mean of the data is usually removed from the original data. The second step is to whiten the data such that the transformed data is uncorrelated with unit variance. For similarity, let's re-denote \(k) a TV-dimensional vector of the observed mixed input patterns at time k. We rewritten the equations above such that these contains time moment as \(k) = [xl(k),x2(k),...,xN(k)f. It is linearly composed of N statistically independent components s(k) = [s,(k),s2(k),....,sN(k)]T by this relationship
x(k) = As(k) = fjsi(kX
(3)
368
where A denotes a NxN unknown matrix and a,, is the z'th column of A . Similar to the PCA projection transformation, Equation (3) views ICA as a projection transformation. That is, the input pattern vectors x(k) are considered to be a linear mixture of statistically independent basis pattern s(k) combined by an unknown mixing matrix A . A is viewed as the ICA projection transformation matrix. In this sense, a, is viewed as the basis vector and s((k) is regarded as the i'h projection component of x(k) onto the basis vector a,. Feature extraction using ICA aims to estimate A such that each component of s(k) becomes maximally independent of the other components. To proceed, the input patterns x(k) are first whitened by PC A [17] z(jfc) = D~'vTx(k)
(4)
where D and V are given as follows: D = diag[A[,A2,---,A,N] and V = [v,,v2,---vJV]. Here A, is the /* largest eigenvalue of the covariance matrix Eix(k)x(k)T\ and v,. is the i',h eigenvector. And D and V satisfy: VDV r =E{x(k)x(k)T]. While z(jfc) is uncorrelated, but still non-independent. The independent components s(k) are then given by s(Jfc) = Wz(ifc) = WD" 2 V r x(£)
(5)
where W is an NxN separation (unmixed) matrix. It can not be computed directly from other matrices like A , D or V . W is typically obtained by iterative approaches which depend on choice of ICA algorithms. For example, for the Infomax algorithm, W is determined by AWoc[WT'+(l-2y)xr (6) Aw0 « 1 - 2y
(7)
where the output vector y = g(Wx + w 0 ) with nonlinearity g(w) = tanh(w) [3]. After convergence of W , the estimated transformation matrix of ICA is constructed as A = VD 1 W
(8)
Where each column a. of A is considered the i'h basis vector and thus used as a feature vector. A block diagram of ICA feature extraction is shown Figure 2.1 Input pattern vector
x(k)
Whitened
D
2
Vr
vector
Z(k)
ICA feature
W
vector
at of A
Figure 2.1: A block diagram of ICA feature extraction To date there have been several application of feature extraction using ICA [1721]. An example for ICA feature extraction is illustrated in Figure 2.2.
369
Figure 2.2; The ICA feature representation of a natural image. st and Sj (i ^ j) are statistically independent components of the original data (adopted from [17]) 2.2
ICA for classification and clustering In pattern classification problems, it is common to come up against the challenges of high dimensional data (i.e., curse of dimensionality). High-dimensional data not only bring computational complexity, but also degrade a classifier's performance. To avoid the problem of dimensionality, the most common approach is to use feature extraction or dimensionality reduction algorithms. There have been several popular methods widely used in dimension reduction for pattern recognition problems. PCA is a well known method for reducing the dimensionality by extracting components which are uncorrelated with each other. Another approach is linear discriminative analysis [22] which can be used to derive a discriminative transformation, making use of second-order statistical information. Feature subset selection [23] is an approach which requires an exhaustive search for feature selection. Support vector machines [24] are a learning approach not suffering from the curse of dimensionality, however the robustness of the probabilistic models is not always guaranteed [25]. ICA, instead, seeks a representation that projects the data into a space where the components have maximal statistical independence. From the viewpoint of pattern recognition, this representation provides high efficiency for pattern classification. There are some advantages of ICA compared with the above methods. First of all, for image compression, the features extracted by ICA have been shown to be similar to the pattern found in the mammalian primary visual system [26]. Secondly, the ICA features have statistically independent activities, consistent with most theories of sensory coding that proposed models to effective internal representation by redundancy [27], The motivation for using ICA for clustering is typically based on the fact the data are linearly transformed by ICA so that the resulting components can be grouped into clusters, such that the components are dependent within cluster and independent between clusters [28]. ICA show to align the context grouping structure well in a human sense [28], thus can be used for clustering. More ICA applications for pattern classification and clustering can see references [18,19,29,30]. 3
Functional M M FMRI is a technique that provides the opportunity to study brain function noninvasively and is a powerful tool utilized in both research and clinical arenas since the early 90s [31]. The most popular technique utilizes blood oxygenation level dependent
370
(BOLD) contrast, which is based on the differing magnetic properties of oxygenated (diamagnetic) and deoxygenated (paramagnetic) blood. When brain neurons are activated, there is a resultant localized change in blood flow and oxygenation which causes a change in the MR decay parameter, T^. These blood flow and oxygenation (vascular or hemodynamic) changes are temporally delayed relative to the neural firing, a confounding factor known as hemodynamic lag. Scientific interest rests primarily with the electrical activity in the neurons, which cannot be directly observed by any variant of the MRI procedure. Since the hemodynamic lag varies in a complex way from tissue to tissue, and because the exact transfer mechanism between the electrical and hemodynamic processes is not known, it is not possible to completely recover the electrical process from the vascular process. Nevertheless, the vascular process remains an informative surrogate for electrical activity. However, relatively low image contrastto-noise ratio (CNR) of the BOLD effect, head movement, and undesired physiological sources of variability (cardiac, pulmonary) make detection of the activation-related signal changes difficult. ICA has shown to be useful for fMRI analysis for several reasons. Spatial ICA finds systematically non-overlapping, temporally coherent brain regions without constraining the temporal domain. The temporal dynamics of many fMRI experiments are difficult to study with functional magnetic resonance imaging (fMRI) due to the lack of a well-understood brain-activation model. ICA can reveal inter-subject and inter-event differences in the temporal dynamics. A strength of ICA is its ability to reveal dynamics for which a temporal model is not available [32]. Spatial ICA also works well for fMRI as it is often the case that one is interested in spatially distributed brain networks. 3.1
Data A cquisition The MRI signal is acquired as a quadrature signal. That is, two orthogonal "detectors" are used to capture the MRI signal [33]. The two outputs from such a system are often put in a complex form, with one output being treated as the real part and the other as the imaginary part. The time-domain data acquired by the spectrometer are, remarkably, equivalent to the spatial-frequency representation of the image data, and so a discrete Fourier transform yields the complex image-space data. It is then common to take the magnitude of this data prior to performing any fMRI analyses. FMRI studies rely upon the detection of small intensity changes over time, often with a contrast-to-noise ratio of less than 1.0. Virtually all fMRI studies analyze the magnitude images from the MRI scanner. A standard approach is to correlate the time-series data with an assumed reference signal [34]. Many generalizations have been proposed, usually involving linear modeling approaches utilizing an estimate of the hemodynamic response [35]. The information contained in the phase images is ignored in such analyses. 3.2
Types of Signal and Noise There are several types of signals that can be encoded within the hemodynamic signals measured by fMRI. Some of these were identified by McKeown in the first application of ICA to fMRI [36]. In this paper, infomax [3] was utilized and separated signals were classified as task-related, transiently task-related, and motion related.
371
In general, fMRI data may be grouped into signals of interest and signals not of interest. The signals of interest include task-related, function-related, and transiently task-related. The task-related signal has already been mentioned and is the easiest to model. A reference waveform, based upon the paradigm, is correlated with the data. The responses of the brain to a given task may not be regular however, for example the signal may die out before the stimulation is turned off or change over time as repeated stimuli are applied, leading to a transiently task-related signal. It is also conceivable that there are several different types of transiently task-related signals coming from different regions of the brain. The function-related signal manifests as similarities between voxels within a particular functional domain {e.g., the motor cortex on one side of the brain will correlate most highly with voxels in the motor cortex on the opposite side of the brain) [37]. An exciting application of this is for identifying synchronous auditory cortex activity [38,39] (see areas corresponding to the top time course in Figure 4.3). Most of these fMRI signals have been examined with ICA and other methods and have been found to be sub-Gaussian in nature (except perhaps the artifacts mentioned in the next section). The signals not of interest include physiology-related, motion-related, and scanner-related signals. Physiology-related signals such as breathing and heart rate tend to come from the brain ventricles (fluid filled regions of the brain) and areas with large blood vessels present, respectively. Motion-related signals can also be present and tend to be changes across large regions of the image (particularly at the edges of images). An example of a motion-related signal occurs during an experiment in which the subjects are mouthing letters, called the rapid automatized naming task [40]. Figure 3.1 depicts an example occurring during the mouthing that was extracted from orbitofrontal and inferior temporal brain regions using infomax algorithm [3]. For comparison, a typical hemodynamic response function is depicted as well. It is clear that the motion is occurring on a time scale too rapid to be related to hemodynamics.
Figure 3.1: Motion-related signal due to mouth movement from inferior temporal and orbitofrontal regions Finally, there are scanner-related signals that can be varying in time (such as scanner drift and system noise) or varying in space (such as susceptibility and radiofrequency artifacts) [41]. A number of such examples including slice dropout, motion artifact, and nyquist ghosting can be found at (http://www.frnrib.ox.ac.uk/ ~beckmann/ homepage/ academic/ littleshop/).
372
There are several types of noise one can characterize in an fMRI experiment. First, there is noise due to the magnetic resonance acquisition which can be discussed as 1) object variability due to quantum thermodynamics and 2) thermal noise. It can be shown that the thermal noise will result in white noise with a constant variance in the image dimension [42]. Additionally there is noise due to patient movement, brain movement, and physiologic noise (such as heart rate, breathing). It has been suggested that physiologic noise is the dominant factor in fMRI studies [43]. In the ICA model these "noises" are often not explicitly modeled, but rather manifested as separate components, (see, e.g., [41,44]). 3.3
Statistical Properties of fMRI Data Properties such as non-Gaussianity and spatial/temporal independence of sources need to be addressed for the application of ICA to fMRI data. If the "activations" do not have a systematic overlap in time and/or space then the distributions can be considered independent [45]. The temporal distribution of a task-related waveform is often nearly bimodal (off/on) and thus the algorithm needs to incorporate this fact. Some other basic assumptions of ICA have been considered in [36]. The assumption that components are spatially independent and add linearly was evaluated and it was concluded that the fMRI signals and noise are non-Gaussian and the accuracy of the ICA model may vary in specific regions of the brain. For example, cortex-based ICA assumes that cortical data are different from non-cortical data and processes a subset of the data determined by a priori information (see section 4.6) [46]. The signals of interest in fMRI are typically focal and thus have a sub-Gaussian spatial distribution. However the artifactual signals will be more varied and potentially super-Gaussian. Many aspects of the fMRI signal are well known and could be incorporated into an ICA analysis. First, local spatial correlation exists in MR images due only to the acquisition process. It is often the case that partial k-space acquisitions involve sampling fewer frequency samples than the desired number of spatial samples. One can use the fact that the matrix of frequency data is Hermitian-symmetric to reconstruct the image using a partially acquired frequency matrix (with the trade-off being a decrease in signal-tonoise-ratio). Another well-known method involves sampling the lower frequencies and padding the high frequencies with zero (with the trade-off being a decrease in spatial resolution). This broadens the well described MRI spatial point spread function in one direction, although it has been suggested that there is a real gain in resolution when zero padding is up to as much as twice the original number of samples [47]. This results in spatial correlation of the MR signal. In addition, spatial correlation is induced by the process being measured. The hemodynamic sources to be estimated have a spatial hemodynamic (vascular) point spread function. This is partially due to the hemodynamics, but is also a function of the pulse sequence and the parameters used. Differing degrees of sensitivity to blood flow and blood oxygenation as well as differences between low and high field magnets will measure different hemodynamics. The pulse sequence, parameters, and magnetic field strength are considered as constant to enable discussion of the hemodynamic point spread function without introducing the complexities of these parameters. There may also be some degree of temporal correlation. Temporal correlation is introduced by: 1) rapid
373
sampling (a scanner parameter) on the time scale of the magnetic equilibrium constant, 7^, 2) the temporal hemodynamic (vascular) point spread function (a physiologic variable), and 3) poorly understood temporal autocorrelations in the data [48]. FMRI provides a non-invasive surrogate measure of the brain's electrical activity. It is a diverse technique and research using fMRI is growing at a rapid pace. The richness of fMRI data is only beginning to be understood. We have provided a brief introduction to the fMRI technique and summarized some of the functionally-related brain signals. It is important to understand the properties of these signals when developing methods for analyzing this data. 4
ICA of fMRI Independent component analysis (ICA) has found a fruitful application in the analysis of functional magnetic resonance imaging (fMRI) data. A principal advantage of this approach is its applicability to cognitive paradigms for which detailed a priori models of brain activity are not available. ICA has been successfully utilized in a number of exciting fMRI applications including the identification of various signal-types (e.g. task and transiently task-related, and physiology-related signals) in the spatial or temporal domain, the analysis of multi-subject fMRI data, the incorporation of a priori information, and for the analysis of complex-valued fMRI data (which has proved challenging for standard approaches). 4.1
Spatial vs. Temporal Independent component analysis is used in fMRI modeling to understand the spatio-temporal structure of the signal. Let the observation data matrix be X, an NxM matrix (where N is the number of time points and M is the number of voxels). The aim of fMRI component analysis is to factor the data matrix into a product of a set of time courses and a set of spatial patterns. In principal component analysis this is achieved by singular value decomposition of the data matrix by which the data matrix is written as the outer product of a set of orthogonal, i.e., un-correlated time courses and set of orthogonal spatial patterns. Independent component analysis takes a more general position and aims at decomposing the data matrix a product of spatial patterns and corresponding time courses where either patterns or time courses are a priori independent. Since the introduction of ICA for fMRI analysis by McKeown et al. [44], the choice of spatial or temporal independency has been controversial. However, the two options are merely two different modeling assumptions. McKeown et al. argued that the sparse distributed nature of the spatial pattern for typical cognitive activation paradigms would work well with spatial ICA (SICA). Furthermore, since the proto-typical confounds are also sparse and localized, e.g., vascular pulsation (signal localized to larger veins that are moving as a result of cardiac pulsation) or breathing induced motion (signal localized to strong tissue contrast near discontinuities: "tissue edges"), the BellSejnowski approach with a sparse prior is very well suited for spatial analysis [49]. Stone et al, proposed a method which attempts to maximize both spatial and temporal independence [50]. An interesting combination of spatial and temporal ICA was pursued by Seifritz et al. [38]; they used an initial SICA to reduce the spatial dimensionality of the data by locating a region of interest in which they then subsequently performed
374
temporal ICA to study in more detail the structure of the non-trivial temporal response in the human auditory cortex. 4.2
A Synthesis/Analysis Model Framework The model shown in Figure 4.1, was introduced in [51] and provides a framework for understanding ICA as applied to fMRI data and for introducing the various processing stages in ICA of fMRI data. The model assumes SICA but can be easily modified for temporal processing. A generative model is assumed for the data including the brain (in a magnet) and the fMRI scanner. Such a model provides a way to monitor the properties of the signals as they propagate through the system and to design the post-processing block, i.e., the analysis stages in a way that matches well with the properties of the source generation mechanisms. The model is also useful for validating ICA results through simulations and hybrid-fMRI data. The data generation block consists of a set of statistically independent (magnetic) hemodynamic source locations in the brain (indicated by si (v) at location v for the *'* source). These sources are a function of magnetic tissue properties such as Tx, T2, r2*, changes in blood flow, changes in blood oxygenation, etc., that are detectable when the brain is placed in a magnetic field. The sources have weights that specify the contribution of each source to each voxel; these weights are multiplied by each source's hemodynamic time course. Finally, it is assumed that each of the N sources are added together so that a given voxel contains a mixture of the sources, each of which fluctuates according to its weighted hemodynamic time course. The first portion of the data generation block takes place within the brain in which the sources are mixed by the matrix A . The second portion of the data generation block involves the fMRI scanner. These sources are sampled ( B ) and represent a function of scan specific MR parameters such as the repeat time (TR), echo time (TE), flip angle, slice thickness, pulse sequence, field-of-view, etc. The data processing block consists of a transformation, T(-), representing a number of possible preprocessing stages, including slice phase correction, motion correction, spatial normalization and smoothing. It is common to perform a data reduction stage ( C ) using PCA or some other approach. The selection of the number of sources is often done manually, but several groups have used information theoretic methods to do order selection [39,52]. The resultant estimated source, s(y'), along with the unmixing matrix A" 1 , can then be thresholded and presented as fMRI activation images and fMRI time courses, respectively. Dala * iuitci.iiion
ytll!,|ts
MM
l->.sUi P]nccssin^
i-Ol'Irfm*™!^
, 1 , , (MlJ **.)«. •)..»
„ , |( 1
Figure 4.1: Model for applying ICA to fMRI data
375
43
Choice ofAlgorithms and Preprocessing As mentioned in the previous subsection, ICA of fMRI involves many preprocessing stages, and there are a number of choices both for those and the ICA algorithms that can be employed. An investigation of how different algorithms and preprocessing stages has been performed by several groups [51,53]. The selection of which algorithm will also depend upon whether one assumes that the sources are sub- or super- Gaussian. In [54], the model described in section 4.2 was utilized to evaluate different preprocessing stages and ICA algorithms using the Kullback-Leibler (KL) divergence between the estimated and "true" distributions. In the case of real fMRI data, validation is difficult as the source distributions are unknown. However, one can move in this direction by superimposing simulated source(s) upon real fMRI data to create a "hybrid" fMRI experiment (see Figure 4.2). Sources are estimated, extracted (by ranking components by their correlation with the known sources) and compared with the actual sources. While this approach is limited, it is useful in providing a quantitative ICA performance measure. Figure 4.2 showing a thresholded "true" source (a) and its mixing function (b). Also shown is a plot of the "hybrid" fMRI data for a voxel close the "true" source maximum (c). The contrast-to-noise level is calculated as the ratio of the source amplitude to the standard deviation (over time) of a voxel within the brain. In general, it is noted that certain choices and combinations make a difference in results. In this work, infomax outperformed (in approximation and variability) fastICA, and PCA outperformed clustering. The best overall combination for this case appears to be infomax and PCA.
Figure 4.2: (a,b,c) Hybrid-fMRI experiment in which a known source is added to a real fMRI experiment (from [32]). (d) Comparison of algorithms and preprocessing using hybrid data
Another approach for comparing algorithms is proposed by Esposito et al m [53]. Linear correlation and receiver operating characteristics are used to compare temporal and spatial outcomes, respectively. The infomax approach appeared to be better suited to investigate activation phenomena that are not predictable or adequately modeled by inferential techniques. 4.4
Group ICA ICA has been successfully utilized to analyze single-subject fMRI data sets, and recently extended for multi-subject analysis [39,55-57]. Unlike univariate methods (e.g., regression analysis, Kolmogorov-Smirnov statistics), ICA does not naturally generalize
376
to a method suitable for drawing inferences about groups of subjects. For example, when using the general linear model, the investigator specifies the regressors of interest, and so drawing inferences about group data comes naturally, since all individuals in the group share the same regressors. In ICA, by contrast, different individuals in the group will have different time courses, and they will be sorted differently, so it is not immediately clear how to draw inferences about group data using ICA. An approach was developed for performing an ICA analysis on a group of subjects [58] which extends the synthesis/analysis model mentioned in 4.2. In order to reduce computational load, data reduction was first performed for each subject's data then a second, aggregate model order reduction was performed. Back-reconstruction and statistical comparison of individual maps is performed following the ICA estimation. This approach is implemented in a Matlab toolbox [59]. Group maps for the fMRI ICA analyses are presented in Figure 4.3. The number of components is estimated to be twenty-one by the two information-theoretic criteria employed: the minimum description length and Akaike's information criterion. Thus, the aggregate data are reduced to this dimension and twenty-one components were estimated. Both maps are thresholded at /><0.001 (/=4.5, df =8). Several interesting components were identified within the data. Separate components for primary visual areas on the left and the right visual cortex (depicted in red and blue, respectively) were consistently taskrelated with respect to the appropriate stimulus. A large region (depicted in green) including occipital areas and extending into parietal areas appeared to be sensitive to changes in the visual stimuli. Additionally some visual association areas (depicted in white) had time courses which were not task related. A comparison of group ICA approaches is found in [60].
Figure 4.3: fMRI Group ICA results (from [39]) 4.5
Multiple Groups In many fMRI experiments, it is desirable to directly compare and contrast two different conditions either within or between subject groups. Methods for performing such comparisons have been developed within the framework of the general linear model; however such comparisons are not intuitive for ICA. Comparisons of two ICA groups can be problematic because the ICA results represent a comparison of two different linear
377
models with different time courses. A method for performing subtractive and conjunctive comparisons of group ICA data is proposed in [61]. An alternative method is given in [62]. One solution involves extracting components of interest using ana priori spatial or temporal template and quantifying whether the components extracted from the two groups have sufficiently unique time courses from the remaining components. An analysis of seven participants performing a three paradigms (visual, motor, and visuomotor experiment) can then be compared between paradigms as seen in Figure 4.4. a Visual - Visuomotor
b Motor - Visuomotor
900 000 *
*
*
*
*
*
*
*
*
*
Figure 4.4: Visual, motor, and visuomotor data comparisons Additional parameterizations are also possible. For example in this study onset latencies were estimated using a weighted least squares technique [63]. A small, but significant latency difference was observed between the onset of visual and motor activation. Group Onset Latencies Visual Experiment Motor Experiment Difference
Left Cortex 3.46±0.33s 3.91±O.30s .450±0.18s (p<0.05, df=7)
Right Cortex 3.19±0.28s 4.43±0.32s 1.24±0.53s (p<0.05, df=7)
Such an approach allows comparisons of both brain activation and time course parameters across paradigms for the flexible modeling approach, ICA. 4.6
Incorporation of a priori Information The incorporation of prior information into ICA methods is important as it can provide improved separability and allow selective exploratory analysis. In addition, ICA methods make assumptions about, e.g. the distributional shape of the sources, and thus it is important to both assess the impact of such assumptions and modify them based upon fMRI data. There have been a number of applications of ICA that have attempted to utilize prior information for fMRI analysis. For example, using a reference function to extract only a single component is proposed in [64]. Stone et al. propose a skewed symmetric nonlinearity (i.e., assume that the source distributions are skewed). This makes sense if one is interested in components that consist largely of either activations or deactivations [65]. Formisano et al. propose performing ICA upon data extracted from the cortex (where the activation is expected to be occurring) using a tessellation model of the brain cortex derived from a high resolution structural image [46]. Duann et al. examine timelocked temporal structure and propose a visualization approach to evaluate trial-by-trial
378
variability [66]. Bayesian methods provide a useful way to incorporate prior information into ICA and may prove useful for FMRI analysis [67]. 4.6.I
Spatial Prior Information The successful applications of ICA to fMRI signals are dependent on the validity of the statistical assumptions implicit in the method. In fMRI data the physiological signals of interest are governed by a small percentage of the whole brain map, whereas the majority of brain regions are governed by less significant homogeneous background with signal of non-interest in the task-related activation maps. While conventional ICA approaches, including a fixed nonlinear function-based algorithm or a high-order cumulant-based algorithm, work well in highly kurtotic or super-Gaussian and subGaussian sources, they might provide poor results with fMRI data due to lack sensitivity to such specifically skewed distributions and low kurtotic signals. Hong & Calhoun propose a source density-driven optimal ICA method, aiming to embody physically realistic assumptions and thus achieve optimal separation results [68]. The central idea is to use a two-stage separation process: 1) Conventional ICA used for all channel sources to obtain initial independent source estimates; 2) source estimate-based "optimal" nonlinearities used for the "refitting" separation to all channel sources. The optimal ICA algorithm is not based on fixed nonlinear functions, but on flexible nonlinearities of density matched candidates. The performance of ICA can be improved by seeking "matched" nonlinearities for each source and incorporating prior information into the ICA algorithm. Figure 4.5 shows the comparison between optimal ICA and Infomax ICA on the fMRI visual stimulation data of a single subject.
(a) Optimal ICA (b) Infomax ICA Figure 4.5: Comparison between Optimal ICA and Infomax ICA 4.6.2
Temporal Prior Information It is also useful to impose constrains directly upon the mixing matrix in a spatial ICA fMRI analysis. For example, a component selective constraint of the ICA model mixing matrix such that one or more specific components are constrained to be "close" to a paradigm-derived time course is shown in Figure 4.6. The degree of closeness is specified by the user based upon amount of confidence placed in the information provided. Results from this approach are shown below for an fMRI experiment for an auditory detection task. The participant was responding to the target with a button press.
379
Figure 4.6: Comparison of ICA and sblCA in one participant The left side of the figure demonstrates the task-related component for an unconstrained ICA analysis. On the right is the constrained (or semi-blind) analysis showing additional regions including motor cortex towards the top of the figure. The temporal correlation with the paradigm is much higher for the constrained analysis as expected. These results demonstrate the utility of incorporating mixing matrix constraints in an fMRI analysis. 4.7
Complex Images Functional magnetic resonance imaging (fMRI) is a technique that produces complex-valued data; however the vast majority of fMRI analyses utilize only magnitude images despite the fact that the phase information has a straightforward physiologic interpretation [69]. A number of ICA algorithms are extended to the complex domain and can be utilized for processing the fMRI data in its native complex domain. The performance of the complex infomax algorithm that uses an analytic (and hence unbounded) nonlinearity with the traditional complex infomax approaches that employ bounded (and hence non-analytic) nonlinearities as well as with a cumulant-based approach has been studied [70]. Results from a magnitude-only and complex-valued analysis are presented in Figure 4.7. The complex-valued approach results in a larger contiguously activated region in all subjects (from [71]). In addition, the phase information is captured by the complex-valued approach (c,d).
380
Figure 4.7: Results from magnitude-only analysis (a) and complex-valued analysis (b,c,d)
5
Conclusion
The application of ICA to fMRI data has proved to be quite fruitful. However there is still much work to be done in order to take full advantage of the information contained in the data. Additional prior information about multiple expected sources (both interesting and non-interesting) and their properties (fMRI properties, physiologic recording, etc) can be utilized. In addition to incorporating appropriate assumptions (and moving towards a semi-blind source separation) it is important to relax inappropriate assumptions (such as having a fixed temporal delay for each source). One of the strengths of ICA of fMRI is its ability to characterize the high-dimensional data in a concise manner. Continuing to do this and developing ways to mine the unexpected information in fMRI data will provide an exciting future for ICA of fMRI. 6
References
[1] A.Jung, "An introduction to a new data analysis tool: Independent Component Analysis," 2001. [2] A.Hyvarinen, "Survey on Independent Componenet Analysis," Neural Computing Surveys, vol. 2, pp. 94-128, 1999. [3] A.J.Bell and T.J.Sejnowski, "An Information Maximisation Approach to Blind Separation and Blind Deconvolution," Neural Comput., vol. 7, pp. 1129-1159, 1995.
381 [4] S.Amari and J.F.Cardoso, "Blind Source Separation-Semiparametric Statistical Approach," IEEE Trans. Acous. Speech, andSig. Proc, vol. 45, pp. 2692-2700, 1997. [5] J.P.Nadal and N.Parga, "Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches," Neural Computation, vol. 9, pp. 14211456, 1997. [6] B.Pearlmutter and L.Parra, Maximum likelihood blind source separation: A context-sensitive generation of ICA. In: Advances in Neural Information Processing Systems, eds. M.C.Mozer, M.I.Jordan, and T.Petsche. 1997,pp. 613-619. [7] H.H.Yang and S.I.Amari, "Adaptive on-Line Learning Algorithms for Blind SeparatiomMaximum Entropy and Minimum Mutual Information," Neural Computation, vol. 9, pp. 1457-1482, 1997. [8] D.Obradovic and G.Deco, "Information Maximization and Independent Component Analysis: Is There a Difference?," Neural Computation, vol. 10, pp. 2085-2101, 1998. [9] A.Hyvarinen and E.Oja, "A Fast Fixed-Point Algorithm for Independent Component Analysis," Neural Comput., vol. 9, pp. 1483-1492, 1997. [10] J.F.Cardoso and A.Souloumiac, "Blind Beamforming for Non Gaussian Signals," IEEProceeding-F, vol. 140, pp. 362-370, 1993. [11] T.W.Lee, et al., "Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources," Neural Comput., vol. 11, pp. 417-441, 1999. [12] S.Choi, et al., "Flexible Independent Component Analysis," Journal of VLSI Signal Processing, vol. 26, pp. 25-38, 2000. [13] R.H.Boscolo, et al., "Non-parametric ICA.," in Proc. Int. Conf. on ICA and BSS, 2001. [14] Bach F. and J.M., "Kernel Independent Component Analysis," Ournal of Machine Learning Research, vol. 3, pp. 1-48, 2002. [15] N.Vlassis and Y.Motomura, "Efficient Source Adaptivity in Independent in Independent Component Analysis," IEEE Trans. Neural Networks, vol. 12, pp. 559-566, 2001. [16] B.Hong, et al, "Identification of Brain Activity in a Visual Stimulation Task - An Adaptive ICA Approach for fMRI Data," in Proc. HBM, 2004. [17] P.O.Hoyer and A.Hyvarinen, "Independent Component Analysis Applied to Feature Extraction From Color and Stereo Images," Computation in Neural Systems, vol. 11, pp. 191210,2000. [18] M.S.Bartlett, et al., "Face Recognition by Independent Component Analysis," IEEE Trans, on Neural Networks, vol. 13, pp. 1450-1464,2002. [19] B.A.Draper, et al, "Recognizing Faces With PC A and ICA," Computer Vision and Image Understanding, vol. 91, pp. 115-137, 2003. [20] A.Hyvarinen, "New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit," Advances in Neural Inf. Proc. Sys., vol. 10, pp. 273-279, 1998. [21] M.Kotani, et al., "Application of independent component analysis to feature extraction of speech," 1999. [22] R.O.Duda, Pattern Classification, John Wiley & Sons, Inc., 2001. [23] R.L.Mason, R.F.Gunst, and J.L.Hess, Statistical Design and Analysis of Experiments, With Applications to Engineering and Science, John Wiley & Sons, Inc., 2003. [24] N.Cristianini and J.Shawe_Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press, 2000.
382 [25] M.Bressan, et al, "Using an ICA representation of high dimensional data for object recognition and classification," 2001. [26] J.H.van Hateren and S.A.van der, "Independent Component Filters of Natural Images Compared With Simple Cells in Primary Visual Cortex," Proc. R. Soc. Lond. B Biol. Sci., vol. 265, pp. 359-366, 1998. [27] H.L.Borgne, et al., "Classification of images: ICA filters vs. human perception," vol. 2, pp. 251-254, 2003. [28] F.R.Bach and M.Jordan, "Beyond Independent Components: Trees and Clusters," Journal of Machine Learning Research, vol. 4, pp. 1205-1233, 2003. [29] C.Liu and H.Wechsler, "Independent Component Analysis of Gabor Features for Face Recognition," IEEE Trans, on Neural Networks, vol. 14, pp. 919-928, 2003. [30] C.A.Shah, et al., "ICA Mixture Model based Unsupervised Classification of Hyperspectral Imagery," 2002. [31] K.K.Kwong, et al, "Dynamic Magnetic Resonance Imaging of Human Brain Activity During Primary Sensory Stimulation," Proc. Natl. Acad. Sci., vol. 89, pp. 5675-5679, 1992. [32] V.D.Calhoun, etal, "Different Activation Dynamics in Multiple Neural Systems During Simulated Driving," Hum. Brain Map., vol. 16, pp. 158-167, 2002. [33] D.I.Hoult, et al, "Quadrature Detection in the Laboratory Frame," Magn. Res. Med., vol. 1, pp. 339-353, 1984. [34] P.A.Bandettini, et al, "Processing Strategies for Time-Course Data Sets in Functional MRI of the Human Brain," Magn. Res. Med., vol. 30, pp. 161-173, 1993. [35] K.J.Worsley and K.J.Friston, "Analysis of FMRJ Time-Series Revisited—Again," Neurolmage, vol. 2, pp. 173-181, 1995. [36] M.J.McKeown and T.J.Sejnowski, "Independent Component Analysis of FMRI Data: Examining the Assumptions," Hum. Brain Map., vol. 6, pp. 368-372, 1998. [37] B.Biswal, et al, "Functional Connectivity in the Motor Cortex of Resting Human Brain Using Echo-Planar MRI," Magn. Res. Med., vol. 34, pp. 537-541, 1995. [38] E.Seifritz, et al, "Spatiotemporal Pattern of Neural Processing in the Human Auditory Cortex," Science, vol. 297, pp. 1706-1708, 2002. [39] V.D.Calhoun, et al, "A Method for Making Group Inferences From Functional MRI Data Using Independent Component Analysis," Hum. Brain Map., vol. 14, pp. 140-151, 2001. [40] M.B.Denckla, "Performance on Color Tasks in Kindergarten Children," Cortex, vol. 8, pp. 177-190, 1972. [41] C.F.Beckmann, et al, "Artefact Detection in fMRI Data Using Independent Component Analysis," in Neurolmage, vol. 11, no. 5, p. S614, 2000. [42] Y.Wang, MR Image Statistics and Model-Based MR Image Analysis PhD. Dissertation, UMBC, 1995. [43] G.Kruger and G.H.Glover, "Physiological Noise in Oxygenation-Sensitive Magnetic Resonance Imaging," Magn Reson. Med., vol. 46, pp. 631-637, 2001. [44] M.J.McKeown, et al, "Analysis of FMRI Data by Blind Separation Into Independent Spatial Components," Hum. Brain Map., vol. 6, pp. 160-188, 1998. [45] V.D.Calhoun, et al, "Spatial and Temporal Independent Component Analysis of Functional MRI Data Containing a Pair of Task-Related Waveforms," Hum. Brain Map., vol. 13, pp. 4353,2001.
383 [46] E.Formisano, et al, "Cortex-based Independent Component Analysis," in Neurolmage, vol. 13, no. 6, p. S199, 2001. [47] R.T.Constable, et al, "High Quality Zoomed MR Images," J. Comput. Assist. Tomogr., vol. 13, pp. 179-181, 1989. [48] E.Zarahn, et al, "Empirical Analyses of BOLD FMRI Statistics. I. Spatially Unsmoothed Data Collected Under Null-Hypothesis Conditions," Neurolmage, vol. 5, pp. 179-197, 1997. [49] K.S.Petersen, et al, "On the Independent Components of Functional Neuroimages," in Proc. Int. Conf. on ICA andBSS, Helsinki, Finland, pp. 615-620, 2000. [50] J.V.Stone, et al, "Spatial, Temporal, and Spatiotemporal Independent Component Analysis of fMRI Data," in Proc. Leeds Statistical Research Workshop, Leeds, England, 1999. [51] V.D.Calhoun, T.Adali, and G.D.Pearlson, "Independent Components Analysis Applied to FMRI Data: A Generative Model for Validating Results," to appear Journal of VLSI Signal Proc. Systems, 2004. [52] C.F.Beckmann, et al, "Investigating the Intrinsic Dimensionality of fMRI Data for ICA," in Neurolmage, vol. 13, no. 6, p. S76, 2001. [53] F.Esposito, et al, "Spatial Independent Component Analysis of Functional MRI Time-Series: to What Extent Do Results Depend on the Algorithm Used?," Hum. Brain Map., vol. 16, pp. 149-157,2002. [54] V.D.Calhoun, et al, "Independent Component Analysis Facilitates fMRI of an Naturalistic Behavior: Hypothesized Neural Substrates of Simulated Driving," in Proc. ISMRM, Honolulu, HI, 2002. [55] V.D.Calhoun, et al, "FMRI Activation In A Visual-Perception Task: Network Of Areas Detected Using The General Linear Model And Independent Components Analysis," Neurolmage, vol. 14, pp. 1080-1088, 2001. [56] M.Svensen, et al, "ICA of FMRI Group Study Data," Neurolmage, vol. 16, pp. 551-563, 2002. [57] D.G.Leibovici and C.F.Beckmann, "An Introduction to Multiway Methods for Multi-Subject FMRI," FMRIB Technical Report (TR01DL1), 2001. [58] V.D.Calhoun, et al, "A Method for Making Group Inferences Using Independent Component Analysis of Functional MRI Data: Exploring the Visual System," in Neurolmage, vol. 13, no. 6, p. S88,2001. [59] E.Egolf, et al, "Group ICA of fMRI Toolbox (GIFT)," in Proc. HBM, Budapest, Hungary, 2004. [60] V.J.Schmithorst and S.K.Holland, "Comparison of Three Methods for Generating Group Statistical Inferences From Independent Component Analysis of Functional Magnetic Resonance Imaging Data," J. Magn Reson. Imaging, vol. 19, pp. 365-368, 2004. [61] V.D.Calhoun, et al, "A Method for Testing Conjunctive and Subtractive Hypotheses on Group fMRI Data Using Independent Component Analysis," in Proc. ISMRM, Toronto, Canada, 2003. [62] A.S.Lukic, et al, "An ICA Algorithm For Analyzing Multiple Data Sets," in Int. Conf. on Image Processing (ICIP), Rochester, NY, 2002. [63] V.D.Calhoun, et al, "A Weighted-Least Squares Algorithm for Estimation and Visualization of Relative Latencies in Event-Related Functional MRI," Magn. Res. Med., vol. 44, pp. 947954, 2000.
384 [64] W.Lu and J.C.Rajapakse, "ICA With Reference," in Proc. Int. Conf. on ICA and BSS, San Diego, CA, pp. 120-125, 2001. [65] J.V.Stone, et al., "Spatiotemporal Independent Component Analysis of Event-Related FMRI Data Using Skewed Probability Density Functions," Neuroimage., vol. 15, pp. 407-421, 2002. [66] J.R.Duann, et al., "Single-Trial Variability in Event-Related BOLD Signals," Neuroimage., vol. 15, pp. 823-835,2002. [67] R.A.Choudrey and S.J.Roberts, "Flexible Bayesian Independent Component Analysis for Blind Source Separation," in Proc. Int. Conf. on ICA and BSS, San Diego, CA, 2001. [68] B.Hong and V.D.Calhoun, "Source Density Driven Adaptive Independent Component Analysis Approach for fMRI Signal Analysis," in Proc. MLSP, 2004. [69] V.D.Calhoun, et al., "Independent Component Analysis of FMRI Data in the Complex Domain," Magn Reson. Med., vol. 48, pp. 180-192, 2002. [70] V.D.Calhoun and T.Adali, "Complex ICA for fMRI Analysis: Performance of Several Approaches," in Proc. ICASSP, Hong Kong, China, 2003. [71] V.D.Calhoun, et al., "On Complex Infomax Applied to Complex fMRI Data," in Proc. ICASSP, Orlando, FL, 2002.
C H A P T E R 4.1
MULTIMODAL E M O T I O N R E C O G N I T I O N Nicu Sebe , Ira Cohen , and Thomas S. Huang 3 Faculty of Science, University of Amsterdam, The Netherlands E-mail: [email protected] 2
HP Labs, USA E-mail: [email protected] Beckman Institute,
University of Illinois at Urbana- Champaign, USA E-mail: [email protected]
Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. Despite important advances, one necessary ingredient for natural interaction is still missing-emotions. Emotions play an important role in human-to-human communication and interaction, allowing people to express themselves beyond the verbal domain. The ability to understand human emotions is desirable for the computer in several applications. This chapter explores new ways of human-computer interaction that enable the computer to be more aware of the user's emotional and attentional expressions. We present the basic research in the field and the recent advances into the emotion recognition from facial, voice, and pshysiological signals, where the diiferent modalities are treated independently. We then describe the challenging problem of multimodal emotion recognition and we advocate the use of probabilistic graphical models when fusing the different modalities. We also discuss the difficult issues of obtaining reliable affective data, obtaining ground truth for emotion recognition, and the use of unlabeled data. 1. I n t r o d u c t i o n Maybe no movie of modern time has explored the definition of what it means to be h u m a n better t h a n Blade Runner. T h e Tyrell Corporation's motto, "More h u m a n t h a n h u m a n " , serves as the basis for exploring the human experience through true humans and created humans, or Replicants. Replicants are androids t h a t were built to look like humans and t o work or fight their wars. In time, t h e y began t o acquire emotions (so much like humans) and it became difficult to tell t h e m apart. W i t h emotions, they began to feel oppressed and many of t h e m became dangerous and committed acts of extreme violence t o be free. Fortunately, Dr. Elden Tyrell, the creator of the Replicants, installed a built-in safety feature in these models: a fouryear life span. 387
388
It is evident from the above story that it is not sufficient for a machine (computer) to look like a human (e.g., have skin, face and facial features, limbs, etc). Something else is also essential: the ability to acquire and show the emotions. Moreover, the machine should learn to recognize faces and to understand the emotions to be able to have a human-like interaction with its human counterpart. Machines may never need all of the emotional skills that people need but they will inevitably require some of these skills to appear intelligent when interacting with people. It is argued that to truly achieve effective human-computer intelligent interaction (HCII), there is a need for the computer to be able to interact naturally with the user, similar to the way human-human interaction takes place. For example, if a machine talks to you but never listens to you, then it is likely to be annoying, analogous to the situation where a human talks to you but never listens. Reeves and Nass 55 have conducted several experiments of classical human-human interaction, taking out one of the humans and putting in a computer. Their conclusion is that for an intelligent interaction, the basic human-human issues should hold. Humans interact with each other mainly through speech, but also through body gestures, to emphasize a certain part of the speech and display of emotions. As a consequence, the new interface technologies are steadily driving toward accommodating information exchanges via the natural sensory modes of sight, sound, and touch. In face-to-face exchange, humans employ these communication paths simultaneously and in combination, using one to complement and enhance another. The exchanged information is largely encapsulated in this natural, multimodal format. Typically, conversational interaction bears a central burden in human communication, with vision, gaze, expression, and manual gesture often contributing critically, as well as frequently embellishing attributes such as emotion, mood, attitude, and attentiveness. But the roles of multiple modalities and their interplay remain to be quantified and scientifically understood. What is needed is a science of humancomputer communication that establishes a framework for multimodal "language" and "dialog", much like the framework we have evolved for spoken exchange. In some applications, it may not be necessary for computers to recognize emotions. For example, the computer inside an automatic teller machine or an airplane probably does not need to recognize emotions. However, in applications where computers take on a social role such as an "instructor," "helper," or even "companion," it may enhance their functionality to be able to recognize users' emotions. In her recent book, Picard 52 suggested several applications where it is beneficial for computers to recognize human emotions. For example, knowing the user's emotions, the computer can become a more effective tutor. Synthetic speech with emotions in the voice would sound more pleasing than a monotonous voice. Computer "agents" could learn the user's preferences through the users' emotions. Another application is to help the human users monitor their stress level. In clinical settings, recognizing a person's inability to expression certain facial expressions may help diagnose early psychological disorders.
389
Speech Recognition Vocal Affect MICROPHONE
Head Movement VIDEO CAMERA
COMPUTER Facial Expression
Gesture Recognition Eye Contact
-*
Body Posture
ANIMATED AGENT
SYNTHETIC SPEECH
Fig. 1.
Multimodal human-computer interaction.
Psychologists and engineers alike have tried to analyze facial expressions, vocal emotions, gestures, and physiological signals in an attempt to understand and categorize emotions. This knowledge can be used to teach computers to recognize human emotions from video images acquired from built-in cameras, and from speech waveforms gathered from on-board microphones. A natural two-way interaction between the human and the computer through multiple modalities is depicted in Figure 1. In this diagram, one of the inputs to the computer is vision (video), from which gaze, posture, gestures, and facial and lip movements can be extracted. Computers may learn to recognize gestures, postures, facial expressions, eye contact, etc. Likewise, speech and voice (audio) through the microphone may convey linguistic as well as paralinguistic information. On the output side, the computer may appear in the form of an "agent"—a computer-animated face or a personified animated character. This "agent" can speak to the human through a synthesized speech and display corresponding facial and mouth movements on the screen. Even if they are not explicitly presented in the figure, some other modalities such as tactile or physiological signals can also be used in conjunction with the video and audio signals. The goal of this chapter is to explore new ways of human-computer interaction by enabling the computer to be more aware of the human user's emotional and attentional expressions. In particular, we concentrate on the problem of integrating audiovisual inputs for the detection the users' facial and vocal emotional expressions and attentive states. By "emotional expression" we mean any outward expression that arises as a response to some stimulus event. These may include typical expressions such as a "smile" to show that one is happy, or to show one likes what one sees.
390
We begin by describing the basic research into the problems of what are emotions, their importance in human-human interaction, and how emotions are expressed by humans (Section 2). Some of this basic research lays the foundation to automatic emotion recognition by computers, and enables the computer science and engineering community to treat the problem as a pattern recognition one. Next, we review the advances in the area of emotion expression recognition from facial, voice, and physiological signals, where the different modalities are treated independently (Section 3). We then describe research into emotion expression recognition from face and voice signals, advocating the use of probabilistic graphical models when fusing the different modalities for achieving true multi-modal emotion expression recognition. We also discuss the difficult problems of gathering good affective data, obtaining ground truth for emotion recognition, and the use of unlabeled data (Section 4). Throughout the chapter we explore and try to provide answers to the following questions: • What clues are there on the face and in the voice that reveal a person's emotions, preferences, and attentional states? • How well can we use these clues to train the computer to recognize human emotion from audio and from video? • Does the use of joint audiovisual input allow for more accurate or efficient emotion recognition than using a single modality? • In realistic scenarios, can the two modalities be treated separately? • How to collect multimodal data with emotional expressions and how should it be labeled? • Can we get away with labeling only small amounts of data and use unlabeled data to help in training the models that recognize emotional expressions? • What data should be collected? Spontaneous or posed data? 2. Human Emotion Research There is a vast body of literature on emotions. The multifaceted nature prevents a comprehensive review, we will review only what is essential in supporting this work. Recent discoveries suggest that emotions are intricately linked to other functions such as attention, perception, memory, decision making, and learning. This suggests that it may be beneficial for computers to recognize the human user's emotions and other related cognitive states and expressions. In this chapter, we concentrate on the expressive nature of emotion, especially those expressed in the voice and on the face. 2.1. Affective
human-computer
interaction
In many important HCI applications such as computer aided tutoring and learning, it is highly desirable (even mandatory) that the response of the computer takes into
391 account the emotional or cognitive state of the human user. Emotions are displayed by visual, vocal, and other physiological means. There is a growing amount of evidence showing that emotional skills are part of what is called "intelligence" 58 ' 25 . Computers today can recognize much of what is said, and to some extent, who said it. But, they are almost completely in the dark when it comes to how things are said, the affective channel of information. This is true not only in speech, but also in visual communications despite the fact that facial expressions, posture, and gesture communicate some of the most critical information: how people feel. Affective communication explicitly considers how emotions can be recognized and expressed during human-computer interaction. Addressing the problem of affective communication, Bianchi-Berthouze and Lisetti 2 identified three key points to be considered when developing systems that capture affective information: embodiment (experiencing physical reality), dynamics (mapping experience and emotional state with its label), and adaptive interaction (conveying emotive response, responding to a recognized emotional state). In most cases today, if you take a human-human interaction, and replace one of the humans with a computer, then the affective communication vanishes. Furthermore, this is not because people stop communicating affect - certainly we have all seen a person expressing anger at his machine. The problem arises because the computer has no ability to recognize if the human is pleased, annoyed, interested, or bored. Note that if a human ignored this information, and continued babbling long after we had yawned, we would not consider that person very intelligent. Recognition of emotion is a key component of intelligence52. Computers are presently affect-impaired. Furthermore, if you insert a computer (as a channel of communication) between two or more humans, then the affective bandwidth may be greatly reduced. Email may be the most frequently used means of electronic communication, but typically all of the emotional information is lost when our thoughts are converted to the digital media. Research is therefore needed for new ways to communicate affect through computer-mediated environments. Computer-mediated communication today almost always has less affective bandwidth than "being there, face-to-face". The advent of affective wearable computers, which could help amplify affective information as perceived from a person's physiological state, are but one possibility for changing the nature of communication. 2.2. Theories
of
emotion
There is little agreement about a definition of emotion. Many theories of emotion have been proposed. Some of these could not be verified until recently when measurement of some physiological signals become available. In general, emotions are short-term, whereas moods are long-term, and temperaments or personalities are very long-term 29 . A particular mood may sustain for several days, and temperament for months or years. Finally, emotional disorders can be so disabling that people
392
affected are no longer able to lead normal lives. Darwin 14 held an ethological view of emotional expressions, arguing that the expressions from infancy and lower life forms exist in adult humans. Following the Origin of Species he wrote The Expression of the Emotions in Man and Animals. According to him, emotional expressions are closely related to survival. Thus in human interactions, these nonverbal expressions are as important as the verbal interaction. James 28 viewed emotions not as causes but as effects. Situations arise around us which cause changes in physiological signals. According to James, "the bodily changes follow directly the perception of the exciting fact, and that our feeling of the same changes as they occur is the emotion." Carl Lange proposed a similar theory independently at around the same time. This is often referred to as the "JamesLange" theory of emotion. Cannon 5 , contrary to James, believed that emotions are first felt, then exhibited outwardly causing certain behaviors. Despite the many theories, it is evident that people display these expressions to various degrees. One frequently studied task is the judgment of emotions—how well can human observers tell the emotional expressions of others, in the voice, on the face, etc? Related questions are: Do these represent their true emotions? Can they be convincingly portrayed? How well can people conceal their emotions? In such tasks, researchers often use two different methods to describe the emotions. One approach is to label the emotions in discrete categories, i.e., human judges must choose from a prescribed list of word labels, such as joy, fear, love, surprise, sadness, etc. One problem with this approach is that the stimuli may contain blended emotions. Also, the choice of words may be too restrictive, or culturally dependent. Another way is to have multiple dimensions or scales to describe emotions. Instead of choosing discrete labels, observers can indicate their impression of each stimulus on several continuous scales, for example, pleasant-unpleasant, attentionrejection, simple-complicated, etc. Two common scales are valence and arousal. Valence describes the pleasantness of the stimuli, with positive (or pleasant) on one end, and negative (or unpleasant) on the other. For example, happiness has a positive valence, while disgust has a negative valence. The other dimension is arousal or activation. For example, sadness has low arousal, whereas surprise has high arousal level. The different emotional labels could be plotted at various positions on a two-dimensional plane spanned by these two axes to construct a 2D emotion model 31 . Scholsberg62 suggested a three-dimensional model in which he had attention-rejection in addition to the above two. Another interesting topic is how the researchers managed to obtain data for observation. Some people used posers, including professional actors and non-actors. Others attempted to induce emotional reactions by some clever means. For example, Ekman showed stress-inducing film of nasal surgery in order to get the disgusted look on the viewers' faces. Some experimenter even dumped water on the subjects
393
or fired blank shots to induce surprise, while others used clumsy technicians who made rude remarks to arouse fear and anger 26 . Obviously, some of these are not practical ways of acquiring data. After studying acted and natural expressions, Ekman concluded that expressions can be convincingly portrayed 17 . A legitimate question that should be considered when doing multimodal emotion recognition is how much information does the face, as compared to voice, speech, and body movement, provide about emotion. Most experimenters found that the face is more accurately judged, produces higher agreement, or correlates better with judgments based on full audiovisual input than on voice or speech input 38 ' 17 . Ekman 17 found that the relative weight given to facial expression, speech, and body cues depend both on the judgment task (e.g., rating the stimulus subject's dominance, sociability, or relaxation) and the conditions in which the behavior occurred (e.g., subjects frankly describing positive reactions to a pleasant film or trying to conceal negative feelings aroused by a stressful film). The whole question of how much information is conveyed by "separate" channels may inevitably be misleading. There is no evidence that individuals in actual social interaction selectively attend to another person's face, body, voice, or speech or that the information conveyed by these channels is simply additive. The central mechanisms directing behavior cut across the channels, so that, for example, certain aspects of face, body, voice, and speech are more spontaneous and others are more closely monitored and controlled. It might well be that observers selectively attend not to a particular channel but to a particular type of information (e.g., cues to emotion, deception, or cognitive activity), which may be available within several channels. No investigator has yet explored this possibility or the possibility that different individuals may typically attend to different types of information.
3. Emotional Expression Recognition for Human-Computer Interaction The mounting evidence of the importance of emotions in human-human interaction provided the basis for researchers in the engineering and computer science communities to develop automatic ways for computers to recognize emotional expression, as a goal towards achieving human-computer intelligent interaction. The labeling of emotions into different states led most research to use pattern recognition approaches for recognizing emotions, using different modalities as inputs to the emotion recognition models. Next we review some of these works. 3.1. Facial expression
recognition
studies
Since the early 1970s, Paul Ekman and his colleagues have performed extensive studies of human facial expressions 18 . They found evidence to support universality in facial expressions. These "universal facial expressions" are those representing happiness, sadness, anger, fear, surprise, and disgust. They studied facial expressions
394 in different cultures, including preliterate cultures, and found much commonality in the expression and recognition of emotions on the face. However, they observed differences in expressions as well, and proposed that facial expressions are governed by "display rules" in different social contexts. For example, Japanese subjects and American subjects showed similar facial expressions while viewing the same stimulus film. However, in the presence of authorities, the Japanese viewers were more reluctant to show their real expressions. Matsumoto 36 reported the discovery of a seventh universal facial expression: contempt. Babies seem to exhibit a wide range of facial expressions without being taught, thus suggesting that these expressions are innate 27 . Ekman and Friesen19 developed the Facial Action Coding System (FACS) to code facial expressions where movements on the face are described by a set of action units (AUs). Each AU has some related muscular basis. Each facial expression may be described by a combination of AUs. This system of coding facial expressions is done manually by following a set prescribed rules. The inputs are still images of facial expressions, often at the peak of the expression. This process is very timeconsuming. Ekman's work inspired many researchers to analyze facial expressions by means of image and video processing. By tracking facial features and measuring the amount of facial movement, they attempt to categorize different facial expressions. Recent work on facial expression analysis and recognition 35>65>32>3>56.20,44,33,42,34,43,12,6,10 has used these "basic expressions" or a subset of them. The recent surveys in the area 21 ' 47 ' 48 provide an in-depth review of many of the research done in automatic facial expression recognition in recent years. Recent work in computer-assisted quantification of facial expressions did not start until the 1990s. Mase 35 used optical flow (OF) to recognize facial expressions. He was one of the first to use image processing techniques to recognize facial expressions. Lanitis et al. 32 used a flexible shape and appearance model for image coding, person identification, pose recovery, gender recognition and facial expression recognition. Black and Yacoob3 used local parameterized models of image motion to recover non-rigid motion. Once recovered, these parameters are fed to a rule-based classifier to recognize the six basic facial expressions. Yacoob and Davis 68 computed optical flow and used similar rules to classify the six facial expressions. Rosenblum et al. 56 also computed optical flow of regions on the face, then applied a radial basis function network to classify expressions. Essa and Pentland 20 also used an optical flow region-based method to recognize expressions. Otsuka and Ohya 44 first computed optical flow, then computed their 2D Fourier transform coefficients, which were then used as feature vectors for a hidden Markov model (HMM) to classify expressions. The trained system was able to recognize one of the six expressions near realtime (about 10 Hz). Furthermore, they used the tracked motions to control the facial expression of an animated Kabuki system 45 . A similar approach, using different features was used by Lien 33 . Nefian and Hayes 42 proposed an embedded
395
HMM approach for face recognition that uses an efficient set of observation vectors based on the DCT coefficients. Martinez 34 introduced an indexing approach based on the identification of frontal face images under different illumination conditions, facial expressions, and occlusions. A Bayesian approach was used to find the best match between the local observations and the learned local features model and an HMM was employed to achieve good recognition even when the new conditions did not correspond to the conditions previously encountered during the learning phase. Oliver et al. 43 used lower face tracking to extract mouth shape features and used them as inputs to an HMM based facial expression recognition system (recognizing neutral, happy, sad, and an open mouth). Chen 6 used a suite of static classifiers to recognize facial expressions, reporting on both person-dependent and person-independent results. Cohen et al. 12 describe classification schemes for facial expression recognition in two types of settings: dynamic and static classification. The static classifiers classify a frame in a video to one of the facial expression categories based on the tracking results of that frame. In this setting, the authors learn the structure of Bayesian networks classifiers using as input 12 motion units given by a face tracking system. The authors also use schemes that utilize data that are unlabeled and cheap to obtain, in conjunction with (expensively) labeled data 1 0 , 1 1 . For the dynamic setting, they used a multi-level HMM classifier that combines the temporal information and allows not only to perform the classification of a video segment to the corresponding facial expression, as in the previous works on HMM based classifiers, but also to automatically segment an arbitrary long sequence to the different expression segments without resorting to heuristic methods of segmentation. These methods are similar in the general sense that they first extract some features from the images, then these features are fed into a classification system, and the outcome is one of the preselected emotion categories. They differ mainly in the features extracted from the video images or the processing of video images to classify emotions. The video processing falls into two broad categories. The first is "feature-based," where one tries to detect and track specific features such as the corners of the mouth, eyebrows, etc.; the other approach is "region-based" in which facial motions are measured in certain regions on the face such as the eye/eyebrow and mouth regions. People have used different classification algorithms to categorize these emotions. In Table 1, we compare several facial expression recognition algorithms. In general, these algorithms perform well compared to trained human recognition of about 87% as reported by Bassili1. In contrast to the classification methods described above, Ueki et al. 65 extracted AUs and used neural networks (NN) to analyze the emotions, mapping seventeen AUs to two dimensions using an identity mapping network, and this showed resemblance of the 2D psychological emotion models. Later on, Morishima 39 proposed a 3D emotion model in order to deal with transitions between emotions, and claimed correlation to the 3D psychological emotion model 62 .
396 Table 1.
Comparisons of facial expression recognition algorithms.
Number of Subjects 1
Performance
kNN
Number of Categories 4
rule-based
6
40
92%
optical flow
rule-based
6
32
95%
optical flow
neural networks
2
32
88%
optical flow
distance-based
5
8
98%
HMM
6
4
93%
distance-based
7
-
74%
Winnow
6
5
86%
Bayesian networks
7
5+53
83%
Author
Processing
Classification
Mase Black & Yacoob Yacoob & Davis Rosenblum et al. Essa & Pentland Otsuka & Ohya Lanitis et al.
optical flow parametric model
Chen Cohen et al.
2D F T of optical flow appearance model appearance model appearance model
86%
Another interesting thing to point out is the problem of the commonly confused categories in the six basic expressions. As reported by Ekman, anger and disgust are commonly confused in judgment studies. Also, fear and surprise are commonly confused. The reason why these confusions occur is because they share many similar facial actions 19 . Surprise is sometimes mistaken for interest, but not the other way around. In the computer recognition studies, some of these confusions are observed 3,68,12 .
3.2. Vocal emotion
recognition
studies
The vocal aspect of a communicative message carries various kinds of information. If we disregard the manner in which the message was spoken and consider the verbal part (e.g., words) only, we might miss the important aspects of the pertinent utterance and we might even completely misunderstand what was the meaning of the message. Nevertheless, in contrast to spoken language processing, which has recently witnessed significant advances, the processing of emotional speech has not been widely explored. Starting in the 1930s, quantitative studies of vocal emotions have had a longer history than quantitative studies of facial expressions. Traditional as well as most recent studies in emotional contents in speech 40,9,13 ' 16,30,59,61 have used "prosodic" information which includes the pitch, duration, and intensity of the utterance 57 . Williams and Stevens66 studied the spectrograms of real emotional speech and compared them with acted speech. They found similarities which suggest the use of acted data. Murray and Arnott 40 reviewed findings on human vocal emotions. They also constructed a synthesis-by-rule system to incorporate emotions in syn-
397 thetic speech 41 . To date, most works have concentrated on the analysis of human vocal emotions. Some studied human abilities to recognize vocal emotions. These findings are very useful for the present work. There has been less work on recognizing human vocal emotions by computers than there has been on recognizing facial expressions by machine. Chiu et al. 9 extracted five features from speech and used a multilayered neural network for the classification. For 20 test sentences, they were able to correctly label all three categories. Dellaert et al. 16 used 17 features and compared different classification algorithms and feature selection methods. They achieved 79.5% accuracy with 4 categories and 5 speakers speaking 50 short sentences per category. Petrushin 51 compared human and machine recognition of emotions in speech and achieved similar rates for both (around 65%). In that work, 30 subjects spoke 4 sentences, with each sentence repeated 5 times, once for each emotion category. Scherer61 performed a large-scale study using 14 professional actors. In this study, he extracted as many as 29 features from the speech. According to Scherer, human ability to recognize emotions from purely vocal stimuli is about 60%. He pointed out that "sadness and anger are best recognized, followed by fear and joy. Disgust is the worst." Chen 6 proposed a rule-based method for classification of input audio data into one of the following emotions categories: happiness, sadness, fear, anger, surprise, and dislike. The input data contained 2 speakers, one speaking Spanish and the other one Sinhala. The choice of these languages was such that the subjective judgments were not influenced by the linguistic content as the observers did not comprehend either language. Each speaker was asked to speak 6 different sentences for each emotion and the contents of the sentences were related in most of the cases to one category and some of them could be applied to two different categories. From the audio signals pitch, intensity, and pitch contours were estimated as acoustic features which were then classified using some predefined rules. Recent studies seem to use the "Ekman six" basic emotions, although others in the past have used many more categories. The reasons for using these basic six categories are often not justified. It is not clear whether there exist "universal" emotional characteristics in the voice for these six categories. Table 2 shows a summary of human vocal affects as reported by Murray and Arnott 40 . This table describes mostly qualitative characteristics associated with these emotions. These are listed in relation to the neutral voice. 3.3. Emotion
recognition
from physiological
signals
Emotion consists of more than outward physical expression; it also consists of internal feelings and thoughts, as well as other internal processes of which the person having the emotion may not be aware. Still, these physiological processes can be naturally recognized by people. A stranger shaking your hand can feel its clamminess (related to skin conductivity); a friend leaning next to you may sense your heart pounding, etc.
398 Table 2.
Summary of human vocal affects described relative to neutral speech.
Speech Rate Pitch Average Pitch Range Intensity Voice Quality
Anger
Happiness
Sadness
Fear
slightly faster very much higher much wider higher breathy
faster or slower much higher much wider higher blaring
slightly slower slightly lower slightly narrower lower resonant
much faster very much higher much wider normal irregular
Disgust very much slower very much lower slightly wider lower grumbled
Physiological pattern recognition of emotion has important applications in medicine, entertainment, and human-computer interaction 53 . Physiological pattern recognition can potentially help in assessing and quantifying stress, anger, and other emotions that influence health. Affective states of depression, anxiety, and chronic anger have been shown to impede the work of the immune system, making people more vulnerable to infections, and slowing healing from surgery or disease. Changes in physiological signals can also be examined for signs of stress arising while users interact with the technology, helping detect where the product causes unnecessary irritation or frustration. This information may help developers to redesign and improve their technology. One of the big questions in emotion theory is whether distinct physiological patterns accompany each emotion 4 . The physiological muscle movements comprising what looks to an outsider to be a facial expression may not always correspond to a real underlying emotional state. This relation between the bodily feelings and externally observable expression is still an open research area, with a history of controversy. Historically, James was the major proponent of emotion as an experience of bodily changes, such as the perspiring hands or a pounding heart 28 . This view was challenged by Cannon 5 and by Schachter60 who argued that the experience of physiological changes was not sufficient to discriminate emotions. According to Schachter 60 , physiological responses such as sweaty hands and a rapid heart beat inform our brain that we are aroused and then the brain must analyze the situation we are in before it can label the state with an emotion such as fear or love. Since these classic works, there has been a debate about whether or not emotions are accompanied by specific physiological changes other than simply arousal level. Winton et al. 67 provided some of the first findings showing significant differences in autonomic nervous system signals according to a small number of emotional categories or dimensions, but there was no exploration of automated classification. Fridlund and Izard 22 appear to have been the first to apply pattern recognition (linear discriminants) to classification of emotion from physiological features, attaining rates of 38-51 percent accuracy (via cross-validation) on subject-dependent classification of four different facial expressions (happy, sad, anger, fear) given four facial electromyogram signals. Picard et al. 53 classified physiological patterns for a set of eight emotions (including neutral) by applying pattern recognition techniques
399 and by focusing on felt emotions of a single subject over sessions spanning many weeks. 4. Multimodal Approach to Emotion Recognition The studies in facial expression recognition and vocal affect recognition have been done largely independent of each other. The aforementioned works in facial expression recognition used still photographs or video sequences where the subject exhibits only facial expression without speaking any words. Similarly, the works on vocal emotion detection used only the audio information. There are situations where people would speak and exhibit facial expressions at the same time. For example, "he said hello with a smile." Pure facial expression recognizers may fail because the mouth movements may not fit the description of a pure "smile." For computers to be able to recognize emotional expression in practical scenarios, these cases must be handled. 4.1. Related
research
Combining audio and visual cues has been studied in recent years for speech recognition 54 . It has been shown that in situations when background noise makes the speech waveforms very noisy, cues from the lip movements improve speech recognition accuracy a great deal. In speech recognition, the lip movements and speech sounds are tightly coupled. For emotional expression recognition, the coupling is not so tight. Very little has been done to utilize both modalities for recognizing emotions. Pelachaud et al. 49 constructed a system that generated animated facial expressions for synthetic speech. Again, this work only emphasized the synthetic aspect and not the recognition of emotions. De Silva and Ng 15 proposed a rule-based method for singular classification of audiovisual input data into one of the six emotion categories: happiness, sadness, fear, anger, surprise, and dislike. Each of their subjects was asked to portray 12 emotion outbursts per category by displaying the related prototypical facial expression while speaking a single English word of his choice. The audio and visual material has been processed separately. They used optical flow for detecting the displacement and velocity of some key facial features (e.g., corners of the mouth, inner corners of the eye brows). From the audio signal, pitch and pitch contours were estimated by using the method proposed by Medan et al. 37 . A nearest neighbor method has been used to classify the extracted facial features and an HMM has been used to classify the estimated acoustic features into one of the emotion categories. Per subject, the results of the classification were plotted in two graphs and based upon these graphs the rules for emotion classification of the audiovisual input material were defined. Chen and Huang 7 proposed a set of methods for singular classification of input audiovisual data into one of the basic emotion categories: happiness, sadness,
400
disgust, fear, anger, and surprise. They collected data from five subjects which displayed 6 basic emotions 6 times by producing the appropriate facial expression right before or after speaking a sentence with the appropriate vocal emotion. Each of these single-emotion sequences started and ended with a neutral expression. Considering the fact that in the recorded data a pure facial expression occurred right before or after the sentence spoken with the appropriate vocal emotion, the authors applied a single-modal classification method in a sequential manner. To conclude, the most surprising issue regarding the multimodal emotion recognition problem, is that although the recent advances in video and audio processing could make the multimodal analysis of human affective state tractable, there were only a few research efforts which tried to implement a multimodal emotion analyzer. Further, there is no record of a research effort that aims at integrating all nonverbal modalities into a single system for affect-sensitive analysis of human behavior. 4.2. Fusing multimodal models
information
using probabilistic
graphical
A typical issue of multimodal data processing so far is that the multisensory data are typically processed separately and only combined at the end. Yet this is almost certainly incorrect; people display audio and visual communicative signals in a complementary and redundant manner. Chen et al. 8 have shown this experimentally. In order to accomplish a human-like multimodal analysis of multiple input signals acquired by different sensors, the signals cannot be considered mutually independent and cannot be combined in a context-free manner at the end of the intended analysis but, on the contrary, the input data should be processed in a joint feature space and according to a context-dependent model. In practice, however, besides the problems of context sensing and developing context-dependent models for combining multisensory information, one should cope with the size of the required joint feature space, which can suffer from large dimensionality, different feature formats, and timing. A potential way to achieve the target tightly coupled multisensory data fusion is to develop context-dependent versions of a suitable method such as the Bayesian inference method proposed by Pan et al. 46 . If we consider the state of the art in audio and visual signal processing, noisy and partial input data should also be expected. A multimodal system should be able to deal with these imperfect data and generate its conclusion so that the certainty associated with it varies in accordance to the input data. A way of achieving this is to consider the time-instance versus time-scale dimension of human nonverbal communicative signals as suggested by Pantic and Rothkrantz 48 . By considering previously observed data (time scale) with respect to the current data carried by functioning observation channels (time instance), a statistical prediction and its probability might be derived about both the information that have been lost due to malfunctioning/inaccuracy of a particular sensor and the currently displayed action/reaction. Probabilistic graphical models, such as hidden Markov Models (in-
401
eluding their hierarchical variants), Bayesian networks, and Dynamic Bayesian networks are very well suited for fusing such different sources of information. These models can handle noisy features, temporal information, and missing values of features all by probabilistic inference. Hierarchical HMM-based systems 12 have been shown to work well for facial expression recognition. Dynamic Bayesian Networks and HMM variants 24 have been shown to fuse various sources of information in recognizing user intent, office activity recognition, and event detection in video using both audio and visual information 23 .
Context
Fig. 2.
Bayesian network topology for bimodal emotion expression recognition.
The success of these research efforts has shown that fusing audio and video for detection of discrete events using probabilistic graphical models is possible. Therefore, we propose the Bayesian network topology for recognizing emotions from audio and facial expressions presented in Figure 2. While the network shown is static, it can be extended to be a dynamic Bayesian network in a straightforward manner. The network topology combines the two modalities in a probabilistic manner. The top node is the class variable (recognized emotional expression). It is affected by recognized facial expressions, recognized vocal expressions, recognized keywords that have an affective meaning, and by the context in which the system operates (if that is available). Vocal emotions are recognized from audio features extracted from the person's audio track. Facial expressions are recognized by facial features tracked using video, but the recognition is also affected by a variable that indicates whether the person is speaking or not. Recognizing whether a person is speaking uses both visual cues (mouth motion) and audio features (using similar techniques as Garg et al. 24 ). The parameters of the proposed network can be learned from data, or manually set for some variables. Inferring the human emotional expression can be performed even when some pieces of information are missing, e.g., when audio is too noisy, or the face tracking loses the face.
402
Another issue which makes the problem of emotional expression recognition even more difficult to solve in a general case is the dependency of a person's behavior on his/her personality, cultural, and social vicinity, current mood, and the context in which the observed behavioral cues were encountered. One source of help for these problems is machine learning: rather than using a priori rules to interpret human behavior, we can potentially learn application-, user-, and context-dependent rules by watching the user's behavior in the sensed context 50 . This leads to another advantage of probabilistic graphical models: well known algorithms exist to adapt the models, and it is possible to use prior knowledge when learning new models. For example, a prior model of emotional expression recognition trained based on a certain user can be used as a starting point for learning a model for another user, or for the same user in a different context. Though context sensing and the time needed to learn appropriate rules are significant problems in their own right, many benefits could come from such an adaptive affect-sensitive HCI tool. While fusing multimodal information for emotion recognition is an important issue, there are some other aspects that are of equal importance. Difficult problems are those of obtaining the ground truth of the data and getting data that genuinely corresponds to a particular emotional state. Even though there are cases when the data can be easily labeled (e.g., a singular strong emotion is captured, such as an episode of rage), in most of the cases the ground truth — which emotion was present — is difficult to establish. We discuss these issues in detail in the following sections.
4.3. Collecting
multimodal
data for emotion
recognition
In general, the goal of emotional expression is to detect the emotional state of the person in a natural situation. However, as any photographer can attest, getting a real smile can be challenging. Asking someone to smile often does not create the same picture as an authentic smile. The fundamental reason of course is that the subject often does not feel happy so his smile is artificial and in many subtle ways quite different than a genuine smile. Picard et al. 53 outlined five factors that influence the affective data collection: • Spontaneous versus posed: Is the emotion elicited by a situation or stimulus that is outside the subject's control or the subject is asked to elicit the emotion? • Lab setting versus real-world: Is the data recording taking place in a lab or the emotion is recorded in the usual environment of the subject? • Expression versus feeling: Is the emphasis on external expression or on internal feeling? • Open recording versus hidden recording: Is the subject aware that he is being recorded? • Emotion-purpose versus other-purpose: Does the subject know that he is a part of an experiment and the experiment is about emotion?
403
Note that these factors are not necessarily independent. The most natural setup would imply that the subject feels the emotion internally (feeling), the emotion occurs spontaneous, while the subject is in his usual environment (real-world). Also, the subject should not know that he is being recorded (hidden recording) and that he is a part of an experiment (other-purpose). Such data are usual impossible to obtain because of privacy and ethics concerns. As a consequence, several researchers 53 ' 64 were trying to use a setup that resembled as much as possible the natural setup. Picard et al. 53 collected data using a posed, closed to real-world (subject's comfortable usual workplace), feeling, open-recording, and emotion-purpose methodology. The key factor that made their data unique is that the subject tried to elicit an internal feeling of each emotion. Sebe et al. 64 were more interested in collecting spontaneous emotion data. They created a video kiosk (lab setting) with a hidden camera (hidden-recording) which displayed segments from recent movie trailers. This setup had the main advantage that it naturally attracted people to watch and could potentially elicit emotions through different genres of video footage — i.e. horror films for shock, comedy for joy, etc. The issue of whether to use posed or spontaneous expressions in selecting facial stimuli, has been hotly debated 17 . Experimentalists and most emotion theorists argue that spontaneous expressions are the only "true" expressions of facial emotion and therefore such stimuli are the only ones of merit. When recording authentic (spontaneous) emotions several aspects should be considered 64 . Not all people express emotion equally well; many individuals have idiosyncratic methods of expressing emotion as a result of personal, familial, or culturally learned display rules. Situations in which authentic emotions are usually recorded (e.g., lab setting) are often unusual and artificial. If the subject is aware of being photographed or filmed (open-recording), his emotional response may not be spontaneous anymore. Even if the subject is unaware of being filmed (hiddenrecording), the laboratory situation may not encourage natural or usual emotion response. In interacting with scientists or other authorities (emotion-purpose), subjects will attempt to act in appropriate ways so that emotion expression may be masked or controlled. Additionally, there are only a few universal emotions and only some of these can be ethically stimulated in the laboratory. On the other hand, posed expressions may be regarded as an alternative, provided that certain safeguards are followed. Increased knowledge about the face, based in large part on observation of spontaneous, naturally occurring facial expressions, has made possible a number of methods of measuring the face. The same situation stands for voice analysis. These measurement techniques can be used to ascertain whether or not emotional behavior has occurred and what emotion is shown in a given instance. Such facial scoring provides a kind of stimulus criterion validity that is important in this area. Additionally, posers can be instructed, not to act or pose a specific emotion, but rather to move certain muscles so as to effect the desired emotional expression. In this way, experimental control may be exerted
404
on the stimuli and the relationship between the elements of the expression and the responses of observers may be analyzed and used as a guide in item selection. It should be noted that the distinction between posed and spontaneous behavior is not directly parallel to the distinction between artificial and natural occurrences. Though posing is by definition artificial, spontaneous behavior may or may not be natural 17 . Spontaneous behavior is natural when some part of life itself leads to the behavior studied. Spontaneous behavior elicited in the laboratory may be representative of some naturally occurring spontaneous behavior, or conceivably it could be artificial if the eliciting circumstance is unique and not relevant to any known real life event. From the above discussion, it is clear that the authentic emotion analysis should be performed whenever is possible. Posed expression may be used as an alternative only in restricted cases and they can be mostly used for benchmarking the authentic expressions.
4.4. Leveraging
unlabeled
data for emotion
recognition
As pointed out in the previous section, collecting emotional expression data is a difficult task. Labeling those data adds an additional challenge, as it is time-consuming, error prone, and expensive. In addition, an emotion expression recognition system that is deployed in a realistic setting would easily obtain an abundance of emotion expressions, but would not be able to obtain manual labeling of that data — if a computer constantly asks a user for his/her emotion, we can be quite sure that eventually the response would be that of anger or annoyance. Therefore, it would be very beneficial to construct methods that utilize both scarcely available labeled data and abundance of unlabeled data — where the labels are the emotional state (or expression) of a user. Again, probabilistic graphical models are ideal candidates for such data, as efficient and convergent algorithms exist for handling missing data in general and unlabeled data in particular. Cohen et al. 11 have shown that unlabeled data can be used for recognizing facial expressions using Bayesian networks with a combination of labeled and unlabeled data. However, they have shown that care must be taken when attempting such schemes. While in the purely supervised case (with only labeled data), adding more labeled data always improves the performance of the classifier, adding more unlabeled data can be detrimental to performance. As shown by Cohen et al. 11 such detrimental effects occur when the assumed classifier's model does not match the distribution generating data. They propose an algorithm for stochastically searching the space of Bayesian networks, converging on a classifier which does utilize positively unlabeled data. To conclude, further research is required to achieve maximum utilization of unlabeled data for the problem of emotion recognition, but it is clear that such methods would provide great benefit.
405
5. Discussion and Conclusion As remarked Salovey and Mayer 58 and Goleman 25 emotional skills are an essential part of what is called "intelligence". This is based on recent scientific findings about the role of emotional abilities in human intelligence and on the way human-machine interaction imitates human-human interaction. As a consequence, emotions, largely overlooked in early efforts to develop machine intelligence, are increasingly regarded as an area of important research. Emotion modulates almost all modes of human communication —facial expression, gestures, posture, tone of voice, choosing of words, respiration, skin temperature and clamminess, etc. Emotions can significantly change the message: sometimes it is not what was said that is the most important, but how it was said. Faces tend to be the most visible form of emotion communication, but they are also most easily controlled in response to different social situations when compared to the voice and other ways of expression. As noted by Picard 52 affect recognition is most likely to be accurate when it combines multiple modalities, information about the user's context, situation, goal, and preferences. A combination of low-level features, high-level reasoning, and natural language processing is likely to provide the best emotion inference. Considering all these aspects, Reeves and Nass 55 and Pentland 50 believe that multimodal context-sensitive human-computer interaction is likely to become the single most widespread research topic of the artificial intelligence research community. Advances in this area could change not only how professionals practice computing, but also how mass consumers interact with the technology. As we discussed in this chapter and pointed out by Pantic and Rothkrantz 48 , although there were significant advances in the fields of video and audio processing, pattern recognition, computer vision, and affective computing, the realization of a robust, multimodal, adaptive, context-sensitive analyzer of human nonverbal affective state is far from being a reality. Currently, the researchers have to cope with the lack of a better understanding of individual- and context-dependent human behavior and with a better integration of multiple sensors and pertinent modalities according to the model of human sensory system. Besides these problems there are other social and ethical issues that should be considered. The context-sensitive multimodal system that is supposed to interact with the human should not invade the user's privacy. Computer technology and especially affect-sensitive monitoring tools might be regarded as "big brother" tools. As remarked by Schneiderman 63 , a large proportion of the population would be terrified by the vision of the universal use of computers in the coming era of ubiquitous computing. Another important aspect is related to teaching the HCI systems our interaction patterns and related behavior and our social and cultural profile. It is obvious that is inefficient and annoying for the human user to train separately all the HCI systems that will be all around us in the future. One way of dealing with this problem is the incorporation of unlabeled data as we pointed out in this chapter. Moreover, the system itself should be able to monitor human nonverbal behavior and to adapt to the current user, to his context,
406 and t o t h e current scenario and environment. By taking all of these aspects into account, we hope t o be able t o develop into the near future multimodal context-sensitive systems t h a t are smart, perceptually aware, recognize t h e context in which they act, can a d a p t t o their users, and can understand how they feel, and respond appropriately. In some sense, these systems will be t h e friendly variants of t h e Replicants from t h e Blade Runner.
References 1. J.N. Bassili. Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37(ll):2049-2058, 1979. 2. N. Bianchi-Berthouze and C. Lisetti. Modeling multimodal expression of user's affective subjective experience. User Modeling and User-Adapted Interaction, 12:49-84, 2002. 3. M.J. Black and Y. Yacoob. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proc. International Conf. on Computer Vision, pages 374-381, 1995. 4. J.T. Cacioppo and L.G. Tassinary. Inferring psychological significance from physiological signals. American Psychologist, 45:16-28, 1990. 5. W.B. Cannon. The James-Lange theory of emotion: A critical examination and an alternative theory. American Journal of Psychology, 39:106-124, 1927. 6. L.S. Chen. Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction. PhD thesis, University of Illinois at Urbana-Champaign, Dept. of Electrical Engineering, 2000. 7. L.S. Chen and T.S. Huang. Emotional expressions in audiovisual human computer interaction. In Proc. International Conference on Multimedia and Expo (ICME), pages 423-426, 2000. 8. L.S. Chen, H. Tao, T.S. Huang, T. Miyasato, and R. Nakatsu. Emotion recognition from audiovisual information. In Proc. IEEE Workshop on Multimedia Signal Processing, pages 83-88, 1998. 9. C.C. Chiu, Y.L. Chang, and Y.J. Lai. The analysis and recognition of human vocal emotions. In Proc. International Computer Symposium, pages 83-88, 1994. 10. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang. Learning bayesian network classifiers for facial expression recognition using both labeled and unlabeled data. In Proc. Conf. on Computer Vision and Pattern Recognition, volume 1, pages 595-601, 2003. 11. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang. Semi-supervised learning of classifiers: Theory, algorithms, and applications to human-computer interaction. IEEE Trans, on Pattern Analysis and Machine Intelligence, to appear, 2004. 12. I. Cohen, N. Sebe, A. Garg, L. Chen, and T.S. Huang. Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 91(1-2):160-187, 2003. 13. R. Cowie and E. Douglas-Cowie. Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In Proc. International Conf. on Spoken Language Processing, pages 1989-1992, 1996. 14. C. Darwin. The Expression of the Emotions in Man and Animals. John Murray, London, 2nd edition, 1890. 15. L.C. De Silva and P.C Ng. Bimodal emotion recognition. In Proc. Automatic Face and
407 Gesture Recognition, pages 332-335, 2000. 16. F. Dellaert, T. Polzin, and A. Waibel. Recognizing emotion in speech. In Proc. International Conf. on Spoken Language Processing, pages 1970-1973, 1996. 17. P. Ekman, editor. Emotion in the Human Face. Cambridge University Press, New York, NY, 2nd edition, 1982. 18. P. Ekman. Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique. Psychological Bulletin, 115(2):268-287, 1994. 19. P. Ekman and W.V. Friesen. Facial Action Coding System: Investigator's Guide. Consulting Psychologists Press, 1978. 20. LA. Essa and A.P. Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans, on Pattern Analysis and Machine Intelligence, 19(7) :757763, 1997. 21. B. Fasel and J. Luettin. Automatic facial expression analysis: A survey. Pattern Recognition, 36:259-275, 2003. 22. A. Fridlund and C. Izard. Electromyographic studies of facial expressions of emotions and patterns of emotion. In J. Cacioppo and R. Petty, editors, Social Psychophysiology: A Sourcebook, pages 243-286, 1983. 23. A. Garg, M. Naphade, and T.S. Huang. Modeling video using input/output markov models with application to multi-modal event detection. In B. Furht, O. Marques, and B. Furht, editors, Handbook of Video Databases: Design and Applications, 2003. 24. A. Garg, V. Pavlovic, and J. Rehg. Boosted learning in dynamic Bayesian networks for multimodal speaker detection. Proceedings of the IEEE, 91 (9): 1355-1369, 2003. 25. D. Goleman. Emotional Intelligence. Bantam Books, 1995. 26. E. Hilgard, R.C. Atkinson, and R.L. Hilgard. Introduction to Psychology. Harcourt Brace Jovanovich, New York, NY, 5th edition, 1971. 27. C.E. Izard. Innate and universal facial expressions: Evidence from developmental and cross-cultural research. Psychological Bulletin, 115(2):288-299, 1994. 28. W. James. The Principles of Psychology. Henry Holt, New York, NY, 1890. 29. J.M. Jenkins, K. Oatley, and N.L. Stein, editors. Human Emotions: A Reader. Blackwell Publishers, Maiden, MA, 1998. 30. T. Johnstone. Emotional speech elicited using computer games. In Proc. International Conf. on Spoken Language Processing, pages 1985-1988, 1996. 31. P. Lang. The emotion probe: Studies of motivation and attention. American Psychologist, 50(5):372-385, 1995. 32. A. Lanitis, C.J. Taylor, and T.F. Cootes. A unified approach to coding and interpreting face images. In Proc. International Conf. on Computer Vision, pages 368-373, 1995. 33. J. Lien. Automatic recognition of facial expressions using hidden Markov models and estimation of expression intensity. PhD thesis, Carnegie Mellon University, 1998. 34. A Martinez. Face image retrieval using HMMs. In IEEE Workshop on Content-based Access of Images and Video Libraries, pages 35-39, 1999. 35. K. Mase. Recognition of facial expression from optical flow. IEICE Trans., E74( 10) :34 74-3483, 1991. 36. D. Matsumoto. Cultural influences on judgments of facial expressions of emotion. In Proc. ATR Symposium on Face and Object Recognition, pages 13-15, 1998. 37. Y. Medan, E. Yair, and D. Chazan. Super resolution pitch determination of speech signals. IEEE Trans, on Signal Processing, 39:40-48, 1991. 38. A. Mehrabian. Communication without words. Psychology Today, 2(4):53-56, 1968. 39. S. Morishima. Emotion model: A criterion for recognition, synthesis and compression of face and emotion. In Proc. Automatic Face and Gesture Recognition, pages 284-289, 1995.
408 40. I.R. Murray and J.L. Arnott. Toward the simulation of emotion in synthetic speech: A review of the literature of human vocal emotion. Journal of the Acoustic Society of America, 93(2): 1097-1108, 1993. 41. I.R. Murray and J.L. Arnott. Synthesizing emotions in speech: Is it time to get excited? In Proc. International Conf. on Spoken Language Processing, pages 1816-1819, 1996. 42. A. Nefian and M. Hayes. Face recognition using an embedded HMM. In IEEE Conf. on Audio and Video-based Biometric Person Authentication, pages 19-24, 1999. 43. N. Oliver, A. Pentland, and F. Berard. LAFTER: A real-time face and lips tracker with facial expression recognition. Pattern Recognition, 33:1369-1382, 2000. 44. T. Otsuka and J. Ohya. Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences. In Proc. International Conf. on Image Processing, pages 546-549, 1997. 45. T. Otsuka and J. Ohya. A study of transformation of facial expressions based on expression recognition from temporal image sequences. Technical report, Institute of Electronic, Information, and Communications Engineers (IEICE), 1997. 46. H. Pan, Z.P. Liang, T.J. Anastasio, and T.S. Huang. Exploiting the dependencies in information fusion. In Proc. Conf. on Computer Vision and Pattern Recognition, volume 2, pages 407-412, 1999. 47. M. Pantic and L.J.M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Trans, on Pattern Analysis and Machine Intelligence, 22(12):14241445, 2000. 48. M. Pantic and L.J.M. Rothkrantz. Toward an affect-sensitive multimodal humancomputer interaction. Proceedings of the IEEE, 91 (9).-1370-1390, 2003. 49. C. Pelachaud, N. Badler, and M. Steedman. Generating facial expression for speech. Cognitive Science, 20:1-46, 1996. 50. A. Pentland. Looking at people. Communications of the ACM, 43(3):35-44, 2000. 51. V.A. Petrushin. How well can people and computers recognize emotions in speech? In Proc. AAAI Fall Symposium, pages 141-145, 1998. 52. R W. Picard. Affective Computing. MIT Press, Cambridge, MA, 1997. 53. R.W. Picard, E. Vyzas, and J. Healey. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans, on Pattern Analysis and Machine Intelligence, 23(10):1175-1191, 2001. 54. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A.W. Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE, 91(9):13061326, 2003. 55. B. Reeves and C. Nass. The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places. Cambridge Univ. Press, 1996. 56. M. Rosenblum, Y. Yacoob, and L.S. Davis. Human expression recognition from motion using a radial basis function network architecture. IEEE Trans, on Neural Network, 7(5):1121-1138, 1996. 57. Y. Sagisaka, N. Campbell, and N. Higuchi, editors. Computing Prosody. SpringerVerlag, New York, NY, 1997. 58. P. Salovey and J.D. Mayer. Emotional intelligence. Imagination, Cognition, and Personality, 9(3):185-211, 1990. 59. J. Sato and S. Morishima. Emotion modeling in speech production using emotion space. In Proc. IEEE Int. Workshop on Robot and Human Communication, pages 472-477, 1996. 60. S. Schachter. The interaction of cognitive and physiological determinants of emotional state. In L. Berkowitz, editor, Advances in Experimental Psychology, volume 1, pages 49-80, 1964.
409 61. K.R. Scherer. Adding the affective dimension: A new look in speech analysis and synthesis. In Proc. International Conf. on Spoken Language Processing, pages 18081811, 1996. 62. H. Schlosberg. Three dimensions of emotion. Psychological Review, 61:81-88, 1954. 63. B. Schneiderman. Human values and the future of technology: A declaration of responsibility. In B. Schneiderman, editor, Sparks of Innovation in Human-computer Interaction, 1993. 64. N. Sebe, M.S. Lew, I. Cohen, Y. Sun, T. Gevers, and T.S. Huang. Authentic facial expression analysis. In Automatic Face and Gesture Recognition, pages 517-522, 2004. 65. N. Ueki, S. Morishima, H. Yamada, and H. Harashima. Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Systems and Computers in Japan, 25(13):95-103, 1994. 66. C.E. Williams and K.N. Stevens. Emotions and speech: Some acoustical correlates. Journal of the Acoustic Society of America, 52(4):1238-1250, 1972. 67. W. Winton, L. Putman, and R. Krauss. Facial and autonomic manifestations of the dimensional structure of the emotion. Journal of Experimental Social Psychology, 20:195-216, 1984. 68. Y. Yacoob and L.S. Davis. Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans, on Pattern Analysis and Machine Intelligence, 18(6):636-642, 1996.
C H A P T E R 4.2 G A I T - B A S E D H U M A N IDENTIFICATION FROM A M O N O C U L A R VIDEO S E Q U E N C E
Amit Kale Center for Visualization and Virtual Environments 1, Quality St Suite 800-B KY 40507 USA E-mail: [email protected]
Aravind Sundaresan' Amit K. RoyChowdhury Department of Electrical Engineering University of California, Riverside CA 92521 USA E-mail: [email protected] Rama Chellappa f Center for Automation Research University of Maryland at College Park MD 2074S USA E-mail:[email protected]. edu Human gait is a spatio-temporal phenomenon that characterizes the motion characteristics of an individual. It is possible to detect and measure gait even in lowresolution video. In this chapter, we discuss algorithms for identifying people by their gait from a monocular video sequence. Human identification using gait, similar to text-based speaker identification, involves different individuals performing the same task and a template-matching approach is suitable for such problems. In situations where the amount of training data is limited, we demonstrate the utility of a simple width feature for gait recognition. By virtue of their deterministic nature, template matching methods have limited noise resilience. In order to deal with noise we introduce a systematic approach to gait recognition by building representations for the structural and dynamic components of gait using exemplars and hidden Markov models (HMMs). The above methods assume that an exact side-view of the subject is available in the probe sequence. For the case when the person walks at an arbitrary angle far away from the camera we present a view invariant gait recognition algorithm which is based on synthesizing a side view of a person from an arbitrary monocular view. 411
412 1. Introduction Automated person identification is an important component of surveillance. An effective approach to person identification is to reduce it to the problem of identifying physical characteristics of the person. This method of identification of persons based on his/her physiological/behavioral characteristics is called biometrics. Established biometric methods range from fingerprint and hand-geometry techniques to more sophisticated methods based on face recognition and iris identification. Unfortunately, no single biometric is perfect or complete. Fingerprints and hand-geometry are reliable but require physical contact. Although, signatures based on face and iris are non-intrusive in nature, the applicability of all these methodologies is restricted to very controlled environments. In fact, current technology is capable of recognizing mostly frontal faces. At the time of writing, iris recognition is being attempted at distances of not more than five meters. When person identification is attempted in natural settings such as those arising in surveillance applications, it takes on a new dimension. Biometrics such as fingerprint or iris are no longer applicable. Furthermore, night vision capability (an important component in surveillance) is usually not possible with these biometrics. Even though an IR camera would reveal the presence of people, the facial features are far from discernible in an IR image at large distances. A biometric that can address some of these shortcomings is human 'gait' or the walking style of an individual. The attractiveness of gait as a biometric arises from the fact that it is non-intrusive and can be detected and measured even in low resolution video. Furthermore, it is harder to disguise than static appearance features such as a face and it does not require a cooperating subject. Early research on gait primarily involved psychophysical studies of gait viz. studying the ability of human observers to recognize gait. The belief that humans can distinguish between gait patterns of different individuals is widely held. Intuitively, it is possible to think of the qualities of walk such as stride length or body swing that help a perceiver identify an approaching figure even before the face becomes discernible. The earliest and most recognized psychophysical study of human perception of gait was the work of Johansson : . Small light bulbs were attached to the body joints of a darkly dressed walker. In this way only gait related cues were available and thus the perception of pure biological motion could be examined. When these point-light displays were static, the random collection of dots were variously interpreted as star constellations. However, as soon as the figures moved, the points of light were immediately perceived to be a human in motion. Motivated by Johanssons work, Kozlowski and Cutting 2 investigated whether observers could identify the gender of a point-light walker. The demonstration that gender could be extracted from gait provided insight into how observers might discriminate between gait patterns of different individuals. The prospect for observers being able to identify individuals from their gaits was thus encouraging. Cutting and Kozlowski 3 demonstrated that perceivers could reliably recognize themselves and their friends
413 from dynamic point-light displays. Barclay et al. 4 suggested that individual walking styles might be captured by differences in a basic series of pendular limb motions. Psychophysical evidence that there exists identity information in gait spurred development of computer vision based algorithms for gait-based human recognition. We attempt to give a summary of some of examples below, but the listing is by no means complete. Most of the methods for gait recognition are appearance based. Appearance based methods work reasonably well in the face of inaccurate background segmentation, changes in speed etc. However, such methods cannot tolerate drastic changes in clothing. Cunado et al. 5 extract a gait signature by fitting the movement of the thighs to an articulated pendulum-like motion model. Huang et al. 6 use optical flow to derive a motion image sequence for a walk cycle followed by eigenanalysis of the binarized silhouette to derive what are called eigen gaits. Benabdelkader et al. 7 use image self-similarity plots as a gait feature. Tolliver and Collins 8 use a spectral partitioning framework for identifying humans by their shape. Lee and Grimson 9 propose an approach in which several ellipses are fitted to different parts of the binarized silhouette of the person and the parameters of these ellipses such as location of its centroid, eccentricity etc. are used as a feature to represent the gait of a person. Hayfron-Acquah et al 10 proposed a method based on analyzing the symmetry of human motion using the Generalised Symmetry Operator. Han and Bhanu n proposed a gait-energy image approach for recognition. 2. Feature Selection An important issue in gait is the extraction of appropriate salient features that will effectively capture the gait characteristics. The features must be reasonably robust to operating conditions and should yield good discriminability across individuals. As mentioned earlier, we assume that the side view of each individual is available. Intuitively, the silhouette appears to be a good feature to look at as it captures the motion of most of the body parts. It also supports night vision capability as it can be derived from IR imagery also. While extracting this feature we can either use the entire silhouette or use only the outer contour of the silhouette. The choice of using either of the above features depends upon the quality of the binarized silhouettes. If the silhouettes are of good quality, the outer contour retains all the information of the silhouette and allows a representation, the dimension of which is an order of magnitude lower than that of the binarized silhouette. However for low quality, low resolution data, the extraction of the outer contour from the binarized silhouette may not be reliable. In such situations, direct use of the binarized silhouette may be more appropriate. We choose the width of the outer contour of the silhouette as one of our feature vectors. In order to generate the width vectors background subtraction 12 is first applied to the image sequence and the resulting motion image is binarized into foreground and background pixels. A bounding box is then placed around the part of the motion image that contains the moving person. Given the binarized silhouettes,
414 Width Pro«9 for Para
(a)
WfdthPrsfitetorPe
(I,)
Fig. 1. Width vector profile for several gait cycles of two individuals
the left and right boundaries of the body are traced. The width along a given row is simply the difference in the locations of the right-most and the left-most boundary pixels in that row. It is easy to see that the norm of the width vector show a periodic variation. In Figure 1 we show plots of the width profiles of two different individuals for several gait cycles. Since we use only the distance between the left and right extremities of the silhouette, the two halves of the gait cycle are almost indistinguishable. Prom hereon, we refer to half cycles as cycles, for the sake of brevity. In Figure 1, the x-axis denotes the frame index while the y-axis denotes the index of the width vector (the row index). The ith horizontal line in the image shows the variations in the ith element of the width vector as a function of time. A brighter gray-scale indicates a higher value of the width. We observe that within each cycle, there is a systematic temporal progression of width vector magnitude, viz. the dynamics. A similar observation has been made in 13 where the gait patterns are analyzed as FVieze patterns. For the two width profile plots shown in Figure 1 , the differences are quite visible. For instance, by observing the bright patterns in the upper region of the two images we see that the brightness is more pronounced in the first image as compared to the second. This area of the plot corresponds to the swings of the hand. Secondly, note that the brightness gradient (which translates to velocity in the video sequence) in the lower part of the images is more pronounced for Person 1 as compared to Person 2. This part of the plot corresponds to the swings of the extremities of the foot. Additionally, note that the height, as well as the visibility of the neck part of the two persons are different. It must be pointed out, however, that the differences need not be so pronounced for all individuals. Thus, the width profile contains structural and dynamic information peculiar to each individual. Also, by definition, the width vector is translation-invariant. Hence, the width of the outer contour of the silhouette is indeed a potentially good candidate as a feature.
415
3. G a i t - B a s e d H u m a n Identification Using A p p e a r a n c e M a t c h i n g A gait cycle corresponds to one complete cycle from rest (standing) position to-rightfoot-forward-to-rest-to-leift-foot-forward-to-rest position. The movements within a cycle consist of the motion of the different parts of the body such as head, hands, legs etc. The characteristics of an individual are reflected not only in the dynamics and periodicity of a gait cycle but also in the size and shape of that individual. Our aim is to build a model for representation and recognition of individual gait. In a pattern classification problem, choice of the feature as well as the classifier is important. As discussed earlier, if the gait data is clean, the width of the outer contour of the silhouette of the person can be a good feature for gait recognition. Prom the temporal width plots, we note that although the width vector changes with time within a gait cycle, there is a high degree of correlation among the width vectors across frames. Most changes occur in the hand and in the leg regions. Hence, it is reasonable to expect that gait information in the width vector can be derived with much fewer coefficients. Given the width vectors {W(l), • • • , W(N)},foi the N frames W(.) £ RM, we compute the eigen vectors { F ( l , ) • • • , V(M)} corresponding to the eigen values of the scatter matrix arranged in the descending order and reconstruct the corresponding width vectors using m(< M) most significant eigen vectors as m
Wr(i) = C£wjV(j)) + W 3=1
where wj =< W(i), V(j) > and W = wM+"+wW, Figure 2 shows the width vectors reconstructed using two eigenvectors. Temporally-ordered sequences of eigens-
>?&*&?
\
(a)
(b)
Fig. 2. Effect of eigen decomposition and reconstruction on the width vectors, (a) Overlapped raw width vectors (b) Smoothed width vectors. Notice that the leg region (the bottom half of the figures) contain a significant portion of the dynamics
moothed width vectors are used for compactly representing the person's gait.
416 For a normal walk, gait sequences are repetitive and exhibit nearly periodic behavior. The gait problem is analogous to text-based speaker identification/verification wherein different individuals utter the same text but differ only in the characteristics of their utterance 14 . A template matching approach is suitable for such problems especially if the amount of training data is limited. Typically, gait cycles when taken at different times tend to be unequal in length due to changes in walking speeds of the individuals. To deal with this issue, dynamic time-warping (DTW) is employed for matching gait sequences. The DTW method was originally developed for isolated word recognition 15 , and later adapted for text-dependent speaker verification 14 . DTW uses an optimum time expansion/compression function for producing non-linear time normalization so that it is able to deal with misalignments and unequal sequence lengths of the probe and the reference gait sequences. A distance metric (usually the Euclidean distance) defined as a function of time is computed between the two feature sets representing the gait data. A decision function is arrived at by integrating the metric over time. Assuming that the first frame of the reference and probe sequence are both indexed as 1 and the last frames of the reference and probe sequences be indexed as X and Y, respectively the match between the two sets can be represented by a sequence of K points C(l),C(2),....,C(k),...,C(K), where C{k) = (x(k),y(k)), and x{k) is a frame index of the probe sequence and y{k) is a frame index of the reference sequence. Here, C{k) represents a mapping of the time axis of probe sequence onto that of the reference sequence. The sequence F = C(l), C(2), ....C(k),...., C(K) is called the warping path. The process of time normalization involves computing the cumulative distance subject to endpoint, local continuity and global path constraints 16
Table 1. Cumulative match scores for the UMD database using different eigen-features. Feature\Rank Eigenvector 1 Eigenvectors 1,2 Eigenvectors 1,2,3 Eigenvectors 1,2,3,4
1 73 80 68 73
2 75 87 80 77
3 80 90 84 84
4 80 90 84 84
5 84 91 84 84
We experimented with different databases to test our method including the UMD, CMU and MIT datasets 1T. Table 1 shows the gait recognition result for the UMD dataset using different number of eigenvectors for reconstruction. Note that by using just the first two eigenvectors an accuracy of 80% is achievable. Other eigenvectors are noisy and, in fact, tend to lower the accuracy. We also considered the USF database a which has been identified as the gait challenge database 18 . The database has variations as regards viewing direction, shoe type, surface type. a
More details about this database can be found at http://figment.csee.usf.edu/GaitBaseline/
417
Also the subjects were asked to carry a briefcase for one testing condition. The USF database has the largest number of individuals among all the databases. It has variations with respect to floor surface (grass (G) or concrete(C)), shoe type (A or B), and camera viewing direction (left (L) or right (R)). The reference for all the experiments was chosen to be (G, A, R). The number of frames corresponding to four half cycles varied from 65 to 90. Different probe sequences for the experiments along with the cumulative match scores are given in Table 2 for the baseline algorithm 19 as well as our method using the eigensmoothed width feature. Note that recognition performance suffers most due to difference in surface characteristics, and least due to difference in viewing angle. An examination of the USF database revealed that the silhouettes provided were noisier compared to the previous datasets. We wanted to see what the performance would be by using the binary silhouettes directly as the feature. In this case we used the binary correlation distance in the local distance computation. As can be seen from the last two columns of Table 2 usage of the binarized silhouettes yields better performance numbers compared to the width vector in this case. Table 2. Probe Sets and match scores for the USF database using the baseline algorithm and our approach using width feature and entire binary silhouette. Experiment (Probe) A(G,A,L) B (G, B, R) C (G,B,L) D (C, A, R) E (G, B, R) F(C,A,L) G (C, B, L)
Baseline Rank 1 Rank 5 96 79 66 81 56 76 61 29 55 24 30 46 10 33
Width Vector Rank 1 Rank 5 91 79 79 67 55 30 42 17 39 15 30 16 31 9
Binary Silhouette Rank 1 Rank 5 97 84 91 83 79 59 41 64 53 24 51 27 38 24
4. A Framework for gait-based person identification using continuous H M M s In the previous section we saw the application of a simple template matching approach to gait recognition. A careful analysis of gait would reveal that it has two important components. The first is a structural component that captures the physical build of a person e.g. body dimensions, length of limbs etc. The second component is the motion kinematics of the body during a gait cycle. A recent paper by Veeraraghavan et al 20 evaluates the contribution from these factors in vision-based gait recognition. In this section we propose a systematic approach to gait recognition by building representations for the structural and kinematic components of gait. A closer examination of the physical process behind the generation of gait signature reveals that, during a gait cycle, it is possible to identify certain distinct phases or stances. In Figure 3, we show five frames that we have picked from a gait cycle for two individuals.
418
Hill Hill (a) (b) Fig. 3. Stances corresponding to the gait cycle of two individuals
In the first stance, the person holds the two feet together. In the second, he is just about to start and his hand is slightly raised. In the third stance, the hands and the feet are separated, while in the fourth, the hands and feet are displaced to a maximum. Finally, in the fifth stance, the person is returning to the rest state. Clearly, every person transits among these successive stances as he/she walks. Although, these stances are generic, there exist differences not only in their image appearance based on the physical build of an individual but also in the way an individual transits across these stances as he/she walks which represents the gait kinematics of the individual. A reasonable way to build a structural representation for a person is to pick N exemplars (or stances) 5 = {ei, • • • ,ejv} from the pool of images that will minimize the error in representation of all the images of that person. Given the image sequence for an unknown person y = {y(l), • • • ,y(T)}, these exemplars can be directly used for recognition as T
ID = arg mm,- ^ m i n n G { l j . . . i i V } d ( y ( £ ) , e £ ) , t=i
where y(t) represents the image of an unknown person at the ith time instant, while e£ represents the nth exemplar of the j t h individual. Note, however, that such a simple discrimination criterion is susceptible to failures not only due to noise but more importantly due to the presence of structural similarities among people in the database. To improve discriminability, the kinematics of the data must be exploited. A closer look at the gait cycle reveals that there is a temporal progression in the proximity of the observed silhouette to the different exemplars. Note that at the start of the gait cycle, a frame is closer to the first exemplar as compared to the other four. As time progresses, the frame will be closer to the second exemplar as compared to the others and so on. Underlying the proximity of the silhouette to the exemplars is a probabilistic dependence across the exemplars. This encompasses information about how long a gait cycle persists in a particular exemplar as well as the way in which the gait cycle transits from one exemplar to the other. For two people who are similar in physical build, this kinematic knowledge can be used to improve the recognition performance. Because the transitions are systematic, it is possible to model this probabilistic dependence by a Markov matrix as shown below. A=[P(ei(t)\ej(t-l))}
(1)
419 for i, j € {1, • • • , N}. The matrix A encodes the kinematics in terms of state duration densities and transition probabilities. Often, in a practical situation, only a finite amount of training data is available and modeling can be difficult if the feature dimensionality is high. The dimension of the feature vectors described in the previous section is at least 100. Directly using the feature vectors to estimate the structure of the person and the kinematics of gait is clearly not advisable. We propose two different approaches to model the structure and kinematics of gait viz. an indirect approach and a direct approach. The choice of nomenclature will become apparent in the following discussion. 4 . 1 . Approach
1: Indirect
Approach
In this approach we pick N exemplars (or stances) £ = {ei, • • • , ejy} from the pool of images that will minimize the error in representation of all the images of that person, different starting positions). In order to do this, we divide each gait cycle into N equal segments. We pool the image features corresponding to the ith. segment for all the cycles. The centroids (essentially the mean) of the features of each part were computed and denoted as the exemplar for that part. Doing this for all the N segments gives the optimal exemplar set £ = {e|, • • • , e ^ } . The next issue is how to choose N. In problems like image compression, it is a common practice to look at the rate-distortion curves to examine the marginal reduction in the distortion as the bits per pixel are increased. We found that increasing JV beyond 5 or 6 does not lead to a significant drop in distortion. In order to reliably estimate the gait kinematics we propose a novel way of compactly encoding the observations observations. Let x(t) denote the feature extracted from the image at time t. The distance of x(i) from the corresponding exemplars e„ G £ can be computed to build a frame-to-exemplar distance (FED) vector, f(t), which serves as a lower iV-dimensional representation of the image at time t. For instance, for the jth individual we compute the nth element of the FED vector as
[ff(t)]n=d(x>(t),e>n), J
(2)
where, t £ {I,--- ,T}, e n denotes the nth exemplar of the j t h person and n e {1, • • • ,N}. Thus, fj~3(t) constitutes an observation vector for person j . Similarly, f ** (t) represents the observation sequence of the person i encoded in terms of the exemplars of person j . Note that for a frame at the start of the gait cycle, the first element of the observation vector will be smaller in magnitude as compared to the remaining four elements. As time progresses, the first element will increase in magnitude because the frame moves closer to the second stance. This temporal variation in the FED vector components corresponds precisely to the transition across exemplars. In particular it is possible to look upon the FED vector sequence fj- (t) as the observed manifestation of the transition across exemplars (a hidden process). An HMM is appropriate for such a signal. HMMs 21 use a Markov process to model the changing statistical characteristics that are manifested in the actual
420
observations. For the gait problem, the exemplars can be considered as analogues to the states of the HMM while the FED vector sequence can be considered as the observed process. Since the feature vectors are transformed to the intermediate FED vector representation, we refer to this approach as an indirect approach. In the proposed model for gait, the primary HMM parameters of interest are the number of states, the initial probability vector (ir), the transition probability matrix (A) and the output probability distribution B which we model as a continuous probability distribution. A = (A, B, it) will be used to compactly represent the HMM. In order to recognize an unknown person the FED vector sequence ff(t) is computed for all j G {l,--- ,M} for him/her using (2). The likelihood that the observation sequence fj was generated by the HMM corresponding to the j t h person can be deciphered by using the forward algorithm 21 as P^logiPifflXj))
(3)
We repeat the above procedure for every person in the database thereby producing Pj, j £ {I,--- ,M}. If the unknown person was m, Pm would be expected to be the largest among all Pj's since the distance between y and the stances of person in will be smaller than that between y and any other person. Also the pattern of transitions between stances/states for 3^ will be closest to that for person m. 4.2. Approach
2: Direct
Approach
In this approach we use the feature vector in its entirety to estimate the HMM A = (A, B, it) for each person. Hence we refer to this approach as the direct approach. One of the important issues in training is learning the observation probability B. As discussed before, the reliability of the estimated B depends on the number of training samples available and the dimension of the feature vector. In order to deal with the high dimensionality of the feature vector, we propose an alternative representation for B. As discussed in the previous section it is possible, during a gait cycle, to identify certain distinct phases or stances. We build a structural representation for a person by picking N exemplars (or stances) from the training sequence, X = {X(l), • • • , X(T)}. We now define B in terms of the distance of this vector from the exemplars as follows. bn(X(t))
= P(X(i)|e n ) = ^ e -^(X(*),o„)
(4)
The probability, P(X(t)|e n ) is defined as a function of D(X(t), e n ) , the distance of the feature vector X(t) from the nth exemplar, e„. The motivation behind using an exemplar-based model in the above manner is that the recognition can be based on the distance measure between the observed feature vector and the exemplars. During the training phase, a model is built for all the subjects in the gallery. Note that B is completely denned by £ if a and (3 are fixed beforehand. An initial estimate of £ and A is formed from X, and these estimates are refined iteratively using Expectation-Maximization 22 . An initial estimate of an ordered set
421 of exemplars from the sequence as described in Section 4.1. A corresponding initial estimate of the transition matrix, A^ (with A^ = AfJmodN+1 = 0.5, and all other A-1 = 0) is also obtained. The initial probabilities 7r„ are set to be equal to 1/N. The iterative refinement of the estimates is performed in two steps. In the first step, a Viterbi evaluation 21 of the sequence is performed using the current values for the exemplars and the transition matrix. We can thus cluster feature vectors according to the most likely state they originated from. Using the current values of the exemplars, £^ and the transition matrix, A^\ Viterbi decoding on the sequence X yields the most probable path Q = {
eii+1) = £ t e T « X(«)
(5)
Given £( t + 1 ) and A^l\ we can calculate A^+l"> using the Baum-Welch algorithm 21 . We set 7Tn — jf a t each iteration. Thus we can iteratively refine our estimates of the HMM parameters. It usually takes only a few iterations to obtain an acceptable estimate. Given the feature vector sequence of the unknown person, y, and the exemplars and HMM model parameters for the different people in the database, the likelihood that the observation sequence was produced by the j t h individual in the database is computed using the forward algorithm as Pj = log(P(^|A,)).
(6)
Note that Xj implicitly includes the exemplar set corresponding to person j . The difference between the direct and indirect methods is that in the former the feat ure vector is directly used as the observation vector for the HMM whereas in the latter, the FED is used as the observation vector. We present the results of both our methods and a comparative analysis on the USF dataset. Different probe sequences for the experiments along with the cumulative match scores are given in Table 3 for the baseline algorithm 19 , our direct and indirect approaches. The image quality for the USF database is worse than the previous two databases in terms of resolution and amount of noise. We experimented with both the width feature as well as the binarized silhouette for the USF dataset. However, the extraction of the outer contour in this case is not reliable and the width vectors were found to be noisy. In Table 3, we report only the results of our methods using the silhouettes as the image feature. From Table 3 we observe that the direct method is more robust to the presence of noise than the indirect method. We also note that the recognition performance suffers most due to differences in surface and background characteristics, and least due to difference in viewing angle. Results from other research groups using this data can be found in 8 and websites (http://degas.umiacs.umd.edu/links.html). From
422
Tables 2 and 3, we note that the HMM approach indeed surpasses the performance using the appearance matching method. Recently, the gallery in the USF database was extended by adding subjects who walked with only one shoe type on grass, which happened to be labelled as Shoe B. Since the shoe type labeling is arbitrary, they were put in the gallery to increase the gallery size to 122. The results for this case are reported in 23 . Table 3. Probe Sets and match scores for the USF database using the baseline algorithm and our indirect and direct approaches. Experiment (Probe) A(G,A,L) B (G, B, C(G,B,L) D (C, A, E (C, B, F(C,A,L) G (C, B,
R) R) R) L)
Baseline Rank 1 Rank 5 79 96 66 81 56 76 29 61 24 55 30 46 10 33
Indirect Approach Rank 5 Rank 1 91 100 81 76 65 76 61 25 29 39 46 24 15 33
Direct Approach Rank 5 Rank 1 100 99 89 90 90 78 35 65 29 65 60 18 50 24
5. View Invariant Gait Recognition The gait of a person is best reflected when he/she presents a side view (referred to in this chapter as a canonical view) to the camera. Hence, most of the above gait recognition algorithms rely on the availability of the side views of the subject. The situation is analogous to face recognition where it is desirable to have frontal views of the person's face. In realistic scenarios, however, gait recognition algorithms need to work in a situation where the person walks at an arbitrary angle to the camera. For a person walking along a non-canonical direction, appearance based features which are used for recognition get distorted. To explain this better we consider the width feature discussed earlier. Temporal plots of the width-vector for the same person walking in the canonical and non-canonical(# = 45) direction are shown in Figures 5 (a) and (b) respectively. A simple gait feature, viz. the stride length or the maximal separation of the feet, can be derived from the width plots by measuring the highest intensity in the leg regions (lower halves of the width plot). Clearly the apparent stride-length is smaller for the non-canonical view. The second effect that is obvious from the plots is a foreshortening effect as the person walks away from the camera. In order to obtain good gait recognition performance, it is necessary to correct for both of these effects through view synthesis. The most general solution to this problem is to estimate the 3-D model for the person. Features extracted from the 3-D model can then be used to provide the gait model for the person. This problem requires the solution of the structure from motion (SfM) or stereo reconstruction problems 24>25, which are known to be hard for articulating objects. In the absense of methods for recovering accurate 3-D models, a simple way to
423
exploit existing appearance based methods is to synthesize the canonical views of a walking person. In 26 , Shakhnarovich et al. compute an image based visual hull from a set of monocular views which is then used to render virtual canonical views for tracking and recognition. Gait recognition is achieved by matching a set of image features based on moments extracted from the silhouettes of the synthesized probe video to the gallery. An alternative to synthesizing canonical views is the work of Bobick and Johnson 27 . In this work, two sets of activity-specific static and stride parameters are extracted for different individuals. The expected confusion for each set is computed to guide the choice of parameters under different imaging conditions (viz. indoor vs outdoor, side-view vs angular-view etc). A cross-view mapping function is used to account for changes in viewing direction. The set of stride parameters (which is smaller than the set of static parameters) is found to exhibit greater resilience to viewing direction. Representation using such a small set of parameters may not give good recognition rates on large databases. In this section we present a view-invariant gait recognition algorithm for the single camera case. Consider a person walking along a straight line which subtends an angle 9 with the image plane (AC in Figure 4). If the distance, ZQ, of the person from the camera is much larger than the width, AZ, of the person, then it is reasonable to replace the scaling factor z ?AZ for perspective projection by an average scaling factor -^-. In other words, for human identification at a distance, we can approximate the actual 3-D human as a planar object. Assume that we are given a video of a person walking at a fixed angle 9 with a translational velocity V = [vx, 0, vz]T(Figure 4). We show that by tracking the direction of motion, a, in the video sequence, we can estimate the 3-D angle 6. Using the planarity assumption, knowing angle 9 and the calibration parameters, we can synthesize side-views of the sequence of images of an unknown walking person without explicitly computing the 3D model of the person. PROJECTION PLANE
Zl»f Fig. 4. Imaging Geometry
424
Tracking Assuming that we can find the location (xref,yref) of the persons head at the start of such a segment, we use a sequential Monte Carlo particle filter 28 to track the head of the person to get {(xl (t), yl (t)), wl (t)} where the superscript denotes the index of the particle and ro' (t) denotes the probability weight for the estimate ( ( * ' ( * ) ,
* ' ( * ) ) •
Estimation of 3-D Azimuth Angle Assume that the motion between two consecutive frames in the video sequence is small. Using the optical flow based SfM equations 29 for constant velocity models vz(t) = vz(^ 0) and vx(t) = » x ( ^ 0), cot(9{t)) = ^- and given the initial position of the tracked point (xref,yref) it can be shown that X
cot,gs
ref
=
~ yrefCOt(a(xref,
yref))
Knowing (xo, yo),cot(a) and 9, f can be computed as part of a calibration procedure. View Synthesis Having obtained the angle 6, we synthesize the canonical view. Let Z denote the distance of the object from the image plane. If the dimensions of the object are small compared to Z, then the variation in 9, d9 « 0. This essentially corresponds to assuming a planar approximation to the object. Let \Xe,Yg,Ze\' denote the coordinates of any point on the person (as shown in the Figure 4) who is walking at an angle 6 > 0 to the plane passing through the starting point [XrefYrefZref}' and parallel to the image plane which we shall refer to, hereafter, as the canonical plane. Computing the 3-D coordinates of the synthesized point involve a rotation about the line passing through the starting point followed by a perspective transformation we can obtain the equations for [xo,yo]' as _
xgcos(9) + xref(l - cos(9)) -sin(9)(xg + Xref) + f
2/o = /
r-7^-7
9
-
r—-7,
(8)
-sm(9)(xg + xref) + J where x = f— and y = f— (8) is attractive since it does not involve the 3D depth; rather it is a direct transformation of the 2D image plane coordinates in the non-canonical view to get the image plane coordinates in the canonical one. Thus using the estimated azimuth angle 9 we can obtain a synthetic canonical view using (8). View synthesis provides for a correction of both the foreshortening and distortion of stride length (see Figures 5 (c) and (d)) and improves the gait recognition performance appreciably. 5.1. Gait-based
Recognition
We present gait recognition results on two databases. U M D 3 database: This consists of 12 people, who walk along straight lines at different values of azimuth angle 9 = 0,15,30 and 45. The image sequences
425
Fig. 5. Width profile as a function of time for (a) Canonical View (0 = 0); (b)Unnormalized sequence for 0 — 45; synthesized views for: (c) 6 = 30 (d) 6 = 45
corresponding to 0 = 0 were used as the gallery while the other sequences were used as a probe. The width profile plot for the canonical view and the view synthesized from 0 = 45 are shown in Figure 5. As can be seen from this plot, our method has compensated for both the foreshortening effect as well as restored the true legswing. Two consecutive cycles in the canonical view are chosen as the gallery to be compared with two consecutive gait cycles in the probe sequence. The DTW technique is used to match a given probe sequence to the different gallery sequences using binary correlation as a local distance measure and a similarity matrix S = s(i,j) is obtained, where s(i,j) refers to the similarity between the probe i and the gallery j . Gait recognition performance for 0 = 30 and 45° is shown in Figures 6(a) and (b) using the synthesized and raw images in terms of a cumulative match characteristic. As noted before, the algorithm results in a broader reproduction of the torso region. The situation can be remedied by assigning a lower weight to the torso region when computing the binary correlation or simply ignoring it. We take the latter approach by computing the binary correlation only over the lower half of the boxed image. The result using only the leg region is shown as the dashed lines in the Figure 6. It can be seen that the gait recognition result is better than what is obtained by using the entire body. Interestingly, 19 notes that the lower 20 % of the silhouette accounts for roughly 90% of the recognition. To boost the gait recognition performance further, certain structural characteristics of the individual that are extracted subsequent to view synthesis e.g. height can be fused with the leg dynamics. The height of the probe sequence is estimated robustly from the synthesized video as h{i) = medianfcJ'(i),j = 1 • • • M M, being the length of the probe sequence. We fuse height information together with the leg dynamics by scaling each entry s(i,j) of the similarity matrix by the corresponding height
426
ratio, max(£!4,2 - j | | ) . The results for this case are shown as the solid line with circles in Figure 6. The fact that the gait recognition results are encouraging upto angles of 45 degrees allows us to hypothesize that it is possible to do reasonable human identification using gait with only two cameras (installed perpendicular to each other).
30
40 With Leg and height Fusion With leg only With complete synthesized images Unnormalized Images
4
6
With Leg and height Fusion With leg only With complete synthesized images Unnotmalized Images 8
10
10
Rank — >
(a)
(b)
Fig. 6. Cumulative Match Characteristics for Original and Synthesized images for (a) 9 = 30 (b) 9 = 45 for UMD3 database N I S T database: This consists of 30 people walking along a S -shaped walking pattern as shown in Figure 7(a). There are two cameras looking at the top horizontal part of the sigma. The camera that is located further away is used in our experiments since the planar approximation we make is more valid in that case. The segment of the sigma next to the top horizontal part is used as a probe. This segment is at an angle 33° to the horizontal part. To do gait recognition we employed the fusion of the leg-dynamics with the height since it gave the best performance for Database 1. The gait recognition result is shown in Figure 7(b). As can be seen the recognition rate is about 60%. One of the reasons for the lower recognition performance in this case is that the image size is rather small. Note however that the recognition goes to 100% within 6 ranks.
6. Conclusions and Future Work In this chapter we investigated the information contained in the video sequences of human gait and how to extract and represent that information in ways that facilitate human identification. Human identification using gait, similar to text-based speaker identification, involves different individuals performing the same task and a template-matching approach is suitable for such problems. In situations where the amount of training data is limited, we showed the utility of a simple feature viz. the width of the outer contour of the binarized silhouette of the subject and its
427 Gallery 100
20 Performance using synthesized images Performance using unnormalized Images 10 \J
(a)
Camera
15 Rank
20 >
25
30
(b)
Fig. 7. (a)£ shaped walking pattern in the NIST database (b)Gait recognition performance on the NIST database
derivatives for gait recognition in a dynamic time warping framework. By virtue of their deterministic nature, template matching methods have limited noise resilience. To improve robustness a systematic approach to gait recognition by building representations for the structural and dynamic components of gait using exemplars and hidden Markov models (HMMs) was discussed. Gait can serve as a cue for recognizing people if the database is small. But for large databases, gait information, by itself, may not be sufficient to recognize an individual. In fact, we must realize that the gait recognition capability of even humans is limited. However, it can be used to narrow down the list of potential matches. In a recent paper 3 0 we demonstrated the use of gait as a filter to achieve faster human identification by limiting the number of candidates being passed to a more accurate face recognition algorithm. Gait can also be used in conjunction with other cues such as the color of clothing etc. for short time verification problems viz "was this the same person who walked in front of this camera t minutes ago?". Finally, a view invariant gait recognition algorithm which is based on synthesizing a side view of a person from an arbitrary monocular view was discussed. We also presented a view invariant gait recognition algorithm. The fact that the gait recognition results are encouraging upto angles of 45 degrees allows us to hypothesize that it is possible to do reasonable human identification using gait with only two cameras (installed perpendicular to each other). This could prove to be less restrictive than the visual hull approach that needs at least 4 cameras. For indoor multiple camera settings it would also be interesting to study the contribution of 3-D information for gait recognition using multi-camera kinematic models 3 1 , 3 2 .
428
References 1. G Johansson, "Visual motion perception," Scientific American, vol. 232, pp. 76-88, 1975. 2. L. Kozlowski and J. Cutting, "Recognizing the sex of a walker from a dynamic point display," Perception and Psychophysics, vol. 21, pp. 575-580, 1977. 3. J. Cutting and L. Kozlowski, "Recognizing friends by their walk:gait perception without familiarity cues," Bulletin of the Psychonomic Society, vol. 9, pp. 353-356, 1977. 4. C D . Barclay, J. E. Cutting, and L.T. Kozlowski, "Temporal and spatial factors in gait perception that influence gender recognition," Perception and Psychophysics, vol. 23, pp. 145-152, 1978. 5. D. Cunado, J.M. Nash, M.S. Nixon, and J. N. Carter, "Gait extraction and description by evidence-gathering," Proc. of the International Conference on Audio and Video Based Biometric Person Authentication, pp. 43-48, 1995. 6. P.S. Huang, C.J. Harris, and M.S. Nixon, "Recognizing humans by gait via parametric canonical space," Artificial Intelligence in Engineering, vol. 13, no. 4, pp. 359-366, October 1999. 7. R. Cutler C. Benabdelkader and L.S. Davis, "Motion based recognition of people in eigengait space," Proceedings of the IEEE Conference on Face and Gesture Recognition, pp. 267-272, 2002. 8. D. Tolliver and R. Collins, "Gait shape estimation for identification," Proceedings of AVBPA, pp. 734-742, 2003. 9. L. Lee and W.E.L. Grimson, "Gait analysis for recognition and classification," Proceedings of the IEEE Conference on Face and Gesture Recognition, pp. 155-161, 2002. 10. J. Hayfron Acquah, M.S. Nixon, and J.N. Carter, "Automatic gait recognition by symmetry analysis," Pattern Recognition Letters, pp. 2175-2183, 2003. 11. J. Han and B. Bhanu, "Statistical feature fusion for gait-based human recognition," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2004. 12. A. Elgammal, D. Harwood, and L. Davis, "Non-parametric model for background subtraction," FRAME-RATE Workshop, IEEE, 1999. 13. R.T. Collins Y. Liu and Y. Tsin, "Gait sequence analysis using frieze patterns," Proceedings of the European Conference on Computer Vision, vol. 2, pp. 657—671, 2002. 14. Sadaoki Furui, "Cepstral analysis technique for automatic speaker verification," IEEE Trans. Acoust., Speech, Signal Processing, vol. 29, no. 2, pp. 254-272, April 1981. 15. H. Sakoe and S. Chiba, "Dynamic programming algorithm optimization for spoken word recognition," IEEE Transactions on Acoustics, Speech and Signal Processing, vol. ASSP-26 no. 1, pp. 43-49, 1978. 16. Jinu Mariam Zacharia, "Text-independent speaker verification using segmental, suprasegmental and source features," M.S. thesis, IIT Madras Chennai India, March 2002. 17. A. Kale, N. Cuntoor, B. Yegnanarayana, A.N. Rajagopalan, and R. Chellappa, "Gaitbased human identification using appearance matching," in Optical and Digital Techniques for Information Security, B. Javidi, Ed. Springer Verlag, December 2004. 18. P. J. Phillips, S. Sarkar, I. Robledo, P. Grother, and K. W. Bowyer, "The gait identification challenge problem: Data sets and baseline algorithm," Proc of the International Conference on Pattern Recognition, 2002. 19. P. J. Phillips, S. Sarkar, I. Robledo, P. Grother, and K. W. Bowyer, "Baseline results for the challenge problem of human id using gait analysis," Proc. of the 5th IEEE Int. Conf. on Automatic Face and Gesture Recognition, 2002. 20. A. Veeraraghavan, A. RoyChowdhury, and R. Chellappa, "Role of shape and kinemat-
429
21. 22.
23.
24. 25. 26.
27.
28. 29. 30. 31.
32.
ics in human movement analysis," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2004. L.R. Rabiner, "A tutorial on hidden markov models and selected applications in speech recognition," Proceedings of the IEEE, vol. 77, no. 2, pp. 257-285, February 1989. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the em algorithm," Journal of the Royal Statistical Society, vol. 39, no. 1, pp. 1-38, 1977. A. Kale, A. Sundaresan, A. N Rajagopalan, N. Cuntoor, A. RoyChowdhury, V.Kruger, and R. Chellappa, "Identification of humans using gait," IEEE Transactions on Image Processing, July 2004. O.D. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint, MIT Press, 1993. R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2000. G.Shakhnarovich, L.Lee, and T.Darrell, "Integrated face and gait recognition from multiple views," Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, December 2001. A.F. Bobick and A. Johnson, "Gait recognition using static activity-specific parameters," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2001. Michael Isard and Andrew Blake, "Contour tracking by stochastic propagation of conditional density," Proceedings of ECCV, , no. 1, pp. 343-356, 1996. Vishwjit Nalwa, A Guided Tour of Computer Vision, Addison-Wesley, 1993. A. Kale, A. RoyChowdhury, and R. Chellappa, "Fusion of gait and face for human identification," Proc. of the ICASSP, 2004. A. Sundaresan, A. RoyChowdhury, and R. Chellappa, "3d modelling of human motion using kinematic chains and multiple cameras for tracking," Proc. of Eighth International Symposium on the 3-D Analysis of Human Movement, Tampa, 2004. C. Bregler and J. Malik, "Tracking people with twists and exponential maps," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 1998.
CHAPTER 4.3 PALMPRINT AUTHENTICATION SYSTEM
David Zhang Biometrics Research Centre (UGC/CRC), Department of Computing Hong Kong Polytechnic University, Kowloon, Hong Kong E-mail: [email protected]. edu. hk
In this chapter, we present a novel biometric authentication system to identify a person's identity by his/her palmprint. In contrast to existing palmprint systems for criminal applications, the proposed system targets at the civil applications, which require identifying a person in a large database with high accuracy in real-time. The system is constituted by four major components: User Interface Module, Acquisition Module, Recognition Module and External Module. More than 7,000 palmprint images have been collected to test the performance of the system. The system can identify 400 palms with a low false acceptance rate, 0.02%, and a high genuine acceptance rate, 98.83%. For verification, the system can operate at a false acceptance rate, 0.017% and a false rejection rate, 0.86%. The execution time for the whole process including image collection, preprocessing, feature extraction and matching is less than 1 second..
1. Introduction Biometrics lies in the heart of today's society. There has been an ever-growing need to automatically authenticate individuals at various occasions in our modern and automated society, such as information confidentiality, homeland security, and computer security. Traditional knowledge based or token based personal identification or verification is so unreliable, inconvenient, and inefficient, that it is incapable to meet such a fast-pacing society. Knowledge-based approaches use "something that you know" to make a personal identification, such as password and personal identity number. Token-based approaches use "something that you have" to make a personal identification, such as passport or ID card. Since those approaches are not based on any inherent attributes of an individual to make the identification, it is unable to differentiate between an authorized person and an impostor who fraudulently acquires the "token" or "knowledge" of the authorized person. This is why biometric identification or verification systems have received more attention in recent years. 431
432
I Enrollment Path User Interface Module
w^
Acquisition Module
Palmprint Capture
Preprocessing
41
Recognition Module
&
Feature Extraction
41
I Identification 1/
External Module
41
Template Creation
~~T>
Matching
rftV> Result
T Master Temphi Database
[
Fig. 1. The breakdown of each module of the palmprint authentication system.
Biometrics involves identifying an individual based on his/her physiological or behavioral characteristics. Many parts of our body and various behaviors are embedded with such information for personal identification. In fact, using biometrics for person authentication is not new, which has been implemented over thousands years, numerous research efforts have been put on this subject resulting in development of various techniques related to signal acquisition, feature extraction, matching and classification. Most importantly, various biometric systems including, fingerprint, iris, hand geometry, voice and face recognition systems have been deployed for various applications [1]. According to the International Biometric Group (IBG, New York), the market for biometric technologies will nearly double in size this year alone. Among all biometrics, hand-based biometrics, including hand geometry and fingerprint are the most popular biometrics gaining 60% market share in 2003 [2]. The proposed palmprint system is also a hand-based biometric technology. Palmprint is concerned with the inner surface of a hand and looks at line patterns and surface shape.
433
A palm is covered with the same kind of skin as the fingertips and it is larger than a fingertip in size. Therefore, it is quite natural to think of using palmprint to recognize a person, like fingerprint, hand geometry and hand vein [3-6]. Because of the rich features including texture, principal lines and wrinkles on palmprints. we believe that they contain enough stable and distinctive information for separating an individual from a large population. We also expect that palmprints are robust to the noise because of the large surface area. Several companies, including NEC and PRINTRAK. have developed palmprint systems for criminal applications [8-9]. On the basis of fingerprint technology, their systems exploit high resolution palmprint images to extract the detailed features like minutiae for matching the latent prints. Such approach is not suitable for developing a palmprint authentication system for civil applications, which requires a fast, accurate and reliable method for personal identification. Based on our previous research work [10], we develop a novel palmprint authentication system to fulfill such requirements. The rest of the chapter is organized as follows. The system framework is shown in Section 2. The recognition engine is described in Section 3. Experimental results of verification, identification, robustness, and computation time are provided in Section 4. Finally, conclusion is given in Section 5.
Fig. 2. The palmprinl authentication system installed al Biometric Research Center (UGC/CRCO. The Hong Kong Polytechnic University.
2. System Framework The proposed palmprint authentication system has four major components: User Interface Module, Acquisition Module, Recognition Module and External Module. Fig. 1 shows the breakdown of each module of the palmprint authentication system. Since March 2003, this palmprint authentication system has been installed at Biometric Research Center. Department of Computing. The Hong Kong Polytechnic University (see Fig. 2). The functions of each component are listed below:
434
A)
User Interface Module provides an interface between the system and users for the smooth authentication operation. We designed a flat platen surface for the palm acquisition. It is crucial to develop a good user interface such that users are happy to use the device. B) Acquisition Module is the channel for the palmprints to be acquired for the further processing. C) Recognition Module is the key part of our system, which will determine whether a user is authenticated. It consists of image pre-processing, feature extraction, template creation, database updating, and matching. Then it gives an identification/ verification result. D) External Module receives the signal coming from the recognition module, to allow some operations to be performed, or denied the operations requested. This module actually is an interfacing component, which may be connected to another hardware components or software components. Our system presents an external interface for physical door access control or an employee attendance system. Since the design philosophy and implementation of the user interface module and the acquisition module have been described in [10], and the external interface is an application dependent component, we are not intended in this chapter to discuss them further, and we will concentrate on the discussion about the recognition module in detail.
3. Recognition Engine After the palmprint images are captured by the Acquisition Module, they are fed into the recognition engine for palmprint authentication. The recognition engine is the key part of the palmprint authentication system, which consists of the stages of: image preprocessing, feature extraction, and matching. 3.1. Image Pre-processing When capturing a palmprint, the position, direction and stretching degree may vary from time to time. As a result, even the palmprints from the same palm could have a little rotation and translation. Also the sizes of palms are different from one another. So, the preprocessing algorithm is used to align different palmprints and extract the corresponding central part for feature extraction [7]. In our palmprint system, both the rotation and translation are constrained to some extent by the capture device panel, which can locate the palms by several pegs. Then the preprocessing algorithm can locate the coordination system of the palmprints quickly by the following five steps: Step 1: Use a threshold to convert the original grayscale image into a binary image, then using a low-pass filter to smooth the binary image. Step 2: Trace the boundary of the holes HI and H2 between those fingers (see Fig. 3). Step 3: Compute the tangent of the holes HI and H2. Tl and T2 are the tangent points of HI and 7/2, respectively. Step 4: Aign 77 and T2 to determine the Y-axis of the palmprint coordination system and making a line passing through the midpoint of the two points (77 and T2), which is perpendicular to this Y-axis to determine the origin of the system.
435
Step 5: Extract a sub-image of a fixed size based on the coordinate system (see Fig. 3). The sub-image is located at a certain area of the palmprint image for feature extraction.
Fig. 3. The major steps on palmprint preprocessing: HI and H2 are boundary of the holes between the two fingers, where Tl and T2 are the tangent point of/// and H2. respectively. The central bo.x is the sub-image of a palm.
3.2. Feature Extraction The feature extraction technique implemented on the proposed palmprint system is modified from [10], where single circular zero DC Gabor filter is applied to the preprocessed palmprint images and the phase information is coded as feature vector called PalmCode. The modified technique exploited four circular zero DC Gabor filters with the following general formula: GD =
1
T-exp 2no~
(* ~x0)
, (>• - v,,)-
(7
a
>|exp(/Ixcax )-exp(-2;r~rtr
(1)
where, x = .\cost9+vsint? and y =-xsinf? + y c o s # : (x(h yu) is the center of the function in the spatial domain of the function; co is the frequency of the sinusoidal plane wave along the orientation, 9: a is the standard deviations of the circular Gaussian function; 9 is the direction of the filter. The four Gabor filters share the same parameters, a and co, only different in 9. The corresponding values of 9 are 0, TI/4,rc/2and 37t/4. In the previous approach, only the phase information is exploited but the magnitude information is totally neglected. The proposed method is to use the magnitude to be a fusion condition to combine different PalmCodes generated by the four Gabor filters. Mathematically, the implementation has the following steps. 1) The four Gabor filters are applied to the preprocessed palmprint image. / described as G*I. where G, (/—1, 2. 3. 4) is the circular zero DC Gabor filter and "*" represents an operator of convolution. 2) The square of the magnitudes of the sample point is obtained by M,{x,y) = Gj(.x,y)*IxG,(x,y)* I, where "—" represents complex conjugate. 3)
According to the fusion rule, k = argmax(M/(.Y,>')), the phase information at point / (x. y) is coded as the followings
436
hr=l
if
Re[G A */]>0,
(2)
hr=0
if
Re[Gt*/]<0,
(3)
h, =1 //
Im[G,*/]>0,
(4)
h, = 0 //
Im[GA. * /] < 0 .
(5)
This coding method is named as Fusion Code, which is represented by a set of bits. Our experiments show that the Fusion Code is more stable and efficient for palmprint authentication. 3.3. Feature Matching The feature matching determines the degree of similarity between two templates - the identification template and the master template. In this paper, the normalized hamming distance is implemented for comparing two Fusion Codes. The normalized hamming distance is represented by: ,
N
X Xp"(:'j)
n Q (; j) n ( P ( ; j ]
" -
( « - ® QK (i> j ) + p i ( i - j ) ® Q'(/- J)))
where PR (QR), PI (QI) and PM(QM) are the real part, imaginary part and mask of the Fusion Code P(Q), respectively; <8> and n are Boolean operators, XOR and AND, respectively [10]. The ranges of normalized hamming distances are between zero and one. Zero represents perfect matching. Because of the imperfect preprocessing, one of the Fusion Code is vertically and horizontal translated to match the other again. The ranges of the vertical and the horizontal translations are defined from -2 to 2. The minimum D0 value obtained from the translated matching is considered to be the final matching score.
4. Performance Evaluation 4.1. Testing Database We collected palmprint images from 200 individuals using our palmprint capture device described in [10]. The subjects are mainly students and staff volunteers from The Hong Kong Polytechnic University. In this dataset, 134 people are male, and the age distribution of the subjects is: about 86% are younger than 30, about 3% are older than 50, and about 11% are aged between 30 and 50. In addition, we collected the palmprint images on two separate occasions. On each occasion, the subject was asked to provide about 10 images each of the left palm and the right palm. Therefore, each person provided around 40 images, resulting in a total number of 8,025 images from 400 different palms in our database. In addition, we changed the light source and adjusted the focus of the CCD camera so that the images collected on the first and second occasions could be regarded as being captured by two different palmprint devices. The average time interval between the first and second occasions was 70 days. The size of all the testing
437 images used in the following experiments is 384x284 with 75dpi, in 256 gray levels.
I
1
1
S a 4 CD
S. 3
0 1
0.2
0.3 0.4 Hamming distance
0.5
(a)
7
f1
6
•-
J-5 0} ay
2 4
-
-
£= OJ
i
u
°- 3 2
/ i
1
0
0.1
0 2
0.3 0.4 Hamming distance
1,
0.5
0.6
(b) Fig. 4. Verification test results, (a) and (b) show the Genuine and imposter distributions for verification tests with one and three registered images per palm, respectively.
4.2. Experimental
Results of Verification
Verification refers to the problem of confirming or denying a claim of individuals. It is also considered as one to one matching. To obtain the verification result, two group experiments are carried out separately. In the first experiment, only one palmprint image of each palm is used for registration, while 3 palmprint images of each palm are used for registration in the second experiment. In the first experiment, each palmprint image is
438 matched with all other palmprint images in the database. A correct matching occurs if two palmprint images are from the same palm, incorrect matching otherwise. The total number of matching is 32,119,735. Number of genuine matching is 76,565 and the rest of them are incorrect matchings. Fig. 4(a) shows the probability of genuine and imposter distributions estimated by the correct and incorrect matchings. Some thresholds and corresponding false acceptance rates (FARs) and false rejection rates (FRRs) are listed in Table 1. According to Table 1, using one palmprint image for registration, the proposed system can be operated at a low false acceptance rate 0.096% and a reasonably low false rejection rate 1.05%. Table 1. False acceptance rates (FARs) and false rejection rates (FRRs) with different threshold values for the palmprints verification results.
Threshold 0.32 0.34 0.36 0.38 0.40
Registered image= 1 FAR(%) FRR(%) 0.000027 8.15 0.00094 4.02 0.011 1.94 0.096 1.05 0.68 0.59
Registered images=3 FAR(%) FRR (%> 0.000012 5.12 0.0016 2.18 0.017 0.86 0.15 0.43 1.03 0.19
Table 2. FARs and FRRs with different threshold values for the l-to-400 palmprints identification results.
Threshold 0.320 0.325 0.330 0.335 0.340 0.345
Trial=1
\R(%) FRR(%) 3.69 0.0049 2.93 0.0439 2.29 0.15 0.37 1.90 0.84 1.51 1.16 1.45
Trial=2
FAR (%) 0.0098 0.088 0.28 0.68 1.43 2.32
FRR(%) 1.80 1.34 1.02 0.72 0.57 0.42
Trial=3
FAR (%) 0.020 0.131 0.42 0.96 1.93 3.02
FRR (%) 1.17 1.06 0.68 0.48 0.37 0.26
In the second experiment, the testing database is divided into two databases, 1) registration database and 2) testing database. Three palmprint images of each palm collected in the first occasion are selected for the registration database. Totally, the registration database contains 1,200 palmprint images and the rest of them are for the testing database. In this verification test, each palmprint image is matched with all the palmprint images in the testing database. Therefore, each testing image produces three hamming distances for one registered palm. We take the minimum of them as the final hamming distance. For achieving statistically reliable results, this test is repeated for three times by selecting other palmprint images for the registration database. Total number of hamming distances from correct matchings and incorrect matchings are 20,475 and 8,169,525, respectively. Fig. 4(b) shows the probability of genuine and imposter distributions estimated by the correct and incorrect matchings, respectively. Some threshold values along with its corresponding false acceptance and false rejection rates are also listed in Table 1. According to Table 1 and Fig. 4, we can conclude that using three templates can provide better verification accuracy. In fact, using more palmprint
439
images of the same palm during registration can provide more information to the system so that it can recognize the noise or deformed features. It is also the reason for commercial verification systems requiring more than one sample for registration. 100 99.5 qq
#< <1>
cr 9B.5 m
>CU) r
98
m
() n
Wb
n ID
o
y/ 9G.5-
10'3
- 10"2
1 aginst 400 identification based on 1 trial 1 aginst 400 identification based on 2 trial 1 aginst 400 identification based on 3 trial
10'' 10° False Acceptance Rate (%)
10'
102
Fig. 5. The ROC curves for a 1-against-400 identification testing with different number of trials.
4.3. Experimental Results of Identification Identification test is a one-against-many, N comparison process. In this experiment, N is set to 400, which is the total number of different palms in our database. Same as the previous verification experiment, the palmprint database is divided into two databases, 1) registration and 2) testing databases. The registration database contains 1,200 palmprint images, three images per palm. The testing database has 6,825 palmprint images. Each palmprint image in the testing database is matched to all of the palmprint images in the registration database. Therefore, each testing image generates 3 correct and 1,197 incorrect matchings. The minimum hamming distances of correct matchings and incorrect matchings are regarded as the identification hamming distances of genuine and impostor, respectively. This experiment is also called a one-trial test since the user only provides one palmprint image in the test to make one decision. In fact, a practical biometric system collects several biometric signals to make one decision. Therefore, in this experiment, we implement one, two, and three-trial tests. In the two-trial test, a pair of palmprint images in the testing database belongs to the same palm is matched to all of the palmprint images in the registration database. Each pair of the palmprint images in the two-trial test generates 6 correct and 2,394 incorrect matchings. The minimum hamming distances of correct matchings and incorrect matchings are considered as the identification hamming distances of genuine and imposter, respectively. Similarly, in the three-trial test, the identification hamming distances of genuine and imposter are obtained from 9 correct and 3,591 incorrect matchings, respectively. Each test is repeated three
440 times by selecting other palmprints from the registration database. In each test, the number of identification hamming distances of genuine and imposter matchings both are 20,475. Fig. 5 shows ROC curves of the three tests and Table 2 lists the threshold values along with its corresponding FARs and FRRs of the tests. According to Fig. 5 and Table 2, more input palmprints can provide more accurate results.
(e)
(D
Fig. 6. Three palmprint images with and without rings on the fingers and their corresponding pre-processed sub-images.
4.4. Computation Time Another key issue for a civilian personal identification system is whether the system can run in real time. In other words, the system should be running as fast as possible. The proposed method is implemented using C language and Assemble language on a PC
441
embedded Intel Pentium IV processor (1.4GHz) with 128M memory. The execution time for image collection, image preprocessing, feature extraction and matching are listed in Table 3. The total execution time for a l-against-400 identification, each palm with 3 templates, is less than 1 second. Users will not feel any delay when using our system. Table 3. Execution time of the paimprint authentication system.
Operations Image collection Preprocessing Feature extraction Matching
Execution Time 340ms 250ms 180ms 1.3jus
Fig. 7. Paimprint images with different texts for testing the robustness of the system.
442 Table 4. The hamming distances of Fig. 7
Figs 6(a) 6(b) 6(c) 6(d) 6(e)
6(b) 0.19
'(c)
6(c) 0.21 0.18
6(d) 0.27 0.27 0.27
6(e) 0.29 0.26 0.28 0.23
6(f) 0.28 0.27 0.28 0.19 0.19
(d)
Fig. 8. Identical twins" palmprints. (a), (b) are their left hands, and (c), (d) are their right hands, respectively.
4.5. Robustness As a practical biometric system, other than accuracy and speed, robustness of the system is another important issue. Here, we present three experiments to illustrate the robustness of our system. The first one is an individual's jewelry such as rings, which may influence the accuracy of some preprocessing algorithms. The second one is the noise on the palmprints, which directly affect the performance of the system. The third is the ability of identifying the identity of identical twins' palmprints. Fig. 6 shows three palmprint images with and without rings on the fingers and their
443
corresponding preprocessed sub-images. It shows that the preprocessing algorithm described in Section 3 is not affected by the rings. However, some preprocessing algorithms such as [16] using such information may not be stable under the influence of the rings. To verify the robustness of noise palmprints, Fig. 7(a) provides a clear palmprint image and Figs. 7(b)-(f) show five palmprint images, with different texts. Their hamming distances are given in Table 3; all of them are smaller than 0.29. Comparing the hamming distances of imposter in Tables 1 and Tables 2, it is ensured that all the hamming distances in Table 4 are relatively small. Fig. 7 and Table 4 illustrates that the proposed palmprint authentication system is very robust to the noise on the palmprint. A test of identical twins is regarded as an important test for biometric authentication, but not all biometrics, including face and DNA, can pass this test. However, the palmprints of identical twins have enough distinctive information to distinguish them. We collected 590 palmprint images from 30 pairs of identical twins' palms. Each of them provides around 10 images of their left palms and 10 images of their right palms. Their age range is between 6 and 45 years old. Some samples of identical twins' palmprints are shown in Fig. 8. Based on this database, we match a palmprint in the twin database with his/her identical twin sibling to produce imposter matching scores, and match the samples of their own to get the genuine scores. The genuine and imposter distributions are given in Fig. 9. From the figure, we can find that identical twins' palmprint can easily be separated, just like twins' fingerprints [11].
Genuine imposter
£4
V. j \ D
0.1
0.2
0.3 Matching scores
0.4
0.5
Fig. 9. The genuine and imposter distributions for measuring the similarity of identical twins' palmprints.
5. Conclusions In this chapter, we have presented a novel biometric system based on the palmprint. The proposed system can accurately identify a person in real time, which is suitable for
444 various civil applications such as access control. Experimental results show that the proposed system can identify 400 palms with a low false acceptance rate, 0.02%, and a high genuine acceptance rate, 98.83%. For verification, the system can operate at a false acceptance rate, 0.017% and a false rejection rate, 0.86%. The experimental results including accuracy, speed and robustness demonstrate that the palmprint authentication system is comparable with other hand-based biometrics systems, such as hand geometry and fingerprint verification system [12-14] and is practical for real-world applications. The system has been installed at the Biometric Research Center, Department of Computing, The Hong Kong Polytechnic University since March 2003 for access control. Acknowledgements This research is partially supported by the UGC/CRC fund from the Hong Kong SAR Government and the central fund from The Hong Kong Polytechnic University. Thank my team members, Guangming Lu, Adams Kong, and Michael Wong for their hard work and unstinting support. References 1. 2. 3.
4.
5. 6.
7. 8. 9. 10.
11. 12. 13. 14.
A. Jain. R. Bolle and S. Pankanti (eds.). Biometrics: Personal Identification in Networked Society. Boston. Mass: Kluvver Academic Publishers. 1999. International Biometric Group's Biometric Market Report 2003-2007: http://w\vw.biometricgroup.com/reports/public/market report.html. R. Sanchez-Reillo. C. Sanchez-Avilla and A. Gonzalez-Marcos. ""Biometric identification through hand geometry measurements". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22. no. 10. pp. 1168-1171. 2000. S.K. Im. H.M. Park. Y.W. Kim, S.C. Han. S.W. Kim and C.H. Kang. "An biometric identification system by extracting hand vein patterns". Journal of the Korean Physical Society, vol. 38.' no. 3. pp. 268-272."2001. A. Jain. L. Hong and R. Bolle. "On-line fingerprint verification'". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19. no. 4. pp. 302-314. 1997. A.K. Jain. A. Ross and S. Prabhakar. ""An introduction to biometric recognition". IEEE Transactions on Circuits and Systems for I 'ideo Technology, Special Issue on Image- and Video- Based Biometrics, vol. 14. no. 1. January 2004. G. Lu. D. Zhang and K. Wang. "Palmprint Recognition Using Eigenpalms Features'". Pattern Recognition Letters. 24. pp. 1473-1477. 2003. http://w\v\v.nectech.com/afis/'download./PalmprintDtsht.q.pdf - NEC automatic palmprint identification system. http://www.printrakintemational.eortvomnitrak.htrn - Prinkrak palmprint identification system. D. Zhang. W.K. Kong. J. You and M. Wong. ""On-line palmprint identification"'. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25. no. 9. pp. 1041-1050. 2003. A. K. Jain. S. Prabhakar and S. Pankanti. "On the similarity of identical twin fingerprints." Pattern Recognition, vol. 35. no. 11. pp 2653-2662. 2002. A.K. Jain. S. Prabhakar. L. Hong and S. Pankanti. "Filterbank-based fingerprint matching". IEEE Transactions on Image Processing, vol. 9. no. 5. pp. 846-859. 2000. A.K. Jain. L. Hong and R. Bolle. "On-line fingerprint verification". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19. no. 4. pp. 302-314. 1997. R. Sanchez-Reillo. C. Sanchez-Avilla and A. Gonzalez-Marcos. "Biometric identification through hand geometry measurements". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22. no. 18. pp.1168-1171. 2000.
C H A P T E R 4.4 R E C O N S T R U C T I O N OF HIGH-RESOLUTION FACIAL IMAGES FOR VISUAL SURVEILLANCE
Jeong-Seon Park and Seong-Whan Lee Korea University Anam-dong, Seongbuk-ku, Seoul 136-701, Korea E-mail: {jspark, swlee} ©image.korea.ae.hr
In this chapter we provide a summary of our previous works concerning the reconstruction of high-resolution facial images for visual surveillance. Specifically we present our methods of reconstructing high-resolution facial image from a low-resolution facial image based on example-based learning and iterative error back-projection. In our method, a face is represented by a linear combination of prototypes of shape and texture. With the shape and texture information about the pixels in a given low-resolution facial image, we can estimate optimal coefficients for a linear combination of prototypes of shape and those of texture by solving least square minimization. Then high-resolution facial image can be obtained by using the optimal coefficients for linear combination of the high-resolution prototypes. Moreover iterative error back-projection is applied to improve the result of high-resolution reconstruction. The encouraging results of our methods show that our high-resolution reconstruction methods can be used to improve the performance of the face recognition by enhancing the resolution of low-resolution facial images captured in visual surveillance systems.
1. I n t r o d u c t i o n There is a growing interest in visual surveillance systems for security areas such as international airports, borders, sports grounds, and safety areas. And various researches on face recognition have been carried out for a long time. B u t there still exist a number of difficult problems such as estimating facial pose, facial expression variations, resolving object occlusion, changes of lighting conditions, and in particular, the low-resolution images captured in visual surveillance systems(see Fig. 1). Handling low-resolution images is one of t h e most difficult and commonly occurring problems in various image processing applications such as analysis of scientific, medical, astronomical, and weather images, archiving, retrieval and transmission of those images as well as video surveillance or monitoring. 1 Numerous methods have been reported in t h e area of estimating or reconstructing high-resolution im445
446
Fig. 1.
Example of low-resolution facial image captured in visual surveillance systems
ages from a series of low-resolution images or single low-resolution image. Superresolution is a typical example of techniques reconstructing a high-resolution image from a series of low-resolution images, 2 - 4 whereas interpolation produces a large image from only one low-resolution image. We are concerned with building a high-resolution facial image from a lowresolution facial image for visual surveillance. Our task is distinguished from previous works that built high-resolution images mainly from scientific images or image sequence of video data. Our reconstruction method is example-based, object-classspecific or top-down approach. It is highly tolerant to sensor noise, 5 incompleteness of input images and occlusion by other objects. The example-based approach to interpreting images of deformable objects are now attracting considerable interest among many researchers. 6,7 The motivation for example-based learning lies in its potential of deriving highlevel knowledge from a set of prototypical examples. This chapter presents our reconstruction methods of high-resolution facial image from a low-resolution facial image based on the iterative error back-projection of example-based learning. The extended morphable face model 11 is used in example-based learning, and a mathematical procedure for solving least square minimization (LSM) is applied to the model. More over, iterative error back-projection procedure is applied to improve the result of high-resolution reconstruction. This paper is organized as follows. Section 2 explains the overview of our reconstruction method based on extended morphable face model. In Sec. 3, we explain the procedure of high-resolution shape estimation using example-based learning, the problem of high-resolution reconstruction and solution to solve the least square minimization of reconstruction error. Then, in the next section, we describe our iterative error back-projection procedure which is composed of estimation, simulation and error compensation. In Sec. 5, experimental results with low-resolution facial
447
images are provided along with an analysis of these results. Finally, conclusions and future works will be discussed in Sec. 6. 2. Overview of High-resolution Reconstruction In this section, we present an overview of our reconstruction methods using examplebased learning based on extended morphable face model. Suppose that sufficiently large amount of facial images are available for off-line training, then we can represent any input face by a linear combination of a number of facial prototypes. 9 Moreover, if we have a pair of low-resolution facial image and its corresponding high-resolution image for the same person, we can obtain an approximation to the deformation required for the given low-resolution facial image by using the coefficients of examples. Then we can obtain a high-resolution facial image by applying the estimated coefficients to the corresponding high-resolution example faces(see Fig. 2). Consequently, our goal is to find an optimal parameter set a which best estimates the high-resolution facial image from a given low-resolution facial image.
no Prototype 1
+
no
Prototype 1 Fig. 2.
Prototype 2
Prototype M
Basic idea of the high-resolution reconstruction based on example-based learning
2.1. Extended
morphable
face
model
Our reconstruction method is based on the morphable face model introduced by Poggio et al.s and developed further by Vetter et a/..9,10 Assuming that the pixelwise correspondence between facial images has already been established, 9 the 2D shape of a face is coded as the displacement fields from a reference face. The shape of a facial image is represented by a vector S = (df ,d\,• • • ,d%,,dyN)Te $t2N, where N is the number of pixels in facial image, (d%,d^) the x,y displacement of a pixel that corresponds to a pixel x^ in the reference face and can be denoted by S(xk). The
448
texture is coded as the intensity map of the image which results from mapping the input face onto the reference face. Thus, the shape normalized(or free) texture is represented as a vector T = (ii,- ,lN) e Jft , where i^ is the intensity or color of a pixel that corresponds to a pixel Xk among N pixels in the reference face and can be denoted by T(xfe). Next, we transform the orthogonal coordinate system by principal component analysis(PCA) into a system denned by eigenvectors sp and tp of the covariance matrices Cs and CT on the set of M faces. Where S and T represent the mean of shape and that of texture, respectively. Then, a facial image can be represented by the following equation. M-l
M-l a s
S = S + 22 P p<
T = TP=i
where a, ^ c R * * - 1 . In order to reconstruct a high-resolution facial image from a low-resolution one, we would like to define an extended morphable face model as an extended face is composed of a pair of low-resolution face and high-resolution one. Then, an extended face is separated by an extended shape and an extended texture according to the definition of morphable face model as shown in Fig. 3.
Reference (ft)
Extended Reference ( / l + )
Extended Input ( / + )
iiife
IICl F Lii5 LJ Fl P L m
•
•is i
Extended Shape ( S + ) (a) M o r p h a b l e face model
a
Fin Extended Texture ( T + )
( b ) E x t e n d e d m o r p h a b l e face model
Fig. 3. Examples of face representations based on extended morphable face model
T h e n we c a n define 5 + = (df, df, • • •, d% dyL, d*L+1, d\+l
•••, d*L+H,
dyL+H)T
to be an extended shape vector by simply concatenating low-resolution shape and
449
high-resolution shape, where L and H is the number of pixels (h, • • • i*Li*L+i»- • • AL+H)T Then, by applying PC A to in Eq. (1) can be expressed as
is the number of pixels in low-resolution facial image in high-resolution one. Similarly, let us define T+ = t° be an extended texture vector. both the shape S+ and texture T+, the face image follows.
M-l
5+ = S+ + J2
M-l
a *pSp
i
-*
— •*•
/? P tp +
(2)
P=i
where a,
0e9tM~1.
2.2. Reconstruction
procedure
Before explaining our example-based reconstruction procedure, we define two types of warping processes, forward and backward warping(see Fig. 4).
forward warpj
backward warping Fig. 4.
Examples of forward warping and backward warping
Forward warping warps a texture expressed in reference face onto each input face by using its shape information. This process results in an input facial image. Backward warping warps an input face onto the reference face by using its shape information. This process results in a texture information expressed in reference shape. The mathematical definition and more details about the forward and backward warping can be found in Ref. 9. Our reconstruction procedure consists of 4 steps, starting from a low-resolution facial image to a high-resolution facial image(see Fig. 5). Here the displacement of the pixels in an input low-resolution face which correspond to those in the reference face is known. Step 1. Step 2.
Obtain the texture of an input low-resolution facial image by backward warping. (a) Estimate a high-resolution shape from a given low-resolution shape.
450
Fig. 5.
Procedure for reconstructing a high resolution facial image from a low-resolution one
(b) Improve the estimated high-resolution shape by iterative error backprojection. Step 3. (a) Estimate a high-resolution texture from the obtained low-resolution texture at Step 1. (b) Improve the estimated high-resolution texture by iterative error backprojection. Step 4. Synthesize a high-resolution facial image by forward warping the improved texture with the improved shape. Step 1 and Step 4 are explained from the previous studies of morphable face models in many studies. 6,9 Step 2(a) and Step 3(b) are carried out by similar mathematical procedure except that the shape about a pixel is 2D vector and the texture is lD(or 3D for RGB color image) vector. Therefore, we will describe only the Step 2 of (a) estimating high-resolution shape information from low-resolution one using example-based learning and (b) improvement the results of initial high-resolution estimation by applying iterative error back-projection procedure.
3. High-resolution Estimation using Example-based Learning In this section, we present our reconstruction method using example-based learning, the problem of high-resolution reconstruction and solution to solve the least square minimization of reconstruction error.
3.1. Problem
definition
Since there is a shape information about only low-resolution facial image, we need an approximation to the deformation required for the low-resolution shape by using coefficients of the bases(see Fig. 2). The goal is to find an optimal set ap that
451 satisfies M-l
S+(xj) = X ] aP4(xi)>
J = 1, • • • , i ,
(3)
P=i
where iCy IS £1 pixel in the low-resolution facial image, L the number of pixels in low-resolution image, and M — 1 the number of bases. We assume that the number of observations, L, is larger than the number of unknowns, M — 1. Generally there may not exist a set of ap that perfectly fits the S+. So, the problem is to choose a so as to minimize the reconstruction error. For this, we define an error function, E(a), the sum of square of errors which measures the difference between the known displacements of pixels in the low-resolution input image and its represented ones. L
M-\
E(a) = £ ( 5 + 0 r , 0 - ] T aps+{Xj)f j=i
(4)
p=i
where x\, • • • , XL are pixels in the low-resolution facial image. Then the problem of reconstruction is formulated as finding a which minimizes the error function : a = arg min E{a).
3.2. Solution
by least square
(5)
minimization
The solution to Eqs. (4) - (5) is nothing more than least square solution. Eq. (3) is equivalent to the following equation. ^(xi).--s^_1(a;1)\/ai \
/5 + (xi)N (6)
\sf(xL)---sl[_1(xL)/\?M-J
\S+(xLy
This can be rewritten as : S + a = S+,
(7)
where j+
4(xi)
••• s i - i ^ i )
,s+(xL)
••• s ^ _ 1 ( x L ) ,
-
3 = ( « ! , - • • ,OIM-l) T ,
S+ = (S+(x1),-.-,S+(xL))T.
(8)
452
The least square solution to an inconsistent S+a = S + of L equation in M — 1 unknowns satisfies S + S+a* = S + S + . If the columns of S + are linearly independent, then ( S + S + ) is non-singular and has an inverse a* = ( S + T S + ) - 1 S + T S + .
(9)
The projection of S + onto the column space is therefore S+ = S + a*. By using Eq. (9), we can obtain a high-resolution shape M-l
S(xL+j)
= S+(xL+j)
+ J2 ^ps+{xL+j),
j = l,...,H,
(10)
P=I
where ££+i, • • • ,XL+H are pixels in the high-resolution facial image, L and H is the number of pixels in the low-resolution facial image and that of high-resolution facial image, respectively. By using Eq. (10), we can get the correspondence of high-resolution shape. Previously we made the assumption that the columns of S are linearly independent. Otherwise, Equation (9) may not be satisfied. If S has dependent columns, the solution a* will not be unique. The optimal solution in this case can be solved by pseudoinverse of S + . 7 But, for our purpose of effectively reconstructing a highresolution shape(or texture) from a low-resolution one, this is unlikely to happen. 4. Improvement by Iterative Error Back-Projection In this section, we explained the procedure of our iterative error back-projection method for improving the results of high-resolution estimation. In order to reconstruct a high-resolution facial image from only one lowresolution image, we used an example-based learning or top-down approach. Also to improve the accuracy of high-resolution estimation, we applied iterative error back-projection procedure to the example-based learning. Numerous kinds of error back-projection have been used to various applications such as super-resolution. 4 According to our example-based learning methods, we approximate a given lowresolution shape with some error, that is defined by Eq. 4 of estimated a*. So, we can easily guess that the estimated high-resolution shape also has some error if we know the original high-resolution shape, H M-\ E{a) = £ ( S + ( * L + J ) - Y, a;s+(xL+j))2 (11) j=i
p=i
where xi, • • • , XH are pixels in the high-resolution facial image. Our iterative error back-projection is applied to reduce the reconstruction error by iteratively compensate the high-resolution error which estimated by similar error reconstruction from simulated low-resolution error. The flowchart of the procedure we have designed for iterative error backprojection is outlined in Fig. 6. The notations showed in this figure are defined as follows.
453
Estimation
-j
H* = Reconstruction(Z/) Dp
Low - resolution : L
=0
.::::
1=1
I
L, = Downsampling(//,_1) Simulation
I
u
1_
£>/=Distance(L 7 , L?) t=r+l
High -resolution://,B
L* = lJ Error compensation
H,
±
= Reconstruction(Lf )
H^H^ Fig. 6.
-L< - t
-Lf
-Dt
-T2
-Lf
- uit
-Lst
I
+ Of-Hf
Flowchart of our iterative error back-projection method
Input low-resolution data (shape or texture) Iteration index, t = 1,2, • • • , T Reconstructed high-resolution data at iteration t Low-resolution data simulated by down-sampling reconstructed highresolution one at iteration t Reconstruction error by measuring Euclidean distance between an input low-resolution data and a simulated low-resolution one at iteration t Threshold value to determine whether the current reconstruction is accurate or not Threshold value to determine whether the current iteration is convergent or not Reconstruction error data by pixel-wise difference between an input lowresolution data and a simulated low-resolution one at iteration t Reconstructed high-resolution error data from a low-resolution error data at iteration t Weight value for error compensation at iteration t
First, as an initial stage, we estimate the high-resolution d&ta,(Hff) from an input low-resolution one(L') by using our solution of least square minimization
454
which described in previous Sec. 3.2. Second, as a simulation stage to verify the reconstruction accuracy of our example-based method, we simulate a low-resolution data(Lf) from the estimated high-resolution one by down-sampling it, then measure the distance(Df) between the input low-resolution and simulated one by distance function. We assume that if a reconstruction is successful, the reconstruction error (measured distance) will be very small. From this assumption, we determine whether the reconstruction is accurate or not by comparing current reconstruction error and one threshold value (Ti) and whether the iteration is convergent or not by comparing the changes of previous and current distances and another threshold v a l u e ^ ) . If one or both comparisons are satisfied, then the current result of reconstruction is considered as a final highresolution shape, otherwise next error back-projection stage is performed. Third, as an error back-projection stage, we create a low-resolution error shape vector between the input low-resolution shape and the simulated one by simple pixelwise difference operation, estimate a high-resolution shape error vector by similar procedure of example-based reconstruction from a low-resolution error vector, then compensate the previously estimated high-resolution shape by adding the currently estimated high-resolution shape error. In this stage, we can use a weight value(u>t) which is smaller than 1 in order to prevent divergence of iterations. The weight can be varied according to the current distance, that is, if the distance is large then the weight is also large. By using these iterative procedures, we tried to improve the result of highresolution estimation by iteratively compensate error vector. 5. Experimental Results and Analysis 5.1. Face
database
For testing the performance of our reconstruction methods, we used 200 facial images of Caucasian faces that were rendered from a database of 3D head models recorded by a laser scanner. 9,10 The original images were color image set of size 256 x 256 pixels. They were converted to 8-bit gray level and resized to 16 x 16 and 32 x 32 for low-resolution facial images by Bicubic interpolation. PCA was applied to a random subset of 100 facial images for constructing bases of the defined face model. The other 100 images were used for testing our algorithm. Next, we use a hierarchical, gradient-based optical flow algorithm to obtain a pixel-wise correspondence.9 The correspondence is defined between a reference facial image and every image in the database. It is estimated by the local translation between corresponding gray level intensity patches. 5.2. Reconstruction
results and
analysis
As mentioned before, 2D-shape and texture of facial images are treated separately. Therefore, a high-resolution facial image is reconstructed by synthesizing the es-
455
timated(or improved) high-resolution shape and the estimated(or improved) highresolution texture.
(a) Input
|\?
(b) Bilinear
(c) Bicubic
©©
0 in G
|9
Example-based learning
Interpolation
*
*
(d) Previous I (e) Proposed
#
#
#
#
(f) Original
'
«
#
Fig. 7. Examples of 256 x 256 high-resolution facial images reconstructed from 16 x 16 lowresolution facial images Figures 7 and 8 show the examples of the 256 x 256 high-resolution facial image reconstructed from 16 x 16 and 32 x 32 low-resolution images, respectively. In these figures, (a) shows the input low-resolution images, (b) to (e) the reconstructed high-resolution images using Bilinear interpolation, Bicubic interpolation, previous method 11 (using only example- based reconstruction) and proposed method (enhanced (d) by applying iterative error-back projection), respectively. And (f) shows the original high-resolution facial images. As shown in Fig. 7, classifying the input low-resolution faces are almost impossible, even with the use of Bilinear or Bicubic interpolations. On the other hand, reconstructed facial images by the example-based learning methods, especially the reconstructed images by proposed method are more similar to the original faces than others. Also, similar but better results were obtained from 32 x 32 low-resolution facial images(see Fig. 8). For quantitative evaluation of the performance of our reconstruction methods, we measured the mean reconstruction errors in shape, texture and facial image from the original high-resolution data, respectively.
456 Interpolation
(b) Bilinear
#05
#19
#71
#99
Example-based learning
uuu nnnnn uuuy n (a) Input
(c) Bicubic
(d) Previous
(o) Proposed
(f) Original
Fig. 8. Examples of 256 x 256 high-resolution facial images reconstructed from 32 X 32 lowresolution facial images
The horizontal axes of Figs. 9 and 10 represent the input low-resolution, two interpolation methods and our previous and proposed reconstruction methods. Vertical axes of them represent the mean displacement error per pixel about shape vectors and the mean intensity error per pixel(for an image using 256 gray level) about texture and image vector, respectively. ErrSx and ErrSy in Fig. 9 are the s-directional mean displacement errors along the a;— and y— axes for shape, respectively. And Err J" and Err A in Fig. 10 implies the mean intensity error for texture and for facial image, respectively. As shown in Figs. 9 and 10, the reconstruction errors of the proposed method are smaller than others. From the encouraging results of the proposed method as shown in Figs. 7-10, we can expect that our reconstruction method can be used to improve the performance of face recognition systems by reconstructing a high-resolution facial image from a low-resolution facial images captured in visual surveillance systems. In order to verify the effect of the resolution enhancement, we carried out simple face recognition experiments under following configurations. The original 256 x 256 facial images were registered, and the reconstructed high-resolution facial images from 16 x 16 facial images are used as recognition data. Figure 11 shows us the correct recognition rates of face recognition experiments.
457 Mean displacement 4-0
error' per pixel
S.5 S.O
\
2.5
s:
2.0
-a-
ErrSr
--&-
ErrS,,
1.5 1.0 0.5 0 (a) Input Fig. 9.
(b) Bilinear
(c) Bicubic (d) Previous (e) Proposed
Comparisons of mean reconstruction errors in shape
Mean intensity
error per
pixel
4.0 S.5 3.0 *v
2.5
\
^ ^
2.0
• ° \ — ~4
1.5
\
_o-
ErrJT
—*-
Err.I
v - • - _, _
—-—-~^. ~" "*
1.0 0.5 i
0 (a) Input Fig. 10.
i
(b) Bilinear
•
(c) Bicubic (d) Previous (e) Proposed
Comparisons of mean reconstruction errors in texture and image
As we can see, the recognition performance is improved by using the proposed reconstruction methods. 6. Conclusions In this chapter, we provided efficient methods of reconstructing high-resolution facial image based on an example-based learning and iterative error back-projection. Our reconstruction method consists of the following steps : computing linear coefficients minimizing the error or difference between the given shape or texture and the linear combination of the shape or texture prototypes in the low-resolution image, and applying the coefficient estimates to the shape and texture prototypes in the high-
458 Recognition
rates(%)
100
(a) Input Fig. 11.
(I)) Bilinear
(c) Bicubic (d) Previous (e) Proposed
Comparisons of recognition performance
resolution facial image, respectively. T h e experimental results are very natural a n d plausible like original highresolution facial images, when displacement among t h e pixels in an input face which correspond to those in the reference face, is known. It is a challenge for researchers t o obtain the correspondence between the reference face and a given facial image under low-resolution vision tasks. In order to apply our example-based reconstruction methods t o real-environment visual surveillance systems, we must develop an automatic algorithm t o overcome restrictions of a prior condition t h a t the displacement among pixels in an input face which correspond t o those in the reference face is known.
Acknowledgments We would like t o thank the Max-Planck-Institute for providing the M P I Face Database.
References 1. B. Tom and A.K. Katsaggelos. Resolution Enhancement of Monochrome and Color Video Using Motion Compensation. IEEE Transactions on Image Processing, Vol. 10, No. 2, pages 278-287, Feburary 2001. 2. S. Baker and T. Kanade. Limit on Super-Resolution and How to Break Them. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24 No. 9, pages 11671183, September 2002. 3. M. G. Kang and S. Chaudhuri. Super-Resolution Image Reconstruction. IEEE Signal Processing Magazine, Vol. 23 No. 3, pages 19-36, May 2003. 4. F. Dekeyser, P. Perez, and P. Bouthemy. Restoration of Noisy, Blurred, Undersampled Image Sequences Using Parametric Motion Model. In Proceedings of the International Symposium on Image/Video Communications over Fixed and Mobile Networks, ISIVC 2000, Rabat, Morocco, April 2000.
459 5. P. S. Windyga. Fast Impulsive Noise Removal. IEEE Transactions on Image Processing, Vol. 10, No. 1, pages 173-178, 2001. 6. M. J. Jones, P. Sinha, T. Vetter, and T. Poggio. Top-down Learning of Low-level Vision Tasks [Brief Communication]. Current Biology, Vol. 7, pages 991-994, 1997. 7. B.-W. Hwang, S.-W. Lee. Reconstruction of Partially Damaged Face Images Based on a Morphable Face Model. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 3, pages 365-372, 2003. 8. D. Beymer, A. Shashua, and T. Poggio. Example-Based Image Analysis and Synthesis. AI Memo 1431/CBCL Paper 80, Massachusetts Institute of Technology, Cambridge, MA, November 1993. 9. T. Vetter and N. E. Troje. Separation of Texture and Shape in Images of Faces for Image Coding and Synthesis. Journal of the Optical Society of America A. Vol. 14, No. 9, pages 2152-2161, 1997. 10. V. Blanz, S. Romdhani, and T. Vetter. Face Identiflcation across Different Poses and Illuminations with a 3D Morphable Model. In Proceedings of the 5th Internatioal Conference on Automatic Face and Gesture Recognition, Washington, D.C., pages 202-207, 2002. 11. J.-S. Park and S.-W. Lee. Resolution Enhancement of Facial Image Based on Topdown Learning. In Proceedings of ACM SIGMM 2003 Workshop on Video Surveillance, Berkeley, CA, pages 59-64, November 2003.
CHAPTER
4.5
OBJECT RECOGNITION WITH DEFORMABLE FEATURE G R A P H S : FACES, H A N D S , A N D C L U T T E R E D SCENES
Jochen Triesch Department of Cognitive Science, UC San Diego 9500 Gilman Drive, La Jolla, California, 92093-0515, USA E-mail: [email protected]
Christian Eckes Fraunhofer IMK NetMedia Schloss Birlinghoven, D- 53754 Sankt Augustin, E-mail: [email protected]
Germany
A fundamental question in invariant object recognition is that of representation. This chapter reviews object representations based on deformable feature graphs that describe particular views of an object as a spatial constellation of image features. These representations are particularly useful in situations of high clutter and partial occlusions. We demonstrate the benefits of these representations in three recognition applications: face analysis, hand gesture recognition, and the interpretation of cluttered scenes composed of multiple partly occluded objects. We conclude by discussing current trends and open challenges. 1. I n t r o d u c t i o n 1.1. The Problem
of Invariant
Object
Recognition
Object recognition is perhaps the most fundamental problem in computer vision. Humans can recognize objects without effort despite background clutter, partial occlusions, or changes in object location, view point, lighting situation, etc. This capability is called invariant recognition because the recognition process has to be invariant to these and other environmental changes t h a t can dramatically change all the pixels of an image while the depicted object remains its identity (Fig. 1). Despite considerable effort it has not been possible to endow computer vision systems with levels of competence t h a t are anywhere close to h u m a n performance. W h a t makes the task so difficult? A p a r t of the answer is t h a t invariant recognition is a computationally very demanding task and today's computers are no match for the processing power of the h u m a n visual system. Another p a r t of the answer may be t h a t we have not found the right way of representing objects t h a t facilitates efficient solutions t o t h e invariant recognition problem. This chapter reviews approaches t h a t 461
462
Fig. 1. Left: invariant recognition. Humans can easily recognize a person or an object despite great variability in the appearance due to pose, lighting, occlusion, clutter, facial expression and other factors. Right: segmenting an object from the background can be quite difficult and may require prior knowledge of the object's appearance — in this case the dalmatian dog in R.C. James's famous picture.
represent particular views of an object as a deformable constellation of certain image features. 1.2. Recognition
and
Segmentation
A problem closely related to the invariant recognition problem is the segmentation problem. Segmentation refers to the process of grouping together all pixels in an image that belong to the same object. The way in which researchers view the relation between recognition and segmentation is a topic of substantial controversy. The traditional view is that segmentation should happen independent of and prior to object recognition. Image elements are grouped together solely based on low level properties of individual pixels or small image patches such as brightness, color, motion, stereo disparity, or texture. Once an object has been segmented recognition proceeds by extracting relevant features from the segmented object and subsequent classification. This approach may be called "segmentation, feature extraction, recognition (SFR)". It is the method of choice in many application domains such as industrial inspection, where a high degree of control over the imaging process is possible. In this case, the proper choice of background and lighting as well as a possibly quite restricted set of objects can make segmentation relatively easy. When it comes to vision in more natural and less controlled environments, however, the SFR approach is severely limited. Despite decades of intensive research, it has been impossible to find robust solutions to the segmentation problem that can handle the complexities and ambiguities of everyday visual scenes 1 . The necessary information for correctly grouping all the pixels of an image to the proper objects may simply not be present in the low level pixel properties, as illustrated by R.C. James's famous picture of a dalmatian dog (Fig. 1, right). There are two alternatives to the SFR scheme. The first one simply ignores the segmentation problem altogether and tries to recognize objects directly from features extracted from the entire image. We will refer to this approach as "direct
463
recognition (DR)". This is the viewpoint we will focus on in this chapter. The second alternative tries to couple segmentation and recognition in a closed feedback loop such that partial or tentative segmentation results aid recognition while provisional recognition results try to improve segmentation. This idea could be called "integrated segmentation and recognition (ISR)". This second approach has hardly been studied in the literature but is likely to become an important topic for computer vision research over the next years. 1.3. Object Representations
for Direct
Recognition
In the following we will contrast two kinds of methods that try to recognize objects on the basis of features extracted directly from the raw unsegmented input image: feature constellation techniques and invariant feature techniques. Our goal is not to provide a comprehensive review but rather to just highlight some important ideas. Feature constellation techniques represent a particular view of an object as a spatial constellation of localized image features 2 ' 3 ' 4 ' 5 ' 6 ' 7 ' 8 ' 9 ' 10 ' 11 ' 12 ' 13,14 ' 15 . During recognition, a similar constellation of image features is searched for in a novel input image. A goodness of fit criterion evaluates the match. These techniques vary in a number of aspects including the kinds of image features used in the representation and the algorithm for matching. In order to reach invariance with respect to pose changes, appearance changes, articulation, etc. these methods have to do one or several of the following: 1) represent very different views of an object with separate models, 2) model the variability in the appearance of individual features, 3) model variability in the spatial arrangement of the features. Depending on the number of degrees of freedom in the models, the matching procedure can become very expensive to compute. An early example of such a technique was proposed by Fischler and Elschlager 2 who proposed a method similar to dynamic programming to match object models to the input image. The matching procedure seeks to minimize a cost function that is a sum of two terms. The first punishes dissimilarities of image features to the stored model features while the second penalizes geometric distortions. It has been shown that such a cost function can be derived from a statistical approach for modeling object appearance under a number of assumptions. A widely used feature constellation technique is Elastic Graph Matching4'6'13 (EGM), which will be discussed in more detail below. There, the image features stored in an object model are typically convolution results of Gabor wavelets16 of different orientations and spatial scales. Matching is usually done in a heuristic coarse to fine strategy to make the optimization more tractable. EGM has been very successful in a number of difficult recognition applications, ranging from face detection and recognition 4 ' 6 ' 17 , over gesture recognition against cluttered backgrounds 18 ' 13 , to the analysis of scenes with partially occluded objects 19 ' 8 , just to name a few. There
has
also
been
significant
progress
in
employing
probabilistic
464
frameworks 7 ' 10 ' 12 ' 20 . For example Lanitis et al. achieve compact descriptions of face shape variations using a point distribution model and use discriminant analysis techniques to aid recognition 7 . Nelson and Selinger9'21 have used a manually constructed "perceptual grouping hierarchy" in a 3D object recognition system. At the lowest level, local edge elements are grouped to curve segments. These are in turn grouped to so-called context patches that describe a local arrangement of curve segments. An arrangement of these context patches then describes a particular view of an object. Finally, several such views are grouped together to obtain a full 3D object description. Their approach is very interesting because of the use of a hierarchical object description and it gives very good results for a 3D object recognition task for uniform backgrounds. However, it is not very robust with respect to cluttered backgrounds. Invariant feature techniques. The origins of these approaches 22 ' 23 ' 24 can be traced back to the work of Swain and Ballard 25 on using color histograms for object recognition. In a sense, they represent an extreme stance for reaching invariant recognition. The idea is to extract a number of invariant features, i.e. features whose responses are supposed to stay the same under a number of object transformations like translation, scaling, rotation, and change of aspect. Early work in this direction focused on color histograms 25 and ways to make them more robust with respect to illumination changes 26 ' 27 . More recently, researchers have also used a number of other more complex features. To this end, the responses of a potentially large number of filters of different scales and orientations are applied to the entire image and the responses are then pooled together. Mel's SeeMore system 22 simply adds all the responses of the same filter type for different scales, and orientations applied to the entire image. It evaluates 102 different feature channels, whose responses signal how frequent features of one type are observed in the image regardless of their position, orientation, or scale. Recognition is performed by comparing the 102 dimensional vectors using a nearest neighbor classifier. Similarly, Schiele and Crowely24 use a histogram representation. The responses of filters of one type but different scales, orientations, and positions are all combined into the same histogram bin. Recognition is done by comparing these histograms with statistical techniques. These approaches have given impressive results for recognition on large object data bases. However, they do not perform very well for cluttered scenes. Typically, researchers use data bases with objects on homogeneous backgrounds, and experiments have shown that heavily cluttered backgrounds can have a strong negative impact on performance. The problem is that as soon as there is cluttered background or there are multiple objects in the image, the invariant feature representation will throw all local image features "into one pot" resulting in a super-position catastrophe. For example, the histogram representation is a super-position of contributions from the background and all objects in the scene from which it is no longer possible
465
to tell the individual sources if several objects can give rise to the same features. There is no way of "grouping" or "binding" together the features stemming from the same object. Several attempts have been made to remedy this problem. The first restricts histogram computation to localized image regions 28,24 . This approach works reasonably well if there are few objects in an otherwise empty scene but it is unclear whether this approach is sufficient to overcome the challenge of unfamiliar cluttered backgrounds without any prior assumptions about object sizes and shapes. The second is the use of highly specific feature detectors that are only triggered for very few objects 29 . The idea is that if several such features are "triggered" this provides very strong evidence for the presence of the object. At the same time false positives are thought to be avoided through the high specificity of the feature detectors. To this date, it remains an open question how feature detectors with the needed specificity and the desired invariance characteristics can be found. A third approach is to introduce semi-local constraints23 between the invariant features extracted at interest points. This approach is quite related to the feature constellation techniques discussed above. Considering that some feature constellation techniques use somewhat invariant features and that some invariant feature techniques consider spatial constraints among features it becomes clear that these approaches really lie on a continuum. The features extracted can have varying degrees of invariance to specific transformations 11 (translation, rotation, scale, etc.). and there can be more or less tight constraints regarding their arrangement in the image. Just how much invariance of what kind is optimal for a specific recognition problem and how proper features can be learned is an open question. Similarly, the right level of complexity and specificity of features is an important unsolved problem. Inspiration regarding the answers to these difficult questions may come from the study of biological vision systems, that seem to use a hierarchy of feature detectors of increasing complexity and invariance properties 30 . 2. Object Recognition with Labeled Feature Graphs In the remainder of this chapter we will focus on a specific kind of feature constellation technique — Elastic Graph Matching 4 (EGM). Elastic Graph Matching (EGM) is an object recognition architecture, which has been successfully applied to object recognition, face and gesture recognition, and the analysis of cluttered scenes. Although being motivated by a theory of neural information processing, EGM is similar to other elastic matching approaches 2 . In EGM views of objects are represented as labeled graphs with an underlying two-dimensional topology. The nodes of a graph are labeled with a local image description. Edges of a graph are labeled with a distance vector. Sometimes, the graphs are chosen to have a simple grid-like topology, while sometimes nodes are placed on fiducial points (interest points, landmarks) on the object of interest as illustrated in Fig. 2.
466
Fig. 2. Representation of faces with labeled graphs. Left image: simple graph with grid-like topology. Right two images: Graphs with flexible topology where nodes are placed on fiducial points on the face. The first image shows a graph manually placed on the face during training. The second image shows the graph matched to a novel input image where the face has a slightly different pose and expression. Each node of the graph is labeled with a local image description, edges between nodes are labeled with distance vectors.
Elastic matching of a model graph to an image means to search for a set of node positions such that 1) the local image description attached to each node matches the image region around the position where the node is placed and 2) the graph is not distorted too much. This is expressed as the optimization of a goodness-of-match function with a term for the similarity of the features and a cost term punishing graph distortions. 2.1. Gabor Wavelet
Transform
As a local image description for labeling the nodes of a graph a Gabor jet is usually used. This is a vector of responses of Gabor wavelets (Fig. 3):
<Mx)=—exp--liJ-
exp (ik T x) - exp ( - y - J
.
(1)
These wavelets represent a plane wave with wave vector k restricted by a Gaussian envelope function of width a; x denotes the two-dimensional image location. The responses of several such filters with different sizes and orientations, parameterized by different k, constitute a jet: k„„ = K (C°S.(Q".)N) with fc„ = kmax/r, aM = ^ . (2) \sin(a p )y D Here, the index v € { 0 , . . . ,L — 1} labels the L different spatial scales or frequency levels, and / J £ { 0 , . . . , D — 1} labels the D different orientations. The value / is the spacing factor between kernels in the frequency domain and kmax is the maximum wave number. A jet is a complex vector composed of the L x D complex filter responses Cj, where the index j is a double index running over different orientations and scales. The c,- are represented as absolute value a,- and phase 4>f- cj = Oj-e'w.
467
Fig. 3. Gabor wavelet transform. From left to right: original image; real (top) and imaginary (bottom) part of Gabor wavelet (not drawn to scale); real part of the result of convolving the image with the complex wavelet; absolute value of the convolution result. The parameters used were |k| = 1, a = 2TT, a = ir/4.
Interestingly, the visual system of primates seems to extract Gabor-like feature at its early processing stages 31 , which may be related to the statistical structure of natural images 32 . The absolute responses of such filters are quite robust to small rotations and translations which introduces a certain amount of invariance into the representation. Two similarity functions are commonly used for comparing Gabor jets 4 , e : 5abs(J,J')=
, ^
^
, ,
(3)
Sabs uses only magnitudes of the complex filter responses. It has the form of a normalized scalar product. 5 p h a uses the magnitude and phase of the complex filter responses. Both functions yield similarity values between zero and one. While SabsiJyJ1) slowly changes when J' is "moved" across the image, Spha,(J,J') varies very rapidly because the phases of filter responses change significantly on a spatial scale corresponding to the wave-vector k of strongly responding kernels. This difference between Sabs and S p h a can be understood by comparing the real part of the convolution result with the absolute value of the convolution result in Fig. 3. The absolute response varies much more smoothly. 2.2. Elastic
Matching
Process
and
Recognition
Let us consider the case of having a set of objects we may want to recognize and we have created one model graph for each object. For recognition, i.e. finding the best model for a given input image, all object models are matched sequentially. For each we need to find the minimum of the similarity or goodness-of-fit function. The similarity of a graph G matched to the image I at node positions x n may be defined
468
as simply the average of the similarities of the individual jets: f
i G
S b
* *(jn>
S (G, I) = jfY,
J
(x"))'
<5)
71=1
Often it is useful to introduce an additional cost term to punish geometric distortions of the graph. Also, allowing for different weightings of different nodes in the evaluation of a match can be beneficial. After matching all models to the image, the model obtaining the highest similarity represents the classification result. For object detection, thresholds for all object models must be learned. An object is detected if the matching process yields a score that is above threshold. During matching we need to simultaneously optimize all degrees of freedom of the deformable object model. If object models with many degrees of freedom are allowed (such as shifting across the image, independent scaling in x- and ydirection, rotation in plane, various non-rigid deformations) then matching time can become an issue making an exhaustive search over the parameter space unfeasible. To combat the adverse effect of many degrees of freedom in the object model, a coarse to fine matching scheme can be used. For example, a strategy may be to first try to determine the position, orientation, and scale only very coarsely, and then to refine the estimate in subsequent matching steps. In a final step, "diffusion" of individual nodes may be allowed to compensate for small "irregular" deformations of the object in the current input image. 2.3. Bunch
Graphs
The bunch graph idea6 was developed to model variability in feature appearance. The natural variability in the jets of corresponding points in several images of the same object or class of objects (e.g. several left eyes of different persons) is captured by labeling each node of a graph with a set or "bunch" of jets (short: bunch jet) extracted from corresponding points in different example images rather than only with a single jet. This method can also be used to model variability due to different backgrounds 18 . For the matching process, we need a similarity function comparing the set of jets attached to each node of the bunch graph with local image information. The similarity of a bunch jet Bn comprising K jets at a node n to a single jet J(x) taken at a point x in an image is defined to be the maximum of the similarities of the K individual jets: S*hs(Bn, J(x)) = max{S a b s (J"(/c), J(x)) ,k = l,...,K}.
(6)
k
5p h a is defined analogously. When a graph G with N nodes is matched to an image i" at the node positions x„, its total similarity is given by the average of the node similarities:
SL(G, I) = jjJ2 S^(Bn, J(*n)) • 71=1
(7)
469
Again, we define 5^ h a in an analogous manner. Modeling the variability in feature appearance by storing a set of prototypes is obviously not the only plausible method and a number of other approaches, e.g. based on principal component analysis, can be used as well. 2.4. Application
to Face
Recognition
Automatic face recognition is an important problem with applications ranging from human computer interaction to surveillance and security. A number of studies have explored the use of EGM for face finding and pose estimation 17 , identification 4 ' 6 ' 33 ' 34 , facial expression recognition, gender estimation and more. Figure 2 shows how a face graph may be constructed from a training image so that the graph can then be used to recognize the face in a novel view. While early systems used simple graphs with a grid topology, substantial improvements can be obtained by using graphs whose nodes are placed on specific fiducial points (or landmarks) on the face. The bunch graph technique can been used to represent a generic face model in a single graph. To this end, jets from corresponding points (say, several left eyes) of several training images from different persons are attached to a single node. The underlying assumption is that feature variation, e.g. variability in the appearance of left eyes, is approximately independent of relational variability, i.e., variability in the relative position of left eyes with respect to other features of the face. The great advantage of this method is that it allows to separate the matching process, where the facial landmarks are found, from the identification process, where features from the found facial landmarkds are compared to a gallery of known persons. A number of other extensions have been made to the basic method described so far that are beyond the scope of this chapter 6,33 ' 34 . 3. Combining Multiple Feature Types While early versions of EGM only used a single shape or texture feature type such as Gabor or Mallat niters, it has also been extended for handling multiple feature types 8 ' 13 . There are several ways of extending EGM for multiple feature types (Fig. 4). First, the feature types used may differ from node to node. For example, a model graph describing a face could employ edge features at the outline of the face and texture features at the inside. Second, all nodes may be labeled with a combination of different feature types which is identical for all nodes. Third, nodes may be labeled with combinations of feature types which may differ from node to node. Here we focus on the second scheme. 3.1. Compound
Jets
In order to allow for multiple feature types at a node we introduce the notion of a compound jet. In a compound jet, several local image descriptions are simply con-
470
Fig. 4. Combining different feature types in graph based object representations can be done in a number of ways. All nodes may be labeled with just a single (a) or multiple (c) features that are identical for all nodes. Or all nodes may be allowed to be labeled with a single (b) or multiple (d) feature types that are different for all nodes.
catenated. When similarities between two compound jets J and J' are computed, first the similarities of their corresponding constituents are considered. These are are then combined by computing a weighted average: S(J,J')
= ^2WFSAJF,J'F), T
$ > j r = l.
(8)
T
The Sj: are similarity functions for comparing jets of a particular feature type T. Figure 5 compares different kinds of features for finding image point correspondences. Combining a number of complementary features into a compound jet can improve the overall performance.
3.2. Compound
Bunch
Graphs
One can easily extend the bunch graph idea to graphs with compound jets as well. Compound bunch graphs are constructed in the following manner: • For every node n of a particular graph of a view of an object the jets of the same feature type T extracted from K different training images are integrated into a bunch jet 5 £ = {Jp-(l), • • • > J?{K)}- With these bunch jets as node labels one bunch graph is created for every feature type, whose geometry is averaged from the constituent graphs. • The bunch jets B1^ of different feature types but from the same node n of the same view are concatenated to a compound bunch jet Bn. With these compound bunch jets as node labels, a compound bunch graph is created, whose geometry is identical to that of the constituent bunch graphs of different feature types.
471
Fig. 5. Finding image point correspondences with local descriptions. Combining different feature types in a node description can be beneficial for establishing image point correspondences. a 5 b : source image and its skin color segmentation. The circle marks a point where a compound jet was extracted. c,ds target image and its skin color segmentation. The goal is to locate the point in c corresponding to the marked location in a. e-f: similarity of marked location in a to all locations in c when using only Gabor jets, only color features, only Gabor jets computed on the skin color segmentation, or compound jets representing a weighted average of the others. Bright pixels indicate high similarity.
For evaluating the similarity «S B (S n , J(JC)) between a compound bunch jet Bn and a compound jet J(x) extracted at a particular location x a weighted average of contributions S B stemming from different feature types is used:
5B(BW, J(x)) = 5>^S£(B£, J H X )), J > * = 1'
(9)
The functions S B compare a bunch jet E^ to a simple jet Jj? of the same feature type T in analogy to (6): S%(Bh M*))
= max{5^(J-(fe), J^(x)),
k = 1 , . . . , K} .
(10)
k
The similarity functions Sj? directly compare two jets of a particular feature type T. The similarity of a compound bunch graph Q with node positions x n to an image I is defined as the average of the similarities of all N nodes, in analogy to the previous section:
sQ(e,i) = jj'£sB{Bn,j{xn)).
(ii)
n
Again5 an additional topological cost term can be introduced to punish large geometric distortions, and nodes can be given different weights.
472
Fig. 6. Person-independent hand posture recognition against complex backgrounds is a challenging problem. Due to differences in hand anatomy and individual performance of a gesture there is substantial variability in the appearance of the hand. In addition, rings, etc. may be present and long sleeves may introduce partial occlusions.
3.3. Application
to Hand
Gesture
Recognition
Gestures are a powerful means of communication among humans. In fact, gesturing is so deeply rooted in our communication that people often continue gesturing when speaking on the telephone. Consequently, the automatic recognition of human gestures is an important topic in human computer interaction. The recognition of hand postures is an important ingredient for building gesture-based interfaces. Although there have been attempts to recognize gestures and sign language without recognizing the specific posture of the gesturing hand it is clear that these approaches are limited. For example, in American Sign Language the same movement can have very different meanings depending on the posture of the gesturing hand. A number of requirements make hand posture recognition a particularly challenging task. First, if a system is intended to be person independent it must cope with shape variability due to different hand anatomy or different performance of the same gesture by different persons. Second, if the system is intended to work in unconstrained environments it must be able to deal with cluttered backgrounds, making bottom-up segmentation of the gesturing hand difficult. Along similar lines, it should not be necessary for subjects to take off rings etc. before interacting with the system but they should be able to just "come as they are". Other desiderata are real-time performance and robustness to varying lighting conditions. Using a deformable feature graph approach such as EGM is a very good starting point for this challenging recognition task because 1) it has an inherent ability to handle geometric deformations, 2) it does not require prior segmentation of the gesturing hand, and 3) it can elegantly handle variability in feature appearance with the bunch graph method. We have used EGM with multiple feature types to address this difficult problem 13 . In particular, we added two kinds of color features to the local im-
473
Fig. 7. Example of a model graph being matched onto an input image, as original graph. b: matched graph, c: skin color segmentation.
age descriptions at the graph nodes. The first additional feature was a local average of the color in the vicinity of the node (in HSI color space), the second additional feature was a Gabor jet extracted from a preprocessed version of the image that emphasized skin tones. The image database consisted of more than 1000 color images of twelve hand postures performed by 19 persons against simple and complex backgrounds with varying amount of skin color (Fig. 6). The images of three subjects signing against uniform light and dark backgrounds formed the training set3 giving 6 training images per posture, the remaining images formed the test set. For the images in the training set, we constructed graphs of 15 nodes. The nodes were manually placed at anatomically significant points. Model graphs extracted from the training images were combined into 12 compound bunch graphs (one for every posture) as described above. The matching procedure had 4 steps: coarse positioning of the graph by scanning it across the image in steps of five pixels; estimating rotation in plane (+/» 15 degrees) while refining the position estimate; estimating scale changes (up to 20 percent); local diffusion of individual nodes to fine tune the matching. An example of a successful match is given in Fig. 7. The nodes usually find their proper positions during the matching, even if the background is very cluttered or contains large regions of skin color.
3.4.
Evaluation
Cross-runs on a test set of 604 images taken against uniform light or dark background and 338 images against complex backgrounds were performed. The results are summarized in Tab. 1. The weighting factors wj? between the three different feature types were systematically varied in an exhaustive manner considering all relative weightings of the types 1:0:0, 4:1:0, 3:2:0, 3:1:1, 2:2:1, 2:1:1, and 1:1:1, and system performance on the test set was measured. The best result was obtained for ^Gabor = ^GoiorGabor = 25%, t^coior = 50%. Since the weighting of cues only intro-
474 Table 1. weighting only only only best
Gabor Color ColorGabor Mixture
Recognition results for gesture recognition. simple background
complex background
82.6% 39.7% 88.2% 92.9%
70.4% 34.6% 76.3% 85.8%
Note: All numbers are percent correct recognition.
duces 2 free parameters, we decided not to use a separate validation set. It turns out that a proper combination of the three feature types outperforms any of them alone. For example, the error for the best combination found is less than half as big as that for using Gabor features alone. Performance does not depend critically on the precise weighting between the features if all are being used. The recognition rates vary smoothly with the weighting and there is a large plateau of weightings yielding recognition rates higher than any of the feature types alone could account for. If only Gabor and colorGabor features but not color features are used, performance is still very good, suggesting that the color features are not essential for the system's performance. An analysis of the system's errors reveals that strong shape variations due to either differences in performance of a posture, different hand anatomy, strong rotation in depth, or combinations of these account for practically all errors. Interestingly, it was found that a greater flexibility of the graph during the final matching step was not effective in reducing the number of errors. The reason seems to be that the geometric distortions are not of a random nature which would be well modeled by a diffusion process, but instead show high correlations. This is due to the hand's kinematics which only allow for certain correlated movements of nodes. A more refined modeling of likely graph distortions thus seems like a good avenue for future research. In its extreme form, this would amount to a full kinematic model of the hand, but this may not always be necessary. 4. Analysis of Cluttered Scenes with Partial Occlusions The analysis of cluttered scenes such as the one depicted in Fig. 8 is one of the most challenging problems in computer vision. Massive background clutter makes it virtually impossible to properly segment the scene into regions corresponding to the objects in a bottom-up fashion, i.e. without using explicit object models. Furthermore, occlusions can render large portions of an object invisible. The ease and efficiency with which the human visual system solves this problem belies the fundamental computational challenges that this task poses. It turns out that for this kind of task, the use of deformable feature graphs as the object representation is again a very good choice. In particular, the use of local image features allows to explicitly discover and model partial occlusions by other objects 19,8 . In this Section
475
Fig. 8. Analysis of cluttered scenes. Given a number of object models in the form of deformable feature graphs (left), the goal is to analyze a cluttered scene with partially occluded objects (middle) and to return a description of the identities, locations, and occlusion relations of all objects in the scene (right).
we will outline an extension of the methods discussed so far to stereo image pairs and apply them to the analysis of cluttered scenes of multiple objects occluding one another. 4 . 1 . Extension
of EGM
to Stereo Image
Pairs
Most work on stereo vision has as its goal the estimation of 3-D structure from 2-D images. In contrast, we are interested in improving object recognition through the use of stereo processing 35 ' 36 . The essence of stereo vision is solving the correspondence problem, i.e. determining which point in the left and right image correspond to the same physical point. Most stereo algorithms consider fairly simple image features when trying to establish the correspondence relations, but in principle features of arbitrary complexity can be used for this purpose — or even entire objects. The basic ideas in our case is that correct object matches in the left and right image are not independent but related through the epipolar geometry of the stereo system. This induces a constraint that can be used to discard false matches if they fall on inconsistent locations in the left and right image (Fig 9). As a side product, the distance of correct matches can be estimated. If absolute size information about the objects is available, then the distance information must in turn be related to the apparent size of the object in the images — an additional constraint that can be used to reject false matches.
4.2. Scene
Analysis
Our goal for object recognition in the presence of partial occlusions is to explicitly detect and discount occlusions when evaluating the quality of a match. One idea for doing so works in a front-to-back fashion. Object models are matched to the image and the object models obtaining good matches enter a set of candidates. Of these candidates, the front-most object, i.e. the one closest to the camera is determined
476
Fig. 9. Simultaneous recognition in stereo image pairs. The exploitation of epipolar constraints can aid recognition in difficult situations with clutter and partial occlusions, a: stereo image pair of cluttered scene with partially occluded object "duck", b : Simultaneous matching of object model duck in both images subject to epipolar constraints manages to correctly locate the object despite the partial occlusion, c: In contrast, independent matching in the left and right image is more prone to errors. In this case, it fails for the right image.
on the basis of stereo disparity, match similarity and possibly other cues. The frontmost candidate is then accepted and removed from the candidate list. Since this object is the closest candidate, it cannot be occluded by other objects other than those recognized on previous iterations. However, it may itself be occluding some objects. To take this into account, the image area covered by the accepted object is labeled as occluded. Now, the matches for other objects are re-evaluated by discounting any nodes that fall into the occluded area. This may lead to additional objects entering the set of candidates. The algorithm now repeats the selection of the front-most candidate until all no more candidates remain. 4.3.
Evaluation
To test the method we have recorded a data base of stereo image pairs of cluttered scenes composed of 18 different objects in front of uniform and complex backgrounds. There is considerable variation in object shape and size, but there are also objects with nearly identical shapes differing only in color and surface texture. The training images comprise one stereo image pair of each object in front of a uniform blue cloth background. The test images comprise scenes composed of 2-5 objects in front of one simple background (80 scenes) and four complex backgrounds (160 scenes). In systematic experiments evaluating the above scene analysis methods we found that the extension to recognition in stereo image pairs and the use of multiple feature types in the form of compound jets can lead to quite dramatic performance
477
Fig. 10. Left: collage of objects. Right: example scenes with simple (a,b) and complex (c-f) backgrounds. Note the variation in size for object "pig".
improvements. In particular, for a fixed level of false positives, a version of the system using both extensions reached roughly twice as many correctly recognized objects as a monocular version of the system employing Gabor features only. An important lesson from these experiments is that the representation of objects as constellations of localized features allows to explicitly discount partial occlusions — a property that is quite appealing. Representations based on "larger" features covering big parts of the object 15 appear less suited for this task because they are more prone to be corrupted by partial occlusions. It may be possible to develop hierarchical architectures that integrate features at different scales 9,21 and combine the advantages of more localized and more global features. Progress in the area of scene analysis has been comparatively slow and we feel that a reason for this is that researchers have usually tried to decompose the problem in the wrong way. A specific problem may have been a frequent insistence on treating segmentation and recognition as separate problems, to be solved sequentially. We believe that this approach is inherently problematic and this work is to be seen as an initial step towards finding an integrated solution for both problems. In the method described above, which is an extension to the work by Wiskott 19 , segmentation is purely model driven, i.e. if an object has been recognized, the corresponding region of the image is labeled as a segment. This need not be the case and current work explores ways of also integrating bottom-up segmentation cues.
5. Discussion This chapter discussed object recognition approaches based on deformable feature graphs — a special case of what we have called direct recognition approaches, that try to recognize objects on the basis of features directly extracted from unsegmented images. Deformable feature graphs represent particular views of objects as spatial constellations of certain image features. Such an object representation has a number of benefits. First, it allows recognition without prior segmentation of the object of interest. Second, robustness to small variations in appearance tends to be very good
478
if features are chosen properly, i.e. there tends to be good generalization. Third, also depending on the features used and the type of recognition scheme employed, it can be quite fast, lending itself to real time implementations with current off-theshelf hardware. Fourth, it allows for explicitly modeling and discounting of partial occlusions. Fifth, the methods described in here have already proved their potential in very different applications ranging from face detection and recognition, over hand gesture recognition, to the analysis of cluttered scenes of partly occluded objects. Overall, these benefits ensure continued interest in these methods.However, there are many fundamental challenges still lying ahead. First, it is desirable to automate the learning process as much as possible. In the gesture recognition example it was necessary to manually label the anatomical landmarks in the training images. How can object models be built without precise feature correspondences but only a coarse localization of the object in the training images? Even better would be a system for which only a label (object is present/not present) has to specified for each image 37 ' 38 . Finally, it should even be possible to discover object categories without any explicit supervision. Human infants seem to be able to spontaneously form categories of visual objects without explicit instruction or supervision. A machine vision system that simply observes its visual environment and learns to cluster objects into the right categories in a completely unsupervised fashion would represent a quantum leap for the field. Second, how can large object data bases be searched efficiently? Methods that engage in elaborate matching procedures for thousands of objects in a sequential fashion seem rather inappropriate. An idea that could lead to benefits in recognition speed is to share partial object descriptions among several models and to search for objects in a bottom-up fashion, where the detection of individual features triggers a more elaborate analysis for all matching candidate objects. In this context, how can partial object descriptions be learned that lend themselves to efficient sharing between objects while still allowing effective discrimination? There may be large benefits in employing hierarchical object representations using features at several levels of complexity. Fairly unspecific lower level features can be shared between object models most effectively while more specific higher level features give the necessary discriminative power for recognition. What are good frameworks for learning such representations? Third, how can recognition and segmentation be integrated within a closed feedback loop to allow for efficient recognition in the presence of clutter and occlusions? These questions are likely to remain of great interest for years to come. Some inspiration may come from the study of biological vision systems, which have mastered object recognition better than any machine vision system that man has ever created. Acknowledgments This work was supported by the National Science Foundation under grants NSF0208451 and NSF-0233200. We thank Christoph von der Malsburg for comments
479 on p a r t s of this chapter.
References 1. J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Trans. PAMI, 22(8):888-905, 2000. 2. M. A. Fischler and R. A. Elschlager. The representation and matching of pictorial structures. IEEE Trans. Computers, 22(l):67-92, 1973. 3. A. Yuille and A. Blake. Deformable templates. In A. Blake and A. Yuille, editors, Active Vision, pages 21-38. MIT Press, 1992. 4. M. Lades, J. C. Vorbriiggen, J. Buhmann, J. Lange, C. von der Malsburg, R. P. Wiirtz, and W. Konen. Distortion invariant object recognition in the dynamic link architecture. IEEE Trans. Computers, 42:300-311, 1993. 5. R. Brunelli and T. Poggio. Face recognition: Features versus templates. IEEE Trans. PAMI, 15(10):1042-1052, 1993. 6. L. Wiskott, J.-M. Fellous, N. Kriiger, and C. von der Malsburg. Face recognition by elastic graph matching. IEEE Trans. PAMI, 19(7), 1997. 7. A. Lanitis, C. J. Taylor, and T. F. Cootes. Automatic interpretation and coding of face images using flexible models. IEEE Trans. PAMI, 19(7):743-746, 1997. 8. J. Triesch and C. Eckes. Object recognition with multiple feature types. In L. Niklasson, M. Boden, and T. Ziemke, editors, Proceedings ICANN 98, pages 233-238. Springer, 1998. 9. R. C. Nelson and A. Selinger. Large-scale tests of a keyed, appearance-based 3-d object recognition system. Vision Research, 38(15-16):2469-2488, 1998. 10. M. C. Burl, M. Weber, and P. Perona. A probabilistic approach to object recognition using local photometry and global geometry. In ECCV'98, pages 628-641, 1998. 11. David G. Lowe. Object recognition from local scale-invariant features. In Proc. of the International Conference on Computer Vision ICCV, Corfu, pages 1150-1157, 1999. 12. M. Zobel, A. Gebhard, D. Paulus, J. Denzler, and H. Niemann. Robust facial feature localization by coupled features. In Proc. 4th Intl. Conf. on Automatic Face and Gesture Recognition, pages 2-7, 2000. 13. J. Triesch and C. v.d. Malsburg. A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans. PAMI, 23(12):1449-1453, 2001. 14. P. Viola and M. J. Jones. Robust real-time object detection. Int. J. of Computer Vision, 57:137-154, 2001. 15. S. Belongie, Jitendra Malik, and Jan Puzicha. Shape matching and object recognition using shape contexts. IEEE Trans. PAMI, 24(4):509-522, 2002. 16. D. Gabor. Theory of communication. J. Inst. Elec. Eng. (London), 93:429-457, 1946. 17. N. Kriiger, M. Potzsch, and C. von der Malsburg. Determination of face position and pose with a learned representation based on labeled graphs. Image and Vision Computing, 15:665-673, 1997. 18. J. Triesch and C. v.d. Malsburg. Robust classification of hand postures against complex backgrounds. In Proc. 2nd Intl. Conf. Automatic Face and Gesture Recognition, pages 170-175. IEEE Computer Society Press, 1996. 19. L. Wiskott and C. von der Malsburg. A neural system for the recognition of partially occluded objects in cluttered scenes. Int. J. of Pattern Recognition and Artificial Intelligence, 7(4):935-948, 1993. 20. P. F. Felzenszwalb and D. P. Huttenlocher. Efficient matching of pictorial structures. In CVPR 2000, pages 66-73, 2000. 21. A. Selinger and R. C. Nelson. A perceptual grouping hierarchy for appearance-based
480 3d object recognition. Computer Vision and Image Understanding, 76(l):83-92, 1999. 22. B. W. Mel. SEEMORE: Combining color, shape, and texture. Neural Computation, 9(4):777-804, 1997. 23. C. Schmid and R. Mohr. Local greyvalue invariants for image retrieval. IEEE Trans. PAMI, 15(5):530-535, 1997. 24. B. Schiele and J. L. Crowley. Recognition without correspondence using multidimensional receptive field histograms. Int. J. of Computer Vision, 36(l):31-50, 2000. 25. M. Swain and D. H. Ballard. Color indexing. Int. J. of Computer Vision, 7(l):ll-32, 1999. 26. B. V. Funt and G. D. Finlayson. Color constant color indexing. IEEE Trans. PAMI, 17(5):522-529, 1995. 27. D. Slater and G. Healey. The illumination-invariant recognition of 3d objects using color invariants. IEEE Trans. PAMI, 18(2):206-210, 1996. 28. F. Ennesser and G. Medioni. Finding waldo, or focus of attention using local color information. IEEE Trans. PAMI, 17(8):805-809, August 1995. 29. B. W. Mel and J. Fiser. Minimizing binding errors using learned conjunctive features. Neural Computation, 12(4):731-762, 2000. 30. T. J. Palmeri and I. Gauthier. Visual object understanding. Nature Reviews Neuroscience, 5:291-302, 2004. 31. J. P. Jones and L. A. Palmer. An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. J. Neurophysiol., 56(6):1233-1258, 1987. 32. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-609, 1996. 33. K. Okada, J. Steffens, T. Maurer, H. Hong, E. Elagin, H. Neven, and C. von der Malsburg. The Bochum/USC Face Recognition System And How it Fared in the FERET Phase III test. In H. Wechsler, P. J. Phillips, V. Bruce, F. Fogeman Soulie, and T. S. Huang, editors, Face Recognition: From Theory to Applications, pages 186205. Springer-Verlag, 1998. 34. K. Okada and C. von der Malsburg. Pose-invariant face recognition with parametric linear subspaces. In Proceedings of the Fifth International Conference on Automatic Face and Gesture Recognition, pages 64-69, 2002. 35. J. Triesch. Vision-Based Robotic Gesture Recognition. PhD thesis, Institut fur Neuroinformatik, Ruhr-Universitat Bochum, Germany, 1999. Also published as book by Shaker Verlag, Aachen, Germany, ISBN 3-8265-6257-7. 36. Efthimia Kefalea. Flexible object recognition for a grasping robot. PhD thesis, Institut fur Neuroinformatik, Ruhr-Universitat Bochum, Germany, 2001. 37. M. Weber, M. Welling, and P. Perona. Towards automatic discovery of object categories. In Proc. IEEE Comp. Soc. Conf. Comp. Vis. and Pat. Rec, CVPR2000, 2000. 38. H. S. Loos and C. von der Malsburg. 1-click learning of object models for recognition. In H. H. Biilthoff, S.-W. Lee, T. Poggio, and C. Wallraven, editors, Biologically Motivated Computer Vision (BMGV 2002), volume 2525 of Lecture Notes in Computer Science, pages 377-386. Springer Verlag, 2002.
C H A P T E R 4.6 H i e r a r c h i c a l C l a s s i f i c a t i o n a n d F e a t u r e R e d u c t i o n for Fast Face Detection
Bernd Heisele Honda Research Institute USA, Inc. Boston, USA E-mail: [email protected]
Thomas Serre Sam Prentice Tomaso Poggio Center for Biological and Computational Learning at M.I.T. Cambridge, USA E-mail: [email protected] [email protected] [email protected]
We present a two-step method to speed-up object detection systems in computer vision that use Support Vector Machines (SVMs) as classifiers. In the first step we build a hierarchy of classifiers. On the bottom level a simple and fast linear classifier analyzes the whole image and rejects large parts of the background. On the top level, a slower but more accurate classifier performs the final detection. We propose a new method for automatically building and training a hierarchy of classifiers. In the second step we apply feature reduction to the top level classifier by choosing relevant image features according to a measure derived from statistical learning theory. Experiments with a face detection system show that combining feature reduction with hierarchical classification leads to a speed-up by a factor of 335 with similar classification performance.
1. I n t r o d u c t i o n A major task in visual scene analysis is to detect objects in images. This is commonly done by shifting a search window over an input image and by categorizing the object in the window with a classifier. T h e main problem in categorization is the large range of possible variations within a class of objects. T h e system must generalize not only across different viewing and illumination conditions but also across different exemplars of a class, such as faces of different people for face detection. This requires complex, computationally expensive classifiers. Further contributing to the computational load of the task is the large amount of input d a t a t h a t has to be processed. A real-time vision system has to deal with d a t a streams in the range 481
482
of several MBytes/sec. Speeding-up classification is therefore of major concern when developing systems for real-world applications. In the following we investigate two methods for speed-ups: hierarchical classification and feature reduction. In 4 we presented a system for detecting frontal and near-frontal views of faces in still gray images. The system achieved high detection accuracy by classifying 19x19 gray patterns using a non-linear SVM. Searching an image for faces at different scales took several minutes on a PC, far too long for most real-world applications. Experiments with faster classifiers (linear SVMs) gave significantly lower recognition rates. To speed-up the system without losing in classification performance one can exploit the following two characteristics common to most vision-based detection tasks: First, the vast majority of the analyzed patterns in an image belongs to the background class. For example, the ratio of non-face to face patterns in the tests in 4 was about 50,000 to 1. Second, many of the background patterns can be easily distinguished from the objects. Based on these two task-specific characteristics it is sensible to apply a hierarchy of classifiers. Fast classifiers remove large parts of the background on the bottom and middle levels of the hierarchy and a more accurate but slower classifier performs the final detection on the top level. This idea is related to the well known coarse-to-fine template matching 7 ' 2 ' 3 . In 8 hierarchical classification is used to speed-up a face detection system. A candidate selection neural network with increased robustness against translation is added as a first layer to an existing face detection neural network. We present a method for building and training a hierarchy of SVM classifiers given a set of classifiers which operate at different image resolutions. The iterative algorithm starts with the topmost classifier at the highest image resolution and adds a new lower layer. At each iteration, the hierarchies are retrained bottom up and a speed test is performed on a validation set of non-face images to choose the hierarchy with the least number of computations. Another possibility of speeding-up classification is to reduce the number of features by selecting a subset of relevant features. Feature reduction is a more generic tool than the above described hierarchical classification and can be applied to any classification problem. There are basically two types of feature selection methods in the literature: filter and wrapper methods *. Filter methods are preprocessing steps performed independently of the classification algorithm or its error criteria. Wrapper methods attempt to search through the space of feature subsets using the criterion of the classification algorithm to select the optimal feature subset 5 . We present a new wrapper method to reduce the dimensions of both input and feature space of a non-linear SVM. For our final detection system we combine feature selection with hierarchical classification by putting a non-linear SVM with feature selection on top of the hierarchy of linear SVMs. A similar idea of combining feature selection with hierarchical classification has been recently proposed in 12 for frontal face detection. They use AdaBoost to train a cascade of linear classifiers and to select features from an over complete set of Haar wavelet features. In contrast to
483
our approach, however, the complexity of the classifiers in the final hierarchy is only controlled by the number of features and not by the class of decision functions (i.e. class of kernel functions). The outline of the paper is as follows: In Section 2 we give a brief overview of SVM classification. In Section 3 we describe how to build and train the hierarchical system. Sections 4 and 5 describe the feature selection methods for the input and the feature space of the SVM, respectively. In Section 6 we present experimental results of the hierarchical system with feature reduction. The paper is concluded in Section 7. 2. Background on Support Vector Machines 2.1.
Theory
An SVM n performs pattern recognition for a two-class problem by determining the separating hyperplane that has maximum distance to the closest points of the training set. These closest points are called support vectors. To perform a non-linear separation in the input space a non-linear transformation <&(•) maps the data points x of the input space IRn into a high dimensional space, called feature space IRP (p > n). The mapping <&(•) is represented in the SVM classifier by a kernel function K(-,-) which defines an inner product in IRP. Given I examples {(XJ, j/j)}f=1, the decision function of the SVM is linear in the feature space and can be written as: e / ( x ) = w • $(x) + b = J2 «°3A# ( * , x) + b. (1) 2= 1
The optimal hyperplane is the one with the maximal distance in space IRP to the closest points $(XJ) of the training data. Determining that hyperplane leads to maximizing the following functional with respect to a: t 2
i a
W (a) = 2j2 i2=1
Y^ aiajyiyjKixi,xj) i,j
(2)
= l
under constraints J2i=i aiVi = 0 a n d C — ai — 0, * = 1> •••>^- An upper bound on the expected error probability EPerr of an SVM classifier is given by n :
Ep
-AE{w)=\E(R2w2{a0)^
(3)
where M = w}ac>\ is the distance between the support vectors and the separating hyperplane and R is the radius of the smallest sphere including all points $ ( x i ) , ...,$(x^) of the training data in the feature space. In the following, we will use this bound on the expected error probability to rank and select features. 2.2.
Computational
Issues
The only non-linear kernel investigated in this paper is a second-degree polynomial kernel K(x, y) = (1 + x • y) 2 which has been successfully applied to various
484
object detection tasks 6 ' 4 . Eq. (1) shows two ways of computing the decision function. When using the kernel representation on the right side of Eq. (1) the number of multiplications required to calculate the decision function for a second-degree polynomial kernel is: Mk,poiy2 = (n + 2) • s,
(4)
where n is the dimension of the input space and s is the number of support vectors. This number is independent of the dimensionality of the feature space. It depends on the number of support vectors which is linear with the size I of the training data n . On the other hand, the computation of the decision function in the feature space is independent of the size of training samples, it only depends on the dimensionality p of the feature space. For the second-degree polynomial kernel the feature space IRP has dimension p = ^ 2 'n and is given by x* = (y/2xi,--- ,y/2xn,x\,--, x"^, ^/2xix2, • • • , V ^ n - i ^ n ) - Thus the number of multiplications required for projecting the input vector into the feature space and for computing the decision function is: Mf,poly2 =
y(n
+ l)n ; 2
(n + 3)n + ^ 2 ' = (n + 2) • n
(5)
From Eq. (4) and (5) we see that the computation for an SVM with second-degree polynomial is more efficiently done in the feature space if the number of support vectors is bigger than n. This was always the case in our experiments; the number of support vectors was between 2 and 6 times larger than n. That is why we investigated not only methods for reducing the number of input features but also methods for feature reduction in the feature space.
3. Hierarchy of classifiers 3.1. System
Overview
In most object detection problems the majority of analyzed image patches belong to the background class. Only a small percentage of them look similar to objects and require a highly accurate classifier to avoid false classifications. It is sensible to apply a hierarchy of classifiers where the complexity of the classifier increases with each level. By propagating only those patterns that were not classified as background, we quickly decrease the amount of data to process. The main issues in designing such a classification hierarchy are how to choose the input features to the classifiers, how to select the number of levels, and finally how to train the classifiers. We used pixel values as inputs to the classifiers and reduced the number of features from top to bottom by decreasing the image resolution, similar to coarse-to-fine matching approaches. An example of a hierarchical system with five layers is shown in Figure 1.
485
19x19 linear classifier
Fig. 1. Example of a hierarchical system with five levels: Starting with the classifier at the lowest resolution patterns which have been classified as background are successively removed from the image going up the hierarchy. The final non-linear classifier processes the image at maximum resolution.
3.2. Building
the
Hierarchy
In the following we describe an algorithm for automatically determining the architecture of the hierarchy. We begin with a set of classifiers which operate at different resolutions and are each trained over the entire training set. In our experiments the resolution of the linear SVM classifiers ranged from 3 x 3 to 19x19. Given this set, the goal is to find the best hierarchy with respect to speed and recognition performance. The algorithm builds the hierarchy in an iterative top-down fashion starting with the topmost classifier and adding a new layer at each iteration. It consists of three steps: a) A d d i n g a n e w layer. We add a classifier to the hierarchy which operates at a lower resolution than the current bottom classifier. For example, if the current hierarchy consists of a 19x19 and an 11x11 classifier we add classifiers with resolutions ranging from 9x9 down to 3x3, resulting in 7 new hierarchies. b) R e t r a i n i n g . We perform bottom-up training of the new hierarchies. We train the bottom classifier on the whole training set and then shift its hyperplane such that a fixed percentage of positive examples8, is on one side. Then we remove all negative examples which lie on the other side of the hyperplane to generate a new a
Was set to 99% in the experiments.
486
training set for the classifier on the next higher level. This is continued for each layer till we reach the top of the hierarchy. c) Speed test. We set the thresholds of all classifiers in the hierarchies such that a fixed percentage of the faces in the training set b is correctly classified. Then we perform a speed test of the hierarchies on the validation set of non-faces and choose the hierarchy with the least number of computations. The iterative process is continued until the lowest resolution is reached or adding a new layer does not increase the speed of the system. In our experiments we used a set of linear SVM classifiers with resolutions ranging from 3x3 to 19x19 and a non-linear SVM at resolution 19x19 which was chosen to be our top level classifier. The non-linear SVM was not included into the bottom-up learning procedure described above. For a realtime system this computationally expensive classifier can only be applied to a very small percentage of the input patterns. Since our bottom-up training method reflects the runtime data flow, the few training examples that remain would not have been sufficient to train an accurate classifier. The training set for the linear classifiers consisted of 9,662 face images and 33,045 non-face images at resolution 19x19. The validation set consisted of 73,089 randomly collected non-face images at resolution 19x19. We applied histogram equalization as proposed in 10 to decrease the variations caused by changes in illumination. To train the classifiers at lower resolutions we downscaled the training images to the proper size. Applying the above described training procedure resulted in a five level hierarchy with the 19x19 non-linear SVM on top, followed by a 19x19, 11x11, 4x4, and finally 3x3 linear SVM classifier. The number of examples removed in the bottom-up training procedure for the final hierarchy is given in Table 1. Table 1. Number of negative training examples used for training in bottom-up training procedures.
Bottom-up:
3x3 33,045
->
4x4 23,598
-*
11x11 18,149
->
19x19 10,568
As mentioned before, the topmost classifier was trained independently. We used the same training set (2,429 faces and 4,548 non-face patterns) as in 4 to allow for comparisons with the results reported there. In Figure 2 we show the ROC curves of the single second-degree polynomial classifier 4 and our hierarchical system on the CMU set l c . We selected the thresholds of the hierarchical classifiers in layers one to four according to the individual ROC curves on the training set d . The hierarchy performs better than the single classifier for recognition rates below 75%. Above b
Was set to 99% in the experiments. This set 9 consists of 118 images including 479 faces. In our experiments searching over multiple scales and locations resulted in processing of about 57,000,000 19x19 patterns. d W e selected the point on the ROC curve with 97% recognition rate. c
487
that, the single SVM classifier is superior, indicating that some of the difficult face patterns in the test set do not reach the last layer of the hierarchy. Figure 3 shows the data flow through the hierarchy and the time spent in each layer. The hierarchical system is about 260 times faster than the system with the single second-degree polynomial SVM.
Training: 2.429 faces / 4.548 non-facos Test: C M U sot 1 / 1 1 8 I m a g e s 1479 f a c o s / 5 6 , 7 7 4 , 9 6 6 w i n d o w s
O
0.00001
00OQO2
OOM03
000004
0O0006
000000
000007
000X0
000009
OOOOI
False positives / number of windows
Fig. 2. ROC curves of the hierarchical system and the original system with a single second-degree polynomial SVM for the CMU set 1.
% of tola! lime spent al each layer % of original patterns after each layer
11.1%
17.3%
47.9%
13.8%
9.9%
80.0%
14.6%
0.6%
0.2%
0.005%
19x19 linear classifier
19x19 poly classifier
faces (0.005%)
2.3%
80.0% JX.» linear classifier
Remaining
|20.0%
97.7%
Whole image 100%
Background (99.995%)
Fig. 3. Data flow and computing time for the hierarchical classifier tested on the CMU set 1.
In layers one to four most of the computation time (~90%) is used for feature extraction. Further optimizing the classifiers would not lead to a significantly faster system. In the last layer about 95% of the computing time is spent on the classification leaving much room for speed-ups using feature reduction methods. In the following sections we explore methods for feature reduction and apply them to the non-linear SVM at the top level of the hierarchy.
488
4. Dimension Reduction in the Input Space 4.1. Ranking
Features in the Input
Space
13
In a gradient descent method is proposed to rank the input features by minimizing the bound of the expectation of the leave-one-out error of the classifier. The algorithm showed superior performance compared to other feature selection methods (filter methods based on Fisher score, Pearson correlation coefficients and Kolmogorov-Smirnov test) for various classification tasks (face detection, person detection, cancer morphology classification). The basic idea is to re-scale the ndimensional input space by a n x n diagonal matrix a such that -^j is minimized. The new mapping function is then 3><x(x) = <J>(cr • x) and the kernel function is Ka(x,y) = K(a • x , a • y) = ($CT(x) •$ CT (y)). The decision function given in Eq. (1) becomes: e. /(x,<x) = w • $CT(x) + b = Y/a°iyiKa(xi,X)
+b
(6)
The maximization problem of Eq. (2) is now given by: t
W2(a,a)
1
i
= m a x ^ at — — ^ ^ ai^jyiyjKa(xi,Xj)
(7)
subject to constraints 5Z i= i aiVi = ® an<^ C > a.i > 0, i = 1, ...,£. The radius around the data is computed by solving the following maximization problem: R2{f3,a) = m a x ^ / ^ ( x , , X i ) - ^ & / ? , • # „ ( X i , x , )
(8)
subject to Y,iA = 1, f3i>0, i = !,...,£. We solve for a, /?, and <x using an iterative procedure: We initialize a as a vector of ones and then solve Eqs. (7) and (8) for the margin and radius, respectively. Using the values for a. and /? and the bound in Eq. (3) we compute a by minimizing W2(a, <J)R2(/3, a) using a gradient descent procedure. We then start a new iteration of the algorithm using the
489 Training: 2.429 ttcw / 4.548 non-focM. To.t: RMocod CMU tot / I IB ImaoM / 472 lata* / 23.873 wkidowa.
FBIID poriMvat / number of wlndowi
Training: 2.429 tstM / 4.54B nonJMM. Te*t: Rodutad CMU * M 111B Imagoa / 472 ( M M / 21,573 windows.
\) \
Fatso positives' number o( windows
Fig. 4. ROC curves for a) gray features and b) PCA gray features with the 60, 80 and 100 ranked features.
4.2. Selecting
Features
in the Input
Space
In Section 4.1 we ranked the features according to their scaling factors
EP.„<\E{^
(9)
To simplify the computation of our algorithm and to avoid solving a quadratic optimization problem in order to compute the radius R, we approximated 6 R by 2p where p is the dimension of the feature space IRP. For a second-degree polynomial kernel of type (1 + x • y ) 2 we get: EPerr < j 2p E {W2(a0))
< ~ n*(n* + 3) E (W2(a0))
(10)
where n* is the number of selected features f . The estimated bound on the expected error is shown in Fig. 5. We had no training error for more than 22 selected features. The bound drops over the first 30 features, stays about the same between 30 to 60 features, and then increases steadily. This bound is considered to be a loose bound on the expected error. To check if it is of practical use for selecting the number of features we performed tests on the CMU set 1. In Fig. 6 we compare the ROC curves obtained for different numbers of selected features. The results show that using more than 60 features does not improve the performance of the system. This observation coincides with the run of the curve in Fig. 5. However, the error on c
We previously normalized all the data in lR n to be in a range between 0 and 1. As a result the points lay within a p-dimensional cube of length %/2 in IRP and the smallest sphere including all the d a t a points is upper bound by \flp. f As we used a second-degree polynomial SVM the dimension of the feature space p = n * ( n * + 3 ) / 2 .
490 Estimation of Bound on Generalization Error
4«E*W y'
X
a t I
x
- * • PCA gray features
*5
70
85
1M
r145
. 170
. 193
. 220
2*5
N u m b e r of selected features
Fig. 5. Approximation of estimated bound on the expected error versus number of principal components. The values on the y-axis are not normalized by t h e number of training samples.
the test set does not change significantly for more than 70 features although the estimated bound on the expected error shown in Fig. 5 increases. An explanation for this could be that the bound gets looser with increasing number of features because of our approximation of R by the dimensionality of the feature space. Training: 2.429 faces / 4,546 non-faces. Test: CMU set 1,118 images / 479 faces / 56.774.966 windows.
D'l
06
ttf^^^Z^^"
0 7
y
0.6
-»- 30 PCA gray features
08
- • - 60 PCA gray features 0".
-*-100 PCA gray features 0.3
- * - complete sei of features (283 gray features)
0,2 0.00001
0.00002
0.00003
0.0O004
0.00005
0.00006
0.00007
0.00008
O.0O009
0.0001
False positives / number of windows
Fig. 6.
ROC curves for different numbers of PCA gray features.
5. Feature Reduction in the Feature Space In the previous Section we described how to reduce the number of features in the input space. Now we consider the problem of reducing the number of features in
491 the feature space. We use a new method based on the contribution of the features from the feature space to the decision function / ( x ) of the SVM. t
/ ( x ) = w • $(x) + b = £ a%iK(^,
x) + b
(11)
i=l
with w = (wi,...,wp). For a second-degree polynomial kernel with K(x,y) — (1 + x • y) 2 , the feature space Htp with dimension p = ("+ 3 )" is given by x* = (y/2x\,--- ,\/2xn,x\,--,Xn,y/2xiX2,--- , \/2a;„_ix n ). The contribution of a feature x*k to the decision function in Eq. (11) depends on Wk- A straightforward way to order the features is by ranking \wk\- Alternatively, we weighted w by the support vectors to account for different distributions of the features in the training data. The features were ordered by ranking \wkYliVix*i,k^ w n e r e x*,k denotes the fc-th component of support vector i in feature space JRP. For both methods we first trained an SVM with a second-degree polynomial kernel on 60 PCA gray features of the input space which corresponds to 1,891 features in IRP. We then calculated E(S) = ^\f(xi)-fs(xi)\
(12)
i
for all s Support Vectors, where / s ( x ) is the decision function using the S first features according to their ranking. Note that in contrast to the previously described selection of features in the input space this method does not require the retraining of SVMs for different feature sets. The results in Fig. 7 show that ranking by the weighted components of w lead to a faster convergence of E(S) from Eq. (12) towards 0. Fig. 8 shows the ROC curves for 500 and 1,000 features. As a reference we added the ROC curve for a second-degree SVM trained on the original 283 gray features. This corresponds to - — ^ — = 40,469 components in the feature space. By combining both methods of feature reduction we could reduce the dimensionality by a factor of about 40 without loss in performance. 6. Experiments In the final experiments we combined feature reduction with hierarchical classification. In Figure 10 we compare the ROC curves of the hierarchical system and the single SVM classifier with second-degree polynomial kernel. The hierarchy performs better than the single classifier up to a recognition rate of 80%. For higher recognition rates, the single SVM classifier performs slightly better because some of the more difficult face patterns in the test set do not reach the last layer of the hierarchy. The average computing time on a Pentium IV with 1.8 GHz for an image of size 320 x 240 is given in Table 2. We sped-up the detection by a factor of 335 compared to the original system achieving a processing speed of 4 frames/sec. The face detector proposed in 12 is about 4 times faster at runtime. However, it requires several weeks for training while our system can be trained in a day.
492 Partial S u m f o r S u p p o r t V e c t o r s
l
fe p—|
7
W
j!
5 4
weighted w
n,* *-««—j
1
li/w^.
0
1
L _
~r^-t
— Number of selected features
Fig. 7. Classifying support vectors with a reduced number of features. The x-axis shows the number of features, the y-axis is t h e mean absolute difference between the output of the SVM using all features and the same SVM using the S first features only. The features were ranked according to the components and the weighted components of the normal vector of the separating hyperplane.
Training: 2.429 faces / 4.548 non-faces. Test: CMU set 1 n 18 Images / 479 facos / 56.774,966 windows.
000001
000003
000003
0CO0O4
000005
000006
000007
001
000009
00001
Falso positives / number of windows
Fig. 8.
ROC curves for different dimension of the feature space.
In Figure 10 we show the results for an example image from CMU set 1. The original image is shown in Figure 10 a), Figures 10 b) to e) show the outputs of the SVM classifiers at levels two to five, respectively. Dark pixels represent high output values of the SVM indicating the presence of a face, white areas have been removed by previous classifiers in the hierarchy. In Figure 10 f) we show the final detection results as computed by the top level second-degree polynomial SVM classifier. Some more example images from CMU set 1 are shown in Figure 11.
493 Table 2. Average computing time per 320 x 240 image for various face detection systems. Each image was processed at five different scales to detect faces at resolutions between 30 x 30 and 70 x 70 pixels. System Single SVM Single SVM with Feature Reduction Hierarchy of SVMs Hierarchy of SVMs with Feature Reduction
Time / Image [ms] 86,875 23,383 391 259
Speed-up Factor
3,7 222.2 335.4
Training: 2,429 f a c e s / 4,548 n o n - f a c e s T e s t : CMU set 1 / 1 1 8 i m a g e s / 479 f a c e s / 56,774,966 w i n d o w s
000001
060002
00OO0J 000004 000005 O.0O0OB 000007 False positives / number of windows
OOOOOB
Fig. 9. ROC curves of the hierarchical system and the original system with a single second-degree polynomial SVM for the CMU set 1.
7. C o n c l u s i o n a n d F u t u r e W o r k
In this paper we presented speed-up methods for a face detection system based on hierarchical classification and feature reduction. To quickly remove large background parts of an image we arranged five SVM classifiers with increasing computational complexity in a hierarchical structure. We proposed an iterative algorithm for automatically building and training a hierarchical system of classifiers. To further speed-up the detection system we applied feature reduction to the non-linear SVM at the top level. This was accomplished by ranking and then selecting PCA gray features according to a classification criterion that was derived from learning theory. Applying these methods to face detection, we removed 99% of the features without loss in classification performance. Experiments show that the combination of feature selection and hierarchical classification results in a speed-up factor of 335 while maintaining classification accuracy. In the current system image preprocessing and feature extraction account for 90% of the runtime. In future work we will explore alternative feature extraction and preprocessing techniques to further speed-up the detection.
494
Fig. 10. Detections at each level of the hierarchy, a) Original Image, b) to e) Outputs of layers two to five. Dark pixels represent high outputs of the classifier, white areas have been removed by previous layers, f) Final detection result computed by layer six.
Fig. 11.
Face detection with the hierarchical system.
Acknowledgements This article is reprinted from Publication P A T T E R N R E C O G N I T I O N , Vol 36, No 9, B . Heisele, T. Serre, S. Prentice, T . Poggio, "Hierarchical classification and feature reduction for fast face detection with support vector machines", p p . 20072017, 2003, with permission from Elsevier.
References 1. 2. 3.
Blum, A., Langley, P., 1997. Selection of relevant features and examples in machine learning. Artificial Intelligence 10, 245-271. Burt, P. J., 1988. Smart sensing within a pyramid vision machine. Proc. of the IEEE 76(8), 1006-1015. Edwards, J., Murase, H., 1997. Appearance matching of occluded objects using
495
4.
5. 6.
7. 8. 9.
10.
11. 12. 13.
coarse-to-fine adaptive masks. Proc. of the IEEE on Computer Vision and Pattern Recognition , 533-539. Heisele, B., Poggio, T., Pontil, M., 2000. Face detection in still gray images. A.I. Memo 1687, Center for Biological and Computational Learning, MIT, Cambridge, MA. Kohavi, R., 1995. Wrappers for feature subset selection. Artificial Intelligence, special issue on relevance 97, 273-324. Oren, M., Papageorgiou, C , Sinha, P., Osuna, E., Poggio, T., 1997. Pedestrian detection using wavelet templates. IEEE Conference on Computer Vision and Pattern Recognition, San Juan, 193-199. Rosenfeld, A., Vanderbrug, G. J., 1977. Coarse-fine template matching. IEEE Transactions on Systems, Man and Cybernetics 2, 104-107. Rowley, H. A., 1999. Neural network-based face detection. Ph.D. thesis, CMU, School of Computer Science, Pittsburgh. Rowley, H. A., Baluja, S., Kanade, T., 1997. Rotation invariant neural networkbased face detection. Computer Scienct Technical Report CMU-CS-97-201, CMU, Pittsburgh. Sung, K.-K., 1996. Learning and example selection for object and pattern recognition. Ph.D. thesis, MIT, Artificial Intelligence Laboratory and Center for Biological and Computational Learning, Cambridge, MA. Vapnik, V., 1998. Statistical learning theory. John Wiley and Sons, New York. Viola, P., Jones, M., 2001. Rapid object detection using a boosted cascade of simple features. Proc. of IEEE Conference on Computer Vision and Pattern Recognition. Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V., 2001. Feature selection for support vector machines. Advances in Neural Information Processing Systems 13.
CHAPTER 5.1 TRACKING AND CLASSIFYING MOVING OBJECTS USING SINGLE OR MULTIPLE CAMERAS
Quming Zhou and J. K. Aggarwal Department of Electrical and Computer Engineering The University of Texas at Austin, Austin, TX 78712, USA This chapter presents methods for tracking moving objects using single or multiple cameras. Objects are classified into three categories: single person, people group or vehicle. For the case of the single camera, the proposed method integrates spatial position, shape and color information to track object blobs. Further, it focuses on establishing a correspondence between objects and templates as the objects come into view. For the case of multiple cameras, we fuse data from individual cameras using an Extended Kalman Filter (EKF) to resolve object occlusion. Our results show that integrating simple features makes the tracking effective and that EKF improves the tracking accuracy when long term or temporary occlusion occurs. Successful results for PETS2001 and other image sequences are presented.
1. Introduction The efficient tracking and classification of multiple moving objects is a challenging and important task in computer vision. It has applications in surveillance, video communication and human-computer interaction. Recently, a significant number of tracking systems have been proposed. Some address lowlevel feature tracking while others deal with high-level description, such as recognition, event detection and trajectory interpretation. The success of highlevel description relies on accurate detection and tracking of moving objects and on the relationship of their trajectories to the background. There is a considerable diversity among the trackers proposed by various researchers. Haritaoglu, Harwood and Davis [15] classified the feature trackers into several categories according to their sensor types (single or multiple camera, color or gray-scale) and their functionality (tracking single or multiple objects, handling occlusion). Cai and Aggarwal [6] proposed a tracker to establish feature correspondence 499
500
between consecutive frames under different views. They tracked coarse human models from sequences of synchronized monocular grayscale images in multiple camera coordinates. Stauffer and Grimson [30] developed a vision system extending the traditional tracking tasks, which dealt only with low-level feature tracking, to sequence classification and unusual event detection. Their system passively observed moving objects and learned patterns of action from those observations. Another event detection method applied to video surveillance was proposed in [31]. The most widely used cues in object tracking are spatial position, shape, color, intensity and motion. Many uncontrollable factors such as lighting, weather, unexpected intruders or occlusion may affect the efficiency of tracking when these cues are used in an outdoor environment. One solution is to combine two or more cues. Another solution is to use multiple cameras. We shall consider both of these options in this chapter. In the following, we discuss additional research relevant to two issues of integration of features and multiple cameras. 1.1 Feature Integration Multi-feature integration has been exploited extensively in featured-based tracking applications. Shi and Tomasi [29] classified a tracked feature as reliable or unreliable according to the residual of the match between the associated image region in the first and current frames. A feature should be abandoned when the residual grows too large. This work [29] was extended by Tommasini et al. [34], who introduced a simple, efficient, mode-free outlier rejection rule for rejecting spurious features. Kaneko and Hori [18] proposed a feature selection method based on the upper bound of the average template matching error. The selection criterion is directly related to the reliability of tracking. More complex integration methods using statistical models have been presented in [40, 24]. Triesch and Malsburg [35] presented a system for integrating features in a self-organized manner. In their system, all features agreed on a result, and each feature adapted to the result agreed upon. The self-organized adaptation led to suppression and recalibration of discordant features, i.e., features that did not agree with the overall result. Experiments showed that the system was robust to sudden changes in the environment as long as the changes disrupted only a minority of the currently used features at the same time, although all features might be affected in the long run. Cai and Aggarwal [6] used a Bayesian classifier to find the most likely match of the subject in the next frame. Multiple features were modeled by a joint Gaussian function. The most likely match was the minimum value among the candidates with a joint Mahalonobis distance less
501
than a certain threshold. Rigoll et al. [25] combined two stochastic modeling techniques. Pseudo-2D Hidden Markov models were used to capture the shape of a person within an image frame. A Kalman filtering algorithm was applied to the output of the first model to track the person by estimating a bounding box trajectory indicating the location of the person within the entire video sequence. Both algorithms cooperated in an optimal way. 1.2 Multi-camera Tracking Single camera tracking is hampered by the camera's limited field of view. Occlusion is always a challenge for single camera trackers, while multi-camera trackers can utilize the different views to obtain more robust tracking, especially for wider fields of view or occluded objects. Multi-camera tracking is a correspondence problem between objects seen from different views at the same time or with a fixed time latency. It implies that all views of the same object should be given the same label. Khan and Shah [19] presented a system based on the field of view lines of the camera to establish equivalence between views of the same object in different cameras. These lines were used to resolve any ambiguity between multiple tracks. The authors claimed that no calibration was required. Cai and Aggarwal [5] used only relative calibration between cameras to track objects over a long distance. Dockstader and Tekalp [8] introduced a distributed, real-time computing platform to improve feature-based tracking in the presence of articulation and occlusion. Their contribution was to perform both spatial and temporal data integration within a unified framework of 3D position tracking to provide increased robustness to temporal feature point occlusion. A full calibration was employed to build the 3D model. Lee [20] recovered the full 3D relative positions of the cameras and the domain plane of activity in the scene using only the tracks of moving objects. For a more thorough discussion of modeling, tracking and recognition for human motion analysis, we refer the reader to Aggarwal and Cai [1], Gavrila [13] and Collins et al. [7]. 1.3 Object Classification Despite the significant amount of literature on video surveillance, little work has been done on the task of classifying objects as single person, people group, or vehicle. Tan, Sullivan and Baker [33] developed an algorithm to localize and recognize vehicles in traffic scenes. Lipton, Fujiyoshi and Patil [21] classified
502
moving targets from a video stream into human, vehicle or background regions. They used dispersedness value as a classification metric based on the assumption that humans were, in general, smaller than vehicles and had more complex shapes. This assumption became somewhat tenuous in cases where the humans were closer than the vehicles to the camera, where humans grouped together, or when vehicles were occluded. Foresti [11] described a visual surveillance system to classify moving objects into car, motorcycle, van, lorry or person. In his system, object classification was based on a statistical morphology operator, the spectrum, which was used to index large image databases containing multiple views of different objects. Since the spectrum was invariant to object translation, rotation, and scale variations, a simple distance measure was applied as a matching function to individuate the correct object class in the database. In the classification system designed by Stauffer and Grimson [30], when given one or more observations of a particular object, the system classified it into a set of classes such that the same type of object tended to be put in the same class. The number of classes was not predetermined but depended on the input sequences. Their classification method used on-line vector quantization to generate a codebook of prototype representations. Then, the automatically tracked sequences were used to define a co-occurrence matrix over the prototypes in the codebook. Following that, the co-occurrence data were used to probabilistically break apart the prototypes in the codebook into a binary tree representation. The final result was a hierarchical classifier that could classify any individual representation. 1.4 Overview of this Chapter In this chapter, we address the issues of tracking and classifying moving objects from video sequences acquired by stationary cameras. Our tracking objective is to establish a correspondence between the image structures of consecutive frames over time to form persistent object trajectories. Our processing includes: 1) automatically determining the initial positions of moving objects, 2) extracting feature information from all moving objects within view, 3) tracking detected objects by features, 4) classifying the moving objects into three categories: single person, people group or vehicle, and 5) fusing data by an Extended Kalman Filter (EKF) to resolve object occlusion. Our tracking system integrates spatial position, shape and color. This integration makes the tracker insensitive to background changes, motion interruption and object orientation. We detect motion and background variation to segment the moving object blobs,
503
and then compare the similarity of the object blobs with different templates, thereby tracking the objects. We use simple thresholds instead of comparing the Mahalanobis distances between the templates and objects. In multi-camera tracking, we focus on developing a methodology for tracking multiple objects in the views of two different fixed cameras, whose transformation matrix and focal length are estimated from five coplanar control points. The single camera tracking uses 2D image coordinates, whereas in multicamera tracking we transform to real world coordinates. We apply an Extended Kalman Filter to fuse the separate observations from the two cameras into a position and velocity in real world coordinates. This enables our tracker to switch from one camera's observation to the other when the target object is occluded from view. We initialize tracking by solving a constrained linear least squares problem. The constraint is that a moving object in the real world always has some height. This helps us to remove the shadow on the ground plane, whose height is zero. Recently, the Kalman filter has been used in multisensor data fusion [12, 26]. The Kalman filter can be applied to each sensor's data separately [22, 38] or it can be applied simultaneously to data from multiple sensors, combined through additional logic [9, 27]. Our underlying assumption for using the Kalman filter to fuse data is that there is a mathematical relationship between the target object's image positions in two synchronized cameras [4]. Furthermore, measurements from two synchronized cameras provide enough information to estimate the state variables of the system, the position and velocity in real world coordinates. After obtaining an accurate description of the observed object, we classify the objects and update the templates, taking into account any occlusion. Our chapter presents two robust classification metrics to classify the target object into single person, people group or vehicle, namely the variance of motion direction and the variance of compactness. These two metrics are independent of the target object size and orientation and the camera used. The classification allows our tracker to know what kinds of objects are moving in the scene and to detect when two people or more come together to form a group, or separate from each other, dissolving a group. The remainder of this chapter is organized as follows. Section 2 describes the algorithm for moving object segmentation in an outdoor environment. Section 3 focuses on extracting features from the moving blobs. Section 4 develops a hierarchical approach to compare the blobs with the templates in order of position, shape and color. Classification metrics are derived in Section 5. Section 6 provides experimental results to demonstrate the accuracy and discusses the problems of tracking using a single camera. The associated classification results
504
are also presented in this section. Section 7 applies an EKF to take advantage of multiple cameras. Experimental results using EKF are given in Section 8. Finally, Section 9 presents conclusions. Fig. l(a)-(b) shows the architecture of our proposed systems using a single camera and multiple cameras, respectively.
Centroid Projection in Camera I
1
Single Camera Module I
^ Tracking
w
Centroid
^ W
Extended Kalman
Initialization
Single Camera Module 11
—w
t
Real
^
W
Filter
Centroid
World Position
^
w
Tracked Objects
W Centroid Proje ction in Cam 2raII
Figure 1. (a) Architecture of the proposed system using a single camera, (b) Architecture of the proposed system using multiple cameras
2. Moving Object Detection The first step in tracking objects is to segment the objects from the background. Two common methods are motion segmentation and background subtraction. Motion segmentation is basically a threshold of the difference between the current image and the subsequent image based on the assumption
505
that the background does not change over successive frames [17, 37]. This method is easy and fast in many applications, but some problems appear when tracking multiple objects or when object motion stops. Hence, we turn to background subtraction at the expense of updating the background [14, 10]. A pixel-wise median filter with L frame length is employed to build the background under the assumption that a moving object would not stay at the same position for more than 0.5L frames. L is typically of the order of twenty. If the object were to stay still more than 0.5L frames, it would be incorporated into the background. In general, I(x,y,t) will denote the intensity at a point (x,y) at time t. This notation is shortened to Ik (m, n) to denote the intensity at pixel (m, n) at frame k. The background model for pixel (m, n) with an intensity of Ik (m, n) at frame k using a median filter is Ik (m, n) = median, (Ik_05L (m, n), Ik_0 5L+X (m, «),•••, Ik+05L (m, n)) ,
(1)
which retains the stationary pixels in the background. Here Ik (m, ri) represents the observed intensity and Ik(m,n) denotes the model for the background. A median filter can build the background even when moving objects exist in the scene, but usually requires a large amount of memory to save L frames at a time. So, it is applied only when we detect a large new blob that lasts for r/h frames, jjh = 5 in our examples. Background subtraction is performed in color and in edge density. We subtract the foreground from the background in each of RGB color channels and then take the maximum absolute values of these three differences as the difference value Diffc in color space as Diffc (Hi, n) = max) | Rf -Rh\,\
Gf -G„\,\ Bf -Bb\\,
(2)
where Rh, Gh and Bh are background color channels, Rf, (Jf and Bf are foreground color channel. We do a similar subtraction using edge density, Dens values, instead of color values, and thus obtain the difference value Diffe in edge density. The edge density Dens is defined as (2w + l ) - ( - , t ^ w ^ .
dx
dy
where 2w + 1 is the size of an average window filter. The binary foreground pixels F{m, n) are computed as F(m,n)
[I if Diffc (m, /I) > Thx OR Diffc, (m, n) > Th, =] . , (^ 0 otherwise
(4)
506
where 77?, and Th2 are fixed thresholds determined empirically. Our experiments show that color and texture are complementary when the target object is small in size relative to the entire image.
Figure 2. (a) Original picture, (b) Background extracted by the median filter, (c) Moving blobs
The resulting foreground image contains noise due to the clutter in the background. The 'close' binary morphological operator is applied to remove noise. Now, we segment the foreground F into several isolated blobs by an eightconnected algorithm. We assume that each initial blob has a moving object, which may be a person, a vehicle or a people group. Fig. 2 (a) - (c) shows an example of the above segmentation. The extracted blobs represent the moving objects in the image sequences. 3. Feature Extraction We extract four types of features for each moving blob. These features are generally used in object tracking except for the variance of motion direction, which is used for classification purposes. The centroid of a blob (M, N ) tells us the spatial position of the blob. It is the average position of all pixels in the blob. (w,w) is the position of a given pixel in a blob for a given time. Since an object will not move far from its last position from one frame to the next, its centroid provides us with a strong and useful feature for tracking objects. Shape features provide us with shape-based information about the objects. An object's shape generally does not change much between consecutive frames. We select four features, length and width, area, compactness and orientation. The object orientation is defined as the angle between the major axis and the horizontal axis. An object with a more complex shape is more likely to change its compactness than an object with a simple shape, assuming that the same segmentation method is used. We define these features by:
507
• •
Length and width (L, W) L = max(w) - min(w) ,
W = max(w) - mm(n) .
(5)
Area
(6)
A= XI^'")' (m,n)eR
where F(m,n) is the binary foreground pixels in a blob. •
Compactness c =
4K* Area Perimeter'
where Perimeter is the number of all boundary pixels including the boundaries of inside holes. • Orientation 1 0 = -arctan(
2wLin ^20
—
—) ,
(8)
^02
where upq = ^ ^ ( w - M ) ' ' ( n - N ) "
for /?,
The use of color as a feature allows us to track objects when shape information is not reliable. Color is independent of the object size and orientation and especially useful when we can detect only partial objects. However, color is likely to change with the lighting. Most previous research [39, 23, 16] has concentrated on how to stabilize color in some color space, e.g. HSV, normalized RGB and YIQ. In our work, we use principal component analysis (PCA) to extract the color feature of the object. PCA is also called eigenspace decomposition and is applied to reduce the number of dimensions of the working space. It projects d-dimensional data onto a lower-dimensional subspace in a way that is optimal under a least square error criterion. We choose the axis that maximizes the variation of the data as our principal axis. This axis is the eigenvector corresponding to the maximum eigenvalue. We define the color similarity measure between objects and templates to be the transformation between their principal axes. All pixel RGB color values are collected from each blob. Three color channels of a pixel are denoted by a 1x3 vector rgb = [R G B]. The average color of all 77 pixels is given as 1 n rgh=-Y,rSbi-
u
(9)
508
Then, a new vector is defined as rgb = rgb - urgb. By concatenating all 77 pixels, an j] x 3 matrix RGB is formed to express the red, green and blue components of the full data set as RGB = [rgbx; rgb2 ;••• rgbn J. Next, a 3x3 covariance matrix E is calculated by 1, = -RGB'RGB
(10)
and the eigenvector and eigenvalue matrices are computed as SO = A®. We take the eigenvector O, corresponding to the largest eigenvalue A, as the principle axis. If two blobs have similar color, their principle axes should be similar. After we extract the above features for each object blob, the feature vector Rlk will represent the blob, whereRlk =[M,N,L,W,A,C,d,^>\]Our tracking is based on the extracted feature vector instead of the blob itself. 4. Object Tracking Our tracking process compares the feature vector Rlk with all templates Tlk_x (i=l, 2... T ). If a match is found, then the template is updated for the next match through an adaptive filter of the form Tlk={\-P)T,k_x+(3R,k
,
(11)
where the learning parameter /? depends upon how fast the template is updated. If no match is found for several successive frames, then a new candidate template TT+] k will be created. A template will be deleted from the candidate template list if it is not matched with any object for successive L frames, the length of the median filer. A hierarchical matching process is performed in the order of centroid, shape and color. As soon as a match occurs, the process of matching terminates for the given blob. Centroid matching In general, an object's next position will be in the neighborhood of its current position. We calculate the distances from a target object to all templates and sort the distances from the minimum to the maximum, namelyd mm ,d 2 ,---d max . Three thresholds, 77?,, Th4 and Th5, are used in our centroid matching. An object is likely to be a new object if itst/min > Th3. A match occurs if a distance is the dmin such that dmm < Th4. Considering the case when occlusion occurs, we add a third constraint to avoid a possible mismatching, the second minimum distance d2 > dma + Th5. This constraint prevents mismatching if there are two or more
509
objects at a distance less than threshold Th4 from the template. If the centroid matching procedure fails to match the target object to any template, then the next matching procedure, shape or color matching, is necessary. To reduce the possibility of a false match, we compare the target object's shape or color only with templates that are the first four minimum distances from the object. Shape matching We compute the distance from the shape feature vector [A, C, 0] to the template as Dis = (A/Atemp - l ) 2 +(C/C,emp - l ) 2 +(0/dlemp - l ) 2 ,
(12)
where A, , C, and tit a r e t n e a r e a ' compactness and orientation of the template, respectively. The object is assigned to the template that yields the least distance if the distance is less than a predetermined threshold. To achieve a good measure of dissimilarity using distance, we normalize the shape feature prior to calculating the distance. Color matching We use a similar function to compare the angles between the principle axes of the color of the object and the template. The normalized inner product
s($„or)=
;
r
en)
P*ll IP r I is used as the similarity function. The value of this metric is larger when blob d>R and template <£r are similar. This measurement, which is the cosine of the angle between
510
5. Object Classification The motion feature is an efficient way to track humans and vehicles [3, 28]. Vehicles are more consistent in their motion because they are rigid objects, whereas humans shift some parts of their bodies backwards when they move forward to maintain balance. So the variance of motion direction is employed to measure motion consistency. We derive this feature from optical flow. Optical flow is widely used to estimate motion based on the assumption that the change in image brightness in the sequence is due only to motion. This assumption leads to a brightness constraint equation as dl, .dx dl, .dyn ,, ^ N dl . — (x,y,t) + — (x,y,t) — + —(x,y,t)-^ = 0 , (14) dt ox dt oy dt where I(x, y, t) is the image intensity as a function of space (x, y) and time t. This equation does not determine the flow field since it only has one restriction for two parameters dx I dt and dyldt. In order to obtain a unique solution, an additional smoothness constraint is imposed as argmin{(|)^(|)2}.
(15)
By Lagrange optimization, the solutions are dl 81 dt dx
dx =VxJ
~dl
ox
dl_dl_ and
dt dy
dy dt ox
oy
oy
where vT, and v v , are the velocities along the x axis and y axis at time t, respectively. The weighted velocity sums of three successive frames with weights, a_x, a0 and a, are v
x,t =a-\vxj-\ +a0vXJ + axvxl+x ,
(17)
Vyl = a_xvyJ_x + a0vvl + axvvl+x .
(18)
Then, a Gaussian filter G is applied, yielding Vl, =VXJ ®G and V*yJ=Vyjt®G
,
(19)
where the symbol ® is a convolution operator. We define motion direction as S = arctm2(Vv,
Vxl) ,
(20)
511 where arctan 2 is the four quadrant arctangent function. We calculate 6 for each pixel in an object blob and compute the variance crj of all 5 ' s in the blob. Our experiments show that crj is an efficient metric to distinguish a single person. The variance of motion direction aj cannot discriminate a people group from a vehicle. We added the variance of compactness cr; into consideration based on the observation that the shape of a people group tends to change dramatically, which is measured by the variance of compactness over frames, denoted as Ths. Then, a people group is differentiated from a vehicle by its large variance of compactness over frames as Th . The remaining objects are classified as vehicles. In our classification, a bicycle is regarded as a vehicle.
Figure 3. (a) The van is still, (b) When the van moves, a new candidate template is created, (c) The candidate template lasts for 3 frames and thus becomes a true template and tracking begins
6. Experiments in Single Camera Tracking and Classification We used the PETS2001 datasets and other videos to evaluate the single camera tracking system. PETS2001 datasets were provided by the second IEEE International Workshop on Performance Evaluation of Tracking and Surveillance. Each video is digitized with a frame rate of 25 frames per second. Datasets include moving people and vehicles. These videos are challenging in terms of significant lighting variation, occlusion and scene activity. Figs. 3-5 show some sample frames from test videos. The van in Fig. 3(a) stayed still for a long time; hence, it was merged into the background. It suddenly
512 started in the next frame. A 20-frame median filter updated the background, and a new candidate template was created, shown in Fig. 3(b) by a rectangular blob. In Fig. 3(c), this new candidate lasted more than three successive frames and thus began to be tracked as a moving object. In Fig. 4(a), a car, which was automatically classified as vehicle, approached a tree. In Fig. 4(b), the car appeared to be separated into two parts by the tree. The system recognized that these parts were the same. Fig. 4(c) shows the car emerging from behind the tree. We detected occlusion by comparing the blob size. If the size reduction exceeded a certain threshold for three successive frames, an occlusion was deemed to have occurred. In this case, only the centroid components of the template were updated. This was done using a linear prediction by assuming the velocity same as the preceding frame.
Figure 4. (a) A car approaches the tree, (b) The car appears to be in two parts, (c) The car emerges from behind the tree
Figure 5. (a) Before grouping, (b) Grouping, (c) After grouping Fig. 5 presents an example of how to track several objects which group
together and then come apart. In Fig. 5(a), a bicycle was passing a car. The bicycle blended with the car in Fig. 5(b). This blending ended at Fig. 5(c), and the objects again were tracked individually. A classification example is drawn from the PETS2001 sequence with a tree presented earlier in Fig. 4. The variance of motion direction aj and the variance of compactness a; are used as the classification metrics. Fig. 6 (a) - (b) shows
513
the variance of motion direction and the compactness of a single person, a people group, a vehicle and a bicycle. A single person is characterized by a high crj while a people group has a high a*. The variance of motion direction of a single person is above 1.2 on average, as shown in Fig. 6(a), while those of others are lower than 0.9. A threshold of variance of motion direction, 77?% =1.0, can be chosen so that a single person is detected when crj > 1.0 . It is clear from Fig. 6(b) that the variance of compactness of a people group is as high as 0.18, while those of other classes are lower than 0.02. Therefore, a threshold, 77? =0.1 can classify a people group bycr,: >0.1. A bicycle is regarded as a vehicle in the classification. The classification error is less than 5% in our experiments [45]. Variance of molion direction
Variance ol compactness
05 0
5
10 Irame
15
20
0
SgggggSSSSSSSSSSSggJ 5
10 frame
15
20
Figure 6. 0: vehicle. o:bicycle. x: single person. *: people group, (a) Variance of motion direction, (b) Variance of compactness
Figs. 7-9 show several examples where individual features failed to track objects. However, by integrating of the centroid, shape and color features, we can track the objects accurately. Fig. 7 shows a sudden action. The predicted position based on constant velocity failed to track the object since it quickly changed its direction of motion. Fig. 8 shows a large group of persons. The constant position predication did not work because many objects changed their positions when several persons met together. Fig. 9 shows a dynamic background with shadow. The shape feature was not reliable because the shadow was included in the shape feature. Our integration of multiple features makes the tracker more robust in these three sequences. All these sequences were tracked correctly as shown in Figs. 7-9. Classification results of Fig. 9 are shown in Fig. 10. Given the thresholds Ths=\.0 and Th_ =0.1, the single person is detected, as itso-j =1.4>Ths, and the people group with three people is also detected, as its variance of compactness is 0.14, greater than the chosen threshold Thg.
514
Figure 7. Sudden action
Figure 8. Large people group
Figure 9. Dynamic background
In summary, the proposed system was used to analyze several videos and gave promising results. The background updating met problems when the lighting changed gradually. In this situation, our median filter kept updating the background without holding a static background. Vehicles were tracked successfully in part because of their large size. People tracking worked reliably through all sequences as long as the people were sufficiently isolated from each other. Groups of people were not handled well in that they significantly occluded each other's outlines in the image. The overall performance of single camera tracking was good. We accurately tracked 13 objects out of 17 moving objects from the PETS 200 l's videos, with an accuracy of 76%. Among the failed objects, three were due to occlusion. Our own testing videos gave an accuracy of 82% from 50 objects.
515 Variance of motion direction
Variance of compactness
0.15
<»»»»»««
W l » l » » « l i « ] '
li I
0.05
"~0
5
10 frame
15
20
n x x x x x x x x x x x x x x x x x x x : "0 5 10 15 20 frame
Figure 10. Classification results from Fig. 9; x: single person, *: people group.
7. Multiple Camera Tracking In the previous sections, we developed a framework to track objects using a single fixed camera. Although we found the results of our single-camera tracker to be encouraging, there are some unresolved problems, mainly due to object occlusion. For example, the bicycle shown in Fig. 4 was tracked as a different object once it passed behind a tree. This section focuses on developing a methodology for tracking objects in the views of two fixed cameras. We consider the tracking problem as a dynamic target tracked by two disparate cameras, each with different measurement dynamics and noise characteristics. We combine the multiple camera views to obtain a joint tracking that is better than the single camera tracking in handling occlusion. In this chapter, we fuse the individual camera observations to obtain combined measurements and then use a Kalman filter to obtain a final state estimate based on the fused measurements [32]. Measurement fusion is accomplished by increasing the dimension of the observation vector of the Kalman filter, based on the assumption that there is a mathematical relationship between the positions of an object Q in two disparate camera views. Since the mapping from one camera's view to the other view is nonlinear, an Extended Kalman Filter is used to estimate the state vector [2].
7.1 Camera Calibration In order to determine the mathematical relationship between the observations from two different views, we calibrate the two cameras using five coplanar control points [36]. Another calibration method using three coplanar points has been reported to track the ground point[42]; however, since the ground point is not always reliable due to noise, we use five points for our calibration. Fig. 11
516
shows the five coplanar control points. Points A, B, D and E are four vertices of a rectangle and point C is the center of the rectangle. For our application, an accurate camera calibration simply means that the 2D image coordinate can be properly predicted given the 3D location of the object.
Figure 11. The control points on the ground plane (z=0)
The transformation from the real world position (xw,yw,zw) to the camera 3D coordinates (xc,yc,zc) is given by a rotation matrix and a translation vector as =
\TU
r21 r31~|
T
T
u
7.3
hi ^23
n
yw z
?33. _ »-.
rr4,i +
743.
where the rotation matrix and the translation vector are called the homogeneous transformation T . The transformation from 3D camera coordinates (,xc,yc,zc) to the ideal (undistorted) image coordinate (Qx,Qy) is obtained using perspective projection with pinhole camera geometry as Qx=^f and £\.= — / • (22) zc zc where / is the effective focal length. Using the calibration technique presented by [36], we know the homogeneous transformation T, focal length fr and the image center (TX,'J\.) for camera I and for camera II, S, fs and (Sx,Sy). Therefore, we have the following equations for image positions(£^(x,£),..), (Qix^Qix) anc* t n e r e a ' world position
(xv,yw,zw).
517
xT
2„ =xwv Tnu n
+T +yj2\ +zJn < !LfT+Tx=^-fT+Tx, + ywT23 + zwT33 + T43 'c,\
_ xwTn + yJn
tii v -
X
Q2,=
+ ZyTyi + Tn f ^+T
;x~
^
n^ll +^^^21
xr ^r~ /?• •* v
+ Z S W 3\
+S
4\
r
__ ^ci ,
x
X
^w^B
+
X"
7,
X
>V^23 + 2».S 33 + £"43
r
+ s
-z„
_ xwSn + yvS22 + zwS32 +Si2 y 2v
(24)
/r
_ -V2
+ s
s
xwSi3+ywS23+zwS33+SA3
(23)
_ yc?_ JS
+
^ i -
^c ,
JS
+
^ i
(25)
(26)
Knowing the cameras' calibration, we are able to merge the object's tracking from two different views into a real world coordinate view. The 3D tracking data is the state vector X k, which cannot be measured directly. The object's positions in the two camera views are the measurement. We assume a constant velocity between two consecutive frames. The dynamic equation related to the state vector Xk is described as follows, XM=Q>kXk+Wk,
(27)
where Xk =[xw yw z„. xw yv zw]' , the spatial position and velocity in real world coordinates, AT denotes the time step in the transition matrix
Ot
1 0 0 0 0 0
0 0 AT 1 0 0 0 1 0 0 0 1 0 0 0 0 0
0
0 AT 0
0 0 AT-
0 1 0
0 0 1
and Wk is white noise with a known covariance matrix. The subscript k in the notation denotes the frame number. It is often omitted for a general frame. We derive the observation equation from the object's positions in two camera views. Let \_QXxk,Qu k\ and [Q2xk,Q2vjl\ be the image positions of object Q in two camera views at kth frame. The observation vector Zk=[Qu,k,Qn^Qi,^
T
=hk(Xk)
+
Vk,
(28)
where Vk is white noise with known covariance matrix, and hk is a non-linear function relating the state vector Xk to the measurement Zk .
518
The dynamic equation is linear but the measurement equation is nonlinear, so the EKF is used to estimate the state vector Xk. Expanding in a Taylor series and neglecting higher order terms, Zk = hk (Xk ) + Vk=hk (Xk]k_x) + H\ (Xk - Xk\k-\klk_x) + Vt H x
+v
kk
k
+
K(Xk\k-\)-HkXk\k-\
(29)
>
where X k\k-\ is the estimation of the state vector from X k_] and Hkis the Jacobian matrix of partial derivations of hk (Xk) with respect to Xk dQu,k dx
Hk =
dX
SQu,k dy
SQix,k dz
dQu,k dx
dQr,,k dy
dQu,k dx dQ2,,k
dz 8Qh,k dz dQ2,,k dz
dx
(30)
The EKF recursive equations are •
Update of the sate estimation Xk]k
•
Prediction of states X•k+\\k
• X k\k-\
+
Lk[Zk-hk(Xk]k_])]
(31) (32)
^kXk\k
Kalman gain matrix Lk = I.k[k_lHk (HkI.kV[_lHk + Qk)' (33)
Update of the covariance matrix of states J
*|i
~^k\k-\
^k\k-\Hk(Hk^k\k-\Hk
+Qk)
Predication of the covariance matrix of states
HkY.k\k-\ (34)
2W=*+*t (35) Initialization is provided by Z0|_, = £|X^o - X0 )(X0 - X0) J and X^_{ = X0. The EKF uses a linearization of the state equations and the observation equations about the current best estimate of the state to produce minimum meansquare estimates of the state when Wk and Vk are white noise. When a filter is actually working, so that quantities trace I.k,k and trace IMJ_, become available, one can use these as guides to \\Xk - X k\k and \\XL ', and this in •k\k-\ turn allows estimating the amount of approximation involved.
519
7.2 Tracking Initialization Tracking initialization is a process of labeling a new object in the view of a camera. If this new object has a correspondence in the other camera, both objects should be assigned the same tag number. The reason behind tracking initialization is that when we recover the 3D coordinates of an object and project the estimated (xw,yw,zw) into Q2y , the estimated Q2y should be near the observed Q2y . We initialized tracking by solving a constrained linear least squares problem, which is described as follows: JTXc,l
mm [xw,yw,z„
/r^cl
~QlyZc,l
fsXc,7
~QlxZc,2
fsJca-Q
such that
(36)
z>zc
2yZc,2
where all the symbols are defined in Equations (23-26). The sum of errors is minimum when all the objects in the view of one camera correctly find their correspondences in the other camera. Rearranging the above equations yields min \\GY ••D Y
T\ifT where G:
T\2JT
^nfs
II2
such that
z > z0
(37)
'
"
Q\x^n T2lfT Q\xT2i ~Q\yT[?, T22jT—QlyT23 'Qix^n
$ 21JS
s'22./S JS
S\2JS Y
= [xw yw
Q2x^23 Q-2y^2i
T^fT QixT3J Ti2fT-QiyT33
hlfs ~Q 2x^33 ^ 332./S 2/:
•Q?2y^33 .S
ZwY -and
D = U2ixM3 —Mi/r'6iy^43 — M2Jr> 62x^43 — S4lfs,Q2yS4J
— S42fs\ .
We add a constraint z > z0 since the moving object in the real world always has a non-zero height. This helps us to remove the shadow on the ground plane, whose height is zero. We regard the object Qx in camera I and the object Q2 in camera II as the same object Q if the sum from the above optimization problem is the minimum of all possible combinations. 8. Experiments in Multiple Camera Tracking Figs. 12 and 13 show two examples of multiple camera tracking. The dots are the observations in each camera and the lines are estimations from the EKF. x and y in the plots are horizontal and vertical coordinates. In Fig. 12, the
520
projections from the real world trajectory estimated by EKF accurately fit the observations from the two cameras. The person appears in both views consistently, therefore, the trajectory is continuous. In Fig. 13, the car is occluded in the left view so its trajectory is interrupted for a while. However, the associated trajectory in the right view is continuous during that period, which can be used to predict the interrupted positions. The overall accuracy is 94% for the 17 moving objects in the PETS 200l's videos. Three objects lost in single camera tracking were tracked properly using two cameras. Only one object was lost for a short time in the multiple camera tracking. There is some divergence between the projections and the observations in Fig. 13. One source of the divergence comes from the camera calibration and would be reduced by using more control points to calibrate the camera [36]. The other source is the time lag from the Extended Kalman Filter. Our experiments demonstrate that the overall performance of multiple camera tracking is better than that of a single camera, especially when occlusion occurs. The EKF works well with most temporary occlusions, and the tracking initialization process can deal with long term occlusion of more than 100 frames. Using multiple cameras, we can track the bicyclist in dataset II, which is occluded for a long time by the tree in camera I and thus would be difficult to track using only camera I. More accurate camera calibration would reduce the tracking error at the cost of having a more complex observation equation in the EKF. Multiple camera tracking relies on the objects existing in at least two camera views most of the time. When the target object is out of the field of one camera permanently, the tracking reduced to the single camera method. 9. Conclusions In this chapter, we have presented a system for tracking and classifying moving objects using single and multiple cameras in an outdoor environment. We combine spatial position, shape and color to achieve good performance in tracking people and vehicles. Principle component analysis is used to extract a color vector that is less sensitive to lighting changes. The variance of motion direction is robust for discriminating a single person. The variance of compactness can efficiently detect a people group. It is encouraging and useful that simple features are successful in tracking and recognition of moving objects. The PETS 2001 datasets have reported an accuracy of 76%. Among the failed tracking, 75% was caused by occlusion. People tracking is more challenging than vehicle tracking due to the smaller object size and higher possibility of occlusion.
521
The Extended Kalman Filter fuses data from multiple cameras and performs quite well despite occlusion. However, the effectiveness of EKF may be reduced when occlusion happens in both camera views. Occlusions only in one camera view are handled successfully. An overall accuracy of 94% using multiple cameras was obtained for the PETS 2001 datasets, better than using a single camera. In addition, none of the thresholds or other parameters were changed when switching from single camera tracking to multiple camera tracking. Based on our success in tracking multiple objects across multiple video streams, the reported research may be extended to recognizing moving object activities from multiple perspectives. Such a system would automatically monitor the moving objects, including humans and vehicles, classify activities, and maintain a video database, recording the time and location of motion.
Trajectory in Camera I
Trajectory* in Camera I observation from Camera estimation from EKF
250
250 200
•/.-«,
.150
>,150
100
100
50
:."
too
200 x
300
Figure 12. Trajectory of the single pedestrian
observation from Camera I estimator! from EKF
100
200
300
522
Trajectory in Camera I
Trajectory in Camera I 140
140 -
130
observation from Camera I estimation fromEKF
120
120
»110
•110
100
100
Oij
'.in
80
80
100
200 x
300
observation trom Camera I estimation from EKF
130
100
200 X
300
Figure 13. Trajectory of the car with occlusion
References [1] J. K. Aggarwal and Q. Cai. "Human motion analysis: A review ", Proc. of IEEE Workshop on Son-Rigid and Articulated Motion, pp. 90-102. Puerto Rico. 1997. [2] B. Anderson and J. Moore. Optimal Filtering. Prentice-Hall. NJ. 1979. [3] A. F. Bobick and J. W. Davis. " The recognition of human movement using temporal templates". IEEE Trans, on PAMl. vol. 23. no. 3. pp. 257-267. March 2001. [4] K. Bradshaw. I. Reid and D. Murray. "The active recovery of 3D motion trajectories and their use in prediction ". IEEE Trans, on PAML vol. 19. no. 3. pp. 219-234. March 1997. [5] Q. Cai and J. K. Aggarwal. "Tracking human motion using multiple cameras". Proc. of Intl. Conf. on Pattern Recognition, vol. 3. pp. 68-72. Vienna. Austin. 1996. [6] Q. Cai and J. K. Aggarwal. "Tracking human motion in structured environments using a distributed-camera system". IEEE Trans, on PAMl. vol. 21. no.ll. pp. 1241-1247. Nov. 1999. [7] R. T. Collins. A. J. Lipton and T. Kanade. "Introduction to the special section on video surveillance". IEEE Trans, on PAML vol. 22. no. 8. pp. 745-746. Aug. 2000. (8] S. L. Dockstsder and A. M. Tekalp. "Multiple camera tracking of interacting and occluded human motion". Proc. of the IEEE. vol. 89. no.10. pp. 1441-1455. Oct. 2001. [9] P. J. Escamilla-Ambrosio and N. Mort. "A hybrid Kalman filter-fuzzy logic architecture for multisensor data fusion". Proc. of Intl. Sym. on Intelligent Control, pp. 364-369. Mexico City. Mexico. 2001.
[10] M. A. T. Figueiredo and A. K. Jain, "Unsupervised learning of finite mixture models", IEEE Trans, on PAMI, Vol. 24, no. 3, pp. 1-15, March 2002. [11] G. L. Foresti, "A real-time system for video surveillance of unattended outdoor environments", IEEE Trans, on Circuits and System for Video Technology, Vol. 8, pp. 697-704, Oct. 1998. [12] Q. Gan and C. J. Harris, "Comparison of two measurement fusion method for Kalmanfilter-based multisensor data fusion", IEEE Trans, on Aerospace and Electronic System, vol.37, no.l, pp. 273-279, Jan. 2001. [13] D. M. Gavrila, "The visual analysis of human movement: A survey", Computer Vision and Image Understand, vol. 73, no. 1, pp. 82-98, Jan. 1999. [14] D. Gutchess, M. Trajkovic, E. Cohen-Sola, D. Lyond and A. K. Jain, "A background model initialization algorithm for video surveillance", Proc. of Intl. Conf. on Computer Vision, vol. 1, pp. 733-740, Vancouver, Canada, 2001. [15] I. Haritaoglu, D. Harwood and L. S. Davis, "W/sup 4/:real-time surveillance of people and their activities", IEEE Trans, on PAMI, vol. 22, no. 8, pp. 809-830, Aug. 2000. [16] Q. Iqbal and J. K. Aggarwal, "Retrieval by classification of image containing large manmade objects using perceptual grouping", Pattern Recognition, vol. 35, no. 7, pp. 1463-1479, July 2002. [17] R. Jain, W. N. Martin and J. K. Aggarwal, "Segmentation through the detection of changes due to motion". Computer Graph and Image Processing, vol. 11, no. 3, 13-34, 1979. [18] T. Kaneko and O. Hori, "Feature selection for reliable tracking using template matching", Proc. of Intl. Conf. on Computer Vision and Pattern Recognition, pp. 796-802, Madison, Wisconsin, 2003. [19] S. Khan and M. Shan, "Consistent labeling of tracking objects in multiple cameras with overlapping fields of view", IEEE Trans, on PAMI, vol. 25, no.10, pp. 1355-1360, Oct. 2003. [20] L. Lee, R. Romano and G. Stein, "Monitoring activities from multiple video streams: establishing a common coordinate frame", IEEE Trans, on PAMI, Vol. 22, no. 8, pp. 758767, Aug. 2000. [21] A. J. Lipton, H. Fujiyoshi, and R. S. Patil, "Moving target classification and tracking from real-time video", Proc. of IEEE Workshop on Application of Computer Vision, pp. 8-14, Princeton, NJ, 1998. [22] O. Loffeld and R. Kramer, "Phase unwrapping for SAR interferometry. A data fusion approach by Kalman filtering", Proc. of Intl. Sym. on Geosciences and Remote Sensing, vol. 3, pp. 1715-1717, Hamburg, Germany, 1999. [23] Y. Raja, S. J. Mckenna and S. Gong, "Tracking and segmenting people in varying lighting conditions using colour", Proc. of IEEE Intl. Conf. on Automatic Pace and Gesture Recognition, pp. 228-233, Nara, Japan, 1998. [24] C. Rasmussen and G. D. Hager, "Probabilistic data association method for tracking complex visual objects", IEEE Trans, on PAMI, vol.23, no.6, pp. 560-576, June 2001. [25] G. Rigoll, S. Eickeler and S. Muller, "Person tracking in real-world scenarios using statistical method", Proc. of Intl. Conf. on Automatic Face and Gesture Recognition, pp. 102-107, Grenoble, France, 2000.
524 [26] Z. Sasiadek and P. Hartana, "Sensor data fusion using Kalman filter", Proc. of Intl. Conf. on Information Fusion, vol. 2, pp. 19-25, Paris, France, 2000. [27] Z. Sasiadek , Q. Wang and M. B. Zeremba, "Fuzzy adaptive Kalman filtering for INS/GPS data fusion", Proc. of Intl. Sym. on Intelligent Control, pp. 181-186, Mexico City, Mexico, 2001. [28] K. Sato and J. K. Aggarwal, "The detection and recognition of events in video", Computer Vision and Image Understand, Special Issue on Event mining, 2004. [29] J. Shi and C. Tomasi, "Good features to track", Proc. of Intl. Conf. on Computer Vision and Pattern Recognition, pp. 593-600, Seattle, Washington, 1994. [30] C. Stauffer and W. E. L. Grimson, "Learning patterns of activity using real-time tracking", IEEE Trans, on PAMI, vol. 22, no.8, pp. 747-757, Aug. 2000. [31] E. Stringa and C. S. Regazzoni, "Real-time video-shot detection for scene surveillance applications'", IEEE Trans, on Image Processing, vol. 9, no. 1, pp. 69-79, Jan. 2000. [32] N. Strobel, S. Spors and R. Rabenstein, "Joint audio-video object localization and tracking", IEEE Signal Processing Magazine, vol. 18, no.l, pp. 22-31, Jan. 2001. [33] T. N. Tan and G. D. Sullivan and K. D. Baker, "Efficient image gradient based object localization and recognition", Proc. of Intl. Conf. on Computer Vision and Pattern Recognition, pp. 397-402, San Francisco, California, 1996. [34] T. Tommasini, A. Fusiello, E. Trucco and V. Roberto, "Making good features track better", Proc. of Intl. Conf. on Computer Vision and Pattern Recognition, pp. 178-183, Santa Barbara, California, 1998. [35] J. Triesch and C. Malsburg, "Self-organized integration of adaptive visual cues for face tracking", Proc. of Intl. Conf. on Automatic Face and Gesture Recognition, pp. 102-107, Grenoble, France, 2000. [36] R. Y. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses", IEEE Journal of Robotics and Automation, vol. RA-3, no.4, pp. 323-344, Aug. 1987. [37] S. Wenstop, "Motion detection from image information", Proc. of Scandinavian Conf. on Image Analysis, pp.381-386, 1983. [38] K. M. Wong and E. Bosse, "Data compression, data fusion and Kalman filtering in wavelet packet sub-bands of multisensor tracking system", Proc. of Radar. Sonar and Navigation, vol. 145, pp. 100-108, 1998. [39] C. R. Wren, A. Azarbayejani, T. Darrel and A. P. Pentland. "Pfinder: real-time tracking of the human body", IEEE Trans, on PAMI, vol. 19, no. 7, pp. 780-785, July 1997. [40] Y. Wu and T. S. Huang, "A co-inferene approach to robust visual tracking", Proc. of Intl. Conf. on Computer Vision, pp. 26-33, Vancouver, Canada, 2001. [41] Q. Zhou and J. K. Aggarwal, "Tracking and classifying moving objects from video", Intl. Workshop on Performance Evaluation of Tracking and Sun'eillance, Kauai, Hawaii, 2001. [42] Q. Zhou, J. Park and J. K. Aggarwal, "Quaternion-Based Tracking of Multiple Objects in synchronized video'', Proc. of Intl. Sym. on Computer and Information Sciences, pp. 430438, Antalys, Turkey, 2003.
CHAPTER
5.2
P E R F O R M A N C E EVALUATION OF IMAGE ALGORITHMS
SEGMENTATION
Xiaoyi Jiang Department of Computer Science, University of Miinster Einsteinstrasse 62, D-48I4S Miinster, Germany Email: xjiang @math. uni-muenster. de
Performance evaluation of computer vision algorithms has attracted much attention in recent years. In this chapter we consider performance evaluation of image segmentation algorithms. Three classes of segmentation tasks are investigated: edge detection, region segmentation, and detection of curvilinear structures. We present a taxonomy of performance evaluation approaches, discuss various performance evaluation methods, and explore several other related issues. Considering the advances achieved so far, it can be expected that continuing efforts in future will make this important research area a mature one. In particular the proposed techniques will find wider acceptance in our community to go along with a sound development of computer vision research.
1. I n t r o d u c t i o n "When you can measure what you are speaking about and express it in numbers, you know something about it; b u t when you cannot measure it, when you cannot express it in numbers, your knowledge is of the meager and unsatisfactory kind." - Lord Kelvin Traditionally, computer vision suffers from a lack of sound experimental work. T y p ically, researchers show their results on a few images and point out why t h e results look good and better t h a n those achieved by other methods. From such studies t h e readers never know whether the results are best examples or representative examples, whether the algorithm consistently outperforms others, and so on. As pointed out by R. J a i n and T . Binford 1 3 : "The importance of theory cannot be overemphasized. But a t the same time, a discipline without experimentation is not scientific. W i t h o u t adequate experimental methods, there is no way to rigorously substantiate new ideas and t o evaluate different approaches." T h e dialogue on "Ignorance, Myopia, and Naivite in Computer Vision Systems" initiated by R. Jain and T . Binford 1 3 and the responses documented the necessity of evaluating theoretical findings, vision algorithms, etc. by using empirical data. Today sound empirical comparison of vision algorithms is increasingly being accepted as original research,
525
526
Performance Evaluation
Experimental
Theoretical
Feature-Based
Non-GT Based Fig. 1.
Task-Based
GT-Based
Taxonomy of performance evaluation of segmentation algorithms
including research aspects such as methods, tools, and databases for performance evaluation, design of experiments, and actual comparison of different algorithms for vision problem, etc. The readers are referred to refs. 2 ' 6 ' 7 ' 23 ' 28 for collections and to ref.3 for an overview of recent works on empirical evaluation techniques in computer vision. This chapter is concerned with performance evaluation of image segmentation algorithms. Concretely, we consider three segmentation problems: edge detection, region segmentation, and detection of curvilinear structures. It is not our intention to provide a comprehensive survey. Instead, we will present a taxonomy of performance evaluation approaches (Section 2), discuss some representative methods for each category (Sections 3-5), and finally address several important issues which have only gained attention in recent years (Section 6-8). An earlier survey of performance evaluation for image segmentation can be found in ref.33. The present chapter emphasizes on a broader coverage of topics and recent developments. The research literature covered in this chapter is taken from the domain of intensity and range images. However, all the approaches are general enough to be applied to other imagery modalities. 2. T a x o n o m y A variety of methods have been proposed for assessing the performance of segmentation algorithms. These methods can be categorized as shown in Figure 1. A theoretical evaluation is done by applying a mathematical analysis without the algorithms ever being implemented and applied to an image. Instead, the algorithm behavior is mathematically characterized and the performance is determined analytically or by simulation. The major limitations of theoretical approaches are the simplistic mathematical models and the difficulty in applying them to many of the more modern segmentation algorithms because of their complexity.
527 An experimental evaluation can be divided into feature-based and task-based. The former category measures the algorithm performance only based on the quality of detected features under consideration, e.g. edges and regions. Within this category we can further distinguish between non-GT based and GT-based approaches. The term ground truth (GT) is used to denote some reference result that represents the expected ideal segmentation. The basic idea of GT-based approaches is then to measure the difference between the machine segmentation result and the ground truth. In contrast non-GT based methods do not assume the availability of GT and compute performance measures directly by means of some desirable properties of the segmentation result. Task-based evaluation follows a very different philosophy. Image segmentation represents only one, although important, step in achieving the high-level goal of a vision system, e.g. object recognition. Of ultimate interest is the overall performance of the system. Instead of abstractly comparing the performance of segmentation algorithms it is thus more meaningful to conduct an indirect comparison based on their influences on the final performance of the entire system. 3. Edge detection Among the three segmentation tasks considered in this chapter, edge detection is doubtless the one that has attracted the most attention in the context of performance evaluation. An example of theoretical evaluation can be found in ref.14. The rationale there is that an optimal edge detector has a convolution kernel that yields the smallest variance on its output for the same input perturbation. Given a vector W containing the elements of a convolution kernel, it is shown that if an image contaminated by an independently and identically Gaussian distributed noise with zero mean and standard deviation
528
values. To overcome this sort of problems, Bowyer et al.4 proposed a one-to-one matching between the machine segmentation and the ground truth. When an edge pixel has multiple potential matches, they choose the closest-distance one. This locally optimal assignment can be further improved by casting the task into a global optimization problem 24 ' 30 . This approach computes the discrepancy measure in a globally optimal manner, however, at a price of substantially higher computational burden. It should be pointed out that there is not always convincing evidence of real improvement compared to the suboptimal edge matching methods. This is particularly true since due to the computational complexity, the global optimization problem is typically solved approximatively 30 . Recently, task-based evaluation becomes popular. In ref.16 edge detectors for range images are not compared alone. Instead, they are put into the context of region segmentation and evaluated in a combination with a subsequent grouper to form a complete region map based on the edge detection result. Task-based evaluation in the domain of intensity images is exemplified in refs. 9,31 . Heath et al.a measures the edge detection performance by visual rating scores which indicate the perceived quality of the edges for recognizing an object. In the work of Shin et al.31 edge detection goodness is judged by how accurately structure from motion algorithms can recover the known structure and motion from the edge detection of image sequences. Interestingly, they noticed that ratings of edge detector performance based on pixel-level measures and the structure from motion task are well correlated.
4. Region segmentation Region segmentation represents another major form of image segmentation; in the domain of range images this is actually the dominant segmentation task. Compared to evaluating edge detectors relatively few performance measures have been proposed for region segmentation. In ref.10 a machine segmentation (MS) of an image is compared to the ground truth specification to count instances of correct segmentation, under-segmentation, over-segmentation, missed regions, and noise regions. These measures are denned based on the degree of mutual overlap required between a region in MS and a region in GT. A correctly segmented region is recorded if and only if an MS region and the corresponding GT region have a mutual overlap greater than a threshold. Multiple MS regions that together correspond to one GT region constitute an instance of over-segmentation while one MS region corresponding to the union of several GT regions is considered as under-segmentation. A MS (GT) region that has no corresponding in GT (MS) constitutes an instance of noise (missing) region. In contrast to the evaluation strategy above, the approach taken in ref.12 delivers one single performance measure. Considering two different segmentations Sx — {R{, R\,...,R^} and 52 = {Rl, Rh • • • > ^2 } ° f t n e same image, we associate each region R\ from £2 with a region R\ from S\ such that R\ n R{ is maximal.
529 The directional Hamming distance from Si to S2 is defined as:
DH(Si^S2) = £
E
1**^1
corresponding to the total area under the intersections between all R2 € S2 and their non-maximally intersected regions E* from Si- The reversed distance DH(S2 =>• Si) can be similarly computed. Finally, the overall performance measure is given by: DH(S1=>S2)+DH(S2=>S1)
1
P = 1
U
where A is the image size and p e [0,1]. Letting MS and GT play the role of Si and 5 2 , respectively, allows us to measure their discrepancy. In ref.25 another single overall performance measure is proposed. It is designed so that if one region segmentation is a refinement of another (at different granularities), then the measure should be small or even zero. Let R(S,Pi) be the set of pixels corresponding to the region in segmentation S that contains the pixel pi. Then, the local refinement error associated with pi is: _ -
P(q q n l E(5l 52 Pi)
'
'
\R(Si,Pi)\R(S2,Pi)\ \RjS^i)\
where \ denotes set difference. Finally, the overall performance measure is defined as: Pi
: -min{
Y, E(Si,S2,Pi), all pixels pi
£ E(S2,Si,Pi) all pixels p{
}
or P2
= J
5Z all pixels
mm{E(Si,S2,Pi),
E(S2,Si,Pi)}
Pi
Note that both measures are tolerant of refinement and they are meaningful only when comparing two segmentations with an approximately equal number of regions. This is generally the case when comparing a machine segmentation with the ground truth. 5. Detection of curvilinear structures The term curvilinear structure denotes a line or a curve with some width. Curvilinear structures are useful features in a variety of applications, e.g. in finding roads or rivers in aerial images, blood vessels or bones in medical images, and characters in text images. Compared to other commonly used features such as edges, there are few methods for measuring the performance of curvilinear structure detection algorithms. In the following our discussion will be exemplified by the task of detecting blood vessels in fundus images. It is important to point out that these approaches are applicable in the general context as well.
530
Fig. 2.
Fig. 3.
M S i i : partial thinning of GT. MS12: deletions in G T
MS21: partial expanding of GT. MS22: insertions in GT
In refs. 11,21 the performance of blood vessel detection is measured as follows. Given a machine-segmented result image (MS) and its corresponding hand-labeled ground truth image (GT), any pixel which is marked as vessel in both MS and GT is counted as a true positive. Any pixel which is marked as vessel in MS but not in GT is counted as a false positive. The true positive rate (TPR) is established by dividing the number of true positives by the total number of vessel pixels in GT. The false positive rate (FPR) is computed by dividing the number of false positives by the total number of non-vessel pixels in GT. This approach is more suitable for comparing large regions and its application to curvilinear structures as elongated and thin regions is questionable. This is illustrated in Figure 2 that shows two modified versions of a GT vessel image. For both MSn and MS 12 we obtain TPR=0.851 and FPR=0.000, indicating an equal rate of 85.1% correct detection and no spurious vessels. But in reality there are substantial differences between the two MS images. The image MSn is generated by thinning GT at some places. In this case the entire vessel network is correctly detected, but some vessels have a smaller width than GT. In contrast MS12 results from deleting parts in GT and therefore perfectly equals GT except the deleted parts. A more objective performance measure would be TPR(MSu)=1.00 and TPR(MSi 2 )<1.00, indicating the percentage of the correctly detected part of the vessel network. The correctly detected parts of the vessel network can be further evaluated with respect to the detection accuracy, i.e. the width error. Then, we would expect a non-zero
531 (zero) width error for MSn (MSi 2 )- A second situation in Figure 3 illustrates a related problem. Again, the two MS images MS21 and MS22 have equal performance measures TPR=1.00 and FPR=0.012, implying a full detection of the vessel network and 1.2% spurious vessels in both cases. Here MS21 emerges from GT by locally expanding GT while MS22 equals GT plus eight spurious (diagonal) vessel parts. Different from MS22, the spurious vessel pixels in MS21 cause vessel width errors, but do not change the vessel network structure in any way. Intuitively, a measure FPR(MS 2 i)=0 and FPR(MS22)>0 thus makes more sense. Accordingly, MS21 (MS22) is associated with non-zero (zero) width error. These two examples clearly show the drawbacks of the evaluation method used in refs. 11,21 . Due to the nature of curvilinear structures being elongated and thin regions, a pixel-wise comparison is not feasible. A mixture of two different aspects of performance, namely detection rate and detection accuracy, in the overall performance measures TPR and FPR result in a biased impression of detection quality. In a recent work20 we have proposed a GT-based approach to considering these two aspects separately. Given a blood vessel image V, we perform a thinning operation to obtain a thinned vessel image Vt of one-pixel width. Obviously, the vessel network structure is described by V t . A supplement of width information for each vessel pixel in Vt provides then a full description of the vessel network. Given a MS and a GT, we propose to measure the detection rate by comparing MSt and GTt only, i.e. how much of the GT vessel network structure is detected in MS. In a second step the width of matched MS* and GTf vessel pixels are compared to give a measure of detection accuracy. A pseudo-code description of the evaluation procedure is given in Figure 4. To measure the detection rate we have to determine how many of the vessel pixels in GTt are detected in MSt- Due to the uncertainty in both manual groundtruthing and vessel detection a pixel-wise comparison is doubtless of no use at all. Traditionally, point matching is based on some threshold, i.e. two points less than the threshold apart from each other are considered as matched. This is particularly true in the evaluation of edge detection algorithms. In our case the characteristic of curvilinear structures gives us a natural limiting threshold without any manual specification. We consider a vessel pixel p in GT t as detected (true positive, TP) if p is also a vessel pixel in MS. In practice we only require that the 3 x 3 window around p contains at least one vessel pixel in MS in order to tolerate positional bias in the case of very thin, say one-pixel wide, curvilinear structures. Then, the true positive rate (TPR) results from dividing the number of true positives by the total number of vessel pixels in GT t . Undetected vessel pixels in GT t are counted as false negatives (FN, missing vessels). A division by the total number of vessel pixels in GT t supplies the false negative rate (FNR). In a similar manner false positives (FP, spurious vessels) are those vessel pixels q in MSt for which there is no vessel pixel in GT in the 3 x 3 window around q. In addition to computing FN and FP it is also interesting to ask about the characteristic, for instance the width, of the missing
532
suml = sum2 = 0; /* Detection rate */ for each vessel pixel p in GTt { if there is at least one vessel pixel in W3X3(p) in MS { T P + + ; mark p as detected; } else { F N + + ; FNHist[width(p)]++; } } TPR = TP / (number of vessel pixels in GT t ); FNR = FN / (number of vessel pixels in GT t ); for each vessel pixel q in MSt { if there is no vessel pixel in W3x3(q) in GT { F P + + ; FPHist[width(g)]++; } } /* Detection accuracy */ for each detected vessel pixel p in GTt { find q in MSt nearest to p; suml + = distance(p,q); sum2 + = |width(p)-width(g)|; } AvgPosAccuracy = suml / #(detected vessels in GTt); AvgWidthAccuracy — sum2 / #(detected vessels in GTt); Fig. 4.
Evaluation procedure for curvilinear structures
and spurious vessels. It is probably more problematic to miss or erroneously detect thick vessels than thin ones. To obtain this information we can establish a width histogram of the false negatives and false positives. Here the width is defined by the Euclidean distance to the nearest pixel in the background. In the second part of the evaluation we intend to measure the detection accuracy. The accuracy consists of that of position and width. Here only those vessel pixels in GTt marked as detected are involved. For each of such points p its matching in MSt is defined to be the vessel pixel q in MSt with the smallest Euclidean distance to p. Then, the position accuracy is simply the Euclidean distance of p and q. Similarly, the width accuracy is given by the difference of the width of p and q. In both cases we finally compute the average accuracy through a division by the total number of the detected vessel pixels in GTtUsing this approach an evaluation of the four images in Figures 2 and 3 lead to the results in Table 1. As wanted, MSn has TPR near 100%, implying a full detection of the vessel network structure. The fact that the detected vessels are thinner than GT is expressed by the width error 0.26 (pixel). In contrast MS12
533 Table 1.
Evaluation results for M S n , MS12, MS21, and MS22 MSu
TPR FNP FNHist FP position accuracy width accuracy
TPR FNP FP FPHist position accuracy width accuracy
99.9% 0.1% (4 pixels) 1:50%, 2:25%, 3:25% 0 0.00 0.26
MS21 100.0% 0.0% 0 0.05 0.28
MS12 77.7% 22.3% (1726 pixels) 1:62%, 2:34%, 3:4% 0 0.01 0.00
MS22 100.0% 0.0% 408 1:8%, 2:8%, 3:8%, 4:21%, 5:20%, 6:29%, 7:6% 0.00 0.00
leads to TPR=77.7% only and accordingly 22.3% of the vessel network structure undetected. Based on the histogram of false negatives we see further that the missing vessels are relatively thin; 96% of the missing parts have a width up to 2. Here the width error is zero, indicating a perfect detection of 77.7% of the vessel network. The interpretation of these evaluation measures is exactly what we postulated for more informative and precise performance evaluation compared to the evaluation method in refs. 11 ' 21 . Similar improvement can also be observed for MS21 and MS22No false positive is found in MS21, compared to 408 in MS22- Furthermore, the spurious vessels in MS22 are mainly thick ones with 76% being at least 4 pixels wide. Although both MS21 and MS 22 have 100% TPR, MS21 is not a perfect detection. The width error 0.28 indicates some detection inaccuracy. In addition to such synthetic situations the work 20 further shows the potential of this evaluation approach on real image segmentation results. 6. Imagery design An essential issue of performance evaluation is the imagery design including acquisition of image data and training/test set partitioning. One concern in imagery design is the possible "dimensions" of the task under consideration, i.e., parameters that (in the ideal case fully) cover the variations of the scenes to be imaged. For range image segmentation, for instance, we are faced with a large number of factors to be considered: • type of range scanners, the associated number of bits per pixel (quantization level) and type and amount of noise (besides quantization); • number of surfaces in the image, size of surfaces (in pixels); • jump edges: amount of depth discontinuity between two adjacent surfaces, edge length (in pixels); • crease edges: angle between two adjacent surfaces, incident angle of edge to
534
viewpoint, edge length (in pixels). In the case of medical image segmentation both healthy and pathological images should be included. Theoretically, testing algorithms on an image set that spans the ranges of all these dimensions would yield "failure points" and "tolerances". In practice, however, acquiring, ground-truthing, processing, and analyzing the necessary image data would require a prohibitive amount of effort. As a consequence, an image set of manageable size is usually acquired that reasonably explores the problem dimensions. A second aspect of imagery design is training/test set partitioning. A given image set is divided into two disjoint subsets for training and test purpose, respectively. The training set is used to fix the parameters of algorithms, while performance measures are computed on images from the test set using the fixed parameters. The training/test set partitioning has been hardly investigated so far and is typically done randomly 10 ' 29 . In the following we describe a systematic, optimization-based approach for this purpose reported in ref.19 The characteristics of an image set II can be expressed in terms of image features. For the range image segmentation task, for instance, some examples of image features are: size, contour length, compactness, etc. of regions; angles between adjacent regions. Each image feature /; is associated with a distribution (probability) of its values, P;, over II. Any partition of II into two disjoint subsets IIi and II2, IIi U n 2 = II, implies distributions Pn and P; 2 over IIi and n 2 , respectively. Assume that IIi (n 2 ) serves as the training (test) image set. Then we are facing the question what properties the distributions Pn and Pa should have in order to make the corresponding partition into training/test set most meaningful. For simplicity the cardinality of IIi is assumed to be specified by the user. The usual practice of using the training images to fix the parameters of algorithms implicitly assumes that the test images have characteristics similar to those of the training images. Expressed in terms of image features this means that the training set II1 and the test set II 2 should have similar distributions Pn and P; 2 for all features fi. But in case of a random partitioning of II into IIi and II 2 , as done in refs. 10 ' 29 for instance, there is a danger of large differences between distributions Pji and Pj 2 of these features, which is clearly in contradiction with the assumption usually made in empirical performance evaluation studies. Assume a function di(Pn, P; 2 ) that defines the distance, or dissimilarity, between two distributions Pn and P; 2 . Then, the discussion above suggests a strategy of finding favorable partitions by minimizing the term: n
^2wi-di(Pa,Pa)
(1)
Here n is the number of image features under consideration and Wi represents a weighting factor that controls the relative influence of the various features.
535 Table 2. Characteristics of three range image sets sensor type #images resolution surfaces imaging volume
ABW
Perceptron
K2T
structured light 40 512 x 512 planar table-top size
T O F laser 40 512 x 512 planar room-size
structured light 60 480 x 640 planar/curved table-top size
This understanding of favorable partitions gives us a systematic method for determining training/test image sets. Interestingly, it opens the door to other evaluation scenarios that may be of interest as well. Let us consider a partition that maximizes the term (1). This would result in an extreme test case in which the test image set differs to the largest extent (with respect to the measure (1)) from the training image set. If we set all weighting factors W{ to be zero except one Wk, then we are potentially able to test the sensitivity of the algorithms concerned with the particular feature fk- This kind of "worst-case" test scenarios and the usual "bestcase" test scenarios together provide a rich set of different scenarios for evaluating algorithms from various view points. Considering both cases we formulate the strategy of finding favorable partitions formally as: Partition problem V: For a given image set II, a number 1 < c < \U\, distance functions di(), and weighting factors Wj, find a partition of U into two disjoint subsets IIi and H2, |XIx | = c, IIi UII2 = II, such that n
i=l
is optimized (minimized or maximized). Unfortunately, this partition problem turns out to belong to the class of MVhard discrete optimization problems. Thus, we are forced to develop approximative methods for finding suboptimal solutions in reasonable time. In ref.19 a genetic search algorithm has been designed for this purpose. The range image segmentation comparison project 10 ' 29 has been used as a testbed for this training/test set design approach. Under consideration of the large number of factors in the test imagery design, three image sets were acquired using different sensors. Table 2 summarizes their characteristics. The ground truth segmentation was generated by human operators. Figure 5 gives an example from the ABW image set, showing the intensity image, the range image, and the ground truth segmentation (from left to right). In refs. 10 ' 29 the ABW/Perceptron/K2T image set was divided into a training set of 10/10/20 images and a test set of 20/20/40 images. The partition was done manually, basically by human inspection of the images. For testing our training/test set design approach we define the following features:
536
Fig. 5.
An example image and ground t r u t h segmentation from ABW set
® region type: planar, cone, cylinder, sphere, torus ® region size: number of pixels ® region perimeter ® region compactness: region shape defined by perimeter 2 /size ® angle between two adjacent regions ® length of contours between two adjacent regions For the ABW image shown in Figure 5, for instance, the segmentation ground truth contributes the following angle values to the overall angle distribution: 346°(1), 54.0°(1), 58.4°(1), 61.3°(1), 66.6°(1), 68.2°(2), 75.1°(2), 84.5°(1), 90.0°(10), where the number in brackets indicates the number of occurrences of the angle. If we only consider the angle feature, i.e. set all weighting factors Wi except that for angle to zero, then the manual partition used in ref.10 leads to M = 0.016472 by using a particular distance function. This is illustrated in Figure 6 which shows the angle distribution of both training and test set.
1 Leaend Training distribution interpolated Training distribution points Test distribution interpolated X Test distribution points
0.7
1
i
1
/
0.8
/-
0.5
-
0.4
/ "
0.3
"
0.2
•
/
"
0.1
-
/
-
/ "
*•**-assaJLi":::.:
Fig. 6.
T ^ — - * * •"*"•*=
i
^ * ^ -"*
;--
J
Angle distribution of training/test set: ABW, manual partition
537 Table 3.
Optimization term M: GA vs. manual partition Best partition
Worst partition
Manual partition
0.001448 0.002409 0.004296
0.116354 0.199991 0.630148
0.016472 0.033738 0.027788
ABW Perceptron K2T
Leqend best partition manual partition worst Dartitbn
K2T angles
-i
•
PERC ang as
—r—|
11
rn !
-i
database/feature
Fig. 7.
Optimization term Ai: GA vs. manual partition
Using the angle feature alone, the genetic algorithm is used to generate a best and a worst partition by minimizing resp. maximizing the optimization term M.. The computed partitions result in the M. values recorded in Table 3, or graphically in Figure 7. For the ABW image set, the angle distributions of both partitions are graphically illustrated in Figure 8, see Figure 6 for a comparison with the manual partition. For comparison purpose the M. values of the manual partition is given in Table 3 as well. In all cases the best partition found by the genetic algorithm is better than the corresponding manual partition. The worst partition provides us, on the one hand, an extreme test situation, in which the training and test images have quite different characteristics with respect to the angle feature. On the other hand, it represents an upper limit of the badness of manual partitions which a human operator may produce in his efforts to create a good partition. Training/test data partitioning is a research issue which has not gained much attention in the literature. The work in ref.19 represents only a step forwards and much work is still needed to reach some degree of maturity. 7. Ground truth Ground-truthing is typically a time-consuming manual task 10 ' 25 . In certain cases the quality of the manually specified ground truth is an issue itself. For range images the ground-truthing is geometrically guided and as such may be well supported by a graphic visualization tool 10 . Other tasks like vessel pixel masking is more delicate. It
538
0.7
"
Leaend Training distribution interpolated Training distribution points Test distribution interpolated Test distribution points
1
1
+ *
/-
0.6
0.5
1" i 0.3
"
/-
-
/ /
0.2
-
/
0.1
*"V-^ - l J ? . " '
X
J
^l*•rff^t^*
l!L 1 *=a«m - " ^
Leaend Training distribution interpolated Training distribution points Test distribution interpolated TBSI distribution points
0.7
-
-
*
'
1
—
"
*
-
—
•
1
* + *
h
/-
0.6
0.5
f
0.4
/-
Pro
H
0.3
-
-
0.2
0.1
-
**^_ - - • - ' i »-%^-
_ » *>~f — • * •
i | »•-..».>,
1
Angles
Fig. 8.
Angle distribution of training/test set: ABW, GA. Best (top) and worst partition (bottom)
is not unusual that even domain experts have non-uniform opinions. Along this line we will obtain different GT results when asking multiple people to do the groundtruthing. There are several ways of handling this situation. In ref.11, for instance, two hand-labelings of vessels in twenty retinal images were made by two different persons. The first hand-labeling, which is used as ground truth in the evaluation, took a more conservative view of the vessel boundaries and in the identification of small vessels than the second hand-labeling. By considering the second handlabeling as "machine-segmented result images" and comparing them to the ground truth, we obtain a detection performance measure which is then regarded as a target performance level. The same strategy was adopted in ref.21 as well. Another solution is to compute some kind of mean (average) of all the groundtruthing results 5 . Of course, this approach poses the interesting, however proba-
539 Table 4. http data http data http data http data
Sources of image data with known ground truth
figment.csee.usf.edu/edge/roc/ 60 real intensity images with G T edge map www.cs.berkeley.edu/projects/vision/grouping/segbench/ a huge number of intensity and color images with G T region map marathon.csee.usf.edu/range/seg-comp/SegComp.html 140 real + many synthetic range images with G T region map www.parl.clemson.edu/stare/probing/ 20 retinal images with G T vessel network
bly non-trivial, problem of mean computation in a problem-specific, typically nonvector, space, which may be a challenge itself. Automatic determination of ground truth is a difficult task and has not gained much attention yet. One attempt of this kind is described in ref.32. To achieve a ground truth edge map, the edge detector is run multiple times using different parameter settings. Then, for each pixel it is counted for how many times among the trials it is considered as an edge pixel. This count is finally thresholded to decide whether the pixel should be an edge pixel in the estimated ground truth. Despite of the high costs of ground-truthing GT-based performance evaluation remains an attractive choice. Basically, it is needed to be done only once for one image database and the GT can then be shared in the research community. In the past years several image sets along with hand-specified ground truth were put for public use. Table 4 summarizes some of these efforts.
8. Parameter selection Usually, a segmentation algorithm has a number of parameters. In order to evaluate its performance these parameters must be specified. In the range image segmentation comparison project 10 ' 17 ' 29 this was done manually in an ad-hoc manner. Manual training of algorithm parameters will produce results that are dependent on the knowledge, skill, and efforts of the experimenter. If two experimenters of the same algorithm spend equal time, their tuning results are likely to differ. For this reason an automated training procedure is preferable to manual training. A reasonable way of doing this is to find out the parameter setting that results in the best performance on a training set according to some performance measure. Fully exploring the parameter space is not a trivial task. In ref.26 an adaptive search procedure is proposed. Assume that the number of parameters be N. The allowed range of each parameter is specified and sampled by D evenly-spaced points. Then, the entire parameter space is first coarsely explored by the DN parameter settings. The segmentation results are evaluated against the ground truth and the highest performing one percent of the DN initial parameter settings are selected for refinement. The refinement in the next iteration creates a 3 x 3 x • • • x 3 sampling around each of the parameter settings carried forward. In this way the resolution of parameter space exploration becomes finer with each iteration, even as the total
540
number of parameter settings considered is reduced. The expanded set of points is then evaluated and the top-performing points are again selected to be carried forward to the next iteration. This process is continued until some termination criteria are fulfilled. The final top-performing point is selected as the trained parameter setting. Although this approach does not guarantee to find the global optimum, a comparison conducted with the exhaustive search found no statistically significant differences in the performance of the selected parameter settings. Basically, any optimization technique may be applied to search the parameter space. As an alternative, for instance, genetic search has been chosen in ref.8 for this purpose. 9. Conclusions Performance evaluation in general and for image segmentation algorithms in particular is an important task in computer vision. Today sound empirical comparison of vision algorithms is increasingly being accepted as original research, including research aspects such as methods, tools, and databases for performance evaluation, design of experiments, and actual comparison of different algorithms for vision problem, etc. In this chapter we have described various aspects of comparing image segmentation algorithms. The research activities so far have resulted in reasonable approaches for conducting experimental evaluation. A series of comparison studies have delivered interesting insights into the relative performance of segmentation algorithms 9 ' 10 ' 17 ' 26 ' 29 . It can be expected that continuing efforts in future will make this important research area a mature one. In particular the proposed techniques will find wider acceptance in our community to go along with a sound development of computer vision research. Acknowledgment. The author wants to thank K.W. Bowyer, H. Bunke, D. Goldgof, A. Hoover, and D. Mojon for many stimulating discussions on various aspects of performance evaluation. References 1. I.E. Abdou and W.K. Pratt. Quantitative design and evaluation of enhancement/thresholding edge detectors. Proceedings of IEEE, 67(5): 753-763, 1979. 2. K.W. Bowyer, and P.J. Phillips (eds). Empirical Evaluation of Techniques in Computer Vision. IEEE Computer Society Press, 1998. 3. K.W. Bowyer, and P.J. Phillips. Overview of work in empirical evaluation of computer vision algorithms. In ref. , 1-11, 1998. 4. K.W. Bowyer, C. Kranenburg, and S. Dougherty. Edge detector evaluation using empirical ROC curves, Computer Vision and Image Understanding, 84(1): 77-103, 2001. 5. V. Chalana and Y. Kim. A methodology for evaluation of boundary detection algorithms on medical images. IEEE Trans, on Medical Imaging, 16(5): 642-652, 1997. 6. H. Christensen, and W. Porstner (eds). Special issue on performance evaluation. Machine Vision and Applications, 9(5/6): 215-340, 1997. 7. H.I. Christensen and P.J. Phillips (eds). Empirical Evaluation Methods in Computer Vision. World Scientific, 2002.
541 8. L. Cingue, R. Cucciara, S. Levialdi, S. Martinez, and G. Pignalberi. Optimal range segmentation parameters through genetic algorithms. Proc. of 15th Int. Conf. on Pattern Recognition, Vol. 1, 474-477, Barcelona, 2000. 9. M.D. Heath, S. Sarkar, T. Sanocki, and K.W.Bowyer. A robust visual method for assessing the relative performance of edge-detection algorithms. IEEE Trans, on PAMI, 19(12): 1338-1359, 1997. 10. A. Hoover, G. Jean-Baptiste, X. Jiang, P.J. Flynn, H. Bunke, D. Goldgof, K. Bowyer, D. Eggert, A. Fitzgibbon, and R. Fisher, An experimental comparison of range image segmentation algorithms, IEEE Transactions on PAMI, 18(7): 673-689, 1996. 11. A. Hoover, V. Kouznetsova, and M. Goldbaum. Locating blood vessels in retinal images by piece-wise threshold probing of a matched filter response. IEEE Trans, on Medical Imaging, 19(3): 203-210, 2000. 12. Q. Huang and B. Dom. Quantitative methods of evaluating image segmentation. Proc. of Int. Conf. on Image Processing, 53-56, 1995. 13. R. Jain and T. Binford. Ignorance, myopia, and naivite in computer vision systems. CVGIP: Image Understanding, 53(1): 112-117, 1991. 14. Q. Ji and R.M. Haralick. Efficient facet edge detection and quantitative performance evaluation. Pattern Recognition, 35: 689-700, 2002. 15. X. Jiang, A. Hoover, G. Jean-Baptiste, D. Goldgof, K. Bowyer, and H. Bunke. A methodology for evaluating edge detection techniques for range images. Proc. of 2nd Asian Conf. on Computer Vision, Vol.11, 415-419, Singapore, 1995. 16. X. Jiang and H. Bunke. Performance assessment of edge-based range image segmentation. Proc. of Int. Conf. on Advances in Pattern Recognition, 83-92, Plymouth, 1998. 17. X. Jiang, K. Bowyer, Y. Morioka, S. Hiura, K. Sato, S. Inokuchi, M. Bock, C. Guerra, R.E. Loke, and J.M.H. du Buf. Some further results of experimental comparison of range image segmentation algorithms. Proc. of 15th Int. Conf. on Pattern Recognition, Vol. 4, 877-881, Barcelona, 2000. 18. X. Jiang. An adaptive contour closure algorithm and its experimental evaluation. IEEE Trans, on PAMI, 22(11): 1252-1265, 2000. 19. X. Jiang, C. Irniger, and H. Bunke. Training/test data partitioning for empirical performance evaluation. In ref. , 23-37, 2002. 20. X. Jiang and D. Mojon. Supervised evaluation methodology for curvilinear structure detection algorithms. Proc. of 16th Int. Conf. on Pattern Recognition, Vol. 1,103-106, Quebec City, 2002. 21. X. Jiang and D. Mojon. Adaptive local thresholding by verification-based multithreshold probing with application to vessel detection in retinal images. IEEE Trans, on PAMI, 25(1): 131-137, 2003. 22. L. Kitchen and A. Rosenfeld. Edge evaluation using local edge coherence. IEEE. Trans, on SMC, 11(9): 597-605, 1981. 23. R. Klette, H.S. Stiehl, M. Viergever, and K. Vincken (eds). Performance Characterization and Evaluation of Computer Vision Algorithms. Kluwer, Amsterdam, 2000. 24. G. Liu and R.M. Haralick. Optimal matching problem in detection and recognition performance evaluation. Pattern Recognition, 35: 2125-2139, 2002. 25. D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its applications to evaluating segmentation algorithms and measuring ecological statistics. Proc. of Int. Conf. on Computer Vision, Vol. 2, 416-423, 2001. 26. J. Min, M. Powell, and K.W. Bowyer. Automated performance evaluation of range image segmentation algorithms. IEEE Trans, on SMC - Part B, 34(1): 263-271, 2004. 27. T.B. Nguyen and D. Ziou. Contextual and non-contextual performance evaluation of
542 edge detectors. Pattern Recognition Letters, 21(9): 805-816, 2000. 28. P.J. Phillips and K.W. Bowyer (eds). Special section on empirical evaluation of computer vision algorithms. IEEE Transactions on PAMI, 21(4): 289-335, 1999. 29. M.W. Powell, K.W. Bowyer, X. Jiang, and H. Bunke. Comparing curved-surface range image segmenters. Proc. of Int. Conf. on Computer Vision, Bombay, 286-291, 1998. 30. M.S. Prieto and A.R. Allen. A similarity metric for edge images. IEEE Trans, on PAMI, 25(10): 1265-1273, 2003. 31. M.C. Shin, D.B. Goldgof, K.W. Bowyer, and S. Nikiforou. Comparison of edge detection algorithms using a structure from motion task. IEEE Trans, on SMC - Part B, 31(4): 589-601, 2001. 32. Y. Yitzhaky and E. Peli. A method for objective edge detection evaluation and detector parameter selection. IEEE Trans, on PAMI, 25(8): 1027-1033, 2003. 33. Y.J. Zhang. A survey on evaluation methods for image segmentation. Pattern Recognition, 29(8): 1335-1346, 1996. 34. Q. Zhu. Efficient evaluations of edge connectivity and width uniformity. Image and Vision Computing, 14: 21-34, 1996.
C H A P T E R 5.3 CONTENT-BASED VIDEO ANALYSIS FOR KNOWLEDGE DISCOVERY
Chia-Hung Yeh, Shih-Hung Lee and C.-C. Jay Kuo Integrated Media Systems Center Department of Electrical Engineering University of Southern California, Los Angeles, CA 90089 E-mail: {chyeh, cckuo} @sipi.use. edu With the rapid growth of multimedia information, video content analysis becomes crucial when dealing with large amounts of multimedia data, including texts, images, audio, video, and graphics. Video content analysis can be used to facilitate the access to desired content for information retrieval, summarization, and knowledge discovery. The study and development of suitable and effective techniques for video content management and access has been conducted extensively over the last decade. In this chapter, we will first review the characteristics of visual and audio contents and their basic processing techniques. Then, we will discuss semantic approaches to video data understanding and provide an overview of major developments in this field. In particular, we emphasize video knowledge discovery using features of multiple modalities, enhancing collaboration among semantic cues from various kinds of information sources.
1. I n t r o d u c t i o n Due to the advance of modern multimedia technologies such as high storage equipment and high performance video streaming techniques, more digital video has become available to serve commercial, entertainment, education, and many other different purposes. For example, there are hundreds of digital T V channels t h a t can be recorded every day at home for future playback and millions of video clips available from the Internet for streaming a n d / o r file download. The popularization of digital equipment such as digital still cameras and digital video camcoders also makes it easy to capture home video for archiving. Generally, there are more and more digital video con543
544
tents available all over the world in every day. To effectively manage these contents, it is important to develop automatic techniques for video indexing, searching, browsing, editing, summarizing, skimming, and abstracting. Content-based video analysis attempts to organize the raw video data into systematic structures so that its semantic content can be effectively represented 1 _ 3 for future usage. Although video content understanding is an easy task for human beings, it is difficult for computers to have the same degree of semantic understanding and knowledge extraction. As a result, previous work has focused more on content description based on low-level features such as motion, shape and color descriptors in MPEG-7 2 ' 3 . Furthermore, techniques 4~8 such as shot change detection, keyframe selection, face recognition, object tracking, speech recognition, and audio segmentation have been developed for low-level video content analysis. Although the low-level image/video processing methodology is relatively mature and many reliable algorithms have been proposed, the way to bridge the gap between low-level syntactic features and high-level semantic meanings is generally known as an open and challenging problem. It is difficult to extract the semantic meaning of video based on low-level features without specific domain knowledge or models. To determine a good semantic model, it is important to incorporate all possible data modes associated with video such as audiovisual information, closed captions, texts, etc. This is known as the multimodal analysis. Multimodal feature extraction and model fitting can enhance the performance of knowledge-oriented detectors such as face, object, speech, and speaker recognition as well as help bridge the gap between low-level features and high-level semantic concepts. One significant application in content-based video analysis is the development of a method to access and browse a large collection of video data 9 _ u quickly and efficiently. The concept of video abstraction has been inspired accordingly. A good video abstraction scheme should have the capability to keep the significant parts of the original data and preserve as much relevant information as impossible. There are two different methods of video abstraction: video summarization and video skimming 9 ~ u . Video summarization, also known as storyboard techniques, employs key frames for navigation while video skimming consists of moving video clips and their corresponding audio information 12 ~ 14 . Video summarization has no timing synchronization issues with audio data. However, it is usually more interesting for humans to watch the trailer than a static picture 15>16. Video skimming presents a more complete concept via a short video clip
545
that contains both audio and visual information in most applications. The success of video skimming relies on the maturity of video content analysis techniques. We will explore this research topic in more depth. The rest of this chapter is organized as follows. We first examine features of audiovisual data and introduce several basic processing procedures in Sec. 2. Then, we explain the knowledge discovery problem by developing models to connect low-level syntactic features and high-level semantic models in Sec. 3. The video browsing, summarization, and skimming applications are discussed in Sec. 4. Finally, concluding remarks are given in Sec. 5. 2. Low-level Feature Extraction and Fusion Most video data contain both audio and visual information. In this section, we will discuss the extraction of features from these two domains, and present several basic processing techniques. 2.1. Visual Feature Extraction
and
Processing
The visual content in video can be decomposed into a hierarchical structure 4,17,18,19 a s s n o w n m pig i At the bottom level, a shot is usually chosen as the basic visual content unit 20 ~ 23 that consists of a set of consecutive and similar frames. One or more keyframes that represent the content can be extracted from each shot 24>25. Although a shot is often too fine a segmentation unit for the ultimate goal of semantic extraction, it is essential to detect shot boundary accurately as a preprocessing step. There are two major types of shot boundaries: abrupt and gradual changes. An abrupt change is resulted from camera break while a gradual change may be due to camera movement (panning, tilting, and zooming) or editing effects. Extensive work has been done in shot change detection 2 0 _ 2 3 . The scene is grouped by a collection of shots that are spatially or temporarily related. It us often used to convey a higher-level concept such as color similarity or time locality. Several works has been done for scene extraction using a model-based approach. However, results are more reliable only in specific application domains since the model-based approach demands good domain knowledge to build accurate models. Furthermore, low-level features of multiple modalities such as visual, audio, text, speech, and music can be integrated for more robust scene extraction. The techniques for scene extraction have become mature recently 26>27. As shown in Fig. 1, the higher the level, the more semantic meaning it should contain. Problems in high-level knowledge discovery become more
546 Event/Tempo
Semantic
i
< i W
Scene I lliHIj'll.'J
Shot ( I I I ! ipill
Frame
Fig. 1.
A schematic illustration of visual content in video.
challenging. Let us use two examples to illustrate this point. An event 28 ' 29 is a subjectively defined concept, which may refer to a dialog scene or a fighting scene in a movie. A tempo 30 is an abstract concept, which attempts to describe human experience in watching a movie. Once we obtain events or tempos, they can use for video abstraction. In general, video content analysis can be viewed as a video segmentation problem. By segmenting video data into different resolutions, we are able to summarize the content in multiple scales. 2.2. Audio Feature Extraction
and
Processing
Several features have been proposed to describe the characteristics of audio data. For example, descriptors were developed to describe silence, timbre, waveform, spectral, harmonics, and the fundamental frequency in the audio framework of MPEG-7 3 . For a given audio clip, it may contain different segments, it is important to segment the audio into different genres such as silence, music, and speech in the temporal domain 31 . There are different features associated with music and speech. Tempo, melody, and rhythm provide the middle level features for music. As to highlevel knowledge, we may be interested in the music genre. Speech data do carry a large amount of semantic meaning. Speech recognition and speaker identification are two major problems that have been well studied 28>29. Many successful techniques have been proposed. There are other techniques and applications for audio analysis. For example, we might want to detect some predefined events for video abstract.
547
In this case, we can make an association between events and special sounds and use it to find these events like the atomic sound effect (e.g. claps during a speech) and squealing brakes in a chasing event.
2.3. Multimodal
Feature
Fusion
The general concept of content-based semantic understanding via multimodal data analysis is depicted in Fig. 2. The fusion of multimodal information should provide basic semantic entities such as "who," "when," "where," "what," "how," and simple relation between entities, etc. for indexing video document 3 2 _ 3 5 . For example, we may want to identify how many people in this program, who they are, where and when this event takes place, what they do/say, etc. The need for multimodal analysis and fusion arises from practical considerations. Most low-level features contain partial description of an event. The amount of raw data is too large to be useful for higher levels. Thus, they need to be effectively combined to result to a concise description of the underlying content.
Video data
Multimodal content segmentation
Semantic units
Fig. 2. R e l a t i o n s h i p of m u l t i m o d a l c o n t e n t segmentation, fusion/integration, a n d basic semantic units.
Content segmentation, that forms the basis of multimodal analysis, should decompose video data in meaningful segments that are useful for the indexing of specific events, although current detectors are suitable for detecting and recognizing some particular events, most schemes belong to the unimodal approach. The multimodal approach can make a significant difference in the detection performance since it considers additional contextual relations into account.
548
3. Semantic Understanding and Knowledge Discovery 3.1. Semantic
Understanding
To fuse different modalities, one approach is based on the probabilistic framework as shown in Fig. 3. Jasinschi et al. 36 proposed to use Bayesian networks to model both content and context. Content that consists of visual, audio, and metadata modalities is used to describe different levels of abstraction while context describes the underlying structural information. A hierarchical organization is assumed in their framework based on low-level, middle-level and high-level detectors. Color, edge, and shape lowlevel features are employed in the visual modality in order to obtain face, keyframe, and videotext detectors. Twenty low-level features are used to obtain salient, noise, speech, music, etc. with detectors in audio modality. In metadata modality, the closed-caption is employed to get keywords and catalog information. For a high-level semantic meaning, TV program will be classified into talk show, financial news, and commercial topics. The Bayesian network is used to processing the content information. In this framework, each node in the network represent low-level features and middle-level detectors; the directed arcs represent the relationship between these nodes. Nodes and arc are related to conditional probability densities. By the joint probability distribution of low-level features and middle-level detectors, high-level semantics can be derived.
Constrained domain
Video data
Input data
Fig. 3.
High-level
Illustration of probabilistic-based networks for multimodal fusion/integration.
549
Another probabilistic approach is based on hidden Markov models (HMM). Huang et al. 37 proposed a graph framework for semantics video indexing, where semantic concepts are modeled via probabilistic detectors called the Multiject. There are three types of multijects: the site (e.g. outdoor or beach), object (e.g. human or car) and event (e.g. walking, flying or explosion). Since semantics concepts are related to each other, a graph probabilistic network called Multinets are used to model the interaction and context between them. By constructing multinets with the individual probabilities of all multijects using factor graphs, the proposed scheme can fit multimedia data into some extent of semantic indexes. HMM are often employed as a statistical classification method for multimodal feature fusion. Significant improvement can be made over the unimodal approach by considering the multimodal information and the probabilistic network jointly. The third popular approach is based on ontology/taxonomy. Zhou et 38 al. proposed an ontology/taxonomy structure to support the semantic content retrieval of sport video data. A generic and unified multi-level video model is used to represent audiovisual data, which includes spatial, temporal, and evolving events. Each level of data is mapped to a video processing module and an ontology structure. The five levels are raw data, video segmentation, perceptual feature content, conceptual content, and knowledge levels. The ontology provides a mechanism for exploiting the syntactic, structural, semantic and lexical-information of video data. Then, the decision tree learning algorithm can discover high-level knowledge.
3.2. Knowledge
Discovery
Knowledge discovery is a high-level task that is different from semantic understanding. Knowledge discovery attempts to find interesting patterns, multidimensional information summary, unusual events, or trend 39 while semantic understanding addresses the issue of video data interpretation. Although video is more accessible than ever via semantic content analysis, there is still a need to extract valuable knowledge from a large amount of video data. For example, the media source may contain hidden information about the speaker (facial expression, gesture, and emotion) and the location, time, and surroundings that might be critical to intelligence applications. Moreover, it might be necessary to correlate one media source with previous and other related sources to comprehend the whole story. Thus, it is desirable to develop innovative ideas for exploring video database to
550
obtain valuable knowledge. This requires the fusion of diverse technologies, including video/audio/text/speech processing and semantic understanding mentioned in Sec. 3.1, advanced data mining techniques, and knowledge representation and reasoning that will be discussed in this section. Although a lot of research has been performed on data mining, knowledge discovery from video data (or called video mining 40 ) has been little addressed before. This is truly an interdisciplinary research field, and there exist many challenging research issues. Figure 4 shows the knowledge discovery process from video data. As shown in the figure, the video mining process can be achieved via video content analysis and five major data mining modules: (1) feature-mining, (2) frequent-mining, (3) sequential-mining, (4) exception-mining and (5) move-mining. They will be detailed below.
Video content analysis
Fig. 4.
The knowledge discovery process from video using data mining techniques.
Feature-mining is used to select and extract high-level semantics and low-level features for classification and prediction. Although semantic understanding can extract and model a large pool of diverse features from a large amount of data, it is still difficult to select essential features in predicting domain-specific knowledge. There is often no obvious connection between high-level semantics and low-level features so that data mining methods should be exploited to discover such links. Commonly used methods include: 41 the hidden Markov chain models, the Markov Chain MonteCarlo (MCMC) method, the Expectation-Maximization (EM) algorithm, and the Bayesian information criterion (BIC) algorithm.
551 Frequent-mining can mine frequent patterns in video databases. Frequent patterns are the set of features t h a t appears relatively frequently in a large d a t a set. For example, terrorist activities may be strongly correlated with certain events, locations, suspects, and victims. Scalable methods for mining frequent patterns have been studied extensively in recent d a t a mining research. Such mining may help disclose the characteristics and hidden patterns of video d a t a in classifying them, finding unusual incidents, and predicting future recurring events 42 > 43 . Sequential-mining can mine sequential patterns in video databases. Sequential patterns are the set of subsequence events t h a t appear frequently in sequence d a t a sets. Mining sequential patterns could help predict future events and find regularities among ongoing events 4 4 . Exception-mining can mine outliers and unusual patterns. For many applications (such as homeland security, intrusion detection, etc.), normal patterns may have been known or not of user's interest, b u t unexpected events and outliers are often of great interest. It is a challenging task t o detect such unexpected events (or outliers) efficiently, effectively, with a low false alarm rate. Mining unexpected events needs good knowledge on what are usual events and general patterns. Thus, the first step for mining unusual patterns could be the extraction of usual ones, by clustering, classification, a n d / o r frequent/sequential pattern mining. Then, mining unusual patterns can be viewed as a complementary process 4 5 . Move-mining can extract space- and time-related video features. In video mining, spatial information is closely related to time since one is often interested in finding regularities and exceptions of moving objects. Mining spatio-temporal information from video d a t a can obtain the knowledge how object clusters and frequent patterns may evolve with time, whether such changes may have any regularity when being associated with multidimensional information to find highly correlated features. Knowledge discovery relies on good models and rules for each application and should be capable of automatically obtaining "interesting events," "exception events," "unknown events," and predicting "future events." Additionally, building up relationships among semantic units from multiple sources is equally important and is the foundation to support knowledge discovery such as video mining and prediction operations 4 0 .
552
4. Applications of Content-based Video Analysis In the last decade, extensive work has been done in content-based video analysis for various applications. The development of video content analysis technology is accredited with the maturity of digital signal and image 46 processing techniques. Many low-level features such as metadata, texts, images, audio, speech, and visual content have been studied, and extensive studies on this topic lay a solid foundation for semantic video processing. In this section, we will present several important applications and their associated technologies. 4.1. Video
Browsing
The concept of content-based image retrieval (CBIR) was proposed in early 90's. Among several experimental systems 47>48, the QBIC (Query by Image Content) system 47 , developed by IBM Almaden Research Center in 1995, provides a good example. The QBIC system supports complex multi-feature and multi-object queries from a large image database. Since images form a basic unit in video, the image retrieval technique is relevant to video analysis. In particular, the keyframe in a shot can be indexed and searched using the same technique. Video browsing aims at quick discovery of the information of interest to users. There are two scenarios: (1) to browse a large collection of video clips and (2) to browse one video clip to identify relevant segments. Generally, visual content browsing is easier than audio/speech browsing since the audio/speech data may not be understandable if the playback speed is too fast. Thus, most research work in this area focuses on visual content browsing. Video browsing can be achieved via various means such as the static storyboard, video skim, etc. Tse et al. 49 proposed a video browsing system by considering temporal and spatial properties of video data jointly. Key frames were extracted and browsing interfaces were constructed for keyframe display. To extract semantic-related concepts, users could participate in a series of experiments by varying parameters such as the key frames display rate, the number of simultaneous displays, and user perception in object recognition and gist determination, etc. Finally, results were generated and displayed via either static keyframes called the "filmstrip" or dynamic key frames called the "slide show." Chang et al. 50 discussed experiences in developing WebClip, Falvor, Zest and WedSEEK systems. These systems were designed to search and browse video data. Their functions include browsing,
553
content-based semantics retrieval, object segmentation, annotation, motion trajectory and video editor, and so on. He et al. 51 proposed a multimedia browsing system with the video-on-demand application as the target application. The work analyzed user's behavior on the intranet, exploited audio speedup, video summarization and annotation techniques, and presented various video browsing schemes suitable for this application.
4.2. Video Summarization
and
Skimming
Video summarization and skimming 9 _ u are somehow different from video browsing. Generally, video summarization/skimming can achieve the goal of video browsing. However, video summarization 12 ~ 14 and video skimming 15,16 should automatically retrieve the most significant and representative frames or segments instead of providing a large number of candidate results. Usually, video summarization and skimming employ various combinations of techniques that include shot detection, scene generation, motion detection, human face recognition, text detection, speech recognition, audio segmentation and music detection to generate a short synopsis of the original. For video skimming, a good skim is much like a movie trailer, which is synopsis of the entire movie and can be employed for the search and browse purposes. We can categorize video summarization techniques into three major classes: frame-based sampling 52>53, shot-based sampling 5 4 ' 5 5 ) and featurebased methods 56 > 57 ' 58 . As mentioned in Sec. 2.1, a shot is a smallest unit and consists of a set of consecutive and similar frames. Therefore, any frame in a shot can be picked up to represent its content meaning. The simplest way is to pick the first frame of each shot as its keyframe. Gong and Liu 56 proposed a video summarization and retrieval systems based on singular value decomposition (SVD). Zhang et al. 57 proposed a video retrieval and browsing scheme by extracting keyframes based on the color-histogram. In their scheme, the first frame in a shot is a default keyframe. Then, the color-histogram between the other frames and the last frame are calculated. If the value is larger than a predefined threshold, the extra new keyframe is obtained. Zhuang et al. 58 reported a method to cluster shots using the color-histogram and a representative frame in each cluster is selected to represent its content. Tseng et al. 59 proposed a video summarization scheme especially for pervasive mobile devices. In the area of video skimming, Smith and Kanade 15 ' 16 proposed a video skimming system based on audio and image characterization. The signif-
554 icant information such as specific objects, audio keywords, and relevant video is first extracted and then selecting video frames that contain human faces, texts, or camera motions creates a skim. Similarly, Toklu and Liou 2 5 proposed a video skimming scheme via visual, audio, and text information. Their work depends heavily on the text information. Pfeiffer et al. 60 proposed a system called VAbstract, which selects video clips from a movie by detecting special events such as dialogs, shots, explosions, and text occurrences. These events are then combined with important shots, where the importance is measured based on an action criterion. Sundaram and Chang 62 developed a semantics skimming system, where the visual complexity of a shot was measured in terms of the Kolmogorov complexity. Then, the complexity of a shot was mapped into the minimum time required for human understanding. Furthermore, they used the cinematic syntax rules to define a concept called the phrase, which consists of a sequence of shots to convey a particular semantic meaning and is adopted as the fundamental semantic unit. Then, three syntax reduction mechanisms were used to extract phrases from different types of scenes. Some further improvements of their algorithms have been made 63 ' 64 recently, and a constraint function of the utility model was applied for skim generation. Chang 6 5 also proposed a content-adaptive streaming concept based on skimming techniques. The proposed scheme allows allocating unequal bandwidth in streamed video according to the importance of segments.
5. Conclusion and Future Work In this chapter, we discussed the characteristics of video data and pointed out major challenging issues as well as ways to narrow the semantic gap between domain-specific knowledge and feature extraction. The use of multimodal features and/or data existing in video contents can provide results that are more robust. However, the key for knowledge discovery relies on the construction of a proper knowledge model for each specific application. Knowledge discovery from text data has become quite mature these days. It is our hope that knowledge discovery from video data can reach the same level of maturity. For example, we may expect to have a video search engines as powerful as Google in the near future. This is a challenging goal and still has many open issues for future research.
555
References 1. B. S. Manjunath, T. Huang, A. M. Teklap and H. J. Zhang, "Guest editorial introduction to the special issue on image and video processing for digital libraries," IEEE Transactions on Image Processing, vol. 9, pp. 1-2, 2000, 2. S. F. Chang, A. Puri, T. Sikora, and H. J. Zhang, "Introduce to the special issue on MPEG-7," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, pp. 685-687, 2001. 3. B. S. Manjunath, P. Salembier and Thomas Sikora, "Introduction to MPEG7," John Wiley & Sons, LTD, England, 2002. 4. A. Hampapur, R. Jain and T. Weymouth, "Digital video segmentation," ACM Multimedia, pp. 357-364, 1994. 5. H. J. Zhang, J. Wu, D. Zhong and S. W. Smoliar, "An integrated system for content-based video retrieval and browsing," Pattern Recognition, vol. 30, no. 4, pp. 62-72, 1994. 6. S. F. Chang, J. R. Smith, M. Beigi and A. Benitez, "Visual information retrieval from large distributed on-line repositories," Proceedings of ACM Communication, vol. 40, no. 12, pp. 63-71, 1997. 7. Y. Rui, T. S. Huang and S. Mehrotra, "Constructing table-of-content for video," A CM Journal of Multimedia Systems, vol. 7, pp. 359-368, 1998. 8. N. Sebe, M. S. Lew, X. Zhou and T. S. Huang, "The state of the art in image and video retrieval," Proceedings of International Conference on Image and Video Retrieval, pp. 1-8, 2003. 9. R. Lienhart, S. Pfeiffer and W. Effelsberg, "Video abstracting," Communications of the ACM, pp. 55-62, 1997. 10. A. Hanjalic and H. J. Zhang, "An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis," IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, no. 8, 1999. 11. Y. Li, T. Zhang and DTretter, "An overview of video abstraction techniques," HP Laboratories Technical Report, HPL-2001-191, 2001. 12. D. Dementhon, V. Kobla and D. Doermann, "Video summarization by curve simplification," Proceedings of ACM Multimedia, pp. 211-218, 1998. 13. X. D. Sun and M. S. Kankanhalli, "Video summarization using Rsequences," Real-time Imaging, pp. 449-459, 2000. 14. C. M. Taskiran, A. Amir, D. Ponceleon and E. J. Delp, "Automated video summarization using speech transcripts," Proceedings of SPIE, pp. 371-382, 2002. 15. M. Smith and T. Kanade, "Video skimming for quick browsing based on audio and image characterization," Technical Report CMU-CS-95-186, pp. 1-12, 1995. 16. M. Smith and T. Kanade, "Video skimming and characterization through the combination of image and language understanding techniques," Proceedings of the IEEE Computer Vision and Pattern Recognition, pp. 775-781, 1997. 17. Q. Wei, H. J. Zhang and Y. Zhong, "A robust approach to video segmentation using compressed data," Proceedings of SPIE, vol. 3022, pp. 448-456,
556 1997. 18. H. Jiang, T. Lin and H. J. Zhang, "Video segmentation with t h e assistance of audio content analysis," Proceedings of IEEE International Conference on Multimedia and Expo, vol. 3, p p . 1507-1510, 2000. 19. H. S u n d a r a m and S. F . Chang, "Video scene segmentation using video and audio features," Proceedings of IEEE Conference on Multimedia and Expo, vol. 2, p p . 1145-1148, 2000. 20. B. L. Yeo and B. Liu, "Rapid scene analysis on compressed video," IEEE Transactions on Circuit and System for Video Technology, vol. 5, no. 6, p p . 533-544, 1995. 21. J. S. Boreczky and L. A. Rowe, "Comparison of video shot b o u n d a r y detection techniques," Proceedings of SPIE on Storage and Retrieval for Image and Video Databases, vol. 2670, p p . 170-179, 1996. 22. U. Gargi, R. Kasturi and S. H. Strayer, "Performance characterization of video-shot-change detection methods," IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, no. 1, 2000. 23. C. L. Huang and B. Y. Liao, "A robust scene-change detection m e t h o d for video segmentation," IEEE Transactions on Circuit and System for Video Technology, vol. 11, no. 12, p p . 1281-1288, 2001. 24. A. Girgensohn and J. Boreczky, "Time-constrained keyframe selection technique," Proceedings of ICMCS, p p . 756-761, 1995. 25. C. Toklu and S. P. Liou, "Automatic keyframe selection for content-based video indexing and access," Proceedings of SPIE, vol. 3972, p p . 554-563, 2000. 26. R. Zabih, J. Miller and K. Mai, "A feature-based algorithm for detecting and classifying scene breaks," Proceedings of the third ACM International Conference on Multimedia , p p . 189-200, 1995. 27. H. S u n d a r a m a n d S. F . Chang, "Determining c o m p u t a b l e scenes in films and their structures using audio-visual memory models," Proceedings of the Eighth ACM International Conference on Multimedia , p p . 95-104, 2000. 28. Y. Li, S. Narayanan and C.-C. Jay Kuo, "Adaptive speaker identification with audiovisual cues for movie content analysis," accepted for publication in Pattern Recognition Letters 29. Y. Li, S. Narayanan and C.-C. Jay Kuo, "Content-based movie analysis a n d indexing based on audiovisual cues," accepted for publication in t h e IEEE Transactions on Circuits and Systems for Video Technology. 30. E. D. Scheirer, "Tempo and b e a t analysis of acoustic musical signals," Journal Acoustics Society of America, vol. 103, no. 1, p p 588-601, 1998. 31. T . Zhang and C.-C. Jay Kuo, "Heuristic approach for generic audio d a t a segmentation and annotation," Proceedings of the Seventh ACM International Conference on Multimedia, p p . 67-76, 1999. 32. S. F . Chang and H. S u n d a r a m , "Structural and semantic analysis of video," Proceedings of IEEE Conference on Multimedia and Expo, vol. 2, p p . 687690, 2000. 33. J. G u o and C.-C. Jay Kuo, "Semantic video object segmentation for contentbased multimedia applications," Kluwer Academic Publishers, 2001.
557 34. Y. Li and C.-C. Jay Kuo, "Video content analysis using multimodal information for movie content extraction, indexing and representation, Kluwer Academic Publishers, 2003. 35. Cees G. M. Snoek and M. Worring, "Multimodal video indexing: a review of the state-of-the-art," Multimedia Tools and Applications, 2001. 36. R. S. Jasinschi, N. Dimitrova, T. McGee, L. Agnihotri, J. Zimmerman, D. Li and J. Louie, "A probabilistic layered framework for integrating multimedia content and context information," Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2057—2060, 2002. 37. M. R. Naphade, I. V. Kozintsev and T. S. Huang, "A factor graph framework for semantics video indexing," IEEE Transactions on Circuit and System for Video Technology, vol. 12, no. 1, pp. 40-2002. 38. W. Zhou, S. Dao and C.-C. Jay Kuo, "On-line knowledge-based and rulebased video classification system for video indexing and dissemination," Information Systems, vol. 27, pp. 559-586, 2002. 39. C. Olaru and L. Wehenkel, "Data mining," IEEE Computer Applications in Power, vol. 12, no. 3, pp. 19-25, 1999. 40. A. Rosenfeld, D. Doermann, D. Dementhon, "Video mining," Kluwer Academic Publishers, 2003. 41. T. Hastie, R. Tibshirani, and J. Friedman, "The Elements of Statistical Learning: Data Mining, Inference, and Prediction," Springer-Verlag, New York, 2001. 42. R. Agrawal and R. Srikant, "Fast algorithms for mining association rules," Proceedings of International Conference on Very Large Data Bases,pp. 487499, 1994. 43. J. Han, J. Pei, and Y. Yin, "Mining frequent patterns without candidate generation," Proceedings of A CM-SIGMOD International Conference on Management of Data, pp. 1-12, 2000. 44. R. Srikant and R. Agrawal, "Mining sequential patterns: generalizations and performance improvements," Proceedings of International Conference on Extending Database Technology, pp. 3-17, 1996. 45. W. Jin, K. H. Tung, and J. Han, "Mining top-N local outliers in large databases," proceedings of ACM SIGKDD International Conference on Knowledge Discovery in Databases, pp. 293-298, 2001. 46. W. K. Pratt, "Digital image processing," John Wiley & Sons, INC, 1991. 47. M. Fichner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafiner, D. Lee, D. Petkovic, D. Steele and P. Yanker "Query by image and video content: the QBIC system," IEEE Computer, vol. 289, pp. 23-32 1995. 48. J. R. Smith, and S. F. Chang, "VisualSEEK: A full automated contentbased image query system," A CM Conference on Multimedia, pp. 87-98, 1996. 49. T. Tse, G. Marchionini, W. Drag, L. Slaughter and A. Komlodi, "Dynamic key frame presentation techniques for augmenting video browsing," Proceedings of the working conference on Advanced Visual Interfaces, pp. 185-194, 1998.
558 50. S. F . Chang, A. Eleftheriadis and R. McClintock, " Next-generation content representation, creation, and searching for new-media applications in education," Proceedings of the IEEE, vol. 86, p p . 884-904, 1998. 51. L. He, A. G u p t a , S. A. W h i t e , and J. Grudin, "Corporate deployment of on-demand video: usage, benefits, and lessons," MSR-TR-98-62, Microsoft Research R e d m o n d , WA. 98052 USA, 1998. 52. M. Mills, J. Cohen and Y. Y. Wong, "A magnifier tool for video d a t a , " Proceedings of the SIGCHI Conference on Human factors in computing systems, p p . 93-98, 1992. 53. Y. Taniguchi, A. Akutsu, Y. Tonomura and H. H a m a d a , "An intuitive and efficient access interface to real-time incoming video based on a u t o m a t i c indexing," Proceedings of the third ACM International Conference on Multimedia, P P . 25-33, 1995. 54. S. W . Smoliar and H. J. Zhang, "Content-based video indexing and retrieval," IEEE Multimedia, p p . 62-72, 1994. 55. F . A r m a n , R. Depommier, A. Hsu and M. Y. Chiu, "Content-based browsing of video sequences," Proceedings of the second ACM International Conference on Multimedia, p p . 97-103, 1994. 56. Y. Gong and X. Liu, "Video summarization and retrieval using singular value decomposition," Multimedia Systems, vol. 9, p p . 157-168, 2003. 57. H. J. Zhang, J. Wu, D. Zhong and S. W . Smoliar, "An integrated system for content-based video retrieval and browsing," Pattern Recognition, vol. 30, no. 4, p p . 643-658, 1997. 58. Y. Zhuang, Y. Rui, T. S. Huang and S. Mehrotra, "Adaptive key frame extraction using unsupervised clustering," Proceedings of IEEE International Conference on Image Processing, vol. 1, 866-870, 1998. 59. B. L. Tseng and C. Y. Lin and J. R. Smith, "Video summarization a n d personalization for pervasive mobile devices," Proceedings of SPIE, vol. 4676, p p . 359-370, 2002. 60. S. PfeifFer, R. Lienhart, S. Fischer and W . EfFelsberg, "Abstracting digital movies automatically," Journal of Visual Communication and Image Representation, vol. 7, no. 4, p p . 345-353, 1996. 61. R. W . Lienhart, "Dynamic video summarization of home video," Proceedings of SPIE, vol. 3972, p p . 378-389, 2000. 62. H. S u n d a r a m and S. F . Chang, "Condensing computable scenes using visual complexity and film syntax analysis, IEEE Conference on Multimedia and Exhibition, p p . 22-25, 2001. 63. H. S u n d a r a m , L. Xie, S. F . Chang, "A utility framework for t h e a u t o m a t i c generation of audio-visual skims, Proceedings of the Tenth ACM International Conference on Multimedia, p p . 189-198, 2002. 64. H. S u n d a r a m and S. F . Chang, "Constrained utility maximization for generating visual skims," IEEE Workshop on Content-based Access of Image and Video Libraries, p p . 124-131, 2001. 65. S. F . Chang, "The Holy Grail of content-based media analysis," IEEE Multimedia, p p . 6-10, 2002.
CHAPTER 5.4 OBJECT-PROCESS METHODOLOGY AND ITS APPLICATIONS TO IMAGE PROCESSING AND PATTERN RECOGNITION
Dov Dori Technion, Israel Institute of Technology, Haifa 32000, Israel, and Massachusetts Institute of Technology, Cambridge, MA 0213, USA E-mail: [email protected] Modeling of complex systems should conveniently combine structure and behavior in a single model. Motivated by this observation, Object-Process Methodology (OPM) is a comprehensive, holistic approach to modeling, study, development, engineering, evolution, and lifecycle support of systems. Employing a combination of graphics and a subset of English, the OPM paradigm integrates the object-oriented, process-oriented, and state transition approaches into a single frame of reference. Due to its structure-behavior integration, OPM provides a solid basis for modeling complex systems in general and pattern recognition systems in particular. This chapter provides an overview of OPM, its ontology, semantics, and symbols. It then describes applications of OPM in various domains with focus on image processing and pattern recognition, and concludes with a sample of a generic image processing example model.
1
The OPM Ontology
Function, structure, and behavior are the three main aspects that any conceivable system exhibits. Function is the top-level utility that the system provides its beneficiaries who use it or are affected by it, either directly or indirectly. The system's function is enabled by its architecture - the combination of structure and behavior. The system's architecture is what enables it to function so as to benefit its users. Modeling of complex systems should conveniently combine structure and behavior in a single model. Motivated by this observation, Object-Process Methodology (OPM) (Dori 1995, 2002) is a comprehensive, holistic approach to modeling, study, development, engineering, evolution, and lifecycle support of systems. Among many fields of application, it was also applied to pattern recognition, as we describe below. 559
560 Employing a combination of graphics and a subset of English, the OPM paradigm integrates the object-oriented, process-oriented, and state transition approaches into a single frame of reference. Structure and behavior coexist in the same OPM model without highlighting one at the expense of suppressing the other to enhance the comprehension of the system as a whole. Rather than requiring that the modeler views each of the system's aspects in isolation and struggle to mentally integrate the various views, OPM offers an approach that is orthogonal to customary practices. According to this approach, various system aspects can be inspected in tandem for better comprehension. Complexity is managed via detail levels, which are generated and traversed through by several abstraction/refinement mechanisms. Due to its structure-behavior integration, OPM provides a solid basis for modeling complex systems in general and pattern recognition systems in particular. This chapter provides an overview of OPM, its ontology, semantics, and symbols. It then describes applications of OPM in various domains with focus on image processing and pattern recognition, and concludes with a sample of a generic image processing example model. The elements of the OPM ontology, shown in Table 1, are divided into three groups: entities, structural relations, and procedural links. 1.1
Entities
Entities, the basic building blocks of any system modeled in OPM, are of three types: objects with states, and processes. The symbols for these three entities are respectively shown as the first group of symbols at the left hand side of Figure 1, which is the symbols in the toolset available as part of the GUI of OPCAT 2 (Dori, Reinhartz-Berger et al. 2003). entities
structural relations
procedural
links
Figure 1. The three groups of OPM symbols in the toolset of OPCAT 2
1.2
OPM things: object and processes
Objects are (physical or informatical) things that exist, while processes are things that transform (create, destroy, or change the state of) objects. Below we provide a set of basic definitions that build on top of each other. An object is a thing that exists. Objects are the things that are being transformed in the system.
561 Transformation is generation (creation) or consumption (destruction) of an object, or a change of its state. Processes are the things that transform objects in the system. A process is a thing that represents a pattern of object
transformation.
Table 1. Things of the OPM ontology and their basic attributes
Thing / Attribute Object
Symbol
Description / OPL sentence A thing (entity) that has the potential of stable, unconditional physical or mental existence.
Object Name
Object Name is an object.
Process
( Processing }
A thing representing a pattern of transformation that objects undergo. Processing is a process.
Obied
j
Essence { Pi<xer',ina )
I
object
;
Affiliation j "
"
"
Processing is physical. An attribute that determines whether the thing is environmental (external to the system, dashed contour) or systemic.
" ~-
\ Processing > %
An attribute that determines whether the thing (object or process) is physical (shaded) or informational.
*- - _ - - '
Processing is environmental.
In OPL, bold Arial font denotes non-reserved phrases, while non-bold Arial font denotes reserved phrases. In OPCAT, various OPM elements are colored with the same color as their graphic counterparts (objects are green, processes are blue, and states are brown). Objects and processes are collectively called things. The first two lines of Table 1 shows the symbol and a description of the two types of OPM things. The next two lines show two basic attributes that things can have: essence and affiliation. Essence is an attribute that determines whether the thing is physical or informational. The default essence is informatical. A thing whose essence is physical is symbolized by a shaded shape. Affiliation is an attribute that determines whether the thing is environmental (external to the system) or systemic.
562
The default affiliation is systemic. A thing whose affiliation is environmental is symbolized by a dashed contour. 1.3
OPM states
Objects can be stateful, i.e., they may have one or more states. A state is a situation at which an object can exist at certain points during its lifetime or a value it can assume. Stateful objects can be, affected, i.e., their states can change. Effect is a change in the state of an object. Table 2. States and values
Description / OPL sentence
Symbol Stateful object with two states
! reachahle |
A situation at which an object can exist.
i
Website
| unreachable 1 1
A value that an object can assume.
Temperature
Value
Website can be reachable or unreachable.
Temperature is 15. ,..„.„„.,,
Stateful object with three states: initial, default, and final
2.
A state can be initial, default, or final. 1
dl
J ( "•" 1 , » - " "
1 jun^ j)
j
Car can be new, which is initial, used, which is default, or junk, which is final.*
"""
OPM Structure Modeling
Structural relations express static, time-independent relations between pairs of entities, most often between two objects. Structural relations, shown as the middle group of six symbols in Figure 1, are of two types: fundamental and tagged. 2.1
The four fundamental structural relations
Fundamental structural relations are a set of four structural relations that are used frequently to denote relations between things in the system. Due to their prevalence and usefulness, and in order to prevent too much text from cluttering the diagram, these relations are designated by the four distinct triangular symbols shown in Figure 1. The four fundamental structural relations are:
563
(1) aggregation-participation, a solid triangle, Aj, which denotes the relation between a whole thing and its parts, (2) generalization-specialization, a blank triangle, A, which denotes the relation between a general thing and its specializations, giving rise to inheritance, (3) exhibition-characterization, a solid inside blank triangle, Aj, which denotes the relation between an exhibitor - a thing exhibiting a one or more features (attributes and/or operations) - and the things that characterize the exhibitor, and (4) classification-instantiation, a solid circle inside a blank triangle, iA, which denotes the relation between a class of things and an instance of that class. Table 3. The fundamental structural relation names, OPD symbols, and OPL sentences Structural Relation Name Forward
Backward
Aggregation
Participation
Root
Exhibition
Characterization
03}
Whole Parts
CD QLJ
DL]
Exhibitor Features
OPL Sentences with 1, 2, and 3 refineables
OPD with 3 refineables
Refin eable s
i
7°} B
j
r C
}
.
! D
i
General
Generalization
Specialization
Classification
Instantiation
Li
Specializat r ions Class Instances
% D) Dd To"i
l
f\
,
CD D J CD
A consists of B. A consists of B and C. A consists of B, C, and D. A exhibits B. A exhibits B and C. A exhibits B, C, and D. B is an A. B and C are As. B, C, and D are As. B is an instance of A. B and C are instances of A. B, C, and D are instances of A.
Table 3 lists the four fundamental structural relations and their respective OPDs and OPL sentences. The name of each such relation consists of a pair of dash-separated words. The first word in each such pair is the forward relation name, i.e., the name of the relation as seen from the viewpoint of the thing up in the hierarchy. The second word is the backward (or reverse) relation name, i.e., the name of the relation as seen from the viewpoint of the thing down in the hierarchy of that relation. Each fundamental structural relation has a default, preferred direction, which was determined by how natural the sentence sounds. The preferred shorthand name for each relation is underlined. Likewise, underlined sentences are the default sentences. They are more useful, as they sound more natural. As Table 3 shows, each one of the four fundamental structural relations is characterized by the hierarchy it induces between the
564
root—the thing attached to the tip of the triangle and the leaves—the thing(s) attached to the base of the triangle, as follows. (1) In aggregation-participation, the tip of the solid triangle, Mi, is attached to the whole thing, while the base—to the parts. (2) In generalization-specialization, the tip of the blank triangle, [A, is attached to the general thing, while the base—to the specializations. (3) In exhibition-characterization, the tip of the solid inside blank triangle, Af, is attached to the exhibitor (the thing which exhibits the features), while the base is attached to the features (attributes and operations). (4) In classification-instantiation, the tip of the solid circle inside a blank triangle, IA, is attached to the thing class, while the base—to the thing instances. The things which are the leaves of the hierarchy three, namely the parts, features, specializations, and instances, are collectively referred to as refineables, since they refine the ancestor, the root of the tree. Refineable is a generalization of part, feature, specialization, instance.
and
The third column in Table 3 lists for each fundamental structural relations the name of the root (whole, exhibitor, general, class) and the corresponding refineables (parts, features, specializations, and instances). The next column contains an OPD with three refineables, while the rightmost column lists the syntax of three OPL sentences for each fundamental structural relations, with one, two, and three refineables, respectively. Having presented the common features of the four fundamental structural relations, in the next four subsections we provide a small example for each one of them separately. 2.2
Aggregation-participation
Aggregation-participation denotes the relation between a whole and it comprising parts or components. Consider, for example, the excerpt taken from Section 2.2 of the RDF Primer (Manola and Miller 2003): ... each statement consists of a subject, a predicate, and an object. This is a clear case of whole-part, or aggregation-participation relation. The OPM model of this statement, which consists of both the OPD and the corresponding OPL, is shown in Figure 2. Note that the OPL sentence, "RDF Statement consists of Subject, Predicate, and Object." which was generated by OPCAT automatically from the graphic input, is almost identical to the one cited from the RDF Primer. The same OPD exactly (disregarding the graphical layout) can be produced by inputting the text of the OPL sentence above. This is a manifestation of the OPM graphics-text equivalence principle.
565
RDF Statement
Subject
PiPtln.ate
0|i|Bl,t
RDF Statement consists of Subject, Pi eJicate, and Object. Figure 2. OPD of the sentence "RDF Statement consists of Subject, Predicate, and Object" The aggregation-participation relation is not always as easy to detect as it is in the example we have just seen, where the OPL reserved phrase "consists of" clearly indicates the existence of this relation. To see a more subtle example, consider another sentence extracted from the same RDF Primer (Manola and Miller 2003): RDF models statements as nodes and arcs in a graph. In order to model this sentence in OPM, using our prior knowledge about graphs and ignoring the possibility of an empty or a single-node graph, we should break it into simpler and more explicit sentences: (1) A graph consists of at least one node and optional arcs. (2) A graph models at least one RDF statement. For now, we will only model the first sentence, since it deals with the aggregationparticipation relation. Sentence (1) above is modeled in the Figure 3. The plus (+) symbol above Node denotes the "at least one" (1..m) participation constraint, while the asterisk (*) symbol above Arc denotes the "optional" (0..m) participation constraint. 2.3
Generalization-specialization
Generalization-specialization is a fundamental structural relationship between a general thing and one or more of its specializations. Continuing our example from the RDF Primer (Manola and Miller 2003), consider the very first sentence from the abstract: The Resource Description Framework (RDF) is a language for representing information about resources in the World Wide Web.
566
.* Gpcat II - RDF basi System
9
Edit View
Natation
MjZtaiJSl j ^ i u > i a J t . MB»- i-SLffllUBaih
Operation
Generation
nJAuMai-.H*Uu3SiMMji
Help
^RDFbaEit E>1 SD
iXO d'd
£2 Giijih consists of optional Arcs and at least one Mode. i OPL Generator Thing ; "l?^OPDTree_
Figure 3. The OPM model of a graph consisting of at least one node and optional arcs Let us take the main message of this sentence, which is that RDF is a language. This is exactly in line with the OPL syntax, so we can input the OPL sentence "RDF is a Language." into OPCAT and see what we get. The result, without any diagram editing, is shown in Figure 4, along with the conversation window titled "Add new OPL sentence," in which this sentence was typed prior to the OPD creation.
2.4
Exhibition-characterization
We continue to scan the RDF Primer (Manola and Miller 2003), where in Section 2.2.1 we find the sentence RDF has a simple data model. To model this statement, we need to rephrase this sentence into the following three sentences: (1) RDF is characterized by a data model. (2) The data model of RDF is characterized by a complexity attribute. (3) The value of this complexity attribute is "simple."
567
System Edit View Natation Operation Generation Help
SBElQ
S, ^ fl iff
S« -A' Q. <=L * O
j a H
Add new OPL sentence (Add new sentence)
I&&J
•w
RDF is a Language Addjj
Update
|| Clear ;
Exit ;
Figure 4. The OPD obtained by inputting into OPCAT the OPL sentence "RDF is a Language" These three sentences are further rephrased to conform to the OPL syntax as follows: 1. RDF exhibits Data Model. 2. Data Model exhibits Complexity. 3. Complexity is simple.
IQSD?
;rf*tf El
Figure 5. The OPD representing the sentence "RDF has a simple data model."
2.5
Classification-instantiation
Reading through the RDF Primer, we find in Section 3.3 on datatypes the sentence: Datatypes are used by RDF in the representation of values, such as integers, floating point numbers, and dates.
568 RDF predefines just one datatype, rdf.XMLLiteral, used for embedding XML in RDF. An OPL interpretation of these two sentences, respectively, is: 1. RDF exhibits many Datatypes. 2.
XMLLiteral is an instance of Datatype.
Figure 6 is the OPM model of XMLLiteral, an instance of a Datatype of RDF. c*DK 11
• QSD Naincsp^rt3 idt
WWW w3 oiaMrjG9/02/:2-rdf-«)/ntdvii5#
""RDF
£y D
T'
in f
|
j Datatype *
—
•
—
—
•
—
•
J—j —
^
^
XMLLiteral i
__L^ZE:
....... -•__JI|
RE F exhibit s many E at ityji P XMLLiteral IC instance of a E aiifYpe.
i
Figure 6. The OPM model of XMLLiteral, an instance of a Datatype of RDF
3.
OPM Behavior Modeling
Procedural links connect entities (objects, processes, and states) to express dynamic, time-dependent behavior of the system. Behavior, the dynamic aspect of a system, can be manifested in OPM in three ways: (1) A process can transform (generate, consume, or change the state of) objects, (2) An object can enable a process without being transformed by it, and (3) An object or a process can trigger an event that might, in turn, invoke a process if some conditions are met. Accordingly, a procedural link can be a transformation link, an enabling link, or an event link. In order to be able to talk about object transformation, we need to first define state and demonstrate how states are used.
569 3.1
Object states
In Figure 7 we added to the object Check two states: The initial state uncashed and the final state cashed. This causes the addition of the following OPL sentence to the OPL paragraph: Check can be uncashed, which is initial, or cashed, which is final.
3.2
Transformation links
A transformation link expresses how a process transforms one or more objects. The transformation of an object can be its consumption, generation, or state change. The transforming process is the transformer, while object that is being transformed is called transformee. 3.3
Input and output links
Having added the states to the object Check, we can now show how the process Cashing affects Check by changing its state. In Figure 8, Cashing was added and linked to the two states of Check: An input link leads from the initial uncashed state to Cashing, while an output link leads from Cashing to the final state cashed.
*• Opcat If - Chfck-Payee System
Edit View
Notation
Operation
iLffl^!!s4ia^J£'t ts&J
Generation
Help
i>ir';. so
R^D* E3 Check
t u r cash >d J
"-1
cash is
Payee
Figure 7. Adding states to Check
The OPL sentence generated automatically by OPCAT as a result of adding these input and output links is: Cashing changes Check from uncased to cashed.
570 Table 4. Summary of the procedural links between two processes
Number of states or timeline
OPD
OPL
Statsful Object
Single state
Stateful Object is singular.
singula! j
Stateful Object
Two states
Three states or more
Three states or more
singular
Stateful Object can be singular or plural.
plural
Stateful Object | first
|
second j
third
j
[fourth 1
Stateful Object
OfO
[second
[ third
j
([fourth ]|
Stateful Object can be first, second, third, or fourth.
Stateful Object can be first, which is initial, second, third, which is default, or fourth, which is final.
Note that from a pure graphics view, the input and output links are identical. Their role is determined by inspecting their context, i.e., their source and destination: The input link leads from the input state to the affecting process, while the output link leads from that process to the output state. Not only do these two links look identical, but the same tool in the OPCAT toolset is used to create them. The input link is drawn by dragging it from the input state to the process, while the output link is drawn by dragging it from the process to the output link. 3.4
Effect link
Sometimes we may not be interested in specifying the states of an object but still show that a process does affect an object by changing its state from some unspecified input state to another unspecified output state. To express this, we suppress (hide) the input and output states of the object, so the edges of the input and output links "migrate" to the contour of the object and coincide, yielding the effect link shown in Figure 9.
571
ESBB
» Opcat !I - Check-Payee system Edit VIPUI
9
Hutdtmn operation
•
. SD SHUheck 2M Payee GKuasHing
Generation Help
rf'tf
so
\\ -_
|
Chyuk
]
sS
""*[
(i^
''<•'•
•<: *
!
»
.
~
J
s»
M^gy^r I
H
rav
"
i
~ ~
^ r
»
•
Caching change* 1 i from i =
w s
JZJ
OPL Generator
!
S /> /
1 .^OPO-II»W i g g f J i J
V
Figure 8. The Cashing process changes the state of Check from the uncashed to cashed The OPL sentence that represents this graphic construct is: Cashing affects Check.
X BSD
d'tif
0 r=
i
nfrrr\
.aoay*w«tt
j
(
Payee
""'""
Cashing
-
_^
j
JL
HI
~
OPL Generator
• oo AAAA^^ f ?SS ?> ? Figure 9. Suppressing the input and output states of Check
3.5
Result and consumption links
We have seen that one type of object transformation is effect, in which a process changes the state of an object from some input state to another output state. When these two states are expressed (i.e., explicitly shown), then we can use the pair of input and output links to specify the source and destination states of the transformation. When the states are
572 suppressed, we express the state change by the effect link, a more general and less informative transformation link. State change is the least drastic transformation that a process can undergo. Two more extreme transformations are generation and consumption, denoted respectively by the result and consumption links. Generation is a transformation which causes an object, which had not existed prior to the process execution, to become existent. For example, Check is born as a result of a Check Making process. As Figure 10 shows, the object Check is generated as a result of executing the process Check Making. The result link is the arrow originating from the generating process and leading to the generated object. The OPL sentence that represents this graphic construct (shown also in Figure 10) is: Check Making yields Check.
Figure 10. The object Check is generated as a result of executing the Check Making process In contrast to generation, consumption is a transformation which causes an object, which had existed prior to the process execution, to become non-existent. For example, Check is consumed as a result of a Destroying process. As Figure 11 shows, the object Check is consumed as a result of executing the process Destroying. The consumption link is the arrow originating from the consumed object and leading to the consuming process. The OPL sentence that represents this graphic construct (shown also in Figure 11) is: Destroying consumes Check.
573 •""Tr" Tif —"
% Opcat li - Check-Payee System
Edit
9B5iQ
View
Notation
WSSMMSISS
Operation
Generation
a ty B IP *>
Help
X -M <%. ^ , fc n
B ^DK
SD
0 |
, 7 ,,itf3*-ig. :G
Check
• a** c J S *
Payep
Figure 11. The object Check is consumed as a result of executing the Destroying process
Note that from a pure graphics view, the consumption and result links look identical. Their role is determined by inspecting their context, i.e., their source and destination: The consumption link leads from the consumed object to the consuming process, while the result link leads from the generating process to the resulting object. Not only do these two links look identical, but they also look identical to the input and output links, so actually the same arrow denotes four different links: input, output, consumption and result. The same tool in the OPCAT toolset is used to create all four of them. The consumption link is drawn by dragging it from the consumed object to the consuming process, while the result link is drawn by dragging it from the generating process to the resulting object.
574 Table 5. Summary of the procedural links between a process and an object or its state Transform ation Link Type
Source Entity
Destination Entity
Input link
Input state
Affecting process
Output link
Affecting process
Output state
Consumpti on link
Consumed object
Consuming process
Consuming Process consumes Consumed Object.
Result link
Generating process
Resulting object
Generating Process yields Generated Object.
Statespecified Consumpti on link
Final state of the consumed object
Consuming process
Statespecified Result link
Generating process
Initial state of the resulting object
OPD
OPL
Affecting Process changes Object from input state to output state.
Consuming Process :;ofi5timinci
\
procsss j consumes terminal Consumed Object.
Generating Process yields initial Generated Object.
3.6 State-specified result and consumption links We have seen that result and consumption links respectively denote generation and destruction of objects. We sometimes wish to be more specific and to state not only that an object is generated by a process, but also at what state that object is generated. Some other times, we might wish to be able to state not only that an object is consumed by a process, but also at what state that object has to be in order for it to be consumed by the process. As Figure 10 shows, the object Check is generated in its unendorsed state as a result of executing the process Check Making. The OPL sentence that represents this state-specified result link graphic construct (shown also in Figure 12) is: Check Making yields unendorsed Check.
575
In comparison, the "regular," non-state-specified result link is the same, except that the (initial) state is not specified: Check Making yields Check.
JfOsir""""""""^^
Figure 12. The object Check is generated in its unendorsed state as a result of executing the Check Making process The difference is the addition of the state name (unendorsed in our case) before the name of the object (Check) that owns that state. Note that from a pure graphics view, the consumption and result links look identical. Their role is determined by inspecting their context, i.e., their source and destination: The consumption link leads from the consumed object to the consuming process, while the result link leads from the generating process to the resulting object. Not only do these two links look identical, but they also look identical to the input and output links, so actually the same arrow denotes four different links: input, output, consumption and result. The same tool in the OPCAT toolset is used to create all four of them. The consumption link is drawn by dragging it from the consumed object to the consuming process, while the result link is drawn by dragging it from the generating process to the resulting object. Table 5 provides a summary of the six procedural links between a process and a (possibly stateful) object. They are divided into three pairs: Input and output links, which always come as a pair, consumption and result links, and state-specified consumption and result links. 4.
Applications of OPM in Pattern Recognition
OPM has been applied in many domains, including education (Dori & Dori, 1996), computer integrated manufacturing (Dori, 1996A; Dori, Gal & Etzion, 1996), The R&D universe and its feedback cycles (Myersdorf & Dori, 1997), real-time systems (Peleg & Dori, 2000), banking (Dori, 2001), requirements engineering (Soffer, Golany, Dori & Wand, 2001), Web applications development (Reinhartz-Berger, Dori & Katz, 2002), ERP modeling (Soffer, Golany & Dori, 2001), axiomatic design (Soderborg, Crawley &
576
Dori, 2002), computational synthesis (Dori & Crawley, 2003), software reuse (ReinhartzBerger, Dori & Katz, 2002), and systems architecture (Soderborg, Crawley & Dori, 2003). In this chapter, we focus on applications of OPM in the domains of image processing and pattern recognition.
4.1
Image Processing and Pattern Recognition applications of OPM
Specifying how automated understanding of engineering drawings can be achieved was done using OPM (Dori, 1994). This provided the basis for the Machine Drawing Understanding System, MDUS (Dori & Wenyin 1999). Dori (1995A) specified MDUS as a case in point for representing pattern recognition embedded systems. Within MDUS, Object-Process Methodology was used to express structural relations among dimensionset components (Dori 1996). Dori, Velkovitch, and Wenyin (1996) applied OPM for segmentation and recognition of ANSI- and ISO-standard dimensioning texts. Wenyin and Dori (1998A) showed how graphics recognition algorithms can be modeled with common features based on OPM principles. Based on this approach, they specified with OPM a generic integrated line detection algorithm (GILDA). GILDA (Wenyin & Dori, 1998; 1999) is based on the generic graphics recognition approach, which abstracts the graphics recognition as a stepwise recovery of the multiple components of the graphic objects and is specified by the object-process methodology. Using an object-process diagram, they defined 12 classes of lines which appear in engineering drawings and used them to construct a class inheritance hierarchy. The hierarchy highly abstracts the line features that are relevant to the line detection process. Based on the "Hypothesis and Test" paradigm, lines are detected by a stepwise extension to both ends of a selected first key component. In each extension cycle, one new component which best meets the current line's shape and style constraints is appended to the line. Different line classes are detected by controlling the line attribute values. As they show in experiments, the algorithm demonstrates high performance on clear synthetic drawings as well as on noisy, complex, real-world drawings. Analysis and representation of the Image Understanding Environment was carried out using Object-Process Methodology (Dori 1996). Dori and Hel-Or (1998) carried out semantic content-based image retrieval using object-process diagrams. Dori, Doermann, Shin, Haralick, Phillips, Buckman, and Ross (1997) used Object-Process Methodology to devise a detailed and accurate conceptual representation of document structure. Through a set of OPDs, they showed how the logical layout of a document can be faithfully and effectively modeled. Development and representation of document analysis systems, as well as syntactic and semantic graphics recognition were done with Object-Process Methodology (Dori 1999; 2000).
577 4.2
An OPM model of a generic image processing system
We conclude the chapter with an example of an OPM model of a small part of a generic image processing system. The objective of this example is not to provide a complete coverage of all the system's details and ramifications, but rather to exemplify how OPMbased conceptual modeling can help understand the intricate relationships among objects and processes in the system. Figure 13 shows The Image Processing system model after the object Image and the process Processing were linked. As the OPL paragraph indicates,
Processing changes Image from raw to processed. This is the fundamental function of the image processing system, and as we shall see, each process is some specialization of this generic pattern of transformation. In Figure 14, we added the attribute Encoding of Image with its three values, which gave rise to the two OPL sentences: Image exhibits Encoding. Encoding can be binary, grey level, or color. Figure 14 also shows the specializations of the process Processing, the Encoding attribute of Image with its three values, and the instrument link between each of these values and the corresponding Processing specializations.
Figure 13. The Image Processing system model after the object Image and the process Processing were linked In Figure 15 the attribute Purpose with its two values edge detection and enhancement has been added to the process Processing such that each Purpose value
578
is an instrument to another type of Processing. Finally, the process Color Edge Detecting in Figure 16 is a result of multiple inheritance from Edge Detecting and Color Processing, and it yields Color Edge Image, a specialization of Image. The process Grey Level Enhancing is a result of multiple inheritance from Image Enhancing and Grey Level Processing, and it yields Grey Level Enhanced Image, another specialization of Image.
Image
A binary
yrey level]
co\o
31 Image can be raw or processed. raw is initial. Image exhibits Encoding. Encoding can be binary, grey level, or color. Processing changes Image from raw to processed. Binary Processing is Processing. Binary Processing requires binary Encoding. Grey Level Processing is Processing. Grey Level Processing requires grey level Encoding. Color Processing is Processing. Color Processing requires color Encoding.
Figure 14. The Image Processing system showing the specializations of the process Processing, the Encoding attribute of Image with its three values, and the instrument link between each of these values and the corresponding Processing specializations
5.
Summary
This chapter has presented an overview of Object-Process Methodology and its applications in a variety of domains with emphasis on image processing and pattern recognition.
579 The domain-independent nature of OPM makes it suitable as a general, comprehensive and multidisciplinary framework for conceptual analysis, design, implementation, and lifecycle support of systems of all kinds and complexity levels. In particular, the example at the end of the chapter hints to the viability and value of using OPM for modeling and systems development in image analysis and pattern recognition, where processes and objects are so intertwined and inseparable. Indeed, it is hard to think of a domain in which structure and behavior are so interdependent. Due to its single model, expressed in both graphics and text, OPM lends itself naturally for such domains, as it is uniquely poised to cater to the tight interconnections between structure and behavior that are so hard to separate.
Figure 15. The attribute Purpose with its two values edge detection and enhancement has been added to the process Processing such that each Purpose value is an instrument to another type of Processing Developers and researchers in image processing and pattern recognition are sure to benefit from unexpected insights and system thinking approach if they adopt the holistic OPM approach in their research and development efforts.
580
Figure 16. The process Color Edge Detecting is a result of multiple inheritance from Edge Detecting and Color Processing, and it yields Color Edge Image, a specialization of Image. The process Grey Level Enhancing is a result of multiple inheritance from Image Enhancing and Grey Level Processing, and it yields Grey Level Enhanced Image, another specialization of Image
References 1. Dov Dori (1994). Automated Understanding of Engineering Drawings: an ObjectOriented Analysis. Journal of Object Oriented Programming, pp. 35-43, Sept. 2. Dov Dori (1995). Object-Process Analysis: Maintaining the Balance between System Structure and Behavior. Journal of Logic and Computation, 5, 2, pp. 227-249. 3.
Dov Dori (1995A). Representing Pattern Recognition-Embedded Systems through Object-Process Diagrams: the Case of the Machine Drawing Understanding System. Pattern Recognition Letters, 16, 4, pp. 377-384.
4. Dov Dori (1996). Expressing Structural Relations among Dimension-set Components Using the Object-Process Methodology. Report on Object Analysis and Design, 2, 6, pp. 20-24. 5. Dov Dori (1996A). Object-Process Analysis of Computer Integrated Manufacturing Documentation and Inspection. International Journal of Computer Integrated Manufacturing, 9, 5, pp. 339-353.
581
6. Dov Dori, Document Analysis Systems Development and Representation through the Object-Process Methodology. Document Analysis Systems, S.W. Lee and Y. Nakano (Eds.), Lecture Notes in Computer Science, 1655, Springer, pp. 271-282 1999. 7. Dov Dori, Syntactic and Semantic Graphics Recognition: The Role of the ObjectProcess Methodology, Lecture Notes in Computer Science, 1941, Springer, p. 277, 2000. 8. Dov Dori (2001). Object-Process Methodology Applied to Modeling Credit Card Transactions. Journal of Database Management, 12, 1, pp. 2-12. 9. Dov Dori (2002). Object-Process Methodology - A Holistic Systems Paradigm, Springer Verlag, Berlin, Heidelberg, New York. 10. Dov Dori and Edward Crawley (2003). Towards a Common Computational Synthesis Framework with Object-Process Methodology. 2003 AAAI Spring Symposium Series: Computational Synthesis: From Basic Building Blocks to High Level Functionality, Stanford University, Stanford, CA, Technical Report SS-03-02. 11. Dov Dori, David Doermann, Cristian Shin, Robert Haralick, Ihsin Phillips, Mitchell Buckman, and David Ross (1997). The Representation of Document Structure: a Object-Process Analysis. In Handbook of Character Recognition and Document Image Analysis, H. Bunke and P.S.P. Wang (Eds.), World Scientific Publishing Company, pp. 421-456. 12. Dov Dori, Avigdor Gal and Opher Etzion (1996). A Temporal Database with Data Dependencies: a Key to Computer Integrated Manufacturing. International Journal of Computer Integrated Manufacturing, 9, 2, pp. 89-104. 13. Dov Dori and Hagit Hel-Or (1998). Semantic Content-Based Image Retrieval Using Object-Process Diagrams. In Advances in Pattern Recognition, A. Amin, D. Dori, P. Pudil and H. Freeman (Eds.), Lecture Notes in Computer Science, 1451, pp. 230241. 14. Dov Dori, Yelena Velkovitch, and Liu Wenyin (1996). Object-Process Based Segmentation and Recognition of ANSI and ISO Standard Dimensioning Texts. In Graphics Recognition: Methods and Application. Rangachar Kasturi and Karl Tombre (Eds.), Lecture Notes in Computer Science, 1072, pp. 212-232. 15. Dov Dori and Yehudit J. Dori (1996). Object-Process Analysis of a Hypertext Organic Chemistry Module. Journal of Computers in Mathematics and Science Teaching, 15(1/2), pp. 65-84. 16. Dov Dori and Liu Wenyin (1999). Automated CAD Conversion with the Machine Drawing Understanding System: Concepts, Algorithms, and Performance. IEEE Transactions on Systems, Man, and Cybernetics, 29, 4, pp.411-416. 17. Doron Myersdorf and Dov Dori (1997). The R&D Universe and Its Feedback Cycles: an Object-Process Analysis. R&D Management, 27, 4, pp. 333-344. 18. Mor Peleg and Dov Dori (2000). The Model Multiplicity Problem: Experimenting with Real-Time Specification Methods. IEEE Transaction on Software Engineering, 26, 8, pp. 742-759.
582
19. Iris Reinhartz-Berger, Dov Dori, and Shmuel Katz, OPM/Web - Object-Process Methodology for Developing Web Applications. Annals of Software Engineering. 13, pp. 141-161,2002. 20. Iris Reinhartz-Berger, Dov Dori, and Shmuel Katz (2002). Open Reuse of Component Designs in OPM/Web. Proc. IEEE 26th Annual International Computer Software and Applications Conference, pp. 19-26. 21. Nathan Soderborg, Edward Crawley and Dov Dori (2002). System Definition For Axiomatic Design Aided by Object-Process Methodology. Proc. 2nd International Conference on Axiomatic Design, Cambridge, MA, USA, pp. 134-140. 22. Nathan Soderborg, Edward Crawley, and Dov Dori (2003). OPM-Based System Function and Architecture: Definitions and Operational Templates. Communications of the ACM, 46(10), pp. 67-72. 23. Pnina Soffer, Boaz Golany and Dov Dori (2003). ERP Modeling: A Comprehensive Approach. Information Systems 28, 6, pp. 673-690. 24. Pnina Soffer, Boaz Golany, Dov Dori and Yair Wand (2001). Modeling Off-theShelf Information Systems Requirements: An Ontological Approach. Requirements Engineering, 6, pp. 183-199. 25. Liu Wenyin and Dov Dori (1998). A Generic Integrated Line Detection Algorithm and its Object-Process Specification. Computer Vision - Image Understanding (CVIU), 70, 3, pp. 420-437. 26. Liu Wenyin and Dov Dori (1998A). Genericity in Graphics Recognition Algorithms. In Graphics Recognition - Algorithms and Systems, Lecture Notes in Computer Science, K. Tombre and A. K. Chhabra (Eds.), Vol. 1389, pp. 9-18. 27. Liu Wenyin and Dov Dori (1999). Object-Process Based Graphics Recognition Class Library: Principles and Applications. Software - Practice and Experience, 29, 15, pp. 1355-1378.
C H A P T E R 5.5 MUSICAL STYLE RECOGNITION —A APPROACH
QUANTITATIVE
Peter van Kranenburg E-mail: [email protected] http://www. lodebar. nl/pvk
Eric Backer E-mail: [email protected] http://www-it.et.tudelft.nl/~backer
In this chapter a study is described in which the possibilities of statistical pattern recognition for musical style recognition are explored. A dataset with compositions of Bach, Handel, Telemann, Mozart and Haydn is investigated. The used featureset consists mainly of features that describe the different sonorities in the compositions. It is shown that with these features it is possible to separate the styles of the various composers. For this a fc-nearest neighbor classifier is used which is trained in a featurespace that is spanned by the fisher-discriminants. In the untransformed featurespace, clusters are found with the fc-means algorithm. It appears that these clusters reflect the styles of the various composers. The pattern recognition tools are also used to learn something about the characteristics of the styles. For this, decisiontrees are built using the C4.5 algorithm. It turns out that each of the styles represented in the dataset can be described with a few features.
1. I n t r o d u c t i o n W h e n listening to the radio, it can happen t h a t a piece of music is broadcasted of which a listener immediately can say: "This must be made by Bach", or "That sounds like the Beatles". This recognition indicates t h a t something in the structure of this music bears the "fingerprint" of its maker. Knowing this, we can ask ourselves if it is possible to develop algorithms with which this kind of attributions can be made automatically. T h e h u m a n p a t t e r n recognition system is very complex, and not yet fully understood. B u t in the past decades many machine learning algorithms have been developed to let computers perform a similar task. In this chapter an experiment is described in which is tried to train a machine to identify the composer of a musical composition. T h e values of a set of features (also called "style markers") are measured in the scores of a set of compositions. Of these 583
584
compositions the composer is known. This is because we need to know the "truth" to determine the error rate of the classifiers. For now, our aim is both to explore the possibilities of machine learning techniques for this kind of attributions and to learn something about the characteristics of the styles of the composers whose compositions are investigated. 2. Non-traditional Authorship Attribution and Stylometry Because this is a relatively new area of research, it is desirable to find a starting point. Fortunately an area of research exists in which very similar problems are investigated: authorship attribution. Of many historical texts the author is not known, or the attribution made by the source of the text is not trusted. This has caused a lot of research during the entire history of written text. As early as in the fourth century BC, librarians of the famous library of Alexandria studied the "authenticity" of texts attributed to Homer. 1 A project that nowadays, after more than 2000 years, is still not finished. Some other well known examples are the attributions made in the Bible, the plays of Shakespeare and the Federalist Papers. In the past decades, the ever increasing power of computers made it possible to execute algorithms that demand a lot of computations. The possibilities of this are used in authorship attribution too, which resulted in a research area called nontraditional authorship attribution. In this kind of research, it is tried to quantize the style of a certain author. The study of this is called stylometry. It is not obvious what exactly has to be quantized. Many so called style markers are developed. Examples of these are word and sentence lengths, frequencies of certain words and richness of vocabulary. The resulting data is used to classify the texts. Although no one widely accepted methodology exists in non-traditional authorship attribution, 2 ' 3 the results achieved so far, indicate a reasonable probability of success when applying this to music. The main problem of stylometry is the lack of an underlying theory.2 Many style markers turn out to be distinctive, but often it is not clear why. In the words of Joseph Rudman: "Authorship studies must not fall into the trap of discarding style markers from their stylistic autoradiogram because they didn't work in some other study. Until the study is done, it is not known which style markers will be the discriminators."4 There are, however, some intuitive guidelines which can help with the search for proper style markers. E.g. they have to be independent of the subject written about and of the genre of the text. When counting the number of occurrences of the word "bee" in a text about insects and also in a text about the stock market from the same author, the results will differ, which is not what we want. So, it is better to count context independent words, like "for" and "if". It is also preferable to use style markers that describe the unconscious processes of text writing or, in our case, composing. While this kind of style markers are independent of decisions of the author when writing the text, these are expected to have the same values for different texts.
585
Another problem is the concept "author" itself. In many cases more than one person was involved in the process of creating the text. Editors, for example, can have a profound influence on various characteristics of the text. And, above all, no author works in a vacuum. The style of a certain author is also determined by his environment. This made it possible for us to divide the history into different style periods, in which most authors share certain characteristics. So, the question of authorship is not a simple one. 3. Music When trying to attribute a piece of music to a certain composer using stylistic evidence, the most important place to look is into the score. This entity is the most direct connection to the activity with the composer. Most of the issues that arise when studying authorship of texts, are also of importance when studying authorship of music. One of the most important issues is the design of an appropriate set of features c.q. style markers. Because there exists no generalized theory for determining which style marker will be successful, we will evaluate some that can be expected to be distinctive. This is the only possible thing we can do for now. A method for (automatically) obtaining style markers is desirable, but this is beyond the scope of the present work. Some studies exist in which the same problem is encountered, but in most of the cases no rationale is given for the choice of style markers. It is the intuition of the investigator that determines them. In a study aimed at interactive improvisation with a computer, Roger Dannenberg et al. used machine learning tools to recognize the "mood" of music, such as "lyrical", "frantic" etc. 5 It is shown that it is very well possible to recognize these moods using machine learning tools. They obtained very low error-rates. Unfortunately they don't mention all the features that were used. Pedro Ponce de Leon and Jose Ifiesta used a fc-nearest neighbor classifier, a Bayesian classifier and a self-organizing neural map to classify musical styles.6 Their dataset consists of fragments of jazz and classical music. They give an overview of the features used for this, which involves basic properties of the melodies they've extracted from the fragments, such as the number of notes, pitch range, etc. With all three classification methods error rates between 10% and 20% could be obtained. An important feature for separating classical and jazz fragments is the number of syncopations. This is an illustration of the remark of Rudman. It is not very surprising that a style marker like this is important for this particular problem. But for other styles this style marker may be less important. Similar quantitative work is also done "by hand". Fred Hofstetter used the Chisquare test to determine whether the difference in interval-characteristics of Easternand Western-Europe nineteenth-century string quartets is significant or not. 7 With this he discovered that Eastern melodies are characterized by more leaps, and that in Eastern melodies relatively more changes of direction can be found.
586
4. Some Preliminaries For a study like this it is necessary to have the music available in a machine readable format. Many different encodings exist. Each with its own possibilities and limitations. The most widely used representation is MIDI. But MIDI is a performancerepresentation and not a representation of the score. In MIDI the on- and off-times of pitches are stored. Because a human being is not very accurate in his timing when playing a piece of music (and in fact, he should not be), some quantization has to be done before a MIDI-representation can be used for our purpose. When a MIDIrepresentation is encoded by hand, this drawback is overcome. But, another even more important problem with MIDI is the lack of precision in indicating pitches. An equal temperature is supposed, so there is no difference between e.g. a F sharp and a G flat, because they have the same pitch number. Since this affects the sizes of most intervals (e.g. an augmented fourth cannot be distincted from a diminished fifth), determination of harmonic characteristics becomes impossible, which is not very useful for our purpose. Some extensions of the MIDI-standard exist in which this limitation is overcome, but these are mutual incompatible and not widely used. Having a machine readable representation of music is one thing. But then this representation has to be parsed and the contents of the music has to be made available for the routines that will compute the feature values. For this Donncha O Maidi'n (university of Limerick) very kindly made his C++-library CPNView available. This library consists of a number of routines to access the score of a musical composition. CPNView can read the *kern-representation developed by David Huron. 8 This is a representation often used for academic research, and a large corpus of encoded music is available from the Center for Computer Assisted Research in the Humanities at Stanford University (CCARH). 9 From this collection a number of compositions is drawn to construct the dataset that is used in the present study. For the execution of the machine learning algorithms PRTools is used. 10 This is a Matlab toolbox, created by Bob Duin, in which many machine learning algorithms are implemented. 5. Repertoire The collection of encoded music at the CCARH consists almost entirely of music from the eighteenth and early nineteenth centuries. Not all of this is suited for our purpose. Many movements from cantatas, oratorios and operas have a basso continuo which is not completely written out. So, some harmonic characteristics cannot be determined. These movements are only used when more than two other voices are active most of the time. In order to reduce the variance in the computed feature values, it is also important not to include too short compositions. After examining the behavior of the feature values a minimum of 30 bars is taken. Another issue is the presence of transposing instruments. Sometimes several parts had to be transposed. Apart from this, many files needed some adaptations before they could
587 be parsed by CPNView. With these limitations in mind, a number of compositions is chosen from the CCARH library. The resulting dataset consists of the following groups of pieces:3, • • • • • • •
J.S. Bach: 40 cantata movements. J.S. Bach: 33 fugues from "Das Wohltemperierte Clavier". J.S. Bach: 11 movements from the "Kunst der Fuge". J.S. Bach: 8 movements from the violin concertos. G.F. Handel: 39 movements from the Concerti Grossi, op. 6. G.F. Handel: 14 movements from trio sonatas, op. 2 and op. 5. G.Ph. Telemann: 30 movements from the "Fortsetzung des Harmonischen Gottestdienstes". • G.Ph. Telemann: 24 movements from the "Musique de table". • F.J. Haydn: 54 movements from the string quartets. • W.A. Mozart: 53 movements from the string quartets.
Of the three baroque composers, works in different genres are added. Orchestral works as well as compositions for small instrumentation, and, in the case of J.S. Bach, works for keyboard. Of Mozart and Haydn only string quartets are added. From this dataset, several smaller datasets can be made. The following subsets are investigated (The classes are indicated with braces): I {Bach}, {Telemann}, {Handel}, {Haydn}, {Mozart}. II {Bach}, {Telemann}, {Handel}. I I I {Bach}, {Telemann, Handel}. I V {Bach}, {Telemann, Handel, Haydn, Mozart}. V {Telemann}, {Handel}. V I {Mozart}, {Haydn}. V I I {Telemann, Handel}, {Mozart, Haydn}. So the main point of interest is the difference between the style of J.S. Bach and the other composers. But it is also interesting to try to distinguish between composers whose style is very likely. Especially the set with Haydn and Mozart will be challenging, since only compositions of the same genre are included. 6. F e a t u r e s For each composition in the dataset, the values of 20 features are computed. Most of these features are low-level properties of counterpoint. When composing polyphonic music, the composer must control the distances between the voices. The way he is doing this, can be expected to be consistent for compositions in different genres and of different dates. The conscious decisions are on higher level. These are e.g. the a
I t is not possible to include a complete list of all compositions here. This list can be obtained from the authors.
588
key, modulations, the development of a theme, the use of certain motifs, etc. These kind of features are expected to be less suitable for our purpose. The following features are computed: Stab
Timeslice
The "stability" of the length of the successive timeslices. With a timeslice, the time interval between two changes in the music is ment. This is shown in fig. 1. The stability is computed by dividing the standard deviation of the lengths of the timeslices by the mean length of the timeslices. This normalization is necessary to compare pieces with different time signatures. So, when having a low value, the music is more like a steady stream, while a larger value indicates more diversity in rhythm.
I l l
Fig. 1.
I I
Boundaries of the timeslices
DissPart The fraction of the score that consists of dissonant sonorities. Consonants are: perfect primes, minor and major thirds, perfect fourths and fifths and minor and major sixths. But a fourth is considered dissonant if it is between the lowest voice and one of the upper voices. All other intervals are considered dissonant. The total duration of dissonant sonorities is divided by the total duration of the composition. BeginBarDiss The fraction of bars that begins with a dissonant sonority. Sonority
Entropy
For this feature, the concept "sonority" is used according to the definition of Robert Mason. 11 In this definition a sonority is a certain type of chord. So e.g. all major triads are the same sonority, regardless of inversion or pitch. Each sonority has an unique number. For each sonority the total duration of all occurrences is computed. Then the probabilities of occurrence are estimated using this weighted frequencies. With this probabilities the entropy is computed according to: N
-^2,Pilog{pi) i=l
589 where N is the total number of sonorities and p, the probability of occurrence of sonority i.
HarmonyEntropy The concept "Harmony" is also defined by Mason. It is much like sonority, but now difference is made in pitch. So e.g. a F-major triad and a G-major triad are the same sonority but different harmonies. Again chord inversion is not taken into account. The value of this feature is computed the same way as the SonorityEntropy.
PitchEntropy A list of occurrences of all pitches is made. Again the occurrences are weighted by the durations. Of the resulting list, the entropy is computed.
VoiceDensity In a polyphonic composition not all voices are active during the whole composition. The voicedensity indicates the average number of active voices. This is normalized with the total number of voices. For this feature only bars that are strictly polyphonic are taken into account. A bar is strictly polyphonic if in that bar no voice has more than one note at a time and if more than one voice is active.
PartSeconds, PartThirds, PartFourths, Part Aug Fourths, PartDimFifths, PartSixths, PartSevenths, PartOctave When combining the different voices of a polyphonic composition, the composer has to obey certain constraints. In many of these constraints the vertical distances between the voices are important. This set of features measures the amount of certain of intervals between the different voice-pairs. Systematically all voice-pairs are examined. The total duration of all occurrences of each specific interval is computed and at the end divided by the total duration of all intervals in all voice-pairs. The intervals are taken modulo one octave. So e.g. a tenth is a third. When the same pitch occurs in more than one voice, it is taken into account once.
ParThirds,
ParFourths,
ParSixths
It can happen that in a voice pair two intervals of the same size succeed each other. This is called a parallel. For this three features the amount of parallel thirds, fourths and sixths is computed in the same way as the previous group of features. The total duration of all intervals involved in these parallels is divided by the total duration of all intervals in all voice pairs.
590 Step
Suspension
When a dissonant is sounding between two voices, it often is suspended into a consonant by lowering the lower voice one step. This feature indicates how many dissonances are suspended this way. It is computed in the same way as the previous features. In the following sections these features are referred to by their index numbers. These can be found in table 1. Table 1.
The featureset
index
feature
index
feature
1 2 3 4 5 6 7 8 9 10
StabTimeslice DissPart BeginBarDiss Sonority Entropy Harmony Entropy PitchEntropy VoiceDensity PartSeconds PartThirds Part Fourths
11 12 13 14 15 16 17 18 19 20
PartAugFourths PartDimFifths PartFifths PartSixths PartSevenths Part Octaves ParThirds ParFourths ParSixths StepSuspension
7. Analysis 7.1. Density
estimation
In our dataset, the number of objects per class is far too less to determine whether the densities are Gaussian or not, so we have to be careful with algorithms and models that suppose this kind of densities. It is better not to use methods that suppose any density at all. So we will use a fc-nearest neighbor classifier, the C4.5 algorithm to build decisiontrees and the fc-means clustering-algorithm. For the fc-nearest neighbor classifier and for the fc-means algorithm, normalized datasets are used. For each feature, the mean is shifted to the origin and all values are divided by the standard deviation. This makes the scales of the features more comparable. 7.2. Feature
reduction
It is expected that not all 20 features are necessary for making correct classifications. So, we have to search for a subset of features that contains enough information to separate the classes. In order to reduce the dimensionality of the data, several algorithms are available. These algorithms try to optimize a certain criterion in one way or another. Two fundamental different approaches exist. One can try to select an optimal set of features (feature selection) or one can compute new features using all
591 the available ones (feature extraction). The second approach has the disadvantage that in most cases the meaning of the resulting new features is not easy to understand. So it can be used to build classifiers, but not to understand what separates classes. 7.2.1. Selection Evaluating every possible subset is very time consuming. So a clever algorithm should be used in order to obtain a solution in reasonable time. For obtaining an optimal solution an algorithm called branch and bound can be used. 12 Since the worst case complexity of this algorithm is exponential, it is not suitable for large datasets. So, a range of sub-optimal alternatives has been proposed. One of these is floating forward selection (FFS), proposed by Pudil et al.13 It has been shown by Jain and Zongker14 that this algorithm is performing nearly as well as branch and bound and outperforms other algorithms on a 20-dimensional two-class dataset with Gaussian densities. This floating selection algorithm is applied to all proposed datasets. The implementation of PRTools is used. The criterion to be optimized is the sum of (estimated) mahalanobis distances between the classes. With this, the high correlation between some of the features is taken into account, which is not the case when we use euclidean distances. For all datasets, subsets of all sizes are selected. This results in 20 feature subsets per dataset. In order to get an impression of the distributions of the "best" features with respect to the isolation of the style of Bach, parzen-estimations of the densities of the first six selected features for dataset IV are shown in fig. 2. Another way to get an indication of which features are important for splitting the data into classes is to build decision trees with the C4.5 algorithm. C4.5 uses the criterion of information gain for selecting features. 15 The result of this can be very insightful. 7.2.2. Extraction In order to obtain good classification results, one can search in the featurespace for directions in which the separation of the classes is highest. A method to do this is called linear discriminant analysis (LDA). For a two-class problem, the datavectors Xj are projected onto a line: j/j = w T Xj such that the following criterion 12 is maximized: J F ( w ) = | w T ( mT i - m 2 ) | 2 w Sww in which Sw is the within-class scattermatrix and m i and ni2 are the class means. This criterion is called Fisher's criterion. It has its maximum value when the projected means are as far apart as possible and the total within-scatter of the
592
0.1
0.2
0.3
0.4
HarmonyEntropy
0 Fig. 2.
0.1
0.2
0.3
Some densities of the features in dataset IV: Bach (solid) and not-Bach (dashed)
projected classes is as low as possible. These "discriminants" are more suitable for classification than principal components. Directions in which the covariance is large, are not necessarily directions in which the class-separability is high. In PRTools a multiclass version of the Fisher-analysis is implemented, This is applied to all datasets. 7.3. Unsupervised
classification
With unsupervised learning, the classlabels are not taken into account. The aim of this is to search coherent structures in the data. Of course, we hope that the structures found will show some similarities with the classes. For all datasets the k -means clustering-algorithm is executed. A number of so called prototypes are placed in the featurespace. The number of prototypes equals the number of desired clusters. All objects are attributed to the nearest prototype. Then the prototypes are replaced to the center of these groups and the process is repeated. The algorithm ends when the prototypes remain at the same positions. This is done with all selected featuresets. The algorithm is initiated with a number of prototypes that equals the number of classes in the dataset. These prototypes are placed at the known class means. Then with each selected featuresubset the clustering-algorithm is executed. After finishing the clustering, a confusionmatrix is built, which reflects the confusion between the resulting clusters and the classes. So, for each dataset we obtain 20 confusionmatrices. Now, we can see which features
593
lead to the lowest amount of confusion. The resulting featuresets and the error made are shown in table 2. Table 2.
Optimal featuresets for clustering
Dataset
optimal feature subset
number of misclassified objects
I II III IV V VI VII
all, except 4 2, 14, 17, 10, 1 2, 1, 17, 13, 4 1, 17, 2 10, 14, 4 13 13, 18
108 (35.3%) 28 (14.1%) 13 (6.3%) 39 (12.3%) 11 (10.3%) 37 (34.6%) 42 (19,6%)
Except for the first dataset, it appears that all classes can be found with just a few features. For an unsupervised method, this is quite successful. Especially when we realize that not all the datasets are symmetric with respect to the numbers of objects in the classes. With overlapping classes, it can be expected that clusters are found with roughly the same number of objects in it. But these results indicate that the various classes really do form coherent structures. Of course, this result is stimulated by placing the prototypes at the "right" places. But when placing them randomly, this could be one of the resulting situations. And, when the clusters wouldn't correspond to the classes, the outcome would be not so nice. It is obvious that separating the string quartets of Mozart and Haydn is the most difficult task. Because the errors on datasets II and VII are relatively low, the large error for dataset I is also caused by this. So within the baroque group and between the classicism and baroque groups separation is very well possible, and within the classicism group, separation is more difficult. This is not surprising because the styles of Mozart and Haydn are very closely related. 7.4. Supervised
classification
With supervised classification the classlabels of the objects are taken into account in the learning phase. Two classifiers are used: a fc-nearest neighbor classifier and a decisiontree (C4.5). The decisiontree is mainly used to get more insight in the available data. The leave-one-out errors of the constructed decisiontrees, are roughly twice as high as the leave-out-out errors of the nearest-neighbor classifiers. So, for more reliable classification, it is better not to use these decisiontrees. 7.4.1. nearest neighbor classifier For the feature-subsets of sizes de{l,2,... ,20}, selected by the floating feature selection algorithm, the leave-one-out error is computed for /c-nearest neighbor classifiers, for each k e { 1 , 3 , 5 , . . . , 53}. The maximum of 53 is because four of the five classes don't contain more than 53 objects.
594 In order to obtain a reliable estimation of the true error of the classifier and to avoid overfitting, train- and testsets have to be made for training and testing the classifiers. The size of these sets has to be chosen in such a way that a reliable indication of the true error of the classifier is obtained. Testing using hold out validation will give a very pessimistic estimate of the true error, because not all objects will be used for learning. Therefore it is better to use more advanced methods to test the classifiers. Since we do not have the luxury of having much data, leave-one-out validation is preferable. One object is isolated. On all other objects a classifier is trained. Then the one isolated object is used to test that classifier. This is done for each object in the dataset. The resulting error rate is the total number of misclassifications divided by the total number of objects in the dataset. This procedure reduces the bias in the estimation of the true error, but increases the variance. But, since our datasets are not very small, the effect of the increased variance will not dominate the results. The smallest datasets (V and VI) have 107 objects each. One more error will result in a change of the leave-one-out error of less then 1% (1/107). For the other datasets this is even lower. The parameters which result in minimal errors are shown in table 3.
Table 3.
minimal errors with nearest neighbor classifiers
Dataset
k
feature subset
loo-error
I
11
2, 13, 8, 9, 1, 17, 5, 10, 14, 19, 11, 6, 7, 20
0.2647
II II
31 13
2, 14, 17, 10, 1,1 9, 13 2, 14, 17, 10, 1, 19, 13, 6
0.0854 0.0955
III III
17 7
2, 1, 17 2, 1, 17
0.0481 0.0529
IV
5
1, 17, 2
0.0662 0.0841
V
7
10, 14, 19, 8, 12, 11, 3, 20, 7, 4
VI
3
7, 11, 13, 1, 6
0.2430
VII VII
3 11
13, 2, 8, 16, 19, 5, 7, 11, 15, 20, 3, 17, 10, 18 13, 2, 8, 16, 19, 5, 7, 11
0.0841 0.0888
When looking to the error-rates, we can see the same pattern as with the results of the A;-means algorithm. Datasets I and VI are the most difficult to classify due to the presence of the compositions of Haydn and Mozart. With Ockham's razor in mind, it is preferable to look for the most simple models. So, in table 3 also some second best options are shown. Especially the 31 neighbors needed for dataset II is large compared to the number of objects per class. When adding one feature, eleven are needed, while the error is increasing with only one percent. For dataset I, no simpler model with comparable error-rate can be found.
595 7.4.2. decisiontree For each of the datasets a decisiontree is built. Since the algorithm is not deterministic, the best of 50 trials is chosen. I.e. the one with the lowest apparent error. These trees give insight in the relevance of the various features. From the tree that is built for dataset I, we can learn that all compositions of Telemann, and 40 of Handel are in the subspace (DissPart < 0.358065) ("1 (PartAugFourths < 0.0239669). In the same subspace 8 compositions of Haydn, 17 of Mozart and 5 of Bach can be found. So, it can be stated that the styles of Handel and Telemann are characterized by consonant sonorities and a low amount of augmented fourths. 73 of the 92 compositions of Bach are in the subspace (DissPart > 0.358065) fl (ParThirds < 0.169492) n (StabTimeslice < 0.414224). Together with 4 compositions of Handel, 2 of Mozart and 2 of Haydn. A scatterplot of this subspace is shown in fig. 3. From this observation, we can conclude that Bach wrote more dissonant sonorities than his contemporaries. The DissPart-feature separates Handel and Telemann from Bach, but, it doesn't separate Bach from Haydn and Mozart. For this, other features are needed. The parallel third is a device to enrich the sonorities, but does not add very much to the musical contents of a composition. The amount of parallel thirds is low in Bach's music. Probably he wouldn't be satisfied by a lot of parallel thirds. The rhythmic "stability" is also a characteristic of Bach's music, which in many cases is like a steady stream. This is especially the case in the polyphonic compositions, which are taken into account here. The combination of these three characteristics leads to the style of Bach.
I (DissPart > 0.358065) 0.25
0.2
£
0.15
Q.
i*
0.1 +
JVoi
+ ++
+H
0.05 0.2
0.4 0.6 StabTimeslice
0.8
1
Fig. 3. Scatterplot with the features that characterize Bach's style ( + ) . Compositions with DissPart<0.358065 are not shown
The compositions of Haydn and Mozart are scattered all over the tree. There is
596
no subspace which is clearly the domain of these two composers. In order to learn more about the styles of these composers, it is better to examine the tree built from dataset VI. Unfortunately this tree is rather deep, which is coherent with our observation that it is difficult to separate the compositions of these two composers. Making a highly pruned tree is more insightful. This is done by setting the maximum number of objects at a leaf to eight. In the resulting tree, the voicedensity is the most important feature. Parzen estimations of the densities are shown in fig. 4. Roughly spoken, in Mozart's string quartets voices are more active than in Haydn's. But, as can be seen, there is much overlap. In the subspace (VoiceDensity < 0.75251) 32 movements of Haydn and 11 of Mozart can be found, while the complementing subspace is occupied by 22 movements of Haydn and 42 of Mozart. 15
10
5
0.5 Fig. 4.
0~6i
" 6.7
0.8
"b.9
1
Parzen estimations of the voicedensity for Haydn (dashed) and Mozart (solid)
To learn what separates the styles of Handel and Telemann we can inspect the tree made from dataset V. From this tree it is clear that the amount of fourths and sixths are the most discriminative features. 39 of the 53 compositions of Handel are in the subspace (PartFourths > 0.0755208) n {PartSixths > 0.15229), while most of Telemann's compositions are in {(PartFourths < 0.0755208) n (PartSixths < 0.201381)} U {(PartFourths > 0.0755208) n (PartSixths < 0.15229)}. So Telemann's compositions are characterized by a low amount of fourths and sixths, and it seems that when the amount of fourths is higher, the amount of sixths tends to be lower. This is not a correct observation. When making a scatterplot (fig. 5), we can see why the tree is made this way. 7.5. LDA using
Fisher-criterion
After finding the fisher "disciminant(s)", the leave-one-out errors of fc-nearest neighbor classifiers for ke {1, 3, 5 , . . . , 53} are computed, with the projected data. The
597
0.08
0.1
0.12
0.14 0.16 PartSixths
0.18
0.2
0.22
Fig. 5. scatterplot of the features PartFourths and PartSixths of classes Handel (+) and Telemann (*), with the discriminant of the part of the decisiontree that isolates most compositions of Telemann
optimal results are shown in table 4. Also the fc-means algorithm is executed for the transformed datasets. This is done in the same way as with the "real" features. The results of this are also shown in table 4. Table 4.
errors for datasets with fisher-discriminants as features
Dataset
k
loo-error (nn)
misclassified objects (fc-means)
I II III IV V VI VII VII
15 17 47 15 9 7 31 11
0.1993 0.0704 0.0385 0.0599 0.0841 0.2056 0.0654 0.0701
68 (22.2%) 19 (9.5%) 9 (4.3%) 27 (8.5%) 9 (8.4%) 23 (21.5%) 15 (7.0%)
—
When comparing these results to the results on the original dataset, one can see that this transformation reduces the errors. So in order to build good classifiers, this transformation is very useful. But for more insight in the data, this transformation is not very helpful. For dataset III as much as 47 neighbors are needed for the lowest error. It appears that three compositions of Bach are clearly mapped in the region of Handel and Telemann. These are BWV 4.5, BWV 6.3 and BWV 550. Although BWV 6.3 is a piece with 3 voices, most of the time only 2 voices are active. The third voice has a chorale melody, but not all the time. This can influence some feature values, since two-voice counterpoint is slightly different from counterpoint with three or more
598
voices. BWV 4.5 and BWV 550 are youth-works of Bach. When removing these three compositions from the dataset, the leave-one-out error of a 1-nearest neighbor classifier is lowest with 2.4%. So, even when these compositions are present, it is better to use a 1-nearest neighbor classifier. For dataset VII the second best classifier is a 11 nearest neighbor classifier. This only affects the leave-one-out error with less than 0.5%. Dataset VII consists of 214 objects, so an increase in the leave-one-out error of 0.5% means that one more object is misclassified. 8. Conclusions The obtained results indicate clearly that it is possible to recognize musical style automatically. So, this kind of research can be a valuable addition to more traditional methods of musical style analysis. It offers a quantitative evaluation of the styles rather than the traditional qualitative descriptions. It is important not to see this as a replacement, but as an addition. Combining results from different viewpoints, will give more robust knowledge. The results of this study are a promise for the future, in which we can expect further increase in the computational power as well as further increase in the understanding and application of pattern recognition techniques. This also means that this kind of research can be helpful in authorship disputes. It reveals characteristics of the compositions that cannot be known from normal perception. And, therefore, also not from the perception of the composers. When one composer imitates the style of another, it is not likely that he had much control on the characteristics that are examined in this study. Compositions which are very like each other for a listener, can be located in different places in the featurespace. With the results of the fc-means clustering-algorithm it is shown that the compositions of the various composers do have tendency to cluster. This is an indication that the chosen features are suited for separating the styles represented in the dataset. It is shown that projection of the data on the fisher-discriminants leads to more accurate attributions. The resulting featurespace is difficult to understand from a music-theory point of view, but when using this in authorship problems, this mapping can be made to get better results. With the investigated datasets, we can learn something about the characteristics of the styles of the composers whose compositions were incorporated in the dataset. The success of this is very dependent of the used style markers. In the present work the focus is on the sonorities. This turns out to be a good choice. The various composers all had their own preferences. It is shown that with these features Bach's style can be distinguished from his contemporaries with great accuracy. The feature that is most important for this is the amount of dissonant sonorities. The combination of a high amount of dissonant sonorities, a low amount of parallel thirds and a "steady" rhythm leads to the style of Bach. Although their styles are closely related, there is not much confusion between the
599 compositions of Handel and Telemann. So this featureset is also suited to uncover the differences between t h e styles of these two composers. It t u r n s out t h a t Handel wrote more fourths and sixths t h a n Telemann. There is much more confusion between H a y d n and Mozart. This could be expected since all compositions considered are in a genre t h a t offers the composer not very much freedom, and from "traditional" research it is known t h a t their styles are closely related. T h e feature t h a t t u r n s out to be most important is t h e voicedensity. Mozart wrote "thicker" textures t h a n Haydn. But, to get a better description of their styles other features may be necessary.
9. F u t u r e w o r k T h e present work indicates the possibilities of success when applying these p a t t e r n recognition techniques to more specific problems. So, it can be a valuable contribution t o some ongoing authorship discussions. It is also possible to follow development of style over a certain period of time. W h a t separates t h e seventeenth form t h e eighteenth century? How does the individual style of a composer develop? Of course, the featureset can be extended. E.g. addition of features t h a t describe the melodic characteristics (intervals, step vs. leap, etc.), features t h a t describe harmonic properties, or more detailed contrapuntal properties. There are many more possibilities. But maybe it is more important to find a more fundamental approach for finding style markers. There is a need for an underlying theory. It might be interesting to develop psychological models for the act of composing. This could give us an indication of which style makers reflects t h e unconscious processes in composing music. It is also interesting to think about what is necessary to describe a musical style entirely. Probably this is not possible, b u t maybe something can be said about t h e completeness of featuresets.
References 1. Love, H., Attributing Authorship: An Introduction, Cambridge, 2002, 15. 2. Ibid., 156ff. 3. Rudman, J., The Hypothetical and Theoretical Underpinnings of Non-traditional Authorship Attribution Studies: Assumptions, Presumptions, and Verifiable Constructs, University of Virginia, 7 may 2004, . 4. Rudman, J., "The State of Authorship Attribution Studies: Some Problems and Solutions", Computers and the Humanities, 31(1997-8), 351-365. 5. Dannenberg, R.B., B. Thom and D. Watson, "A Machine Learning Approach to Musical Style Recognition", Proceedings of the 1997 International Computer Music Conference, 344-347. 6. Ponce de Leon, P.J. and J.M. Ifiesta. "Musical style classification from symbolic data: A two-styles case study", Lecture Notes in Computer Science, 2771, 2004, 166-177.
600 7. Hofstetter, F.T., "The Nationalistic Fingerprint in Nineteenth-Century Romantic Chamber Music", Computers and the Humanities, 13(1979), 105-119. 8. Everything You Need to Know About The Humdrum "**kern" Representation, Ohio State University, School of Music, 7 may 2004, 9. Center for Computer Assisted Research in the Humanities, Stanford University, 7 may 2004, 10. Duin, B., PRTools, a Matlab Toolbox for Pattern Recognition, Delft University of Technology, Faculty of Applied Sciences, 7 may 2004, 11. Mason, R.M., Modern Methods of Music Analysis using Computers, Peterborough, 1985. 12. Webb, A., Statistical Pattern Recognition, Chichester, 2 n d edition, 2002. 13. Pudil, P., J. Novovicova and J. Kittler, "Floating Search Methods in Feature Selection", Pattern Recognition Letters, 15(1994), 1119-1125. 14. Jain, A. and D. Zongker, "Feature Selection: Evaluation, Application, and Small Sample Performance", IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(1997), 153-158. 15. Quinlan, J., C4-5: programs for Machine Learning, San Mateo, 1993.
CHAPTER 5.6 AUTO-DETECTOR: MOBILE AUTOMATIC NUMBER PLATE RECOGNITION
Giovanni B. Garibotto Elsag spa, Via Puccini, 2,1-16154 Genova, Italy e-mail: giovanni. garibotto @ elsag. it Auto-Detector is a revolutionary mobile Automatic Number Plate Recognition (ANPR) system, installed on board of patrol cars, to continuously detect and read the license plates of all the vehicles found during patrol (both parked and in-motion), without affecting normal operational duties. All recognized license plates are immediately compared against one or more hot-lists of selected number-plates (on-board) and an alarm signal is immediately issued to the patrol crew in case of a match found. The system is already in use by most important Italian security forces (Carabinieri Army and Road Police) for Land Security and traffic monitoring. International applications are in progress (US, UK, Chile, Switzerland, Spain, etc). Experimental results from current installations are referred.
1
Introduction
Increasing traffic density and congestion represents one of the main problems of everyday life in the crowded downtown. Automatic Number Plate Recognition (ANPR) technology is already a useful solution for access control to save time and alleviate congestion. Moreover, it is extremely important for security and surveillance activities as well as to assist patrol in law enforcement. Actually, the possibility to track vehicles in the traffic flow by using the only universal identification tag available, their number plate, is recognized as the most powerful tool in the area of Homeland Security. ANPR is a consolidated technology 1 , widely used in a variety of applications of ITS (Intelligent Transportation Systems). From the long list of suppliers, it is possible to identify two different types of products. The first one is made of software packages of Optical Character Recognition (OCR), mainly oriented to system developers and system integrators. The second category contains 601
602
integrated solutions, where OCR is a component of a complete system, including image sensors, lighting, and special image processors, for Traffic Control or Police Security applications. Most referred applications in the literature are based on fixed installations for access control, electronic pay-toll collection and law-enforcement (red-light violation, average speed control) and security control (stolen-car detection and crime investigation). Currently, the new application frontier is the development of mobile ANPR systems, installed on standard patrol vehicles, to increase data-collection in normal traffic flow, and extend inspection and monitoring possibilities during patrol. A mobile system should in a fully automatic way, and immediately, identify particular vehicles of interest, hidden amongst the huge volume of traffic on today roadways or congested urban areas. This function is extremely important for security control in downtown (for the detection of non-authorized vehicles) as well as along high-speed traffic highways or any kind of public parking lots. The recognition task from a moving platform is much more complex than from fixed installations, and represents a real challenge for Computer Vision technology. In fact, all critical conditions are present, like sudden, unpredictable changes of lighting conditions during patrols (transition from sunlit to shadow areas, tunnels, night patrols, etc.), arbitrary license plate 3D orientation, noise and variable contrast. Moreover, on-line response is a must, to allow immediate intervention by patrol crew when a searched number-plate is found in the traffic flow. Some examples of mobile LPR systems can be found in 2'3, where special sensor configuration (illumination and video camera) are installed on a servicecar for License Plate control of parked vehicles in parking lots as well as along the road side. Unfortunately, low-speed limitations, and the large size of the acquisition and processing units, are affordable for parking control, but represent unacceptable constraints for general road security applications. The paper describes "Auto-Detector", a new mobile on-board LPR system, that has been designed to detect and recognize the license plates of parked as well as moving vehicles, within the field of view of its micro video cameras. Elsag 0 2 CR technology (Outdoor Optical Character Recognition), developed around the embedded Intel X-Scale platform, plays a key role in solving these problems. It performs continuous recognition at video frequency, achieving more than 25 license plate readings per second, even in a cluttered environment. The vision sensor is integrated with a flash-pulsed infrared LED illuminator in an extremely compact package for easy installation on patrol vehicles, fully
603
compatible to existing equipment configurations. The system may be connected to standard geo-positioning and communication subsystems to satisfy specific application needs, from autonomous peripheral units, to centrally controlled car fleets. Auto-Detector can easily integrate also any optional video-recording system to collect a panoramic view of the scene. Main functional and technical requirements of mobile ANPR applications are briefly summarized in the following, with a technical description of AutoDetector system and its performance in real life patrol situations. Auto-Detector is already in operation (beginning of 2004) in more than 1300 patrol surveillance vehicles of the main Security Institutions in Italy (Carabinieri Army and RoadPolice). The referred results are based on the outcomes of a real testing phase in the field during years 2002 and 2003. 2
Requirements of a Mobile ANPR System
The use of a mobile ANPR system provides some important advantages in security and ITS applications. Data collection is no more limited to a fixed position along the road (like a tripod or a bridge), but can be performed almost everywhere along the traffic flow. As a consequence system requirements turn out to be much more severe: • Miniaturization: the imaging sensor must be realized small as much as possible, to be installed into quite narrow space, to harmonize with existing vehicle components. • Image processing units must provide low-power consumption to avoid overloading existing battery resources. • Recognition of slanted and skewed plates: because of 3D arbitrary orientation, the imaged license plate characters are heavily distorted and the recognition system must be able to achieve high recognition performance with severe perspective distortion conditions. • Sudden, unpredictable changes of external light conditions during patrols, • Real time processing at fast speed (over 25 fps, frames per second) to allow ANPR at normal cruise speed (approximately 50 km/h) in downtown control applications as well as at high cruise speed (more than 100 km/h) on the highway. • On-line immediate comparison against a huge data-base of searched license plates (million of LP's strings) for security apps (stolen car detection) and immediate transmission of alarms to central supervision units.
604
Ergonomic constraints to simplify user interaction, avoid any extra workload to patrol crew, and achieve an effective data management (upload/download of collected data). 3
Description of the Auto-Detector System
The block diagram in Fig. 1 illustrates the two main components of the system, namely the onboard license plate recognition, for alarm detection and image recording, and the central unit for data update and consultation and alarm management. Radio positioning station
Operations Centre
License plate control station
Vehicle communications
GSM/GPRS Modem
Vehicle radio positioning
Entry and deletion of hot-listed license plates LAN
Alarm management
WirelessOnboard vehicle GSM /GPRS Modem
RS232
LAN
RS 232
Navigation
System «
02CR LPR System
Left camera
• Right camera
Fig. 1. Auto-Detector - functional block diagram. The onboard unit integrates license plate recognition functions with the vehicle's onboard navigator system, which is used also for online communication with the control centre, as well as for man-machine interaction, on the existing onboard display (hot-list licence plate detected and recognised during patrol).
605
3.1 Database search and query Operations Centre personnel can consult all data collected during patrol, following the privacy rules valid in the different countries. According to some legislation only searched license plates in the hot-list can be stored and managed. Same criteria apply to downtown access control applications, where only license plates that are found outside the white-list of authorized vehicles can be stored for violation detection and prosecution. Certain legislations allow the storage of license plates for a limited period of time, under full responsibility of restricted institutions and legally authorized personnel. Auto-Detector system provides all necessary software tools for number plate search in the database (both complete and partial matches) and report where and when the searched plate was found. The use of a cartographic system enables operators to search for license plates by simply selecting the area of interest. It is also possible to display the image of the vehicle corresponding to the recognised licence plate, plus the map position of the site in which it was detected. Additional viewing functions include zoom and image enhancement tools. Other important processing tools are devoted to update the search-list of license plates, to add and remove number plate strings and entering comments or special annotations on the type of alarm (stolen vehicle or investigation in process, etc.) or the type of permission (temporary or permanent authorization). Password protection ensures that only authorised personnel can have access to the system.
Wna* To-ewo* SWMIO-I T M A *
*"
ifvffy*tM.v*Jt
Fig. 2. Licence plate consultation and blacklist update screens.
606
3.2
Camera orientation
The imaging sensor can be oriented in different ways inside the sensor assembly, mounted on the patrol vehicle's roof. The number of cameras and their optimal arrangement has been selected in such a way to allow the maximum number of plates to become visible, in the camera field of view, during patrol, according to the following classification: A. vehicles moving in the same direction as the patrol car in the lanes to the right and left (overtaking). This situation is typical on urban throughways and highways (limited to the lanes immediately adjacent to the patrol vehicle). B. vehicles moving ahead in the same direction and the same lane of the patrol car. C. vehicles moving in the opposite direction to the patrol car in the oncoming lane to the left. This situation is typical in narrow urban contexts with twodirectional traffic. D. vehicles parked in the direction of traffic flow to the right of patrol vehicles, and also to the left in narrow one-way streets (parallel parking). E. diagonally parked vehicles to the right and left depending on the type of road (one-way or two-ways). Such a situation is typical in some urban contexts to optimise available parking space and is very frequent in large factory, and commercial car-parks. F. perpendicular parking. This is a quite common situation in large commercial and airport car-parks where large space is available. Our selected solution, as shown in figure 3, represents a compromise to collect the largest possible number of license plates during patrol. Only category B seems not to be managed by this configuration, since vehicles immediately ahead of patrol car, in the same traffic lane, are outside the image field of view of the cameras. Anyway, normal traffic manoeuvres (left/right turning, overtaking) increase the probability of those plates to be framed sometimes, since they remain for a long time in front of the patrol car. Hence, it was found ineffective to include an additional camera oriented ahead for such a small number of occurrences. 3.3 Imaging sensor A key issue of Auto-Detector design was the selection of the most appropriate camera sensor configuration to allow its effective installation on patrol vehicles without affecting their functional characteristics. Moreover, a big problem is
607
raised by the wide, unpredictable variations in lighting conditions, from just a few lux in shadow, tunnels and at night, to more than 100 Klux in full sunlight. Reflective properties of modern number plates in all countries of the world allow the use of an integrated acquisition system combining digital micro-cameras coupled to a programmable high-speed pulsed LED illuminator. To deliver maximum acquisition efficiency, the pulses of light are synchronised with the camera's acquisition system so that they occur at the same time and have the same duration as the optical sensor's exposure time. The beam of light emitted by the LED is in the near infrared spectrum (solutions using LED's in the range 730 to 880 nanometres are available) to limit ambient light interference. This makes it possible to compensate for low light conditions, and reduce the interference caused by shadows thrown on license plates in full sunlight.
3 m. c.a.
3 m. c.a. Fig. 3. Onboard camera configuration.
The achieved degree of miniaturisation, as depicted in fig. 4, has been possible using high level LED on-chip integration technology, from StockerYale4 (more than 100 LED's inside a 2.5 cm diameter cylinder) and the use of high-sensitivity digital micro-cameras (progressive scan). 4
License Plate Processing and Recognition
The key feature of the Auto-Detector system is the availability of an extremely effective Automatic Number Plate Recognition system able to detect the maximum number of instances of the LP's in transit with high accuracy. The ANPR system should be able to work continuously on the input video sequence
608
at the maximum rate (more than 25 frame per second fps, for progressive cameras) to read license plates from the video flow, without the help of any external triggering device. Elsag has developed its first ANPR system at the beginning of the 90's in a variety of traffic control applications, from pay-toll highway applications (Telepass by Autostrade spa), parking control, access control to downtown restricted traffic areas. This technology is based on the long experience on Intelligent Character Recognition applied to document and form processing5 as well as for address understanding and handwriting in mail sorting6.
Fig. 4. Using a high level of miniaturisation Auto-Detector system can easily be adapted for use in a broad range of tailor-made applications.
There are many system-engineering issues to be addressed when dealing with license plate recognition and tracking. It requires multidisciplinary competence of electro-optics, Computer Vision and Pattern recognition. Adaptive exposure control is another essential feature of the acquisition system. Image Processing technologies are used to achieve the most effective image normalization and optimise plate detection and character localization. The recognition process
609
(more details can be found in 7) consists in three main blocks as described in fig.5. A temporal post-processing unit is arranged to perform data fusion and tracking among different images of the video sequence. The context verification process exploits both spatial and syntactic information in order to select the best hypothesis for the number-plate. Let us suppose that a given country allows number-plate with K characters in fixed position /,,=fX„F,j (expressed in a given system of coordinates). If the image under processing contains N validated characters, the general idea is to extract all choices of K elements from N and to evaluate them both geometrically and syntactically. The final temporal postprocessing stage aims to extract a single number-plate for a given vehicle. The presence of a vehicle is detected by the observation of a group of video frames affected by image motion and showing a sufficient number of recognized characters. Such definition allows the detection of a vehicle even if the contextual process does not extract a valid number-plate hypothesis. All possible number plate hypotheses, with both spatial and syntactic information, are gathered from the images belonging to the selected video frames and a clustering approach is used in order to compute the best choice. Video-rate processing allows also instantaneous speed estimation from multiple readings of the same License Plates8. Continuous Digital Video Acquisition
^ ^
Exposure control
1r
• • • • •
Video Processing and ROI detection 3D perspective ROI normalization Radiometric: contrast norma ization and character segmentation Grey-scale Character Recog nition and nneasure Contextual Analysis and Un derstanding
Data base creation and maintenance Alarm list management
+
Application Interface (MMI and Apps Management)
Fig. 5. Block diagram of the 02CR License Plate Reader.
4
^
610
Main characteristics of Elsag 0 2 CR ANPR are: • Adaptive control of the exposure time of the camera to optimize the contrast image of the license plate • Data temporal integration and feedback loop control to recover character segmentation errors. • Continuous processing on the video sequence (free-running) and simultaneous multiple license reading, without the need of external triggering systems. • Integration of high sensitivity imaging sensors with IR-LED integrated illumination system. • Detection and Recognition performance (certified according to Italian standards UNI10772) has been measured above 95% according to the different operating conditions, with reading tolerance over 35° in both space orientations. Accuracy has been measured above 99.5% (Italian context) for the detected license plates. • Multinational license plate (multifont training is required to deal with different contextual information). A general context-free configuration has been developed with similar recognition performance. This is absolutely necessary for applications (like US vanity plates) where no context is available at all. Auto-Detector is configured with two acquisition sensors (left and right oriented). To achieve high recognition performance across two independent channels considerable processing power is required, combined with low consumption (much lower than usually offered by standard industrial components) to avoid onboard battery drain. The solution adopted implements an extremely effective processing configuration, based on INTEL X-Scale embedded technology, developed by Tattile srl9: • INTEL X-Scale integer processor (880 MHz, 32 bits, 64 MB RAM) • 100 Mb/s network connection low consumption (<10 W with 12-24 V. supply) • extremely compact solution • easy remote link for maintenance and upgrades. As illustrated in fig. 6, the system acts as a network server. Communications with the operations centre are handled over a wireless-LAN link at the start/end of patrols. On-line communication is managed through standard radio-link or public networks based on GSM-GPRS connections. System configuration is continuously evolving by following to the increasing power of embedded processing technology. Thanks to the high level of miniaturisation achieved, it
611
has been possible to install all processing instrumentation in the small space available in the baggage compartment of the patrol car.
License plate reading unit Micro camera + IR Flash
Smart-Reader and controller
Micro camera + IR Flash
Smart-Reader and controller
Wireless LAN bridge
•!-• LAN
Ethernet Frame-Vision Store and harddisk
Colour Micro camera
Videorecording (optional)
Navigation System and Display
Composite PAL Video 4—
RS232 serial link
Fig. 6. O CR onboard license plate recognition unit - block diagram.
5
On-Board Functional Operations
The paragraphs below describe a typical patrol mission of Auto-Detector. Data and hot-list downloads are performed from the Operations Centre using a PC workstation server with a user interface and applications software to update and consult the licence plate database. Data exchange from the central station to the onboard vehicle is handled by a wireless-LAN link in a restricted operation area e.g. inside the patrol car-park or workshop), which minimises human intervention and makes the data exchange process as automatic as possible. 5.1
Start of patrol
Patrol officers are not required to perform any supplementary operations when starting their vehicles. All data update operations are performed completely automatically over a wireless-LAN link. Upon completing all data upload, the
612 0 2 CR is ready to acquire, recognise and store license plate numbers from the traffic flow. 5.2
License plate recognition during patrol
During patrols, police officers receive and send messages from and to the Operations Centre using the onboard navigator terminal, which in addition to its usual functions, also offers the following services: u the licence plate recognition unit stores data referring to vehicle transits in the lanes to the right and left of the patrol vehicle. The string of the licence plate most recently recognised by each camera is updated continuously on the available display monitor. • Transit data recorded by the onboard unit contains: the license plate string recognised, a compressed image of the vehicle and the date, time and position (geographical position data from the onboard navigator). When a license plate on the alam-list is recognised, the system sounds an alarm and the "best" transit image is immediately displayed on the screen to provide the human operator an evidence of the positive detection. The alarm signal is automatically sent also to the Operations Centre through the onboard communication system. A marker is automatically entered on the onboard navigator map to display the position (on the navigation map) where the license plate was detected and recognised.
Fig. 7. When a license plate has been found in the alarm-list, its image is displayed on the monitor.
Q During patrols, the Operations Centre can also send supplementary license plates to search for, in addition to those stored in the onboard hot-list.
613
5.3 End of mission On returning from patrols, the system automatically uploads all collected data to the central unit over the wireless-LAN link, and then shuts the system down, without any workload for the crew.
6
Results and Performance
The license plate reading system is designed to recognise standard approved license plate formats. Recognition performance can be affected by non-standard formats, damaged or dirty plates. Performance figures given below are based on the principle of subjective "readability" by a human observer sitting in the patrol vehicle next to the driver. 6.1 Experimental results (Italian applications) Auto-Detector has been integrated into a series of experimental pilot systems since March 2002 for the investigation security forces (Carabinieri Army) as well as for the Italian Road Police, in a variety of typical patrol missions, downtown (in the main Italian cities) as well as along the highways or high-speed roadways. More than 800 patrol cars have been already in operations at the end of 2003 through all the country, and the achieved results have been judged very satisfactory by official Institutions in terms of alarms detection and support to investigation (quantitative results are confidential). Officially disclosed information, through the media, refers that more than 1000 stolen cars have been found in less than a year of operation and even greater benefit is expected in the future, when the full control service will be completed. Recognition performance has been measured in normal license plate conditions (level of dirt build-up, shadows and reflections, colour, etc.). The system can work successfully in all environmental conditions thanks to the sensitivity of the micro-cameras and the use of optical filters in the near infrared (IR) spectrum. Performance figures have been computed by comparing the number of license plates, recognized by the onboard ANPR system, against the number of license plates considered "readable" by a human observer, from a recorded video stream, collected by a normal video camera placed nearby the sensor assembly. The following average results have been measured: a motorway traffic: at a cruise speed of 110 km/h the average percentage of successful recognitions vs. "readable" license plates was > 90%
vehicles parked at roadside (in the direction of traffic flow) along urban road sections; at an average speed of about 50 km/h? the average percentage of successful recognitions vs. "readable" license plates by an onboard operator was greater than 90% diagonally parked vehicles: at an average speed of about 50 km/h, the average percentage of successful recognitions vs. license plates "readable" by an onboard operator was approximately 95%
Fig. 8. Examples of LP's collected by Auto-Detector during patrol missions.
i^BilBpiiMB
IIIIBIllll^^^MII
Fig. 9. Diagonally parked vehicles: day and night processing.
o •
Perpendicular parked vehicles: a 45° pan orientation of the camera has been used to achieve an average percentage of successful recognitions > 90% Night time recognition is affected only by very dirty or damaged plates. Otherwise the near infrared LED illumination produces images that are even more favourable for recognition than at daytime by strongly attenuating all objects without retro-reflectivity properties.
615 •
a
there was no appreciable performance reduction in rain or overcast conditions. In these cases environmental conditions make the images acquired more uniform, like those at dusk or night the error rate for all license plates recognised by the system is extremely low (lower than 0.5 %), despite the highly variable lighting of the scene and the trajectory of the vehicles encountered during patrols. This result has been achieved by the explicit use of geometrical context constraints and license plate number syntax, in agreement with the rules established by national certification authorities.
6.2 An experimental mobile access control to downtown City Centre There are already many implementations of downtown access control and Road-pricing, by using ANPR fixed installations in most of European countries10, (London, Stockholm, Rome, Milan, and other important Italian historical centres). The common feature of these implementations is the automatic recognition of all authorized license plates from a white-list collected by Administration Authorities. Main problems of conventional systems are the need of triggered systems, the environmental impact of installations (encumbering support poles and processing control boxes), the need of traffic restrictions (single lane constraints). By using a mobile ANPR, like Auto-Detector, on board of local police patrol cars, a thorough control and monitoring of downtown centres is possible, without the need of any fixed infrastructure. This feature is particularly important to save the artistic and architectural value of most important historical centres in Europe. Moreover, an immediate notification of the violation is possible by police officers as soon as a non-authorized vehicle is found in the traffic flow (either in motion or parked), beside the normal collection procedure of traffic violations. Finally, the level of monitoring and control can be decided by the local police administration, for instance by sending more patrol cars in downtown in the more busy traffic hours. An experimental installation of Auto-Detector is in progress in Verona, since January 2004, to test the effectiveness of the proposed solution and its integration in the conventional traffic control procedures. 6.3 International applications Auto-Detector has been successfully experimented in many different countries in Europe (UK, Spain, Slovenia, Croatia, Greece, Switzerland) in US (Ohio and Washington D.C.) in South America (Chile). For each application a
616
suitable training of the different character fonts has been performed in a very short time. Typically two-days data collection has proved to be sufficient to achieve a satisfactory level of recognition performance, just slightly lower than the best figures measured in the consolidated Italian installations. The introduction of the appropriate syntactic constraints is possible by using the international standard rules described in Europlate11. Such international experimental activity has been supported by the availability of a demo-configuration of the Auto-Detector system as a "modular transportable system". It consists of a case containing the sensor configuration (camera and IR illumination system) with magnetic support, to be easily placed on the roof of any standard vehicle. Another case contains the processing unit, to be placed in the boot of the car, with 12V power supply (from the cigarettelighter). A standard lap-top PC, with wireless-LAN connection, is used to provide man-machine interface as well as to allow data management and hot-list update as a simulation of the supervisory Central station. This democonfiguration can be installed in a few minutes on any standard car, and can be used by anyone, with no special skill requirements. 6.4
Future developments
A complex project as the mobile Auto-detector system is never finished. There are always improvements from many different viewpoints: hardware and sensor technology, Computer Vision and Image processing, system integration as well as the identification of new applications. A multiple camera configuration has been already implemented to allow different camera orientation according to special application requirements (highspeed far-range vision, slow-speed side view, etc.). An experimental 4-camera sensor prototype has been developed. By combining together camera orientation and optimized focal lengths it is possible to greatly increase the range of inspection and monitoring around the patrol car. Further improvements of grey-scale character recognition are in progress by exploiting, even more, temporal integration of multiple readings. Higher resolution and improved robustness of recognition are the main goal of this activity. New applications of the mobile sensor of Auto-Detector are under investigation. One of such research goal is the automatic traffic sign detection from a service car equipped with a quite similar IR sensor configuration. Retroreflective properties of the traffic signs are used to improve signal detection along the roadside. The final objective is the detection and recognition of traffic
617
signs with geo-reference position along highways, to allow an automatic monitoring of the efficiency of the roadway. 7
Conclusions
Computer Vision has been always considered as one of the most promising technologies towards Intelligent Sensor and Systems, able to solve automatically complex and relevant perception recognition tasks. There are many practical examples of applications (Machine Inspection, Robotics, Metrology) where Computer Vision systems are successfully used to mimic human perception capabilities with higher performance (objective and quantitative results, accuracy, processing speed). In this context, Optical Character Recognition, OCR, is still the most effective example of successful exploitation of Vision technology, in document processing as well as in Mail sorting, with an extremely high reading performance (more than 15 mail address recognition per second). Moreover, the economic value of these applications are extremely high with many installations in all countries in the world. Auto-Detector is another success story of Intelligent Computer Vision since License Plate detection and recognition in a busy and cluttered traffic flow far exceeds human perception possibilities. Multiple plate tracking and reading at a rate of 25 frames/second, and the immediate on-line check against a list of millions of license plates represents a complete new service provided to patrol Surveillance and Security Institutions to support monitoring and investigation activities in the new challenging field of Homeland Security. This data collection system can rapidly alert patrols and the operations centre when hot-listed numbers are detected. Moreover, Auto-Detector has been developed to satisfy current operational constraints and achieve a high level of acceptance by the users. The system perfectly integrates with existing on-board components, without adding any extra effort to the workload of patrol personnel, who is free to operate as usual in his/her duties. As such it can be easily defined as an auxiliary officer on-board of patrol cars. References 1. 2. 3.
Y. Hofman "License Plate Recognition - A Tutorial", Hi-Tech Solutions, May 20, 2003, http://www.licenseplaterecognition.com/ AutoFind - Mobile License Plate Recognition LPR, from AutoVu Technologies, http://www.autovu.com/website/content/products autofind.html Mobile LPI (License Plate Inventory) http://www.pipstechnology.com/
618 4. 5.
6.
7.
8.
9. 10. 11.
Stocker Yale Awarded Contract for Specialized LED Applications", Press release Nov.2002, http://www.stockeryale.com/company/news/pr_l 1072002.htm E. Appiani, L. Boato, S. Bruzzo, A.M. Colla, M. Davite and D. Sciarra, "STRETCH": A System for Document Storage and Retrieval by Content, Proc. DEXA '99, Workshop W07: DAUDD'99, Florence, Italy, Sep. '99, E.Bruzzone, E.Furfaro, B.Ghiolo, S.Spaggiari, "Korean Flat sorting", pp. 177-188, Proceedings of the Int.Conference on Mail Technology - Tomorrow's World, ImechE Conference Transactions, ISBN 1 86058 220 6. G.Garibotto, P.Castello, E.Del Ninno, "Dynamic Vision for License Plate Recognition", chap 6.6, "Multimedia Vision-based Surveillance Systems", Kluwer Academic Publisher, 1999. G.Garibotto, P.Castello, E.Del Ninno, P.Pedrazzi, G.Zan, "Speed-Vision: speed measurement by License Plate reading and tracking", 2001 IEEE Intelligent Transportation Systems Proceedings, pp 585 590. Tattile, Smart Reader, "Machine Vision based on SAllOO Intel StrongARM Processor is now ready for your application", http://www.tattile.com. Urban Transport Pricing in Europe, http://www.transport-pricing.net/ The Interpol Guide to Vehicle Registration Plates - by N.Parker & J. Weeks -EUROPLATE - May 2003
Chapter 5.7 OMNIDIRECTIONAL VISION Department
Hiroshi Ishiguro of Adaptive Machine Osaka University
Systems
Omnidirectional vision sensors, which have been proposed in 1970, are recently studied in computer vision and multimedia research. This chapter discusses features of previously developed omnidirectional vision sensors and their applications. Omnidirectional vision is one of the key issues in advanced computer vision techniques and enables us to develop various applications of computer vision.
1. Introduction Physical agents living in complex environments, such as humans and animals, need two types of visual sensing abilities. One is to gaze particular objects with a precise but small retina, the other is to look around the environment with a wide but coarse retina. Both visual sensing mechanisms are required for realizing robust and flexible visual behaviors. Especially, the omnidirectional visual information obtained by looking around is necessary to monitor wide areas and to avoid dangerous situations. On the other hand, cameras developed for TV broadcasting so far have been used for monitoring systems and vision systems of robots in previous computer and robot vision studies. The standard TV cameras, which have a limited visual field of about 30 - 60 degrees, can be used to observe local areas, but it cannot observe wide areas. In order to extend the TV camera applications and develop robust and flexible vision systems like animals, special cameras that can take omnidirectional visual information is needed. Such OmniDirectional Vision Sensors (ODVSs) (or omnidirectional cameras) is proposed by Rees [10] in the patent submitted to US government in 1970. Then, Yagi [12], Hong [2] and Yamazawa [13] developed again in 1990, 1991 and 1993, respectively. Recently, Nayar [7] has geometrically analyzed the complete class of single-lens single-mirror catadioptric imaging systems and developed an ideal ODVS using a parabola mirror. Figure 1 shows an example of the omnidirectional image. 619
620
Figure 1: Omnidirectional image 2. Omnidirectional vision sensors The history The original idea of the ODVSs using a mirror in combination with a conventional imaging system has been proposed by Rees in the U.S. Patent No. 3, 505, 465 in 1970 [10] (see Figure 2(c)). The idea is to use a hyperboloidal mirror for acquiring an ODI that has a single center of projection. That is, the ODI can be transformed into normal perspective images. In 1990, progress of computer technologies enabled real-time process of vision data and researchers made again several types of ODVSs as vision systems for computers and robots. Yagi and Kawato [12] made an omnidirectional vision sensor using a conic mirror (see Figure 2(a)). Hong and others [2] made an omnidirectional vision sensor using a spherical mirror (see Figure 2(b)). Their purpose was to navigate mobile robots with the ODVSs. The omnidirectional vision of a robot is convenient for detecting moving obstacles around the robot and for localizing itself. Then, Yamazawa and others [13] made
621 again an ODVS by using a hyperboloidal mirror. By utilizing the merit of the hyperboloidal mirror that the ODI can be transformed into perspective images, they proposed a monitoring system with the ODVS.
Telecentr lens
(b) Spherical miiror
(c)
Hyperboloidal mirror
(d) Parabolla mirror
Figure 2: Omnidirectional mirrors
Nayar and Baker [7] theoretically analyzed imaging systems using mirrors and developed an ideal ODVS using a parabola mirror and a telecentric lens. The ODVS using a hyperboloidal mirror can generate an image taken from a single center of projection in combination with a standard perspective camera. However, it has a demerit that one of the two focal points of the hyperboloid has to be set on the camera center as shown in Figure 2(c). This demerit makes it difficult to design the ODVS. On the other hand, the imaging system proposed by Nayar and Baker does not have such a demerit since it is using the telecentric lens as shown in Figure 2(d). As known well, the parabola mirror has a focal point for light which is parallel to the main axis of the parabola mirror. Further, the imaging system is superior in acquisition of non-blurred images and it can eliminate internal reflections of a hollow cylindrical or spherical glass which supports the mirror. Another method to acquire ODIs The ODVSs using mirrors acquire omnidirectional images (ODIs) in real time. However, the resolution is not so high. In order to acquire high resolution ODIs, methods swiveling a camera have been proposed by Sarachik [11] and Ishiguro [3]. The original idea has been given by the panorama camera which takes panoramic scene photographs by swiveling a slit camera. For taking images in static environments, this method is very effective and it is being recently used in multimedia applications. Another problem of the ODVSs is control of camera parameters, especially control of iris. In the method to swiveling a camera, the
622
camera observes local environments, but the ODVS observes a wide environments and the ODI taken with the ODVS contains various intensities. Therefore, the camera used in the ODVS need a wide dynamic range. As discussed here, the ODVS which can take ODIs has several advantages against the previous vision sensors, but it has also two major demerits, the low resolution and the requirement of a wide dynamic range. With current CCD sensors, it is difficult to obtain high resolution ODIs and its dynamic range is not so wide. We need to improve the CCD itself. 3. Omnidirectional images The history The origin of the methods to acquire the ODIs was a panoramic camera which takes omnidirectional photographs though a slit filter attached in the front of the camera lens while swiveling the camera. Zheng and Tsuji [14] used this idea with a CCD camera. The image obtained by arranging image data along a vertical line on the image center is called Panoramic Image. They analyzed the features of the panoramic images and proposed applications for mobile robot navigation. When the camera moves along a circular path in the method for acquiring panoramic images, an ODI is obtained. The ODIs is a cylindrical projection and it can contain precise angular information if the camera precisely moves. Early studies on the ODIs were mainly done by Nelson, Zheng and Ishiguro. Zheng and others [14] proposed a Circular Dynamic Programming for identifying features between two ODIs. The circular dynamic programming robustly finds correspondences by iterating a conventional dynamic programming method based on the periodicity of the ODIs. Ishiguro and others [3] proposed two types of Omnidirectional Stereo. By rotating a camera along a circular path, motion pallarax is observed by tracking feature points on the image plane and omnidirectional range information can be obtained. This stereo method does not have any blind spots outside the circular path. Another stereo is realized with two ODIs taken at different locations. Although the method using two ODIs has a problem of feature identification, it can obtain more precise omnidirectional range information. The optical flow field The flow field of ODIs is also interesting properties. Nelson and others [8] analyzed the flow field of the Gauss sphere retina and proposed methods to estimate camera motion parameters. On the other hand, Ishiguro and others focused upon just the FOEs and proposed methods to precisely navigate mobile robots [3] and to estimate robot motion parameters [4] based on the important feature of the ODIs that two FOEs, FOE and FOC, are observed in the flow field and the angle between them is 180 degrees. The periodicity An ODI is a periodical signal around the rotation axis. That is, Fourier transform
623
of the ODI does not require window functions. This means the transform is precise and efficient data compression is possible for the ODIs. By Fourier transform, an ODI is divided into magnitude components and phase components. The magnitude and phase components depend on location of the ODVS and the direction of the reference axis of the ODVS, respectively. Based on the magnitude and phase components, mobile robot navigation that does not refer to the internal sensor data can be realized [5]. First, the robot moves randomly in the environment and takes ODIs at various locations. Then, it executes Fourier transform for the ODIs and divides them into magnitude and phase components. By comparing the magnitude components of the ODIs, positions where the ODIs are taken can be estimated. The positions cannot be precisely estimated but it is topologically correct. The map that represents the topological positions of the observation points can be used for the robot navigation. Here, in order to use the map, the robot needs to know its direction against the environment. The direction can be estimated from the phase components of the ODIs. That is, the robot can memorize locations as a map and navigate itself by using it only with the ODVS.
4.
Designs of ODVSs
An ODVS consists of two major components, a mirror which is symmetrical on rotation and an apparatus which supports the mirror. This section discusses merits and demerits on various designs of the two major components of previously developed ODVSs Designs of mirrors There are four types of the previously developed mirrors as shown in Figure 2. Merits and demerits of the mirrors can be discussed from the following aspects: Whether the mirror can generate an ODI which has a single center of projection (The ODI can be transformed to normal perspective images). • How small the astigmatism of the optical system consisting of the mirror and a camera is. • Whether the optical system uses a standard lens and camera. • How large the vertical viewing angle is. Spherical mirror Generally, mirrors are made by depositing aluminum film onto a shaped glass. An important issue in the machining is how easy it is to process the glass. A normal lens is a part of a spherical glass, therefore it is easy to make spherical mirrors with the conventional lens process. In addition to the merit in the machining, another important merit of the spherical mirror is the astigmatism. Comparing with other mirrors as shown in Figure 2, the astigmatism is rather small since it can be considered as a flat surface near the optical axis of the camera (of course, it is not small in the peripheral). Further, as discussed in the next section, the spherical mirror does
624
not require a long focal depth for acquiring a focused image. That is, the spherical mirror is superior to making low cost ODVSs which can acquire clear images. However, the ODI acquired with the spherical mirror does not have a single center of projection and cannot be transformed into normal perspective images. The vertical viewing angle is also so large. Although the ODVS can observe over the horizontal plane, the image is distorted in the peripheral of the ODI.
Conic mirror A conic mirror is second to the spherical mirror in the easy machining. The feature of the conic mirror is to have normal reflection in the vertical direction. Therefore, it is easy to combine several mirrors. For example, stereo images can be acquired as shown in Figure 2(a2). Further, the conic mirror enables a special optical system which observes horizontally in combination with a telcentric lens as shown in Figure 2(a3). However, the astigmatism is large and the ODI cannot be transformed into normal perspective images. Further, it needs a long focal depth to acquire focused ODIs. A spherical mirror has a focal point like a normal lens, on the other hand, the conic mirror does not have it and needs a lens which characteristics are near to them of a pin hole (the detail is discussed in the next section). Hyperboloidal mirror The machining of a hyperbolidal mirror is difficult, but it has a single center of projection. An ODI taken with the hyperboloidal mirror can be transformed to normal perspective images, cylindrical images, and so on. Further, if the curvature is small, the astigmatism is not so large. The hyperboloidal mirror is the best for optical systems using normal cameras. However, it has serious demerits that the design is not so flexible since the focal point of the hyperboloid needs to be set on the camera center. Parabola mirror An ideal optical system can be realized with a parabola mirror and telecentric lens. The optical system has a single center of projection and the astigmatism is small for a small curvature. Further, the parabola mirror is the best for acquiring focused ODIs. The telecentric lens also brings two merits. Since the projection is orthogonal, the distance between the mirror and the lens can be set flexibly in the design and the lens eliminates internal reflections of a glass cylinder or sphere which supports the mirror (the detail is discussed in the next subsection). However, it is a demerit for making a compact and low-cost system to use the telecentric lens. The telecentric lens is generally expensive and the size is not so small.
625
Based on the discussion in the previous sections, the author has developed a low-cost and compact ODVS as shown in Figure 3. An important component of the ODVS is an appara-tus supporting the mirror. For the supporting apparatus, there are two requirements: • Eliminating internal reflections by the supporting apparatus. • Precise surface for acquiring non-distorted ODIs. Many types of ready-made clear hollow cylinders, which are made from glass and plastic, are available and the surface precision is sufficiently high. However, the clear hollow cylinder has a serious problem of internal reflections. One of the ideas to solve this problem is to support the mirror with a clear hollow sphere of which center locates on the focal point of the mirror. However, the problem is in the precision of the surface. Generally, precise machining for the clear hollow sphere is very difficult. Another idea to eliminate internal reflections by the clear hollow cylinder is to equip a black needle along the main axis of the cylinder. The light which is reflected by the internal surface of the clear hollow cylinder and then passes through the camera center closes the main axis of the cylinder. Therefore, the black needle equipped along the main axis completely eliminates internal reflections. 5. Applications of ODVSs Multimedia applications In the recording of round table meeting scenes, the previous camera cannot acquire images which contains faces of all participants. The ODVS attached at the center of the table makes it possible. Nishimura and others [9] used the ODVS for recognizing human gestures in round table meetings. Their approach can be extended for several applications. For example, indexed database of
Figure 3: C-ODV
626
Figure 4: Distributed vision system for recognizing human behaviors round table meetings with the simple human behavior recognition and teleconference systems which enable communication between groups can be considered. Monitoring applications The ODVS with a hyperboloidal or parabola mirror can be used in spite of the conventional camera system using a pan-tilt gaze controller. The ODI taken with the ODVS can be transformed into perspective images taken in any directions. Although there exist several problems, especially the resolution of the images, the ODVSs can be used as gaze control camera systems. The systems with the ODVSs is very compact and it can change the gaze direction in real time with a powerful computer. The ODVSs have a demerit of the low image resolution, therefore the author considers ODVSs should not be used as TV cameras, but as vision sensors. As an application in which ODVSs are used as vision sensors, a moving object tracking system using multiple ODVSs can be considered [6]. Figure 4 shows an overview of the system. The use of multiple ODVSs solves the following problems of vision systems with the wide viewing range. 1. 2.
The wide environment can be covered with a smaller number of ODVSs. The ODVSs can be observed each other, therefore the relative positions of the ODVSs can be precisely estimated with error minimization methods, such as a least square method. 3. Extended trinocular vision, namely N-ocular vision, can be realized. With the N-ocular vision, multiple ODVSs can robustly identify objects between the ODIs. The robust and real-time tracking system can be developed by using
627
constraints of the environment in addition to the above fundamental merits. For example, this system can be used for human behavior recognition. Previous research approaches for human behavior recognition focused upon rather local behavior, such as facial expressions and gestures. The system which recognizes global behaviors of humans in a room gives useful information to the previous system for recognizing local behaviors. In the case where ODVSs are used as vision sensors, the approaches to use many ODVSs are promising and C-ODVSs are useful for such applications which require the many number of ODVSs. Mobile robot navigation Other major applications of the ODVS are in mobile robot navigation. The wide viewing range is useful for predicting collisions with moving obstacles. Yagi and others [12] proposed a method to avoid collisions with moving obstacles by continuous observation with the ODVS. In the case where both the robot and obstacles move along linear paths with constant velocities and they collide each other, the viewing angle from the robot to the obstacles is constant. By feeding the viewing angle to the steering, the robot can avoid the collision. The ODIs are also useful for memorizing particular locations. Hong and others [2] proposed a method for image-based homing. The robot memorizes the goal location with a vertical edge image observed by the ODVS. Then, the robot goes from a distant location toward the goal by comparing input images with the memorized image. This approach is interesting as a method which shows rich properties of ODIs. Further, ODVSs can be used for other functions of robots, such as pose stabilization and so on. The author considers the most serious problem of previous vision-guided mobile robots is in the limited visual field of the vision sensors. The ODVSs are expected as key tools for solving such difficulties. 6. Conclusion In this chapter, the author has discussed features of previously developed omnidirectional vision sensors and proposed designs of low-cost and compact ODVSs. Further, their novel applications have been briefly discussed. Although a few problems still remained for the ODVSs, the author considers utilization of omnidirectional vision sensor will be a key issue in computer vision and multimedia applications. References 1. D. H. Ballard, Reference frames for animate vision, Proc. Int. Joint. Conf. Artificial Intelligence, pp. 1635-1641, 1989. 2. J. Hong, and others, Image-based homing, Proc. Int. Conf. Robotics and Automation, 1991. 3. H. Ishiguro, M. Yamamoto and S. Tsuji, Omni-directional stereo, IEEE Trans. Pattern Analysis & Machine Intelligence, Vol. 14, No. 2, pp. 257262, 1992.
628
4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
H. Ishiguro, K. Ueda and S. Tsuji, Omnidirectional visual information for navigating a mobile robot, Proc. IEEE Int. Conf. Robotics and Automation, pp. 799-804, 1993. H. Ishiguro and S. Tsuji, Image-based memory of environment, Proc. Int. Conf. Intelligent Robots and Systems, pp. 634-639, Nov. 1996. H. Ishiguro, T. Sogo and T. Ishida, Human behavior recognition by a distributed vision system, Proc. DiCoMo Workshop, pp. 615-620, 1997. (in Japanese) S. K. Nayar and S. Baker, Catadioptiric image formation, Proc. 1997 DARPA Image Understanding Workshop, pp. 1431-1437, 1997. R. C. Nelson and J. Aloimonos, Finding motion parameters from spherical flow fields, Proc. Workshop on Computer Vision, pp. 145-150, 1987. T. Nishimura, T. Mukai and R. Oka, Spotting recognition of gestures performed by people from a single time-varying image, Proc. Int. Conf. Robots and Systems, pp. 967-972, 1997. D. W. Rees, Panoramic television viewing system, United States Patent No. 3, 505, 465, Apr. 1970. K. Sarachik, Characterizing an indoor environment with a mobile robot and uncaliblated stereo, Proc. Int. Conf. Robotics and Automation, pp. 984-989, 1989. Y. Yagi and S. Kawato, Panoramic scene analysis with conic projection, Proc. Int. Conf. Robots and Systems, 1990. K. Yamazawa, Y. Yagi and M. Yachida, Omnidirectional imaging with hyperboloidal projection, Proc. Int. Conf. Robots and Systems, 1993. J. Y. Zheng and S. Tsuji, Panoramic representation for route recognition by a mobile robot, Int. J. Computer Vision, Vol. 9, No. 1, pp. 55-76, 1992.
Index
2-class clustering, 264 2-D, 79, 137, 157, 197, 325, 395,454 2-D Fourier transform, 394 2-D wavelet transform, 325 3-D, 70, 72, 77, 87,115, 137, 157, 395, 422,454,464 4-D, 157
parallel Contextual, 55 Parallel External Contextual (PEAC), 59 artificial intelligence, 96 artificial neural network (ANN), (see neural network) aspect ratio, 284 authorship attribution, 587 authorship of music, 585 Auto-Detector, 601 automated personal identification, 412 automatic handwritten text processing, 259 automatic knowledge acquisition, 219 Automatic Number Plate Recognition (ANPR), 601 mobile system, 602 auto-regressive model, 181 average recognition rate, 282
active contour model (snake), 309 AdaBoost, 482 adaptive exposure control, 608 adaptive region growing, 308 affiliation, 561 affine-invariant algorithm, 74 affine-invariant semi-local parts, 87 affine transformation, 74 agent, 389 aggregation-participation, 560 algorithm, 2-D wave-front, 173 Bayesian relaxation labeling, 310 Baum-Welch, 25 gradient-based optimization, 68 £-mean clustering, 287 LBG, 264 match-filtering, 350 parallel, 171 sharpening, 353 skeletonization 137 watershed, 153 appearance based method, 413 array grammar, context, 56
background characterization, 347 background clutter, 461 background subtraction, 504 bank cheques, 259 Bayes decision rule, 44 Bayesian information criterion (BIC), 550 Bayesian network, 316, 401, 548 dynamic, 401 bicubic interpolation, 454 bigram text file, 278 binarized silhouette, 413 binary decision tree, 316 629
630
binary tree representation, 502 biological vision systems, 465 biometrics, 412 biorthogonal wavelet, 326 blind source separation, 366 blood vessel detection, 530 bootstrap procedures, 20 bound on expected error, 489 boundary method 8 bounding box (BB) method, 260 brain function, 365 breakpoint, 180 breast cancer detection, 303 brightness gradient, 414 bunch graphs, 468 compound, 470 camera calibration, 115,515 camera coordinate system, 120 camera modeling, 115, 119 cartographic system, 605 case-based reasoning (CBR), 95 CBR maintenance, 102 cellular automata, 208 CENPARMI IRIS database, 271 central moments, 182 Centre of a maximal ball (CMB), 141, 167 centroid of blob as feature, 506 Chamfering, 171 change of lighting condition, 445, 602 character segmentation, 254 Chinese character OCR, 241 Chinese character recognition, 241 classification-instantiation, 567 classification risk, 44 classifier Bayes, 3, 12, 500, 585 combining, 14 Fisher, 8
^-nearest neighbor, 8 linear, 8, 11 logistic, 6 minimum Euclidean distance, 234 multi-class, 12 multi-level HMM, 395 non-linear, 12 normal-based linear, 8 normal-based quadratic, 7 off-the-shelf, 22 one-class, 3, 13 plug-in Bayes, 7 quadratic, 7 support vector, 11 two-class 5 "weak", 14 coarse to fine strategy, 463, 482 color, 471,545,577 color-histogram, 553 CMUsetl, 487 combining multiple feature types, 469 compactness, 507 compactness hypothesis, 4 complex system modeling, 559 complex-valued fMRI data, 365 computed tomography (CT) image, 96 computer-aided diagnosis (CAD) system, 303 connected components, 153,220,261 of graphs, 294 connectedness, 140 cononical representation affine, 131 Euclidean, 133 projective, 129 content-based classification, 288 content-based video analysis, 543 content segmentation, 547 contrast ratio, 305
control point, 516 convex hull, 185, 188,260 co-occurrence statistics, 199,502 co-registration, 349 corner detection, 74 correspondence between object and template, 499 CPNView, 586 cross-correlation, 77 cross-entropy, 203 cross-validation, 20 curse of dimensionality, 245 curvature extrema, 180 database search and query, 605 decision tree, 583 withC4.5, 583 deformable feature graphs, 461 detection of curvilinear structure, 525 detection rate false negative (FNR), 531 false positive (FPR), 318,530 true positive (TPR), 318, 530 detector entropy, 86 Harris-Laplacian, 73 scale invariant, 72 digital distance transform, (see distance transform) digital mammogam, (see mammogram) directional Hamming distance, 529 discriminant function, 45 discriminative training, 42 dissimilarity measure, 21,203 distance city block, 138 D6 distance, 141 Euclidean, 149, 181,261,532
local, 159 distance measures, 259 combined, 266 graph-based, 291 modified, 262 vector-based, 290 distance transform (DT), 138,157 chessboard, 158 city block, 161 Euclidean, 158, 165 octagonal, 161 path generated, 159 reverse (rDT), 172 salience, 174 weighted, 159, 169 document analysis by, 219 hierarchical methods, 226 non-hierarchical methods, 226 document layout analysis, 253 document structure, 219 geometric, 219 logical, 219 document understanding, based on 2 description language, 231 formatting knowledge, 230 tree transform, 229 dynamic programming, 463 circular, 622 dynamic system of 2-D images, scale , 3 3 3 wavelet-packet-based, 337 dynamic time-warping, 416 edge coherence, 527 edge detection, 356, 525, 579 eigengait, 413 eigen-smoothed width vector, 415 Elastic Graph Matching (EGM), 463 elastic matching, 465
632
elongatedness, 167 embedded Intel X-Scale Platform, 602 emotion recognition, 387 from physiological signals, 397 multimodal, 387 vocal, 396 emotional expression, 389 ensemble averaging, 29 entities, 560 entropy harmony, 589 pitch, 589 sonority, 588 epipolar constraint, 124 error curves, 21 essence, 561 event detection, 499 exhibition-characterization, 566 experimental evaluation, 527 feature-based, 527 task-based, 527 Euclidean geometry, 115 example-based learning, 446 Expectation-Maximization, 19, 26, 82, 420, 550 extended trinocular vision, 626 extensible Markup Language (XML), 229 face analysis, 212 face recognition, 412,445 Facial Action Coding System (FACS), 394 facial expression, 389 facial images, 454 false alarms, 362 fast face detection, 481 fastICA, 367
feature affine-invariant, 80 blob-like affine, 74 geometric, 71 intensity, 312 high-level, 91 local invariant, 71 low-level, 97 shape, 313 symbolic, 96 texture, 314 feature-based tracking, 500 feature constellation technique, 463 feature extraction, 17, 244, 304, 432, 506 feature integration, 500 feature matching, 436 feature ranking, 488 feature reduction, 481 feature selection, 16, 304, 315, 413, 500, 590 stepwise, 315 feature space, 4, 583 dimensionality, 4 Fisher discriminant, 583 Fisher ratio, 8,591 fitting, 183 RMS, 183 Least Median of Squares, 184 floating forward selection (FFS), 591 form document processing, using 219 Form Description Language, 233 wavelet transform approach, 232 Fourier descriptors, 181 Frame-to-exemplar distance vector (FED), 419 frequency representations, 295
633
Functional magnetic resonance imaging (fMRI), 365 Fundamental matrix, 78, 124 fusion at data level, 347 fusion at feature level, 347 fuzzy region growing, 310 gait-based human recognition, 413 gait cycle, 414 Gabor features, 474 Gabor filter, 252,435 Gabor jet, 466 Gabor wavelet transform, 466 Gaussian mixture model, 249 generalization-specialization, 565 Generalized Symmetry Operator, 413 Generative Topographic Mapping (GTM), 18 genuine and imposter distributions, 443 geometric complexity, 224 geometric distances, 474 geometric primitives, 183 geometric structure, 226 gesture, 389 gesture recognition, 463, 472 global shape measures, 177 graph-based kNN approach, 297 graphs, 289 grids, non-isotropic, 170 rectangular, 167 ground truth (GT), 389,527 GT based performance evaluation, 539 G-statistic, 203 hand-based biometric technology, 432 hand gesture recognition, 461 Haar wavelet features, 482
Harris-Laplace detector, 72 hemography, 117 hidden Markov model (HMM), 25, 248,411,501,549 classification, 30 continuous, 417 embedded, 395 learning, 26,28 robust parameter estimation, 28 topology or structure, 27 hierarchical classification, 481 hierarchical methods, 226 hierarchical object representations, 478 high-level knowledge discovery, 546 high-level semantics, 548 high-resolution image, 445 high-resolution imaging (HRJ), 347 high-resolution reconstruction, 446 HMM-MLP hybrid system, 260 holistic approach, 559 Homeland Security, 601 HSI and HRI data fusion, 352 human-computer interaction (HCI), 387,499 human gait, 411 human vision, 273 hyperspectral image (HSI) sensors, 347 image enhancing, 580 image fusion, 325 multiresolution, 325 multisensor, 347 image sharpening algorithms, 353 image interpretation, 96 logical, 96 model-based, 104 numerical, 96 image interpretation system, 95
634
Image Processing system model, 577 image retrieval, 211 content-based (CRIR), 552 image segmentation, 97, 306 image signature, 80 imagery design, 533 independent component analysis (ICA), 365 group, 375 kernel, 367 non-parametric, 367 spatial, 370 ICA feature extraction, 367 indicator function, 45 industrial visual inspection, 210 information entropy theory, 242 information system, 219 Informax principle, 367 integrated segmentation and recognition (ISR), 463 integrating simple features, 488 Intelligent Transportation System (ITS), 601 invariant recognition, 461 iris filter, 309 iterative error back-projection, 452 iterative pixel classification, 307 iterative thinning, 138 Kalman filtering, 325 Extended (EKF), 499, 522 scale recursive, 329 wavelet-packet based, 326 kernel function, 12 Gaussian, 12 Parzen, 48 smoothed 0-1 loss, 52 yfc-mean clustering , 289, 293, 583 ^-nearest neighbors, 292, 583
knowledge-based approach, 431 knowledge discovery, 549 Kruppa equations, 135
Labeled feature graphs, 465 Landsat TM image, 325 Laplacian-of-Gaussian (LoG) operation, 72 law enforcement, 601 learning algorithm performance, 31 learning in CBR system, 108 least square minimization, 451 leave-one-out estimation, 19 legibility of type-fonts, 273 license plate, 601 line detection, linear discriminant analysis (LDA), 19, 245,316,591 linear non-orthogonal transformation, 365 links, 560 effect, 570 input and output, 569 procedural, 560 results and consumption, 571 state-specified result and consumption, 574 transformation, 569 local binary pattern (LBP) operator, 197 multi-resolution, 206 opponent color (OCLBP), 209 local contrast measure, 197 logic filter, 309 logical structure, 225 look-up table (LUT), 342 low-level features, 548 low-level video content analysis, 544
635
low-resolution image, 445 macrotexture, 199 magnetic resonance angiography (MRA) image, 139 MAP (maximum aposterior probability), 26 Entropic estimator, 29 Mahalanosis distance, 78, 500 mammograms, 303 Markov random field (MRF) model, 199,307 mass detection, 303 matching centroid, 508 color, 508 shape, 508 machine segmentation (MS), 528 material identification, 347 maximal ball, 160 maximum likelihood estimation (MLE), 41 Maximum Mutual Information (MMI) approach, 42 mean reconstruction error, 415 measure of chirality, 190 circularity, 182 convexty, 188 eccentricity, 182 ellipticity, 183 elongatedness, 182 rectangularity, 185 sigmoidality, 187 squareness, 183 symmetry, 189 triangularity, 185 medical imaging, 325 microcalcification detection, 303
micro-texton, 200 middle level detectors, 548 MIDI-representation, 586 minimum classification error (MCE) recognition, 41 loss function, 44 minimum Hamming distance, 439 mining, data, 550 exception, 551 feature, 550 frequent, 551 move, 551 sequential, 551 mirror, conic, 624 hyperboloidal, 624 parabola, 624 spherical, 624 mobile robot navigation, 627 model selection algorithm, 78 modified fractal signature, 228 moment invariants, 77,177 momentum terms, 10 monocular video sequence, 411 "mood" of music, 585 morphable face model, 447 extended, 447 motion segmentation, 504 moving platform, 602 MPI face database, 458 multi-camera tracking, 501 multidimensional scaling, 298 multijet, 549 Multi-Layer Perceptron, 42 multimedia application of ODVS, 625 multimodal analysis, 544 multimodal context-sensitive system, 406
636
multimodal feature fusion, 547 multinet, 549 multiple moving objects, 499 multi-feature integration, 500 multi-resolution analysis (MRA), 327 multi-resolution LBP, 206 multi-scale techniques, 72, 197,305, 311 multi-sensor data fusion, 503 multi-sensor measurements, 325 music features, 587 musical style recognition, 583 mutual information measures, 298 neural network, 3, 310, 316, 395 multi-layered feed-forward, 9 Probabilistic (PNN), 42 Radial Basis Function (RBF), 316 NIST database, 426 non-Gaussianity, 372 non-parametric classification, 203 non-rigid deformation, 79, 468 non-rigid texture, 79 object, 222,560 object occlusion, 445 Object-Process Methodology (OPM), 559 behavior modeling, 568 ontology, 559 states, 562 structural modeling, 562 object representation, 461 object tracking, 508 omnidirectional stereo, 622 omnidirectional vision, 619 on-line vector quantization, 502 ontology/taxonomy structure, 549 opponent color LBP, 209
optical center, 122 optical flow, 394 Optical Character Recognition (OCR), 220,247,273, 277, 601 Outdoor (0 2 CR), 602 order stochastic filtering, 228 palmprint authentication system, 431 palmprint image, 441 panoramic image, 620 paper inspection, 211 partial object description, 478 pattern recognition system, 4 Parzen density, 8, 596 Parzen window, 41, 42 "memory-based", 42 partial occlusion, 474 partially exposed target, 349 performance of segmentation algorithms, 526 perimeter, 182, 187 peripheral branch, 154 PETS2001 database, 571 photometric information, 71 physiology-related signal, 371, 389 pose estimation, 469 pre-processing, 304, 434 primitive-based model, 199 principal component analysis, 16, 350, 365, 448,488 for feature extraction, 17 print fonts, 273 prior information for ICA, 365, 377 probabilistic based network, 548 probabilistic graphic models, 387 process, 560 projective geometry, 115 projective transformation, 129 PRTools, 586
637
pruning, 139, 153 pulsed LED illuminator, 607 quadratic discriminant function (QDF), 246 modified, 247 range image sets, 535 Receiver Operating Characteristics (ROC), 86,317,439,487 free response (FROC), 317 recognition-based segmentation, 260 reconstruction, 128 affme, 129 Euclidean, 131 projective, 128 region clustering, 307 region descriptors, 181 regularization parameters, 11 remote sensing, 325 resampling, 14 re-segmentation approach, 260 Resource Description Framework (RDF), 565 re-training, 485 rotation invariant, 197,204 rotation invariant descriptor, 76 rule-based classification method, 397 run length/Euclidean with heuristics (RLEH), 260 scanner-related signal, 371 scattering matrix within, 9 between, 9 scene analysis, 475 second moment matrix, 75 eigenvalues, 75 segmentation, 177, 462
moving object, 503 range image, 535 region, 526 self-organizing map (SOM), 18, 203, 585 semantic model, 544 semantic understanding, 548 semantics, 559 semi-local constraints, 465 shape, 157, 177 shape decomposition, 178 shot, 545 SIFT descriptor, 77 signal (of interest in fMRI) function-related, 371 task-related, 371 transiently task-related, 371 similarity, 105 overall, 105 partial, 105 similarity function, 467 similarity measure, 105 singular value decomposition (SVD), 553 skeleton branch, 153 skeletonization, 137 curve, 137 surface, 137 solid object, 140 sonorities, 583 spatial gap inter-character, 264 inter-letter, 284 inter-word, 264 spatial gap distance, 261 spatial resolution, 347 spatial-spectral analysis, 357 spatial-temporal structure of signal, 373
638
spatio-temporal recognition, 25, 32 spectral angle, 356 spectral dimensionality reduction, 349 spectral distance, 356 speech recognition, 43 SPOT images, 341 state transition probability, 26, 36 statistical pattern recognition, 3, 96, 583 stereo image pairs, 475 string grammar Chomskian, 56 trajectory guided contextual, 70 stroke structure analysis, 242 structure-behavior integration, 560 structure from motion (SfM), 422 style of composers, 583 stylometry, 584 support vectors, 483 support vector machine (SVM), 248, 482 linear, 482 nonlinear, 482 synthesis/analysis model, 374 synthetic-aperture radar (SAR), 347 syntactic method, 55 tactile, 389 template matching, 310, 416 texton, 199 texture analysis, 197 texture descriptors, 79 texture measure statistical, 199 structural, 199 unified, 200 texture primitives, 199
token-based approach, 431 topology preserving removal operation, 138 touching characters, 254 tracking, 424 tracking moving objects, 499 training/test set partitioning, 534 transformation, 561 affine, 118, 183 inverse, 183 projective, 117 similarity, 118 tree, 229 UMD3 database, 424 unlabeled data, 387 unsupervised classification, 592 USF database, 416 video abstraction, 544 video browsing, 552 video content management and access, 543 video gesture recognition, 29 video knowledge discovery, 543 video processing feature based, 395 region based, 395 video retrieval system, 553 video search engine, 554 video skimming, 544 video summarization, 544 video (visual) surveillance, 445, 501 view invariant gait recognition, 411 visual sensory mechanisms, 619 Viterbi algorithm, 28 Viterbi Path Counting, 29
voxels, 138 border, 134 curve, 143 edge, 144 induced skeletal, 143 internal, 144 intrinsic, 145 junction, 144 saddle, 142 significant, 139 skeletal, 145 tunnel, 145
wave-front propagation, 173 weakly supervised, 85 web documents, 229, 287 graph representation, 288,295 wide viewing range, 627 width vector profile, 414 wireless-LAN link, 610 wood inspection, 210 word extraction, 264 world coordinate system, 121
