THE DISSIMILARITY REPRESENTATION FOR PATTERN RECOGNITION Foundations and Applications
SERIES IN MACHINE PERCEPTION AND ARTIFICIAL INTELLIGENCE* Editors:
H. Bunke (Univ. Bern, Switzerland) P. S. P. Wang (Northeastern Univ., USA)
Vol. 50: Empirical Evaluation Methods in Computer Vision (Eds. H, 1. Christensen and P. J. Phillips) Vol. 51: Automatic Diatom Identification (Eds. H. du Buf and M. M. Bayer) Vol. 52: Advances in Image Processing and Understanding A Festschrift for Thomas S. Huwang (Eds. A. C. Bovik, C. W. Chen and D. Goldgof) Vol. 53: Soft Computing Approach to Pattern Recognition and Image Processing (Eds. A. Ghosh and S. K. Pal) Vol. 54: Fundamentals of Robotics - Linking Perception to Action (M. Xie) Vol. 55: Web Document Analysis: Challenges and Opportunities (Eds. A. Antonacopoulos and J. Hu) Vol. 56: Artificial Intelligence Methods in Software Testing (Eds. M. Last, A. Kandel and H. Bunke) Vol. 57: Data Mining in Time Series Databases y (Eds. M. Last, A. Kandel and H. Bunke) Vol. 58: Computational Web Intelligence: Intelligent Technology for Web Applications (Eds. Y, Zhang, A. Kandel, T. Y. Lin and Y. Yao) Vol. 59: Fuzzy Neural Network Theory and Application (P.Liu and H. LI) Vol. 60: Robust Range Image Registration Using Genetic Algorithms and the Surface Interpenetration Measure (L. Silva, 0. R. P. Bellon and K, L. Boyer) Vol. 61 : Decomposition Methodology for Knowledge Discovery and Data Mining: Theory and Applications (0.Maimon and L. Rokach) Vol. 62: Graph-Theoretic Techniques for Web Content Mining (A. Schenker, H. Bunke, M. Last and A. Kandel) Vol. 63: Computational Intelligence in Software Quality Assurance (S. Dick and A. Kandel) Vol. 64: The Dissimilarity Representation for Pattern Recognition: Foundations and Applications (flzbieta Pekalska and Roberi P. W. Duin) Vol. 65: Fighting Terror in Cyberspace ( f d s . M. Last and A. Kandel)
*For the complete list of titles in this series, please write to the Publisher.
THE DISSIMILARITY REPRESENTATION FOR PATTERN RECOGNITION Foundations and Applications
Elibieta Pekalska
Robert P. W. Duin
Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology Delft, The Netherlands
vp World Scientific NELVJERSEY
*
LONDON
*
SINGAPORE
-
RElJlNG
-
ShAYGHAI
-
HONG K O N G
-
TAIPEI
-
CHEWKAI
Published b-y
World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA qflce: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
U K @ice: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
THE DISSIMILARITY REPRESENTATION FOR PATTERN RECOGNITION Foundations and Applications Copyright 02005 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or p u t s thereoj may not be reproduced in any,form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval s.ysrem now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN 981-256-530-2
Printed by FuIsland Offset Printing (S) Pte Ltd, Singapore
To t h e ones who ask questions and look for answers
This page intentionally left blank
Preface
Progress has not followed a straight ascending line, h u t a spiral with rhythms of progress and retro,qression, o,f evolution and dissolution. JOHANN WOLFGANG
VON
GOETHE
Pattern recognition is both an art and a science. We are able to see structure and recognize patterns in our daily lives and would like to find out how we do this. We can perceive similarities between objects, between people, between cultures and between events. We are able to observe the world around us, to analyze existing phenomena and to discover new principles behind them by generalizing from a collection of bare facts. We are able to learn new patterns, either by ourselves or with the hclp of a teacher. If we will ever be able to build a machine that does the same, then we will have made a step towards an understanding of how we do it ourselves. The two tasks, the recognition of known patterns and the learning of new ones appear to be very similar, but are actually very different. The first one builds on existing knowledge, while the second one relies on observations and the discovery of underlying principles. These two opposites need to be combined, but will remain isolated if they are studied separately. Knowledge is formulated in rules and facts. Usually, knowledge is incomplete and uncertain, and modeling this uncertainty is a challenging task: who knows how certain his knowledge is, and how can we ever relate the uncertainty of two different experts? If we really want to learn something new from observations, then a t least we should use our existing knowledge for their analysis and interpretation. However, if this leads to destruction of all inherent organization of and relations within objects themselves, as happens when they are represented by isolated features, then all what is lost by (not incorporated in) the representation has to be learned again from the observations. These two closely related topics, learning new principles from observavii
...
Vlll
T h e dissinailarity representation for p a t t e r n recognition
tioris and applying existing knowledge in recognition, appear to be hard to combine if we concentrate on these opposites separately. There is a need for an integrated approach that starts in between. We think that the notion of proximity between objects might be a good candidate. It is an intuitive concept that may be quantified and analyzed by statistical means. It does not a priori tear the object into parts, or more formally, does not require neglecting the inherent object structure. Thereby, it offers experts a possibility to model their knowledge of object descriptions and their relations in a structural way. Proximity may be a natural concept, in which the two opposites, statistical and structural pattern recognition, meet. The statistical approach focuses on measuring characteristic numerical features and representing objects as points, usually in a Euclidean or Hilbert feature space. Objects are different if their point representations lie sufficiently far away from each other in this space, which means that the corresponding Euclidean distance between them is large. The difference between classes of objects is learned by finding a discrimination function in this feature space such that the classes, represented by sets of points, are separated as well as possible. The structural approach is applicable to objects with some identifiable structural organization. Basic descriptors or primitives, encoded as syntactic units, are then used to characterize objects. Classes of objects are either learned by suitable syntactic grammars, or the objects themselves are compared by the cost of some specified match procedure. Such a cost expresses the degree of difference between two objects. One of the basic questions in pattern recognition is how to tell the differencc between objects, phenomena or events. Note that only when the difference has been observed and characterized, similarity starts to play a role. It suggests that dissimilarity is more fundamental than similarity. Therefore, we decided to focus more on the concept of dissimilarity. This book is concerned with dissimilarity representations. These are riurncrical representations in which each value captures the degree of' conimonality between pairs of objects. Since a dissimilarity measure can be defined on arbitrary data given by collections of sensor measurements, shapes, strings, graphs, or vectors, the dissimilarity representation itself becomes very general. The advantages of statistical and structural approaches can now be integrated on the level of representation. As the goal is to develop and study statistical learning methods for dissimilarity representations, they have to be interpreted in suitable mathematical frameworks. These are various spaces in which discrimination
Preface
ix
functions can be defined. Since non-Euclidean dissimilarity measures are used in practical applications, a study beyond the traditional use of Euclidean spaces is necessary. This caused us to search for more general spaces. Our work is founded on both mathematical and experimental research. As a result, a trade-off had to be made to present both theory and practice. We realize that the discussions may be hard to follow due to a variety of' issues presented and the necessary brevity of explanations. Although some foundations are laid, the work is not complete, as it requires a lot of research to develop the ideas further. In many situations, we are only able to point to interesting problems or briefly sketch new ideas. We are optimistic that our use of dissimilarities as a starting point in statistical pattern recognition will pave the road for structural approaches to extend object descriptions with statistical learning. Consequently, observations will enrich the knowledgebased model in a very generic way with confidences and natural pattern classifications, which will yield improved recognition. This book may be useful for all researchers and students in pattern recognition, machine learning and related fields who are interested in the foundations and application of object representations that integrate structural expert knowledge with statistical learning procedures. Some understanding of pattern recognition as well as the familiarity with probability theory, linear algebra arid functional analysis will help one in the journey of finding a good representation. Important facts from algebra; probability theory and statistics are collected in Appendices A D. The reader may refer to [Fukunaga, 1990; Webb, 1995; Duda et al., 20011 for an introduction to statistical pattern recognition, and to [Bunke and Sanfeliu, 1990; Fii, 19821 for an introduction to structural pattern recognition. More theoretical issues are treated in [Cristianini and Shawe-Taylor, 2000; Devroye et al.: 1996; Hastie et al.; 2001; Vapnik, 19981, while a practical engineering approach is presented in [Nadler and Smith, 1993; van der Heiden et d., 20041. Concerning mathematical concepts, some online resources can be used, such as http: //www .probability.net/, http: //mathworld.wolfram. corn/, http : //planetmath. org/ and http : //en. wikipedia. org/. ~
Credits. This research was supported by a grant from the Dutch Organization for Scientific Research (NWO). The Pattern Recognition Group of the Faculty of Applied Sciences at Delft University of Technology was a fruitful arid inspiring research environment. After reorganization within the university, we could continue our work, again supported by NWO, in the
X
T h e dissamilarity r e p r e s e n t a t t o n for patterm. recognttton
Information and Communication Theory group of the Faculty of Electrical Engineering, Mathematics and Computer Science. We thank both groups and their leaders, prof. Ian T. Young and prof. Inald Lagendijk, all group members and especially our direct colleagues (in alphabetic order): Artsiom Harol, Piotr Juszczak, Carmen Lai, Thomas Landgrebe, Pavel Paclik, Dick de Ridder, Marina Skurichina, David Tax, Sergey Verzakov and Alexander Ypma for the open and stimulating atmosphere, which contributed to our scientific development. We gained an understanding of the issues presented here based on discussions, exercises in creative thinking and extensive experiments carried out in both groups. We are grateful for all the support. This work was finalized, while the first author was a visiting research associate in the Artificial Intelligerice group at University of Manchester. She wishes to thank for a friendly welcome. This book is an extended version of the PhD thesis of Elzbieta Pekalska and relies on work published or submitted before. All our co-authors are fully acknowledged. We also thank prof. Anil Jain and Douglas Zongker, prof. Horst Biinke arid Simon Gunter, Pavel Paclik and Thomas Landgrebe, and Volker Rotli for providing some dissimilarity data. All the data sets are described in Appendix E. The experiments were conducted using PRTools [Diiiri et ul., 2004b1, DD-tools [Tax, 20031 and own routines. To the Reader. Our main motivation is to bring the attention to the issue of representation and one of its basic ingredient: dissimilarity. If one wishes to describe classes of objects within our approach this requires a mental shift from logical and quantitative observations of separate features to an intuitive and possibly qualitative perceiving of the similarity betwcen objects. As the similarity judgement is always placed in some context, it can only be expressed after observing the differences. The 1110ment the (dis)similarity judgements are captured in some values, one may process from whole (dissimilarity) to parts (features, details or numerical descriptions). As a result, decision-theoretic methods can be used for learning. The representation used for the recognition of patterns should enable integration of both qualitative and quantitative approaches in a balanced manner. Only then, the process of learning will be enhanced. Let it be so. Dear Reader, be inspired! Wishing You an enjoyable journey,
Elibieta Pqkalska and Robert P.W. Duin, Chester / Delft, June 2005.
Notation and basic terminology
Latin symbols
matrices, vector spaces, sets or random variables scalars, vectors or object identifiers vectors in a finite-dimensional vector space basis vector estimated mean vectors Gram matrix or Gram operator estimated covariance matrix dissimilarity function, dissimilarity measure dissimilarity matrix functions identity matrix or identity operator centering matrix number of clusters space dimensions kernel number of objects or vectors, usually in learning neighborhood sets i-th object in the representation set R probability function projection operator, projection matrix or probability orthogonal matrix representation set R = { P I ,p 2 , . . . ,p,} similarity function, similarity measure similarity matrix or stress function i-th object in the training set T training set T = { t l , t a , . . , tiv} weight vectors
xi
The dissimilarity representation for pattern recognition
xii
Greek symbols scalars or parameters vectors of parameters Kronecker delta function or Dirac delta function evaluation functional dissimilarity matrix used in multidimensional scaling trade-off parameter in mathematical programming field, usually R or C, or a gamma function regularization parameter i-th eigenvalue diagonal matrix of eigenvalues mean, probability measure or a membership function mean vector mappings covariance matrix dissimilarity function set or a closed and bounded subset of Rm
Other symbols
A
c cm 2)
3
G, K: 'FI
z J
Q
w R+
% R"
ST I
u.v,x z
a-algebra set of complex numbers m-dimensional complex vector space domain of a mapping set of features Krein spaces Hilbert space indicator or characteristic function fundamental symmetry operator in Krein spaces set of rational numbers set of real numbers set of real positive numbers
R+ u (0) rn-dimensional real vector space m-dimensional spherical space, m+l 2 2 = {xERm+l: 2 , - 7- } transformation group subsets. subspaces or random variables set of integers
s,m
cz=l
xiii
Notation and basic terminology
Sets and pretopology A, D, . . . , Z
sets
( A i , A z , .. . , A n } cardinality of A generalized interior of A generalized closure of A set union of A and B set intersection of A and B set difference of A and B set symmetric difference, AAB = (A\B) U (B\A) Cartesian product,, A x B = { ( a ,b ) : ~ E AA b E B } power set, a collection of all subsets of X neighborhood system neighborhood basis neighborhood, pretopological or topological space defined by the neighborhoods n/ (pre)topological space defined by a neighborhood basis neighborhood, pretopological or topological space defined by the generalized closure algebraic dual space of a vector space X continuous dual space of a topological space X generalized metric space with a dissimilarity p metric space with a metric distance d &-ballin a generalized metric space (X, p ) , B E ( Z )= { y E X : p ( y 9 z )< F }
collection A of subsets of the set s2 satisfying: (1) Q E A , (2) A E A=+ (R\A)cA, (3) ( V k A k E A A A = U g I A k ) * A E A p : A 4 RO,is a measure on a a-algebra A if p ( @ )= 0 and p is additive, i.e. 1-1 Ak) = C kp ( A k ) for pairwise disjoint sets Ak measurable space; R is a set and A is a a-algebra measure space; /I, is a measure probability space normal dist,ribution with the mean vector p and the covariance matrix C probability of an event A E R
(uk
P(A)
The dissimilarity representation f o r p a t t e r n recognition
conditional probability of A given that B is observed likelihood function is a function of 6’ with a fixed A, such that L(0lA) = cP(AIB = 6’) for c>O expected value (mean) of a random variable X defined over ( 0 ,A, P ) ;E [ X ]= xdP variance of a random variable X , V ( X ) = E[(X-E[X])Z] standard deviation of a random variable X ,
s,
4x1 = d r n
k-th central moment of a random variable X ,
“-wl)kl
Pk(X) = cumulative distribution function probability deriisty function
Mappings and functions &:X+Y
4 is a mapping
(function) from
X
to
Y;
X is the domain of q5 and Y is the codomain of q5 range of 4 : X + Y ,R$= { ~ E YjZrx : y = 4(x)} 407 injection
surjectioii bijection homomorphism eridomorphism isomorphism automorphisrri monomorphism linear form functiorial irri ( 4 ) ker ( 4 ) concave function corivex function logistic function logarithmic function
composition of mappings 4 : X ---t Y such that (21# Z Z ) + (4(21)#4(xz)) holds for all ~ 1 ~ ExX2 ; 724 # Y 4 : X + Y ,X onto Y , such that 724 = Y injection which is also a surjection linear mapping froni one vector space to another linear mapping from a vector space to itself homomorphism which is a bijection endoniorphism which is an isomorphism homomorphism which is an injection homomorphism from a vector space X to the field r linear form iniage of a homomorphism 4 : X + Y ,724 kernel of a homomorphism 4: X + Y ker(q5) = { z E X :q5(x) = 0} f ( CY 2 1 (1- C Y ) Z Z ) 2 ~ f ( 2 2 ) (1- ~ ) f ( 2 2 holds ) for all 2 1 , xz E Df and all a: E [0,1] f is convex iff -f is concave f ( x ) = 1/(1+ e x p ( - c z ) ) f ( x ) = log(z); here log denotes a natural logarithm
+
+
Notation and basic terminology
sigmoid function gamma function
xv
f(x) = 2 / ( l + exp(-z/a)) - 1 st-ie-zds,t > o
r ( t )=
Jr
Vectors and vector spaces
u,v,x,y, z Z = X x Y Z=X$I'
Z=X@Y
V. W , X ,
Y,Z
{xt1I=1 0 1 ei
XT X+ X'Y
xi Y X* X' C(X7
L ( X ,y > CJX, r) LJX, Y )
vector spaces Cartesian product of vector spaces direct sum of vector spaces; each z E Z can be uniquely decomposed into z E X and y E Y such that z = .r y and X n Y = (0) tensor product of vector spaces; for any vector space U and any bilinear map F : X x Y + U , there exists a bilinear map H : Z 4 U such that F ( x ,y) = H ( x @ y) for all z E X and y E Y vectors in finite-dimensional or separable vector spaces {Xi x2 7 . . . , x n 1 column vector of zeros column vector of ones standard basis vector, e , = 1 and e3 = 0 for j # 1: transpose of a real vector conjugate transpose of a complex vector inner product of vectors in R'" inner product of vectors in Cm algebraic dual space of a vector space X cont,inuous dual space of a topological space X space of linear functiorials from X onto the field I?, equivalent to algebraic dual X * space of linear operators from X onto Y space of continuous linear functionals from X onto the field r, equivalent to continuous dual X ' space of continuous linear operators from X onto Y
+
1
Inner product spaces and normed spaces
L:
closed and bounded set, R c R" set of all functions on R set of all continuous functions on R set of function classes, Lebesgue measurable: on f 2 L; = { f E C ( R ) : (J, I f ( z ) l P dz)i/p< oo}, p 2 1
LF
LpM
0
an) C(R) M(R)
=
{ f E M ( R ) : (J, I f ( z ) l P p ( d ~ ) ) ~ < / Pm}, p 2 1
The dzssimilarity representatzon f o r p a t t e r n recognition
inner product norm &norm of X E I W ~ lIxIJP , = (Czl)x,)*)~/P, p 21 !,-norm of f E L F ; llf /I” = (J, If ()”.I dz)l’p, P L 1 space X equipped with the inner product (., .) space X equipped with the norm I / . I / space X equipped with the dissimilarity p orthogonal complement to X Hilbert space reproducing kernel Hilbert space with the kernel K Banach space (Rm,/ I . lip), p 2 1 Banach space (Rm, / I . l i p ) . p >. 1 Indefinite inner product spaces Hilbert spaces (Ic+. (., .)) and (Ic-, -(., .)) Krein space, Ic = Ic+ @ K - and Ic- = Ic; Hilbert space associated with a Krein space K IIcl = Ic+ @ IIc-1, where Ic- = K i and 1K-l = (L, (.. .)) pseudo-Euclidean space with the signature ( p ,q ) inner product in a K r e h space K inner product in a pseudo-Euclidean space E reproducing kernel Krein space with the kernel K fundamental projections identity operator in a Krein space; I = P+ + Pfundamental symmetry in a K r e h space; J = P+ - Pfundamental symmetry in Iw(P14) H-scalar product, [z, y] = (3%. y ) ~ H-norm, IlxlirL = [I.+ Operators in inner product spaces and normed spaces
( u t J ) matrix or an operator A with the elements atJ 2-th row of a matrix A j - t h column of a matrix A a I,A determinant of a matrix A det(A) A*B Hadaniard product, A * B = (aLII btJ) A*” Hadaniard power, A*P = ( a f J ) * .B Hadaniard power, a*B = ( n b 7 2 )where , UER AT transpose of a real matrix A
A
=
02. , A ,
Notation and basic terminology
At AX A hermitian A symmetric
A orthogonal A unitary A cnd
A cpd
A nd A nsd A Pd A psd
xvii
conjugate transpose of a complex niatrix A adjoint A in a Hilbert space; A X = AT or A X = At A = At A = AT A A T = I and ATA = I A A t = I and AtA = I A = At is conditionally negative definite if x t A x 5 0 and x t 1 = O for x # 0 A = At is conditionally positive definite if x t A x 2 0 and x t 1 = o for x # o A = At is negative definite if x t A x < 0 for x # 0 A = At is negative sernidefiriite if x t A x 5 0 for x # 0 A = At is positive definite if x t A x > 0 for x # 0 A = At is positive semidefinite if x t A x 2 0 for x # 0
Operators in indefinite inner product spaces
A*
A A A A A
J-self-adjoint 3-isomctric J-coisometric J-symmetric J-unitary
space of continuous linear functionals from a Kreiri space K into thc field I? space of continuous linear operators from a Krein space Ic into a Krein space G adjoint of an operator A, A t C(K,G ) is such that (A f ,g)B = ( f .A * ~ ) holds K for all f E K a11d 9 E
A = A* A E G ( K , G ) is isometric if A*A = Ic A E C(K, G ) is coisometric if AA* = 1, (4, g)lc = ( f >A S ) K for all f , g IC ( A f , A!dK = ( f , S ) K for all f , g K
Dissimilarities d
D D*2 D ( T ,R ) S d2
D E ,D2 dP DP dmax
dissimilarity measure dissimilarity matrix, D = ( d t J )
D*2 = ( d ; J ) dissimilarity representation similarity matrix, S = (sz7) Euclidean distance Euclidean distance matrix .$-distance &distance matrix &-distance
xviii
T h e dissimilarity representation for p a t t e r n recognition
&distance matrix Hausdorff distance modified-Hausdorff distance square Mahalanobis distance Levenhstein distance, normalized Levenhstein distance Kullback-Leibner divergence J-coefficient information radius divergence Bhat t acharayya distance Chernoff distance Hellinger coefficient Tversky dissimilarity and Tversky similarity cut semimetric based on the set V Graphs and geometry
cut on X G = (V,E) ad,jacent nodcs linear hull cone convex hull hyperprism
hypercylinder
partition of a set X into V arid X\V graph with a set of nodes V and a set of edges E = { ( u , w )u: , u E V } two nodes in a graph joined by an edge huiir(x)={C;=l p t z Z :X , E V A ~ , E I ' } , ~ C R { X : huIlR+( X ) = X} Pt J,: 2%EV, Pt 2 0 A Pt = I} figure generated by a flat region in Rm, moving parallel to itself along a straight line hyperprisni for which the flat region in Rm-' is a hypersphere
c;=1
{c:=l
rn
hypersphere hyperplane
{ x E R ~llx:ll; = R2}with the volume V = 2KkZ mr(F) and the area A = 2Rm-1T' r(?) m-dimensional hyperplane, {xERm+l:
parallelotope
polyhedron polyhedral cone polytope simplex
c : ; '
w,x,= wo}
collection of points in Rm bounded by m pairs of ( m- l)-dimensional hyperplanes (a generalization of a parallelogram) { x EA~ xP 5 b, : AER""" A bER"} {xER": A x 5 0, AEIW"~"} collection of points bounded by m-dimensional hyperplanes (a generalization of a triangle in 2D) polytope; a collection of points in Rm enclosed by (m+1) (m- 1)-dimensional hyperplanes
Abbreviations
’
iff cnd CPd nd nsd Pd Pdf psd k-CDD k-NN
NN k-NNDD AL CCA CH CL CNN
cs
CPS DS GNMC GMDD LLE LogC LP LPDD LSS MAP MDS ML MST NLC NMC NQC NN PCA
if and only if
conditionally negative definite conditionally positive definite negative definite negative semidefinite positive definite probability density function positive semidefinite &Centers Data Description &Nearest Neighbor rule Nearest Neighbors k-Nearest Neighbor Data Description Average Linkage Curvilinear Component Analysis Compactness Hypothesis Complete Linkage Condensed Nearest Neighbor Classical Scaling Classifier Projection Space Dissimilarity Space Generalized Nearest Mean Classifier Generalized Mean Data Description Locally Linear Embedding Logistic regression linear Classifier Linear Programming Linear Programming Dissimilarity data Description Least Square Scaling Maximum A Posteriori Multidimensional Scaling Maximum Likelihood Minimum Spanning Tree Normal density based Linear Classifier Nearest Mean Classifier Normal density based Quadratic Classifier Nearest Neighbor rule Principal Component Analysis
xix
xx
RKHS RKKS RNLC RNQC QC QP SL SOM SRQC
sv
SVM SVDD
so
WNMC
The dissamilarity representation for pattern recognition,
Reproducing Kernel Hilbert Space Reproducing Kernel Krein Space Reqularized Normal density based Linear Classifier Reqularized Normal density based Quadratic Classifier Quadratic Classifier Quadratic Programming Single Linkage Self-organizing Map Strongly Reqularized Quadratic Classifier Support Vector Support Vector Machine Support Vector Data Description Support Object Weighted Nearest Mean Classifier
Contents
Preface
vii
Notation and basic terminology
xi
A bbreuintions
xix
1. Introduction
1
1.1 Recognizing the pattern . . . . . . . . . . . . . . . . . . . . 1.2 Dissimilarities for representation . . . . . . . . . . . . . . . 1.3 Learning from examples . . . . . . . . . . . . . . . . . . . . 1.4 Motivation of the use of dissinlilarity representations . . . . 1.5 Relation to kernels . . . . . . . . . . . . . . . . . . . . . . . 1.6 Outline of the book . . . . . . . . . . . . . . . . . . . . . . . 1.7 In summary . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 . Spaces
1 2 4 8 13 14 16 23
2.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 A brief look a t spaces . . . . . . . . . . . . . . . . . . . . . 2.3 Generalized topological spaces . . . . . . . . . . . . . . . . . 2.4 Generalized metric spaces . . . . . . . . . . . . . . . . . . . 2.5 Vector spaces . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Normed and inner product spaces . . . . . . . . . . . . . . . 2.6.1 Reproducing kernel Hilbert spaces . . . . . . . . . . 2.7 Indefinite inner product spaces . . . . . . . . . . . . . . . . 2.7.1 Reproducing kernel Krein spaces . . . . . . . . . . . 2.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 . Characterization of dissimilarities 3.1 Embeddings, tree models and transformations . . . . . . . . xxi
25 28 32 46 56 62 69 71
85 87 89
90
The dissimilarsty representation for p a t t e r n recognition
xxii
3.2 3.3
3.4
3.5
3.6
3.7
3.1.1 Embeddings . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Distorted metric embeddings . . . . . . . . . . . . . Tree models for dissimilarities . . . . . . . . . . . . . . . . . Useful transformations . . . . . . . . . . . . . . . . . . . . . 3.3.1 Transformations in sernimetric spaces . . . . . . . . . 3.3.2 Direct product spaces . . . . . . . . . . . . . . . . . . 3.3.3 Invariance and robustness . . . . . . . . . . . . . . . Properties of dissiniilarity matrices . . . . . . . . . . . . . . 3.4.1 Dissimilarity matriccs . . . . . . . . . . . . . . . . . 3.4.2 Square distances and inner products . . . . . . . . . Linear embeddings of dissimilarities . . . . . . . . . . . . . 3.5.1 Euclidean embedding . . . . . . . . . . . . . . . . . . 3.5.2 Correction of non-Euclidean dissimilarities . . . . . . 3.5.3 Pseudo-Euclidean embedding . . . . . . . . . . . . . 3.5.4 Generalized average variance . . . . . . . . . . . . . . 3.5.5 Projecting new vectors to a n embedded space . . . . 3.5.6 Reduction of dimension . . . . . . . . . . . . . . . . 3.5.7 Reduction of complexity . . . . . . . . . . . . . . . . 3.5.8 A general embedding . . . . . . . . . . . . . . . . . . 3.5.9 Spherical enibeddings . . . . . . . . . . . . . . . . . . Spatial representation of dissimilarities . . . . . . . . . . . . 3.6.1 FastMap . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.2 Multidiniensional scaling . . . . . . . . . . . . . . . . 3.6.3 Reduction of complexity . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 . Learning approaches 4.1 Traditional learning . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Data bias arid model bias . . . . . . . . . . . . . . . 4.1.2 Statistical learning . . . . . . . . . . . . . . . . . . . 4.1.3 Inductive principles . . . . . . . . . . . . . . . . . . . 4.1.3.1 Empirical risk minimization (ERM) . . . . . 4.1.3.2 Principles based on Occam’s razor . . . . . . 4.1.4 Why is the statistical approach not good enough for learning from objects? . . . . . . . . . . . . . . . . . 4.2 The role of dissimilarity representations . . . . . . . . . . . 4.2.1 Learned proximity representations . . . . . . . . . . 4.2.2 Dissimilarity representations: learning . . . . . . . . 4.3 Classification in generalized topological spaces . . . . . . . .
90 95 95 99 99 102 103 105 105 116 118 118 120 122 124 125 127 128 129 130 132 133 135 143 144 147 148 148 151 154 156 160 163 166 171 172 175
Contents
xxiii
4.4 Classification in dissimilarity spaces . . . . . . . . . . . . . 4.4.1 Characterization of dissimilarity spaces . . . . . . . . 4.4.2 Classifiers . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Classification in pseudo-Euclidean spaces . . . . . . . . . . 4.6 On generalized kernels and dissimilarity spaces . . . . . . . 4.6.1 Connection between dissimilarity spaces and psendoEuclidean spaces . . . . . . . . . . . . . . . . . . . . 4.7 Disciission . . . . . . . . . . . . . . . . . . . . . . . . . . . .
209 211 215
5. Dissimilarity measures 5.1 Measures depending on feature types . . . 5.2 Measures between populations . . . . . . 5.2.1 Normal distributions . . . . . . . . 5.2.2 Divergence measures . . . . . . . . 5.2.3 Discrete probability distributions . 5.3 Dissimilarity measures between sequences 5.4 Information-theorctic measures . . . . . . 5.5 Dissimilarity measures between sets . . . 5.6 Dissimilarity measures in applications . . 5.6.1 Invariance and robustness . . . . . 5.6.2 Example nieasures . . . . . . . . . 5.7 Discussion and conclusions . . . . . . . . .
180 180 185 196 205
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
6 . Visualization
216 228 228 229 233 234 237 238 242 242 242 250 255
6.1 WIultidimensional scaling . . . . . . . . . . . . . . . . . . . . 257 259 6.1.1 First examples . . . . . . . . . . . . . . . . . . . . . . 6.1.2 Linear and nonlinear methods: cxamples . . . . . . . 261 267 6.1.3 Implemeritation . . . . . . . . . . . . . . . . . . . . . 6.2 Other mappings . . . . . . . . . . . . . . . . . . . . . . . . . 268 6.3 Examples: getting insight into the data . . . . . . . . . . . 274 6.4 Tree models . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 7 . Flirther da.ta exploration
7.1 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Standard approaches . . . . . . . . . . . . . . . . . . 7.1.2 Clustering on dissimilarity representations . . . . . . 7.1.3 Clustering examples for dissimilarity representations
289 290 290 295 303
xxiv
T h e dissimilarity representation for pattern recognition
7.2 Intrinsic dimension . . . . . . . . . . . . . . . . . . . . . . . 309 7.3 Sampling density . . . . . . . . . . . . . . . . . . . . . . . . 319 7.3.1 Proposed criteria . . . . . . . . . . . . . . . . . . . . 320 7.3.2 Experiments with the NIST digits . . . . . . . . . . . 325 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 8 . One-class classifiers 8.1 General issues . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Construction of one-class classifiers . . . . . . . . . . 8.1.2 Onc-class classifiers in feature spaces . . . . . . . . . 8.2 Domain descriptors for dissimilarity representations . . . . 8.2.1 Neighborhood-based OCCs . . . . . . . . . . . . . . 8.2.2 Generalized mean class descriptor . . . . . . . . . . . 8.2.3 Linear programming dissimilarity data description . 8.2.4 More issues on class descriptors . . . . . . . . . . . . 8.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Experiment I: Condition monitoring . . . . . . . . . 8.3.2 Experiment 11: Diseased mucosa in the oral cavity . . 8.3.3 Experiment 111: Heart disease data . . . . . . . . . . 8.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 . Classification 9.1 Proof of principle . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 NN rule vs alternative dissimilarity-based classifiers . 9.1.2 Experiment I: square dissimilarity representations . . 9.1.3 Experiment 11: the dissiniilarity space approach . . . 9.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Selection of t.he representation set: the dissimilarity space approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Prototype selection methods . . . . . . . . . . . . . . 9.2.2 Experimental setup . . . . . . . . . . . . . . . . . . . 9.2.3 Results and discussion . . . . . . . . . . . . . . . . . 9.2.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . 9.3 Selection of the representation set: the embedding approach 9.3.1 Prototype selection methods . . . . . . . . . . . . . . 9.3.2 Experiments and results . . . . . . . . . . . . . . . . 9.3.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . .
333 336 337 341 346 348 350 353 359 366 366 374 377 379 383 384 384 388 389 395 396 398 401 404 416 417 418 421 428
Contents
XXV
9.4 On corrections of dissimilarity measures . . . . . . . . . 9.4.1 Going more Euclidean . . . . . . . . . . . . . . . 9.4.2 Experimental setup . . . . . . . . . . . . . . . . 9.4.3 R.esults and conclusions . . . . . . . . . . . . . . 9.5 A few remarks on a simulated missing value problem . . 9.6 Existence of zero-error dissimilarity-based classifiers . . 9.6.1 Asymptotic separability of classes . . . . . . . . 9.7 Final discussion . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
428 429 430 432 439 443 . 444 451
10. Combining
453
10.1 Combining for one-class classification . . . . . . . . . . . 10.1.1 Combining strategies . . . . . . . . . . . . . . . . 10.1.2 Data and experimental setup . . . . . . . . . . . 10.1.3 Results and discussion . . . . . . . . . . . . . . . 10.1.4 Summary and conclusions . . . . . . . . . . . . . 10.2 Combining for standard two-class c1assificat)ion . . . . . . 10.2.1 Combining strategies . . . . . . . . . . . . . . . . 10.2.2 Experiments on the handwritten digit set . . . . 10.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . 10.2.4 Conclusions . . . . . . . . . . . . . . . . . . . . . 10.3 Classifier projection space . . . . . . . . . . . . . . . . . . 10.3.1 Construction and the use of CPS . . . . . . . . . 10.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 11. Representation review and recommendations
11.1 Representation review . . . . . . . . . . . 11.1.1 Three generalization ways . . . 11.1.2 Representation formation . . . . 11.1.3 Generalization capabilities . . . 11.2 Practical considerations . . . . . . . . . 11.2.1 Clustering . . . . . . . . . . . . . 11.2.2 One-class classification . . . . . 11.2.3 Classification . . . . . . . . . . .
12. Conclusions and open problems
455 456 459 462 465 466 466 468 470 473 474 475 483 485
. . . . . . . . . 485 . . . . . . . . . . 486 . . . . . . . . . . 489 . . . . . . . . . . 492 . . . . . . . . . . 493 . . . . . . . . . 495 . . . . . . . . . . 496 . . . . . . . . . 497 503
12.1 Summary and contributions . . . . . . . . . . . . . . . . . 505 12.2 Extensions of dissimilarity representations . . . . . . . . . 508 12.3 Open questions . . . . . . . . . . . . . . . . . . . . . . . . 510
The disszrnilarity representation f o r pattern. recognition
xxvi
Appendix A
515
On convex arid concave functions
Appendix B Linear algebra in vector spaces
519
B . l Some facts on matrices in a Euclidean space . . . . . . . . . 519 B.2 Some facts on matrices in a pseudo-Euclidean space . . . . 523
Appendix C
Measure and probability
527
Appendix D
Statistical sidelines
533
D.l D.2 D.3 D.4
Likelihood arid parameter estimation . . . . . . . . . . . . Expectation-maximization (EM) algorithm . . . . . . . . Model selection . . . . . . . . . . . . . . . . . . . . . . . . . PCA and probabilistic models . . . . . . . . . . . . . . . . D.4.1 Gaussian model . . . . . . . . . . . . . . . . . . . . . D.4.2 A Gaussian mixture model . . . . . . . . . . . . . . D.4.3 PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . D.4.4 Probabilistic PCA . . . . . . . . . . . . . . . . . . D.4.5 A mixture of probabilistic PCA . . . . . . . . . . .
Appendix E
Data sets
E . l Artificial data sets . . . . . . . . . . . . . . . . . . . . . . . E.2 Real-world data sets . . . . . . . . . . . . . . . . . . . . . .
. 533 . 535 .
. . .
536 538 538 539 541 542 543 545 545 549
Bihliqqraphy
561
Index
599
Chapter 1
Introduction
T h u s all h u m a n knowledge begin,s with, intuitions, then goes t o concepts, and is completed in ideas. IMVIMANUEL KANT
1.1
Recognizing the pattern
We recognize many patterns’ while observing the world. Even in a country never visited before, we recognize buildings, streets, trees, flowers or animals. There are pattern characteristics, learned before, that can be applied in a new environment. Sometimes, we encounter a place with objects that are alien t o us, e.g. a garden with an unknown flower species or a market place with strange types of fish. How do we learn these patterns so that this place will look more familiar on our next visit‘? If we take the time, we are able to learn some patterns by ourselves. If somebody shows us around, points out and explains what is what, we may learn faster and group the observations according to the underlying concepts. What is the first step in this categorization process? Which principle is used in the observations to constitut,e the first grouping? Are these descriptive features like color, shape or weight? Or is it our basic perception that some objects are somehow different and others are similar? The ability to observe the differences and the similarities between objects seems to be very basic. Discriminating features can be found once we are familiar with Similarities. This book is written from the perspective that the most primary observation we can make when studying a group of objects or phenomena is that some are dissimilar and others are similar. From this starting point, we aim to define a theory for learning and recognizing patterns by automatic means: sensors and computers that try to imitate the human ability We will use the word ‘pattern’ exclusively to refer to quantitativelqualitative characteristics between objects. In the literature, however, ‘pattern’ is also used to refer to a single object for which such characteristics are studied. We will avoid this usage here.
The disszmzlarity representation for pattern recognition
2
A
B
C
D
E Figure 1.1
F
G
H
I
J
K
Fish contours.
of pattern recognition. We will develop a framework in which the initial
representation of objects is based on dissimilarities, assuming that a human expert can make explicit how to measure them from sensor data. We will develop techniques to generalize from dissimilarity-based representations of sets of objects to the concepts of the groups or classes that can be distinguished. This is in contrast to the traditional paradigm in automatic pattern recognition that starts from a set of numerical features. As stated above, features are defined after dissimilarities have been observed. In a featurebased approach, more hunian expertise may be included. Consequently, if this is done properly. a feature description should be preferred. If, however, this expertise is not available, then dissimilarities have to be preferred over arbitrarily selected features. There are already many applied studies in this area based on dissimilarities. They lack a foundation and, consequently, consistent ways for building a generalization. This book will contribute to these two. In the first part, Chapters 2 to 5, concepts and theory are developed for dissimilarity-based pattern recognition. In the second part, Chapters 6 to 10. they are used for analyzing dissimilarity data and for finding and classifying patterns. In this chapter, we will first introduce our concepts in an intuitive way.
1.2
Dissimilarities for representation
Human perception and inference skills allow us to recognize the common characteristics of a collection of objects. It is, however, difficult to formalize such observations. Imagine, for instance, the set of fish shape contours [Fish contours, site] as presented in Fig. 1.1. Is it possible to define a simple rule that divides them into two or three groups? If we look at the contours. wc firid that some of the fish are rather long without characteristic fins (shape C, H and I), whereas others have distinctive tails as well as fins, say a group
3
htroduction
A (a) Fish shapes
H
(b) Area difference
9
(c) By covers
(d) Between skeletons
-i B/
Figure 1.2 Various dissimilarity measures can be constructed for matching two fish shapes. (b) Area difference: the area of non-overlapping parts is computed. To avoid scale dependency, the measured difference can be expressed relative to the sum of the areas of the shapes. (c) Measure by covers: one shape is covered by identical balls (such that the ball centers belong to it), taking care that the other shape is covered as well. The shapes are exchanged and the radius of the minimal ball is the sought distance. 111 both cases above, B is covered such that either A or H are also covered. (d) Mcasure between skeletons: two shape skeletons are compared by summing up the differences between corresponding parts, weighting missing correspondences more heavily.
of fin-type fish. Judging shapes F and K in the context of all fish shapes presented here, they could be found similar to other fin-type fish: A, B, D, E, G and J. By visual inspection, they do riot really appear to be alike. as they seem to be thinner and somewhat larger. If the exairiples of C, H and I had been absent, the differences between F and K and other fin-type fish would havc been more pronounced. Furthermore. shape A could be considered similar to F and K , but also different due to the position arid shape of its tail and fins. This simple example shows that without any extra knowledge or a clear context, one cannot claim that the identification of two groups is better than the identification of three groups. This decision relies on a free interpretation of what makes objects similar to be considered as a group. For the purpose of automatic grouping or identification, it is difficult to determine proper features, i.e. mathematically encoded particular properties of the shapes that would precisely discriminate between different fish
4
The dissimilarity representation for pattern recognition
and at the same time emphasize the similarity between resembling examples. An alternative is to compare the shapes by matching them as well as possible and determining the remaining differences. Such a match is found with respect to a specified measure of dissimilarity. This measure should take on small values for objects that are alike and large values for distinct objects. There are many ways of comparing two objects, and hence there are many dissimilarity measures. In general, the suitability of a measure depends on the problem at hand and should rely on additional knowledge one has about this particular problem. Example measures are presented in Fig. 1.2, where two fish shapes are compared. Here, the dissimilarity between two similar fish, A and B, is much smaller than between two different fish, B and H. Which to choose depends on expert knowledge or problem characteristics. If there is no clear preference for one measure over the other, a number of measures can be studied and combined. This may be beneficial, especially when different measures focus on different aspects of patterns.
1.3
Learning from examples
The question how to extract essential knowledge and represent it in a formal way such that a machine can ‘learn’ a concept of a class, identify objects or discriminate between them, has intrigued and provoked many researchers. The growing interest inherently led to the establishment of the areas of pattern recognition, machine learning and artificial intelligence. Researchers in these disciplines try to find ways to mimic the human capacity of using knowledge in an intelligent way. In particular, they try t o provide mathematical foundations and develop models and methods that automate the process of recognition by learning from a set of examples. This attempt is irispircd by the human ability to recognize for example what a tree is, given just a few examples of trees. The idea is that a few examples of objects (and possible relations between them) might be sufficient for extracting suitable knowledge to characterize their class. After years of research, some practical problems can now be successfully treated in industrial processing tasks such as automatic recognition of damaged products on a conveyor belt, or to speed up data-handling procedures. or the automatic person identification by fingerprints. The algorithms developed so far are very task specific and, in general, they are
Introduction
5
still far from reaching the human recognition performance. Although the models designed are becoming more and more complex, it seems that to take them a step further, one will need to analyze t,heir basic underlying assumptions. An understanding of the recognition process is needed; not only the learning approaches (inductive or deductive principles) must be understood, but mainly the basic notions of class, measurenient, process and the representation of objects derived from these. The formalized representation of objects (usually in mathematical terms) and the definition of classes determine how the act of learning should be modeled. While many researchers are concerned wit,h various algorithmic procedures, we would like to focus on the issue of representation. This work is devoted t o part,ic.iilar representations, na.mely dissimilarity representations. Below and in the subsequent sections, we will give some insight into the nature of basic problenis in pattern recognition and machine learning and motivate the use of dissimilarity representations. While dealing with entities to be compared, we will always refer to them as to object,s, elements or instances, regardless of whether they are real or abstract. For instance, images, textures and shapes are called objects in the same way as apples and chairs. An appropriate representation of objects is based on data. These are usually obtained by a measurement device and encoded in a numerical way or given by a set of observations or dependencies; presented in a structural form, e.g. a relational graph. It is assumed that objects can, in general, be grouped together. Our aim then is to identify a number of groups (clusters) whose existence supports an understanding of not only the data, but also the problem itself. Such a process is often used to order information and to find suitable or efficient descriptions of the data. The challenge of automatic object recognition is to develop computer methods which learn to identify whether an object belongs to a specific class or learn to distinguish between a number of classes. Typically, the system is first presented with a set of labeled objects, the training set,, in some convenient representation. Learning consists of finding the class descriptions such that t,he system can correct,ly classify novel examples. In practice, the entire system is trained such that the given examples are (mostly) assigned to the correct class. The underlying assumption is that the training examples are representative and sufficient for the problem at hand. This implies that the system can extrapolate well to previously unseen examples, that is, it can generalize well. There are two principal directions in pattern recognition, statistical arid
6
T h e dissimilarity representation f o r pattern recognition
Table 1.1 Basic differences between statistical and structural Pattern Recognition [Nadler and Smith, 19931. Distances are a common factor used for discrimination in both approaches. __ Properties
Statistical
Structural
Foundation
Well-developed mathematical theory of vector spaces Quantitative Numerical features: vectors of a fixed length Element position in a vector Easily encoded Vector-based methods Metric, often Euclidean Relies on distances or inner products in a vcctor space Due to improper features and probabilistic models
Intuitively appealing: human cognition or perception Qualitative: structural/syntactic Morphological primitives of a variable size Encoding process of primitives Needs regular structures Graphs, decisions trees, grammars Defined in a matching process Grammars recognize valid objects; distances often used Due to improper primitives leading to ambiguity in the description
Approach Descriptors Syntax Noise Learning Dissimilarity Discrimination Class overlap
structural (or syntactic) pattern recognition [Jain et al., 2000; Nadler and Smith, 1993; Bunke and Sanfeliu, 19901. The basic differences are summarizcd in Table 1.1. Both approaches use features to describe objects, but these features are defined difEerently. In general, features are functions of (possibly preprocessed) measurements performed on objects, e.g. particular groups of bits in a binary image summarizing it in a discriminative way. The statistical, decision-theoretical approach is (usually) metric and quantitative, while the structural approach is qualitative [Bunke and Sanfeliu, 1990; Nadler and Smith, 19931. This means that in the statistical approach, features are encoded as purely numerical variables. Together, they constitute a feature vector space, usually Euclidean, in which each object is represented as a point2 of feature values. Learning is then inherently restricted to the rria,thexnatical methods that one can apply in a vector space, equipped with additional algebraic structures of an inner product, norm and the distance. In contrast, the structural approach tries to describe the structure of objects that intuitively reflects the human perception [Edelman et al., 1998; Edelnmn, 19991. The features become primitives (subpatterns), fundamental structural elements, like strokes, corners or other morphological elements. 21n this book, the words ‘points’ and ’vectors’ are used interchangeably. In the rigorous mathematical sense, points and vectors are not the same, as points are defined by fixed sets of coordinates in a vector space, while vectors are defined by differences between points. In statistical pattern recognition, objects are represented as points in a vector space, but for the sake of convenience, they are also treated as vectors, as only then they define the operations of vector addition and multiplication by a scalar.
Introduction
7
Characterization I Decision
I
t Generalization/ inference
I
Representation
Segmentation
1
..
..__
i
Measurements
...*
~ - -
I
t
Objects
Figure 1.3 Components of a general pattern recognition system. A representation is either a numerical description of objects and/or their relations (statistical pattern recognition) or their syntactical encoding by set of primitives together with a set of operations on objects (structural pattern recognition). Adaptation relies on a suitable change (simplification or enrichment) of a representation, e.g. by a reduction of the number of features, relations or primitives describing objects, or some nonlinear transformation of the features, to enhance the class or cluster descriptions. Generalization is a process of determining a statistical function which finds clusters, builds a class descriptor or constructs a classifier (decision function). Inference describes the process of a syntax analysis, resulting in a (stochastic) grammar. Characterization reflects the final decision (class label) or the data description (determined clusters). Arrows illustrate that a building of the complete system may not be sequential.
Next, the primitives are encoded as syntactic units from which objects are constructed. As a result, objects are represented by a set of primitives with specified syntactic opcrations. For instance, if the operation of concatenation is used, objects are described by strings of (concatenated) primitives. The strength of the statistical approach relies on well-developed concepts and learning techniques, while in the structural approach, it is much casier to encode existing knowledge on the objects. A general description of a pattern recognition system is illiistratcd in Fig. 1.3; see also [Duin et al., 20021 for a more elaborate discussion and [Nadler and Smith, 19931 for an engineering approach. The description starts from a set of measurements performed on a set of objects. These measurements may be subjected to various operations in order to extract the
8
T h e dissimdarity representation f o r p a t t e r n recognition
essential information ( e g . to segment an object from the image background arid identify a number of characteristic subpatterns), leading to some nunierical or structural representation. Such a representation has evolved from an initial description, derived from the original measurements. Usually, it is not directly the most appropriate one for realizing the task, such as identification or classification. It may be adapted by suitable transformations, e.g. a (nonlinear) rescaling of numerical features or an extension and redefinition of primitives. Then, in the generalization/inference stage, a classifier/identifier is trained, or a grammar3 is determined. These processes should include a careful treatment of unbalanced classes, non-representative data, handling of missing values, a rejection option, combining of inforrnatiori and combiriing of classifiers and a final evaluation. In the last stage, a class is assigned or the data. are characterized ( e g . in terms of clusters and their relations). The design of a complete pattern recognition system may require repetition of some stages to find a satisfactory trade-off between the final recognition accuracy or data description and the computational and storage resources required. Although this research is grounded in statistical pattern recognition, we recognize the necessity of combining numerical and structural information. Dissimilarity measures as the common factor used for discrimination, Table 1.1, seems to be the natural bridge between these two types of information. The integration is realized by a representation. A general discussion on the issue of representation can be found in [Duin et al., 2004al.
1.4
Motivation of the use of dissimilarity representations
The notion of similarity plays a pivotal role in class formation, since it might he seen as a natural link between observations on objects on the one hand arid a judgment on their shared properties on the other. In essence, similar objects can be grouped together to form a class, and consequently u class is a set of sim,ilar objects. However, there is no such thing as a general object similarity that car1 be universally measured or applied. A comparison of two objects is always with respect to a frame of reference, i.e. a particular point of view, a context, basic characteristics, a type of domain, or attributes considered (see also Fig. 1.1). This means that background information, or 3Primitives are interpreted as syntactic units or symbols. A grammar is a set of rules of syntax that enables the generation of sentences (structures) from the given symbols (units).
Introduction
(
9
Measurements or intermediate representation
1
1 Feature-based representation
1
Define a set of features
Dissimilarity-based representation
1 Define a dissimilarity measure \I
Represent objects as points in a feature vector space
i
Impose the geometry, e.g. of the Euclidean distance between the points
1 Interpret the dissimilarities in a suitable space to reflect the distance geometry
Figure 1.4 The difference with respect to the geometry between the traditional featurebased (absolute) representations and dissimilarity-based (relative) representations.
the existence of other classes; will influence the way objects are compared. For instance, two brothers may not appear to resemble each other. However, they may appear much more alike if compared in the presence of their parents. The degree of similarity between two objects should be determined relative to a given context or a procedure. Any measurement of similarity of objects will be based 011 certain assumptions concerning the properties of their relation. Such assumptions come from some model. Similarity can be modeled by a measure of sirriilarity or dissimilarity. These are intimately connected; a small dissimilarity and a large similarity both imply a close resemblance of objects. There exist ways of changing a similarity value into a dissimilarity value and vice versa, but the interpretation of the measure might be affected. In this work. we mostly concentrate on dissimilarities, which by their construction, focus on the class and object differences. The choice for dissimilarities is supported by the fact that they can be interpreted as distances in suitable vector spaces, and in many cases, they may be more intuitively appealing. In statistical pattern recognition, objects are usually encoded by feature values. A feature is a corijunction of measured values for a particular attribute. For instance, if weight is an attribute for the class of apples, then a feature consists of the measured weights for a number of apples. For a set T of N objects, a feature-based representation relying on a set .F of m features is then encoded as an N x m matrix A(T, F), where each
10
The dissimilarity representation f o r pattern recognition
row is a vector describing t8hefeature values for a particular object. Feat,ures 3 are usually interpreted in a Euclidean vector space equipped with the Euclidean metric. This is motivated by the algebraic structure (defined by operations on vectors) being consistent with the geometric (topological) structure defined by the Euclidean distance (which is then defined by the norm). Then all traditional mathematical concepts and methods, such as continuity, convergence or differentiation are applicable. The continuity of algebraic operations ensures that the local geometry (defined by the Euclidean distance) is preserved throughout the space [Munkres, 2000; Kothe, 19691. Discrimination techniques operating in vector spaces make use of their homogeneity and other properties. Consequently, such spaces require that up to scaling all the features are treated in the same way. Moreover, there is no possibility to relate the learning to the geometry defined between the raw representations of the training examples. The geometry is simply imposed beforehand by the nature of the Euclidean distance between (reduced) descriptions of objects, i.e. between vectors in a Euclidean space; see also Fig. 1.4. The existence of a well-established theory for Euclidean metric spaces made researchers place the learning paradigm in that context. However, the severe restrictions of such spaces simply do not allow discovery of structures richer than affine subspaces. From this point of view, the act of learniiig is very limited. We argiic here that the notion of proximity (similarity or dissimilarity) is more fundamental than that of a feature or a class. According to an intuitive definition of a class as a set of similar objects, proximity plays a crucial role for its constitution, and not features, which may (or may not) come later. From this point of view, features might be a superfluous step in the description of a class. Surely, proximity can be specified by features, such as their weighted linear combination, but the features should be meaningful with respect to the proximity. In other words, the chosen combination of features should reflect the (natural) proximity between the objects. On the other hand, proximity can be directly derived from raw or pre-processed measurements like images or spectra. Moreover, in the case of syrnbolic objects, graphs or grammars, the determination of numerical features might be an intractable problem, while proximity may be easier to define. This emphasizes that a class of objects is represented by individual examples which are judged to be similar according to a specified measure. A dissimilarity representation of objects is then based on pairwise comparisons and is expressed e.g. as an N x N dissimilarity matrix D ( T , T ) .Each entry of D is a dissimilarity value computed between pairs of objects; see
Introduction
11
OBJECTS
\ I / ensor measurements
f
e
y
ABSOLUTE REPRESENTATION
A(T,F)
feature 2
ilarity measure
RELATIVE REPRESENTATION
feature 3
feature 1
Figure 1.5 Feature-based (absolute) representation vs. dissimilarity-based (relative) representation. In the former description, objects are represented as points in a feature vector space, while in the latter description, objects are represented by a set of dissimilarity values.
also Fig. 1.5. Hence, each object z is represented by a vector of proximities
D ( z ,T ) to the objects of T (precise definitions will be given in Chapter 4). For a number of years, Goldfarb and colleagues have been trying to establish a new mathematical formalism allowing one to describe objects from a metaphysical point of view, that is, to learn their structure and characteristics from the process of their construction. This aims at unifying the geometric learning models (statistical approach with the geometry imposed by a feature space) and symbolic ones (structural approach) using dissimilarity as a natural bridge. A dissimilarity measure is determined in a process of inductive learning realized by so-called evolving transformation systems [Goldfarb. 1990; Goldfarb and Deshpande. 1997; Goldfarb and Golubitsky, 20011. Loosely speaking, such a system is composed of a set of primitive structures, basic operations that transform one object
12
The dissimilarity representation f o r p a t t e r n recognition
into another (or which generate a particular object) and some composition rules which permit the construction of new operations from existing ones [Goldfarb et al., 1995, 1992, 2004; Goldfarb and Deshpande, 1997; Goldfarb and Golubitsky, 20011. This is the symbolic component of the integrated model. The geometric component is defined by means of a dissimilarity. Since there is a cost associated with each operation, the dissimilarity is determined by the minimal sum of the costs of operations transforming one object int,o another (or generating this particular object). In this sense, the operations play the role of features, and the dissimilarity - dynamically learned in the training process - combines the objects into a class. In this book, t,he study of dissimilarity representations has mainly an epistemological character. It focuses on how we decide (how we make a model to decide) that an entity belongs to a particular class. Since such a decision builds on the dissimilarities, we come closer to the nature of what a class is, as we believe that it is proximity which defines the class. This approach is much more flexible than the one based on features, since now, the geometry and the structure of a class are defined by the dissimilarity measure, which can reflect the structure of the objects in some space. Note that the reverse holds in a feature space, that is, a feature space determines the (Euclidean) distance measure, and hence the geometry; see also Fig. 1.4. Although, dissimilarity information is further treated in a numerical way, the development of statistical methods dealing with general dissimilarities is the first necessary step towards a unified learning model, as the dissimilarity measure may be developed in a structural approach. Notwithstanding the fact that integrated model may be constructed for objects containing an inherent, identifiable structure or organization, like apples, shapes, spectra, text excerpts etc., current research is far from being generally applicable [Korkin and Goldfarb, 2002; Goldfarb and Golubitsky, 2001; Goldfarb et al., 2000b, 20041. On the other hand, there are a number of instances or events which are mainly characterized by discontinuous numerical or categorical information, e.g. gender, or number of children, etc. Therefore, wc may have to consider heterogeneous types of information to support decisions in medicine, finance, etc. In such cases, the symbolic learning model cannot be directly utilized, but a dissimilarity can be defined. This emphasizes the importance of techniques operating on general dissimilarities. The study of proximity representations is the necessary starting point from which to depart on a journey into alternative inductive learning methodologies. These will learn the proximity measure, and hence a class description, from examples.
Introduction
13
1.5 Relation to kernels Kernel methods have become popular in statistical learning [Cristianini arid Shawe-Taylor, 2000; Scholkopf arid Smola, 20021. Kernels are (conditionally) positive definite (cpd) functions of two variables, which car1 be thought to encode similarities between pairs of objects. They are originally defined in vector spaces, e.g. based on a feature representation of objects, and interpreted as generalized inner products in a reproducing kernel Hilbert space (RKHS). They offer a way to construct non-linear decision functions. In 1995, Vapnik proposed an elegant formulation of the largest margin classifier [Vapnik, 19981. This support vector machine (SVM) is based on the reproducing property of kernels. Since then, many variants of the SVM have been applied to a wide range of learning problems. Before the start of our research project [Duin et al., 1997, 1998, 19991 it was already recognized that the class of cpd functions is restricted. It does riot accommodate a number of useful proximity measures already developed in pattern recognition and computer vision. Many existing similarity measures are not positive definite and many existing dissimilarity measures are not Euclidean4 or even not metric. Examples are pairwise structural alignments of proteins, variants of the Hausdorff distance, and normalized edit-distances; see Chapter 5. The major limitation of using such kernels is that the original formulation of the SVM relies on a quadratic optimization. This problem is guaranteed to be convex for cpd kernels, and therefore uniquely solvable by standard algorithms. Kernel matrices disobeying these requirements are usually somehow regularized, e.g. by adding a suitable constant to their diagonal. Whether this is a beneficial strategy is an open question. Although our research was inspired by the concept of kernel, the line we followed heavily deviates from the usage of kernels in machine learning [Shawe-Taylor and Cristianini, 20041. This is caused by the patternrecognition background of the problems we aim to solve. Our starting point is a given set of dissimilarities, observed or determined during thc development of a pattern recognition system. It is defined by a human expert and his/her insight into the problem. This set is, thereby, an alternative to the definition of features (which also have to originate from such expertise). A given Euclidean distance matrix may be transformed into a kernel and interpreted as a generalized Gram matrix in a proper Hilbert space. 4The dissimilarity measure being Euclidean is inherently related t o t,he corresponding kernel being positive definite; this is explained in Chapter 3.
The dissimilarity representation for p a t t e r n recognition
14
[
(
Characterizationof dissimilaritymatrices
Chapter3
Learning aspects
I
I
Reprera;;;;,review
j [T)
j
ConclusionsI open problems Chapter 12
Figure 1.6 Conceptual outline of the book
However, many general dissimilarity measures used in pattern recognition give rise to indefinite kernels, which have only recently become of interest [Haasdonk, 2005; Laub and Miiller, 2004; Ong et al., 20041, although we had already identified their importance before [Pckalska et al., 2002bl. How to handle these is an important issue in this book.
1.6
Outline of the book
Dissimilarities play a key role in the quest for an integrated statisticalstructural learning model, since they are a natural bridge between these two approaches, as explained in the previous sections. This is supported by the theory that (dis)similarity can be considered as a link between perception and higher-level knowledge, a crucial factor in the process of human recognition and categorization [Goldstone, 1999; Edelman et ul., 1998; Wharton et al., 19921. Throughout this book, the investigations are dedicated to dissimilarity (or similarity) representations. The goal is to study both methodology and approaches to learning from such representations. An outline of the book is presented in Fig. 1.6.
Introduction
15
The concept of a vector space is fundamental to dissimilarity representations. The dissimilarity value captures the notion of closeness between tjwo objects, which can be interpreted as a distance in a suitable space: or which can be used to build other spaces. Chapter 2 focuses on mathematical characteristics of various spaces, among others (generalized) metric spaces, norm spaces and inner product spaces. These spaces will later become the context in which the dissimilarities are interpreted arid learning algoritlinis are designed. Familiarity with such spaces, their properties and their interrelations is needed for further understanding of learning processes. Chapter 3 discusses fundamental issues of dissimilarity measures and generalized metric spaces. Since a metric distance, particularly the Euclidean distance, is mainly used in statistical learning, its special role is explained and related theorems are given. The properties of dissimilarity matrices are studied, together with some embeddings, i.e. spatial represeritations (vectors in a vector space found such that the dissiniilarities are preserved) of symmetric dissimilarity matrices. This supports the analysis of pairwise dissimilarity data D ( T ,T ) based on a set of examples T . Chapter 4 starts with a brief introduction into traditional statistical learning, followed by a more detailed description of dissimilarity reprcsentations. Three different approaches to building classifiers for such representations are considered. The first one uses dissimilarity values directly by interpreting them as neighborhood relations. The second one interprets them in a space where each dimension is a dissimilarity to a particular object. Finally, the third approach relies on a distance-preserving embedding to a vector space, in which classifiers are built. In Chapter 5, various types of similarity and dissimilarit,y measures are described, together with their basic properties. The chapter ends with a brief overview of dissimilarity measures arising from various applications. Chapters 6 and 7 start from fundamental questions related to exploratory data analysis on dissimilarity data. Data visualization is one of the most basic ways to get insight into relations between data instances. This is discussed in Chapter 6. Other issues related to data exploration and understanding are presented in Chapter 7. They focus on methods of unsupervised learning by reflecting upon the intrinsic dimension of the dissimilarity data, the complexity of the description and data striicture in terms of clusters. A possible approach to outlier detection is analyzed in Chapter 8 by coiistructing one-class classifiers. These methods are designed to solve problems, where mainly one of the classes, called the target class, is present.
T h e dissimilarity representation for p a t t e r n recognition
16
Objects of the other, outlier, class occur rarely, cannot be well sampled, e.g. due the measurement costs or are untrustworthy. We introduce the probleni arid study a few one-class classifier methods built on dissimilarity represent at ions. Chapter 9 deals with classification. It practically examines three approaches to learning. For recognition, a so-called representation set is used instcad of a complete training set. This chapter explains how to select such a set out of a training set arid discusses the advantages and drawbacks of t hc studied techniques. Chapter 10 investigates combining approaches. These either combine different dissiniilarity representations or different types of classifiers. Additionally. it briefly discusses issues concerning nieta-learning, i.e. conceptual dissimilarity representations resulting from combining classifiers, one-class classifiers or weak models. in general. Chapter 11 discusses the issue of representation in pattern recognition and provides practical recommendations for the use of dissimilarity representations. Overall conclusions are given in Chapter 12. Appendices A-D provide additional information on algebra, probability and statistics. Appendix E describes the data sets used in the experiments.
1.7
In summary
Dissimilarity representations are advantageous for identification and recognition, especially in the following cases: 0 0
0 0 0 0
0 0 0
sensory data, such as spectra, digital or hyperspectral images data represented by histograms, contours or shapes, phenomena that can be described by probability density functions, binary files, text-related problems, when objects are encoded in a structural way by trees, graphs or strings. when objects are represented as vectors in a high-dimensional space, when the features describing objects are of mixed types, as a way of constructing nonlinear classifiers in given vector spaces.
Mathematical foundations for dissimilarity representations rely on:
(1) topology and general topology [Sierpiliski, 1952; Cech, 1966; Kothe, 1969; Willard, 1970; Munkres, 2000; Stadler et al., 2001; Stadler and Stadler, 2001b],
Introduction
17
(2) linear algebra [Greub, 1975; Bialynicki-Birula, 1976; Noble and Daniel, 1988; Leon, 1998; Lang, 20041, ( 3 ) operator theory [Dunford and Schwarz, 1958; Sadovnichij, 19911. (4) functional analysis [Kreyszig, 1978; Kurcyusz, 1982; Conway, 1990; Rudin, 1986. 19911, ( 5 ) indefinite inner product spaces [BognBr, 1974; Alpay et al., 1997; Iohvidov et al., 1982, Dritschel and Rovnyak, 1996; Constantinescu and Gheondea, 20011, (6) probability theory [Feller, 1968, 1971; Billingsley, 1995; Chung, 20011, (7) statistical pattern recognition [Devijver and Kittler, 1982; Fukunaga, 1990; Webb, 1995; Devroye et al., 1996; Duda et nl., 20011, (8) statistical learning [Vapnik, 1998; Cherkassky and Mulier, 1998; Hastie et al., 20011. (9) the work of Scholkopf and colleagues [Scbolkopf. 1997, 2000; Scholkopf et al., 1999b, 1997a, 1999a, 2000b], (10) the results of Goldfarb [Goldfarb, 1984, 1985, 19921, and inspiration from many other researchers. We will present a systematic approach to study dissimilarity representations and discuss some novel procedures to learning. These are inevitably compared to the nearest neighbor rule (NN) [Cover and Hart, 19671, the method traditionally applied in this context. Although many researchers have thoroughly studied the NN method and its variants together with design of perfect dissimilarity measures (appropriate to the character of the NN rule), to our knowledge little attention was dedicated to alternative approaches. An exception are the support vector machines. These rely on a relatively narrow class of (conditionally) positive definite kernels, which, in turn, are special cases of similarity representations [Duin et al., 1997, 19981. Only recently the interest has arisen in indefinite kernels [Haasdonk, 2005; Laub and Muller, 2004; Ong et al., 20041. The methods presented here are applicable to general (dis)similarity representations, and this is where our main contribution lies. A more detailed description of the overall contributions is presented below.
Representation of objects. A proximity representation quarititatively encodes the proximity between pairs of objects. It relies on the representa-
18
T h e dissimilarity representation f o r p a t t e r n recognition
tion set, R, a relatively small collection of objects capturing the variability in the data. Each object is described by a vector of proxirnities to R. In the beginning, the representation set may consist of all training examples as it is reduced later in the process of instance selection. Here, a number of selection criteria are proposed and experimentally investigated for different learning frameworks. In this way, we extend the notion of a kernel t o t,hat of a proximity representation. If R is chosen to be the set of training examples, then this representation becomes a generalized kernel. When a suitable similarity measure is selected, a cpd kernel is obtained as a special case. Using a proximity representation, learning can be addressed in a more general way than by using the support vector machine. As such, we develop proximity representations as a first step towards bridging the statistical and structural approaches to pattern recognition. They are successfully used for solving object recognition problems.
Data understanding. Understanding data is a difficult task. The main consideration is whether the data sampling is sufficient to describe the problem domain well. Other important questions refer to intrinsic dimension, data structure, e.g. in terms of possible clusters and the means of data visualization. Since there exist marly algorithms for unsupervised learning, our primary interest lies in the former questions. In this book, three distinct approaches to learning from dissimilarity representations are proposed. The first, one a.ddresses the given dissimilarities directly. The second addresses a dissimilarity representation as a mapping based on the representation set R. As a result, the so-called dissimilarity space is considered, where each dinlension corresponds to a dissimilarity to a particular object from R. The third one relies on an approxiniate embedding of dissimilarities into a (pseudo-)Euclidean space. The approaches are introduced, studied arid applied in various situations. Domain description. The problem of describing a class has gained a lot of attention, since it can be identified in many applications. The area of interest covers all problems where specified targets have to be recognized and anomalies or outlier situations have to be detected. These might be examples of any type of fault detection, abnormal behavior, or rare diseases. The basic assumption that an object belongs to a class is based on the idea that it is similar to other examples within this class. The identification procedure can be realized by a proximity function equipped with a threshold, determining whether or not an instance is a class member. This proximity function can be e.g. a distance to a set of selected prototypes.
Introduction
19
Therefore, the data represented by proximities is more natural for buildirig concept descriptors, since the proximity function can directly be built on these proximities. To study this problem, we have not only adopted known algorithnis for dissimihrity representa,tions, but have also implemented and investigated new methods. Both in terms of' efficiency and performance issues. our methods were found to perform well.
Classification. We propose new methodologies to deal with dissiniilarity/similarity data. These rely either on approximat,e embedding in a pseudo-Euclidean space and construction of the classifiers there, or on building of the decision rules in a dissimi1arit)yspace, or on designing of neighborhood-based classifiers, e.g. the NN rule. In all cases, foundations are established, that allow us t o handle general dissimilarity measures. Our methods do not require metric constraints, so their applicability is quite universal. Combining. The possibility to combine various types of information has proved to be useful in practical applications; see e.g. [MCSOO,2000; NICS02, 20021. We argue that combining either significantly different dissimilarity representations or classifiers different in nature on the same representation can be beneficial for learning. This may be useful when there is a lack of expertise of how a well-discrimination dissimilarity measure should be designed. A few measures can be considered, taking into account differeiit characteristics of the data. For instance, when scanned digits should hc compared, one measure focuses on the contour information, while others on the area or on statistical properties. Applications. The proximity measure plays an important role in many research problems. Proximity representations are widely used in many areas, although often indirectly. They are used for text or iniage retrieval, data visualization, the process of learning from partially labeled sets, etc. A number of applications is discussed where such measures ares found to be advantageous. In essence. The study on dissimilarity representations applies to all dissimilarities, independently of the way they have been derived, e.g. from raw data or from an initial representation by features, strings or graphs. Expert knowledge oil the application can be used to formulate this initial representation and in the definition of the proximity measure. This makes the dissimilarity representations developed natural candidates for combining
20
T h e dzssamzlaraty r e p r e s e n t a t z o n for p a t t e r n recognztion
the strengths of structural and statistical approaches in pattern recognition and machine learning. The advantage of the structural approach lies in encoding both domain knowledge and the structure of an object. The benefit of the statistical approach lies in a well-developed mathematical theory of vector spaces. First, a description of objects in the structural framework can be found. This can then be quantized to capture the dissimilarity relations between the objects. If necessary, other structurally and statistically derived measures can be designed and combined. The final dissimilarity representation is then used in statistical learning. The results in this work justify the use and further exploration of dissimilarity information for pattern recognition and machine learning.
PART 1
Concepts and theory
Budowatem na piasku I zawalito sie. Budowatem na skale I znwalilo sie. Teraz budujgc Zaczn,e od dymu z komina.
I built on the sand And at tumbled down. I built on 0; rock And it tumbled douin,. Now when I build, I shall be,qi,n With the smoke from. the chimney.
“PODWALINY” , LEOPARDSTAFF
“FOUNDATIONS”, LEOPOLDSTAFF
This page intentionally left blank
Chapter 2
Spaces
Ring the bells that still can ring Forget yo’w perfect offeer.ing There i s a crack: in ever-ything That’s h,o,w the light gets in. “ANTHEM>’LEONARDCOHEN
Many dissimilarity measures have been designed and are used in various ways in pattern recognition, machine learning, computer vision and relat,ed fields. What is missing, however, is a general and unified framework for learning from examples that are represented by their dissimilarities to a set of representation objects. Different aspects of the measures, such as showing of the Euclidean behavior, metric or asymmetric properties, may lead to different learning approaches. In the statistical approach to pattern recognition, objects are represented as points in a vector space, equipped with an additional algebraical structure of an inner product and the associated norm. This is iisually a Euclidean or a Hilbert space. The distance between the points is then naturally measured by the Euclidean or Hilbert distance. If beneficial. other metric distances niay be introduced, usually froni the family of the &distances or C,-distances. Classifiers are functions defined by firiit,e vector representatioris in this vector space. Usually, they are designed, i)ased on the assumed model, applied probabilistic reasoning or used pairwisc distances. The question we begin with is more difficult. How can a learning task be performed given a set of pairwisc dissimilaritics? Dissimilarities are measured according to a specified dissimilarity measure, which is not necessarily a metric and not necessarily a measure in the strict niathcmatical sense. It quantifies the similarity or commonality between two objects by taking small values for two similar objects and large values for two distinct objects. Additionally, when possible, sensor measurements or other intermediate description of the set of examples niay be given. The challenge is
23
24
T h e dissimilarity representation f o r p a t t e r n recognition
to discover the structure in the data, identify objects of a particular class or learn t o distinguish among the classes, knowing the procedure according to which the dissimilarity is computed and the dissimilarity values between a set of (training) examples. As no vectorial representation of the objects is provided, the challenge is now to use the dissimilarities in a meaningful way. To make use of statistical learning, we must find an appropriate framework for the interpretation of dissimilarities. The concept of a (vector) space is important for the development of a theoretical foundation, both from the representational and algorithmic point of view, since we will rely on numerical procedures and deal with numerical representations of the problems. Dissimilarities quantitatively express the differences between pairs of objects. while learning algorithms usually optimize some error or loss function for a chosen numerical model. Dissimilarities, therefore, have a particular meaning within a frame of specified assumptions and models. Spaces possessing different characteristics will allow different interpretations of the dissimilarity data, which will lead to different learning algorithms. Therefore, before discussing dissimilarity representations and learning methods, we need essential concepts arid properties of various spaces. This chapter is motivated by the lack of a consistent and clearly ideritifiable mathematical theory on general dissimilarity measures, not only in the pattern recognition field, but also in mathematics. In its foundations, such a theory should rely on the notion of nearness between two objects. Therefore. the theory of spaces plays a key role, since suclu a nearness can easily be introduced here. Most of the existing theories deal with norms, which are often used to define metrics. Usually, Euclidean, city block or max-norm distances are considered. Other interesting contributions can be fourid in various subfields of mathematics, such as non-Euclidean geornetries [Blumenthal, 1953; Coxeter, 19981, differential geometry [Kreyszig, 1991; Struik, 19881, algebras [Paulsen, 20021 and operator spaces [Effros and Ruan, 2000; Pisier, 20031. Additional inspiration can be found in the fields of experimental psychology and artificial intelligence. These, however, remain of interest for future study. To oiir knowledge, no book yet exists that explains the theoretical background of general dissimilarity measures and studies learning problems from such a perspective (although a general study on pattern theory in this direction by Grenander is available [Grenander, 1976, 1978, 19811). Therefore, this chapter is meant to fill this gap. It not only introduces spaces with their basic properties, but it also shows the relations between them. Conse-
Spaces
25
quently, the concepts are presented from a mathematical point of view and supported, if possible, by examples from pattern recognition. The purpose of this chapter is to bring together and present a basic theory on spaces in the context of general dissimilarities, both metric and non-metric. The spaces described here will serve as interpretation frameworks of dissimilarity data. The connections will become clear in Chapters 3 arid 4. We will start by recalling basic notions from set theory.
2.1
Preliminaries
Throughout this book, a set X is a collection of objects of any kinds, both real and abstracts, such as real-word objects, digital images, binary strings, points in a plane, real numbers or functions. These are called elements of X . In some cases, a set can be determined by a means of a property of its elements, such as the set of convex pentagons, non-decreasing functions or scanned handwritten digits of '1'. The set of natural numbers is W. The sets of real, positive real and nonnegative real numbers are denoted by R, B+ and R : , respectively. The set of complex numbers is denoted by @. If X and Y are two sets, then X U Y is their union, A n B is their intersection, X\Y is their difference and X A Y = ( X \ Y ) U (Y \X ) is their symmetric difference. X x Y = { (2, y) : .?:EXA y E Y } denotes a Cartesian product. P ( X ) is a power set, which is a collection of all subsets of X . An index set I defines a correspondence between i E I and either an element ai of a set A or a subset Ai of A. A family of sets in A is denoted by A = {Ai : i E I } . The union, intersection and Cartesian product can be extended to a family of sets as UiEIAi, Ai and Hi,, Ai,respectively.
ni,,
Definition 2.1 (Mapping, function) Let X and Y be two sets. If wit,h each element zE X we associate a subset F ( z ) of Y , then the correspondence z d F ( z ) is a mapping of X into Y or a function from X to Y . If the set F ( z ) consists of a single element; then the mapping is single-valued, and multi-valued, otherwise. Mapping, function or transformation will be used interchangeably. Definition 2.2 (Basic facts on functions)
0
Let f : X + Y be a function from X to Y . X is the domain, of f a.nd Y is the codomain o f f . The range of f is Rf= {y E Y : gZEx y = q5(z)}. The inverse function of f is a mapping f-' : Y 4 X that satisfies f - ' ( f ( z ) ) = z and f ( f - l ( y ) ) = y for all Z E X and ~ E Y .
26
0
The dissimilarity representation for pattern recognztzon
The image of z is f ( i c ) . The preimage of y are all z E X whose image is y. i.e. fpl(y) = {:I; E X : f ( z ) = y}. The image of A c X is the set f ( A ) c Y consisting of all elerncrits of Y which equal f ( a ) for some (1, E A. The preimage (inverse image) of B c Y is the set f - l ( B ) c X consisting of all elements ic E X such that f(x)E B.
Definition 2.3 (Composition) Let f : X + Y and g : Y 4 2 be fuiictions. Then g o f : X 4 Z is a composition of mappings such that (.9 O .f)(.) = 9
(.f.I).(
Definition 2.4 (Injection, surjection, bijection) 0 A fiinction f : X t Y is injective or one-to-one if it maps distinct arguments to distinct images, i.e. x1 # 2 2 + f ( z 1 ) # f ( i c 2 ) holds for all :I: 1, :1'2 t x . 0 A function f : X + Y is ective if' its maps to all images, i.e. if for evcry y t Y , there exists Z E X such that f ( z ) = y. In other words, f is 0
surjective if its rangc is cqual to its codonlain. A function f : X --f Y is bijective if it is both injective and surjective, i.c. if' for cvery Y ,there exists exactly one Z E X such that f ( x ) = y.
The composition of two injections (surjections, bijections) is again an injection (surjection, bijection).
Definition 2.5 (Binary relation) Let X and Y be two sets. A binary relation, R is a subset of the Cartesian product X X Y ,i.e. R C X X Y . A subsct of X x X is a binary relation on X. One writes zRg to indicate that :1:
is in relation with y.
Definition 2.6 (Equivalence relation) is a binary relation
N
An equivalence relation on X
which is
(1) reflexive: .7: z or all EX. (2) syrrirnetric: (x y) + ( y x) for all z . y ~ X . ( 3 ) transitive: (x y A y z) + (x z) for all 2 , y, zE X . N
-
- -
-
-
-
The set of all elements of X equivalent to z is an equivalen,ce class of n: and denoted hy This means that [z] = {y : y t X A y z}.
[XI.
Definition 2.7 (Partially ordered set) A partially ordered set is a pair ( X ,5 ) ;where X is a set arid 5 is a partial order on X , which is: (1) reflexive: x 5 z or all z t X. (2) antisyrnrnetric: (x 5 y A y 5 x) + x = y for all Z, EX. ( 3 ) transitive: (z 5 p A y 5 z)+ (z 5 z) for all z, y, Z E X .
Spaces
27
Definition 2.8 (Upper bound, lower bound ) Let ( X ,5)be a partially ordered set and Y c X.Y is partially ordered under Y is bounded from above (from below) if there exists z E X for which y 5 z (x 5 y) holds for all y E Y.:2: is an 'upper (lo,wer) bound of Y .
s.
Definition 2.9 (Directed set) A partially ordered set X is a directed set if for any z, y E X , there exists z E X such that x 5 z and y 5 z . X is inversely directed if for any x,y E X , there exists z E X such that z 5 x and
z
An iriiportaiit axiom in set theory is the axiom of choice. It is generally accepted as true. Theorem 2.1 (Zermelo/Zorn) A x i o m of ch,oice can, be equivalen,tly stated as: ( I ) Let A be a set of non-empty sets. T h e n there exists a cho%cefimction, f defined o n A, such that f o r each set A in A, f ( A ) E A. (2) Given a n y set of mutually disjoint non-empty sets, there exists at least one set that contains exactly one element in c o m n o n with each of the non-empty sets. (3) If every totally ordered subset of a partially ordered set X has un upper bound th,en. X h,us a. m.azimal element. Definition 2.11 (Finite, countable and uncountable sets) Let A be any set. 0 A is finite if it consists of a finite number of elements. Otherwise, it is infinite. 0 A is countable if there exists an injection f : A ---t N. A is in,finitely countable if there exists a bijection f : A + N. 0 A is uncountable iff A is not countable. All uncountable sets are, therefore, infinite.
T h e dzssimilarity representation for pattern recognztzon
28
Example 2.1 Every subset of a countable set is countable. Cartesian product of finitely many countable sets is countable. The set, of natural numbers is infinitely countable and its cardinality if defined as No (aleph-null). The set of prime numbers, the set of integers or the set of rational numbers arc also countable and have the cardinality of No. The set of real numbers is an uncountable set with the cardinality denoted by c or 21 (beth-one). An uncountable set has the cardinality strictly greater than N o . Example sets with the cardinality of c are [0,1],the set of irrational numbers, the power set of natural numbers, the set of all infinite sequences consisting of' zeros and ones (or of integers), the set of all open sets in R" or the set of all continuous functions from R to B. The set of all functions from R 4 R has tlie cardinality of 2 2 = 2'l, which is larger than XI.
2.2
A brief look at spaces
111general, a space is a set of elements augmented with a type of relation or an additional structure. From a pattern recognition point of view, a space shonld posscs some properties such that a finite representation of objects can be characterized for learning. This niearis that some of the characteristics may be induced or imposed on tlie data instances considered to ensure a generalization' for future examples. Intuitivcly, a space should possess a notion of closeness between its elements, which is compatible with the algebraic structure', whenever such a structure is available. A space is often considered to already posses a structure of a high degree, as those of' metric vector spaces. Euclidean vector spaces are the most widely used example of these, as they reflect our intuitive understanding of a space: vectors are elements that can be added, multiplied by a scalar or projected onto each other. The Euclidean metric is a distance experienced 'To generalize is 'to derive or induce (a general conception or principle) from particulars' [Webster dictionary, site]. 2For instance, the structure of a vector space is based on the algebraic operations of vector addition arid multiplication by a scalar. Thanks to them, a linear combination is defined and, consequently, a hyperplane. In general, the operations defined and permitted in a space, can possibly lead to a creation of constructs, which are different (more complex) than the basic elements themselves, but still reside in this space.
Spaces
29
neighborhood space pretopological space topological space Hausdotff space ‘ (=-space
Figure 2.1
Some generalized topological spaces
by us daily. Usually, more primitive spaces are explained using high-level concepts, such as a norm or metric. Note that a metric can be introduced not only in a vector space, but on an arbitrary set. An example is a set of binary strings or a set of proteins, where a distance measure ca.pt,urcsthe similarity (likeliness) betwecn two strings or t,wo proteins. In such cases. the most natural iiieasure, derived in agreement with the physical process (e.g. evolutionary changes) behind the creation of set examples, may iiot be metric. Therefore, we will use the word ‘dissimilarity’ to indicat,e that metric constraints, i.e. reflexivity, definiteness, syrnmetry and triangle inequality, Def. 2.38, are not necessarily satisfied. Practical problems ask for an investjigation of spaces that are more primitive than metric (or Euclidean). A bottom-up approach, starting from a notion of a neighborhood or of a convergence, is then needed. However, as a commoii approach to learning is to use metric (Euclidean) distances or to impose tlicrn by a suitable correction of‘ the given dissimilarity measure, we will discuss then1 well. Sections 2.3-2.7 will briefly introduce basic coriccpts of gerieralizcd topological spaces, generalized metric spaces and inner product spaces, as well as their essential properties. We will briefly mention the spaces to be introduced in the subsequent sections. The riotion of a neighborhood3 (or of a generalized closure) is the basis for the construction of more coinplex spaces, among others neighborhood spaces, pretopological spaces arid topological spaces. For a general illustration of the interrelations between generalized topological spaces, see Fig. 2.1. The idea of such a pictorial schema is to present how we can build from a very general space satisfying a few constraints, more specific spaces, possessing more structiire. ‘Even more primitive concepts can be used, like filter, convergence or nearness [Gastl and Hammer, 1967; Cech, 1966; Stadler and Stadler, 2001al.
30
The dissimilarity representation for pattern recognitzon
..........................
I/
inner product space
;
Hilbert space
,
\
\
Euclidean space
normed space
\ I\ 1
pseudo-Euclidean space I
space ..........Pontryagin .... Krein space ~
Krein mace
J
,
indefinite inner product space
\
Fiaure 2.2 Schematic relations between some classes of generalized inner product spaces. RKHS denot,es a reproducing kernel Hilbert space and RKKS stands for a reproducing kernel Krein space. I
Basically, if one pictorial space is ‘encapsulated’ by another, the former fulfils more requirements and possesses more properties than the latter, and, consequently, it is more specific and its structure is richer. A set with a neighborhood system creates a neighborhood space. If the intersection of two neighborhoods belongs to the neighborhood system, this leads to a pretopological space. Adding the concept of a ’proper’ boundary, i.e. an idempotent closure operator, gives rise to a neighborhood basis consisting of open scts. As a result, we get a topological space. Imposing the existence of disjoint neighborhoods for distinct elements (which implies that thc sequences of elements have at most one limit) yields a Hausdorff space. By requiring more and more separation axioms (by the means of topological operations) between disjoint sets and distinct elements, more advanced spaces arc obtained, finally leading to a metric space4. By providing the Euclidean distance to a vector space, a Euclidean space is obtained. This brief presentation shows that a Euclidean space is highly structured. Having int,roduced generalized topological spaces, we will consider the linear space as the foundation for more complex spaces. The following spaces are briefly discussed: normed and (indefinite) inner product spaces with their relations to a metric space. Our attention is specifically devoted to Euclidean (Hilbert) and pseudo-Euclidean (Krein ) spaces. Since the inner product and metric are essential concepts for describing relations between object representations, we consider the dependencies between 4The ordering of spaces from extremely general to very specific is by no means unique. One may arrive a t a metric from a uniform structure and a proximity structure [Kothe, 1969: Willard, 19701.
Spaces
'
31
hollow space (1)
- {
'd
~.~...
/f semimetric space (1),(2),(4) , ,
I
, :, ,, ,, ,, ,
......................................
if
,
, , ,, ,,
~
'
inner product space
,#
Hilbert space
/
j
i' 2 :a, 6
:'
jC
I
,, ,I .-.-
>,: m8 i%
normedspace
'
,0 , ,
h
I 7
metric space (1)-(4)
\
(Euclidean spa&
quasimetric space (1)-(3)
-..~
,j
Figure 2.3 Schematic relations between some classes of generalized metric spaces. The numbers correspond to the conditions of Def. 2.38.
some classes of spaces from these two perspectives; see Figs. 2.2 and 2.3 for schematic diagrams. In these pictorial schenias, if one space is 'embraced' by another, it is either more restricted than or a special case of the first. For instance, a (finite-dimensional) Euclidean space is a particular case of' a Hilbert space, which, in turn, is an inner product space and a special case of a Banach space. The latter is an example of nornied spaces, which, if metric is defined, becomes a metric space. If the metric requirements are weakened, then more general spaces, like quasimetric or premetric spaces are obtained. Scctioris 2.3-2.7 will mainly provide definitions, theorenis and basic characterizations of various spaces. The presentation is by no means complete. Only basic facts are chosen to show how spaces, rich in structure, are created from simple ones. Most of the proofs are omitted as they can be found in standard textbooks. The ones presented here are either new or have an educational value. We think that this chapter is essential to build an understanding of various spaces, even if the theory can be mostly found in standard books on algebra, functional analysis and linear spaces. The goal of our presentation is to prepare for the introduction of Krein spaces and dissiniilarity representations later on, and to make the reader aware of specific properties of a Euclidean space, which do not necessarily hold in other spaces. This is an important point as our scientific intuition often relies on early experiences with low-dimensional Euclidean spaces. This niay lead to hidden assumptions or expectations, which prevent us from investigating more general learning theories. Although Euclidean (Hilbert) spaces proved to be good frameworks for statistical learning paradigms, there are many practical problems that can-
32
The dzssimilarity representation foT pattern recognition
not be directly explained there. Such spaces have strong limitations, as they are rich in algebraic structure. One, therefore, needs to make compromises or to investigate other possibilities, which would hopefully lead to alternative approaches. 2.3
Generalized topological spaces
Stantlard t,extbooks on topology define a topology on a set X by the means of a collection of open sets. Open sets are the basic notion of topology. For instance, in application to digital image processing, they are used to construct a digital topology on the 'integer planei ZXZ; see [Khalimsky, 1987; Khalirnsky et al., 1990; Kong et al., 1991, 19921. When topology is discussed in norined vector spaces, the norm defines a metric distance, which is used to construct open ball neighborhoods, defined as B,(z)= {y E X : d ( z . y ) < E } , for positive E . These open sets determine the natural topology in metric spaces. The concept of neighborhood is, however, more fundarricntal than the concept of distance, since a metric (normed) space is already a high-level construction with a high degree of geometric structure; see Figs.2.1, 2.2 and 2 . 3 . Topology can be derived in a bottom-up way, whcrc the notion of a norm or a distance is not yet available. This can be achieved by the use of neighborhoods or generalized closure operators. For an introduction to standard topology, see e.g. the books of [Willard, 1970; Munkres, 20001. For more general topics, please refer to the following books [Sierpiiiski, 1952: G a d , 1964: Cech, 1966; Kothe, 1969; Kelley, 19751, as well as the articles [Day, 1944; Gastl and Hammer, 1967; Gnilka, 1994, 1995; Stadler et al., 2001, 2002; Stadler and Stadler, 2001bl. Our point is that neighborhoods or generalized closure operators can be considered as basic concepts t o build a (pre)topological space and t o express the relations between objects. This can be especially advantageous when one directly works with a represcntation domain of objects, such as a collection of strings. Since our analysis starts from dissimilarity relations between a set of' examples, the neighborhoods will be defined by the use of dissimilarities in generalized metric spaces. One of the most crucial characteristics a space should reflect is the notion of closeness (nearness), i.e. being able to tell whether two elements are near or not. Note that, at the most basic level it might be impossible to distinguish that an element n: is nearer to z than to y , although it can be judged that z is near to both y and z . So: the relation of nearness may be based on the relations between sets and may not be quantitative. It does
Spaces
33
not need tjo be symmetric, i.e. J: can be near to y, but not vicc versa. (The nearness can also be seen as an asymmetric resemblance relation, e.g. of a child to a parent.) A further study in this direction may lead to the so-called proximity spaces [Willard, 1970; Cech, 19661. A possible formalization of the notion of nearness for the set X can be made by defining for each element II: E X a collection of subsets of X , called neighborhoods of 2 . Intuitively, the basic properties of neighborhoods should be that each element, 5 is contained in all its neighborhoods, any set containing a neighborhood is a neighborhood, so consequently the entire set is the largest neighborhood of each of its points. Below, formal definitions are presented. Definition 2.12
(Generalized topology via neighborhoods)
Let
P ( X )be a power set of X , i.e. the set of all subsets of X . The neighborhood function hf: X + P ( P ( X ) )assigns to each x E X the collection n/(z)of all its neighborhoods of z such that:
(1) Every
:L'
belongs to all its neighborhoods:
Y 2 c x Y N E N ( ~2 )E N . (2) Any set containing a neighborhood is a neighborhood: =+ hfEN(2)). vINEN(Z)v'nrcx ( N c ( 3 ) The intersection of t,wo neighborhoods is a neighborhood: ~ I N ; M a v ( 2N ) nM E N ( Z ) . (4) For any neighborhood of IC, there exists a neighborhood of a neighborhood of each of its elements:
2
that is
Y N E N ( Z j 3M€N(Zj Yiy€,R.l ME"Y).
The pair ( X ,N ) with N satisfying the first two requirements is a neiyhborhood space [Gastl and Hammer, 19671. The pair ( X , N ) ,obeying conditions (1) ( 3 ) is called a pretopological space. If all conditions are satisfied, then ( X ,N ) becomes a topological space. ~
Definition 2.13 (Neighborhood basis) A subfamily &(x) of the neighborhood system N ( x )is a neighborhood basis (or a local basis) at :I: if the following conditions are fulfilled:
(1) V N E I v B ( 5 ) Z E N . (2) YN,"tND(Z) 3MENB(2)M
c N n N'.
A neighborhood basis uniquely defines a pretopological space. This follows, since a neighborhood system satisfying the conditions (1) ( 3 ) of Def. 2.12 is built by taking all subsets of X larger than the basis neighborhoods, i.e. N ( z ) = (111 C X : 3 N E ~ B ( zN) c M } . Therefore, instead ~
34
The dissimilarity representation f o r pattern recognition
I
I
(b)
Figure 2.4 Illustration on neighborhoods. (a) Examples of neighborhoods of z from the set X . (b) A nested neighborhood basis of X . (c) A neighborhood N of a set Y C X ; dashed ovals correspond to neighborhoods of elements ?/ E Y.
of considering a complete neighborhood system, only a neighborhood basis can be used for the definition of pretopology. Note that a pretopological space niay have many bases, each of them capable of describing the entire space. Neighborhood systems are a general tool to represent knowledge on relations between the elements of a set X . A neighborhood of an element II: is somewhat similar to 2 , however, since it is a set, its elements are not distiriguishable froin 2 . When additional structure is added, the e1ement.sin neighborhoods may become distinguishable. For instance, neighborhoods can be defined by the use of binary relations; siniilarity and dissimilarity measiires or hierarchic systems. See also Fig. 2.4 for an illustration on neighborhoods.
Example 2.2 (Neighborhood bases) (1) Let X = { a ,b, c, d , e}. The neighborhood basis emphasizes particular rclations between the elements. For instance, for the relations on the right side below, it is defined as:
Ni?(a)= { { a > ,{ a ,b, .>I. NB(0)= { { u , b : c } } .
NB(c) = { { c } ,{a?b; c}, { c , d. e } } . N B ( d ) = {{c.d,e}}.
N B ( e ) = { { e } , { c ,d , e } } Extension of the above neighborhood relations to a set of integers is the Khalimsky line: used to define a digital topology [Khalimsky, 1987; Khalimsky et al.: 19901. (2) Let p : X x X + 'w? be a general dissimilarity measure as in Def. 2.45, such that p ( z , x) = 0. Then B~(z) = {y t X : p(x,y) < S } is a neighborhood of z for a given 6> 0. The neighborhood basis is then defined as
NB(2)= {&(z): &>0}.
Spaces
35
( 3 ) Let X be a set. A hierarcliical clustering (see Sec. 7.1) can be seen as a successive top-down decomposition of X, represented by a tree. The root describes the complete set and it is the largest cluster. Its children nodes point to a decomposition of X into a faniily of pairwise disjoint, clusters. Each cluster can be further deconiposed into smaller clusters, represented by nodes in t,he tree, until the single elements in the leaves. In this way, sequences of nested clusters are created. A neighborhood of II: is a cluster ch at the level h in the subtree containing the leaf 2 . Then &(x) = {CtL: z E Ch,}. Notc that the requiremerit of disjoint clusters at each level is riot essential for the definition of N B ( Z ) .
Definition 2.14 (Neighborhood of a set) Let ( X ; N )be a pretopological space and let Y C X . Then N is a neighborhood of 1’ iff N contains a neighborhood Nu of each E Y . The neighborliood system for I’ is then n/(g). See also Fig. 2.4(c). given by N ( Y )= Definition 2.15 (Open and closed sets via neighborhoods) Let X be a set. A 2 X is an open set if it is a neighborhood of each of its elements, i.e. V l z E ~A E N ( z ) . A is a closed set if (X\A) is open.
A neighborhood function N defines a generalized topology on the set X , as presented in Def. 2.12. Neighborhoods can be used to define genersliaed interior and closure operators, which may further define open arid closed sets, the basic concepts in a topological space. Since the properties of t,hc neighborhood, closure and interior functions can be translated into each other, t,hey are equivalent constructions on X . For instance, a generalized closure can be considered as a principal concept to define other operators on sets [Gastl and Hammer, 1967; Stadler et al., 2001; Stadler arid Statiler, 20021. Definition 2.16 (Generalized closure) Let P ( X ) be a powcr set of X. A genegrulized closure is a function P ( X ) + P(X)which for each A c X assigns A - c X such that 0- = 0 arid A c A-.
The generalized closure is not idempotent. This means that for A c X . the condition A - - = A- does not necessarily hold, as required for the topological closure. The interior function and neighborhood system N can be now defined by the generalized closure. Definition 2.17 (Generalized interior) Let P ( X )be a powrr set of X . A generalized znterzor is a function P ( X ) 4 P ( X ) which for each subset A
T h e dissimilarity representation for pattern recognition
36
Table 2.1 Equivalent axioms for the neighborhood system and the generalized closure operator. X is a set and A , B , N , M represent any of its subsets. Axioms (1)-(3) describe neighborhood spaces, axioms (1)-(4) define pretopological spaces and axioms (1)--(5) define topological spaces. Closure A -
Propcrties
(5) Idempotent
of X assigns a subset A" of X such that A" one can write that A- = X\(X\A)".
=
X\(X\A)-.
Equivalently,
Definition 2.18 (Neighborhood system) The neighborhood N : X + P ( P ( X ) )is a function which for each Z E X assigns the collection of neighborhoods defined as N(z) = { N E P ( X ) : II: $ ( X \ N ) - } . Equivalently, one can write that Z E N (X\N)$N(z).
*
Definition 2.19 (Generalized topology via closure) Let P ( X ) be the power set of X . Consider a generalized closure - : P ( X ) + P ( X ) with the following properties: (1) 0- = 0. ( 2 ) Expansive: VACX A C: A-. (3) Monotonic: V A , n-c x A C B jA(4) Sublinear: VA.BCX ( AU B ) - C A(5) Idempotent: VACX A-- = A-.
B-
u B-
If axioms (1) (3) are fulfilled, then ( X . - ) is a neighborhood spacc. If axionis (1) (4) hold, then ( X ,-) is a pretopological space. If all conditions are satisfied, (X. -) defines a topological space; see also Table 2.1. -
~
Corollary 2.1 Axioms given in Table 2.1 are eyuzvalen,t. Proof. Let X be a set and let N . M be any subsets of X . Then the basic fact in set theory is that following equivalence N C:M @ ( X \ M ) C ( X \ N ) holds. In the proof, we will make use of this and Def. 2.18, in which the generalized closure is defined by the neighborhood system. The latter means that Z E N - H ( X \ N ) @N(z). The proof follows.
Spaces
37
(1) Assume @ = 0- holds for every zEX. From Def. 2.18, 'dZEx~9'0-H ' d z a z @ ( X \ W @ Y 7 x x X EN(Z). (2)
Assume that the generalized closure is expansive. Let
:1: E
X arid
N E N ( ~ By ) . Def. 2.18, this is equivalent to z @ ( X \ N ) - . Making use of the expansive property of the closure, on has (X\N) C: (X\N)-. I t follows that X\(X\N)C ( X \ ( X \ N ) ) = N. For any z E X the following equivalence z @ (X\N)- H z E X\(X\N)holds. Since X\(X\N)- C (X\(X\N)) = N, then z E X\(X\N)+ :I: E N . Therefore, z; @ (X\N)z E N . Hence, we have proved that N E N(z) X€N.
*
x E X and N E N ( z ) . Assume that N E N ( x ) + z E N holds for any N C X . By Def. 2.18, for any 1c one has 1c E N + :I:# (X\N) + (X\N) @ N ( x@ ) z E N-.As z E N + z E N - consequently, N C N-.
-+== Let
( 3 ) Let z E X . Assume that N EN(^) and N C M . The latter is equivalent to (X\n/l) C ( X \ N ) . Since the generalized closure is monotonic, N C A!! @ ( ( X \ M ) C ( X \ N ) ) + ( ( X \ M - 2 ( X \ N ) - ) holds for all N , M C X . The latter set relation is equivalent to stating that x 9' (X\N)- + n: @ ( X \ M ) - , which by Def. 2.18: is equivalent to N E N(z)+ A f ~ " ( z ) . Since ( N € N ( z )A N C &I), then MEN(^).
C N - U M - hold for all N ; Af i X . Assume that N , M E N ( z ) . Replacing N by (X\N) and M by ( X \ M ) , one gets: ( ( X \ N ) U ( X \ M ) ) - 2 (X\N)- U (X\A,f)-. Herice 12: E ( ( X \ N ) U (X\M))- + ( Z E (X\N)- V z E ( X \ M - ) , which is equivalent to { z # ( X \ N ) - A z g ( X \ M ) - + z $ ( ( X \ N ) U ( X \ M ) ) - } . Since N , MEN(rc) and from de Morgan's law (X\N)U(X\M)= X \ ( N n M ) , the latter implication is equivalent to (N E N ( z ) A M E N(x:))+ ( N Ti M) € N ( z )by Def. 2.18.
(4) Let ( N U A d -
(5) Let z E X and N E N ( z ) . Assume that the generalized closure is idenipotent for all subsets of X . Therefore, one can write ( X \ N ) - = ( X \ N ) ) . BasedonDef. 2 . 1 8 , o n e h a s N E N ( x ) w z ; @ ( X \ N ) - -s (X\N)-- ++ (X\(X\N)-) ~ n / ( z )Let . M = X\(X\N)-. Then M E n/(:c) by the reasoning above. For all y, the following holds y E Ail ++ y @
( X \ W @ Y e (X\N)- @ Y @ (X\N)-- ++Y @ X\(X\(X\N)-)'++ (X\Ad)- @ M E N ( y ) , by Def. 2.18. Hence, we have shown that Y N E N ( ~3~.r=(x\(x\,v-)~,v(~) ) 'dy~ns EN^). 0 y@
T h e dissimilarity representation f o r p a t t e r n recognition
38
N(z) A EN(z)
Neighborhood
A c X A is open
Closure A-
InteriorA'
A = X\(X\A)-
A = A'
The difference between pretopological and topological spaces lies in the notion of a closiirc operator. In a topological space, the closure of any set A is closed, A-- = A - , and the interior of any set is open, (A")" = A". In a pretopological space, this is not necessarily true: so the basis neighborhoods are not open. Here, the generalized closure operator expresses the growth phenomenon, where the cornposition of several closures results in successive augmentations, i.e. A 5 A- C A-- C . . . .
Example 2.3 (Pretopological and topological spaces) (1) Let X be any set and let S : X X X + P ( X ) be a symmetric relation, i.e. S ( x , y ) = S ( y , x ) . Assume a generalized closure of A C: X be S ( x ,y). Then (X, - ) is a neighborhood space, defined as A- = since the generalized closure obeys conditions (1)-(3) of Def. 2.19. (2) Let X be a finite set and ( X , E ) be a directed graph. Let F ( x ) be a set of the forward neighbors of x, i.e. F ( x ) = EX: ( T C , ~E) E } . Let A X . By axioms of Def. 2.19 it is straightforward to show that the closure A - = U Z E A ( F ( xU) {x}) defines a pretopological space ( X ,-). (3) Let N B ( ~ = ){ ~ E R Ix : - y1 < E A E > 0). Then (R,&) defines a topological space. (4) Let &(z) = { ( u . m ) : U E I W A X E ( a , ~ ) }Then . (R,NB) defines a topological space.
c
Corollary 2.2 (Open and closed sets) Let ( X ,- ) be n neighborhood space de,fined by t h e generulized closure, i.e. conditiom (1)-(3) of De,f. 2.19 hold. A 2 X i s an open set if A" = A. A is a closed set if A- = A; see also Table 2.2. Th,e followin,g holds:
(i)
AEN(TC H )A = X\(X\A)-.
Y Z E ~
(2) A
=
A"
A
= X\(X\A)-.
Pro0f. (1) Assiimc that Y r E ~A E N ( x ) holds. By Def. 2.18, Y r E ~A E N ( x ) @ Y l r g ~x $ ( X \ A ) - @ Y l s € ~xEX\(X\A)-. Hence A = X\(X\A)-. (2) A = A" = X\(X\A)- by Def. 2.17.
Spaces
39
Lemma 2.1 Let ( X , N ) be a neighborhood space. T h e assertions: (1) V I N E , V (3~A)4 C N ( z ) V y t &JEN(Y) ~ and (2) VN(NE N ( z ) @ No E N(2:)) are equivalent. The proof of Corollary 2.1, point (5) shows that VINE~u(a) VyEnr hf E N(y). Since M = N o by Def. 2.17, 3M=(X\(X\N)-)EN(zL.) then No E N ( x ) . 0
Proof.
A collection of open sets containing z constitutes a neighborhood basis in a topological space, which can be proved by Lemma 2.1. Equivalently, since the closure operator is dual to the interior operator, a neighborhood basis in a topological space can be built by a collection of closed sets coritaining x. Lemma 2.2 Let (X,&) be a pretopological space. If all neighborhoods oj = No} ,for all x E X , t h e n (X,NB) is a topological space.
NB are open sets, or NB(x) = {N C X : x E N A N
Corollary 2.3 (Closure on neighborhoods) Let ( X , - ) be a neighborhood space. T h e n a genm-alized closure operator i s a function, P ( X ) + P ( X ) , defined as gcl(A) = {x E X : V N t ~ ( r ) A n N # Moreover, gcl(A) = A-.
a}.
In order to prove that gcl(A) = A- holds for any A C X, we will equivalently show that (z$gcl(A)) H (x$A-) holds for all z t X .
Proof.
=+ z $gcl(A) + 3 ~ ~ , vN (n~A )= 0.By Def. 2.18, the latter is equivalent t o (x$(X\N)- A N n A = 0). Since (N n A = 0) + (A C X \ N ) , then by the monotonic property of -, A- C (X\N)- holds. Since z $ (X\N)-, then x#A-. +=
By Def. 2.18, then z@gcl(A).
(.$Ap) + ((X\A) @ N ( x )holds. )
Since (X\A) n A = 0,
0
Definition 2.20 (Limit point) Let ( X , N ) be a neighborhood space. An element y E X is a limit of A C X iff for every neighborhood N EN(^), N intersects A\{y}. The set of all limits points is called the derived set, der(A) = EX: V N c ~ ( y )(A\{y}) n N # 0} [Sierpinski, 19521. Corollary 2.4 In a neighborhood space, der(A) tains all its limit elements and conversely.
C A - . A closed set
con-
The notion of corivergencc is important in neighborhood (pretopological arid topological) spaces. Recall that a sequence x, in X is a function from N to X: hence f ( n ) = zn. The order of elements in z , is, thereby, important.
40
T h e dzssimilarity representataon f o r p a t t e r n recognatzon
The sequence x,, is different from a set { x : n } ~ ?which l, is simply indexed by W. One would say that a sequence xn converges to x E X in a neighborhood space ( X . N ) if for every neighborhood N c N ( z ) ,there exists k E N such that xn E N for all n 2 k . The problem with this definition is, however, that neighborhoods may have an uncountable number of elements arid countable sequences may not capture the idea of convergence well. In general, convergence is defined by the use of filters, which are generalization of sequcnces.
Definition 2.21 (Filter) A filter on a set X is a collection of X such that (1) (2)
YFE3
F
F of subsets
# @.
~ F" ~ C g (~F n F'). ( 3 ) b ' p ~v ~~F ( C F' + F ' E 3 . ' d p . p i e 3~ p
If 3 satisfies only the first two conditions, then
F defines a filter basis.
Note that given a filter 3 on a set X and a function f : X + Y , the set f ( 3 )= {f(A): A t 3 } forms a filter base for a filter of the function f .
Definition 2.22 (Convergence) Let ( X ,JV)be a neighborhood space. A filter 3 converges to z E X , 3 4x if V N E ~ / ( s ~) F F~ 2 FN . One may easily verify that a neighborhood system N(z) of an element X in a prctopological space ( X , A f ) ;compare t o Def. 2.12. One may, therefore, imagine a set of nested subsets (neighborhoods) of an element 2 that defines the convergence to x. If one is given a sequence of elements 2 , for n E W, then a filter basis can be defined as {PI,k EN},where Fk is a subsequence of x ,starting from the element x k , i.e. (xk,zk+l,. . .). :c is a filter on
Definition 2.23 (Hausdorff space) A neighborhood space ( X . N )is Hausdorfl or T2 if every two distinct elements of X havc disjoint neighborhoods. i.e. Yr,yEX 3 N z E ~ ( Nz )g ,E ~ ( y )Nzn & = 0. Lemma 2.3 Every convergent filter in a Hausdorfl space has u unique limit. Functions and, especially, continuous functions are basic tools in applications of various spaces. The basic intuition of a continuity is that small changcs in the input produce small changes in the corresponding function output. where 'small' is expressed by a chosen distance. In general neighborhood spaces, one can only work with sets.
41
Spaces
Definition 2.24 (Continuity by filters) Let f : ( X , N )+ ( Y . M ) be a function between two pretopological spaces. f is continuous at :I; E X if for all filters 3 on X if .F + x, then f ( F ) f (x). --f
Definition 2.25 (Continuity by neighborhoods) Let f : ( X , N ) + (Y.M ) be a function between two neighborhood spaces. f is continuous at z E X if for each neighborhood M of f ( z ) in Y , there exists a neighborhood N of z in X , whose image lies entirely in M . f is contiriiious on X if it is continuous at every x E X . Formally, f is continiloils if holds for all EX. Yin/ic,u(f(.)) ~ N E N ( . ) f ( N ) C Theorem 2.2 (On continuous functions) Let f : ( X , N )+ ( Y , M ) be a f u n c t i o n between two neigh,borhood spaces. T h e following assertions are equi.ualent [Gnilka, 1997; Munkres, 20001: 1. f i s continuous at x. 9. For all x E X , B E M ( f ( 2 ) + ) f - ' ( l ? ) ~ N ( z. ) 3. For every set A E P ( X ) ,f ( A - ) C ( f ( A ) ) -. 4. For eiie7-77 set B E P ( Y ) ,( f - l ( B ) ) -C f - l ( B - ) . 5. For every set B E P ( Y ) ,f - l ( B " )C ( f - l ( B ) ) " . Note that in topological spaces, continuity of a function translates to the fact that the preimage of an open (closed) set is an open (closed) set.
Remark 2.1 T h e composition of finitely many contin,uous mappin,gs i s a continuous mapping. Definition 2.26 (Regular space) 0
0
A neighborhood space ( X , N ) is regular if for each neighborhood N of z E X , there exists a smaller neighborhood M of z whose closure is contained in N ? i.e. Y N ~ N ( ~31\.lc~(.) ) M - C N. A topological space is regular if every neighborhood of n: contains a closed neighborhood of x. It means that the closed neighborhoods of' I(: forin a local basis at z. In fact. if the closed neighborhoods of each point in a topological space form a local basis at that point, then the space milst be regular.
Definition 2.27 (Normal space) 0
A pretopological space is normal if the separation of the closures of two sets imposes the existence of their disjoint neighborhoods, i.e. if for nonempty sets A and B , one has (A-nB-= 0) ( ~ N * , N( ~A C NA)A(BC N B ) A (NAn N B = 0)) [Cech, 1966: Stadler and Stadler, 20021.
*
42
The dissimilarity representation for p a t t e r n recognition
Table 2.3 Properties in Neighbood spaces
Regularity axioms
Separation axioms
A topological space is normal if the separation of two closed sets imposes the existence of their disjoint neighborhoods. Neighborhood and (pre)topological spaces can be classified with respect to the degree to which their points are separated, their compactness, overall size and connectedness. The separation axioms are the means to distinguish disjoint sets and distinct points. A few basic properties are presented in Table 2.3 and scheniatically illustrated in Fig. 2.5 [Cech, 1966: Stadler and Stadler, 2001b; Munkres, 20001.
Definition 2.28 (Completely within) A set A is completely within B in a neighborhood space ( X , N ) if there is a continuous function 4 : ( X . N )+ [O. 11 such that 4 ( A )C (0) and 4(X\B) C (1). Therefore, A C B. Different pretopological spaces can be distinguished by the way they 'split up into pieces'. The idea of connectedness becomes therefore useful.
Definition 2.29 (Connectedness) A space X which is a union of two disjoint non-empty open sets is disconmected, and connected, otherwise. Equivalently, a space X is connected if the only subsets of X which are both open and closed are the empty set arid X . Definition 2.30 (Cover) Let X be a set. A collection of subsets w C X is a coiier of' X if X = U w . A cover is finite if finitely many sets belong to it. If w and w' arc covers of X , then w' is a subcover if w' c w .
Spaces
REG
QN
43
--
TO
TI 1
T2
T2 t
T3
T,
t
0
Figure 2.5 A pictorial illustration of the regularity and separation properties; based on [Stadler and Stadler, 2001bl. Neighborhoods are drawn as ovals and closures are indicated as filled ovals. The regularity condition REG demands for each neighborhood N the existence of a smaller neighborhood whose closure is contained in N. The quasinormality axiom Q N requires that the separation of the closures of two sets iniposes the existence of their disjoint neighborhoods. To means that for two distinct elements, there exists a neighborhood of one of them such that it does not contain the other element. TI states that any two elements have neighborhoods with the property that the neighborhood of one element does not contain the other element. Tz imposes the existence of disjoint neighborhoods for any two elements. T' asks for the existence of neighborhoods such that their closures axe disjoint, for any two elements. T3 demands that for each neighborhood N, there is a set h!! which is completely within N.
Definition 2.31 (Compact space) A topological space X is compact if every open cover has a finite subcover. A topological space is locally compact if every element has a compact neighborhood. Theorem 2.3 Let f : X + Y be a continuous function betweesn topological spaces. If A is a compact subset of X , then f ( A ) i s a, compact subset of Y . Theorem 2.4 A closed subset of a compact set is compact. A compact subset of a Hausdorfl space i s closed. Definition 2.32 (Dense subset) Subset A of a topological space ( X .- ) is dense in X if A- = X . Equivalently, whenever N , is an open neighborhood of Z E X the ~ set NT n A is non-empty. Definition 2.33 (Size) 0
0
A topological space is
separable if it is a closure of a countable subset of itself, or in other words if contains a countable dense subset. first-countable if every element has a countable local basis; see also Def. 2.13.
44
T h e dissimilarzty representation f o r pattern recognition
second-countable if‘ it has a countable basis for its topology. Second-countable spaces are separable, first-countable and every open cover has a countable subcover. Example 2.4 1. Every topological spaces is dense in itself. 2. Let (R,Ng) be a topological space withNB(x) = ( ( ~ - - E , Z + E ) : E > O } . Then by Corollary 2.3, the set of rational numbers Q is dense5 in R, i.e. Q- = R. Consequently, Iw is separable, as Q is countable. More generally, R” is separable. 3. A discrete topological space ( X , N g ) is a space with Ng(x) = {z}, i.e. the basis consists of single elements. This means that every subset of X is both open and closed. Every discrete space is first-countable and second countable iff it is countable.
Definition 2.34 (Topological product space) Suppose X i , i = 1:2. . . . , *rL are given sets. The set X of all n-tuples ( X I ,2 2 , . . . , x T L )z ,iE X i is a Cartesian product X = X ~ X X ~ . X xX, . .= X i . Let ( X i , N i ) , i = 1 , 2 , . . . , n be (pre)topologicaI spaces. (X,n/) is a (pre)topological product space if n/(z)= hii(xi) is a neighborhood basis of Z E X .
n;=,
Remark 2.2 T h e definitions above can be extended t o a n y (countable or n o t ) ,family of topological spaces. T h e mapping 7ri : x + z i i s a projection o,f X onto X i . It is a continuous mapping and the topology defined o n X is the weakest topology for which all the projections ~i are continuous [Kothe, 19691. Topology (pretopology) can be introduced on a set in many ways. It can be defined by a collection of open sets, or generated by a neighborhood basis, a (generalized) closure or other operators. The way it is introduced specifies particular ‘closeness relations’. One should, however, remember , that new topologies can always be added to a set. Some topologies can be cornpa,red, however not all of them are comparable.
Definition 2.35 (Weaker and stronger topologies) Let X be a set arid let H , M be two neighborhood systems defined for every x E X . The topology defined by N , the N-topology is stronger (finer) than the topology defincd by M , the M-topology if for each x E X every neighborhood ‘Informally, one may think that a subset A is dense in X if the elements of A can ‘approximate’ the elements of X with the arbitrary precision with respect to X .
Spaces
45
M E M ( x ) is also a neighborhood of n/(z). It means that N has more neighborhoods than M . The M-topology is then ,weaker (coarser) than the N-topology. If' neighborhood bases NB arid M B are considered, then the N-topology is stronger than tlie M-topology if for each z E X and every basis neighborhood M B E M B ( Z ) there , is a basis neighborhood of N B E N B ( such ~ ) that N B c MB. If finitely or infinitely many topologies are defined by N, on a set X : there is tlie strongest, (finest,) topology specified by n/ among the topologies on X which are weaker (coarsest) than every n/,-topology. This nieaiis that every neighborhood of n/(x) is a neighborhood of N, for every 0.
Definition 2.36 (Homeomorphism) A bijective function6 f : X i Y between two topological spaces (X,n/) and (Y,M ) is a ho,meomorphism if both f and f - l are continuous. The spaces X arid Y are homeoniorph,ic. The homeomorphisms form an equivalence relation on the class of all topological spaces. Therefore, homeomorphic spaces are iridistinguishahlc as topological spaces; they belong to the same equivalence class. Two homeomorphic spaces share the same topological properties. e.g. if one is compact, connected or Hausdorff, then the other is as well. This also means that a set N g N ( z ) is open in X iff the set f ( N )~ M ( f ( x is ) )open in Y . Moreover, a sequence z, converges to z iff the sequence f ( ~ converges , ~ ) to
f (x). Remark 2.3 T h e identity m a p I : ( X , N ) ( X , N ) ,where I(%) = z is a homeomorphism when the same topology (neighborhood systems) are used over the domain and the range of the map. In general, it i s not true, if two di.fferent topologies are defined o n X . Let N B ( z )= X and M n ( x ) = {x} be the neigh,borhood bases for all 2 E X . T h e n N consists of X and M is u power set of X (without the empty set). B y Dgf. 2.25, I i s con,tinuous at z z f f o r all hl E M ( z ) there exists N E N ( : c ) such, that f ( N ) C A f . As N = X fm- all z and there exists hf = {x} such that f ( X ) @ {x}, then I : ( X ,N ) 4 ( X ,M ) i s discontinuous at each point IC.
Proposition 2.1 Let n/ and M be two neighborhood systems de,finined o n a topological space X . Th,e identity m a p I : ( X ,N ) 4 ( X ,M ) is continluous iff the N-topology is stronger than the M-topology. 6A bijective function f always has an inverse f-', even if f is.
but not necessarily continuous,
46
The dassemalaraty representataon f o r p a t t e r n recognition
Equivalence relation on a set is a binary relation between its elements, such that some of them become indistinguishable by belonging to the same class. In the study of spaces, a quotient space is the result of identifying such classes by an equivalence relation. This is usually done to construct new spaces from given ones.
-
-
Definition 2.37 (Quotient space) Let ( X , N )be a topological space and let be an equivalence relation on X. Denote by X/ the set of equivalence classes of X under -. Let i7 : X + X/ be the projection map which sends each element of X to its equivalence class. The quotient topology on X/ is the strongest topology (having the most open sets) for which 7r is continuous.
-
Remark 2.4 If X is a topological space and A c X , we denote by X / A a quotient space of the equivalence classes X/ under the relation x y zf x = y or 2 , y E A. So for x @ A , {x} is a n equivalence class and A is a single class.
-
2.4
-
Generalized metric spaces
A set can be augmented with a metric distance, or a structure weaker than metric, which leads to generalized metric spaces. A metric can also be introduced to vector spaces. They are, however, discussed in thc subsequent section. Most of the material presented here relies on the following books [Bialynicki-Birula, 1976; Blumenthal, 1953; Dunford and Schwarz, 1958; Kiithe, 1969; Kreyszig, 1978; Willard, 19701.
Definition 2.38 (Metric space) A metric space is a pair ( X ,d ) , where X is a set and d is a distance function d : X X X i R; such that thc following conditions are fulfilled for all x,y, z E X : (1) Reflexivity: d ( x : x )= 0. (2) Symmetry: d(x,y) = d ( y , z). ( 3 ) Definiteness: ( d ( z ,y) = 0) + (x = y). (4) Triangle inequality: d ( z , y ) d(y, z ) 2 d(x,z ) .
+
For instance, X can be R", Z", [a,bIm, or a collection of all (bounded) subsets of [ a ,b]". If X is a finite set, e.g. X = { X I ,5 2 , . . . , x n } >then d is specified by an n x n dissimilarity matrix D = (&), i,, j = 1,.. . , n such that di,i = d ( x i , zj).Consequently, the matrix D is nonnegative, symmetric and has a zero diagonal.
Spaces
47
Example 2.5 Examples of metric spaces: 1. Let X be any set. For x,y E X , the discrete distance metric on X is given by d(x,y) = Z ( x # y), where Z is the indicator (or characteristic) function. If X is a finite set, then all the pairwise distances can be realized by points lying on an equilateral polytope (extension of an equilateral triangle and of a tetrahedron). 2. Let X be a set of all binary sequences of the length 111. Given two binary strings s = ~ 1 ~ 2. s,, . and t = t l t 2 . . . t,, the Hamming distance is defined as d H a m ( s , t ) = Z(sk # t k ) . 3. Metrics in a vector space R". To emphasize that a vector x comes form a finite-dimensional vector space Rm,we will mark it in bold:
c;;r"=,
(a) d, (x,y)= (CEl (xi- yzi")h, p 2 1; a general Minkowski distance. (b) dl (x,y) = Ixi - yil,the city block distancc. (c) dz (x,y) = d~ (x,y) = (CE,(zz- yi)')+, the Euclidean distancc. (d) d, (x.y ) = d, ( x y~) = maxi (xi- gi(,the max-norm distance.
El"=,
4. Let F ( 0 ) be a set of real-valued functions defined on a bounded and closed set 0. Let M ( 0 ) c F ( 0 ) be a set of function cl which are Lebesgue measurable on R. Then L f = {.f E M(62) : (Jn If(x)l"dz)l/l-'
(hi
( f ( t )- d t ) I pd t ) (a) F = Lf, d,M(f,3) = (b) 7 = d 3 f ; d = "PtEn lf(t) - s(t)l.
Lg>
Similar metrics can be defined on the set of continuous fuiictiori C(n). 5. Let X be a set and (Y,d)be a metric space. The set of all bounded functions f : X i Y can be considered as a metric space with the distance d ( f , g ) = suparX d ( f ( z ) , g ( z ) ) for any bounded functions f and y.
Theorem 2.5 (Backward triangle inequality) In a m,etrzc space ( X ,d ) , th,e backward triangle inequality, (d(.lc,z ) - d ( y , z ) ( 5 d ( z . y), holds for a11 x,y . z r X . 7Note that i f f is zero nearly everywhere on R except for a countable number of points, then f is a non-zero function, but I f ( z ) I P d z ) ' / p = 0 for p 2 1. Hcnce, such f is indistinguishable from a zero-constant function. Thercfore, an equivalence relation is introduced such that f g if (,[, if(.) - g ( z ) l p d z ) ' / P = 0. One, therefore, considers function classes defined by the equivalence classes.Since now on, M(CLL)will denote such
-
function classes
(s,
-
T h e dissimilarity representation f o r pattern recognition
48
Proof. By triangle inequality, d ( z ,z ) d(y, z ) 5 d ( z ,y) and d ( y , z ) d ( x , z ) 5 d(y,x). Since d ( y , x ) = d ( z , y ) , the latter inequality becomes 0 - ( d ( z , 2 ) d ( y , z ) ) 5 d ( z , y ) . Hence, the inequality follows. ~
~
~
Theorem 2.6 (Natural topology in metric spaces) Every metric space (X, d ) with a n open bull neighborhood basis i s a topological Hausdorff space.
Proof. Let B E ( x ) {y E X : d ( z ,y) < E } be an open ball. To show that ( X ,d) is a topological space, it is sufficient to prove that the neighborhood basis NB(z)= { B E ( z:) E > 0) defines a topology on X. Below, we show that indeed NB(II;) is a neighborhood basis, i.e. the axioms of Def. 2.13 are fulfilled. ( I ) Obviously, d(x,x) = 0 < E , which means that z E B,(x)for any E > 0, so axiom ( I ) is satisfied.
(2) Consider any BE(x) and B,(z) for ~ , >q 0. Let C 5 min{E,q}. Then d ( x , y ) < < 5 rnin{&,q}, which means that if ~ E B then c , y~B,(x)flB,(z), hence the inclusion B<(z) C B E ( x )fl B,(z) holds. Therefore, axiom (2) is fulfilled and the &-ballsare the local basis. By Lemma 2.2 we need to prove that the &-ballsare open sets. By Def. 2.15, B E ( x )is an open set iff for all y E BE(x) + B E ( z )E N(y). We will first show that for each y ~ B , ( x )there exists q such that B,(y) C B,(x).Let y ~ B , ( x ) .Let q be such that 0 < q 5 &-dd(z,y). Let z ~ B , ( y ) .This means that d ( y , 2 ) < rl 5 E - d ( z ,y), which leads to d ( z ,y) +d(y, z ) < E . By the triangle inequality, we have d ( z , z ) < E , which stands for ,zEB,(z). Hence, we show that z E B , ( y ) =+ z E B , ( z ) , hence B,(y) C BE(x). By axiom (2) of Def. 2.12, any set enclosing B,(y) belongs to N(y),hence B E ( z )EN(^) for all y ~ B , ( z ) Therefore, . N B ( ~consists ) of open sets. Consequently, by Leinrna 2.2, &(z) defines a topological space. The fact that every metric space ( X ,d ) is Hausdorff (see also Def. 2.23) can be shown as follows. Let x , y E X and E = d(z,y)/2. Then the opcn balls 0 are disjoint, i.e. B E ( x )n B,(y) = 8 and z ~ B , ( z and ) yEB,(y). Since a metric space is Hausdorff, every sequence has at most one limit and every subsequence is convergent to the same limit. This has an impact on applications. Solutions to many practical problems can be expressed as iterated function systems in some metric space. These properties ensure that if such systems are convergent, they are convergent to a unique solution. In practice. however, an additional property of completeness,
Spaces
49
Def. 2.42, must be required, which takes care that the limit exists in the domain of interest.
Definition 2.39 (Dense subset in a metric space) Let ( X , d ) be a metric space. The subset Y of X is dense in X if it has a finite cover by &-ballsfor any positive E . Formally, V E x VzExSyty d(x,y) < E holds. Remark 2.5 X i s separable i f X contains a countable subset th,at i s dense
in X . Definition 2.40 (Distance between two sets) Assume a metric space ( X , d ) . Let A and B be two subsets of X . Then the distance between two sets is defined as & ( A , B) = infaEA,btB d(a, b ) . Note that & ( a , B) = inf/,,B d ( a , b ) . Theorem 2.7 A m e t r i c space ( X ,d ) is a nmrmal space and ,first-countable (with, a cowntable neighborhood base a t each poin,t).
Proof. We consider the topology defined by open hall neighborhoods, B,(n:) = {y E X : d ( z ,y) < E } , E > 0. Let A and B be two disjoint closed sets. Define U = { x E X : d s ( z , A ) < $ d s ( z , B ) and V = { ? / E X: & ( v , B ) < i d s ( y , A ) . Note that A c U and B c V . Moreover, U and V are disjoint open neighborhoods of U and V . Hence, by Def. 2.27, every metric space is normal. A countable neighborhood basis is formed as N B ( ~ = ){B;(z) : n = 1, 2 , . . .} and BL(x)= { y E X : d ( z , y) < k}. 0 Definition 2.41 (Convergence) Let ( X , d ) be a metric space. The z, = z if limTsioo d ( x n 33;) = 0. sequence z, converges to z E X , limTL+m Equivalently, 2 , converges to z if the open ball B E ( %=) { g EX: d(z,y) < E } contains a tail of 2 , . In a metric space, limit points of a set are always limits of convergent sequences of the elements of the set. As filters are needed for a convergence in a general topological space, in a metric spaces, sequences can be used.
Definition 2.42 (Cauchy sequence, complete space) Let (X,d) be a metric space. A sequence 2, E X is C a u c h y iff limn.m--tmd ( ~ znl,) , ~ , = 0, which is equivalent to stating that VEx, 3~ b ' n , 2 , m > ~d ( z n , 2 , )< E . A space (X,d ) is complete if every Cauchy sequence converges in X . Theorem 2.8 (On metric spaces) 1. I n a m e t r i c space every convergent sequence i s Cathy, but n,ot conversely. T h e latter is illustrated in E x a m p l e 2.6.
50
The dissimilarity representation for pattern recognition
2. In a m e t r i c space, t h e distance d i s continuous, i.e. t h e convergence of a n y t w o sequences x,, yn E X t o x and y, respectively, implies t h a t limn--iood ( z n , yn) = d ( z , y). 3. A closeds subset of a complete m e t r i c space i s complete u n d e r the i n duced metvic. 4. A complete subset of a m e t r i c space i s closed. 5. E v e r y m e t r i c space i s a subset of a complete space.
Example 2.6 Examples of complete and non-complete spaces: is Cauchy, but not 1. ((0.11, dz) is not complete. The sequence 5 , = convergerit in (0, 11, since limn-oo x, = 0. 2. (RTr'> d 2 ) is complete. 3. Let 12 be a closed and bounded set in Wm and C(R) be a set of continuous functions on 62. (C(R), d,) is complete. 4. Let R be a closed and bounded set in Rm. (C(fl), d p ) for 1 5 p < 00 is not cornplet,e,since some of the Cauchy sequences converge to discontinuous functions [Kurcyusz, 19821. If M(R) is a set of classes of functions measurable in the Lebesgue sense, then (M(R),d p ) is complete. 5 . Since every metric space is a subset, of a complete space [Sierpiriski. 1952; Munkres, 20001, it can be completed by adding the limits of all convergent sequences. For instance, ((0,1],dz) can be completed to ([0,I], dz).
Definition 2.43 (Bounded, totally bounded subset) Let ( X , d ) be a metric space. A subset Y of X is bounded if there exists M > 0 such that d(y1. yz) 5 M for all y l , yz E Y . Y of X is totally bounded if for all E > 0 there is a finite subset { ~ i } r =of~Y ( n depends on E ) for which Y C U ~ ~ " = , B , ( pholds. i) Note that if Y is a bounded subset of
W",then it is totally bounded.
Theorem 2.9 A m e t r i c spa,ce ( X ,d ) i s compact $ f i ti s com,plete a n d totally bomded. Theorem 2.10 (Heine-Borel) closed a n d bounded.
A subset Y of IWTL i s compact 2 8 it i s
An interesting example of a metric space of a practical importance in evolutionary biology and clustering problems is an ultrametric space; see also Sec. 3.2. The reason is their close relationship to trees (hierarchical organizations) [Hughes, 2004; Fiedler, 19981. s A closed subset of X is a closed subset in the natural topology of X.
Spaces
51
Definition 2.44 (Ultrametric space) An ultrametric space is a pair (X,d ) , where X is a set and d is an ultrametric distance function d : XXXi Rt.d is a metric satisfying the ultrameti-ic inequality, also called a strong trian,gle inequality: d ( z , z) I max { d ( z , y), d ( y , z ) } ,
zt X
2, y 3
Note that the inequalit,y above imposes the triangle inequality. Recall that a ball neighborhood in a metric space (X, d ) is defined ils BE(:c)= {y E X : d ( z , y ) < E } . A neighborhood basis NB(x) = { B E ( x:) E > 0) defines a natural topology on X ; Theorem 2.6. So, the ball neighborhoods, also called open balls, are open sets, while closed hails, B ; ( z ) = EX: d(x,y) I E } , are closed sets.
Theorem 2.11 (Properties of ultrametric spaces) The ,followin,g properties h,old in an ultrametric space ( X ,d ) [Hughes, 2004]. If two open, ball neighborhoods intersect in X , then o’ne contains the other. If’ t u o closed ball neighborhoods in,tersect in, X , then, on,e contains the other. Every point in an) open bull rLeighborh,ood i s i t s center, a.e.
Definition 2.45 (Generalized metric spaces) Let X be a set and p : X X X -R: be a dissimilarity function. If t,lie requirements of Def. 2.38 hold, then p is a distance function. If these requirements are weakened, spaces with less constraintsg are considered; see also Fig. 2.3: 1. hollow space - a space (X,d ) obeying the reflexivity condition. 2. premetric space - a hollow space ( X ,p) obeying the symmetry constraint. gTerminology is not unified; it varies between authors and contexts
T h e dissimilarity representation f o r p a t t e r n recognition
52
3. quasimetric space - a premetric space ( X , p ) obeying the definiteness constraint. 4. semimetric space - a prenietric space ( X , p ) satisfying the triangle inequality. 5. A hollow space ( X ,p ) satisfying the triangle inequality [Bonsangue et al., 19981.
Example 2.7 Examples of generalized metric spaces: 1. Let X = { N ( p ,D ) } be a space of one-dimensional normal distributions. The Mahalanobis distance, or the Fisher ratio [Duda et al., 20011, defined as dA/f(N(pl,Dl),H(pZ,0 2 ) ) = (o:+m,",f 'cL1-cLz' is premetric; see Fig. 2.6.
R" and k , 1 2 k 2 7 n is a fixed integer. Then the distance measuring the absolute difference along the k-th dimension, defined as dkPrank(x,y)= / z k- ykl, is semimetric. Let ( R , d ; p ) bc a measurable space, in which R is a set, A is a 0algebra of subsets on R and p is a measure. Then d,(A, B ) = p (AAB), AAB = (AU B ) \ ( Afl B ) , is semimetric [Willard, 19701. Let X be a sct of closed subsets of RmL.Similarly, as above, the mdimensional volume symmetric difference d,,l(A, B ) = vol (AAB)is semimetric. The definiteness condition is not fulfilled, since d,,~ (A, B ) = 0 for A and B being finite collections of points. In the pattern recognition area, this dissimilarity can be computed between two matched shapes as the area of non-overlapping parts; see also Fig. 1.2(b). Let. ( X , p ) be a semimetric space. Let the equivalence relation N be defined as z N y iff p ( z , y) = 0. If X" is the set of equivalence classes [x] in X under this relation, then p- defined on X" such that p"([z], [y]) = p(x,y) is a metric on X" [Willard, 19701. The space (R"', d p ) , where dp(x,y) = (Czl1~ ~ y i l " ) ; and p~ ( 0 , l ) is quasirnetric.
2. Let X
=
dkPrank
3.
4.
5.
6.
Proof. We will prove that the triangle inequality does not hold. Consider m = 2 and A = [O, 1IT,B = [O,0IT and C = [1,0IT. Then d p ( A , B )= d p ( B , C )= 1 and d p ( C ; A )= 2 ; . Finally, d,(A,B) d p (B, C ) = 2 < 2 = d p (C,A ) , since p < 1. Hence, the triangle inequality is violated. 0
+
In generalized metric spaces, the definition of convergence and of a Cauchy sequence are adopted from the metric case, Def. 2.41 and Def. 2.42.
Spaces
53
Figure 2.6 Mahalanobis distance between one-dimensional normal distributions is premetric. The reflexivity and symmetry conditions are satisfied, but the definiteness arid triangle inequality are not. Although A and B are different, d ( A , B ) = 0. Let u = U B = uc and lp~g- ~ c= / a. Then dn,r(B,C) = a / ( f i c ) and d n f ( C , A ) = ./(a2 + a : ) $ . Since U A >u,d n r ( C , A ) i d n f ( A ,B ) < d n f ( C ,B ) .
Definition 2.46 (Convergence) Let ( X ,p ) be a quasimetric space. An clement z E X is called a limit of an infinite sequence x,, limn+m x,, = 2 if limn-m p(z,, x) = 0. Definition 2.47 (Continuity of a dissimilarity) Let ( X ,p ) be a genis continueralized metric space. Dissimilarity function p : X X X + ous a t x and y if for any two sequences x,. y, E X , xn = z and limn+m y, = y implies that limn+m p(x,, y n ) = p ( z , y). Moreover. p is continuous in X if it is continuous for each pair from X . Note that all ‘nice’ properties of a dissimilarity measure, such as continuity. convergence of a sequence to one limit, Cauchy convergent seqiiencrs can be considered for the metric only. A generalized metric space may not fulfill these conditions.
Example 2.8 Let ( X ,p) be a quasimetric space. 1. X is not necessarily a Hausdorff space. A sequcnce may have more than one limit. Proof. Consider a quasimetric space ([0,1],p ) such that p ( z , y) = I y - d if z,yE [0, I ) , p(z. 1) = p(x,O) if n: E (0, I ) , p(1,O) = p(O,l) = 1 and p ( l . 1 ) = 0. Then the sequence T , = converges to both 0 and 1, since 0 both p ( : , 0) and ~($1) have the limit zero if n 00. 2. The dissimilarity p is not necessarily continuous. Proof. Consider a quasimetric space ([0, 11.p) such that p ( r , y ) = 2 if x , y E (0, l} and x # y, and p(x, y) = /z - yl, otherwise. Then p is we discontinuous for the pair (0, l ) , since for xn = and yT1 = 1 have limn-wx, = 0 and limTl+wyn = 1, but p(z,\y,) = 1, while p ( z , y ) = 2. --f
~
k,
54
T h e dissimilarity representation f o r p a t t e r n recognition
3 . An infinite sequence of elements from X might be convergent without being Cauchy. Proof. Consider a space ([0,l ] , p ) , such that p ( z , y ) = 1 if 5 = 1 y = and n # m,, and p ( z , y ) = Iz - yl, otherwise. Then for IT1 1 2 , = ,; limn + ooz, = 0. So, z, is convergent, but not Cauchy, since p(k, = 1. 0
i,
A)
Theorem 2.12 If ( X , p ) is a quasimetric space with a continuous dissirnilarity p, t h e n f o r all 2 E X and all E > 0 , &(z) = { y E X : p ( z , y) < E } is urL open set [Sierpi.riski, 19521. Proof. We will use Corollary 2.4 stating that a, closed set contains all its limit elements arid conversely. To prove that B,(z) is open, we will show that the complementary set Y = X\B,(z) = {y E X : p ( z , y) 3 E } is closed. Let z be a limit element of Y , which means that there exist elements z, E Y such that z, converges to z . From continuity of p, p(z,,z) + 0. Since z, E Y , then for any z E X , one has p ( z , z,) 2 E . From continuity of p, it follows that p ( z , z ) = lirrinioop(z3zn,) 2 E . This proves that z E Y . Consequently, Y is a closed set, as it contains its limit elements. Hence, BE(x) is open. 0
Theorem 2.13 (Pretopology in generalized metric spaces) ( X ,p ) be a space with a dissim,ilarity measuie p.
Let
1. Hollow space is pretopological. 2. Prenietric space is pretopological. 3. Quasimetric space with u, continuous dissimilarity p i s topological. 4. Semimetric space i s topological. 5. Hollow space with p satisfying the triangle inequality is topological.
Proof. To show that hollow, prernetric and quasimetric spaces are pretopological, one needs to prove that the &-ballsdefine a neighborhood basis. Such a proof directly follows the proof given in the metric case by Theorem 2.6. A continuous dissimilarity measure in a quasinietric space assures that thc &-halls are open sets by Theorem 2.12, hence the axioms of the topological space are fulfilled. The proof that a semimetric space is topological follows the same reasoning as in the metric case; see the proof of Theorem 2.6. The proof that a hollow space satisfying the triangle inequality is topological is given in [Bonsangue et al., 19981. 0
Spaces
55
Since generalized metric spaces are pretopological, continuous functions between such spaces can be defined adequately; see Dcf. 2.25 and also Corollary 2.2. Making use of neighborhood balls, we have:
Definition 2.48 (Continuity of a function) Let ( X , d ) and ( Y . p ) be generalized metric spaces. A function f : X + Y is continuous at 5 E X if vie>^ 36>0 y E Bs(x) f ( y ) E BE(f(x)), where neighborhood balls are defined as Bb(z)= ( 2 : d ( z , z ) < S } arid BE(f(x)) = ( f ( z ) : p ( f ( z ) , f(2:)) < E } , respectively. In the case of metric spaces, the E- and &balls are open sets. The function f is continuous if it is continuous at every Z E X .
*
Corollary 2.5 (On continuous functions) Let ( X . d ) and ( Y . p ) he metric spaces (or generalized metric spaces with continuous d issimilarity measures). The assertions below are equivalent: 1. ,f i s cont.in,uous at x. 2. For every neighborhood M of f ( z ) E Y , f - l ( M ) is a n,eighbor,hood of X E X , k t N ( f ( z ) )f-'(W M ( x ) . 2 , = z, th,eri limnioo f ( x n )= f(:c).
3. If limn-oo
Corollary 2.6 (Continuity of a composed mapping) Let ( X , d x ) > (Y,d y ) and (2, d ~ be) generalized m,etricspaces with continuous dissimilari t y measures a n d let f : X + Y , g : Y + Z and h,; X + Z be mappings. I f f a n d g are coritinuous, then the composed mappin,g h = go f , h,(x) = g ( . f ( z ) ) , i s continuous as well. Sketch of proof. The proof follows disrectly frosrri considering thp equirualence between the continuity arid the converge of a seqmnce based o n Corolla?-y 2.5. Direct product spaces can he used for the construction of a new spacc by combining two (or more) spaces. In the context of (finite) generalizcd metric spaces, if the measures refer. the same set of objects, a new dissimilarity measure can be created, e.g. by their summation.
Definition 2.49 (Product space) Let (X,dx) and ( Y , d y ) he generalized metric spaces. Then a product generalized niet'ric space X x Y with a dissimilarity d can be defined as ( X X Y d, x o d y ) , where 0 is thc sum or max operator. This means that (dxody)((x~,y~), ( 2 2 , ~ ~= ) )dx(:1:1:x2)+ dY (Yl,Y2) or ( d x o d y)((:El, Yl), ( 2 2 , Y2)) = max { d x ( Q .: c 2 ) ,dY (Yl>?In)> for ~ I , Z ~ and E Xy l , y 2 E Y . Extension of the concepts of neighborhoods, convergence arid continuity to a product space is straightforward. For instance, U is a neighborhood of
56
The dissimilarity representation f o r pattern recognition
the pair (x.y) if there exist a neighborhood N of x E X and a neighborhood M of y E Y such that N x M C: U . Also, the convergence of a sequence (xn.yYc)E X x Y is equivalent to the convergence of sequences 2 , E X and Yn E Y .
2.5
Vector spaces
Generalized topological spaces and generalized metric spaces defined on sets were described in the previous sections. The necessity, however: arises to consider sets on which meaningful binary operations are allowed. This leads to groups and fields. When the operations become the addition of elements and scalar multiplication, vector spaces can be defined. When, additionally, a topology or a metric is introduced to a vector space, its algebraic structure is enriched. The reader is referred to [Bialynicki-Birula, 1976; Dunford and Schwarz, 1958; Garrett, 2003; Greub, 1975; Kothe, 1969; Larig, 2004; Willard, 19701 for more details.
Definition 2.50 (Group) A group ( G , o ) is a nonempty set G with a binary operation 0 : GxG + G, satisfying the group axioms: (1) Associative law: V i n , b , c E ~ ( a o b ) o c= a o ( b o c ) . (2) Existence of a unique identity element: 3 i d E ~' d l a Eaoid ~ = idoa = a. ( 3 ) Existence of an inverse element: ' d a E ~3,- E~ n o aa- o a = id. If additionally the commutative law, a o b = boa, holds for all a, b E G , then the, group G is Abeliun.
+, +
Definition 2.51 (Field) A field (r, *) is a nonempty set I? together with the binary operations of addition and multiplication * satisfying the following conditions: (1) (I', +) is an Abeliari group with the 0 additive identity element. ( 2 ) (F\{O},*) is ari Abeliari group with the unit multiplicative identity clement. ( 3 ) Distributive laws: a*(b+c) = (n*b)+(n*c) and (a+b)*c = (n*c)+(b*c) hold for all a , b, c E I?.
Example 2.9 (Fields and groups) (1) Let Z be a set of integers. (Z, +) is a group, but (Z, *) is not. ( 2 ) Let R be a set of real numbers and C be a set of complex numbers. (R, +, *) and (C,+, *) are fields.
57
Spaces
Definition 2.52 (Vector space) A vector space (a linear space) X over the field r is a set of elements, called vectoi-s, with the following algebraic struc,ture:
+
(1) There is a function X x X 4 X , mapping (z,y) t o 2 y , such that ( X ,+) is an Abelian group with the zero additive identity. ( 2 ) There is a function r x X 4X , mapping ( X , z ) t o Xz, such that the following conditions are satisfied for all x, EX and all A, p € r : (a) Associative law: (A p ) 2 = X ( p z ) . (b) Distributive laws: X(z+u) = X x + X y , and (X+p)n: = Az+pz. (c) Existence of multiplicative identity element 1 E r: 1x = z. If the field r is not explicitly mentioned,
r is assumed to be either R or @.
Definition 2.53 (Linear combination, span and independence) Let X be a vector space. The vector x is a linear combination of vectors {x1,x:2:. . . ,xn} from X if there exist { a l , a 2 , .. . ,a,) E r such that z = C,"=,ajxJ. The span of ( 2 1 : ~ 2 ,. .. , x,} is a collection of all their linear E X is linearly independent combinations. A finite set of vectors if C,"=,ajx,? = 0 implies that) all aj = 0. Otherwise, the set is linearly dependent. An infinite set is linearly independent if every finite subset is liriearly independerit. Definition 2.54 (Basis and dimension of a vector space) Let X be a vector space. The set B of vectors b, E X forms a Hamel basis of' X if B is linearly independent and each vector x is in the span of V = { b 3 } for some finite subset V of B. The dimension of X , diniX, is the cardinality of B . Definition 2.55 (Subspace) A subspace V of a vector space X is a subset of X , closed for the operations of vector additions and scalar multiplication. Example 2.10 Examples of vector spaces: 1. Iw and C with usual operations of scalar addition and multiplication. 2 . Rrn and C", with the elements z = (zl, zz,.. . ,x,) and the elementwise addition and multiplication by a scalar, are m-dimensional vector spaces. 3. A set of nxm, matrices with the matrix addition and multiplication by a scalar. 4. The set 3 ( a )of all functions defined on a closed and bounded set 62, with the pointwise addition ( f g)(z) = f(x) g(z) and the scalar multiplication ( c f ) ( z ) = c f ( n : ) .
+
+
58
T h e dissimilarity representation f o r pattern recognition
5. The set Pn of all polynomials of the degree less than n is a vector space a,nd a,n n~-dimensiorialsubspace of F(52). 6. The set C(52) of continuous functions on R arid the set M(R) of classes of functions measurable in the Lebesgue sense" are infinite dimensional vect,or spaces and subspaces of F(i1). 7. = { f E M(12) : (<Jnl.f(x)I"dx); < m} for p 2 1 is an infinite dirriensional vector space and a subspace of F ( 0 ) .
C ,F
Definition 2.56 (Quotient vector space) Let X be a vector space over a field r and let Y be a subspace of X. Consider an equivalence relation o i l X such that :cl x2 if ( 2 1 x2) E Y . X / Y , X mod Y , defined by the relation is a quotient vector space. Let [:c] denote the equivalence class of 3;. The addition on the equivalent classes is defined as [XI] [ 2 2 ] = [xi 221 for all il:2 E X and the multiplication by a scalar is defined as a[.] 1 [ax] for all CY Er and Z E X .
-
N
~
N
+
+
~
If X is an n-dimensional space and Y is an m-dimensional space, then X / Y has the dimension n, rri. ~
Definition 2.57 (Linear map) Let X and Y be vector spaces over the field I'. A linear m,ap (linear operator) from one vector space to another is a function 4 : X 4 Y , also called homomorph,ism,, such that for all x 1 , 5 2 E X and all X t I?, the following conditions are fulfilled:
+
(1) Additivity: 4(x1 x2) = qh(il;X:l) ( 2 ) Honiogencity: q5(Xx) = @ ( m ) .
+ q5(x2).
Note that the above conditions are equivalent t o stating that f preserves linear combinations, i.e. $(CzlXixi) = Czl &4(zi) for all xi E X and all X i E r. If'Y = I',then 4 is called a linear functional. X arid Y can also be defined over different fields.
Remark 2.6 If X and Y are finite dimensional vector spaces with chosen hases, th,en any linear map can be represented by a m,atrix. For instanm, a h e a r transformation Rk + R" is represented by an k x m matrix A such that y = Ax for .?: ER'"and ' ~ E I W ' " ~ . Definition 2.58 (Kernel and image) Let f : X i Y be a linear trmsformation between two vectors spaces. The kernel or a null-space of f is a subspace of X consisting of vectors whose image is 0, i.e. ker(f) = "Two functions are in the same equivalence class if they agree almost everywhere, i.e. if they disagree on a set of a measure zero. From now on, M ( n )refers to such classes of functions measurable in the Lebesgue sense.
59
Spaces
{ x : X : f(x) = O}. The image of f is a subspace of Y consisting of images of vectors from X , i.e. i m ( f ) = { ~ E YI z:E X f ( x ) = y}.
+
Lemma 2.4 If X is finite dimensional, then dim(ker(f)) dim(irn(f)) = dim(X). If additionally Y is finite dimensional and the bases are chosen. then the linear map is represented by the matrix F . dim(im(F)) is the rank of F and dim(ker(F)) is the nullity of F . The notion of a dual vector space is important in niost applications. It is especially useful for inner product and nornied spaces; see also Sec. 2.6.
Definition 2.59 (Dual space) Let X be a vector space over the field I? (Ror C).The dual space, also called algebraic dual, X * = C ( X .I?) of X is a set of linear functions , f : X + I',also called lin,ear functionals. Remark 2.7 The collection X * of linear functionals on X over r is a vector space over r with the pointwise addition (.f g)(x) = f ( x ) g ( z ) and scalar niultiplicatio,n ( a . f ) ( x )= a f ( z ) for all f , g E X * , a ~ and r ZEX. The 0-vector in X* is the linear functional that maps every vector x E X to zero. The additive inverse ( - f ) is defin,ed by (-f)(x) = - f ( x ) . The associative law and distributive laws, De,f. 2.52, can easily be uerified b y struightforward computations.
+
+
As we will later deal with finite samples, our focus is on finitedimensional spaces. If X is finite-dimensional, then both X and X * havc the same dimension. Moreover, X is isomorphic'' to X * . The isomorphism depends on the basis B of X , which defines a dual basis B*of X* arid a bijection B + B*. So, given a basis of X , there exists a unique corresponding dual basis. Definition 2.60 (Dual basis) Let X be an n-dimensional vector space with a basis B = { b l , b 2 , . . . , bn}. A dual basis { f l , f 2 , . . . , f n } of X * with respect to B is a basis for X * with the property that 1, if i fj(b2) =
=j
,
0 , otherwise,
"An informal definition by Hofstadter [Hofstadter, 19791: T h e word 'isomorphism' applies when two complex structures can be mapped onto each other, in such a way that t o each part of one structure there is a corresponding part in the other structure, where 'corresponding' means that the two parts play similar roles in their respective structures. Formally, the isomorphism f is a bijective map (one-to-one and onto) such that both f and its inverse f p l are linear maps.
T h e dissimilarity representation f o r p a t t e r n recognition
60
{fz}r=l
The linear functionals of X* are formally defined as f t : X + r by fL(CT=l ~ . k b k )= 2,. Then fz are nonzero elements of X* and span X*. If X is infinite-dimensional, then the dimension of X* is strictly larger than that of X [Kothe, 19691. A simple illustration of this fact is a space X of infiriitc real sequences (21 zz, . . .) with a finite number of non-zero elements. The dual space X* consists of infinite real sequences (x?,x:, . . .) of a n y elements, hence its dimension niust be larger than this of X. ~
Definition 2.61 (Bilinear map) Let X. Y' and 2 be vector spaces over the field I?. A bilinear m a p (bilinear operator) is a function f :X X Y + 2 such that (1) For any fixed z E X the map y transformation from Y to 2. (2) For any fixed y E Y the map z transformatioil from X to 2.
4
f ( z ,y), f 3 ; ( g ) = f ( z ?y), is a linear
+
f ( z ,y), f,(x)
=
f ( z ,g ) , is a linear
If X = Y and f ( x , y) = f ( y , z) for all z, y E X , then f is s y m m e t r i c . If r = C and f(z, y) = f + ( y ,z) for all x,y E X: where t denotes complex c o n j u g a t i o ~ ithcn ~ ~ , f is H e r m i t i a n .
Definition 2.62 (Bilinear form) Let X be a vector space over the field I?. A bilinear f o r m , is a bilinear transformation f : X X X+ I?. Notc that any rcal n x m matrix A can be regarded as a matrix of a bilinear form X X Y + R such that X = R" and Y = R" and f ( x . y ) =
c:=,x;:1
AL1XZY.l.
Definition 2.63 (Non-degenerate bilinear form) Let f : X X X + r be a bilinear form over thc vector space X. f is non-degenerate when the following conditions hold: (1) If f ( ~ l , x z=) 0 for all (2) If f(xl,x2)= 0 for all
x1 E X , then .c2
E X . then
xz
= 0.
z 1 = 0.
Thc spaces X and X* are dual with respcct to a bilinear function X * x X + I?, called a scalar product or i n n e r product, and denoted as (., .), such that ( f ,x) = f ( x ) for :I:E X and f E X*. For instance, if X = R" = X* (R" is self-dual), then ( x * ,x ) = C711x:xi for x* E X " and x E X . This scalar product is linear in both arguments and its properties are analogous ''If
zE@ such that
t=
a+bz, then
z t = a-bi, i2 = -1.
Recall that IzI = z z t = a2+b2.
Spaces
61
to the properties of a well-known scalar product studied in analytical geonietry (that is, given two vectors x and y, their scalar product, is computed by multiplying their lengths and the cosinus of the angle betwecn thcrn). Note, however, a subtle difference. In general, the arguments of ( f ,:c) belong to diflerent spaces and they cannot be exchanged, which rrieaiis that this inner product is not symmetric. One, however, uscs the same notioii of inner product to strengthen the analogy to the traditional geoiiietric inner product. Formal definitions on inner product will follow in Sec. 2.6.
Definition 2.64 (Evaluation functional) Let X * = C(X, r)be a space of linear functionals. An evaluation ,functional 6, evaluates each function f € X * at a point Z E X as 6 , [ f ] = , f ( x ) . One can, therefore, write that 6,[f = ( . f , z ) for f e X * arid Z E X . Any isomorphism q5 :X + X* defines a unique non-degenerate bilinea function on a finite-dimensional vector space X by ( : x , g ) = 4 ( x ) ( g ) for x,!/EX such that for the fixed II:, q5(z): X 4I?. R.emind that X* consists of linear functionals X 4 r and Y * consists of linear functionals Y + r. Note that a bilinear map f : X X Y + r can be characterized by the left linear map f~ E C ( X , Y*). i.e. f~ : x f r arid the right linear map f~ E C(Y,X*),i.e. f ~g :+ f, such that f~(x)(y) = fz(y) = f ( z , y ) and f ~ ( y ) ( x = ) f,(x) = f ( z , y ) for all Z E X and ~ E Y . ---f
Theorem 2.14 (Dual map) 0 Consider a homomorphism l i, : X + I/ over th,e ,field r. T ~there L exists an associated (unique) dual m a p $* : Y *+ X* bekween the dual spaces Y * and X * such thmt $ * ( g ) ( : c ) = g(,dJ(x))for all g E Y * and X E X . 0 T h e dual m a p i s h e a r , hence ($I 4)* = $* 4* and (a$)*= ~ 4f o*r the h e a r maps $ and 4, and a ~ rAdditio,nally, . (4)o 4)* = 4* o $*.
+
+
Definition 2.65 (Second dual space) Let X be a vector space over the field I?. The second dual space X** of X is the dual of its dual spacc. This means that the elements of X** arc linear fimctionals f : X * 4I?. There exists a vector space homomorphism : X + X** defined by q(x)(f) = f ( x ) for all II: E X and f E X*. If X is finite-dimensional: then 17 is an isomorphism, called canonical isomorphism, arid dimX = diniX* = diniX**.
Definition 2.66 (Quadratic form) Let X be a vector space. A mapping q : X 4 R is a qu,adratic form if for all ~ 1 ~ ExX2 arid N E R,the following conditions hold:
62
The dissimilarity representataon for p a t t e r n recognition
(1) f ( z 1 , ~=) y ( z l (2) q(nz)= fY2y(z).
+2 2 )
-
y(x1)
-
y(z2) is bilinear in z1 and
22.
Note that f is a symmetric bilinear form.
Definition 2.67 (Continuous dual space) Continuous dual C,(X, r') of a topological vector space X is a subspace of the dual space X* = C(X,I?) consisting of all continuous linear functiona1sl3. Definition 2.68 (Topological vector space) A vector space X over the field r is a topological vector space if there exists a neighborhood system N such that ( X , N )is a topological space and the vector space operations of addition ( z , y ) + z + y of X X X + X and multiplication by a scalar (A, z) 4Xz of FxX + X are continuous in the topology. Note that in a topological vector space, the topology is determined by the neighborhoods of 0. The neighborhood base NB(0) is defined by open sets of 0 such that every neighborhood of 0 contains a base neighborhood from NB(O).All the neighborhoods of other points are unions of translatcd base neighborhoods: U,,o(za Bp, z, E X and Bp ~ N g ( 0 ) .
+
Definition 2.69 (Convex set) Let X be a set in a real vector space. X is co'ri'uex if QZ (1 - a ) y E X for all z, y E X and all a: E [0,1].
+
Definition 2.70 (Locally convex topological vector space) A topological vector space X is locally convex if every point has a local base consistirig of convex sets. X is locally com,pact if every point has a local base consisting of compact neighborhoods. These definitions can also be simplified tjo consider only a local base of 0. 2.6
Normed and inner product spaces
Metric spaces are already richer in stzructurethan topological spaces, still more structure can be introduced; see Fig. 2.1 and Fig. 2.2. Normed and irnier product spaces are special cases of metric vector spaces, where metric is dcfined either by a norm or an inner product. The algebraic and geometric structures of such spaces are richer than those of metric spaces only. Inner product spaces are important, since there exists a welldeveloped mathematical theory which places the pattern description and I3For any finite-dimensional normed vector space (to be defined in Sec. 2.6) or any t,opological vector spacc, such as a Euclidean space, the continuous dual and the algebraic dual coincide. L , - ( X ) is then a normed vector space, where the norm I l f I I of a continuous linear functional f on X is defined as IIfII = sup{lf(z)i:llxll 5 1).
Spaces
63
learning in their context. Details can be found in [Dunford and Schwarz, 1958; Garrett, 2003; Greub, 1975; Kreyszig, 1978; Kothe, 1969; Pryce, 1973; Sadovnichij, 19911. Definition 2.71 (Normed space) Let X be a vector space over the field r. A norm on X is a function I I . I I : X + eS: satisfying for all 2 , y E X and all a E F the following conditions: (1) Nonnegative definiteness: 11x1I 2 0. (2) Non-degeneration ((x(( = 0 iff J: is a zero vector. ( 3 ) Homogeneity: ( \ a x l (= ( a /( ( X I ( . (4) Triangle inequality: / ( z+ yl( _< (1z/( ( ( y ( ( .
+
A vector space with a norm ( X , I / . 11) is called a normed space. If only the axioms (1); (3) and (4) are satisfied, then ( 1 . ( 1 becomes seminorm and ( X , 1 I . 1 I) a seminormed space. Example 2.11 Examples of seminornied spaces: 1. (F([-l,1 1 , ) )(1). with ( I f 1 1 = ! f ( O ) ( is a seminormed space. p l . 2. (R", ~ ~ ~withp ~ ~ 2p 1,)where , llxilp = (C:Ll lxil ). is anormed space. 3. (Kim3 /I . llm), where l\xllm= maxi=1,..., lxil, is a normed space. 4. Let C(Q) be a set of continuous functions on a closed and bounded set, R c R". (C(Q), 11 . l i p ) , where l i f l i P = (J: l f ( x ) l p d z ) iand p 2 1, is a normed space.
Theorem 2.15 (Seminormed spaces are topological) A seminormed space is a topological vector space, where the open ball neighborhoods are defined asB,(z) = { ? / E X : ! ( x - y y J l < ~ }E,> O . Remark 2.8 I n a seminormed space with the topology induced b p a semin,orm, all neighborhood systems can be constructed by the translation of the neighborhood system for 0 , i.e. N ( x ) = N(0)+ x. This is also tr.ue wh,en the topology Is defined by a translution invuariant (serni)metric. Remark 2.9 A n,ormed space is a locally convex topological vector space, since a norm is a convex function, Example A . l . Therefore, the open ball neighborhoods BE(x) are convex sets. Lemma 2.5 (On seminormed spaces)
1. The (serna)norm is a continuous function, i.e.
The dissamalarity representation for pattern recognition
64
Figure 2.7
Example of a metric open ball in
R2,B l ( 0 )=
:
Jm+Jm
3. Not every metric space is a n,ormed space.
4.
Sketch of proof. Let X = R and d(x,y ) = I ( x # y). Suppose that d ( z , y) = I/x - yll i s true f o r some norm 1 1 . 11. T h e n f o r all a EIW and Z E R , llazll = IallIzII should hold. L e t z = 2-y, then llzll = 1. Consider cy = 2 . T h e n we have 2 = IcvII/zII= I/czI/ = 1, hence a contradiction. Conseguen,tly, thxre is n,o n,orm th,at generates this metric. If a metric distance d in a uector space X is translation invariant, i.e. d(n: z , y z ) = d ( z , y ) and d ( a x , a y ) = \ a \ d ( z , g )holds for all x , y , z c X and (WE&%,then ~~x~~ = d ( z ; O ) defines a norm.
+
+
Remark 2.10 W h e n metric distances are discussed in, vector spaces, th,eg are I L S I L ~ ~ ~ defined IJ by a 'rLorm. This m a y lead t o a false in,tuition that a n open ball BE(x) = {y E X d(x,y ) < E } defined by a metric d i s a convex set. Only if d(x,0 ) defines a norm, then, the metric space ( X ,d ) i s locally coniiex. Otherwise, it i s not true. An example is a vector space R2 with the metric d(x:y ) = An, open ball B l ( 0 ) is sho3wn in Fig. 2.6. It can ea,sily be checked that d is a metric (the triangle inequality holds thanks to the inequality Jusb L & & f o r all a , b 0 ) . Let E > 0. To see that any ball i s not convex, by Def. 2.69, it is suficient t o show that there exist y , z E B E ( x ) such that ( a y (1 - a).) $2 B E ( x ) for some a t [0,1]. Let x = ( ~ 1 . 2 2 )E R2.Define y = ( 5 1 & E ~ , x and ~ ) z = ( 2 1 , ~ &E~). T h e n d(x,y) = $ E < E and d ( x , z ) = $ E < E . Hence, y , z E B,(x). Bwt d(x;7jy 1 TZ) 1 = T3 E4 > & . So, ( a y (1 - Q)Z)$BE(X)f o r Q = +. Any metric space (R",dg),where dg(x,y) = C:, Izi - y#, p < 1, is not localky conuex. dg i s metric by Corollary 3.2.
Jm+
Jm.
+
>
+
+
+
+
+
Spaces
65
Definition 2.72 (Bounded operator) Let (X, 11 . 1 1 ~ ) and (Y.I ( . / l y ) be normed vector spaces. A linear operator A : X 4Y is bounded if there x all Z E X . A linear fiinctiorial exists aER+ such that llAzlly 5 a / ( x ( J for f : X + r is hounded if there exists a € R + such that If(s)l 5 (1 ) ~ I c for ))x all Z E X .
Iy)
Corollary 2.7 Let (X, I 1 . I 1 5 ) and (Y,1 1 . 1 be normed vector spaces. A linear m a p T : X + Y is bounded ifl it i s continuous. Definition 2.73 (Operator norm) Let (X, 11 . 115) arid (Y,/ I . I ?,) be normed vector spaces and A : X + Y a continuous linear map. Then a uniform norm of A is llAlj = sup^^^.^,r - llT~11~. Definition 2.74 (Continuous dual space of a normed space) Let (X, ( 1 11) be a normed vector space over the field I?. The continuoils dual space X' = C , ( X , r) consists of all continuous linear fiinctionals .f : X + r. X' is it,self a normed vector space with the uniform norm defined as l l f l l = suPII,((
r).
Definition 2.75 (Banach space) A normed s p x e for which the associated metric induced by the norm is complete, i.e. cvery Cauchy sequence converges in this space. is called a Ban,ach space. Example 2.12 Examples of Bariach spaces: 1. (R",I ( . 112) is a Banach space. 2. Let t?, p 2 I, be a vector space of real sequences II: =
(XI, 2 2 . .
(czl
. .)
such that Czl (zi(P
Example 2.13 (On Banach and continuous dual spaces) 1. Let (X, / I . 11) be a normed space. The continuous dual X ' = Cc(X,r) I ~ ( I c ) ~ for EX' and z E X is a Banach with the norm I l f l l = space.
T h e dissimilarity representatzon for pattern recognition
66
(Rn,. 1 I ,112) is a Banach space, sirice I I . I l2 induces the Euclidean distance. Its continuous dual space is also (Rm,/ I . 112). Hence the Euclidean space is self-dual. 3. Let ”, p 2 1, be a vector space of real sequences z = ( 2 1 . 2 2 , . . .) such t,hat Erll z i / P < c c with the norm given by lIzllP = (C,“=, Izilp)k. This norm induces the Minkowski metric d,. Consequently, and t:’,= (R”,d p ) are Banach spaces. The continuous dual of p > 1 is a space t? for q > 1 such that 1 + 1 = 1. Hence, ty is self-dual. P q 4. Let tz be a vector space of real sequences z = (zI,z2,. . .) with the norin given by ((z((, = supi lzi!.This norm induces the metric d,. Therefore. tz is a Banach space. Consequently, if finite sequences z = ( 2 1 , ~. ~ . . ., z m )are considered in R’”, with the norin 1 1 .,,I/ the space tz = (R””, d,) is Banach, as well. The continuous dual of lr is e;M. 2.
lr
lr,
Definition 2.76 (Inner product space) Let X be a vector space over @. An znner product (., .) is a bilinear function X X X + C satisfying the following axioms for all z, y, z E X and all a , PEC: (1) (2) (3) (4)
Nonnegative definiteness: (2, z) 2 0. Non-degeneration: (z, z) = 0 iff z is a zero vector. Hermitian symmetry: (z. y) = (y, z)+. Linearity in X and sesquilinearity over C: ( a x + P y , z ) = a t ( z ~ z ) f p(y,z) t and (z,ay+Pz) = a ( z . y ) + P ( z . z ) .
If X is a real vector space, then (., .) : X X X + R is a symmetric bilinear form. A vector space with an inner product ( X , (.)) is an inner product vector space. Lemma 2.6 (On inner products) 1. T h e i n n e r product in a n i n n e r product space i s a continuous function. 2. Eiiery irrner product space is a normed space with t h e norm defined as I l 5 j l = (z,z)k
3. E v e r y iririer product defines the associated m e t r i c d ( z ,y)
=
1 Ix - yl 1.
4 . Parallelogram law. ( ( z + U ( l 2 + j ( z - y [ ( ’= 2 ( ( 2 ( ( ’ + 2 ( ( y j ( ’holdsforthe ~ O T T T (I j
~ (= ( (z>~)i.
5. Polarization identity.
T h e real i n n e r product (., .) can be determined ‘from t h e corresponding n o r m as ( x , y ) = 2(11z+y1121 11~11’-11y11’). T h e complex i n n e r p r o d u c t
Spaces
67
can be determ*i.nedf r o m t h e cor,r-espon,ding norm as (x.y) 1 1 . ~- y/)I2- illz + i y ) I 2- ilia - i y l l ’ ) , where i 2 = -1.
=
( / ( l e t -IJ(
l2
+
Theorem 2.16 (Cauchy-Bunyakovski-Schwars inequality) Let (X, (., .)) be an inner product vector s p c e . The following inequality I ( 2 ,y) I 5 (x,z) 4 (y, y ) i holds for all 2 ,y E X . The equality holds if y = f o r some
~EC.
Definition 2.77 (Hilbert space, pre-Hilbert space) An inner product space for which the induced norm gives a complete metric space is a Hilbert space. A non-complete inner product space is a pr-e-Hdbert space. Example 2.14 (On Hilbert spaces) m
q y , is a Hilbert space. 1. (R7n,(., .)) with ( 2 ,y) = 2. ly is a Hilbert space with an inner product defined as (.~;,y) = C,“=, zi yi. The metric becomes d ( z . y ) = l ~ z ; - y ~=~ (xi- ? y j 2 ) 2 ) + . 3. The space L p defined on a set M(f2) of Lebesgue rneasurablc classes b of functions with (f.g) = ( J a f(x)y(x) p ( d x ) ) + is a Hilbert spacc. Note that L;, defined on a set of continuous functions, since riot complete, is only a pre-Hilbert space. 4. The space l; (and le,”) with p # 2 is riot an inner product space, hence not a Hilbert space.
(cEl
Sketch of proof. The proof i s based on the contradiction of the parallelogram law for z = (1,1,0, 0, . . .) and y = (-1, 1.0, 0, . . .). Then IIzllP= Ilyllp = 2; and 112 +yllp = 112 - yllp = 2 . So, 2 llzll;+ 2 1 1 ~ 1 ; = 4l+5 and /Iz + yilg + /Iz - yyl(p2 = 8, so the equality is satisfied only if p = 2. 0 5. The space L& on R = [u,b]with the norm l i f l l m = max,E[n,b][ f ( r )is l not an inner product space, hence not a Hilbert space.
Sketch of proof. The proof is ba,sed on the contradiction of the parallelogram law for the functions f(x) = a and g ( z ) = 2--a defined on [a,,b ] . Then = a , ~ ~ =g b -~a , ~Ilf+gllm m = b, and I l f - g i l m = a . Then 2 ilfll2,+2((gil& = 2 ~ ’ + 2 ( b - a ) ~and I l f + g l ( & + l ( f - g ( ( & = u 2 + b 2 , 0 so the equality is not satisfied for any a < b.
ilfiloo
Bounded linear functionals are defined in analogy to Def. 2.72. Contimiity of a fLinct,ionalis equivalent to its boundness, as stated in Corollary 2.7.
Definition 2.78 (Orthogonality, orthogonal complement) ( X , (., .)) be an inner product space.
Let
68
T h e dissimilarity representation for pattern recognition
(1) Vectors 1c and y are orthogonal in X , zly if ( x l y ) = 0. Hence, a zero vector is orthogonal to every vector. (2) A subspace V of X is orthogonal if all vectors of V are orthogonal. ( 3 ) Let X be a subspace of X . The set XI = {y E X : k f z E x ( y , ~ = ) 0} is the orthogonal complement of X .
Definition 2.79 (Orthonormal basis) Let 'Ft be a Hilbert space. The set {e,} of elements in 7-l is an orthonormal basis if ( e z , e j ) = bij for all . . z , . ~ and is the Kroiiecker delta, bij = Z(i = j ) , arid every II: € 3-1 can 00 be uniquely written as x = a i e i , which is equivalent t o stating that 00 x = limN-,m aieLfor some a , € @ and . IaiI2.
EL,
be a n orthonorTheorem 2.17 (Orthogonal expansions) Let {ei}zl ma1 basis in a Hilbert space IFI. The following dependencies hold ,for all x,y€R:
C,"=, 1 (x,e i ) j i lIx/12. This inequality holds also .for a pre-Hzlbert space. 2. Purseval formflula: /1x/I2= C,"=, I(x,ei)l. 3. Planch,erel inequality: (2, y) = ( x , e i ) ( e i , y). I . Bessel in,equality:
c,"=,
Definition 2.80 (Adjoint of a continuous linear map) Let ( X , (., .)x)arid (Y,(.; . ) y ) be pre-Hilbert spaces. Let AEC,(X, Y )be a continuous linear map. An adjoint A* € L,(Y*, X * ) , if exists, is a continuous linear map such that (Ax, y ) y = (2, A * y ) x . Theorem 2.18 If ( X , (., .)J is a Hilbert space and (Y,(., . ) u ) is a preHilbert space, then, A E C,(X. Y ) has a unique adjoint A*. Definition 2.81 (Self-adjoint, unitary operator) Let (El(., .)) be a Hilbert, space. A E C,(IFI,IFI) is self-adjoint or Hermitian if A* = A, ( A x ,y) = ( x , Ay) for all 2 , ~ E I F I . A is unitary if AA* = A*A = 1. Theorem 2.19 (Projection theorem) Let V be a closed subspace o,f x E 'Ft, there exist unique x?,E V and x l E V' such t h d z = 2 , : + 21, Define x, = P x, where P is the orihogonal projection, of x on,to V . P has the followin,g properties:
IFI. T h e n f o r every
1. P 2 = P (idempotent). 2. ( P x ,y ) = (x.P y ) (self-adjoint).
3. ( P z ,( I - P ) x) = 0. :I; = Px + ( I - P ) x and PI( I - P).
4.
Only the ,first two conditions are required for P to be a projection.
Spaces
2.6.1
89
Reproducing kernel Halbert spaces
Reproducing kernels are used in a variety of applications like function estimation, function approximation or model building. They uniqiicly define so-called reproducing kernel Hilbert spaces (RKHS). which are spaces of bounded linear functionals, see Def. 2.72 and Def. 2.74. Reproducing kernels are used in statistical learning theory [Vapnik, 19981 for the coristruction of support vector machines; see also Chapter 4. Here. we will provide basic definitions and facts. More details can be found in [Berg et al., 1984; Dunford and Schwarz, 1958; Schaback, 1999, 2000; Schaback and Wendland, 2001; Wahha, 19991.
Definition 2.82 (Positive definite function or kernel) [Berg ef al., 1984; Wahba, 19991 Let X be a set. A Hermitian function K : X x X + C is positive definite (pd) iff for all n c N , { x z , C X and {c2.}= :, C C,one has Cc,=,c,cb K ( J , ,x J ) > 0, where t denotes complex conjugation. Such a function is called a k n e l l 4 . Additionally, K is conditionally positive definite (cpd) iff thc above condition is satisfied only for { c ~ } , "such , ~ that n c3 = 0. Depending on the sign of c,ci K ( z , , z,), also (conditionally) negative, nonnegative and nonpositive functions can be defined.
)rxI
c:,=,
Note that if X is an n-element finite set, such as X = ( p 1 ,pz,. . . ,p n ) , then K is pd iff the nxn, matrix K ( X , X ) is pd. Moreover, if K is pd, then K ( p , , p , ) 2 0 for all p , E X .
Theorem 2.20 (Riesz representation theorem) [Rudin, 1986; Debnath and Makusinski, 19901 Let X be a pre-Hilbert space over the field I?. For every continuous linear functional $(z) : X + r (for a fixed z), there exists a uniqwe y in the com,plet%onX- of X s*uch that $(z)(g) = ( 2 ,y) f o r all z E X . be a Hilbert space over the field I?. For every continuou,s linear Let fun,ctional + ( x ) : IFI 4 r (for a fixed x), there exists a unique y E 'R such that $(z)(y) = (z.g) for all X E X . Definition 2.83 (Reproducing kernel Hilbert space) Let X be a set and CX denote a space of functions f : X 4 C. Let 'RK c C X be a Hilbert space of bounded (hence continuous) linear functionals. A 14Kernel K originates from the study of integral operators, where (LKf ) ( z ) = K ( z , y ) f ( y ) d g . K is called a kernel of the operator LK.
70
The dissimilarity representation for p a t t e r n recognition
Hermit,ian function K : X X X i
C is a reproducing kernel for ‘HK if
(1) K ( ~ , . ) E N for K all Z E X and ( 2 ) K ( x , . ) is the representer of evaluation at z in ‘ H K , that is f ( z ) = (f,K(z,.))xK for all f E N K and all (fixed) Z G X .
NK equipped with K is called the reproducing kernel Hilbert space (RKHS). Example 2.15 1. Every finite-dimensional Hilbert space is a RKHS for some kernel K . 2. The space Lp defined on a set M ( R ) of Lebesgiie measurable classes of functions with ( f ,9 ) = f ( z ) g(z) d z ) is a Hilbert space, but not a RKHS. The reason is that the elements of L F are defined over equivalence classes of functions they agree almost everywhere and not the individual functions, herice the evaluation is not defined. Although the Dirac delta functionl5 6(x) is the representer of evaluation in as f ( z ) = S f ( t ) S ( z - t)& but 6 @ L p . The reason is that S should be in the equivalence class of functions h(z),which take zero for all :c # 0 arid some non-zero value for x = 0. However, S f ( z ) d x = 0, but f S(x)dx = 1, hence contradiction.
Ly
The reproducing kernel map is realized by a linear map li/: z 4 K ( z ,.) such that $(y) = K ( z ,y). Since K ( y , .) is the representer of evaluation a t y, then $(y) = (+, K ( y ,. ) ) x K = ( K ( z ,.), K(y, . ) ) x KAs . a result, one gets K ( z ,y ) = ( K ( z ,.): K ( y , . ) ) x K This . means that a pd kernel K can be seen as a Gram opcrator in ‘ H K , i.e. there exists a function li/ in a Hilbert space 3 - t ~such that the evaluation of the kernel at x and y is equivalent to taking the inner product between $(z) and +(y). If X is a set of a finite cardinality, say n, then the functions are cvaluated only at a finite number of points. Consequently, the RKHS becomes an n,-dimensional space, where the linear functions become n-dimensional vectors. As a result, the reproducing kernel K simplifies to an n x n Hermitian (or symmetric) pd matrix. Corollary 2.8 Let XK = .C,(X, I?) be a Hilbert space of bounded functionals defined over the donialn X . If the evaluation functional S,, 6,[ f ] = f ( x ) is defined and continuous for every x E X and f E ‘ H K , then NK is a RKHS. Hence, there exists K ( z ,.) ~ 7 - such l ~ that S,[f] = f ( z ) = ( K ( z ,.), f ( . ) ) x K . ‘“The Dirac delta function 6 is defined as 6(z) = 0 for z # 0 and J’ 6 ( z ) d z = 1. For any continuous function f one has the following property .I’6(z - t ) f ( t ) d t = f ( z ) .
Spaces
71
Theorem 2.21 (Mercer theorem) Let ' H K be a Hilbert space of functions f : X 4 C and let K : X X X + G be a Hermitian kerne1l6. If ( K ( x ,.), K ( x ,.))xK 5 03, then K can be expanded b y a countable sequence of orthonormal eigenfunctions $i and real positive eigenvalues X i such that the bilinear series K ( z ,y) = &$i(x)& (y)t converges uniformly and ab~olutely'~.
c,e"_,
The theorem above means that the eigenfunctions and eigenvalues are found as a solution to the eigen-equation ( K ( z .),+i(.))x, , = Xi&(z) or, in the integral form, JX K ( x ,y)$%(y)dy = Xi&(z), if K corresponds to an inner product defined by the integral. In practice this requires that X is a compact subset of R" or an index set. As the eigenfunctions {&}Elare linearly independent functions (an orthonormal basis of XK);then any function f in the space 'HK can be written as f ( z ) = C,"=, a i & ( x ) . The inner product between f and g in the Hilbert space 'HK is defined as (f(x),g(x)).~~~ = * ( a , i h j ) , where g(z) = C,"=1bZ$2(z). Such a space of functions with the kernel K is indeed a RKHS, since ( f ,K ( x , = ( . f ( y ) ,K ( z ,Y ) ) R ~= ( f k )K(Y, , z ) ' ) x K= C,"=, ((&&(x))t)t = C,"=, ~ g ) ~ (=z )f ( z ) , because K is Hermitian,
cEl
2
.)hK
i.e. K(z, y) = K(y,z)?. Note that = ( f ( x )f,( z ) ) & , = C"z = 1 (u?(2 A, and IIKil&, = ( K ( z ,.), K ( z ,.))&, = Xi. There is an equivalence between choosing a specific ' H K , reproducing kernel K and defining the set of X i and $ i .
, : c
Theorem 2.22 (Moore-Aronszajn theorem) [Wahba, 19991 For every p d kernel K on X x X (X is a compact set), there exists a unique RKHS ? f ~over X for which K is the reproducing kernel and vice versa.
2.7
Indefinite inner product spaces
Indefinite inner product is a generalization of a (positive definite) inner product (., .), Def. 2.76, by requiring that only the (Hermitian) symmetry and (sesqui)linearity conditions hold. The facts presented here are based on the books [Alpay et al., 1997; BognBr, 1974; Iohvidov et al., 19821 and 161n the integral form the positive-definiteness means that (Kf, f ) ~ , = Jxxx wx>Y)f(x)f(Y)+dxdY2 0. I7Let {u,} be a set of functions X + C. A series C & u,(z), converges uniformly to u ( z ) iff for every .E > 0, there exists a natural number N , such that for all z t X arid all n 2 N , Iun(z)- u(x)l < E . For a fixed z,a series C ,u z ( z )converges absolutely if the series C 1 u,(z) I converges.
72
T h e dissimilarity representation for p a t t e r n recognition
the following articles [Constantinescu and Gheondea, 2001; Dritschel and Rovnyak, 1996; Rovnyak, 1999; Goldfarb, 1984, 19851.
Definition 2.84 (Indefinite inner product space) Let V be a vector space over C. An indefinite i n n e r product (., .)v is a map V X V ---f CC such that for all II:, y , z EV and a , P E C , one has: (1) Hermitian symmetry: (z,y ) = ~ (y, x)t. (2) Linearity in X and sesquilinearity over C:( a2
P+(Y,.)V
and
( 5 ,a
y
+PZ)V =
(II:, Y)v
+ /3g, z
) =~at
(x,Z ) V
+ P (z, " ) V .
If V is a real vector space, then (., .)v : V x V form; see Def. 2.61.
4
R
+
a symmetric bilinear
Since (x,I C ) V can have any sign, there is a distinction among positive, negative and neutral vectors and the corresponding subspaces. For the material presented below, V is assumed to be an indefinite inner product space equipped with the inner product (., .)v. We will write (., .) only if the traditional positive definite inner product Def. 2.76 is meant.
Definition 2.85 (Positive, negative and neutral vectors) A vector 5 E V is positive if (IC, z)V > 0 , negative if ( 2 ,z ) <~0 or neutral if (IC, x)v = 0. A subspace X c V is called positive, negative or neutral if all its elements are so, respectively. Every indefinite inner product space contains at least one non-zero neutral vector [Bognk, 19741
Definition 2.86 (Orthogonality, orthogonal complement) ( V , (., .)) be an indefinite inner product space.
Let
(1) Vectors II: and y are orthogonal in V if (z, y )= ~ 0. (2) A subspace X of V is orthogonal if all vectors of X are orthogonal. (3) Lct X be a subspace of V . The set ' X = { y E V : VZEx( y , z ) v = O} is the orthogonal complement of X .
Definition 2.87 (Isotropic subspace, degenerate subspace) ( V ,(.; .)) be an indefinite inner product space.
Let
(1) A vector %r E V is isotropic if it is a non-zero vector orthogonal to every vector in V . ( 2 ) Let, 0 be the zero vector. Let X C V . The isotropic subspace Xo of X consists of isotropic vectors, i.e. Xo = X n XI. If Xo # 0, then X is degenerate and (., .)v is degenerate on X. The entire space V is degenerat,e if V' # 0.
Spaces
73
Example 2.16 Inner product spaces: Let V be a vector space of pairs of real numbers. Let (x,y ) = ~ 5 1 y1 2 2 y 2 for 2 = (zl,z2) and y = ( y l , y z ) . Then ( V ,(.,.),,,) is indefinite. Note also that if X = { (21,za) E V : z1 + x2 = 0}, then XI = X. Hence X is a degenerate subspace of V . Let V be a vector space of number sequences (ul, v 2 , .. .) satisfying 1 ~ ilvi12 l < 00. Then (x,y ) =~ E~ xi y j defines an inner product. Depending on the signs of ~ i ( x, , y ) V may be positive, negative or indefinite. If ~i are of different signs, then (z, y ) is~ indefinite. Moreover, if there exists at least one zero E ~ then , (x,y ) is~ degenerate. Let L( [a,b ] ) be a vector space of real valued functions that are measurable and square-summable with respect to some function p . Depending b on the function p , ( f , g ) = f ( x ) g ( z ) d p ( x ) defines an indefinite or definite inner product; see also [Halnios, 19743. ~
Cpl
Cpl
s,
Definition 2.88 (Fundamental decomposition) Let (V, (., .)v)be an indefinite inner product space. If V is represented as a direct orthogonal decomposition'* V = V+ @ V- @ Vo such that V+, V - and Vo are positive, negative and neutral subspaces, respectively, then such a decomposition is called a fundamental decomposition and V is decomposable. Not every space V admits a fundamental decomposition, but every finitedimensional inner product space does [Bognk, 19741. Spaces which yield a fundamental decomposition are called Krein spaces and are of our interest. Pseudo-Euclidean spaces are the simplest examples of these. See also Fig. 2.2.
Definition 2.89 (Pseudo-Euclidean space) A pseudo-Euclidean space & = IW(P>q)is a real vector space equipped with a non-degenerate, indefinite inner product (., . ) E [Greub, 19751. & admits a direct orthogonal decornposition & = E+ @ E - , where &+ = IWP and E- = IW4 and the inner product is positive definite on €+ and negative definite on E-. The space & is, therefore, characterized by the signature ( p ,q ) [Goldfarb, 19841. '*A direct sum V = X @ Y @ Z means that every v t V can be uniquely decomposed into z E X , y E Y and t E 2 such that v = z y z and X n Y = {0}, Y n 2 = ( 0 ) and X n 2 = (0). An orthogonal sum of X , Y and 2 is their direct sum such that they are pairwisc orthogonal. Here, a direct orthogonal decomposition V = V+ V - @ Vo means that V - = V$ and Vo = (V+ n V:)', i.e. Vo = V+ n V $ consists of neutral vectors orthogonal t o all other vectors in V .
+ +
The dissimilarity representation for pattern recognition
74
Definition 2.90 (Orthonormal basis) Let € = Iw(P,Q)be a pseudoEuclidean space. An orthonormal basis {el,e2, . . . , eP+q}in € is defined as
i
1, f o r i = j = 1 , 2 , . . . ,p , ( e i , e , ) = -1, f o r i = j = p + l , . . . l p + q , 0; for i # .I.
The inner product between two vectors x and y in R(P.Q)can be expressed by the standard inner product (., .) in a Euclidean space.
Lemma 2.7 (Pseudo-Euclidean inner product via the standard inner product) L e t € = R(Piq) be a pseudo-Euclidean space. T h e n (., .)& can be expressed by t h e traditional (., .) in n Euclidean space RP+q as P t9
where
and I,,,
and I,,, are t h e adeiitzty matrrces.
If x+ and x- stand for the orthogonal projections of x onto RP arid Rq, respectively, then (xl y ) = ~ ( x t ly + ) - (x-,y-).The indefinite Liior~n' of a non-zero vector x becomes llxll; = ( X , X ) E = xTJpqx,which can have any sign. Based on the inner product, the pseudo-Euclidean distance is defined analogous to tlie Euclidean case. Definition 2.91 (Pseudo-Euclidean square distance) Let € be a pseudo-Euclidean space. Then
d2X.Y)
=
llx - YII; = (x - Y1 x - Y) & = (x - Y)
T
Jpq
(x - Y ) ,
= Iw(P,Q)
(2.2)
is a yseiido-Euclidean square distance. It can be positive. negative or zero.
am,
The distance d is either real or in the form of where i2 = -1. Note that the square distance between distinct vectors x and y may equal zero. Note that an orthonormal basis of the pseudo-Euclidean space is chosen as it is convenient for the representation. The rcason is that Jpq has a siniplr form arid it is both symmetric and orthogonal in tlie Euclidean space RP+q and in the pseudo-Euclidean space PQ(p,q) (this will be explained
Spaces
75
Figure 2.8 Left: a pseudo-Euclidean space & = R ( l > l ) = R1 x iR1 with d2(x,y) = (~-y)~J11(x-y).Orthogonal vectors are mirrored versus the lines 5 2 = 2 1 or xz = -1, for instance ( O A , O C ) E= 0. Vector v defines the plane 0 = ( v , x )=~ vTJ1lx. Note that the vector w = J ~ I va,‘flipped’ version of v, describes the plane as if in a Euclidean space 8’. Therefore, in any pseudo-Euclidean space, the inner product can be interpreted as a Euclidean operation, where one vector is ‘flipped’ by Jpg. The square distances can have any sign, e.g. d 2 ( A , C )= 0, d 2 ( A , B )= 1, d 2 ( B , C )= -1, d 2 ( D , A ) = -8, d 2 ( F ,E ) = -24 and d 2 ( E ,D ) = 32. Right: A pseudo-sphere 11x11; = x: - x; = 0. From the Euclidean point of view, this is an open set between two conjugated hyperbolas. Consequently, the rotation of a point is carried out along them.
later on). One may, however, consider another basis. Let V = IWTL be a vector space and let { v ~ } ? =be~ any basis. Consider two vectors of V ; x = C:k,zivi and y = Cr=lyivi, as expressed with respect the basis vectors. Let 4 : V X V + IR be a symmetric bilinear form in V. Then 4(x,y ) = C:=‘=, Cy=lziyi 4(vi,vj) = xTMy, where A,f = M ( 4 ) such that Mij = 4(vi, vj) for all i ,j = I, . . . , n is a matrix of the form 4 with respect to the basis { V ~ } F = ~ .Assume that 4 is non-degenerate, which means that the rank of M is n. If M is positivc (negative) definite, i.e. if 4(x,x) > 0 ( d ( x , x )< 0) for all x E V, then qh (4) defines a traditional inner product in V. If A f is indefinite, i.e. $ ( x , x ) is either positive or negative for x E V, then 4 defines an indefinite inner product in V. We will denote it as (x,y)bf = xTMy. If M is chosen to be J P q , then { V ~ } Y = ~ is an orthonormal basis in R(”q), p g = n. This means that any symmetric non-degeneratc bilinear form 4 defines a specific pseudo-Euclidean space. Any other such form $ will define either the same or different pseudo-Euclidean space, depending on the signature, i.e. the number of positive and negative eigenvalues of M ( $ ) . If the signatures of M(qh) and AT($) are identical, then the same pseudo-Euclidean space is obtained.
+
76
T h e dissimilarity representation f o r p a t t e r n recognition
Note that if the basis of R" is changed, then the matrix of the bilinear form changes as well. If T is a transformation matrix of the basis { V ~ } F = ~ to the basis { W ~ } F = ~ ,then M"(4) = TTM"(q5)T is the matrix of q5 with respect to the new basis. This directly follows by substituting x by (Tx) and Y by (TY) in ( X , Y ) M . By introducing algebraical structures to a vector space V = R",specific vector spaces are obtained, depending on a form of a bilinear map or of a metric. One may introduce both an inner product (., .) and an indefinite inner product (., .)E to the same vector space. Such inner products are naturally associated with the (indefinite) norm and the (indefinite) distance. Additional metrics or norms can also be introduced. In this way, a vector space may be explored more fully by equipping it with various structures. A pseudo-Euclidean space R(P>q)can also be represented as a Cartesian product IWP x i Rq. It is, thereby, a ( p q)-dimensional real subspace of the ( p y)-dimensional complex space CP+q, obtained by taking the real parts of the first p coordinates and the imaginary parts of the remaining q coordinates. This justifies Eqs. (2.1) and (2.2), and allows one to express the square distance as d2(x,y ) = d i p (x,y) - d& (x,y ) , where the distances on the right side are square Euclidean. A Euclidean space is a special case of the pseudo-Euclidean space as RP = IR(P3').
+
+
Definition 2.92 (Isometry between pseudo-Euclidean spaces) Let ( X ,(., and (Y,(., . ) E z ) be pseudo-Euclidean spaces. A mapping q5 : X + Y is an isometry if (4(z), 4(y))e, = (x,y ) ~ ~ . The notions of symmetric and orthogonal matrices should be now properly redefined. Sirice the matrix JPq plays a key role in the definitions below, we will denote them as 3-symmetric and 3-orthogonal matrices to make a distinction between matrices in indefinite and traditional inner product spaces.
Definition 2.93 (3-symmetric, 3-orthogonal matrices) an n x n matrix in IW(P>'J), n = p y. Then
+
1. A is J-sym,metric or 3-self-udjoint if 2. A is J-orthogon,al if J&AT&, A = I .
JPq
Let A be
ATJpq= A.
A 3-symmetric or 3-orthogonal matrix in a pseudo-Euclidean sense is neither symmetric nor orthogonal in the Euclidean sense. If, however, pQ(P>q) coincides with a Euclidean space, i.e. q = 0, then the above definitions simplify to the traditional ones, as Jpq becomes the identity operator I . For instance, by straightforward operations one can check that the matrix
Spaces
[
I : ]
is 3-symmetric in
IR(l.l)
with
77
3
=
[
and that
5 [ 2 1;l i s
3-orthogonal in IR(lil). If we denote A* = JP4 ATJp4,then the conditions above can be reformulated as A* = A for a J-symmetric matrix A and as A*A = I for a J-orthogonal matrix A . This already suggests that A* plays a special role of the adjoint operator, which will be discussed below. An extension of a pseudo-Euclidean space leads to a K r e h space, which is a generalization of a Hilbert space as a pseudo-Euclidean space is a generalization of a Euclidean space.
Definition 2.94 (Kreh and Pontryagin spaces) a vector space K over C such that
A K r e k space is
(1) There exists a Hermitian form, an indefinite inner product (.; . ) X on K , such that the following holds for all z, y; Z E K and Q, [ ~ E C :
(a) Hermitian symmetry: (z, y ) =~ (y, x ) i , (b) Linearity over IC and sesquilinearity over a!i (z, .)X
+ P'
@I:
(ax+ /j y 3z ) =~
(y, Z ) X .
(2) Ic admits a direct orthogonal decomposition K: = K+ @ K - such that ( K + , (., .)) and ( L-(.,.)) , are Hilbert spaces'' and ( Z + , L ) ~ = 0 for any z+ E IC+ and 2- E Ic-. The space IC- is also called an antispace with respect to (., .). If K is a vector space over also Def. 2.61.
R,then (., -)K is a
symmetric bilinear form; see
It follows that K admits a fundamental decomposition with a positive subspace Ic+ and a negative subspace K - . Therefore, Ic+ = ( K - ) l . Let dimIc+ = K,+ and dimIC- = 6- be the ranks of positivity and negativity, respectively. Krein spaces with a finite rank of negativity are called Pontrgagin spaces (in other sources, e.g. [Bognar, 19741, the rank of positivity is assumed to be finite). A Pontryagin space with a finite K - is denoted by II,. Note that if (.; .)Ic is positive definite or zero for zero vectors only. then K is a Hilbert space. Example 2.17 (Pseudo-Euclidean, Kre'in and Pontryagin spaces) Let V be a vector space of real sequences ( u l , v 2 , . . .) satisfying C,"=,I ~ i jv,l j 2 < 00. Then ( 2 ,y ) = ~ Czl ~i z i yi defines an inner product. If ~1 = 1 and ~j = -1 for all j> 1, then the inner product is given ans 03 ( xy)v ~ = z1y1 zi yyi and V becomes a Pontryagin space. If ~ 2 >j 0
xi=*
IgAll Hilbert spaces discussed here are assumed to be separable, i.e. they admit countable bases.
The dissimilarity representation for pattern recognitaon
78
and & z J - 1 < O for all j,then V equipped with (z,.y)vdefines a Kre'in space. If V is a vector space of finite sequences (v1, vz, . . . , v,) and all E~ # 0, then V with (2, y)v = ~i zi yi is a pseudo-Euclidean space.
c;:,
Definition 2.95 (Fundamental projections and fundamental symmetry) Let Ic = Ic+ @ Ic-. The orthogonal projections P+ and P- onto Ic+ and Ic- , respectively, are called f u n d a m e n t a l projections. Therefore, any x E K can be represented as x = P+ x + -'J x where I K = P+ P- is the identity operator in K . The linear operator 3 = P+ - P- is called t>he fundam,ental s y m m e t r y .
+
Corollary 2.9 (Indefinite inner product by the traditional one)
(x,w)lC
=
(x,3 , ) .
In Hilbert spaces, the cla.sses of symmetric, self-adjoint, isometric and unitary operators are well known [Dunford and Schwarz, 19581. Linear operators. carrying the same names can also be defined in Krein spaces. The dcfinitions are analogous and many results from Hilbert spaces can be generalized to KreYn spaces. However, due to indefiniteness of the inner product, the classes of special properties with respect to the inner product are larger. We will only present the most important results; see [Bognjr, 1974; Iohvidov e t al., 1982; Pyatkov, 2002; Goldfarb, 1984. 19851 for details. Definition 2.96 (H-scalar product, H-norm) Let z, y t IC. The H scalar product is defined as [x,y] = (2, J ~ ) Kand the H - n o r m is 11x1i~ = [x;:x] 4.
+
Let x E Ic be represented as x = x+ x-, where x+ E Ic+ and x- E Ic-. Since [x,y] = (2, Jy)x, we can write [z, y] = (z+, y + ) K - (z-, y - ) ~= (:x+,yj+) - (-(z-,y-)) = ( z , ~ ) This . means that [x,g]is equivalent to the traditional (Hilbert) inner product and Ic+ and Ic- are orthogonal with rcspect to [z, y]. Moreover, the associated Hilbert space IFI is then such that IFI = IIcI = Ic+ @ IIc-1, where 1Ic-I stands for (K-,(., .)). Formally, there is a close 'bound' between a Krein space arid its associated Hilbert space: Lemma 2.8 A decomposable, non-degenerate i n n e r product space Ic i s u K r e k space ifl f o r every f u n d a m e n t a l s y m m e t r y J ,t h e H-scalar product t u r n s it into a Hilbert space [Bognur, 19741. H-scalar product is a Hilbert inner product, therefore Ic can be regarded as a complete Hilbert space (Banach space) with the H-scalar product (Hnorm). As a result, the (strong) topology of K: is the norm topology of the associated Banach space, i.e. the H-norm topology. This topology is
79
Spaces
simply defined by the nornis in the associated Hilbert space 1KI. does not depend on the choice of fundamental symmetry2'. Therefore. continuity, convergence and other notions can be defined for K with respect to the H-norm.
Definition 2.97 (Convergence, Cauchy sequence) (1) The sequence x, in K: converges to x E K with respect to thc H-norm iff lim7L-m(z7h,y)K= (x,y)~ for all y E K: and limn400(xfl,,.x,)K: = (x,Z ) K . (2) The sequence x ,in K is Cauchy with respect to the H-norm iff (xT1x,,xn - x,)~ 40 and ( z T L , y form ) ~ a Cauchy sequence for y € K . Corollary 2.10 Since (x,y ) =~ [x+,y+] - [ L , g-1, then (z, y ) is~ continUOTLS with respect t o the H-norm in both x aad y. Theorem 2.23 (Schwarz inequality) /1x11~ l l v l l ~holds for all r c , y E K . Proof.
(1 .+1 2
2
The inequality l(z, y ) ~ l5
l [ ~ + ? ~ + 1 F [ ~ - ~ Y2 -5l l(1 .+1 I / ~ + I / + I I ~ ~ - ll lIY -l l )2 + 11~-/12)(llY+l/2 + IlY-112) = I l ~ l l ~ l l Y l l ~ . I(.,V)Kcl
5
Definition 2.98 (Orthonormal basis in a KreYn space) Krein space. If K+ and K - are separable Hilbert spaces, then a countable orthonormal basis {ei}g1 in K such that any z uniquely written as IC = a i e i for some E r and means that
czl
(ei,e,j)lc =
{
I
Let K be a there exists E K can be lat12. This
c,"=,
1, if i = j and P+ei is an orthonormal vcctor in K+, -1, if i = j and P-e, is an orthonormal vector in IC-. 0, otherwise.
Theorem 2.24 (Orthogonal expansions) I f K + and Ic- ure sepurable Hilbert spaces, then there exists a countable orthonmrmal basis in K . For x,y E IC, one has [Boyna'r, 19741:
{e,}zl
(1)
CF1 Ib,+ I 2
< 0.
(21 -Cz(e,.e,)KC=-l I(X, 4 K 1 2 5 (3)
(Z.?/)K: 5
c : ,
(Z.Z)K
5
Ci(e,,e,)K=l I(x7 +I3.
(ez, e i ) K ( x ,e i ) K ( e t ,Y ) K .
201na Krein space, there are infinitely many fundamental decompositions, hence fundamental symmetries and, consequently, infinitely many associated Hilbert spaces. However, the decompositions yield the same ranks of positivity and negativity, the same H-norm topologies; simply, they are isomorphic.
80
T h e dissimilarity representation f o r p a t t e r n recognition
Definition 2.99 (Adjoint operator) Let C,(IC, G ) be a space of continuous linear operators from the Krein space K onto the Krein space G. If G is K:, then C,(K) will be used. Note that C,(K:) is a dual space of K .
1. A* E C,(G,K) is a unique 3 - a d j o i n t of A E C,-(K,G) if ( A x , y ) g = ( x , A * y ) ~for : all z t K and all ~ E G . 2. A E &(K) is 3-self-adjoint ( 3 - s y m m e t r i c ) if A* = A , ( A z , y ) ~= (x,Ay)x for all rc, y E K . Definition 2.100 (Isometric and unitary operators) [Alpay et al., 1997; BognBr, 19741 Let A E L,(K,G) be a continuous linear operator K: + 4. A is 3 - i s o m e t r i c if A*A = I , and 3-coisometric if AA* = I,. A E L c ( K ) is 3 - u n i t a r y if (Arc, Ay), = (rc, y ) for ~ all rc, y E K , or in other words. if it is both 3-isometric and J-coisometric. Remark 2.12 Th,e fundamental s y m m e t r y 3 fulfills 3 Hence, 3 is J-symm,etric and J-unitary.
=
J* =
3-l.
Theorem 2.25 (Factorization) [Bogncir, 19'74l Every 3 - s y m m e t r i c operator A E C,(K:) can be expressed as A = T T * , where T E C,(V, K ) f o r some K r e k space V and ker(T) = 0 . Since a Krein space is inherently connected t o its associated Hilbert space, both the 3-adjoint and 3-unitary operators can be expressed through operators in this Hilbert space. Hence, the condition ( A x ,y)g = (z, A * ~ ) Kis: equivalent to stating that ( A x ,J g ) = (z,JA*y). This is further equivalent to ( J A z ,g) = (z, JA*g), since J is self-adjoint (symmetric) with respect t o (., .) in the associated Hilbert space \ K \ . This means that in IKl, the adjoint of ( J A ) is (JA*). Let A X be a Hilbert adjoint of A. (This means that A X = AT or A X = At, depending whether the Hilbert space is over W or C.)Then (JA)X = A X J = JA* and finally
A* = J A X J . For a 3-unitary operator in a Krein space K , we have (Arc,Ay)~:= (rc, y ) ~which , is equivalent to stating that (Ax,JAy) = (x,Jy) in the associated Hilbert space. Since J is self-adjoint in 1x1, then ( ( J A ) z ,(JA)y) = ( x ?y). So, ( J A ) is a unitary operator in IKI, which means that ( J A ) " = ( J A ) - l . Then Apl = J A X J ,which is equivalent to A-l = A*. Formally, we have: Theorem 2.26 Let A E & ( K , G ) , then A E Cc,(lKl,IGl) f o r the associated If A X is a Halbert adjoint of A, t h e n A* = Hilbert spaces 1x1 and IS/. Jx A XJ G , where JK:an,d JG are the fundamental symmetries. Moreover,
IIA*IIH = / I A X / I H= IlAIIH.
Spaces
81
Definition 2.101 (Krein regular subspace) Let K be a Krein space. A Krein regular subspace of K is a subspace X which is a Krein space in the inner product of K , i.e. (x,y ) x = ( x ,y ) for ~ all 2 , Y E X . Definition 2.102 (Positive, uniformly positive subspaces) A closed or non-closed subspace V E K is positive if ( x ,x ) >~0 for all x E V and V is ,unifownly positzue if it is positive and (x,Z)K >a(/xil&for a positive cv depending on X and the associated H-norm. Similar definitions can be made for negative, uniformly negative, nonnegative etc. subspaces. The term maximal, if added, stands for a subspace which is not properly contained in another subspace with the same property. Every maximal positive (negative) subspace of a Krein space is closed. If K = K+ 6? K - is the fundamental decomposition, then the subspaces K+ and K - are maximal uniformly positive or negative, respectively. Any maximal unifornily positive or negative subspace arises in this way [Bognk, 19741.
Definition 2.103 (Positive definite operator) A J-self-adjoint operator A E C , ( K ) is positive definite ( 3 - p d ) in a Kreiri space if (x,A ~ )>K0 for all x E K . The negative definiteness ( 3 - n d ) or semi-definiteness is defined accordingly. The above condition is equivalent to 0 < (2, AZ)K = (x,JAz). This means that A is J-pd if ( J A ) is pd in the associated Hilbert space 1 K 1. For instance, the fundamental symmetry J is J-pd, since it is J-symmetric and JJ = I.
Theorem 2.27 (Projection theorem) Let V be a closed, non-degenerate subspace of a K r e k space K . Th,en f o r every z E K , there exist unique zvE V and x 1 E V' such that z = x, 21, where 2 , = Px and P is the orthogonal projection of x onto V [Bogncir, 1974; Iohvadov et al., 19821. P h,us th,e following properties:
+
1. 2.
P2
=P.
(Px, y ) ~= : (x,Py)x. (J-self-adjoint)
c?,
( P z ,( I K
4.
z = Pz
~
P)Z ) K
+ (I;c
~
= 0.
P ) z and PI( I x - P ) .
Only the first two conditions are required f o r P t o be u projection. Definition 2.104 (Gram and cross-Gram operators) Let V be a linear subspace of K: spanned by linearly independent vectors { u l , v 2 , . The Gram operator, or the inner product operator, is defined as G,,,, =
82
T h e dassimilarity representataon for pat t er n recognition
( ( ~ ~ , v , ~ ) x ),..., i ,., j = 1 Assume further that a subspace ZA C IC, spanned by ( 7 ~ 1 , I L Z , . . . , u t } . is given. Then G,[,, = ( ( u i , u j ) ~ ) i , l , . _t :_j =_l ,..., is the cross- G r a m operator.
Theorem 2.28 (Projection onto a subspace) Let V be a linear subspace of a Krein space IC spanned by the vectors (w1,112, . . . , un}. Hence, V = [ ~ q + * u 2. ., .,ti,] is the basis of V . If the Gmrn operator G,, = ( ( v i , v j ) )i,,i=l....,n ~ is nonsingular, then the orthogonal projection of x E I C on,to V is unique and given by 5,
= V G:,
g,,
(2.3)
where g , is a n n x 1 vector of the elements (x,w i ) ~ ,i = 1 , 2 , . . . , ri,. If the G r a m operator G,, is singular, then either the projection does not exist or x ,= Vz, w h ~ r ez i s a solution t o the linear system, G,, z = g,. Proof. Let J be the fundamental symmetry of K . Let x, be the projection of x onto V . Based on Theorem 2.27, X can be uniquely decomposed as z = x,, ZL, such that z, E V and x i E VL. Moreover. ( x l ~ u z=) 0.~ Hence, ( : c , ’ u , ) ~= ( x t , > t i i ) xwhich > are the elements of g , . Since the vectors ( 7 i i } are linearly independent (as the span of V ) , then there exists a , siich t,hat x,,= C/4,a p ~= , V a , where a is a colurnn vector. The elements of g, become (x,,wi)x = (Va,’ui)x,‘i = 1,.. . , 72. This gives rise to g , = V t J V a = G,, a. If G,, is nonsingular. then a can be determined uniquely as G :; g , , hence x,, = V Gzf g , . If G,, is singular then either there is no solution to the cquation g , = G,,,, a or there are many solutions. 0
+
Remark 2.13 T h e same formulation as Eq. ( 2 . 3 ) holds for a projection onto a subspace in a Hilbert space, provided that the indefinite i r i n w product (.>.)x %sreplaced b y the usual inner product (., .). In a Hilbert space, the singularity of the Gram operator G,, means that { . u ~ } are ~ = linearly ~ dependent. In the case of a Krein space, this means that V contains an isotropic vector, i.e. there exists a linear combination of {ZI~}:=~which is orthogonal to every vector in V . In other words, to avoid the singularity of the Gram operator, the subspace V niust be nondegenerate.
Remark 2.14 Since ( x , ~ i i ) x= ( x > J u i )= d J w i , then by th,e use of the Hilbert operations only, we can write that g, = V i J x and also G,, = V t J V . As a result, x, = V(VtJ’V)plVtJ’xand the projection operator P onto the subspace V i s expressed as P = V ( V t J V ) - l V t J .
Spaces
83
Corollary 2.11 Let V = span(v1, va,. . . , v,} and Li = s p a n { u l ,u ~. ... , u,} be linear subspaces of K . Assume the Gram operator G,,, and the crossGra.m operator G,,,, = ( ( u i , u,)lc)i=l..t.j=l..,. If G,, is nonsingular, th,en by Theorem 2.28 the orthogonal projections of the elements f r o m K: onto V are given by Qv = G,, G,: V. Theorem 2.29 (Indefinite least-square problem from a Hilbert perspective)'l. Let V be a linear non-degenerate subspace of a Krefn space K spanned by the vectors { u l , v2,.. . , vn}. T h e n for the basis V = [2/1.v2,. . . ,un] of V and f o r u E K , the function, F ( x ) = IIu - Vxli$ reuch,es its m i n i m u m iff G,,, = V t J V is positive de,fin,ite in, a Hilbert sense in the uicinity of xS2'. x, is the sought solution such that 5 , = G;Jg, and g , = VtJu. Otherwise, n o solution exists.
+
2 x t V t J u z t V tJVz . From mathematProof. Jju- Vxllg = utJu ical analysis [Birkholc, 1986; Fichtenholz, 19971, .z', is a stationary point of F ( x ) if the gradient VFlz=zs = 0. By a straightforward differentiation of F , one gets 2VtJVx - 2 V t J u = 0, hence VtJVx, = V t J u . Since V is non-degenerate, then G :; exists. Therefore, by Remark 2.14, the potential solution is given as 2, = (VtJV)plVtJu = Gzfg,. Traditionally, the stationary point x, is a unique minimum iff the nxn>Hessian is positive definite in a Hilbcrt sense. The Hessian H = a2F H = 2 Vt JV = 2 Gu,. Since the matrix of indefinite inner products equals G,, is generally not positive definite, H 2 G,, is also not. Consequently, 2 , cannot be a global minimum. However, H is positive definite at the point 5 , . Observe that zLHz, = u ~ J V ( V ~ J V ) - ~ = V u~tJJ P Uu , where P is the projection matrix onto the space spanned by the column vectors of V ;see Remark 2.14. By Theorem 2.27, P is J-self-adjoint, hence JP is ~
(-)&=lz5
21For comparison, an equivalent formulation is given for the Hilbert case: (Least-square problem in a Hilbert space) Let V = span(v1, vz,. . . , w,,} be a linear subspace of a Hilbert space 'h and let V = [v]vz . . . v,]. Then for u E 7-1, the norm F ( z ) = IIu V t ~ 1 1is~minimized for z such t h a t , V z = u,, i.e. z is the orthogonal projection of 1~ onto V . The unique solution is zs = Gr:gu, where G,, if the Gram matrix (in a Hilbert space) and g, is a nxl vector of the elements ( u , v z ) , for i = 1 , 2 , . . . , n. roof. 1 ( u- V Z 1'~ = 1 / u - un u, - Vzl1' = ( /u- uu1 j 2 I (u, - V Z /1;' since (u - u v , u o V z ) = 0. From Theorem 2.28, we know that the projection of u onto V is unique and it is given by uz,= VGL: g,. F ( z ) is then minimized for llu?,- V z / I 2 = IlVGr; g , - V s / J 2being equal t o zero, if the sought solution is x g = G,;,'g,. 22From a Hilbert point of view, the minimum of F cannot be found for a n arbitrary indefinite space. Assume, for instance a K r e h space K = W(',') with the indefinite norm = z: - 'c;. Then for a particular z = [l z2], the minimum of 110 - ziig = 1 - xg is reached at --oo ~
+
~
Ilzllk
+
84
T h e dissimilarity representation f o r pattern recognition
positive definite in the Hilbert space lKI. Therefore, x i H x , = u t J P u holds for any U E K ,which means that H is positive definite at 2 , .
>0 0
Below, we present an interpretation of the indefinite least-square problem, but from the indefinite point of view. The solution does not change, however, the interpretation does:
Proposition 2.2 (Minimum in the Kre'in sense) L e t K be a K r e h space over t h e field F and let f (x)= Ilb - Axil: be a, quadratic f u n c t i o n in K . T h e minimum o,f f in K i s a special saddle p o i n t xs in t h e associated Hllbert space JKl. T h i s space i s specified by t h e indefiniteness of J. T h i s m e a n s t h a t f Ix+ takes t h e minimum a t x , ~ , + a n d f 1 ~ - takes t h e m a x i m u m a t x,,_, iihere x,,+ and x,,_ are t h e f u n d a m e n t a l projections of x, E K o n t o K+ and K - , respectively.
+
+
Proof. Givcn that J = P+ (-P-), we have: f ( x ) = f + ( x ) f - ( x ) , whcre f+(x) = ztAtP+Ax - 2xtAtP+b b t P + b and f _ ( x ) = -(xtAtP_Ax 2xtAtP-b btP_b) are the restrictions of .f to Ic+ and K - , respectively. As f+ and f - are defined in the complementary subspaces (L is the orthogonal coniplement of K + ) , the minimum of f is realized by determining x,,+ for which f+ reaches its minimurn and finding :I;,,- for which f - reaches its maximum. The final solution is then 2, = z,,+ x , ~ , -(this is due to K being the direct orthogonal sum of K+ and K - ) . The critical points are the ones for which the gradients of f+ and f - are zero. This leads to x,,+ = (iltP+A)_lAtP+b and x;,,_ = (AtP-A)-lAtP_b. The Hessian matrices become, correspondingly, H+ = 2AtP+A and H - = -2AtP-A. Thanks to the properties of projection operators, P+ = Pip+ and P_ = Pip-, Theorem 2.27, one has H+ = 2 (P+A)t(P+A),which is positive definite by the construction, and H - = -2 (P_A)t(P-A), which is negative definite. Hence, f+ reaches its 0 rnininium for x,>+ and f- reaches its maximum for zs,-. ~
+
+
+
Theorem 2.30 (Indefinite least-square problem) L e t V be a linear non-degenerate su.bspace of a K r e i n space K spanned by th,e vectors ,vn] as t h e basis of V . Th,en for 712, (711, 112,. . . , u , , ~ } .D e n o t e V = [UI, the f u n c t i o n F ( x ) = liu, - Vxllg i s m i n i m i z e d in t h e Kreiiz sense for x, being t h e orthogonal projection o f 'u on,to V . T h e u n i q u e solution i s f o u n d 0,s x, = G;Jgu.
Proof. Similarly as in the proof above, we have: IIu - Vxllc = u + J u2 ztVtJu+xtVtJVz. x, is a stationary point of F ( z ) if the V7Fl,=,3 = 0. This leads to the equation VtJVz, = VtJu. By Remark 2.14, the solution
Spaces
85
is then given as x, = G,;i,lg,. We require that the Hessian, equal to 2 V t J V , is indefinite with the indefiniteness specified by J.This holds as VtP+V is positive definite in a Hilbert space K+, hence z , , ~ +yields a niiriimurn there and -VtP-V is negative definite in a Hilbert space IK-1, hence x S , ~ 0 yields a maximum there; see Proposition 2 . 2 . Remark 2.15 Note that the system of linear eguation,s V t J V x = V t J u solved in an inde5nite least-square problem can be expressed as Q'Qz = Q*u, where Q = V and Q* = V t J . This can be interpreted as a system of normal equations i n a Krefn space. Consequently, G;JVtJ is a pseudoinverse o f V in this space. 2.7.1
Reproducing kernel Krez'n spaces
Reproducing kernel Krein spaces (RKKS) are natural extensions of rcproducing kernel Hilbert spaces (RKHS). The basic intuition here relies on the fact that a Krein space is composed as a direct orthogonal siini of two Hilbcrt spaces. hence the reproducing property of the Hilbert kernels can he extended to a KreYn space, basically by constructing two reproducing Hilbert kernels and combining them in a usual way. We will present facts on reproducing kernel Pontryagin spaces (RKPS), which are Krein spaces with a finite rank of negativity (in other sources, e.g. [BognAr, 19741: a rank of positivity is assumed to be finite). Here, we will only present the most important issues, for details and proofs, see [Alpay et al., 19971 and also the articles [Constantinescu and Gheondea, 2001; Dritschel and Rovnyak, 1996; Rovnyak, 19991. All Hilbert spaces associated to Krein spaces are considered to be separable. Definition 2.105 (Hermitian kernel) Let X be a Krein space. A function K defined on X x X 4 CC of coritinuous linear operators in a Krein space X , is called a Hermitian kernel if K ( z ,y) = K ( z ,y)* for all 2 , y E X . K(z,y) has K negative squares, where K is a nonnegative integer, if every matrix { K ( x i , ~ ~ ) }based ? ~ =on ~ { Q , x ~ ., . . , xn} E X and n = 1 , 2 , . . . has at most K negative eigenvalues and there exists at least one such a matrix that has exactly K negative eigenvalues. Lemma 2.9 Let IIK be a Pontryagin space and let 2 1 , 2 2 , . . . 2, E IT,. The Gram operator G = ( ( ~ i , x j ) n J & = cwn ~ have n o more than K negative eigenvalues. Every total set in ITK contains a finite subset whose Gram matrix has exactly K negative eigenvalues [Alpav ct al., 19971.
86
T h e dissimilarity representation f o r p a t t e r n recognition
Lemma 2.10 Let 5 1 , 5 2 , ,zn,belong t o a n inner product space ( K . (.. . ) K c ) . T h e n the n,umber of negative eigenvalues of the G r a m operator G = ((xi,z j ) ~ ) ~ j coincides ,l with the dimension of the maximal negative subspace of s p a n ( x 1 , . . . , z T L[Alpay } et al., 19971. Definition 2.106 (Reproducing kernel Kreln space) Let X be a KreTn space arid let C X be a space of functions f : X + CC. Assunie K K c C X is a Kreiri space of continuous linear functionals on X . A Hermitian fiinctiori K : X X X + C is a reproducing kernel K K if (1) K ( X ; ) E K K for all Z E X and (2) K ( z . .) is the representer of evaluation at z in K K : ( f , K ( z , . ) ) K ,for all ~ E K and K all (fixed) Z E X .
,f(x) =
K K equipped with K is a reproducing kernel Krein space (RKKS). If K K is a Poritryagin space, then the resulting space of functions is called a reproducin,g kxrnel Poritryagin space (RKPS).
Corollary 2.12 Let K = Lc(X,CC) be a K r e f n space of continuous linear f u n c t i o n d s defined over the dom,ain X. If the eiJaluation functional 6,, 6 , r [ f ]= f (z) is defined and continuous for every z E X , t h e n K is a RKKS. Hence, there exists K ( z , . )E K such that 6, : z K ( z ; . )or 6 J f ] = ,f (x) = ( K ( x ,.), ,f ( . ) ) K . Therefore, the reproducing kernel is unique arid can he written as K ( z :y ) = 6, 6;, where 6; EL,-(@,K K ) is the 3-adjoint of the evaluation mapping E ( x ) for any fixed z E X . Similarly to the Hilbert case, m e has ( K ( z ,.), K(y, . ) ) K ~= K ( z ,y ) . In the case of the Pontryagin space, K ( z , y ) has at most /c. negative squares, Def. 2.105, where m is the rank of negativity. --f
Theorem 2.31 (On reproducing kernels) [Rovnyak, 19991 Let K ( z ,y) be a Hermitian kernel X X X+ C. T h e following assertions are equivalent:
1. K ( x ;y ) is a reproducing kernel for some K r e f n space K K consisting of functions over the domaisn X . 2. K ( z ,y) has a nonnegative majorane3 L ( s ,y) o n X X X . 3. K ( z .y) = K+(x,y ) - lip(.,y) for some nonnegative definite kernels K+ and lipo n X X X .
If the above holds, then for a given nonnegative majorant L ( z , y ) for K ( x ,y), there exists a K r e f n space K K with a reproducing kernel K ( x ,y ) , 23A nonnegative majorant L for K is a nonnegative definite kernel L such that L - K and &K are nonnegative definite kernels in the ‘Hilbert’sense, i.e. according to Def. 2.82.
Spaces
87
which as continuously contained in the Hi,lbert space XL with, the reprodli~cin.9 kernel L ( x . y).
+
Note that L(z,y) can be chosen as K+(s,y) K-(z.y). Note also that the consequence of this theorem is that the decomposition K ( x ,y ) = K + ( z ,y)-K-(x, y) can be realized such that K+ is a reproducing (Hilbert) kernel for ( K K ) + and K- is a reproducing (Hilbert) kernel for I(ICI<)-I in a fundamental deconiposition K K = ( K K ) + @ l ( K ~ ) - l . Practically, this means that the K* can be chosen as reproducing kerncls for the spaces in a fundamental decomposition.
Theorem 2.32 (On reproducing kernels in RKPS) Supjmse that K l ( x ,y) and K ~ ( xy), are reproducing kernels f o r Pontryagin spaccs n,, and 6 2 , a n d nK2of linear ,functions o n X with the ranks of negativity respectively. Then K ( z ,y) = Kl(x, y) K2(x, 9) is the reproducing kernel for a Pontryayin space nKwith K 5 K I ~ 2 Eqimlity . holds iff 7t = rlK,n l l K 2is a Hilbert space with the inner product: ( 2 ,y ) =~ (x.y ) n K l (2. y)n,., f o r z;y ~ 7 - t .
+ +
2.8
+
Discussion
Some classes of spaces have briefly been described. These are pretopological and topological spaces, generalized metric spaces, norrned and inner product, spaces. Normed and metric spaccs are topological. As a norm can be associated to an inner product, hence also a topology can be associated to it. Euclidean, Hilbert and Banacli spaces arc the usual examples of inner product and normed spaces, respectively. Most of the learning methodology developed deals with vectorial (feature-based) representations of objects either in Euclidean or Hilbert spaces. The reason is that the iriner product, the norin (defined by the inner product), t,he metric (defined by the norm) and topology (tlrfincd by the metric balls) coincide in these spaces. Since the probabilistic framework and other learning approaches are well developed, a natural requirerncnt for dissimilarity data seems to be their metric behavior. As a result, dissirnilarity measures used in statistical learning are either constructed or corrected to obey this requirement. In addition to the Euclidean distance. other tp metrics are considered in vector spaces. usually the city block or inax-norm distance. However, marly general dissimilarity measures have been derived for object comparison or matching in computer vision, pattern recognition
88
T h e dissimilarity representation f o r pattern recognition
and related fields, as briefly described in Chapter 5. Therefore, there is a need for learning paradigms applicable to general dissimilarity measures. Only if' a proper mathematical foundation is established for metric and non-metric dissimilarities, more general measures may be used and developed further. It is our aim to present general learning methods and to apply them to a number of problems. They will be constructed in a mathematical framework relying on generalized metric spaces and Krein spaces, which reduce to pseudo-Euclidean spaces for finite data representations. Since K r e h spaces are extensions of Hilbert spaces, Krein spaces accommodate a more general interpretation of dissimilarity data than Hilbert spaces. Since our starting point is a dissimilarity representation, all the relations and close links between generalized metric spaces and (indefinite) inner product spaces, as well as those between generalized metric spaces and generalized topological spaces are important. The most essential properties have ,just been introduced. How these spaces are used for learning is the topic of Chapters 3 and 4.
Chapter 3
Characterization of dissimilarities
A rock pile ceases t o be a rock pile the moment a single man co.ntemplates it, bearing within him the image of a cathedral. “FLIGHT TO ARRAS” ANTOINEDE: SAINT-EXUPERY ~
Various spaces in the context of generalized metric spaces were introduced in Chapter 2 . These are pretopological spaces, normed and (indefinite) inner product spaces. This chapter focuses on theoretical aspects of dissimilarities and relations between generalized metric spaces and inner product spaces. Isometric embeddings, as well as semimetric and metric transformations are described, as they are basic means for characterizing dissimilarities. A theory is also presented that deals with transformations preserving metric properties or allows one to test, whether a, pa,rticula,rdistance is Euclidean. Shortly, this chapter introduces some tools that check or enhance particular properties of dissimilarity matrices. It prepares the ground for the data exploration techniques and learning algorithms discussed in Chapters 6 10. In practice, we will always deal with finite samples, i.e. a finite collection of numerically represented data entities. Such a finite representation is used to define a space or a more general framework having suitable properties, in which learning techniques will be applied. Given a set R of n objects, this representation becomes an n x n dissiniilarity matrix D ( R ,R ) whose elements are denoted by d i j ) . Each entry d i j is a dissimilarity value bctweeri the i-th and j - t h objects. Consequently, the properties of dissimilarity measures and possible interpretation spaces are mainly discussed in the context of such finite collections. Metric distances have advantageous properties, and therefore many methods work in (Euclidean) metric spaces. Section 3.1 briefly introduces basic aspects of city block and Euclidean embeddings. Such isometric mappings find correspondences between an abstract space defined by a (fit1it.e) representation of distances and a chosen metric space. Seniinietric and metric transformations are also considered. Next, tree models for t,he rep~
~
90
T h e dissimilaraty representation for p a t t e r n recognition
resentation of dissimilarity relations are introduced. Section 3.4 presents basic relations and properties of dissimilarity matrices, especially with respect, to metric arid Euclidean behavior. Many traditional learning methods are designed in a Hilbert space or in a Euclidean space equipped with a Euclidean distance. Therefore, given a distance measurc, it is important to know whether it shows Euclidean behavior. For the Euclidean distance, every finite representation D can perfectly be embedded in a Euclidean space. This means that a configuration in a Euclidean space can be found such that the original distances are preserved. If the measure is non-Euclidean, then either it is corrected to become Euclidean or it is used as such. Any premetric non-Euclidean measure, i.e. any measure satisfying the definiteness arid symmetry constraints, Def. 2.45, can be interpreted as a indefinite distance in a pseudoEuclidean space (KrcYn space). Section 3.5 explains how both isometric and approximate embeddings in a pseudo-Euclidean space can be realized. Such mappings are examples of spatial models of the dissimilarity data. A few ot,lier projection techniques are presented in See. 3.6. Additionally, spherical embeddings are discussed in Sec. 3.5.9. 3.1
Embeddings, tree models and transformations
Both enibeddirigs arid tree models are means to represent generalized metric spaces in spatial organizations. The main purpose of (isometric) embeddings is to detcrrniric whether a given space ( X , d ) with a dissimilarity measure d is isometrically equivalent to a predefined space possessing some ~isefiillproperties. 3.1.1
Embeddings
Ernbeddings are B useful tool in practical problems where finite dissimilarity representations are considered, that is, firiitc (generalized) metric spaces ( X ,d ) defined by the corresponding dissimilarity matrix D . If an equivalence between such spaces arid other known spaces is established, the latter. if possessing favorable properties, can be used to set learning paradigms there. Although many spaces can be considered in this context, Euclidean and Hilbert spaces are the ones most extensively investigated. The reason of their applicability is that they are simultaneously topological, inner product, norrned, arid metric spaces, where the inner product is used to define
Characterization
of
dissimilaritres
91
the norm, which further defines the metric and topology. Because of these properties, many theoretical models and a probabilistic framework exist, which are used to solve pattern recognition problems formulated in such spaces. Studying the questions related to embeddings allows one to better characterize commonly used metric spaces. The essential work on Euclidean and Hilbert embeddirigs was done in [Cayley, 1841; Menger, 1931; Schoenberg, l935,1938a,b;Blurnenthal, 19531. Since Euclidean embeddings require a thorough treatment, they are the subject of Sec. 3.4 and Sec. 3.5. Here, we will briefly describe a few aspects of the e,-embeddings, with el-embeddings in particular. The el-embeddings rely on the additive property of the el metric, which. in turn. can be represented by x i additive tree; see also Def. 3.10. Let us recall some basic definitions; see also Example 2.5. Assiinie that M ( 0 )is a set of function classes on a closed and bounded set 52 measurable in the Lebesgue sense. Formally, one has: 0
! : = (RW1rL,dp), where d,(x,y)
!z = (RvL,d,), where d , ( x , 0
=
(C’,”=, Izi - y$’);
and p > O .
y ) = dmax(x,y) = max, )xi - y,J.
(s,
( M ( [ u , b ] ) , d pwhere ), d,(f,g) = If(.) - g ( z ) l ” d z ) b , p > O . L g = ( M ( [ u , b ] ) , d , ) whered,(f,g)=sup,If(x) , -g(x)ldz.
Lf
=
tp defines an sra-dirneIisiona1 space, while lr describes an infinite dirneiisional space. For simplicity, we will also write ep instead of, ; ! where the dimension m is fixed. I f p 2 1, then tpand L r are metric spaces, othcrwise, they are quasinietric spaces. el stands for the city block metric; while !22 is the Euclidean metric. Let ( X ,dx) arid (Y,d y ) be Definition 3.1 (Isometric embedding) metric spaces. ( X ,d x ) is isometrically enibeddable into (Y,d y ) if there exists an isometry 4 : X + Y , i.e. , a mapping 4 siicli that d x ( z l . : x : 2 ) = d ~ ’ ( 4 5 ( a 4) ,5 ( ~ ) )for all x 1 , n EX.
Definition 3.2 (&embedding) A metric space ( X ,d ) is !,-embeddable if ( X , d ) is isometrically embeddable into the space e; for some intcger m 2 1. The smallest such integer is called the !,-dimension. Isometrics are injective’. Two spaces are isometrically isomorphic (see footnote 11 on page 59) if there exists a bijective2 isometry between them. ‘An injective function maps distinct input values to distinct output values. 2 A bijective function f : X + Y is both injective and surjective. A surjective function has its range equal to its codomain, i.e. for every y t Y there exists z E X with f(z)= y.
92
The dzssirnilarity representation for pattern recognition
Example of a distance matrix D ( X , X ) describing a metric space ( X , d ) , X = { I , J , K , L } that, cannot be embedded in a Hilbert space.
Figure 3.1
In this case, the two spaces are essentially identical. Every metric space is isometrically isomorphic to a subset of some normed vector space. Every complete metric space is isometrically isomorphic to a closed subset of some Banach space.
Definition 3.3 (Lipschitz continuous mapping and contraction) Lct (X,dx) and ( Y , d y ) be metric spaces. A mapping 4 : X + Y is Lipschitz continuous if there exists a constant K such that dy ( q ! ~ ( z4(z2)) ~), < n dx ( X I , x 2 ) holds for all 21, x2 E X . If n < 1, then 4 is a contraction. Theorem 3.1 (On Lipschitz continuous mappings) ( I ) E ? i e ~ yLipschitz con,tinuous mapping is continuous. The reverse is not true. Sketch of proof. To see th,ut a continuous mupping may not be L%pschitz,assume X = R a.nd d(z,y) = lx-yl. Th,e function f ( z )= x2 on X i s continuous. However, no n exists such that I z2- y2 1 5 n /x-yl for all x , y E X . Consider y = 0 . Then for 15/ 5 1, x 2 5 1x1, but for 1x1 > 1, x 2 > 2 , hence a contradiction,. f is continuous, but not Lipschitz. (2) Let ( X , d ) be a metric space. The mapping x + d ( z . z ) , z E X , i s Lipsch,itz with = 1. Theorem 3.2 A Euclidean space IWvL can be embedded an a Hilbert space. Every finite subset of rri, elements in a Hilbert space can be embedded in ~ m - 1[BIunienth,al, 19531.
Not every metric space (X,d) can be embedded in a Hilbert space. A counterexample is a metric space consisting of four elements X = {I,,I, I(,L} and represented by a distance matrix in Fig. 3.1. From the definition of d , it follows that there are two points J and L which should be considered as the middle points between I and K . In a Hilbert space, howevcr, every pair 3: and y determines a unique middle point z = i ( x + y) between them such that d ( z , z ) = d ( z ,y) = d(x.y) [Blumenthal, 19531.
4
Hcnce, a contradiction.
ChaTaCteTZZatzOn
of dissamilarities
93
Theorem 3.3 (Schoenberg) Let y E (0.21 and r E (0, t ] . Then, th,e spaces (Rm,d;) and ( M ([a,b ] ) ,d;), where d; i s t h e r-th power o,f th,e P,- or' C,- distance, are isometrically embeddable in a Hilbert space [Schoen,berg, 19371. This theorem covers classes of both nori-metric and metric spaces with the dissimilarity d,, whose distances can be transformed by an appropriate power function such that they become enibeddable in a Hilhcrt space. If a finite collection of points is considered, then Theorem 3.3 refers to a Euclidean ernbedding. This theorem justifies the validity of a common sense approach, where non-metric or non-Euclidean finite spaces ( X ,d ) are transformed by a power transformation with the power r E ( 0 , l ) . This is done in practice, sincc such a transformation may be capable of making the space metric or even Euclidean; see Sec. 3.3. After such a transforrriatioii, a prenietric space may also become semirnetric. Lemma 3.1 (On embeddability of metric spaces into max-norm spaces) Any finite m e t r i c space ( X ,d ) is e,-embeddable. Assume that a collection of ri objects is given as X = Then a metric space ( X ,d) can be ernbedded in 8&. Define a data-dependcnt mapping 4 : X 4 R" such that $(:I;) = [d(x,x1),d(z,z2), . . . d ( x , z n ) l T . Denote zi = q5(xZ)for i = 1,2 , . . . , n . Then one has dm(4(zi),4(zj)) = dm(zi,z3) = rriaxl
{ X I ,5 2 . . . . , x n } .
~
Theorem 3.4 Let ( X ; d ) be a metric space.
The following implicatlon,s hold [Deza and Laurent, 1997; Ball, 1990; Bretagnolle et al., 1966; W e l l s and Williams, 197'51:
94
The dissimilarity representatton for pattern recognition
d is ta-embeddable
4
U for 2
d i s el-embeddable
4
U
E d is !?,-embeddable
for 15 p < m d i is t,-embeddable
l
Definition 3.4 (Cut semimetric) A partition of a set X into two subsets V c X and X\V is called a c u t . Such a cut defines a cut s e m i m e t r i c as 6 v ( z , y ) = Z(1V n {z,v}\ = l ) ,where 1 . 1 denotes the cardinality of tkie set and Z is the indicator function. A cut serriiriietric (metric without the definiteness condition required) is used for a further characterization of the !I-distance. A distance d is lgernbeddable if d can be decomposed as a nonnegative linear combination of nr cut metrics. Moreover, there exists a nonnegative measure space3 such that d is the measure of the symmetric difference. Formally, one has [Deza and Laurent, 19941: Theorem 3.5 (!I-distance characterized) L e t X = { X I , 2 2 , . . . , x,} a n d dz, = d ( z z , z , ) . Assume ( X , d ) i s a f i n i t e m e t r i c space. L e t i , j = 1;2 , . . . . n . T h e ,following assertions are equivalent:
Cv:vcx
d(xi,z j ) = XV 6 ~ ( z i , z j )where , XV .:%LE (1) d i j (2) There exists a nonnegative measure (probability) space (n,A, p ) and A l , A 2 , . . . , A , E A s u c h t h a t d,, = p (Ai AA,) = p((Ai\Aj) U
( 4 7 \A)).
() ( X ,d ) is -P;'L-embeddable, i. e. there exist vectors {vl, v2, . . . , v,} E Rm f o r ,some integer 'ru 2 1 s u c h that d i j = lvik - v j k 1.
cy=l
(a,
'A measure space is defined as a triple A, p ) , where R is a set, A is a 0-algebra on (a collection of subsets A such that E A and if A E:A, then a\A E: A, and a union of any number of subsets of A belongs to A, as well).
Characterazation of dassamilarities
95
Although the el-distance admits a decomposition by cut seniimetrics. it
does not, simplify the process of deciding whether a particular distance is isometrically el-embeddable or not. The difficulty of devising a polynomial algoritlim for an (;"-embedding is due to the non-uniqueness of such a dccomposition. Given n points, there are 2n-1 possible cuts. This problem is known to be NP-complete [Karzanov, 19851.
3.1.2
Distorted metric embeddings
Not every metric space is isometrically embeddable into the or Y, spaces. This is, however, possible with distortion. Let ( X ,dx) and (Y,d y ) be mctric spaces. Then a mapping 4 : X 4 Y is an embedding with a distortion c 2 1, called c-embedding, or bi-Lipschitz embedding, if there exists T > 0 such that r d x ( x , y ) 5 d y ( 4 ( z ) , $ ( y ) ) 5 c r d ~ ( x , yholds ) for all x , y ~ X . There has been a lot of research devoted to the problems of distorted embeddings into the &I, 2! and too spaces. For instance, a classical result is that any finite semimetric space ( X , d ) of n points, = n, can be embedded into L1 with distortion c = O(log, n ) [Bourgain, 19851. Anotlier result is that a finite n-element metric space ( X , d ) can be embedded into $ for p 2 1 and k = U((log,n)') dimensions with distortion c = U(log, s n ) . c-embeddings are not treated in this book. Nevertheless, they remain of high interest for further study as such embeddings to low-dimensional vector spaces may be useful for learning. An overview of import,ant results concerning low-distortion embeddings with the algorithmic emphasis can be found e.g. in the following works [Indyk, 2001; Linial, 2002; MatouSek, 2002; Krauthgamer et al., 20041.
1x1
3.2
Tree models for dissimilarities
Representing dissimilarities by trees is an important issue in many scientific fields, like data analysis, mathematical psychology, historical linguistics, bioinforrnatics and evolutionary biology; see e.g. [Sneath arid Sokal, 1973; Barthdemy and Guknoche. 1991; Kim and Warnow, 19991. A tree structure of the dissimilarity matrix allows one for a natural interpretation of relations between the objects. It is a useful tool for understanding the data structure, especially for a smaller number of objects. where the resiilts can be presented visually. Further on, tree models support the hierarchical clustering scheme based 011 proxirnities; see Chapter 6. In brief. tree mod-
96
T h e dassimilarity representation f o r pattern recognition
els order the dissimilarity information in ternls of organizational aspects, hierarchical and nested structures. Formal definitions follow.
Definition 3.5 (Graph) A graph, (V, E ) consists of a set of nodes, V = { v l , va;. . . . w,} arid a set of edges E { ( v i ,wj) : vJ,vJ E V} connecting two nodes. In a directed graph, the pairs (w,,w3) in E are ordered, in an undirected graph, they are not. The degree of a node vi is the number of edges incident in 7 ) i . Nodes with degree one arc called leaves. Nodes with larger degrrc are called inkrnal. A weighted graph (V,E , W ) is a graph (V,E ) with nonnegative weights wij E W associated to the edges (vi,vj). Definition 3.6 (Path and minimum path distance in a graph) Let (V,E ) be a graph. A p a t h between two nodcs 'oil and vtk is a sequence of connected edges (vil, u i 2 ) , (wi2, (vikpl, uik).A graph is connected if there is a path between any of the vertices. The length of a path is then a sum of the weights of the connected edges between the nodes vil and < q kThe . minimum p a t h distance between any two nodes is defined as the minimum length of the paths connecting them. Definition 3.7 (Tree) A tree is a connected graph, where each pair of nodes is connected by a unique path.
A key model is the additive tree model, which represents objects by nodes of a tree and defines dissimilarities as path lengths between two nodes, computed by the sum of the weights of the edges on the path. A special case of a rootcd additive tree is an ultrametric tree; in which the distance from the root to every leaf is identical. The formal definitions follow. Definition 3.8 Let ( X ,d ) be a metric space. If X = { X I , 2 2 , . . . ,z,}, then d is described by an rixn, distance matrix D = ( d t 3 ) such that dij = d (xi:x3).Additional constraints can be considered for all IC, y, z , EX: ultrametric neyuuliky: d(z,z ) 5 max { d( z, y ) , d ( y , z ) } .
A metric space satisfying the above inequality is called ultrametric or ri,ori,-Archirr~.edean; see also Def. 2.44 and fact 2.11. ( 2 ) four-point property: d ( z ,y) d ( z , u)I max { d ( z , u ) d ( y , z ) ,d ( z , z) d(Y. u)1. (3) hypermetric inequality: y T D y 5 0 and y E Z n such that yT1 = 0; where D is an ' n x n , distance matrix. A metric space obeying the above inequality is called hypermetric. An infinite metric space X is hypermetric if the inequality holds for every finite subspace of X .
+
+
+
Characterrzation
of
drssrmrlarrties
97
(4) neya,tiwe t.y;ne: y T D y 5 0 for y E Rn such that
yT1= 0, where D is an nxn distance matrix. A metric space obeying the above inequality is called of ne,qati,ue trjpe. An infinite metric space X is of negative type if the inequality holds for every finite subspace of X .
The ultrametric and four-point properties can also be considered for prcmetric spaces. The four-point property and ultrametric inequality are iriliereritly connected to a tree structure of the distance data. Definition 3.9 (Additive, ultrametric trees) An additive tree is a connected, undirected graph where each pair of nodcs is connected by a unique path. An ultrametric tree is an additive tree in which each leaf is equidistant (along the path) from the root. Let ( X ,d ) be a finite metric space and let D be the corresponding distance matrix. D defines a unique additive tree iff the four-point inequality holds for any quadruple from X. D defines a unique ultrametric tree iff the ultrametric inequality holds for any triplet from X . See Fig. 3.2 for examples. Formally, one has [Barthdemy arid Gui.noche, 19911: Definition 3.10 (Additive distance tree) Let D = ( d t J )be it11 n x n symmetric distance matrix between the elements of X . Let To be an edgc weighted tree with at least n nodes, where n distinct nodes of To arc labelccl by the elements of X. To is an additive tree for the matrix D if for every pair of the labeled nodes ( i , j ) ,the path from the node i to the node j has the total weight equal to d,, . In an additive tree the root is not determined and choosing differcrit roots may suggest different interpretations or relations iri the clata. As the root will likely distinguish a few significant groups in the data, it should be chosen to enhance the interpretability of the data. This, however. requires some prior knowledge on cluster tendencies of some objects. Another possibility is to place the root a t the midpoint between two most distant objects in thc tree or at a node which minimizes the variance of the distances from the root to the leaf nodes, so it splits the data into homogeneous groups. In a weighted graph, the path metric defines the shortest path, judged by the total sum of weights, between two nodes. A metric satisfying the four-point property is the path metric of nonnegative weighted trees.
98
T h e dissimilarity representation for p a t t e r n recognition
7 j
D I - J
(a) Additive distance tree
K
L
D I
J
K
L
(b) Ultrametric distance tree
Figure 3 . 2 Tree examples. (a) Additive distance tree for the given D satisfying the four-point property. (b) Ultrametric distance tree for the given ultrametric distance matrix D .
Theorem 3.6 E v e r y p a t h distance, i e . t h e shortest-path distance in a tree, i sI! -embeddable Proof. Let T = (V, E ) be a tree, where '11 is a set of vertices arid E is a set of edges. Every edge e = (vk,vl)introduces a partition of V into two sets V k and Kc = V\Vk such that v k E V k and YL E 4". The path metric of T caii be decomposed as d T ( u i , v j ) = C ( V k , V ~ ) E E b V ~ ( ~ where k , , u l ) 6vk, , is the cut metric, Def. 3.4. In the case of a weighted tree, the values of 6v,,( i i k , ,ul) are multiplied by the weights ~ i k l .Since the path metric can be decomposed as a linear combination of cut metrics, then it is I! -embeddable by Theorem 3.5. 0 This statement makes a connection between the embeddability into a l1space and a path metric of an additive tree. Consequently, any distance matrix D which is !,-embeddable can be represented by a path metric of an additive tree T . NIoreover, if a path metric is embeddable in !?, then the tree has at most 2m leaves [Hadlock and Hoffman, 19781. This establishes a discrete model of distance relations between the objects.
Definition 3.11 (Ultrametric distance tree) Lct D = ( d i j ) be an n,xn symmetric distance matrix between the elements of X . An ultram,etric distance tree for the matrix D , also called dendrogram, is a rooted tree TD with the following properties: 1. TD contains n. leaves, each labeled by a unique element of X . 2. Each internal node is labeled by distance values from D such that d i j is the label of the least coninion ancestor of the leaves i and j . 3 . Along any path from the root to a leaf, the numbers labeling internal nodes strictly decrease. 4. Each internal node has at least two children.
Characterization of dissimilarities
99
In practice, there might be no additive tree to represent the distance data D , since there might be no path metric coinciding exactly with D . A solution can be offered by finding a tree which models the givcn dissimilarities as well as possible by the path distances. Such a tree imposes a metric D T , which should provide the best approximation of D iinder some criterion, defined e.g. by the !I, e, or ! , norms. This is a formulation of a numerical taxonomy problem, which has received a great deal of attention over the years; see e.g. [Barth4lemy and Guknocl-ie, 1991: Kim and Warnow, 19991. The additive or ultrametric tree fitting problems are known to be NP-hard under the (1 arid t 2 norms [Sneath and Sokal, 1973; Kim and Warnow, 1999; Cliepoi and Fichet, 20001. In the case of tjhe t m norm, the same holds for an additive tree [Agarwala e t al., 19991, however the optimal ultrametric tree can be computed in a polynomial time [Farach et al., 19951. There exists a number of other methods trying to construct either an ultrametric or additive tree such that the path distance approximates the given distance as well as possible. Refer t o [Barth4lemy and Guknoche, 1991; Gordon, 1996; Kim and Warnow, 1999; Sneatli and Sokal, 19731 for general descriptions or to [Agarwala et al.: 1999: Cohen and Farach, 1997; Farach et al., 1995; Gascucl, 1997, 2000; de Soete, 1 9 8 4 ~ ; de Soete and Caroll, 19961 for more specific algorithms. See also Sec. 6.4.
3.3
Useful transformations
Some transformations are considered, which either preserve the (semi)metric properties or, in particular cases, change a dissimilarity rneasure into a metric. Properties of convex and concave functions are introduced in Appendix A. 3.3.1
Transformations in semimetric spaces
Theorem 3.7 ((Semi)metric transformation) Let (X,d) be a s e m i m e t r i c space. Then the composition of mappings, f o d , i s also sem%metrici f f :Rt+ Rt is a non-decreasing a n d concave function such t h a t f ( 0 ) = 0. If ( X ,d ) is a m e t r i c space, th.en f o d is m,etric, if additionally f i s positive o n R+.
Proof. We focus on a metric space, as the proof for a semimetric space is analogous. Let ( X ,d ) be a metric space. Since f(x) > 0 for positivc x and
100
The dissimilarity representation for pattern recognition
f(0) = 0, then f o d directly fulfills the positivity, reflexivity and symmetry constraints: see Def. 2.38. It suffices to prove the triangle inequality. Let d,, = d ( z .y), d,, = d ( y , z ) arid d,, = d ( z , 2 ) for any 5 , y, z E X . Assume that d,, + d,, 2 d,, holds for every triplet ( d z y ,d,,, d Z z ) . Since f(d,, + d,,) 2 f ( d Z Z )and ,f is non-decreasing, one suffices to show that f(d,,) f(d,,) 2 f ( d , , d,,). The inequality f ( d x , ) f(d,,) 2 f(d,,) is then straightforward. Based on the concavity of f , one has that f ( a t (1 Q ) U ) ~ o l f ( t ) + ( l - o l ) f ( u ) f o r a l l a ~ [ O , l ] a n d a l l t0. , uL~e t o l = *, u = d,, d,, arid t = 0. We have:
+
+
+
+
+
(3.1)
Also, . f ( d y Z )2 f(&,+d,z). which finishes the proof.
Hence, f ( d x , )
+ f ( & ) 2 f(dxy+dyz), 0
f is required to be non-decreasing and concave. In fact, instead of concavity, the subadditive property of f is needed, i.e. f ( x 1 x2) 5 f ( q ) f ( x 2 ) for all x1 and z2 iri the donlain o f f .
+
+
Corollary 3.1 L e t ( X ,d ) be a m e t r i c space. T h e n ( X ,f o d ) i s also m e t r i c f o r t h e following f u n c t i o n s f :R: +: R ( 1 ) f(x) = c z , c > 0 . (2) f(.) = z C Z ( ~ > O ) , c>o. (3) f ( z ) = rniii{c,x}, c>O. (4) f(x) = z7‘>O
0 .
+
(G) f ( z ) = sigm ( x ) = 2 / ( 1 +exp($))
(7)f(.)
-
1, a>O.
log(l+z).
Proof. It is easy to verify that all the functions above are rrionotoriically growing and f ( 0 ) = 0. Concavity of f ’ s are easily proved by showing that f ” are n e p t i v e for x > 0 ; see Theorem A.l. 0 Theorem 3.8 (Blumenthal) L e t ( X , d ) be a m e t r i c space and let f T ( z ) = xT be a m e t r i c t r a n s f o r m w i t h r E ( 0 , $1. T h e n d‘ = f T o d is metric and any f o u r p o i n t s of (X,d‘) can be isometrically embedded in a Euclidean s p c e [Blumenthal, 19361.
Characterization of dissim,ilarities
101
Note that for any metric, every three points can be isometrically embedded in a Euclidean space; which follows from the triangle inequality: see also Sec. 3.4. The above theorem explains that the power transformation f T ( x ) = xT with 0 < T 5 $ makes the metric ‘inore’ Euclidean, sirice the &-embeddability holds for any four points. Corollary 3.2 Let d, be the !,-distance
and p E ( 0 , l ) . T h e n the space
(Rm,dF) is metric. Proof. Assume that x, y , z E R“. Note that dE(x,y) = Czl I:ci - z / i ( ” . It suffices to show that the triangle inequality holds, since (R”,d,) with p~ ( 0 , l ) is quasimetric, as shown in Example 2.7. Based on the Minkowski m inequality4, lai bilp 5 (ail?’ (bilJ’,we have:
+
+
+
The latter inequality is equivalent to stating that dF(x,z ) 5 d:(x, y) 0 dg(y,z). Hence, d; is metric. Remark 3.1 Since ( R m dg), , with p E ( 0 , l ) is a metric spece, then based o n Corollary 3.1, ( R m , d g ) with r E ( 0 , p ] is a metric space as well. Definition 3.12 Let
G denote a set
of functions g ( z ; p ) of one real nonnegative variable z and one real parameter p > 0 such that g ( 0 ; p ) = 0 and g ( x ; p ) is a continuous strictly increasing function of 2 . Moreover. g is such that for m y x > 0 and any real E > 0, there exists a such that p 5 a + jg2(x;p)- I ] 5 E [Courrieu, 20021.
4 are [Courrieu, 20021: with s u p s < 00. Let E < 1. Then a > 0 for
Example 3.1 Examples of the functions from 0
0
Power function: g ( z ; p ) = ICJ’ z = 1 and a = [log (1 sign(z - I ) ~ ) ] / ( 2 l o(x)), g otherwise. Weibull function: g ( z ; p ) = 1 - exp ( -:cr/p) with T > 0. Let & < I . Then a = - z T / l o g ( l - fi).
+
Theorem 3.9 (Courrieu) Let ( X ,p ) be a n n-element finite quasirnetr.ic space. Let be a set of functions as in Def. 3.12. Consider a funxtaon g E 8. T h e following statements hold: 4The proof of this inequality is as follows. Let a 2 b 2 0 and p E (0,1]. Let -1 g(z) = (1 + cf). with c E (0,1] and z 2 1. g’(z) = -5 log(1 + c 5 ) log(c)g(z) 2 0, so g is non-decreasing. g takes the minimum value of (1 + c ) a t z = 1. For z =
I and
c = $ < 1 , ( a p + b P ) ~ = a g ( ~P ) >- a g ( l ) = a + b h o l d s . H e n c e , ( a + b ) p < u p + b ~ ’ .
102
T h e dassamilarity representation f o r p a t t e r n recognition
( I ) There exists a real
n(X)> 0 s u c h that f o r a n y positive
p
2 a ( X ) , the
space ( X .g ( p ; p ) ) is %sometricallyembeddable in a Euclidean space Rk of t h e d i m e n s i o n I; 5 n- 1. (2) T h e r e exists a real p(X)s u c h t h a t 0 < p ( X ) 5 a(X)and for a n y posit%uep 5 p(X)the space ( X , g ( p ; p ) )i s isometrically embeddable in a Euclidean space Rk of a dim,ension k = n- 1 [Courrieu, 20021. Thc above theorem explains that any finite quasimetric space can be transformed into a Eiiclideari space by a suitable function g ( . ; p ) . It does not: however, explain how a proper parameter p can be determined. This depends on the set X and cannot be captured by a general formula. In practice, p < 1. For p approaching zero, the quasimetric space resembles more arid more a discrete metric space, defined as d ( z , y) = Z(x # y). This means that thc structure in the data is weakencd in the embedded Euclidean space, since the points move towards the corners of an equilateral polytope. Still for any p > 0, some structural information is present.
3.3.2
Direct product spaces
Direct product spaces allow one for a construction of' a new space by combining two (or more) spaccs; see also Sec. 2.4. In the context of generalized metric spaces, this means that given two (or more) such finite spaces describing the same objects, a new dissimilarity measure can be created by their summation or by inaxinium operator. Now: some conclusions can be drawn for the combined spaces.
Theorem 3.10 (Metric direct product space) L e t (X,dx)and (Y.d y ) be m e t r i c spaces. The direct product space ( X x Y .d x d y ) , defined as (dx.dY)((lC1:y1),(xzrY2)) = dx(~llm).dY(Yl,Y2),w h e r e ~ 1 , x z E X and y1 , y2 E Y , and i s either t h e s u m o r m a x operator, i s m e t r i c .
Proof. The proof is straightforward by checking the conditions of 0 Def. 2.38. Theorem 3.11 L e t (X,px) and ( Y , p y ) be m e t r i c spaces. Suppose that t h e direct product space (XxY, px @ p y ) i s defined s u c h that ( p x @ ~ ~ ) ( ( ~ 1 ~(YI,Y~)) 1 ~ ~ 2 = ) , p x ( ~ i , x 2 ) + p y ( y iy2) 1 x1,m E X and y1,y2 E Y . T h m t h e space ( X X Ypx , @ p y ) is
(1) !,-embeddable (f ( X ,px) and (Y, py) are !I-embeddable. (2) i s &permetric $7' ( X ,px) and (Y, p y ) are hypermetric. (3) i s of n,egative type if ( X ,p x ) and (Y, p y ) are both of negative type.
103
Characteritatton of dissimilarities
Proof. Let Z = X X Yand p = px @ p y . (1) ==+ Assume that (2, p ) is (1-erribeddable. Then p ( ( x l ,yl), (xz,;y2)) = p x ( s 1 , x z ) 0 PY(Y1,YZ) 0 = px(z1,xz) p x ( G ! > x 2 ) pY(Yl>:Y1) PY ( ~ 1W,Z ) = P ((xi,N ) , (W , ~ ) ) + p ( ( yi, m ) ,(xz,yz)). Consequently, (2, p) is ( X X { ~ z } , p xx )({yi}xY,py), which is equivalent to ( X , p x ) x (Y,py). Hence, ( X ,px) and (Y,p y ) are (1-embeddable.
+ +
+
+
+
+
Assume that ( X , p x ) and ( Y , p y ) are el-embeddable. Let, 4x and denote the L1-embeddings of the spaces ( X , p x ) and (Y,py), correspondingly. Then the embedding 4 of ( 2 , p ) into el can be obtairied by d x , Y) = [4x(x) 4~ ( v ) ] . Since p = px @ p y , then p ( ( ~ 1 ~ x 2(yl ) . y2)) = +==
4y
Px(Z1.22)
+ II4Y(Y1) [ # ) x ( x 2 )4y(!h)/Ii = ll4(a,w) ~(z~.Y~)III.
+ PY('Y1,YZ)
iI[d)x(xi) b ( Y i ) ]
-
= II#)x(Z1) - 4x(22)111
-
$y(y2)I11 =
Hence,
-
( X X Y p, ) is ti-enibeddable. (2) ==+ Let (2, p ) be hypermetric. This means that for y1 E Y , (Xx{yl}, p ) is hypermetric as well. Then p ( ( x l , y l ) ,(z2,yz)) = ~ X ( : ~ . ' I , J : Z )so , (X,px) is hypermetric. The same reasoning holds for (Y,p y ) . +== Let (X,px)and ( Y , p y ) be hypermetric spaces. Define z E Zz as a function of (x,y) E X X Y to satisfy C(zl,yl)EZ z(x1,yl) = 1. Define u E Zx arid u E Zy such that u(z1) = C y l t Y z ( ~ l , y land )
4Yl) =
C z l t X z(x1,Yl). Consewntly, CZIEX 4 x 1 ) = CYIEY 'C'(V1)
Then
C ~ ( z l ; z 2 ) E X A ( y 1 . Y 2 ) E Y4}Z 1 , Y l )
4 x 2 . Y2) P((Z1, za), ( Y l .
y2))
C(z,,z2)€X 4 x 1 ) U ( Y l ) P X ( Q , Y 1 ) + C ( Y l , y 2 ) E4yY 1 ) 4 Y 2 ) P Y ( ! l 1 , Y 2 ) Hence, be Def. 3.8, ( 2 , p ) is hypermetric. (3) The proof is similar to the one above.
= =
5 0.
0
If X = Y, then the above theorem states that p x @ p y , the summation of distances, preserves the tl-embeddability, hypermetric property and the propcrty of being of negative type. Let 2 = X X Y . We also know that if (2, p ) is of negative type, then (2, p 4 ) is tz-embcddable by Theorem 3.13. This means that if (2, p x ) and (2.p y ) are tz-embeddable, then thc space (2, (p; &)a) is also tz-embeddable.
+
3.3.3
Invariance and robustness
Invariance is an important issue for designing informative dissimilarity measures. It may be studied in the context of invariant pattern recognition or for matching purposes [Rodrigues, 2001; Hagedoorn and Vcltkamp. 1999a,b]. Here, we will briefly discuss it in terms of (semi)metric properties.
104
The dissimilarity representation for pattern recognition
Definition 3.13 (Invariance) A semimetric space ( X ,d ) is invariant under the transformation group 7 if it is invariant for each element t from 7, i.e. d ( t ( z )t,( y ) ) = d ( z , y) holds for all z, y E X . By the definition of a group, Def. 2.50, the identity transformation id7 belongs to 7 . Also, for each t E 7 ,there exists the inverse transformation t-l E 7 such that t o t-' = t-' o i = id7. Consequently, d ( t ( z ) , y ) = d ( z l t p ' ( y ) ) and d ( z ,t ( y ) ) = d ( t - ' ( z ) , y ) . For example, if 7 is a group of rotations and translations, then Euclidean distance is invariant under 7. The identity is the zero rotation (or the zero translation). The city block distance is not rotation invariant, although it is still translation invariant. Theorem 3.12 (On invariant measures) Let 7 be a transformation group in a semimetric space ( X ,d ) such that ( X ,d ) is invariant under 7. Then
(x,d T ) is a semimetric space, where d 7 ( z , y) = inftGT d ( t ( z )y) , (2) vitl,tzE7 d7(t1(z),t2(u)) = d T ( z , y ) . (1)
Proof. (1) Note that d7(z,y) 2 0. Then 0 5 d7(z,z) 5 d(idl(z),z) = 0 and the reflexivity constraint is fulfilled. Since d is symmetric and invariant under 7 ,then d7(z, y) = i n f t E i d ( t ( z )y) , = i n f t c l d(y, t ( z ) )= infit,7 d ( t - ' ( y ) , z ) = inft,,7 d ( t ' ( y ) , z ) = d7(y,x). Hence, d7 is symmetric. Let t1,tz E 7. Since d is semimetric and invariant under 7, one can write d((t1 o t 2 ) ( z ) , z ) = d(tl(z),t;l(z))5 d ( t l ( z ) , ~ ) d(g,t,'(z)) = d ( t l ( z ) , y ) f d ( t 2 ( y ) , z ) .Denote t = t l o t z . Then d 7 ( z , z ) = i n f t l , t 2 a -d((t1 0 t2)(z),z ) 5 inftl,t2Eid ( t l ( z ) ,y ) + d ( t ~ ( y ) z: ) 5 i n f t l E i d ( t ~ ( z ) , y ) + i n f 'd(ta(y), t ~ ~ ~ z ) = d*(z,.y)+d7(;y, z ) , which proves that the triangle inequality is satisfied. Hence, d7 is a semimetric.
+
(2) Consider any t l ? t 2€ 7 .d 7 ( t l ( z ) , t 2 ( y ) )= inft,gd((t o tl)(z):t2(y)) = i n f t c 7 d ( ( t z 1 0t o t l ) ( z ) , y ) = inft!€.rd(t'(z),y) = d7(z,y), wlicre t' = t;1otot1.
0
This theorem has a practical implication when objects defined by a set of points or by contours need to be compared. For instance, this is useful for objects in digitized images. It justifies the common sense reasoning of computing the distance between two objects as the value of a smallest mismatch. One of the objects is usually transformed to match the other in the best possible way. If there are invariant transformations, then the semimetric properties of the distance measure are preserved; see also Sec. 5.5.
Characterization of dassamilarities
3.4
105
Properties of dissimilarity matrices
This section discusses some properties of dissimilarity matrices with respect to metric behavior, metric transformations and Euclidean erribeddings. Also corrections of dissimilarities, which impose the metric or Euclidean constraints, are discussed. This is done explicitly for dissimilarity matrices, since our learning algorithms will be later based on such finite representations. A substantial part of the presented theory comes from the work of Gowcr [Gower, 1986, 19821. Basic notions and facts of traditional matrix algebra, as well as of pseudo-Euclidean algebra (which is relatively new)? are collected in Appendix B. The reader should consult books on linear algebra, such as [Bialynicki-Birula. 1976; Greub, 1975; Lang, 2004; Noble and Daniel, 19881, if more details are needed. 3.4.1
Dissimilarity matrices
Here, we will only discuss matrices over the field of real numbers. Concerning the notation, e3 E is a standard basis vector, i.e. e j = 1 and ei = 0 for i # j , 1 denotes a vector of all ones and 0 stands for a vector consisting of all zeros. Moreover, if a vector representation X is mentioned, we will follow the convention from pattern recognition, where n vectors in
RnLare placed in rows of an nxm matrix X , i.e. X
=
[
xq . The consex:
quence of this convention is that the Gram matrix G, Def. B.3, becomes now G = XXT. This notation is a bit unfortunate for matrices in pseudoEuclidean spaces, as the adjoint operator is not the transpose. However, to be consistent with the description of learning techniques, in both Euclidean and pseudo-Euclidean spaces, a vector representation X will be meant as an n x n i matrix of 71, transposed vectors in R”’. The Gram matrix in a pseudo-Euclidean space R(P.4) becomes G = XJp,XT. Consider an n x n dissimilarity matrix D = ( d i j ) and an n x ’ n similarity matrix S = ( s t 3 )for i , j = 1 , .. . , n . In all discussions below, we assume that D is nonnegative and has a zero diagonal. D = D ( X , X ) is now undcrstood as corresponding to a finite generalized metric space ( X ,d ) , X = { z i , z 2 , . . . , z n } , such that the elements of D are d,, = d ( q , z j ) , i, j = 1,2 , . . . , n. In this book, an R X R distance matrix D is called Euclidean if the elements d i j of D are Euclidean distances between pairs of vectors in some real vector space. Note that in the literature, other authors may call D*’ Euclidean.
T h e dissimilarity representation for p a t t e r n recognition
106
Recall from Def. 2.45 that premetric obeys the reflexivity and symmetry constraints, and quasimetric additionally fulfills the definiteness axiom. If the triangle inequality is satisfied, then a quasimetric becomes metric. Semimetric is a metric without the definiteness axiom.
Definition 3.14 (Metric for D) Let D be a symmetric dissimilarity matrix with positive off-diagonal elements. D is metric if the triangle inequality dij d j k dik holds for all triplets ( i , j , k ) .
+
>
Remark 3.2 L e t D be a s e m i m e t r i c . (1)
I l f d i j = E,
(2) If
d,ij = 0 ,
then then
ldik -
d j k / 5 E for a n y k . for a n y k .
dik = d j k
Pro0f.
+
+
(1) By the triangle inequality, d i k d i j 2 d k j and d j k d j i 2 d k i hold for any k . Based on the symmetry condition and dij = E , one obtains: dik E 2 d j k and djk E 2 d i k , which after a simple transformation, gives dik E 2 d j k 2 d i k - E and finally ldik - d j k / 5 E .
+ +
+
(2) Trivial, by the same reasoning as in (1).
0
These properties of metric dissimilarities are important from a practical point of view. If two objects are similar, which means that the dissimilarity between them is small (close to zero or equal to zero), then any other object will have a similar relation to them both. As a result, one of them niay become a prototype to represent the information of both of them. This property is used for approximate nearest neighbor searches in a Euclidean space; see e.g. [Moreno-Seco e t al., 20031.
Remark 3.3 D i s a m e t r i c i f every triplet is Euclidean. Any metric triplet ( d i j , d i k , d k j ) is Euclidean, as it constitutes a Euclidean triangle. However, if n > 3 , not every n x n metric distance matrix D has a Euclidean representation. A counterexample is a 4 x 4 matrix D shown in Fig. 3.3. D can be isometrically embedded in 15: but not in l 2 , so no Euclidean representation of D exists.
Corollary 3.3 I f D i s quasimetric (the triangulation inequality does n o t hold), t h e n t h e m a t r i x D' = D f c (llT-I ) , c 2 maxp,q.rId,, d,, - d,,l i s m>etric.
+
Proof. It suffices to show that D' fulfills the triangle inequality, since other properties can easily be checked. Let ( i , j ,k ) be a triplet for which the triangulation inequality does not hold, i.e. dij + d j k < d i k . Since c 2
Characterization of dissimilarities
107
I
I
0
3
3 1.6
J
3
0
3 1.6
K 3
3
0 1.6
1’5/J -0.1
L 1.61.6 1.6 0
(c) Il representation
(b) No Euclidean representation
Figure 3 . 3 Example of (a) metric distances with (b) no Euclidean representation and (c) a possible el representation. In order to get a 2 0 or 3 0 Euclidean representation, = &f. the distances from the point L to other points should be at least equal to They are smaller, so no Euclidean embedding exists.
59
+
rriaxp,q,rl d p q d,, - d,J, then (djk c) prove that (dij + c ) dij djk 1di.j d j k - d i k l = c is nonnegative, then ( d i j +c) proof.
+
+
+
+
+
C
2
2
(dik
+ djk + c). Note
NOW,we should that d i j djk c 2 d i k , since Iz/ = -z for z < 0. Because ( d j k + c ) 2 ( d i k + C ) , which finishes the ldij
-
+
+
+
If c is relatively small, then the dissimilarity matrix D is only slightly non-metric. If? however, c is large, then the analysis should take into account its non-metric properties. The triangle inequality is the most burdensome to check; in the worst case scenario, all the triplets need to be investigated. In practice, c is not the smallest possible value that will make D metric. Having estimated c, one may follow a sort of bisection rnethod with a specified precision to determine a smaller value C’E (0, c).
Remark 3.4 An i m p o r t a n t question refers t o t r a n s f o r m a t i o n s of a m e t r i c dissimilarity s u c h t h a t t h e m e t r i c properties are preserved. F r o m Theor e m 3.7, w e kn,oiii t h a t if D i s m e t r i c , th,en so i s D f = ( f ( d i g ) ),for f being a non-decreasing and concave f u n c t i o n s u c h t h a t f ( 0 ) = 0 and f (x)> 0 f o r 2 > 0 . Consequently, if D = ( d i j ) is m e t r i c , t h e n f o r c > 0 t h e follouiing dissimilarity matrices also are: ( c d i j ) , ( d i j + c ( I - & j ) ) , (min(1, d i j } ) , ( &23. ) with r E ( 0 , I], ( d i j / ( d i j c ) ? (sigm ( d i j ) ) , (log(l+ d i j ) ) ; see Corollary 3.1.
+
Below we present some results, mostly related to the Euclidean behavior of a distance matrix and its vector representation. A more thorough explanation can be found in Sec. 3 . 5 .
Definition 3.15 (Euclidean behavior) An R X R distance matrix D = ( d i j ) is Euclidean if it can be embedded in a Euclidean space (IFn:d 2 ) : where 7n. 5 n. In other words, a configuration {XI x2,.. . , xn} m n be determined in IwVL such that da(xi,xj) = llxi - xjllz = di,?for all :‘.;j.
108
T h e dissimilarity representation f o r p a t t e r n recognition
Theorem 3.13 (Test I for Euclidean behavior) A symmetric n x n niatrix D with a, zero diagonal is Euclidean a# D*’ = (d:j) is conditionally negative definite (cnd). This means that zTD*’z 5 0 holds for all vectors z E R” such that zT1 = 0 . Equivalently, a symmetric n,xn matrix D with a zero diagonal is Euclidean i;rf -D*2 is conditionally positive definite (cpd). Proof. Let 0”’ be a square Euclidean distance matrix. This means that there exist vectors XI,x2,. . . , x, in an m-dimensional Euclidean space R” such that d:j = I/xi-xJ’. Let gEPSTLconsists of the elements gi = I(xi11’, i = 1 , 2 , . . . n. Then,
=
2zTlgTz - 2 IIzTXI1’.
Note that IIzTXI1’ 2 0 and zT1gTz2 0, as l g T is a psd matrix. Since zTD*’z = 2(zT1gTz- I/zTX1l’), then to assure that z ~ D * ~ 5 z0, one should require that zT1 = 0. Hence, D*’ should be cnd. This finishes the proof. El
Theorem 3.14 (Test I1 for Euclidean behavior) [Gower, 19861 An distance matrix D is Euclidean iff the matrix D:’ = J,D*’J,: with J , = ( I - I s T ) is negative semidefinite (nsd) for sT1 = 1. Equivalently, D i s Euclidean ifl the matrix S;’ = - $ D , is positive semidefinite (psd) for sT1 = 1. 11x71
Proof. ==+ For any x E R”,the vector z = ( I - IsT)x is orthogonal to 1 , i.e. zTl = xT(I- slT)1 xT1 - xT1sT1= xT1 - ( ~ ~1 1= )0. Then by Thcorem 3.13, zTD*’z 5 0; which yields xT[(I- IsT)D*’(I- s l T ) ] x 5 0. This proves that 0:’is negative semidefinite.
+=
Assume be nsd. This means that 0 2 zTDi2z for zT1 = 0. One has zTD;’ z = zT[(1- I s T ) D*’(I- slT)]z= z ~ D * -~ 2z zT1sTD*’+ zTlsTD*2s1Tz = zTD*’z. Hence is cnd and by Theorem 3.13, D is Euclidcan. 0
Remark 3.5 D is Euclidean ifl 0,“’ = J D*’,J is nsd. J = ( I - 11’) is knoiun as the centering matrix. This is a special case of Theorem ,?.14 .for
Characterization of dissimilarities
109
il.
s= Another special case holds for s = ei, where ei ER" is a standard basis vector. Remark 3.6 Let D be an r i x n distance matrix. If Theorem 3.14 i s true f o r a particular vector s , e.g. s = $1, then it is true for any s s,uch that sTl = 1. Some further intuition, is presented in Sec. 3.4.2. Theorem 3.15 (Vector representation) Let D be an nx'ri Euclidean distance matrix. Then there exists m 5 n and a vector representation of the distances d,,j in Rm, defined b y the rows of an r i x m matrix X , such that X X T = -$(I- I s ~ ) D * ~ (slT)and IsT1 = 1.
-+
Proof. Indirectly, the goal is to prove that S,? = ( I - 1sT)D*'(I- slT) is a matrix of inner products (Gram matrix) of some vector representation X in Kim. Let h = 0 " ' s - $ lsTD*2s. It can easily be check that h is the diagonal of S,;see also Sec. 3.4.2. After straightforward calculations, we get S, = -;(I- I S ~ ) O * ~ (slT) I - = -L(D*' 2 - h l T - lhT). Since (e, - e,)Tl = 0, then hlT(ei ej) = 0 as well. Consequently, -
T 1 T *2 (ei - e,j) S,(ei - ej) = -; (e( - ej) D (e, - e J )
by the fact that D*' has a zero diagonal (& = 0 for any i ) . On the other hand, according to Def. 3.15, there exist a vector configuration X = [x:; x 2 . . . ; ] : x such that d:? = JIx,- x,llz. So, we ca.n w r i k d2. ag = ((XTei - X'ejiI; = ((XT(e, - e,)IJ," = (ei - e,)TXXT(e, - e3).
Since, d:j = (e,-ej)TS3(ei-ej), then X can be related to S,as S, = XXT. Note that the dimension m of X is determined by the rank of S,s. 0 Theorem 3.16 (Test 111 for Euclidean behavior) Let D be an n x n , non-zero symmetric matrix with a zero diagonml. D is Euclidean ifl has exactly one negative eigenvalue.
[
-:;'I',
Proof. This theorem and its proof follows directly from the consideratioris of on seniidefiriiteness of quadratic forms in [Chabrillac and Crouzeix, 19841. Given an n x n real symmetric matrix A, they show that requiring that zTAz 2 0 for all z such that BTz = 0 is equivalent to stating that the matrix [ has exactly r = rank negative eigenvalues. By Theorem 3.13; D is Euclidean iff D*2 is cnd, that is zT(-D*')z 2 0 for
tT t]
(a)
110
The dissimilarity representation for pattern recognition
all 0 = zT1 = l T z . By substituting A = -D*'
[ -:;'t] has exactly one negative eigenvahie.
and
I?
= 1, we get that
0
Lemma 3.2 If a n nxn symmetric matrix A is cnd, then A has at most one positive eigenvalue. Proof. A contrario. By eigendecomposition, A = Q AQT, where A = diag(Xi) is a diagonal matrix of eigenvalues in a non-decreasing order, XI 2 A 2 2 . . . 2 X, and Q = [ql q, . . . q,] is an orthogonal matrix of the corresponding eigenvectors. Suppose now that XI and Xz are both positive. Let Y c R" such that Y = span{q,, qz}. Then dim(Y) = 2, as the two eigenvectors constitute a basis. Note that y T A y > 0 for all non-zero y E Y ,since there exist a1 and 0 2 , where at least one is rion-zero, such that y = a1 q, azq, and y T A y = [a1q1+ mqJTQAQT[al q1+ azqzl = [DI CYZ 0 . . . 01 A [a1a2 0 . . . 0lT = a: A1 a; A2 > 0, since the set of eigenvec-
+
+
tors {qi}i=l.."is orthonormal. Let 2 = {z E R" : zT1 = O}. Then d i m ( 2 ) = n - 1. A is cnd if for all z E 2:zTA z .< 0. Y and 2 are subspaces of R" and there exist a I I U I I - X ~ ~ O X E Y n 2,since dirri(k') = 2 and dini(2) = n - 1. However, one has that xTA x > 0 holds, since x E Y and also xTA x 5 0 holds, since x E 2 and A is 0 cnd. Hence. a contradiction.
Remark 3.7 If an nxn symmetric matrix D with a zero diagonal is Euclidean, then, -D*2 ha.s exactly one negative eigenvalue. T h e rewe'we does riot hold. Proof. Assume that {Xi}:=.=, are the eigenvalues of -D*'. Since the trace is, on the one hand, a sum of the diagonal elements and, on the of -0"' other hand, the sum of eigenvalues, then t r (-D*2) = 0 = C:='=, Xi. It follows that -D*2 must have at least one negative eigenvalue. By Lemma 3.2 a n x n , cpd matrix has at most one negative eigenvalue. Hence, -D*2 has exactly one negative eigenvalue.
To provc that the reverse does not hold, consider a nonnegative symmetric matrix A = 3 o 1 . The cigerivalues of -A*2 are {9,0.217, -9.217}, so
[::I:
+
+
Since a13 a32 = 1 1< 3 = a12, then A cannot be Euclidean as the triangle inequality is not fulfilled. 0
-A*, ha,s exactly one negative eigenvalue.
Theorem 3.17 (Test IV for Euclidean behavior) [Crouzeix and Ferland7 i982] Let D be an n x n non-zero symmetric matrix with a zero diag-
Characterzzation of disszmilarities
111
onal. D i s Euclidean iff D*2 has one positive eigenvalue and there exists a vector z such that D*’ z = 1 and zT1 2 0. Theorem 3.18 (Constructing D from S) inite similarity matrix.
Let S be a positive serraidef-
(1) If S = ( s i j ) such that 0 5 s z j 5 1 and sii = 1, th,en, the dissimilarity 5’)”;is Euclidean. Also the matrix 0 2 = ( l l T - S ) matrix D1 = (llTis Euclidean. (2) The dissimilarity matrix D = ( d z 3 ) with, d i j = ( s i i ,?j3 - 2s,,)4 is Euclidean,.
+
If S i s not psd, then the corresponding ciissimilarity matrices shown aboii~ are not Euclidean, yet they can still be constructed. Proof. By Theorem 3.13, it is sufficient to prove that DT2 and D;2 are cnd for all z such that zT1 = 0. (1) Let zT1 = 0. Since DT2 = l l T -S . then zTDT2z= zT1lTz- zTSz = -zTSz 5 0. The latter inequality holds since S is psd, which means that zTSz 2 0 for any z. Hence, 0;’ is cnd. Consequently, D1 is Euclidean. Thanks t o 0;’ = l l T -2s S*’, we get z ~ D , *=~ -zT(2S z - S*’) z. Now. we need to require that zT(2S-S*’)z 2 0. which is equivalent t o (2S-S*2) being psd. See Gower [1986] for proof.
+
( 2 ) Let s = diag (S). Then D*’ = slT+IsT - 2s.Lct zT1 = 0. Consequently, zTD*’z = zTslTz zTlsTz- 2zTSz = 0 0 - 2zTSz< 0, since S is psd. Hence, D*’ is cnd. As a result, D is Euclidean. 0
+
+
Corollary 3.4 Let S = ( s i j ) be an n x n similarity matrix with the elements si3 satisfying 0 5 si3 5 1 and sit = 1 for i , j = I , 2,.. . , n. If the matrix D = ( l l T -S)*+is either non-metric or non-Euclidean metric, then S i s not psd. Moreover, if D = (llT-S ) is eith,er non-metric o r nm-Euclidean metric, then 2 s - S*2 i s not psd [Gower, 19861.
Theorem 3.19 (Correcting D to make it Euclidean) Let D be a nonEuclidean symmetric dissimilarity matrix and let S ( D ) = . I D J , where J = (IDenote Xmin as the smallest eigenvalue ofS(D*’) and A,
illT).
as the largest eigenvalue of the matrix
-+
where
is the zero matrix and I n x n is the identity matrix. Then D can be corrected
112
The dissimilarzty representation for pattern recognitzon
I&K
I5
d
K
J
Figure 3.4 Any non-Euclidean distances can be corrected to become Euclidean. Let D be the dissimilarity matrix from Fig. 3.3. Then the matrix = [ D * 2 + 2(llT-1)]*3 ~ is Euclidean for T 2 0.33 and the matrix D f ) = D IE. (llT I ) is Euclidean for n 2 0.3124; see Theorem 3.19. The plots present Euclidean embeddings of the corrected distances. Note that by using 7 = 0.33 and K = 0.3124, 2-dimensional representations are obtained; see plots (a) and (b), while for T = IE. = 0.5 the number of dimension increases; see plots (c) and (d).
+
~
such that the matrices D:') and DL') are Euclidean5 [Gower, 19861':
(1) D.I-" = [D*2+ 2 r ( 1 1 T - I ) ] * b . (2) D L 2 ) = D + ~ ( l l T - I ) , K ,
r
2 -Amin,
> > ~ ~ ~ .
Proof. (1) Assume that D is a non-Euclidean symmetric dissimilarity matrix. We will use Remark 3.5 to prove tjhat is Euclidean.
-+
illT)
02'
Let S, = (ID*2(I- ;1lT)and h = i D * 2 1 Then we find t h a t S , = -$, (D*z - h l T- lhT).Note that 1
& 1 lTD*'l.
1
.
diag (S,) = - - [diag (0"') 2
~
-
diag (hlT)- diag (lhT)]= ~-[0 -h -h] = h. 2
5There are mistakes (misprints?) in the formulation of this theorem in [Gower, 19861.
113
Characterization of dissimilarities
Therefore, D*2 can be expressed as
D*2 = h l T + l h T- 2 S, = diag ( S c )IT -t 1diag (S,)T
-
2 S,
+
Let S, = ( s Z j ) , then d:j = sii s j j - 2 s,] holds for all i ,j = 1, . . . , ri. From the latter equation follows that adding a constant ‘T to the diagonal of S, is equivalent to adding 27 to the off-diagonal elements of D*2. An eigendecomposition of Sc is given as S, = Q A&’, where A = diag ( A i ) is a diagonal matrix of the eigenvalues in a non-increasing order and Q is an orthogonal matrix of the corresponding eigenvectors. If S, is psd, then all eigenvalues are nonnegative, hence Amin 2 0. However, since D is nonEuclidean, then S, is not psd by Theorem 3.14. This means that there exist some negative eigenvalues, thereby Xlllirl < 0. Let ‘T 2 -Amin, where ‘ T > O if S, is not psd (note that 7 2 0 if S,: is pstl). Then A 71 is a nonnegative diagonal matrix. As a result, S, = Q [A 7 1 ] Q T i s psd. Note furthcr that S, = Q h Q T + Q r 1 Q T = S+7QQT= S, ‘TIand by the observation above, S, = S, 71 = D*2 2 ‘T (llT I). Since S, is psd, then by Remark 3.5, [D*’ 2 7 (ll -T I ) ] * $is Euclidean.
+
+
+
+
+
+
+
(2) Here, the smallest yi > 0 is sought such that D f ) = D 6 (llT - I ) is Euclidean. By Remark 3.5 this means that /c. should be chosen such that the smallest eigerivalue of S, = - a J ( D p ) ) * 2 J , where J = ( I is zero. Let q be the eigenvector of S, corresponding to the zero eigenvalue. Then S, q = 0. Since = D*2+2n0+rc2(11T-I) and J J = J , then after simple transformations, we obtain -$ ( J D * 2 J + 2 y i J D J - ~ ~g =50. ) Let p be a vector such that p = - k D * 2 J q or, equivalently, -6Jp = 2n.JDJq - K 2 J q = 0. After the diviJD*2Jq. One gets -nJp sion by 6 (remember that n > 0), -Jp + 2 J D J q = yiJq is obtained. Consequently, thanks to J J = J , one needs to solve the following sys-JD*’J ( J q ) = K, (Jp) which is equivalent t o tern of equations -Jp 2 J D J (Jq) = yi ( J q ) ,
illT).
+
+
O,,., -JD*2J -In X n 2 J D J
][ d [ ,]);?;:
value of the matrix
Jp
= yi
( : : :
. This means that
K
is thc largest eigen-
which finishes the proof.
Note that both corrections defined above yield different solutions, as illustrated in Fig. 3.4. In the first case, the correction of dissimilarities is linearly related to the corresponding matrix of inner products S, (see also
Th,e dissimilarity representation f o r pattern recognition
114
Theorem 3.15), such that S, case.
=
S,
+ T I . This does not hold in the latter
Theorem 3.20 Let ( X ,d ) be a finite metric space with, the associated dissinrilarity matrix D . Consider the assertions [Barthe'lerny and Gue'noche, 1991; Kelly, 1970; Hjort et al., 1998; Everitt and Rabe-Hesketh, 19971:
( I ) ( X ,d ) is ultrametric. (2) ( X ,d ) possesses the four-point property. (3) ( X ,d ) is Pa-embeddable. (4) ( X ,d ) is (1 -embeddable. (5) ( X , d ) is hypermetric. (6) ( X , d ) is of negative type. (7) ( X ,d i ) is (2-embeddable. The follo.uiin,y implications: ( 1 ) +
(3)
+ (4) + (5) + (6) + (7)hold.
Proof. (1) + (2) Ultranietric space is realized by an ultrametric tree, which is additive by Def. 3.9. Hence, the four point property is fulfilled. (1) + (3) The proof that every finite ultrametric space of n distinct points can be isometrically embedded into an (n- 1)-dimensional Euclidean space can be found in [Leniin, 19851 and the effective construction in [Fiedler, 19981.
(2)
+ (4) See Theorem 3.6 and also [Barthdemy and Guknoche, 19911.
(3) + (4) See [Bretagnolle et al., 1966; Critchley and Fichet, 19971 for proofs.
(4) + (5) Let X = {zI,z2, , z n } . Based on Theorem 3.5, it suffices to show that every cut metric, Def. 3.4, satisfies the hypermetric inequalities, Def. 3.8. Let V and V" = X\V define the cut. Then the cut metric &,(x7,zj) equals 1 if J V n { z i , z j } J = 1 and 0, otherwise. Let y E Zn such that yT1= 1. Then Ci C, yi yi & ( x i , x j ) = 2 CJgV yi y j = 2 ( C 7 , E VY * ; ) ( C j @ VY j ) = 2 ( C Z E V Y i ) ( l - CiEV3%) I 0. The latter inequality holds, since CzEv yi is an integer and, therefore, either both CzEV yi and (1 yi) have opposite signs or one of them is zero. Hence, the cut metric is hypermetric.
xiEv
xzEV
+
(5) (6) Let D be hypermetric. Then by Def. 3.8, y T D y 5 0 for y E 2" such that yT1= 1. Let x E R n be any vector. Then z = (I - lyT)X E W . Moreover, it is easy to check that zT1 = 0. Hence, by Def. 3.8, D is of negative type.
Characterization of dissimilaraties
115
(6) + (7) There is an equivalence of d being of negative type and D*2 being cnd. This implication is true based on Theorem 3.13. 0 Many dissiniilarity measures are constructed by combining the measure applied to all the attributes separately. Given m features (attributes), the dissimilarity can be expressed in the form of d(z,y) = CyLl f ( x l r y Y 7 ) . where f ( z J , z J=) 0 a nd f ( z , , yj) = f ( y J , x J 2 ) 0 for all ,j. Then we have:
Corollary 3.5 Let f i s metric in R.
2,
~ E B P .Then d ( z , y ) = Cy=lf ( z j ,y j ) is m,etr.%ci#
Proof. + Since f is nonnegative, symmetric and f (u, u)= 0 for 71 ER. then the axioms of reflexivity, symmetry and definiteness are fulfilled. Since d is metric, then d(.x. y) d(y, z ) 2 d ( z .z ) for all z, y, z. Consider z’. y. z such that xj = c,, yl = cy, zl = c, for all J and some constants c,,cy and ri. The triangle inequality for d reduces to f ( x C cyC) , + f(yc, z,) 2 j”(rr> zC)> hence f is metric. +=== Trivial. 0
+
EL,
Theorem 3.21 Let f : X x Y Rt be a function. d ( z , y) = f ( x f .y r ) is metric zflp(~:,y)= [ f ( ~ ~ , y i ) ] is ~ )metric + [Gower, 14861. --f
(cF=i
Remark 3.8 Direct product spaces allow us for a construction o,f a ne’ii) space b y combining two (or more) spaces; see also See. c3.3.2. Given a number of square dissimilarity matrices, a new dissimilariky matrix can be creded b y applying an elernmt-wise operator, szLch as sum or m,ax, t o them. For instance, D = D1 + Dz. I n the light of Sec. 3.3.2, this m,eans that finite generalized metric spaces are combined into a new one. The spaces are assumed to be defined on th,e same finite set X , yet they ure distinguished by the dissimilarities measures used. Now,the consequences from Th,eorems 3.10 and 3.11 and the mathematical induction are: The ma2 and sum operators preserve the metric properties. A square dissimilarity matrix, resulting from the summation of dissirnilaritg matrices preserves the t ,-embeddability, hypermetric a d negatiue type properties. If DT2 and Dg2 are o,f negative type, then the matrix (DT2+ D;’)*k is t2-embeddable. This follows from the preservation of the negative type property by summation and Theorem 3.20(7).
116
3.4.2
The dissimilarity representation for p a t t e r n recognition
Square distances and inner products
Assume a configuration X of n vectors {XI,x2,.. . , x,~}in a Euclidean space. d2(x,,xj)= (xi - xj,xi- xj) holds thanks to the definition of R Euclidean distance and an inner product. Therefore, &(Xi, X,j)= (Xi, Xi)
+ (Xj,Xj)
-
+ d2(Xj,0)
= d2(Xz:0)
2(Xi,Xj) -
2 (Xi,Xj),
(3.2)
where 0 is the origin in this space. Conscquently, 1
( X i ,Xj) = -2 (&Xi,
Xj) - &(Xi,
0) ~
d2(Xj;0)).
(3.3)
Based on the well known properties of inner products arid the above formula, the square distance &(xi, X) of xi to the mean sf of the configuration X can be expressed by the square distances as follows: d 2 ( X i , X ) = I/Xi
= (Xi-
~
= d 2. %.
x:xi
-
X) = (Xi,Xi) + (x,x) - 2 (Xi,%)
- -d2 1
(3.4)
2"'
where, abusing somewhat the notation, d;, stands for the mean computed over the 1:-th row of the matrix D*2 and d2, is the overall mean [Torgerson, 1967; Goldfarb; 19851. Let 11s assume, without loss of generality, that the mean vector coincides with the origin, i.e. X = 0. This implies that d2(xi,0)= d2(xz,%). By conibiiiirig this with Eqs. (3.3) and (3.4), one gets
1 rr s = l "'
d2(Xi:Xj) ~
- Cd2(Xz,Xs)
(3.5) n
s=l
p,s=l
Characterization of dissimilarities
117
valid for all i , j = 1 , 2 . . . .,n. Let X E R n x k be a representation of all vectors (x: is the i-th row of X ) and let G be the matrix of inner products, i.e. G = X X T . such that gtJ = (x,,xJ).Eq. (3.5) simplifies to
(3.6) Let D*2 be an n x n square Euclidean distance matrix. By substituting:
1 dz, = - 1D*21T
n2
and after straightforward calculations, G becomes
Alternatively, by Eq. ( 3 . 2 ) ,D*2 can be defined by the Gram matrix G as D*'
= glT+lgT-
2G,
(3.8)
where g is a vector of the diagonal elements of G, i.e. g = diag(G), or g = (G * 1)1,where * is the Hadamard product. In this way, an explicit linear relation between the Gram matrix G and the matrix of square Eiiclidean distances 0*' is expressed. The assumption on a zero mean of the configuration X is not essential, since the configuration can be shifted such that the origin coincides with any other vector lying in a convex hull of X . This means that instead of ST = XTkl = 0. one requires that XTs = 0 with sT1 = 1. As a result, J from Eq. (3.7) becomes J = I - 1sT arid in a bottom-up way. we have reached Theorems 3.14 and 3.15, and Remark 3.6. The same reasoning holds for a pseudo-Euclidean space R(F',q) , because of the linear relation between square pseudo-Euclidean distances and the corresponding indefinite inner products. Therefore, Eq. ( 3 . 5 ) is valid for a pseudo-Euclidean space, where an indefinite inner product . ) E , defined by Eq. (2.1), is used instead of (., .), and a pseudo-Euclidean distance Eq. (2.2) is used instead of the Euclidean distance. As a consequence, the matrix of indefinite inner products G becomes G = X [ - 7 ~] XT. This leads to the conclusion that Eqs. ( 3 . 7 ) and (3.8) remain true for a pseudo-Euclidean space, as well. Further discussion follows in Sec. 3.5.3. ( 3 ,
118
3.5
The dissimilarity representation f o r pattern recognition
Linear embeddings of dissimilarities
Dissimilarity data can be embedded into a Euclidean space in a number of' ways. Since we are interested in a faithful configuration, an embedding is found such that the distances are preserved as well as possible. Linear embeddings are considered, the isometric ones first and then their approximate variants. Since it is not always possible to isometrically embed the data in a Euclidean space, a pseudo-Euclidean space will be used as well. From such a perspective, any finite prenietric space can be isometrically embedded in a pseudo-Euclidean space.
3.5.1
Euclidean embedding
Consider a set R of n objects, R = { p l , p 2 , . . . , p n } . These objects may not be yet represented for the use of computer algorithms, therefore, you think of R as of' an index set. X , on the contrary, is a representation of the objects froni R in a Euclidean (pseudo-Euclidean) space R k . Hence, it is a set of vectors {XI,x2,. . . , xn} in this space. Given a Euclidean pairwise distance matrix D = D ( R ,R ) E I R n x n between the objects of R, a distance preserving linear mapping into a Euclidean space can be found. Such a projcction is known as classical scaling (CS) [Young and Householder, 1938; Cox and Cox, 1995; Borg and Groenen, 19971. The dimension k , k 5 n and a configuration X E R n x k have to be determined such that the (squared) Euclidean distances are preserved. Note that when one configuration is found. any other can be created by a rotation or a translation, as the Euclidean distance is translation- and rotation- invariant. Without loss of generality, the projection is constructed such that the origin coincides with the mean vector o f t h e configuration X . To determine X , the relation between the Euclidean distances and inner products are used. We know from Sec. 3.4.2 that D*' = g l T l g T 2 G , where G is the Gram matrix of the underlying configuration X, G = XX', arid g = diag (G). G can also be expressed as G = JD*'J, where J is the centering matrix J = I - ~ l l T E R n X nJ . projects6 the data such that the final configuration has a zero mean vector. Then the factorization of G by its eigendecomposition is found as
+
~
-;
G=QAQ~,
(3.9)
6A more general projection can be achieved imposing that a weighted mean of X becomes zero; see also Sec. 3.4.2. Then J = I - 1sT, where s is such that sT1 = 1 and G = - J D*' JT. By choosing a proper s , any arbitrary vector of X can be projected at the origin, as well.
Characterrzation of dassamilai~ties
119
where R is a diagonal matrix with the diagonal consisting of nonncgative eigenvalues (G is positive definite by Theorem 3.14) ranked in descending order arid followed by the zero values. and Q is an orthogonal matrix of the corresponding cigenvectors; see Theorem 3.15. G is a niatrix of inner products, so G = XXT. As a result. given k , k 5 n,non-zero cigerivalucs. a k-dimensional representation X is determined as
x = Q~ A;,
(3.10)
where Q k E W L Xisk the rnatrixlof k leading eigerivectors (i.c. corresponding to k largest eigenvalues) and A$ € p S k x k contains the square roots of the coiresponding eigenvalues. This is the result of classical scaling. Note that X determined in this way is unique up to rotation (the centroid is now fixed). since for any orthogonal matrix U , XXT = X U UTX = ( X U )(XU)T. Also the features of X are uncorrelated. since the columns of Q k are orthonormal. The estimated covariance matrix of X becomes then
Sincc C is a diagonal matrix, the vector configuration X is equivalent to the Principal Component Analysis7 (PCA) result as shown below [Fukunaga, 1990; Duda e t al., 2001]. Moreover, the eigcnvalues of G play a key role here as they linearly scale the resulting features (basis eigenvectors) and, therefore, they decide which of them are significant and which not. Note t,liat an uncorrelated vector representation X is obtaincd only for J = (Iwhich corresponds t o s = i l in Theorem 3.14 for J = ( I - IsT). Only then the vector mean of X is set to the origin. This justifies why this particular s is in favor. Note also that G is a reproducing kernel for the Euclidean space R k ;see also Sec. 2.6.1. This means that the erribeddirig procedure can be directly performed on a positive serriidefinite matrix G, hence a kernel, treated as a similarity matrix.
illT),
Proposition 3.1 (On the equivalence between the PCA and CS results) Assume u vector configumtion Y in a Euclidean space Rk and a Euclidean distance m a t r i x D ( Y , Y ) . Let t h e m e a n vector of Z lie a t the origin. T h e n t h e classical scaling resudt, Eq. (3.10), i s equivalent to t h e PCA projection based o n th,e estimated covariance m a t r i x of 2 , c o i i ( 2 ) =
12%. n-1 7More details on PCA can be found in Appendix D.4.3.
120
The dissimilarity representation for pattern recognition
o Adding2.r
0.12 0.1 ** ~
o.08-o * 0.061, f;
i
O 04/. 0.02
1
Figure 3.5 Eigenvalues resulting from the embedding of a 100x100 modified Hausdorff dissimilarity representation D of the NIST-38 digit data (described in Appendix E.2) and the corrected representations Dzr and D,. The eigenvalues are sorted according to their magnitudes, so the order of eigenvalues coming from different embedding is different. Remember that 7 =,I,A,I, where, , ,A is the smallest eigenvalue. Adding 27 t o the off-diagonal elements of D*2 is equivalent t o adding 7 t o all the original eigenvalues (hence the smallest one becomes now zero). Eigenvectors remain the same. The relation between original eigenvalues and eigenvalues of D , is nonlinear.
Proof. We start from an nxn distance matrix D and the (Gram) matrix G= JD"2.J. Let (Az, 9,) be an eigen-pair of G. Then YYTq, = A,q, and further by the multiplication by &YT, one obtains &(YTY) (YTq,) = "-(YTq,), n- 1 which is further equivalent to cov (Y) (YTq,) = &(YTq,). Drfine qrCA = YTq,/&, z = 1 , 2 . . . . , n. It is straightforward to check that the vectors qfCA4 are orthonormal. This means that qrcn)is an cigen-pair of cov(Y). The solution of the PCA projection, xrCAis given as x, = Y q y A = YYTq,/&. which is equivalent to x, = Aq,.In the
-+
(5,
1
matrix iiotation. X = Q k A f , which is the classical scaling result. 3.5.2
0
Correction of non-Euclidean dissimilarities
The Gram matrix, G = -$ JD*2J , is positive (semi)definite (pd or psd) if the dissimilarity matrix D E EXnxn is Euclidean; see Theorem 3.14 and Remark 3.5. Therefore, a non-Euclidean dissimilarity matrix D gives rise to an indefinite Gram matrix (which is not psd). Since G has negative eigenvalues, then D cannot be isometrically embedded into a Euclidean space. Algebraically, the Euclidean representation X cannot be constructed by Eq. (3.10) as it relies on square roots of the eigenvalues. However, D can be corrected such that the corresponding G becomes psd. Possible approaches to address this issue are:
Characterization
0
0
of
dissimilarities
121
Only p positive eigenvalues are taken into a y o u n t , resulting in a pdimensional Euclidean configuration X = Q , A;, p < k . Since the actual dissimilarities D are nonnegative, the magnitude of the smallest negative eigenvalue of G is smaller than the largest positive one. Also the slim of the positive eigenvalues is larger than the sum of magnitudes of the riega,tive ones. Hence, after neglecting the negative contributions, the resulting Euclidean distances are overestimated. This might be a justified approach if the negative eigenvalues are relatively small with respect to the positive ones; see Sec. 3.5.6, where the issue of noise influence is discussed. We argue that the distances which are directly measured may be noisy and, therefore, not perfectly Euclidean. This will result in small negative eigenvalues of G. Therefore, noise is diminished, when they are disregarded. By Theorem 3.19, there exists a positive constant T 2 - A m i n , where Amin is the smallest (negative) eigenvalue of G, such that the distance matrix Dz, = [D*2+ 2 7 (llT - I ) ] * $ is Euclidean. This means that the corresponding Gram matrix G, is pd. The eigenvectors of G and G, are identical, but the value 7 is added to the non-zero eigenvalues, giving rise to a corrected diagonal eigenvalue matrix A, = A k TI. This is equivalent t o ‘regularizing’ the covariance matrix of our final configuration X by C = (A, 71) and changing X , respectively. Note that the original dissimilarities are distorted significantly if T is a large value. See also Fig. 3.5. There exists a positive constant 6 2, , , ,A max, where, , A max is defined in Theorem 3.19, such that D , = D K (llT - I ) is Euclidean. After the correction of D , the corresponding Gram matrix G, yields eigenvalues and eigenvectors which are different than those of G. See also Fig. 3.5. By Theorem 3.9, there exists a parameter p for a function g defined as in Def. 3.12 such that the matrix D, = ( g ( d i , j ; p ) )is Euclidean. In practice, p is determined only by trial-and-error, although p < 1, in general. In principle, an indication of the value of p is given by (1- T ) , where T is the ratio of the absolute value of the smallest negative eigenvalue to the largest positive one. An algorithm to determine p is also proposed in [Courrieu, 20021.
+
+
0
0
+
These approaches transform the original dissimilarity data such that a Euclidean configurat’ioncan be found. This is especially useful when the negative eigenvalues are relatively small in magnitude, which suggests that the original distance measure is nearly Euclidean. In such cases, the negative eigenvalues can be interpreted as noise contributions. If the negative
122
The dasszmalaraty representataon f o r p a t t e r n recognataon
eigeiivalues are relatively large (in magnitude), then by neglecting them, important information might be disregarded [Laiib and Muller, 2004; Pqkalska rt nl., 2002bI. There is still an open question whether it is beneficial for learning to transform the dissiniilarity data D to exhibit the Euclidean behavior, either by neglecting the negative eigenvalues or by directly enlarging D*2 by a constant. In general, a hollow dissimilarity matrix D, Def. 2.45, can be corrected to have the Euclidean behavior. First, to make it definite, any zero dissimilarity between two different objects should change into a small fixed value E , depending on the overall distances, e.g. 0.01. Alternatively, if the zero dissiniilarity is obtained for different but nearly identical objects of the same class only, one may consider them as belonging to the same equivalerice class with the equivalence relation z y iff d ( z , y ) = 0; see also Example 2.7. Next, to make the dissimilarity symmetric, an operation like avcraging of di,i and d j 2 or taking their niaxirnurri value can be performed. Sirice D has become quasirnetric, any of the corrections described above will make it Euclidean. It is also possible that the corrections applied are less than required for guaranteeing the Euclidean behavior (i.e. by adding a constant to the offdiagonal elements of D*2 smaller than the necessary one). In such cases, the measure is simply rnade ‘more’ Euclidean (hence, also ‘more’ metric), since the infliicncc of negative eigenvaliies will become smaller after the discussed transformations; hence it may become negligible.
-
3.5.3
Pseudo-Euclidean embedding
When a Euclidean space is not ‘largc enough’ to cnibed the dissimilarity data (due to negative eigenvalues of the corresponding Gram matrix), Goldfarb proposed to embed D into a pseudo-Euclidean space [Goldfarb. 1984, 19851. Such a procedure can be applied to any premetric finite dissimilarity representation. A pseudo-Euclidean space is a direct orthogonal ciecorriposition of two vector spaces IWP and Rq,for which the indefinite inner product is positive definite on IWP and negative definite on IWq; see See. 2.7 for details. To find this embedding, the same reasoning as in the Euclidt.an case is applied. The essential differencc rcfers to the notion of an inner product arid a pseudo-Euclidean distance. Now G = - JD*’J is the Gram matrix, but cxpressed as (3.12)
Characterization of dissimilarities
123
where JPq is the fundamental symmetry matrix in a pseudo-Euclidean space IR(P.q). The eigendecomposition of G becomes then (compare also to Eq. (3.9)):
where p f q = k . A is a diagonal matrix of p positive and q negative eigenvalues, presented in the following order: first the positive eigenvalues with decreasing values, then the negative ones with decreasing magnitudes followed by zeros. Note that this is a proper eigendecornpositiori in a pseudoEuclidean space. Remark 3.9 (Eigendecomposition of the Gram matrix) In a pseudo-Euclidean .space, t h e G r a m m a t r i x G i s positive de,finite in t h e Euclidean sense, but it i s not J-positive defin,ite. However, GJPpcl is J-pd. T h e eigendecomposition of G& c a n be therefore performed in a psewdoEuclidean space. T h i s gives GJpq = Q h Q * , which leads t o t h e traditional eigendecomposition of G as G = Q(AJPq)QT,,where Q i s th,e m a t r i x of eigenualues of G and (hJpq) is th,e diagonal m a t r i x o,f corresponding eigenvalues; see A p p e n d i x B.2 f o r details. The configuration X can be therefore determined in a pseudo-Euclidean space IW' = I[R(p>q)of the signature ( p , 4 ) as (3.14) where only k non-zero eigenvalues in A h are taken into account. Q k is the matrix of k leading eigenvectors. Otherwise. additional zero eigenvalucs would describe X in a degenerate indefinite inner product space as explained in Sec. 2.7.) Additional explanation on eigendecomposition in a pseudo-Euclidean space can be found in Appendix B.2. The estimated pseudo-Euclidean covariance matrix is defined by (see Proposition B.3) (3.15)
Consequently, X is an uncorrelated representation. Although C is not positive definite in the Euclidean sense, it is positive definite in the pseudoEuclidean sense, i.e. J-pd by Def. 2.82. This means that X is can be interpreted in the light of a PCA projection. hence the whole embedding procedure becomes a non-degenerate indefinite-kernel-PCA approach [Schiilkopf
124
The dissimilarity representation for p a t t e r n recognition
et al., 1999b; Schiilkopf, 19971, where the kernel G is a reproducing kernel of the pseudo-Euclidean space R(P,q); see also Sec. 2.7.1. Similarly to the Euclidean space, such an uncorrelated vector representation is obtained only for J = ( I or in other words, for s = il, when the following Gram matrix G = -iJ,D*’J; with J , = ( I - I s T ) is considered. Corriputing the square distance in a pseudo-Euclidean space R(P,y)can be realized by computing the square Euclidean distance in a ‘positive’ space R P and subtracting tlie square Euclidean distance found in a ‘negative’ space Rq. The distances derived in the ‘positive’ space only are overestimated, therefore, the purpose of the ‘negative’ space is to correct them, i.e. makc them non-Euclidean. Since in this work, pseudo-Euclidean spaces will result from an embedding of nonnegative dissimilarities, the ‘negative’ contributions to the overall distances are smaller than the ‘positive’ contributions.Hence, by the construction of the configuration X , the projected vectors of X take smaller (in magnitude) values in Rq than in RP.Practice confirms that many summation-based measures arc close to the Euclidean distance, giving risc to relatively small negative eigenvalues in the embedding. On the contrary, the dissimilarity measures based on minimum or rnsxirnum operations, such as variants of Hausdorff distance, Sec. 5.5, are vcry different and give rise to large (in magnitude) negative eigenvalues. Note that the proposed embedding is very general. Any symmetric dissimilarity matrix can be embedded in a pseudo-Euclidean space. And any asymmetric dissirnilarity matrix D can easily be made symmetric
illT),
3.5.4
Generalized average variance
Assiinic a set of vectors {xl,xz.. . . , xn} in a pseudo-Euclidean space E = R(”>‘J). p + y = k , determined in the pseudo-Euclidean embedding of D . Let t h y he stored as row vectors in the matrix X . The generalized average variance of the configuration X is the trace oftlie covariance matrix of X , i.c. tlie s l i m of variances, as
Remember that 1 ) . I I E = ( . , . ) E . Since X reflects the same geometry as imposed by the square distancc matrix D * 2 ( R , R ) based , on the set R = ( ~ 1 . ~ 2 ., .., p n } , it is possible to express V d ( X ) only in terms of such distances. Formally. one has:
Characterization o.f d i s s i m i l a r i t i e s
125
Corollary 3.6 Given the dissimilarity matrix: D = D(R, R ) , the generalized average variance of the embedded pseudo-Euclidean corifigurution X is equivalent to the average square dissimilarity: (3.17)
Proof. We will now show the equivalence between Eq. (3.17) and Eq. (3.16). Making use of Eq. (3.8) and the facts that lT1= rI and lTg = tr(G) = Cy=,llx,ll$,one gets 1 rL Vd(T)= 7 2n
=
1 C C d2(pj,;pl,) = 1TD*21 j=1 k=l
2n2
1
1
1
27L2
2n,
2n
- [lTglT1+ lT1gT1 - 2 lTG 11 = - ITg + -gTI
-
I rL2
- lTG 1
as lTG 1 = lTX Jpq XT1 = ETJpq X = jI%j,:I for Jpq being the fundamental symmetry in the space €. The above reasoning remains valid for an Euclidean distance matrix D as 0 one uses JPq = I .
3.5.5
Projecting new vectors t o an embedded space
Let X E I t n x k , k = p + q , be a configuration in a pscudo-Euclidean space R'" = R(p,q) that preserves all pairwise distances expressed by D ( R .R). Remember that the Euclidean case is included for y = 0. Given a matrix D, E R t X Tof L the dissimilarities between t new objects arid tlie objrcts of R. new objects can be projected to an cnibcdded space. Let X, be tlie new configuration to be determined. First, the cross-Gram matrix G,, relating all new objects to the objects from R should be found.
Corollary 3.7 Let D ( R ,R ) be isometrically em,bedded into ( X .R'")), where R'"= R(P,q)). Let D,,be a dissimilarity matrix between t 'new objects and the objects of R. The cross-Gram matrix G, sf indefinite inner products is given as G, = - i [ D i 2 J - U D * 2 J ] ,wh,ere J = ( I E I W " " ~ and u = 111TERtxa. t
illT)
T h e dissimilarity representation for p a t t e r n recognition
126
Proof. Assume that {yl, y 2 , .. . ,yt} is a vector representation of new objects projected into the space Rk. It follows from Eq. (3.3) that the inner product between a new vector and the original vectors is given by ( y z , x 3 )=~ -i[d;(y,,x,) -d:(y,,O) -d2(x3,0)]. Making use of Eq. (3.4) and the fact that the mean coincides with the origin, the indefinite inner product becomes then
(3.18) 1 (di(Y,, X j ) 2
= --
-
(di,i.
-
d2j
+ d2. ),
where ( d i ) i , stands for the mean computed on the i-th row of the dissimilarity matrix D i 2 , d2j stands for the mean computed over the j - t h row of the matrix D*2 and d2, is the overall mean. Let G,, €iRtXnbe the matrix of inner products between t new vectors arid n original ones. Using elementary matrix operat,ions, Eq. (3.18) can be rewritten as: G, = -$(D:2 J - U D * 2 J ) ,where J is the centering matrix arid u = 1 1 T ~ ~ f ~ 7 ~ . 0
+
As a result, the cross-Gram matrix G,, is G, = - ; ( D i 2 J-U D * 2 J ) . On the other hand, G, is the matrix of indefinite inner products and, thereby, it can be expressed as:
Gn = X,, JPqXT,
41q =
{[
I ERkXk, I,,, o
-IqxjE I W " ~ ,
if Rk is Euclidean. if Rk is pseudo-Euclidean
(3.19) Therefore, X,, is determined as the solution of an indefinite least square problem X , J p q XT= G,, i.e. X , = G, X ( X T X ) - l J p q ; see Theorem 2.28, Remark 2.14 and Corollary 2.11 for justification. Knowing that XTX = ( A ( and X = Q k IRI,14, X,, is alternatively expressed as
xn= G,
X 1Al-l
Jpq
or
x, = Gn Q k
1A1,l-t
Jpq.
(3.20)
Assuming that U D*2J is pre-computed, the computational complexity of determining the cross-Gram values of G, for a single object is O ( n ) . Since X (Alp' Jpq can be pre-computed as well, then O ( n k ) operations are required for a projection of a new object.
Characterization of dissimilarzties
127
Dissimilarities D(R, Ri
Vector representations R(2.1)
Figure 3.6 By adding one object to the set R, the dimension k of the vector represcntation of D ( R , R ) in a pseudo-Euclidean space might increase by more than one. The points of R = { I , J , K , L } lie on a line, but the point M does not (two upper plots). The embedding of D ( R , R ) reveals a 1-dimensional configuration (the leftmost bottom plot). After enlarging R by M , a pseudo-Euclidean configuration in R(2x1) is obtained (the rightmost bottom plot, where the z-axis describes the ‘negative’ contribution), increasing the dimension by 2. The circles in the rightmost plot correspond t o the points I L projected into a plane, parallel to the xy-plane, on which iZ.1 lies. ~
3.5.6
Reduction of dimension
By enlarging the set R by one object, one vector is added to a finite pseudo-Euclidean space, but the dimension k of the vector representation resulting from the enlarged D might increase by more than one, contrary to the Euclidean case; see Fig. 3.6 for an illustration. This happens if the pseudo-Euclidean space does not yet fully reflect the variability in the dissimilarity data. In practice, when new vectors are added, they are projected into the space determined by the starting configuration X . Note also that both outliers and noise can significantly contribute to the resultirlg dimension k . Tlierefore: the reliability of X , i.e. whether D ( R jR ) is sufficiently well sampled, plays an essential role in the process of representing new data, and consequently, the performance of learning algorithms applied later on. Originally, the pseudo-Euclidean configuration X is found such that the distances are preserved exactly and the dimension of X is determined by the number of non-zero eigenvalues of G. However? there might be many relatively small non-zero eigenvalues as compared to the large ones. Knowing
128
The dissimilarity representation for p a t t e r n recognition
that dissimilarities are noisy measurements, the small eigenvalues, as they scale the features of X , reflect non-significant directions in this vectorial representation. Therefore, neglecting small eigenvalues stands for noise reduction (see Fig. 3.7 for an illustration) or for determining a representation with the intrinsic dimension. In both cases, the distances will be preserved only approzirnately. One has, however, a control over the dimension of the reduced vector representation. Basically, the dimension reduction can be achieved by an $orthogonal projection, governed by the indefinite PCA. Recall that X = Q k l A k / + is constructed such that it is an uncorrelated vector representation, i.c. the covariance matrix C = --&& is a diagonal matrix. This guarantees that X has already the form of the 3-orthogonal PCA projection; see Eq. (3.15). It means that the reduction of dimension is performed in a simple way by neglecting directions corresponding to eigenvalues, which are small in magnitude. The reduced representation is then determined by p’ dominant positive eigenvalues and q’ dominant (in magnitude) negative eigenvalues8. Therefore, X ( m )E R”’”, rn < k , is found as X ( m ) = Qm lAmli, where m = p ’ f q ’ and A,, is a diagonal matrix of first, decreasing positive eigenvalues and then increasing negative eigenvalues, m d Qm is the matrix of the corresponding eigenvectors. How to choose p’ and q’ is an important point. In practice, the number of significant dimensions can be determined by detecting the position when the eigenvalue curve, (interpolating the eigenvalues decreasing in magnitude), flattens. Some theoretical analysis of eigenvalue spectra of covariance and Gram matrices can be found in [Hoylc and Rattray, 2003a, 2004a,b].
3.5.7
Reduction of complexity
Reduction of dimension described above is useful for data representation, since both noise and non-significant information are neglected. Still, the reduced configuration in EXm = R(P’>q’)is determined by all n objects. However, for the definition of an m-dimensional pseudo-Euclidean space only m 1 objects are in principle necessary: one to define the origin and m objects to be the basis vectors. Given a reduced configuration X ( n L )with respect to the principal axes, the question arises how to choose a reduced
+
8Remember that X is an uncorrelated representation only if the origin coincides with the mean vector of X , i.e. obtained by using the centering matrix J = ( I - I s T ) with s = i 1 in Eq. ( 3 . 7 ) :see also Theorem 3.14. If some other s is used, then the indefinite PCA should be performed in a pseudo-Euclidean space. See also Sec. 3.5.8
Characterization of dissimilarities
W.R)
Theoretical data R
-
129
O
O
,
D(R,R)+noise
o,
-5
D(R+noise, R+noise)
5
5 0
1
+ +
0
-5
-9b A: -54283,1462,506,467 0 5
-1
3
Figure 3.7 Noise influence on the eigenvalues of G. The leftmost upper plot presents a theoretical banana data of 160 points, for which the Euclidean distance matrix D has been computed. The rightmost upper plot shows the embedding of D into a 2D space (note that the retrieved configuration is exact up to a rotation). The leftmost bottom plot presents the projection into the first 2 dimensions of the 159D data obtained via embedding of the distorted distances D, (where & j = d i j ~ i for j i # j and ~ i j N ( 0 , 1)) (taking care that min+j l ~ i j l< mini#j dij, i.e. no negative distances arise), which become non-Euclidean. The average distortion is 0.8, while the average Euclidean distance is 7.57. The rightmost bottom plot presents the projection into the first 2 dimensions of the 4D data obtained via embedding of D ( k ) , where R consists of the theoretical data R to which 2 noisy features were added giving rise to the average distortion of 0.9. Note that the first 2 largest eigenvalues, as presented in the plots, arc relatively the same for the non-distorted as well as distorted data, which practically gives the same results in all the cases. Therefore, by neglecting relatively small eigenvalues, noise is diminished.
+
+.
set Rrrd of T = m + l (or more) objects such that the projection defined by R r e d gives a good approximation of the configuration X ( T ( l . To avoid an intractable search over all possible subsets, an error nieasiire bctween the reduced and approximated configurations can he defined and then rninimized, e.g. in a greedy approach. Such criteria are proposed and analyzed in an experimental study in Sec. 9.3. 3.5.8
A general embedding
An uncorrelated vector representation X (with a diagonal covariance matrix) is obtained by an embedding of D which relies on the eigendeconipo-
130
T h e dassamilarity representation f o r p a t t e r n recognition
-;
illT).
sition of G = JD*2J for J = ( I Another vector representation X,? can be obtained from the Gram matrix G = ( I - I s ~ ) D * ~ (sIl-T ) , wherc sT1 = 1, following the steps as described in Sec. 3.5.3. In general, the determined representation X , will not be uncorrelated. In such cases, finding the principal components may be helpful.
-+
Proposition 3.2 (Pseudo-Euclidean PCA) Let R(p14) be a pseudoEuclidean space. A s s u m e that a configuration of n vectors is shifled s3uch th,at the X lies at the origin. Hence the pseudo-Euclidean covarG see also Proposition 13.3. BY Propoance ni,atriz is C E = &xTxzPq; sition, B.2, the eigendecomposition CE = QRQ*. A J-orthogonal m,atrix Q and the diagonal matrix of eigenvalues A can be ,found as suggested in Remark B.l. Let P be a projection t o a subspace R(p'.g'), p' 5 p a n d q' 5 q spanned by the dominant k = p' + q' eigenvectors, correspondi n g t o the f i r s t p' largest positive eigenvalues followed by the q' largest an magnitude negative eigenvalues. Let t h e m be described by a matrix Q k = [qlq 2 . .. qP,+,,]. By Proposition B.1,such a projection i s defined as P = Qk(Q~Jp,Qk)-'QzJP, = Q k J p ~ q / Q ~ J p qas , the vectors in Q k are orthonormal, Q I J p q Q k = J p ~ q ~ T. h e resulting matrix of principal components Y (rernem,ber th,at the transposed vectors are placed n o w in th,e rows of the matrices) is Y = X & , Q & I ~ , Q ~ . 3.5.9
Spherical embeddings
Nonlinear projection methods can rely on the geodesic distance, i.e. the shortest path between two points on a manifold; see e.g. [Tenenbaum et al., 2000; Lee et al., 2000. 20021. Euclidean distance is the geodesic distance on a hyperplane. Since there exists a natural connection between the spherical geodesic distance and a linear Euclidean embedding, also spherical embeddings are bricfly considered.
Definition 3.16 (Spherical distance) Let SF c IWTn+' be an rrdirrierisional spherical space, such that ~ ~ ~ = ~r 2 . 'Wex assume f that the renter of the sphere lies at the origin. The spherical distance d, between two vectors x,y E Sy is d , ( ~ y, ) = r arccos ( 5 x ' y ) . This is the geodesic distance on the sphere, which coincides with the angle between the vectors. Given an nxn dissimilarity matrix D and a positive r , a question arises whether there exist points { X I , x 2 >... , x n } on a sphere such that the
SF
Characterization of dissimilarzties
131
spherical distance d,5(xi,xj) = d i j . This can bc t>ransforniedinto the problem of embedding a suitable distance matrix in a Euclidean space:
Theorem 3.22 (Schoenberg) L e t D be a n n x n dissimila D can be embedded into u spherical space SF for a posikiiie r ifl di,i 5 r r f o r all i, j = 1 , 2 , . . . n and t h e m a t r i x G = (cos( is p s d [Schoenberg, 19371. T h e n the smallest nz s u c h that D embeds into a: spherical space Sp i s m = r a n k ( G ) - 1. T h e solution i s undefiried if mnk( G ) = 1.
F))
Proof. Although the proof can be found in [Schocnberg, 19371,we present it here since it has an educational value. A problem of tlie ernbedding of D into the spherical space Sp can be transformed into thc problem of embedding of some transformed matrix into a Euclidean space. The requirement of d,, 5 r T is obvious, since no distance on the sphere with the radius T can exceed 7r T . Suppose that {XI , x2,. . . , xrL} lie on a sphere SF. Let, xo be t,he center of Sp.Then all xi form an n-simplex in RT1,+lwhose edges have the lengthsg: poi = = r arid ptJ = = 2 T (sin $) for i , j == 1,.. . , n. Consequently, we have to prove that the distance niatrix D , = ( p i 3 ) is Euclidean, or equivalently, that D , ernbeds into a Euclidean space. Based on Theorems 3.14 and 3.15, D, is Eiiclideari if S = - L 2 (I-IsT) D i 2 (I-slT) for sT1 1 is psd. Since in our case, x g shoidd become the origin (i.e. the center of the sphere), then we choose s = el. As D;2el = r 2 ( 1- el), then after straightforward basic transformations we get S = r 2 ( ( 1- e l ) l T- ;Di2),. Given that xo = 0, it is sufficient to consider a matrix G which is tlie matrix S without the first row and the first column. Then G = !r2(llT-2 sin2(dzj/(2r))= r2 (cos(d7,J/r)). Thc condition that S is psd imposes that G should be psd, which firiislics the proof. 0 ~
Consequently, we have also proved the following:
Theorem 3.23 An, n x n , dissimilarity matrix D embeds into a spherical space iff D, = ( p i j ) , i , j = 0 , 1 , . . . , n,, with pol = fr a n d p i j = 2 r (sin(dz3/(2r)),i = 1,.. . n is Euclidean. Note that the embedded points are found by applying Theorem 3.15 (with s = e l ) to D,. Corollary 3.8 Spherical distances D e z a and Laurent [1997].
(SF,d,)
are !l-embeddable; see also
'From geometry pz", = p&+pi3 -2 paz pol cos ( $ d z 3 )which , after basic transformation gives ptJ = 2 sin($dZ3).
132
The dissimilarity representation f o r pattern recognition
Figure 3.8
Illustration of the spherical distance
Sketch of proof. Based on Theorem 3.5, it is sufficient to show the existence of a nonnegative measure space such that d, is the measure of the symmctric difference. Let S y be an m-dimensional sphere with the radius r = 1. Define tlie measure p on S;" as the fraction of rn-dimensional wol(A) hypervolunie such that p ( A ) = v .10 E [O, 11 for A 2 S;". Let A be a collection of subsets of ST. Consequently, (S;",A,p) is a probability space. Consider further a hemisphere H;"(x) = {y E ST : d,(x.y) 5 $ } centered at 2 . Then the measure of volume symmetric difference between 1
arccos(XTY)
2.rr vol(ST) = two hemispheres is p ( H ; " ( x )AH;"(y)) = 2 $ d,(x,y). On the pictorial illustration, Fig. 3.8, p (N;"(x) A H y ( y ) )is the volume of the shaded regions. Based on Theorem 3.5, the space (S;",d,) is Y1-embeddable.
3.6
Spatial representation of dissimilarities
Assunie that we have a set of raw data examples R, e.g. typed words, shapes. digitized voice cxcerpts, and a dissimilarity measure provided by an exptrt. Since the computation of dissimilarities is usually costly, in practical applications, R is a rclatively small sct of objects c.g. chosen from a larger set T . Given tlie dissimilarity data D ( R ,R), a spatial representation of D is a configuration of points representing the objects in a space. Usually. it Euclidean (pseudo-Euclidean) space is considered or, alternatively, R" equipped with an t p metric. Spatial representations are in fact approximate ernbeddings into suitable low-dimensional vector spaces, which should reflect the dissimilarity relations between the objects. Hence, they are often used as a visualization tool. Such spatial representations are visually appealing and often enhance interpretability of the relations in the data. The configurations are believed to reflect significant characteristics,
Characterazation of dissimilarities
133
as well as ‘hidden structures’ of the data. Therefore, objects judged to he similar result in points being close to each other in such a space. Thc larger the dissimilarity between two objects, the furt,her apart they should be in the resulting map of points. More generally, spatial representations are interpreted as (possibly reduced in complexity) feature-space configurations of the overall dissimilarity structure in the data. They are often used in clustering, classification or data-mining techniques. From the previous section we already know that (approximate) linear embeddings of the dissimilarities are methods for obtaining spatial representations. In this section, we will discuss two more techniques, namely a linear projection FastMap [Faloutsos and Lin, 19951 and nonlinear multidimensional scaling (MDS); see e.g. [Kruskal and Wish. 1978; Cox and Cox, 1995; Borg and Groenen, 19971. It is important to emphasize that the resulting axes in such spatial maps are, in themselves, meaningless. What is important is the relative positions of the points, representing objects. In the case of a Euclidean space, additionally, the orientation of the projection is arbitrary, since any rotation of the configuration does not change the distances (the same is valid for pseudo-Euclidean spaces, however, in terms of an appropriate rot,ation, represented by a J-orthogonal matrix, defined there). See Fig. 3.10 for an illustration of the basic spatial models on a theoretical banana data. A number of other techniques exists for obtaining spatial representat,ions. These will be briefly introduced in Ckiapter 7 )in which also practical aspects of spatial models are considered. 3.6.1
FastMap
FastMap was introduced in [Faloutsos and Lin, 19951 in the Data Mining conirniinity and it is originally meant for vectorial data accompanied by a distance measure. Assume that a set R = { P I , . . . , p r L } and an n x n
134
The d i s s i m i l a r i t y representation for pattern recognition
Euclidean distance matrix D ( R , R ) are given. Then there exists an mdinierisional Euclidean space, rn 5 n such that the distances are preserved perfectly; see Sec. 3.5.1. The idea is to project the data on m mutually orthogonal directions. This is realized in an incremented way, starting from the first dimension. The basic principle is to orthogonally project p i into a line in IwTlL determined by two pivot objects, T I and 7'2. Pivot objects should bc the ones which yield the 1a.rgest distance. The projection xi of the object pi into this line can be determined from the cosine law of the Euclidean geometry (as illustrated in Fig. 3.9) as:
(3.21) Since thc objects will lie in a Euclidean space RnL1 the projection method is extended as follows [Faloutsos and Lin, 19951. Let H be an ( m- 1)dimensional hyperplane perpendicular to the line defined by T I and r2 (or the remaining EXmp1 space). Then after mapping all objects on this hyperplane (or in fact updating the distances appropriately), the problem to be solved is identical to the original one, but with the dimension ( m - l ) , instead. Hence, the solution can be found recursively using Eq. (3.21) to determine the coordinates of the dimension of interest. Since the square Euclidean distances are additive, the distances of the objects projected into the hyperplane H becorne then [Faloutsos and Lin, 19951:
In t,he next step, D = D H , defining the same problem, but for the space . Although the dimension m should he specified beforehand, the
~ 7 r i1
algorithm may stop when the distances d H become practically zero. New points can be added to the existing map in the same recursive manner, based on the distances to the pivot objects. The algorithm requires the computation of 2m distances, so the complexity is C?(m). Note that the cosine law captures the same relation as the one given by Eq. (3.2). The Euclidean embedding realized by classical scaling, as described in Sec. 3.5.1, makes also use of the cosine law, however, the projection is optimized for all the triplets (defining Euclidean triangles) simultaneously, instead of in an incremental way as FastMap does. Note that for non-Euclidean dissimilarities the derived configuration approxirnat,cs the original distances, since the cosine law (which is the founda,tion of FastMap) is valid for the Euclidean distances only. In the mapping process, at some point the distances d H of the objects projected into the
Characterization of dissimilarities
L... 1 1 _ 1
Theoretical data
,
-I
Classical Scaling --I
,
135
. Fast Map
'- +
Sammon map S-1
Sammon map 5'1
Figure 3.10 Spatial 2D maps of 200 x 200 dissimilarity matrix D for theoretical banana data (leftmost subplot). D is defined by the city block distance. Since D is nonEuclidean, Theorem 3.20, the 2D maps are only approximate embeddings. The scale is preserved in all subplots.
hyperplane H may become negative, which indicates that H exists in a pseudo-Euclidean space. Yet, the projection is always done to a Euclidcan space. Formally. if the dissimilarity data D can be embedded into the R(piq) space, then thc dimension m used in FastMap should be such that m 5 p , since FastMap preserves, in fact, the Euclidean distances corresponding to the embedding into BP.In summary, FastMap is less optimal than classical scaling with respect to the preservation of distances. However, it is fast, as an incremental mapping, which has a possibility to an early stopping. 3.6.2
Multidimensional scaling
Multidimensional scaling (MDS) refers to a group of' linear and nonlinear projection methods of the dissimilarities. Although the theory of MDS was developed in behavioral and social sciences [Kruskal and Wish, 1978; Cox and Cox. 1995; Borg and Groenen, 19971, its applications were extended to pattern recognition and other related fields. The reason is that the MDS methods facilitate data visualization and exploration. These projection techniques aim to preserve all pairwise, symmetric dissimilarities between data objects, resulting in a low-dimensional representation of the geornetrical relations between the points. Such a configuration is usually found in a Euclidean space. although any other 8, space, p 2 1, can also be considered [Cox and Cox, 1995; Borg and Groenen, 19971. The MDS output is a spatial
136
The dissimilarity representation for pattern recognition
representation of the data. Most of the concepts presented here as well as the discussion on the MDS algorithms can be found in the books of [Borg and Groenen, 19971 and [Cox and Cox, 19951. The latter book provides a good, concise introduction into the subject, while the former book is a thorough compendium. Our work is concerned with Sammon mapping a,nd it relies on [Pekalska et ul., 1998a,c,b, 19991. Metric MDS is a description of methods which assume that both the input data and the output configuration are metric, or rather that the dissimilarities are described by quantitative values. Suppose an n x n dissimilarity matrix is given. The aim is to find a possibly low-dimensional space such that the discrepancy between the original dissimilarities arid the est,ima.teddist,a,ncesis minimized. Intuitively, each pairwise distance corresponds to a ‘spring’ between two anchors (points) in this low-dimensional space. Then the MDS technique tries to rearrange the points such that the overall ‘stress’ of a fully connected spring system is minimized. The dissimilarities can describe the relations between objects represented originally in a high-dimensional space, measured (e.g. matching costs of image patterns, road distances) or given (human judgments). When the observed or measured dissimilarities convey qualitative instead of quantitative information, they give rise to non-metric MDS methods. In essence, they are solved in a similar way as metric MDS methods with the exception that the nature of dissimilarities is different, such as preferences or ranks [Borg and Groenen, 1997; Cox and Cox, 1995; Kruskal, 19771. These methods are not discussed here. Sirice now on, MDS will stand for metric MDS. There are different ways of preserving the structure of the data, giving rise to somewhat different techniques of MDS. Traditional classical scaling (CS) is the most simple, linear MDS algorithm. It has been already int,roduced in Sec. 3.5, where the embeddings of pseudo-Euclidean distances are discussed. Also that FastMap can be considered as a linear MDS example.
Nonlinear MDS. Nonlinear MDS projections rely on the minimization of an appropriate nonlinear function. This is a loss function. called stress (acronym for standard residual sum of squares), which measures the diflerelice between the Euclidean distances (or P,-distances) of the present configuration of n points in RnLand the actual (given) dissimilarities. Hence, the problem of finding the right spatial configuration resolves itself into an optimization problem. where a Configuration yielding the minimum of the strcss is sought. Here, for convenience, we will adopt the notation used
Characterization of dissimilarities
137
in MDS. Let A be the actual nxn dissimilarity matrix and let D he the estimated distance matrix for the projected configuration. We will write d,, ( X ) to indicate that the distances are computed for a retrieved configuration X . The most elementary MDS loss function is the raw stress, defined as [Kruskal, 1964; Borg and Groenen, 19971:
(3.23) It yields a square badness-of-fit measure for the entire representation. .f is a continuous parametric monotonic function, a transformation applicd to the given dissimilarities 6 i j . In many cases, f is the identity function, but it may be some other function such as polynomial or logarithmic. Usually, = f ( S i j ) , called disparity, is adopted. Although the raw the notation of &, stress is used in practice e.g. [Borg and Groenen, 19971, in our opinion, it is not an informative function to be rnininiized iteratively as it reflects the absolute error. The differences between actual and estimated dissiniilarities should rather be expressed in relative terms to avoid that large absolute differences contribute significantly to the error function, while small differences do not. Large differences do not necessarily indicate a bad approximation. Therefore, the stress should be normalized in a way that avoids a scale dependency. This leads to a least squa’res scaling (LSS) loss function [Cox and Cox, 1995; Kruskal and Wish, 1978; Kruskal, 19771: n,-1
TL
where fwrj are appropriately chosen weights. For instance, the weights can be used to shift the emphasis to small dissimilarities by choosing w,,~= l/&j for non-zero 6,. Concerning disparities, straightforward choices arc c.g. a linear or logarithmic function, i.e. & j = Q BSij or & j = Q log(&j), where Q and /3 are estimated in the least square sense by modeling the perfect relation d i j ( X ) = S,,. The normalization by estimated distances makes the error measure invariant under rigid transformations, like shifts and rotations, arid non-rigid transformations, like uniform stretching or shrinking, of the derived configuration. The stress is optimal when all original disparities 8i.j are equal to the estimated distances d r J ( X ) . Since this is unlikely to happen, d i j ( X ) will be a distorted representation of the relations within the data. The larger the stress, the greater the distortion. The optimization procedure for the
+
+
138
T h e dissimilarity representation f o r p a t t e r n recognition
LSS is an iterative process of two alternating stages: fitting & j to d i j for a present configuration X (hence dij are considered as fixed for that moment) and minimization of the stress function, i.e. updating X , given 6 i j . As from an application point of view, one is interested in the relative positions of objects in the spatial map, a general suggestion in the MDS area is to consider a ratio MDS [Borg and Groenen, 19971, where & j = pbij for > 0. It means that the ratio of two disparities should be equal to the corresponding dissimilarities: & j / i k ~ = bij/bkl. For the SLss stress, the optimal pFSs can be derived analytically as the one minimizing SLss, provided that D ( X ) is fixed. By setting up the derivative of SLss over /3 to zero, its optimal value is found as pcSs= Cj Cdi: j ( X ) / CjCi &jdij(X). Alternating the computation of pcSswith an iterative improvement to the stress provides an efficient procedure for finding the solution to the ratio MDS . Most of the minimization algorithms are based on gradient methods [Kruskal arid Wish, 1978; Kruskal, 1977; Borg and Groenen, 19971, but also other techniques have been especially adopted for tlie MDS purposes, such as iterative majorization [Cox and Cox, 1995; Borg and Groenen, 19971. In our experience [Pckalska et al., 1998a1, this algorithm has a slow convergence. An interesting modification for vectorial representations is studied by Webb [Webb, 1995, 19971. He looks for a nonlinear transformation in tlie reduced space Rm in which the approximated &distances are close to the actual !,-distances in terms of the weighted raw stress. The transformation is defined by radial basis functions, hence the iterative majorization technique determines its parameters. This results in a mapping that is applicable to new data. Another way to normalize the raw stress is to use the original dissimilarities instead of the approximated ones. This lea,ds to loss functions being variants of the Sammon mapping. Sammon mapping. The original Sarnmon mapping was proposed in pattern recognition by Sammon [Sammon Jr., 19691 as a tool for a nonlinear projection from a high-dimensional Euclidean space to a low-dimensional space. To our knowledge, it is not mentioned in books and articles devoted to the MDS research. However, it may be considered as a method in this area, if interpreted as a projection technique which tries to preserve the original dissimilarities. For the sake of simplicity, we will account the variants of Sammon mappings as the MDS examples in this book. Samnion mapping is a nonlinear projection realized by the minimization of the
Characterization of dissimilarities
139
following loss function:
In general, the stress can be defined in a number of ways, e.g. as studied by us in [Pekalska et al., 1998a,c,b]:
for t = . . , . -2, - l , O , 1 , 2 , . . ., which results in the following measures for the identity function f,i.e. 6,, = f ( & ) = dt3:
S-I(X)
=
S(X) n-1
n
We will refer to all of them as (variants of) Samrnon mappings. Each of’ the loss functions mentioned above emphasizes a different aspect of the geometric relations between points, i.e. it emphasizes. to some extent, either smaller or larger distances, which directly influences either local or global aspect of the method. For instance, S-2 emphasizes very small distances. i.e. it penalizes the error in representing small dissimilarities more than the same error for large ones. Therefore, S-2 focuses on local details, hence it is very nonlinear. On the other hand, 5’2 emphasizes larger distances, hence it tends to present more global map of relations. SO provides a balance between large and small distances, i.e. errors in representing s r d l and large dissiniilarities are penalized equally. Depending on the application requirements, the loss function can be chosen appropriately.
140
T h e dissimilarity representation for p a t t e r n recognition
By applying the ratio approach to the Sammon stresses, i.e. the discrepancy 6 i j = P&j, one gets
for t = . . . , -2, - l , O , 1,2 , . . .. Note also that scaling of 6 i j by [j is equivalent to scaling of d i j ( X ) by which is further equivalent to scaling of‘ X by i.e. St(X,p) = S t ( i X , l ) . The optimal p* can be determined as the point yielding minimum of St for the present configuration X (hence also D ) . By setting the first derivative of S t ( X , P ) with respect to ,3 to zero, after straightforward calculations, one obtains [j* = (CIl<, . . 6t. ’Il d : j ( X ) ) , / ( C j <6jT11dij(X)). i After simplifications, inserting y* into St(X,p) yields
i>
i,
for t = . . . , -2, - 1 , 0 , 1 , 2 , . . .. Note that 0 5 S t ( X , p * )5 1 by the nonnegativity of dissimilarities and the Schwartz inequality. Theorem 2.16, since
c,,, 6:~z+1(6:ydzIl(x))5 (C,<, 4, (C,,, s:Ild;3(X))+. In order to compare the Sammon stress functions to the LSS loss funct+2
1
1 2
tions Eq. (3.24). let us introduce the variants of the SLss,similarly to the variants of Sammon mapping. So, we can introduce a general LSS loss function as:
for t = . . . . - 2 . - 1 . 0 , 1 , 2 . . . .. Then by considering the ratio MDS, 6,, = BS,,, one can express the optimal p (minimizing S:ss(X,p)) as prSs = 6,)df,fl(X))(CIl<, 6:Jdt3 ( X ) ) . The substitution of prSs into SkSs gives then
for t = . . . . - 2 , - 1 . 0 , l . 2 . . . .. If X * is the optimal configuration (corresponding to a local minimum) of the Sammon error St, then the LSS stress S:ss(X*,,3;ss) is equal to the Sammon stress S t ( X * . p * )for t = 0. This does riot hold for other t , although for t < 0 , the Sammon stress St at the
Characterization of disszmzlarities
Classical scaling ~ _ _ ~
141
Sammon map So __
~
Figure 3. 1 2D spatial maps of the Euclidean distance representation of 400 points uniformly distributed in a 10-dimensional space. The scale is preserved.
local riiiiiirnuiii of X * would be smaller than the corresponding 5 ' ;' and thc other way around for t > 0 . This can be directly deduced from tlie formulations of Eq. (3.32) and Eq. (3.301, taking into account that the MDS distances d,, ( X ) underestimate the actual dissimilarities, which leads to the following inequalities CJ<% S,, > C,<,d,, and C3<7 > 1. Note that except for the raw stress, both SOL' and So are the loss functions traditionally applied in MDS. Practically, they give the same (up to scaling and rotation) results. In general, due to the normalization by tlie actual dissimilarities, St will emphasize smaller dissimilarities than Skss foi t < 0 and the other way around for t > 0. This is not a problem, since when needed, additional weights can be used. This means that the Sarnmon stress functions can be generalized in order to incorporate the nonnegative weights of individual pairs as
2
Usually, the weights are chosen to be either 0 or 1, where 0 is used to accomniodate for missing values (here: dissimilarities). However, t,he weights can also be set to J- or $- for non-zero S i j . For instance, in the latter 67 1
case, the weighted stress
1J
Sg becomes the unweighted stress S-2.
Implementation. Sirice the optimization of the Saiiinioii stress fimctioris is easier defined in the gradient terms, we give preference to Saniniori mappings. To find a Sammon representation, one starts from t,lie initial configuration of points X (e.g. randomly chosen or from the classical scaling result) in Rm for which all the pairwise distances are computed. Next, the points are adjustled so t,hat the stress decreases. In an iterative process, the configuration is improved by shifting around all points to approximate
The dissimilarity representation for pattern recognition
142
better arid better the model relation & j = d i j for i , j = 1 , 2 , . . . , n, iinti1 a (local) minimum of the stress is reached. In such a procedure, a steepest descent, Newton-Raphson algorithm [Press et al., 19921, iterative majorization [Borg and Groenen, 1997; Heiser, 19911, conjugate gradients [Shcwchuk, 19941 or scaled conjugate gradients (SCG) [Mder, 19931 can be used to search for the minimum. In our experiments with artificial and real data [Pckalska et al., 1998a,c,b],we found out that concerning the convergence rate, the scaled conjugate gradients and Newton-R.aphson techniques are preferable. The former technique is characterized by large improvements in the first iterations, and a slow convergence to a minimum later on. Therefore, a hybrid algorithm can be considered, which switches to the Newton-Raphson minimization after the first few iterations. The found minimum depends on tlie initialization. Usually, the output of classical scaling is a, good starting point, since it is the global minimizer of the raw stress (in a linear way). However, it is useful to compare its result to the Samniori output obtained from a random initialization, since the optimization algorithm may get stuck in a local minimum close to the initial configuration. A 'better' initial configuration, i.e. a scaled version of the classical scaling result X C S i.e. t* X C X was suggested in [Trosset and Mathar, 2000; Malone et al., 20021. It is, however, not new, since their t* equals Sijdij(X)/ d $ ( X ) , which is equivalent to in the ratio MDS with the stress Skss (and also SO).A good initialization is still an open problem. From our experience is follows that Sammon mappings are less sensitive to the starting configuration than the LSS mappings. In summary, it is important to emphasize that the MDS techniques hased on tlie minimization of' the normalized square differences will produce maps, where projected points will tend to be enclosed in circular or ellipsoidal shapes. This can be clearly observed for a Euclidean distance matrix D computed for an artificial example of 400 points unifornily distributcd in a 1OD hypercube. The MDS result can be seen in Fig. 3.11. See [Hughes and Lowe, 20031 for the formal proofs referring to the raw stress based on the square Euclidean distances. To avoid this artifact, other types of error measure can be considered, for instance in the form of
xjci
cj
&
c3<,
IS, - d 7 , ( X ) / / C 3 <6%, 7 01' E ( X ) = C3<% 167, - ~%?(X)I/6%,. The measiires bascd on absolute values are, however, difficult to optimize (due to discontinuous derivatives). Another MDS technique. which is more robust against outliers can also be designed by considering the following fit F ( X ) = rnedian,,,fr [Cox and Cox, 19951.
E ( X )=
'
6
z
J
r
61,
d
'
J
(
x
)
'
Characterization of dissimilarities
143
Two different spatial configurations can be matched by the use of Procrustes analysis. This might be useful to compare two configurations derived from optimizations of different loss functions or to indicate how a configuration changes when the similarity between objects changes over time (as e.g. human preferences of some products). Basically, the configurations are matched be determining the optimal translations, rotations and scalings; see [Borg and Groenen, 1997; Cox and Cox, 19951 for details.
3.6.3
Reduction of complexity
Given objects, a nonlinear MDS method requires the computation of O ( n 2 )distances in each iteration step arid the same memory storage. However, for a low, m-dimensional representation, only mn values should be determined. This suggests that the total number of constraints on distances is redundant, so some of them can be neglected. This leads to the idea that only distmces to a subset of all object,s are preserved, for wliicli a modified version of the MDS mapping is considered. Although X ,derived from MDS, has tlie dimension 'm, it is deterrriiricd by n > m objects. In general, a linear space can be defined by m+l linearly independent objects. If they were placed such that one lies in tlie origin arid the others lie on the axes, they would determine the space exactly. Since this is unlikely, the space retrieved will be an approximation of the original one. When more objects are used, the space is better defined. Following [Cho arid Miller. 20021, objects having relatively many close neighbors (lying in the areas of high density) can be selected for the representation set R C T of the size T > m, 011 which the (non-)linear mapping could be based. For a dissimilarity representation A ( T , T ) ,a natural way to proceed is the k centers algorithm (Ypma and Duin, 19981. It looks for k center objects. i.e. exarriples that minimize the rriaximurn of the distances over all objects to thcir nearest neighbors; see also Sec. 7.1.2. It uses a forward search strategy, starting from a raiidoni initialization. Note that the k-means algorithm [Duda et al., 20011 cannot be used since no feature rcpresent8atjiori is assumed, only the dissimilarities A. For a chosen set R, the linear mapping of A(R,R)into an m-dimensional space is defined by Eqs. (3.7) - (3.14). The remaining objects A(T\R, R ) can then be added to the map by the use of Corollary 3.7 arid Eq. (3.20). In case of the Sanimon mapping, a modified version should be defined, which generalizes to new objects. Following [Clio arid Miller, 20021, first the Sarrimon mapping of A ( R ,R) into the space Wm is performed, yielding
144
The dissimilarity representation for pattern recognition
the configuration X g . The remaining objects can be mapped to this space, while preserving the dissimilarities to the set R, i.e. A' = A(T\R, R). This can be done via an iterative minimization procedure of the modified stress Adt, using the found representation X A as
(3.34) for t = . . . , - 2 , - 1 , O , 1 , 2 , . . .. Equivalently, the modified loss functions of tlie LSS and MDS can be defined as: n
r
(3.35) Thanks to these procedures, new objects can be added to an existing map. Their complexity reduces from O ( m n 2 )computing , O ( n 2 )distances in the R" space, to O(nrnr+nr2)in each iteration step. Another possibility to define a Sanimon mapping is by the use of neural networks, as studied in [Mao and Jain, 1995; de Ridder and Duin, 19971.
3.7
Summary
This chapter presents ways of characterizing dissimilarity measures, especially for those represented as finite generalized metric spaces. The basic concern was whether a dissimilarity measure is metric or not, which can easily be checked for a finite representation. Also transformations preserving the metric properties are considered. Usually, such transformations are able to make a non-Euclidean dissimilarity measure 'more' Euclidean. An essential question, however, is whether a given distance measure is Euclidean or city block. The Euclidean distance is important because a Euclidean space is both a metric and an inner product space. Hence, there exists a natural connection between the traditional inner product and the Euclidean distance, which allows one to embed any Euclidean distance matrix in a finite-dimensional Euclidean space. Both isometric and approximate linear and nonlinear embeddings are presented here, as well as their generalizations for the projection of new examples. These are the multidimensional scaling techniques. If a measure is non-Euclidean, no isometric projection into a Euclidean space exists. Some solutions are presented, where either tlie dissirnilar-
Characterization of dissimilaritaes
145
ity is corrected such that it becomes Euclidean or it is projected into a pseudo-Euclidean space. Any premetric non-Euclidean measure (satisfying the definiteness and symmetry constraints) can be formalized in such an indefinite inner product space. This builds a general framework where any symmetric dissimilarity representation can be explained. The city block distance is important because, on the other hand, due to its additivity property. Finite generalized metric spaces can also be represented by weighted, fully connected graphs, where the weights correspond to the given dissimilarity values. A city block distance is perfectly structured by an additive tree model, where the dist,arice is understood in terms of the shortest path in this tree. Other dissimilarity measures can also be interpreted via such tree models; however, only approximately. See Chapter 7 for more discussion. I11 short, t,his chapter deals with the characterization of generalized metric spaces, especially finite spaces represented by n x n dissimilarity matrices. It introduces useful tools for checking metric or Euclidean properties and for finding the dependencies in the family of &-distances. In particular it discusses the issue of (approximate) embeddings into pseudo-Euclidean spaces which can be carried out for any symmetric dissimilarity measures. In this way, it establishes the foundation for designing learning algorithms on spatial representations in Euclidean and pseudo-Euclidean spaces, as will be seen in Chapter 4. Also the process of data exploration is supported, either by visualization of two-dimensional spatial maps or by visualization of the organization of objects and their underlying structure as given by a tree model.
This page intentionally left blank
Chapter 4
Learning approaches
The leaming and knowledge that we have, is, at the most, but little compared with that of which we are ignorant. PLAT0
Although objects may have different initial or intermediate representations, in the form of sensor measurements, shapes, relative graphs or numerical features, we will ultimately describe them by pairwise dissirnilarities. By now, the ground for learning methodologies for dissimilarity representations has been established. First of all, various spaces arid the relations between them were characterized in Chapter 2 . They prepare mathematical frameworks, in which dissimilarities are explained. Next, Chapter 3 discussed basic properties and transformations of dissimilarity matrices as representations of finite generalized metric spaces, especially in the context of metric or Euclidean distances. Finally, also isometric arid approximate embeddings into pseudo-Euclidean space were presented. General statistical learning aspects for vectorial representa.tions are briefly summarized in Sec. 4.1. Statistical learning is described to establish the context in which learning froin dissimilarities will take place. Section 4.2 formally introduces dissimilarity representations and explains their role in unifying of the statistical and structural approaches to learning. Also an extension of dissimilarity representations is mentioned, based on the 'true' inductive learning, as illuminated in [Goldfarb, 1990; Goldfarb and Golubitsky, 2001; Goldfarb et a,l.; 1995,20041. Next, three main dissimilarity-based learning approaches are presented. They refer to three interpretations of such representations in some spaces, for which particular statistical learning niethodologies can be adapted. In the first approach, the dissimilarity values are interpreted directly, hence they are characterized in (pre)topoIogical spaces. The second approach focuses on dissimilarity spaces in which each dimension corresponds to a dissimilarity to a chosen object. The third approach finds a spatial representation, i.e. an embedded (pseudo-)Euclidean corifiguration such that the dissimilarities are preserved as well as possible.
147
148
The dissimilarity representation for pattern recognition
More details on these niethodologies can be found in Secs. 4.3 - 4.5. This chapter ends with additional remarks on generalized kernels as well as some insights in the connections between dissimilarity spaces and the underlying pseudo-Euclidean spaces, as given in Sec. 4.6. The purpose of this chapter is not only educational; it also presents the ideas behind the learning from dissimilarity representations and explains the basic methods. Although this material relies on our publications [Pqkalska and Duin, 2002a; Pqkalska et ul., 2002a,b, 2004a1, there are many new insights and observations presented here. Also new is the perspective from which it is discussed.
4.1
Traditional learning
Learning from examples is the process of discovering, distinguishing, detecting or describing patterns present in the data. It relies on both extraction and representation of information from the measurements collected in order to understand the process (phenomenon) that created them. The result of learning is that the knowledge already captured in mathematical terms is used to describe the present independencies such that the relations between patterns are better understood or used for generalization. The latter means that a certain concept, e.g. that of a class, is formalized such that it can be applied to unseen examples of the same domain, inducing new information, e.g. of a class label. In this process, new data objects should obey the same deduction procedure and follow the same reasoning as the original examples. Note that the word ‘pattern’ refers to both a property or a characteristic of an individual object (i.e. its structural or mathematical representation) arid a property of the entire set of objects given by their characteristics.
4.1.1
Data bias and model bias
Pattern recognition is usually concerned with the learning of a concept from a set of examples. Here, a concept is a general notion of an entity serving to designate a class of instances or another type of relations. More practically, an abstract or real set of all possible examples of the concept to be learned is a domain. For instance, if one wants to learn the concept of a Dutch tulip (in fact of a tulip class), then this domain consists of all types of tulips ever grown in the Netherlands. So a domain is a complete representation of the
Learning approaches
149
concept considered. In practical applications, domains cannot he studied in their entirety due to their complexit,y7t,he costs of the collection process, the physical limitations of both measuring and storing devices, and the measurement costs. Consequently, domains are sampled. This means that only some examples are provided to represent a domain, and, as a result, only a limited amount of data is available for learning purposes. So, data represent i n f o r m a t i o n and knowledge available f o r a particular d o m a i n . A (concept of a) class1 is represented by a finite collection of instances, but it is not yet described by this. The description of a class has to be based on description of each single instance in the measurement process, where each instance is characterized by a set measurements arid additional knowledge about the class. Measurements in general refer to the outputs of measuring tools, algorithms, or procedures, and they can be performed directly on objects or inferred from raw measurements. Raw measurements refer to the raw outputs of sensors or devices which record signals. images, hyper-spectra images etc. All such outputs can be used for a definition of relational descriptions, features or a proximity measure. On the other hand, an abstract domain might be represented by some example structures, order in the data or inference rules, provided from outside. In such cases, the standard measurement process may not play a direct role; instead, the support is given by structural representations. In those cases, it is often very difficult to define numerical features. Still, a proximity measure can usually be constructed. Data introduce a bias (‘a systematic error introduced in sampling or testing by selecting or encouraging one outcome over others’ [Webster dictionary]) of the domain we wish to learn. We have a bias with respect to the chosen representation and to the model chosen such as a learning approach. The latter is caused by a dissonance between the learning procedure imposed on the data and the validity of the assumptions. Such a model bias is related to some error measuring the discrepaiicy between the assumed and learned values, so it is related to a bias of an estimator of an ideal model; see also Sec. 4.1.2. D a t a bias refers to both domain and data description. The first one, a sampling bias, is caused by assuming that data exaiiiples are representative for the domain. Since it is often impossible to supply instances describing the whole domain variety, a finite sample gives ] A class is either a natural category, i.e. present in reality, like a class of tomatoes or mugs, or an abstract category consisting of objects or instances sharing common properties considered for the application’s need, e.g. articles on sport, human silhouettes, people with a particular disease, etc.
150
The dissimilarity representation for pattern recognition
rise to a sampling bias. The representation bias results from a selection of characteristic features, a proximity measure or a structural representation. Taking into account the efficiency of both data, representation and learning algorithms, as well as the resolution of measuring devices, it is impossible to consider an infinite set of features, an infinitesimally precise proximity measure or complex and detailed structural information. The necessary simplification or redundancy of the data representation introduces a representation bias. Data bias has important implications for the learning algorithms. It strongly contributes to the model bias; if the data examples are a poor representation of the domain, then the selected model, optimized by using the given examples, does riot describe the reality well. Data are well described if siniilar objects are close in their representations (e.g. if two similar objects are represented by two vectors which lie close together in a vector space), the so-called compactness hypothesis [Arkadiev and Braverman, 1964; Duin, 1999; Duin and Pekalska, 20011 and if two close descriptions correspond t o the objects that resemble each other, the so-called true representation. The basic principle is that the objects do not posses random descriptions; on the contrary, the neighbors of a particular object in the representation are similar to it in reality. Note that true representation implies that distinct objects lie far away (with respect to the chosen dissimilarity) in the representation. This means that the measurements contain sufficient information not only to support the resembling objects, but also to tell them apart from distinct objects. Moreover, data are well sampled if all instances in the domain are somehow described in data or, in other words, if adding new instances will not change this description significantly. Given a lot of data relevant to the problem at hand (actually with respect to a chosen model; see also footnote 4 on page 155), the learning task becomes relatively easy (in the methodological sense; the computational cost may increase), since the data bias becomes smaller. Consequently, if data are representative and well sampled, there is enough support and information in the data to model their functional dependencies, hence the model bias becomes smaller as well. Only such data will assure a good generalization of a learning algorithm. The problematic situations are those where the amount of data is small or where there are many unlabeled examples (sometimes the collection of data can be automated, while the labeling process is slow and expensive since it should be done by humans). Conventionally, data are described by features. For instance, the class (concept) of apples (domain) can be represented by features (obtained in
Learning approaches
151
the measurement process) such as weight, size and color. A feature-based representation of a concept relies on selecting n instances to represent the domain and on defining, say, rn features for the description. We can think of vertical and horizontal samplings. where these samplings coincide with the choice of objects and features, respectively. Such data are often expressed as an nxm matrix A, where A is interpreted as a configuration of n points (feature vectors) in an rn-dimensional feature space R"'. usually Euclidean. This representation is mainly used in statistical pattern recognition [Fukunaga. 1990; Duda et al., 20011, where it is assiinied that the distribution of pattern classes can be derived from a represcntative set of such points ( a training sct) with sufficient accuracy. This often requires (strict) additional assumptions on the distribution characteristics.
4.1.2
Statistical learning
Statistical learning is usually understood as the process of determining an unknown dependency between inputs and outputs given a limited nuinher of observations. i.e. training examples. A probabilistic framework is often considered for this task, as it is mathematically appealing for haridling w certainty. Input vectors J: E X , usually X = RTrL> are assumed to be drawii independently from a fixed but unknown, probability density function ~ ( 3 ; ) . The functional dependency between outputs y E Y and iriput,s .7: is g''lven as a fixed conditional density p(yl~:),which is also unknown. Depending on the domain of Y , different learning problems can be presented. If Y is discrete, then it is a classification problem. wliilc if it is continuous, it is a regression problem. If no Y is present, then the learning problem becomes a density estimation. The training exaniples T,, = { ( J : ~yi) , :i = 1 are considered to be iid (independent and identically distributed) according to the joint probability density2 p ( x , y ) = p ( x ) p ( y l z ) . The difficulty relies on determining the relationship between X and Y , based on T, only. For a more elaborate introduction to statistical learning, see tlic books [Hastie et al., 2001; Cherkassky and Mulicr, 1998; Vapnik, 1995. 19981 and [Scholkopf and Smola, 20021. We are often interested in prediction and, therefore, in modeling the conditional probability of observing a particular y given a specific 2 . By 'All these assumptions, although general, are in fact strong. They actually assume a fixed (stationary) distribution from which the examples arc sampled. This is often violated in practical applications, e.g. when the data are collected in various conditions or even by differently calibrated sensors.
152
The dissimilarity representatton for p a t t e r n recognition
Bayes formula, one can write p(ylz) = p ( z ~ $ ( y ) . Assuming that the quantity p(y111:)can be computed, the most appealing approach for assigning an output to a new II: is the value of y which yields the maximum a posterior probability p(y1z); see Appendix D . l . This is known as a theoretical optimal Bayes rule. In practice, since the true distributions are unknown, the Bayes optimal rule cannot be found. One, therefore, tries to estimate this ideal by a function g(z) coming from a general hypothesis space of functions G = ( 9 : X + Y’}, where Y’ is e.g. R1,(0, l} or {-13I}, depending on the task. The goal of learning is then formulated as a selection of g* E G which best approximates the outputs y, given a finite set of examples. To measure the discrepancy (hence define the best fit) betwecn the estimated outcome g(z) and the original output y for a given z,a loss function L : Y x Y ’ + [0,MI is needed. A single output L ( g ( z ) y), , however, is not very informative over a particular function g. Rather tlie overall expected loss should be used to infer about g. This is the true loss of the hypothesis g, given by the error or rosk functional as:
which minimizes Ideally, the learning is a process of estimating g* E the error E ( g ) . This requires the integration over the complete probability distribution of all possible inputs II: and outputs y. Since p ( x ,y) is unknown and the only information is a set of available training examples T,, the learning problem is ill-posed. To make the learning task feasible, one usually considers a specified class of functions { g a } ( e g . polynomials), where a are parameters indexing the functions (e.g. polynomial degrees). Then go* E G miiiirriizes the error € ( g o ) . Note that the true, optimal Bayes solution y, does not necessarily belong to { g a } . To tackle such a learning problem, an empirical error (or risk) is minimized. It is expressed as:
For a given finite training set T,, there might be infinitely many functions minimizing the empirical error, since they need to behave identically only for the training examples. Therefore, by the selection of a class of functions {gcy},i.e. narrowing the scope of interest, the learning task is better formulated. Note, however, that this is purely a choice made to be able to tackle the learning problem, unless some other prior knowledge exists.
Learning approaches
153
Depending on the loss function, basic learning problems such as classification, regression, density estimation and clustering can be set up in this statistical framework. Since knowing p ( z , y ) would allow one to solvc any learning problem expressed by the minimization of the risk, the density estimation is the most general (hence most difficult) problem. Later, we will focus on predictive learning such as classification and regression.
Classification. A general multi-class classification problem can be decomposed into a number of two-class problems [Fukunaga, 19901. Hence, a two-class problem is considered as the basic one. Assume a set of training examples { (xi,~ i ) } r = with ~ , the corresponding labels yi E (0, 1} (sornetimes also yi E {-1,l)). The hypothesis space becomes then a set of indicator functions. The most common loss function is then L(y,g,(z)) = I ( y # ga(x)). The corresponding risk or the t r u e error ET = €(go) denotes the probability of misclassification (given equal costs). The empirical error, also called the training or apparent error, becomes then E A = Eemp(gn,Tn) = ~ ~ Z(yi~# ga(xi)). = 1 As a result, the learning can be simplified to finding a classzjier g&* (z) that minimizes the empirical error. Regression. Regression is based on estimating a functional dependence f between inputs z and outputs y in the form of y = f ( z ) + E ; where E is such that E ( E J z=) E P ( ~ ~dy Z )= 0, i.e. rmdom noise with a zero mean. f is then seen as the expectation of the output conditional probability f ( x ) = J y p ( y l z ) dy. The risk measures the dissonance between the actual outputs and the expected predictions with the common loss function being L(y,g,(z)) = (y - go(.))’. Under the assumption of a zero mean noise and based on the fact that y - g a ( z ) = y - f(x) + f ( x ) - g a ( z ) ? the risk Eq. (4.1) can be decomposed3 as a sum of two contributions, noise variance a,nd approximation accuracy as:
s,
where en is a fixed value, since it does not depend on g a . So, learning can be Note that the latter term equals zero: since is a random noise with a zero mean. Note also that
154
T h e d i s s i m i l a r i t y representation f o r p a t t e r n recognition
now stated as determining ga* E 6 that best approximates the (unknown) ,f. Tlic empirical risk with respect to the set of functions { g a } is expressed (!It - ga(2,))2. as EccrrLp(cJa,Tn)= Note also that tlie true error & ( g a ) can be expressed as
;c:=l
where the last two contributions arc due to the squared bias and the variance. respectively. Since the classification can be considered as a special case of regression (with y being discrete), thc decomposition above is an important phenomenon.
Density estimation. The process of estimating densities is concerned with tlie input vectors T = { x % } ; = ~ only. The output represents the density itself. The loss function is usually given as L(g(1:)) = g(z) or L ( g ( z ) ) = -logg(z), yielding in the latter case, the following risk: E ( g ) = - logg(z)p(z)dz, which in a finite casc simplifies to E,,,(g,),T =
-; c;=1logYa(&).
4.1.3
I n d u c t i v e pr inciples
Predictive learning such as regression or classification consists of two steps: thc process of learning, i.e. the estimation of an (unknown) dependence between inputs and outputs, and tlie process of generalization, i.e. prediction of outcomes for newly coming examples based on the discovered concept. In practice, the first step is closely related to induction (‘inference of a generalized coriclusion from particular instances’ [Webster dictionary, site]), while the second step rcfers to deduction (‘derivation of a conclusion by reasoning’ [Webstcr dictionary, site]). In a general form, however, the deduction is much simplified; it involves only the computation of outcomes based on thc derived parameters in the learning stage. Probably, that is why such a process is called an inductive learning paradigm. The minimization of the expected risk relies on this principle. Hence, the entire problem is put in the framework of a global function estimation. Another approach is based on estimating the risk functional by using the training set at the moment, when a new example appears. This requires a reformulation of the learning problem such that additional unlabeled examples are treated in the context of the given training set. So, the dependence between the training data and test examples is estimated when required and may differ from instance to instance. This approach is called trunsductive
155
Learnang approaches
(
tnductive learning Training data
i
\
7-___
)
Transductive learning
Test Training examples’ data
A priori assumptions
\
A priori assumptions
J
Model I dependence \____
Tep ;e ,s-
___-
t
[-j-Predicted the learned concept
outputs
L-
_
_
~
Figure 4.1 Inductive (left) and transductive (right) learning paradigms. A priori assumptions are here understood in terms of the specified assumptions on a set of learning algorithms and related parameters.
inference [Vapnik, 19981. Such an inference might be applied locally (the unknown examples are related to the objects in local neighborhoods), but not necessarily. If it is applied globally, the comput)ational burden might become high (under the inductive paradigm, only one final funct,ional dependence is estimated). Examples of this approach are the cases of learning from partially labeled sets or designing linear classifiers in local neighborhoods. This inference can also be reduced to the deduction step only, like in the k-NN rule for a fixed k . A schematic illustration of the inductive and transductive learning principles is shown in Fig. 4.1 Statistical learning theory is mostly developed for inductive principles. This is somewhat surprising, since in a general context of inference from srnall sarriple size training data4, only restricted information is available. Vapnik [Vapnik, 19981 formulated the main learning principle as: ‘‘lfyou posses a restricted amount of information f o r solving some problem,, t r y to solve the problem directly and never solve a m,ore general problem, as an, intermediate step. It is possible that the available information i s suficientf o r a direct solution, but’it is i n s u f i c i e n t f o r solving a more general intermediate problem. ” Following this rule, we conclude that, not only a predictive learning problem should be approached directly (instead of e.g. estimating the probability density function first as usual parametric methods do), but, more importantly, that it should be solved only f o r the points of interest, instead of estimating a single function globally at the entire domain. This 41n the classical sense, a small sample size problem is understood as an inference from n data vectors for an estimation of M free parameters of the approximating function, wherc n / M is small, e.g. 2 or even 5 1. Vapnik defines it with respect to a class of approximating functions of the VC dimension h,, as a problem, where n / h v c is small, such as 10 [Vapnik, 19981. See Sec. 4.1.3.2 for more details.
156
T h e dissimilarity representation f o r pattern recognition
(a) Training set
(b) Test set
Figure 4.2 Overtraining. (a) A zero-error classifier. (b) This classifier is overtrained, since it yields a high error on an independent test set and its boundary is too complex for such a srnall training set.
means that the learning problem is solved ‘at the spot’. Consequently, the application of this principle naturally leads to a transductive learning. This type of learning has not yet evoked sufficient interest of researches, probably due to the expected computational cost in a testing stage. Yet, it becomes one of the open issues for further research. We will focus on inductive learning methods. These provide a general prescription for handling the data vectors and the assumptions on the approximating functions in the learning process. Here, the empirical risk minimization and a few paradigms based on the Occam’s razor principle are considered within the framework of inductive principles.
4.1.3.1 Empirical risk minimization (ERM) In this paradigm, a function g&* is sought such that the empirical error, i.e. t,he training error E A = Eemp(gru,T,) L ( y z , g a ( x z )is ) minimized. The training error is a rough (biased) approxiniation of the (unknown) true error. We assume that an optirrial function g* (hence the Bayes rille) exists in S,although the class of functions { g a } E S might not contain it. In classification, the actual risk &(g*) is the minimal risk ever possible, called also the Bayes error. The Bayes error is the error of a theoretical optirrial Bayes rule, which assigns a vector z to the class yielding the highest postcrior probability p(ylx). In practice, since the true distributions are unknown, the Bayes error cannot be computed. Depending on the loss function and a set of chosen g a l the ERM can be employed in a riurriber of ways, e.g. based on the maximum likelihood estimators, Appendix D. 1, or linear regression. Usually, this principle is iiscd in the parametric methods, where a model is specified first (e.g. a normal density-based linear classifier assuming normal distributions) and
Learning approaches
157
then the parameters are estimated from the training data. This works well, provided that t,he number of training examples is large wit,h respect, to the model complexity (Vapnik's approach: related to the VC dimension, see Sec. 4.1.3.2; classical approach: related to the number of free parameters, which agrees with the Vapnik's approach for polynomial chssifiers). Such models do not have enough flexibility, hence they can result in a largc bias (see below). The difficult'y of applying the ERM for limited t,raining chta is tha.t, it does not yet guarantee a small expected risk, i.e. the true error. In the classification case it rneans that a small error on the training set docs riot imply a small error on an independent test set. The phenoinerion that y&, yields a small empirical risk, but still shows a large true error on an independent test set is called o.ciertraining or ouerfitting . A siifficiently flexible function can perfectly fit the training data, completely adapting to all the information available there, reaching a zero empirical error. As a result, this function can describe structures (due to the noise) which in fact are not present in the data; see Fig. 4.2. Hence, to avoid overtraining for fixed and srriall sample sizes, simple models are preferred to the complex ones. The problem is much more pronounced when the riiirnber of features, hence the dimension m, is very large with respect to the nuinbcr of data vectors. ASSUKIEa fixed number of objects. From a classical point of view, adding of new features may give worse results on an independent test set. This is caused by a poor estimation of the function parameters due to insufficient amount of the data vectors, and called the m r s e of dimensionality (Jain and Chandrasekaran, 1987; Jain et al., 20001. See also Fig. 4.3. A number of solutions is proposed to treat the curse of dimensionality, such as feature selection [Devijver and Kittler, 19821 or feature extraction [Duda et al., 20011 techniques. The first wies find the hest few features, while the latter construct new features functionally depending on the old ones, e.g. as their linear combination. Such procedures still might not be sufficient to guarantee a good generalization for complex functions.
The empirical risk depends on training exBias-variance dilemma. amples, hence different training sets will yield different models ,y&* ( T n ) . Consequently. the loss function is also a function of a training set. This dependency can be removed by averaging over training sets of a fixed size. Then the expected empirical risk with respect to all the training sets of cardinality n becomes E,[E,,,(g,, T T L )where ], En[.]denotes the exprctation. In the case of regression, there is a clear decomposition of the latter
T h e dissimilarity representation f o r pattern recognition
158
Number of training examples
c
d
0 Number of features m (or parameters M)
Figure 4.3 Curse of dimension.
quantity into a (squared) bias term5, measuring ‘the accuracy or a quality of the match’ of the learning algorithm to the problem [Duda et al., 20011 and a variance term, measuring ‘the precision or specificity of the match’ [Duda et al., 20011. Additionally, there is an irreducible term eo, independent from the training sets as derived from Eq. (4.3) by using a summation instead of an integral. Hence, we have that ErL[lernp(ga, Tn)]= eo +En [; (sa ( X i ) - f(4)2] +En ;[ (scu( X i ) -En [sa( X i ) ] 12] ’ This decomposition indicates that there exists a bias-variance trade-off, which is a fundamental problem while fitting a model to the data [Geman et al., 19921. The practical implication of such a trade-off is that a flexible function g a , i.e. a function which is able to model the irregularities well, will have a high variance since it will tend to fit the desired outputs well (yielding a snialler bias). Consequently, it will vary dramatically between various training sets. Conversely, an inflexible model will tend to behave similarly with respect to the training sets, yielding small variance, but its inflexibility might cause a high bias [Hand, 19971. A bias-variance decomposition becomes more complicated for the zeroone loss function in t,he classification case. Although it is possible to extend the reasoning behind the square loss to a classification problem by assuming that the interaction between the bias and the variance is multiplicative [Geman et al.; 1992; Duda et al., 20011, there is no clear interpretation. A recent unified bias-variance decomposition is proposed in [Domingos, 2000b1, of which the zero-one loss is a special case. There, the variance contribution is additive for unbiased examples (i.e. examples wrongly classified) and subtractive, otherwise. This means that the zero-one loss allows for a larger tolerance of a learning algorithm with respect to variance than in the
ci
ci
5Here, bias is understood as a bias of an estimator. If 8 is a random variable, then the estimator 8 is biased if the bias b = E[d]- 8, where E[.]is the expectation, is non-zero.
Learning approaches
$1
.
\+True
159
error / Expected risk e T
Number of training examples n
Figure 4.4
Consistency of the empirical risk regularization principle
case of'the square loss. This follows from the offset contribution to the averaged loss (i.e. empirical error) by the biased examples. This explanation is logical, since in the end, the classification problem directly focuses on a proper assignment of objects to classes and not on a proper estimation of probability functions. See [Domingos, 2000b,a] for details. Another general framework for the additive bias-variance decomposition with different loss functions is proposed in [James, 20031. Additional insights can be found in [Heskes, 1998; Hansen and Heskes, 20001.
Consistency. To assure that a small empirical error guarantees a sniall true error, a consistency between the true and empirical risks is needed. For a fixed function gal the empirical error Eq. (4.2) will converge to the true risk E(g,) Eq. (4.1) by the law of large numbers. But this is not enough, since it should hold for any gCY.Let go* minimize the true error, i.e. go* = argmingeEG€ ( g o ) and let g&* minimize the empirical risk (hence it depends on Tn). Then the consistency of the ERM principle requires that €(g&,) = limn+m Eemp(gh,, T,) = €(gCY*) holds in probability. This requires a one-sided uniform convergence of the empirical risk to the actual risk in probability [Vapnik, 19981: Theorem 4.1 (Vapnik and Chervonenkis) T h e necessary a n d SUBcient condition for t h e consistency of t h e empirical risk to the a c t i d risk: is t h e on,e-sided u n i f o r m convergence in probability:
This is illustrated in Fig. 4.4. Note that the asymptotic error might differ from the Bayes error.
160
4.1.3.2
T h e dissimilarity representation for p a t t e r n recognation
Principles based o n Occam’s razor
Statistical approaches have been developed to assure convergence (4.5). Tliese often rely on the Occam’s razor principle. Assume a learning problem and a set of functions {gol}, depending on the parameters a , analyzed to find a solution. The learning problem is now coniplex since it relies on the estimation of both: the model structure or complexity (the degree of a polynomial), called model se1ectio.n of the parameters (coefficients) in some optimization procedure. Such methods are put in paradigms more general than the ERM. It is assumed that the best prediction is achieved for a model of the right complexity, found by applying the Occam’s ruzor principle. This principle states that one should not presume more things than the required minimum; in the selection process, among otherwise equivalent models, it advocates to choose tlie simplest one. The Occam’s razor principle can be implemented in a number of ways, taking into account that there is a trade-off between the model complexity (e.g. the number of free parameters) and the model fit to the training data. The most typical examples are: structural risk minimization, regularization principle, Bayesian inference and minirriurri description length. We will focus on the first two principles.
Structural Risk Minimization (SRM). The approximating functions are ordered according to their complexity (like ordering polynomials by the degrce) such that a nested structure is formed. The complexity of functions linear in parameters is related to the number of parameters. I Kgeneral, ~ it is cstirnated by the so-called Vapnik-Chervonenkis (VC) dimension h,, [Vapnik, 19981, which describes the capacity of a set of functions { g a } EG. In case of a binary classification, h,, is equivalent to the maximal number of points N which can be separated into two classes in all 2N ways by using functions from the considered set {gCY}.It means that for each possible labeling of N points into two classes, there exists a function from { g a } which takes 1 for examples coming from one class and -1 (or 0) for examples from tlie other class. An analytic upper-bound based on the VC dimension is provided by Vapnik [Vapnik, 19981 to estimate the expected risk. Given n training points, with the probability at least 1-q, the bound below remains t,rue:
Learning approaches
161
The estimate above is used for the model selection of the optimal complexity in the following way. For n training examples, the expected risk is controlled by two quantities: the cnipirical risk, which depends on the choseii function for particular Q and the VC dimension h,,, of the considered set of functions. Therefore, in order to control h,,, the approximating functions are ordered according to their complexity such that if G k = { g o : Q E A k } , where Ak: is a set of parameters, and GI c G2 c G3 c . . ., the corresponding VC dimensions fulfill h$ 5 hzz 5 h!: 5 . . .. Tlie SRM principle chooscs the function from a subset G k for which the bound yields minimum. Note that the bound derivation is based on the worst-case scenario, since the VC dimension considers all possible labelings of an arbitrary configuration of points. The importance of this bound, however, is that it guarantees the uniform convergence of €e7rLp to the actual risk, Eq. (4.5), for a finite h,, (which is a necessary and sufficient condition) [Vapnik, 1998; Evgeniou et al., 20001 and for thc indicator functions (hence a classification problem). It might not be true for other functions [Evgeniou et al., 20001. Regularization principle. This principle assumes a flexible set of approximating functions { g o } , but the restriction in the solution result,s from an additional term capturing the complexity of the function y,. Comequently, a penalized risk is minimized:
where 4 is a rioiiriegative functional and the nonnegative A, indepertderit of the training data, controls the strength of the regularization. For X = 0: the penalized risk reduces to the empirical risk, while for a large A: a simple solution is obtained, mostly ignoring the training examples. Hence, the model estimate is described as a trade-off between fitting the data and a priori knowledge on the function’s complexity (regularization term). The q5 functional can be selected in many ways. The simplest method counts the number of free parameters in the function, while a more sophisticated method uses the l2 norm of the parameters cr or the curvature estimator of g a . See also [Girosi, 19981 for the relation between the regularization principle and the SRM. In statistical learning, data are assumed to be represented by vectors in bounded regions in a feature space, so the learned function slioiild change smoothly over the space, avoiding high oscillations. Tlie smoother the function? the lower its complexity. So, functions of lower coniplexity are preferred for finite sample sizes (the regularization t,erm is niea,nt,to penal-
162
T h e dissimilarity representation f o r pattern recognition
,,
\,Classifier's
\\
complexity
True error e-
Asymptotic errors --i Bayes error
1 '/
.classifier's complexity L
Number of training examples n
Figure 4.5
Complexity of classifiers vs. cardinality of the training set.
ize coniplex functions more). In the limit (when the nuniber of training examples grows to infinity) complex functions offer better solutions (the bias is small). In the case of classification, this phenomenon is illustrated in Fig. 4.5. Bayesian inference. This principle assumes that a model M , such as a function g a , is adequately selected to describe the problem. The vector of parameters a of the model M is assumed to be drawn from a theoretical parameter distribution, which means that a is considered as a random variable. A prior distribution over this unknown a should be then specified to capture the beliefs about the problem before seeing the data. For instance, one may require that a is smooth and takes reasonably small values. In practice, Gaussian prior distribution is frequently used. In the simplest case it can be chosen to be a spherical Gaussian, for which the prior distribution is p ( a ) = N(0,a21).The hyperparameter 0' can be now treated again as a random variable, or assumed fixed. The Bayesian inference is then based on the Bayes formula for updating the priors given the evidence from the data as p(M1data) = where p ( M ) = p ( a ) . p(data) is the probability of observing the data, p(data)M) is the likelihood, i.e. the probability that the data are generated by the model M and p(M1data) is the posterior probability of a model 11.1given the data. Hence, one tries to find a complete density function for the vector of parameters a . As a result, one never actually estimates or chooses any value for a ; all possible parameter values, although in different degree, play some role. A preference for simpler models is encoded by encouraging particular prior distributions. More details can be found in [Jensen; 1996; Robert, 2001; MacKay, 20031.
p(dazz
Learning approaches
163
Minimum description length (MDL). This principle is based on the information-theoretic basis and the concept of algorithmic coniplexity characterizing t,he randomness of t,he datja [MacKay, 20031. Its basic insight is that statistical learning is related to firiding regularities in the data, which can be used to compress the data, i.e. describe them by using fewer symbols than originally needed. Learning is, therefore, related to data compression [Griinwald, 20051. Roughly speaking, given a number of hypotheses (models) and a data set, the hypothesis H is chosen which compresses the data most. Models are assumed to describe tlie regularities in the data and should contain a few easily encoded parameters. N is identified as the model which minirriizes L ( H ) +L(datulH), where L ( H ) is the shortest binary code describing H and L(data1H)is the shortest binary code describing D when encoded with the help of H . For an introduction to MDL, the reader is referred to [Griinwald, 20051. Relations between the MDL arid the SRM can be found in [Vapnik, 19951. 4.1.4
Why is the statistical approach not good enough f o r learning f r o m objects?
Even after such a brief review on statistical learning theory, the reader can be convinced that. the methods developed in a proper framework guararitee good solutions. The a n s w e r is afirmative, provided t h a t t h e prohlems to be solved are oiiginally generated a s p o i n t s in, a suitable vector spacc, such as Euclidean. There i s a missing link between a collection of real osr abstract objects to be learned from and th,eir proper represen,tat%on,sto be used as a basis in a learning paradigm. In the statistical approach (except for the MDL priiiciple), the description of objects is often drarnatically reduced to points in a vector space. The analysis starts at this level. often neglecting the (one-way) correspondence between the objects and the points. Also such a simplification of an object to its riunierical description (i.e. without any structural information) precludes any inverse mapping to be able to retrieve the object itself (this is partly possible in tlie structural representation of objects, e.g. from the skeleton of an object in an irnsge, its shape can be retrieved well enough). The objects are simply treated as if they are already generated as points in the space. Note that these assumptions are very strong. Since the connection between the points and the objects is forgotten, any learning in such a framework is in fact learning with respect to the assumed distributions realized by a sample in the space R". Hence, this learning is in a purely mathematical sense. Besides the
164
The dissimilarity representation SOT pattern recognition
guarantees on providing good learning solutions, one should be concerned with the guarantees that the representation of objects by points enables to achieve that. For all these reasons, Goldfarb and his colleagues [Goldfarb, 1990; Goldfarb et al., 1995, 1992; Goldfarb and Deshpande, 1997; Goldfarb et nl., 2004; Goldfarb and Golubitsky, 20011 strongly oppose the statistical approach to learning, separated from the ‘real’ objects themselves. Real objects possess their internal structure, organization or ‘interconnectivity’, as observed in their shapes, for instance. This property can be reflected by the connectivity of neighboring samples in the sensory data such as an image. However, in the traditional feature space representation, all the continuity of an object, all the structure is lost [Chan and Goldfarb, 1992; Goldfarb, 1992; Goldfarb et al., 19951. The structure information may partially be encoded in some feature values, e.g. when features are defined as the responses of several image filters, but in the representation itself it is not available anymore. This also holds for the vectorial pixel representation of images, in which each pixel defines a separate dimension in a feature space. The complete image resides there, but the fact that some pixels are neighboring and others are remote is not expressed in the representation6. The Euclidean vector space assumes independent feature contributions and, therefore, it precludes the possibility of reflecting the structure of an image. The structure may be rediscovered to some extent by computing correlations between pixel-features from a set of images or by trying to find a low-dimensional manifold on which the set of images, represented as points, lies. Still, this is not the original structure. Moreover, the necessity of learning them in such a way is disputable if the primary structure of an image, reflected by the connectivity of neighboring pixels, is already given. Structural representations, on the other hand, are specified in terms of instance’s components and their interconnections. For a real object, it might be its structure. The structure, however, should be regular enough to be described by a relatively small number of primitives, i.e. fundamental structural element>s,such as strokes, corners or other shape elements. For instance, shapes can be represented as their skeletons, contours by their string rcpresentatioris, where each character in the string corresponds to some kind of a stroke. Also more abstract instances, such as articles in 6The fact that consecutive features correspond to neighboring pixels in an image cannot be used in the feature-based representation. Features can be permuted and, in a Euclidean space, the configuration of points (representing the images) is the same up to the rotation, while the structure of an image is completely lost by shuffling the pixels.
Learning approaches
165
a database can he represented structurally. An article might be organized in a hierarchical way, expressing the fact that it is composed of a title, an introduction, body, conclusions and references, The body might be e.g. an interview, a comment, a letter, a speech or a general writing. In turn, an interview can be made with a famous artist, actor, writer, scientist, etc. In this way, the detailed information on articles can be represented by trees. Other type of phenomena, e.g. a financial condition of a family, might he captured by graphs, expressing the relations between all the important factors, such as incomes of the family members, mortgage, loans, the number of children and their age, etc. In general, structural pattern recognition [Fu, 19821 assumes that there exists sufficient and suitably formulated knowledge to build a structural description of objects and classes. This knowledge is defined and encoded either explicitly by an expert or implicitly by a set of (training) exarnples. In order to relate new objects to the described classes, a (dis)siiiiilarity measure between objects and the structural description of objects and/or classes is needed. Like in the sta.tistica1 approach, the demands here are strong: suitable knowledge should be available t o build the structural model and an informative dissimilarity measure should be defined between the model and real-world observations. The research of pattern recognition is meant to establish a link between object representations, derived from their (sensor) measurements of objects or structural descriptions, and a learning algorithm; see also [Duin et ul., 20021 for perspectives. In the statistical learning theory, the bounds ensuring a good generalization (in classification: a small test error. given a small training error) are based on the notion of a classifier complexity, which can be related to the VC dimension. For a binary classification problem, this notion is derived for the worst configuration of n (training) points arid considering all 2n labelings of them. This is a possible scenario to coilsider if the association to the (real or abstract) objects one started from is neglected. From the pattern recognition point of view, this is unrealistic t o postulate a class of (similar) objects being described by arbitrarily labeled points. If this had been the case, one would have chosen another representation. Basically, the representation of objects is not accidental. it should be such that similar objects are close in their representations (of course, one should first define what the closeness means). This is the compactness hypothesis [Arkadiev and Braverman, 1964; Duin, 19991. Ideally, also the true representation hypothesis should hold, which is the reverse of the compactness hypothesis. It says that two close object representations
166
The dissimilarity representation ,for pattern recognition
Sensors / Raw measurements
1
Sensors Data
I Sensors /
t R ~ W measurements
Figure 4.6 Proximity as unifying statistical and structural approaches t o learning. This work focuscs on optimized dissimilarity representations.
correspond to ohjccts that resemble each other. Note that in order to solve a r e d pattern recognition problem, such a subjective hypothesis is necessary to cornplcment the available objective information. For that reason, the bounds derived by Vapnik are strongly over-pessimistic. 111 surnrnary, the answer to the question of this section is: Suitable representations of objects should be considered first before using or adapting th,e desueloped leurning meth,odology. S u c h representations should possibly ernbody a priori knowledge orn the class of objects as well as their possible structural information. There might be hybrid representations as well. Our work is concerned with an example representation based on the notion of (t1is)similarity and by this relying on the compactness hypothesis. It is pioneering, riot directly in the following methodology, but in the statement of thc learning problem and resulting adaptations. 4.2
The role of dissimilarity representations
Pattern recognition relies on the description of regularities in observations of classcs of objects. A class i s a set of similar objects (e.g. sharing similar characteristics or commonalities). This implies that the notion of ‘similarity‘ is more fundamental than of a ‘feature’ or of a ‘class’, since it is the similarity which groups objects together and, thereby, it should play a crucial role in the class constitution [Edelman and Duvdevani-Bar, 1997; Edelman et al., 1998; Edelman, 19991. Such a proximity should be possibly modeled such that a class has an efficient and compact description. In applications, however, numerical features often come before proximity is taken into account. Using the notion of proximity (instead of features) as a prirnary concept renews the area of automatic learning in one of its founda-
Learning approaches
167
tions, i.e. the representation of objects [Mottl e t al., 2001b,a; Gukrin-Dugud et al., 1999; Duin et al., 19981. Conceptually, it is a novel approach, but other researchers are conscious of the essential role that proximity plays for the class description. Examples can be found in [Bunke et al., 2001; Strehl, 2002; Mottl et al., 2001b,a; Jacobs e t al., 2000; Edelman et al., 1998; Edelman, 1999; Goldfarb, 1984, 1990; Goldfarb et al., 2004; Goldfarb and Gohibitsky, 20011. Proximity measures have the capability to capture both the statistical and structural information of' patterns and, thereby, they form a natural bridge between these approaches. Recently, also a universal distance measure was proposed in the MDL learning framework, based on Kolmogorov complexity [Bennett e t al., 1998; Li e t al., 20031. The distance, as such is impossible to compute, however, can be approximated by a normalized compression distance [Cilibrasi and V i t h y i , 2004, 20051. This is an important fact, as a n y binary files or any strings can compared by a chosen compressor. This finding extends the universality of proxirnity measures as a basic and most general way of representing information for learning, as they can be derived in all learning principles. proximity representations are then the starting point. Two main types of representations are considered: the ones which are learned and the ones which are optimized (or fixed). This work builds some foundations for the latter, called simply proximity representation,s. Learned representations remain an issue for further research. Proximity representations can be divided into relative and conmptual. In a relative representation, pairs of objects are related to each other to measure the proximity value between them. Consequently, each object is described by a set of proximities to other objects [Duin et al., 1998, 1999; Pekalska and Duin; 2002a; Pekalska e t al., 2002bl. They may be defined on a feature-based (vectorial) representation, see Fig. 4.6 by using the distances between fcature vectors, but also on the structural representation by the distances between graphs or other structural models, or dircctly on the raw data, e.g. by similarities between shapes in images. So, proximity representations arc very general as they conibirie all types of approaches. Briefly, they describe the sampled domain in a relative way, based on the pairwise comparison of objects. Remember that an object is a general notion of a real object, any entity process, phenomenon or any abstract instance, as long as one is able to compare them in a qumtitativc way. Proximity representations can be extended to depict a relation of one entity to a number of them or a relation of a model to the whole concept. Such representations are called conceptual. Examples arc a similarity of an
168
T h e dissimilarity representation f o r p a t t e r n recognition
PI
Set T = ( p l ,p2, .,p7}
P
T$:
p7
R P
P3
x
x
x
x
x
x
DFR) Representation set R = (PI ,p2,p3]
Figure 4.7 Dissimilarity representation D ( T , R ) . The representation objects are elemerits of the set T .
object to a (sampled) domain such as a resemblance of a particular niug to a class of mugs, a siniilarit,y of a language to a group of European languages, a growth and development of a child to the model development or an image query for retrieving similar images in a process of redefining the query. Also, in the statistical sense, the posterior probabilities of' an object .'c (or in fact its feature-based representation) with respect to C classes, form a similarity conceptual representation [P(n:Iclass I), . . . P(zlc1ass C)]. Conceptual representations will appear in Chapter 8, where one-class classifiers are built based on a proximity of an object to a class, and in Chapter 10 in the context of classifier combining techniques. Now, we discuss relative representations, where our main focus is on dissimilarity representations. ~
Definition 4.1 (Dissimilarity representation) Assume a collection of objects R = { p l ; p 2 , . . . , p n } , called a representation s e t , or a set of prot o t y p e s , and a dissimilarity7 measure d . The dissimilarity d is computed or derived from the objects directly, their sensor representations, string representations or other intermediate representations. To maintain generality, a notation of d ( p i , p j ) is used, instead of d(f(pi),f ( p j ) ) , where f ( p i ) corresponds to some possible intermediate representation of p i . A dissimilarity representation of an object z is a set of dissimilarities between z and the objects of R expressed as a vector D ( z ,R ) = [ d ( z ; p l ) ,d ( z , p z ) , . . . , d ( z ; p , ) ] . Consequently, for a collection of objects T , it extends to a dissimilarity matrix D ( T ,R). The idea of a representation set is that R is a relatively small set of representative objects for the domain considered. The most simple 71f d is a similarity (or a proximity) measure, the corresponding representation is called appropriately. d is expected to capture the notion of closeness between two objects, however it might be non-metric. In general, we require that d is nonnegative and obeys the reflexivity condition; see Def. 2.38.
Learning approaches
169
dissimilarity representation is then D ( R ,R), hence a square dissimilarity matrix with a zero diagonal. In general, R might be a subset of T ( R C T ) or they might be completely distinct sets. See also Fig. 4.7. Although there exists a resemblance between dissimilarity and featurebased representations in their matrix notation, the meaning is completely different; see Fig. 4.8 for an illustration. Dissimilarity representations will be used in (statistical) learning. An important question refers to the characteristics of informative dissimilarity measures. For instance, for a robust real-world object description, a measure should incorporate the necessary invariance, like translation, rotation, scale and illumination invariance. Essentially, the measure should be such that the compactness hypothesis holds, i.e. representations of similar objects are similar. This means that a small variation of an object should impose only a small change of a proximity value, hence the natural variation of objects of the same class should be captured there. Many dissimilarity measures are constructed by solving object matching problems, often defined in terms of the minimization of the mean square error or mean absolute error by the use of affine transformations. This corresponds to the Euclidean or city block distance, which may not fully integrate the mentioned invariances. Such computed distances can not directly capture the structural information of the objects since they are based on sums of (weighted) independent contributions, referring only to some object properties. On the other hand, non-Euclidean or non-metric measures have become more popular, e.g. for measuring shape distances [Dubuisson and Jain, 1994; Jain and Zongker, 19971 or others [Jacobs et al., 2000; Edelman et al.: 1998; Guitrin-Dugu6 et al., 1999; Santini and Jain, 19991. Due to measurement noise in sensory data or other uncontrolled factors, there might be a necessity to improve the resulting dissimilarity measures. Noise can be reduced either in a pre-processing stage of nieasiireinerits or by the use of (non-)linear transformations, if the measures are just given or directly result from an earlier analysis. Such transformations may be also applied to impose a more compact class description, e.g. by making large distances smaller, or (if required) by imposing particular characteristics of distances such as a Euclidean behavior. Some of such transformations are described in Sec. 3.1. Since different proximity measures, as defined in feature spaces, between strings, graphs and on the sensor data may reflect various aspects of data characteristics. as well as various types of expert knowledge, their
170
T h e dassimilarity representation foT p a t t e r n recognition
Feature-based representationA(T,F) f,
f,
f,
Dissimilarity representationD(T,R) P< P. P.
Figure 4.8 Left: feature-based representation A(T,F).Right: dissimilarity representation D ( T ,R). Assume T = { t l , .. . , t n } is a set of training objects and F = { f l , . . . , fm} are the features. An object t, is then represented as a vector of its feature values a ( t i ,f,) i.e. A(t,,F)= [ a & , f l ) , . . . , a ( t i , fm)].The feature f, is represented as a vector A ( T ,f,). Hencc, features correspond t o dimensions in a (Euclidean) vector space, where objects become points. A dissimilarity representation describes the relations between objects, hence additionally, a collection of representatives R = { P I , . . . , p r } is needed. In the most simple case, R = T and for a quasimetric measure, the resulting D ( R , R) is a symmetric matrix with a zero diagonal. R might be a subset of T or a distinct set. An object t , is represented by a vector of its dissimilarities d ( t i , p , ) to the objects from R,i.e. D ( t Z R) , = [ d ( t , , p l ) , . . . , d @ i , p r ) ] . D ( T , p , ) = [ d ( t l , p , ) , . . . ,d(tn,p,)IT refers to dissimilarities to a particular object p,. Any entry in A is a feature value for a particular object, while any entry in D is a similarity value between two objects.
conibination might be beneficial. They may be considered either jointly or exclusively, or they may form a new proximity representation. The possibility of a combination makes a dissiniilarity representation a more universal representation due to the increased flexibility. Now, a complex problem can be described by a number of dissimilarity representations between their different aspects or characteristics. For instance, an article in a database can be represented (in an intermediate stage) as a point in a vector space, where each feature corresponds to the frequency of the specified keyword, but also a5 a trec organization of a title, an introduction. body, conclusions and references, c-tc. Next. two different dissimilarity measures can be designed in the statistical and structural approaches, yielding two distinct representations. which can be further combined. Combining proximity measures (or their transformed versions) [Pqkalska and Duin, 2001bl is closely related to the area of combining classifiers [Kittler et al.. 19981. Examples can be found in Chapter 10. Another fundamental question refers to the learning paradigms, especially those which deal either with non-metric or non-Euclidean measures. Basically, they take place in spaces, already introduced in Chapter 2. More precisely. they built on methods of linear algebra and functional analysis,
Learning approaches
171
as well as statistical learning [Hastie et al., 2001; Diida e t al.. 2001: Vapnik, 19951, kernel methods [Cristianini and Shawe-Taylor, 2000: Scholkopf and Smola, 2002; Vapnik, 19981 and approximate erribeddings in pseudoEuclidean and KreYn spaces [BognAr. 1974; Iohvidov et al.. 1982; Duin and Pekalska, 2002; Goldfarb, 1985; Pekalska et al., 2002b: Pqkalska and Duin. 2002~1,as presented in Chapter 3. Further on, the usefulness of pretopological spaces, offering poorer axioms than Euclidean spaces, can be studied as well. The compactness hypothesis may serve as a basic demand for building pretopological spaces from more general neighborhood relations. The learning approaches are discussed in the next section. 4.2.1
Learned p r o x i m i t y representations
We realize that the developed framework for dissimilarity representations is only a first step in the direction of integrating both statistical arid structural approaches, the problem of constructing an informative representation and proper learning methodologies. For dissimilarity representations, the mcasure itself is assumed to be given. To some extent it can be optimized with respect to a set of objects, but, rather in a limited way, such as the determination of parameters. The next step is to investigate how dissimilarity measures can be learned from a set of examples. For this purpose, a learned representation can be considered, primarily based on the structure present in real objects [Goldfarb, 1992: Goldfarb and Golubitsky, 2001; Goldfarb e t al., 20041. Two possibilities are considered: to learn a relative representation or to learn a conceptual representation. The first focuses on defining a dissimilarity measure and a set of prototypes to which other objech will refer. Such a representation is further used for learning. The conceptual representation descrihcs a dissimilarity of an object to a class. Such a dissimilarity is related to the costs (weights of transformations) of generating an object from a set of primitives (basic descriptors) in the context of other objects within a class, as well as objects outside this class. This is an attempt of a truly inductive way of learning [Goldfarb et al., 20041, where not wily the essential transformations and the weights are learned. but primitives as well. Siich a formulation is close to the problem of one-class classification [Tax, 2001; Tax and Duin, 20041. Another simpler approach is to combine the strengths of' the structural arid statistical frameworks on the level of a relative representation. Assume that one deals with objects that possess an identifiable structure,
172
T h e dissimilarity representation for p a t t e r n recognition
Interpret D(.,R) as a dissimilarity vector space and design a classifier there
W,R)
Figure 4.9
d(;pJ
Classification in a dissimilarity space: an illustration
such as spectra, time-signals, images or text documents. The first step is to define a small collection of fundamental structural detectors, yet general enough to be applicable in many problems, independent of a specific expert knowledge of the application. This means that such detectors are defined for the given measurement domain, e.g. spectra or images. The useful subpatterns should then be identified by the detectors when applied to the consecutive measurement values. The inter-relationships between the subpatterns should be captured in a relational intermediate representation (e.g. by a graph or by a string). These would be the basis for the matching proccss and the derivation of the final dissimilarity. The learning relies then on the learning of proper weights (contributions) assigned to the identified subpatterns such that the specified dissimilarity is optimal for the discriniination between the classes. The simplest example is the edit-distancc between string descriptions of objects, however, more general approaches are needed to be developed. Note that one may also consider statistical feature extractors (such as wavelets or Gabor filters), which work on the consecutive measurements, to be the building blocks of the learned dissimilarity. How to learn such measures is open for research. 4 .2 .2
Dissimilarity representations: learning
Statistical learning approaches can be adapted for dissimilarity representations. The added value of a dissimilarity-based framework lies not always in the following methodology, but in the representation itself. As we discussed in tlie previous section, a dissimilarity representation can include both the statistical and structural properties of data. Hence, instead of a single representation of a problem, one may also consider either a complex representation, built from many dissimilarity representations, or as a hybrid representation, where different aspects of tlie data are described
Learning approaches
173
in various ways such as by features, dissimilarities, and inference rules. Then one needs to face the task of combining various expertise [MCSOO, 2000; MCS02, 20021, as discussed in Chapter 10. In this section, we will concentrate on the classification task. Given a training set of K classes, a classifier tries to model a functional dependence between the data representation and the class indicators (labels) such that a new object is assigned to a specific class. The goal is the minimization of (the cost of) misclassification such that novel examples are as correctly labeled as possible. The problem to be faced in establishing classification methodologies for dissiniilarities is that the measures used in practice are often non-Euclidean or even non-metric. Nevertheless, they may perform well and it remains of practical interest to study their properties fiindameritally. Since dissimilarity representations encode the information on objects dissiniilasit,ies in a numerical way, the nature of learning is unavoidably numerical, which leads to the use of spaces. In general, we can distinguish three main approaches to dissimilarity representations, interpreted in the context of suitable spaces. They are briefly introduced below and trcated more thoroughly in the subsequent chapters. Assume a dissimilarity representation D ( T ,R), where R is a representation set and T is a training set. The measure d is general, as our basic requirements are only the nonnegativity and reflexivity, Def. 2.38. In the first neighborh,ood-based approach, making use (directly or not) of pretopological spaces, the dissimilarities between the objects are interpreted directly. This means that a dissimilarity representation describes an abstract space, where the basic neighborhoods or generalized closure operators play a key role; see also Sec. 2.3. The space is abstract in the sense that it is usually not explicitly given or it is not a vector space. It is defined by a set of available objects, e.g. a set of finite binary strings, performed measurements and the number of additional factors, such as camera positions or lighting conditions, playing role in the measurement process. So, this abst,ract space is either a. set of objects or some nieasiirement space. The neighborhoods arid generalized closure operators are defined by the use of dissimilarities to the objects from R. An example classifier is the k-nearest neighbor rule (k-NN). The second dissimilarity space approach, addresses a dissirriilarity representation as a data-dependent mapping specified by the representation set R. A mapping $(..R) : X + R" is defined as $ ( x , R ) = [ d ( z , p l )d ( z , p 2 ) . . . d ( z , p n ) ] . Note that X denotes either objects therriselves (e.g. a set of convex subsets of a finite-dimensional space), or
T h e dissimilarity representation f o r pattern recognition
174
iIll
Find a spatial representation In the underlying space E
*
Dfi ,/
)-
*.
D(RV
TRAIN
T
... D(T-R,R) R
TEST
Project the test data D(S R) onto E and apply the classifier
S
*
.
D(S.R)
Figure 4.10
Classification in a embedded space: an illustration.
a feature-based vectorial representation of objects. Note that X might not be given explicitly. The dimension of such a space is controlled by the cardinality of R.Using this formulation, decision functions can be directly constructed on dissimilarity representations, as in a vector space, in which each dimension corresponds to a dissimilarity to a representation object d ( . , p i ) . This is possible, since a dissimilarity vector space is assumed to be endowed with the traditional inner product and the associated norm and Euclidean metric. Additionally, if beneficial for learning, other distance measures, e.g. from the (,-distance family are considered. Since dissimilarities are nonnegative, all the data are mapped as points to a nonnegative ortharit of a vector space. Many traditional classifiers can be now applied [Pqkalska and Duin, 2000, 2002a; Pqkalska et al., 2002bI; as discussed in Sec. 4.4. See also Fig. 4.9. The third embedding approach requires that a dissimilarity representation D ( T , R ) is such that R C T. First, a spatial representation of the syiiimetric D ( R , R ) is found, i.e. a vector space V , where the objects are mapped as points such that their distances reflect the actual dissimilarities, as presented in Sec. 3.6. Then the remaining objects T\R, if exist, are projected there. Next, a decision function is determined in the this embedded vector space; see also Fig. 4.10. This is possible, as the embedded vector space is equipped with an inner product and additional algebraic structures, if needed. In learning, new objects are first projected to V and subjected to the classifier. Further details are presented in Sec. 4.5. In summary, to build a classifier for dissimilarity representations, the training set T of N objects and the representation set R of n, objects are used. R is a set of prototypes, possibly covering all the present classes. R is usiially considered to be a subset of T , R C: T , although R and T might
Learning approaches
175
be different for the first two approaches. R might be chosen from T either randomly or in a systematic way. For instance, n objects can be chosen such that thc minimum distance between them is maximized. Another possibility is based on a greedy approach. In an iterative mariner, starting from a randomly chosen object, an object is selected which is the most dissimilar to all objects already chosen. This might be applied globally or for each class separately. Such objects are likely to be atypical or positioned between the classes. If computationally tractable, one may also usc the complete representation D ( T ,T ) and try to select R to optimize a particular classifier on d ( T , R). Selection methods will be discussed in Chapter 9. In the learning process, a classifier is constructed by making use of the N x n D ( T , R ) ,relating all training objects to all the prototypes. The information on a set T, of s new objects is provided by their. dissiniila,rit>ies to R , i.e. as an s x n matrix D ( T SR). ,
4.3 Classification in generalized topological spaces Let X be either a finite set or a vector space. Consider a generalized metric space (X, p) with a, dissimilarity measure p: X X X4 R$ such that p ( z , z ) = 0. Let, B b ( z ) = { y E X : p ( z , y ) _< S} be a S-ball ('closed' neighborhood ball) for some positive 6. For each z E X define its miriinial neighborhood as B ~ ~ r z ( z=l ({yz )E X : p ( z , y ) 5 dnn(z)},where 6,r,,(z) = p ( n : , n n ( x ) )is, the dissimilarity of z to its nearest neighbor n n ( z ) . By the . growth function gr(A) is now reflexivity property of p, z E B ~ , , , ( z ) A defined on the power set P(X)as a generalized closure operator such that the following axioms hold for all z E X and all A E P ( X ) : (1) gr(G?) = 8.
(2) gr(z) =
" BiL,J&)
= B&)(4
= UzEA gr(z).
(3)
It is straightforward to check that this growth function fulfills tlie axioms (1) (4) of Def. 2.19, hence ( X ,gr) is a pretopological space. Such a closure operator describes the d,,-neighbors pretopology. Imagine now that d,, does riot depend on x , hence 6 ,, ( x ) = E > 0 for all z E X . If E is chosen as the smallest distance inX, E = minzEX p ( z , n n ( z ) ) then , the growth of the element z is gr(x) = BE-(z),hence is described by tlie &-ball. If X is a Banach vector space equipped with a metric distance p, e.g. such as the [,-distance, p 2 1, then thanks to the property of convex neighborhoods on which the &-ballrelies, gr2(z) = gr(gr(z)) = gr(B,(z)) = ~
176
t
The dissimilarity representation for p a t t e r n recogitition
Metric distance in a normed space
t
Non-metric dissimilarity
Figure 4.11 Assume a vector space X with the additional dissimilarity p (it might be for instance a measurement space). Example growth operators gr(z) = B E ( z )with a fixed E > 0 are shown, when p is a metric distance (left) or when p is a non-metric dissimilarity , in this (right). In the non-metric case, gr(gr(z)) is not necessarily identical t o B z E ( z ) as metric case.
BZE(z)and, more generally, gr"(x) = B,,(x) for m E Ns. For a metric not defined by a norm or for non-metric measure p, this is not true, in general. Still it may happcn that B,,(x) C gr"(z). See also Fig. 4.11 for ari illustration. Alternatively, one can define the k-nearest neighbor pretopology, where the growth function of z is defined by its k nearest neighbors, gr(.z) = {y : y E { n n l ( z )nn2(x),. , . . , n n k ( x ) } ,where nni(z) is the i-th neighbor of z. It additionally satisfies gr(0) = 8 and gr(A) = UzEAgr(z). It is then straightforward to check that gr fulfills the axioms of pretopology in Def. 2.19. Another possibility is to use neighborhoods to define the neighborhood basis at 2 . Let BE(x) = {y E X : p ( z , y ) < E } be an e-ball for a positive E. Then the neighborhood basis is defined as N B ( ~=){ B E ( z:) E = c p ( x , n n k ( x ) ) , k = 1 , 2 , . . .}, where c = 1 S is a constant and S > 0 is very small. Consequently, (X,NB) is a pretopological space. Consider now a training set T and a dissimilarity representation D ( T ,R ) . A generalized closure (growth) operator or neighborhood basis can be defined for every class wi based on D(T2,RZ),where Ti c T and
+
8We will show that & ( 5 ) = gr2(z) for a metric distance p in a Banach vector space. Note that BzE(z)= ( 2 : p ( z , x) < 2 ~ and ) gr2(z) = UZIEB,(2)&(y) = {(z,Y) : p ( z , Y) < E
A P(Y>z)<E). z t &(z)
+ Let
e p ( z , z ) < 2 ~ In . a Banach vector space with the metric p , there exists a unique middle point such that p(z, z) = p ( z , y) p(y, z). Hence, p(z, y) < E and p ( y , z ) < ~ .It follows that z t g r 2 ( z ) and consequently, B z E ( z )C gr2(z). + L e t zEgr2(z) H p ( z , y ) < & A p ( y , z ) < c . Since pisnonnegative, then p ( z , y ) + p ( y , z ) < 2 ~ By . the triangle inequality, p ( z , z)< p ( z , y) p(y, z). Hence, p ( z , z ) 5 Z E . It follows that z E B z E ( z )and, consequently, gr2(z) C B z E ( z ) .
+
+
Learning approaches
177
R, c R correspond to the objects from w,. So, BELis the neighborhood basis for the class w,. An unknown object is assigned to the class wk if it belongs to a generalized closure or a neighborhood of one or more objects from the class wk only. If no single class exists. then the sets Bme, for rn E N can be used instead as an approximation of the successive growth by a repetitive use of the generalized closure (in a metric vector space with a growth function defined by a metric distance, B,,(z) C gr"(z) holds). If an object belongs to the intersection of neighborhoods (or closures) of two (or more) classes, then the final decision should be made by looking at the majority of objects from a particular class within the neighborhoods. It means that the decision rules built on the dissimilarities directly can be interpreted as classifiers in pretopological spaces. Examples are variants of the nearest neighbor rules. A classifier based on the repetitive generalized closure operators in pretopological spaces (hence based on growing neighborhoods) is also discussed in [Lebourgeois and Emptoz, 1996; Frillicot and Emptoz, 19981, Nearest neighbor (NN) rule. A straightforward approach to dissimilarities leads to a nearest neighbor rule [Cover and Hart. 1967; Fukunaga, 19901 or, more generally, to an instance-based learning [Aha et al., 19911. In its simplest form, the 1-NN rule assigns a new object to the class of its nearest neighbor. Originally, this nearest neighbor is chosen from the training set T . The k-NN rule is based on majority voting. An unknown object becomes a member of the class the most frequently occurring among the k nearest neighbors. Usually, k is assumed to be odd to avoid ties (for two-class problems). Notc that when k is fixed, no training is involved. Traditionally, the k-NN rule is applied to data represented as vectors in a vector space, for which either Euclidean or city block metrics arc derived. This means that indirectly the corresponding dissimilarity representation is used. This nearest neighbor principle can be extended to other (nonmetric) dissimilarity measures, obeying the compactness hypothesis: see also Sec. 4.1. The k-NN classifier is attractive, since it is simple, intuitively appealing arid no prior knowledge of the data distributions is required. It can estimate complex boundaries locally and differently for each new instance (hence its adaptations can be seen as an example of transductive learning). Moreover, it is known [Devroye et al., 1996; Diida et al., 20011 that for the k-NN rule fk" in metric vector spaces, the empirical risk € , , , ( f k ~ ~ ) converges uniformly to the actual risk € ~ N Nwith increasing n, Eq. (4.5), such that
178
T h e dissimilarity representation for pattern recognition
. . . & 3 N N 5 El” 5 E(f*)(2 -&E(f*)), where &(f*) is the Bayes error and K is the number of classes. This means that the k-NN rule is asymptotically at most twice as bad as the Bayes rule. Please refer to [Devroye et al., 19961 for proofs and other bounds. In practice, when one deals with finite sample sizes, the asymptotic inequalities will not hold. The k-NN rule is expected to perform well, provided that the domain of a problem, hence the data are well sampled. In cases, where a t least one of the classes is undersampled or badly sampled, the k-NN rule deteriorates. The k-NN rule can also be interpreted as the one which locally tries to estimate the posterior probabilities. These estimates rely on a neighborhood determined by the k-furthest neighbor. For small k , the nearest neighbors might often lie further away due to the data sparseness or the estimates might be poor due to noisy examples. Increasing k allows one to reduce the noise influence. However, the nearest neighbors with large dissimilarities in the voting scheme may lead to an unnecessary error. Therefore, a weighted voting [Devroye et al., 19961 might be an option, where the neighbor contributions are weighted accordingly to their dissimilarities to a particular object. In our approaches, the nearest neighbors will be found among objects from the representation set R C T . For a test example t,, the k-NN rule will be then applied to D ( t s ,R).
&(f*) 5 . . . 5
&21+1NN
5
12Z-lNN
5
Weighted nearest neighbor (WNN) rule. The use of k nearest neighbors in the k-NN rule is based on the assumption that they are relatively close to the object z at question. If this is not the case, it might be sensible to weight the neighbor contributions accordingly to their dissimilarities to z. Asymptotically, for a fixed k , all the k neighbors should be very close to 2 , hence the weights should not have a large impact for the classifier performance. On the contrary, for finite training sizes, it niight be helpful. Let T L ( ~ ) ( denotes ~ ) the i-th nearest neighbor of x (under a specified dissiniilarity d ) ) with the label yi and let w1 2 w2 2 . . . wk be the corresponding weights. Formally, for a two-class problem and the labels (0, 1}, the k-WNN rule (assume odd k to avoid ties) is:
Weights should emphasize the neighbors which are nearby x, but also the contributions of the far-away neighbors should be counted. So, d(x.n(%) (x))
Learning approaches
179
should be weighted by a monotonically decreasing function. For exam1 if the denominator is zero),
1where E > 0 is a small constant to avoid the division by d, +E ' zero. 7rJ = e C d ( x , n ( ~ ) ( z ) ) where /u, n > 0 determines the neighborhood or u3 = e - N d ( z ~ n ( , ) ( 2 ) ) 1a,P 3 , > 0. In empirical study of [Zavrel, 19971, the weights chosen according to the first possibility mentioned above, gave rise to the improved k-WNN rule also performing better than the k-NN rule. VJ =
Edited and condensed nearest neighbor rules. Despite the simplicity and good performance of the k-NN rule, the criticism points a t both the space requirement to store the entire training set, and the computational expense of' computing dissimilarity to all training examples. Consequently, there has been an interest in condensing the training set in order to reduce its size [Hart, 1968; Dasarthy, 1991; Wilson and Martinez, 20001. In oiir terminology this is equivalent to a selection of a proper representation set R out of the training set T , hence it niight be called a prototype selection. as well. Also editing [Devijver and Kittler, 19821 is considered to increase the accuracy of the k-NN predictions, in the presence of' noise in the training data. A basic editing algorithm removes noisy instances as well as close border cases, leaving smoother decision boundaries. It also retains all 'internal' points; i.c. it does not reduce the number of objects as much as most other reduction algorithms. More on editing and condensing can be fourid in Sec. 9.2. Many variants of the "-rule, taking into account the local structure of the data or weighting the neighbor contributions appropriately, are invented or adopted for feature based representations; see Sec. 5.6 for a, brief information. The question on how such measures should be constructed is beyond the scope of this work.
In conclusion. The potential of generalized closure operators and neighborhood systems as the means of defining pretopological spaces is not explored yet. Although Chapter 2 provides basic definitions and some intuition in this area, learning methods are not developed. Currently, only generalized variants of the nearest neighbor approaches are considered. We think that the study of pretopological spaces may bring additional understanding on the perspective of existing statistical methodologies. It may open new ways of defining classification rules, in which relations between objects are described by some binary function, such as a general proximity measure.
180
4.4
The dissimilarity representation f o r pattern recognition
Classification in dissimilarity spaces
The novelty of our approach relies on interpreting D ( T ,R) as a representation of a vector space, called a dissimilarity space, where each dimension describes the dissimilarity to a particular object. More formally, let X denote a set of objects (e.g. a set of fruits, a set of finite strings over a particular alphabet or a set of gray-value images of particular size), or a feature vector representation of objects. Let R = ( p 1 , p 2 , . . . , p n } be a collection of representation objects, chosen from X and let d be a dissimilarity measure used to compare pairs of objects from X. We will assume that d is bounded, i.e. there exists a positive constant M such that d(x,y) 5 M for all 2 , y E X . If this is not true, then a suitable semimetric transformation, Theorem 3.7, can be used to bound it. A data-depending mapping 4 ( . .R ) : X + RTL> described as d(x,R) = [ d ( z , p l )d ( z , p z ) . . . d ( x , p n ) l T def i r m an n-dimensional vector space, a dissimilarity space denoted D ( . ,R). 4.4.1
Characterization of dissimilarity spaces
In order to use the apparatus of statistical learning, a dissimilarity vector space will be equipped with the traditional inner product and the associated norm and Euclidean metric. Additionally, other dissimilarity measures, especially from the family of the !,-distance will be considered. Important observations can be made for metric distances. Proposition 4.1 Let D ( . ,R ) be a dissimilarity space built b y a metric distance d . Consider ( D ( . ,R ) ,d,), the dissimilarity space equipped with the max-norm distan,ce. Then the space ( D ( . ,R ) ,d,) results from an embedd i n g of a metric distance representation D ( R ,R ) .
Proof. This follows from Lemma 3.1, where it is proved that the max-norm distance in a dissimilarity space D ( ., R ) , defined as d m ( d ( P 7 ;R ) ,d ( P j , R ) )= maxp€R ld(P2,Pr) - d(Pj,Pr)I is equal to the 0% irial distance d(pi,p ] ) . 0 Proposition 4.2 Let ( X ,d ) be a set of objects X with a metric distance d . ASTUTE that R c X and D ( . ,R ) is a dissimilarity space built by d . Consider a scaled dissimilarity space ( D ( s ) ( R),d,), ., defined as D ( , ) ( x ,R ) =
C k D ( x ,R ) , equipped with the !,-distance d,, p 2 0. Then the metric d majorizes d, or, in, other words, the mapping $ : ( X ,d ) + ( D ( s ) ( R . , ) ,d,) is a contraction (Lipschitz continuous with IE. = 1).
Learning approaches
181
Proof. The statement above means that in a suitably scaled dissimilarity space the !,-distances take values not larger then the original distances d. This also holds for p~ (0, I), hence non-metric l,-distances. Since d is metric, the backward triangle inequality, Theorem 2 . 5 , Id(z. z ) d(y, z)l 5 d ( x , y) holds for any IC, y, z E X. Since distances are nonnegative and the power function f, = x p is monotonically increasing on R : , this is equivalent to Id(x, z ) - d(y. .)I” 5 d ( z ,y), for p 2 0. Consider now the !,-distance in a dissimilarity space. Consider any z. y E X. Using the facts mentioned above. one has
(4.9)
Hence, d P ( D z ( zR), , D(sl( y , R ) ) 5 d ( x , 1 ~ ) and by Def. 3.3, the mapping 0 $ : (X, d ) i ( D ( s , ( .R), , d p ) is a contraction.
Remark 4.1 If the original dissim,ilarity space i s considered, then $ : ( X ,d ) + ( D ( . ,R ) ,d,) i s Lipschitz for K = ni. This holds as for th,e chosen I finite representation set R,n* i s a constant, n o t depending o n X . A S n,; m a y be a large constant, it is u s e f d t o linearly scale the given dissimilarities. Remark 4.2 If the dissimilarity measure d is non-metric, one m a y make at first symmetric, ,for instance by averaging, davT(x,y) = $ ( d ( x ,y) d ( y , x ) ) (otherwise two asymmetric dissimilarity spaces can be studied). If th,e definiteness and triangle inequality are disobeyed, then there exists a constant c > 0 , such that the distance d,,, defined as d,(x, y ) = davT(x,y) c for x # y and d c ( z , x ) = 0, is metric. This follows f r o m Corollary 3.3 and the fact that d is bounded. If d ( x ,y) 5 M for all x , y E X , th,en c E ( 0 ,MI. If the mapping $ : ( X ,d,) + ( D ( s , ( .R):d,) , as defined in Proposit%on4.2 is considered, then d, mujorizes the !,-distance in th,e scaled dissimilarity space.
+
+
Note that the mapping $ of ( X , d ) into a dissimilarity space is a type of a distorted embedding, Sec. 3.1.2. Note also that since is Lipschitz: it is continuous by Theorem 3.1. $J
182
The dzsszmzlarzty representatzon f o r pattern recognztzon
Figure 4.12
Metric 2D dissimilarity space.
If‘ the distance measure d is a metric, then all vectors D ( z , R ) lie in an ,ri,-diniensionalprism, bounded from below by a hyperplane on which the objects from R are and which is bounded from above in case of bounded dissimilarities. Consider a 2D representation D ( z ,R ) , where R = [ p i , p j ] . For brevity, denote that d,, = d ( p i , p j ) and z = d ( z , p i ) , and y = d ( z , p j ) for an object z. The following triangle inequalities should hold: z f y 2 d i j , d i j n: 2 y and d i j y 2 z for a metric d. Depending on z and y, a prism is formed as shown in Fig. 4.12. Note that in higher-dimensional spaces, the (1iyper)prism is asymmetric and the vertices of its base do not lie on the axes. For instance, in a 3D space, the vertices lie in the zy-, yz- and z z - planes. In principle, z may be placed anywhere in the nonnegative ortliant of a dissimilarity space D ( . , R) only if the triangle inequality is completely violated. This is, however, impossible from the practical point of view, because then the compactness hypothesis would not be fulfilled. Consequently, this would mean that d would have lost its discriminating properties of being (relatively) small for similar objects. Therefore, the measure d, if not metric, it has to be sufficiently close to a metric and, thereby, D ( z , R ) will still lie either in the prism or in its relatively close neighborhood’. See Fig. 4.13 and Fig. 4.14 to get some intuition. A justification for the construction of classifiers in dissimilarity spaces is as follows. The property that dissimilarities should be small for similar objects, i.e. belonging to the same class, and large for distinct objects, gives a possibility for a discrimination. Thereby, D ( . , p i ) defined by thc dissimilarities to the representative p i can be interpreted as a ‘feature’. If p , is a characteristic object of a class w , then the discrimination power of
+
+
’This is not always true. If one considers a power transformation of a metric distance, which as a monotonic transformation preserves the order of dissimilarities, t,hen a large deviation from the triangle inequality can be expected. For instance, this happens for d = d;O for the Euclidean distance dz taking values in [0,5].
Learning approaches
Theoretical data
Dissimilarity space
Theoretical data
5
5
0
0
-5
183
Dissimilarity space
-5 0
5
1 0 1 5 0
5 DLpJ
10
0
5
1 0 1 5 0
5
10
DLp,)
Figure 4.13 Simple illustration of 2D dissimilarity spaces. The first and third plots show the theoretical artificial data with a quadratic classifier. The t5/2-distance reprcsentations D ( T , R ) were considered defined by two representation objects R = [ p l r p 2 ] and the measure d(ti,pj) = Jt,k The second and thc fourth plots present the dissimilarity spaces D ( . , R ) , where the objects p l and p2 are marked by circles on the first and third plots. For a well chosen R, a linear classifier in a dissimilarity space D ( . . R ) separates the data well. In this example, a dissimilarity representation may be used to encode nonlinearities from the original vectorial representation. Here; a dissimilarity representation is derived for the given vector space. In general, we assume the other way around: only a dissimilarity representation is given for which is interpreted in a suitable space.
(C;=,
D ( . , p i ) can be large, i.e. the dissimilarity values for the objects from w become small. If pi is an atypical object of its class, then D ( . , p i ) may not be informative. On the other hand, as the usefulness of D ( . , p i ) is judged with respect to objects of other classes, atypical exarnples of w may still be distinctive. The overall strength lies in using the complete representation set R. Another reasoning relies on the following fact. If the objects 3: and y are alike and the dissimilarity value d(z,y) is small, then for othcr objects z , the dissimilarities d ( z ,z ) and d ( y , z ) might not ta,ke similar values if the measure d is non-metric. However, is the dissimilarities of z and y to a given set of prototypes R are inspected, one can expect that, although the individual values will differ, the vectors D ( z ,R) and D ( z ,R) are correlated in their entirety. If so, then the representations D ( z ,R ) and D ( z , R ) arc close in a dissimilarity space (as judged e.g. by the Eiiclideaii distance there). Consequently, the dissimilarity space approach should be useful for non-metric mea,sures. One may wonder what the added value of such a representation over a feature-based representation is, if the traditional classifiers designed for vectors spaces may be applied in the end. The strength of a dissimilarity representation relies on the flexibility of a dissimilarity measure to be defined on quite arbitrary sets or measurements, such as sequences? text documents, digital images, spectra or features. Moreover, a dissimilarity
184
The dissimilarity representation for pattern recognition
Euclidean
Dissimilarityl o 8
Modified Hausdorff
Dissimilarity to 8
Figure 4.14 Examples of 2D dissimilarity spaces and linear classifiers for a subset of handwritten digits 3 and 8. Two dissimilarity representations D ( T , R ) are shown based on the Euclidean distance (metric) computed between blurred images and the modified Haiisdorff distance (non-metric) between digit contours; see also Sec. 5.5. R is randomly chosen and consist of two examples, one for each digit.
measure has a potential to encode statistical or/and structural characteristics of data instances. Also dissimilarity measures can be naturally combined, e . ~by . their weighted sum. A dissimilarity representation allows one to capture the properties of objects more adequately, as more emphasis and knowledge is put to a class of similar objects. Dissimilarity representations are numerical descriptions and are interpreted in suitable spaces. The choice of a vector space D ( . ,R ) is in agreenient with its mathematical concept in the following way. The dimensions of such a space are now dissimilarities to the prototypes which are derived according to a specified measure. Hence, they convey homogeneous type of information. This is riot valid for a general feature-based representation, where features have different character and range, e.g. weight or Icngth. Another advantagc of a dissimilarity representation is that since a dissimilarity measure already possibly encodes the object structure and/or other characteristics, the designed classifiers might be chosen to be simple, e.g. linear models. Defining a well-discriminating dissimilarity measure for a non-trivial learning problem is difficult. Designing such a measure is equivalent to defining good features in the traditional feature-based classification problem. If a good measure can be found and a training set is representative, then the k-NN rule is expected to perform well. The decision of the k-NN is based on local neighborhoods and it is, in general, sensitive to noise. It means that k nearest neighbors found might not be the best representatives
Learning approaches
185
for making a decision to which class an object should be assigned. For small or non-representative training sets, a better generalization can be achieved by a classifier built in a dissimilarity space. For instance, a linear classifier in a dissimilarity space is a weighted linear combination of dissimilarities between an object and the representation examples. The weights are optimized on the training set and large weights (in magnitude) emphasize objects which essentially influence the final decision. By doing this, a more global classifier can be built, by which its sensitivity to noisy representative examples is reduced. Our experience confirms that a linear or quadratic classifier can often generalize better than the k-NN rule; especially for a small representation set R; see also [Pqkalska and Duin, 2001al. 4.4.2
Classifiers
A dissimilarity representation D ( T ,R ) is an N x n dissimilarity matrix. D ( x ,R ) is a row vector in D ( T ,R ) , for simplicity reasons, however, it will be treated as an n x 1 column vector without any further notice. A general linear function in a dissimilarity space D ( . ,R ) has the following form:
c n
g ( D ( x ,R ) ) =
wj d ( z , p j )
+ wo = wTD(z,R ) + wo.
(4.10)
j=l
In the training process, y is determined as a decision boundary between two classes. A classifier is a function returning the class assignments. Usudly, one assumes that the equation g ( D ( x ,R ) ) = 0 defines a decision boundary, hence for a two-class problem, the sign of g ( D ( z , R ) )indicates to which class the object z belongs. Assume a, t,wo-classproblem with a set of labels {+l,-1). Then Decision boundary: g ( D ( z ,R ) ) = 0. Classifier: f(D(.? R ) ) = sign ( g ( D ( x R)). ,
(4.11)
For the sake of convenience, since now on, we will not make any distinction between a boundary function g and its indicator function f , both denoting as c2ass$ers. We will also assume that a dissimilarity measure d is scaled suitably to build a dissimilarity space, e.g. as described in Proposition 4.2; by a semimetric transformation or in some other way to guarantee that the dissimilarities are bounded by a not too large constant. In the description below, K classes, w l , w2;. . . ,W K , are distinguished. Usually two classes are considered, K = 2, since any multi-class classifi-
186
T h e dzssamilarity representation for p a t t e r n recognition
cation problem can be decomposed into a number of two-class problems. Thc k-th class with cardinality rj,k is denoted by wk,k = 1 , 2 , . . . , K , and its prior probability by p ( w k ) . Thc mean vectors per class, determined in the dissimilarity space D ( . ,R) are denoted as mi, and the overall mean is given as m. In features vector spaces, the training pairs {(xi,yi)}yE1 are corisidercd such that x L is a vector arid y2 = (-1, I} is a label. We will make an additional observation here. Assume that each class is modeled by a proximity function f w , capturing proximity of an object to thc class w,.This, in fact, means that, a onc-class model is designed for each class. An example is the log-likelihood, when tlie class is niodeled by a single or a mixture of Gaussians or the reconstruction error, when the class is modeled by a PCA-subspace. The boundary betwecn two classes, w , and wj is then defincd by f u , ( D ( z ,R ) ) = f w , ( D ( x ,R)), which means that g ( D ( z ,R) = fu, ( D ( z ,R ) ) f u , ( D ( z ,R)). Note that in such a situation, a multi-class problem does not require the training of all two-class classifiers. As each proximity function evaluates commonality of a particular objects :I: to the given class, the final decision is made by the maximum evidence (provided that these values can be compared). Assuming that f w , is a. similarity fiinctioii, the classifier f is defined as f ( D ( x ,R ) ) = class wk if fu, ( D ( x ,R ) ) = maxi{fut(D(x,R ) ) } .If f u , is a dissimilarity function, then tlie final decision relies on the minimum dissimilarity. In the probabilistic reasoning, proximity functions estimate a posteriori probability, herice the decisioii and the final classifier are based on the maximum a posteriori estimates. Consequently, all density-based class$ers c a n directly be design,ed as m,ulti-class classi6ers. Classifiers originally designed in (inner product) vector spaces can be adopted to dissimilarity representations. Below, several decision functions, used in statistical pattern recognition, is described. The list is by no means complete. For instance, neural networks are not covered here, as they are highly nonlinear classifiers [Bishop, 1995; Ripley, 19961. Since the knowledge on tlie problem is assunied to be incorporated to the dissimilarity measure, simpler classifiers ar