edited by
C H Chen Electrical and Computer Engineering Department, University of Massachusetts Dartmouth, N. Dartmouth...
145 downloads
3638 Views
25MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
edited by
C H Chen Electrical and Computer Engineering Department, University of Massachusetts Dartmouth, N. Dartmouth, MA, USA
L F Pau Ericsson, Sweden
P S P Wang College of Computer Science, Northeastern University, Boston, MA, USA
orld Scientific New Jersey. London Hong Kong
Published by World Scientific Publishing Co. Pte. Ltd. P 0 Box 128, Farrer Road, Singapore 912805
USA ofJice: Suite 1B, 1060 Main Street, River Edge, NJ 07661 UK oflice: 57 Shelton Street, Covent Garden, London WC2H 9HE
Library of Congress Cataloging-in-Publication Data Handbook of pattern recognition & computer vision /edited by C.H. Chen, L.F. Pau, P.S.P. Wang -- 2nd ed. p. cm. Includes bibliographical references and index. ISBN 9810230710 (alk. paper) 1. Pattern recognition systems. 2. Computer vision. I. Chen, C. 11. Pau, L.-F. (Louis Franpis), 1948H. (Chi-hau), 1937111. Wang, Patrick S.-P. (Shen-pai) IV. Title. Handbook of pattern recognition and computer vision. TK7882.P3H35 1999 006.4--dc21 98-51616 CIP
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library
First published 1999 Reprinted 2001
Copyright 0 1999 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, orparts thereoJ may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the Dublisher.
Printed i n Singapore by U t e P r i n t
PREFACE TO THE SECOND EDITION
The progress in pattern recognition and computer vision since the publication of the first edition in 1993 has been enormous. While the first edition is never out of date, t o incorporate new activities in the field, and to incorporate many valuable comments by the readers of the first edition, it is useful now to present a new edition. More than half of the pages of this volume are new, reporting a number of new activities. This edition also contains five parts: (1) Basic Methods in Pattern Recognition, (2) Basic Methods in Computer Vision, (3) Recognition Applications, (4) Inspection and Robotic Applications, and (5) Architectures and Technology, with a total of 34 chapters of original work on various topics. Part 1 starts with a chapter on cluster analysis which is followed by a chapter on statistical recognition and a chapter on syntactic pattern recognition. Until about twenty years ago, these three areas are what fundamental pattern recognition was about. The re-discovery of artificial neural networks in the mid-eighties has made it a n essential topic of modern pattern recognition. The fourth chapter addresses particularly neural network computing from the viewpoint of implementing pattern recognition algorithms. Additional basic pattern recognition methods are presented in the next two chapters on Gaussian Markov random field model and 3-D object pattern representation respectively. The first two chapters of Part 2 deal with texture analysis and segmentation. We agree with some readers’ comments that the first edition placed a lot emphasis on texture analysis and in fact Chapters 1.5; 2.1; 2.2; 4.2 of this edition are texture-based. This reflects that more progress has been made on this topic. The third chapter on color in computer vision has been fully rewritten from the first edition t o provide some latest progress. The next two chapters are on projective geometry and 3-D motion analysis. The last two chapters of Part 2 present aspects of 3-D shape representation and 3-D surface-based systems. Part 3 provides a comprehensive study of nine recognition applications including: nondestructive testing of materials, speech recognition, remote sensing application, fingerprint processing using an opto-electronic system, human chromosomes classification, document processing, biomedical signal recognition, geographic data analysis, and face recognition. Obviously it is not possible to present all application areas and the readers are kindly requested to refer t o the first edition and other publications for additional recognition applications. The first two chapters of Part 4 are in fish industry inspection and textured surface inspection with industrial objects. The next chapter on context related issues in image understanding is of concern t o all inspection and robotic applications. The chapter on computer vision in postal automation represents one aspect of robotic application. The last V
vi
Preface to the Second Edition
chapter on vision-based automatic vehicle guidance presents an area of emerging importance in intelligent transportation systems and again robotic vision can play an important role. Part 5 deals with several aspects of architecture and technological development. The first chapter on vision provides a tutorial discussion of technological issues in computer vision. The next chapter presents highlights of optical pattern recognition. Infrared imagery technology has been around for some time and its application has not been limited t o military and medical areas. The third chapter deals with this technology and its use in classification problems. Another major technological development is the video content analysis and retrieval which is discussed in the fourth chapter. The VLSI architecture for computing central moments for pattern recognition and image processing is presented in the final chapter with applications to breast cancer detection and road pavement distress detection. We understand that even with all chapters in this volume, there are still a lot of activities in pattern recognition and computer vision not presented here. Our view is that a handbook is not a dictionary and it is not desirable to cover all topics superficially. The in-depth treatment of a topic in each chapter can provide readers with a good understanding on the subject while collectively the chapters capture well all major developments in this field. The second edition is specially dedicated to the memory of Professor King Sun Fu. There are five chapters written by Professor Fu’s former students. During his career, Professor Fu has approached almost every area of recognition application in his time. By following his emphasis on applications, we have expanded the application part considerably. We are fortunate to have Professor Azriel Rosenfeld, who knew Professor Fu well, to write on the topic of vision with some speculations. Professor Rosenfeld’s words of wisdom will always be helpful to the research and education community. We are also fortunate to have Professor C. C. Li, who was Professor Fu’s classmate at the National Taiwan University, to prepare a memorial article. We thank both of them for their contributions. We like to take this opportunity to express our deep gratitude to all new and old contributors to this handbook series. This volume continues to represent the most comprehensive and most up to date handbook publication in pattern recognition and computer vision. The book will certainly help us to prepare for a new century of dynamic development in this field.
The co-editors June 1998
PREFACE
The area of pattern recognition and computer vision, after over 35 years of continued development, has now reached its maturity. The theories, techniques and algorithms are mostly well developed. There are a number of applications which are still being explored. New approaches motivated by applications and new computer architectures available are still being studied. Also the recently renewed and intensive efforts on neural networks have had great and positive impact on pattern recognition and computer vision development. Pattern recognition and computer vision will definitely play a very major role in advanced automation as we enter the 21st century. Amid all of these activities now going on, this new Handbook of Pattern Recognition and Computer Vision is much needed to cover what has been well developed in theory, techniques and algorithms and the major applications of pattern recognition and computer vision as well as the new hardwarelarchitecture aspects of computer vision and the related development in pattern recognition. The previous Handbook of Pattern Recognition and Image Processing, edited by T. Y. Young and the late K. S. Fu (Academic Press, 1986) was well received. The progress in pattern recognition and computer vision has been particularly significant in the recent past. We believe this new handbook that reflects more recent developments especially in computer vision will serve well the increasingly larger community of readers in the area. As students and friends of Prof. Fu, we remember well his vigorous efforts to broaden the frontiers of pattern recognition and computer vision in both theories and applications, to build it as an interdisciplinary area, and to lay down the foundation of intelligent and automated systems based on pattern recognition and computer vision. The book, in keeping up with his vision for the area, provides an extensive coverage of major research progress since the publication of Young and Fu’s book. The book is organized into five parts. Part 1 presents a thorough coverage of the basic methods in pattern recognition including clustering techniques, statistical pattern recognition, neural network computing, feature selection, and syntactic, structural and grammatical pattern recognition. Part 2 presents comprehensively the basic methods in computer vision including texture image analysis and model based segmentation, color and geometrical tools, 3-D motion analysis, matheniatical morphology, and parallel thinning algorithms. Part 3 presents several major pattern recognition applications in nondestructive evaluation, geophysical signal interpretation, economics and business, underwater signals, character recognition and document understanding, biomedical image recognition and medical image vii
viii
Preface
understanding. Part 4 focuses on unique applications in inspection and robotics with topics on computer vision in the food processing industry, context modeling and position estimation for robots, and related issues. Part 5, on the other hand, examines the broader system aspects, including designing computer vision systems, optical pattern recognition, spatial knowledge representation, neural network architecture for image segmentation, architectures for computer vision and image information systems. More than 85 per cent of the chapters are original and unpublished work while the remaining reprint chapters provide complementary coverage. There is no doubt that a single volume handbook like this cannot examine every aspect of pattern recognition and computer vision, nor can it present the contributions of all leading researchers. However, we believe the book has captured both the scope and depth of progress in this highly dynamic and multidisciplinary area. In preparing the book, we are most fortunate to bring together all contributors who are among the leaders in the area. We would like t o take this opportunity to express our deep gratitude to their unselfish and timely efforts to share their expertise with the readers. We also like to thank Dr. K. K. Phua and Ms. Jennifer Gan of World Scientific Publishing for their help and encouragement throughout the preparation of this volume. C. H. Chen L. F. Pau P. S. P. Wang September 1992
FOREWORD VISION: SOME SPECULATIONS
AZRIEL ROSENFELD C e n t e r for A u t o m a t i o n Research University of Maryland, College Park, M D 20742-3275, USA
1. Introduction The purpose of vision is t o extract useful information about the world from images. Computer vision attempts t o do this by computer analysis of digitized images. Efforts along these lines have been under way for over 40 years; but even the domains first studied in the 1950s (which included handwriting, photomicrographs of biological specimens, and aerial photographs of built-up areas) still present many unsolved problems. This very brief note presents some comments, many of them intended t o be provocative, about the reasons for the slow progress in the field and the ways in which successes are likely to be achieved. 2. Vision and Mathematics
As an hommage t o Wigner [l],this section might have been entitled “On the unsurprising ineffectiveness of mathematics in the visual sciences”. Most vision problems, even those that were first tackled in the 1950s, are mathematically ill-defined (reading handwritten words, counting cells, recognizing buildings). Real-world visual domains do not satisfy simple mathematical (even probabilistic) models. Even when such models are assumed, problems that involve inferring information about a scene from images are often mathematically ill-posed or computationally intractable; but a more serious difficulty is that the models themselves are unrealistic, and are likely to remain so for a long time to come [2]. On a more positive note, mathematical and statistical tools do have their uses in formulating vision problems; in particular, they provide methods of describing image formation processes and image analysis algorithms. There also exist domains (typically involving machine-made scenes: printed documents, integrated circuits, mechanical arts, all with controlled lighting) that can be mathematically modelled quite accurately (as long as the scene is not too dirty!). In these sorts of domains, vision systems can be designed t o perform quite successfully, provided they take the model seriously and bring adequate computer power to bear on the task. ix
x
Foreword
3. Vision and Biology In this section we put mathematics aside and consider what might be called the “multilegged existence theorems” for vision [3]. Animals use vision quite effectively in the real world. Apparently, simple algorithms can be quite useful for extracting useful information about natural environments (for example [4],insects rarely collide with the underbrush even when flying through it at many body lengths per second); though an organism may have trouble coping if its environment changes significantly (for example, a frog surrounded by unmoving flies will probably starve to death [ 5 ] ) . Simple computer vision algorithms can in fact be designed, sometimes with the aid of learning techniques, that will usually perform usefully in real-world environments; but they may fail disastrously if they encounter unusual situations. For simple organisms, nature overcomes their individual limitations by providing large populations in which there are variations among the individuals. Thus even if many individuals fail, others survive (or learn to survive) and may pass their successful characteristics on to their descendants. (It has been suggested, halfseriously, that this might provide a biological justification for the hundreds of “yet another ...” computer vision algorithms in the literature; but note that the analogy is valid only if these algorithms are compared on a variety of tasks, so their potential advantages can be discovered!) Only a few pieces of research are truly innovative and can be regarded as major mutations; as in nature, such breakthroughs are sometimes lethal, but they may sometimes succeed in producing new successful species of vision systems. This “biological” approach could be used to discover useful computer vision algorithms through extensive experiments with real data; but it does not seem practical to develop robot vision systems in this way, since the cost of the failures would probably be unacceptable. A possible strategy for developing useful vision algorithms, without incurring many costly failures, might be t o initially use large sets of recorded real-world data t o design and test the algorithms. Higher-level organisms are more flexible in their uses of vision. Their flexibility may be based on an ability to evaluate and combine the information extracted from the images in multiple visual “areas” or “pathways”. This suggests that computer vision systems could benefit from the deliberate use of multiple techniques an approach that has been generally avoided in the past because of its computational cost. Another possible basis for the successful visual performance(s) of organisms is that they make use of redundant visual data. Computer vision systems often t,ry to reduce computational cost by analyzing only single (sets of) frames, and (for dynamic scenes) using results obtained from earlier frames as predictors t o simplify the analysis of later frames. Biological organisms, on the other hand, process all the data that their visual systems provide (though results obtained from earlier processing may serve to direct attention to subsets of the results obtained from later processing). The availability of redundant data allows an organism to discover
Foreword
xi
processing errors, since they give rise to results that are not persistent. Computer vision systems are just reaching; the levels of processing power that will allow them to handle, in real time, amounts of input data comparable to those handled by biological visual systems, and t o apply multiple processing techniques to the data. As these levels are reached, vision system performance may significantly improve. 4. Concluding Remarks Over the past four decades, the performance of computer vision systems has largely kept pace with available computer power. Many domains remain intractable from a theoretical standpoint, but systems that take advantage of data and algorithm redundancy may eventually achieve performances comparable t o those of biological organisms, which serve as living demonstrations of the effectiveness of vision in the real world. References [l] E. Wigner, The unreasonable effectiveness of mathematics in the natural sciences, Comm. Pure Appl. Math. 13 (1960). [2] A. Rosenfeld, Some thoughts about image modeling, In K. V. Mardia and G. K. Kanji, (eds.), Statistics and Images 1 (Advances in Applied Statistics, Supplement to J. Appl. Stat. 20 (5/6), Carfax Pub. Co., Abingdon, Oxfordshire, UK, 1993) 19-22. [3] R. A. Kirsch (personal communication, June 1967) used the phrase “two-legged existence theorem” as an argument for the feasibility of computer vision (or, as we were calling it then, “pictorial pattern recognition”). He probably had in mind only featherless bipeds, but an even earlier example (an existence proof for the air-to-ground ATR problem) is Skinner’s 1945 demonstration that pigeons could be trained to serve as effective bombsights. [4] J. A. Albus, personal communication, about 1990. [5] J. Y. Lettvin, H. R. Maturana, W. S. McCulloch and W. H. Pitts, What the frog’s eye tells the frog’s brain, Proc. Ire 47 (1959) 1940-1957.
A MEMORIAL TO T H E LATE PROFESSOR KING-SUN FU
Twelve years have gone by since the passing away of Professor King-Sun Fu, an eminent scholar and a pioneer in the field of pattern recognition, computer vision, and machine intelligence. This volume is dedicated to his memory by his friends and students as a testimonial to his profound and enduring influence. I am honored to be asked to write a memorial article. Here I will give a biography* of King-Sun as well as a short addendum t o the bibliography of his published works compiled in 1985. King-Sun was born on October 2, 1930, in Nanking, China, the second son of his parents, General and Mrs. Tzao-Jen Fu. The Fu family’s native city, however, is Hangzhou, Zhejiang. He pursued his middle school education at the Chinese Air Force Youth Preparatory School. After his father died in 1949, his mother took him and two younger brothers to Taiwan, where he matriculated in the National Taiwan University in the Fall of 1949. He was one of the top students in the Electrical Engineering Department. Not only did he excel in mathematics and engineering subjects, but he also cultivated interests in classical music and literature. Perhaps the latter pursuit laid the foundation for his own prolific writing in serial journals and scientific books. His writing was distinct: clearly expressive and succinct. The demands of academia did not serve to stifle his personal exuberance and vitality in life, however, and, in finding a balance to his studies, King-Sun was also active on the basketball and volleyball teams in the school. He was indeed a versatile student on the campus. After graduating from the National Taiwan University with a B.S.E.E. degree in 1953 and completing one year of ROTC training, King-Sun received a graduate assistantship from the Electrical Engineering Department of the University of Toronto, Canada, and went there in September 1954 to start his graduate study. He wrote his master’s degree thesis on dynamic analysis of large electric machines and received his M.A.Sc. degree in the summer of 1955. In September 1955 he transferred to the University of Illinois, Urbana, IL, USA, for his doctoral study in Electrical Engineering, and completed his Ph.D. dissertation in network theory on “An Approximation Method for Both Magnitude and Phase by Rational Functions” in February 1959 under Professor M. E. Van Valkenburg. During his three and a half years at the University of Illinois, King-Sun became immensely interested in statistical methods, information theory, abstract algebra, and modern analysis. He was a top student in Professor J . L. Doob’s course *Portions of the biography are reprinted, with permission, from IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-8, No. 3, pp. 291-294, May 1996. @ 1986 IEEE.
Xiii
xiv
A Memorial to the Late Professor King-Sun Fu
on stochastic processes which was intended for mathematics graduate students only. This extraordinary background paved the way for his later research in sequential methods of statistical pattern recognition and machine learning. On the Urbana campus, King-Sun fell in love with Miss Viola Ou, then a graduate student in library science. They were married in Urbana, Illinois, on April 7, 1958. After receiving his Ph.D. degree from the University of Illinois in 1959, King-Sun worked for a year and a half as a research engineer at Boeing Airplane Company, Seattle, Washington, from February 1959 to August 1960. He also taught as a special lecturer at the Seattle University during the Spring Semester of 1960. In September 1960 he accepted a faculty position a t Purdue University, West Lafayette, Indiana, as an assistant professor in the School of Electrical Engineering. The following semester he was selected by Purdue to be a visiting scientist with the Research Laboratory of Electronics at Massachusetts Institute of Technology from February to June 1961. During that summer he was with the IBM Thomas J. Watson Research Center at Yorktown Heights, NY. After he returned to Purdue University in September 1961, he began to pursue his research in pattern recognition and machine intelligence - a field in which he played a prominent role in its development during the next quarter of a century. He became a n associate professor in September 1963 and was promoted to the rank of professor of electrical engineering in September 1966. In 1967 he was a visiting professor of electrical engineering and computer science at the University of California, Berkeley. He was the assistant head for research a t the School of Electrical Engineering a t Purdue from 1969 to 1972. In 1972 he was awarded a prestigious Guggenheim fellowship and was a visiting professor of electrical engineering a t both Stanford University and the University of California, Berkeley. After returning to Purdue, he established the Advanced Automation Research Laboratory in the School. He was named the Goss Distinguished Professor of Engineering at Purdue University in 1975. During the Fall of 1984 he, along with other colleagues at Purdue, initiated the highly innovative program of research in intelligent manufacturing. This program resulted in the startup of the National Science Foundation Engineering Research Center for Intelligent Manufacturing Systems during early 1985. King-Sun was the founding director of the Center. At the peak of his career, his sudden death in April 1985 was a tremendous loss to our scientific and engineering community. During the earlier years he first focused his study on statistical pattern recognition and learning systems. From 1961 t o 1970 he and his students developed sequential methods for feature selection and pattern recognition, non-parametric procedures for pattern classification, a stochastic approximation approach to learning control systems, and stochastic and learning automata. His first research monograph entitled “Sequential Methods in Pattern Recognition and Machine Learning” was published in 1968. By the late 1960’s he began his unique research on syntactic pattern recognition, which was introduced by the earlier efforts of Murray Eden, R. Narasimhan, R. A. Kirsch, Robert S. Ledley, and Alan Shaw. King-Sun initiated and launched in-depth studies on stochastic context-free
A Memorial to the Late Professor King-Sun Fu
xv
programmed languages and stochastic syntax analysis for pattern recognition and image analysis. His book, “Syntactic Methods in Pattern Recognition”, was published in 1974. In the ensuing years he and his students made the greatest and foremost impact on syntactic pattern recognition research. His school developed fundamental methodologies of stochastic error-correcting syntax analysis, error-correcting parsers for formal languages and, in particular, for attributed and stochastic tree grammars, and error-correcting isomorphisms of attributed relational graphs for pattern recognition. The syntactic methods for texture analysis, shape recognition, and image modeling were introduced in the late 1970’s, and the three-dimensional plex grammar in 1984. Attributed grammars were developed from the viewpoint of combining syntactic and statistical pattern recognition. In the meantime, the contextual information was also introduced into statistical pattern recognition. The unification of both syntactic and statistical approaches was always in his thoughts. Inference procedures of context-free programmed grammars, multi-dimensional grammars, transition network grammars, and stochastic tree grammars were developed one after another in the late 1970’s through early 1980’s. It is probably appropriate to say that all these constitute what we may call Fu’s theory of syntactic pattern recognition. His treatise, “Syntactic Pattern Recognition and Applications”, published in 1982, made this subject material more easily understandable to researchers and practitioners in various disciplines. King-Sun and his colleagues also made important contributions to pattern recognition applications. His work on pattern classification of remotely sensed agriculture data (1969) and earth resources (1976) is considered classic in the field. During the mid 1970’s through the early 1980’s, his biomedical pattern recognition research extended to chest radiographic image analysis, automatic recognition of irradiated chromosomes, nucleated blood cell classification, and Pap smear and cervical cell image analysis and classification. The Moayer-Fu paper on fingerprint pattern recognition based on the syntactic approach received the 1976 outstanding paper award of the IEEE Transactions on Computers. His work on seismic signal discrimination and bright spots detection appeared in 1982 and 1985. His research on industrial automatic inspection and computer vision included IC chip inspection (1980), metal surface inspection (1984), and inspection of industrial assemblies (1985). An expert system was developed by his group for the assessment of structure damages caused by earthquakes (1983). Since the late 1970’she envisioned the importance of integrated and special computer architectures and parallel algorithms for pattern recognition, image processing, and database management. This led t o his works in 1980’son parallel parsing of tree languages, query languages for image database systems, and VLSI implementation of parallel parsing algorithm and hierarchical scene matching. In the meantime, his research on three-dimensional object representation and shape description, orientation estimation, overlapping workpiece identification, knowledge organization, and robotic vision for path planning laid the foundation for the establishment of
xvi
A Memorial to the Late Professor King-Sun Fu
the Engineering Research Center on Intelligent Manufacturing Systems by Purdue University and the National Science Foundation in 1985. As mentioned earlier, King-Sun was the chief architect and the first director of the research center. He wrote six books, edited or co-edited eighteen books, authored or co-authored forty-four book chapters, and one hundred sixty-two serial journal papers. In addition, he authored and co-authored two-hundred forty-eight conference papers. Seventy-two Ph.D. dissertations were completed under his supervision. His activities in professional societies started in 1965-67 as the chairman of the Institute of Electrical and Electronic Engineers (IEEE) Discrete Systems Committee and the chairman of the Fifth Symposium on Discrete Adaptive Processes. Under his leadership he organized and served as the first chairman (1967-69) of the IEEE Automatic Control Group’s Learning and Adaptive Systems and Pattern Recognition Technical Committee. He was on the official American delegation to the International Conference on Artificial Intelligence, Moscow, USSR, in 1967, and an official American delegate to the 1969 International Federation of Automatic Control (IFAC) International Congress held in Warsaw, Poland. He served on the administrative committee of the IEEE Automatic Control Group (1969-71) and later of the IEEE Control System Society (1974-76), was the chairman of the 1969 IEEE International Convention, a director (for IEEE) of the American Automatic Control Council in 1972, and the general chairman of the 1977 IEEE Conference on Decision and Control. He took part in the IEEE Systems, Man, and Cybernetics Society activities, beginning in 1969 when he served as the chairman of the Adaptive Systems Technical Committee of its predecessor, the IEEE Systems Science and Cybernetics Group, and was on the administrative committee of the Group (1970-72). He became the Cybernetics Technical Committee chairman (1972-76) and then the Society’s Vice President for Technical Committees (1978-79). He was an associate editor of IEEE Transactions on Systems, Man, and Cybernetics (1969-1985). In order to provide an international forum to promote advances in pattern recognition, he and the contemporary leaders in the field organized the first International Conference on Pattern Recognition in Washington, DC, in 1973, for which he served as chairman. The biannual conferences evolved into the formation of the International Association for Pattern Recognition (IAPR) by 1976. He was elected to be its president for 1976-78, a member of its executive committee (1976-80), chairman of its long range planning committee (1979-81), and a member of its governing board (1976-85). (In memory of his distinctive contributions, the IAPR has established since 1986 the “K. S. Fu Award, to be given to a distinguished contributor in the field once every two years.) In the meantime, he reorganized the Pattern Recognition Committee into the Machine Intelligence and Pattern Analysis Technical Committee (later renamed as the Pattern Analysis and Machine Intelligence Technical Committee) of the IEEE Computer Society and was its first chairman (1974-77). He was an associate editor of IEEE Transactions on Computers during 1977-78. His initiative led to the founding of the IEEE Transactions on Pattern
A Memorial to the Late Professor King-Sun Fu xvii
Analysis and Machine Intelligence, and he served as its first editor-in-chief (1978-81) as well as a member of the editorial committee (1981-85). In addition, he served on editorial boards (editor, associate editor, and advisory board) of many other scientific journals. These include Pattern Recognition (associate editor, 1971-85); International Journal on Information Sciences (associate editor, 1970-82; editor 1982-85); Journal of Cybernetics of the American Society of Cybernetics (editorial board, 1970-85); International Journal on Computer and Information Sciences (advisory editor, 1971-85); Journal of Information Processing (editorial advisory committee, 1978-81); Journal of Analytical and Quantitative Cytology (editor, 1978-85); International Journal of Fuzzy Sets and Systems (advisory editor, 1979-1985); International Journal of Cybernetics and Systems (advisory board, 1980-85); Computer Vision, Graphics and Image Processing (associate editor, 1981-85); Pattern Recognition Letters (advisory editor, 1982-85); IEEE Transactions on Geoscience and Remote Sensing (associate editor, 1984-85); IEEE Computer (editorial board, 1983-85); and Journal of Parallel and Distributed Computing (editorial board, 1984-85). He chaired and co-chaired the Engineering Foundation Conference on Pattern Information Processing in 1972, on Algorithms for Image Processing in 1976, and on Algorithms for Image and Scene Analysis in 1978. He was Program Chairman of the 1975 IEEE-ACM Conference on Computer Graphics, Pattern Recognition and Data Structure, of the 1978 IEEE Computer Society Conference on Pattern Recognition and Image Processing, and of the 1979 IEEE Computer Society COMPSAC Conference. He chaired the 1980 IEEE Picture Data Description and Management Workshop, and initiated and chaired the IEEE Computer Society Workshop on Computer Architecture for Pattern Analysis and Image Database Management in 1981 and 1983. He was the general chairman of the 1984 IEEE Workshop on Language for Automation; Honorary Chairman of the 1984 IEEE Workshop on Visual Languages at Hiroshima, Japan; and general chairman of the 1985 IEEE International Conference on Robotics and Automation. His leadership was well recognized for organizing many international joint seminars and workshops. He served as the coordinator of the NSF supported U.S.-Japan Seminar on Learning Processes in Control Systems at Nagoya, Japan, in 1970 and the Second U.S.-Japan Seminar on Learning Control and Intelligent Control at Gainesville, Florida, in 1973; as the vice-coordinator of the US-Japan Seminar on Fuzzy Sets and Their Applications at Berkeley, California, in 1974; as a co-director of the NATO Advanced Study Institute on Pattern Recognition and Applications in 1975; as the co-chairman of the Dahlem Konferenzen on Biomedical Pattern Recognition and Image Processing at Berlin, West Germany, in 1979; and the coordinator of the NSF sponsored U.S.-France Seminar on the Applications of Pattern Recognition and Machine Intelligence to Automatic Testing at Alexandria, Virginia, in 1983. As a guest editor, he helped put together the special issue on “Feature Extractions and Selection in Pattern Recognition” of IEEE Transactions on Computers, September 1971; the special issue on “Syntactic Pattern Recognition” of Pattern Recognition, Part One,
xviii
A Memorial to the Late Professor King-Sun Fu
November 1971, and Part Two, January 1972; special issue on “Pattern Recognition” of IEEE Computer, May 1976; and on “Robotics and Automation” of IEEE Computer, December 1982. He was on the IEEE Computer Society Governing Board (1978-81), and served as the Society’s Vice President for Publications and a member of the Executive Committee (1982-83) and Fellow Committee (1972-76, 1984-85). He served as the president of the Chinese Language Computer Society (1983-85). He was the Vice President (1984-85) and President-elect of the then newly formed IEEE Robotics and Automation Council. He was on the IEEE Fellow Committee (1977-79), IEEE TAB Awards and Recognition Committee (1979-78), American Federation of Information Processing Societies (AFIPS) Harry Goode Memorial Award Committee chairman (1982-85), American Society of Engineering Education (ASEE) Award Committee (1983-85), and IEEE Award Board, Education Medal Committee (1983-85). King-Sun was literally showered with honor in recognition of his monumental research contributions and contributions to the profession. He was elected a Fellow of the Institute of Electrical and Electronic Engineers in 1971. He was elected a member of the National Academy of Engineering in 1976 and a member of the Academia Sinica in 1978. He served on the National Science Foundation’s Advanced Automation Panel in 1973, Automation Research Council for 1972-78, and Committee on Cytology Automation of the National Institutes of Health, 197881. Among the many awards which he received are the Herbert N. McCoy Award in 1976 for Contributions to Science; the American Society of Engineering Education Senior Research Award in 1981 for outstanding loyalty and contributions as a pioneer in the contemporary engineering disciplines of pattern recognition, image processing and machine intelligence; IEEE Education Medal in 1982 for contributions to engineering education through inspired teaching and research in computer engineering, system theory and pattern recognition; American Federation of Information Processing Societies Harry Goode Memorial Award in 1982 in recognition of his contributions in pattern recognition and its applications and his leadership in education in information processing; Chinese Institute of Engineers - USA (CIE-USA) Achievement Award in 1983 for leadership in engineering education and contribution to pattern recognition; and the IEEE Centennial Medal in 1984. King-Sun helped Taiwan, China, with his scientific advice in various ways. Over a period of fifteen years (1970-1985), he gave invited lectures there almost every year. He was the Program Chairman of the Academia Sincia International Computer Symposium at Taipei in 1978. He helped found the Institute of Information Science, Academia Sinica and was instrumental in establishing the Microelectronics and Information Science and Technology Research Center at the National Chiao Tung University, Taiwan, in 1984. He nurtured a number of young scholars who have become the principal researchers and engineers for the vital development of computer engineering and information science in Taiwan. Likewise he educated a number of scholars from mainland China during 1979 through 1985. He was invited
A Memorial to the Late Professor King-Sun Fu xix
to give lectures to the Institute of Automation, Chinese Academy of Sciences in 1979, and was honored as a Distinguished Visiting Professor of Beijing University, an Honorary Professor of Tainghua University, Beijing, and an Honorary Professor of Fudan University, Shanghai. King-Sun took great pride in his two sons, Francis and Thomas, and one daughter, June. When they were young he always spent his leisure time playing ball or other sports with them. Together with Mrs. Fu, he provided their children with the best education in the home and at school. They are all grown up now: Francis is a computer engineer; Thomas, an oceanographer; and June, a biochemist. They have their own accomplishments in their respective professions. In spite of his overwhelming achievements, King-Sun was a modest man with great sensitivity. Considerate and generous to his friends and students, he exemplified the notion of greatness both professionally and in his personal relationships with others. Standing alongside his outstanding contributions to the scientific world, King-Sun’s great wisdom and human warmth will always be remembered. Ching-Chung Li Department of Electrical Engineering University of Pittsburgh Pittsburgh, PA 15261, USA January 1997
Addendum to “A Bibliography of Published Works of the Late Professor King-Sun Fu,” I E E E Trans. Pattern Analysis and Machine Intelligence, Vol. PAMI-8, No. 3, pp. 295-300, May 1986. At the time of compiling the bibliography, eight items were missed (seven of which appeared later in 1986 and 1987). To make his bibliography complete, these are listed below.
Books: K. S. Fu, T-X Cai and G-Y Xu, Artificial Intelligence and Its Applications, (in Chinese) (Beijing: Tsinghua University Press, 1987).
Book Chapters: K. S. Fu, M. Ishizuka and J. T. P. Yao, “Application of Fuzzy Sets in Earthquake Engineering,” in Fuzzy Set and Possibility Theory, R.R. Yager, Ed. (New York: Pergamon Press, 1982) 504-523. Serial Journal Articles E. K. Wong and K. S. Fu, “A Hierarchical Orthogonal Space Approach t o ThreeDimensional Path Planning,” I E E E Journal of Robotics and Automation, Vol. RA-2, March 1986, 42-53.
xx A Memorial to the Late Professor King-Sun Fu
W. C. Lin and K. S. Fu, “A Syntactic Approach to Three-Dimensional Object Recognition,” IEEE Trans. Syst., Man, Cybern., SMC-16, May/June 1986, 405-422.
H-S Don and K. S. Fu, “A Parallel Algorithm for Stochastic Image Segmentation,” IEEE Trans. Pattern Anal. Machine Intell., PAMI-8, Sept. 1986, 594-603.
M. A. Eshera and K. S. Fu, ‘&AnImage Understanding System Using Attributed Symbolic Representation in Inexact Graph-Matching,” IEEE Trans. Pattern Anal. Machine Intell., PAMI-8, Sept. 1986, 604-618. K-Y Huang and K. S. Fu, “Decision-Theoretic Approach for Classification of Ricker Wavelets and Detection of Seismic Anomalies,” IEEE Trans. Geoscience and Remote Sensing, GE-25, March 1987, 118-123. S. Basu and K. S. Fu, “Image Segmentation by Syntactic Method,” Pattern Recognition, 20, No. 1 (1987), 33-44.
CONTENTS
Preface to the Second Edition
V
Preface
vii
Foreword
ix
A Memorial to the Late Professor King-Sun FU
xiii
Contents
xxi
PART 1. BASIC METHODS IN PATTERN RECOGNITION 1.1 Cluster Analysis and Related Issues Richard C. Dubes
3 33
1.2 Statistical Pattern Recognition
Keinosuke Fukunaga 61
1.3 Syntactic Pattern Recognition
Kou- Yuan Huang 1.4 Neural Net Computing for Pattern Recognition
105
Yoh-Han Pao 1.5 On Multiresolution Wavelet Analysis using Gaussian Markov Random
Field Models C. H. Chen and G. G. Lee
143
1.6 A Formal Parallel Model for Three-Dimensional Object Pattern Representat ion
183
P. S. P. Wang PART 2. BASIC METHODS IN COMPUTER VISION 2.1 Texture Analysis
207
Mihran Tuceryan and Anil K. Jain 2.2 Model-Based Texture Segmentation and Classification
249
R. Chellappa, R. L. Kashyap and B. S. Manjunath 2.3 Color in Computer Vision: Recent Progress
283
Glenn Healey and Quang- Tuan Luong 2.4 Projective Geometry and Computer Vision
Roger Mohr xxi
313
xxii
Contents
2.5 3-D Motion Analysis from Image Sequences using Point Correspondences
339
John. J. Weng and Thomas S. Huang 2.6 Signal-to-Symbol Mapping for Laser Rangefinders Kenong W u and Martin D. Levine
387
2.7 3-D Vision of Dynamic Objects S.-Y. Lu and Chandra Kambhamettu
425
PART 3. RECOGNITION APPLICATIONS 3.1 Pattern Recognition in Nondestructive Evaluation of Materials
455
C. H. Chen 3.2 Discriminative Training - Recent Progress in Speech Recognition
473
Shigeru Katagiri and Erik Mcdermott 3.3 Statistical and Neural Network Pattern Recognition Methods for Remote Sensing Applications
507
Jdn Atli Benediktsson 3.4 Multi-Sensory Opto-Electronic Feature Extraction Neural Associative Retriever H.-K. Liu, Y.-H. Jan, Neville I. Marzwell and Shaomin Zhou 3.5 Classification of Human Chromosomes - A Study of Correlated Behavior in Majority Vote
535
567
Louisa Lam and Ching Y. Suen 3.6 Document Analysis and Recognition by Computers Yuan Y. Tang, M. Cheriet, Jiming Liu, J. N . Said
579
and Ching Y. Suen 3.7 Pattern Recognition and Visualization of Sparsely Sampled Biomedical Signals
613
Ching-Chung Li, T. P. Wang and A. H. Vagnucci, M.D. 3.8 Pattern Recognition and Computer Vision for Geographic Data Analysis
625
F. Cavayas and Y. Baudouin 3.9 Face Recognition Technology
667
Martin Lades PART 4. INSPECTION AND ROBOTIC APPLICATIONS 4.1 Computer Vision in Food Handling and Sorting Hordur Arnarson and Magnth Asmundsson
687
Contents
4.2 Approaches to Texture-Based Classification, Segmentation and Surface Inspection Matti Pietakainen, Timo Ojala and Olli Silven 4.3 Context Related Issues in Image Understanding L. F. Pau 4.4 Position Estimation Techniques for An Autonomous Mobile Robot-A Review Raj Talluri and J. K. Aggarwal 4.5
Computer Vision in Postal Automation G. Garibotto and C. Scagliola
4.6 Vision-Based Automatic Road Vehicle Guidance Dieter Koller, Quang-Tuan Luong, Joseph Weber and Jitendra Malilc
xxiii
71 1 737
765 797 817
PART 5. ARCHITECTURE AND TECHNOLOGY 5.1 Vision Engineering: Designing Computer Vision Systems Rama Chellappa and Azriel Rosenfeld
857
5.2 Optical Pattern Recognition for Computer Vision David Casasent
869
5.3 Infra-Red Thermography: Techniques and Applications M. J . Varga and P. G. Duclcsbury
891
5.4 Viewer-Centered Representations in Object Recognition: A Computational Approach Ronen Basri 5.5 Video Content Analysis and Retrieval Hongjiang Zhang 5.6 VLSI Architectures for Moments and Their Applications to Pattern Recognit ion Heng-da Cheng, Chen-Yuan Wu and Jaguang Li
Index
925 945
979
1003
PART 1
BASIC METHODS IN PATTERN RECOGNITION
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 3-32 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 1.1I CLUSTER ANALYSIS AND RELATED ISSUES
RICHARD C. DUBES Department of Computer Science, Michigan State University, East Lansing, M I 48824-1087, U S A This chapter explains how cluster analysis organizes information in applications such as Computer Vision and Pattern Recognition. Information is represented as points in multidimensional feature spaces where each coordinate represents a measurement. Some tools from exploratory data analysis are discussed, with an emphasis on linear projections derived from the covariance matrix. Two types of clustering are reviewed - hierarchical and partitional. Hierarchical clustering leads to nested partitions of the data. SAHN algorithms for hierarchical clustering are defined and some of the common characteristics are explained. Partitional clustering arranges data in separate clusters, as with the K-Means algorithm. The chapter ends with a discussion of validation that centers on external and internal tests of validity and tests for the number of clusters. A bibliography is provided for further reading. Keywords: Proximity, exploratory data analysis, projection, hierarchies, dendrograms, K-means, cluster validity, algorithms.
1. Introduction
Organizing information is an essential part of any learning task. Cluster analysis is the formal study of methods and algorithms for objectively organizing numerical data. One finds cluster analysis in the literature of almost all disciplines, including engineering, statistics, psychology, sociology, biology, astronomy, business, medicine, archeology, psychiatry, geography, anthropology, economics, and computer science, to name a few. No single definition of “cluster” is universally accepted. Cluster analysis includes the process of “looking” at data, known as exploratory data analysis, which is a tool for igniting creativity and suggesting alternative models for the data. This chapter views cluster analysis as the initial step in organizing numerical data so as to abstract the essence of the data and describe the data as simply as possible. The discussion is informal and omits mathematical proofs. Computational issues are mentioned only briefly. Cluster analysis is sometimes called “unsupervised learning” because only actual observations affect the data organization. By contrast, pattern recognition uses a priori labels to “learn” the parameters of models for the categories, or pattern classes, present in the data. A pattern recognition algorithm seeks to define a good 3
4
R. C. Dubes
decision rule for labeling patterns of unknown origin, based on information gleaned from labeled patterns. The algorithms of cluster analysis make no decisions but fit various structures, such as partitions and hierarchies, to the data. Although the literature in several fields of application carry papers on clustering, the only journal exclusively devoted to the methodology of clustering is the Journal of CZasszfication, published by the Classification Society of North America since 1984. Some general books on the topic are [1,2,3,4,5]. One application, image segmentation, will help explain the context of this chapter. Each pixel, or each small sub-image, is characterized by a set of numbers [6]. Candidates for the numbers are co-occurrence features?gray-level intensities, measures of fractal dimension? estimates of Markov random field-parameters?and other indices popular in the computer vision community. Cluster analysis labels each pixel or sub-image so that regions from the same underlying class, such as land-use category in remote sensing, have the same label and regions from different classes have different labels. The organization is done in the feature space, in which each axis represents one of the measurements. One must develop faith in the clustering algorithm and must formally validate the results. The cluster labels are then transferred to the image for interpretation. This chapter will concentrate on the process of assigning the cluster labels, and not on the choice of features. Figure 1 is an overview of the most important aspects of a typical cluster analysis. Once data have been gathered and some type of exploratory data analysis has been applied to evaluate the data representation, one can apply clustering tendency algorithms to ensure that the data are not random. This avoids the embarrassment and futility of imposing sophisticated procedures for analyzing data that contain no clusters [4]. This chapter covers two types of clustering: hierarchical clustering creates a complete hierarchy, or nested sequence of partitions; partitional clustering creates one partition of the data. Omitted due to lack of space are treatments of fuzzy clustering [7,8], conceptual clustering [9,10], and any mention of neural nets [ l l ] . The validation step, in which one applies statistical tests to ensure that the structure recovered from the clustering algorithm is “real” in some sense, is the most difficult step of the entire process. The interpretation of the results requires experience and interaction with the expert in the field of application. The entire process, or any part of it, may need to be repeated until one is satisfied with the result. All this effort should reveal the underlying structure of the data so that sharper and more definitive studies can be planned. 2. Data
The procedures and algorithms of cluster analysis are geared towards the type and scale of the data, so this section begins by reviewing some basic definitions about data in Section 2.1 and about normalization in Section 2.2. The most important
1.1 Cluster Analysis and Related Issues 5
I
Xrtrix
Clurtrring
Birrrrchicil
Prrtitionrl Clurt-ring
Clurtrring
I A
Partition
Vilidrtion
Intrrprrtrtion
Fig. 1. Methodology of cluster analysis.
characteristic of a set of data is its dimensionality, which is briefly explained in Section 2.3. 2.1. Representing Numerical Information
2.1.1. Scale and type Data occur in several types and scales. The simplest unit of data is a number. Vectors and matrices are built from numbers, but all numbers should be on the same scale and have the same type. The scale of a number refers to its relative significance. Usually recognized are nominal, ordinal, interval, and ratio scales. A number on a nominal scale is simply a numerical tag, such as “1” for “Ford”, “2” for “Chevy”, and “3” for “Toyota”. Numbers on an ordinal scale have significance only in their relative positions. Numbers on nominal and ordinal scales are sometimes called qualitative, whereas numbers on the interval and ratio scales are called quantitative. The gap between numbers has significance when the numbers are on an interval scale. If, for example, a person were asked to state his preference for soda on a scale of “1” to “10” with “10” being most preferred, then responses 1,5,9 and 1,2,9 would have different meanings on an interval scale, but not on an ordinal scale.
6
R. C. Dubes
The most important data scale in engineering work is the ratio scale, which is the interval scale with a natural zero. Data from sensors, and numbers which can be placed on the real line are examples. For example, distance is measured on a ratio scale. Doubling the distance between two towns means using twice as much gas to get between them, whatever the unit of distance. Temperature, on the other hand, is an interval measurement because its significance depends on the unit. Measuring in degrees Kelvin is a ratio-scale measurement, while temperature in degrees Celsius is an interval-scale measurement. Data type refers t o degree of quantization. The three types recognized here are binary, or two-valued, discrete, or multi-valued, and continuous, or data taken from the mathematical real line. Binary data, also called dichotomous data, are for situations where the possible responses are (“yes”, “no”), or (“on”, “off”). A discrete type has a small number of values, where “small” depends on the situation. Since instruments have finite resolution, all data measured in the real, as opposed to the mathematical, world are discrete. Calculus ordinarily requires that data be continuous. Thus, we often assume data are continuous and ignore the unpleasant reality. 2.1.2. Patterns and proximity Whatever the scale and type, data are collected in one of two basic formats, called a pattern matrix and a proximity matrix. A pattern matrix represents each object under examination as a set of measurements. Each measurement is called a feature and a pattern is a set of feature values measured on an object. The set of d measurements form the feature space, each feature corresponding to one orthogonal axis. A pattern matrix is thus an n x d matrix, where n is the number of patterns and d is the number of features. The notation for the j t h feature of pattern i will be xij and the ith pattern itself will be denoted by the column vector xi. Letting superscript T denote matrix transpose, xi = [ZZl xi2
. . . ZZd]T .
We require n >> d and think of the patterns as a swarm of n points floating in a d-dimensional space. The transpose of the n x d pattern matrix X is written as:
A pattern matrix, called the speaker data, that consists of 40 rows (patterns) and 5 columns (features) will be used to demonstrate several procedures. A listing is given in Table 1. Each pattern represents a spoken phrase and each feature represents a measurement taken on the spectrum derived from the phrase. There are eight categories, one for each person involved in the study. Eight male speakers spoke the same phrase five times; patterns 1-5 are from speaker number 1, patterns 6-10 from speaker number 2 , and so forth. The category labels are on a nominal scale
1 . 1 Cluster Analysis and Related Issues 7 Table 1. Speaker data. Features
Category
Pattern
1
1 2
107.0000
1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5
105.5000
18.0000 15.0000
87.0000 90.0000
94.6500 96.4750
89.0000 91.0000
3
106.5000
13.0000
88.0000
94.9500
93.0000
4 5
102.5000
13.0000 17.0000
85.0000 85.0000
91.1250 93.1250
90.0000 87.0000
17.0000 13.0000
97.0000 97.0000
103.3000 103.3500
100.0000 103.OOOO
6 7
106.0000 118.0000 116.5000
8
123.0000
16.0000
100.0000
105.7250
104.0000
9 10
119.0000
15.0000 16.0000
98.0000 101.oooo
103.5000 105.1750
104.0000 104.0000
11
103.5000
83.9000
81.0000
104.OOOO 104.5000
10.0000 1 2.oooo
78.0000
12
81.0000
86.9750
81.0000
10.0000
77.0000
91.0000
11.0000 11.0000
84.5250 86.3000
13 14
122.0000
104.5000 100.0000
15 16
109.0000
16.0000
78.0000 79.0000 82.0000
17 18
113.0000 111.0000
22.0000 16.0000
85.0000 85.0000
19
113.0000
15.0000
78.0000
94.0250 92.6750 94.3250
20 21
112.0000 122.5000
20.0000 30.0000
84.0000 87.0000
93.5500 89.6500
5
22
119.0000
81.0000
87.0750
93.0000 90.0000
5
23
87.0000
92.1750
97.0000
5 5 6 6 6 6 6 7 7 7 7 7 8 8 8 8 8
24 25
122.5000 125.5000
28.0000 24.0000 32.0000
87.0000 83.0000
94.3000 89.7000
90.0000 96.0000
72.0000
78.9000 79.0250
74.0000
87.0000 91.6000
83.0000 84.0000 87.0000 90.0000 90.0000 100.0000 90.0000
26
119.0000 95.5000
23.0000 23.0000
27
94.0000
28 29
103.0000
15.0000 24.0000
76.0000 77.0000
82.7500
78.0000 78.0000
106.5000 102.5000
28.0000
76.0000
84.2750
78.0000
19.0000
75.0000
79.9750
84.0000
31 32
111.oooo 110.0000
25.0000 26.0000
77.0000 78.0000
86.7750 89.7500
82.0000 78.0000
33
109.0000 109.0000
22.0000 23.0000
78.0000
87.5250
84.0000
34
110.5000 107.0000
22.0000 16.0000
89.3250 88.7750
83.0000
35 36
78.0000 81.0000 79.0000
87.3250
82.0000 81.0000
37
107.0000
16.0000
77.0000
86.3500
81.0000
38
105.0000
20.0000
78.0000
80.0000
39 40
111.0000 110.0000
18.0000 15.OOOO
80.0000 85.0000
87.6500 87.5000
30
86.2000
84.0000 88.0000
8
R. C.Dubes
while the features are on a ratio scale since all have natural zeros. The features are assumed to be continuous. A proximity matrix is a square, symmetric matrix. Its rows and columns both correspond to patterns, or t o features. The (2, j) entry contains an index of proximity that denotes the degree of closeness or alikeness between the objects corresponding to row i and column j . If the proximity matrix is a dissimilarity matrix, then the larger the entry (i, j ) , the less items i and j resemble one another, as when Euclidean distance measures the proximity between two patterns. In a similarity matrix, a large value indicates a close resemblance between the two objects, as when a correlation coefficient measures the proximity between two features. 2.1.3. Indices of proximity Clustering algorithms require that an index of proximity he established between all pairs of items. Anderberg [l]defines several such measures of proximity. This chapter covers only the case when proximities are computed from a pattern matrix. An index of dissimilarity d(q, r ) between patterns xq and x, is a real-valued function satisfying the following for all q and r .
An index of similarity satisfies the first two conditions, hut replaces the third with 4%Q ) 2 my{d(q, .)I. The most common index of dissimilarity in engineering work is the Minkowski metric (2.4). (llm)
.>(jcr,
4%
=
b q j - Z,jIrn
)
(2.4)
See [12] for a full discussion of this and related indices of dissimilarity. Common parameter values are m = 2, or Euclidean distance, m = 1, or Manhattan distance, also called taxicab and city-block distance, and m + 00, or “sup” distance. The Euclidean distance must be carefully separated from the squared Euclidean distance. The Minkowski metric is for continuous data on a ratio scale and is itself on a ratio scale. The Minkowski metric satisfies the triangle inequality.
d ( q , r ) 5 d ( q , s)
+ d(s, r ) for all ( 4 , r, s)
(2-5)
A common index of similarity between features u and w is the sample correlation coefficient.
1 . 1 Cluster Analysis and Related Issues 9
The sample means, mu and m,, and the sample standard deviations, s, and s,, are defined in (2.7). The sample correlation coefficient indicates the degree of linear dependence between two features, with a value of 0 indicating linear independence. The absolute value measures the degree of resemblance between two features since negative and positive correlations with the same magnitude have the same interpretation. A number of indices of proximity for binary data, such as the simple matching coefficient and the Jaccard coefficient, have been proposed [1,4]but are not discussed here. 2.2. Normalization and Standardization
A pattern matrix is normalized t o equalize the contributions of the features to a projection or a clustering. Milligan and Cooper [13] noted that some normalizations that appeal to our intuition may affect performance in unexpected ways. In this section, a “*’, superscript will denote the raw, or un-normalized, data, as in x:~. Throughout the remainder of the chapter, the context must make clear which normalization has been applied. One common normalization is t o move the origin of the feature space t o the grand mean vector, which is the vector of sample means for the d features. The sample mean m3,and the sample standard variance 532, for feature j are:
The origin is shifted to the grand mean vector by defining, for all i and j = 1, 2, . . . , d: 2 . . - xt. - m j . 23 - a j
=
1, 2, . . . , n (2.8)
This shifting of the origin does not affect Euclidean distance and merely simplifies equations. The z-score normalization divides each feature by its standard deviation s j in (2.9). x*. - m x . . - 23 3 (2.9) 23 sj
This normalization stretches or squeezes each of the coordinate axes t o equalize the spreads along all axes. A third normalization, called a range method, reduces all features to the range [0,1] by subtracting the smallest value in each column and dividing by the range of each column. x.. ” -
xij - min{xij} max{xij} - min{xij}
The min and max are taken over the j t h column of the pattern matrix. Miligan and Cooper [13] found that the range normalization outperformed the z-score normalization when extracting the structure of the data.
10 R. C. Dubes
2.3. Dimensionality
The dimensionality of data refers to t,he number of independent parameters required t o describe the data. These parameters for each pattern are often taken to be the feature values on the d coordinate axes. However, data can sometimes be projected to fewer than d dimensions, as discussed in Sections 3.1 and 3.2. The number of dimensions in the target space is called the intrinsic dimensionality [14,15] since it represents the dimensionality suggested by the data themselves and provides a parsimonious characterization of the data. Dimensionality has taken on new importance with the emergence of fractal geometry as a cross-disciplinary field of study [16]. The true dimensionality of the data, whether fractional or integer, may become an important characteristic of the data in exploratory data analysis. The estimation of fractal dimensionality [17] is beyond the scope of this paper. 3. Exploratory Data Analysis
Cluster analysis includes a variety of heuristic procedures for getting to know data. Whereas clustering algorithms strive to be objective and quantitative, exploratory data analysis is purposefully subjective. The goal is to use whatever tools are available t o look a t the data, with emphasis on graphs, charts, projections, and any graphical representation that assists the visual system and nudges intuition. Everitt [18] describes several techniques that go beyond simple graphing of functions. Recent advances in graphical computer displays have made the techniques of exploratory data analysis widely available. Two that I have found useful are the S program [19] originally developed at AT&T and the MacSpin program for the Macintosh.” A few standard techniques are explained in this section to indicate the flavor of exploratory data analysis. 3.1. Linear Projections
Why represent d-dimensional data in two or three dimensions? One reason is to be able to see the data. A second reason is to simplify the data by eliminating redundancy and isolating the important characteristics of the data. The process of representing data in a new space is sometimes called ordination. No two- or three-dimensional representation can fully capture the intricacies and complexity of, say, ten-dimensional data. A linear projection does little “violence” to the data since relative distances are preserved and certain geometrical characteristics are maintained. Several schemes for representing data have been proposed [20], including discriminant analysis [4]. The transformation discussed here has been called the princzpal component, the Karhunen-Loeve, and, simply, the eigenvector transformation. It is based on the eigenvalues and eigenvectors of the d x d sample covariance matrix R computed aThe S software is available from Statistical Sciences, Inc., 1700 Westlake, Seattle, WA 98109, USA. The MacSpin program is marketed by Abacus Concepts.
I . 1 Cluster Analysis and Related Issues
11
from the given n x d pattern matrix, X.
Equation (3.1) assumes that either (2.8) or (2.9) has been applied to the raw data. If (2.8) has been applied, R is a covariance matrix whose diagonal entries are the variances of the columns in X. If (2.9) has been applied, R is a correlation matrix meaning that all diagonal entries are unity and off-diagonal entries are correlation coefficients between -1 and 1. Most linear algebra texts show that the eigenvalues of R are solutions for the scalar X to the determinant equation (3.2).
Here, I is a unit matrix of order d x d and 0 is a vector of zeros. If R has full rank of d , and we denote its (necessarily real and non-negative) eigenvalues by 2 2 . . . 2 A d , then the eigenvectors c1, cz, . . . , Cd are orthonormal (column) vectors satisfying, for each j from 1 to d:
(R - X j ) c j
=0
C T C =~ 0 i f j # IC =
1 if j = k .
(3.3) (3.4) (3.5)
Eigenvectors are not unique. The eigenvector transformation from d to m 5 d dimensions is defined by a coefficient matrix whose rows are eigenvectors corresponding to the m largest eigenvalues of R. If xi is a pattern in the original d-dimensional space, its image yz is the m-vector defined in (3.6).
Yi =
If the rank of R is full and m = d, then (3.6) rotates the coordinate axes and decorrelates the features. That is, the covariance matrix computed from the new vectors { y i } is a diagonal matrix whose diagonal entries are the eigenvalues XI, X2, . . . , A d . Thus, the eigenvalues of R can be interpreted as sample variances in the rotated space. The d-dimensional patterns can be projected to two or three dimensions for viewing by setting m to 2 or 3 in (3.6). Why the top rows? A reasonable criterion for projecting data is square-error, meaning that the target space saves as much of
12
R. C. Dubes
the variance as possible. The total variance is the sum of the diagonal elements of R . d
d
j=1
j=1
Since the eigenvalues are ordered by size, it makes sense to save eigenvectors corresponding to the largest eigenvalues. Tou and Heydorn [21] phrase this problem nicely and prove these facts. Figure 2 shows the eigenvector transformation for the speaker data both with pattern numbers and with category labels. Some clustering of the patterns is suggested, especially in Fig. 2(b).
u 7
2
' >
r)
0
4
2
4
:
e8
rq
I
P29
*'
25
.
*
' 6
1
17
s 6
7'
23
u
6
6 u
I
7 -
n
5 21
5
s
s24
5 160
110
110
190
?m
210
fir.,
Fig. 2. Eigenvector projections of speaker data; (a) pattern labels; (b) category labels.
3.2. Nonlinear Projections Nonlinear projections share the goals of linear projections but the details differ. The projection introduced by Sammon [22] illustrates these differences. Sammon's projection begins with a n n x n dissimilarity matrix whose entries are Euclidean distances between all pairs of patterns in the &dimensional feature space. A set of n points is scattered randomly in a portion of the two-dimensional target space, with each point representing one pattern. Given such a configuration of n points, the following stress criterion can be computed.
1.1 Cluster Analysis and Related Issues 13 Here, d(q, r ) is the Euclidean distance between patterns q and r in the ddimensional feature space and D(q, r ) is the Euclidean distance between the points representing patterns q and T in the two-dimensional target space. The sums are over all pairs of patterns 1 5 q 5 r 5 n. Stress is a function of the 2n coordinates of the points in the target space. The idea is to move the points around so as to minimize E . The configuration a t which E is minimum is taken t o be the Sammon projection. A configuration for which D(q, r ) = d(q, T ) for all q and r would be ideal, but it is seldom possible to match all d-dimensional distances in two dimensions. The stress function resembles the criterion function in multidimensional scaling [23]. However, multidimensional scaling begins with ordinal data and Sammon’s method begins with ratio data. Since E uses the difference between two distances, one must have ratio-scale dissimilarities between all pairs of patterns. The denominators normalize the stress. The term d ( r , q ) in the sum on the right weights small distances more heavily than large ones and tends to preserve local structure. The multiplier on the left, which is fixed throughout the minimization procedure, tends to make E insensitive to changes in scale and sample size. Several algorithms exist for minimizing functions of many variables. Sammon [22] proposed a gradient descent algorithm. Simulated annealing has also been tried [24] but the results were disappointing. Whatever the minimization technique, one encounters the usual problems of stopping at local, rather than global, minima and of dependence on the starting configuration. Any minimization should be run several times and the one achieving the smallest stress should be retained. Our implementation requires that the user supply the maximum number of iterations allowed in the gradient descent algorithm and a “magic factor”, which influences the internal stopping criterion. An advantage of a nonlinear projection over a linear projection is its greater flexibility and ability to “see” complex structures. A disadvantage is that a nonlinear method can distort the data unduly and paint a misleading picture. In addition, extra points cannot be easily located in the target space of a nonlinear projection whereas the entire feature space can be projected to two dimensions with (3.6). Figure 3 pictures the speaker data with Sammon’s projection. The gradient descent algorithm was run ten times and the best of the ten runs is exhibited. The stress values for the ten runs ranged from 0.02512 to 0.0009157. The algorithm always stopped because it had achieved a minimum, not because it had reached the maximum number of iterations. Figures 2 and 3 agree on the structure of the data and provide evidence for the conclusion that the clustering by category is real, and not an artifact of an algorithm. One can try to name the variables on the Sammon plot in Fig. 3 from characteristics of the pattern, but no conclusions about correlation between these new variables can be drawn. The last step in the Sammon projection is to apply the eigenvector rotation to decorrelate the variables.
14
R. C. Dubes 8 10
8 -
2 2 8
1 7
e
2
2
a-
0
Ji
-
n
24 z1
n
26
11
2
11
20
in
7
I
R4
0 -
*
o
1
3
0
3
' 3
6 6
1
B
3 6
6 6
Fig. 3. Sammon projection of speaker data; (a) pattern labels; (b) category labels.
3.3. Graphical Procedures Everyone has their favorite graphical representation of data. A few possibilities are mentioned in this section. Both the columns (features) and the rows (patterns) of the pattern matrix can be depicted by graphical methods. Box plots and histograms provide quick looks at the distribution of a feature. Figure 4(a) shows box plots for the first three features of the speaker data. Each box identifies the 25th and 75th percentiles of the data. The horizontal line inside the box is at the median value. The dashed vertical lines are drawn from the minimum value to the 25th percentile and from the 75th percentile to the maximum. A star is drawn to signal that an outlier has occurred according to a particular criterion. Figure 4b gives histograms for the same three features. The number of bins is about filor 7 in this case. The ordinate is the number of feature values in each bin. Box plots are best for quick comparisons among features. Differences in ranges and locations are obvious a t a glance. The fact that feature 3 has a larger right tail than left is also obvious. Histograms provide more detailed information and exhibit the modes in multi-modal data. Graphical representations of the patterns can also be useful. Figure 5 exhibits Chernoff faces [25] for the first 20 patterns from the speaker data. Each face depicts one row in the pattern matrix by associating one characteristic of the face with each feature. Feature 1 is proportional to the area of the face, feature 2, to the shape
1.1 Cluster Analysis and Related Issues 15
-0
T
I
88 -
8 -
0
.
T
0
-
_-(4
-.
1 :,
(b)
Fig. 4. Graphical views of the first three features of the speaker data. (a) Box plots; (b) histograms.
Fig. 5. Chernoff faces of the speaker data.
of the face, feature 3, to the length of the nose, feature 4, to the location of the mouth, and feature 5 is related to the curve of the smile. The faces are arranged in Fig. 5 so that each column contains faces from a different category. Associations
16 R. C. Dubes
among the patterns can be recognized by mentally clustering the faces. Outliers can sometimes be quickly identified. For example, pattern 19 in Fig. 15 might be considered different from other faces. All features were normalized t o fall between 0 and 1 before the faces were drawn. 4. Hierarchical Clustering Clustering data means grouping either patterns or features such that items in the same group are more alike than are items in different groups. The objective is to abstract the essence of the data by isolating groups, or clusters, which explain the data. A hierarchical clustering is a sequence of nested groupings. In some biological applications, the hierarchical structure itself is fitted to the data. In many engineering problems, one searches the hierarchy for a single significant grouping. This section explains the basic mathematical structure for describing hierarchical clustering and presents some standard clustering algorithms. For more complete treatments, see [1,4,2,26]and Chapter 5 of [27].
4.1. Hierarchies and Dendrograms
A hierarchical clustering method is a mathematical procedure for creating a sequence of nested partitions from a proximity matrix, assumed to be a dissimilarity matrix for patterns. Let X denote the set of n patterns. A partition C = {Cl, C2, . . . , Cm} of X is a set of disjoint, non-empty subsets of X which, taken together, constitutes X. That is, if i # j ,
c,n cj = a ; c1u c2u . . . u cm= x.
(4.1)
where lLn” denotes set intersection, “U” denotes set union and W7’denotes the empty set. Each set Ci is called a component of the partition. A clustering is a partition of X ;its components are formally called clusters. A hierarchical clustering is a sequence of nested partitions starting with the trivial clustering in which each pattern is in a unique cluster and ending with the trivial clustering in which all patterns are in the same cluster. Partition B is nested into partition C if every component of B is a subset of a component of C, so C is formed by merging components of B. A dendrogram is a binary tree that depicts a hierarchical clustering. Each node in the tree represents a cluster. Cutting the dendrogram horizontally creates a clustering. A dendrogram represents n clusterings of n patterns, including the two trivial clusterings. Figure 6 is a simple dendrogram for five patterns. The five pattern labels can be permuted in several ways without altering the information in the dendrogram. Murtagh [28]enumerated several types of dendrograms. Dendrograms can simply picture the clusterings, as in Fig. 6, or can have a scale showing the level of dissimilarity at which each clustering is formed.
1 . 1 Cluster Analysis and Related Issues 17
Fig. 6 . Dendrogram for five patterns.
4.2. Recovered Structure and Ultrametricity
A hierarchical clustering method tries to fit a dendrogram to the given dissimilarity matrix. The dendrogram imposes a measure of dissimilarity called the cophenetic dissimilarity on the patterns as follows. Let {CO,C1, . . . , Cn-l} be the sequence of partitions in a dendrogram where Co is the trivial clustering that puts each pattern in its own cluster and C,-1 places all patterns in the same cluster. The clusters in the mth clustering are denoted {Cml,Cmz, ... , Cm(n-m)}.A level function L ( m ) is defined on the partitions as the dissimilarity level a t which clustering m first forms. Specific level functions are defined by each clustering method. The cophenetic dissimilarity d c between patterns x q and x, is defined in (4.2).
where k q , , = min[m : ( x q ,x,) E Cmt, some t].
The cophenetic dissimilarity is a dissimilarity index in the sense of (2.1). However, it also satisfies the ultrametric inequality.
The nesting of partitions to form the hierarchy and the monotonicity of the level function ensure that (4.3) is satisfied. This inequality is stricter than the triangle inequality (2.5). For the distances between patterns in a feature space t o satisfy the ultrametric inequality, for example, all triples of patterns must form isosceles triangles with the one side being shorter than the two sides of equal length. This demands that many ties in proximity occur at just the right places.
4.3. Hierarchical Clustering Algorithms A hierarchical clustering algorithm is a process for constructing a sequence of partitions from a proximity matrix. A particular clustering method can be implemented
18 R. C. Dubes
by several algorithms. Some hierarchical clustering algorithms for constructing dendrograms by hand are based on graph theory [4]. This section is limited to a class of algorithms commonly known as “SAHN” (Sequential, Agglomerative, Hierarchical, Nonoverlapping) algorithms. A SAHN algorithm begins with a n n x n dissimilarity matrix [d(q, r ) ] between patterns. All SAHN algorithms are appropriate when the dissimilarities are on a ratio scale, as with distances in a feature space. In addition to a sequence of clusterings, a SAHN algorithm creates a level function. The algorithm begins with L(0) = 0 and each pattern in a unique cluster. A cluster is denoted (s).
SAHN Algorithm for Hierarchical Clustering 1. Set the sequence number of the clustering: m = 0. Repeat the following steps until m = n. 2. Find the pair of clusters [(s), (t)] for which
4 ( s ) , (t)l = min{d[(q),
(.)I)
(4.4)
where the minimum is taken over all pairs of clusters. 3. Increment m by 1. Merge clusters (s) and ( t ) into a single cluster to define clustering m. Define the level of this clustering as:
L ( m ) = d [ ( s ) , (t)I.
(4.5)
4. Update the dissimilarity matrix by deleting the rows and columns corresponding to clusters ( s ) and ( t ) and adding a row and column for the newly formed cluster (s, t ) . The dissimilarity between this new cluster and an existing cluster, cluster ( k ) ,depends on the clustering method being employed and is given by (4.6).
4.4. Characteristics of Hierarchies and Algorithms
Day [29] explains the computational complexities of hierarchical and partitional clustering algorithms. Some comments on practical aspects of hierarchical clustering are given below without justification. The primary application of hierarchical clustering in engineering work is to create a representative sequence of clusterings while searching for a single “good” clustering. It is seldom the case that data are hierarchically related on more than a few levels.
1.1 Cluster Analysis and Related Issues 19 Clustering method Single-link Complete-link UPGMA WPGMA
fft
P
Y
112
112
0
-112
112
112
0
112
ns ns f nt
ns ns nt
+
0
0
112
112
0
0
1J2
-114
0
ffS
UPGMC WPGMC Ward’s Flexible
112
ns
+ nk
ns f n t ff
+nk
nt +nk ns +nt ff
+n k
-nk
ns f n t
+nk
1 - 2ff
0 0
Fig. 7 . Coefficients for SAHN hierarchical clustering algorithms.
The single-link method is also called the connectedness method, the minimum method, and the nearest neighbor method. The updating equation uses the minimum of the two dissimilarities. Single-link clusters can be “straggly” since the smallest of the pairwise dissimilarities between two clusters determines when the clusters join. Other algorithms for the single-link method involve the minimum spanning tree [30] and graph theory [31]. By contrast, the complete-link method is also called the completeness method and the maximum method. The updating equation translates into taking the maximum of the dissimilarities between pairs of patterns in different clusters. Complete-link clusters often form in small clumps because the largest of the pairwise distances determines when two clusters are merged. Figure 8 shows the complete-link dendrogram for the speaker data. The visual impact of the dendrograms is obvious. It should also be clear why dendrograms are not very useful for more than 200 patterns. The scale on the left is on the same scale as the dissimilarity matrix. The patterns were normalized by (2.9). Some of the clusters in Fig. 8 can be identified in Figs. 2 and 3. With the single-link and complete-link methods, hierarchies merge only at dissimilarities that occur in the given dissimilarity matrix. Other clustering methods do not have this property. Ward’s method 1321 is also called the minimum square error method because it merges, at each level, the two clusters which will minimize the square error from among all mergers of pairs of existing clusters. Ward’s method does not create the clustering which minimizes square error among all clusterings with that number of clusters, however. Two characteristics of Ward’s method should be noted. First, it minimizes square error as described above only when the dissimilarity index is the squared Euclidean distance. Second, the scale on the dendogram is not the same as the dissimilarity scale between patterns. Hierarchies can merge at levels several hundred times the largest dissimilarity between patterns. The visual effect on dendrograms is to accentuate the “lifetime” of individual clusters.
20
R. C. Dubes
Fig. 8. Complete-link dendrogram for speaker data.
The acronym “PGM’ refers to “pair group method” since clusters are merged in pairs. The prefixes “U” and “W’ refer t o “unweighted” and “weighted” methods, repectively. A weighted method treats all clusters the same in (4.6), so patterns in small clusters achieve more individual importance than do patterns in large clusters. An unweighted method takes the size of the clusters into account, so the patterns are treated equally. The suffixes “A” and “C” refer to “arithmetic averages” and “centroids”, respectively. Thus, the full name for the UPGMA method is “unweighted pair group method using arithmetic averages” , sometimes called the group average method. The UPGMC and WPGMC methods have direct geometric interpretations with patterns in a feature space [27]. The UPGMA and WPGMA methods have no simple geometric interpretation. An implicit assumption underlying this entire discussion is that no two entries in any of the matrices encountered in matrix updating are the same. Ties in dissimilarity can produce unexpected and baffling dendrograms for all methods except the single-link method. Jardine and Sibson [33] show that the single-link method is the only method having a continuity property, whereby dendrograms merge smoothly as dissimilarities approach one another. Unfortunately, the single-link method has performed much worse that other methods in extracting the true structure of the data. If the dissimilarity matrix contains ties, either break the ties or use the single-link method. A clustering method is monotone if, when merging clusters (s) and ( t ) into cluster (s, t ) , then for all clusters (Ic) distinct from ( s ) and ( t ) ,
1.1 Cluster Analysis and Related Issues 21 Monotonicity permits dendrograms to be drawn as binary trees, as in Figures 6 and 8. The cophenetic dissimilarity is satisfied only for monotone clustering methods. Without monotonicity, crossovers or reversals can occur in which two clusters merge at a level lower than the level a t which one of the two clusters was formed. This counter-intuitive phenomenon has nothing to do with the data, but is a characteristic of the clustering method. Interpreting a non-monotone dendrogram is extremely difficult. A simple condition for monotonicity, proved by Milligan [34], is that the coefficients in (4.6) satisfy (4.7).
Which clustering method is best? No clear answers exits, even though several comparative studies have been conducted [35]. If the input dissimilarity matrix satisfies the ultrametric inequality then the single-link and complete-link dendrograms will be exacty the same. This is usually taken to mean that the data are organized in a perfect hierarchy. The degree to which the two dendrograms resemble one another is an indication of how appropriate a hierarchical structure is for the data. Defining a quantitative measure of similarity between dendrograms is a very difficult task. 5. Partitional Clustering
A dendrogram displays n clusterings, or partitions, of n patterns, but how do we find a single good clustering of the data? A partitional clustering method handles a large number of patterns and applies an objective criterion in an attempt to achieve the “best” clustering. A solution to the clustering problem is easy to state. Select a criterion and evaluate it for all clusterings, saving the optimal result. This solution is impractical for two main reasons. First, the number of possible clusterings is astronomical [4]. For example, there are about 11,259,666,000 clusterings of 19 objects into 4 clusters. Evaluating a criterion for all clusterings is out of the question. Even if one could enumerate all clusterings, the choice of a single criterion function raises severe difficulties. Clusters of patterns in d dimensions can have a variety of shapes, from spherical to line-like [36]. No single criterion can search for shapes simultaneously. Before choosing a clustering criterion, one must determine what is meant by “cluster” in the application at hand. One general definition is that a cluster is a set of patterns whose inter-pattern distances are smaller than interpattern distances for patterns not in the same cluster. This idea of “cluster” leads to accepting the square-error criterion stated below. One can also reasonably decide that a cluster is a region of high density in the feature space, surrounded by regions of low density. Square-error is not the only criterion. However, square-error is relatively easy to compute, makes good intuitive sense, and the clusters can be interpreted as hyperspheres in the feature space.
22
R. C.Dubes
Given n patterns {XI,x2, . . . , x,} in the d-dimensional feature space, define an indicator function as follows. Let z i k = 1 if pattern xi is a member of the kth cluster and 0 if not. Every pattern must belong to one cluster and the clusters are labeled sequentially. The center of the kth cluster is the centroid of the patterns belonging to the cluster.
Here, n k = C:==, &k is the number of patterns in cluster k . The square-error e$ for cluster k is the sum of the squared Euclidean distances between patterns in cluster k and mk. n
The square-error for the entire clustering is the sum of the square-errors for the individual clusters. K
(5.3) k=l
The clustering problem can be stated as the problem of minimizing E L , for K fixed, by selecting the binary weights { z i k } in such a way that each row of the n x K matrix of weights has exactly one “1” in each row and a t least one “1” in each column. That is, each pattern can belong to only one cluster and no cluster can be empty. In fuzzy clustering [7] zik is taken to be the degree of belonging for pattern i in cluster k and z i k in (5.2) is replaced by z:k where q is chosen empirically, and is usually 2. One can approach this optimization in many ways, including simulated annealing [24]. Gordon and Henderson [37] translate the constrained optimization problem into an unconstrained problem to simplify the solution. The computational demands of formal minimization techniques have encouraged the development of simple, heuristic algorithms [1,4]. The most popular of these is the K - m e a n s method.
K-Means Algorithm for Partitional Clustering 1. Select an initial clustering of the n patterns with K clusters and initialize the cluster centers. Repeat until the clustering stablizes: 2. Assign cluster labels to all patterns by finding the closest cluster centers. 3 . Compute cluster centers for all clusters with (5.1). Repeat Steps 2 and 3 until no cluster labels change. 4. Apply heuristic splitting and lumping criteria. End the repeat. 5 . Compute statistics of the final clustering.
1.1 Cluster Analysis and Related Issues 23 The clustering with the smallest square-error is retained as the solution. Although the convergence of the algorithm can be proven [38] no assurances that the solution is optimal can ever be given. It is easy to propose data that will defeat the algorithm [4]. One hopes that running the algorithm several times for different starting configurations leads to a reasonable solution. The problem of choosing the number of clusters and of formally validating the results is discussed in Section 6. Steps 2 and 3 constitute a K - m e a n s pass. An alternative is to recompute the cluster centers after each cluster label is changed. Step 4 changes the number of clusters by merging small clusters, removing outliers, or splitting large clusters. Criteria for these operations are chosen heuristically. The initial partition can be chosen in several ways [l].For example, one can choose k patterns at random as seed points. One can also choose k patterns that are reasonably separated from one another as seed points. In any event, the algorithm should be run with different seed points to seek the best clustering. Any program for implementing a partitional clustering method involves several parameters. The primary ones are K and any parameters associated with splitting clusters, lumping clusters, and identifying outliers. An exception is the CLUSTER program [4] which creates a sequence of clusterings and uses no parameters. Only experience can dictate selection of these parameters. The manner in which the data are normalized must also be considered. Normalization (2.9) can reduce the effects of spreads in the individual variables and equalize the contributions of all variables to Euclidean distance. The results of any partitional clustering algorithm is a set of tables of numbers showing the cluster centers, the square-errors for the individual clusters, the covariance matrices for the clusters, and various statistics which try to quantify the characteristics of the clustering [4]. Clustering algorithms are often run to see if patterns are clustered according to some a priori category information and a clusterby-category table is displayed to see if the clusters correspond to categories in any way. The cluster centers can be taken as a sampling of the original data and one can represent the data by projecting the cluster centers t o two dimensions. Figure 9 shows the cluster-by-category table from the CLUSTER program with the speaker data, after normalizing by (2.9). Also shown are the square-errors for all clusters. Comparing the table entries to Fig. 2, 3, and 8 may justify some of the entries in the table. For example, categories 1, 2, 3, and 5 are in unique clusters. However, cluster 3 has a much larger square-error than clusters 1, 2, and 5. This suggests that clusters 1, 2, and 5 have smaller dispersions than cluster 3 . 6. Validation and Interpretation
Validation refers to the objective assessment of a clustering structure so as to determine whether a structure is meaningful. The structures under consideration are hierarchies, clusterings, and individual clusters. The sense of validity explained
24
R. C. Dubes
here calls a structure valid if it is unusual under the circumstances of the study. That is, a structure is valid if it cannot reasonably be assumed to have occurred by chance or to be an artifact of a clustering algorithm. Validation is accomplished by carefully applying statistical methods and testing hypotheses.
Cluster 1 2
1 0 0
2
3
4
5
0
0
0
0
5
0
0
3 4
0
0
0
0
0 5 0 0 0
0
5
0 0
5
6 7 8
7
8
0
0
0
6 3 0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
5
1
0
0
0
0
0
0
0
0
4 1
0
0
0
4
e: 274.1 139.5 441.4 177.4 192.4 438.3 296.9 221.5
6.1. A n Attitude Towards Validation
One might argue that a structure is “valid” if it makes sense or is useful or can be interpreted or provides new insight. The list of ways t o justify a result is endless. This section considers only objective measures of validity that can be tested statistically. Formal testing may not be required in every application. One might use experience and judgement t o interpret a clustering structure or use clustering in an exploratory manner. In addition to the three types of structure (hierarchy, clustering, cluster), three types of validation studies called external, internal, and relative can be defined [4]. An index must be chosen to reflect the sense of validity being examined, with the structure and type of study in mind. An external assessment of validity objectively compares the recovered structure t o an a praora structure and tries t o quantify the match between the two. For example, one might test how closely cluster labels match category labels. An internal examination of validity uses no a priori information but tries t o determine if the structure is intrinsically appropriate for the data. For example, one might try to determine if a cluster derived from the single-link method is unusually compact or isolated, as compared t o other single-link clusters of the same size in random data. A relative test compares two structures and measures their relative merit. For example, one might compare a 4-cluster clustering to a 5-cluster clustering without using any a prior2 information. Several indices have been proposed for this purpose and some are explained in Section 6.4. The
1.1 Cluster Analysis and Related Issues 25 paradigm for testing the validity of a clustering structure is summarized below.
Validity Paradigm 1. Identify the clustering structure (hierarchy, clustering, cluster) and the type of validation (external, internal, relative). 2. Select an index. 3. Select an hypothesis of “no structure”. 4. Obtain (by theory or simulation) the baseline distribution of the index under the “no structure’’ hypothesis. 5. Compute the index for the structure being tested. 6. Formally test the hypothesis of “no structure’’ by determining whether the observed index is LLunusual”.
Establishing the baseline distribution might require extensive statistical sampling such as Monte Carlo estimation or bootstrapping. Although the same index can apply to all three types of tests, the circumstances of the application are very different. External tests of validity are generally much easier to apply than internal tests because hypotheses of randomness are easier to propose and baseline distributions are easier to derive than for internal tests. One example of a n external test is the Bailey profile [39,4] which measures the validity of an individual cluster that is defined before the analysis begins. The natural indices of validity for an individual cluster are simple measures of compactness and isolation derived from graph theory. Bailey profiles are restricted to ordinal data and use the hypergeometric distribution to derive baseline distributions. Baseline distributions of compactness and isolation indices cannot be derived when the cluster is obtained by applying a clustering method because the cluster depends on the method itself, among other things. Even though data are purely random, a diligent clustering method might uncover an unusual cluster. An internal test of validity should recognize such clusters as being artifacts. Three validation problems of practical interest are discussed in this section. 6.2. External Tests of Validity f o r a Partition
Suppose two partitions of n patterns are to be compared. One is from category information, obtained before the analysis is begun. The other is obtained from some clustering method. How well does the clustering match the categories? The labels themselves are not important, so renaming categories or clusters cannot affect the degree of match. Hubert and Arabie [40] studied the Rand index as a means for assessing the match between two such clusterings. Let a denote the number of pairs of objects that are in the same cluster in both clusterings and let d denote the number of pairs in different groups. There are
26
R. C. Dubes
n(n - 1)/2 pairs of objects to check. The Rand index [41] measures the degree of match.
Other statistics have been suggested for this purpose but they are linear functions of one another [4]. A clustering is termed “valid” if R is unusually high, as measured with respect to some baseline distribution. To make R less sensitive to problem parameters. Hubert and Arabie [40] corrected it for chance. The hypergeometric distribution was applied to find E ( R ) ,the expected value of R under the baseline distribution. The maximum possible value of R is 1, so the corrected Rand index is :
R’
=
R - &(R) 1 - & ( R ).
Detailed formulas are given elsewhere [40,4]. Using the Rand index, corrected or not, in a test for external validity requries that a baseline distribution be developed. The variance of R‘ is known when one of the partitions is assumed to be assigned a t random. However, the full baseline distribution of R’ is required t o formally test this hypothesis of randomness. 6 . 3 . Internal Tests of Validity for a Partition
The paradigm for external tests of validity is the same as for internal tests. The hypothesis of randomness and the baseline distributions differ. This has led t o some confusion in the literature. For example, the distribution of certain Fstatistics and x2 statistics are listed in standard books on multivariable statistics. These distributions assume that the groups have been chosen without reference to the data, as when one assigns a priori category labels. These distributions are not applicable to the internal validation of clusterings found by sifting through the data. Using them can create misleading results [35]. Milligan [42] compared the performances of 30 internal indices of validity. Three primary difficulties arise in obtaining the baseline distribution needed for an internal test of validity. The first difficulty lies in choosing a hypothesis of “no structure” or “randomness”. To create purely random data, one must choose the region of space in which the random data are to be generated. To be fair, this region must match the characteristics of the data. The second difficulty is the necessity to match all the data parameters. This implies that one must estimate the baseline distribution anew in every application, usually by Monte Carlo sampling. Bock [43] has derived asymptotic distributions for some indices, but it is seldom clear when the assumptions of the derivation are satisfied. The third difficulty is a bit subtle. It is not fair to compare a clustering obtained from a clustering algorithm t o just any clustering of random data. One should compare it to the best clustering of random data. That is, before calling a result valid, one should be reasonably certain that the same result could not be obtained
1.1 Cluster Analysis and Related Issues 27 from any random data, not simply that the result could not be obtained from some random data. Engelman and Hartigan [44] demonstrated this point in estimating the distribution of the ratio of between t o within scatter for one-dimensional data. Given a set of random data (from a normal distribution), they found the ratio for the best separation of the data into two clusters. They published the percentage points for the distribution of the best value of the ratio. However, the result is only applicable t o one-dimensional data. The methodology cannot be extended to more than one dimension easily because the number of clusterings to be examined increases exponentially with dimension. These difficulties may explain why more internal validation is not performed. 6.4. Relative Tests of Validity -How
M a n y Clusters?
The problem of determining the “true” number of clusters has been called the fundamental problem of cluster validity [45,46]. This question is particularly important in image segmentation, where the number of categories, such as land-use categories, is not know in advance. The question “HOWmany clusters are there in the data?” can be asked in at least three meaningful ways. 0
0
0
Do the data contain the number of clusters I expect? This clearly calls for an external test of validation and one might use the procedures of Section 6.2. Is it unusual to find this many clusters with data of this sort? This somewhat vague statement can best be answered with an internal criterion, as in Section 6.3. The basis for comparison must be defined and a baseline distribution must be derived- two difficult tasks. Which of a few clusterings is best? This is more difficult to answer than the first question, but is more specific than the second. The question implies that several clusterings, such as those derived by cutting a dendrogram at several levels, or running a partitional clustering algorithm several times, be considered as candidates. The number of clusters in the best of these clustering is taken to be the “correct” number.
This section considers the third question by examining a sequence of clusters as the number of clusters changes monotonically. For example, one might seek a stopping rule for choosing the best level for cutting a dendrogram. What is a good index? One possibility is to pick the clustering that minimizes square-error (5.3). Square-error is a strong function of the dimensionality, the number of patterns, and the number of clusters [4]. As the number of clusters increases, the square-error has a tendency to decrease whether or not one clustering is better than another. Milligan and Cooper [47]compared 30 indices as stopping rules by applying each to a wide variety of data and ranking them according to the one which found the correct answer most frequently. The correct answer was known because they generated their own data. Any index that performed well should, logically, be trusted with real data. Dubes [48] made more detailed comparisons of two other indices. Zhang
28
R. C.Dubes
and Modestino [49] integrate formal estimation of the number of clusters into image segmentat ion. Three representative indices are defined below; n is the number of patterns, K is the number of clusters in the clustering being evaluated, and E$ is the square-error of the clustering (5.3). To estimate the number of clusters, plot the chosen index as K varies and look for either a peak, a valley, or a knee in the curve, depending on the index. Two underlying assumptions are that the data are not random and at least two clusters exist. The Calinski-Harabasz index, CH(lc) was the best of the 30 indices tested by Milligan and Cooper [13].
The index will always be positive and will be zero for K = 0. Its upper bound depends on problem parameters. The value of K that maximizes C H ( K )is chosen to estimate K . This index normalizes the square-error and tends t o depend less on problem parameters than does the square-error itself. The C index is a normalized form of the r statistic [50] proposed to measure the correlation between spatial observations and time. Let c ( q , r ) be 1 if patterns xq and x, are in the same cluster and 0 if not. Let d(q, r ) denote the dissimilarity, or Euclidean distance, between the two patterns. The “raw” r statistic is: n-1
n
g=l
r=q+l
The dissimilarities need not be distances. Let a K be the number of pairs of patterns in which both patterns are in the same cluster. Define the following two statistics as the smallest and largest possible values of I? for a clustering of the kind being examined. min(r) = sum of
UK
smallest dissimilarities
m a x ( r ) = sum of
UK
largest dissimilarities
The C index is defined as:
C ( K )=
r - min(r) max(r) - min(r) .
The range of the C index is limited to [0,1]. The value of K that minimizes C(K) estimates the number of clusters. 0 The Goodman-Kruskal y statistic [51,4] measures the rank correlation between Euclidean distances d ( q , r ) and function f ( q , r ) for the clustering being evaluated, where f ( q , r ) = 1 - c(q, r ) is 1 if xq and x, are in different clusters and 0,
1 . 1 Cluster Analysis and Related Issues 29 if in the same cluster. In standard notation, S(+) denotes the number of concordant quartets and S ( - ) , the number of discordant quartets. This requires some explanation. A “quartet” is two pairs of numbers. One is a pair of dissimilarities, say [d(q, r ) , d(s, t ) ] ,and the other is the corresponding pair of indicator values, [ f ( q , r ) , f(s, t ) ] . A quartet is concordant either if d ( q , r ) < d(s, t ) and f ( q , T ) < f ( s , t ) or if d(q, r ) > d(s, t ) and f ( q , r ) > f(s, t ) . A quartet is discordant either if d(q, r ) < d(s, t ) and f ( q , r ) > f(s, t ) or if d(q, r ) > d(s, t ) and f ( q , r ) < f(s, t ) . If either pair is tied, the quartet is neither concordant nor discordant. Then,
This index is limited t o the range [-1,1]. The value of K then maximizes r ( K ) estimates of the number of clusters. Studies [13] have shown that some indices proposed in the literature perform very poorly, while some perform very well. It is impossible to claim optimality for any of them, because the characteristics of the data can affect performance in unknown ways.
7. Final Comments Cluster analysis is a valuable tool for organizing, summarizing, and exploring multivariate data. Among the problems that appear in applications are the choice of a clustering criterion that recognizes only clusters of interest, and the choice of a clustering algorithm. Validating the results objectively is the most difficult problem of all. Notwithstanding these real problems, cluster analysis has proved enlightening, especially when invoked by a careful practioner who is aware of inherent limitations and has the proper computer tools.
Acknowledgements
I acknowledge the support of the National Science Foundation, most recently through grant IRI-8901513 and grant CDA-8806599. References [I] M. R. Anderberg, Cluster Analysis for Applications (Academic Press, New York, NY, 1973). [2] A. D. Gordon, Classification (Chapman and Hall, London, 1981). [3] J. A. Hartigan, Clustering Algorithms (John Wiley & Sons, New York, NY, 1975). [4] A. K. Jain and R. C. Dubes, Algorithms for Clustering Data (Prentice-Hall, Engelwood Cliffs, NJ, 1988). [5] L. Legendre and P. Legendre, Numerical Ecology (Elsevier Scientific, Amsterdam, 1983).
30
R . C. Dubes
[6] 3. M. Jolion, P. Meer and S. Bataouche, Robust clustering with applications in computer vision, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 791-802. [7] J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms (Plenum Press, New York, NY, 1981). [8] X. L. Xie and G. Beni, A validity measure for fuzzy clustering, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 841-847. [9] R. S. Michalski and R. E. Stepp, Automated construction of classifications: Conceptual clustering versus numerical taxonomy, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 396-410. [lo] G. Matthews and J. Hearne, Clustering without a metric, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 175-184. [ll]Institute of Electrical and Electronics Engineers, Special Issue of Proceedings o n Neural Networks, I: Theory and Modeling, Sept. 1990. [12] J. C. Gower and P. Legendre, Metric and Euclidean properties of dissimilarity coefficients, J . Classification 3 (1986) 5-48. [13] G. W. Milligan and M. C. Cooper, A study of standardization of variables in cluster analysis, J. Classification 5 (1988) 181-204. [14] K. Pettis, T. Bailey, A. K. Jain and R. Dubes, An intrinsic dimensionality estimator from near-neighbour information, ZEEE Trans. Pattern Anal. Much. Intell. 1 (1979) 25-37. [15] N. Wyse, R. Dubes and A. K. Jain, A critical evaluation of intrinsic dimensionality algorithms, in E. S. Gelsema and L. N. Kana1 (eds.), Pattern Recognition in Practzce (North-Holland Amsterdam, 1980) p. 415-425. [16] K. Falconer, Fractal Geometry (John Wiley & Sons, New York, NY, 1990). [17] J. Theiler, Estimating fractal dimension, J. Opt. SOC.Am. A 7 (1990) 1055-1073. [18] B. S. Everitt, Graphical Techniques for Multivariate Data (Elsevier North-Holland, New York, NY, 1978). [19] R. A. Becker, J. M. Chambers and A. R. Wilke, The New S Language (Wadsworth & Brooks/Cole, Pacific Grove, CA, 1988). [20] T. Okada and S. Tomita, An optimal orthonormal system for discriminant analysis, Pattern Recogn. 18 (1985) 139-144. [21] J. T. Tou and R. P. Heydorn, Some approaches to optimum feature extraction, in J. T. Tou (ed.), Computer and Information Sciences I 1 (Academic Press, New York, NY, 1967) 57-89. [22] J. W. Sammon, A nonlinear mapping for data structure analysis, IEEE Trans. Comput. 18 (1969) 401-409. [23] J. B. Kruskal, Multidimensional scaling and other methods for discovering structure, in K. Enslein, A. Ralston, and H. S. Wilf (eds.), Statistical Methods f o r Digital Computers (John Wiley & Sons, New York, NY, 1977) 296-339. [24] R. W. Klein and R. C. Dubes, Experiments in projection and clustering by simulated annealing, Pattern Recogn. 22 (1989) 213-220. [25] H. Chernoff, The use of faces to represent points in k-dimensional space graphically, J. Am. Stat. Assoc. 68 (1973) 361-368. 126) A. D. Gordon, Hierarchical classification, in P. Arabie and L. Hubert (eds.), Clustering and Classification (World Scientific, Singapore, 1992).
1.1 Cluster Analysis and Related Issues 31 [27] P. H. A. Sneath and R. R. Sokal, Numerical Taxonomy (W. H. Freeman and Company, San Francisco, CA, 1973). [28] F. Murtagh, Counting dendrograms: A survey, Disc. Appl. Math. 7 (1984) 191-199. [29] W. H. E. Day, Complexity theory: An introduction for practitioners of classification, in P. Arabie and L. Hubert (eds.), Clustering and Classification (World Scientific, Singapore, 1992). [30] J. C. Gower and G. J. S. Ross, Minimum spanning trees and single-linkage cluster analysis, Appl. Stat. 18 (1969) 54-64. [31] L. J. Hubert, Some applications of graph theory to clustering, Psychometrika 39 (1974) 283-309. [32] J. H. Ward, Hierarchical grouping to optimize an objective function, J. Am. Stat. ASSOC. 58 (1963) 236-244. [33] N. Jardine and R. Sibson, Mathematical Taxonomy (John Wiley & Sons, New York, NY, 1971). [34] G. W. Milligan, Ultrametric hierarchical clustering algorithms, Psychometrika 44 (1979) 343-346. [35] G. W. Milligan, A review of Monte Carlo tests of cluster analysis, Multivar. Behav. Res. 16 (1981) 379-407. [36] C. T. Zahn, Graph-theoretical methods for detecting and describing gestalt clusters, I E E E Trans. Comput. 20 (1971) 68-86. [37] A. D. Gordon and J. T. Henderson, Algorithm for Euclidean sum of squares classification, Biometrics 33 (1977) 355-362. [38] S. Z. Selim and M. A. Ismail, K-means type algorithms: A generalized convergence theorem and characterization of local optimality, I E E E Trans. Pattern Anal. Mach. Intell. 6 (1984) 81-87. [39] T. A. Bailey and R. C. Dubes, Cluster validity profiles, Pattern Recogn. 15 (1982) 61-83. [40] L. J. Hubert and P. Arabie, Comparing partitions, J . Classification 2 (1985) 193-218. [41] W. M. Rand, Objective criteria for the evaluation of clustering methods, J. Am. Stat. ASSOC. 66 (1971) 846-850. [42] G . W. Milligan, A Monte Carlo study of 30 internal criterion measures for clusteranalysis, Psychometnka 46 (1981) 187-195. [43] H. H. Bock, On some significance tests in cluster analysis, J . Classification 2 (1985) 77-108. (441 L. Engelman and J. A. Hartigan, Percentage points of a test for clusters, J. Am. Stat. ASSOC. 64 (1969) 1647-1648. [45] R. 0 . Duda and P. E. Hart, Pattern Classification and Scene Analysis (John Wiley & Sons, New York, NY, 1973). [46] B. S. Everitt, Unresolved problems in cluster analysis, Biometrics 35 (1979) 169-181. [47] G. W. Milligan and M. C. Cooper, An examination of procedures for determining the number of clusters in a data set, Psychometrika 50 (1985) 159-179. [48] R. C. Dubes, How many clusters are best?-An experiment, Pattern Recogn. 20 (1987) 645-663. [49] J. Zhang and J . W. Modestino, A model-fitting approach to cluster validation with application to stochastic model-based image segmentation, I E E E Trans. Pattern Anal. Mach. Intell. 12 (1990) 1007-1009.
32
R. C.Dubes
[50] L. J. Hubert and J. Schultz, Quadratic assignment as a general data-analysis strategy, British J . Math. Stat. Psychol. 29 (1976) 19G-241. [51] L. A. Goodman and W. H. Kruskal, Measures of association for cross-classifications, J . Am. Stat. ASSOC. 49 (1954) 732-764.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 33-60 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
[ CHAPTER 1.2 I STATISTICAL PATTERN RECOGNITION
KEINOSUKE FUKUNAGA School of Electrical Engineering, Purdue University West Lafayette, IN 47907, USA In the introductory Section 1, the problems of statistical pattern recognition are defined, and a flow chart is presented to show how a classifier ought to be designed. In Section 2, the theoretically optimal (Bayes) classifier and its variations are introduced. The discussion is extended to show how the resulting classification (Bayes) error can be computed for some limited cases. Also, the upper and lower bounds of the Bayes error are shown. Section 3 discusses how a classifier is designed in practice. Linear, quadratic and piecewise classifiers are included. These are based on the expected vectors and covariance matrices of underlying probability distributions. In practice, these vectors and matrices are not given, and must be estimated from an available set of samples. Consequently, the designed classifier variates and the classification error becomes a random variable. Section 4 discusses how the number of available samples affects the classification performance, and also how to allocate the samples into design and test. In Section 5, nonparametric techniques are presented. They are needed in the estimation of the Bayes error and the structure analysis of data, where a mathematical formula such as Gaussianness cannot be applied. Both the Parzen and Ic nearest neighbor approaches are discussed. Feature extraction and clustering are discussed in other chapters. Keywords: Statistical pattern recognition, classifier, probability of error, hypothesis tests, effect of sample size, nonparametric.
1. Introduction The purpose of statistical pattern recognition is to determine to which category or class a given sample belongs. Through observation and measurement processes, we obtain a set of numbers which make up the measurement vector. The vector is a random vector and its density function depends on its class. The design of a classifier consists of two parts. One is to collect data samples from various classes and to find the boundaries which separate the classes. This process is called classifier design, training, or learning. The other is to test the designed classifier by feeding the samples whose class identities are known. Figure 1 shows a flow chart of how a classifier is designed [I]. After data is gathered, samples are normalized and registered. Normalization and registration are very important processes for a successful classifier design. However, different data require different normalization and registration, and it is difficult to discuss 33
34
K. Fukunaga Y
DATA GATHERING
---
-I-
REGISTRATION
.-----
1 SEARCH FOR NEW MEASUREMENTS
ERROR ESTIMATION
NONPARAMETRIC PROCESS
ERROR ESTIMATION (NONPARAMRRICI DATA STRUCTURE ANALYSIS
7
$. FEATURE EXTRACTION CLUSTERING STATISTICAL TESTS MODELING LINEAR CLASSIFIER QUADRATIC CLASSIFIER PlECEWlSE CLASSIFIER NONPARAMETRICCLASSIFIER
PARAMETRIZATION PROCESS ERROR ESTIMATION
Fig. 1. Process of designing a classifier. (From [l],reprinted with permission.)
these subjects in a generalized way. Therefore, these subjects are not included in this chapter. After normalization and registration, the class separability of the data is measured. This is done by estimating the Bayes error, the overlap among different class densities, in the measurement space. Since it is not appropriate a t this stage to assume a mathematical form for the data structure, the estimation procedure must be nonparametric. If the Bayes error is larger than the final classifier error we wish to achieve (denoted by E O ) , it means the data does not carry enough classification information to meet the specification. Selecting features and designing a classifier in the later stages merely increase the classification error. Therefore] we must go back to data gathering and seek better measurements. Only when the estimate of the Bayes error is less than E O , may we proceed to the next stage of data structure analysis in which we study the characteristics of the data. All kinds of data analysis techniques are used here. They include feature extraction, clustering] statistical tests, modeling, and so on. Note that each time a
1.2 Statistical Pattern Recognition 35 feature set is chosen, the Bayes error in the feature space is estimated and compared with the one in the measurement space. The difference between them indicates how much classification information is lost in the feature selection process. Once the structure of the data is thoroughly understood, the data dictate which classifier must be adopted. Our choice is normally either a linear, quadratic, or piecewise classifier, and rarely a nonparametric classifier. Nonparametric techniques are required in off-line analyses to carry out many important operations such as the estimation of the Bayes error and data structure analysis. However, they are often too complex for any on-line operation. After a classifier is designed, the classifier must be evaluated. The resulting error is compared with the Bayes error in the feature space. The difference between these two errors indicates how much the error is increased by adopting the classifier. If the difference is unacceptably high, we must re-evaluate the design of the classifier. At last, the classifier is tested in the field. If the classifier does not perform as was expected, the database used for designing the classifier is different from the test data in the field. Therefore, we must expand the database and design a new classifier. In this chapter, only the boldfaced portions of Fig. 1 will be discussed briefly. More details are found in [I]. Clustering and feature extraction are discussed in other chapters. Also, unless otherwise stated, only the two-class problem is discussed in this chapter, although the results for a more general multi-class problem are listed whenever available. The notations, which are frequently used in this chapter, are summarized as
Dimensionality Number of classes Number of total samples Number of class i samples Class i A priori probability of wi Vector Random vector Conditional density function of wi Density function A posteriori probability of wi given X Expected vector of wi Expected vector Covariance matrix of wi Covariance matrix
36
K. Fukunaga
2. The Bayes Classification In this section, the classification algorithms and the resulting errors are presented, based on the assumption that pi(X) and Pi are known. The classification algorithms are also known as hypothesis tests. 2.1. Likelihood Ratio Classifier
The probability of the classification error can be minimized by classifying X into either w1 or w2, depending on whether q1(X) > q2(X) or q1(X) < q2(X) is satisfied. That is, qi(X)
qz(X)
(Bayes classifier) .
W2
The resulting risk at X is
r * ( X ) = min[ql(X), q2(X)]
(Bayes risk).
(2.2)
The overall error is obtained by taking the expectation of (2.2) over X: E* = E{r*(X)} =
Pi
p2(X)dX
sr2
(Bayes error)
s,
where ~1 = p l ( X ) d X and ~2 = pz(X)dX are called the w1 and respectively. ri is the region where X is classified to wi. For the multiclass problem, qk(X) = max 2 q,(X) + X E E* =
w2
(2.3) errors,
(2.4)
Wk
-L
E 1 - max q,(x)}.
(2.5)
A more convenient form of the Bayes classifier is obtained by applying the Bayes theorem, q2(X)= P$pp(X)/p(X),and taking negative logarithm: h(X) = - ln[pi(X)/pz(X)]
3 ln[Pi/P2]
(2.6)
W2
The h ( X ) combined with a threshold is called the likelihood ratio classifier. When X is distributed Gaussianly with M , and C, for w,, 1 -In pi(X) = - ( X 2
-
1 Mi)TC;l(X - M i ) + - In lCil 2
+
(:)
1n27-r.
(2.7)
The threshold of the classifier could be changed according to various requirements as follows: Bayes classifier for minimum cost. Let c j i be the cost of classifying a wi sample into wj. The expected cost of classifying X into wi is
1.2 Statistical Pattern Recognition 37 The classification rule and the resulting cost are ck(X) = min z ci(X) + X E wk
(2.9) (2.10)
For the two-class problem, w1
h(X)
= - ln[pl(X)/p2(X)]
5 ln[(c12 - C I I ) ~ ' I / ( C ~ I - c22)P2].
(2.11)
w2
This is a likelihood ratio classifier with a new threshold. Neyman-Pearson test. Let ~1 and ~2 be the error probabilities from w1 and w2, as shown in (2.3). The likelihood ratio classifier minimizes ~1 subject to ~2 being equal to a given constant, say, EO. The threshold value must be selected t o satisfy ~2 = EO and is normally determined empirically. A plot of ~1 vs. E Z for the likelihood ratio classifier with varying threshold is called the operating characteristics and is used frequently as a visual aid to see how two errors are traded by changing the threshold. In the Neyman-Pearson test, ~2 = EO is the operating point and the corresponding threshold value is chosen. When w2 represents a target t o be identified against the other class ( w l ) , E I , E Z , and 1 - ~2 are called the false alarm, the leakage, and the detection probability, respectively. Minimax test. We can make t,he expected cost invariant even when Pa varies unexpectedly after the classifier has been implemented. This is done by selecting the threshold of the likelihood ratio classifier to satisfy (c11 - c22) i- (c12 - C 1 1 ) E l
- (C2l -
C22)&2
=0
f
(2.12)
Particularly, when c11 = c22 and c12 - c11 = c21 - c22, the threshold is chosen t o satisfy ~1 = ~ 2 .This classifier eliminates the possibility of having an unexpected large error due t o the unexpected variation of Pi, In all of these three cases, the likelihood ratio classifier is commonly used, and only the threshold varies. This may be interpreted as replacing the true Pi's of (2.6) by artificial Pi's. Therefore, theoretically, all these cases may be treated as the Bayes classifier, assigning the different meaning to Pi for each application. Some other subjects related to hypothesis tests are as follows. Independent measurement set. When X consists of statistically independent measurement sets as X T = [XTXr.. . XL], the Bayes classifier becomes
This suggests how to combine, for classification, seemingly unrelated information such as radar and infrared signatures.
38
K. Fukunaga
One-class classifier. When one clearly defined class is classified against all other (sometimes not well-defined) possibilities, the boundary may be determined from the knowledge of one class only. A typical example is a hyperspherical boundary around a Gaussian distribution with M = 0 and C = I. This technique could work when the dimensionality of the data, n, is very low (such as 1 or 2). However, as n increases, the error of this technique increases significantly. The mapping from the original n-dimensional space to a one-dimensional distance space destroys valuable classification information which existed in the original space. For an example with n = 64, the error increases from 0.1% t o 8.4%. Reject. When X falls in the region where the Bayes risk r * ( X ) is high, we may decide not to classify the sample. This concept is called reject. The reject region r,(t) and the resulting probability of rejection R ( t ) are specified by the threshold
tas
r,(t) = { X : r * ( X ) > t } R(t)= Pr{r*(X) > t } = 1 - Pr{r*(X) 5 t } .
(2.14) (2.15)
Note that P r {r*(X) 5 t } is the distribution function of a random variable r * ( X ) . The probability of error with reject is the integration of r * ( X ) p ( X )in outside r,, and thus depends on t. The error may be evaluated directly from the reject probability as
r,,
1 two. t
4t)= -
(2.16)
The error decreases as the reject probability increases, and vice versa. A plot of ~ ( tvs.) R ( t ) is called the error-reject curve, and is used as a visual aid to see how E ( t ) and R(t) are traded by changing the threshold t. With the largest possible t = 1 - 1/L for the L-class problem, R(1- 1/L) = 0 and ~ ( 1 -1/L) is equal t o the Bayes error,
E*.
Model validation. The distribution function of the random variable r* (X) is a simple and good parameter to characterize the classification environment, determining both E ( t ) and R ( t ) .Thus, when samples are drawn and a mathematical model is assumed, two distribution functions of r * ( X ) may be obtained: one empirically from the samples and the other theoretically from the model. The comparison of these two distribution functions could be used to test the validity of the mathematical model. 2.2. The Bayes Error The Bayes error of (2.3) is generally hard to compute, except for the following two cases. (1) Gaussian X with C1 = Cz = C. For this case, the Bayes classifier becomes a linear function of X as 1 h ( X ) = (Mz - M I ) ~ C - ~ X- ( M T C - ' M i - M,TC-'M2) 6 t . (2.17) 2
+
1.2 Statistical Pattern Recognition 39 Since X is Gaussianly distributed, h ( X ) is also a Gaussian random variable. Therefore,
where ml = E { h ( X ) l w l } =
m2
= E{h(X)Iwz} =
g: = Var{h(X)Iwi} =
-i1 (M2 - M l ) T C - 1 ( M 2
-
+-21 (M2
-
M1)
MI)~C-~ - M1) (M~
( M~ M ~ ) ~ c - ~- (M IM) ~ ( i = 1, 2 ) .
(2.19) (2.20) (2.21)
(2) Gaussian X with C1 # C2. The Bayes classifier for this case is 1 1 h ( X ) = - ( X - Ml)TC,l(X - M I ) - - ( X - M2)TC,1(X - M2) 2 2 (2.22) The distribution of h ( X ) is no longer Gaussian, but the errors for a general h ( X ) can be expressed as follows:
where the unspecified integral regions are the entire domain for X and [-m, +co] for w. Particularly, when h ( X ) is the quadratic function of (2.22) and p i ( X ) ’ s are Gaussian, we can integrate (2.23) and (2.24) explicitly with respect to X to obtain
(2.25)
(2.26) where (2.27)
L&(w)
=
1 1 tan-’(ujiw) - - w[cji 2 2
-
+ (ujib’$w2)/(1 + w2u’$)]
(2.28) (2.29) (2.30)
40
K. Fukunaga
The A’S are the diagonal components of A which are obtained by simultaneously diagonalizing C1 and Cz as
AT&A = I and AT&A
=A
(2.31)
and the (dzi - d l i ) is the i t h component of the vector AT(Mz - M I ) . Equations (2.25) and (2.26) must be integrated numerically, but they are one-dimensional integrations.
Upper and lower bounds. The computation of the Bayes error is very complex unless h ( X ) is linear. Furthermore, since a numerical integration is involved, the Bayes error cannot be expressed explicitly. An alternative is to use the upper and lower bounds of the Bayes error as a measure of class separability. Some of popular bounds are listed as follows: E{yl(X)yZ(X)} :
2 nearest-neighbor error
E{min[ql(X) , yz(X)]} : 2E{yl(X)yz(X)} :
Bayes error
Nearest-neighbor error
2k2E{yi(X)ln yl(X) + yz(X)lnyz(X)} :
--
E{}-/,
:
Bhattacharyya bound
(2.32) (2.33) (2.34)
Equivocation
(2.35)
(2.36)
The inequalities (2.32) 5 (2.33) 5 (2.34) 5 (2.35) 5 (2.36) hold regardless of the distributions. One of the popular bounds is the Bhattacharyya bound, which has an explicit expression for Gaussian distributions: (2.37)
The first term of (2.38) indicates the class separability due to the mean difference, and the second term gives that due to the covariance difference. When the distributions are non-Gaussian, (2.37) with (2.38) is no longer guaranteed to bound the Bayes error. Still, in order to use (2.38) as an effective class separability measure, one may transform each variable to a Gaussian-like one. For example, power transforms, yi = xy (i = 1, . . . , n ) , may be used for causal distributions. Such a variable transformation is useful not only for measuring the class separability, but also for designing a better classifier. Designing the Bayes classifier for Gaussian distributions, even with additional variable transformations, is often easier than designing the Bayes classifier for non-Gaussian distributions.
Scatter measures. The bounds discussed above are valid only for two class problems. There exist no well accepted extensions of the above bounds t o multiclass. Scatter measures, introduced here as an alternative, are intuitively derived, simple, but not
1.2 Statistical Pattern Recognition 41 directly related to the Bayes error. They are for multiclasses. Let us define for L-class: L
Pi(hfi- M ) ( M i - hf)T Between-class scatter matrix
Sb =
(2.39)
i=l
L
Pi&
S, =
Within-class scatter matrix
(2.40)
i= 1
s, = E{(X
-
M)(X - k f ) T= } s b f sw Mixture scatter matrix
(2.41)
The class separability can be measured by the combinations of these matrices: trSLlSb,
t r SL'S,,
In ~S~'swl etc.
(2.42)
where trace and log-determinant are for converting a matrix to a number. The first and second ones are often used for feature extraction and clustering respectively. All combinations, SG'Sb, s;'Sw, sk'sb etc., share the same eigenvectors, and their eigenvalues are well related. The first one of (2.42) measures class separability based on the scatter of class means, normalized by S,, and the others are the variations. Therefore, the scatter measures can be applied only for the cases where classes are separated mainly by mean-difference. There are no measures available for multiclasses mainly separated by covariance-difference.
3. Classifier Design Once the structure of data is studied thoroughly, it is easy to select a proper classifier for the data. This section presents how several typical classifiers can be designed. 3.1. Linear Classifiers
The Bayes classifier becomes linear for the following two cases. (1) Gaussian X with C1 = Cz = C: For this case, the Bayes classifier is expressed by (2.17), which is linear. In particular, when C = I ,
h ( X ) = (MZ - M1)TX
+ 51 ( M T M 1 -
M,TM2) 5 t .
(3.1)
This classifier is also known as the distance classifier in which ( X - M 1 ) T ( Xw1
w1
w2
W2
M I ) g ( X - M z ) ~ ( -XM z ) , or the correlation classifier in which M F X 2 M T X with an energy constant condition MFM1 = MFMz. In both cases, PI = Pz and thus t = 0 is assumed. When C # I , the distance or correlation classifier may still be applied, but only after X is linearly transformed to Y = ATX in order to make C y = ATCA = I . (2) Binary independent x,'s: For independent xJ7staking either +l or -1, n
(1+2,)/2
p z ( X )=
wy j=1
(1 - wz3)(1-z')/2[s(s, - 1)
+ b(z, + l)]
(3.2)
42
K. Fulcunaga
where wiJ
= Pr{zj = fllwi}.
Substituting (3.2) into (2.6),
For Gaussian X with C1 # C2 and more general non-Gaussian X, a linear classifier is not the best one. However, because of its simplicity and robustness, a linear classifier is frequently adopted. The design procedure is as follows. W1
h ( X )= V T X 5 t
(3.4)
w2
Equation (3.4) indicates that X is linearly mapped down t o a variable h and the distributions of h for w1 and w2 are separated by a threshold t. Thus, the optimum V is found by minimizing the probability of error in the h-space. Because of complexity in the error computation, simpler criteria such as f ( m l , 77x2, g f , a;) are often used, where mi = E{h(X)lwi} = VTMi and ~7: = Var{h(X)lwi} = VT&V. The typical examples are
f=
Pl(m1 - may + P2(m2 - m0l2 PI.-::
+P 2 4
between class scatter within class scatter
)
(3.6)
+
where mo = P1ml P2m2 is the mixture mean. These criteria measure the class separability of the distributions of h. The solution of d f /dV = 0 is
where
That is, the optimum V always takes the form of (3.7) regardless of the functional form of f . The effect of f is observed only in the averaging coefficient of covariance matrices, s. For example, s = 0.5 for (3.5) and s = PI for (3.6). V can be found even without specifying f . Since the form of V is known as in (3.7), we change s from 0 to 1 with a certain increment, say, 0.1, compute the empirical distribution functions of h = V T X for w1 and w2 from the given data set, select the value of the threshold, and count the number of misclassified samples. The optimum s is the one which gives the smallest error in this operation. The Bhattacharyya bound of (2.38) gives a simple test t o decide whether or not a linear classifier is appropriate. When the first term of (2.38) is dominant, the classifiability comes mainly from the mean difference. Therefore, a linear classifier is
1.2 Statistical Pattern Recognition 43
a proper choice. However, if the second term is significant, the covariance difference plays an important role, and a quadratic classifier is called for. 3.2. Quadratic Classifiers
For Gaussian X, the Bayes classifier becomes quadratic, as shown in (2.7) or (2.22). In practice, the quadratic classifier of (2.22) is widely adopted in many applications, even without checking the Gaussianness of X, and with much success. Probably, this is the classifier everyone may try first, even before conducting data structure analysis. However, it is not known how to design the optimum quadratic classifier for non-Gaussian distributions, as the linear classifier was designed. The optimization of f(m1,m2, c;,c;) for h = X T Q X + VTX with respect to a matrix Q and a vector V is too complex. If quadratic terms xJxk's are treated as new variables y,, h = CCqJkxJxk C v , x , becomes a linear equation as h = Ca,y, Cv,x,. However, for high-dimensional cases, the number of y z ' s becomes prohibitively large.
+
+
Two-dimenszonal display. One of the procedures used to improve the performance of the quadratic classifier is t o plot X in a two-dimensional display where d : ( X ) = ( X - Mz)TC,l ( X - M,) for i = 1, 2 are used as the J: and y axes. If X is Gaussian, the Bayes classifier becomes a 45" line with a proper y-crossing point. When the distribution is not perfectly Gaussian, we can observe in the display that the 45" line is not the best boundary to minimize the number of misclassified samples. Then, visually we can find a better line to classify samples by changing the slope and the y-crossing point of the line. It corresponds to adjusting (Y and p of the following quadratic classifier:
Once samples are plotted and examined, the boundary in the display need not be restricted to a line. Any curve could be drawn. This flexibility is the advantage of seeing the data on the display.
Fourier transform. When a stationary random process is time-sampled, the coefficients of the discrete Fourier transform are uncorrelated, and its covariance matrix becomes diagonal. Thus, the quadratic classifier of the Fourier coefficients y , is reduced to h = Cr=,(q,ly,12 v,y,) vo. This is the Bayes classifier, if y,'s are Gaussian.
+
+
Approximation of covariance matrices. If we can assume a structure for a covariance matrix, we can simplify the design of a quadratic classifier. In addition, the classifier becomes less sensitive to the parameter variation due to the estimation process by using a finite number of design samples. This will be discussed in the next section. One of the possible structures is the toeplitz f o r m for the correlation matrix, allowing each variable to have its distinct variance. In this case, parameters must be selected to assure that the toeplitz matrix be positive-definite. In particular, when
44
K. Fukunaga
the correlation coefficient between the ith and j t h variables, ~ i j is, approximated by pli-jl ( p : a constant), the entire correlation matrix is characterized by one parameter, p, and its determinant and inverse matrix may be expressed explicitly.
3.3. Piecewise Classifiers For the multiclass problem, the boundary must have a piecewise structure as follows.
Piecewise quadratic classifiers. If X is Gaussian, the classifier becomes, from (2.7), piecewise quadratic:
The first term of (3.10) is widely used even for non-Gaussian distributions. However, it must be noted that the normalized distance of X from each class mean, M i , must be adjusted by two constant terms, In [Cil and 1nPi.
Piecewise linear classifiers. When Xi's are similar, the quadratic term X T C i ' X is eliminated from (3.10) to get a piecewise linear classifier:
Or, replacing Ci by the averaged covariance, C = (Cl
+ . .. + C L ) / L , (3.12)
Another possibility is to design the optimal linear classifier for each pair of classes. In this case, L(L - 1)/2 classifiers must be designed, instead of L in (3.11) or (3.12).
Clustering. In some applications, each class distribution would be handled better by dividing it into several clusters. For example, take the signatures of a target viewed from the front and side. Since they are so different, it may be more appropriate t o separate them into several clusters rather than to treat all of them as one class. Considering each cluster as a class, we can form a new multiclass problem with a significantly increased number of classes. However, the details of designing such a classifier depend very much on how clusters are defined and obtained and how many classes are generated. Therefore, although important, the subject is not discussed in this chapter.
IC nearest neighbor (ICNN). The ICNN classifier forms a piecewise linear boundary, although it is very complex and data dependent. The simpler boundary could be obtained by merging samples into a smaller number of representatives and then applying the ICNN classifier t o these representatives.
1.2 Statistical Pattern Recognition 45
3.4. Sequential Classifiers When m consecutive observation vectors, XI, . . . , X,, are known as coming from the same class, we can use this additional information to reduce the classification error. That is, the number of variables is extended from n for one vector t o m x n for m vectors. Thus, we can form a new random vector with m x n components and design a classifier in the (m x n)-dimensional space. However, when these vectors are statistically independentl a simpler formula could be adopted:
That is, the likelihood ratio classifier is applied t o the incoming sample X i , and the output is accumulated. Rewriting the left-hand side of the inequality as (3.14)
the expected values and variances of s and h E{s(wi} = E{h(wi} and
= h(X)
are related by
1 Var{s(wi} = - Var{h(wi}. m
(3.15)
Thus, we can reduce the variances of s by increasing m, while maintaining the expected values of s. Furthermore, the density function of s becomes close to a Gaussian by the central limit theorem. Two important properties of the sequential classifier emerge from the above discussion. One is that we can make the error as small as we like by increasing m. The other is that the error is determined by a small number of parameters, E{hlwi}, Var{hlwi}, and m, and is little affected by the higher-order moments of h. In practice, the true p i ( X ) ’ s are never known, and h ( X ) = - l n [ p ~ ( X ) / p z ( X ) ] must be replaced by some function i ( X ) . A desired property for i ( X ) is E{k(X)(w1} 5 0 and
E{L(X)Iwz}
2 0.
(3.16)
regardless of the distribution of X . As long as (3.16) is satisfied, the random variable i ( X ) carries classification information, however small, regardless of the distribution of X. The classifiable information can be enhanced as much as we like by increasing m in the sequential operations. Two k ( X ) ’ s are known to satisfy (3.16) for all distributions of X, whose expected vectors and covariance matrices are A41 and C1 for w1 and A42 and Cz for w2: 1
- (M,TC-lA41-
2
- MI)
1 (X 2
- -
-
A4,Tc-’A42) A42)TC,1(X
-
(3.17) A42)
(3.18)
46
K. Fukunaga
Any positive-definite matrix C in (3.17) satisfies (3.16). But, the averaged covariance matrix such as the one in (3.7) would be a better C to achieve the same performance with a smaller m. Equation (3.17) could be used, if the first term is dominant in the Bhattacharyya bound (2.38), but (3.18) is more appropriate otherwise. Note that these equations are the same as (2.17) and (2.22), respect ively. One of the most important aspects in classifier design is t o make the classifier robust. That is, the performance of the classifier must be maintained, even if the distribution of test samples becomes somewhat different from the one used for design. The sequential technique can compensate the degradation of the performance of h ( X i ) 5 t by increasing m. 4. Estimation of Classification Errors
So far, we have discussed the design of parametric classifiers, assuming that Mi and Ci are given. In practice, these parameters are estimated by using a finite number of available samples, and the estimates are random variables. Consequently, the classifier designed with these estimates variates, and its performance is also random. Therefore, it is important to know how the sample size affects classifier design and performance. 4.1. Eflect of Sample Size o n Estimation
General formula. First let us consider the problem of estimating f = f (y1, . . . , yq) by f = f (yl, . . . , y q ) ,where f is a given function, the yi's are the true parameter values, and yi's are their estimates. When the deviation of yi from yi is small, f may be expanded by a Taylor series as follows:
where Ayi
= yi
-
yi. If the estimates are unbiased,
In most parametric cases, the yi's are the components of M , and C, ( r = 1, 2), and Y T = [ y l . . . yq] can be expressed as
Y T = [ml(1)
7
m(l) " ' 7
7L
7
(1) J1) (2) m,(2) 7 .. ., m,(2) 1 c11 > . .. , n7L7 c11, . . . ,
?.A]
(4.4)
where mi') and cii) (i 5 j ) are the components of M , and C, respectively. Their unbiased estimates are obtained by the sample m e a n and sample covariance matrix
1.2 Statistical Pattern Recognition 47 as
where X i T ) is the ith sample from w,. When X:')'s are drawn from Gaussian distributions, E{AyiAyj} for the y;'s of (4.4) are known, and (4.2) and (4.3) become
(4.7) where both C1 and Cz are assumed to be diagonal with ( r = 1, 2) as their diagonal components. Without loss of generality, any two covariance matrices can be simultaneously diagonalized. Also, N1 = N2 = N is assumed for simplicity. Note that both the bias of (4.6) and the variance of (4.7) are proportional to 1/N. The other terms are determined by the underlying distributions. This is true even when XiT)'sare drawn from non-Gaussian distributions. Estimation procedure o f f . Equation (4.6) can be rewritten as U
E{f}" f + - N .
(4.8)
This equation suggests the following procedure t o estimate f : (1) Change N to N1, . . . , Ne. For each Ni, compute MTand 2,and subsequently f . Repeat this T times independently, and approximate E{f} by the sample mean of the 7 results. (2) Plot the empirical points of E{f} vs. 1/N. Then, find the line best fitted to these points. The slope of the line is u , and the y-crossing point is the estimate off.
Bhattacharyya distance. The Bhattacharyya distance of (2.38) is a function of M , and C,. Thus, the bias of (4.6) can be further reduced by computing the partial derivatives of this function. Treating the first and second terms separately, r
48
K. Fukunaga
where M I = 0, M2 = [ml . . . m,IT, Xi1) = 1 (A1 = I) and Xi2) = X i (A2 = A) are used without losing generality. For example, when mi = 0 and X i = 1, E{Apl} %’ n/(4N) and E{Ap2} n(n 1)/(8N). This is the case where the Bhattacharyya distance is measured between two sample sets generated from the same Gaussian distribution. Although p1 = p2 = 0 for N = 00, a finite N creates the biases. Note that E{Ap1} is proportional to l / k ( k = N / n : ratio of sample size and dimensionality) while E(Ap2) depends on (n+l ) / k . In order to maintain the same amount of bias (E{Ap} = E{Apl} E{Ap2}), a larger k must be chosen as n increases. For example, t o meet E{Ap} 5 0.223, k must be larger than 6.2 and 39.6 for n = 8 and 64 respectively. The variances of (4.7) also can be computed similarly.
+
+
4.2. Estimation of Classification Errors
The classification errors of (2.23) and (2.24) are the members of the family of functions presented in (4.1) and (4.4), when h ( X ) and p , ( X ) are functions of M , and C,. However, in this case, the randomness comes from two sources: the finite design-sample set to make h ( X ) random and the finite test-sample set to make pi(X) random. Since these two affect the error differently, we need t o discuss their effects separately.
Effect of test samples. When a finite number of samples is available for testing a given classifier, a n error-counting procedure is the only feasible possibility in practice. That is, each sample is tested by the classifier and the number of misclassified samples is counted. Then,
Et(6,) = E ,
and
Vart{h,} =
Er(1 -
Nr
~ r )
(7- =
1, 2)
(4.11)
where Et and Vart indicate that the expectations are taken with respect to test samples, and N, is the number of test samples from w,. This is a n unbiased estimate. Furthermore, (4.11) is valid regardless of functional forms for h ( X ) and Pr(X).
Effect of design samples. When a finite number of design samples is used to compute M,, 9, and then h ( X ) , Ed{&.} and Vard{i,} can be obtained through (2.23) and (2.24) for given test distributions, p , ( X ) , where Ed and Vard indicate the expectations with respect to design samples. The resulting bias is
Ed{&} 2 E
+
v +N
(4.12)
where E = P1~1 P ~ E and z , v is determined by the underlying distributions of design samples, given test distributions and the functional form of h ( X ) . The number of design samples is denoted by N (= N1 = N2) and distinguished from the test sample size N . Although v is a complicated function and can be computed
1.2 Statistical Pattern Recognition 49 explicitly only for simple special cases, v can be obtained empirically by the estimation procedure of (4.8). When h ( X ) is the quadratic classifier of ( 2 . 2 2 ) with C1 = C2 = I and PI = Pz, v becomes 1
vq 2
4
J
2-1
m
(4.13) where M = M2 - M I . On the other hand, when h ( X ) is the linear classifier of (2.17) with C1 and PI = P2, v becomes
ve =
1 4JGimiT
e--MTM/8
[
(1
--+
+ MTM
- 1]
.
=
C2 = I
(4.14)
When C1 = C2, the quadratic h ( X ) of (2.22) becomes the same as the linear h ( X ) of (2.17). However, when the estimated covariance matrices, 21 # 2 2 , are used, h(X) of (2.22) differs from that of (2.17). As a result, E{AE} for the quadratic classifier is proportional to n 2 / N while E{&} for the linear classifier tends t o n / N when n gets large. This implies that many more samples are needed to properly design a quadratic classifier than a linear classifier. More generally, (4.6) suggests that the bias could be proportional to n2 because of the double summation of the last term. This is due to the fact that n2 correlations are estimated. This number could be significantly reduced, if we assume a structure of the covariance matrix and estimate a smaller number of parameters. As for Vard{i}, we have for the Bayes classifier Vard{d}
(4.15)
0:
N
otherwise
Effect of independent design a n d t e s t samples. When both design and test samples are finite and they are independent, the bias and variance of 2 are
E{d}
(4.16)
Z Ed{€}
(4.17) where E and Var indicate that the expectations are taken with respect t o both design and test samples. Note that Vart{d,} of (4.11) is obtained from the first term of (4.17) by replacing Ed{&} by E,. Since Ed{&,} Z E , v,/N, we can conclude as follows:
+
(1) the bias of the classification error comes entirely from the finite design set, and ( 2 ) the variance comes predominantly from the finite test set.
50
K. Fukunaga
4.3. Holdout, Leave- One- Out, and Resubstitution Methods When only one set of samples is available and the performance of the specified classifier is to be estimated, we need to decide how to divide the samples into two groups, design and test.
Upper and lower bounds of the Bayes error. In general, the classification error is a function of two sets of data, the design set PD and the test set PT, and may be expressed as E ( P DP , T ) where E is an operator to compute the Bayes error. We assume that both PD and PT are drawn from the same set of underlying distributions P = { p l ( X ) , p 2 ( X ) } . If P is used for design, the resulting classifier is the Bayes which produces the Bayes error by testing P . That is, the Bayes error is expressed by & ( P P , ) . Letting and +2 be two different sets of samples independently drawn from P , E ( P ,P ) can be bounded as (4.18)
The rightmost term indicates that, as long as design and test sample sets are independent, the resulting error is larger than the Bayes in expectation with respect to test samples. The leftmost term suggests that, if the same set is used for both design and test, the resulting error is smaller than the Bayes in expectation. These procedures are called the holdout (H) and resubstitute (R) methods respectively. The H method works well, if many data sets can be generated by a computer. However, in practice, with only one set of data, we need t o divide the available data into two independent groups. This reduces the number of samples available for each of design and test. Also, it must be assured that the distributions of design and test samples are close. Another problem is how to allocate samples to design and test. This is normally done by balancing the bias due t o the design sample size and the variance due to the test sample size. The leave-one-out (L) method alleviates the above difficulties of the H method. In this method, one sample is excluded, the classifier is designed on the remaining N - 1 samples, and the excluded sample is tested by the classifier. This is repeated N times to test all N samples. The number of misclassified samples is counted to obtain the estimate of the error. Since each test sample is excluded from the design sample set, the design and test sets are independent. Also, since all N samples are tested and N - 1 samples are used for design, the available samples are more effectively utilized. Furthermore, we do not need t o worry about dissimilarity between the design and test distributions. Although the L method requires N classifiers (one for each sample), these classifiers may be computed with little extra computer time as the perturbations from the classifier designed from N samples. This will be shown for the quadratic classifier next and for nonparametric cases later.
R and L methods for the quadratic classifiers. In the R method, all available samples, X J T )( r = 1, 2 ; i = 1, . . . , N T ) ,are used t o compute MT and 2, of (4.5).
1.2 Statistical Pattern Recognition 51
Then, the same samples are tested as to whether or not the following inequality is satisfied:
for testing
Xk
E { X (1) , , .. ., X N ( 11 ) ,XI ( 2 ), . . . , X N ( 2 )Z } .
(4.19)
Then, the number of misclassified samples is counted. This error is supposed t o be smaller than the true error of the classifier. On the other hand, in the L method, X i 1 ) E w1 is excluded from the computations of k 1 and 21, and the modified k l and 51 are used in (4.19) to test X:'). Similarly, k 2 and 2 2 are modified for testing X r ) E w2. The resulting quadratic equation ~ L ( X I becomes ,)
where
and
d $ (X,) = ( X ,
- k,)T2,1(XI,
-
k,) .
(4.22)
Equation (4.20) indicates that, for X I , E w1, h ~ ( X k is) larger than h ~ ( X k )and , the chance of X I , being misclassified is increased. The same is true for XI, E w2. Therefore, the L error is always larger than the R error. When the R method is used to count the error, h ~ ( X kand ) d z ( X ~ ,()r = 1, 2) must be computed for all (N1 N2)Xk's. The L method requires an additional computation of (4.21) for each 5. However, since (4.21) is a scalar function, the computation time for this part is negligibly small. Thus, when h ~ ( X kis) computed and tested for each XI,, h ~ ( X kis) also computed and tested at the same time with little additional computer time.
+
5. Nonparametric Procedures As Fig. 1 shows, nonparametric procedures are necessary for error estimation in both measurement and feature spaces and data structure analysis before a
52
K. Fukunaga
parametric structure of the data is determined. Nonparametric procedures are based on the estimation of a density function without assuming any mathematical form.
5.1. Estimation of a Density Function There are two approaches for density estimation: one is the Parzen approach and the other is the k-nearest neighbor (NN) approach. They have similar statistical properties with minor differences.
Parzen density estimate. When N samples X I , . . . , X N , are given, but no mathematical form can be assumed for the density function, the value of the density function at X may be estimated by p ( x )=
1
c N
Ic(X
-
Xi)
i=l
where K ( . ) is called a kernel function. In practice, selection of the kernel function is limited to either Gaussian or uniform particularly in a high-dimensional space. A more general form for K ( . ) is
(5.2) where I?(.) is the gamma function. When m = 1 and 00, K ( . ) of (5.2) are reduced to the Gaussian and uniform kernels respectively. The matrix A , which is called a metric, determines the shape of the hyperellipsoid, and r controls its size (both in the uniform case). Otherwise coefficients are selected to satisfy two conditions: tc(X)dX = 1 (which is required to satisfy p ( X ) d X = 1) and X X T l c ( X ) d X = r 2 A (the covariance matrix of .(X)). The bias and variance of (5.1) are
s
s
s
The control parameters of the Parzen density estimate are m, r , A and N and their optimal choices could be found by minimizing E{ [ p ( X )-p(X)]’} which is (Bias)2+ Var. However, the optimal selection of parameters for density estimation does not coincide with the one for classification.
kNN density estimate. In this approach, the kth N N sample of X is found, and the distance d k (or the corresponding volume U k ) is measured, where d i = (XkNN X ) T A - ’ ( X k ~-~X ) , uk = CIA^^/^^;, A is a metric, and c i s a constant. Then, the
1.2 Statistical Pattern Recognition 53 density estimate a t X is k-1 (5.5)
p(x) = NVk(X)
where vk is a random variable and the function of X . Defining u as the probability of a sample falling within ?&, the density function of u is known as (5.6) Since u g u k p ( X ) for a small ?&, the density function of vk can be computed from (5.6). Thus, the bias and variance of the k N N density estimate are also obtained, resulting in
[ +a
E{p(X)} 2 p(X) 1
Var{p(X)}
- tr
':rv;
{ -A }
(5.7)
(Ncl:l&x))2/n]
P2 (XI
E -.
k-2
The term (.)'//" of (5.7) is d; by (5.5). Also, Var {p(X)} of (5.4) becomes p2(X)/k for a uniform kernel in which w is l / w . That is, the biases and variances of the Parzen and k N N density estimates are very similar. The k N N density estimate could be considered as the Parzen one with a uniform kernel whose kernel size is adjusted by p(X). The control parameters of the k N N density estimate are k , A and N , and their optimal choices are found by minimizing E{p(X) - p(X)I2}.
Moments of the kNN distance. Since the density function of moments of d k can be computed, resulting in
Vk
is known, the
where the integrals for some distributions with the covariance matrix C are Gauss: Uniform:
(27r)m/21C1"/2"(1 -- m / n ) - n / 2
(zn-)m/2jCIm/2nr--m/n (1
+n/2)(1+ n / 2 1 ~ / ~ .
(5.10) (5.11)
When m / n is small in a high-dimensional space, and A is selected as A = C, E{dr} is determined predominantly by n and m. The effects of k and N are minimal as r ( k m / n ) 2 r ( k ) and r ( N 1 m / n ) 2 r ( N 1). Also, E{drlX} is computed by (5.9) without taking the integration, but is little affected by p(X) because of a small m/n in power. The variance of d k is very small and all dk's are close to the expected value.
+
+ +
+
Estimation of the local dimensionality. The ratio of two averaged k N N distances depends only on k and n, but not on N and p ( X ) as follows: (5.12)
54
K. Fukunaga
The n computed from d k ’ s and d k + l ’ s by (5.12) depends only on neighboring information, and thus indicates the local dimensionality (or intrinsic dimensionality). Generally, the dimensionality plays a dominant role in determining the statistical properties of any nonparametric estimation. For example, E{dy} of (5.9) with A = C is predominantly determined by n and m. However, it must be kept in mind that the n of (5.9) means the local dimensionality but not the global one.
Very large number of classes. When the number of classes is very large as in the hundreds, we may consider class expected vectors Mi (i = 1, . . . , L ) as random vectors drawn from a distribution, p ( M ) . The classification error between a pair of classes, wi and w j , is determined by the distance between Mi and Mj and the amounts of noises around these M’s. The overall error depends on how many neighboring classes contribute the error. On the other hand, the previous discussion indicates that, in a high-dimensional space, the k N N distance is not affected by k , L and p ( M ) . That is, each class is surrounded by many other neighboring classes with almost equal distances. Thus, almost-equal pairwise errors are added up to form the total error which could become large. Classification of a very large number of classes must be handled with special care. It is not enough to confirm that each pair of classes can be classified with a reasonably small error. 5.2. Classification
Parzen classifier. Substituting the Parzen density estimates into the likelihood ratio classifier of (2.6),
This is called the Parzen classifier and can be used to classify X , when a set of samples {Xi’), . . . , X N(1)1, X ,(2) , . . . , X N(2), } is given. Each class may have a distinct kernel function. In order t o find the upper and lower bounds of the Bayes error, we may adopt the resubstitution ( R ) and leave-one-out ( L ) methods for the Parzen classifier. In the R method, the same samples, Xi‘) ( r = 1, 2 ; i = 1, . . . , N‘), are tested by (5.13), and the number of misclassified samples is counted. On the other hand, when Xe(l) is tested in the L method, Xj’) is excluded t o form the Parzen density estimate of w1. Therefore, the numerator of (5.13) must be replaced by
The denominator stays the same as l;z(Xe( 1 )). Again, X,(’) (f? = 1, . . . , N1) are tested and the number of misclassified samples is counted. Note that the amount
1.2 Statistical Pattern Recognition 55 subtracted in (5.14), ~ 1 ( 0 )does , not depend on !. When an w2 sample is tested, the denominator of (5.13) is modified in the same way. It can be proved that P1L(Xj1))5 fil(Xjl))if K ( X )5 ~ ( 0 which ) is satisfied for the kernel functions of (5.2). Therefore, the tested sample has more of a chance to be misclassified in the L method than in the R method. Also, note that the L density estimate of (5.14) can be obtained from the R density estimate by simple scalar operations -subtracting ~ l ( 0 and ) dividing by N1 - 1. Therefore, the computation time needed to obtain both the L and R density estimates is almost the same as that needed for the R density estimate alone.
kNN classifier. Using the k N N density estimates of (5.5), the likelihood ratio classifier becomes
where us)= clA,I1/”dL)’” and dL)’(X) = ( X i : h N - X)TA-l(XL:,,N - X ) . That is, in order to test X , the k l N N from w1,X k(1)l N N ,and the k 2 N N from w2, X l(c2 )z N N , are found, the distances from X to these neighbors, dg’ and d t ) , are measured, and
d t ) and d g ) are inserted into (5.15) to test whether the left hand side is smaller or larger than t. For simplicity, kl = k2 = k is used in this chapter. Normally, the class covariance matrix is used for A, and therefore A1 # A2. The R and L methods of the k N N classifier are used to bound the Bayes error. In the R method, all samples are included in the list of design samples from which the k N N of the test sample is found, and the same samples are tested. When X,(l) is tested, X,(’) itself is the closest sample in the list. Therefore, dFll(Xjl)) is inserted into the denominator of (5.15) while df)(Xjl)) is inserted into the numerator. On the other hand, when Xi1) is tested in the L method, Xk’) must be excluded from the list of design samples. Therefore, d r ) ( X { ’ ) ) and d f ) ( X j ’ ) ) are compared. Since drjl(Xjl)) 5 d P ) ( X j l ) ) , Xk‘) has more of a chance to be misclassified in the L method than in the R method. Also, note that in order to find the k N N , the distances to all samples are computed and compared. When d r ) ( X j ’ ) ) is obtained, d,&(Xj’)) is also available. This means that the computation time needed to get both the L and R results is practically the same as the time needed for the R method alone. Similarly, for testing a w2 sample, X ,( 2 ), d f j l ( X j 2 ) )and d f ) ( X j 2 ) )are compared with d r ) ( X d ” ) in the R and L method respectively. Voting kNN classifier. Instead of selecting k N N from each class separately and comparing the distances, the k N N ’ s of a test sample are selected from the mixture of classes, and the number of neighbors from each class, ki, among the k N N is counted. The test sample is then classified to the class represented by a majority
56
K. Fukunaga
of the kNN’s. That is,
k,
= max(k1,
..., k L }
+
X E w,
(5.16)
In order to avoid confusion between these two k N N procedures, we will call (5.16) the voting k N N procedure and (5.15) the volumetric kNN procedure. For the voting kNN procedure, it is common practice t o use the same metric to measure the distances to samples from all classes, although each class could use its own metric. Since the kz’s are integers and a ranking procedure is used, it is hard to find a component of (5.16) analogous with the threshold of (5.15). It can be shown that, with t = 0 in (5.15), the volumetric k N N and voting (2k - 1)” procedures give identical classification results for the two-class problem using the same metric for both classes. For example, let k and (2k - 1) be 3 and 5 respectively. In the voting 5NN procedure, a test sample is classified to w1, if 3 , 4, or 5 of the 5”’s belong to w1. This is equivalent to saying that the third N N from w1 is closer to the test sample than the third N N from w2. In the voting k N N classification for the two-class problem, k must be odd. Otherwise, k l = kz could occur and we cannot decide which class the test sample is classified to. This problem may be alleviated by introducing the concept of rejection. That is, when kl = k2 occurs, the test sample is rejected, and not counted as misclassified one. As a result, the classification error becomes smaller than even the Bayes error. This happens because some of the to-be-misclassified samples are rejected and not counted as the error. The asymptotic ( N , = m) performance of the voting k N N is known. For the two-class problem, the risk of the voting k N N classification given X, r k ( X ) , is (5.17)
(5.18) where <(X) = ql(X)qZ(X). On the other hand, the Bayes risk given X is
= g;( 1
i=l
22-2 i-1
)m
(5.19)
Using (5.17)-(5.19), it is not difficult to prove that these conditional risks satisfy the following inequalities
1.2 Statistical Pattern Recognition 57 Taking the expectation of these risks with respect to X, the corresponding errors can be obtained. Therefore, these errors also satisfy the inequalities of (5.20). Thus,
where &*
= E{T*(X)}
and
Ek"
= E{Tk(X)}.
Equation (5.21) indicates that asymptotically (N, = co)the Bayes error is bounded by the voting kNN errors; the upper bounds for odd k's and the lower bounds for even k's.
5.3. Selection of Parameters
As is seen in Fig. 1, one of the major objectives of nonparametric classification is to estimate the Bayes error. This is done by finding the upper and lower bounds, using the L and R methods. However, normally nonparametric estimates are heavily biased unless the parameters are carefully chosen, and the results might not bound the Bayes error. Simply increasing the sample size is not the way to reduce the biases. The required number of samples could be astronomical particularly in a high-dimensional space. Bias of the voting kNN classification. The asymptotic kNN error is obtained by assuming (rr(Xe") = q,(X) where xe" is the CNN of X. However, as (5.9) suggests, X ~ N N and X are not close in a high dimensional space. Using a better approximation, qT(xeNN) q,(X) VTq,(X)(Xe" -X ) tr{V2q,(X)(XeNN X)(Xe" - X)T}, we can compute the simplest case, the bias of the NN error, for a finite sample size, resulting in
+
++
where ENN is the asymptotic NN error, A is the metric, B1(X) is a matrix determined by the underlying distributions, and p1 is a constant related to N and n as follows:
For given distributions (given B l ( X ) and n ) , the bias is controlled by N and A. However, the reduction of ,& in (5.23) by increasing N is very slow for a large n. For example, for n = 64, N-2/n is 0.81, 0.65 and 0.52 for N = lo3, lo6 and lo9 respectively. The optimal A should be obtained by minimizing Ex{.} of (5.22) for the global metric and (A(-l/ntr{AB1(X)} for the local metric. However, since B l ( X ) is too complex to compute in practice, we do not know how to select A.
58
K . Fukunaga
The above result can be extended to the 2NN error as (5.24) 4/n)
r(N I'(N
+ 1)
+ 1 + 4/n)
FZ
(5.25)
where &(X) is another matrix determined by the underlying distributions. In (5.25), ,& is proportional t o N P 4 / " while is t o N P 2 I n . T h a t is, as N increases, the 2NN error converges to its asymptotic value more quickly than the N N erroras if n were half as large. Also, note that pz is significantly smaller than 01, because r2/"/nn(0.071 for n = 64) is squared. Since their asymptotic errors are related by &" = 2&2" from (5.17) and (5.18), a better estimate of ENN could be obtained by estimating &2" first and doubling it.
Parzen classajiers. In the L method, design and test samples are independent, and the bias of the classification error comes from the finite design samples. In this case, h ( X ) in (2.23) and (2.24) is replaced by h(X) of (5.13) in which the bias and variance of p,(X) are given in (5.3) and (5.4). Therefore, we can express the bias of the error of the Parzen classifier in terms of T and N as E{Ad}
Z'
a1r2
+ a2r4 + a3 r-n __ N
(5.26)
where a l , a2 and a3 are determined by the underlying distributions, the metrics A1 and A2 for w1 and w2, the parameter t o control the shape of the kernel m, and the threshold t in (5.13). Recall in (5.3) and (5.4) that the bias of the Parzen density estimate is a function of V 2 p ( X ) ,A and r2 while the variance is a function of p ( X ) , N and w (which is a function of r , A and m). The a1r2 and a2r4 terms indicate how biases in the density estimates influence the performance of the classifier, while the a3r-"/N term reflects the role of the variance of the density estimates. For small values of r , the variance term dominates (5.26), and the observed error rates are significantly above the Bayes error. As T grows, however, the variance term decreases while the u1r2 and a2r4 terms play an increasingly significant role. Thus, for a typical plot of the observed error rate versus r, € decreases for small values of r until a minimum point is reached, and then increases as the bias terms of the density estimates become more significant. The r-" in the third term of (5.26) for a small r and a large n is astronomically large (for example, r-" = 1.8 x lo1' for r = 0.5 and n = 64). It is futile t o attempt to bring down this term by selecting a large N and reducing a3. The optimal r can be obtained by taking the derivative of (5.26) with respect to r and equating it to zero. The more practical solution is t o compute the L and R errors for the various values of T , plot the curves, and find the optimal r. As r increases, the biases of both L and R errors increase, and they do not bound the Bayes error. In order to reduce this bias, we must make a1 and a2 of (5.26) small by selecting parameters, t, A1, A2 and m.
1.2 Statistical Pattern Recognition 59 (1) Selection of the decision threshold t. Changing t is a very effective way t o reduce a1 and 132. Since a1 and a2 are very complicated functions, it is more practical to find the optimal value for t experimentally. The different optimal value for t must be computed for each selection of r. Although better but more complex procedures are available, a simple procedure to determine t is presented as follows: For each value of r , find the value o f t which minimizes the R error, and then use this value o f t to find the L error. Since the selection o f t is isolated from the actual values of the L estimate of the likelihood ratio, using this method helps to maintain the independence of the test operation from the design one. (2) Selection of metrics, A1 and A2. Minimization of a1 and a2 with respect t o A1 and A2 does not give any easy answer. Also, since A1 and A2 are n x n matrices, empirical try-and-error cannot be applied easily. An intuitive selection, which is normally used, is A, = C,. But, this is far from the optimal solution. Another alternative is Ai = Ci - y i ( X - M i ) ( X -Mi)T with a constant yi, the properties of which have not been studied extensively. Even if A1 = C, is a good selection of the metric, the estimation of C, needs special care. That is, if the same sample set is used for both estimating C, and computing the L and R errors, both L and R errors are severely and optimistically biased, and often they do not bound the Bayes error. In order to avoid this optimistic bias, we must use two independent sample sets, one for each of the above two operations. A better but more complex procedure is to exclude Xi') from estimating C,, when X i.' is tested by the L and R classifiers. (3) Selection of m. As m increases, the L error curve is little affected. But, the R curve steadily rises closer to the Bayes error line until m = 4, and then stops to improve. It means that (1) both Gaussian and uniform kernels give the similar L errors, (2) the uniform kernel is better to compute the R error than the Gaussian kernel, and (3) the uniform-like kernels are obtained by selecting m = 4.
kNN classifiers. The parameters of the k N N classifier, k, t, A1 and A2, are chosen in the same way as the ones of the Parzen classifier. That is, k and t are determined empirically. We do not know how to select A1 and A2. Samples to compute the L and R errors must be independent from samples to estimate the metrics. References [l] K. Fukunaga, Introduction to Statistical Pattern Recognition, Second Ed. (Academic Press, San Diego, CA, 1990). [2] R. 0. Duda and P. E. Hart, Pattern Classijication and Scene Analysis (Wiley, New York, 1973).
60
K. Fukunaga
[3] P. R. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach (PrenticeHall, Englewood Cliffs, New Jersey, 1982). [4] A. K. Agrawala (ed.), Machine Recognition of Patterns (IEEE Press, New York, 1977). [5] P. R. Krishnaiah and L. N. Kana1 (eds.), Handbook of Statistics 2: Classzfication, Pattern Recognition and Reduction of Dimensionality (North-Holland Publ., Amsterdam, 1982). [6] T. Y .Young and K. S. Fu (eds.), Handbook of Pattern Recognition and Image Processing (Academic Press, San Diego, 1986). [7] B. V. Dasarathy (ed.), Nearest Neighbor (NN)Norms: NN Pattern Classification Techniques (IEEE Computer Society Press, Los Alamitos, CA, 1991).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 61-103 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 1.3 I SYNTACTIC PATTERN RECOGNITION
KOU-YUAN HUANG Department of Computer and Information Science, National Chiao Tung University Hsinchu, Taiwan Sabbatical Leave at Rice University and University of Houston In pattern recognition problems, the structural information that describes the pattern is important, so we can use syntactic methods to recognize the pattern. The pattern can be described as a representation, i.e., a string, a tree, a graph, an array, a matrix, or an attributed representation. We can parse the representation and assign the pattern to its correct class. In this chapter, we present the algorithms of the finite-state errorcorrecting parsing for the recognition of the 1-D string; the modified error- correcting Earleys parsing and the parsing using a match primitive measure for the recognition of the 1-D attributed string; the weighted minimum-distance structure preserved errorcorrecting tree automaton and the modified maximum-likelihood structure preserved error-correcting tree automaton for syntactic parsing of the 2-D patterns.
Keywords: Syntactic pattern recognition, pattern representation, string distance, attributed string, finite-state automaton, tree automaton, error-correcting parsing, minimum distance, maximum likelihood.
1. Introduction Syntactic pattern recognition has been developed over two decades, received much attention and applied widely to many practical pattern recognition problems, such as (1) English and Chinese character recognition, ( 2 ) fingerprint recognition, (3) speech recognition, (4) remote sensing data analysis, (5) biomedical data analysis in chromosome images, carotid pulse waves, EEG signals,. . . , etc., (6) scene analysis, (7) texture analysis, (8) 3-D object recognition, (9) two-dimensional mathematical symbols, (10) spark chamber pictures, (11) chemical structures, (12) geophysical seismic signal analysis, . . . , etc. [3,4,8~11,13-17,21,26,28,30,3537, 39-46,51,52,57-60,64-67,70,74,76,77,85,86,89]. In pattern recognition problems, besides the statistical approach, the structural information that describes the pattern is important, so we can use syntactic methods to recognize the pattern. A pattern can be decomposed into simpler subpatterns, and each simpler subpattern can be decomposed again into even simpler subpatterns, and so on. The simplest subpatterns are called primitives (symbols, terminals). A pattern can be described as a representation, i.e. a string of primitives, a tree, a graph, an array, a matrix, or an attributed string,. . . , etc. [23,32, 61
62
K.-Y. Huang
44-46,48,69,84]. We can parse the representation and assign the pattern to its correct class. A basic block diagram of the syntactic pattern recognition system is shown in Fig. 1 [30]. The system consists of two major parts: training and recognition. The training part consists of primitive (and relation) selection, grammatical inference, automata construction from the training patterns, while the recognition part consists of preprocessing, segmentation or decomposition, primitive (and relation) recognition, construction of pattern representation, and syntactic parsing analysis for the input testing pattern. Pattern representation PreproPrimitive Representation Syntax Input bcessing -+ Segmentation +(and relation) +construction -+analysis (parsing) pattern recognition
-Training Patterns
Classification
’
~
Primitive Grammatical Automata (and relation) +inference +construction selection
-
Fig. 1. Block diagram of a syntactic pattern recognition system.
The finite-state grammar, context-free grammar and context-sensitive grammar of the formal language are adopted in the description of 1-D string representation of the pattern [2,30]. The 1-D string grammars also include programmed grammar, indexed grammar, grammar of picture description language, transition network grammar, operator precedence grammar, pivot grammar, plex grammar, attributed The syntactic parsing analgrammar, . . . , etc. [11,25,26,28,30,37,40,68,70,73,75,78]. yses include finite-state automata, pushdown automata, top-down parsing, bottomup parsing, CockeYounger-Kasami parsing, Earley’s parsing,. . . , etc. [2,30]. The description power can be extended from 1-D string grammars t o highdimensional pattern grammars for the analysis of 2-D and 3-D patterns. The high-dimensional pattern grammars include tree grammar, array grammar, web grammar, graph grammar, shape grammar, matrix grammar,. . . , etc. [12,23,30,32, 46,48,66,69,72,84]. The syntactic parsing analyses include tree automata, array automata,. . . , etc. For consideration of substitution, insertion, and deletion errors in the pattern, the automata can be expanded to error-correcting automata to accept the noisy pattern or distorted pattern [1,44,45,63,78,79,82,88].The 1-D string grammars and high-dimensional pattern grammars also include stochastic grammars, languages, and the corresponding parsers [20,29,33,65,78,81].In this chapter, we present some fundamental algorithms for parsing 1-D strings, 1-D attributed strings, and 2-D patterns.
1.3 Syntactic Pattern Hecognition 63 2. Grammar and String Distance Computation
A grammar is a 4-tuple G = (VN, VT, P, S), where VN is the nonterminal set, VT is the terminal set, P is a finite set of production rules, and S E VN is the starting symbol. From the starting symbol S, a string (sentence) is derived by using the production rules of P, S + a0 a1 + ... + a , = a. The language generated by G is defined as L(G) = {ala is in V; and S a}, where * represents several derivation steps using production rules in G [2,30]. Depending on the form of the production rules, the grammars can be divided into finite-state grammar, context-free grammar, context-sensitive grammar, and unrestricted grammar. We can use a string to represent a pattern. Due to noise and distortion in the practical applications of syntactic pattern recognition, misrecognition of primitives is regarded as substitution errors, and segmentation errors are regarded as deletion and insertion errors [30]. For a given input string y and a given grammar G, we can find the minimum distance between y and z using a parsing technique, where string z is in L(G). The parsing technique is called minimum-distance error-correcting parser (MDECP) [1,30]. We use MDECP in the finite-state parsing, attributed string parsing and tree automaton.
*
Definition 2.1. Error transformations [30]. For two strings x,y , E Vk we can define three error transformations on T : Vk V; such that y E T(x). (1) Substitution error transformation: w1 aw2
(2) Deletion error transformation: w1 a w2 ( 3 ) Insertion error transformation: w1w2 w2 E v;.
+
T
4 w1 bw2, for all a , b E VT, a # b
T
4w1 w2, for all a E VT T tf w1 b w z , for all b E
VT, where w1,
Definition 2.2. Distance between two strings [1,30,54,83]. The distance between two strings x,y E Vk, dL(x,y), is defined as the smallest number of error transformations required to derive y from x. Example 2.1. Given a sentence x = abcd and a sentence y = accbd, then x = T T abcd 4 accd +-$accbd = y. dL(x,y) = 2. The graphical interpretation is as follows. y = a c
x-;
B a
_I
____
b
c
d
. + Insertion
,____,___-,
r _ _ _ _ I
6
I
I
.
.
.
64
K.-Y. Huang
Definition 2.3. Weighted error transformations [30]. For two strings x , y E V$, we can define three weighted error transformations on T: V$ + V$ such that y E T(x). w1, wz E V; are shown as follows: TI
(1) Weighted substitution error transformation w1 a wz S ( a ,b) w1 bw2, for a , b E VT, a # b, where S(a, b) is the cost of substituting a by b. Let S(a, a ) = 0. (2) Weighted deletion error transformation WI awz where D(a) is the cost of deleting a from w1 a w 2 . (3) Weighted insertion error transformation w1 w2 I(b) is the cost of inserting b.
T
4’ D(a) w1 w2,
for a
E
VT,
T,
4I(b) w1 bwz, for b E VT, where
Definition 2.4. Weighted distance between two strings. The weighted distance between two strings x, y E V;, d W ( x ,y), is defined as the smallest cost of weighted error transformations required to derive y from z. Algorithm 2.1: Weighted distance between two strings [83] Input: Two strings x = a1a2.. . a, and y = blb2.. . b,, substitution error cost S ( a , b ) , S(a,a ) = 0, deletion error cost D(a), and insertion error cost I(a), a , b E VT. output: d(x, y). Method: Step 1. D(0,O) = 0. Step 2. Do i = 1,n. D(i, 0) = D(i - 1 , O ) D ( u ~ ) Do j = 1,m. D(0,j) = D ( 0 , j - 1)-t- I(b3) Step 3. Do i = 1 , n ; do j = l , m . el = D(i - 1 , j- 1) + S ( a i , b j ) e2 = D(i - 1,j) D(ai) e3 = D ( i , j - 1) I&) D ( i , j ) = min(el,ez,es) Step 4. d(x,y) = D ( n , m ) . Exit.
+
+ +
We may consider context-deletion and context-insertion errors, then the deletion cost Del(a,b), deleting a in front of b or after b, and the insertion cost I(a,b), inserting b in front of a or after a , must be included. The relation between error probability and string distance has previously been presented in the detection of wavelets [41,42]. 3. Finite-State Grammar and Error-Correcting Finite-State Automata In a 1-D training wavelet (signal), pattern representation performs pattern segmentation and primitive recognition to convert a wavelet into a string of primitives (symbols, terminals). Grammatical inference infers finite-state grammar from a set of training strings. The finite-state grammar is expanded to contain three types of error symbol transformations: deletion, insertion, and substitution errors. The
1.3 Syntactic Pattern Recognition 65
Input seismic signals
Location of wavelets
’
- Recognition -----Training
String pattern representation Amplitude Segmen- +dependent --p + tation encoding
-
Error-correcting finite-state parsing
Classification results
+
I---finite-state grammar
grammar and automaton
Fig. 2. A classification system of seismic wavelets using error-correcting finite-state parsing.
automaton can be constructed from error-correcting finite-state grammar. Then, the minimum-distance error-correcting finite-state automaton can perform syntactic parsing and classification of input wavelets. The system of Fig. 2 is the example that can be used for the recognition of 1-D seismic Ricker wavelets [45]. We describe the techniques in the following sections. 3.1. Grammatical Inference
3.1.1. Inference of finite-state grummur
After the primitive recognition, each training wavelet can be represented by a string of primitives (symbols, terminals). A basic canonical definite finite-state grammar can be inferred from a set of training strings [30]. The canonical definite finite-state grammar G, associated with the positive sample set S+ = { z 1 , 2 2 , . . . , 5 , ) is defined as follows:
G , = (VN, VT, P, S) , where S is the starting symbol, and VN,VT, and P are generated using the following steps. (1) Check each string xi E Sf and identify all of the distinct terminal symbols used in the strings of S+. Call this set of the terminal symbols, VT. (2) For each string zi E S+,xi = uilui2 . . . ai,, generate the corresponding production rules:
Each
Zi,j
represents a new nonterminal symbol.
66
K.- Y. Huang
(3) The nonterminal symbol set VN consists of S and all the distinct nonterminal symbols Zi,j produced in Step (2). The set P consists of all the distinct production rules generated in Step (2).
Example 3.1. Given a training string abbc, the inferred canonical definite finite, P, s) is as follows: state grammar G ~ ( V NVT,
The production rule set P:
3.1.2. General expanded finite-state g r a m m a r
Due t o noise and distortion problems, three types of error symbol transformations may occur in the strings. After the inference of finite-state grammar, the grammar is expanded t o include the three types of error production rules. And the grammar is called the general expanded finite-state grammar. The following are the construction steps. (1) The original production forms of a finite-state grammar are A + aB, or A + a. Change A + a to A + aF, where F is the new nonterminal with ending terminal a. (2) The production forms added to account for substitution errors are A + bB, where a # b. ( 3 ) The production forms added to account for insertion errors are A 4 aA. (4) The production forms added to account for deletion errors are A + XB, where X is the empty terminal. We could put weights to production rules, especially the different weights in the three types of error production rules. The following is the algorithm.
Algorithm 3.1: Construction of general expanded finite-state grammar. Input: A finite-state grammar G = (VN,VT, P, S). Output: The general expanded finite-state grammar G = (V”, VT,, P’, S’), where P’ is a set of weighted production rules. Method: (1) For each production rule in P with the form A + a in the grammar G, change the rule to the from, A -+ aF, where F is a new nonterminal. (2) Set VTJ = VT U {A}, VN, = VN U {F}, S’ = S. ( 3 ) Let the production rule set P‘ = P with a weight of zero for each original production rule. (4) For each nonterminal A in VNJadd t,he production A + XA (with the weight 0) to P’.
1.3 Syntactic Pattern Recognition 67 (5) Substitution error production: For each production A -+ a B in P do For each terminal b in VT do If A t bB is not in the P’ then add the production A t bB (with the weight 1) t o P’. (6) Insertion error production: For each nonterminal A in V” do For each terminal a in VT do If A t aA is not in the P’ then add the production A t aA (with the weight 1) to P’. (7) Deletion error production: For each production A t a B in P(A # B) do If A -+ XB is not in the P’ then add the production A -+ XB (with the weight 1) to P’. (8) Add the production F -+ X (with weight 0) to P’. (9) Output G’.
Equal unit weight is assigned to each error production rule in the algorithm. Different weights may be assigned in Steps (5), (6) and (7).
Example 3.2. From the above Example 3.1, given the training string abbc, the inferred general expanded finite-state grammar G’(VN?,VTJ,P’, S’) using Algorithm 3.1 is as follows: VN, = { S , A , B , C , F } VTJ = {a,b,c,X} S’
=
{S}
The production rule set P’ is: (0) S t aA, 0 (1) A -+ bB, 0 (2) B -+ bC, 0 (3) C -+ cF, 0 (4) s + AS, 0 (5) A t XA, 0 (6) B -+ XB, 0 (7) c t XC, 0 (8) F -+ XF, 0 (9) S -+ bA, 1 (10) S t c A , 1 (11) A - + a B , 1 (12) A t cB, 1
(13) B t aC, 1 B -+ cC, 1
(14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25)
C ta F , 1 C -+ bF, 1 S -+ as,1 S t bS, 1 s -+ cs, 1 A t aA, 1 A t bA, 1 A -+ cA, 1 B t aB, 1 B t bB, 1 B -+ cB, 1
(26) (27) (28) (29) (30) (31) (32) (33) (34) (35) (36)
C -+ aC, 1 C + bC, 1 C -+ cC, 1 F ta F , 1 FtbF,1 F -+ cF, 1 S -+ XA, 1 A -+ XB, 1 B -+ XC, 1 C t XF, 1 F -+ A, 0
Production rules (9) to (16) handle the substitution errors, rules (17) to (31) handle the insertion errors, and rules (32) to (35) handle the deletion errors. The corresponding error-correcting finite-state automaton is shown in Fig. 3.
68
K.-Y. Huang
Fig. 3. Finite-state transition diagram of the general expanded grammar of Example 3.2.
3.1 3. Restricted expanded finite-state grammar
For insertion error, we can insert an error terminal symbol before or after some arbitrary terminal symbol. Then we can expand the form of the production rule A -+ aB with restricted insertion error as follows:
or
A
+ bB1, B1 -+ U B
(Insert b in front of a ) ,
A
+ uB2, B2 -+ bB
(Insert b after u )
The proposed algorithm is described in the following.
Algorithm 3.2: Construction of restricted expanded finite-state grammar Input: A finite-state grammar G = (VN,VT, P, S). Output: The restricted expanded finite-state grammar G’ where P‘ is a set of weighted production rules. Method:
=
(V”, VT,, P’, S’),
(1) For each production rule in P with the form A -+ a in the grammar G, change the rule to the form A + aF, where F is a new nonterminal. (2) Let P’ = P with the weight 0 for each original production rule. (3) Substitution error production: For each production A -+ aB in P do For each terminal b in VT do If A -+ bB is not in the P’ then add the production A -+ bB (with the weight 1) to P’. (4) Insertion error production: For each production A + aB in P do { Insert b in front of a }
1 . 3 Syntactic Pattern Recognition
(5)
(6) (7) (8) (9)
69
For each terminal b in VT do add the production A + bB1 (with the weight 1) t o P’, add the production B1 t a B (with the weight 0) to P’, and { Insert b after a } For each terminal b in VT do add the production A + aBz (with the weight 0) to P‘, add the production B2 tbB (with the weight 1) t o P’. Deletion error production: For each production A taB in P (A # B) do If A + XB is not in the P’ then add the production A -+ XB (with the weight 1) to P’. Set S’ = S,VT, = VT U { X},V” = all the distinct nonterminal symbols in P’. For each nonterminal A in V” do add the production A tXA (with the weight 0) to P’. Add the production F -+ X (with weight 0) t o P’. Output G’.
Example 3.3. From Example 3.1, given the training string abbc, the inferred restricted expanded finite-state grammar G”(VN”,V y , P”, S”) using Algorithm 3.2 is as follows: VN” = { S , A , B , C , D , E , F , G , H , I ,J , K , L } VT” = {a,b,c,X} S” (0) S + aA, 0 (1) A + bB, 0 (2) B tbC, 0 (3) C -+ cF, 0 (4) S + bA, 1 (5) S + cA, 1 (6) A + aB, 1 (7) A t cB, 1 (8) B + aC, 1 (9) B --t cC, 1 (10) C + aF, 1 (11) C +bF, 1 (12) S -+ XA, 1 (13) A -+ XB, 1 (14) B + XC, 1 (15) C + XF, 1 (16) S t a D , 1 (17) S t bD, 1 (18) S -+ cD, 1 (19) D + aA, 0 (20) S -+ aE, 0
(21) E t aA, 1 (22) E t bA, 1 (23) E -+ cA, 1 (24) A t aG, 1 (25) A + bG, 1 (26) A t cG, 1 (27) G t bB, 0 (28) A -+ bH, 0 (29) H + aB,1 (30) H + bB, 1 (31) H t cB, 1 (32) B + aI, 1 (33) B + bI, 1 (34) B -+ cI, 1 (35) I t b C , 0 (36) B 4 bJ, 0 (37) J + a C , 1 (38) J -+ bC, 1 (39) J + cC, 1 (40) C --t aK, 1 (41) C t bK, 1
(42) (43) (44) (45) (46) (47) (48) (49) (50) (51) (52) (53) (54) (55) (56) (57) (58) (59) (60) (61)
=
{S}
C t cK, 1 K + cF, 0 c -+ CL, 0 L + aF, 1 L -+ bF, 1 L -+ cF, 1 S +AS, 0 A -+ XA, 0 B t XB, 0 C -+ XC, 0 D tAD, 0 E tXE, 0 F -+ XF, 0 G tXG, 0 H tAH, 0 I + XI, 0 J + XJ, 0 K + XK, 0 L -+ XL, 0 F -+ A, 0
The transition diagram of the error-correcting finite-state automaton is similar to Fig. 3.
70
K.-Y. Huang
3.2. Minimum-Distance Error- Correcting Finite-State Parsing Input testing pattern strings can be analyzed by a finite-state automaton that can accept strings derived by finite-state grammar. Given a finite-state grammar G , there exists a finite-state automaton M such that the set of strings accepted by M is equal to the set of strings L(G) derived by the finite-state grammar G. The automaton M can be represented by a finite-state transition diagram [30]. The production rule A + aB in the finite-state grammar corresponds to the transition 6(A, a ) = B in the automaton. An input string can go from the initial state to the final state of the automaton if the string is accepted by the automaton. Here, each transition of the error-correcting finite-state automaton has two attributes, input terminal symbol and weight value. For the production rule A + aB with weight w , we use C A B ( U )= w as the cost representation in the transition of the automaton. We want t o parse the input string from the initial state to the final state with minimum cost. The following algorithm is proposed to compute the minimum cost (distance) by using the dynamic programming technique [30].
Algorithm 3.3: Minimum cost of error-correcting finite-state parsing with two attributes in each transition. Input: An error-correcting finite-state automaton with n nodes numbered 1 , 2 , . . . , n, where node 1 is the initial state and node n is the final state. Two attributes, terminal symbol and its cost function C i j ( a ) ,for 1 5 i, j 5 n, a E (V, U {A}), with Ci j(a) 2 0, for all i and j . An input testing string S. Output: MI, the minimum cost of the path from node 1 to node n when the parsing sequence is equal to the terminal sequence of the input string S. Method: (1) MI1 = 0 , M l j = maxint (a large number), 1 5 j 5 n. (2) For 1 5 k 5 n do For 1 5 j 5 n do M1j = min {Mlk
+ Ckj(A), 1 5 k 5 n }
( 3 ) Set h = 1 (4) Let Mik = Mlk, 1 I k 5 n.
For all 1 5 j 5 n do M1j = min {Mik C k j ( b ) ,1 5 k 5 n } , where b = S(h), the hth terminal symbol of S.
+
( 5 ) For 1 5 k 5 n do For 1 5 j 5 n do Mlj = min {Mlk
+ Ckj(A), 1 5 k 5 n}
(6) If h < IS(, then increase h by 1 and go to Step (4). If h = S, then go to Step (7). (SI is the number of terminal symbols in S. (7) Output MI, which is the minimum cost from node 1 to node n following the terminal sequence of the input string S. Then stop.
1.3 Syntactic Pattern Recognition
71
This algorithm is a minimum-cost error-correcting finite-state parsing when insertion] deletion, and substitution error transformations are considered. The cost function Cij(a) denotes the cost of moving from state i to state j when the input symbol is ‘a’. It is noted that C i j ( a ) = maxint, a large number, if there is no direct path from state i to state j under the symbol ‘a’. The value of C,j(a) comes from the production rule of the expanded grammar. M l j is the minimum cost from state 1 to state j. Steps (2) and (5) handle the transitions (i.e. deletion error). Step (4) handles the substitution and insertion errors.
Example 3.4. From the above Examples 3.1, 3.2, and 3.3, given the training string abbc, and its inferred error-correcting finite-state automata from the above two expanded grammars] an input testing string aabc is parsed by using Algorithm 3.3 and the parsing results from both automata with minimum cost are shown in Figs. 4 and 5. The minimum parsing costs are 1 from both automata.
Fig. 4. Parsing result of test string aabc using Algorithm 3.3 according to the state transition of general expanded grammar of Example 3.2 in Fig. 3. Minimum cost is 1.
Fig. 5. Parsing result of test string aabc using Algorithm 3.3 according to the state transition of restricted expanded grammar of Example 3.3. Minimum cost is 1.
72
K.-Y. Huang
4. Attributed Grammar and Minimum-Distance Error-Correcting
Earley’s Parsing
4.1. Introduction In the syntactic approach] after segmentation and primitive assignment of the digital signal, the same primitive may be repeated several times. This often makes the size of the pattern strings and inferred grammars unnecessarily large. Instead of keeping trace of all these identical primitives, we can use one syntactic symbol to represent the type of primitive with a n attribute t o indicate the length of that primitive. This leads to the application of the length attribute t o seismic and other similar digital signal analyses [44,59]. Attributed grammar has been applied in pattern recognition. You and Fu [87] implemented attributed grammars in shape recognition. Shi and Fu [76] proposed the use of attributed graph grammars. A grammar of which production is associated with a set of semantic rules is called an attributed grammar [30]. Both the inherited and the synthesized attributes often lead to significant simplification of grammars [30]. The advantages of using attributed grammars for pattern recognition are two-fold. First, the inclusion of semantic information increases flexibility in pattern description. Second, it reduces the syntactic complexity of the pattern grammar. The resulting grammar size for each pattern class is reduced. Similarity and dissimilarity measures between two strings have been discussed in many articles [49,50,54,83]. Here, the similarity measure between two attributed strings is proposed and is called the match primitive measure (MPM). Two parsing methods for attributed strings are proposed. One is the modified minimum distance error-correcting Earley’s parsing and the other is the parsing algorithm using match primitive measure (MPM). The system of Fig. 6 shows the example of two parsing methods that can be used for the recognition of seismic wavelets [44].
Input seismic signals
Location b of wavelets
Training wavelets
Attributed string representation
Attributed string representation
Error-correcting Earley’s parsing or parsing using match primitive measure
-
Inference of b attributed
grammar
Classification results
1.3 Syntactic Pattern Recognition
73
4.2. Attributed Primitives and Strings
In this chapter, each primitive is accompanied by a length attribute. That is, each pattern primitive, a , can be represented by a two-tuple,
where s is a syntactic symbol denoting the basic segment, and y represents the length of a. For example, a pattern string is aaadgggggeeaaagg
It can be simplified by merging the identical symbols. Thus, the above string becomes (a,3 ) ( 4 l)(g, 5)(e,2 ) ( a ,3)(9,2)
as the attributed string where each number represents the number of duplications for that symbol. This idea leads to some storage improvement in string representation.
4.3. Definition of Error Transformations for Attributed Strings In minimum-distance error-correcting parsing for context-free languages, three different types of error transformations (insertion, deletion, and substitution) have been defined [1,54,83]. In order to handle these three error transformations in the parsing of attributed strings, error transformations for attributed strings have been defined. Errors can be classified as global and local deformations. For global deformations, errors consist of insertion and deletion errors, and each of them can deform syntactically and semantically. (1) A syntactic insertion error is the replacement of a null string X (the length of X is zero) by a syntactic symbol, i.e.
(2) A semantic insertion error is the addition of a length attribute, i.e.
( s , y l ) 4(s,y2) where y l
< y2
When a syntactic insertion error has occurred, the associated semantic length is also added. However, a semantic insertion error can occur without any syntactic insertion error. (3) A syntactic deletion error is the replacement of a syntactic symbol by a null string A, i.e. ( s ,y) -+(X,O) .
74
K.-Y. Huang
(4) A semantic deletion error is the removal of an attribute from a semantic length, i.e. (s, y1) + (s,y2) where y l > y 2 . When a syntactic deletion error has taken place, the corresponding semantic length is also deleted. A semantic deletion error can occur without a syntactic deletion error. For local deformations, a substitution error can take place. (5) A syntactic substitution error is defined as the replacement of primitive s by another primitive t , i.e. (s,
Y) + (tl Y) .
A semantic substitution error is not defined, because it may be counted as a semantic insertion or deletion error. 4.4. Inference of Attributed Grammar
An attributed context-free grammar is a 4-tuple G = (V,, Vt, P, S), where V, is the nonterminal set, Vt is the terminal set, P is a finite set of production rules, and S E V, is the starting symbol. In P, each production rule contains two parts: one is a syntactic rule, and the other is a semantic rule. Each symbol X E (V, U V,) is associated with a finite set of attributes A(X); A(X) is partitioned into two disjoint sets, the synthesized attribute set Ao(X) and the inherited attribute set Al(X). The syntactic rule has the following form:
where k means the kth production rule. The semantic rule maps values to the attributes of X ~ , O Xk,l, , X ~ J ,. . ,Xk,nk. The evaluation of synthesized attributes is based on the attributes of the descendants; therefore it proceeds in bottom-up fashion. On the other hand, the evaluation of inherited attributes is based on the attributes of the ancestors; therefore it proceeds in top-down fashion. To explain the inference procedure, let us consider the example of the previous string aaadgggggeeaaagg,
where each primitive has a unit length attribute 1. First it will be converted into the following string by merging identical primitives:
Let the upper case character be nonterminal and the lower case character be terminal. Then, we can infer the following attributed grammar.
1.3 Syntactic Pattern Recognition 75
Syntactic rules
Semantic rules
(1) S -+ ADGEAG (2) A + a A (3) A + a (4) D + dD (5) D +d (6) E + eE (7) E + e (8) G + 9G (9) G + 9 where L denotes the inherited length attribute, and y denotes the synthesized length attribute. The number right after the nonterminal symbol in the semantic rules is used to distinguish between occurrences of the same nonterminal. For example, in the production rule (2), A1 represents the nonterminal A on the left side; A2 represents the nonterminal A on the right side of the syntactic part. It is noted that the inherited length attribute, L, is not down to the descendents as it usually is; rather it is used t o maintain the semantic information of the training string and as a reference for comparison in parsing. For simplicity, let y(a) = 1 for all a E Vt. Consider the second input string
aakkdddf f eeeea. We convert it into
( a ,2)@, 2 x 4 3 ) ( f , 2)(e, 4)(a, 1) and add the following productions to the inference grammar. Syntactic rules
Semantic rules
S -+ AKDFEA
L(A1) = 2, L(K) = 2, L(D) = 3, L(F) = 2, L(E) = 4, L(A2) = 1 K + kK Y(K1) = Y(k) + Y(K2) K-+k Y(K) = Y(k) F + fF y(F1) = Y ( f ) + Y(F2) F-+f Y(F) = Y ( f ) For the new input string, there will be no need to add those production rules, A -+ a A , A + a , . . . , etc. One production rule is created for each input string, i.e. the first production rule in the above example. In fact, there are (2m+n) production rules for a set of n training strings, where m is the number of nonterminal symbols. We now formulate the inference algorithm of attributed grammar which uses the length attribute [59].
Algorithm 4.1: Inference of an attributed context-free grammar. Input: A set of training strings. Output: An inferred attributed context-free grammar. Method:
76
K.-Y. Huang
(1) Convert each input string t o the attributed string by merging identical primitives. (2) For each input attributed string a l a 2 a 3 . . ’ a k , add the production S + AlAzA3.. . A k t o the inference grammar, where Ai is the nonterminal corresponding to terminal ai, and the semantic rule L(Ai) = yi, 1 5 i 5 k, where yi is the length attribute of primitive ai. (3) For each primitive a,add the production rule A -+ aA, y(A1) = y(a)+y(A2) and A + a,y(A) = y ( a ) to the inference grammar, if they are the new production rules. This inferred grammar will generate excessive strings if we only apply syntactic rules. However, we can use semantic rules (inherited attributes) t o restrict the grammar so that no excessive strings are generated. 4.5. Minimum-Distance Error-Correcting Earley ’s Parsing f o r
Attributed Strings
A modified Earley’s parsing algorithm is developed here for attributed contextfree languages. Here, errors of insertion, deletion, and substitution transformation are all considered in the derivation of Earley’s item lists. Let the attributed grammar G = (V,, V t ,P, S) be a CFG (context-free grammar), and let z = b l b 2 . . . b, be an input string in Vt*. The form [A + a . p, 5 , y, i] is called a n item for z if A + ap is a production in P and 0 5 i 5 n [ 2 , 2 2 ] . The dot in a . ,L? between a and p (we use a . p in the program output) is a metasymbol not in V, or V t ,which represents the parsing position; 5 is a counter for local syntactic deformation which accumulates the total cost of substitution of terminal symbols. When A = S,y is used as a counter for global deformation which records the total cost of insertion and deletion errors. On the other hand, if A # S then y is used as the synthesized attribute of A. The meaning of index i is the starting parsing position of the string, and it is the same pointer as the conventional item of Earley’s parsing algorithm. The parsing algorithm for an input string z is shown in the following. Algorithm 4.2: Minimum-distance error-correcting Earley’s parsing for an attributed string. Input: An attributed grammar G = (V,, Vt, P, S) and a test string z = blb2 . . . b, in Vt*. Output: The parse lists 10,11,. . . ,I,, and the decision whether or not z is accepted by the grammar G together with the syntactic and semantic deformation distances. Method: (1) Set j = 0 and add [S + . a , 0, 0, 01 to I j if S + a is a production in P. (2) Repeat Steps (3), (4) and (5) until no new items can be added t o Ij. (3) If [B + ( . , x l , y l , z ] is in Ij, B # S, and
1.3 Syntactic Pattern Recognition 77 (a) if [A + a . BP, 2 2 , y2, Ic] is in Ii and A # S, then add item [A + a B . p, 21 2 2 , yl y2, k] to Ij. (b) if [S -+ a . BP, 2 2 , y2, k] is in Ii, then add item [S + a B . p, 21 2 2 , y2 IL(B) - yll, k] to Ij. (c) if [S + a . Cp, 2 2 , y2, Ic] is in Ii, C # B, then add item [S + a . Cp, 2 2 , yl y2, Ic] t o Ij. (d) if [S 4a .,2 2 , y2, Ic] is in Ii, then add item [S + a ., 2 2 , yl y2, Ic] to Ij. If B + [ is a production in P, and if [A + a . BP, 2,y, i ] is in Ij, then add item [B 4 .[, O , O , j] to Ij. If [S + a . BP, 2 , y, i ] is in I j , then add item [S -+a B . p, 2,y L(B), i ] t o Ij. If j = n, go to Step (8); otherwise increase j to j 1. For each item [A + a . a p , z , y , i ] in Ij-1, add item [A + a a . p , z + S ( a , b j ) , y + y(a),i]to Ij, where y ( a ) is the synthesized attribute of a. For simplicity, let y ( a ) = 1 for all a in Vt. S(a, b j ) is the substitution cost, and S ( a ,a ) = 0. Go to Step (2). If item [S -+ a ., 2 , y, 01 is in I,, then string z is accepted by grammar G where z is the local deformation distance, and y is the global deformation distance; otherwise, string z is not accepted by grammar G. Exit.
+ +
+ +
+
+
(4) (5) (6) (7)
(8)
+
+
In the above algorithm, Step (3b) handles the semantic insertion and deletion errors, Steps (3c) and (3d) handle the syntactic insertion errors, Step (5) handles the syntactic deletion errors, and step (7) handles the syntactic substitution errors. It is possible for collision t o occur in the process of developing a new item, i.e. the old item has already been in the list when a new item is to be put in the list. Under this situation, the one with the minimum distance (minimum of z + y) is selected for that item. Actually, collision may occur with only the items related to S-productions, because insertion and deletion transformations are allowed for those items only. Training string : ahhc The attributed grammar G(Vn,Vt,P,S)is as follows : Vn = ( S , A, B, C) V, = la. h, c )
s
=IS}
The production set P i s as follows : Syntactic rules Semantic rules
Fig. 7. Training string abbc and its inferred attributed grammar for Earley’s parsing.
78
K.-Y. Huang
‘est string aabc and its item lists :
I[O] contains [S->ABC. [C->.c [C->.cc [S->AB.C [B->.b [B->.bB [S->A.BC [A->.a [A->.aA [S->.ABC
I[3] contains [A->aA. [B->bB. [C->cC. [C->.c [C->.cC [B->.b [B->.bB [A->.a [A->aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->AB.C [S->ABC. [S->A.BC [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
,0,4,01 0 0 0I ,O,O,O] ,0,3,0l ,O,O,OI ,O,O,O] 7
2
7
,0,1,0]
,O,O,OI ,O,O,Ol ,O,O,O]
,1,3,01 ,2,3,01 ,3,3,01 ,0,0,31 ,0,0,3] ,0,0,31 ,0,0,3l .0,0,3l ,0,0,31 ,2,2,1] .1,2,1l ,0,3,0] ,1,2,1] ,l,O,OI , I ,1,0] ,0,2,0] ,1,1,2] > 1,1,2l ,0,1,2] ,0,1,21 ,1,1,21 ,1,1,21
I[1] contains [C->.c [C->.cc [B->.b [B->.bB [A->.a [A->.aA [S->.ABC [S->A.BC [S->AB.C [S->ABC. [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
I[4] contains [C->cC. [B->bB. [A->aA. [A->aA. [B->bB. [C->cC. [C->.c [C->.cC [B->.b [B->.bB [A->.a [A->aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->ABC. [S->AB.C [S->A.BC [A->a.A [A->a. [B->b.B [B->b. rc->c.c
,O70,1l ,0,0,1]
,0,0,11 ,0,0,1]
,o,o,11 ,0,0,1] ,0,1,0] ,O,O,O] .0,2,0] ,0,3,0] ,0,1,0] ,o,1,OI ,1,1,0] ,1,1 ,OI ,1,1,0] ,1,1 ,OI
I[2] contains [C->.c [C->.cc [B->.b [B->.bB [A->.a [A->.aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->ABC. [S->AB.C IS->A.BC [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
.0,0,21 ,0,0,21 ,0,0,2] ,0,0,2] ,0,0>21 ,O,O,21 ,2,2,0] ,2,2,0] ,0,2,0] ,0,2,0l .1,2,0] , I , I ,O] ,O,l ,O] ,0,1,1I 8 . 1 >1l ,1,1,1] .1,1,1l ,l,l,ll ,l,~?~l
.3,4,0] .3,4,0] .2,4,0] ,2,3,1] ,2,3,1l ,2,3,11 ,0,0,4l ,0,0,4] ,0,0,41 ,0,0,4l ,0,0,4l ,0,0,41 ,1,2,2l ,1,2,2] ,0,4,0] ,2,2,2] ,1,0,0] .1,1,0] ,0,3,0] ,1,1,3] ,I, 1 3 1 ,0,1,3] ,0,1,31 .I,M
Fig. 8. Item lists of t h e Earley’s attributed parssing on the test s t r i n g a u k .
1.3 Syntactic Pattern Recognition 79 Since the error-correcting grammar is ambiguous, the time complexity is O ( n 3 ) , and the space complexity is O ( n 2 ) ,where n is the length of the input string. The parsing is inefficient if the length of input string is large. 4.6. Experiment
Given a training string abbc and using Algorithm 4.1, the inferred attributed grammar is shown in Fig. 7. An input test string aabc is parsed by Algorithm 4.2. The corresponding item lists are shown in Fig. 8. As we can see from the derived item lists, all three kinds of errors are considered. The corresponding items are generated for each possible error transformation. Because the item [S + ABC ., 1 , 0 , 0 ] is in the 14 list, the string aabc is accepted with local syntactic deformation distance 1 and global deformation distance 0. 5. Parsing of Attributed Strings Using Match Primitive Measure (MPM) Although the modified Earley’s parsing algorithm considers all three types of errors, the parsing speed is inefficient. Here, the parsing of an attributed string using the match primitive measure (MPM) is proposed. The similarity measure of attributed strings is discussed in the following.
Fig. 9. The partial MPM f[i,j] computed from f [ i , j - 11 and f[i
-
1, j ]
5.1. Match Primitive Measure- Similarity Measure of Attributed String Matching The match primitive measure (MPM) is defined as the maximum number of matched primitives between two strings. Here, the similarity measure between two attributed strings is proposed. The computation of MPM between two lengthattributed strings can be implemented by the dynamic programming technique on grid nodes as shown in Fig. 9. For each node, three attributes are associated, i.e.
80
K.-Y. Huang
(f,h, w). Let a be an attributed string, where a[i] denotes the ith primitive in a; a[i].s and a[i].y denote the syntactic symbol and length attribute of a[i], respectively. Let ( i l j )indicate the position in the grid. f [i,j]represents the MPM value from point (0, 0) to ( i , j ) ,i.e. the MPM value between two attributed substrings (a[l] . s, a [ l ] . y)(a[2].s,a[2].y). . . (a[i].s,a[i].y)and (b[l].s,b[l]-y)(b[2].s,b[2].y).. . ( b [ j ] . sb[j].y) , of attributed strings a and b. h [ i , j ]and w [ i , j ] represent the residual length attributes of primitive a[i]and b [ j ] ,respectively, after the match primitive measure (MPM) between two attributed substrings (a[l] . s, a [ l ] . y)(a[2]. s, a[2]. y) . . . ( a [ i ]. s, a[i]. y) and (b[l]. s, b [ l ] . y)(b[2] . s , b[2] . y) . . . ( b [ j ]. s, b [ j ]. y) of attributed strings a and b. The partial MPM f [i,j]can be computed from the partial MPM’s f [i - 1,j] and f [i,j - 11 as shown in Fig. 9. The following algorithm is proposed t o compute the MPM between two attributed strings. Algorithm 5.1: Computation of the match primitive measure (MPM) between two attributed strings. Input: Two attributed strings a and b. Let a = (a[l] . s, a[I] . y)(a[2] . s, a[2] . y) . . . (a[rn] . s, a[m] . y) b = (b[l] . S, b[l] . y)(b[2] . S , b[2] . Y ) . . . ( b [ n ]. S , b[n]. Y), where m,n are the number of primitives of a and b, respectively. Outout: The maximum MPM S(a, b). Method: (1) f[O, 01 := 0; h[O,01 := 0; w[O,01 := 0; (2) for i := 1 to m do begin f [ i , O ] := 0; h[i,O]:= a[i]. y; w[i,O] := 0; end; (3) for j := 1 to n do begin f [O, j] := 0; h[O,j]:= 0; w [ O , j ] := b[j] . y; end; (4) for i := 1 to m do for j := 1 to n do begin nodl:= hmove ( i ,j ) nod2:= vmove ( i , j ) if nod1.f >nod2.f then node [Z,j]:= nod1 else node [Z,j]:= nod2; end; (5) output S ( a , b ) := f [m,n]/dGqE;where y1 = a[i] . y,y2 = cj b[jl . y Functions hmove and vmove are written as follows: function hmove (i, j ) : node-type; {node (i - 1,j)+ node ( i , j ) } begin
1.3 Syntactic Pattern Recognition 81 if a[i]. s # b[j] . s then d l := 0; else d l := min(v[i - I , j ] , a [ i ] .y); hm0ve.f := f[i - l , j ] d l ; hm0ve.h := a[i] . y - d l ; hm0ve.v := v[i - l , j ]- d l ; return(hmove) ;
+
end; function vmove (i,j): node-type; {node ( i , j - 1) + node ( i , j ) } begin if a[i] . s # b [ j ]. s then d l := 0; else dl: = min(h[i,j - I],b [ j ]. y); vmove-f := f[i,j - 11 d l ; vm0ve.h := h [ i , j- 11 - d l ; vmovew := b [ j ]. y - d l ; return (vmove); end;
+
J
\
0
f=O h=O v=o
f=O h=3 v=o
1
f=O h=O v=4
f=3 h=O v=l f=3
2
f=O h=O v=2
h=O v=2
3
f=O h=O v=2
f=3 h=O v=2
4
f=O h=O v=4
f=3 h=O v=4
5
f=O h=O v=1
f=3 h=O
6
f=O h=O v=3
f=3 h=O v=3
v=l
In the above algorithm, two functions are used. Function hmove is used t o compute the variation of attributes ( f , h , u ) from node (i - 1,j) to node ( i , j ) . Function vmove is used to get the value of (f,h, u)at point (i, j ) from point (i, j-1). An example of the MPM computation of two attributed strings is shown in Fig. 10. The normalized MPM value is calculated. 5.2. Inference of Attributed Grammar For the parsing of an attributed string using the property of the MPM, the attributed grammar for the training strings is inferred first.
Algorithm 5.2: Inference of attributed grammar. Input: A set of training strings. Output: An attributed grammar. Method: (1) Convert each input string to the attributed string by merging identical primitives. (2) For each input attributed string a1a2a3 . . . ak, add to the grammar the production S -+AlA2As . . . Ak, where Ai is the nonterminal corresponding to terminal a,; and the semantic rule L(Ai) = yi, 1 5 i 5 k , where yi is the length attribute of primitive ai. (3) For each primitive a , add the production rule A + a , y(A) = y ( a ) and y(a) = y, where y is the length attribute of primitive a. The example is shown in Fig. 11
Training string : abbc The attributed grammar G(Vn,Vt,P,S) is as follows Vn = {A,B, C, SJ Vt = {a, b, c ) s ={S) The production set P is as follows : Syntactic rules Semantic rules
Fig. 11. Training string abbc and its inferred attributed grammar for the M P M parsing.
1.3 Syntactic Pattern Recognition 83 5.3. Top-down Parsing Using M P M Given an attributed grammar G and input attributed string z , the value of MPM between z and L(G), the language generated by the grammar G, is calculated. Consider an S-production rule in the grammar, which has the form
For each nonterminal a t the right-hand side of S-production rule, two attributes are associated with it. f [ k ]denotes the MPM value calculated from the beginning up to the parse of kth nonterminal. h [ k ]is a kind of residual attribute used for the calculation later on. The proposed algorithm to compute the MPM between z and L(G) is described in the following.
Algorithm 5.3: Top-down parsing using the MPM. An attributed grammar G = (Vn,Vt, P, S) and an input string z . Let m = the number of primitives in z . n = the length of z = z [ i ]. y. Output: The maximum MPM between z and L(G). Method:
Input:
xi
(1) Set N = the number of S-production rules, and max-MPM = 0. (2) Set f [ O ] = 0 and h[O]= 0. (3) For all 1 5 k 5 N do Steps (4)to (10). (4) Apply the kth S-production rule with the form Sk + Ak,lAk,Z. . . Ak,,, , where m k is the number of nonterminals at the right-hand side of the kth S-production rule to do Steps (5) to (8). (5) For all 1 5 i 5 mk do
f [;I = 0; Wl = L(Ali,z); 1. (6) For all 1 5 j 5 m do Steps (7) and (8). (7) Set v0 = z [ j ]. y and v = v0. (8) For all 1 5 i 5 mk do Apply production rule Ak,i t Uk,i. (a) if z [ j ]. s = ak,i, then d l = min(y(ak,i),ZI) else d l = 0; f l = f [ i - 11 d l ; hl = Y ( a k , i ) - d l ; w l = Y - dl; (b) if z [ j ]. s = u k , i , then d l = min(h[i],v0) else d l = 0; f2 = f [ i ] dl; h2 = h[i]- d l ; v2 = vo - d l ;
+
+
84 K.-Y. Huang
(c) if f l
> f 2 then
{ f[iI = f l ; h[i]= h l ; v = vl; } else{ f[i] = fa; h[i] = h2; v = v2; } (9) MPM = f [ m k ] / d x , where l k = CzlL(Ak,i). (10) If MPM>max-MPM, then max-MPM = MPM. (11) Output max-MPM. Here the normalized MPM is calculated. Algorithm 5.3 is obtained from the comparison between the input string and the string generated by the S-production rule.
Example 5.1. The training string abbc and its inferred attributed grammar are shown in Fig. 11. One input string aabc has been tested, and the parsing result is shown in Fig. 12. The MPM value is 0.75 after normalization. Test string : aabc = (a,2)(b,l)(c,l) 1
2
3
0
0 1
1 1
0
1
0 1
1
2
1
1
3
1
1 2 2
2 3
Fig. 12. Parsing result using M P M for the test string aabc, the MPM value is 0.75.
6. Tree Grammar and Automaton 6.1. Introduction Tree grammars and the corresponding recognizers, tree automata, have been successfully used in many applications such as: English character recognition, LANDSAT data interpretation, fingerprint recognition, classification of bubble chamber photographs, and texture analysis [30,57,64,67,89]. Fu pointed out that “By the extension of one-dimensional concatenation to multidimensional concatenation strings are generalized to trees.” [30]. Comparing with other high dimensional pattern grammars: web grammar, array grammar, graph grammar, plex grammar, shape grammar,. . . , etc. [30], tree grammar is easy and convenient to describe a pattern using data structure of the tree, especially in the tree traversal and in the substitution, insertion, and deletion of a tree node.
1.3 Syntactic Pattern Recognition 85
Input testing seismogram
Recognition Training
-
Preprocessing: (1) Envelope (2) Thresholding (3) Compression (4) Thinning
~
1 Training seismic
Pattern representation: (1) Pattern extraction (2) Primitive recognition (3) Tree construction
Pattern representation: ( 1 ) Pattern extraction
Error-
conecting
Classification
tree automata
grammar
Fig. 13. A tree automaton system for seismic pattern recognition.
An example of applying tree grammar and automata t o recognize 2-D synthetic seismic patterns is presented. The system of a tree automaton is shown in Fig. 13. In the training part of the system, the training seismic patterns of known classes are constructed into their eorresponding tree representations. Trees can infer tree grammars [5,7,55,56,65].Several tree grammars are combined into one unified tree grammar. Tree grammars can be used t o generate the error-correcting tree automaton. In the recognition part of the system, each input testing seismogram passes through preprocessing [38,47],pattern extraction, and tree representation of the seismic pattern. Then each input tree is parsed and recognized by the errorcorrecting tree automaton into the correct class. Three kinds of tree automatons are adopted in the recognition: weighted minimum distance structure preserved errorcorrecting tree automaton (SPECTA), modified maximum-likelihood SPECTA, and minimum distance generalized error-correcting tree automaton (GECTA). We have some modifications on the methods of weighted minimum distance SPECTA and maximum-likelihood SPECTA. 6.2. Tree Grammar and Language
A tree domain (tree structure) is shown below [30]. Each node has its ordering position index and is filled with a terminal symbol. 0 is the root index of a tree. 0
0.1
/ I \ 0.2
/ \ 0.1.1
0.1.2
...
0.1.1.2
. ..
/ \ 0.1.1.1
0.3 . ..
86
K.-Y. Huang
Each node has its own children, except the bottom leaves of the tree. The number of children at each node is called the rank of the node. Although there are different kinds of tree grammars [30], we use the expansive tree grammar in the study because of the following theorem. Theorem 6.1. For each regular tree grammar Gt, one can effectively construct an equivalent expansive grammar Gt’, i.e. L(Gt’) = L(Gt) [30]. An expansive tree grammar is a four-tuple Gt = (V, T , P, S), where = V N u VT, VN = the set of nonterminal symbols, VT = the set of terminal symbols, S : the starting nonterminal symbol, T : the rank of terminal symbol, i.e. the number of children in the tree node, and each tree production rule in P is of the form
v
(11x0 -+
x
(2)Xo -+ x,
or
/I...\
x1 x2
&(z)
where x E VT and XO,X I , X2,. . . , XT(z)E VN. For convenience, the tree production rule (1) can be written as XO + XXlX2 . . . Xr(z) [30]. From the starting symbol S, a tree is derived by using the tree production rules of P, s -+(Yo =+ a1 =+ ... =+ (Ym = a. The tree language generated by Gt is defined as L(Gt) = { a is a tree IS (Y in G t } , where * represents several derivation steps using tree production rules in Gt.
>
6.3. B e e Automaton
The bottom-up replacement functions of the tree automaton are generated from the expansive tree production rules of the tree grammar. Expansive tree grammar is Gt = ( V , r , P , S ) , and tree automaton is Mt = ( Q , f , S ) , where Q is the set of states, f are the replacement functions, and S becomes the final state.
xo -+ If tree production rule
X
/ I
x1 xz
... \
is in P, then the bottom-up
x,
replacement function in the tree automaton can be written as
xo 4/ XI
x
I .. . \ x2
xn
, or
f ( 2 14x0 / I . .. \ , or fz(X1,X2,. .. , X n ) XI
x2
xn
-+
XO.
The tree automaton is an automatic machine to recognize the tree and has the tree bottom-up replacement functions which are the reverse direction of the tree production rules. Tree grammar uses forward and top-down derivation to derive the tree. The tree automaton uses a backward replacement of the states from the bottom to the root of the tree. If the final replacement state is in the set of the final
1.3 Syntactic Pattern Recognition 87 states, then the tree is accepted by the automaton of the class. Otherwise the tree is rejected.
Example 6.1. The following tree grammar Gt = (V, T , P, S) where V = { S , A , B , $ , a 1 b } , V ~= {*,$,.L a , + b } , r ( a ) = { 2 , 0 } , r ( b ) = {2,0}, r ( $ ) = 2, and P: (1) S + $
(2) A
+a
/ \ A
(3) B
+b
A
+a
(5) B
+b
/ \
/ \
B
(4) A
B
A
B
can generate the patterns, for example, using productions (l),(4), and (5),
or using productions ( l ) ,( 2 ) , ( 3 ) , (4), (5), (4),and (5). $
1
a
b I \ a b
b
The tree automaton which accepts the set of trees generated by Gt is
Mt = ( Q , f a , f b , f $ , F ) , w h e r e Q = { 4 A , q B , Q S } , F = { q ~ } ,a n d f : f a = q A , f b = qB, f a ( Q A , q B ) = ~ Af bI( q A , q B ) = q B , f$(qA,
qB) = 4%.
Example 6.2. The following tree grammar can be used to generate trees representating L-C networks.
Gt
=
(V, T , P, S), where V = {S, A, B, D, E, $, Vin, L, C, W}, VT = {$, Vin, L, C,W}, 2, r(Vin) = 1,r(L) = {1,2},r(C) = 1,T(W) = 0, and P:
T($) =
(1) S + $
(2) A
/ \
A
B
+ Vin I E
(3) B
+L
(4) B
/ \
D
B
+L I
D
(5) D
+C I
E
(6) E
+W
88
K.-Y. Huang
For example, after applying productions (l),(2), (3), ( 6 ) , (5), (4), (6), (5), and (6), the following tree is generated.
$
I\ i'n L I I\ w C L I
I
ni'
w c
The tree automaton which accepts the set of trees generated by Gt is Mt =
(Q, fw, fc, f ~fv,,, , f$,F), where Q = {QE, 4 D , QB, q.4, qs}F = { q s } , and f:
fW()=qE,fC(qE)=qD,
fL(qD)=qB,fL(qD,QB)=qB,
fV,,(qE)=qA,
f$(qA,qB)=qS-
Example 6.3. Tree representation and tree grammar of seismic bright spot pattern. (A) Bright spot pattern: The seismogram of the primary reflection of bright spot is generated from a geologic model [19,71]. The geologic model is shown in Fig. 14 and the seismogram Distance
0.0
Density=2 0 gm/cm**3 Velocity=2.0km/sec
D=2.3 V=2.3
-1.0
I
/
D12.270 V=2.225
D=2.8 V=2.8
.." Fig. 14. Geologic model.
\
1 . 3 Syntactic Pattern Recognition
89
Fig. 15. Bright spot seismogram.
is shown in Fig. 15. After preprocessing, thresholding, and compression in the vertical direction, the bright spot pattern can be shown below. We can scan the pattern from left to right, then top t o bottom. The segments (branches) can be extracted in the tracing. Eight directional Freeman’s chain codes [27] are used to assign primitives t o the segments. From expanding the segments (branches), the tree representation of the seismic bright spot pattern is constructed. And the tree can infer the production rules of the tree grammar. $0
X
xxxxx xxxxx I xxx xxx xxx xxxxxxxxxxxx xxx 5 xxx 0 xxx 7 xx xxxxxxxxxxxxxxxxyxxxxxx xx 5 xxxxx 0 xxxxx 7 xx xx 5
Primitives: eight directional Freeman’s chain codes [27] and terminal symbol 0 (the neighboring segment have already been expanded), 3
2
1
4 1% O
5
--
-6 7 Primitives
90
K.-Y. Huang
(B)(1) Tree representation of bright spot after scanning and tree construction: $
I \ 5 7 I\ I \ 5 0 0 7 I \ I \
5
0
@ 7
(2) Corresponding tree node’s position: 0
/ \ 0.1 I \ 0.1.1 0.1.2 I\ 0.1.1.10.1.1.2
0.2 I \
0.2.1 0.2.2 I\ 0.2.2.1 0.2.2.2
(C) Tree grammar: Gt = (V, r, P, S), where V = set of terminal and nonterminal symbols = {$, 0, 5, 7 , 0, S, A, B, C, D, E, F, GI HI I, J > , VT = the set of terminal symbols = {$, 0, 5, 7, Q},$: the starting point (root) of the tree, @:represents that the neighboring segment has already been expanded, S: the starting nonterminal symbol, T : r ( 5 ) = r(7) = { 2 , 0 } , r ( $ )= 2 , r ( @ ) = O,r(O) = 0, and P: (2)A -+ 5 (3)Bj 7 (4)C+ 5 I \ I \ I \ I \ G H C D E F A B (7)F-+ 7 (8)G-+5 (9)HjO (lO)I+@ I \
(1) S-+ $
I
(6)E-+@
(5)D+O
(11)J-+7
J
The tree derivation steps are as follows:
j
I\ A
B
(4,5 , 6, 7, 8, 9, 10, 11)
(3)
(2)
(1) S + $
$
I \ 5 B I \ C D
$
-+
I 5
_ _ - - . _ - - . -> -._ $- . . - - . -
\
7 I \ I \ C D E F
I
\
5 7 I \ I \ 5 0 @ 7 I \ I \ 5 0 @ 7
1.3 Syntactic Pattern Recognition 91 So following the steps as described in (A) and (B), each seismic pattern can be represented as a tree. From the steps (B) to (C), a tree can infer tree production rules. The tree production rules can derive trees. Each tree corresponds t o its pattern class. (D) Tree automaton from tree production rules of (C): A tree automaton generated by Gt is Mt = (Q, f$, fo, f5, f7, f ~S),, where Q = {S, A, B, C, D, E, F, G, H, I, J}, S: the final state, and the bottom-up replacement function f: (11) f7 + J (10) f~ + 1 (9) fo + H (8) f5 + G (7) f7(I,J) + F (6) f~ + E (5) fo + D (4) f5(G,H) -+ c (3) f 7 ( W + B (2) f5(C,D) + A (1) f$(A,B) + s.
The number on the left hand side of the bottom-up replacement function corresponds to the number of production rule of the tree grammar, and the bottom-up replacement function is the reverse of the corresponding production rule. The above tree in (B) can be replaced by replacement functions step by step from the bottom to the root of the tree and accepted by this tree automaton Mt as the seismic bright spot pattern. 6.4. Tree Representations of Patterns
In the tree automaton system, patterns must be extracted from image data and constructed as the tree representations. In order to construct the tree representation of a pattern automatically, a scanning algorithm is proposed. The following Algorithm 6.1 is the construction of tree representation from scanning an input pattern. The algorithm works for both four-neighbor and eight-neighbor connectivity. The scanning is from left t o right and top to bottom on the binary image. In the algorithm, breadth-first tree expansion is adopted to construct the tree representation such that the depth of the tree will be shorter and the parsing time of the input tree by tree automaton will also be shorter in parallel processing.
Algorithm 6.1. Construction of a tree representation from a pattern. Input: Image of a pattern after thinning. Output: Tree representation of a pattern. Method: (1) While scanning image from left to right and then top to bottom, (a) If the scanning reaches a point of the pattern, then the point is the root (node) of a tree. (b) Trace all following branches (segments) from a node. And assign the terminal symbol to each branch (segment) by the chain code. (c) If the lower end of each branch (segment) has sub-branches (sub-segments), then go to Step (b), trace the sub-branches (sub-segments) and expand all children nodes from the left-most child node in the same tree level. After the whole children nodes in the same level are expanded, then go t o Step (b) to expand the descendants from the left-most in the next down tree level.
92
K.-Y. Huang
Expand level by level until there is no node to be expanded. Then a pattern is extracted and its corresponding tree representation is constructed. There may exist several patterns in the image data. Algorithm 6.1 can extract a pattern. The following Algorithm 6.2 can extract all the entity patterns in the image and construct their tree representations of all the patterns in the image.
Algorithm 6.2. Extract all patterns and construct their tree representations from the binary image. Input: Image data after thinning. Output: Tree representations of all patterns. Method: (1) Scan image from left t o right, then top to bottom. (2) When the scan reaches a point of a pattern, follow Algorithm 6.1 to extract one pattern and construct its corresponding tree representation, then erase the position of current pattern from the image. (3) Go to Step (l),continue to scan, extract the next pattern, and construct the tree representation until there is no pattern to be extracted.
6.5. Inference of Expansive Free G r a m m a r In the training part of tree automaton system of Fig. 13, training patterns must be given in order t o infer the production rules. The following Algorithm 6.3 is presented to infer expansive tree grammar from tree representation of the pattern.
Algorithm 6.3. Inference of expansive tree grammar. Input: Tree representation of a pattern. Output: Expansive tree grammar. Method:
(1) From top to bottom of a tree, for every node of the tree, derive a tree production rule:
x+
a
/ I
x1
x2
... \ x n
where X: the nonterminal symbol assigned to the node, a: the primitive (terminal) of the node, and XI, Xa,. . . ,Xn: the nonterminals of the direct descendants (children) to cover the next level subtrees. (2) Go to Step (1) to handle the other nodes in the same level of the tree, until every node is reached. Handle each node level by level of the tree. This algorithm can be implemented in one recursive procedure.
1.3 Syntactic Pattern Recognition 93
6.6. Weighted Minimum-Distance SPECTA Due to noise and distortion there are some possibilities that terminals may be recognized as the neighboring terminals in the process of primitive recognition. The tree may have substitution error terminals, then tree automaton must be expanded to recognize error trees. Given a tree W below, node b at position 0.1 is substituted by node x of W'. But the trees W and W' have the same tree structure. This
'2W'.
substitution error is written as W
Tree W = b I d
a
0
a I \
Substitutenode b
c = 0.1 0.2 I 0.1.1
at position 0.1 by x
I
\
I d
For a given tree grammar Gt or tree automaton Mt, the minimum-distance SPECTA is formulated to accept the input tree and to generate a parse that consists of the minimum number of substitution error tree production rules or error replacement functions, while the tree structure is still preserved. Assume that W' is an input tree, the parsing of the input tree by minimum-distance SPECTA is a search for the minimum distance and is a backward procedure of constructing a tree-like transition table with all candidate states and their corresponding costs recorded from the leaves to the root of W'. For each tree node a (index position in tree), there is a corresponding transition box t, which consists of triplet items (X, # k , c) in the transition table, where X is the state, # k is the kth production rule or replacement function, c is the accumulated error cost. An example is shown as follows. Example 6.4. Parsing by minimum-distance SPECTA. Given tree production rules P: (1)s -+
(2)A
$
I \
A
B
-+
5
( 3 ) B -+ 7
$
I \
, and given an input tree,
5
7
we can generate the parsing of the input tree by minimum-distance SPECTA as below.
K.-Y. Huang
94
includes state A (nonterminal) in (A, 2, 0) and state B (nonterminal) in (B, 3, 1). Triplet item (A, 2, 0) represents that we can use A in production rule (2) A -+ 5 to derive terminal 5 with 0 substitution error because input terminal is 5, triplet item (B, 3, 1) represents that we can use state B in production rule (3) B + 7 to derive terminal 7 with 1 substitution error because input terminal is 5. t o . 2 has a similar explanation. t o has state S in triplet item (S, 1, 0) so that we can use S in
to.1
$
production rule (1)to derive
/ \ with 0 substitution error, 0 in (S, 1, 0) is the
A B sum of 0 from (A, 2, 0) and 0 from (B, 3, 0). Although there are other derivations from the combinations of the states (nonterminals) A and B of the triplets, i.e. $
$
$
$
/ \ , only S in production rule (1) can derive
/ \ , / \ , and
/ \
A
A B A B B A B with minimum error, others are counted as larger errors and neglected. If X is a candidate state of tree node at position index a , then each triplet item (X, #lc, c) is added to a box t,, #lc specifies the lcth production rule or bottom-up replacement function used, and c is the accumulated minimum number of substitution errors from the leaves to the node a in subtree of W' from node a when tree node a is represented by state X. The algorithm of minimum-distance SPECTA was in Fu [30]. For consideration of the weighted substitution error costs of pair terminals, Basu and Fu [6], presented the minimum weighted distance SPECTA. Here we have some modification. We can expand each production rule to cover the substitution error production rules and embed cost t o each production rule. So the expanded tree grammar with weighted error costs are generated. Production rule X 4z is expanded to substitution error production rule with error cost c, X 4 y, y # z; X 3 z, c = 0, and production rule C
x+ I
I
...
x,x*
\
X"
isexpandedto
/
XI
Y I ... \ x2
,y#x;c=Oify=x.
X"
Initially the expanded tree grammar with error costs must be generated. Then the algorithm of weighted minimum distance SPECTA is presented as follows.
Algorithm 6.4. Weighted minimum distance SPECTA. Input: An expanded tree grammar Gt (or a tree automaton Mt) with error costs, a tree W'. Output: Parsing transition table of W' and minimum distance. Met hod: (1) Replace each bottom leaf of the input tree W' to a state. If the bottom leaf (terminal) of the input tree is 2, i.e. r[W'(a)]= 0 (rank of bottom node a in tree W' is 0) and W ' ( a ) = z (terminal at bottom node a
1.3 Syntactic Pattern Recognition 95 in tree W’ is x), then for any terminal y and production rules ((#k)X % y ) , replace the leave (terminal) x of the input tree t o state X, and store (X, #k, c) to the box t a .
Do Steps (2) and (3) until there is no replacement to the root of the tree. Replace the subtree to a state using bottom-up replacement function. m
L .
If subtree is
/ 1 . .. \
x1 xz n) and W’(a)
, i.e. r[W’(a)] = n > 0 (rank of node a in tree W’ is
xn
(terminal at node a in tree W’ is x), then for any terminal y x% y X and production rules ( ( # k ) / I . . . \ ), replace subtree / I . . . \ =x
x1 xz
xn
x1
x2
+ + +
x,
to state X, and store (X, # k , c‘) to the table box t,, c’ = c1 c2 . . . c,, where ci is the cost in table box t,,j for state Xi, z = 1,.. . ,n. Whenever more than one item (X,#k,c) in t, has the same state X, keep the item with smaller error cost, and delete the items with larger error costs. If items (S, # k , c) are in t o , choose one item with minimum distance c, then the input tree is accepted with distance c. If no item is associated with the starting nonterminal S in t o of the form (S, # k , c), then the input tree is rejected.
Example 6.5. Parsing of error bright spot pattern by minimum distance SPECTA. (A) Bright spot pattern with primitive errors: $0
X
5
xxxxx xxxxx
7 xxx xxx xxxxxxxxxxx xxxxxxxxxxxxxxxx 0 xxx 5 xxx 0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx xx xxx 0 xxxxxxxxxx 0
xxx
5
xx
(B) Tree representation with primitive errors: Using eight directional F’reeman’s chain codes [27] and @, tree representation of bright spot with primitive errors is shown below. $
f 5
\
7 I \ 5 O @ 0 I \ I \ 5 0 0 0 I\
96
K.-Y. Huang
(C) Parsing by minimum distance SPECTA: Using the tree automaton inferred from bright spot pattern in Example 6.3, the transition table of parsing on tree of error bright spot pattern of Example 6.5 using minimum distance SPECTA can be generated as follows. Here the costs of terminal substitution errors are set and equal t o 1. The explanation of each box from the bottom-up replacement is the same as that of Example 6.4
6.7. Modified Maximum-Likelihood SPECTA When the probabilities of tree production rules and the substitution deformation probabilities on terminal pair symbols are available, maximum-likelihood SPECTA can be used for the recognition of patterns [30]. The stochastic expansive tree grammar G, = (V, T , P, S) has the form of production rules in P : (1) xo
3
or
z
/I
x1 x2
(2)
XO
3 z,
... \ &(z)
where p is the probability of production rule, z E VT and Xo,X1, Xz,. . . ,XT(z)E VN.
I . 3 Syntactic Pattern Recognition 97 The major steps of the maximum-likelihood SPECTA in Fu are described as follows [30]. Given stochastic expansive tree grammar G,, terminal substitution probabilities q(y/x) and input tree W’. (1) Replace each leaf of the input tree t o a state. If r[W‘(a)]= 0 (rank of node a in tree W’ is 0), W’(a) = y (symbol at node a in tree W’ is y), and X 3 x is the lcth production rule in P, then add (X, #lc,p’) to t, and p’ = p x q(y/x). (2) Replace the subtree to a state using bottom-up replacement function. If r[W’(a)]= n > 0 (rank of node a in tree W’ is n ) , W’(a) = y (symbol at node a in tree W’ is y), and
x 4
x
x1
x2
/
I . . .\
is the lcth production rule in P,
x n
then add (X, #lc,p’) t o table box t,, p’ = p i x p i x . . . x p; x p x q ( y / z ) , where p’, is the probability in table box t,.i for state Xi, i = 1,.. . , n. The calculation of probability p‘ was from the multiplication of the production rule probability p and the terminal substitution probability q(y/x). Instead of the calculation from the multiplication, p x q(y/x), here a modification is proposed. Similar to the previous expanded tree grammar with error costs, each production rule is expanded t o cover substitution error production rules with the probabilities,
x+
i.e. if
X S
x
/ 1 .. . \
x1 x2
is in P, then expand to
x n
X
/ I . ..\
x1 x2
for all
xn
terminals y, y = x or y # x. Summation of the probabilities from expansion of one tree production rule is 1. There are some possibilities that terminals may be recognized as the neighboring terminals in the process of primitive recognition. The value of probability of substitution pair terminals is inversely proportional to the angle of pair terminals. Each tree production rule of tree grammar Gt is expanded to cover the substitution error production rules with probabilities. Based on the expanded grammar, the maximum-likelihood SPECTA is modified here for the recognition of seismic patterns. The algorithm is presented as follows.
Algorithm 6.5: Modified maximum-likelihood SPECTA. Input: (1) Expanded tree grammar Gt (or tree automaton M,) with probability on each production rule, (2) input tree W’. Output: Parsing transition table of W’ and maximum probability. Method: (1) Replace each bottom leave of the input tree W’ t o a state. If r[W’(a)]= 0 (rank of node a in tree W’ is 0), W’(a) = x, and X lcth production rule in P, then add (X, #lc,p’) t o t, and p’ = p .
5 x is the
Do Steps (2) and (3) until there is no replacement at the root of the tree.
98
K.-Y. Huang
(2) Replace the subtree to a state using bottom-up replacement function. If r[W’(a)] = n > 0 (rank of node a in tree W’ is n),W’(a) = x, (terminal at X2; 5 node a in tree W’ is x), and / I . . . \ is the kth production rule
x1
x2
x,
in P, then add (X, # k , p ’ ) to table box t,, p’ = p i x p i x . . . x p k x p , where pi is the probability in table box t,.i for state Xi, i = 1,.. . ,n. ( 3 ) Whenever more than one item (X, # k , p ’ ) in t , has the same state X, keep the item with larger probability and delete the item with smaller probability. (4) If items (S, # k , p ’ ) are in t o , choose one item with maximum probability p’, then the input tree W’ is accepted with probability p’. If no item in t o is associated with the starting nonterminal S, then the input tree W‘ is rejected. 6.8. Minimum Distance GECTA
Due to noise, distortion, and interference of the wavelets, the tree has an error structure. The error may cause the tree to become the preserved or non-preserved tree structure. If the tree structure is preserved, then weighted minimum distance SPECTA and modified maximum-likelihood SPECTA can be applied in the recognition of patterns. If the tree structure is not preserved, then minimum distance GECTA [30] can be applied in the recognition of patterns. The syntax errors between two trees may include substitution, deletion, and insertion errors. The insertion error includes three types of errors: stretch, branch, and split errors. Totally there are five types of syntax errors on the tree. The distance between two trees is defined to be the least cost sequence of error transformations needed t o transform one t o the other [30,61,80].Because there are five possible error transformations to transform one tree into the other tree, each production rule of the tree grammar must be expanded to cover all five syntax errors on trees. Then the expanded grammar can generate a recognizer, i.e. the minimum distance generalized error-correcting tree automaton (GECTA). Similar to the weighted minimum distance SPECTA, the parsing of an input tree W’ using minimum distance GECTA is also to construct a tree-like transition table with all candidate states and their corresponding costs recorded. The procedure is a backward from the leaves to the root of W’ for the least cost solution [30].
7. Conclusions and Discussions Theoretical studies in syntactic pattern recognition were effective in handling abstract and artificial patterns [30,89]. We need simultaneous progress in both theoretical studies and real-data applications in the future. Combining the syntactic and the semantic approaches can expand the power of syntactic pattern recognition. Semantic information often provides spatial information, relation, and reasoning between primitives, subpatterns, and patterns, and can be expressed syntactically, for example, by attributed strings and attributed graphs. The attributed 2-D and
I . 3 Syntactic Pattern Recognition 99 3-D pattern grammars such as attributed tree, graph, and shape grammars, may be the subject of future study [9,31,76,86-881. The distance computation between two attributed patterns (attributed strings, attributed trees, attributed graphs,. . . , etc.) may also be studied in the future [44]. The error-correcting finite-state parsing, Earley’s parsing, tree automaton,. . . , etc. may also be expanded for attributed strings, trees,. . . , etc. [44,45,59]. The distance can be computed between input pattern y and language L(G), or between input pattern and training pattern. Using a distance or similarity measure, clustering methods, such as minimum-distance classification rule, nearest neighbor classification rule, K-nearest neighbor classification rule and method of hierarchical clustering, can be easily applied to syntactic patterns [30,34,61,62,89]. If the pattern has inherent structural property, globally we can use the syntactic approach t o recognize the pattern. Locally we can use neural network techniques in the segmentation and the recognition of primitives, such that the syntactic pattern recognition will improve and become more robust against the noise and distortions. In the study of certainty effect, besides the probability approach, fuzzy logic may be considered in grammar and automaton, for example, fuzzy tree automaton [53]. Parallel parsing algorithms can speed up the parsing time [12,18,20]. For example, the tree automaton can be parsed from the bottom leaves to the top root of the tree in parallel. Further, a syntactic approach to the time-varying pattern recognition may also become one of the research topics in the future [24].
References [l] A. V. Aho and T. G. Peterson, A minimum distance error-correcting parser for context-free languages, SIAM J. Comput. 1 (1972) 305-312. [a] A. V. Aho and J. D. Ullman, The Theory of Parsing, Translation, and Compiling, Vol. 1: Parsing (Prentice-Hall, Englewood Cliffs, NJ, 1972). [3] F. Ali and T. Pavlidis, Syntactic recognition of handwritten numerals, IEEE Trans. Syst. Man Cybern. 7 (1977) 537-541. [4] K. R. Anderson, Syntactic analysis of seismic waveforms using augmented transition network grammars, Geoexploration 20 (1982) 161-182. [51 A. Barrero, Inference of tree grammars using negative samples, Pattern Recogn. 24, (1991) 1-8. [6] S. Basu and K. S. Fu, Image segmentation by syntactic method, Pattern Recogn. 20, (1987) 33-44. [7] J. M. Brayer and K. S. Fu,A note on the K-tail method of tree grammar inference, IEEE Trans. Syst. Man Cybern. 7 (1977) 293-299. [8] I. Bruha and G. P. Madhavan, Use of attributed grammars for pattern recognition of evoked potentials, IEEE Trans. Syst. Man Cybern. 18 (1988) 1046-1089. [9] H. Bunke, Attributed programmed graph grammars and their application to schematic diagram interpretation, IEEE Trans. Pattern Anal. Mach. Intell. 4 (1982) 574-582. [lo] H. Bunke and A. Sanfeliu (eds.), Special Issue: Advances in Syntactic Pattern Recognition, Pattern Recogn. 19, 4 (1986). [ll] H. Bunke and A. Sanfeliu (eds.), Syntactic and Structural Pattern Recognition Theory and Applications (World Scientific, 1990).
100
K.-Y. Huang
[12] N. S. Chang and K. S. Fu, Parallel parsing of tree languages for syntactic pattern recognition, Pattern Recogn. 11 (1979) 213-222. (131 C. H. Chen (ed.), Special Issue: Seismic Signal Analysis and Discrimination, Geoexploration 20,1/2 (1982). [14] C. H. Chen (ed.), Special Issue: Seismic Signal Analysis and Discrimination 111, Geoexploration 23, 1 (1984). [15] C. H. Chen, L. F. Pau and P. S. Wang (eds.), Handbook of Pattern Recognition and Computer Vision (World Scientific, 1993). [16] C. H. Chen (ed.), Special Issue: Artificial Intelligence and Signal Processing in Underwater Acoustic and Geophysics Problems, Pattern Recogn. 18,6 (1985). [17] J. C. Cheng and H. S. Don, A graph matching approach to 3-D point correspondences, Znt. J. Pattern Recogn. A d i f . Intell. 5 (1991) 399-412. [18] Y . C. Cheng and S. Y. Lu, Waveform correlation by tree matching, IEEE Trans. Pattern Anal. Mach. Intell. 7 (1985) 299-305. [19] Y . T. Chiang and K. S. Fu, Parallel parsing algorithm and VLSI implementations for syntactic pattern recognition, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 302-314. [20] M. B. Dobrin and C. H. Savit, Introduction to Geophysical Prospecting, 4th edn. (McGraw-Hill, New York, 1988). (211 H. S. Don and K. S. Fu,A parallel algorithm for stochastic image segmentation, I E E E Trans. Pattern Anal. Mach. Intell. 8 (1986) 594-603. 1221 D. Dori, A syntactic/geometric approach to recognition of dimensions in engineering machine drawings, Comput. Vision Graph. Image Process. 47 (1989) 271-291. [23] J . Earley, An efficient context-free parsing algorithm, Commun. of A C M 13 (1970) 94-102. [24] M.A. Eshera and K. S. Fu, A graph distance measure for image analysis, IEEE Trans. Syst. Man Cybern. 14 (1984) 398-408. [25] T. I. Fan and K. S. Fu, A syntactic approach t o time-varying image analysis, Comput. Graph. Image Process. 11 (1979) 138-149. [26] T. Feder, Plex languages, I f . Sci. 3 (1971) 225-241. [27] G. Ferrate, T. Pavlidis, A. Sanfeliu and H. Bunke (eds.) Syntactic and Structural Pattern Recognitions (Springer-Verlag, 1988). [28] H. Freeman, On the encoding of arbitrary geometric configurations, I E E E Electron. Comput. 10 (1961) 260-268. [29] K. S . Fu, Syntactic Methods in Pattern Recognition (Academic Press, New York, 1974). [30] K. S. Fu, Syntactic image modeling using stochastic tree grammars, Comput. Graph. Image Process. 12 (1980) 136-152. [31] K. S . Fu, Syntactic Pattern Recognition and Applications (Prentice-Hall, Englewood Cliffs, NJ, 1982). [32] K. S. Fu, A step towards unification of syntactic and statistical pattern recognition, I E E E Trans. Pattern Anal. Mach. Intell. 5 (1983) 200-205. [33] K. S. Fu and B. K. Bhargava, Tree systems for syntactic pattern recognition, I E E E Tkans. Comput. 22 (1973) 1087-1099. [34] K. S . Fu and T. Huang, Stochastic grammars and languages, Int. J. Comput. In$ Sci. 1 (1972) 135-170. [35] K. S . Fu and S. Y . Lu, A clustering procedure for syntactic patterns, I E E E Trans. Syst. Man Cybern. 7 (1977) 734-742. [36] J. E. Gaby and K. R. Anderson, Hierarchical segmentation of seismic waveforms using affinity, Pattern Recogn. 23 (1984) 1-16.
1.3 Syntactic Pattern Recognition 101 [37] P. Garcia, E. Segarra, E. Vidal and I. Galiano, On the use of the morphic generator grammatical inference (MGG) methodology in automatic speech recognition, Int. J. Pattern Recogn. Artif. Intell. 4 (1990) 667-685. [38] R. C. Gonzalez and M. G. Thomason, Syntactic Pattern Recognition (Addison Wesley, Reading, MA, 1978). [39] K.-Y. Huang, Branch and bound search for automatic linking process of seismic horizons, Pattern Recogn. 23 (1990) 657-667. [40] K.-Y. Huang, Pattern recognition to seismic exploration, in Automated Pattern Analysis in Petroleum Exploration, eds. I. Palaz and S. K. Sengupta (Springer-Verlag, New York, 1992) 121-154. [41] K. Y. Huang, W . Bau and S. Y. Lin, Picture description language for recognition of seismic patterns, SOC. Exploration Geophysicists Int. 1987 Mtg., New Orleans, 326-330. [42] K. Y. Huang and K. S. Fu, Syntactic pattern recognition for the classification of Ricker wavelets, Geophysics 50 (1985) 1548-1555. [43] K. Y. Huang and K. S. Fu, Syntactic pattern recognition for the recognition of bright spots, Pattern Recogn. 18 (1985) 421-428. [44] K. Y. Huang, K. S. Fu,S. W. Cheng and Z. S. Lin, Syntactic pattern recognition and Hough transformation for reconstruction of seismic patterns, Geophysics 52 (1987) 1612-1620. [45] K. Y. Huang and D. R. Leu, Modified Earley parsing and MPM method for attributed grammar and seismic pattern recognition, J . Inf. Sci. and Eng. 8 (1992) 541-565. [46] K. Y. Huang and D. R. Leu, Recognition of Ricker wavelets by syntactic analysis, Geophysics 60 (1995) 1541-1549. [47] K. Y. Huang and T. H. Sheen, A tree automaton system of syntactic pattern recognition for the recognition of seismic patterns, 56th Annu. Int. Mtg., SOC.Expl. Geophys. (1986) 183-187. [48] K. Y. Huang, T. H. Sheen, S. W. Cheng, Z. S. Lin and K. S. Fu, Seismic image processing: (I) Hough transformation, (11) Thinning processing, (111) Linking processing, Handbook of Geophysical Exploration: Section I. Seismic Exploration, 20, Pattern Recognition €9 Image Processing, (ed.) F. Aminzadeh (1987) 79-109. [49] K. Y. Huang, J. J. Wang and V. M. Kouramajian, Matrix grammars for syntactic pattern recognition, 1990 Telecomm. Symp. Taiwan, 576-581. [50] J. W. Hunt and T. G. Szymansky, A fast algorithm for computing longest common subsequences, Commun. A C M 20 (1977) 350-353. [51] S. Kiram and C. Pandu, A linear space algorithm for the LCS problem, Acta Informatica 24 (1987) 353-362. [52] A. Koski, M. Juhola and M. Meriste, Syntactic recognition of ECG signals by attributed finite automata, Pattern Recogn. 28 (1995). [53] L. H. T. Le and E. Nyland, An application of syntactic pattern recognition to seismic interpretation, in Computer Vision and Shape Recognition, A. Krzyzak, T. Kasvand and C. Y. Suen (eds.) (World Scientific, 1988) 396-415. [54] E. T. Lee, Fuzzy tree automata and syntactic pattern recognition, I E E E Trans. Pattern Anal. Mach. Intell. 4 (1982) 445-449. [55] V. I. Levenshtein, Binary codes capable of correcting deletions, insertions and reversals, Sou. Phys. Dokl. 10 (1966) 707-710. [56] B. Levine, Derivatives of tree sets with applications to grammatical inference, IEEE Trans. Pattern Anal. Mach. Intell. 3 (1981) 285-293. [57] B. Levine, The use of tree derivatives and a sample support parameter for inferring tree systems, IEEE Trans. Pattern Anal. Mach. Intell. 4 (1982) 25-34.
102
K.-Y. Huang
[58] R. Y. Li and K. S. Fu, Tree system approach for LANDSAT data interpretation, Symp. Mach. Process. Remotely Sensed Data, West Lafayette, Ind., June 29-July 1, 1976. [59] W. C. Lin and K. S. Fu, A syntactic approach to 3-D object representation, I E E E Trans. Pattern Anal. Mach. Intell. 6 (1984) 351-364. [60] H. H. Liu and K. S. Fu,A syntactic approach to seismic discrimination, Geoexploration 20 (1982) 183-196. [61] S. W. Lu, Y . Reng and C. Y . Suen, Hierarchical attributed graph representation and recognition of handwritten Chinese characters, Pattern Recogn. 24 (1991) 617-632. [62] S. Y. Lu, A tree-to-tree distance and its application to cluster analysis, I E E E Trans. Pattern Anal. Mach. Intell. 1 (1979) 219-224. [63] S . Y. Lu and K. S. Fu, A sentence-to-sentence clustering procedure for pattern analysis, IEEE Trans. Syst. Man Cybern. 8 (1978) 381-389. [64] S. Y. Lu and K. S. Fu, Error-correcting tree automata for syntactic pattern recognition, I E E E Trans. Comput. 27 (1978) 1040-1053. [65] S. Y. Lu and K. S. Fu, A syntactic approach to texture analysis, Comput. Graph. Image Process. 7 (1978) 303-330. [66] S . Y . Lu and K. S. Fu,Stochastic tree grammar inference for texture synthesis and discrimination, Comput. Graph. Image Process. 9 (1979) 234-245. [67] W. Min, Z. Tang and L. Tang, Using web grammar to recognize dimensions in engineering drawings, Pattern Recogn. 26 (1993) 1407-1416. [68] B. Moayer and K. S. Fu, A tree system approach for fingerprint pattern recognition, I E E E Trans. Comput. 25 (1976) 262-274. [69] R. Mohr, T. Pavlidis and A. Sanfeliu (eds.) Structural Pattern Recognitions (World Scientific, 1990). [70] T. Pavlidis, Linear and context-free graph grammars, J. ACM 19 (1972) 11-12. [71] T. Pavlidis, Structural Pattern Recognition (Springer-Verlag, New York, 1977). [72] C. E. Payton (ed.), Seismic Stratigraphy - Applications to Hydrocarbon Exploration (AAPG Memoir 26, Tulsa, OK, Amer. Assn. Petroleum Geologists, 1977). [73] J. L. Pfaltz and A. Rosenfeld, Web grammars, Proc. 1st Int. Joint Conf. Artif. Intell., Washington, D.C., (1969) 609-619. [74] A. Rosenfeld, Picture Languages (Academic Press, New York, 1979). [75] A. Sanfeliu, K. S. Fu and J . Prewitt, An application of a graph distance measure to the classification of muscle tissue patterns, Int. J. Pattern Recogn. Artif. Intell. 1 (1987) 17-42. (761 A. C.Shaw, The formal picture description scheme as a basis for picture processing system, Znf. Control 14 (1969) 9-52. [77] Q. Y. Shi and K. S. Fu, Parsing and translation of (attributed) expansive graph languages for scene analysis, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 472485. [78] L. Stringa, A new set of constraint-free character recognition grammars, ZEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 1210-1217. (791 P. H. Swain and K. S. Fu, Stochastic programmed grammars for syntactic pattern recognition, Pattern Recogn. 4 (1972). [80] E. Tanaka and K. S. Fu,Error-correcting parsers for formal languages, I E E E Trans. Comput. 27 (1978) 605-616. [81] E. Tanaka and K. Tanaka, The tree-to-tree editing problem, Int. J. Pattern Recogn. Artif. Intell. 2 (1988) 221-240. [82] M. G. Thomason, Generating functions for stochastic context-free grammars, Int. J. Pattern Recogn. Artif. Intell. 4 (1990) 553-572.
1.3 Syntactic Pattern Recognition 103 [83] M. G. Thomason and R. C. Gonzalez, Error detection and classification in syntactic pattern structures, ZEEE Trans. Comput. 24 (1975) 93-95. [84] R. A. Wagner and M. 3. Fischer, The string to string correction problem, J. ACM 21 (1974) 168-173. 1851 P.S. P. Wang (ed.), Special issue on array grammars, patterns and recognizers, Znt. J. Pattern Recogn. Artif. Intell. 3,3&4 (1989). [86] G. Wolberg, A syntactic omni-font character recognition system, Znt. J. Pattern Recogn. Artif. Intell. 1 (1987) 303-322. [87] A. K. C. Wong, S. W. Lu and M. Rioux, Recognition and shape synthesis of 3-D objects based on attributed hypergraph, I E E E Trans. Pattern Anal. Mach. Intell. 11 (1989) 279-290. [88] K.C.You and K. S. Fu, A syntactic approach to shape recognition using attributed grammars, I E E E Trans. Syst. M a n Cybern. 9 (1979) 334-345. [89] K. C. You and K. S. Fu, Distorted shape recognition using attributed grammars and error-correcting techniques, Comput. Graph. Image Process. 13 (1980) 1-16. [go] T. Y . Young and K. S. Fu (eds.), Handbook of Pattern Recognition and Image Processing (Academic, New York, 1986).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 105-142 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 1.4 I NEURAL NET COMPUTING FOR PATTERN RECOGNITION
YOH-HAN P A 0 Electrical Engineering and Computer Science, Case Western Reserve University 10900 Euclid, Cleveland, Ohio 44106-7221, USA In this chapter we discuss Artificial Neural Net computing from the viewpoint of its being an enabling methodology for pattern recognition research and practice. The four functionalities of clustering, learning functional mappings, classification through associative recall, and optimization are discussed in a comparative manner relative to other practices in pattern recognition and also relative to each other. In addition to references, two bibliographies, one for books and the other for journals, are provided as guides for further reading.
Keywords: Neural net computing, ART, Hopfield net, optimization, functional m a p ping, supervised learning, Boltzmann machine, simulated annealing, functional-link net, associative memory.
1. Introduction In this chapter, we address Artificial Neural Net (ANN) computing from the perspective of its being a tool for implementing pattern recognition algorithmic practices. The primary context of our discussion is that of pattern recognition, but the topic of specific interest is how neural net computing can be used for attaining pattern-based information processing objectives, especially those which have been established over the years t o be of central interest and importance to the pattern recognition research and practitioner communities. Researchers in information processing have long recognized the strikingly different information processing propensities of serial digital computers and of biological systems. The former rely on speed and accuracy and on ability to execute vast amounts of detailed programmed instructions precisely. But they are, nevertheless, easily overwhelmed by algorithmic tasks, which are of exponential or greater complexity. Unfortunately most real-world perception/cognition tasks, if approached in a direct manner, are of such a nature. In contrast, the nature of biological systems is that of distributed parallel processing systems, made up of large numbers of interconnected elemental processors of rather slow processing speed. In addition, information processing seems to depend on the ability to discern what is cogent and relevant, and to focus on that while
106
Y.-H. Pao
sustaining a minimal degree of maintenance on other matters. Situations, circumstances, and events seem to be evaluated on the basis of the “pattern-ness” of things and on similarities between patterns, and on associations between patterns. This is in marked contrast to the operational strategies of the high-speed, general-purpose, serial-digit a1 computers. At the risk of overstating the case, it almost seems that in approaching the performance of a task, serial-digital computer algorithms tend t o search all of the system space to find a reasonably good path from start state to goal state. We know such approaches are doomed to failure because of the combinatorial explosion in the number of paths to be tried. In contrast to the systematic, frontal-attack approach, biological systems seem to rely more on experience and education so that any good path or even a segment of a good path is remembered, and that knowledge is transmitted through generations, either genetically or through education. In this latter mode of information processing, individual operations are of limited significance but patterns, both spatial and temporal, are of central importance. The significance of patterns is established by associations between a pattern (or a set of patterns) and other patterns (or sets of patterns). Accordingly, the formation of such associations and the activation of such linkages are matters of critical importance. One of the practical objectives of pattern recognition researchers has always been the ability to design and implement machine systems, which are able to perform perception tasks competently t o degrees of proficiency comparable t o that of biological systems. To date it cannot be said that progress in that respect has been as substantial as desired or as expected. If we try to identify reasons for this relative lack of success, we might include the following. It would seem that detailed studies of information processing architectures and procedures in actual biological neuronal studies are so difficult that progress comes a t a very slow pace, indeed. Therefore guidance from that source, though much valued, is limited. In addition, tragically, one of the few initial attempts at artificial neural net computing was so thoroughly discredited at its onset that no academic research in that topic could be sustained for the past decades, until recently. For example, pattern recognition texts have always taught the WidrowHoff algorithm [l]as a procedure for learning a linear discriminant but never with any suggestion that it might also be considered to be a representation of a net capable of learning functional mappings. These matters and others contributed to the absence of a coherent body of commonly-shared knowledge of adaptive and associative pattern-based, information processing practice, even when it was clear that such knowledge and activity were critical to further progress in pattern recognition research. The most recent resurgence in artificial neural net computing is due to initiatives from the cognitive psychology sciences and from researchers interested in biological information processing matters.
1.4 Neural Net Computing for Pattern Recognition 107 It is a huge and high risk jump t o go from well-accepted, highly professional psychological or biological studies to the dubious practice of postulating some drastically simplified “neuronal” computational models and to try to establish some relevancy between the two types of endeavors. However, at any rate, as is well known, such initiatives were carried out over the past decade and have stimulated a powerful resurgence of interest and activity in artificial net computing [2]. Of primary significance to us, is the outcome that regardless of whether the artificial neural net computing paradigm models biology or not, it is of intrinsic value to information processing researchers especially pattern recognition researchers who are interested in the “pattern-ness” of matters and in the rapid distributed parallel processing of associated nets of such patterns [2]. Currently there is not only interest in basic matters in artificial neural net computing, but also extensive activity in the application of this technology to practical tasks, with reports of considerable success. This chapter is primarily in the nature of a n annotated guide to the knowledge which comprises the core of the state-of-the-art in this field at this time. The guide is, therefore, selective rather than comprehensive, and the notation reflects our personal biases and viewpoints, as indeed must be the case for the annotation to be meaningful. The organization of our presentation of materials is described in Section 2. The topical matters themselves are discussed in subsequent sections. These rather sparse schematic discussions are stressed in a section on comments and bibliographic remarks and by a list of titles for further reading. 2. Organization of Chapter In a manner consonant with accepted practice, we divide the architectures and algorithms of artificial neural net computing into four parts characterized by the headings of unsupervised learning, supervised learning, associative memory, and optimization. In addition we list a fifth area, which addresses systems level issues. In Table 1 we list for each such topical area some typical architectures, algorithms, and functionalities supported by the algorithms and corresponding activities and results in traditional pattern recognition research. We believe that Table 1 indicates that neural net computing does indeed address issues of interest to pattern recognition and might indeed provide effective means for realizing the computational objectives of pattern recognition. There are aspects of artificial neural net computing which have been well discussed in literature and even in books. There is no need for us to repeat such discussions in this brief chapter. For accepted background material, we refer the reader to the referenced works and also to the additional bibliographies. In the following sections, we discuss each of these areas.
Pao and Hafez algorithm for concept formation
0
System Level Issues
ART 1, 2 or 3
0
Hopfield and Tank approach
Hopfield net
Generalized delta rule/back propagation-of-error Functional-link net
0
0
Topologically correct mapping
LVQ
0
0
ART 1 & 2
0
Representation Algorithms
Optimization
Associative Memory
Supervised Learning
Unsupervised Learning
ANN Computing Area
Classification
Disconcerning regularities in data
Classification
0
Max nets
Forming clusters
Topologically correct
Classification
0
0 0 0
Feature extraction Concept formation
D D
Associative memory
Learning discriminant
Feature extraction
Underdeveloped in pattern recognition
Undeveloped in pattern recognition
Distributed matrix associative memories
No direct correspondence
This area is of great importance but underdeveloped in traditional pattern recognition
Non-parametric estimation (usually limited to estimation of density distribution functions)
Neural net computing does not deal with feature extraction
Modifying clusters
Vigilance factor
K-means and ISODATA
0
New concepts:
Comments
Data reduction
Traditional Pattern Recognition Issues
0
0
0
Inductive learning
Optimal activation to :omplex problems :radient search
Associative recall
0
Restoration of corrupted patterns
Learning a functional mapping from a set of examples
0
0
Functionality
Table 1. Neural net computing and pattern recognition.
cc
0
F
1.4
Neural Net Computing f o r Pattern Recognition
109
3. Unsupervised Learning We can distinguish between three types of unsupervised learning represented by the algorithms of the types of ART [3-51, LVQ [6], and topologically correct mapping [7,8]. To some this area of neural net computing contributes the least to pattern recognition, because in a sense nothing significantly new is added to the principal functionality of cluster formation. Indeed, it might be argued that existing methods, such as the K-means [9] algorithm or the ISODATA [lo] algorithm, can do just as well if not better than the corresponding neural net algorithms. However to others it is exactly this close correspondence which is satisfying and stimulating. Currently in neural computing, clustering is established on the basis of some metric defined in the actual pattern space in question. This means that we establish a rule for calculating the “distance” between two patterns, and decide whether they should be considered sufficiently similar t o be grouped within one and the same cluster or whether they should be in different clusters. This is illustrated in Fig. l ( a ) for some geometric but not necessarily isotropic metric. If the metric is isotropic, meaning that the rule for calculating distances is the same regardless of the direction in which we look from any one pattern, then the result is a partitioning of pattern space into distinctive nonoverlapping hyperspherical regions or clusters, as shown in Fig. l ( b ) , for a Euclidean distance metric, in two dimensions. Even in such a straightforward simple procedure, we can introduce variety by specifying different cluster radius thresholds for different regions of the pattern space. That can and in general does result in the need for special procedures for resolving conflict and for ensuring convergence. To date no neural net algorithm provides the capability of shaping clusters of the form shown in Fig. l(c) in a meaningful and adaptive manner. 3.1. ART
The well-accepted ART algorithms might seem to differ from the above Euclidean distance approach but actually deviate from it only slightly, being exactly the Euclidean distance approach if all of the vectors are of the same length. As shown in Fig. 2, in the ART algorithm each input pattern vector x is projected on each and all of the prototype vectors bj, and the cluster (prototype) node with the largest projection sum y j = C bjizi is identified with the use of the MAXNET. The proposition that the input vector x belongs to cluster j is then checked by forming the vigilance factor t j z z i . If that exceeds a threshold value (say) p, then the vector x is accepted as an additional new member of that j t h cluster, and the values bj are updated. The top down vigilance factor components are updated also. The ART algorithms are well explained in the literature [3-51, but we advocate and practice a slightly modified version of these especially in so far as updating is concerned [ 111.
Xi
110
Y.-H. Pa0
limitof duster exteni
0 ..... .
..
(c)
Fig. 1. Formation of clusters in unsupervised learning: (a) essentials of a cluster, (b) formation of distinctive non-overlapping clusters, and ( c ) more general cluster formation.
The projection procedure is adequate as long as both the b vectors and the input vectors are all of the same length. Under such circumstances, the scalar product of the two vectors b and x do, indeed, provide a measure of t-he similarity. Also we note that the square of the Euclidean distance between the bj vector and x vector is
= bgbj - 2bSx
+X ~ X .
(3.1)
Clearly the larger the value of (bjx),the smaller the Euclidean distance between bj and x or, in other words, the more similar they are. Also clearly all the previous remarks are also vaIid for the case of binary valued features as in ART 1.
1.4 Neural Net Computing for Pattern Recognition
111
In general, however, we advocate the practice described in Box 1, which is compatible with standard pattern recognition practice and with ART 2.
Box 1 Activate all output nodes j , j = 1, 2, . . . , J . Initialize weights b,, = E,, where E,, are random numbers (-1 < E,, < 1). Input pattern {z,} i = 1, . . . , N . Calculate the square of the Euclidean distance ED;, = 1,(b,, - y,)’. Determine that j for which ED:, < ED;, for all k = 1, 2, . . . , J ; K # j. Assign pattern {z,} as belonging to node j if ED:, also is equal to or less than ED;, (limit), where ED:, (limit) is a more or less arbitrary chosen limiting radius beyond which patterns are not considered to be of that cluster. n 1 z, (n = 0 at initialization). 7. Update b,,(n 1) = - b,z(n) + n+l Therefore, after the input of the first pattern b,,(1) = z,. 8. Input the next pattern, determine t o which unsupervised learning node it belongs, and update corresponding { b , , } .
1. 2. 3. 4. 5. 6.
+
We note that the cluster centers, the bj vectors, are only slightly perturbed by the inclusion of a new member, especially if the cluster already contains a number of members. The updating of bj is weighted so that
n 1 bj(n + 1) = -bj(n)+ n+l n+lX when the j t h cluster with n members in the cluster adds an additional member x to the cluster. The top-down vigilance vector tj is then taken to be equal to b, and is updated in the same manner. There are different ways of exercising this algorithm depending on whether one should activate all cluster prototype nodes initially or activate additional ones only as needed. In contrast to concerns which might dominate if we were endeavoring to build models of the brain, in artificial neural net computing it would seem that the latter practice, that of activating each additional new prototype cluster as needed, is more reasonable and usually convergence to stable cluster centers occurs in a straightforward manner. Also in the case of artificial neural net computing, there may be circumstances where determination of maximum similarly might be carried out more simply than with use of the MAXNET 1121. This type of algorithm corresponds closely to the K-means and ISODATA algorithms, and more to the former than to the latter. Our interest in the ART type of algorithm lies in the suggested net architecture and in the fact that the procedure corresponds to that of the K-means algorithm.
112
Y.-H. Pao y2
Y1
.................
YM
MAX-NET (0 detmine dustermembership. Not allcavrecbnsdwmn.
7oPdoW'
tF link.not shorm. used in verification.
f
f ...........
f ...............
XI
x2
xi
t
XN
Fig. 2. Some aspects of the ART algorithm. (a) Schematic illustration of the ART net, (b) two-dimensional illustration of the equivalence of the projection and distance measures when all pattern vectors are of the same length.
The ART algorithm has been extended to hierarchical ART structures [5] in work which addresses systems issues in the use of such algorithms. What is a little disappointing is the lack of opportunity to shape the clusters and to merge or split them as in the case of ISODATA, as illustrated in Fig. l(c). 3.2. Learning Vector Quantization
The Learning Vector Quantization (LVQ) algorithm builds [6] on the ART type of algorithm and mixes supervised learning with cluster formation. In a manner similar to that of the Widrow-Hoff [l]algorithm of pattern recognition, it refines the structure of a cluster by examining the class membership of each of the members in turn. The assumption is that nearly all of the cluster members belong t o one and the same class. Now as each member is examined in turn, the cluster prototype is modified to move closer to the current member under consideration or away from it depending on whether that member is or is not a member of the majority class.
1.4
Neural Net Computing for Pattern Recognition
113
That is m(n
+ 1) = m(n) + a(x
-
m(n)) if x is of the class of the cluster
(3.3)
+ 1) = m(n)
-
a(x - m(n)) if x is not of that class
(3.4)
or
m(n
where n is the number of cluster members already checked, and m is the vector denoting the cluster center. The parameter a is a fractional quantity which decreases with n so that there is convergence. This situation is depicted schematically in Fig. 3. X X
X
X
X
0 X
/Ax
X
x o
O X
x 0
cluster center adapts by moving A x
- patterns of class c (the majority)
o - patterns aof class C
A
- cluster "center of cluster of class c patterns (biased by presence of c patterns)
Fig. 3. The Learning Vector Quantization (LVQ) algorithm incorporating supervised classification into the cluster procedures.
3.3. Topologically- Correct Mapping The topologically correct mapping approach to clustering allows us t o investigate relationships between the following matters: 0 0 0
0
patterns defined in an N-dimensional positional space X , a process (or metric) defined on X and on the patterns described in X , nodes (or neurons) located spatially in an M-dimensional positional space Y , and an interactional process (or metric) defined for the neurons described in Y .
These entities are described in Fig. 4 where we show pattern vectors {x} defined in pattern space X . The ordering process we impose on top of the patterns in that space is not limited to a determination of the inter-pattern Euclidean distance, but can be quite general indeed. Given the patterns in X and the ordering process, we ask how the consequences of that ordering might be reflected in another space. In particular, for instance, for an array of fixed position neurons in space Y with
114
Y.-H. Pao
. .:. . .
. ..: . .
f
.
..