Advances in Pattern Recognition
Advances in Pattern Recognition is a series of books which brings together current developments in all areas of this multi-disciplinary topic. It covers both theoretical and applied aspects of pattern recognition, and provides texts for students and senior researchers. Springer also publishes a related journal, Pattern Analysis and Applications. For more details see: http://link.springer.de The book series and journal are both edited by Professor Sameer Singh of Exeter University, UK. Also in this series: Principles of Visual Information Retrieval Michael S. Lew (Ed.) 1-85233-381-2 Statistical and Neural Classifiers: An Integrated Approach to Design Sˇaru¯nas Raudys 1-85233-297-2 Advanced Algorithmic Approaches to Medical Image Segmentation Jasjit Suri, Kamaledin Setarehdan and Sameer Singh (Eds) 1-85233-389-8 NETLAB: Algorithms for Pattern Recognition Ian T. Nabney 1-85233-440-1 Object Recognition: Fundamentals and Case Studies M. Bennamoun and G.J. Mamic 1-85233-398-7
Bir Bhanu and Ioannis Pavlidis (Eds)
Computer Vision Beyond the Visible Spectrum With 156 Figures
Bir Bhanu, PhD, Fellow IEEE, AAAS, IAPR, Senior Honeywell Fellow (Ex.) Center for Research in Intelligent Systems, University of California at Riverside, CA, USA Ioannis Pavlidis, PhD Department of Computer Science, University of Houston, TX, USA
Series editor Professor Sameer Singh, PhD Department of Computer Science, University of Exeter, Exeter, EX4 4PT, UK
British Library Cataloguing in Publication Data Computer vision beyond the visible spectrum. — (Advances in pattern recognition) 1. Computer vision 2. Pattern recognition systems I. Bhanu, Bir II. Pavlidis, Ioannis 006.3′7 ISBN 1852336048 Library of Congress Cataloging-in-Publication Data CIP data available. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. Advances in Pattern Recognition ISSN 1617-7916 ISBN 1-85233-604-8 Springer London Berlin Heidelberg Springer is a part of Springer Science+Business Media springeronline.com © Springer-Verlag London Limited 2005 The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Printed and bound in the United States of America 34/3830-543210 Printed on acid-free paper SPIN 10868565
Preface
Traditionally, computer vision has focused on the visible band for a variety of reasons. The visible band sensors are cheap and easily available. They are also sensitive in the same electromagnetic band as the human eye, which makes the produced data more interesting from the psychophysiology point of view. In fact, computer vision was pre-occupied for a long time with the problem of understanding and imitating the human visual system. Recently, this obsession subsided and computer vision research focused more on solving particular application problems with or without the help of the human visual paradigm. A case in point is the significant progress achieved in object tracking. It so happens that many imaging applications cannot be addressed in the visible band. For example, visible sensors cannot see in the dark; thus, they are not very useful in military applications. Visible radiation cannot penetrate the human body and, therefore, cannot be a viable medical imaging modality. Other electromagnetic bands and sensor modalities have been identified and developed over the years that can solve all these problems, which are beyond the reach of the visible spectrum. Initially, it was primarily phenomenological and sensory work that was taking place. Later came algorithmic work, and with that computer vision beyond the visible spectrum was born. In this book, we explore the state-of-the-art in Computer Vision Beyond the Visible Spectrum (CVBVS) research. The book is composed of nine chapters which are organized around three application axes: 1. Military applications with an emphasis on object detection, tracking, and recognition. 2. Biometric applications with an emphasis on face recognition. 3. Medical applications with an emphasis on image analysis and visualization. Although the chapters describe research, they are not written as typical research papers. They have a tutorial flavor appropriate for a book. The book opens with the military applications since they represent the birthplace of CVBVS. All the major modalities used in military applications V
VI
Preface
are represented in the first five chapters. These include SAR (Synthetic Aperture Radar), laser radar, hyperspectral, and infrared. The first five chapters also address fundamental issues with regard to object detection, tracking, and recognition, sometimes in more than one modality. This allows comparative evaluation of these important computational imaging questions across the electromagnetic spectrum. In Chapter 1, Boshra and Bhanu et al. describe a theoretical framework for predicting the performance of object (target) recognition methods. The issue of identifying military targets in imagery is of great importance in military affairs. For years, target recognition was based purely on heuristics, and as a result performance was brittle. Boshra and Bhanu’s work is representative of a more rigorous methodological approach, which promises to transform target recognition from art to science. In Chapter 2, Bhanu and Jones unveil specific methods for improving the performance of an SAR target recognition system. SAR is probably the most successful imaging modality for military applications, because of its allweather capability. Bhanu and Jones’ methods conform to the model-based framework and involve incorporation of additional features, exploitation of a priori knowledge, and integration of multiple recognizers. In Chapter 3, Arnold et al. present target recognition methods in a different modality, namely, three-dimensional laser radar. Three-dimensional laser radars measure the geometric shape of targets. The main approach described in this chapter is quite appealing because it bypasses detection and segmentation processes. In Chapter 4, Kwon et al. deal with target recognition in the context of hyperspectral imagery. The basic premise of hyperspectral target recognition is that the spectral signatures of target materials are measurably different than background materials. Therefore, it is assumed that each relevant material, characterized by its own distinctive spectral reflectance or emission, can be identified among a group of materials based on spectral analysis of the hyperspectral data. Kwon et al. use independent component analysis (ICA) to generate a target spectral template. ICA is a method well-suited to the modular character of hyperspectral imagery. In Chapter 5, Vaswani et al. close the sequence of military application papers by presenting a method for object detection and compression in infrared imagery. The proposed solution is guided by the limitations of the target platform, which is an infrared camera with on-board chip. The object detection method is computationally efficient, to deal with the limited processing power of the on-board chip. It is also paired with a compression scheme to facilitate data transmission. Chapter 6 deals with biometrics and signals a transition from the military to civilian security applications. Wolff et al. present a face recognition approach based on infrared imaging. Infrared has advantages over visible imaging for face recognition, especially in the presence of variable lighting conditions.
Preface
VII
Wolff et al. provide quantitative support for this argument by unveiling a system that performs comparative evaluation. Chapter 7 opens the medical applications part of the book. It refers to cardiovascular image analysis of magnetic resonance imagery (MRI). While SAR is probably the most successful modality for military applications, one could make the case that MRI is the most successful modality for medical applications. Initially, MRI was treated much like x-rays. A radiologist, without any machine assistance, was interpreting the raw imagery. Increasingly, however, computer vision methods aid in this interpretation. In this chapter, Sonka et al. present techniques for 3D segmentation and quantitative assessment of left and right cardiac ventricles, arterial and venous trees, and arterial plaques. In Chapter 8, Fenster et al. present segmentation and visualization techniques in another very important medical imaging modality, that is, ultrasound. Specifically, the authors describe methods to reconstruct ultrasound information into 3D images to facilitate interactive viewing. They also describe automated and semi-automated segmentation methods to quantify organ and pathology volume for monitoring disease. In Chapter 9, Berry et al. introduce some very interesting image analysis work on a novel medical imaging modality, namely, terahertz pulsed imaging. Vis-a-vis the more established MRI and ultrasound modalities, terahertz pulsed imaging is the “new kid on the block”. Berry et al. propose Fourier transforms and wavelets to analyze spectroscopic information of materials. They actually demonstrate that these methods perform as well as traditional analysis methods for material properties and predict a number of biomedical applications that stand to benefit form this technology. The book can be used for instruction in graduate seminars or as a reference for the independent researcher. Although CVBVS is a broad and fast moving field, the balanced selection of key theoretical and practical issues represented in the chapters of the book will maintain their relevance for some time. It is our sincere hope that the book will serve as a springboard for the individual researcher who is interested in CVBVS research. A number of people have contributed in our effort and we are deeply grateful to all of them. These certainly include the authors of the individual chapters and the reviewers who patiently went through three review cycles. We are especially grateful to Pradeep Buddharaju who handled most of the last minute editing and thanks to whom the book assumed its finished form. Houston, Texas Riverside, California USA January 2004
Ioannis Pavlidis Bir Bhanu
Contents
Chapter 1 A Theoretical Framework for Predicting Performance of Object Recognition Michael Boshra, Bir Bhanu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
Chapter 2 Methods for Improving the Performance of an SAR Recognition System Bir Bhanu, Grinnell Jones III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Chapter 3 Three-Dimensional Laser Radar Recognition Approaches Gregory Arnold, Timothy J. Klausutis, Kirk Sturtz . . . . . . . . . . . . . . . . . . . 71 Chapter 4 Target Classification Using Adaptive Feature Extraction and Subspace Projection for Hyperspectral Imagery Heesung Kwon, Sandor Z. Der, Nasser M. Nasrabadi . . . . . . . . . . . . . . . . . 115 Chapter 5 Moving Object Detection and Compression in IR Sequences Namrata Vaswani, Amit K Agrawal, Qinfen Zheng, Rama Chellappa . . . 141 Chapter 6 Face Recognition in the Thermal Infrared Lawrence B. Wolff, Diego A. Socolinsky, Christopher K. Eveland . . . . . . . 167 Chapter 7 Cardiovascular MR Image Analysis Milan Sonka, Daniel R. Thedens, Boudewijn P. F. Lelieveldt, Steven C. Mitchell, Rob J. van der Geest, Johan H. C. Reiber . . . . . . . . 193 Chapter 8 Visualization and Segmentation Techniques in 3D Ultrasound Images Aaron Fenster, Mingyue Ding, Ning Hu, Hanif M. Ladak, Guokuan Li, Neale Cardinal, D´ onal B. Downey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
IX
X
Contents
Chapter 9 Time-Frequency Analysis in Terahertz-Pulsed Imaging Elizabeth Berry, Roger D Boyle, Anthony J Fitzgerald, James W Handley . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
Chapter 1 A Theoretical Framework for Predicting Performance of Object Recognition Michael Boshra1 and Bir Bhanu2 1 2
Center for Research in Intelligent Systems, University of California, Riverside, California 92521,
[email protected] Center for Research in Intelligent Systems, University of California, Riverside, California 92521,
[email protected]
Summary. The ability to predict the fundamental performance of model-based object recognition is essential for transforming the object recognition field from an art to a science, and to speed up the design process for recognition systems. In this chapter, we address the performance–prediction problem in the context of a common recognition task, where both model objects and scene data are represented by locations of 2D point features. The criterion used for estimating matching quality is based on the number of consistent data/model feature pairs, which we refer to as “votes.” We present a theoretical framework for prediction of lower and upper bounds on the probability of correctly recognizing model objects from scene data. The proposed framework considers data distortion factors such as uncertainty (noise in feature locations), occlusion (missing features), and clutter (spurious features). In addition, it considers structural similarity between model objects. The framework consists of two stages. In the first stage, we calculate a measure of the structural similarity between every pair of objects in the model set. This measure is a function of the relative transformation between the model objects. In the second stage, the model similarity information is used along with statistical models of the data distortion factors to determine bounds on the probability of correct recognition. The proposed framework is compared with relevant research efforts. Its validity is demonstrated using real synthetic aperture radar (SAR) data from the MSTAR public domain, which are obtained under a variety of depression angles and object configurations.
1.1 Introduction Model-based object recognition has been an active area of research for over two decades (e.g., see surveys [1, 2, 3, 4]). It is concerned with finding instances of known model objects in scene data. This process involves extracting features from the scene data, and comparing them with those of the model objects using some matching criterion. Performance of the recognition process depends on the amount of distortion in the data features. Data distortion
1
2
Michael Boshra and Bir Bhanu
can be classified into three types: (1) uncertainty: noise in feature locations and other feature attributes; (2) occlusion: missing features of data object of interest; and (3) clutter: spurious data features which do not belong to the data object of interest. In addition to data distortion, recognition performance depends on the degree of structural similarity between model objects. The often-overlooked similarity factor can have a profound impact on performance. Intuitively, the difficulty of recognizing a specific object is proportional to the degree of its similarity with the rest of the objects in the model set. data-distortion models
real test images
model objects
model objects
Image Synthesis Module synthetic test images
Object Recognition System
performance results
(a)
Object Recognition System
performance results
(b)
Figure 1.1. Empirical approaches for estimation of object recognition performance: (a) using real data, (b) using synthetic data.
Performance of object recognition is typically estimated empirically. This is done by passing a set of scene images containing known model objects to a recognition system, and then analyzing the output of the system. The set of scene images can be either real [5, 6, 7], or synthetic with artificial distortion introduced to them [8, 9, 10]. Both scenarios are illustrated in Figures 1.1(a) and 1.1(b), respectively. Empirical performance evaluation has a number of limitations: 1. It does not provide an understanding of the relationship between object recognition performance and the various data and model factors that affect it. In other words, the empirical approach can provide an answer to the question of what performance to expect, for a given set of model objects and specific data distortion rates. However, it does not explain why this is the expected performance. Such an understanding is critical for designing better object recognition systems, as it can provide fundamental answers to questions such as: (a) When does performance break down as a function of the amount of data distortion? (b) What are the
Chapter 1 Predicting Performance of Object Recognition
3
performance limits when using a specific sensor? (c) Is a given featureselection scheme sufficient for achieving desired levels of performance? (d) What is the largest size of a model set that can be accommodated without significantly degrading performance? Fundamental understanding of the relationship between performance and the factors affecting it is essential for the advancement of the field of object recognition from an art to a science. 2. The performance estimated empirically is dependent upon the actual implementation of the object recognition system. This implementation can be based on recognition approaches such as alignment [11, 12], hypothesis accumulation [13, 14], or tree search [15, 16]. Note that the performance obtained using these approaches can be different, even if they use similar matching criteria. For example, systems that use a vote accumulator (Hough space) will generate different performance estimates depending on the resolution of the accumulator. Another example, alignment-based systems, achieve polynomial-time complexity by using a “looser” notion of data/model feature consistency. 3. Empirical evaluation requires the presence of an actual object recognition system. Obviously, this can considerably slow down the design process. In this chapter, we address the performance–prediction problem in the context of a typical object recognition task. It can be described as follows. (1) Both model objects and scene data are represented by discretized locations of 2D point features. (2) A data object is assumed to be obtained by applying a 2D transformation to the corresponding model object. Notice that the space of possible 2D transformations is naturally discretized, since we are dealing with discrete 2D point features. (3) The data/model matching quality is estimated using a vote-based criterion. In particular, the quality of a given match hypothesis is estimated by counting the number of consistent data/model feature pairs, which we refer to as “votes.” We present a statistical method for formally predicting lower and upper bounds on the probability of correct recognition (PCR) for the task outlined above. The proposed method considers data distortion factors such as uncertainty, occlusion, and clutter, in addition to model similarity. Integrating these data and model factors in a single approach has been a challenging problem. The performance predicted is fundamental in the sense that it is obtained by analyzing the information provided by both the data and model features, independent of the particular vote-based matching algorithm. A schematic diagram of the prediction method is shown in Figure 1.2. It can be contrasted with the diagrams of the empirical approaches shown in Figure 1.1. The validity of the proposed method is demonstrated using real synthetic aperture radar (SAR) data from the MSTAR public domain. This data set is obtained under a variety of depression angles and object configurations. The remainder of this chapter is organized as follows. The next section reviews related research efforts, and highlights our contributions. Section 1.3
4
Michael Boshra and Bir Bhanu data-distortion models
model objects
Performance Prediction Method
performance results
Figure 1.2. Formal estimation of object recognition performance.
presents an overview of the proposed method. Sections 1.4 and 1.5 describe the statistical modeling of the data distortion factors, and the object similarity, respectively. Derivation of lower and upper bounds on PCR is presented in Section 1.6. The validity of those bounds is demonstrated in Section 1.7, by comparing actual PCR plots, as a function of data distortion, with predicted lower and upper bounds. Finally, conclusions and directions for future research are presented in Section 1.8.
1.2 Relevant Research Several research efforts have addressed the problem of analyzing performance of feature-based object recognition. Most of these efforts focus on the problem of discriminating objects from clutter. We present here a representative sample of these efforts. Grimson and Huttenlocher [17] presented a statistical method for estimating the probability distribution of the fraction of consistent data/model feature pairs for an erroneous hypothesis. They derived such a distribution using a statistical occupancy model (Bose–Einstein model), assuming bounded feature uncertainty and uniform clutter models. This distribution was used to determine the minimum fraction of consistent feature pairs required to achieve a desired probability of false alarm. Sarachik [18] studied the problem of predicting the receiver operating characteristic (ROC) curve for a specific recognition algorithm. The ROC curve described the relationship between the probability of correct recognition and that of a false alarm. The chosen algorithm used a weighted voting criterion based on Gaussian feature uncertainty. A statistical analysis was presented to determine the probability distributions of the weighted votes for both valid and invalid hypotheses, assuming uniform occlusion and clutter models. These distributions were used along with the likelihood-ratio test to predict the ROC curve. Alter and Grimson [8] used statistical knowledge about sources of data distortion to design a recognition criterion, based on the likelihood-ratio test. The likelihoods of observed data-feature set, conditioned on hypothesis validity and
Chapter 1 Predicting Performance of Object Recognition
5
invalidity, were calculated by assuming both bounded and Gaussian feature uncertainty models, in addition to uniform occlusion and clutter models. Lindenbaum [19] extended the modeling of clutter to include background objects of known shape, in addition to uniformly distributed random features. This hybrid model was incorporated into a statistical analysis to predict the number of features needed to guarantee recognition of a data object at a given confidence level. The analysis considered bounded feature uncertainty, as well as structural similarity between a given data object and background ones. Irving et al. [20] derived a theoretical bound on the ROC curve of an object detection task. The generalized likelihood-ratio test was used to discriminate between model objects at various poses and random clutter. The likelihoods of both clutter and model objects were modeled using 2D Poisson processes. Modeling object likelihood as a Poisson process was based on the assumption of independence of object views at discretized poses. This work considered bounded feature uncertainty, and uniform occlusion and clutter models. The problem of discriminating objects from other model objects has received considerably less attention than object/clutter discrimination. This problem is obviously central to integrated performance prediction of object recognition. It requires consideration of not only data distortion but also object similarity. In addition, it requires consideration of the interaction between object similarity and data distortion. Lindenbaum [21] presented a probabilistic analysis for predicting lower and upper bounds on the number of data features required to achieve a certain confidence level in object localization or recognition. It explicitly considered the similarity between different model objects, as well as the self-similarity between a model object and an instance of itself at a different relative pose. The data distortion factors considered were bounded uncertainty and occlusion. The analysis considered extreme cases in modeling the interaction between occlusion and similarity, thus resulting in the generation of relatively loose bounds. We note that the analysis presented in [19], outlined above, can be used in the context of object/object discrimination considering uncertainty and clutter, as well as object similarity. Grenander et al. [22] addressed the problem of predicting fundamental error in object pose estimation. In their work, objects were represented by templates at the pixel level. A minimum mean-square-error estimator, the Hilbert–Schmidt estimator, was used to estimate object pose in the presence of pixel uncertainty. Performance of object/object discrimination was determined partially empirically through the synthesis of distorted templates of one object, and then using the likelihood-ratio test, based on the Hilbert–Schmidt estimator, as a recognition criterion (refer to Figure 1.1(b)). The methods outlined above are summarized and compared with our method in Table 1.1. This table highlights the main contribution of our work, namely the integration of uncertainty, occlusion, clutter, and similarity factors in a single approach for performance prediction. As shown in the table, previous methods considered only a subset of these factors. It can also be seen that our method is unique among other object/object discrimination methods
6
Michael Boshra and Bir Bhanu
Table 1.1. Comparison between performance-analysis methods (U , O, C, and S denote uncertainty, occlusion, clutter, and similarity, respectively). Work
Discrimination Data/Model Features Transform. Factors U OC S Grimson et al. [17] object/clutter 2D/2D lines rigid X X Sarachik [18] object/clutter 2D/2D points affine XXX Alter and Grimson [8] object/clutter 2D/3D points & lines weak persp. X X X Lindenbaum [19] object/clutter 2D/2D boundary pts. affine X XX Irving et al. [20] object/clutter 2D/2D points 2D transl. X X X Lindenbaum [21] object/object 2D/2D boundary pts. rigid XX X Grenander et al. [22] object/object pixel-level templates rotation X This work object/object 2D/2D discretized pts. 2D transl. X X X X
in that it considers point features. Another unique aspect of this work is not just the new theory but also the validation using real data. We note that parts of this work have appeared in [23, 24, 25].
1.3 Overview In this section, we present an overview of the proposed performance–prediction method. Our problem can be formally defined as follows. We are given the following: 1. A set of model objects, MD = {Mi }, where each object Mi is represented by discretized locations of 2D point features, Mi = {Fik }. 2. Statistical data distortion models. 3. A class of data/model transformations, T . Our objective is to predict lower and upper bounds on PCR as a function of data distortion. We consider recognition to be successful only if the selected hypothesized object is the actual one, and the difference between the hypothesized pose and the actual one is small. The pose error can be represented by the relative pose of the hypothesized object with respect to the actual one. It is considered acceptable if it lies within a subspace, Tacc ⊂ T . We assume in this work that Tacc = {0}, i.e., only exact object location is acceptable. A block diagram of the proposed method is shown in Figure 1.3. The main elements in this diagram can be described as follows: • Data-Distortion Models: The data distortion factors are statistically modeled using uniform probability distribution functions (PDFs). 1. Uncertainty: The location of the data feature corresponding to a model feature is described by a uniform distribution. Notice that the uncertainty PDFs are discrete, since the feature locations considered in this work are discretized. We further assume that the PDFs associated with different
Chapter 1 Predicting Performance of Object Recognition
Data-Distortion Models occlusion clutter uncertainty
7
Transformation Class Model Objects
Computation of Object Similarity similarity histograms
Computation of Performance Bounds
Lower and Upper PCR Bounds
Figure 1.3. Block diagram of performance–prediction method.
model features are independent. We argue that such independence assumption is reasonable in most practical applications. 2. Occlusion: We assume that every subset of model features is equally likely to be occluded as any other subset of the same size. This assumption is more appropriate for modeling features that are missing due to inherent instability or imperfections of feature extraction. It is less suitable for modeling features that are missing due to being occluded by other objects, since it does not consider the spatial-correlation aspect among occluded/unoccluded features. Spatial correlation can be captured by using Markov random fields [9, 26, 27], at the expense of significantly increasing the complexity of the analysis. In Section 1.7, we outline a simpler approach that can implicitly consider the spatial-correlation factor, without increasing the analysis complexity. 3. Clutter: We assume that clutter features are uniformly distributed within a region surrounding the object. This distribution is useful for modeling random clutter, which does not have specific spatial structure. Modeling “structural” clutter requires analyzing its similarity with model objects. We note that the similarity-modeling concepts presented in this work can be used in modeling of structural clutter. This topic is a subject for future research. • Computation of Object Similarity: The purpose of this stage is to compute the structural similarity information among all pairs of model objects. Our definition of object similarity depends on the amount of feature uncer-
8
Michael Boshra and Bir Bhanu
tainty. In particular, the similarity between two model objects is directly proportional to feature uncertainty. This agrees with the intuitive observation that as different objects become more “blurred,” it becomes more difficult to differentiate between them, which is, in a sense, equivalent to saying that they become more “similar.” The similarity between a model object, Mi , and another one, Mj , is defined as the number of votes for Mj given an uncertain instance of Mi , i.e., an instance of Mi that is obtained by randomly perturbing its features. Accordingly, the number of votes for Mj is a random variable. The chosen definition of similarity depends on the relative transformation between Mi and Mj , defined by transformation class T . Accordingly, the similarity between Mi and Mj can be viewed as a probabilistic function, which we call the similarity function. The similarity information is encoded in two histograms, which we call all-similarity and peak-similarity histograms, to be described in Section 1.5.2. These histograms are used for predicting lower and upper PCR bounds, respectively. • Computation of Performance Bounds: The objective of this stage is to compute PCR bounds. The computation is based on estimating the PDF of the votes for a specific erroneous object/pose hypothesis, given a “distorted” instance of a given model object. The estimation process takes into account the structural similarity between the model object and the erroneous hypothesis. The vote PDF is used to determine the probability of a recognition failure, which occurs if the erroneous hypothesis gets same or more votes than the distorted object. This information is integrated for potential erroneous hypotheses to determine the PCR bounds.
1.4 Data-Distortion Models We formally model the effects of the three distortion factors considered in this work on a “perfect” model object. This modeling is used to determine the vote PDF in Sect. 1.6.2. • Uncertainty: The effect of the uncertainty factor is to perturb locations of model features according to some PDF. Since this PDF is assumed to be uniform, it can be represented by a region. Let Fik ∈ Mi be a model feature, and Fik be a distorted instance of it. Define Ru (Fik ) to be the consistency region associated with Fik . Such region bounds the possible locations of Fik ¯ u (Fik ), where given Fik , i.e., Fik ∈ Ru (Fik ). Likewise, we can say that Fik ∈ R ¯ Ru (·) is the reflection of Ru (·) about the origin. Practically, Ru (·) is the same ¯ u (·), because Ru (·) is symmetric about the origin (e.g., circle, square). as R ¯ u (·) = Ru (·). An Accordingly, we assume in this work for simplicity that R uncertain instance of Mi can be obtained by uniformly perturbing each of its features within corresponding consistency region. This can be formally represented as: Du (Mi , Ru (·)) = {Pu (Ru (Fik )) : Fik ∈ Mi },
Chapter 1 Predicting Performance of Object Recognition
9
where Pu (R) is a function that returns a feature selected randomly within region R. • Occlusion: The effect of occlusion is the elimination of some model features. An occluded instance of Mi can be formally defined as Do (Mi , O) = Mi − Po (Mi , O), where Po (Mi , O) is a function that returns a subset of O features selected randomly from Mi . For a fixed O, all subsets generated by Do (Mi , O) are equally likely to occur, since we are assuming uniform occlusion. • Clutter: The effect of clutter on a model object is the addition of spurious features to it. They are assumed to be uniformly distributed within a clutter region, Rc , surrounding the model object. This region can have an arbitrary shape (e.g., bounding box of model features, convex hull, etc). A cluttered instance of Mi can be defined as Dc (Mi , C, Rc , Rx ) = Mi ∪ Pc (C, Rc − Rx ), where Pc (C, R) is a function that returns C features selected randomly within region R, and Rx is a region that clutter features are excluded from falling into. The reason for including Rx is explained below. • Combined Distortion: Consideration of the combined effects of uncertainty, occlusion, and clutter on a model object raises an ambiguous situation. It takes place when a model feature gets occluded and then a spurious feature falls within its consistency region. The ambiguity arises from the fact that this situation can not be distinguished from the no-occlusion/no-clutter case. In order to simplify the analysis, we assume the latter case. This can be modeled by restricting the clutter features to lie outside region Rx , defined as the union of the consistency regions of occluded features. We refer to Rc − Rx , or simply Rc , as the effective clutter region. A distorted instance i (Ru (·), O, C, Rc ), can be obtained by first occluding O features of of Mi , M Mi , perturbing unoccluded ones within their consistency regions Ru (·), and then randomly adding C clutter features within the effective clutter region Rc . This can be represented formally as: i (Ru (·), O, C, Rc ) = Dc (Du (Do (Mi , O), Ru (·)), C, Rc , Rx ), M where Rx = ∪k Ru (Fik ), ∀Fik ∈ (Mi − Do (Mi , O)). Figure 1.4 shows an example of the distortion process.
1.5 Computation of Object Similarity In this section, we formally define a measure of the structural similarity between model objects, and outline the method used to construct the similarity histograms.
10
Michael Boshra and Bir Bhanu
M
cR
M
i
(a)
M
cR i
(b)
cR x
i
x
M
Rc i
x
(c)
(d)
x
x
M
Rc i
x
(e) Figure 1.4. An illustration of the different stages of the distortion process: (a) original object consisting of six features (dark circles), (b) after occlusion (O = 2; small circles represent occluded features), (c) after perturbation (Ru (·) = a circle; small double circles represent locations of features before perturbation), (d) after clutter (C = 3; small crosses represent clutter features; notice absence of clutter features inside consistency regions of the two occluded features), (e) distorted object.
1.5.1 Definition of Object Similarity We introduce a number of definitions that lead to a definition of the similarity between a pair of model objects.
Chapter 1 Predicting Performance of Object Recognition
11
τˆ • Vote-Based Criterion: Let Mτiˆ = {Fik } be object Mi at pose τˆ ∈ T with respect to a data object, M. We refer to Mτiˆ as a hypothesis of object is the number of features in Mi at location τˆ. The votes for Mτiˆ , given M, A model feature, Mτiˆ that are “consistent” with at least a data feature in M. τˆ τˆ if Fl falls Fik ∈ Mi , is said to be consistent with a data feature, Fl ∈ M, τˆ τ ˆ within the consistency region of Fik , i.e., Fl ∈ Ru (Fik ). Accordingly, we can as follows: formally define the votes for Mτiˆ given M
=|{F τˆ : F τˆ ∈ Mτˆ and ∃Fˆl ∈ M s.t. Fˆl ∈ Ru (F τˆ )}| (1.1) . VOTES(Mτiˆ ; M) ik ik i ik • Feature/Feature Similarity: Let us assume that we are given a pair of model features, Fik ∈ Mi and Fjlτi ∈ Mτj i , where Mτj i is a hypothesis of object Mj at location τi ∈ T with respect to object Mi . The similarity between Fik and Fjlτi , denoted by Sf f (Fik , Fjlτi ), is defined as the probability that an uncertain measurement of Fik is consistent with Fjlτi . Formally, we have Sf f (Fik , Fjlτi )
=
AREA(R(Fik ) ∩ R(Fjlτi )) AREA(R(Fik ))
,
where AREA(R) is area of region R. Obviously, Sf f (Fik , Fjlτi ) lies in the range [0, 1]. It is proportional to the extent of overlap between the consistency regions of Fik and Fjlτi (R(Fik ) and R(Fjlτi )). Figure 1.5 illustrates Sf f (Fik , Fjlτi ) as a function of τi , for a sample of three consistency regions. In some cases, we refer to feature pairs with overlapping/nonoverlapping consistency regions as similar/dissimilar feature pairs, respectively. • Object/Feature Similarity: We define the similarity between an object, Mi , and a feature, Fjlτi ∈ Mτj i , as the probability that Fjlτi is consistent with an uncertain measurement of any feature in Mi . We can formally define object/feature similarity, denoted by Sof (Mi , Fjlτi ), as Sof (Mi , Fjlτi ) = 1 −
(1 − Sf f (Fik , Fjlτi )). k
• Object/Hypothesis Similarity: Let us denote the similarity between Mi and Mτj i as Soh (Mi , Mτj i ) or simply Sjτi . We define Sjτi as the number of votes for hypothesis Mτj i , given an uncertain instance of Mi , which is Du (Mi , Ru (·)) (refer to Section 1.4). Formally, Sjτi = VOTES(Mτj i ; Du (Mi , Ru (·))). It is obvious that Sjτi is a random variable. The minimum value of Sjτi is the number of coincident feature pairs of Mi and Mτj i . It can be expressed as τi τi min(Sjτi ) = | {Fjk : Sof (Mi , Fjk ) = 1} | .
12
Michael Boshra and Bir Bhanu
1
R (F ) u ik
0.8
Sff
0.6
F τ
ik
0.4
i 0.2
τ
F
R (F i ) u
0 2
τ
i
2
1
jl
jl
1
0 0 −1
−1 −2
τ
−2
τ
iy
ix
(a)
1
R (F
u ik
)
0.8
Sff
0.6
0.4
F
ik
0.2
τ
τ
0 2
Fi
i
jl
2
1 1
0 0 −1
τ
R (F u
jl
i
−1 −2
τ
)
−2
τ
iy
ix
(b)
1
0.8
Sff
0.6
0.4
R (F u
τ jl
0.2
i)
τ
R (F
u ik
F
)
τ
F i
0 2 2
1
i
1
0 0
jl
−1
τiy
ik
−1 −2
−2
τ
ix
(c) Figure 1.5. : An illustration of feature/feature similarity for a variety of consistency regions assuming T is the space of 2D translations: (a) a circle of unit radius, (b) a discrete eight-neighbor region, (c) a point region (implies absence of positional uncertainty). Note that the components of τi along the x- and y-axes are represented τi by τix , and τiy , respectively. We assume here for simplicity that Fik = Fjl when τi = 0.
Chapter 1 Predicting Performance of Object Recognition
13
On the other hand, the maximum value of Sjτi is the number of features of Mτj i that are similar to features in Mi (i.e., whose consistency regions overlap with consistency regions of features in Mi ). Thus, τi τi max(Sjτi ) = | {Fjk : Sof (Mi , Fjk ) > 0} | .
The expected value of Sjτi can be approximated as τi E(Sjτi ) ≈ Sof (Mi , Fjk ), k
where ity.
τi Fjk
∈
Mjτi .
Figure 1.6 shows an example of object/hypothesis similar-
Mi
τi
Mj
Figure 1.6. An illustration of object/hypothesis similarity. Notice that there are three similar feature pairs with feature/feature similarity values of approximately 1 2 , and 1. Accordingly, we have Sjτi ∈ [1, 3], and E(Sjτi ) ≈ 2. 3 3
• Uniform Model of Object/Hypothesis Similarity: In order to make the prediction of PCR bounds mathematically tractable, we make the following reasonable assumptions about Mi , Mτj i and the structure of their similar feature pairs: 1. The consistency regions of the features that belong to each of Mi and Mτj i are not overlapping. 2. The correspondence between similar features in Mi and Mτj i is bijective (one-to-one). 3. The feature/feature similarity between every pair of similar features is a constant value, Pjτi . It is the average object/feature similarity of the features in Mτj i that are similar to features in Mi . The above assumptions result in a “uniform” view of the structural similarity between object Mi and hypothesis Mτj i . As an illustration, Figure 1.7 shows the uniform model corresponding to the object/hypothesis pair shown in Figure 1.6. The uniform similarity model leads to the approximation of the PDF of Sjτi by the following binomial distribution:
14
Michael Boshra and Bir Bhanu
PS τi (sτj i ) = BS τi (sτj i ; Njτi , Pjτi ), j
j
where PX (x) = Pr[X = x], BX (x; n, p) = K(n, x)px (1 − p)n−x , K(a, b) = a! (a−b)! b! , Njτi = max(Sjτi ), and Pjτi =
E(Sjτi ) . Njτi
Mi
τi
Mj
Figure 1.7. Uniform similarity model for object/hypothesis pair shown in Figure 1.6. Notice that similar feature pairs have constant feature/feature similarity, Pjτi ≈ 2 and Njτi = 3. 3
• Object/Object Similarity: The similarity between a pair of objects, Mi and Mj , is defined as the object/hypothesis similarity Sjτi , for all τi ∈ T . Thus, object/object similarity can be viewed as a probabilistic function. As an illustration, Figure 1.8(a) shows a pair of simple model objects. The corresponding expected-similarity function, E(Sjτi ), is shown in Figure 1.8(b). Note that peaks in the expected-similarity function correspond to object hypotheses that have a higher degree of similarity with Mi than neighboring ones. A sample of these hypotheses, referred to as peak hypotheses, is shown in Figure 1.9. In our work, peak hypotheses are used for predicting an upper bound on PCR. 1.5.2 Construction of Similarity Histograms As discussed in the previous section, we describe the object/hypothesis similarity between Mi and Mτj i by two parameters, (Njτi , Pjτi ). For our purpose of performance prediction, we add two more parameters: 1. The size of Mi , | Mi |. 2. The effective size of Mτj i , | Mτj i ∩ Rc |, which is the number of features of Mτj i that lie inside the clutter region Rc .
Chapter 1 Predicting Performance of Object Recognition Mi
2
Mj
2
9
15
9
X
x
2
2 F
F
F
F
F
i2
i1
j1
7
7 i3
F
F
j3
i4
j2
Y
Y
(a)
3
← Peak B ← Peak C
2.5
E(Sτj i)
2 1.5 1 0.5 0 8
← Peak A
6
10
4
5
2 0
0
−2 −4
−5 −6
τiy
−8
−10
τix
(b) Figure 1.8. An illustration of object/object similarity: (a) model objects Mi and Mj , (b) corresponding expected-similarity function, E(Sjτi ), assuming four-neighbor consistency region, and 2D translation space.
Thus, we encode the information of object/hypothesis similarity using tuple (| Mi |, | Mτj i ∩ Rc |, Njτi , Njτi Pjτi ). Accordingly, the similarity information is accumulated in 4D histograms 3 . Two similarity histograms are needed in our work, one for storing similarity information corresponding to all erroneous hypotheses, and the other 3
When calculating the effective size of Mτj i , we have also included features of Mτj i that lie outside Rc but are similar to features in Mi .
16
Michael Boshra and Bir Bhanu M (j -7
-5)
t
X
Mi
(0 0) t
Mj
X
Mi
X
Y
Y
Y
Peak A
Peak B (7 0) t
Mi
X
Mj
X
Y
Y
Peak C Figure 1.9. Three hypotheses corresponding to peaks A, B and C shown in Fig. 1.8(b), assuming four-neighbor consistency region.
for storing information corresponding to peak hypotheses only. They are referred to as all- and peak-similarity histograms, respectively. The algorithm used to construct these histograms is outlined in Figure 1.10. It calculates the similarity between every model object Mi , and all the erroneous hypotheses competing with it. The erroneous hypotheses are selected to satisfy the following two criteria: 1. Each has at least one feature inside clutter region Rc . 2. For hypotheses that belong to Mi , the relative pose, τi , lies outside Tacc , defined in Section 1.3. The similarity information associated with Mi is accumulated in local all- and peak-similarity histograms, ASHi and P SHi , respectively. These histograms, for all Mi ∈ MD, are then added to form global similarity histograms, ASH and P SH, respectively.
Chapter 1 Predicting Performance of Object Recognition
17
Initialize global similarity histograms ASH and P SH for each model object Mi ∈ MD do Initialize local similarity histograms for Mi , ASHi and P SHi for each model object Mj ∈ MD do for each τi ∈ T such that | Mτj i ∩ Rc |> 0 do if (i = j) ∨ ¬(τi ∈ Tacc ) then Compute similarity parameters (Njτi , Pjτi ) Increment ASHi (|Mi|, |Mτj i ∩ Rc|, Njτi , Njτi Pjτi+12 ) by 1 if Mτj i is a peak hypothesis then Increment P SHi (|Mi|, |Mτj i ∩ Rc|, Njτi , Njτi Pjτi+12 ) by 1 end if end if end for end for Add ASHi to ASH Add P SHi to P SH end for Figure 1.10. Similarity-computation algorithm.
1.6 Computation of Performance Bounds In this section, we derive the PDF of votes for an erroneous hypothesis, and use this PDF for predicting lower and upper bounds on PCR. 1.6.1 Motivating Example We start by presenting an example to illustrate the combined effects of data distortion and object similarity on the vote process. This example, illustrated in Figure 1.11, assumes the uniform model of similarity between Mi and Mτj i , which is defined in Section 1.5.1. It can be described as follows: •
Prior to being distorted, Mi has five votes, since it consists of five features. On the other hand, Mτj i does not have any features of Mi within the consistency regions of its features. Accordingly, it gets no votes. • The first distortion step involves occlusion of two features in Mi . Obviously, this reduces the number of votes for Mi from five to three. At this point, Mτj i still does not get any votes. Notice that the number of similar feature pairs between Mi and Mτj i decreases from three (which is Njτi ; refer to Section 1.5.1) to two. • The second step involves randomly perturbing the three unoccluded features in Mi within their consistency regions. This keeps the number of votes for Mi at three. On the other hand, observe that both of the two unoccluded similar features of Mi move to the regions that overlap with the consistency regions of their corresponding similar features in Mτj i . This contributes two votes to Mτj i .
18
Michael Boshra and Bir Bhanu
VOTES Mi Mi
τ Mj i
τ Mj i
5
0
3
0
3
2
3
4
two features occluded
Mi
τ Mj i
(No = 2)
three features perturbed
Mi
τ Mj i
(V s = 2)
four features added Mi x
x
x
τ Mj i
(V c = 2)
x
Rc
Figure 1.11. An example showing the vote process for object Mi and erroneous hypothesis Mτj i , as Mi gets distorted.
• In the final distortion step, four clutter features are randomly added within clutter region Rc . Two of these features happen to fall within the consistency regions of two new features of Mτj i . This contributes two extra votes for Mτj i , thus bringing its total number of votes to four. The number of votes for Mi stays the same (recall from Section 1.4 that clutter features are excluded from falling into consistency regions of occluded ones). The above example shows how data distortion and model similarity can result in a recognition failure by reducing the number of votes for the correct hypothesis, and increasing them for an erroneous one. It also provides us with the following valuable insight into the distribution of votes for both correct and incorrect hypotheses, as a function of data distortion: • The number of votes for Mi , denoted by Vi , is simply the number of unoccluded features of Mi . That is, for a distorted instance of Mi , i , we have i (Ru (·), O, C, Rc ) or simply M M Vi = | Mi | −O.
(1.2)
Chapter 1 Predicting Performance of Object Recognition
19
• The number of votes for Mτj i , denoted by Vjτi , comes from two different sources: (1) object Mi , due to structural similarity (second distortion step in Figure 1.11), and (2) clutter features, due to random coincidence (third distortion step in Figure 1.11). Thus, Vjτi is a random variable that can be expressed as follows: Vjτi = Vs + Vc ,
(1.3)
where Vs and Vc are random variables that represent similarity and clutter votes for Mτj i , respectively. • The number of similarity votes, Vs , is bounded by the number of similar feature pairs that remain unoccluded, which we denote by No (obviously, No ≤ Njτi ). In the above example, it can be seen that Vs = 2, Vc = 2, and No = 2. In the next section, we use these three random variables to determine the PDF of Vjτi . 1.6.2 Probability Distribution of Hypothesis Votes In order to determine the PDF of Vjτi , we need to determine the PDFs of Vs (number of similarity votes), Vc (number of clutter votes), and No (number of unoccluded similar features). In the previous section, we have seen that Vs depends on No . Accordingly, we can express the PDF of Vs as PVs (vs ; no )PNo (no ), (1.4) PVs (vs ) = no
where PVs (vs ; no ) = Pr[Vs = vs ; No = no ]. From (1.3) and (1.4), we can represent the PDF of Vjτi as PV τi (vjτi ) =
j
no
PV τi (vjτi ; no )PNo (no ), j
(1.5)
where PV τi (vjτi ; no ) = j
PVs (vs ; no )PVc (vjτi − vs ; no , vs )
vs
and PVc (vc ; no , vs ) = Pr[Vc = vc ; No = no , Vs = vs ]. We estimate the PDF of No and the conditional PDFs of Vs and Vc based on the uniform models of data distortion and structural similarity, presented in Sections 1.4 and 1.5.1, respectively. • PDF of No : The process of occluding O features in Mi can be viewed as picking O balls from an urn, which contains Njτi white balls and (| Mi | −Njτi ) black balls, with no replacement. In our case, the white (black) balls represent features in Mi that are similar (dissimilar) to features in Mτj i . Based on the
20
Michael Boshra and Bir Bhanu
uniform occlusion and similarity models, the PDF of No can be described by the following hypergeometric distribution, PNo (no ) = HNo (Njτi − no ; O, Njτi , | Mi | −Njτi ), where HX (x; n, a, b) = no ∈
K(a,x)K(b,n−x) . K(a+b,n)
[max(0, Njτi
(1.6)
Note that
− O), min(Njτi , | Mi | −O)].
• Conditional PDF of Vs : It can be easily shown that the conditional PDF of Vs is represented by the following distribution!binomial binomial distribution: PVs (vs ; no ) = BVs (vs ; no , Pjτi ). This distribution is obtained based on the assumptions of uniform uncertainty and similarity models. Notice that Pjτi < 1 implies vs ∈ [0, no ], while Pjτi = 1 implies vs = no . • Conditional PDF of Vc : The estimation of the PDF of Vc is considerably more involved than those of No and Vs . It can be outlined as follows. Let RV c ⊂ Rc be the largest region such that a clutter feature falling within it will contribute a vote for Mτj i . Region RV c is the union of the consistency regions of features in Mτj i ∩ Rc that do not have any features of Mi within their consistency regions. They are basically all the features of Mτj i ∩ Rc minus those that have similar features of Mi within their consistency regions. A slight complexity arises from our clutter modeling explained in Section 1.4: features in Mτj i ∩ Rc that are similar to occluded features in Mi are effectively associated with “truncated” consistency regions. Figure 1.12 shows an example of RV c . Based on the assumption of uniform similarity, we can show the following: 1. The area of a truncated consistency region is AREA(Ru (·))(1 − Pjτi ). 2. The numbers of potential vote-contributing features with truncated and full consistency regions are nt = Njτi −no , and nf =| Mτj i ∩Rc | −vs −nt , respectively. Splitting the effective clutter region Rc into two subregions, RV c and Rc −RV c , we can approximate the conditional PDF of Vc by the following binomial distribution, AREA(RV c ) PVc (vc ; no , vs ) ≈ BVc vc ; C, , (1.7) AREA(Rc ) where AREA(RV c ) = AREA(Ru (·))(nf + (1 − Pjτi )nt ), and AREA(Rc ) = AREA(Rc ) − O × AREA(Ru (·)).
The lower bound of vc is 0, while the upper bound is either min(nf + nt , C) if Pjτi < 1, or min(nf , C) if Pjτi = 14 . 4
The area of Rc is calculated by assuming, for simplicity, that clutter region Rc totally covers the consistency regions of the features of Mi .
Chapter 1 Predicting Performance of Object Recognition
21
Figure 1.12. An illustration of the clutter vote region RV c , assuming the uniform model of similarity between Mi and Mjτi .
1.6.3 Lower Bound on PCR Let Hi be the set of erroneous object/pose hypotheses corresponding to Mi . It can be defined as Hi={Mτj i : Mj ∈ MD and τi ∈ T s.t. |Mτj i ∩ Rc|> 0} − {Miτi : τi ∈ Tacc }. We can express the probability of misinterpreting a distorted instance of Mi , i , as any hypothesis in Hi , as M i ] = Pr[∃ Mτi ∈ Hi s.t. V τi ≥ Vi ]. Pr[Hi ; M j j
(1.8)
The probability that Mτj i “beats” Mi (i.e., Mτj i reaches or exceeds votes for Mi ) can be obtained from (1.2) and (1.5): i ] = Pr[Mτj i ; M PV τi (vjτi ). (1.9) τ
vj i ≥|Mi |−O
j
From (1.8) and (1.9), we obtain the following upper bound on the probability of recognition failure: i ] < i ]. Pr[Hi ; M Pr[Mτj i ; M τ
Mj i ∈Hi
The above inequality directly leads to the following lower bound on PCR: i ] > 1 − i ]. Pr[Mτj i ; M (1.10) Pr[Mi ; M τ
Mj i ∈Hi
From the derivation of the vote PDF discussed in the previous section, we i ] depend on only four objectcan observe that Vjτi and, in turn, Pr[Mτj i ; M dependent parameters: size of Mi , effective size of Mτj i , and the two similarity parameters (Njτi , Pjτi ). Define
22
Michael Boshra and Bir Bhanu
i ], W (a, b, c, d) = Pr[Mτj i ; M such that a =| Mi |, b =| Mτj i ∩Rc |, c = Njτi and d = Njτi Pjτi + 12 . Equation (1.10) can be rewritten as i ] > 1 − Pr[Mi ; M ASHi (a, b, c, d)W (a, b, c, d). (1.11) a
b
c
d
Taking the average of (1.11) over all model objects in the model set MD, we obtain the following lower bound on average PCR for MD: PCR(MD) > 1−
1 ASH(a, b, c, d)W (a, b, c, d), (1.12) |MD| a c b
d
where ASH(a, b, c, d) =
ASHi (a, b, c, d).
i
1.6.4 Upper Bound on PCR In this section, we present three possible upper bounds on PCR. These bounds differ from each other in the degree of their tightness, and their reliance on assumptions. The first upper bound can be obtained by observing that recognition fails if any hypothesis in Hi beats Mi . The probability that this event takes place i ], which is defined in (1.9). The for a given hypothesis, Mτj i , is Pr[Mτj i ; M maximum of these probabilities among all hypotheses in Hi forms a lower bound on the probability of recognition failure. This directly leads us to the following upper bound on PCR: i ] < 1 − max Pr[Mτi ; M i ], Pr[Mi ; M j τi ,j
(1.13)
where Mτj i ∈ Hi . Obviously, we do not expect this bound to be tight, since it involves only a single erroneous hypothesis. One possible approach for obtaining a bound that is tighter than the one in (1.13) is to consider a subset of the hypotheses in Hi , and make the assumption that the vote PDF’s for these hypotheses are independent. Obviously, the voteindependence assumption is not reasonable among adjacent hypotheses due to the overlap of their consistency regions. We propose to consider hypotheses that correspond to peaks in the expected similarity function (refer to Section 1.5.1). The rationale behind such choice can be outlined as follows: •
Peak hypotheses tend to occur at random locations, which makes the voteindependence assumption among them more reasonable than when applied to adjacent ones.
Chapter 1 Predicting Performance of Object Recognition
23
• Peak hypotheses have a higher degree of similarity with given model object Mi , than neighboring ones. Accordingly, we can say that a distorted instance of Mi is more likely to be misinterpreted as a peak hypotheses than an off-peak neighbor. Let Hpi be the set of peak hypotheses associated with model object Mi . That is, Hpi = {Mτj i : Mτj i ∈ Hi , and τi is a peak in E(Sjτi )}. Based on the vote-independence assumption, we can obtain the following upper bound on PCR: i ] < min (1 − Pr[Mτi ; M i ]), Pr[Mi ; M (1.14) j j
τi
Mτj i
where ∈ Hpi . This upper bound is tighter than the one defined in (1.13), since it considers a representative subset of model hypotheses that belong to a single object, instead of just a single hypothesis as in (1.13). In order to obtain a bound that is tighter than the one in (1.14), we need to consider hypotheses in Hi that belong to all model objects, not just a single object as in (1.14). One possible way of achieving this goal is to make the assumption that the vote PDFs for peak hypotheses that belong to different model objects are independent. This assumption leads us to the following upper bound, i ] < i ]), Pr[Mi ; M (1 − Pr[Mτj i ; M (1.15) j,τi
Mτj i
where ∈ Hpi . We note that the above assumption can be invalid in some extreme cases, such as when two of the model objects are identical or very similar. However, we argue that it is reasonable in many practical scenarios. For example, let us consider the domain of object recognition using SAR data, which is the main motivation behind our efforts (see the experimental results in Section 1.7). It is well known that the features extracted from a SAR image are extremely sensitive to the geometry of the object shape. Accordingly, even in the presence of similar objects, we can expect the corresponding feature sets to be considerably different [10]. In our experiments, we use the tightest bound defined in (1.15). This bound can be rewritten as i ] < Pr[Mi ; M (1 − W (a, b, c, d))P SHi (a,b,c,d) . (1.16) a
b
c
d
Taking the geometric mean of (1.16) for all model objects in MD, we obtain an estimate of the upper bound of PCR corresponding to MD, P SH(a,b,c,d) PCR(MD) < (1 − W (a, b, c, d)) |MD| , (1.17) a
b
c
d
24
Michael Boshra and Bir Bhanu
where P SH(a, b, c, d) =
P SHi (a, b, c, d).
i
1.7 Experimental Validation In this section, we validate the proposed prediction method by comparing predicted bounds on performance with actual ones determined empirically. 1.7.1 Recognition Task The recognition task that we consider in this work involves recognition of objects using SAR data [10]. We use real data from the MSTAR public domain [28]. The model set consists of a number of objects, which are military targets. Each object is represented by a number of SAR views that sample its signature at several azimuth angles and a specific depression angle. We consider each view as an independent 2D “object.” Each model or data view is represented by locations of scattering centers, which are image peaks. These scattering centers correspond to eight-neighbor peaks. The features of each object are the 30 peaks that are strongest in magnitude of radar returns. Recognition involves comparing peaks extracted from data and model views assuming that the space of applicable transformations, T , is discrete 2D translations in the image plane [5, 10]. 1.7.2 Model and Test Sets Our model set consists of three objects, which are T72, BMP2, and BTR70. The numbers of views for these objects are 231, 233, and 233, respectively, and so the total number of views is 697. All of these views are obtained at depression angle 17◦ . Figure 1.13 shows a representative sample of a single view for each object, along with associated scattering centers. The test data consists of seven sets, T D1 through T D7 . They are classified into two groups depending on whether the data distortion in the set is synthetic or real. 1. Synthetic-Distortion Group: This group consists of sets T D1 through T D5 . It is obtained by synthetically distorting the model set, following the distortion process outlined in Section 1.4. We have added the constraint that no features are eight-neighbors, in order to simulate the process of peak extraction. The distortion in each set is characterized by specific consistency and clutter regions, and a number of occlusion/clutter (O/C) values. Notice that the numbers of occluded and clutter features are always the same in our case, since we are considering a fixed number of features
Chapter 1 Predicting Performance of Object Recognition
(a) T72
(b) BMP2
25
(c) BTR70
Figure 1.13. Examples of SAR images and associated scattering centers at depression angle 17◦ , and azimuth angle 132◦ .
for both model and data views. The test subset corresponding to a specific O/C value consists of four randomly generated distorted instances for each model view. Accordingly, the size of each test subset is 4 | MD |, and the total size of the test set is 4noc | MD |, where noc is the number of O/C values considered in the set. The upper section of Table 1.2 describes the five synthetically distorted test sets. Note that the fourth column in this table consists of two subcolumns: the first one describes the basic shape of the clutter region, while the second describes the scale of that shape. In particular, the clutter region is obtained by scaling the basic clutter region by some factor. Furthermore, notice that the consistency region Ru (·) for test set T D1 is the zero-neighbor region, which implies the absence of any positional uncertainty. 2. Real-Distortion Group: This group consists of two sets, T D6 and T D7 . These sets correspond to variants of the model set involving real distortion. The first set, T D6 , is obtained by changing the configurations of the model objects [5]. Examples of changing the object configuration are using different flash lights, changing numbers of barrels, etc. The second set, T D7 , is obtained by changing the depression angle from 17◦ to 15◦ . Note that due to the nature of the SAR imaging process, such a small angle change can result in a significant variation in the object signature [28]. For both sets, the distortion parameters are determined as follows. The
26
Michael Boshra and Bir Bhanu Table 1.2. Description of the test sets used in the experiments.
Set T D1 T D2 T D3 T D4 T D5 T D6 T D7
Distortion Consistency Region synthetic 0-neighbor synthetic 4-neighbor synthetic 8-neighbor synthetic 4-neighbor synthetic 4-neighbor ∆ config. 4-neighbor ∆ dep. angle 4-neighbor
Clutter Region Shape Factor convex hull 1 convex hull 1 convex hull 1 convex hull 2 convex hull 3 convex hull 1 convex hull 1
Occluded/ Clutter Features 18, 19, · · · , 27 9, 10, · · · , 20 0, 1, · · · , 15 9, 10, · · · , 24 9, 10, · · · , 24 estimated estimated
Size 4 × 697 × 10 4 × 697 × 12 4 × 697 × 16 4 × 697 × 16 4 × 697 × 16 464 581
consistency and clutter regions, Ru (·) and Rc , are empirically chosen to be four-neighbor region and convex hull of view features, respectively. The O/C value is estimated for each test view through finding the best matching model view within a difference of ±3◦ azimuth angles, and counting the number of unmatched features. In the case of the absence of a model view within ±3◦ azimuth angles, the test view is matched with the available model one that is closest in azimuth. The lower section of Table 1.2 summarizes the two real sets. Note that our models are defined from real data that may in fact contain some distortions. Thus, our notion of occlusion and clutter is relative, not absolute. That is, a spurious feature in the model view that does not match any feature in the test view is considered as a true model feature that is occluded. Furthermore, a true feature in the test view that is missing in the model view is considered as a clutter feature. Obviously, recognition performance would be improved through learning the true model features, or by giving a variety of weights to the model features depending on the probability that they correspond to true features. This topic is beyond the scope of this work. 1.7.3 Results Our selected performance measure is PCR as a function of data distortion, in particular, the occlusion/clutter rate (O/C). As mentioned, the consistency and clutter regions are assumed to be fixed for each test set. The empirical performance is determined using an object recognition system, which uses the vote-based criterion defined in (1.1). The recognition system examines all the relevant translations between a given test view, and each model one. The translations examined are defined by the bounding box on the translations that lead to at least a single match between a test feature and a model one. Accordingly, the performance determined by the recognition system is optimal, assuming the given vote-based criterion. The predicted bounds are obtained as described in Section 1.6 with two minor modifications. The first one involves the clutter region. Notice that in our experiments, the exact shape of the
Chapter 1 Predicting Performance of Object Recognition
27
clutter region is not fixed, but differs from one view to another. Accordingly, AREA(Rc ) is replaced by the average of clutter-region areas for all model views. Recall that AREA(Rc ) is used when calculating the vote PDF of an erroneous hypothesis (refer to Section 1.6.2). The second modification involves the conditional PDF of clutter votes, defined in (1.7). Notice that in our recognition task, image peaks cannot be eight-neighbors. This fact needs to be considered in the estimation of the clutter vote PDF, in order to obtain more accurate performance bounds. An approximate method for estimating such a PDF is presented in the Appendix. We first analyze the results involving the synthetic-distortion test group. Figures 1.14 and 1.15 show actual and predicted performance plots for the test sets T D1 through T D5 . From these figures, we observe the following: •
In all cases, our method consistently succeeds in predicting reasonably tight bounds on actual performance. • The predicted lower bounds consistently predict the actual breakpoint in performance with high accuracy. In all cases, the breakpoint in the lowerbound plot either coincides with the actual one, or occurs very slightly before it. • The prediction method confirms the intuitive observation that performance degrades when the size of the consistency region increases. This can be observed by comparing Figures 1.14(a)–(c). The method also confirms the intuitive observation that performance improves when the density of clutter features decreases, which takes place when the size of the clutter region increases. Again, this can be observed by comparing Figures 1.14(b), 1.15(a), and 1.15(b). • The predicted lower bound is consistently very close to the actual PCR plot along the knee section, defined as the section between the first two breakpoints in the plot. In all cases, the difference between the lower bound and corresponding PCR in the knee section is less than 3%. Beyond the knee section, the lower bound diverges. This observation can be explained as follows. Let Hi (D) ⊂ Hi be set of erroneous hypotheses that simultaneously beat true hypothesis at distortion level D, where D = O = C. At relatively low levels of distortion, the probability that more than one erroneous hypotheses will simultaneously beat the true one is negligible compared to that of only a single erroneous hypothesis 5 . Accordingly, in case of recognition failure at low distortion levels, Hi (D) has only a single element in most cases. This makes the lower bound defined in (1.10) an estimate of the actual performance. However, as distortion increases, the above assertion ceases to be valid, which makes the lower bound in (1.10) a strict one. • The predicted upper bound becomes less tight beyond the breakpoint of performance, but then becomes very tight towards the end of the plot. This can be explained as follows. It is easy to see that the size of Hi (D) is 5
This is assuming we do not have identical or very similar model objects.
28
Michael Boshra and Bir Bhanu 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
70
80
90
100
70
80
90
100
70
80
90
100
Distortion Rate
(a) 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
Distortion Rate
(b) 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
Distortion Rate
(c) Figure 1.14. Actual and predicted PCR plots for synthetic-distortion sets involving different consistency regions: (a) T D1 , (b) T D2 , (c) T D3 .
Chapter 1 Predicting Performance of Object Recognition
29
100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
30
20
40
50
60
70
80
90
100
70
80
90
100
Distortion Rate
(a) 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
Distortion Rate
(b) Figure 1.15. Actual and predicted PCR plots for synthetic-distortion sets involving different clutter regions: (a) T D4 , (b) T D5 .
proportional to D. At low distortion levels, the size of Hi (D) is small, and the probability that it has no peak hypothesis is generally high. Accordingly, the upper bound defined in (1.15) becomes a less tight one (recall that this upper bound represents the probability that no peak hypothesis beats the true one). Note that we can quantitatively measure the tightness of the upper bound using the ratio F =
1 − predicted upper bound , 1 − actual PCR
where F ∈ [0, 1]. As distortion increases, the size of Hi (D) increases as well, and the probability it has no peak hypothesis decreases. This increases the tightness of the upper bound, until it eventually coincides with the actual PCR, at which case F becomes equal to one.
30
Michael Boshra and Bir Bhanu
We now turn to analyzing the real-distortion group. Actual and predicted PCR plots for real-distortion test sets T D6 and T D7 are shown in Figures 1.16(a) and 1.16(b), respectively. These figures show that our prediction method succeeds in predicting reasonably accurate bounds on actual performance. For example, the breakpoint in performance is predicted accurately in both cases. However, we also observe that the predicted bounds are over-optimistic in the knee section of the plot. The reason is obviously the existence of some differences between the actual distortion models and the assumed uniform ones. One of the important differences is that the assumed uniform occlusion model does not account for the spatial correlation among occluded/unoccluded features. 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
70
80
90
100
70
80
90
100
Distortion Rate
(a) 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
Distortion Rate
(b) Figure 1.16. Actual and predicted PCR plots for real-distortion group: (a) T D6 , (b) T D7 .
Chapter 1 Predicting Performance of Object Recognition
31
A goal of our research is to develop statistical techniques for learning uncertainty, occlusion, and clutter models. The topic of learning distortion models has received some attention in the literature, particularly in learning distributions of feature positional uncertainty [29]. We present here an initial method for learning the statistical model of occlusion, for the purpose of performance prediction. A possible direct approach is to learn the spatially correlating occlusion model from training data (e.g., using Markov random fields [26]), and then use it to determine the prediction bounds. In particular, the occlusion model can be used to determine the PDF of the number of unoccluded similar features (PDF of No ), which is subsequently used to determine the PDF of votes for an erroneous hypothesis (see Section 1.6.2). The difficulty with this approach is that it involves not only a spatially correlating occlusion model, but also a spatially correlating similarity model, one that considers the spatial correlation among similar/dissimilar feature pairs. Fortunately, in the context of our problem, there is a significantly simpler approach: Instead of learning the spatially correlating occlusion and similarity models, and then estimating their combined effect on the PDF of No , we directly learn the PDF of No . The learning process can be outlined as follows: 1. We use half of the test set for the learning process (odd-numbered views). i , we search for the best matching model 2. For each selected test view, M view, Mi , as described earlier when estimating the distortion rate for a test view (see Section 1.7.2). From this match, we eliminate unmatched features of Mi . The resulting view, Mi , corresponds to model view Mi after occlusion, but without either uncertainty or clutter. 3. For each erroneous hypothesis corresponding to Mi , Mjτi ∈ Hi , we calculate the number of similar feature pairs with each of Mi and Mi , denoted ¯ τi , respectively. by Njτi and N j 4. For each tuple (| Mi |, O =| Mi | − | Mi |, Njτi ), we histogram corre¯ τi to estimate the conditional PDF of No . sponding values of N j We have replaced the PDF of No defined in (1.6) by the learned PDF. The resulting new predicted bounds are shown in Figures 1.17(a) and 1.17(b) for test sets T D6 and T D7 , respectively. Comparing these figures to the corresponding ones in Figure 1.16, we observe that the predicted bounds obtained using the learned PDF of No are considerably more accurate than the ones obtained by assuming the uniform distortion models. In particular, the extent of over-optimism between actual and predicted plots almost disappears for set T D6 , and decreases by about 65% for set T D7 . This is assuming that the extent of over-optimism is measured by the area between the actual plot and the predicted lower-bound plot at the knee section. More accurate prediction can be obtained through comprehensive learning of the PDFs of all the distortion parameters involved.
32
Michael Boshra and Bir Bhanu 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
70
80
90
100
70
80
90
100
Distortion Rate
(a) 100
90
80
PCR × 100
70
60
50
40
30
20
Predicted Actual
10
0
0
10
20
30
40
50
60
Distortion Rate
(b) Figure 1.17. Actual and predicted PCR plots for real-distortion group using learned PDF of No : (a) T D6 , (b) T D7 .
1.8 Conclusions Object recognition performance is typically evaluated empirically. A major limitation of empirical evaluation is that it does not give us an insight into the recognition process. Such an insight is fundamental for transforming the field of object recognition from an art to a science. Most efforts for formal analysis of object recognition performance focus on the problem of object/clutter discrimination. This work extends those efforts by also considering the prob-
Chapter 1 Predicting Performance of Object Recognition
33
lem of object/object discrimination. An integrated approach is presented for predicting lower and upper bounds on recognition performance. Such an approach simultaneously considers data distortion factors such as uncertainty, occlusion, and clutter, in addition to model similarity. This is in contrast to the other few object/object discrimination approaches, which consider only a subset of these factors. The method is validated in the context of a recognition task involving real SAR data, and point features for matching. Validation is done by comparing predicted lower and upper PCR bounds with actual PCR plots determined experimentally. The selected test sets involve both synthetic and real distortion. In all cases, the results demonstrate that the prediction method consistently succeeds in predicting reasonably tight bounds on actual performance. The predicted bounds, however, are slightly over-optimistic in the case of test sets involving real distortion. More accurate bounds can be obtained through comprehensive learning of all the distortion models involved in the recognition process. As an initial step towards achieving this goal, we have presented a method for learning the statistical distribution of the parameter that encodes combined effect of occlusion and similarity on performance. The impact of using such a learned distribution on the accuracy of the predicted bounds has been demonstrated. To the best of our knowledge, this work is the first that validates a prediction method for object recognition using real data. We finally discuss possible ways to extend the work presented here. The recognition task considered in this work involves discretized 2D point features for both data and model objects, and a 2D translation space. The proposed method, however, can be extended in a number of ways. We explore some of them below: • Consideration of more general data/model transformations (e.g., rigid, affine) is straightforward. It would mainly involve extending the set of erroneous hypotheses for a model object, to reflect the extra degrees of freedom (see Sect. 1.6.3). • Replacement of discretized feature sets by non discretized sets is a more involved process. This is because, in such a case, the erroneous-hypothesis set for a model object would be of infinite size. One possible approach is sampling the erroneous hypotheses in the transformation space, and then “dilating” the consistency regions associated with model features to account for non-sampled hypotheses (e.g., [21]). We note that consistencyregion dilation would be needed only when computing lower bounds on performance. • It is also possible to incorporate feature attributes [5], and/or increase dimensionality of feature locations. Both cases would mainly involve increasing the dimensionality of consistency and clutter regions to be equal to the sum of location dimensionality and number of attributes. Otherwise, the method remains basically the same.
34
Michael Boshra and Bir Bhanu
• Another interesting possibility is to describe the positional uncertainty by a Gaussian distribution, instead of a uniform one. This would involve replacing the uniform vote-based criterion by a weighted one, which depends on the distance between corresponding data and model features, as well as the standard deviation of the Gaussian distribution [18]. This would require making modifications to both the measure of object/hypothesis similarity, and the method used to calculate the vote distributions for both true and erroneous hypotheses. The prediction method, however, remains conceptually the same. • The performance theory presented here can be integrated with adaptive algorithms to optimize object recognition performance in practical scenarios (e.g., with respect to a desired trade-off between probability of false alarm and probability of correct recognition [30]). In conclusion, the work presented here is believed to lay a theoretical/conceptual foundation that is important for developing a general theory for performance prediction of object recognition.
Appendix We present an approximate method for estimating the conditional PDF of clutter votes Vc , considering the constraint that no two features can be eight neighbors (imposed by the feature extraction process). We can represent the feature-adjacency constraint by a separation region, Rs (·). In our case, Rs (·) is a 3 × 3 window centered at the feature’s location. The method can be outlined as follows: •
It can be seen that the feature-adjacency constraint reduces effective clutter and clutter-vote regions, Rc and RV c , to smaller ones, Rc and RVc , respectively (see Figure 1.18). We can express the area of Rc as AREA(Rc ) ≈ AREA(Rc ) − AREA(Rs (·))(| Mi | −O). For clutter-vote region RVc , let us first consider features in Mi that are similar to those in Mjτi , but are lying outside the overlapping regions (e.g., upper right feature of Mi in Figure 1.18). Generally, the separation region corresponding to one of these features partially covers the consistency region corresponding to the similar feature in Mτj i . We can approximate the area of the covered section, denoted by Ed , as follows. (1) Both the consistency and separation regions are modeled as squares, having the same areas as their original respective regions. (2) The relationship between square consistency regions corresponding to similar feature pairs is modeled by movement of one square with respect to the other parallel to one of the edges, until the area of the overlapping region is the same as the original one (which corresponds to feature/feature similarity). (3) Assuming the
Chapter 1 Predicting Performance of Object Recognition
35
square modeling of consistency regions, we can easily estimate expected location of a feature of Mi , assuming that it is outside the overlapping region. Given the expected location, we can easily estimate the area of the covered section, Ed . We can then estimate the area of RVc as AREA(RVc ) = AREA(RV c ) − (no − vs )Ed .
Figure 1.18. An illustration of clutter vote region RVc , in the presence of separation regions, shown as squares.
• The above discussion, as well as that in Section 1.6.2, implies that features of Mτj i that can result in votes for Mτj i are effectively associated with consistency regions of various sizes. In order to simplify the calculations, we assume that we have an “effective” number of these features, ne , such that each feature is associated with a full consistency region. It can be easily shown that ne = AREA(RVc )/AREA(Ru (·)) + 0.5. In such a case, the effective area of the consistency region, Eu , is AREA(RVc )/ne , which is approximately the same as AREA(Ru (·)). • Next, we need to estimate the average effective area of a separation region, Es . This area is simply the ratio between the area corresponding to the union of the separation regions of the C clutter features, to C. If the clutter features are spread apart from each other, then Es is close to AREA(Rs (·)). However, if their density is high, which is typical in our case, then there can be significant overlap between their separation regions, thus making Es considerably smaller than AREA(Rs (·)). In our work, we have Es ≈ min(AREA(Rs (·)), AREA(Rc )/C). •
(1.18)
Now, we are finally at a position to approximate the conditional PDF of Vc : PVc (vc ; no , vs ) ≈ K(C, vc ) × L(AREA(RVc ), Eu , vc )L(AREA(Rc − RVc ) − vc (max(0, Es − Eu )), Es , C − vc ) L(AREA(Rc ), Es , C)
36
Michael Boshra and Bir Bhanu
where L(X, Y, n) =
n−1 i=0
(X − iY ).
Acknowledgments This work was supported in part by AFOSR grant F49620-02-1-0315. The contents and information do not reflect positions or policies of the U.S. Government.
References [1] Arman, F., Aggarwal, J.: Model-based object recognition in dense-range images: A review. ACM Comput. Surveys 25 (1993) 5–43 [2] Besl, P., Jain, R.: Three-dimensional object recognition. ACM Comput. Surveys 17 (1985) 75–145 [3] Chin, R., Dyer, C.: Model-based recognition in robot vision. ACM Comput. Surveys 18 (1986) 67–108 [4] Suetens, P., Fua, P., Hanson, A.: Computational strategies for object recognition. ACM Comput. Surveys 24 (1992) 5–61 [5] Bhanu, B., III, G.J.: Recognizing target variants and articulations in synthetic aperture radar images. Optical Engineering 39 (2000) 712–723 [6] Boshra, M., Zhang, H.: A constraint-satisfaction approach for 3-D object recognition by integrating 2-D and 3-D data. Comput. Vision Image Understand. 73 (1999) 200–214 [7] Wells, W.: Statistical approaches to feature-based object recognition. Int. J. of Computer Vision 21 (1997) 63–98 [8] Alter, T., Grimson, W.: Verifying model-based alignments in the presence of uncertainty. In: Proc. IEEE Conf. Comput. Vision and Patt. Recogn., San Juan, Puerto Rico (1997) 344–349 [9] Boykov, Y., Huttenlocher, D.: A new Bayesian framework for object recognition. In: Proc. IEEE Conf. Comput. Vision and Patt. Recogn. Volume 2., Fort Collins, Colorado (1999) 517–523 [10] III, G.J., Bhanu, B.: Recognition of articulated and occluded objects. IEEE Transactions on Pattern Anal. and Mach. Intell. 21 (1999) 603– 613 [11] Dhome, M., Richetin, M., Lapreste, J., Rives, G.: Determination of the attitude of 3-D objects from a single perspective view. IEEE Transactions on Pattern Anal. and Mach. Intell. 11 (1989) 1265–1278 [12] Huttenlocher, D., Ullman, S.: Recognizing solid objects by alignment with an image. Int. J. of Computer Vision 5 (1990) 195–212 [13] Dhome, M., Kasvand, T.: Polyhedra recognition by hypothesis accumulation. IEEE Trans. on Pattern Anal. and Mach. Intell. 9 (1987) 429–439 [14] Stockman, G.: Object recognition and localization via pose clustering. Comput. Vision Graphics Image Process. 40 (1987) 361–387
Chapter 1 Predicting Performance of Object Recognition
37
[15] Faugeras, O., Hebert, M.: The representation, recognition and locating of 3-D objects. Int. J. of Robotic Res. 5 (1986) 27–52 [16] Grimson, W., Lozano-Perez, T.: Localizing overlapping parts by searching the interpretation tree. IEEE Trans. on Pattern Anal. and Mach. Intell. 9 (1987) 469–482 [17] Grimson, W., Huttenlocher, D.: On the verification of hypothesized matches in model-based recognition. IEEE Transactions on Pattern Anal. and Mach. Intell. 13 (1991) 1201–1213 [18] Sarachik, K.: The effect of Gaussian error in object recognition. IEEE Transactions on Pattern Anal. and Mach. Intell. 19 (1997) 289–301 [19] Lindenbaum, M.: An integrated model for evaluating the amount of data required for reliable recognition. IEEE Transactions on Pattern Anal. and Mach. Intell. 19 (1997) 1251–1264 [20] Irving, W., Washburn, R., Grimson, W.: Bounding performance of peakbased target detectors. In: Proc. SPIE Conference on Algorithms for Synthetic Aperture Radar Imagery IV. Volume 3070. (1997) 245–257 [21] Lindenbaum, M.: Bounds on shape recognition performance. IEEE Transactions on Pattern Anal. and Mach. Intell. 17 (1995) 665–680 [22] Grenander, U., Miller, M., Srivastava, A.: Hilbert-Schmidt lower bounds for estimators on matrix lie groups for ATR. IEEE Transactions on Pattern Anal. and Mach. Intell. 20 (1998) 790–802 [23] Boshra, M., Bhanu, B.: Predicting performance of object recognition. IEEE Trans. on Pattern Anal. and Mach. Intell. 22 (2000) 956–969 [24] Boshra, M., Bhanu, B.: Validation of SAR ATR performance prediction using learned distortion models. In: Proc. SPIE Conference on Algorithms for Synthetic Aperture Radar Imagery VII. Volume 4053., Orlando, Florida (2000) 558–566 [25] Boshra, M., Bhanu, B.: Predicting an upper bound on SAR ATR performance. IEEE Trans. on Aerospace and Electronic Syst. 37 (2001) 876–888 [26] Li, S.: Markov Random Field Modeling in Image Analysis. SpringerVerlag, New York (2001) [27] Ying, Z., Castanon, D.: Feature-based object recognition using statistical occlusion models with one-to-one correspondence. In: Proc. Int. Conf. on Comput. Vision. Volume 1., Vancouver, Canada (2001) 621–627 [28] Ross, T., Worrell, S., Velten, V., Mossing, J., Bryant, M.: Standard SAR ATR evaluation experiments using the MSTAR public release data set. In: Proc. SPIE Conference on Algorithms for Synthetic Aperture Radar Imagery V. Volume 3370., Orlando, Florida (1998) 566–573 [29] Pope, A., Lowe, D.: Probabilistic models of appearance for 3-D object recognition. Int. J. of Computer Vision 40 (2000) 149–167 [30] Bhanu, B., Lin, Y., Jones, G., Peng, J.: Adaptive target recognition. Machine Vision and Applications 11 (2000) 289–299
Chapter 2 Methods for Improving the Performance of an SAR Recognition System Bir Bhanu1 and Grinnell Jones III2 1 2
Center for Research in Intelligent Systems, University of California, Riverside, California 92521,
[email protected] Center for Research in Intelligent Systems, University of California, Riverside, California 92521,
[email protected]
Summary. The focus of this chapter is on methods for improving the performance of a model-based system for recognizing vehicles in synthetic aperture radar (SAR) imagery under the extended operating conditions of object articulation, occlusion, and configuration variants. The fundamental approach uses recognition models based on quasi-invariant local features, radar scattering center locations, and magnitudes. Three basic extensions to this approach are discussed: (1) incorporation of additional features; (2) exploitation of a priori knowledge of object similarity represented and stored in the model-base; and (3) integration of multiple recognizers at different look angles. Extensive experimental recognition results are presented in terms of receiver operating characteristic (ROC) curves to show the effects of these extensions on SAR recognition performance for real vehicle targets with articulation, configuration variants, and occlusion.
2.1 Introduction In this chapter we are concerned with methods for improving the recognition of vehicles in Synthetic Aperture Radar (SAR) imagery. The recognition systems start with real SAR chips (at one foot resolution) of actual military vehicles from the MSTAR public data [1] and end with the identification of a specific vehicle type (e.g., a T72 tank). Several major challenges for identifying the vehicles are the presence of significant external configuration variants (fuel barrels, searchlights, etc.), articulated configurations (such as a tank with its turret rotated), and partial occlusion. The detection theory [2, 3], pattern recognition [4, 5, 6], and neural network [7] approaches to SAR recognition all tend to use global features that are optimized for standard, non-articulated, non-occluded configurations. Approaches that rely on global features are not appropriate for recognizing occluded or articulated objects because occlusion and articulation change global features like the object outline and major axis [8]. Our previous work [9, 10, 11, 12], relied on local features to successfully recognize articulated and highly occluded objects. SAR recognition results for 39
40
Bir Bhanu and Grinnell Jones III
our basic approach are compared (in [9]) to other different approaches using real SAR images from the MSTAR public data. In our research on SAR automatic target recognition (ATR), we initially started using invariant locations of SAR scattering centers as features [12] and later developed our basic recognition approach based on using quasi-invariant locations and magnitudes of the scattering centers [9, 10, 11]. This followed the traditional approach to improving recognition performance by finding additional features that can help to distinguish between the objects. This is the first method of improvement discussed in this chapter. The second method of exploiting model similarity was inspired by the research on predicting the performance of recognition systems by Boshra and Bhanu [13] that introduced the idea that recognition performance depends on the distortion in the test data and the inherent similarity of the object models. Related to the third method for improving recognition performance, we had previously shown that a significant number of SAR scattering center locations do not typically persist over a few degrees of rotation [10]. However, this had been viewed as a problem for modeling, rather than a potential opportunity for independent observations at different look angles. In this chapter, we integrate results from multiple look angles to improve recognition performance. This chapter discusses three basic approaches to improve the performance of an SAR recognition system: 1. Incorporation of additional features. 2. Exploitation of a priori knowledge of object similarity. 3. Integration of multiple recognizers at different look angles. The next section discusses SAR target characteristics; Section 2.3 gives a description of the basic SAR recognition system; Section 2.4 introduces the additional feature of peak shape factor and shows performance improvements for additional features; Section 2.5 describes techniques to measure and utilize model similarity to improve recognition performance; Section 2.6 demonstrates the independence of multiple look angle SAR recognition and the results for performance improvements; and the final Section 2.7 provides the conclusions of the chapter.
2.2 SAR Target Characteristics The typical detailed edge and straight line features of man-made objects in the visual world do not have good counterparts in SAR images for subcomponents of vehicle-sized objects at one-foot resolution. However, there is a wealth of peaks corresponding to scattering centers. The relative locations of SAR scattering centers, determined from local peaks in the radar return, are related to the aspect and physical geometry of the object, independent of translation and serve as distinguishing features. Target regions-of-interest (ROI) are found in the MSTAR SAR chips by reducing speckle noise using
Chapter 2 Improving an SAR Recognition System
41
the Crimmins algorithm (see [14]), thresholding at the mean plus two standard deviations, dilating to fill small gaps among regions, eroding to have one large ROI and little regions, discarding the small regions with a size filter and dilating to expand the extracted ROI. The scattering centers are extracted from the SAR magnitude data (within the boundary contour of the ROI) by finding local eight-neighbor maxima. The parameters used in extracting ROIs are held constant for all the results reported. Objects from the MSTAR public data used in this research include: BMP2 armored personnel carriers (APCs), a BTR70 APC, T72 tanks, a ZSU23/4 anti-aircraft gun, and a BRDM2 APC. Photo images of the MSTAR articulated objects used in this paper, T72 tank serial number (#) a64 and ZSU 23/4 anti-aircraft gun #d08, are shown in Figures 2.1 and 2.2. Example SAR images and the ROI, with the locations of the scattering centers superimposed are shown in Figure 2.3 for baseline and articulated versions of the T72 and ZSU (at 30◦ radar depression angle, 66◦ target azimuth).
(a) turret straight.
(b) turret articulated. Figure 2.1. T72 tank #a64.
(a) turret straight.
(b) turret articulated.
Figure 2.2. ZSU 23/4 anti-aircraft gun #d08.
42
Bir Bhanu and Grinnell Jones III
(a) T72: image.
(b) ROI.
(c) articulated image. (d) articulated ROI.
(a) ZSU: image.
(b) ROI.
(c) articulated image. (d) articulated ROI.
Figure 2.3. MSTAR SAR images and ROIs (with peaks) for T72 tank #a64 and ZSU 23/4 #d08 at 66◦ azimuth.
2.2.1 Azimuthal Variance of Scatterer Locations The typical rigid body rotational transformations for viewing objects in the visual world do not apply much for the specular radar reflections of SAR images. This is because a significant number of features do not typically persist over a few degrees of rotation. Since the radar depression angle is generally known, the significant unknown target rotation is (360◦ ) in azimuth. Azimuth persistence or invariance can be expressed in terms of the percentage of scattering center locations that are unchanged over a certain span of azimuth angles. It can be measured (for some base azimuth θo ) by rotating the pixel locations of the scattering centers from an image at azimuth θo by an angle ∆θ and comparing the resulting range and cross-range locations with the scatterer locations from an image of the same object at azimuth θ0 + ∆θ. More precisely, because the images are in the radar slant plane, we actually project from the slant plane to the ground plane, rotate in the ground plane, and project back to the slant plane. Since the objects in the chips are not registered, we calculate the azimuth invariance as the maximum number of corresponding scattering centers (whose locations match within a given tolerance) for the optimum integer pixel translation. This method of registration by finding the translation that yields the maximum number of correspondences has the limitation that for very small or no actual invariance it may find some false correspondences and report a slightly higher invariance than in fact exists. To determine scattering center locations that persist over a span of angles, there is an additional constraint that for a matching scattering center
Chapter 2 Improving an SAR Recognition System
43
to “persist” at the kth span ∆θk , it must have been a persistent scattering center at all smaller spans ∆θj , where 0 ≤ j < k. Averaging the results of these persistent scattering center locations over 360 base azimuths gives the mean azimuth invariance of the object. Figure 2.4 shows an example of the mean azimuth invariance (for the 40 strongest scatterers) as a function of azimuth angle span using T72 tank #132, with various definitions of persistence. In the “exact match” cases the center of the rotated scatterer pixel from the image at θo azimuth is within the pixel boundaries of a corresponding scatterer in the image at θ0 + ∆θ. In the “within 1 pixel” cases, the scatterer location is allowed to move into one of the 8 adjacent pixel locations. Note that for a 1◦ azimuth span, while only 20% of the scatterer locations are invariant for an “exact match,” 65% of the scatterer locations are invariant “within 1 pixel.” The cases labeled “persists” in Figure 2.4 enforce the constraint that the scatterer exist for the entire span of angles and very few scatterers continuously persist for even 5◦ . In the upper two cases (not labeled “persists”) scintillation is allowed and the location invariance declines slowly with azimuth span. The “within 1 pixel” results (that allow scintillation) are consistent with the one-foot ISAR results of Dudgeon et al. [15], whose definition of persistence allowed scintillation. Because of the higher scatterer location invariance with 1◦ azimuth span, in our research we use azimuth models at 1◦ increments for each target, in contrast to others who have used 5◦ [16], 10◦ [17], and 12 models covering specific azimuth ranges [6]. 100 within 1 pixel exact match within 1 pixel, persists exact match, persists
90
Invariance (percent)
80 70 60 50 40 30 20 10 0 0
1
2
3 4 5 6 7 Azimuth span (degrees)
8
9
10
Figure 2.4. Scatterer location persistence, T72 #132.
The fact that the SAR scatterer locations do not persist over a span of even a few degrees, demonstrated in Figure 2.4, strongly indicates that observations at different azimuth angles are independent. Thus, what had previously been viewed as a “problem” for modeling, now presents a significant opportunity
44
Bir Bhanu and Grinnell Jones III
to improve recognition performance by integrating the results of SAR observations at multiple look angles. This is the basis for the approach which will be discussed later in Section 2.6. 2.2.2 Scatterer Location and Magnitude Invariance Many of the scatterer locations and magnitudes are invariant to target conditions such as articulation or configuration variants. Because the object and ROI are not registered, we express the scattering center location invariance with respect to articulation or configuration differences as the maximum number of corresponding scattering centers (whose locations match within a stated tolerance) for the optimum integer pixel translation. Figure 2.5 shows the location invariance of the strongest 40 scattering centers with articulation for T72 tank #a64 and also for ZSU 23/4 anti-aircraft gun #d08 (at a 30◦ depression angle) as a function of the hull azimuth. The combined average invariance for both articulated vehicles is 16.5% for an exact match of scattering centers and 56.5% for a location match within one pixel (3x3 neighborhood) tolerance. (Note that not all 360 degrees are present in the MSTAR data, so the missing azimuth angles are ignored in this research.) Similarly, Figure 2.6 shows the percent of the strongest 40 scattering center locations that are invariant for two example configuration variants, T72 #812 vs. #132 and BMP2 vehicle #9563 vs. #C21, at a 15◦ depression angle. (While the MSTAR data has 11 configurations of the T72, it only has three configurations of the BMP2, so to avoid biasing the experiments toward the T72, we use only three of the T72 configurations.) The combined average invariance for configuration variants of the T72 (#812 and #s7 vs. #132) and BMP2 (#9563 and #9566 vs. #c21) is 15.3% for an exact match of scatterer locations and 57.15% for a location match within one pixel. Since less than 20% of the SAR scattering center locations exactly match for object articulation and configuration variants, while over 50% of these locations are quasi-invariant within a 3∗3 pixel tolerance, in our research we accommodate this 3∗3 tolerance for scattering center locations in the recognition system. Because of the very large dynamic range for scatterer magnitudes we use a scaled scatterer amplitude (S), expressed as a radar cross-section in square meters, given by S = 100 + 10 log10 (i2 + q 2 ), where i and q are the components of the complex radar return, and we define a percent amplitude change (Ajk ) as: Ajk = 100(Sj − Sk )/Sj . Note that this form allows a larger variation for the stronger signal returns. Figure 2.7 shows the probability mass functions (PMFs) for percent amplitude change for the strongest 40 articulated vs. nonarticulated scattering centers of T72 tank #a64 and ZSU 23/4 gun #d08. Curves are shown both for the cases where the scattering center locations correspond within one pixel tolerance and for all the combinations of scatterers whose locations do not match. Similarly, Figure 2.8 shows the PMFs for percent amplitude change for the strongest 40 scattering centers with configuration variants, T72 #812 vs. #132 and BMP2 #9563 vs. #C21,
Chapter 2 Improving an SAR Recognition System T72 tank #a64 articulated vs. straight
ZSU 23/4 gun #d08 articulated vs. straight
100
100 within 1 pixel exact match
within 1 pixel exact match 80 Invariance (percent)
80 Invariance (percent)
45
60
40
20
60
40
20
0
0 0
45
90
135 180 225 270 Hull azimuth (degrees)
315
360
0
45
(a)T72 tank.
90
135 180 225 270 Hull azimuth (degrees)
315
360
(b)ZSU 23/4.
Figure 2.5. Scatterer location invariance with articulation. T72 tanks #812 vs. #132
BMP2 vehicles #9563 vs. #C21
100
100 within 1 pixel exact match
within 1 pixel exact match 80 Invariance (percent)
Invariance (percent)
80
60
40
20
60
40
20
0
0 0
45
90
135 180 225 270 Hull azimuth (degrees)
(a)T72 tank.
315
360
0
45
90
135 180 225 270 Hull azimuth (degrees)
315
360
(b)BMP2.
Figure 2.6. Scatterer location invariance with configuration.
at a 15◦ depression angle. If we define scatterer magnitude invariance as the number of scatterers with corresponding locations whose magnitudes match (within a stated percent amplitude change tolerance), then the overall average scatterer magnitude invariance (within ± one pixel location and ± nine percent amplitude change) is 50.8 percent for the articulation cases and 51.7 percent for the configuration variant cases. 2.2.3 Target Occlusion There is no real SAR data with occluded objects available to the general public (limited data on vehicles in revetments [18] and partially hidden behind walls [19] has been reported to exist, but it has not yet been released for unrestricted use). In addition, there is no standard, accepted method for characterizing or simulating occluded targets. Typically occlusion occurs when a tank backs up into a tree line, for example, so that the back end is covered by trees and only the front portion of the tank is visible to the radar. Thus, the “bright target” becomes a much smaller sized object to the recognition
46
Bir Bhanu and Grinnell Jones III ZSU 23/4 gun #d08 articulated vs. straight
match within 1 pixel no match Probability mass function
Probability mass function
T72 tank #a64 articulated vs. straight 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 -50
-40
-30
-20 -10 0 10 20 Percent amplitude change
30
40
50
0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 -50
match within 1 pixel no match
-40
(a)T72 tank.
-30
-20 -10 0 10 20 Percent amplitude change
30
40
50
40
50
(b)ZSU 23/4.
Figure 2.7. Scatterer magnitude invariance with articulation. BMP2 vehicles #9563 vs. #c21
match within 1 pixel no match Probability mass function
Probability mass function
T72 tanks #812 vs. #132 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 -50
-40
-30
-20 -10 0 10 20 Percent amplitude change
(a)T72 tank.
30
40
50
0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 -50
match within 1 pixel no match
-40
-30
-20 -10 0 10 20 Percent amplitude change
30
(b)BMP2.
Figure 2.8. Scatterer magnitude invariance with configuration.
system. In addition, the tree tops can produce “bright” peaks that are similar to the strength of target peaks at many azimuths. The occluded test data in our research is simulated by starting with a given number of the strongest scattering centers and then removing the appropriate number of scattering centers encountered in order, starting in one of four perpendicular directions di (where d1 and d3 are the cross-range directions, along and opposite the flight path, respectively, and d2 and d4 are the up range and down range directions). Then the same number of scattering centers (with random magnitudes) are added back at random locations within the original bounding box of the chip. This keeps the number of scatterers constant and acts as a surrogate for some potential occluding object. Our approach, using simulated occlusion provides an enormous amount of data with varying known amounts of occlusion for carefully controlled experiments.
Chapter 2 Improving an SAR Recognition System
47
2.3 Basic SAR Recognition System The basic SAR recognition algorithm is an off-line model construction process and a similar on-line recognition process. The approach is designed for SAR and is specifically intended to accommodate recognition of occluded and articulated objects. Standard nonarticulated models of the objects are used to recognize these same objects in nonstandard, articulated, and occluded configurations. The models are a look-up table and the recognition process is an efficient search for positive evidence, using relative locations of the scattering centers in the test image to access the look-up table and generate votes for the appropriate object (and azimuth pose). Establishing an appropriate local coordinate reference frame is critical to reliably identifying objects (based on locations of features) in SAR images of articulated and occluded objects. These problems require the use of a local coordinate system; global coordinates and global constraints do not work, as illustrated in Figure 2.3, where the center of mass and the principal axes change with articulation. In an SAR image the radar range and cross-range directions are known and choosing any local reference point, such as a scattering center location, establishes a reference coordinate system. The relative distance and direction of the other scattering centers can be expressed in radar range and cross-range coordinates, and naturally tessellated into integer buckets that correspond to the radar range/cross-range bins. The scale is determined by the bin size, which is a function of the frequency of the radar (e.g., X-band for the one foot resolution used in this research). The recognition system takes advantage of this natural system for SAR, where a single basis point performs the translational transformation and fixes the coordinate system to a “local” origin. The model construction algorithm for the recognition system is outlined in Figure 2.9 (where the “origin” is the stronger of a pair of scatterers and a “point” is another weaker scatterer). Because of the specular radar reflections in SAR images, a significant number of features do not typically persist over a few degrees of rotation (as shown in Figure 2.4). Consequently, we model each object at 1◦ azimuth increments. The relative locations and magnitudes of the N strongest SAR scattering centers are used as characteristic features (where N , the number of scattering centers used, is a design parameter). Any local reference point, such as a scattering center location, could be chosen as a basis point (“origin”) to establish a reference coordinate system for building a model of an object at a specific azimuth angle pose. For ideal data, picking the location of the strongest scattering center as the origin is sufficient. However, for potentially corrupted data where any scattering center could be spurious or missing (due to the effects of noise, target articulation, occlusion, nonstandard target configurations, etc.), we use all N strongest scattering centers in turn as origins to ensure that a valid origin is obtained. Thus, to handle occlusion and articulation, the size of the look-up table models (and also the number of relative distances that are considered in the test image during
48
Bir Bhanu and Grinnell Jones III
recognition) are increased from N to N (N − 1)/2. Using a technique like geometric hashing [20], the models are constructed using the relative positions of the scattering centers in the range (R) and cross-range (C) directions as the initial indices to a look-up table of labels that give the associated target type, target pose, “origin” range and cross-range positions, and the magnitudes (S) of the two scatterers. Since the relative distances are not unique, there can be several of these labels (with different target, pose, etc., values) at each lookup table entry. 1. For each model Object do 2 2. For each model Azimuth do 3, 4, 5 3. Obtain the location (R, C) and magnitude (S) of the strongest N scatterers. 4. Order (R, C, S) triples by descending S. 5. For each origin O from 1 to N do 6 6. For each point P from O+1 to N do 7, 8 7. dR = RP − RO ; dC = CP − CO . 8. At look-up table location dR, dC append to list entry with: Object, Azimuth, RO , CO ,SO , SP . Figure 2.9. Model construction algorithm.
The recognition algorithm is outlined in Figure 2.10. The recognition process uses the relative locations of the N strongest scattering centers in the test image to access the look-up table and generate votes for the appropriate object, azimuth, range, and cross-range translation. (In contrast to many model-based approaches to recognition [21], we are not “searching” all the models.) Further comparison of each test data pair of scatterers with the model look-up table result(s) provides information on the magnitude changes (between the data and the model) for the two scatterers. Limits on allowable values for translations and magnitude changes are used as constraints to reduce the number of false matches. The number of scattering centers used and the various constraint limits are design parameters that are optimized, based on experiments, to produce the best recognition results. (Another approach to optimizing these tuning parameters, based on reinforcement learning is presented in [22].) Given the MSTAR targets are “centered” in the chips, a ±5 pixel limit on allowable translations is imposed for computational efficiency. The experimentally determined optimum limit on the allowable percent difference in the magnitudes of the data and model scattering centers was ±9%, which is consistent with the measured probability mass functions of scatterer magnitude invariance with target configuration variants and articulations (previously shown in Figures 2.8 and 2.7). To accommodate some uncertainty in the scattering center locations, the eight-neighbors of the nominal range and cross-range relative location are also probed in the look-up table, and the translation results are accumulated for a 3∗3 neighborhood in the translation subspace. A city-block weighted voting method reduces the
Chapter 2 Improving an SAR Recognition System
49
impact of the more common small relative distances. The recognition process is repeated with different scattering centers as basis points, providing multiple “looks” at the model database to handle spurious scatterers that arise due to articulation, occlusion, or configuration differences. The recognition algorithm actually makes a total of 9N (N − 1)/2 queries of the look-up table to accumulate evidence for the appropriate target type, azimuth angle, and translation. The models (labels with object, azimuth, etc.) associated with a specific look-up table entry are the “real” model and other models that happen by coincidence, to have a scatterer pair with the same (range, crossrange) relative distance. The constraints on magnitude differences filter out many of these false matches. In addition, while these collisions may occur at one relative location, the same random object-azimuth pair doesn’t keep showing up at other relative locations with appropriate scatterer magnitudes and mapping to a consistent 3∗3 neighborhood in translation space, while the “correct” object does. 1. Obtain from test image the location (R, C) and magnitude (S) of N strongest scatterers. 2. Order (R, C, S) triples by descending S. 3. For each origin O from 1 to N do 4 4. For each point P from O+1 to N do 5, 6 5. dR = RP − RO ; dC = CP − CO . 6. For DR from dR − 1 to dR + 1 do 7 7. For DC from dC − 1 to dC + 1 do 8, 9, 10 8. weighted vote = |DR| + |DC|. 9. Look up list of model entries at DR, DC. 10. For each model entry E in the list do 11 11. IF |tr = RO − RE | < translation limit and |tc = CO − CE | < translation limit and |1 − SEO /SO | < magnitude limit and |1 − SEP /SP | < magnitude limit THEN increment accumulator array [Object, Azimuth, tr, tc] by weighted vote. 12. Query accumulator array for each Object, Azimuth, tr and tc, summing the votes in a 3x3 neighborhood in translation subspace about tr, tc; record the maximum vote sum and the corresponding Object. 13. IF maximum vote sum > threshold THEN result is Object ELSE result is “unknown.” Figure 2.10. Recognition algorithm
The basic decision rule used in the recognition is to select the objectazimuth pair (and associated “best” translation) with the highest accumulated vote total. To handle identification with “unknown” objects, we introduce a criteria for the quality of the recognition result that the votes for the potential winning object exceed some minimum threshold vmin . By varying the
50
Bir Bhanu and Grinnell Jones III
decision rule threshold we obtain a form of receiver operating characteristic (ROC) curve with probability of correct identification, PCI = P {decide correct object|object is true}, vs. probability of false alarm, Pf = {decide any object|unknown is true}. We call the algorithm a 6D recognition algorithm since, in effect, we use the range and cross-range positions and the magnitudes of pairs of scattering centers. (When using 40 scatterers, this 6D algorithm takes an average of 2.5 seconds to process a test chip on a Sun Ultra2 without any optimizations.) More formally, a radar image of object c at azimuth pose a consists of N (or more) scatterers, each scatterer k with a magnitude Sk and range and crossrange locations Rk and Ck , which (for consistency) are ordered by decreasing magnitude such that Sk ≥ Sk+1 where k = 1, . . . , N . A model M of object c at azimuth a is given by M (c, a) = {V1 (c, a), V2 (c, a), . . . , VN (N −1)/2 (c, a)},
(2.1)
which is comprised of the set of all pairwise observations, Vi , Vi (c, a) = {f1 , f2 , . . . , f6 }i ,
(2.2)
where i = 1, 2, . . . , N (N − 1)/2, f1 = RP − RO , f2 = CP − CO , f3 = RO , f4 = CO , f5 = SO , f6 = SP , and with the individual scatterers in each pair labeled O and P so that SO ≥ SP for consistency (see Figure 2.11). magnitude S
f5 = S O
f6 = S P f3 = R O
RP range R
f4 = C O f2 = CP - CO CP
f 1 = RP - R O
cross-range C
Figure 2.11. Observation for a pair of scatterers O and P .
We define a match, H, as
Chapter 2 Improving an SAR Recognition System
H(Vi , Vj ) =
w if |(fb )i − (fb )j | ≤ δb , ∀ b = 1, . . . , 6, 0 otherwise
51
(2.3)
where the weight w = |(f1 )i |+|(f2 )i | and the match constraints are δ1 = δ2 = 0 pixels, δ3 = δ4 = 5 pixels and δ5 = δ6 = L percent. The recognition result, T , for some test image (with a test class, x, and test azimuth, y, to be determined) is a maximal match that is greater than a threshold, D, given by T =
[c, a] , if arg maxc,a,t
9
l=1
l n H (V (x, y), V (c, a)) > D, m t k n=1
N (N −1)/2 9 k=1
“unknown,” otherwise (2.4)
where Vm ∈ M (c, a) ∀m such that |(f1 )Vkn − (f1 )Vm | = 0 and |(f2 )Vkn − (f2 )Vm | = 0, and the subscript t applied to a match denotes that the match, Ht , is associated with the relative translation t(R, C) = (∆f3 , ∆f4 ) of the stronger scatterers in the two observations. Note that in this formulation the constraint on m avoids exhaustive search of all the models and can be implemented as a look-up table. The nine observations (denoted by the superscript n in Vkn ) are made to account for location uncertainty by taking the 3∗3 neighbors about the nominal values for the relative locations f1 and f2 of scatterer pair k in the test image. Similarly, the nine matches (denoted by the superscript l in Htl ) are computed at the 3∗3 neighbors located ±1 pixel about the resulting nominal value for translation, t(R, C), of the scatterers in the test image from the model.
2.4 Incorporation of Additional Features A traditional approach to improving recognition performance is to find additional features that distinguish between the objects. The basic SAR recognition system, the 6D system described in Section 2.3, successfully evolved from an earlier simpler 2D version, [12] that used only relative distances and the “exact match” scatterer locations. The 6D version adds: (1) the scatterer magnitudes as additional features; 2) the within-one-pixel quasi-invariance of scatterer locations; and (3) the consistent translation constraint. In addition to the scattering center locations and magnitude features, a “shape factor” can be used as a measure of the sharpness of the local peak in the radar return 8 associated with a scattering center. We define the shape factor F = Sk / i=1 Si , where Sk is the amplitude of the peak and the Si ’s are the amplitudes of the eight neighbors. Figure 2.12 shows the PMFs for percent shape factor change for the strongest 40 scattering centers of T72 #812 vs. #132 (at 15◦ depression angle). Curves are shown both for cases where the scattering center locations correspond within one pixel tolerance and for all the combinations of scatterers whose locations do not match. For the cases with locations that match within one pixel, the percent shape factor change
52
Bir Bhanu and Grinnell Jones III
Probability mass function
T72 tanks #812 vs. #132 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0
match within 1 pixel no match
-70
-60
-50
-40
-30
-20 -10 0 10 20 Percent shape factor change
30
40
50
60
70
Figure 2.12. Shape factor change with configuration.
mean and standard deviation are 1.3 and 15.7, while for the nonmatching cases they are 5.3 and 31.3, respectively. The peak shape factors can be used as additional features to create an 8D version of the basic recognition system in a manner similar to the way that the magnitudes of the peaks are handled in the 6D system. Figure 2.13 shows ROC curves for the MSTAR T72 and BMP2 configuration variants with the BTR confuser using the 2D, 6D, and 8D recognition systems. Each of the systems was optimized for the forced recognition configuration variant case: the 2D system at 20 scatterers; the 6D system at 36 scatterers (with ±5 pixel translation and ±9% amplitude change limits); the 8D system at 50 scatterers (with a ±30% shape factor change limit). Both the 6D and 8D system results are a substantial improvement over the earlier 2D system results. While Figure 2.13 shows that the 8D system gave worse results than the 6D system in the region below 0.1 Pf , reoptimizing the operating parameters (e.g., using 45 scatterers) gives the 8D system better results in the region below 0.1 Pf at the cost of a slightly reduced forced recognition rate.
2.5 Exploitation of Model Similarity 2.5.1 Similarity Measurement Model similarity can be measured in terms of collisions collisions, where a collision is an instance when observations of two different objects map into the same location (within some specified region of uncertainty) in feature space, i.e., if H(Vi , Vj ) = 1 and ci = cj . The recognition system described in the preceding section has a 6-dimensional (6D) feature space based on the range and cross-range positions and magnitudes of pairs of scatterers (see equation (2.2)). As noted before (in equation (2.1)), the model of an object at some azimuth, with N scatterers, is represented by N (N − 1)/2 observations
Chapter 2 Improving an SAR Recognition System
53
Probability of correct identification
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 8D system, 50 scatterers 6D system, 36 scatterers 2D system, 20 scatterers
0.2 0.1 0 0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 Probability of false alarm
0.8
0.9
1
Figure 2.13. ROC curves for configuration variants with 2D, 6D and 8D systems.
using pairs of scatterers with each pair mapped into the 6D feature space. While the 6D feature space could be represented by a simple 6D array in concept, the large range of potential feature values and high dimensionality make other implementations more practical. The nature of the SAR problem, with discrete pixel values for distances and a large dynamic range for scatterer magnitudes, leads to a natural model implementation, shown previously in Figure 2.9, where the relative range and cross-range locations of a scatterer pair are direct indices to a physical 2D array of lists that contain another 4D of information and the label with the object and pose. Thus, the model construction algorithm of Figure 2.9 does not directly provide collisions in all six dimensions of feature space. In order to determine if two objects map to the same location in feature space we need to apply the same constraints as are used in the recognition algorithm (see step 10 of Figure 2.10 and equation (2.3)), because the constraints dictate the size of the region or bucket in feature space that is considered the same location. The general approach to measure the similarity of one model object with respect to several other objects is to first build the look up table models of the other objects using the normal model construction algorithm of Figure 2.9, and then use a modified version of the recognition algorithm of Figure 2.10 with the subject model object (at all the modeled azimuths) as the test conditions to obtain a histogram of the number scatterer pairs that have various numbers of collisions. Basically the modified algorithm uses the first ten steps of Figure 2.10, with the consideration of each pair of scatterers as a separate occurrence (starting a new count of collisions at step 5) and if the constraints are satisfied (at step 10) then a collision is counted. The total number of observations is equal to AN (N − 1)/2, where A is the number of azimuths modeled (some of the MSTAR data was sequestered, so not all 360◦ were available).
54
Bir Bhanu and Grinnell Jones III
Figure 2.14 shows example model collision histograms (at N = 39 and L = 9) for four MSTAR vehicles (at 15◦ depression angle): BMP2 armored personnel carrier (APC) serial number (#) c21; BTR70 APC #c71; T72 tank #132 and ZSU23/4 anti-aircraft gun #d08. Note that the ZSU23/4 has significantly fewer collisions with the other vehicles, because the ZSU23/4 SAR scatterers cover a larger area than the other objects, and thus, have fewer collisions. 4 object table (39 scatterers)
number of scatterer pairs
1000 BMP2 BTR70 T72 ZSU 23/4
800
600
400
200
0 0
100
200 300 400 500 600 number of collisions Figure 2.14. Example recognition model look-up table collision histograms.
The similarity of a pair of scatterers of given object (at a given azimuth) to the other objects modeled can be measured by the number of collisions with other objects in the look-up table. This can be expressed as a relative measure by using the collision histogram. For convenience, the population of collisions for a particular object is mapped into equal partitions (each with 10% of the total number of collisions). As an example, for the collision histograms in Figure 2.14 we obtain the results in Table 2.1, which shows the number of collisions for a given percent of the population. For the BMP2, for example, 27 collisions or less is in the 10% of the population that is the least similar to the other three models (whereas 90% of the BMP2 scatterer pairs have 274 or less collisions). 2.5.2 Weighted Voting The a priori knowledge of the similarities between object models, expressed as the number of collisions for a given percent of the population, can be captured
Chapter 2 Improving an SAR Recognition System
55
Table 2.1. Number of collisions for a given percent of the population (example for N = 39, L = 9). Object BMP2 BTR70 T72 ZSU 23/4 Population percent
27 21 27 0 10
46 37 48 0 20
Number of collisions 66 87 110 136 167 209 274 53 70 91 116 148 192 266 68 89 111 137 168 209 271 0 0 0 1 3 18 78 30 40 50 60 70 80 90
676 712 667 760 100
by assigning weighted votes to model entries in the look-up table, based on collisions with other objects. This is accomplished off-line by again using a version of the recognition algorithm to obtain the number of look-up table collisions for a particular observation with a pair of scatterers from a subject model and azimuth, as before, and then based on the number of collisions determine the population partition (e.g., using Table 2.1) and finally a given weight function is used to assign a weight label to that instance of the particular model observation entry in the look-up table. Thus, in this approach the model similarities, collisions and associated weightings are all precomputed and appropriate weightings are stored in the look-up table during the off-line modeling process. The similarity-weighted models are obtained using the weighted version of an observation (similar to equation (2.2)) given by Vˆi (c, a) = {w, f1 , f2 , . . . , f6 }i ,
(2.5)
where w is the weight. The similarity weighted version of a match (similar to equation (2.3)), given by ˆ i , Vˆj ) = w if |(fb )i − (fb )j | ≤ δb , ∀ b = 1, . . . , 6 H(V (2.6) 0 otherwise can be substituted in equation (2.4) to obtain weighted recognition results. The various weight functions, used in this research to specify w, are shown in Figure 2.15 which plots the weight value assigned vs. percent of collision population. Function 1 (Figure 2.15(a)) applies equal weight to all the values and is later referred to as unweighted. Functions 2–4 (Figures 2.15(b)–(d)), the convex weight functions, penalize the most similar features (in the right tail of the histogram). Function 5 (Figure 2.15(e)), with equal steps is linear. While functions 6–7 (Figures 2.15(f)–(g)), which reward uniqueness (the left tail of the histogram) are concave. These weight functions illustrate a range of possibilities from function 2, which penalizes only the most similar 10% of the population, to function 7, which rewards only the most dissimilar 10%. These seven weight functions are used and comparative performance results are obtained in experiments described in the next subsection.
10 20 30 40 50 60 70 80 90 100
percent of collision population
10 20 30 40 50 60 70 80 90 100
10 9 8 7 6 5 4 3 2 1 0 0
percent of collision population
10 20 30 40 50 60 70 80 90 100
(c)Function 3.
10 9 8 7 6 5 4 3 2 1 0 0
percent of collision population
(e)Function 5.
weight value assigned
(d)Function 4.
10 9 8 7 6 5 4 3 2 1 0 0
10 20 30 40 50 60 70 80 90 100
percent of collision population
(b)Function 2.
weight value assigned
weight value assigned
10 20 30 40 50 60 70 80 90 100
10 9 8 7 6 5 4 3 2 1 0 0
percent of collision population
(a)Function 1.
10 9 8 7 6 5 4 3 2 1 0 0
weight value assigned
10 9 8 7 6 5 4 3 2 1 0 0
weight value assigned
10 9 8 7 6 5 4 3 2 1 0 0
Bir Bhanu and Grinnell Jones III weight value assigned
weight value assigned
56
10 20 30 40 50 60 70 80 90 100
percent of collision population
(f)Function 6.
10 20 30 40 50 60 70 80 90 100
percent of collision population
(g)Function 7. Figure 2.15. Table weighting functions.
2.5.3 Configuration Variant Experiments Our previous results [9] (using a distance-weighted voting technique where the weight was proportional to the sum of the absolute values of the relative range and cross-range distances between the scatterer pair) showed that for the real vehicles used in the MSTAR data, the differences of configurations for an object type are a more significant challenge for recognition than articulation (where the model and the test data are the same physical object under different conditions). Similarly, the previous results [11] on occluded objects (using an unweighted voting technique) demonstrated significantly better recognition results than the configuration variant cases. For these reasons, in this research we follow a similar approach and optimize the recognition system for the difficult configuration variant cases and then utilize the same system parameters for the articulation and occlusion cases. In these (15◦ depression
Chapter 2 Improving an SAR Recognition System
57
angle) configuration variant experiments, the two object model cases use T72 tank #132 and BMP2 APC #C21 as models, while the four object model cases add BTR70 APC #c71 and ZSU23/4 gun #d08. The test data are two other variants of the T72 (#812, #s7) and two variants of the BMP (#9563, #9566). In addition, BRDM2 APC #e71 is used as an unknown confuser vehicle. The forced recognition results for MSTAR configuration variants are shown in Figure 2.16 for both two object and four object look up table models using various weight functions (defined earlier in Figure 2.15). These results use the optimal parameters (N ,L) for each weight function and table size. For the two object cases, function 3 gives the best results, a recognition rate of 95.81%, compared to the unweighted case of 95.17%. For the four object cases, the convex and linear weighting functions all provide better forced recognition performance than the unweighted case. The concave weighting functions result in worse performance than the unweighted case. The best four object result is 94.17% for function 2, compared to the unweighted case of 92.27%. Thus, increasing the number of objects modeled from two to four, reduces the forced recognition rate by 2.9% (95.17 - 92.27) for the unweighted case, while using model similarity information in the optimum weight function reduces that loss to 1% (95.17 - 94.17). 0.98 2 objects 4 objects
probability of correct recognition
0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 unweighted
linear convex
1
2
3
concave 4 5 6 weight function
7
Figure 2.16. Effect of table size and weighting function on forced recognition of MSTAR configuration variants.
Table 2.2 shows example confusion matrices that illustrate the effect of going from a two object recognition system to a four object model recognition system for the MSTAR configuration variant data. In both cases the system parameters (N ,L) are optimized for forced recognition (2 objects at (38,11) and 4 at (38,12)), both are unweighted cases (constant weight of 10), and both are for d = 1700. (At least 1700 votes, with a weight of 10, is equivalent
58
Bir Bhanu and Grinnell Jones III
to 19 or more scatterers that “matched”.) Comparing the two object results on the left of Table 2.2 with the four object results on the right, we observe that basically a large number of confusers and a few targets move from the Unknown column to the additional models. Thus, while the recognition results are similar for 2 and 4 models (PCI = 0.773 and 0.790 respectively) there are increased false alarms (Pf = 0.13 and 0.32 respectively) which would move the knee of the ROC curve to the right. Table 2.2. Effect of 2 and 4 models on MSTAR configuration variant confusion matrices (unweighted, d = 1700). Identification results Identification results (config. modeled) (configuration modeled) test targets BMP2 T72 Unk. BMP2 T72 BTR70 ZSU23/4 [serial number] (#C21) (#132) (#C21) (#132) (#C71) (#d08) BMP2 [#9563,9566] 189 3 25 189 2 8 0 T72 [#812,s7] 8 131 58 11 138 1 0 BRDM2 (confuser) 28 4 214 27 5 47 0
Unk. 18 47 167
Table 2.3 shows an example MSTAR configuration variant four object confusion matrix for weight function 4. The system parameters (37,9) are optimized for forced recognition with weight function 4 and a d of 1100 is chosen to yield a PCI of 0.776, which is similar to the results shown in Table 2.2. (At least 1100 votes, with an average weight for function 4 of 7.3, is equivalent to 18 or more scatterers matched.) Comparing the earlier four-object unweighted results, shown on the right of Table 2.2, with the weighted results of Table 2.3, we observe that half the misidentifications (11 of 22) are moved to the unknown column. This reduction in misidentifications shows that the model-weighting approach is increasing the distinguishability of the modeled objects. This reduction in misidentifications does not show up directly in the ROC curve results, which treat the off-diagonal target misidentifications the same as the misses where a target is called unknown (i.e., both are cases where the target was not correctly identified). However, the weight function (which effectively reduces the average weighting) allows a similar PCI to be achieved with a lower vote threshold (1100 votes vs. 1700 votes) and results in fewer false alarms. Thus, the lower Pf of 0.276 for the weighted case, vs. 0.321 for the unweighted case, would move the ROC curve to the left. ROC curves are generated for the four-object configuration variant cases by using the optimum parameters for the forced recognition case and varying the vote threshold. Figure 2.17 shows that the ROC curves for the convex and linear weight functions provide generally better performance than the unweighted case. In addition, Figure 2.18 shows that the concave weight functions give worse performance than the unweighted case (except for the region
Chapter 2 Improving an SAR Recognition System
59
Table 2.3. Example MSTAR configuration variant confusion matrix for weight function 4 (d = 1100). Identification results (configuration modeled) test targets BMP2 T72 BTR70 ZSU23/4 Unknown [serial number] (#C21) (#132) (#C71) (#d08) BMP2 [#9563,9566] 179 6 1 0 32 T72 [#812,s7] 4 143 0 0 50 BRDM2 (confuser) 30 6 32 0 178
where PCI < 0.5, Pf < 0.05). The convex weight functions penalize the most common features and so are not much affected by noise (due to configuration differences or other confuser vehicles). On the other hand, the concave weight functions reward (very strongly reward in function 7) the relatively unique features, which makes them susceptible to conditions where noise is strongly rewarded. 1 probability of correct identification
0.9 0.8 0.7
unweighted fn2 fn3 fn4 linear
0.6 0.5 0.4 0.3 0.2 0.1 0 0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 probability of false alarm
0.8
0.9
1
Figure 2.17. MSTAR configuration variant ROCs for beneficial weight functions (four objects).
2.5.4 Articulation Experiments In the articulation experiments the models are nonarticulated versions of T72 #a64 and ZSU23/4 #d08 and the test data are the articulated versions of these same serial number objects and BRDM2 #e71 as a confuser vehicle (all at 30◦ depression angle). Figure 2.19 shows the ROC curves, with excellent articulated object recognition results for both the weight function 2 and the unweighted cases. Since
60
Bir Bhanu and Grinnell Jones III 1 probability of correct identification
0.9 0.8 0.7
unweighted fn6 fn7
0.6 0.5 0.4 0.3 0.2 0.1 0 0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 probability of false alarm
0.8
0.9
1
Figure 2.18. MSTAR configuration variant ROCs for concave weight functions (four objects).
weight function 2, with N = 39 and L = 9, gives the optimum ROC results for the two object (T72, BMP2) configuration experiments and the optimum unweighted parameters are N = 38 and L = 11, these same parameters are used for the articulation experiments. 1 probability of correct identification
0.9 0.8 0.7
unweighted fn 2
0.6 0.5 0.4 0.3 0.2 0.1 0 0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 probability of false alarm
0.8
0.9
1
Figure 2.19. Articulation recognition results.
2.5.5 Occlusion Experiments The occlusion experiments use the same four models as the configuration variant experiments: T72 tank #132, BMP2 APC #C21, BTR70 APC #c71 and ZSU23/4 gun #d08 (all at 15◦ depression angle). The occluded test data is
Chapter 2 Improving an SAR Recognition System
61
generated using the technique described previously in Subsection 2.2.3. In our previous work on occluded objects [11], the confuser vehicle was occluded. However, while the target may be occluded, the confuser vehicle may not necessarily be occluded in practical situations. Hence, in this case the BRDM2 APC (#e71) is an unoccluded confuser vehicle, which is a more difficult problem. Figure 2.20 shows the effect of occlusion on ROC curves for weight function 2, with N = 40 and L = 9 (while N = 40 is not optimum, it yields occlusion in 5% increments). Here with the unoccluded confuser, excellent recognition results are achieved for less than 45 percent occlusion, compared with the prior 70 percent occlusion with an occluded confuser [11]. 1 probability of correct identification
0.9 0.8 0.7 0.6
40 percent occlusion 45 percent occlusion 50 percent occlusion 55 percent occluded
0.5 0.4 0.3 0.2 0.1 0 0
0.1
0.2
0.3 0.4 0.5 0.6 0.7 probability of false alarm
0.8
0.9
1
Figure 2.20. Effect of occlusion on receiver operating characteristics.
2.6 Multiple Recognizers at Different Look Angles 2.6.1 Independence of Multiple Look Angle SAR Recognizers The azimuthal variance of SAR scatterer locations, previously demonstrated in Figure 2.4, strongly indicates that observations at different azimuth angles are independent. Given the probability of one SAR recognizer failing, F 1, where F 1 = 1 − P CI; then if two recognizers are independent, the probability that both recognizers are wrong, F 2, is simply F 2 = F 12 . In order to obtain the most failures, we pick the object configuration variant case, which is the most difficult case for our SAR recognition approach, compared to the depression angle change and object articulation cases. In the configuration variant experiments a single configuration of the T72 (#132) and BMP2 (#C21) vehicles are used as the models and the test data
62
Bir Bhanu and Grinnell Jones III
are two other variants of each vehicle type (T72 #812, #s7 and BMP2 #9563, #9566), all at 15◦ depression angle. These results are obtained using the optimum parameters of 38 scattering centers and a percent magnitude change of less than ±11%. These parameter settings are slightly different from the corresponding (36 and 9%) values used previously in Section 2.4, because here we use unweighted voting, as opposed to the previous distance-weighted voting method. Figure 2.21 takes the experimental forced recognition results for the configuration variant cases with all available combinations of two different azimuths and plots the probability that two recognizers are both wrong as a function of the difference in azimuth angle of the object from the two recognizers. (While Figure 2.21 emphasizes the small angles by only showing up to ±60 degrees, the results out to ±180 degrees are similar.) The single recognizer result, 19 failures in 414 trials, is an F1 failure rate of 0.0459, which is plotted for reference as point “a” in Figure 2.21. For an F1 of 0.0459 the predicted value of F2 is 0.0021, which is very close to the overall experimental average F2 of 0.0025 . The other interesting observation from Figure 2.21 is that the results are independent of the angle difference, even for very small values like 1 degree. This demonstration that multiple-look-angle SAR recognition results are independent, even for small angles down to 1 degree, provides the scientific basis for both measuring and improving the quality of recognition results. 0.05 a
Probability both are wrong
0.045 0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 0 0
5
10 15 20 25 30 35 40 45 50 55 60 Azimuth delta
Figure 2.21. Probability that two recognizers are both wrong.
2.6.2 Multiple-Look-Angle Configuration Variant Results In contrast to the forced recognition case described in the previous subsection, in these configuration variant experiments the BTR70 armored personnel carrier (#c71) is used as an unmodeled confuser vehicle to test the recognition
Chapter 2 Improving an SAR Recognition System
63
system and the vote threshold parameter is used to generate “unknown” results. The other test conditions and parameters are the same as the forced recognition case (most significantly, the models are one configuration of the T72 and BMP2 vehicles and the test vehicles are two different configurations). Figure 2.22 shows the effect of multiple look angle recognizers on the probability of false alarm using the BTR70 as a confuser. (The BTR70 is a more difficult case than other confusers such as the BRDM2 armored personnel carrier or the ZSU 23/4 anti-aircraft gun [9].) In the cases with two recognizers, the decision rule is that if either gives results above the vote threshold, the result is declared a target (which, for these BTR70 confusers, would be a false alarm). Thus, with this “target bias” decision rule the multiple recognizer cases have higher false alarms than a single recognizer. It is important to note that the penalty in increased false alarms is small for the left tail of the curve. Figure 2.22 also shows that the false alarm rates are similar for all the two-look angle recognizer cases and that agreement on the “target” is basically irrelevant for a false alarm. 1
Probability of false alarm
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2
both agree best of two single two disagree
0.1 0 240 220 200 180 160 140 120 100 Votes
80
Figure 2.22. Effect of multi-look on probability of false alarm.
Figure 2.23 shows the effect of multiple-look-angle recognizers on the probability of correct recognition for the configuration variant case where the test data are different configurations of the T72 and BMP2. The top curve shows the results for the 91.7 percent of the time when two recognizers at different look angles agree on the result. The bottom curve is for the remaining 8.3 percent of the time when the two recognizers disagree and the answer that gets the most votes is chosen. The second curve, labeled “best of two,” uses a decision level fusion rule that simply picks the target based on which of the two recognizers got the most votes. (This case is also the weighted average of the agree and disagree cases). In Figure 2.23 the probability of correct recognition decreases as the vote threshold increases (to the left in Figure 2.23), because
64
Bir Bhanu and Grinnell Jones III
the higher threshold causes more targets to be classified as “unknown.” The recognition results for using the best of two recognizers at different look angles are substantially better than the results for a single recognizer. This is basically the result of fewer “misses,” where a target object is classified as an “unknown”; because there are two opportunities to get above the vote threshold and declare a “target.”
Probability of correct recognition
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2
both agree best of two single two disagree
0.1 0 240 220 200 180 160 140 120 100 Votes
80
Figure 2.23. Effect of multi-look on probability of correct recognition.
Figure 2.24 combines the results of Figures 2.22 and 2.23 and shows the effect of using multiple-look-angle recognizers on the ROC curve for the configuration variant cases. These recognition results, using the best-of-two recognizers at different look angles, are substantially better than the results for a single recognizer. For example, at a 0.10 Pf the PCI for the best of two look angles is 0.8324, compared to 0.7091 for a single recognizer. The performance improvement is because the cost in increased false alarms (in Figure 2.22) is low compared to the benefits in increased recognition (in Figures 2.23), due to fewer targets being classified as “unknown.” 2.6.3 Multiple-Look-Angle Articulation Results In the articulated object experiments the models are nonarticulated versions of T72 tank #a64 and ZSU 23/4 anti-aircraft gun #d08 (with the gun turret straight forward) and the test data are articulated versions of these same serial number objects (with the turret rotated) and BRDM2 armored personnel carrier #e71 as a confuser vehicle (all at 30◦ depression angle). The results of applying the same techniques (and all the same recognition system parameters) in these articulated object experiments are shown as ROC curves in Figure 2.25. Again the results for using two recognizers at different look angles and picking the answer with the largest number of votes are better than the single recognizer results.
Chapter 2 Improving an SAR Recognition System
65
Probability of correct recognition
1 0.9 0.8 0.7 0.6 0.5 0.4
both agree best of two single two disagree
0.3 0.2 0.1 0 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Probability of false alarm
1
Figure 2.24. Effect of multilook on configuration variant ROC curve.
Probability of correct recognition
1 0.9 0.8 0.7 0.6 0.5 0.4
best of two single
0.3 0.2 0.1 0 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Probability of false alarm
1
Figure 2.25. Effect of multilook on articulated object ROC curve.
2.6.4 Multiple-Look-Angle Occlusion Results The occlusion experiments use four models: T72 tank #132, BMP2 APC #C21, BTR70 APC #c71 and ZSU23/4 gun #d08 and the unmodeled confuser vehicle is BRDM2 APC (#e71) (all at 15◦ depression angle). The occluded test data is generated using the technique described previously in Subsection 2.2.3, while the BRDM2 APC (#e71) is the more difficult unoccluded confuser vehicle. Figure 2.26 shows the effect of multiple-look-angle recognizers on the probability of correct recognition for the case of 50% occluded targets and an unoccluded confuser. The same techniques (and all the same recognition system parameters) used in the prior configuration variant and articulation experiments are applied to these occluded object experiments. Here again, using two recognizers at different look angles and a decision-level fusion rule of picking
66
Bir Bhanu and Grinnell Jones III
the answer with the largest number of votes gives better results than a single recognizer.
Probability of correct recognition
1 0.9 0.8 0.7 0.6 0.5 0.4
best of two single
0.3 0.2 0.1 0 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Probability of false alarm
1
Figure 2.26. Effect of multi-look on 50% occluded object ROC curve.
2.7 Conclusions The locations and magnitudes of a significant number of SAR scatterers are quasi-invariant with target configuration variations and articulations. A model-based recognition system, using inexact match of local features can successfully handle difficult conditions with object configuration variants, articulation, and occlusion. A comparison of experimental recognition results for MSTAR configuration variants is shown in Table 2.4 for different methods of improving performance of SAR recognition systems involving incorporation of additional features; exploitation of knowledge of model similarity; and integration of multiple recognizers at different look angles. Recognition rates are shown for forced recognition cases and for cases with a 0.10 confuser false alarm rate. The use of additional scatterer magnitude features, the consistent translation constraint, and the ability to handle inexact matches of these local features are fundamental to the success of the basic 6D recognition system, compared to the earlier 2D system [12] which only used relative scatterer locations and required an exact location match. Additional quasi-invariant features, such as peak shape factor in the 8D system, can provide some increase in recognition performance. The similarities between object models can be effectively quantified using histograms of collisions in feature space. This a priori knowledge of object similarity can be successfully used to improve the performance of SAR target
Chapter 2 Improving an SAR Recognition System
67
Table 2.4. Comparison of results for MSTAR T72 and BMP2 configuration variants (6D systems, except as noted by *). recognition rate forced at 0.10 Pf Additional features [Figure 2.13]: * 2D system 6D system * 8D system Use model similarity: function 2 [Figure 2.17], 4 models 2 models Multiple look angles: best of two [Figure 2.24]
0.712 0.947 0.957
0.058 0.554 0.636
0.942 0.949
0.609 0.744
0.975
0.832
recognition. The approach can increase the distinguishability of the modeled objects, reduce misidentifications, and result in decreased false alarms. This is especially beneficial in cases where the number of modeled objects is large. One set of results shown in Table 2.4 for “model similarity” is for a more difficult case with four models, while all the other cases in the table are for two models. In the most difficult configuration variant cases, the convex and linear weight functions, which penalize the most common features, give better performance than the concave weight functions, which strongly reward relatively unique features. The fundamental azimuthal variance of SAR scatterer locations can be successfully used as the basis for a principled and effective multiple-look-angle SAR recognition approach. The experiments demonstrate that SAR recognition results at different azimuths are independent, even for small azimuths, such as one degree. In addition, using decision level fusion of two observations at different look angles can substantially increase SAR recognition performance for target configuration variants. These techniques have also been successfully applied to recognition of articulated objects and occluded objects. In experiments using MSTAR SAR data with a BRDM2 APC confuser and articulated versions of a T72 tank and ZSU 23/4 anti-aircraft gun, the basic 6D recognition system with nonarticulated T72 and ZSU 23/4 models achieved a 95.6−95.7% recognition rate (depending on system parameters) with a 0.10 Pf . Comparable rates using the model similarity and multiple-look-angle approaches increased to 98.0% and 99.0% respectively (see Figures 2.19 and 2.25). Similarly, for experiments with 50 percent occluded objects (with an unoccluded confuser), using the model similarity approach achieved a 70.1% recognition rate at 0.10Pf and multiple look angles achieved 100%, while the basic 6D system only achieved a 20.7% recognition rate (see Figures 2.20 and 2.26).
68
Bir Bhanu and Grinnell Jones III
Higher-resolution SAR data, instead of the one-foot resolution now commonly available, would greatly increase the discriminating power of the current scatterer location and magnitude features. Additional features such as valleys or ridges could also be used, especially with higher-resolution data. The approach of exploiting knowledge of model similarity will become more critical in scaling the recognition problem from 2 to 4 objects to a more realistic 20 to 40 objects. The multiple-look-angle approach could be readily extended to use more than two look angles and more formal methods, like Dempster–Shafer and Bayesian techniques could be used for decision-level fusion. In addition, while the research described in this chapter independently addresses three basic approaches to improve recognition performance: (1) incorporation of additional features; (2) exploitation of a priori knowledge of model similarity; and (3) integration of multiple recognizers at different look angles, these techniques could be combined in practice to address difficult operating conditions involving large numbers of different objects with the combined effects of non-standard configurations, articulation and occlusion.
Acknowledgements This work is supported by AFOSR grant F49620-02-1-0315; the contents and information do not necessarily reflect the position or policy of the U.S. Government.
References [1] Ross, T., Worrell, S., Velten, V., Mossing, J., Bryant, M.: Standard SAR ATR evaluation experiments using the MSTAR public release data set. In: SPIE Proceedings: Algorithms for Synthetic Aperture Radar Imagery V. Volume 3370. (1998) 566–573 [2] Carlson, D., Kumar, B., Mitchell, R., Hoffelder, M.: Optimal trade-off distance classifier correlation filters (OTDCCFs) for synthetic aperture radar automatic target recognition. In: SPIE Proceedings: Algorithms for Synthetic Aperture Radar Imagery IV. Volume 3070. (1997) 110–120 [3] Casasent, D., Shenoy, R.: Synthetic aperture radar detection and clutter rejection MINACE filters. Pattern Recognition 30 (1997) 151–162 [4] Meth, R., Chellappa, R.: Automatic classification of targets in synthetic aperture radar imagery using topographic features. In: SPIE Proceedings: Algorithms for SAR Imagery III. Volume 2757. (1996) 186–193 [5] Ryan, T., Egaas, B.: SAR target indexing with hierarchical distance transforms. In: SPIE Proceedings: Algorithms for Synthetic Aperture Radar Imagery III. Volume 2757. (1996) 243–252 [6] Verly, J., Delanoy, R., Lazott, C.: Principles and evaluation of an automatic target recognition system for synthetic aperture radar imagery
Chapter 2 Improving an SAR Recognition System
[7]
[8]
[9] [10] [11] [12]
[13]
[14] [15]
[16]
[17]
[18]
[19]
[20]
[21] [22]
69
based on the use of functional templates. In: SPIE Proceedings: Automatic Target Recognition III. Volume 1960., Orlando, FL (1993) 57–71 Casasent, D., Shenoy, R.: Feature space trajectory for distorted-object classification and pose estimation in SAR. Optical Engineering 36 (1997) 2719–2728 Yi, J.H., Bhanu, B., Li, M.: Target indexing in SAR images using scattering centers and the Hausdorff distance. Pattern Recognition Letters 17 (1996) 1191–1198 Bhanu, B., Jones, G.: Recognizing target variations and articulations in synthetic aperture radar images. Optical Engineering 39 (2000) 712–723 Jones, G., Bhanu, B.: Recognizing articulated targets in SAR images. Pattern Recognition 34 (2001) 469–485 Jones, G., Bhanu, B.: Recognizing occluded objects in SAR images. IEEE Transactions on Aerospace and Electronic Systems 37 (2001) 316–328 III, G.J., Bhanu, B.: Recognition of articulated and occluded objects. IEEE Transactions on Pattern Analysis and Machine Intelligence 21 (1999) 603–613 Boshra, M., Bhanu, B.: Predicting an upper bound on performance of target recognition in SAR images. IEEE Transactions on Aerospace and Electronic Systems 37 (2001) 876–888 : Khoros Pro v2.2 User’s Guide. Addison-Wesley-Longman (1998) Dudgeon, D., Lacoss, R., Lazott, C., Verly, J.: Use of persistent scatterers for model-based recognition. In: SPIE Proceedings: Algorithms for Synthetic Aperture Radar Imagery. Volume 2230., Orlando, FL (1994) 356–368 Novak, L., Halversen, S., Owirka, G., Hiett, M.: Effects of polarization and resolution on SAR ATR. IEEE Transactions on Aerospace and Electronic Systems 33 (1997) 102–115 Ikeuchi, K., Shakungaa, T., Wheelera, M., Yamazaki, T.: Invariant histograms and deformable template matching for SAR target recognition. In: Proc. IEEE Conf. on Comp. Vision and Pattern Rec. (1996) 100–105 Mossing, J., Ross, T.: An evaluation of SAR ATR algorithm performance sensitivity to MSTAR extended operating conditions. In: SPIE Proceedings: Algorithms for SAR Imagery V. Volume 3370. (1998) 554–565 Wissinger, J., Ristroph, R., Diemunsch, J., Severson, W., Freudenthal, E.: MSTAR’s extensible search engine and inferencing toolkit. In: SPIE Proceedings: Algorithms for SAR Imagery VI. Volume 3721. (1999) 554– 570 Lamden, Y., Wolfson, H.: Geometric hashing: A general and efficient model-based recognition scheme. In: Proc. International Conference on Computer Vision. (1988) 238–249 Grimson, W.E.L.: Object Recognition by Computer: The Role of Geometric Constraints. The MIT Press, Cambridge, MA (1990) Bhanu, B., Lin, Y., Jones, G., Peng, J.: Adaptive target recognition. Machine Vision and Applications 11 (2000) 289–299
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches Gregory Arnold1 , Timothy J. Klausutis2 , and Kirk Sturtz3 1 2 3
Air Force Research Lab, AFRL/SNAT, Bldg. 620, 2241 Avionics Circle, Dayton, Ohio 45433
[email protected] Air Force Research Lab, AFRL/MNGI, Bldg. 13, 101 West Eglin Blvd., Eglin AFB, FL 32542
[email protected] Veridian Incorporated, 5200 Springfield Pike, Suite 200, Dayton, Ohio 45431
[email protected]
Summary. Three-dimensional laser radars measure the geometric shape of objects. The shape of an object is a geometric quality that is more intuitively understood than intensity-based sensors, and consequently laser radars are easier to interpret. While the shape contains more salient (and less variable) information, the computational difficulties are similar to those of other common sensor systems. A discussion of common approaches to 3D object recognition, and the technical issues (called operating conditions), are presented. A novel method that provides a straightforward approach to handling articulating object components and multiscale decomposition of complex objects is also presented. Invariants (or more precisely covariants) are a key element of this method. The presented approach is appealing since detection and segmentation processes need not be done beforehand, the object recognition system is robust to articulation and obscuration, and it is conducive to incorporating shape metrics.
3.1 Introduction Lasers provide many advantages for the object recognition problem, especially when compared to passive electro-optical (video) sensors. For robust object recognition, it is desirable for the sensor to provide measurements of the object that are stable under many viewing and environmental conditions. Furthermore, these sensor measurements, or signatures, should be easily exploitable and provide enough richness to allow object-to-object separability. Specifically, 3D imaging laser radar (ladar) greatly simplifies the object recognition problem by accurately measuring the geometric shape of an object in 3D (preserving scale). In contrast many other sensing techniques inherently suffer a loss of information by projecting 3D objects onto 2D or 1D images. Another advantage of ladar shape measurements for object recognition is that the object signature is far less variable than other sensing modalities (e.g., the shape 71
72
Gregory Arnold et al.
of the object does not vary due to lighting, diurnal affects, thermal loading, range, etc.). However, there is still difficulty since the ladar effectively samples an object’s surface slightly differently each time. Another complicating factor is that ladar does not sample a scene on a uniform sampling lattice. Furthermore, a general complication for recognition is the lack of a general theory of discrimination (i.e., how to tell objects apart). This chapter will present the fundamentals of generic ladar systems, and detail a method for handling the differences in images due to articulation and viewpoint changes. Generically, ladars can be thought of as an orthographic projection of the world onto the sensor (see Figure 3.1). Many 3D imaging ladars provide a range and intensity measurement at every point in the sampling lattice. Direct detection and coherent detection are two common ladar detection techniques. A complete treatise of ladar detection techniques is beyond the scope of this chapter [1]. The intensity value is a measurement of the amount of energy reflected from the appropriate region of the sampling lattice. This measurement is directly related to the monostatic bidirectional reflectance distribution function (mBRDF) of the material illuminated by the laser pulse. The intensity image is effectively a narrow-band, actively illuminated 2D image. This chapter focuses on the range measurement. The inherent data coordinate system for 3D ladar is {angle, angle, range}={θ, φ, ρ}, where θ is the depression angle and φ is the azimuth angle from which the transmitted laser energy propagates from the sensor for each point in the sampling lattice. This is a polar coordinate system that can be transformed into a rectilinear {x, y, z} coordinate system. Many different types of ladars exist, but for simplicity a flash ladar constructed with a focal plane array (FPA) of detectors will be assumed as the standard in this chapter. The term flash implies that the whole range and intensity image is measured at one time by spotlight illuminating the entire scene with one laser pulse. Alternately, a scanning 3D ladar images a scene by scanning one or several Laser beams over the entire sampling lattice. Although scanning ladars will be briefly described, the assumption is that appropriate motion compensation for platform motion has been done such that the resulting range image from the scanning ladar is equivalent to a (3-D) flash ladar. This assumes that the scanning mechanism operates in a linear fashion. Operating conditions [2] have been discussed in many papers since their inauguration during the DARPA Moving and Stationary Target Acquisition and Recognition (MSTAR) program. Essentially the operating conditions are an attempt to describe everything that can affect the sensed image. They include sensor, target, and environmental parameters. Understanding all the standard and extended operating conditions is a first step to the development of an object recognition system. For the purposes of this chapter, they are categorized as (i) conditions whose effect on the image can be modeled (i.e., by a group action), (ii) conditions that obscure the image of the object (but are not easily modeled), or (iii) conditions that do not affect the part of the image corresponding to the object (i.e., changes in background). The goal is to
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
LADAR SEEKER
OUT-GOING LIGHT PULSE
IN-COMING LIGHT PULSE SLANT RANGE ~ ROUND TRIP TIME
73
RANGE IMAGE
}
LIGHT SCATTERED FROM FOG, SMOKE, OR DUST
FALSE RETURNS OR FILTER WITH RANGE GATE REFLECTED LIGHT PULSE
INTENSITY IMAGE
Figure 3.1. General ladar scene. This conceptual diagram illustrates the typical scenario and primary drivers of the sensed imagery. Most ladars return both a range and intensity at each pixel.
model and mitigate sufficient geometric effects to create a robust recognition system. The approach is model–based and object-centric. A 3D ladar object recognition system can be thought of as a “simple” data fusion technique. It is simpler than more general fusion techniques because (typically) only one sensor and one image is involved. Even for this simple case the algorithm must efficiently fuse the information collected from each image pixel (or voxel, short for volumetric picture element, since it contains a third dimension). Enough pixels must be considered simultaneously to remove the unknown nuisance parameters of the model. For example, a single range return (without any other knowledge) cannot contain any information about the shape of the object. For the purposes of this chapter, “shape” is defined as what is left after translation and rotation have been removed. At least four points are needed (for 3D) to extract the local shape. Noise and small changes in the sampling grid can seriously affect the measured shape if the four points are in close proximity. Close proximity is preferred to mitigate obscuration and articulation, but dispersed locations are preferred to mitigate noise and achieve separability. Thus, the algorithm must make intelligent and efficient use of multiscale information to balance these issues. The goal of 3D ladar object recognition systems is to classify images based on how similar or dissimilar the shape from the image is as compared to the shape of each object in a database. A metric is required to quantitatively compare and sort objects based on shape. The triangle inequality property of a metric enables efficient sifting through a large database by removing whole classes of objects that are different enough to be immediately removed from consideration. A modified form of the Procrustes metric is used for this object recognition system. This metric is constructed by considering the quotient space that is invariant to translation and rotation.
74
Gregory Arnold et al.
While 2D and 1D ladar systems exist and have many useful applications, for the purposes of this chapter ladar will imply 3D imaging ladar. A 3D model will be referred to as an object, and an {angle, angle, range} projection of that object (with a ladar at an arbitrary orientation) as an image or range image.
3.2 Ladar Sensors 3.2.1 Hardware ladar sensors are active devices that avoid numerous issues inherent in passive systems and in stereo 3D reconstruction approaches. The two primary types of active range sensors are direct detection (also known as time-of-flight) and coherent detection. Most direct-detection sensors periodically emit a short intermittent pulse and the distance is calculated by measuring the round-trip time (given the speed of light). This eliminates one type of shadow region that occurs with standard stereo 3D reconstruction approaches when a portion of the scene is not visible to both sensors [3]. However, shadow regions will still occur due to self-occlusion and occlusion of the background by objects in the foreground. This is one of the major differences in 3D image formation using ladar and the sensors used in the medical community. ladar is a reflective sensor while many 3D medical imaging sensors are partially transmissive. Coherent detection techniques measure distance with a continuous (or multipulse) laser beam by measuring the phase shift of the reflected signal with respect to the original signal. Continuous-beam lasers require more power and are less covert. Furthermore, coherent detection techniques require more complex hardware than direct detection techniques. Many ladar sensors have a single emitter and detector that are scanned across the scene at a constant angular resolution (see Figure 3.2). Although this is (currently) less expensive than the flash arrays that follow, there are numerous disadvantages. The primary disadvantage is that it takes longer to image a scene by scanning. A secondary problem is the nonlinearities induced by the scanning motion and by scene and object motion during the scanning process. 3.2.2 Projection The appropriate choice of the projection model for ladar is muddled since the projection model will be different depending upon which coordinate system is utilized! In the native {angle, angle, range} coordinate system the projection model is a pinhole camera (full perspective model). Consider two parallel line segments in a plane orthogonal to the line of sight of the ladar. As the lines are moved farther away from the sensor, the angle subtended with respect
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
75
OPTICAL DETECTOR TELESCOPE
LASER BEAM SPLITTER
SCANNER
Figure 3.2. Typical ladar sensor hardware. A simplified diagram of the most common scanning, direct–detect system. 1 to the sensor decreases as range (in each dimension). Therefore the angular extent of the lines within the image decreases as a function of their range. The farther an object is from the sensor, the fewer pixels that will be on the object’s surface (commonly called pixels-on-target). Lines that are parallel in 3D (but not parallel to the FPA) converge to a vanishing point in the {angle, angle, range} image. This is known as the “train tracks phenomenon.” Standard video cameras are also perspective projection. A fundamental difference is that video cameras cannot recover the absolute size of the object. In contrast, since a ladar measures the range, the size of the object can be calculated (up to the pixelation error). An orthographic projection model is appropriate for a rectangular coordinate system. It is not possible to perform this coordinate transformation directly with video cameras. The addition of range information enables the conversion for ladar from polar to rectangular coordinates. This is not precisely correct since the rectangular voxels should grow larger as the depth increases analogous to the change in size of the polar voxels. Alternately, the x–y precision decreases for voxels that are farther from the sensor. For those familiar with radar systems, which are also orthographic, this is analogous to the fact that the signal-to-noise ratio decreases as the object moves farther away from the sensor.
3.2.3 Angle–Angle A flash ladar’s FPA is analogous to the CCD arrays used in digital cameras and video cameras where each array element contains a tiny receiver. Unlike CCD cameras, ladar is an active sensor and a single laser is used to spotlight illuminate the entire scene with each laser pulse. The laser is co-located with the receiver to form a monostatic system. Therefore, each element in the focal plane array can be viewed as containing both a receiver and a transmitter. The FPA structure is inherently different from scanning systems as illustrated in Figure 3.3. Lenses can be used to transform to either array format, but FPAs are easier to build with a constant array size. A constant angular
76
Gregory Arnold et al.
size has a modest advantage during the polar-to-rectangular transformation that is commonly done in software. Neither system will obtain a uniform sampling of an observed surface in general. In Figure 3.3 a constant array size would obtain a uniform sampling of a flat surface that is parallel to the FPA, but as the surface rotates out of plane it will be sampled non uniformly. The only surface that would be sampled uniformly in general is a spherical surface with its origin placed at the focal point of a constant angular size ladar system. In summary, most ladar systems are natively {angle, angle, range} (i.e., polar coordinates). Algorithms that are conducive to polar coordinate systems will have an inherent computational and noise advantage, because it is not possible to transform from the (discretized) polar coordinate system to the (discretized) rectangular coordinate system without some interpolation scheme.
(a)
(b)
Figure 3.3. ladar focal plane arrays. A notional one-dimensional focal plane array is portrayed with a constant array size (a), and with a constant angular size (b). The choice of coordinate system will affect sensor design and the corresponding algorithm development. For FPAs, uniformly sized arrays, (a), are easier to construct and can achieve a constant angular size using a lens. For scanning ladar systems, it is easier to use constant angular step scan mirrors creating a constant angular array (b). Caution is still necessary since neither a constant angular or constant array size imply a uniform sampling of the object surface.
3.2.4 Range Each pixel of the ladar measures the energy returned as a function of time. This is called the “range profile,” and a notional example of a range profile for one pixel is illustrated in Figure 3.4. The majority of systems return the location and intensity of the nth-peak, where n is typically the first or last peak. This is commonly called “first bounce” or “last bounce.” Some newer systems return the entire range profile or the first m detected pulse returns. This has advantages for reasoning about obscuration and validating edges and surfaces more complex than flat plates. The returned pulse is the convolution of the transmitted signal with the object surface. Thus, the determination of the actual range is accomplished
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
77
E n er gy
1 s t bounc e
T i me Figure 3.4. ladar range profile for one pixel. The range profile is the returned signal as a function of time (or distance). When multiple objects are within the field of view, more than one peak will occur. Current ladar systems typically return the first or last bounce peak of the range profile. Newer systems return multiple peaks or the entire profile.
by deconvolving the transmitted signal from the returned signal. This is generally an ill-posed problem. The common solution is to assume that the object surface is flat and therefore a simple peak detector is used to determine the estimated range. Hardware designers employ several common pulse detection strategies and these directly affect the location of the sensed return pulse. Furthermore, the selection of pulse detection technique will change the noise statistics. When multiple surfaces are within the same resolution cell, multiple peaks occur within the returned signal such as are illustrated in Figure 3.4. The nth-peak approach is a trade-off between accuracy and speed. 3.2.5 Transmission Atmospheric effects such as scintillation (atmospheric transmission) and semitransparent obscuration (clouds and smoke) limit the useful range of ladar systems. Platform vibration mitigation can also be difficult to achieve for high-resolution systems. Currently, ranges beyond 5 km are typically relegated to 1D or 2D systems unless advanced techniques, like pulse-doublets, is used. Experimental high-resolution 3D systems have greater operational ranges. The reflectance properties of the object can severely affect the returned signal, or cause a complete dropout (i.e., where no return is recorded for the given pixel). Dropouts can occur if the reflectance is too high (shiny metal reflects the emitted energy away from the detector) or too low (the material attenuates the reflected energy to a peak value below the threshold of the detector). Range gating is a technique that is commonly applied to mitigate the affects of obscuration and semitransparent obscurants. The concept is to only examine the part of the signal that could have come from something near the estimated object location. In other words, if some portion of the energy is returned too quickly, then it is probably the result of reflecting off nearby
78
Gregory Arnold et al.
clouds or smoke. Multiple reflections can cause a portion of the energy to return late. The finer the ability to accurately range gate, the better the signal-to-noise ratio will be. The associated peak detector or more advanced algorithms are more robust by applying this simple geometric constraint to the data. The wavelength of the ladar is an interesting question from the point of view of computer vision beyond the visible spectrum. Lasers in the visible wavelengths are obviously very common; however, most sensor platforms would prefer to be covert. Atmospheric transmission and lack of generic tunable band-gap materials also limit the choice of wavelength. A common wavelength today is 1.06µm. However, this wavelength is not eye-safe and therefore the potential applications are limited. Longer wavelengths are being investigated not only for eye safety, but also for better weather and aerosol penetration. 3.2.6 Synopsis Many other topics with regard to ladar system design could be addressed here. For example, polarization, noise statistics, and hardware limitations are all very important considerations for the design and exploitation of ladar data. However, this section is intended to be a basic initiation into key features of ladar hardware and phenomenology. The primary purpose of this chapter is to discuss object recognition capabilities given a ladar sensor. ladar sensor engineers often want to know how to optimize their sensor design for a particular application. However, a discrimination theory does not currently exist and therefore sensor optimization for object recognition is often based on standard pattern recognition techniques applied to very limited data sets. Approaches that are beginning to address this shortcoming are discussed after the presentation of existing algorithm approaches.
3.3 Is 3D Ladar Object Recognition a Solved Problem? ladar object recognition is not a solved problem, primarily due to computational complexity issues. It has been argued that computer vision is generally an ill-posed problem that will never be solved. From an information-theoretic point of view, 3D sensors contain more information than 2D sensors because 2D sensors suffer a loss of information from the projection of R3 → R2 . Still, the computational problems are no easier for 3D than they are for 2D or 1D. 3.3.1 Technical Challenges The computation complexity issues are primarily due to the operating conditions briefly mentioned in Section 3.1 and further illustrated in Table 3.3
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
79
on p. 92. Ross [2] is an excellent paper on defining operating conditions. The following list represents the subset of the operating conditions that are key to the 3D ladar object recognition problem: • • • • • • • • • •
Translation Rotation 2.5D projection Surface resampling Number of pixels-on-target Point correspondence (labeling, registration) Obscuration Articulation Fidelity Unknown objects
An algorithm that can efficiently and effectively handle all of these problems would be a major advancement in the fields of object recognition and computer vision. Note that “noise” is not explicitly listed. This is intentional since it is the belief of the authors that noise is not the fundamental limitation of current object recognition approaches. Added to all these difficulties, the development process is seldom conducive to solving these hard problems. Typically, most of the money is spent on the hardware and data collections. The exploitation efforts are not begun until the final stages of the development. This leads to an additional set of issues: • • • • • • • • • •
There is never enough data. The sensor parameters are undefined or incorrect. The ground truth will be incomplete or incorrect. The operational “requirements” will be way beyond the state-of-the-art. The money will be limited. Time will be limited. Expectations will be too high. The capabilities will be oversold. The sensor models will not be understood. Mother Nature is enigmatic.
Each of the technical challenges will now be addressed in more detail. Translation The first challenge refers to the fact that the coordinate system is sensorcentered, not object-centered. Therefore, the {x, y, z} or {θ, φ, ρ} pixel locations for a particular point on the object can change arbitrarily. In other words, the algorithm should recognize the object whether it appears at the top of the image or the bottom of the image or anywhere in between. Translation is the easiest challenge to solve, but caution is necessary. The standard approach is to move the centroid of a region or volume of interest to the origin.
80
Gregory Arnold et al.
If the object is consistently segmented from the background every time then this simple method will suffice. However, approaches that are more complicated are needed if the algorithm is to successfully match partially obscured or articulated objects. Rotation The above method of removing translation can be interpreted as putting the data into a standard position. A proof exists [4] that the equivalent approach cannot be done to handle rotation. In general, however, the eigenvectors of the inner product of the (rectangular) coordinates are invariant to rotation. The primary difficulty is with degenerate objects or completely symmetric objects such as a sphere. The difficulty manifests itself as the need for heuristic algorithms to choose the sign and ordering of the eigenvectors. Mathematically, quotienting out 3D rotation from the ladar data does not result in a smooth manifold. 2.5-D Projection The term 2.5D is used to convey the fact that ladar sensors cannot see through an object [1]. As noted in Section 3.2.1, ladar is a reflective sensor, not a transmissive sensor like most medical 3D sensors. A “true” 3D sensor would return a computer-aided design (CAD) model of the object (i.e., front, back, and all the information in between). A ladar only returns the range to the first dispersive surface. Thus, there is a requirement to match an “image” to a model in the database. Ideally, the ladar data can be compared directly to the appropriate portion of the CAD models that are used to describe the objects of interest. Surface Resampling The sampling grid’s relative location on the surface is slightly different each time the surface of the object is sampled. Sometimes these differences are negligible, but a shift of half a pixel can be significant, especially for low resolutions or near, sharp edges. Typically, algorithm developers consider each point in a point cloud as the distance to the object at that particular grid point. This assumption would be correct if the instantaneous field of view (IFOV) of the detectors in the FPA were infinitesimally small. However, this is never the case and the detectors see a portion of the target larger than an infinitesimally small point. Therefore, the range measurement returned is the average distance of the surface(s) within the pixel’s extent (depending upon the peak extraction algorithm). This simplification is harder to make when working with the complete range profile. The implication is that approaches (like graph matching) based on vertex locations will fail without compensating for this affect. Point correspondence algorithms must carefully handle this affect too.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
81
Number of Pixels-on-Target The absolute range from the sensor to the surface of the object is measured and therefore size (scale) is a known quantity. However, the farther the object is from the sensor, the fewer the pixels per unit surface area. For example, for a fixed angular resolution ladar, halving the range to an object quadruples the number pixels on the object. It is possible to calculate the area of the pixel at the object’s surface. However, at the initial stages of the object recognition system it is often more computationally efficient to normalize out the number of pixels-on-target. This is a trade-off between discrimination capability and computational cost. Point Correspondence Even ignoring the issues about resampling the surface, registration is a computationally complex problem—potentially factorial in the number of points to be corresponded. There are some nice algorithms available from the operational research and pattern recognition literature (called the bipartite matching or Hungarian algorithm)[5, 6, 7]. A nice summary and approach is provided in [8] and a powerful new approach is presented in [9]. However, even these have complexity that is polynomial in the number of points to be imaged. An additional consideration is that “point correspondence” typically implies that there are equal numbers of points to register. The problem becomes even more complex when the goal is to match as many points as possible from two different point clouds. The above references have various approaches to solve this problem. An elegant solution to this problem is required before any algorithm is computationally feasible. Two (previously mentioned) complications are that corresponding points will never be precisely the same due to noise, and the points may not physically represent precisely the same region from the scene or CAD model, i.e., the image grid shifted. The first complication necessitates the use of a metric, and the second complication implies that the ultimate goal is to register the measured point cloud directly to a CAD model. In other words, there may not be an advantage to converting the CAD model to a point cloud for the purposes of matching. Obscuration Can an object still be recognized if it is not completely visible? Of course one can never see all of an object simultaneously, and algorithms must work even if everything that is expected to be visible in an image is not. Some parts of the object are more salient than others, so a true understanding of an algorithm’s capability to handle obscuration can only be made with an understanding of the saliency of the visible surfaces. Jones [10] shows relatively good results against obscuration and articulation in 2D. Hetzel [11] presents an approach for 3D data that has an additional advantage as segmentation is not required.
82
Gregory Arnold et al.
Articulation Articulation and obscuration each force a tradeoff between local and global features for object recognition algorithms. Articulation refers to the fact that objects have specific ways of changing. For example, cars have doors that can open, and people can move their arms, legs, and head. It can also be something that may or may not appear on an object, such as a spoiler on a sports car. The ability to recognize an object is typically limited to one instantiation without the ability to recognize each of these variations as allowable changes of the same object. Fidelity A physical and geometric model of the world, the sensor, and everything in between may be conceivable, but it is computationally unachievable. Each step in the processing chain, from the ladar probing the world to the output of the probability of a matching database model, makes simplifications to achieve computability and potentially compromising fidelity. Ideally these simplifications would be achieved while preserving the fundamental discrimination capability or be constructed in such a way as to provide incremental steps back to achieving the ideal discrimination capability. The goal is to be able to recognize all the objects (and corresponding poses) in the image with a finite number of computations. In general, increased fidelity requires greater computation. The link between complexity and database indexing (search) is very strong. The indexing step is an n-class recognition problem, whereas the final validation is a oneclass hypothesis verification. Most of the computations are consumed by the search (throwing out the models that do not match). The fundamental question is how to minimize the required fidelity and still guarantee that potential matches cannot be incorrectly pruned. Unknown Objects This problem has been addressed the least in current literature. The question is how to realize that something is not represented in the database. Ultimately, it boils down to drawing a threshold in a one-class recognition problem, but that threshold should make sense geometrically and with respect to the appropriate noise model. All that can typically be done with current systems is to decide which database object the image looks most like. Thus, the final threshold is drawn based on experiments with limited data sets. As mentioned in the previous technical challenge, the indexing step must be dependent upon the objects in the database, but the final step of verifying the hypothesized identity should be independent of the other objects in the database. Only a controlled environment, where unknown objects cannot occur, would allow the algorithm to bypass the final verification step.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
83
3.3.2 Shape Representations Object representation is a critical first issue in the construction of an object recognition system. From an information-theoretic point of view, the algorithm should use the raw data directly from the sensor since each transformation potentially introduces additional noise into the system or loses relevant information. A rough hierarchy can be imposed upon the different representations that have been proposed by various authors. The following list of shape representations is ordered in an ascending level of abstractness: 1. 2. 3. 4. 5.
Point cloud. Features (points, edges, corners, normals). Triangular mesh. NURBS (biquadratic surfaces). Superquadrics/generalized cylinders (geons).
Note that the number of parameters necessary to describe the shape of an object decreases as the level of abstractness increases. Consequently, object matching based on the more abstract representation is less complicated. A useful survey of techniques for data representation can be found in [12, 13, 14]. Each representation is briefly discussed subsequently. Point Cloud A point cloud generally denotes the raw data that is available directly from the sensor. Features Features are a first-level abstraction of point clouds. Feature points could be as simple as pruning the point cloud to only include points at “interesting” places, such as along edges or corners. The features could also be an augmentation of the point cloud with a local surface normal. Triangular Mesh A triangular mesh is the most common form of mesh that is used. Triangular meshes can be interpreted as a linear approximation (and interpolation) of the point cloud data. Ideally, the choice of vertices for the triangles is made to minimize the difference between the measured point cloud and the corresponding point cloud that would be generated from the mesh. A mesh is typically constrained to be continuous; however this is generally not sufficient to enforce a unique mesh representation. Additional constraints are often applied in order to achieve a unique representation, such as attempting to make the all of the triangles nearly equilateral. However, this is not possible in general, so corresponding recognition algorithms must be able to handle different representations for the same object.
84
Gregory Arnold et al.
NURBS Nonuniform rational B-splines (NURBS) [15] are a generalization of biquadratic surfaces. These representations typically model a 2D surface embedded in 3D [16]. Patches of NURBS can be thought of as the generalization of meshes. The patches form a complete covering of the object’s (measured) surface. Then, the surface within the bounds of each of these patches is represented by a NURBS surface. An immediate complication is defining a robust and unique decomposition of data into model patches, similar to meshes. Thus, the recognition algorithm cannot rely on the same object being modeled the same way for every instantiation of the object. Superquadrics/Generalized Cylinders Superquadrics [17] and generalized cylinders [18] (or geons) model a volume of data. They define specific mathematical forms to model the data. For example, generalize cylinders estimate a cross-section and then sweep the cross-section along the measured data. At each step along the swept out path the centroid of the cross-section is located, and the size of the cross-section is estimated. Objects are constructed by assembling multiple generalized cylinders together. This is clearly an ill–posed problem for obscured or unseen portions of the object. Generally, a symmetry assumption is made, or efforts are made to demarcate what portions of the surface were constructed from measured data. Synopsis All of these higher-level object representations have great promise; however, none of them have lived up to that promise to date. While a successful higherlevel representation would greatly simplify the object recognition process, in practice these representations have simply traded simplicity in one portion of the overall system for added complexity in another with zero or negative gain. The “negative gain” comes from the fact that these representations are exactly that — representations. Object recognition is based on discrimination. While one can argue that there is a functional dependence between representation and discrimination, it is easy to generate examples such that any given representation is the worst possible choice to differentiate the exemplars. However, this is not meant to encourage the other common extreme, which is to collect enough data to distinguish the classes (i.e., data-driven approaches). The goal is to find the right representation to optimize the discrimination capability with respect to the storage or speed requirements. This can only be discovered by modeling the entire object recognition system and using the scientific method such that the data is simply an experiment to validate or refute a hypothesized system model.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
85
3.3.3 Shape Recognition/Indexing Approaches Two obvious approaches to recognition are image-based and model-based. Jain [19] provides an excellent overview of statistical pattern analysis techniques. The most common image-based approach is template matching, and correlation is the most common similarity measure used in template matching. Some common approaches are briefly described below. Every approach has three common problems: 1. How to detect and segment the object from the background. 2. How to build and search a large database (avoiding local minima). 3. How to interpret results when the match is not perfect. The following list and detailed descriptions represents many common approaches to shape recognition that have been developed and tested. References [12] and [13] are good survey papers. 1. 2. 3. 4. 5. 6. 7. 8. 9.
Image-based matching (matched filter, template match). Model-based matching. Geometric hashing. Hough transform. Evidence accrual. Learning approaches (neural networks, genetic algorithms, etc.). Tree search (graph matching). Principal components (eigenspace, eigenface, appearance-based). Invariance (Fourier, moments, spherical harmonics, spin images).
Table 3.1. A summary of some typical characteristics of various approaches to shape recognition. A more detailed description of each method follows. Approach Image-based Model-based Hashing Hough Evidence accrual Learning Tree Principal Components Invariance
Model √ √
√ √ √
Empirical √ √ √ √ √ √ √ √
Voting √ √ √ √ √
√
Alignment √ √
√ √ √ √
Image-Based Matching Image-based or pixel-level matching generally implies the most common matchedfilter-type comparison. This could be a comparison between two different measurements, but more generally is a comparison between a synthesized signature and the measured image using a “hypothesize and verify” type approach.
86
Gregory Arnold et al.
Intuitively, a matched filter is the average of the squared error between each measured pixel and the corresponding template pixel, or a nearest-neighbor type algorithm in a very high-dimensional space. A plethora of variations on this approach is available for purposes such as robustness to outliers and improving relative separability. “Image-based” refers to the fact that the algorithm is trained on measured or synthetic image data. Image-based approaches suffer from both the complexity of the correspondence problem (finding the template in the image) as well as the complexity of building the template database. Moreover, it has been shown that “every consistent recognition scheme for recognizing 3D objects must in general be model based” [20]. Although this was referring to 2D imagery, the pretext still stands and thus pixel-level matching is best suited as a final validation step. Pixel-level validation promises the maximum discrimination capability that is achievable (again from an information theoretic point of view). Several additional considerations are implied by pixel-level validation that are not commonly lumped into standard matched filtering techniques. First, a template is generated from a modeling and simulation capability since precise information is required which cannot be created apriori by data collections (due to cost and complexity). Second, the projection of the model into the image is used to determine the best estimated segmentation. This information should then be used to (a) perform a pixel-level validation for the visible parts of the object, (b) ignore the portions of the object that are obscured, and (c) provide a consistency check that the background looks like background (as opposed to unmodeled portions of the object). A background consistency check is crucial to avoid the “box-in-a-box” problem. This problem appears when trying to recognize different sized objects. For example, a small box looks exactly like a larger box, except (from the sensor point of view) there is additional information that would be ignored without the consistency check. In other words, it would be easy to “recognize” the smaller box within an image of the larger box (however, it would be hard to recognize the larger box given an image of the smaller box). Alternately, is half a car still a car? It is possibly still a car if the other half of the car is obscured, but not likely if the measurements indicate the other half of the car is missing (as evidenced by measurements coming from behind where the other half of the car should be). Even if the algorithm was intelligent enough to realize two known objects are easily confused for each other, a background consistency check is still required so that unknown objects will successfully be rejected. Model-Based Matching The fundamental premise of model-based approaches is that the desired object and few other things will obey the constraints defined by the model. The system’s ultimate performance will be bounded by how well the models can predict the real world and all of its subtleties. As with other methods, inverse
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
87
methods do not currently exist for efficiently matching measured data back to these models. Therefore, current versions of model-based matching are very similar to template matching. The basic approach is “hypothesize and verify,” where the geometry and physics models are used to synthesize an image corresponding to the hypothesis and then a verification technique is applied to compare the measured and synthesized data. Most of these approaches can be classified as 1. voting-based (geometric hashing, pose clustering, or generalized Hough transforms): voting in a parameter space for potential matches, or 2. alignment-based: searching for additional model-to-image matches based on the transformation computed from a small number of hypothesized correspondences. The selectivity and an error analysis based on these approaches has been reported [21, 22, 23]. These techniques are discussed in more detail subsequently. The different voting-based techniques primarily differ in their choice of a transformation space in which to tabulate votes to elect potential matches [24, 10]. These techniques are computationally efficient, accurate, and robust theoretically. The difficulty is in applying these techniques with a noise model. Most of the demonstrated systems assume that the robustness of the system will handle the noise. However, Grimson [23] demonstrates that the affect of the noise is dependent on location. This violates a fundamental assumption of the geometric hashing technique that the noise is independent of where the selected basis was located. Grimson suggests a modified voting technique, but the computational expense and loss in selectivity is clear in comparison to the alignment-based approach, especially as the noise increases. Huttenlocher and Ullman [25] presents an alignment-based approach. Jacob and Alter [21] demonstrates that an alignment-based approach has better error characteristics than voting–based techniques, primarily due to the error approximations being much more accurate in the alignment-based approach (the transformation into the voting space generally make the error very difficult to estimate). Grimson [23] demonstrates, based on the selectivity of a matching set, that a large number of matches are necessary to reduce the probability of false match toward zero. Although Jacobs [21] developed a linear approach for matching points, this still does not handle the combinatorics associated with determining the initial minimal match between the image and object. To conclude, the model-based approach has significant advantages such as limited dependence on measured data (so it is extendible to unsampled data regimes), and orders of magnitude reduction in online storage requirements. The typical disadvantage to model–based approaches is the online cost of synthesizing images corresponding to the hypothesis, especially when complex interactions occur between objects and backgrounds.
88
Gregory Arnold et al.
Geometric Hashing Geometric hashing is essentially an index tabulating an exhaustive enumeration of feature values. The goal is to build an index offline so that the occurrence of a feature in the image is linked to the occurrence of those features for a particular model or models in the database. For example, given a set of features with integer labels between one and five, a hash table would record which models have which features. In addition, it may record the relative frequency of occurrence of the objects for each features value (using some apriori information about the relative frequency of occurrence of the models). Given the following occurrence data: • Model 1: Features 2,4 • Model 2: Features 1,5 • Model 3: Features 5,5,4,2 the hash table in Table 3.2 would be constructed. Table 3.2. An example hash table. The table contains the number of times a feature is present for each model. The occurrence of a feature in the image can be directly indexed into potential matches from the database. Features 1 2 3 4 5
Model 1 0 1 0 1 0
Model 2 1 0 0 0 1
Model 3 0 1 0 1 2
Now, given an image containing a feature of type 1, it is obvious that Model 2 is the only possible match in the database. Furthermore, if a feature of type 3 is found, no models match. If feature type 5 is found, Model 3 is more likely than Model 2. Additional information, such as the look angle from which the feature was visible, may also be stored in the hash table. This is useful for further refining the hypothesis to include pose, especially when there is significant overlap between the features and models. This is a simplistic model of geometric hashing. The principal difficulties are: (a) enumerating and binning all the images so that they can be recorded in the hash table, (b)the separability of the hash table once extensive enumeration has been accomplished, and (c) noise forces the discrete features to be treated probabilistically. This makes the matching much more complex, and once again highlights the need for metrics. References [26] and [27] implement hash tables to achieve efficient indexing.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
89
Hough Transform The Hough transform converts images into the parameter space of lines. This technique is presented in most pattern recognition and computer vision books [28]. The generalized Hough transform extends this technique to parameter spaces for other object descriptors (generally geometric curves and surfaces). The concept is that an edge extraction technique has been applied to the image, and now the goal is to combine the evidence to identify lines despite noise and missing segments. Lines in a 2D image could be parameterized by an angle and the location of an intercept along an axis. However, this parameterization does not uniquely identify lines that are parallel to the axis of intercept. Many different parameterizations are possible, but for demonstration a simple angle–angle representation will be used. A notable benefit of this particular representation is that both parameters have the same domain and range. Thus, the bin size and number of bins should be the same for each parameter. The relative proportion of bins is not as obvious if, for example, slope and intercept are used as the parameters. The general goals in choosing the parameterization include minimizing the number of parameters, nondegeneracy of the parameters, computational complexity, boundedness of the parameters, and uniformity of the parameter distributions. Figure 3.5 illustrates a binning scheme that might be used for the angle–angle parameterization. Notice that the middle bins are effectively more forgiving (coarser quantization) than the bins near the edges. Also, some bins cannot possibly have any support from the image. Thus, an improvement upon this scheme would make the bins a uniform size in the image.
Figure 3.5. Hough transform binning. Here are three examples of the 30×30 different bins corresponding to sampling the circle every 6 degrees. The square represents the image and the circle is shown to demonstrate the construction of the bins. The bins correspond to approximately equivalent lines in the image. Any pixels falling within a bin gives support to that particular line.
Lines in a 2D image intersect a circle circumscribing the image in two locations (the circle must be around the outside of the image in order to
90
Gregory Arnold et al.
avoid lines tangent to the circle). These intersections uniquely define the line, and provide a minimal global parameterization of lines in the image. The circle is discretized into 30 × 30 bins, and a two-dimensional Hough space is created, each axis corresponding to the bin locations along the circle. Each feature in the image votes for all the bins that contain that appropriate point in the image. The relation between the image locations and Hough bins can be precomputed offline so that the online computation is minimal. Bins containing multiple features will always have more votes than bins containing few or no features. Thus, the bins with the most votes are the most likely to contain a line. The presence of a line still needs to be verified. Evidence Accrual Evidence accrual is most commonly another name for Bayesian statistical inference methods [29, 26], although Dempster–Schafer is also common. The approaches and applications are well beyond the scope of this chapter. In general, they are similar to Hough transforms in that ultimately a voting space is established (appropriately normalized to produce a valid probability). Both positive and negative evidence can be accrued. “Negative evidence” refers to evidence that contradicts the hypothesis. Evidence accrual can be used both for determining what models are most likely given an image, and for deciding whether to accept or reject a particular hypothesis given an image. Dempster– Schafer’s advantage in this respect is that it has a framework to incorporate the unknown class and it can generate confidence levels with so-called belief functions [30]. Learning Approaches Neural networks are another area that are well beyond the scope of this chapter [31]. Many different types of networks (generally biologically inspired) have been designed, all with the basic idea of taking the available input data as well as truth information and producing the desired output data. A learning (training) process is used along with feedback (when available) to generate the desired output for a given input. The networks are generally constructed by multiple levels of massively interconnected computational nodes and thresholds. Neural networks are an excellent tool for a quick assessment of the computability of a desired process. However, the inability to guarantee robust decisions and handle unknown objects limits their applicability to object recognition. The reliance of the training process on available data is its fundamental weakness, both because a model is required [20] and because it is typically difficult to predetermine how the decision boundaries generalize to new data. In particular, it is necessary to avoid overtraining, a condition in which the network memorizes the training data and recalls it perfectly but does not generate the true class boundaries.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
91
Genetic algorithms are another biologically inspired approach to object recognition. Genetic algorithms are essentially an optimization technique akin to simulated annealing and gradient descent. The “training” is an iterative process of creation and destruction of potential solutions. Creation involves taking existing solutions and randomly mutating or hybridizing to create new solutions. All the solutions are evaluated based on a measure of success, and those that fall below some threshold are destroyed. This very useful optimization technique is appropriate when other methods of modeling and simplifying the problem are not possible. Tree Search Tree search algorithms are easily as numerous as pixel-level validation variations. The most common approach is to assume the data can be efficiently and correctly segmented into chunks based on local similarity constraints. Then a tree structure is used to describe the relative geometric relation between the chunks. Labels are added to the nodes of the tree that correspond to information extracted from the individual chunks. Finally, trees may be replaced by linked graphs so that missing information will not be detrimental to the matching process [16]. The primary difficulty is how to handle the nonuniqueness of the various partitionings of the image without making the search computationally intractable. “Decision trees” are an automated approach to building trees [32]. Principal Components Principal components, also known as eigenspaces, Hotelling, and Karhunen– Loeve transforms, minimize the mean-squared error in the reconstructed data as elements are dropped from a linear basis. The concept is developed in most pattern recognition and computer vision books [32]. Variations on the basic concept have been suggested that are invariant to translation, rotation, and other useful group actions. It is both an advantage and a disadvantage that the optimal (linear) transformation is data dependent. This enables the data to be transformed such that the principal components are linearly independent. However, the ability to determine and remove any linear dependence between the coefficients representing the objects is limited to the available data. Furthermore, any functional dependence between the components cannot be addressed. Finally, the standard technique is not appropriate for object recognition, since it is optimized to minimize representational error, not discrimination. Section 3.4.2 will detail how Procrustes analysis successfully uses the eigenspace to determine the distance between objects. Reference [33] is the seminal paper on eigenfaces, and [34] develops appearance-based techniques.
92
Gregory Arnold et al.
Invariance Some constraints are required so that not everything can be matched to everything. Data-driven approaches infer the constraints from the available data, whereas invariance-based approaches use a model (group action) of how the objects transform. The invariance approach is to equivalence sets of images that differ only by some (predefined) group actions. The concept is best expressed in the statistical shape analysis literature, shape is what is left after translation, rotation, and scale are removed [35]. Therefore, if the goal is to measure changes in shape, then invariance is a natural tool. Invariance to a particular group action can be achieved many different ways. This includes both variations in the actual invariant function, but also in methods of calculating the function. In particular, invariants can be explicit or implicit. Implicit invariants are functions that are independent of the group parameter. For example, x1 − x2 is invariant to translation along the x-axis. Explicit functions achieve invariance by fixing the group parameter. This is also known as “standard position.” The best example is moving the centroid of the data to the origin. No matter how the data is translated, by moving the centroid back to the origin, the compensated data is explicitly invariant to translation. It only takes a quick example to demonstrate why invariant approaches are important. Consider a standard hypothesis-and-verify technique. Even if the operating condition parameters can be safely quantized as in Table 3.3, the total number of hypothesis is beyond exhaustive computational capabilities. Table 3.3. Enumeration of quantized operating conditions. The total number of hypothesis is beyond exhaustive computational capabilities. Parameter Quantized Bins Object type 20 Object aspect 72 Depression angle 5 Articulation (1 DoF) 36 Configuration (4 binary) 16 Obscuration 400 Correspondence 20 Netting 5 Total Hypotheses 165, 888, 000, 000 = 1.6 × 1011
The most common problem with invariant-based techniques is a lack of understanding of the affect of the transformations into an invariant space. It is trivially true that a constant function is invariant to any group action, so it is only illustrative to point out that many objects are mapped to the same equivalence class by this function. An invariant basis can be derived such that it only equivalences objects that are the same up to the desired
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
93
transformation; however, the affect of noise can be substantially different for each invariant basis. Therefore the choice of invariant basis must be carefully considered. Furthermore, it is not trivial to determine the distribution of the objects in invariant space given a distribution of objects in parameter space. Invariants provide the most promising theoretical approach, but current approaches have not achieved their potential for solving the object recognition problem. However, progress is being made using several different techniques. Johnson’s thesis [36] is the seminal paper on spin images and their application to ladar object recognition. Funkhouser et al. [37] present spherical harmonics. Lo and Don [38] present an excellent summary of invariants formed using moments, and [39] makes significant advances.
3.4 Shape Metrics 3.4.1 Background The motivation for metrics is illustrated by some common questions, “What is the potential efficacy of this sensor to my application?” and “How close is my algorithm to achieving the maximum achievable separability?” However, no theory of discrimination exists and therefore it has not been possible to answer these questions in a general context. Object recognition is fundamentally driven by the ability to differentiate objects. Alternately, a method is needed to measure the difference between objects. The metrics of interest for object recognition are invariant to the modeled group action (e.g., rotation and translation). This is intuitively obvious as one would not expect the distance (measurement of shape difference) between two objects to change when those objects have been rotated. Shape metrics provide a basis for answering these questions. However, metrics for object recognition have been notoriously difficult to find and very little work has been done in developing methods for constructing them. An exception comes from the field called statistical shape analysis or morphometrics [4, 35, 40]. Specifically, these books summarize and extend the research (primarily for biological applications) that although founded in work developed throughout the century, really accelerated with papers by Kendall [41] and Bookstein [42]. The goal in this field is to develop a shape metric based on “landmark” features. From the object recognition point of view, these are simply pixel locations of extracted features. Since shape is of fundamental interest, two objects are considered similar independent of translation, rotation, and scale. In other words, two objects are considered equivalent if they can be brought into correspondence by translating, rotating, and scaling. This is called the similarity group. The metric developed for statistical shape analysis is commonly called the Procrustes, Procrustean, or (in one specific case) the Fubini–Study metric.
94
Gregory Arnold et al.
The proof that it is a metric is developed in Section 3.4.2. The Procrustes metric is a quotient metric. Intuitively, it is easy to conceptualize considering the space of objects modulo the similarity group. Quotienting the group action out of objects can be very difficult both analytically and numerically. Despite these difficulties, this does provide a constructive approach to developing metrics invariant to other group actions. Although an object recognition system must eventually make a hard decision (i.e., make a binary decision as to whether an image is consistent with a particular hypothesized object), the concept is that this decision will occur in the final stages of the algorithm when a pixel-level validation and background consistency is performed. This chapter is applicable to both this final validation stage and the weeding or indexing problem where the goal is to order the most likely candidates first. The indexer should not predetermine how many candidates the algorithm will examine before making a final decision. The metric will naturally rank order the models based on their similarity to the image. Therefore the system will validate the best matches first (based on the extracted data or features) and continue until either the validation is highly confident in a match, it rejects all the potential matches, or a time constraint is exceeded. A preliminary study on the fundamental separability of objects (sets of landmarks) based on this metric can be found in [43]. The need for appropriate metrics is a major theme of this chapter. There have been numerous metrics, measures, and distances proposed throughout the history of object recognition, computer vision, and pattern recognition. Alternative 3D pseudometrics are presented in [3] and [36]. More general metrics can be found in [44] which provides a nice summary of different metrics. Csisz´ar argues for a unique choice of metrics in [45]. Finally, [46] questions whether human perception satisfies any of the axioms of a metric, and [47] contains an excellent summary of what is known about biological vision systems. 3.4.2 Procrustean Metrics In this section the partial Procrustes metric is derived and compared relative to other choices of a metric. The three metrics commonly used in statistical shape analysis are the full Procrustes, the partial Procrustes, and the Procrustes metric. Each metric corresponds to a different model and the appropriate selection of a given metric, in the context of the object recognition problem, depends upon the sensor being employed. Let G be the similarity group — the group generated by the rotation, translation, and scale (dilation)— and consider the componentwise group action Rm × · · · × Rm × G → Rm × · · · × Rm . The associated quotient space, k Σm , is called the shape-space of order (m, k) k Σm ≡ Rm × · · · × Rm /G,
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
95
where k corresponds to both the number of features and the number of copies of Rm . Note that m = 2 represents a two-dimensional “image” space, m = 3 represents three-dimensional “volume” space, and higher values of m represent higher-dimensional feature spaces. The procedure to calculate the (representative) equivalence classes of this quotient space as well as a metric on this space follows. The first step in determining the shape of an object, X, is to quotient out translation by moving the centroid along each axis to a standard position (the origin) ¯ X → X − X, ¯ represents the centroid of X. This where X product ⎡ 1−k 1 1 ... ⎢ 1 1 − k 1 ... 1⎢ ⎢ 1 1 − k ... X→− ⎢ 1 k ⎢ .. .. . . .. ⎣ . . . .
1
1
C
1
can be rewritten as a matrix ⎤⎡
⎤ z1 z2 ⎥ ⎥ z3 ⎥ ⎥ .. ⎥ . ⎦ ... 1 − k xk yk zk 1 1 1 .. .
x1 ⎥ ⎢ x2 ⎥⎢ ⎥ ⎢ x3 ⎥⎢ ⎥ ⎢ .. ⎦⎣ .
y1 y2 y3 .. . X
where C is the k × k centering matrix (the matrix that takes the centroid to the origin), which is symmetric and idempotent (so C = CT = CC). The centering matrix has rank k−1 and taking the Cholesky decomposition of C results in the first row consisting entirely of zeros. Therefore, the Cholesky decomposition is a method for removing a linearly dependent row from C. In particular, the Helmert submatrix, H, is defined by HT H = C, where C is the centering matrix presented above, and H is the Cholesky decomposition into a lower triangular matrix followed by the removal of the row of zeros. Thus H removes translation and reduces the dimension of the object (analogous to dropping one point). This is exactly the number of parameters that were removed from the group action (one parameter for the translation ˆ = HX, is of dimension of each axis). Therefore, the Helmertized object, X (k − 1) × m and is a representative of the original object X in the quotient space for translation. Note that H can be computed directly [35]. The scale is then removed by the scaling operation x → x/x, where · is the l2 -norm as this norm is invariant to rotation. The resulting space, Rm × · · · × Rm modulo translation and scale, is called the preshape space and the equivalence class of X is W = HX/HX. Elements of the preshape space are denoted by W and will also be of dimension (k − 1) × m. A continuous global quotient space does not exist for rotation [4]. However, the distance between the equivalence classes of W modulo SO(m) can be defined and does exist,
96
Gregory Arnold et al.
dP (X1 , X2 ) =
inf
R1 ,R2 ∈SO(m)
W1 R1 − W2 R2 ,
(3.1)
where the matrix norm W = Tr(WT W) is once again the l2 -norm, and SO(m) is the set of m × m rotation matrices. This function can be simplified −1 by defining R = R1 R−1 ) = Tr(A) to obtain 2 and using Tr(BAB dP (X1 , X2 ) = = = =
inf
R1 ,R2 ∈SO(m)
inf
R1 ,R2 ∈SO(m)
inf
R,R2 ∈SO(m)
inf
R∈SO(m)
W1 R1 − W2 R2 (W1 R1 R−1 − W2 )R2 2
(W1 R − W2 )R2
(W1 R − W2 ).
The properties of a metric can be found in any standard math reference book [48]. The properties of the Procrustes metric are ultimately induced by the properties of the norm. In particular, the triangle inequality property of the metric follows from the triangle inequality for the norm dP (X1 , X3 ) = = ≤
inf
R∈SO(m)
W1 R − W3
inf
W1 R − W2 R2 + W2 R2 − W3
inf
(W1 R − W2 R2 + W2 R2 − W3 )
R,R2 ∈SO(m) R,R2 ∈SO(m)
because by defining R = RR−1 2 inf
(W1 R − W2 R2 + W2 R2 − W3 ) W2 R2 − W3 inf inf W1 R − W2 +
R,R2 ∈SO(m)
=
R2 ∈SO(m)
R∈SO(m)
= dP (X1 , X2 ) + dP (X2 , X3 ). Thus this function defines a metric on the shape space. Expanded out in terms of the l2 -norm, the distance is dP (X1 , X2 )2 = = =
2
inf
W1 R − W2
inf
Tr((W1 R − W2 )T (W1 R − W2 ))
inf
Tr(WT1 W1 + WT2 W2 − 2RT WT1 W2 )
R∈SO(m) R∈SO(m) R∈SO(m)
2
2
= Tr(W1 + W2 − 2 = 2(1 −
sup R∈SO(m)
sup R∈SO(m)
Tr(RT WT1 W2 )).
RT WT1 W2 ) (3.2)
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
97
Globally Optimal Solution Finding the argument of the supremum in (3.2) is equivalent to finding the argument of the infimum in (3.1). An analytic globally optimal solution to this optimization problem follows using singular value decomposition WT1 W2 = UT ΛV, where U is orthonormal, V ∈ SO(m), and Λ is zero with a diagonal of singular values. The singular values are positive except the smallest, λm , is the negative if and only if det(WT1 W2 ) < 0 [35]. Note that the singular values, {λi }m i=1 , T T are the square roots of the eigenvalues of the matrix W W W W and has 2 1 1 2 m a maximum i=1 λi = 1 (which corresponds to W1 and W2 matching). Now the metric can be written as dP (X1 , X2 )2 = 2(1 −
Tr(RT UT ΛV)).
sup R∈SO(m)
Since Tr(RT UT ΛV) ≤ Tr(Λ) with equality occurring when R = VUT , the supremum occurs at R = VUT . Thus m (3.3) λi . dP (X1 , X2 ) = 2 1 − i=1
This metric is called the partial Procrustes metric in the statistical shape literature [35]. The full Procrustes metric between two objects X1 and X2 can be written with respect to their corresponding preshapes W1 and W2 as dF (X1 , X2 ) =
inf
Γ∈SO(m),β∈R
W2 − βW1 Γ.
Expressing this metric in terms of the eigenvalues of WT1 W2 WT2 W1 , as done above for the partial Procrustes metric, gives m 2 dF (X1 , X2 ) = 1 − λi . (3.4) i=1
The Procrustes metric is defined as the closest great circle distance between W1 and W2 on the preshape sphere. Kendall [41] shows that the preshape k space is indeed a sphere. In fact, the Procrustes distance on 2 is equivalent to the Fubini-Study metric on CPk−2 (4). From trigonometry it follows m d(X1 , X2 ) = arccos (3.5) λi . i=1
Finally, another distance measure is the full generalized Procrustes metric defined by
98
Gregory Arnold et al.
dg (X1 , X2 ) =
inf Γ1 ,Γ2 ∈SO(m)
(β1 X1 Γ1 + 1k γ 1 ) − (β2 X2 Γ2 + 1k γ 2 )
β1 ,β2 ∈R γ 1 ,γ 2 ∈Rm
= dF (X1 , X2 ). where an additional constraint such as β1 β2 = 1 is necessary to prevent degenerate solutions. The full generalized form is typically used in statistical shape analysis for estimating the mean shape of a set of objects. This development demonstrates it for comparing just two objects. Therefore, the only difference between the full generalized form and the full ordinary form (as it is called in [35]) is the inclusion of the translation in the optimization. For two objects, the full generalized form is equivalent to the full ordinary Procrustes metric (by using the fact that the objects can be centered, i.e., have their centroids moved to the origin, without loss of generality). This follows since the cross terms in the norm due to the translation will disappear because C1k = 0 for any centering matrix C such as the Helmert matrix (C = HT H). The three metrics can be interpreted as a path in shape space corresponding to the chord length (partial Procrustes), arc distance (standard Procrustes), and orthogonal length (full Procrustes). It should be noted that these three metrics are order preserving relative to each other, i.e., dF (X1 , X2 ) < dF (X1 , X3 ) dP (X1 , X2 ) < dP (X1 , X3 ) d(X1 , X2 ) < d(X1 , X3 ). Therefore, any of these metrics can be used if all that matters is the relative ordering of the objects. Dryden and Mardia [35] argues that the full Procrustes distance is the natural choice as it optimizes over the full set of similarity parameters and because it “appears exponentiated in the density for many simple probability distributions for shape” (p.44). Point Correspondence The existing literature on shape metrics presumes the point correspondence between the objects is known or is handled elsewhere. This is also known as the labeling problem. Conceptually, this is the internal labeling problem (matching k image features to k model features) as opposed to the external labeling problem (choosing which subset of the image features and which subset of the model features to compare). An exhaustive computation of the external problem requires Pk computations (where P is the number of image features) for each of the K k feature sets (where K is the number of model features) of each
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
99
model in the database. The internal problem requires k! computations (exhaustive). See [49] for more information. Algorithms for point correspondence that are polynomial in complexity were referenced in Section 3.3.1, however the ultimate goal is to develop an analytic solution to this problem so that theoretical analysis of the label-invariant metric is possible. An implicit assumption has been made that the same subset of points can be found on the image and the object to compare. Zhang [50] presents a technique for consistently extracting the same number of points using Legendre polynomials. The internal labeling problem for Procrustes is explored in [51].
3.5 Current Approach The subsequently described algorithm handles most of the technical challenges previously discussed for a successful 3D object recognition system. The conclusions and future research sections will discuss desired improvements. The method is based on looking at “spheres” of data extracted from the image and comparing these to “spheres” of data extracted from the models. This method specifically avoids feature detection for improved robustness, and this approach specifically handles articulated objects. Recognizing such an object can involve a search in a high-dimensional space that involves all the articulating degrees of freedom, in addition to the usual unknown viewpoint. This algorithm uses invariants to reduce the search space to a manageable size. The input is a range image (in rectangular coordinates) of an unknown object in an unknown articulation position, such as a backhoe with each link in an arbitrary position. The desired output of the system is the identity of the object, and the articulation and viewpoint parameters. The object is identified using a database of known models. Since an object, like the backhoe, could have ten degrees of freedom (DoF) it is infeasible to store all images of each articulated object. Subsampling and storing 10 images of each DoF for just this one object would require 1010 images, which is prohibitive. Consequently an efficient procedure for representing and searching a database of objects is described. Unique features of the method described in this chapter include an integral approach to feature detection and recognition that is more efficient than considering them separately and improves robustness to noise. The approach includes scanning the image, so it avoids the combinatorial explosion of hypothesizing feature correspondences [52, 53, 54]. This approach simultaneously uses the data from all the points in a neighborhood (thus the robustness to noise and obscuration). 3.5.1 The Sensor Model The sensor model characterizes the transformation group acting on the object, and the projection from R3 to R3 . The model presented here is written for
100
Gregory Arnold et al.
four points on an object undergoing the same rigid motion. This is a good starting point, with the articulation being incorporated subsequently. The transformation group is the rigid transformation (rotation and translation). The relation between the measured feature location, {u, v, r}, and the model feature location, {x, y, z} (expressed in rectangular coordinates) is ⎡ ⎤ ⎡ ⎤ ⎤⎡ u1 u2 u3 u4 a x1 x2 x3 x4 ⎢ v1 v2 v3 v4 ⎥ ⎢ R b ⎥ ⎢ y1 y2 y3 y4 ⎥ ⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎣ r1 r2 r3 r4 ⎦ = ⎣ c ⎦ ⎣ z1 z2 z3 z4 ⎦ 1 1 1 1 1 1 1 1 0 0 01 such that R ∈ SO(3) (rotations), and {a, b, c} is a vector denoting a rigid translation of R3 . The trailing 1’s are included to permit writing the translation as a matrix product. As written, both the front and back of the object would appear in the image. The fact that a ladar cannot see the through an object is why the projection is referred to as 2.5D in Section 3.3.1. 3.5.2 Invariants and the Object–Image Relations Object–image relations express a geometric relation (constraint) between a 3D object and its image. This is a general approach and the particular invariants and number of points required for an invariant depends upon the transformation group associated with the sensor model. Although constructing invariants is difficult, once an invariant has been found it is typically simple to compute. Object–image (O-I) relations are an application of invariance theory as applied to the world viewed by a sensor (ladar in this case). O-I relations provide a formal way of asking, “What are all possible images of this object?” and “What are all possible objects that could produce this image?” Clearly, this is a very powerful formalism and it is well suited to object recognition. Recent research has yielded the fundamental geometric relation between “objects” and “images” (for RADAR, SAR, UHRR, EO, IR, and other sensors) [55, 56, 57, 58, 59, 52]. Although the object–image relations are very simple for ladar, they are useful for demonstrating the benefits of using covariants. Fundamentally the O-I relations can be viewed as the result of elimination of the unknown parameters in the model describing the projection of the 3D world onto the sensor. Thus, the ladar O-I relation can be derived by eliminating the group parameters associated with object motion, and the parameters associated with the ladar look direction (for this case, the group actions are the same). The result is an equation relating the object to its ladar range image written in terms of their associated invariants. More specifically, the equation relates 3D rigid invariants (inter-scatterer 3D Euclidean distances, determinants, or inner products) to rigid invariants measured from the ladar image.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
101
3.5.3 The ladar Object–Image Relation The above model of a ladar is essentially an orthographic projection from 3D to 3D. An invariant of 3D objects (undergoing rigid transformations) requires a minimum of four points (in general position), Pi = {Xi , Yi , Zi , 1}. Let the point Pi correspond to the i − th column of a 4 × 4 matrix. By translating and rotating appropriately, one can always transform the points into the standard position, ⎡ ⎤ 0 I1 I2 I3 ⎢ 0 0 I4 I5 ⎥ ⎢ ⎥ ⎣ 0 0 0 I6 ⎦ 1 1 1 1 where {I1 , I2 , I3 , I4 , I5 , I6 } are the object model invariants. They are invariants under 3D rigid transformations. Linear algebra shows a unique transformation (up to sign) exists to make this change of basis. It is not obvious without further explanation, but the invariants are functions of the Euclidean distance, determinants, and inner products. The range image can also be transformed to the standard position, ⎡ ⎤ 0 i1 i2 i3 ⎢ 0 0 i4 i5 ⎥ ⎢ ⎥ ⎣ 0 0 0 i6 ⎦ 1 1 1 1 where {i1 , i2 , i3 , i4 , i5 , i6 } are the image invariants. They are also invariant under 3D rigid transformations. The fundamental object–image relation can now be determined by solving the model projection equations with respect to the unknown rotation and translation. The resulting object–image relations are ij = Ij ,
∀j ∈ {1, 2, 3, 4, 5, 6}
(3.6)
An algorithm that uses these object–image relations does not need to worry about any rigid transformations. In other words, use of the object– image relations guarantees that all the continuous group actions that were modeled have been factored out of the resulting equalities. What remains are the discrete permutations, i.e., the correspondence and ordering of sets of features between the object and image. However the formulation, as presented above, is not feasible for two reasons: 1. Noise issues: As presented, these equations treat the initial points as “special.” A formulation is desired that treats all points equivalently, as this should inject some robustness to noise. 2. Complexity issues: Arnold [49] demonstrated that the complexity of a brute force search of the discrete permutations is not computationally feasible in general.
102
Gregory Arnold et al.
Therefore a modified formulation is desired that provides an invariant relation between the object and image, and also handles the noise and complexity issues. 3.5.4 Covariants Covariants provide an alternative approach to determining O-I relations by considering equivariant functions. This section will formalize the approach, including how projection is naturally handled in the technique. Let G be a Lie group (a continuous transformation group) acting on an n-dimensional manifold M , φ : G × M → M : (g, p) → gp. An important example of such an action is G = GL3 (R), the set of invertible matrices, 3 acting on the function space M = i=1 R3i . An element of this function space is simply an ordered triple,{pi }3i=1 , where each “point” pi is an ordered 3-tuple, {xi , yi , zi }T . Here the action is a componentwise matrix multiplication—hence the action is linear. A G-invariant function is a function Φ satisfying Φ(gp) = Φ(p) ∀p ∈ M ∀g ∈ G. The following figure characterizes an invariant function Φ under an action φ : G × Rn → Rn g ∈G n n
ℜ
ℜ
Φ
Φ
ℜ
where the horizontal arrow denoted by g is the induced map φg : Rn → Rn : p → φ(g, p). With respect to how this linear action is used in applications, view p as the object variables (i.e., 3D coordinates), and q = ρ(gp) as the image variables (i.e., 2D coordinates), where ρ is a projection from 3-space to 2-space, e.g., 100 ρ= 01c and where c is a fixed parameter. The variable q is embedding into 3-space— thus enabling standard invariant theoretic techniques. Embedding ρ → ρ, where ⎡ ⎤ 100 ρ = ⎣0 1 c⎦ 001 and noting that this gives an automorphism of G = GL3 (R) allows one to write q = (ρg)p = gp. Using the property characterizing an invariant function, Φ(gp) = Φ(p) ∀p ∈ M ∀g ∈ G, it follows Φ(q) = Φ(p). This equation provides the relationship between the “object” variables p and the “image” variables q. Thus the O-I relation is f (p, q) = Φ(q) − Φ(p) = 0. Computationally, this idea can be implemented by (1) finding the invariants of the given
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
103
group action, and (2) “equating invariants” Φ(q) = Φ(p), and (3) eliminating any artificial components associated with the embedding (in the example, an artificial component to q was introduced, namely the third component). Note that the invariants found in step one have already eliminated the group parameters from the second step. It should be noted that there are a set of fundamental invariants Φi i = 1, . . . , k for each group action. Hence in practice there is a set of O-I relations Φi (q) − Φi (p) = 0 for i = 1, . . . , k. The technique of Lie group analysis makes determination of the invariants relatively simple. This directly avoids the conceptually simple but often computationally difficult task of eliminating the group parameters and camera parameters directly. The latter approach naturally lends itself to consideration of covariant functions. A covariant involves two actions. The figure below characterizes the definition of a covariant function Φ under the two actions φ : G × Rn → Rn and ψ : G × R → R
ℜn
φ ∈G
Φ
ℜ
ℜn Φ
ψ
ℜ
where the two horizontal arrows correspond to the two induced maps φg : Rn → Rn : p → φ(g, p) and ψg : R → R : q → ψ(g, q). Only nontrivial choices of Φ, φ, and ψ are interesting. Choosing ψ as the identity is equivalent to absolute invariants. As with invariants, the goal is to find a basis from which all other covariants can be written. Vector-valued covariants, Φ : Rn → Rm , are a fairly simple extension that will be used below. The desire is to find a covariant with a small m. The defining property of a covariant function Φ is Φ(gp) = gΦ(p) ∀p ∈ M ∀g ∈ G. The covariants are constructed into a set of O-I relations Φi (gp) = gΦi (p) for i = 1, . . . , k. Similar to the case with invariant O-I relations, any artificial components associated with the embedding must be eliminated. Note that the group parameters have not been eliminated. The benefit of using covariants is that it allows one to explicitly solve (estimate) for the group transformation to move back to the standard position. This advantage follows from the discussion on alignment versus voting techniques in Section 3.3.3. Similarly, noise analysis is easier to perform in the original parameter space. Covariants arise with the group, En = SOn Rn , acting componenrigid m twise on the product space i=1 Rn . Consider the case n = 2. The two group actions are φ : (SO2 R2 ) × :
m
R2i → R2
i=1 i m ((A(θ), b), {p }i=1 )
→ {A(θ)pi + b}m i
104
Gregory Arnold et al.
and µ : (SO2 R2 ) × R2i → R2 : ((A(θ), b), q) → A(θ)q + b,
where A=
cos(θ) − sin(θ) , sin(θ) cos(θ) t b= x , ty
xi . p = yi
and
i
For brevity, denote p = {pi }m i=1 .
Object G
Space
G -1
Φ
Φ g -1
Cova rian t Spac e
g
Figure 3.6. Transformations in the object space induce transformations in the covariant space. The top row, from left to right, represents points in object space being translated and rotated by G. The bottom row shows the covariant (the dark point in the middle; the object points are shown for reference) representation, specifically the centroid for this example. From left to right, the centroid is transformed by g, the action induced by G, the object transformation. The inverse g −1 is readily calculated, and by applying G−1 to the object it can be returned to its canonical coordinate system.
This system gives the covariant function “centroid” m i 1 x i=1 Φ= . m i m i=1 y An example is shown in Fig. 3.6. This result easily generalizes to 3D. Similarly it can be shown that the eigenvectors are covariant under rotation in 3D. These two covariants form the basis of the object recognition algorithm to be described subsequently.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
105
Advantages of Covariants The advantage of using a covariant-based method is that for a large number of points the transformations can still be computed efficiently. Whereas invariants require searching pr combinations of features, where p is the number of image points and r is the number of features required to compute the invariants, the covariant-based technique scans the image (a neighborhood is defined by a sphere) thereby avoiding the combinatorial problem. Therefore, the covariant-based approach is computationally more efficient and more robust to noise. Also, as previously mentioned, the covariant-based approach is conducive to alignment versus voting techniques. Finally, noise analysis is easier to perform in the original parameter space. Covariants in the Algorithm Covariants are simply an (improved) approach to achieving invariance. Specifically, calculating covariants is an intermediate step toward transforming the selected data into a canonical coordinate system. To summarize, covariants appear in two places in the algorithm that follows: 1. Translation: A covariant of translation is the centroid. 2. Rotation: The eigenvectors are covariant with respect to rotation. 3.5.5 Articulation Invariants? Invariance can be used to reduce the articulation problem. When considering invariants, the imperative question is “what transformations should the function be invariant with respect to?” For instance, when the same object is viewed from different viewpoints, invariance with respect to the viewpoint transformation is useful. In ladar range images, the viewpoint invariants are the rigid invariants (in a rectangular coordinate system). This is a well-defined group of transformations and it applies to any (static) object in the ladar range images, i.e., the transformation group is independent of the object. Thus, the viewpoint invariants can be applied generically to all objects. When it comes to articulation this is no longer the case. Each object has different articulation degrees of freedom, i.e., a different transformation group. An object’s DoF (and therefore its transformation group) cannot be determined from a single image. Therefore, while it is mathematically possible to find articulation invariants for each individual object, generic articulation invariants that apply to all objects do not exist. Lacking articulation invariants, the goal is to turn as many of the articulation DoF into generic viewpoint DoF as possible. The remaining DoF are parameterized and used to define a manifold with respect to the canonical coordinate system.
106
Gregory Arnold et al.
3.5.6 Dividing the Object To simplify the articulation problem, viewpoint invariants are applied to parts of the object (subobjects). To avoid explicit segmentation, these smaller parts are not necessarily the “functional” object parts. They are arbitrary sections of the object as partitioned by a sphere of a certain center and radius. For example, one sphere may contain part of the body of a backhoe, and another may contain a joint of a backhoe’s arm. The sphere that contains a rigid body part has viewpoint DoF but no articulation. The sphere that contains a joint has both viewpoint and articulation DoF. However this joint has only one articulation parameter, namely the angle between the two segments of the arm. All other DoF of these arm segments have been turned into viewpoint DoF. In other words, using this approach, the articulation of each arm segment is independent of the backhoe’s body. The joint within each sphere is viewed as a separate object that is seen from an unknown viewpoint. Consequently dividing the backhoe into subobjects reduces the number of articulation DoF from ten to usually at most two. Viewpoint invariance methods can now be used for each subobject such as the joint, obtaining invariants that depend only on the angle of the joint. These invariants are a smooth function of the angle. A major advantage of the object division is the use of so-called “global” invariants of each subobject. A global invariant is a function that depends on the entire subobject rather than on isolated features such as points or lines, i.e., a global invariant is a function of all the voxels within the sphere. This achieves two purposes: (a) it avoids the problem of feature extraction, with the high sensitivity associated with feature-based methods; (b) it avoids calculating global invariants of the whole object, which would be sensitive to occlusion and missing parts. 3.5.7 Method Summary 1. Divide each modeled object into subobjects. 2. Find global viewpoint (rigid) invariants for each subobject as functions of its articulation parameters. 3. Use these invariant manifolds for storage, matching, and identification. A brief description of these steps follows in the next section.
3.6 Implementation 3.6.1 Object Division Each modeled object is divided into parts that are contained within spheres. Spheres are utilized since they are preserved under rotation. An important
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
107
question is how to choose the spheres’ centers and radii. Ideally all possible centers and radii would be used, but of course this is infeasible. Consequently, a finite set of radii starting from the biggest, containing the whole object, to the smallest, in which the data looks planar, are selected. This results in the description of the object on different scales. For each radius a set of centers is chosen. Sufficient numbers of centers are necessary to describe an object uniquely at a given scale. Although the object is 3D, its visible range image is described as a 2D array of ranges. Currently, a rectangular grid is draped over this array to describe where the sphere centers will be placed. Nominally, the grid spacing is some integer subsampling of the array at some fraction of the radius. The sphere centers are placed at the grid coordinates, {θ, φ, ρ}. Obviously the division is dependent on the image since each image will have a different grid and it is different from the grid on the image stored in the database. However, the invariant functions calculated on each sphere vary smoothly from one center to another. Consequently, it is easy to interpolate between grid points when matching is performed. 3.6.2 Finding Invariants Once the object has been partitioned into (generally overlapping) spheres, the invariants of these subobjects are calculated. Note that invariants of a 3D object as a whole cannot be used since only partial views of the object are visible to the range sensor. Thus, at a minimum, a few different views of each object, such as front, back, and sides are required. The views are chosen such that any other view has the same invariants as one of these views. Thus, invariants are calculated for representative views, at various scales, and stored in a database. These are used as reference “models.” There are several ways to calculate such invariants. Ideally, the ones chosen are the least sensitive to changes in the boundaries of the subobjects, resulting from changing the radius or center of the sphere. Currently, the covariants explained in Section 3.5.4 are used to transform the sphere into a canonical, or standard, coordinate system. Specifically, the centroid of the subobject is the canonical origin, and the eigenvectors form the canonical axes. The new origin and axes are independent of the viewpoint; therefore the new coordinate system is viewpoint invariant. See Figure 3.7 for a simplistic example. Transforming the subobject into this system, the grid point (sphere center) now has new coordinates that are invariant since they are given in the invariant coordinate system. Hence for each subobject (or sphere) three invariants are extracted, namely the 3D coordinates of the sphere’s center in the invariant coordinate system. By using invariants to describe every grid point, the full description of the object is invariant. This description does not depend on point features and is insensitive to occlusion. A noteworthy complication is the affect of image discretization. This approach assumes planar patches connect the data points in order to facilitate the necessary integration and normalization to marginalize the affects of discretization [3].
108
Gregory Arnold et al.
X
Figure 3.7. A 2D analogy to the ellipsoid fit. The dashed circle determines the data to be fitted by the ellipse. The axes of the ellipse (eigenvectors) define a canonical coordinate system. Thus any point X within the sphere is invariant in these coordinates.
The current approach and results shown here are based on this simple choice of invariants. This choice was made to provide maximum robustness to low numbers of pixels-on-target. As previously mentioned, all of the points are invariant once they have been expressed in the canonical coordinate system. The results are excellent; however, further discrimination and robustness would be achievable by using more information about the data. Obvious alternative approaches include using the spin images, spherical harmonics, or moments of the “sphere of data.” Ultimately, the chosen method should include a shape metric, such as the Procrustes metric presented in Section 3.4.2. 3.6.3 Indexing The invariants found above are functions of the articulation parameters. Denoting the invariants of each grid point by a vector x and the articulation parameters by a vector u, the invariants are x(u). Indexing amounts to inverting these functions, i.e., given the invariant coordinates x in the image, find the articulation parameters u(x). To do that, the above relations are represented as a surface in a hyperspace, namely f (x, u) = 0. To build this surface, the articulation parameters are varied, and for each vector u all the corresponding vectors x are found. In this hyperspace, the voxels lying on the surface f are marked by 1 and all other voxels are left as 0. This is a digital representation of the hypersurface. This is done off-line for every model in the database. The functions for all models are thus represented in the hyperspace. Thus indexing has been obtained such that given the invariant coordinates x, the corresponding models can be found with articulations u. This can be done by intersecting all the hypersurfaces with the hyperplane x = constant. For most points xi there will be relatively few corresponding models, since most models do not go through all points in the hyperspace even with articulation. Thus the indexing space is rather sparse. 3.6.4 Matching Given an image, the algorithm should match it to the closest model in the database. An initial scale is picked, which fixes the spheres’ radii. Then the
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
109
invariant coordinates xi are computed at each grid point as described above. Note that the grid used for the matching step can be much more sparse than the grid used to build the database. The next step is to find a model in the database with an articulation u that has the same invariant spatial coordinates xi . The algorithm starts with the invariant coordinates x1 of one point of the given range image. For the given point x1 , the surfaces u(x1 ) for all models in the neighborhood of this point are extracted. The whole hypersurface need not be extracted, only the portion in the neighborhood of x1 is necessary. Several techniques make it possible to reconstruct the surface in the neighborhood of x1 although the original surface was constructed using a slightly different grid [3]. Next, the intersection of all the model surfaces with the hyperplane x1 = constant is computed. This provides a list of all models with all articulation parameters uj that match the given point x1 . The process is repeated for another image point x2 . This new point will have a different set of matching models. Continuing with other points xi , all the models (and articulations) are collected in a voting table. Each additional point xi will contribute votes to certain models. The models with the most votes will be the best candidates for possible further verification.
(a)
(b)
(c)
(d)
Figure 3.8. Matching of a backhoe. In each row matching is done using a progressively smaller sphere radius. Column (a) shows crosses at the spheres’ centers. Column (b) shows representative subobjects (hatched areas). Column (c) shows crosses on a matching model. The invariant coordinates match in the neighborhood of each cross. Column (d) is a projection of the model to match the images’ pose.
110
Gregory Arnold et al.
3.7 Conclusions The current algorithm, as presented, efficiently handles nine of the ten primary technical challenges. These include translation, rotation, surface projection, surface resampling, varying numbers of pixels-on-target, point correspondence, obscuration, articulation, and fidelity. The final technical challenge, unknown objects, is partially achieved. The extensive verification step would reject unknown objects, but to do this reliably requires a metric. Ideally, the verification will contain more information than the location of the grid point in the canonical coordinate system. Substantially different surfaces could produce the same grid point, so shape metrics are required to remove this possibility and further refine the indexing procedure. The major contribution of this work is an approach that simultaneously addresses segmentation, recognition, and articulation in an efficient manner. The efficiencies are achieved by decomposing the image into subobjects, applying invariants, using a multiresolution decomposition, and hypothesis voting. The final algorithm explicitly avoids segmenting objects from the background, reduces the articulation parameters to a small number that are found within small subobjects, and encapsulates the benefits of a scanning-based approach (i.e., it is not combinatoric). This approach has demonstrated robustness in experiments.
3.8 Future Research Future research includes developing and testing an alignment-based technique for comparison with the current voting technique. Future goals include developing an improved data comparison technique that has the desirable properties of metrics. However, this will in turn require readdressing the questions of point correspondence, surface resampling, and varying numbers of pixelson-target. Ultimately, the goal is to minimize the overall computational cost for the same performance. The tradeoff is computational complexity versus saliency of the invariant features. Quantifying and qualifying the affects of noise on the system have not been studied extensively, and this is necessary in order to understand the robustness of the system. Initial results have been promising. At the lowest level, this would include estimating the confidence region of the range and angular measurements from the ladar (this is dependent upon the range and the relative surface orientation). Finally, a large-scale assessment of the algorithm on both background and modeled objects remains to be completed.
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
111
3.9 Acknowledgements This work was supported in part by the U.S. Air Force Office of Scientific Research under laboratory tasks 93SN03COR and 00MN01COR. The authors also acknowledge and appreciate the input of Dr. Isaac Weiss and Dr. Manjit Ray for their description of the algorithm and Figure 3.8. Dr. Peter Stiller has been instrumental in forming the viewpoint expressed in this chapter.
References [1] Jelalian, A.V.: Laser Radar Systems. Artech House (1992) [2] Ross, T.D., Bradley, J.J., Hudson, L.J., O’Conner, M.P.: SAR ATR — so what’s the problem? — an MSTAR perspective. In Zelnio, E., ed.: Proceedings SPIE International Conf. Algorithms for Synthetic Aperture Radar Imagery VI, Bellingham, WA, SPIE—The International Society for Optical Engineering (1999) [3] Ray, M.: Model-based 3D Object Recognition using Invariants. PhD thesis, University of Maryland, College Park (2000) [4] Small, C.G.: The Statistical Theory of Shape. Springer-Verlag New York (1996) [5] Papadimitriou, C.H., Stieglitz, K.: Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall (1982) [6] Ettinger, G.J., Klanderman, G.A., Wells, W.M., Grimson, W.E.L.: A probabilistic optimization approach to SAR feature matching. In Zelnio, E., Douglass, R., eds.: Proceedings SPIE International Conf. Volume 2757 of Algorithms for Synthetic Aperture Radar Imagery III., Bellingham, WA, SPIE—The International Society for Optical Engineering (1996) 318–329 [7] Hung-Chih, C., Moses, R.L., Potter, L.C.: Model-based classification of radar images. IEEE Transactions on Information Theory 46 (2000) 1842–1854 [8] van Wamelen, P.B., Li, Z., Iyengar, S.S.: A fast expected time algorithm for the 2D point pattern matching problem. see http://www.math.lsu.edu/˜wamelen/publications.html (2002) [9] Maciel, J., Costeira, J.P.: A global solution to sparse correspondence problems. IEEE Transactions on PAMI 25 (2003) 187–199 [10] Jones III, G., Bhanu, B.: Recognition of articulated and occluded objects. IEEE Transactions on PAMI 21 (1999) 603–613 [11] Hetzel, G., Leibe, B., Levi, P., Schiele, B.: 3D object recognition from range images using local feature histograms. In: Proc. IEEE CVPR. Volume 2., Seattle, WA, IEEE Computer Society (2001) 394–399 [12] Campbell, R.J., Flynn, P.J.: A survey of free-form object representation and recognition techniques. Computer Vision and Image Understanding 81 (2001) 166–210
112
Gregory Arnold et al.
[13] Arman, F., Aggarwal, J.K.: Model-based object recognition in denserange images: A review. ACM Computing Surveys 25 (1993) 5–43 [14] Bhanu, B., Ho, C.C.: CAD-based 3D object representation for robot vision. Computer 20 (1987) 19–35 [15] Piegl, L.A., Tiller, W.: The Nurbs Book (Monograph in Visual Communications). 2nd edn. Springer-Verlag (1997) [16] Fan, T.J., Medioni, G., Nevatia, R.: Recognizing 3D objects using surface descriptions. IEEE Transactions on PAMI 11 (1989) 1140–1157 [17] Solina, F., Bajcsy, R.: Recovery of parametric models from range images: The case for superquadrics with global deformations. IEEE Transactions on PAMI 12 (1990) 131–147 [18] Binford, T.O., Levitt, T.S.: Quasi-invariants: Theory and exploitation. In Firschein, O., ed.: DARPA Image Understanding Workshop Proceedings, Washington DC, Morgan Kauffman (1993) 819–830 [19] Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: A review. IEEE Transactions on PAMI 22 (2000) 4–37 [20] Moses, Y., Ullman, S.: Limitations of non-model-based recognition schemes. A.I. Memo 1301, Massachusetts Institute of Technology (1991) [21] Jacobs, D.W., Alter, T.: Uncertainty progation in model-based recognition. A.I. Memo 1476, Massachusetts Institute of Technology (1994) [22] Basri, R., Weinshall, D.: Distance metric between 3D models and 2D images for recognition and classification. A.I. Memo 1373, Massachusetts Institute of Technology (1992) [23] Grimson, W.E.L., Huttenlocher, D.P., Jacobs, D.W.: Affine matching with bounded sensor error: A study of geometic hashing and alignment. A.I. Memo 1250, Massachusetts Institute of Technology (1991) [24] Stockman, G.: Object recognition and localization via pose clustering. Computer Vision Graphics Image Processing 40 (1987) 361–387 [25] Huttenlocher, D.P., Ullman, S.: Object recognition using alignment. In: Proc. 1st Int. Conf. Computer Vision, Cambridge, MA, IEEE Computer Society (1987) 102–111 [26] Rigoutsos, I., Hummel, R.: A bayesian approach to model matching with geometric hashing. Computer Vision and Image Understanding 62 (1995) 11–26 [27] Lamdan, Y., Schwartz, J.T., Wolfson, H.J.: Affine invariant model-based object recognition. IEEE Trans. on Robotics and Automation 6 (1990) 578–589 [28] Sonka, M., Hlavac, V., Boyle, R.: Image Processing: Analysis and Machine Vision. 2nd edn. Brooks Cole (1998) [29] Binford, T., Levitt, T., Mann, W.: Bayesian inference in model-based vision. In Kanal, L., Levitt, T., Lemmer, J., eds.: Uncertainty in AI, 3, Elsevier (1989) 920–925 [30] Hummel, R.A., Landy, M.S.: A statistical viewpoint on the theory of evidence. IEEE Transactions on PAMI 10 (1988) 235–247
Chapter 3 Three-Dimensional Laser Radar Recognition Approaches
113
[31] Haykin, S.: Neural Networks: A Comprehensive Foundation. 2nd edn. Prentice-Hall (1999) [32] Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. 2nd edn. Wiley-Interscience (2001) [33] Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Proc. IEEE CVPR, New York, IEEE, Computer Society (1991) 586–591 [34] Pope, A.R., Lowe, D.G.: Probabilistic models of appearance for 3D object recognition. Int. Journal of Computer Vision 40 (2000) 149–167 [35] Dryden, I.L., Mardia, K.V.: Statistical Shape Analysis. John Wiley & Sons (1998) [36] Johnson, A.: Spin Images: A Representation for 3D Surface Matching. PhD thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (1997) [37] Funkhouser, T., Min, P., Kazhdan, M., Chen, J., Halderman, A., Dobkin, D., Jacobs, D.: A search engine for 3D models. ACM Transactions on Graphics 22 (2003) 83–105 [38] Lo, C.H., Don, H.S.: 3D moment forms: Their construction and application to object identification and positioning. IEEE Transactions on PAMI 11 (1989) 1053–1064 [39] Flusser, J., Boldyˇs, J., Zitov´ a, B.: Moment forms invariant to rotation and blur in arbitrary number of dimensions. IEEE Transactions on PAMI 25 (2003) 234–246 [40] Lele, S.R., Richtsmeier, J.T.: An Invariant Approach to Statistical Analysis of Shape. Chapman and Hall/CRC (2001) [41] Kendall, D.G.: Shape manifolds, procrustean metrics, and complex projective spaces. Bulletin of the London Mathematical Society 16 (1984) 81–121 [42] Bookstein, F.L.: Size and shape spaces for landmark data in two dimensions (with discussion). Statistical Science 1 (1986) 181–242 [43] Arnold, G., Sturtz, K., Velten, V.: Similarity metrics for ATR. In: Proc. of Defense Applications of Signal Processing (DASP-01), Adelaide, AU, AFOSR (2001) [44] Mumford, D.: Mathematical theories of shape: Do they model perception? In Vemuri, B.C., ed.: Proceedings SPIE, Geometric Methods in Computer Vision. Volume 1570., Bellingham, WA, SPIE—The International Society for Optical Engineering (1991) 2–10 [45] Csisz´ar, I.: Why least squares and maximum entropy? an axiomatic approach to inference for linear inverse problems. The Annals of Statistics 19 (1991) 2032–2066 [46] Santini, S., Jain, R.: Similarity measures. IEEE Transactions on PAMI 21 (1999) 871–883 [47] Wallis, G., Rolls, E.T.: Invariant face and object recognition in the visual system. Progress in Neurobiology 51 (1997) 167–194 [48] Weisstein, E.W.: The CRC Concise Encyclopedia of Mathematics. 2nd edn. CRC Press (2002) see http://www.mathworld.com.
114
Gregory Arnold et al.
[49] Arnold, G., Sturtz, K.: Complexity analysis of ATR algorithms based on invariants. In: Proceedings Computer Vision Beyond the Visible Spectrum (CVBVS), Hilton Head, SC, IEEE Computer Society (2000) [50] Zhang, X., Zhang, J., Walter, G.G., Krim, H.: Shape space object recognition. In Sadjadi, F., ed.: Proceedings SPIE International Conf. Volume 4379 of Automatic Target Recognition XI., Bellingham, WA, SPIE–The International Society for Optical Engineering (2001) [51] Levine, L., Arnold, G., Sturtz, K.: A label-invariant approach to procrustes analysis. In Zelnio, E., ed.: Proceedings SPIE International Conf. Volume 4727 of Algorithms for Synthetic Aperture Radar Imagery IX., Bellingham, WA, SPIE–The International Society for Optical Engineering (2002) 322–328 [52] Weiss, I.: Model-based recognition of 3D objects from one view. In: Proc. DARPA Image Understanding Workshop, Monterey, CA, Morgan Kauffman (1998) 641–652 [53] Weiss, I.: Noise-resistant invariants of curves. IEEE Transactions on PAMI 15 (1993) 943–948 [54] Rivlin, E., Weiss, I.: Local invariants for recognition. IEEE Transactions on PAMI 17 (1995) 226–238 [55] Stiller, P.F.: General approaches to recognizing geometric configurations from a single view. In: Proceedings SPIE International Conf., Vision Geometry VI. Volume 3168., Bellingham, WA, SPIE–The International Society for Optical Engineering (1997) 262–273 [56] Stiller, P.F., Asmuth, C.A., Wan, C.S.: Single view recognition: The perspective case. In: Proceedings SPIE International Conf., Vision Geometry V. Volume 2826., Bellingham, WA, SPIE–The International Society for Optical Engineering (1996) 226–235 [57] Stiller, P.F., Asmuth, C.A., Wan, C.S.: Invariants, indexing, and single view recognition. In: Proc. ARPA Image Understanding Workshop, Monterrey, CA (1994) 1432–1428 [58] Stuff, M.A.: Three dimensional invariants of moving targets. In Zelnio, E., ed.: Proceedings SPIE International Conf. Algorithms for Synthetic Aperture Radar Imagery VII, Bellingham, WA, SPIE–The International Society for Optical Engineering (2000) [59] Stuff, M.A.: Three-dimensional analysis of moving target radar signals: Methods and implications for ATR and feature aided tracking. In Zelnio, E., ed.: Proceedings SPIE International Conf. Algorithms for Synthetic Aperture Radar Imagery VI, Bellingham, WA, SPIE–The International Society for Optical Engineering (1999)
Chapter 4 Target Classification Using Adaptive Feature Extraction and Subspace Projection for Hyperspectral Imagery Heesung Kwon, Sandor Z. Der, and Nasser M. Nasrabadi U.S. Army Research Laboratory, ATTN: AMSRL-SE-SE, 2800 Powder Mill Road, Adelphi, MD 20783 {hkwon,sder,nnasraba}@arl.army.mil Summary. Hyperspectral imaging sensors have been widely studied for automatic target recognition (ATR), mainly because a wealth of spectral information can be obtained through a large number of narrow contiguous spectral channels (often over a hundred). Targets are man-made objects (e.g., vehicles) whose constituent materials and internal structures are usually substantially different from natural objects (i.e., backgrounds). The basic premise of hyperspectral target classification is that the spectral signatures of target materials are measurably different than background materials, and most approaches further assume that each relevant material, characterized by its own distinctive spectral reflectance or emission, can be identified among a group of materials based on spectral analysis of the hyperspectral data. We propose a two-class classification algorithm for hyperspectral images in which each pixel spectrum is labeled as either target or background. The algorithm is based on a mixed spectral model in which the reflectance spectrum of each pixel is assumed to be a linear mixture of constituent spectra from different material types (target and background materials). In order to address the spectral variability and diversity of the background spectra, we estimate a background subspace. The background spectral information spreads over various terrain types and is represented by the background subspace with substantially reduced dimensionality. Each pixel spectrum is then projected onto the orthogonal background subspace to remove the background spectral portion from the corresponding pixel spectrum. The abundance of the remaining target portion within the pixel spectrum is estimated by matching a data-driven target spectral template with the backgroundremoved spectrum. We use independent component analysis (ICA) to generate a target spectral template. ICA is used because it is well suited to capture the structure of the small targets in the hyperspectral images. For comparison purposes a mean spectral template is also generated by simply averaging the target sample spectra. Classification performance for both of the above-mentioned target extraction techniques are compared using a set of HYDICE hyperspectral images.
115
116
Heesung Kwon et al.
4.1 Introduction Progress in automatic target recognition (ATR) techniques is crucial to the rapid development of digital battlefield technologies. Targets are generally man-made objects whose constituent materials and internal structures are substantially different from natural objects (i.e., backgrounds). The differences in materials and appearance between the targets and backgrounds lead to distinctive statistical and structural characteristics in images. A large number of target classification/detection techniques have been developed based on broadband passive forward-looking infrared (FLIR) sensors that sense electromagnetic radiation in the 3–5 or 8–12 µm bands [1]. A commonly used multiple-band CFAR (constant false alarm rate) detection algorithm for a known signal pattern was introduced in [2]. An adaptive feature fusion technique has been recently developed, in which several local properties of the targets and backgrounds were jointly exploited within a local dual window and then fused to integrate different local features [3]. Statistical approaches have also been studied, in which the local statistical characteristics of the target and background regions were exploited by estimating, e.g., a probability density function [4], a co-occurrence matrix [5], or an eigen transformation matrix [6]. However, most previously attempted IR-based target detection techniques fail to produce satisfactory performance in the presence of high thermal clutter backgrounds or camouflaged targets. Hyperspectral imaging sensors have been used in many remote sensing applications, such as reconnaissance, surveillance, environmental monitoring, and agriculture, because they capture more spectral information than a broadband IR sensor. Spectral information is obtained through a large number (often over a hundred) of narrow contiguous spectral channels, each channel capturing electro-magnetic reflectance or emission within the corresponding spectral range. The basic premise of hyperspectral target classification is that the spectral signatures of target materials are measurably different than background materials, and most approaches further assume that each relevant material, characterized by its own distinctive spectral reflectance or emission signature, can be identified among a group of materials. Anomaly detectors assume only the first, while spectral matching algorithms assume the latter as well. We describe a two-class classification algorithm in which each pixel is labeled either target or background. The hyperspectral images we use were generated with the HYDICE (HYperspectral Digital Imagery Collection Experiment) sensor whose spectral range spans from 0.4 to 2.5 µm. In this spectral range the signal energy is dominated by reflected solar radiation rather than photon emission. The operation of the HYDICE sensor is normally limited to the daytime and the sensor cannot distinguish between hot and cold targets. Targets in our HYDICE images are combat platforms whose spectral characteristics depend largely on paint put onto a surface of the target.
Chapter 4 Target Classification Using Adaptive Feature Extraction
117
Based on the linear spectral mixing models discussed in [7], the algorithm assumes that a reflectance spectrum of each pixel is a linear mixture of constituent spectra from disparate material types present in the pixel area. If a pixel spectrum includes a target spectrum, the pixel is labeled a target pixel. The pixel is classified as the background, if the target spectral contribution within the pixel spectrum is zero. Consequently, decomposition of the pixel spectrum into the constituent spectra is required in order to determine whether the target spectrum is present within the corresponding pixel spectrum. In order to successfully unmix the pixel spectrum, the spectral characteristics of the constituent materials have to be known a priori or be obtained from the data. In the proposed algorithm we obtain spectral information of the material types directly from the data. The reflectance spectra in a spatial neighborhood, even from pixels within the same class of materials, are not identical mainly because of material variations and different illumination conditions. In addition, the background spectra generally consist of the combined reflectance spectra from a number of different natural objects, such as trees, grass, water, soil, etc. In order to address the spectral variability and diversity of the background spectra, a background subspace model is used to integrate key spectral signatures that spread over the background areas, into a subspace with substantially reduced dimensionality. Principal component analysis (PCA) is used to reduce the dimensionality of the original background space [8, 9, 10]; a relatively small number of significant eigenvectors are used to generate the background subspace. In the proposed algorithm a target spectral template is created based on sample spectra. The target spectral template is then matched with the target spectral portion of the input pixel spectrum. Two methods are used to generate the target spectral template. A target spectral template generated by averaging the target sample spectra will not closely represent the true spectral characteristics of the target. Instead, we use independent component analysis (ICA) [11, 12, 13] to generate the target spectral template. ICA is applied to a portion of the HYDICE training images containing a small number of targets. The associated basis vector of the independent component showing the highest target prominence, is considered to represent the spectral feature of the target. The target spectral portion within each pixel spectrum is obtained by removing the background spectral portion from the pixel spectrum. The removal of the background is performed by projecting the pixel spectrum onto the orthogonal background subspace. The orthogonal subspace projection for the subpixel target classification was first introduced by Harsanyi and Chang [14]. The amount of target spectrum within each pixel spectrum is then estimated by measuring the correlation between the target spectral portion and the target spectral template obtained from the ICA method. This chapter is organized as follows. The HYDICE imaging system is briefly introduced in Section 4.2. A linear spectral mixing model for sub-
118
Heesung Kwon et al.
pixel target classification is described in Section 4.3. In Section 4.4 we present a brief introduction to ICA and ICA-based feature extraction for the hyperspectral data. In Section 4.5 an adaptive target classification algorithm based on orthogonal subspace projection is presented. Localized subspace estimation for the background is also introduced in Section 4.5. Two-class target classification results for the HYDICE images using the proposed algorithm are presented in Section 4.6. Conclusions are summarized in Section 4.7.
4.2 Hyperspectral Imaging System Hyperspectral sensors operate in a variety of spectral bands, such as the visible, short-wave IR (SWIR: 1–2.5 µm), mid-wave IR (MWIR:3–5 µm), and long-wave IR (LWIR: 8–12 µm). The spectral range of the HYDICE sensors spans the visible and short-wave IR wavelengths where reflected solar radiation rather than object emission is the predominant source of energy. In the HYDICE imaging system, a group of spatially registered images, called a hyperspectral cube, is taken by an imaging spectrometer at a spectral range of 0.4 to 2.5 µm with a step size of 10 nm. The imaging spectrometer splits the reflected light (e.g., reflected sunlight and skylight) into narrow contiguous spectral channels, each channel generating an associated band image, as shown in Figure 4.1. Two-hundred-ten images are generated over the whole spectral range. For every pixel in the cube a spectral curve is then formed, which represents the spectral characteristics of the corresponding pixel, as shown in Figure 4.2.
4.3 Spectral Mixing Model Airborne or tower-based hyperspectral imaging sensors normally view large areas on the ground, so the spatial resolution of the sensors is coarse in many applications. The area covered by each pixel in the hyperspectral images includes several constituent materials rather than a single material type. Consequently, a pixel spectrum is a mixture of the spectral signatures of the different constituent materials present. The detection (or classification) problem that deals with a mixed-pixel (subpixel) model has been intensively researched [7, 14, 15, 16]. We propose a target classification algorithm based on the spectral mixing model that estimates the abundances of the constituent spectra via spectral unmixing and matching to find whether the spectrum of the material of interest is present. Our algorithm focuses on a two-class classification problem in which the pixel spectrum is considered a linear combination of individual spectra of two different material types — the target and background. In order to effectively address the spectral variability of the background spectra, we estimate a low-dimensional background subspace. The subspace is spanned by linearly
Chapter 4 Target Classification Using Adaptive Feature Extraction
119
Band images 0.4 µm
Reflected light from a scene
2.5 µm
Imaging spectro− meter
0.4 µm
HYDICE cube
2.5 µm
Figure 4.1. Hyperspectral imaging system
independent vectors b1 , b2 , . . . , bK , which are the first K eigenvectors of a background covariance matrix; K is equal to the intrinsic dimensionality of the original background space. The target spectrum varies, due mainly to the material variations and different illumination conditions. However, a single target spectral feature vector st , an M × 1 column vector, may still be capable of representing the target spectral characteristics in the data; M is the number of the spectral bands. Based on [17], the pixel spectrum x, an M × 1 column vector, is given by st a + Bc + n, if 0 < a < 1, (4.1) x= Bc + n, if a = 0, where a is a scalar for target abundance, B = [b1 b2 · · · bK ] is an M × K matrix, c = (c1 , c2 , . . . , cK )T , 0 ≤ c1 , c2 , . . . , cK < 1, is a K-dimensional column vector whose components are the coefficients that account for the abundances of the corresponding endmember spectra b1 , b2 , . . . , bK , and n is an M -dimensional column vector, representing Gaussian random noise. If x is
120
Heesung Kwon et al. Reflectance curves
Background
2.5 µm
Target 0.4 µm
10 nm HYDICE images
Figure 4.2. Creation of spectral reflectance curves from a hyperspectral cube.
the pixel spectrum from the target regions, the target abundance a is a nonzero value. On the other hand, a is zero if x is from the background region; in this case, x is a mixture of the background spectra only, as described in Equation (4.1). An accurate decision for the presence of the target spectrum within the pixel spectrum, therefore, can lead to successful classification of the individual pixel spectra. The target feature vector st can be obtained from a spectral library or spectral samples in the given hyperspectral data. In the proposed algorithm the target feature is created from the sample spectra via an ICA-based feature extraction technique introduced in Section 4.4.
4.4 ICA-Based Feature Extraction ICA, a relatively new concept in data representation, exploits high-order statistical dependencies among the data [11, 12, 13]. In this section, we introduce the ICA mixing and unmixing models and an ICA-based feature extraction technique. The feature extraction technique is based on the well-known ICA unmixing process [12], designed to find a feature vector that can closely represent the target spectral characteristics of the given data.
Chapter 4 Target Classification Using Adaptive Feature Extraction
121
4.4.1 Independent Component Analysis (ICA) Data representation in signal processing generally entails efficient removal of correlations present in the data. Principal component analysis (PCA) [8] is a well-known data transform technique that decorrelates the input data by exploiting pairwise second-order dependencies (e.g., covariance) and produces linearly independent variables, called principal components (or eigenvectors). Because the eigenvectors are only linearly independent from one another, nonlinear (high-order) statistical dependencies of the input data — as can be easily observed from sparse structures in an image, such as edges and small areas with relatively high contrast — cannot be removed [18]. ICA is a generalization of PCA where the input data is expressed as a linear mixture of statistically independent components which are nonlinearly decorrelated [11]. The independent components are, therefore, highly nongaussian (e.g., the Laplace distribution) and well suited to represent the sparse structures of the input data. We use ICA to better capture targets scattered in HYDICE images, each target occupying a small number of pixels. Detailed information on ICA and its applications for signal and image processing can be found in [19, 11, 12, 13, 20, 21, 18]. Suppose we are given the data X, consisting of M input images; the number of pixels in each image is N . Then X can be represented by an M × N matrix [x1 x2 · · · xN ], each row, a N -dimensional vector, representing the corresponding input image. Each column xi = (x1 , x2 , . . . , xM )T is an M -dimensional vector whose components are the pixel values of the M images at the corresponding pixel location i. We denote by S = [s1 , s2 , . . . , sM ]T an M × N matrix, each row si T , an N -dimensional vector, representing the corresponding independent component image. The ICA mixing model is then defined by X = AS, (4.2) where A = [a1 , a2 , . . . , aM ]T represents the M × M mixing matrix, each row ai T , an M -dimensional basis vector, representing mixing coefficients that account for the abundances of the independent components for the ith input image. Equation (4.2) indicates that any given input image can be expressed as a linear combination of the independent component images whose abundances are specified in the corresponding row of A. Figure 4.3 shows a simplified schematic diagram of the ICA mixing and unmixing processes. In order to calculate A, we first estimate the unmixing matrix W of X using the basic assumption of ICA — the statistical independence of the components. Let the unmixing matrix W = [w1 , w2 , . . . , wM ]T be the inverse of the mixing matrix A. Equation (4.2) can then be expressed as W X = S.
(4.3)
Figure 4.4 illustrates how matrix multiplications in Equation (4.3) are performed to unmix X. The unmixing process is basically a linear projection
122
Heesung Kwon et al.
MIXING PROCESS
A M I X I N G M A T R I X
x
UNMIXING PROCESS =
S
W
x
X
=
S
U N M I X I N G
? ? ?
M A T R I X
? ? −1
W
Figure 4.3. Simplified schematic diagram for the ICA mixing and unmixing processes
of X onto wi T , i = 1, 2, . . . , M , which are the basis vectors (axes) in a new coordinate system represented by Equation (4.3). wi T are actually the orthogonal basis vectors, and the input images are realigned along wi T to create the corresponding ICA component images. The independent components should be nongaussian, otherwise ICA cannot be achieved [11]. Accordingly, the directions of wi T are set such that nongaussianity of the ith independent component si = wi T X is maximized. 4.4.2 ICA-Based Target Feature Extraction Sample reflectance spectra collected from the pixels in the target regions (the targets in the scene are assumed to be the same kind) can be used directly to estimate the target spectral signature, e.g., by averaging them. However, in practice the material variations and different illumination conditions result in a broad range of spectral variability, as shown in Figure 4.5. Therefore, the individual spectra from the target regions, even from the same region, could be different. In this context the arithmetic mean of the sample reflectance spectra is not an adequate representation of the target spectral signature. We use an ICA-based feature extraction technique that focuses especially on extracting
Chapter 4 Target Classification Using Adaptive Feature Extraction
X
W wT1 wT2 wT3
x1 x2 x3 x4
=
123
S s T1 s T2 s T3
xN
wMT
s MT
Pixel vectors
Hyperspectral cube Figure 4.4. Matrix multiplications for the ICA unmixing process. 700
600
500
400
300
200
100
0
0
50
100
150
Figure 4.5. Example of spectral variability of the target reflectance spectra.
the independent components that form the target. In this technique a small hyperspectral cube is used to generate the target spectral feature vector. Suppose we are given a relatively small hyperspectral cube X = [x1 x2 · · · xN ], an M × N matrix, including the targets and the neighboring backgrounds. Normally, before unmixing X two preprocessing steps need to be taken — dimensionality reduction and whitening. Both the steps are based on the eigenvalue–eigenvector factorization described in [11]. The number of independent sources present in the data is closely associated with the intrinsic dimensionality L of the input data. L can be successfully determined by finding an optimum subspace of the input data, which is equivalent to a problem of finding a number of eigenvectors to be retained. Important studies have been performed to determine L based on probabilistic PCA models [9, 22]. We use the maximum likelihood PCA model introduced by Bishop [9] to cal-
124
Heesung Kwon et al.
culate L. L is normally far smaller than M ; the input vectors in X are the column vectors xi = (x1 , x2 , . . . , xM )T , i = 1, 2, . . . , N . L has to be known a priori to decide how many independent components need to be created in the unmixing process. The dimensionality reduction of X is normally accomplished by projecting X onto the first L eigenvectors of the covariance matrix of X. Whitening is a process in which the input data are uncorrelated and their variances become unity. Both the dimensionality reduction and whitening are performed by Xs = U −1/2 E T X,
(4.4)
where Xs =[xs1 xs2 · · · xsN ], an L × N matrix, is the transformed input data in the L-dimensional space, E is an L × M matrix, representing the first L eigenvectors, and U is a diagonal matrix whose main diagonal elements are the eigenvalues of E. The corresponding independent components Ss = [ss1 , ss2 , . . . , ssL ] can be obtained by W s X s = Ss ,
(4.5)
s where Ws = [w1s w2s · · · wL ] is the unmixing matrix in the L-dimensional space. Figure 4.6 illustrates the dimensionality reduction of the sample hyperspectral cube and the associated ICA unmixing process. Each independent component image represents a set of corresponding projection values si = wi T X. The distributions of si are close to the Laplace distribution which has more pixels in tails and less pixels around the mean than the Gaussian distribution. Therefore, edges and small areas with high contrast are better represented by the independent component images. The structural details of the small targets in the sample hyperspectral cube are detected during the unmixing process, generating one or more ICA component images with greater target prominence (See the ICA component inside the rectangular box in Figure 4.6). The target prominence is measured by calculating the difference between the two values — the averages of the background pixels and target pixels in the ICA component image. As shown in Figure 4.6, the component images with higher target prominence include highly suppressed background where the mean and variation in pixel values are generally low. Once the component image with the highest target prominence ss ∗ is found, the corresponding ws ∗ T , the unmixing vector onto which Xs is projected to create ss ∗ , is identified. The associated dewhitened vector of ws ∗ is then obtained by g∗ = EU 1/2 ws ∗ . (4.6)
gi , i = 1, 2, . . . , L, are the column vectors of the mixing matrix A (gi are shown in Figure 4.6). Each component of the M -dimensional vector g∗ accounts for the abundance of ss ∗ in the associated band image. Therefore, g∗ naturally represents the spectral characteristics of the target relative to those of the background in the given data. If the background spectral characteristics in
Chapter 4 Target Classification Using Adaptive Feature Extraction
125
X Xs in lower dimension
30th
Ws in lower dimension
ICA components s S
Dewhitened vectors in original dimension
0.5
0
0
−50
−0.5
50th
−1
−100 1
2
3
4
5
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0.5
*
0
S
−0.5 −1
80th
P C A
0
g*−50 −100
1
2
3
4
5
1
0
0.5 0
−50
−0.5
−100 1
2
3
4
5
1
0
100th
0 −1
−50 −100 1
2
3
4
5
1
0
0.5
−50
0
130th
−0.5
1
2
3
4
5
−100
Unmixing Figure 4.6. Schematic diagram of ICA-based feature extraction via the ICA unmixing process.
test images are not substantially different from those of the training images, we can apply g∗ as a spectral template to the test images. If the targets are frequently situated among different background types, e.g., grass and soil, we generate multiple spectral templates, each template being adapted to the corresponding background type.
4.5 Subspace-Based Adaptive Target Classification In this section, we estimate a low-dimensional subspace to address the background spectral variability. The subspace is spanned by a small number of linearly independent vectors (eigenvectors) directly estimated from the background spectra. Each pixel spectrum is projected onto the orthogonal background subspace to remove the spectral contribution due to the background materials. The target abundance of the pixel spectrum is then estimated by adaptive template matching.
126
Heesung Kwon et al.
4.5.1 Background Subspace Model Suppose x = (x1 , x2 , . . . , xM )T is an M -dimensional vector, representing a background reflectance spectrum. We denote by Cx the covariance matrix of the vector population of x. M is equal to the number of spectral bands. We estimate the low-dimensional subspace B spanned by first K eigenvectors B = [b1 , b2 , . . . , bK ] of Cx ; K is equal to the dimension of the subspace (K M ), and is set such that B accounts for most of the energy of the original background space. The construction of B ensures that the most of the spectral information from the input space is integrated into the compact subspace with greatly reduced dimensionality. 4.5.2 Orthogonal Subspace Projection and Adaptive Spectral Matching Orthogonal subspace projection is a geometrical approach originally devised to determine the least squares approximations to inconsistent linear algebraic equations (or linear systems) [23]. It has also been used for hyperspectral target classification/detection applications [14, 16]. Because the pixel spectrum is assumed to be a linear mixture of the target and the background constituent spectra, and B is spanned by the key eigenvectors of the background covariance matrix, we can estimate the background portion Bc within the pixel spectrum x by projecting x onto the background subspace B, as shown Figure 4.7. The spectral contribution by the target and noise w is then estimated by simply subtracting the background contribution PB x from x: x − PB x = (I − PB )x, = PB ⊥ x,
(4.7a) (4.7b)
where I is the identity matrix and PB = B(BT B)−1 BT is a background projection matrix, and PB ⊥ is an orthogonal projection matrix. For every pixel, adaptive template matching is performed to estimate the correlation between the target spectral contribution PB ⊥ x within the pixel spectrum and the target spectral template g∗ T , which was obtained from Section 4.4.2: F (x) = g∗ T PB ⊥ x.
(4.8)
The pixels with higher values of F (x) than a predefined threshold are then classified as the target. 4.5.3 Localized Background Subspace The background subspace is estimated based on the sample spectra collected from various terrain types in the hyperspectral training images. Successful subspace modeling ensures robust and enhanced classification performance.
Chapter 4 Target Classification Using Adaptive Feature Extraction
Pixel reflectance spectrum x
127
xB :background portion x − xB : target portion
e = x − xB
Background subspace
E1 Ej xB Ei
E2
Figure 4.7. Orthogonal background projection.
Collecting the background spectra from the training images is normally performed by a human operator. Figure 4.8 shows the various background regions from which the sample spectra were collected. However, human intervention in a target classification process often results in inconsistent performance. It can be eliminated by replacing it with an automated procedure, in which the background subspace is estimated using the sample spectra directly collected from the hyperspectral sensor. In this section we introduce an unsupervised segmentation technique developed by the authors [24] to estimate the background subspace adapted to the local characteristics of the hyperspectral test images. In the technique the reflectance spectra in the test images are first segmented into two classes based on spectral dissimilarity. The spectral dissimilarity is measured within a local window. A relatively large fixed-size local window is placed around each pixel. Note that each pixel corresponds to a pixel vector (spectrum) in the spectral domain. The spectral dissimilarity di associated with pixel location i is defined as j∈B sj − si , (4.9) di = Ni where · represents the Euclidean norm, B represents a set of pixels selected randomly from within the local window, as shown in Figure 4.9. The vectors si and sj represent the corresponding pixels at locations i and j, respectively, and Ni represents the number of the selected pixels (the randomly selected pixels and neighboring pixels) required to estimate di . After the spectral dissimilarity of every pixel is obtained, a spectral dissimilarity image D is formed, as shown in Figure 4.10. Each element of the image represents the amount of the average spectral difference between the center pixel and its randomly selected neighbors. The spectral dissimilarity image provides a set of spectral-
128
Heesung Kwon et al.
Figure 4.8. Sample background spectra from the hyperspectral images to estimate the sample-based background subspace; the samples were collected from the rectangular regions.
feature values di suitable for clustering. The pixels in the targets regions tend to have greater values than those of the background pixels, because of the spectral difference between the two material types. The pixels associated with smaller di are then identified as the background whose spectra are used to estimate the background subspace. Segmentation is performed by applying a simple thresholding method to the feature values in D whose statistics are closely associated with the contents of the HYDICE images — mainly the complexity of the backgrounds. The threshold is calculated by averaging the mean and the maximum value of D.
Chapter 4 Target Classification Using Adaptive Feature Extraction
129
Local window Neighboring pixels
Center pixel
Randomly selected pixels Figure 4.9. Randomly selected pixels and neighboring pixels within the local window for calculation of spectral dissimilarity.
Figure 4.10. Spectral dissimilarity image D of the small hyperspectral cube.
4.6 Experimental Results In this section we apply the proposed ICA-based adaptive matching technique to the HYDICE images to detect targets of interest. The HYDICE imaging sensor generates 210 band images across the whole spectral range (0.4–2.5 µm), each band covering a narrow spectral range of 10 nm. We, however, use only 150 band images by discarding water absorption and low signal-to-noise ratio (SNR) bands; the band images used are the 23rd–101st, 109th–136th, and 152nd–194th. The low SNR bands were identified by human observation of sample spectra. The test cubes were provided with the associated ground truth maps. The coordinates of the centers of the targets in the ground truth map were compared with those of the detected targets to check if the classification was accurate.
130
Heesung Kwon et al.
4.6.1 Target Feature Extraction Figures 4.11–4.13 show typical reflectance curves from the regions of the disparate material types, such as targets, trees, and grass. Even though a distinguishable spectral pattern can be found among a group of reflectance curves of the same material type, a wide range of spectral variability, prevailing in all the material types, hinders correct and accurate classification of the targets from the surrounding backgrounds. In particular, the reflectance curves from the target region, as shown in Figure 4.11, display substantial intraclass variability, showing how seriously material variations and different illumination conditions interfere with the hyperspectral target classification process. Hyperspectral Cube
20
40
60
80 100 20
40
60
80 100
Spectral Reflectance
Pix Value
400 300 200 100 0
20
40
60
80 Band #
100
120
140
Figure 4.11. Spectral variability in the target region.
In order to estimate the target abundance within the individual pixel spectra, we generate a spectral feature vector, used as a correlation template in the classification process, to represent the target spectral characteristics in the given data. Two methods are used to create the spectral feature vector — mean curve generation and ICA-based feature extraction. We compare the target classification performance associated with the two feature generation methods. Figure 4.14 shows the sample target spectral reflectance curves and the corresponding mean spectral curve mTt . The ICA-based spectral feature extraction technique, based on [20], uses a set of training images X = [x1 x2 · · · xN ] which are usually a small-sized hyperspectral cube with the single background type, as shown in Figure 4.6. The intrinsic dimensionality L of the input vectors xi (the pixel spectrum vector) was obtained based on the maximum likelihood PCA model [9]; L was 5 instead of 150, meaning that the column vector xsi , i = 1, 2 . . . , N , are in the 5-dimensional space, and there exists not more than five independent sources in the training im-
Chapter 4 Target Classification Using Adaptive Feature Extraction
131
Hyperspectral Cube
20
40
60
80 100 20
40
60
80 100
Spectral Reflectance 400
Pix Value
300 200 100 0
20
60
40
100
80 Band #
120
140
Figure 4.12. Spectral variability in the tree region. Hyperspectral Cube
20
40 60
80
100 20
40
60
80 100
Spectral Reflectance 500
Pix Value
400 300 200 100 20
40
60
80 Band #
100
120
140
Figure 4.13. Spectral variability in the grass region.
ages. Figure 4.6 shows the training images X and a schematic diagram of the ICA unmixing process, in which the training images are decomposed into the five ICA component images. In the unmixing process the first five principal component images Xs were actually used to calculate the corresponding ICA component images. The direction of each of the five unmixing vectors wis T , i = 1, 2, . . . , 5, was set such that nongaussianity of the corresponding ICA component ssi was maximized. The 150-dimensional vectors gi were then obtained by first dewhitening wis and then projecting them back to the original measurement space. The ICA component image with the highest target prominence ss ∗ and its associated basis vector g∗ T are shown in Figure 4.15.
132
Heesung Kwon et al.
700
700
600
600
500
500
400
400
300
300
200
200
100
100
0
0
50
100
0
150
0
50
(a)
100
150
(b)
Figure 4.14. (a) Sample target reflectance curves and (b) the corresponding mean curve mTt .
g∗ T is used as a target spectral template in the target classification process
5 10 15 20 25 10
20
30
40
50
60
70
(a) 40 20 0 −20 −40 −60
0
50
100
150
(b) Figure 4.15. (a) ICA component image with the highest target prominence and (b) its corresponding basis vector g∗T .
to measure the significance of the target contributions within the individual background-removed pixel spectra. The spectral shape of g∗ T is quite different from that of mTt . This is because the direction of g∗ T is set such that it suppresses the background spectra, while emphasizing the target spectra. mTt , in contrast, is generated only to detect the target spectra, regardless of the spectral characteristics of the backgrounds of the given hyperspectral data.
Chapter 4 Target Classification Using Adaptive Feature Extraction
133
4.6.2 Background Subspace Estimation Figure 4.16 shows two hyperspectral band images, each from either of the two hyperspectral cubes, Cube I and II, to be used as test images for target classification. Cube I includes five targets (four vehicles and one man-made
(a)
(b)
Figure 4.16. Sample band images from two hyperspectral test cubes, (a) Cube I and (b) Cube II.
object) and two different types of background (trees and grass). Cube II is smaller than Cube I in size and includes three targets (all of them are vehicles) and the single background type of grass. We first estimate the background subspace to remove the background portion from every pixel spectrum in the test cubes. Two different techniques were used to estimate the background subspace: localized subspace estimation and sample-based subspace estimation. In order to estimate the localized subspace Bl , we first apply the unsupervised segmentation, described in Section 4.5.3, to the corresponding hyperspectral test cube. For each test cube, Bl is spanned by the first eight eigenvectors of the covariance matrix of the pixel spectra in the regions classified as the background. Figure 4.17 shows the spectral dissimilarity matrices and unsupervised segmentation results for the two test cubes. The size of the local sliding window was half the size of the test image. We used 80 randomly selected pixels and four neighboring pixels of the corresponding input pixel in the local window to estimate the spectral dissimilarity. A simple threshold-based technique was used to segment the dissimilarity matrices. The segmentation of Cube II was successful since most of the background areas were segmented out of the image. Cube I includes more diverse backgrounds than Cube II. In the segmentation of Cube I the area covered by grass was successfully classified as background, while most of the tree regions were not. This is mainly because of the irregular spectral reflectivity of the tree regions. Accordingly, for Cube I most of the background spectra used to estimate the localized subspace were collected from the grass areas. The sample-based background subspace Bs was estimated for both the test cubes based on the sample spectra collected from various types of the
134
Heesung Kwon et al.
(a)
(b)
(c)
(d)
Figure 4.17. Spectral dissimilarity images for (a) Cube I and (b) Cube II and the corresponding segmentation results for (c) Cube I and (d) Cube II.
backgrounds in the hyperspectral training images (inside the rectangular boxes), as shown in Figure 4.8; the same number of eigenvectors used to estimate Bl was used to estimate Bs . It should be noted that modeling of Bs involves a human operation, incurring inconsistent classification performance. Once Bl and Bs are estimated, the associated orthogonal projection matrixes PBl ⊥ and PBs ⊥ , respectively, are readily calculated. Figure 4.18 shows three band images from Cube I and their corresponding backgroundremoved images, in which the background portion of every pixel spectrum is discarded by projecting the pixel spectrum x onto PBl ⊥ and PBs ⊥ . The background removal was quite successful for both the subspace estimation methods. The localized subspace estimation is preferable for the given test cubes because it provides comparable performance to that of the samplebased method and allows automated target classification. However, as HYDICE images include increasingly complex backgrounds (e.g., urban areas), the dissimilarity measure for the localized subspace estimation fails to work properly because of a high level of the spectral dissimilarity inside the background. In general, the choice between the sample-based and the localized subspace estimations needs to be made in the context of background complexity. Therefore, it is necessary to further develop an appropriate measure of the background complexity for hyperspectral imagery as future work.
Chapter 4 Target Classification Using Adaptive Feature Extraction
135
Figure 4.18. Example of the background-removed images based on the localized and sample-based subspace estimation. The images in the first column are the three band images from Cube I. The images in the second and third column are the corresponding background-removed band images based on PBl ⊥ x and PBs ⊥ x, respectively.
4.6.3 Adaptive Target Classification After removing the background portion from every pixel in the hyperspectral test cube, we estimate the target abundance within the pixel spectrum by calculating mTt PB ⊥ x and g∗ T PB ⊥ x; mTt and g∗ T serve as the target spectral templates. Quantitative performance of the proposed algorithm, such as the receiver operating characteristic (ROC) curves, was not evaluated mainly because of a lack of HYDICE images. Figure 4.19 shows the target abundance images and the corresponding classification results for Cube I and Cube II using the mean-based spectral template mTt , which is shown in Figure 4.14. The sample-based background subspace Bs was used in the mean-based target classification. Because the mean curve cannot inherently address a wide range of spectral variability of the target, only partial targets were detected for both
136
Heesung Kwon et al.
(a)
(b)
(c)
(d)
Figure 4.19. Target classification based on the mean-based spectral template and the sample-based subspace estimation. (a) Target abundance image and (b) classification results for Cube I. (c) Target abundance image and (d) classification results for Cube II.
the test cubes. Figures. 4.20 and 4.21 show the target abundance images and the corresponding classification results using the ICA-based spectral template g∗ T for both the sample-based and the localized background subspace models, respectively. A complete set of the targets was successfully detected for both the subspace models, demonstrating the superiority of the ICA-based classification over the mean-based technique.
(a)
(b)
(c)
(d)
Figure 4.20. Target classification results based on the ICA-based spectral template and the sample-based subspace estimation. (a) Target abundance image and (b) classification results for Cube I. (c) Target abundance image and (d) classification results for Cube II.
Chapter 4 Target Classification Using Adaptive Feature Extraction
(a)
(b)
(c)
(d)
137
Figure 4.21. Target classification results based on the ICA-based spectral template and the localized subspace estimation. (a) Target abundance image and (b) classification results for Cube I. (c) Target abundance image and (d) classification results for Cube II.
The ICA-based target classification without background removal from the individual spectra was also applied to the test cubes by calculating g∗ T x. The experiments were performed to find how significantly the background removal process contributed to the target classification process. The projection g∗ T x was not able to suppress the background portion within the pixel spectra, drastically increasing the false classification in the background regions, as shown in Figure 4.22.
4.7 Conclusions We have presented a two-class target classification algorithm, in which an individual pixel spectrum is linearly decomposed into constituent spectra and then labeled either target or background. The background and target spectral characteristics are represented by the low-dimensional background subspace and the target feature vector generated by the ICA-based feature extraction, respectively. The background subspace model has been used to address a wide range of spectral variability by integrating key spectral features of various terrain types into the low-dimensional subspace. In order to fully automate the target classification process, the localized background subspace model has been also developed, in which the eigenvectors spanning the subspace are calculated based on the reflectance spectra directly from the background regions of the test images. Unsupervised segmentation based on the spectral dissimilarity is applied to the test images to identify the potential background regions. In our
138
Heesung Kwon et al.
(a)
(b)
(c)
(d)
Figure 4.22. Target classification results without background removal. (a) Target abundance image and (b) classification results for Cube I. (c) Target abundance image and (d) classification results for Cube II.
experiments, the localized and sample-based subspace models in the proposed classification technique provides comparable classification performance. The background portion is removed from each pixel spectrum by projecting it onto the orthogonal background subspace. The background removal process greatly improves target classification performance, as supported by the simulation results. The abundance of the remaining target spectral portion is estimated by measuring the correlation between the background-removed individual pixel and the target template to identify the intrinsic spectral nature of the corresponding pixel surface. The target features obtained by both the ICA-based and mean-based feature extraction techniques are used as the target spectral templates. The nonlinear data structures of the targets are exploited by the ICA-based feature extraction. Consequently, ICA-based feature extraction results in better classification performance than mean-based extraction. Due to the lack of HYDICE images available for experiment, only the qualitative performance was evaluated. In order to further investigate the usefulness of the proposed algorithm, a quantitative performance measure such as the ROC curve needs to be calculated using a larger data set that includes blind test images. The types of paint used for the targets in our data set is unknown (it may be the three-color camouflage paint). Analysis of the spectral characteristics of different types of paint on the surface of military targets is desirable and needs to be conducted as future work. It will help develop robust tar-
Chapter 4 Target Classification Using Adaptive Feature Extraction
139
get detection/classification methods that can be used in various battlefield environments, e.g., wooded (and/or grassy) areas and barren desert.
4.8 Acknowledgment This research was sponsored by the U.S. Army Research Laboratory (ARL) and was accomplished under the ARL/ASEE postdoctoral fellowship program, contract DAAL01-96-C-0038. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of ARL or the U.S. government. The U.S. government is authorized to reproduce and distribute reprints for government purposes, notwithstanding any copyright notation herein.
References [1] Bhanu, B.: Automatic target recognition: State-of-the-art survey. IEEE Transactions on Aerospace and Elect. Syst. 22 (1986) 364–379 [2] Reed, I.S., Yu, X.: Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution. IEEE Transactions Acoustics, Speech and Signal Process. 38 (1990) 1760–1770 [3] Kwon, H., Der, S.Z., Nasrabadi, N.M.: Adative multisensor target detection using feature-based fusion. Optical Engineering 41 (2002) 69–80 [4] Schachter, B.J.: A survey and evaluation of FLIR target detection/segmentation algorithm. In: Proc. of DARPA Image Understanding Workshop. (1982) 49–57 [5] Aviram, G., Rotman, S.R.: Evaluating human detection performance of targets and false alarms, using a statistical texture image metric. Optical Engineering 39 (2000) 2285–2295 [6] Chan, L., Nasrabadi, N.M., Torrieri, D.: Eigenspace transformation for automatic clutter rejection. Optical Engineering 40 (2001) 564–573 [7] Manolakis, D., Shaw, G., Keshava, N.: Comparative analysis of hyperspectral adaptive matched filter detector. In: Proc. SPIE. Volume 4049. (2000) 2–17 [8] Joliffe, I.T.: Principal Component Analysis. Springer-Verlag, New York (1986) [9] Bishop, C.M.: Baysian PCA. In: Neural Information Processing System. Volume 11. (1998) 382–388 [10] Rajan, J.J., Rayner, P.: Model order selection for the singular value selection and the discrete Karhunen-Loeve transform using a bayesian approach. IEE Proc. Image Signal Process 144 (1997) 116–123 [11] Hyv¨ arinen, A., Oja, E.: Independent component analysis: Algorithms and applications. Neural Networks 13 (2000) 411–430
140
Heesung Kwon et al.
[12] Hyv¨ arinen, A.: Survey on independent component analysis. Neural Computing Surveys 2 (1999) 94–128 [13] Hyv¨ arinen, A.: Sparse code shrinkage: Denoising of nongaussian data by maximun likelyhood estimation. Neural Computation 11 (1999) 1739– 1768 [14] Harsanyi, J.C., Chang, C.I.: Hyperspectral image classification and dimensionality reduction: An orthogonal subspace projection approach. IEEE Transactions Geosci. Remote Sensing 32 (1994) 779–785 [15] Ashton, E.L.: Detection of subpixel anomalies in multispectral infrared imagery using an adaptive bayesian classifier. IEEE Transactions Image Process. 36 (1998) 506–517 [16] Chang, C.I., Zhao, X.L., Althouse, M., Pan, J.J.: Least squares subspace projection approach to mixed pixel classification for hyperspectral images. IEEE Transactions Geosci. Remote Sensing 36 (1998) 898–912 [17] Manolakis, D., Shaw, G.: Detection algorithms for hyperspectral imaging applications. IEEE Signal Processing Magazine 19 (2002) 29–43 [18] Bartlett, M.S.: a dissertation: Face Image Analysis by Unsupervised Learning and Redundancy Reduction. University of California, San Diego (1998) [19] Comon, P.: Independent component analysis: A new concept? Signal Processing 36 (1997) 287–314 [20] Hyv¨ arinen, A.: Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions Neural Networks. 10 (1999) 626–634 [21] Lee, T.W., Girolami, M., Sejnowski, T.J.: Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Computation 11 (1999) 417–441 [22] Minka, T.P.: Automatic choice of dimensionality for PCA. M.I.T Media Laboratory Perceptual Computing Section Technical Report (2000) [23] Strang, G.: Linear Algebra and Its Applications. Harcourt Brace & Company (1986) [24] Kwon, H., Der, S.Z., Nasrabadi, N.M.: An adaptive unsupervised segmentation algorithm based on iterative spectral dissimilarity measure for hyperspectral imagery. In: Proc. SPIE. Volume 4310. (2001) 144–152
Chapter 5 Moving Object Detection and Compression in IR Sequences Namrata Vaswani, Amit K Agrawal, Qinfen Zheng, and Rama Chellappa Center for Automation Research, University of Maryland, College Park, MD, {namrata,aagrawal,qinfen,rama}@cfar.umd.edu
Summary. We consider the problem of remote surveillance using infrared (IR) sensors. The aim is to use IR image sequences to detect moving objects (humans or vehicles), and to transmit a few “best-view images” of every new object that is detected. Since the available bandwidth is usually low, if the object chip is big, it needs to be compressed before being transmitted. Due to low computational power of computing devices attached to the sensor, the algorithms should be computationally simple. We present two approaches for object detection — one which specifically solves the more difficult long-range object detection problem, and the other for objects at short range. For objects at short range, we also present techniques for selecting a single best-view object chip and computationally simple techniques for compressing it to very low bit rates due to the channel bandwidth constraint. A fast image chip compression scheme implemented in the wavelet domain by combining a non-iterative zerotree coding method with 2D-DPCM for both low-and high-frequency subbands is presented. Comparisons with some existing schemes are also included. The object detection and compression algorithms have been implemented in C/C++ and their performance has been evaluated using the Hitachi’s SH4 platform with software simulation.
5.1 Introduction Remote monitoring of activities of stationary or moving vehicles and humans is a critical component in surveillance applications. The sensors used in practice are typically of low quality and the available bandwidth for transmission is quite limited. We consider the problem of remotely monitoring a battlefield with IR sensors and present two approaches for object detection. The first approach is useful when the objects are at large distances, are very small, appear
Prepared through collaborative participation in the Advanced Sensors Consortium sponsored by the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD19-01-2-0008. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation thereon.
141
142
Namrata Vaswani et al.
to move slowly, and their signatures have low contrast over the background. In such cases, traditional methods based on the analysis of the difference between successive frames and/or image and background or image intensity change will not work. We present a novel algorithm (referred to as the MTI algorithm) based on variance analysis which is useful at long ranges, when the object is small and slowly moving. The algorithm uses temporal variance to detect potential moving spots, spatial variance analysis to suppress false alarms caused by sensor vibration, and object geometrical constraints to filter out false alarms caused by tree branches and sensor noise. In other scenarios, when the objects are not so far away, the problem of moving object detection is formulated as one of segmenting an image function using a measure of its local singularity as proposed in [1]. When the detected objects are very small (see Figures. 5.1, 5.2), all views are equally good or bad and hence the problem of best-view selection becomes irrelevant. Also, since the object chip is already very small, compression is not necessary and hence any chipped image of the object can be transmitted. Thus, in such cases all the computational power available on the sensor can be used to solve the object detection problem. When the object chips are larger so that over time the object pose changes, we choose one best-view of the object, compress it using computationally simple techniques and then transmit the compressed best view chip. The challenge here is to solve the best view selection problem and to develop techniques which can compress the image to very low bit rates and are yet computationally simple since they have to be implemented in realtime and on hardware with limited computing power. Note that in this work, we do not study techniques for channel coding since we assume a reliable channel is available for transmission. Performance evaluation of algorithms is critical for determining the memory and computation needs of the algorithm. We present these measures for the algorithms using the Hitachi’s SH-4 microprocessor. The rest of the chapter is organized as follows. Section 5.2 describes the MTI object detection algorithm for long-range, small-size object detection along with results followed by the technique for detecting short-range and larger objects. Section 5.3 describes techniques for best-view selection for large objects. Section 5.4 discusses compression techniques for real-time, low-power, and very low bit rate compression of object chips. Simulations results on SH4– 7751 microprocessor for performance characterization are presented in Section 5.5 and conclusions in Section 5.6.
5.2 Object Detection We present two object detection algorithms, the first one deals with the more difficult problem of long-range object detection when objects are very small and the second one for detecting objects at short range.
Chapter 5 Moving Object Detection and Compression in IR Sequences
143
5.2.1 Long-Range Object Detection (MTI) Techniques based on optical flow and image intensity analysis may not work well for long-range object detection. In typical scenarios, a moving object size can be as small as 2×3 pixels and the motion between two adjacent frames can be less than 0.1 pixels. Several challenging issues need to be addressed. The first challenge is to compensate for sensor motion using electronic stabilization methods. Over the years, two dominant approaches have been developed — flow-based and feature based. The feature-based methods extract point, edge or line features and solve for an affine transformation between successive frames, while the flow-based methods compute optical flow and then estimate an affine model. These methods do not perform well in IR images as feature extraction in IR images is not reliable, and flow estimates for IR images suffer from severe bias due to noise. Also, these methods are too complex for real-time computation. Even if stabilization can be solved using one of these approaches, the problem of separating moving objects from the stabilized background is challenging due to low signal-to-clutter ratio in surveillance applications. Other factors that complicate the problem are changes in the background as the sensor is panning, false motion detection due to atmospherics, and other confusing motion (motion of tree leaves, etc). We assume a stationary sensor. The proposed algorithm uses a variance analysis based approach for moving object detection that can effectively integrate the object motion information over both temporal and spatial domains. Each input frame is first checked for possible errors and then used to update the temporal variance. When the variance for a pixel increases above a given threshold, the pixel is labelled as corresponding to a potential moving object. All candidate-moving pixels are then grouped to form disjoint moving objects. The detected potential moving objects (regions) are verified for their size, shape, and orientation to filter out false detection. After false detection verification, the remaining change regions are reported as moving objects. The temporal variance at a pixel (i, j) at frame k is computed as 2 σi,j,k = Si,j,k − µ2i,j,k ,
where
1 2 L−1 fi,j,k + Si,j,k−1 , L L 1 L−1 µi,j,k−1 , = fi,j,k + L L
(5.1)
Si,j,k =
(5.2)
µi,j,k
(5.3)
2 , Si,j,1 = fi,j,1
(5.4)
µi,j,1 = fi,j,1 .
(5.5)
Here, fi,j,k is the image value at frame k and location (i, j) and L is the temporal window parameter. A pixel is detected as belonging to potential moving object if the following conditions are satisfied:
144
Namrata Vaswani et al. 2 2 σi,j,k ≥ σi,j,k−1 , 2 σi,j,k
≥ T0 (i, j) + Th ,
(5.6) (5.7)
or 2 2 σi,j,k ≥ σi,j,k−1 ,
max
−1≤δx ≤1,−1≤,δy ≤1
(5.8)
2 σi,j,k
≥ T0 (i, j) + Tl ,
(5.9)
2 σi+δ x ,j+δy ,k
≥ T0 (i, j) + Th ,
(5.10)
where T0 (i, j) =
max
|fi+δx ,j+δy ,0 − fi,j,0 |,
(5.11)
σt = vari,j∈N ×N (T0 (i, j)), Th = γh × σt , Tl = γl × σt .
(5.12) (5.13)
−2≤δx ≤2,−2≤δy ≤2
(5.14)
In our experiments, we set γh = 3.4, γl = 0.8 ∗ γh , and L = 16. 5.2.2 MTI Algorithm Results Figure 5.1 shows the object detection result for frame 200 and 300 on an IR sequence (IRseq10) for the case of people walking at a long distance. Even though the width of the objects (people) is 2 to 3 pixels, the algorithm correctly detects and tracks them. In this sequence, the sensor was stationary and its direction was fixed. Figure 5.2 shows the detection of a moving vehicle for frames 160 and 250. In this case the sensor pans and stops and is quickly able to find the object again. Hence, the algorithm is capable of recovering after sensor panning motion. As a comparison with background subtraction techniques, Figure 5.3 shows the result using background subtraction based on nonparametric background modelling [2] on IRseq10. We can see that there are a lot of false detections whose sizes are the same or even greater than the object size. So, the objects cannot be detected reliably. Also background subtraction techniques need some number of frames for background modelling before they can start the detection process. Figure 5.4 shows the ROC curve for the MTI algorithm for different IR sequences for γh varying from 3.0 to 5.5. The detection percentage is calculated as the ratio of object detections in all frames and the total objects actually present in all frames. The false alarms percentage is calculated as the ratio of the false alarms in all frames and the total objects in all frames. Since we do not have ground truth, it was generated manually as follows. For all the sequences considered, the objects are far away and are moving approximately parallel to the image plane with uniform speed. The objects locations were hand-picked every thirty frames and are interpolated
Chapter 5 Moving Object Detection and Compression in IR Sequences
145
Figure 5.1. Object detection: Frames 200 and 300 of IRseq10 along with detected objects.
Figure 5.2. Frames 160 and 250 of a moving vehicle IR sequence along with detected objects (sensor recovering after panning motion).
in between assuming constant object velocity. Table 5.1 shows the effect of parameter L on detection rates and false alarms for different sequences. The MTI algorithm has been implemented on a Dual Pentium 550-MHz personal computer. It can perform moving object detection for 974×436 image sequences at a sustained rate of 12 frames per second (including reading data from the hard drive and sending image to a video display). The algorithm has been tested on fourteen 974 × 436 × 3600 sequences with objects at various distances, moving speeds, moving directions, and motion patterns, and with different numbers of moving objects. From the ROC curves (Figure 5.4) and
146
Namrata Vaswani et al.
Figure 5.3. Detection result using background subtraction on IRseq10. ROC for MTI algorithm 1
0.9 IR seq 1 IR seq 2 IR seq 9 IR seq 10 IR seq 14
0.8
0.7
Detection %
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5 0.6 False Alarms %
0.7
0.8
0.9
1
Figure 5.4. ROC curve for the MTI algorithm for different IR sequences.
Chapter 5 Moving Object Detection and Compression in IR Sequences
147
Table 5.1. Effect of parameter L on detections and false alarms for various IR sequences. Sequence
L
IRSeq1 IRSeq1 IRSeq9 IRSeq9 IRSeq2 IRSeq2
8 16 8 16 8 16
Detections(%) 98.33 99.83 97.70 97.70 99.80 99.60
False Alarms(%) 13.8 28.3 2.24 0.05 32.09 57.33
Table 5.1, we see that the MTI algorithm gives good results (high detection rate and very low false alarm rate) on almost all of the test sequences with no running parameters to be adjusted. 5.2.3 Short-Range Object Detection The problem of object detection is formulated as one of segmenting an image function using a measure of its local singularity as proposed in [1]. The method combines the problem of stabilization, object detection, and tracking into a single process when interframe motions are restricted to lateral translations or tilts and scale changes, and has the advantage of exploiting these sensor motion constraints for performing simultaneous activity detection and stabilization. The algorithm makes use of the Holder exponent of a hybrid capacity (derivative of Gaussian along the X- and Y- axis). Using this measure, Lipschitz signatures which reflect the singularity of the image function along each spatial axis are defined. The Lipschitz signatures are used for detection and tracking of objects. The proposed measure is obtained by applying the operators Gx,σ and Gy,σ to the images which are the derivatives of the Gaussian applied along the x- and y- axes, respectively. The Lipschitz signatures can then be defined as the projection of the these measure along the x- and y- axes. The main assumption of the algorithm is that the “active regions” of the image exhibit some higher level of singularity in the Lipschitz signatures. In other words, the singularities can be detected and tracked over time. The algorithm is robust to image scale variations and can handle multiple moving objects. It also involves projection along spatial axis and hence can be done in real time. Spatio-temporal information on the objects in the scene can also be inferred.
5.3 Best View Selection As mentioned earlier, the best-view selection is not required when the detected object chip is small (few pixels), but becomes important in case the object chip is large and the object pose changes from time to time.
148
Namrata Vaswani et al.
Three different techniques were tried for best-view selection. We work on the assumption that the side view is the “best view” since it has most of the identifiable features (See Figure 5.5). Eigenspace classification has successfully been used for pose detection [3] and face recognition applications [4, 5]. In the first approach, we formulate the best-view selection problem as a pose matching problem in eigenspace. Another approach to best-view selection would be to wait while the object size keeps increasing (it is approaching the sensor) and transmit the largest-sized image or transmit the last frame before the object takes a turn. This can done by estimating the focus of expansion (FOE) which can be used to calculate the velocity direction. Since both techniques mentioned above are computationally intensive, they are not suitable for realtime hardware implementation. The eigenspace technique also suffers from the drawback that it is not generalizable, i.e., to use it for a different type of vehicle would require a new training phase. It works well only when similar objects are available in its training database. Hence, the third approach, a size-based detection method, was finally implemented. In what follows, we describe these techniques in detail. 5.3.1 Eigenspace Classification This technique is useful for best-view selection when there are multiple sensors capturing an object from different orientations and a database of multiple views of the object is available. In our experiments we have used multiple views of tanks taken from different directions. A view closest to the side view is classified as the best view.
Figure 5.5. Tank2 (side view).
Algorithm Construct an eigenspace using images of tanks in various orientations. Place the camera at 45-degree spacings around the tanks to obtain eight possible
Chapter 5 Moving Object Detection and Compression in IR Sequences
149
views of each tank. Obtain the mean image of each of the eight orientations and save its coordinates in eigenspace. For a query image, classify it in eigenspace and calculate the distance from each of the orientations using a distance metric. Since the required data for doing this was not available, we constructed an eigen-space using all the tank images available and obtained class means corresponding to front, side and back views. On classification of a query image, it got mapped to the correct class most of the times. Eigenspace Construction Instead of directly obtaining the eigenvectors of an N 2 ×N 2 covariance matrix, the first M (M is the number of training vectors) eigenvectors can be obtained by first calculating the eigenvectors of an M × M covariance matrix of the transpose of the image data and then obtaining the image eigenvectors by taking linear combinations of the training images weighted by the eigenvector component as described in [4, 5, 6]. Scale and Intensity Invariance Since eigenspace classification is sensitive to scale variations, all images were scaled down to a fixed size before classifying and since the image chip contained the tank only, scaling down the image to a fixed size implied obtaining the tank image of a fixed size. Both for eigenspace construction and classification, the images were normalized by their total energy. Distance Metrics The distance metric can either be the simple Euclidean distance (ED) or the distance along each component normalized by the corresponding eigenvalue (ND). The latter gives better results since it gives more weight to those directions where the noise variance is lower. A better solution is to obtain variance along each component in each class and calculate the distance from a particular class using those eigenvectors which have low variance in that class but overall high variance in eigenspace. For scaling the distance from class k, the variance in class k is used rather than the global eigenvalue for scaling. In this way, the intraclass variance can be suppressed while the interclass variance can be emphasized. We call this measure the class normalized distance (CND). All the three distance metrics for some sample images are tabulated in Table 5.2. It can be seen from the results that the CND is the best metric for classification for the reason stated above.
150
Namrata Vaswani et al. Table 5.2. Eigenspace Classification Results Class tank2 tank6 tank9 btank12 sftank5
Class Normalized Distance(CND)
Euclidean Distance(ED)
Normalized Distance(ND)
4 15 15 30 17
10 20 19 44 21
24 35 28 56 31
5.3.2 Focus of Expansion Estimation The focus of expansion is the point in the image sequence of a moving body from which all the motion vectors appear to diverge. The FOE may not always lie inside the image boundary. Mathematically, the FOE(Xf ,Yf ) is defined as Xf =
Tx , Tz
(5.15)
Yf =
Ty , Tz
(5.16)
where Tx ,Ty , Tz are the translational motion vectors. Therefore, assuming the ground is the X-Z plane, the direction of motion (velocity angle) of the tank is given by Vx Tx θ = tan−1 = tan−1 = tan−1 Xf . (5.17) Vz Tz A modification of the partial search technique for FOE estimation developed by Srinivasan in [7] was used. The equations to be solved are u(x, y) = −(x − xf )h(x, y) + xyωx − (1 + x2 )ωy + yωz ,
(5.18)
v(x, y) = −(y − yf )h(x, y) + (1 + y 2 )ωx − xyωy − xωx ,
(5.19)
where u(x, y) and v(x, y) are the optical flow estimates at (x, y), xf , yf are the x and y coordinates of the FOE in the image(pixel) coordinates, h(x, y) is the inverse of the depth at point (x, y) and ωx , ωy , ωz are the x-, y-, and z- direction rotations. The FOE estimation algorithm requires calculation of the optical flow which is done using the overlapped basis functions technique developed by Srinivasan and Chellappa [8]. The FOE in the world coordinates is given by xf + xoffset − (N − 1)/2 Xf = (5.20) f which is used for direction of motion calculation in (5.17). In the modified partial search FOE estimation technique, we use only the optical flow estimates of the part of the image containing the moving object
Chapter 5 Moving Object Detection and Compression in IR Sequences
151
since the flow estimates of the background are unreliable. Since the size of the image for which optical flow is available is smaller, the FOE is also searched over a smaller region. This speeds up the FOE calculation. Also, the FOE estimate of the previous frame can be used to select an initial offset for the FOE in the current frame to speed up the FOE calculation over successive frames. The direction of motion is estimated at regular intervals and we keep waiting if the tank is approaching towards the camera. The frame at which the tank takes a turn away from the camera or the tank’s size increases above a certain threshold could be chosen as the “best view” in this approach. 5.3.3 Size-Based Best-View Selection Since the application requires best view selection and compression to be done in realtime on hardware with low computing power, we need very simple techniques for best-view selection. Thus, the final algorithm that was implemented simply waits till the size of the image chips exceeds a predefined threshold. If the size starts decreasing, it simply chooses the maximum size frame in the last 90 frames and sends it. Actually a single frame buffer is used to store the maximum sized chip in the last few frames, and as soon as the size starts decreasing or increases beyond the threshold the stored frame is compressed and transmitted. The algorithm rejects chips which are very close to the image boundary (the tank may not be complete). Spurious frames (produced as a result of wrong object detection) are also rejected based on thresholding the height-to-width ratio. 5.3.4 Best-View Selection Results In this section, we present the results of best-view selection using the three approaches discussed above. Eigenspace Classification An eigenspace of front, side, and back views of various tanks is constructed and the class means for each class are precalculated. In Table 5.2, results for distances from the perfect side view (“tank2”) class are shown. Figure 5.6 shows the mean image of the tank2 class. The distance of a query tank2 image (side view, see Figure 5.5) is a minimum. Distance of tank6 (side-back view) shown in Figure 5.8 is higher than tank2 but lower than tank12 (back view) shown in Fig. 5.7. Hence tank2 is the “best-view” in this case. As can be seen from Table 5.2, the CND (class normalized distance) has the maximum variation (4 for perfect side view and 30 for back view), and thus it is the best metric for classification among the metrics used. The eigenspace classification is not a very good method for best-view selection because it is sensitive to the lighting conditions, the type of IR sensor
152
Namrata Vaswani et al.
Figure 5.6. Mean image of the itank2 class.
Figure 5.7. Query image 3: tank12 (back view).
Figure 5.8. Query image 2: tank6 (back-side view).
used, and to the scale of the image. If the actual sensor is different from the sensor used in the database, classification could fail. Moreover a very large database of tank images in various poses is required for a robust eigenspace construction which may not be feasible. Focus of Expansion Estimation The FOE estimation algorithm was run on IR video sequences of the tanks. Since most of the tanks are moving almost horizontally in front of the camera in the test sequences, the FOE values are very large (tending to infinity). The FOE in pixel coordinates for a sample sequence is shown in Table 5.3.
Chapter 5 Moving Object Detection and Compression in IR Sequences
153
Table 5.3. FOE estimates using the optical flow shown in Figure 5.10. Frame No.
F OEx
F OEy
121 123 125 127 129 131
−244 −369 −201 −253 −553 −583
145 152 140 136 110 110
5.9. 0
50
100
150
200
0
50
100
150
200
250
300
Figure 5.9. Optical flow estimate of a typical frame (125) for FOE Estimation.
The optical flow estimate for the full frame 125 is shown in Figure The region of significant motion that is segmented out and used for FOE estimation finally is shown in Figure 5.10. The FOE estimation technique is not very suitable for our application because both optical flow calculation and FOE estimation are computationally intensive. Also for the IR images, the optical flow estimates are not very accurate and as a result the FOE estimates are also not accurate enough. Size-Based Best-View Selection Figure 5.11 shows a best view selected by this approach.
154
Namrata Vaswani et al. 0
10
20
30
40
50
60
70
80
90 0
20
40
60
80
100
120
140
160
180
200
Figure 5.10. Segmented optical flow estimate for frame 125 which is finally used in FOE estimation (magnified view).
Figure 5.11. Size-based best-view selection.
5.4 Compression For long-range objects, since the detected image chip is already very small, compression is not really necessary. Also since the object chip contrast can be very low compared to the background, it is preferable to just transmit the binary image of the object (instead of transmitting a grayscale one) which itself provides an 8 : 1 compression ratio. If more computational power is available, the binary image could be compressed by run-length coding else just the raw bits can be transmitted. For objects at short range (which are large), the image chip chosen by the “best-view selection” algorithm has to be compressed before transmission.
Chapter 5 Moving Object Detection and Compression in IR Sequences
155
Since the available channel bandwidth is low, we have tried to develop compression schemes which can provide very high compression ratios while at the same time maintaining a reasonable image quality. Since the algorithm is to be implemented in real-time using limited computing hardware, the computational complexity should be low. We have developed realtime algorithms for image compression in the wavelet domain. We first provide a background of existing image compression schemes and discuss the theoretical background for wavelet transforms and the compression techniques. Following that, we discuss the compression schemes implemented and compare against other techniques. We compare our scheme, combined zerotree and DPCM coding, against three existing schemes all of which have low computational complexity, viz. scalar quantization (SQ), zerotree coding, and DPCM coding. The more efficient compression schemes like JPEG and LZW are not compared here because they have a much higher computational cost associated with their implementation. We then provide a performance analysis of the coding schemes. Finally, experimental results are provided followed by computational cost analysis. We use PSNR (peak signal-to-noise ratio) which is a standard metric for image compression schemes to compare decompressed image quality. 5.4.1 Previous Work A wavelet zerotree coding scheme for compression is presented in [9], but since it uses vector quantization (VQ), it cannot be used for our application due to high computational cost. Reference [10] presents an embedded predictive wavelet image coder. But it uses arithmetic coding which is not suitable for implementation on a embedded processor such as Hitachi’s SH4. So we have developed algorithms using the Haar wavelet transform followed by noniterative zerotree coding [9] and 2D-DPCM for all subbands. The computational complexity of these algorithms is only marginally higher than simple scalar quantization (SQ) of the entire image. 5.4.2 Wavelet Transform Properties The wavelet transform is an atomic decomposition that represents a signal in terms of shifted and dilated versions of a prototype bandpass wavelet function and shifted versions of a low-pass scaling function. In discrete time, the bandpass wavelet function is a high-pass filter at different scales and the scaling function is a low-pass filter. The image is low-pass and high-pass filtered first along rows and then along columns to generate LL, LH, HL, and HH images each of which is subsampled by two. This process is repeated on the subsampled LL image. The wavelet transform has the following properties which make it suitable for compression. • Multiresolution: The image is decomposed into wavelets at N scales, and only the top few coarsest scales need to be transmitted to obtain a reasonable image quality. Depending on the available channel bandwidth, more
156
•
•
Namrata Vaswani et al.
finer scale coefficients can be transmitted to improve the reconstructed image quality. Entropy Reduction: The wavelet transform of a real image generates a large number of small coefficients (which can be set to zero) and a small number of large coefficients which can be encoded. This property is based on the fact that a real world image will not have information in all frequencies at all points in space. At most points except edges, the higher-frequency information is almost zero. Clustering and Persistence: The wavelet transform attempts to decorrelate the image, but the decorrelation is not complete (since the filters are constant, not data dependent). There is a residual dependency between adjacent coefficients at the same scale (clustering) and between coefficients in adjacent scales but the same spatial location (persistence). Our coding schemes attempt to remove these correlations in the image.
5.4.3 A-DPCM for Scaling Coefficient(LL) Encoding The scaling coefficients (LL subband) contain the maximum information and thus more bits are allocated for its encoding. But it is also the most highly correlated subband and this fact can be exploited to maximize compression. An adaptive-DPCM scheme is used for encoding the LL subband. The current pixel is predicted based on a linear combination of three causal nearest ˆ is obtained as neighbors. The predicted value of the pixel, X ˆ = l(Q) ¯ = w. ¯= X ¯Q wk Qk . (5.21) The predictor coefficients w ¯ are calculated to minimize the mean squared prediction error as ¯ ¯Q ¯ T )−1 E(X.Q). (5.22) w ¯ = E(Q where X is the pixel to be predicted, Qi are the quantities based on which the pixel would be predicted (in this case the nearest neighbors), and w ¯ are the predictor coefficients. Instead of quantizing the pixel value, the error between ˆ is quantized, which requires fewer the actual and the predicted value (X − X) bits since the error would be much smaller than the original pixel value if the prediction is good. Calculation of LMSE predictor coefficients can be done offline on a set of similar images. 5.4.4 Zerotree Coding In multiresolution wavelet decomposition, each coefficient Xi , except those in the LL subband and the three highest subbands, is exactly related to 2 × 2 coefficients of the immediately higher subband. These four children coefficients correspond to the same orientation and spatial location as the parent coefficient Xi . Each of the four children coefficients is in turn related to 2 × 2
Chapter 5 Moving Object Detection and Compression in IR Sequences
157
coefficients in the next higher subband, and so on. These coefficients are collectively called the descendants of the parent Xi . All coefficients with magnitude less than threshold T are called insignificant coefficients and their collection is known as a zerotree. In order to obtain a real-time implementation [9], the search for insignificant coefficients is started in the lowest-frequency subbands except baseband and continued in higher-frequency subbands. When a coefficient is decided as insignificant and set to zero, all its descendants are also set to zero. Thus, one needs to transmit only the escape code for the zerotree root vector besides encoding the nonzero coefficients. The zerotree root positions at each scale can be encoded efficiently using the Run-length Coding(RLC). The non-zero coefficients can be scalar quantized and transmitted. This type of simple threshold based zerotree coding, RLC and SQ are computationally simple algorithms for hardware implementation. However, it is possible that there are significant descendants even though their parent is insignificant. These mispredictions are inevitable in a noniterative search method, but the conditional probability of this happening is very small, as discussed in [9]. The misprediction error can be reduced by predicting the value of a “zero” coefficient based on its nearest nonzero neighbors (causal and noncausal) while decoding. 5.4.5 DPCM on Wavelet Coefficients The zerotree coding exploits the persistence property of wavelet coefficients. But there is also a residual correlation in the high-frequency subbands, especially the LH and the HL bands with horizontal and vertical neighbor, respectively. Hence applying an A-DPCM scheme like that discussed for the LL subband can give additional compression. Also while obtaining the prediction value for the current pixel we can exploit both clustering and persistence properties, i.e., obtain a prediction for the current pixel based on its vertical (for HL) or horizontal neighbor(for LH) and its parent coefficient. Again as in the case of the LL subband, the predictor coefficients can be calculated offline for a sequence of similar images using (5.22). In this case the predictors are the parent coefficient at the same spatial location and the horizontal or vertical neighbor. This scheme is motivated by a similar scheme discussed in [10] for visual images. 5.4.6 Compression Schemes Four different schemes for encoding the wavelet coefficients were compared. In all cases the LL subband was encoded using the A-DPCM scheme discussed in Section 5.4.3.
158
Namrata Vaswani et al.
Scalar Quantization This scheme involves scalar quantization (SQ) of the wavelet coefficients and DPCM encoding of the LL coefficients. Variable bits are allocated to the subbands based on their variances as discussed in [9]. Zerotree Coding Zerotree coding is applied as discussed in Section 5.4.4. This not only gives a significantly reduced bits per pixel (BPP) value than the SQ (as expected), but also gives reduced MSE value compared to SQ. The reason is that the quantization error is higher than the thresholding error for high-frequency subbands which are coarsely quantized. 2D Predictive DPCM on Wavelet Subbands Only DPCM coding is applied as discussed in Section 5.4.5 with no zerotree coding. The performance of this scheme is bad because the “noisy data” close to zero, cannot be predicted correctly and hence the prediction errors obtained are sometimes larger than the original pixel value. Hence the MSE is significantly higher. Combined Zerotree and DPCM Coding We propose to combine zerotree coding and the DPCM encoding (ZT/DPCM) of wavelet coefficients to achieve maximal compression. First a simple zerotree coding is applied to the subbands. This is followed by DPCM coding of the “nonzeroed” coefficients. The value of a “zeroed” neighbor is predicted as follows. If we predict Cx,y based on Cx−1,y which is “zeroed” and the zeroing threshold is T , we estimate Cx−1,y as follows S = Cx−2,y + Cx−1,y−1 , ⎧ ⎨ 0 if S = 0, Cˆx−1,y = −T if S < 0, ⎩ +T if S > 0. This is based on the assumption that since the next coefficient is nonzero, the previous one would be close to the threshold. DPCM combined with zerotree coding works much better because the noisy coefficients have been set to “zero” and we do not try to predict their value. The prediction model is applicable only to those subbands for which enough (> 2) bits have been allocated and the prediction error energy obtained while calculating the predictor coefficients is less than 25% of the subband energy. For other subbands, SQ is used.
Chapter 5 Moving Object Detection and Compression in IR Sequences
159
5.4.7 Performance Analysis The aim of any compression scheme is to minimize the mean squared error (maximize the PSNR) and the entropy per pixel (entropy rate, ER). In SQ, each pixel is coded independently and the correlation in the image is not exploited. So the entropy rate is higher. Entropy rate will be minimized if each pixel is coded based on all past pixels on which it depends, i.e. (for a 1D signal) h(Xn ) > h(Xn |Xn−1 ) > h(Xn |Xn−1 , ..., 1). (5.23) If we assume a one-step Markov model, h(Xn |Xn−1 · · · 1) = h(Xn |Xn−1 )
(5.24)
For 2D data (assuming a Markov random field model), this translates to Xn,n depending only on Xn−1,n and Xn,n−1 . The quantization MSE will be minimized for a given bit rate if the mean square value of the quantity to be quantized is minimum. Hence instead of quantizing Xn,n , in 2D predictive DPCM, we predict a value (Xˆn,n ) based on past values and quantize the difference (Xn,n − Xˆn,n ). Xˆn,n is calculated as discussed in (5.21) to minimize E[Xn,n − Xˆn,n ]2 and hence the quantization MSE over all linear estimators. Also for a given quantization step size (fixed MSE), reduced data variance means reduced entropy. In zerotree coding, the PSNR is higher than SQ because the zeroing error is lower than the quantization error for high-frequency subbands which are coarsely quantized. Zeroing also reduces entropy since the number of symbols to be compressed is reduced. The 2D MRF model with second-order dependencies (correlations) fits well for the LL subband, but does not fit well for the wavelet subbands and the prediction fails completely for very small values (only noise). This is the reason why DPCM on wavelet subbands gives the worst PSNR values. Combined zerotree and DPCM (ZT/DPCM) gives best results in terms of PSNR and entropy rate. The noisy coefficients are zeroed and hence not predicted and thus the quantization error remains low. Because of LMSE prediction, the entropy is minimum and zerotree coding further reduces the entropy rate by reducing the number of symbols to be coded. 5.4.8 Image Compression Results Various types of low-pass and high-pass filters satisfying the prefect reconstruction property can be used. In our implementation, the Haar transform is used because of its simplicity and ease of hardware implementation. Using a longer length filter will not be useful because the low-pass filter will tend to average over a very large area and thus lose the localization property of wavelet transforms. The Haar wavelet is built using a two-tap low-pass filter [1, 1] and a two-tap high-pass filter [1, −1].
160
Namrata Vaswani et al.
Figure 5.5 shows the original tank2 image and Figure 5.12 shows the compressed tank2 images using combined zerotree/DPCM coding and zerotree coding.Table 5.4 shows the compression results for two sample IR images and the Lena image. Table 5.4. The bpp, PSNR[10 log10 2552 /M SE] and entropy for three sample images using zerotree (ZT), zerotree and DPCM (ZT/DPCM), scalar quantization (SQ) and only DPCM coding schemes. Image
Coder
Total BPP
PSNR
Entropy (Non-zero)
RLC BPP
tank2
ZT/DPCM ZT ZT/DPCM ZT ZT/DPCM ZT SQ DPCM
0.5628 0.5628 0.5232 0.5232 0.5066 0.5066 0.7947 0.7947
31.73 31.61 31.75 31.65 29.40 29.30 13.07 25.34
0.0920 0.2112 0.0880 0.2045 0.0851 0.1542 0.3156 0.1508
0.2757 0.2757 0.2649 0.2649 0.2286 0.2286
tank12 lena
Figure 5.12. Compressed tank2 by (a) combined zerotree and DPCM coding (b) zerotree coding.
Chapter 5 Moving Object Detection and Compression in IR Sequences
161
The results have been obtained by allocating a total of 0.5 BPP to various subbands proportional to the logarithm of their variances. Since for this low value of BPP, the lowermost subbands get negative bits allocated to them (which are set to zero), the actual BPP obtained is higher than 0.5. In Table 5.4 we have compared the total BPP, PSNR, BPP, for RLC coding and entropy rate for three different images (two from the IR sequence and the Lena image). The entropy rate is the minimum bits/pixel that can be theoretically achieved for the image. Due to hardware constraints we have not implemented any form of entropy coding (Arithmetic/Huffman). As can be seen from the values of RLC BPP, almost half the bits are used up in encoding the zerotree information. More efficient binary encoding schemes can be employed to reduce this value and this could considerably improve the BPP. Also, in most cases the combined zerotree and DPCM scheme gives the best results both in terms of PSNR values and entropy. In some cases like the Lena image, the PSNR is higher for simple zerotree coding, but the entropy of the combined scheme is less. Ideally, one would assume that applying a DPCM encoding would cause a significant reduction in PSNR. This is definitely true for the LL subband, but the reduction for higher-frequency subbands is not so much because of lesser correlation. Another reason is the uncertainty in predicting the value of a pixel based on a neighboring “zeroed” pixel. We are experimenting with better methods to improve the prediction model for combined zerotree and DPCM encoding. The BPP without zerotree coding is consistently higher for all the images and hence the zerotree coding is advantageous even though half the BPP is used up in RLC coding of the zerotree. Also surprisingly, the PSNR for SQ is lower than for zerotree coding even when the BPP is higher. The reason for this is that the quantization error in SQ is higher than the zeroing error for the high-frequency subbands which are coarsely quantized. From Table 5.4, we observe that DPCM encoding without zeroing is the worst scheme. This is because a lot of the coefficients below the zeroing threshold in the high-frequency subband are actually “noise.” Thus, in DPCM we are trying to predict the value of these “noise” pixels or use them to predict other pixels and hence the predictions are very bad, thus leading to a higher PSNR. Hence the DPCM model fails in the absence of zerotree coding, while it provides a reasonably good model for the image when combined with zerotree coding. 5.4.9 Computational Complexity: Hardware Issues The entire coding scheme is computationally very simple. The Haar wavelet transform involves a single addition operation per pixel. Zerotree coding requires one comparison to a threshold (per pixel) and a multiplication by two (a shift operation) to calculate the descendant position. The DPCM operation involves three real multiplications and two additions to calculate the predicted vale and one subtraction to obtain the error. Run-length coding is
162
Namrata Vaswani et al.
again a counting operation requiring one addition per pixel. Thus the additional cost over scalar quantizing the entire image is (which is the minimum one has to do to compress an image) is three multiplications and a few additions per pixel. For an N 2 ∗N 2 image with a three-level wavelet decomposition, the additional cost for our scheme is given by ACHaar = (N 2 + N 2 /4 + N 2 /16)CA ,
(5.25)
ACZeroing = (N 2 + N 2 /4 + N 2 /16)(CC + CS ),
(5.26)
ACRLC = N 2 CA ,
(5.27)
2
(5.28)
ACDP CM = N (3CM + 3CA ),
where AC is additional cost. CA is the cost for one addition, CM is the cost for one multiplication, CC is the cost for one comparison, and CS is cost for one shift (multiply by two) operation . Since comparison and shift are single operations, CC = CS = 1. Hence, total additional cost is AC = (N 2 + N 2 /4 + N 2 /16)(CA + 2) + N 2 (3CM + 4CA ).
(5.29)
5.5 Performance Evaluation of Algorithms We have developed a C/C++ implementation of the object detection and compression/decompression algorithms. The C/C++ implementation of image compression part of the system currently uses zerotree coding with SQ and RLC. Performance evaluation of the C/C++ code was done using Hitachi’s SH4–7751 microprocessor. SH4–7751 is a high-performance superscalar RISC microprocessor designed for embedded applications. Some of the features of SH4 include 167-mHZ clock frequency, upto 360 MIPS capability, and on-chip cache for instruction and data. More information on SH4 can be found at http://semiconductors.hitachi.com. We evaluated the code performance using the Hitachi Embedded Workshop (HEW) which is an SH4 simulator provided by Hitachi. The results were obtained using the profile utility of HEW. Note that these results are obtained by directly cross-compiling the C code using Hitachi’s cross-compiler. Hence, certain features of SH4 architecture (such as floating point unit) which can enhance real-time performance are not used. In practice, the performance can be improved by designing the assembly language code for computationally expensive procedures so as to take advantage of such features of the object microprocessor. The performance results of the algorithms in terms of code size, run-time memory requirement and instruction cycles on SH4 are as follows.
Chapter 5 Moving Object Detection and Compression in IR Sequences
163
5.5.1 MTI Algorithm The code size required for the MTI algorithm was 9.8 KB. For an input frame size of 120 × 160 maximum run-time memory required was 1.086 MB. Cycles required per frame were equal to 8.5 M which corresponds to 19.54 frames per second (30 fps frame rate). 5.5.2 Image Chipping Algorithm Motion Detection and Best-View Selection For motion detection using the technique described in Section 5.2.3 and bestview selection, the code size required was 16 KB. The run-time memory required for frame size of 240 × 320 was 2.4 MB and the processing frame rate (for 120 × 160 frame size) was 4–5 frames per second. Compression The compression part of algorithm required a code size of 30.5 KB. For a typical image chip of size 60×120, run-time memory required was 0.75 MB and cycles on SH4 needed to compress the images was 31.5 M which corresponds to run-time of 0.188 sec. The run-time is calculated as cycles/clock frequency.
Decompression Decompression required a code size of 21.5 KB and for a typical image chip of size 60 × 120, run-time memory required was 0.75 MB. Note that this memory requirement will increase with chip size. Table 5.5 gives the instruction cycles required for different PSNR values of reconstructed sample tank chip. Table 5.5. Cycles and run-time required for decompression of a sample target chip of size 60 × 120 for different PSNR. PSNR(db)
Cycles(million)
Runtime on SH4(sec)
30.21 34.36
20.73 21.00
0.120 0.125
164
Namrata Vaswani et al.
5.6 Conclusions A novel variance-analysis-based algorithm has been presented for moving object detection. The algorithm is especially suitable for detection of long-range, small, slow moving objects. An initial test on several IR sequences has revealed high detection rates with low false alarms for vehicles at distances of several kilometers. The algorithm is robust and requires no tuning of parameters by the operator. For objects at short distances, technique based on detecting and tracking image singularities have been discussed. Also for objects at short range, methods for best-view selection and compression were presented. Three different approaches to the best-view selection problem were compared. The approach based on classification in eigenspace is suitable when multiple views of the same object are available but has limitations when significant scale and illumination variations are present. The FOE estimation approach would be a useful method for direction of motion calculation but is computationally expensive. The size-based technique is the fastest and gives reasonable results and is used in our implementation. A new scheme combining noniterative zerotree coding with 2D DPCM for LL and for the high-frequency subbands was presented. This method gives better results than simple scalar quantization and simple zerotree coding both in terms of BPP and PSNR at a marginally increased computational cost. The algorithms were implemented in C and their performance results on SH4 processor were presented. The image compression results can be further improved by using some form of entropy coding (since the entropy rate of our scheme is significantly lower) and by replacing the run length coding method with more efficient binary coding techniques. Also the zeroing thresholds can be calculated for the required PSNR values.
References [1] Shekarforoush, H., Chellappa, R.: A multi-fractal formalism for stabilization, object detection and tracking in FLIR sequences. In: International Conference on Image Processing. Volume 3. (2000) 78–81 [2] Elgammal, A., Harwood, D., Davis, L.: Nonparametric background model for background subtraction. In: Proc. 6th European Conf. Computer Vision. Volume 2. (2000) 751–767 [3] Murase, S., Nayar, S.: Visual learning and recognition of 3-d objects from appearance. International J. Computer Vision 14 (1995) 5–24 [4] Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognitive Neuroscience 3 (1991) 71–86 [5] Turk, M., Pentland, A.: Face recognition using eigenfaces. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition. (1991) 586–591 [6] Sirovich, L., Kirby, M.: Low dimensional procedures for the characterization of human faces. J. Optical Society of America 4 (1987) 519–524
Chapter 5 Moving Object Detection and Compression in IR Sequences
165
[7] Srinivasan, S.: Extracting structure from optical flow using fast error search technique. International J. Computer Vision 37 (2000) 203–230 [8] Srinivasan, S., Chellappa, R.: Noise-resilient optical flow estimation using overlapped basis functions. J. Optical Society of America 16 (1999) 493– 509 [9] Paek, S., Kim, L.: A real-time wavelet vector quantization algorithm and its vlsi architecture. IEEE Transactions on CSVT 10 (2000) 475–489 [10] Buccigrossi, R., Simoncelli, E.: Image compression via joint statistical characterization in the wavelet domain. IEEE Transactions on Image Processing 8 (1999) 1688–1700
Chapter 6 Face Recognition in the Thermal Infrared
Lawrence B. Wolff, Diego A. Socolinsky, and Christopher K. Eveland Equinox Corporation, 9 West 57th Street, New York, New York 10019 {wolff,diego,eveland}@equinoxsensors.com
Summary. Recent research has demonstrated distinct advantages of using thermal infrared imaging for improving face recognition performance. While conventional video cameras sense reflected light, thermal infrared cameras primarily measure emitted radiation from objects such as faces. Visible and thermal infrared image data collections of frontal faces have been on-going at NIST for over two years, producing the most comprehensive face database known to involve thermal infrared imagery. Rigorous experimentation with this database has revealed consistently superior recognition performance of algorithms when applied to thermal infrared, particularly under variable illumination conditions. Physical phenomenology responsible for this observation is analyzed. An end-to-end face recognition system incorporating simultaneous coregistered thermal infrared and visible has been developed and tested indoors with good performance.
6.1 Introduction Accelerated developments in camera technology over the last decade have given computer vision researchers a whole new diversity of imaging options, particularly in the infrared spectrum. Conventional video cameras use photosensitive silicon that is typically able to measure energy at electromagnetic wavelengths from 0.4 µm to just over 1.0 µm. Multiple technologies are currently available, with dwindling cost and increasing performance, which are capable of image measurement in different regions of the infrared spectrum, as shown in Figure 6.1. Figure 6.2 shows the different appearances of a human face in the visible, shortwave infrared (SWIR) midwave infrared (MWIR), and longwave infrared (LWIR) spectra. Although in the infrared, the near-infrared (NIR) and SWIR spectra are still reflective and differences in appearance between the visible, NIR and SWIR are due to reflective material properties. Both NIR and SWIR have been found to have advantages over imaging in the visible for face detection [1] and detecting disguise [2].
This research was supported by the DARPA Human Identification at a Distance (HID) program under contract #DARPA/AFOSR F49620-01-C-0008.
167
168
Lawrence B. Wolff et al.
Wavelength 0.7 SWIR in Microns 0.4 1.0 2.5 3.0
MWIR
LWIR 5.0
8.0
14.0
Atmospheric Absorption Visible Near-Infrared
UV Shortwave Infrared
Midwave Infrared
Longwave Infrared
Figure 6.1. Nomenclature for various parts of the electromagnetic spectrum.
(a)
(b)
(c)
(d)
Figure 6.2. A face simultaneously imaged in the (a) visible spectrum, 0.4–0.7 µm, (b) shortwave infrared, 0.9–1.7 µm, (c) midwave infrared, 3.0–5.0 µm, and (d) longwave infrared, 8.0-14.0 µm.
At wavelengths of 3 µm and longer imaged radiation from objects becomes significantly emissive due to temperature, and is hence generally termed the thermal infrared. The thermal infrared spectrum is divided into two primary spectra, the MWIR and LWIR. Between these spectra lies a strong atmospheric absorption band between approximately 5 and 8 µm wavelength, where imaging becomes extremely difficult due to nearly complete opaqueness of air. The range beyond 14 µm is termed the very longwave infrared (VLWIR) and although in recent years it has recieved increased attention, it remains beyond the scope of this chapter. The amount of emitted radiation depends on both the temperature and the emissivity of the material. Emissivity in the thermal infrared is conversely analogous to the notion of reflective albedo used in the computer vision literature [3, 4]. For instance, a Lambertian reflector can appear white or grey depending on its efficiency for reflecting light energy. The more efficient it is in reflecting energy (more reflectance albedo) the less efficient it is in thermally emitting energy respective to its temperature (less emissitivity). Objects with perfect emissivity of 1.0 are completely black. Many materials that are poor absorbers transmit most light energy while reflecting only a small portion. This applies to a variety of different types of glass and plastics in the visible spectrum. As detailed in the following section, the spectral distribution of energy emitted by an object is simply the product of the Planck distribution for a given temperature, with the emissivity of the object as function of wavelength [5]. In the vicinity of human body temperature (37◦ C), the Planck
Chapter 6 Face Recognition in the Thermal Infrared
169
distribution has a maximum in the LWIR around 9 µm, and is approximately one-sixth of this maximum in the MWIR. As we will show through empirical measurement, the emissivity of human skin in the MWIR is at least 0.91, and at least 0.97 in the LWIR. Therefore, face recognition in the thermal infrared favors the LWIR, since LWIR emission is much higher than that in the MWIR. Thermal infrared imaging for face recognition first used MWIR platinum silicide detectors in the early 1990s [6]. At that time, cooled LWIR technology was very expensive. By the late 1990s, uncooled microbolometer imaging technology in the LWIR became more accessible and affordable, enabling wider experimental applications in this regime. At that time, cooled MWIR technology was about ten times more sensitive than uncooled microbolometer LWIR technology, and even though faces are more emissive in the LWIR, in the late 1990s MWIR could still discern more image detail of the human face. At present, uncooled microbolometer LWIR technology coming off the assembly lines is rapidly approaching one-half of the sensitivity of cooled MWIR. For face recognition in the thermal infrared, this is a turning point as for the first time the most appropriate thermal infrared imaging technology (i.e. LWIR) for studying human faces is also the most affordable. For over two years, data collections of both visible and thermal infrared imagery of faces have been taken and continue to take place at regular intervals of 6 months at the National Institute of Science and Technology (NIST). This effort is supported by the DARPA HID program [7]. Section 6.2 describes the comprehensive database resulting from these collections, consisting of over 100,000 images of over 300 individuals so far. This database has provided the empirical foundation with which to rigorously compare the performance of various face recognition algorithms between visible and thermal infrared imagery. Some of these results are summarized in Section 6.6. Also described in Section 6.6 is a recently completed full-working prototype of the equinox access control environment (ACE) face recognition system, which uses fused coregistered visible and LWIR imagery from a novel sensor system. The main advantage of thermal infrared imaging for boosting face recognition performance is its apparent invariance to changing illumination. Section 6.4 attempts to characterize how well thermal infrared images of the human face are invariant to illumination changes. Section 6.5 delves deeper into explaining the physical phenomenology responsible for this invariance by computing the emissivity of human skin in the MWIR and the LWIR. This culminates in a preliminary thermal model for human skin. Finally, Section 6.7 briefly overviews some of the remaining challenges that thermal infrared imaging does not immediately remedy for face recognition.
170
Lawrence B. Wolff et al.
6.2 The Equinox Visible/Infrared Face Database Figure 6.3 shows the experimental set-up for imagery being collected on a regular basis at NIST, simultaneosuly in the visible, SWIR, MWIR, and LWIR spectra. The objectives for these ongoing data collections are as follows: 1. To be able to directly and rigorously compare the performance of face recognition algorithms between visible imagery and imagery in the various modalities of the infrared spectrum. 2. To produce face imagery simultaneously in these modalities under variable illumination conditions. 3. To produce face imagery with significant intrapersonal variation for each imaging condition and modality. Towards the first objective a configuration of four different sensors of respective modalities and interface software has been set-up for simultaneous acquisition of visible, SWIR, MWIR, and LWIR imagery. All infrared cameras are of the Indigo Merlin Series with 320 ∗ 240 resolution. The visible camera is a Pulnix 6710 with 640∗480 resolution. A special optical design insures precise pixel coregistration of visible and LWIR imaging modalities [8, 9]. Although the SWIR and MWIR imaging modalities are boresighted2 , impairing precise coregistration at close distances, the physical separation between these cameras has been minimized beyond what is shown in Figure 6.3 so that views are nearly identical. Optically coregistered sensors in the NIR were in use [2, 1] for face detection. The complexity of coregistering visible and LWIR wavelengths, however, is much greater due to the larger disparity between them. Towards the second and third objectives, collection of image data was repeated for three different illumination conditions: (i) Frontal, (ii) frontal-Left, (iii) frontal-Right, with lamps shown in Figure 6.3 using standard 3200 K color temperature photographic bulbs. Figure 6.4 shows the emission curve for these bulbs in the wavelengths of interest. Forty image frame sequences of visible, SWIR, MWIR, and LWIR were digitized simultaneously at 10 frames/second (i.e., 4-seconds duration), while a human subject was reciting the vowels “a,” “e,” “i,” “o,” “u.” This creates a continuous image sequence with changes in expression throughout providing significant intrapersonal variation over the course of multiple frames. At the same time there is little facial movement between consecutive image frames 1/10 second apart, allowing for analysis of image variations due to temporal sensor noise. Figure 6.2 is an example of one such multimodal frame within this 40-frame sequence. After the acquisition of each 40-frame contiguous image sequence, for each illumination three more static images are taken of individuals told to make extreme expressions of “smile,” “frown,” and “surprise.” 2
The term boresighted typically refers to cameras that have been placed alongside each other and aimed in the same direction. Due to the separation between the cameras, it is impossible to obtain the same view of a 3D object from both sensors. This is often exploited in stereo vision to compute depth maps.
Chapter 6 Face Recognition in the Thermal Infrared
171
Figure 6.3. Camera and illumination equipment set-up used for simultaneous data collection of visible, SWIR, MWIR, and LWIR imagery.
Prior to data collection, the radiometric calibration procedure described in Section 6.3 was performed for the Indigo Merlin series MWIR and LWIR cameras using a Model 350 Mikron blackbody source. Software was developed to convert raw MWIR and LWIR image grayscale values directly into respective thermal emission values from ground-truth blackbody images. Raw image gray values for the MWIR and LWIR cameras are 12-bit integers from which floating point thermal emission values were computed and then rounded back to 12-bit values with appropriate dynamic range. At present, the Equinox visible/infrared database consists of over 300 individuals imaged over five separate data collections at NIST. At least 60 individuals have participated in two or more of these data collections so that intrapersonal variations over 6 months or more can be analyzed. To summarize, for each individual a 40-frame sequence plus three static images were taken for three different illuminations and four spectral image modalities. Not including duplicate individuals, at least 340 individuals ∗ 43 images/modality ∗ 3 illuminations/individual ∗ 4 modalities = 175,440 images are contained in the Equinox database. Almost all of this database was collected indoors, with outdoor imagery beginning to be collected during the last data collection at NIST in April 2002. A portion of this database is available on the Internet at http://www.equinoxsensors.com/hid.
Lawrence B. Wolff et al.
5500 Deg. 3200 Deg. 307.3 Deg. 258 Deg.
K. K. K. K.
172
(a)
(b)
Figure 6.4. (a) Blackbody Planck curves comparing thermal IR emission from common natural and artificial illumination sources to thermal IR emission from human skin. (b) Comparison of an ideal blackbody Planck curve with a nonideal emitter at the same temperature.
6.3 Calibration Of Thermal IR Sensors All objects above absolute zero temperature emit electromagnetic radiation. In the early 1900s Planck was the first to characterize the spectral distribution of this radiation for a blackbody, which is an object that completely absorbs electromagnetic radiation at all wavelengths [5]. According to Planck’s law, the spectral distribution of emission from a blackbody at temperature T , is given by 2πhc2 W (λ, T ) = (6.1) [Watts/cm2 ]µm−1 , hc λ5 (e λkT − 1.0) Q(λ, T ) =
2πc hc λ4 (e λkT
− 1.0)
[Photons/cm2 − sec]µm−1 ,
(6.2)
expressed in two different units of energy flux which are commonly used. In the above formulas, h is Planck’s constant, k is Boltzmann’s constant, c is the speed of light, and λ is wavelength. Figure 6.4(a) shows a comparison of
Chapter 6 Face Recognition in the Thermal Infrared
173
blackbody spectral distributions corresponding to the various temperatures for the Sun, artificial lightbulb illumination at 3200 K color temperature [10], human skin, and average temperature for the atmosphere. In reality, only very few objects are near perfect energy absorbers, particularly at all wavelengths. The proportional amount of energy emission with respect to a perfect absorber is called the emissivity (T, λ, ψ), which takes values in the range [0, 1]. In addition to temperature T and wavelength λ, this can also be a function of emission angle ψ. Kirchoff’s law states that the emissivity at a point on an object is equal to the absorption α(T, λ, ψ), namely: (T, λ, ψ) = α(T, λ, ψ) . This is a fundamental law that effectively asserts the conservation of energy. Blackbody objects are therefore the most efficient radiators, and for a given temperature T emit the most energy possible at any given wavelength. Expressed in the same units as equations ( 6.1) and ( 6.2) above, the spectral distribution of emission from an object with emissivity (T, λ, ψ), is given by: (T, λ, ψ) × W (λ, T ),
(T, λ, ψ) × Q(λ, T ) .
For illustrative purposes, Figure 6.4(b) compares the spectral distribution of the emission of an ideal blackbody at 500 K (227◦ C) with that of a nonideal emitter (e.g., could be a piece of bare metal) also at the same temperature. In this case the nonideal emitter has low emissivity at wavelengths in the MWIR spectral region (3-5 µm) and generally high emissivity in the LWIR spectral region (8-14 µm). Under most practical conditions, 2D imaging array thermal IR sensors (i.e., what are termed staring arrays) measure simultaneously over broadband wavelength spectra, as opposed to making measurements at narrow, almost monochromatic, wavelengths (e.g., an IR spectrophotometer which measures only one point in a scene). With a staring array sensor it is possible to measure average emissivity over a broadband spectrum (e.g., 3-5 µm, 8-14 µm), which in Figure 6.4(b) is simply the ratio of the area under the nonideal curve to the area under the Planck curve over the respective wavelength spectrum. Some of these principles can be observed in Figure 6.5 (in Section 6.4). Plastic materials transparent in the visible spectrum that compose glasses are opaque in the LWIR and appear dark. Emissivity of this material is small in the visible spectrum while being significantly above 0.80 in the MWIR and LWIR spectral regions. The dark appearance of glasses in the LWIR and the MWIR relative to thermal emission from human facial skin is mostly due to the glasses being close to room temperature, about 15◦ C cooler than body temperature. We performed simple experiments whereby these same pair of glasses were heated close to body temperature. Sure enough, the glasses appeared thermally much brighter, but did not show as much thermal emission as facial skin at the same temperature. Also, from Figure 6.5 the influence of reflection of external illumination from glasses is far more prominent than
174
Lawrence B. Wolff et al.
that from facial skin. All of this initially suggests that facial skin has very high emissivity, significantly higher than that of the material comprising glasses. A quantitative estimate of the average emissivity of facial skin in the MWIR and LWIR is developed in Section 6.5, supporting this assertion. Just like visible video cameras, thermal IR cameras measure energy of electromagnetic radiation, the main difference being that because thermal IR cameras sense at such long wavelengths, they measure radiation that has been typically thermally emitted. Of course, visible cameras see radiation emitted from very hot sources (e.g., the sun or artificial lightbulbs which are thousands of degrees Kelvin) but the primary scene elements of interest in the visible are objects from which such light is reflected. Sometimes there is the misconception that thermal IR cameras directly measure temperature, which would be true if all objects were blackbodies. Temperature can be determined indirectly from a thermal IR camera by measurement of energy of emitted radiation, using precise knowledge of emissitivity of the object, which is dependent upon a number of parameters.
Figure 6.5. A qualitative demonstration of the illumination invariance for LWIR imagery of a face under different illuminations. Top row: Visible imagery of a face under three illumination conditions respectively front, left, and right. Bottom row: Co-registered thermal IR imagery simultaneously acquired for each of the three images in top row respectively.
Thermal IR cameras can be radiometrically calibrated using a blackbody ground-truth source. Radiometric calibration achieves a direct relationship between the gray value response at a pixel and the absolute amount of thermal emission from the corresponding scene element. This relationship is called responsivity. Depending on the type of thermal IR camera being used, thermal emission flux is measured in terms of watts/cm2 or photons/(cm2 − second) [11]. The gray value response of pixels for a MWIR camera with an in-
Chapter 6 Face Recognition in the Thermal Infrared
175
dium antimonide (InSb) focal plane array is linear with respect to photons/(cm2 − second). The gray value response of pixels for an LWIR camera using a microbolometer focal plane array is linear with respect to watts/cm2 . Two-point radiometric calibration uses a blackbody plate filling the field of view of the thermal IR camera and capturing images for the blackbody at two different temperatures. Given that human body temperature is 37◦ C, two good temperatures to use for calibrating the imaging of humans in a room temperature scene would be 20◦ C and 40◦ C (293 K and 313 K), as these are relatively evenly spread about the temperature of skin. A relatively large difference between the calibration temperatures will insure numerical stability of the linear regression, while a choice of temperatures nearby the temperature of interest minimizes possible effects from a secondary nonlinear response of the focal plane array. Since absolute thermal emission is known by computing the area under the Planck curve for the corresponding temperature and wavelength spectrum, a responsivity line is generated at each pixel by two (greyvalue, thermal emission) coordinate values. The slope of this responsivity line is called the “gain” and the vertical translation of the line is “offset.” The gain and offset for each pixel on a thermal IR focal plane array can be significantly variable across the array. Radiometric calibration standardizes thermal emission measurement by generating a responsivity line for each pixel. Figure 6.6 shows responsivity lines respective to different integration times, for a single pixel near the center of a MWIR InSb focal plane array that was used to collect face imagery. Eight different temperatures of a blackbody were used to generate multiple data points demonstrating the highly linear response. It is clearly important to record all thermal IR camera parameters for a given radiometric calibration. Note that the responsivity lines for different integration times intersect at the same point, related to various DC bias control settings on the camera. Beyond camera parameters, if an MWIR or LWIR camera is originally radiometrically calibrated in an indoors environment, taking it outdoors where there is a significant ambient temperature difference, the gain and offset of linear responsivity of focal plane array pixels will change as the optical lens temperature in front of the focal plane array changes. Radiometric calibration standardizes all thermal IR data collections, whether they are taken under different environmental factors or with different thermal IR cameras or at different times.
6.4 Measuring Illumination Invariance Variation in illumination is one of the biggest factors that confounds face recognition algorithms in the visible spectrum [12, 13]. It has been recognized in the past [2, 14, 6, 15] that changes in illumination appear to play less of a role in the thermal infrared, but how does one quantify this invariance in terms that are meaningful to face recognition? One way is to quantitatively compare
176
Lawrence B. Wolff et al.
the effect that variation in illumination has on face images in the thermal infrared with other factors that contribute to changes in face imagery, such as variations in facial expression and more subtle variations due to camera noise. Illumination invariance of the human face in the termal infrared can be qualitatively observed in Figure 6.5 for a coregistered LWIR and visible video camera sequence of a face under three different illumination conditions. For this sequence a single 60-W light bulb mounted in a desk lamp illuminates a face in an otherwise completely dark room and is moved into different positions. The top row of visible video imagery shows dramatic changes in the appearance of the face. The bottom row shows LWIR imagery which, unlike its coregistered visible counterpart, appears to be remarkably invariant across different illuminations, except in the image area corresponding to the glasses. As we will see, illumination invariance in the thermal infrared, while not being completely ideal, is nonetheless strongly approximate.
Figure 6.6. Responsivity curves for different integration times for the Indigo Merlin Series MWIR camera used for collecting face images.
Figure 6.7 shows simultaneously acquired MWIR and LWIR images of a subject from the Equinox database, together with corresponding gray value histograms of an individual under the three illumination conditions previously described. These images are the third image frame out of each respective 40image frame sequence. Gray values in the histograms are represented as 16-bit integers with the high 12-bits being the actual image gray value. The gray level histograms are remarkably stable across different illuminations for both the MWIR and the LWIR images. Of the variations that are present in the respective histograms, which are due to change in illumination and which are
Chapter 6 Face Recognition in the Thermal Infrared
177
due to other factors? For instance, note the darker mouth region in the MWIR image for right illumination as compared to the mouth region in the MWIR images for other illuminations. The darker mouth region is due to the subject breathing-in room temperature air at the moment, thereby cooling down the mouth. This has nothing to do with any illumination condition. The histograms in Figure 6.7 can be compared with those in Figure 6.8, which shows gray value histograms corresponding to the fourth and twentieth image frame out of the 40-image frame sequence respective to the frontal illumination condition. In this case, illumination is the same but the fourth frame being consecutive with the third frame isolates changes due to camera noise, and the twentieth frame occuring just under two seconds later means the subject has changed facial expression. The variations in the gray level histogram due to camera noise and to different facial expression under the same illumination are of similar magnitude to variations occurring under different illumination. A quantitative analysis of invariance in the framework of hypothesis testing was also performed. The following analysis is repeated for two different distance measures between images. Firstly we consider the L2 distance between normalized images taken as vectors. Secondly, we use the Kullback–Leibler divergence3 between the histograms of the normalized faces, given by $ P I(P, Q) = P log , Q where P and Q are the respective normalized histograms. For each video sequence of 40 + 3 frames4 , we compute the 43 · 42/2 = 903 distances between normalized faces for distinct pairs of frames. Also, we compute the 43 · 43 = 1849 distances between normalized faces for sequences of the same subject and modality, one sequence with frontal illumination and the other with lateral illumination. From these computations we estimate (nonparametrically) the distribution of distances for images with the same illumination condition and with different illumination conditions. Figures 6.9, 6.10, 6.11, and 6.12 show the estimated distributions for the L2 distance and KL-divergence for two subjects in our database. With an infinite supply of images, we would expect the distances to behave according to a χ distribution with the number of degrees of freedom matching the number of pixels in the normalized faces, and indeed the experimental estimates approximate χ distributions. It is clear from Figures 6.9, 6.10, 6.11, and 6.12 that the distances between normalized visible faces with different illumination conditions are much larger 3
4
The Kullback–Leibler divergence does not satisfy the triangle inequality, and thus is not strictly a distance. However, it provides an information-theoretic measure of similarity between probability distributions. Recall that 40 consecutive video frames were collected while subjects recited the vowels, and then three additional static frames were acquired while the subjects were asked to act out the expressions “smile,” “frown,” and “surprise.”
178
Lawrence B. Wolff et al.
MWIR FRONTAL ILLUMINATION
LWIR
MWIR
LWIR RIGHT ILLUMINATION
MWIR
LWIR LEFT ILLUMINATION
Figure 6.7. MWIR and LWIR imagery of a face for three illumination conditions and respective histograms of the third frame out of a sequence of 40 images.
than those for visible faces with the same illumination condition. This indicates that the variation in appearance due to change in illumination is much larger than that due to change in facial expression. The corresponding statement for LWIR imagery does not hold. That is, looking once again at Figures 6.9, 6.10, 6.11, and 6.12, one can see that the distribution of distances between normalized faces with different illumination conditions is comparable (but not equal; see below) to the distribution obtained by using images acquired with the same illumination condition. In other words, the variation in appearance introduced by changes in illumination and expression is comparable to that induced by changes in facial expression alone. Phrasing these statements as formal hypothesis, we can reject the null-hypothesis of illumination invari-
Chapter 6 Face Recognition in the Thermal Infrared
MWIR
179
LWIR FOURTH FRAME
MWIR
LWIR TWENTIETH FRAME
Figure 6.8. MWIR and LWIR imagery of the same face as Figure 6.7 respective to frontal illumination for the fourth frame (top row) and twentieth frame (bottom row) out of a sequence of 40 images.
0.03
0.008 Front-Front Left-Left Front-Left
Front-Front Left-Left Front-Left 0.007
0.025 0.006 0.02 0.005
0.015
0.004
0.003 0.01 0.002 0.005 0.001
0
0
Figure 6.9. Distribution of L2 distances for visible (left) and LWIR (right) images of subject 2344.
180
Lawrence B. Wolff et al.
0.014
0.008 Front-Front Left-Left Front-Left
Front-Front Left-Left Front-Left 0.007
0.012
0.006 0.01 0.005 0.008 0.004 0.006 0.003 0.004 0.002
0.002
0.001
0
0
Figure 6.10. Distribution of Kullback–Leibler divergences for visible (left) and LWIR (right) images of subject 2344.
0.03
0.006 Front-Front Left-Left Front-Left
Front-Front Left-Left Front-Left
0.025
0.005
0.02
0.004
0.015
0.003
0.01
0.002
0.005
0.001
0
0
Figure 6.11. Distribution of L2 distances for visible (left) and LWIR (right) images of subject 2413.
0.014
0.007 Front-Front Left-Left Front-Left
Front-Front Left-Left Front-Left
0.012
0.006
0.01
0.005
0.008
0.004
0.006
0.003
0.004
0.002
0.002
0.001
0
0
Figure 6.12. Distribution of Kullback–Leibler divergences for visible (left) and LWIR (right) images of subject 2413.
Chapter 6 Face Recognition in the Thermal Infrared
181
ance for visible imagery with a p-value smaller than 0.015 , whereas we are unable to reject the null-hypothesis for LWIR imagery with any significant confidence. The slight shift in the distributions to the right for variable illumination suggests that illumination invariance in the LWIR is not completely ideal.
6.5 Emissivity Of Human Facial Skin Figure 6.4(a) shows that the amount of thermal emission from a common lightbulb is three to four orders of magnitude greater than the thermal emission from skin in both the 3–5 µm MWIR region and the 8–14 µm LWIR region. Empirical observation with our own MWIR and LWIR cameras showed that direct illumination from an incandescant filament through lightbulb glass and plastic diffuser is at least 300 times greater than thermal emission from human facial skin. This is a rather striking fact given that thermal IR imagery of faces is highly illumination invariant. Human skin must absorb a large quantity of radiation in both the MWIR and the LWIR implying that skin has very high emissivity. Figure 6.13 shows a human subject in the same scene with a 6in ∗ 6in square blackbody (Mikron Model 345) imaged in the MWIR and LWIR spectra. Separate images are taken for the blackbody at two different temperatures: 32◦ C and 35◦ C. The corresponding histograms show gray value modes for the facial skin image region and for the blackbody image region. Prior to imaging, an Anritsu thermocouple was used to make contact temperature measurements on the forehead, on both cheeks and on the chin of the human subject. An average skin surface temperature of 32◦ C was observed. Note, however that the face thermally emits more energy than does a 32◦ C blackbody. Recall that a blackbody is, by definition, a perfect emitter at all temperatures and wavelengths. Therefore, we have a physical contradiction unless we can account for the extra radiation. Since the path self-emission from the atmosphere between the subject and the sensor is negligible compared to the emission from the subject, we conclude that the extra radiation must be originating below the skin surface (where body temperature is around 37◦ C) and shining through the translucent skin layer and onto the sensor. This may reveal an important aspect of how thermal emission arises from human anatomy and perhaps even a physical mechanism for why skin has such high absorption in the thermal IR. Figure 6.14 illustrates a preliminary high-level model of human skin in terms of optical and thermal properties. Evidently, skin layers must be significantly transmissive to thermal emission from underlying internal anatomy 5
This means that the likelihood of our rejecting the hypothesis of illumination invariance for visible imagery while at the same time the hypothesis being true is lower than 1%[16].
182
Lawrence B. Wolff et al.
MWIR
LWIR
MWIR
LWIR
Figure 6.13. Direct comparison of MWIR and LWIR imagery of a face with a groundtruth blackbody at two different temperatures, 32◦ C and 35◦ C.
which is at a higher temperature. This is qualitatively evidenced from thermal observation of prominent vasculature beneath the skin particularly in the neck and forehead. Just how far below the skin surface thermal emission is transmitted is unclear and is an avenue for future research. If at least the outer layers of skin are transmissive, then incident thermal IR illumination must be first transmitted and then absorbed within deeper layers of skin or other anatomy. This may explain why the amount of thermal emission from skin seems to be independent of external skin color in the visible spectrum.
LOW INDEX OF REFRACTION AT AIR/SKIN INTERFACE
32 Deg. C.
SIGNIFICANT TRANSMISSION
HIGHER ABSORPTION
37 Deg. C.
EPIDERMIS
MIDDLE LAYER SUBCUTANEOUS FAT LAYER
INTERNAL ANATOMY
Figure 6.14. Preliminary thermal mo