Feature Extraction and Image Processing
Dedication We would like to dedicate this book to our parents: To Gloria and Joaquin Aguado, and to Brenda and the late Ian Nixon.
Feature Extraction and Image Processing Second edition
Mark S. Nixon Alberto S. Aguado
Amsterdam • Boston • Heidelberg • London • New York • Oxford Paris • San Diego • San Francisco • Singapore • Sydney • Tokyo Academic Press is an imprint of Elsevier
Academic Press is an imprint of Elsevier Linacre House, Jordan Hill, Oxford OX2 8DP, UK 84 Theobald’s Road, London WC1X 8RR, UK First edition 2002 Reprinted 2004, 2005 Second edition 2008 Copyright © 2008 Elsevier Ltd. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email:
[email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress ISBN: 978-0-12372-538-7 For information on all Academic Press publications visit our web site at books.elsevier.com Printed and bound in Hungary 08 09 10 10 9 8 7 6 5 4 3 2 1
Working together to grow libraries in developing countries www.elsevier.com | www.bookaid.org | www.sabre.org
.
Contents
.
Preface
xi
1
1
2
Introduction 1.1 Overview 1.2 Human and computer vision 1.3 The human vision system 1.3.1 The eye 1.3.2 The neural system 1.3.3 Processing 1.4 Computer vision systems 1.4.1 Cameras 1.4.2 Computer interfaces 1.4.3 Processing an image 1.5 Mathematical systems 1.5.1 Mathematical tools 1.5.2 Hello Mathcad, hello images! 1.5.3 Hello Matlab! 1.6 Associated literature 1.6.1 Journals and magazines 1.6.2 Textbooks 1.6.3 The web 1.7 Conclusions 1.8 References
1 1 3 4 6 7 9 10 12 14 15 16 16 21 24 24 25 28 29 29
Images, sampling and frequency domain processing
33
2.1 2.2 2.3 2.4 2.5
33 34 37 43 47 47 49 54 54 56 56 57
Overview Image formation The Fourier transform The sampling criterion The discrete Fourier transform 2.5.1 One-dimensional transform 2.5.2 Two-dimensional transform 2.6 Other properties of the Fourier transform 2.6.1 Shift invariance 2.6.2 Rotation 2.6.3 Frequency scaling 2.6.4 Superposition (linearity)
v
3
4
2.7 Transforms other than Fourier 2.7.1 Discrete cosine transform 2.7.2 Discrete Hartley transform 2.7.3 Introductory wavelets: the Gabor wavelet 2.7.4 Other transforms 2.8 Applications using frequency domain properties 2.9 Further reading 2.10 References
58 58 59 61 63 64 65 66
Basic image processing operations
69
3.1 Overview 3.2 Histograms 3.3 Point operators 3.3.1 Basic point operations 3.3.2 Histogram normalization 3.3.3 Histogram equalization 3.3.4 Thresholding 3.4 Group operations 3.4.1 Template convolution 3.4.2 Averaging operator 3.4.3 On different template size 3.4.4 Gaussian averaging operator 3.5 Other statistical operators 3.5.1 More on averaging 3.5.2 Median filter 3.5.3 Mode filter 3.5.4 Anisotropic diffusion 3.5.5 Force field transform 3.5.6 Comparison of statistical operators 3.6 Mathematical morphology 3.6.1 Morphological operators 3.6.2 Grey-level morphology 3.6.3 Grey-level erosion and dilation 3.6.4 Minkowski operators 3.7 Further reading 3.8 References
69 70 71 71 74 75 77 81 81 84 87 88 90 90 91 94 96 101 102 103 104 107 108 109 112 113
Low-level feature extraction (including edge detection)
115
4.1 Overview 4.2 First order edge detection operators 4.2.1 Basic operators 4.2.2 Analysis of the basic operators 4.2.3 Prewitt edge detection operator 4.2.4 Sobel edge detection operator 4.2.5 Canny edge detection operator
115 117 117 119 121 123 129
vi Contents
5
4.3 Second order edge detection operators 4.3.1 Motivation 4.3.2 Basic operators: the Laplacian 4.3.3 Marr–Hildreth operator 4.4 Other edge detection operators 4.5 Comparison of edge detection operators 4.6 Further reading on edge detection 4.7 Phase congruency 4.8 Localized feature extraction 4.8.1 Detecting image curvature (corner extraction) 4.8.1.1 Definition of curvature 4.8.1.2 Computing differences in edge direction 4.8.1.3 Measuring curvature by changes in intensity (differentiation) 4.8.1.4 Moravec and Harris detectors 4.8.1.5 Further reading on curvature 4.8.2 Modern approaches: region/patch analysis 4.8.2.1 Scale invariant feature transform 4.8.2.2 Saliency 4.8.2.3 Other techniques and performance issues 4.9 Describing image motion 4.9.1 Area-based approach 4.9.2 Differential approach 4.9.3 Further reading on optical flow 4.10 Conclusions 4.11 References
137 137 137 139 144 145 146 147 152 153 153 154
Feature extraction by shape matching
183
5.1 Overview 5.2 Thresholding and subtraction 5.3 Template matching 5.3.1 Definition 5.3.2 Fourier transform implementation 5.3.3 Discussion of template matching 5.4 Hough transform 5.4.1 Overview 5.4.2 Lines 5.4.3 Hough transform for circles 5.4.4 Hough transform for ellipses 5.4.5 Parameter space decomposition 5.4.5.1 Parameter space reduction for lines 5.4.5.2 Parameter space reduction for circles 5.4.5.3 Parameter space reduction for ellipses 5.5 Generalized Hough transform 5.5.1 Formal definition of the GHT 5.5.2 Polar definition
183 184 186 186 193 196 196 196 197 203 207 210 210 212 217 221 221 223
156 159 163 163 163 166 167 167 168 171 177 178 178
Contents
vii
6
7
viii
5.5.3 The GHT technique 5.5.4 Invariant GHT 5.6 Other extensions to the Hough transform 5.7 Further reading 5.8 References
224 228 235 236 237
Flexible shape extraction (snakes and other techniques)
241
6.1 Overview 6.2 Deformable templates 6.3 Active contours (snakes) 6.3.1 Basics 6.3.2 The greedy algorithm for snakes 6.3.3 Complete (Kass) snake implementation 6.3.4 Other snake approaches 6.3.5 Further snake developments 6.3.6 Geometric active contours 6.4 Shape skeletonization 6.4.1 Distance transforms 6.4.2 Symmetry 6.5 Flexible shape models: active shape and active appearance 6.6 Further reading 6.7 References
241 242 244 244 246 252 257 257 261 266 266 268
Object description
281
7.1 Overview 7.2 Boundary descriptions 7.2.1 Boundary and region 7.2.2 Chain codes 7.2.3 Fourier descriptors 7.2.3.1 Basis of Fourier descriptors 7.2.3.2 Fourier expansion 7.2.3.3 Shift invariance 7.2.3.4 Discrete computation 7.2.3.5 Cumulative angular function 7.2.3.6 Elliptic Fourier descriptors 7.2.3.7 Invariance 7.3 Region descriptors 7.3.1 Basic region descriptors 7.3.2 Moments 7.3.2.1 Basic properties 7.3.2.2 Invariant moments 7.3.2.3 Zernike moments 7.3.2.4 Other moments 7.4 Further reading 7.5 References
281 282 282 283 285 286 287 289 290 292 301 305 311 311 315 315 318 320 324 325 326
Contents
272 275 276
8
Introduction to texture description, segmentation and classification
329
8.1 8.2 8.3
329 330 332 332 332 335 337 339 339 343 343 345 346
8.4
8.5 8.6 8.7
9
10
11
Overview What is texture? Texture description 8.3.1 Performance requirements 8.3.2 Structural approaches 8.3.3 Statistical approaches 8.3.4 Combination approaches Classification 8.4.1 The k-nearest neighbour rule 8.4.2 Other classification approaches Segmentation Further reading References
Appendix 1: Example worksheets
349
9.1 9.2
Example Mathcad worksheet for Chapter 3 Example Matlab worksheet for Chapter 4
349 352
Appendix 2: Camera geometry fundamentals
355
10.1 Image geometry 10.2 Perspective camera 10.3 Perspective camera model 10.3.1 Homogeneous coordinates and projective geometry 10.3.1.1 Representation of a line and duality 10.3.1.2 Ideal points 10.3.1.3 Transformations in the projective space 10.3.2 Perspective camera model analysis 10.3.3 Parameters of the perspective camera model 10.4 Affine camera 10.4.1 Affine camera model 10.4.2 Affine camera model and the perspective projection 10.4.3 Parameters of the affine camera model 10.5 Weak perspective model 10.6 Example of camera models 10.7 Discussion 10.8 References
355 355 357 357 358 358 359 360 363 364 365 366 368 369 371 379 380
Appendix 3: Least squares analysis
381
11.1 The least squares criterion 11.2 Curve fitting by least squares
381 382 Contents ix
12
Appendix 4: Principal components analysis
385
12.1 12.2 12.3 12.4 12.5 12.6 12.7 12.8 12.9 12.10 12.11
385 385 386 388 389 390 391 392 392 393 398
Index
x
Contents
Introduction Data Covariance Covariance matrix Data transformation Inverse transformation Eigenproblem Solving the eigenproblem PCA method summary Example References
399
.
Preface
.
Why did we write this book? We will no doubt be asked many times: why on earth write a new book on computer vision? Fair question: there are already many good books on computer vision in the bookshops, as you will find referenced later, so why add to them? Part of the answer is that any textbook is a snapshot of material that exists before it. Computer vision, the art of processing images stored within a computer, has seen a considerable amount of research by highly qualified people and the volume of research would appear even to have increased in recent years. This means that a lot of new techniques have been developed, and many of the more recent approaches have yet to migrate to textbooks. But it is not just the new research: part of the speedy advance in computer vision technique has left some areas covered only in scanty detail. By the nature of research, one cannot publish material on technique that is seen more to fill historical gaps, rather than to advance knowledge. This is again where a new text can contribute. Finally, the technology itself continues to advance. This means that there is new hardware, and there are new programming languages and new programming environments. In particular for computer vision, the advance of technology means that computing power and memory are now relatively cheap. It is certainly considerably cheaper than when computer vision was starting as a research field. One of the authors here notes that the laptop that his portion of the book was written on has more memory, is faster, and has bigger disk space and better graphics than the computer that served the entire university of his student days. And he is not that old! One of the more advantageous recent changes brought about by progress has been the development of mathematical programming systems. These allow us to concentrate on mathematical technique itself, rather than on implementation detail. There are several sophisticated flavours, of which Mathcad and Matlab, the chosen vehicles here, are among the most popular. We have been using these techniques in research and teaching, and we would argue that they have been of considerable benefit there. In research, they help us to develop technique more quickly and to evaluate its final implementation. For teaching, the power of a modern laptop and a mathematical system combines to show students, in lectures and in study, not only how techniques are implemented, but also how and why they work with an explicit relation to conventional teaching material. We wrote this book for these reasons. There is a host of material that we could have included but chose to omit. Our apologies to other academics if it was your own, or your favourite, technique. By virtue of the enormous breadth of the subject of computer vision, we restricted the focus to feature extraction and image processing in computer vision, for this not only has been the focus of our research, but is also where the attention of established textbooks, with some exceptions, can be rather scanty. It is, however, one of the prime targets of applied computer vision, so would benefit from better attention. We have aimed to clarify some of its origins and development, while also exposing implementation using mathematical systems. As such, we have written this text with our original aims in mind. xi
Why did we produce another edition? There are many reasons why we have updated the book to provide this new edition. First, despite its electronic submission, some of the first edition was retyped before production. This introduced errors that we have now corrected. Next, the field continues to move forward: we now include some techniques which were gaining appreciation when we first wrote the book, or have been developed since. Some areas move more rapidly than others, and this is reflected in the changes made. Also, there has been interim scholarship, especially in the form of new texts, and we include these new ones as much as we can. Matlab and Mathcad are still the computational media here, and there is a new demonstration site which uses Java. Finally, we have maintained the original format. It is always tempting to change the format, in this case even to reformat the text, but we have chosen not to do so. Apart from corrections and clarifications, the main changes from the previous edition are: • Chapter 1: updating of eye operation, camera technology and software, updating and extension of web material and literature • Chapter 2: very little (this is standard material), except for an excellent example of aliasing • Chapter 3: inclusion of anisotropic diffusion for image smoothing, the force field operator and mathematical morphology • Chapter 4: extension of frequency domain concepts and differentiation operators; inclusion of phase congruency, modern curvature operators and the scale invariant feature transform (SIFT) • Chapter 5: emphasis of the practical attributes of feature extraction in occlusion and noise, and some moving-feature techniques • Chapter 6: inclusion of geometric active contours and level set methods, inclusion of skeletonization, extension of active shape models • Chapter 7: extension of the material on moments, particularly Zernike moments, including reconstruction from moments • Chapter 8: clarification of some of the detail in feature-based recognition • Appendices: these have been extended to cover camera models in greater detail, and principal components analysis. As already mentioned, there is a new JAVA-based demonstration site, at http://www.ecs.soton. ac.uk/∼msn/book/new_demo/, which has some of the techniques described herein and some examples of how computer vision-based biometrics work. This webpage will continue to be updated.
The book and its support Each chapter of the book presents a particular package of information concerning feature extraction in image processing and computer vision. Each package is developed from its origins and later referenced to more recent material. Naturally, there is often theoretical development before implementation (in Mathcad or Matlab). We have provided working implementations of most of the major techniques we describe, and applied them to process a selection of imagery. Although the focus of our work has been more in analysing medical imagery or in biometrics xii
Preface
(the science of recognizing people by behavioural or physiological characteristics, like face recognition), the techniques are general and can migrate to other application domains. You will find a host of further supporting information at the book’s website (http://www.ecs.soton.ac.uk/∼msn/book/). First, you will find the worksheets (the Matlab and Mathcad implementations that support the text) so that you can study the techniques described herein. There are also lecturing versions that have been arranged for display via an overhead projector, with enlarged text and more interactive demonstration. The example questions (and, eventually, their answers) are also there. The demonstration site is there too. The website will be kept as up to date as possible, for it also contains links to other material such as websites devoted to techniques and to applications, as well as to available software and online literature. Finally, any errata will be reported there. It is our regret and our responsibility that these will exist, but our inducement for their reporting concerns a pint of beer. If you find an error that we do not know about (not typos such as spelling, grammar and layout) then use the mailto on the website and we shall send you a pint of good English beer, free! There is a certain amount of mathematics in this book. The target audience is third or fourth year students in BSc/BEng/MEng courses in electrical or electronic engineering, software engineering and computer science, or in mathematics or physics, and this is the level of mathematical analysis here. Computer vision can be thought of as a branch of applied mathematics, although this does not really apply to some areas within its remit, but certainly applies to the material herein. The mathematics essentially concerns mainly calculus and geometry, although some of it is rather more detailed than the constraints of a conventional lecture course might allow. Certainly, not all of the material here is covered in detail in undergraduate courses at Southampton. The book starts with an overview of computer vision hardware, software and established material, with reference to the most sophisticated vision system yet ‘developed’: the human vision system. Although the precise details of the nature of processing that allows us to see have yet to be determined, there is a considerable range of hardware and software that allow us to give a computer system the capability to acquire, process and reason with imagery, the function of ‘sight’. The first chapter also provides a comprehensive bibliography of material on the subject, including not only textbooks, but also available software and other material. As this will no doubt be subject to change, it might well be worth consulting the website for more up-to-date information. The preferred journal references are those that are likely to be found in local university libraries or on the web, IEEE Transactions in particular. These are often subscribed to as they are relatively low cost, and are often of very high quality. The next chapter concerns the basics of signal processing theory for use in computer vision. It introduces the Fourier transform, which allows you to look at a signal in a new way, in terms of its frequency content. It also allows us to work out the minimum size of a picture to conserve information and to analyse the content in terms of frequency, and even helps to speed up some of the later vision algorithms. Unfortunately, it does involve a few equations, but it is a new way of looking at data and signals, and proves to be a rewarding topic of study in its own right. We then start to look at basic image-processing techniques, where image points are mapped into a new value first by considering a single point in an original image, and then by considering groups of points. We see not only common operations to make a picture’s appearance better, especially for human vision, but also how to reduce the effects of different types of commonly encountered image noise. This is where the techniques are implemented as algorithms in Mathcad and Matlab to show precisely how the equations work. We shall see some of the modern ways to remove noise and thus clean images, and we shall also look at techniques which process an image using notions of shape, rather than mapping processes.
Preface
xiii
The following chapter concerns low-level features, which are the techniques that describe the content of an image, at the level of a whole image rather than in distinct regions of it. One of the most important processes is edge detection. Essentially, this reduces an image to a form of a caricaturist’s sketch, but without a caricaturist’s exaggerations. The major techniques are presented in detail, together with descriptions of their implementation. Other image properties we can derive include measures of curvature and measures of movement. These also are covered in this chapter. These edges, the curvature or the motion need to be grouped in some way so that we can find shapes in an image. Our first approach to shape extraction concerns analysing the match of low-level information to a known template of a target shape. As this can be computationally very cumbersome, we then progress to a technique that improves computational performance, while maintaining an optimal performance. The technique is known as the Hough transform, and it has long been a popular target for researchers in computer vision who have sought to clarify its basis, improve its speed, and increase its accuracy and robustness. Essentially, by the Hough transform we estimate the parameters that govern a shape’s appearance, where the shapes range from lines to ellipses and even to unknown shapes. Some applications of shape extraction require the determination of rather more than the parameters that control appearance, but require the ability to deform or flex to match the image template. For this reason, the chapter on shape extraction by matching is followed by one on flexible shape analysis. This is a topic that has shown considerable progress of late, especially with the introduction of snakes (active contours). The newer material is the formulation by level set methods, and brings new power to shape-extraction techniques. These seek to match a shape to an image by analysing local properties. Further, we shall see how we can describe a shape by its skeleton, although with practical difficulty which can be alleviated by symmetry (though this can be slow), and also how global constraints concerning the statistics of a shape’s appearance can be used to guide final extraction. Up to this point, we have not considered techniques that can be used to describe the shape found in an image. We shall find that the two major approaches concern techniques that describe a shape’s perimeter and those that describe its area. Some of the perimeter description techniques, the Fourier descriptors, are even couched using Fourier transform theory, which allows analysis of their frequency content. One of the major approaches to area description, statistical moments, also has a form of access to frequency components, but is of a very different nature to the Fourier analysis. One advantage is that insight into descriptive ability can be achieved by reconstruction, which should get back to the original shape. The final chapter describes texture analysis, before some introductory material on pattern classification. Texture describes patterns with no known analytical description and has been the target of considerable research in computer vision and image processing. It is used here more as a vehicle for material that precedes it, such as the Fourier transform and area descriptions, although references are provided for access to other generic material. There is also introductory material on how to classify these patterns against known data, but again this is a window on a much larger area, to which appropriate pointers are given. The appendices include a printout of abbreviated versions of the Mathcad and Matlab worksheets. The other appendices include material that is germane to the text, such as camera models and coordinate geometry, the method of least squares and a topic known as principal components analysis. These are aimed to be short introductions, and are appendices since they are germane to much of the material. Other related, especially online, material is referenced throughout the text.
xiv Preface
In this way, the text covers all major areas of feature extraction in image processing and computer vision. There is considerably more material on the subject than is presented here; for example, there is an enormous volume of material on 3D computer vision and 2D signal processing which is only alluded to here. Topics that are specifically not included are colour, 3D processing and image coding. But to include all that would lead to a monstrous book that no one could afford, or even pick up! So we admit that we give a snapshot, but we hope that it is considered to open another window on a fascinating and rewarding subject.
In gratitude We are immensely grateful to the input of our colleagues, in particular to Prof. Steve Gunn and to Dr John Carter. The family who put up with it are Maria Eugenia and Caz and the nippers. We are also very grateful to past and present researchers in computer vision at the Information, Signals, Images, Systems (ISIS) Research Group under (or who have survived?) Mark’s supervision at the School of Electronics and Computer Science, University of Southampton. As well as Alberto and Steve, these include Dr Hani Muammar, Prof. Xiaoguang Jia, Prof. Yan Qiu Chen, Dr Adrian Evans, Dr Colin Davies, Dr David Cunado, Dr Jason Nash, Dr Ping Huang, Dr Liang Ng, Dr Hugh Lewis, Dr David Benn, Dr Douglas Bradshaw, Dr David Hurley, Dr John Manslow, Dr Mike Grant, Bob Roddis, Dr Andrew Tatem, Dr Karl Sharman, Dr Jamie Shutler, Dr Jun Chen, Dr Andy Tatem, Dr Chew Yam, Dr James Hayfron-Acquah, Dr Yalin Zheng, Dr Jeff Foster, Dr Peter Myerscough, Dr David Wagg, Dr Ahmad Al-Mazeed, Dr Jang-Hee Yoo, Dr Nick Spencer, Stuart Mowbray, Dr Stuart Prismall, Dr Peter Gething, Dr Mike Jewell, Dr David Wagg, Dr Alex Bazin, Hidayah Rahmalan, Xin Liu, Imed Bouchrika, Banafshe Arbab-Zavar, Dan Thorpe, Cem Direkoglu (the latter two especially for the new active contour material), John Evans (for the great hippo photo) and to Jamie Hutton, Ben Dowling and Sina Samangooei (for the Java demonstrations site). We are also very grateful to other past Southampton students of BEng and MEng Electronic Engineering, MEng Information Engineering, BEng and MEng Computer Engineering, MEng Software Engineering and BSc Computer Science who have pointed our earlier mistakes, noted areas for clarification and in some cases volunteered some of the material herein. Beyond Southampton, we remain grateful to the reviewers of the two editions and to Prof. Daniel Cremers, Dr Timor Kadir and Prof. Tim Cootes for observations on and improvements to the text and for permission to use images. To all of you, our very grateful thanks.
Final message We ourselves have already benefited much by writing this book, and this second edition. As we already know, previous students have also benefited, and contributed to it as well. But it remains our hope that it will inspire people to join in this fascinating and rewarding subject that has proved to be such a source of pleasure and inspiration to its many workers. Mark S. Nixon University of Southampton
Alberto S. Aguado University of Surrey
Preface
xv
This page intentionally left blank
1
.
.
Introduction 1.1 Overview This is where we start, by looking at the human visual system to investigate what is meant by vision, then on to how a computer can be made to sense pictorial data and then how we can process an image. The overview of this chapter is shown in Table 1.1; you will find a similar overview at the start of each chapter. There are no references (citations) in the overview, citations are made in the text and are collected at the end of each chapter. Table 1.1 Overview of Chapter 1 Main topic
Sub topics
Main points
Human vision system
How the eye works, how visual information is processed and how it can fail.
Sight, lens, retina, image, colour, monochrome, processing, brain, visual illusions.
Computer vision systems
How electronic images are formed, how video is fed into a computer and how we can process the information using a computer.
Picture elements, pixels, video standard, camera technologies, pixel technology, performance effects, specialist cameras, video conversion, computer languages, processing packages. Demonstrations of working techniques.
Mathematical systems
How we can process images using mathematical packages; introduction to the Matlab and Mathcad systems.
Ease, consistency, support, visualization of results, availability, introductory use, example worksheets.
Literature
Other textbooks and other places to find information on image processing, computer vision and feature extraction.
Magazines, textbooks, websites and this book’s website.
1.2 Human and computer vision A computer vision system processes images acquired from an electronic camera, which is like the human vision system where the brain processes images derived from the eyes. Computer vision is a rich and rewarding topic for study and research for electronic engineers, computer scientists and many others. Increasingly, it has a commercial future. There are now many vision systems in routine industrial use: cameras inspect mechanical parts to check size, food is inspected 1
for quality, and images used in astronomy benefit from computer vision techniques. Forensic studies and biometrics (ways to recognize people) using computer vision include automatic face recognition and recognizing people by the ‘texture’ of their irises. These studies are paralleled by biologists and psychologists who continue to study how our human vision system works, and how we see and recognize objects (and people). A selection of (computer) images is given in Figure 1.1; these images comprise a set of points or picture elements (usually concatenated to pixels) stored as an array of numbers in a computer. To recognize faces, based on an image such as Figure 1.1(a), we need to be able to analyse constituent shapes, such as the shape of the nose, the eyes and the eyebrows, to make some measurements to describe, and then recognize, a face. (Figure 1.1a is perhaps one of the most famous images in image processing. It is called the Lena image, and is derived from a picture of Lena Sjööblom in Playboy in 1972.) Figure 1.1(b) is an ultrasound image of the carotid artery (which is near the side of the neck and supplies blood to the brain and the face), taken as a cross-section through it. The top region of the image is near the skin; the bottom is inside the neck. The image arises from combinations of the reflections of the ultrasound radiation by tissue. This image comes from a study that aimed to produce three-dimensional (3D) models of arteries, to aid vascular surgery. Note that the image is very noisy, and this obscures the shape of the (elliptical) artery. Remotely sensed images are often analysed by their texture content. The perceived texture is different between the road junction and the different types of foliage seen in Figure 1.1(c). Finally, Figure 1.1(d) is a magnetic resonance imaging (MRI) image of a cross-section near the middle of a human body. The chest is at the top of the image, the lungs and blood vessels are the dark areas, and the internal organs and the fat appear grey. Nowadays, MRI images are in routine medical use, owing to their ability to provide high-quality images.
(a) Face from a camera
(b) Artery from ultrasound
(c) Ground by remote sensing
(d) Body by magnetic resonance
Figure 1.1 Real images from different sources
There are many different image sources. In medical studies, MRI is good for imaging soft tissue, but does not reveal the bone structure (the spine cannot be seen in Figure 1.1d); this can be achieved by using computed tomography (CT), which is better at imaging bone, as opposed to soft tissue. Remotely sensed images can be derived from infrared (thermal) sensors or synthetic-aperture radar, rather than by cameras, as in Figure 1.1(c). Spatial information can be provided by two-dimensional arrays of sensors, including sonar arrays. There are perhaps more varieties of sources of spatial data in medical studies than in any other area. But computer vision techniques are used to analyse any form of data, not just the images from cameras. Synthesized images are good for evaluating techniques and finding out how they work, and some of the bounds on performance. Two synthetic images are shown in Figure 1.2. Figure 1.2(a) 2
Feature Extraction and Image Processing
is an image of circles that were specified mathematically. The image is an ideal case: the circles are perfectly defined and the brightness levels have been specified to be constant. This type of synthetic image is good for evaluating techniques which find the borders of the shape (its edges) and the shape itself, and even for making a description of the shape. Figure 1.2(b) is a synthetic image made up of sections of real image data. The borders between the regions of image data are exact, again specified by a program. The image data comes from a well-known texture database, the Brodatz album of textures. This was scanned and stored as a computer image. This image can be used to analyse how well computer vision algorithms can identify regions of differing texture.
(a) Circles
(b) Textures
Figure 1.2 Examples of synthesized images
This chapter will show you how basic computer vision systems work, in the context of the human vision system. It covers the main elements of human vision, showing you how your eyes work (and how they can be deceived). For computer vision, this chapter covers the hardware and the software used for image analysis, giving an introduction to Mathcad and Matlab, the software tools used throughout this text to implement computer vision algorithms. Finally, a selection of pointers to other material is provided, especially those for more detail on the topics covered in this chapter.
1.3 The human vision system Human vision is a sophisticated system that senses and acts on visual stimuli. It has evolved for millions of years, primarily for defence or survival. Intuitively, computer and human vision appear to have the same function. The purpose of both systems is to interpret spatial data, data that is indexed by more than one dimension. Even though computer and human vision are functionally similar, you cannot expect a computer vision system to replicate exactly the function of the human eye. This is partly because we do not understand fully how the vision system of the eye and brain works, as we shall see in this section. Accordingly, we cannot design a system to replicate its function exactly. In fact, some of the properties of the human eye are useful when developing computer vision techniques, whereas others are actually undesirable in a computer vision system. But we shall see computer vision techniques which can, to some extent, replicate, and in some cases even improve upon, the human vision system. Introduction
3
You might ponder this, so put one of the fingers from each of your hands in front of your face and try to estimate the distance between them. This is difficult, and I am sure you would agree that your measurement would not be very accurate. Now put your fingers very close together. You can still tell that they are apart even when the distance between them is tiny. So human vision can distinguish relative distance well, but is poor for absolute distance. Computer vision is the other way around: it is good for estimating absolute difference, but with relatively poor resolution for relative difference. The number of pixels in the image imposes the accuracy of the computer vision system, but that does not come until the next chapter. Let us start at the beginning, by seeing how the human vision system works. In human vision, the sensing element is the eye from which images are transmitted via the optic nerve to the brain, for further processing. The optic nerve has insufficient bandwidth to carry all the information sensed by the eye. Accordingly, there must be some preprocessing before the image is transmitted down the optic nerve. The human vision system can be modelled in three parts: • the eye: this is a physical model since much of its function can be determined by pathology • a processing system: this is an experimental model since the function can be modelled, but not determined precisely • analysis by the brain: this is a psychological model since we cannot access or model such processing directly, but only determine behaviour by experiment and inference.
1.3.1 The eye The function of the eye is to form an image; a cross-section of the eye is illustrated in Figure 1.3. Vision requires an ability to focus selectively on objects of interest. This is achieved by the ciliary muscles that hold the lens. In old age, it is these muscles which become slack and the eye loses its ability to focus at short distance. The iris, or pupil, is like an aperture on a camera and controls the amount of light entering the eye. It is a delicate system and needs protection, which is provided by the cornea (sclera). This is outside the choroid, which has blood vessels
Choroid/sclera Ciliary muscle Lens
Fovea Blind spot Retina
Optic nerve
Figure 1.3 Human eye
4
Feature Extraction and Image Processing
that supply nutrition and is opaque to cut down the amount of light. The retina is on the inside of the eye, which is where light falls to form an image. By this system, muscles rotate the eye, and shape the lens, to form an image on the fovea (focal point), where the majority of sensors are situated. The blind spot is where the optic nerve starts; there are no sensors there. Focusing involves shaping the lens, rather than positioning it as in a camera. The lens is shaped to refract close images greatly, and distant objects little, essentially by ‘stretching’ it. The distance of the focal centre of the lens varies from approximately 14 mm to around 17 mm, depending on the lens shape. This implies that a world scene is translated into an area of about 2 mm2 . Good vision has high acuity (sharpness), which implies that there must be very many sensors in the area where the image is formed. There are nearly 100 million sensors dispersed around the retina. Light falls on these sensors to stimulate photochemical transmissions, which results in nerve impulses that are collected to form the signal transmitted by the eye. There are two types of sensor: first, the rods, which are used for black and white (scotopic) vision; and secondly, the cones, which are used for colour (photopic) vision. There are approximately 10 million cones and nearly all are found within 5 of the fovea. The remaining 100 million rods are distributed around the retina, with the majority between 20 and 5 of the fovea. Acuity is expressed in terms of spatial resolution (sharpness) and brightness/colour resolution and is greatest within 1 of the fovea. There is only one type of rod, but there are three types of cone. These are: • S (short wavelength): these sense light towards the blue end of the visual spectrum • M (medium wavelength): these sense light around green • L (long wavelength): these sense light towards the red region of the spectrum. The total response of the cones arises from summing the response of these three types of cone, which gives a response covering the whole of the visual spectrum. The rods are sensitive to light within the entire visual spectrum, giving the monochrome capability of scotopic vision. Accordingly, when the light level is low, images are formed away from the fovea, to use the superior sensitivity of the rods, but without the colour vision of the cones. Note that there are very few of the bluish cones, and there are many more of the others. But we can still see a lot of blue (especially given ubiquitous denim!). So, somehow, the human vision system compensates for the lack of blue sensors, to enable us to perceive it. The world would be a funny place with red water! The vision response is logarithmic and depends on brightness adaptation from dark conditions where the image is formed on the rods, to brighter conditions where images are formed on the cones. One inherent property of the eye, known as Mach bands, affects the way we perceive images. These are illustrated in Figure 1.4 and are the bands that appear to be where two stripes of constant shade join. By assigning values to the image brightness levels, the cross-section of plotted brightness is shown in Figure 1.4(a). This shows that the picture is formed from stripes of constant brightness. Human vision perceives an image for which the cross-section is as plotted in Figure 1.4(c). These Mach bands do not really exist, but are introduced by your eye. The bands arise from overshoot in the eyes’ response at boundaries of regions of different intensity (this aids us to differentiate between objects in our field of view). The real cross-section is illustrated in Figure 1.4(b). Note also that a human eye can distinguish only relatively few grey levels. It has a capability to discriminate between 32 levels (equivalent to five bits), whereas the image of Figure 1.4(a) could have many more brightness levels. This is why your perception finds it more difficult to discriminate between the low-intensity bands on the left of Figure 1.4(a). (Note that Mach bands cannot be seen in the earlier image of circles, Figure 1.2a, owing to the arrangement Introduction
5
(a) Image showing the Mach band effect
200
200
mach0,x 100
seenx 100
0
50
100
x
(b) Cross-section through (a)
0
50
100
x
(c) Perceived cross-section through (a)
Figure 1.4 Illustrating the Mach band effect
of grey levels.) This is the limit of our studies of the first level of human vision; for those who are interested, Cornsweet (1970) provides many more details concerning visual perception. So we have already identified two properties associated with the eye that it would be difficult to include, and would often be unwanted, in a computer vision system: Mach bands and sensitivity to unsensed phenomena. These properties are integral to human vision. At present, human vision is far more sophisticated than we can hope to achieve with a computer vision system. Infrared guided-missile vision systems can have difficulty in distinguishing between a bird at 100 m and a plane at 10 km. Poor birds! (Lucky plane?) Human vision can handle this with ease.
1.3.2 The neural system Neural signals provided by the eye are essentially the transformed response of the wavelengthdependent receptors, the cones and the rods. One model is to combine these transformed signals by addition, as illustrated in Figure 1.5. The response is transformed by a logarithmic function, mirroring the known response of the eye. This is then multiplied by a weighting factor that controls the contribution of a particular sensor. This can be arranged to allow combination of responses from a particular region. The weighting factors can be chosen to afford particular filtering properties. For example, in lateral inhibition, the weights for the centre sensors are much greater than the weights for those at the extreme. This allows the response of the centre sensors to dominate the combined response given by addition. If the weights in one half are chosen to 6
Feature Extraction and Image Processing
Logarithmic response
Weighting functions
Sensor inputs p1
log(p 1)
w 1 × log(p 1)
p2
log(p 2)
w 2 × log(p 2)
p3
log(p 3)
w 3 × log(p 3)
p4
log(p 4)
w 4 × log(p 4)
p5
log(p 5)
w 5 × log(p 5)
Output ∑
Figure 1.5 Neural processing
be negative, while those in the other half are positive, then the output will show detection of contrast (change in brightness), given by the differencing action of the weighting functions. The signals from the cones can be combined in a manner that reflects chrominance (colour) and luminance (brightness). This can be achieved by subtraction of logarithmic functions, which is then equivalent to taking the logarithm of their ratio. This allows measures of chrominance to be obtained. In this manner, the signals derived from the sensors are combined before transmission through the optic nerve. This is an experimental model, since there are many ways possible to combine the different signals together. Visual information is then sent back to arrive at the lateral geniculate nucleus (LGN), which is in the thalamus and is the primary processor of visual information. This is a layered structure containing different types of cells, with differing functions. The axons from the LGN pass information on to the visual cortex. The function of the LGN is largely unknown, although it has been shown to play a part in coding the signals that are transmitted. It is also considered to help the visual system to focus its attention, such as on sources of sound. For further information on retinal neural networks, see Ratliff (1965); an alternative study of neural processing can be found in Overington (1992).
1.3.3 Processing The neural signals are then transmitted to two areas of the brain for further processing. These areas are the associative cortex, where links between objects are made, and the occipital cortex, where patterns are processed. It is naturally difficult to determine precisely what happens in this region of the brain. To date, there have been no volunteers for detailed study of their brain’s function (although progress with new imaging modalities such as positive emission tomography or electrical impedance tomography will doubtless help). For this reason, there are only psychological models to suggest how this region of the brain operates. It is well known that one function of the human vision system is to use edges, or boundaries, of objects. We can easily read the word in Figure 1.6(a); this is achieved by filling in the Introduction
7
missing boundaries in the knowledge that the pattern is likely to represent a printed word. But we can infer more about this image; there is a suggestion of illumination, causing shadows to appear in unlit areas. If the light source is bright, then the image will be washed out, causing the disappearance of the boundaries which are interpolated by our eyes. So there is more than just physical response, there is also knowledge, including prior knowledge of solid geometry. This situation is illustrated in Figure 1.6(b), which could represent three ‘pacmen’ about to collide, or a white triangle placed on top of three black circles. Either situation is possible.
(a) Word?
(b) Pacmen?
Figure 1.6 How human vision uses edges
It is also possible to deceive human vision, primarily by imposing a scene that it has not been trained to handle. In the famous Zollner illusion (Figure 1.7a), the bars appear to be slanted, whereas in reality they are vertical (check this by placing a pen between the lines): the small cross-bars mislead your eye into perceiving the vertical bars as slanting. In the Ebbinghaus illusion (Figure 1.7b), the inner circle appears to be larger when surrounded by small circles than it is when surrounded by larger circles. There are dynamic illusions too: you can always impress children with the ‘see my wobbly pencil’ trick. Just hold the pencil loosely between your fingers then, to whoops of childish glee, when the pencil is shaken up and down, the solid pencil will appear to bend. Benham’s disk
(a) Zollner
(b) Ebbinghaus
Figure 1.7 Static illusions
8
Feature Extraction and Image Processing
(Figure 1.8) shows how hard it is to model vision accurately. If you make up a version of this disk into a spinner (push a matchstick through the centre) and spin it anticlockwise, you do not see three dark rings, you will see three coloured ones. The outside one will appear to be red, the middle one a sort of green and the inner one deep blue. (This can depend greatly on lighting, and contrast between the black and white on the disk. If the colours are not clear, try it in a different place, with different lighting.) You can appear to explain this when you notice that the red colours are associated with the long lines and the blue with short lines. But this is from physics, not psychology. Now spin the disk clockwise. The order of the colours reverses: red is associated with the short lines (inside) and blue with the long lines (outside). So the argument from physics is clearly incorrect, since red is now associated with short lines, not long ones, revealing the need for psychological explanation of the eyes’ function. This is not colour perception; see Armstrong (1991) for an interesting (and interactive) study of colour theory and perception.
Figure 1.8 Benham’s disk
There are many texts on human vision. One popular text on human visual perception is by Schwartz (2004) and there is an online book, The Joy of Vision (http://www.yorku. ca/eye/thejoy.htm): useful, despite its title! Marr’s seminal text (Marr, 1982) is a computational investigation into human vision and visual perception, investigating it from a computer vision viewpoint. For further details on pattern processing in human vision, see Bruce and Green (1990); for more illusions see Rosenfeld and Kak (1982). Many of the properties of human vision are hard to include in a computer vision system, but let us now look at the basic components that are used to make computers see.
1.4 Computer vision systems Given the progress in computer technology, computer vision hardware is now relatively inexpensive; a basic computer vision system requires a camera, a camera interface and a computer. These days, some personal computers offer the capability for a basic vision system, by including a camera and its interface within the system. There are specialized systems for vision, offering high performance in more than one aspect. These can be expensive, as any specialist system is. Introduction
9
1.4.1 Cameras A camera is the basic sensing element. In simple terms, most cameras rely on the property of light to cause hole/electron pairs (the charge carriers in electronics) in a conducting material. When a potential is applied (to attract the charge carriers), this charge can be sensed as current. By Ohm’s law, the voltage across a resistance is proportional to the current through it, so the current can be turned in to a voltage by passing it through a resistor. The number of hole/electron pairs is proportional to the amount of incident light. Accordingly, greater charge (and hence greater voltage and current) is caused by an increase in brightness. In this manner cameras can provide as output, a voltage that is proportional to the brightness of the points imaged by the camera. Cameras are usually arranged to supply video according to a specified standard. Most will aim to satisfy the CCIR standard that exists for closed circuit television (CCTV) systems. There are three main types of camera: vidicons, charge coupled devices (CCDs) and, more recently, CMOS cameras (complementary metal oxide silicon, now the dominant technology for logic circuit implementation). Vidicons are the older (analogue) technology which, although cheap (mainly by virtue of longevity in production), are being replaced by the newer CCD and CMOS digital technologies. The digital technologies now dominate much of the camera market because they are lightweight and cheap (with other advantages) and are therefore used in the domestic video market. Vidicons operate in a manner akin to a television in reverse. The image is formed on a screen, and then sensed by an electron beam that is scanned across the screen. This produces an output which is continuous; the output voltage is proportional to the brightness of points in the scanned line, and is a continuous signal, a voltage which varies continuously with time. In contrast, CCDs and CMOS cameras use an array of sensors; these are regions where charge is collected which is proportional to the light incident on that region. This is then available in discrete, or sampled, form as opposed to the continuous sensing of a vidicon. This is similar to human vision with its array of cones and rods, but digital cameras use a rectangular regularly spaced lattice, whereas human vision uses a hexagonal lattice with irregular spacing. Two main types of semiconductor pixel sensor are illustrated in Figure 1.9. In the passive sensor, the charge generated by incident light is presented to a bus through a pass transistor.
VDD
Tx Reset
Incident light Column bus
Incident light
Select
Column bus (a) Passive
Figure 1.9 Pixel sensors
10 Feature Extraction and Image Processing
(b) Active
When the signal Tx is activated, the pass transistor is enabled and the sensor provides a capacitance to the bus, one that is proportional to the incident light. An active pixel includes an amplifier circuit that can compensate for limited fill factor of the photodiode. The select signal again controls presentation of the sensor’s information to the bus. A further reset signal allows the charge site to be cleared when the image is rescanned. The basis of a CCD sensor is illustrated in Figure 1.10. The number of charge sites gives the resolution of the CCD sensor; the contents of the charge sites (or buckets) need to be converted to an output (voltage) signal. In simple terms, the contents of the buckets are emptied into vertical transport registers which are shift registers moving information towards the horizontal transport registers. This is the column bus supplied by the pixel sensors. The horizontal transport registers empty the information row by row (point by point) into a signal conditioning unit, which transforms the sensed charge into a voltage which is proportional to the charge in a bucket, and hence proportional to the brightness of the corresponding point in the scene imaged by the camera. The CMOS cameras are like a form of memory: the charge incident on a particular site in a two-dimensional lattice is proportional to the brightness at a point. The charge is then read like computer memory. (In fact, a computer RAM chip can act as a rudimentary form of camera when the circuit, the one buried in the chip, is exposed to light.)
Vertical transport register
Vertical transport register
Vertical transport register
Horizontal transport register Signal conditioning
Video output
Control
Control inputs
Pixel sensors
Figure 1.10 CCD sensing element
There are many more varieties of vidicon (Chalnicon, etc.) than there are of CCD technology (charge injection device, etc.), perhaps owing to the greater age of basic vidicon technology. Vidicons are cheap but have a number of intrinsic performance problems. The scanning process essentially relies on moving parts. As such, the camera performance will change with time, as parts wear; this is known as ageing. Also, it is possible to burn an image into the scanned screen by using high incident light levels; vidicons can also suffer lag, that is, a delay in response to moving objects in a scene. The digital technologies are dependent on the physical arrangement of charge sites and as such do not suffer from ageing, but can suffer from irregularity in the charge sites’ (silicon) material. The underlying technology also makes CCD and CMOS cameras Introduction
11
less sensitive to lag and burn, but the signals associated with the CCD transport registers can give rise to readout effects. Charge coupled devices only came to dominate camera technology when technological difficulty associated with quantum efficiency (the magnitude of response to incident light) for the shorter, blue, wavelengths was solved. One of the major problems in CCD cameras is blooming, where bright (incident) light causes a bright spot to grow and disperse in the image (this used to happen in the analogue technologies too). This happens much less in CMOS cameras because the charge sites can be much better defined and reading their data is equivalent to reading memory sites as opposed to shuffling charge between sites. Also, CMOS cameras have now overcome the problem of fixed pattern noise that plagued earlier MOS cameras. The CMOS cameras are actually much more recent than CCDs. This begs a question as to which is better: CMOS or CCD? Given that they will be both be subject to much continued development, CMOS is a cheaper technology and it lends itself directly to intelligent cameras with on-board processing. This is mainly because the feature size of points (pixels) in a CCD sensor is limited to be about 4 m so that enough light is collected. In contrast, the feature size in CMOS technology is considerably smaller, currently at around 01 m. Accordingly, it is now possible to integrate signal processing within the camera chip and thus it is perhaps possible that CMOS cameras will eventually replace CCD technologies for many applications. However, the more modern CCDs also have on-board circuitry, and their process technology is more mature, so the debate will continue. Finally, there are specialist cameras, which include high-resolution devices, which can give pictures with a great number of points, low-light level cameras, which can operate in very dark conditions (this is where vidicon technology is still found), and infrared cameras, which sense heat to provide thermal images. Increasingly, hyperspectral cameras are available, which have more sensing bands. For more detail concerning modern camera practicalities and imaging systems, see Nakamura (2005). For more detail on sensor development, particularly CMOS, Fossum (1997) is well worth a look.
1.4.2 Computer interfaces This technology is in a rapid state of change, owing to the emergence of digital cameras. Essentially, the image sensor converts light into a signal which is expressed either as a continuous signal or in sampled (digital) form. Some (older) systems expressed the camera signal as an analogue continuous signal, according to a standard, often the CCIR standard, and this was converted at the computer (and still is in some cases). Modern digital systems convert the sensor information into digital information with on-chip circuitry and then provide the digital information according to a specified standard. The older systems, such as surveillance systems, supplied (or supply) video, whereas the newer systems are digital. Video implies delivering the moving image as a sequence of frames and these can be in analogue (continuous) or discrete (sampled) form, of which one format is digital video (DV). An interface that converts an analogue signal into a set of digital numbers is called a framegrabber, since it grabs frames of data from a video sequence, and is illustrated in Figure 1.11. Note that cameras that provide digital information do not need this particular interface (it is inside the camera). However, an analogue camera signal is continuous and is transformed into digital (discrete) format using an analogue-to-digital (A/D) converter. Flash converters are usually used owing to the high speed required for conversion, say 11 MHz, which cannot be met by any other conversion technology. Usually, 8 bit A/D converters are used; at 6 dB/bit, this gives 48 dB, which just satisfies the CCIR stated bandwidth of approximately 45 dB. The output of the A/D converter is often fed to look-up tables (LUTs), which implement 12
Feature Extraction and Image Processing
Input video
Signal conditioning
A/D converter
Control
Look-up table
Image memory
Computer interface Computer
Figure 1.11 A computer interface: a framegrabber
designated conversion of the input data, but in hardware rather than in software, and this is very fast. The outputs of the A/D converter are then stored. Note that there are aspects of the sampling process that are of considerable interest in computer vision; these are covered in Chapter 2. In digital camera systems this processing is usually performed on the camera chip, and the camera eventually supplies digital information, often in coded form. IEEE 1394 (or firewire) is a way of connecting devices external to a computer and is often used for digital video cameras as it supports high-speed digital communication and can provide power; this is similar to universal serial bus (USB), which can be used for still cameras. Firewire needs a connection system and software to operate it, and these can be easily acquired. One important aspect of Firewire is its support of isochronous transfer operation which guarantees timely delivery of data, which is of importance in video-based systems. There are many different ways to design framegrabber units, especially for specialist systems. Note that the control circuitry has to determine exactly when image data is to be sampled. This is controlled by synchronization pulses that are supplied within the video signal: the sync signals, which control the way video information is constructed. Television pictures are constructed from a set of lines, those lines scanned by a camera. To reduce requirements on transmission (and for viewing), the 625 lines in the PAL system (NTSC is of lower resolution) are transmitted in two fields, each of 312.5 lines, as illustrated in Figure 1.12. (Currently, in high-definition television,
Aspect ratio 4 3
Television picture
Even field lines
Odd field lines
Figure 1.12 Interlacing in television pictures
Introduction
13
there is some debate between the corporations who do not want interlacing, and those who do, e.g. the television broadcasters.) If you look at a television, but not directly, the flicker due to interlacing can be perceived. When you look at the television directly, persistence in the human eye ensures that you do not see the flicker. These fields are called the odd and even fields. There is also an aspect ratio in picture transmission: pictures are arranged to be 1.33 times longer than they are high. These factors are chosen to make television images attractive to human vision, and can complicate the design of a framegrabber unit. Some conversion systems accept PAL or NTSC video and convert it to the firewire system. Nowadays, digital video cameras can provide digital output, in progressive scan (without interlacing), delivering sequences of images that are readily processed. Or there are webcams, or just digital camera systems that deliver images straight to the computer. Life just gets easier! This completes the material we need to cover for basic computer vision systems. For more detail concerning the practicalities of computer vision systems see, for example, Davies (2005) (especially for product inspection) or Umbaugh (2005) (and both offer much more than this).
1.4.3 Processing an image Most image processing and computer vision techniques are implemented in computer software. Often, only the simplest techniques migrate to hardware, although coding techniques to maximize efficiency in image transmission are of sufficient commercial interest that they have warranted extensive, and very sophisticated, hardware development. The systems include the Joint Photographic Expert Group (JPEG) and the Moving Picture Expert Group (MPEG) image coding formats. C, C++ and JavaTM are by now the most popular languages for vision system implementation, because of strengths in integrating high- and low-level functions, and the availability of good compilers. As systems become more complex, C++ and Java become more attractive when encapsulation and polymorphism may be exploited. Many people use Java as a development language partly because of platform independence, but also because of ease in implementation (although some claim that speed and efficiency are not as good as in C/C++). There is considerable implementation advantage associated with use of the Java Advanced Imaging API (application programming interface). There is an online demonstration site, for educational purposes only, associated with this book, to be found of the book’s website at http://www.ecs.soton.ac.uk/∼msn/book/new_demo/. This is based around Java, so that the site can be used over the web (as long as Java is installed and up to date). Some textbooks offer image processing systems implemented in these languages. Many commercial packages are available, although these are often limited to basic techniques, and do not include the more sophisticated shape extraction techniques. The Visiquest (was Khoros) image processing system has attracted much interest; this is a schematic (data-flow) image processing system where a user links together chosen modules. This allows for better visualization of information flow during processing. However, the underlying mathematics is not made clear to the user, as it can be when a mathematical system is used. There is a textbook, and a very readable one at that, by Efford (2000), which is based entirely on Java and includes, via a CD, the classes necessary for image processing software development. Other popular texts include those that present working algorithms, such as Seul et al. (2001) and Parker (1996). In terms of software packages, one of the most popular is OpenCV, whose philosophy is to ‘aid commercial uses of computer vision in human–computer interface, robotics, monitoring, biometrics and security by providing a free and open infrastructure where the distributed efforts of the vision community can be consolidated and performance optimized’. This contains a wealth 14
Feature Extraction and Image Processing
of technique and (optimized) implementation; there is even a Wikipedia entry and a discussion website supporting it. Then there are the VXL libraries (the Vision-something-Libraries, groan). This is ‘a collection of C++ libraries designed for computer vision research and implementation’. Finally, there is Adobe’s Generic Image Library (GIL), which aims to ease difficulties with writing imaging-related code that is both generic and efficient. Note that these are open source, but there are licences and conditions on use and exploitation. A set of web links is shown in Table 1.2 for established freeware and commercial software image processing systems. Perhaps the best selection can be found at the general site, from the computer vision homepage software site at Carnegie Mellon (repeated later in Table 1.5).
Table 1.2 Software websites Packages (freeware or student version indicated by ∗ ) General Site
Carnegie Mellon
http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html (large popular index including links to research code, image processing toolkits, and display tools)
Visiquest (Khoros)
Accusoft
http://www.accusoft.com/
Hannover University
http://www.tnt.uni-hannover.de/soft/imgproc/khoros/
AdOculos (+ Textbook)
The Imaging Source
http://www.theimagingsource.com/
CVIPtools∗
Southern Illinois University
http://www.ee.siue.edu/CVIPtools/
LaboImage∗
Geneva University
http://cuiwww.unige.ch/∼vision/LaboImage/labo.html
∗
∗
TN-Image
Thomas J. Nelson
http://brneurosci.org/tnimage.html (scientific image analysis)
OpenCV
Intel
http://www.intel.com/technology/computing/opencv/index.htm and http://sourceforge.net/projects/opencvlibrary/
VXL
Many international contributors
http://vxl.sourceforge.net/
GIL
Adobe
http://opensource.adobe.com/gil/
1.5 Mathematical systems Several mathematical systems have been developed. These offer what is virtually a wordprocessing system for mathematicians. Many are screen based, using a Windows system. The advantage of these systems is that you can transpose mathematics pretty well directly from textbooks, and see how it works. Code functionality is not obscured by the use of data structures, although this can make the code appear cumbersome. A major advantage is that the system provides low-level functionality and data visualization schemes, allowing the user to concentrate on techniques alone. Accordingly, these systems afford an excellent route to understand, and appreciate, mathematical systems before the development of application code, and to check that the final code works correctly. Introduction
15
1.5.1 Mathematical tools Mathcad, Mathematica, Maple and Matlab are among the most popular of current tools. There have been surveys that compare their efficacy, but it is difficult to ensure precise comparison owing to the impressive speed of development of techniques. Most systems have their protagonists and detractors, as in any commercial system. Many books use these packages for particular subjects, and there are often handbooks as addenda to the packages. We shall use both Matlab and Mathcad throughout this text as they two very popular mathematical systems. We shall describe Matlab later, as it is different from Mathcad, although the aim is the same. The website links for the main mathematical packages are given in Table 1.3. Table 1.3 Mathematical package websites General Guide to available mathematical software
NIST
http://gams.nist.gov/ Vendors
Mathcad
MathSoft
http://www.mathcad.com/
Mathematica
Wolfram Research
http://www.wolfram.com/
Matlab
Mathworks
http://www.mathworks.com/
Maple
Maplesoft
http://www.maplesoft.com/
1.5.2 Hello Mathcad, hello images! Mathcad uses worksheets to implement mathematical analysis. The flow of calculation is very similar to using a piece of paper: calculation starts at the top of a document, and flows left to right and downwards. Data is available to later calculation (and to calculation to the right), but is not available to prior calculation, much as is this case when calculation is written manually on paper. Mathcad uses the Maple mathematical library to extend its functionality. To ensure that equations can migrate easily from a textbook to application, Mathcad uses a WYSIWYG (what you see is what you get) notation [its equation editor is not dissimilar to the Microsoft Equation (Word) editor]. Mathcad offers a compromise between many performance factors, and is available at low cost. There used to be a free worksheet viewer called Mathcad Explorer which operated in read-only mode, which is an advantage lost. An image processing handbook is available with Mathcad, but it does not include many of the more sophisticated feature extraction techniques. Images are actually spatial data, data which is indexed by two spatial coordinates. The camera senses the brightness at a point with coordinates x,y. Usually, x and y refer to the horizontal and vertical axes, respectively. Throughout this text we shall work in orthographic projection, ignoring perspective, where real-world coordinates map directly to x and y coordinates in an image. The homogeneous coordinate system is a popular and proven method for handling threedimensional coordinate systems (x, y and z, where z is depth). Since it is not used directly in the text, it is included as Appendix 2 (Section 10.3). The brightness sensed by the camera is transformed to a signal which is then fed to the A/D converter and stored as a value within the computer, referenced to the coordinates x,y in the image. Accordingly, a computer image is a 16
Feature Extraction and Image Processing
matrix of points. For a greyscale image, the value of each point is proportional to the brightness of the corresponding point in the scene viewed, and imaged, by the camera. These points are the picture elements, or pixels. Consider, for example, the matrix of pixel values in Figure 1.13(a). This can be viewed as a surface (or function) in Figure 1.13(b), or as an image in Figure 1.13(c). In Figure 1.13(c) the brightness of each point is proportional to the value of its pixel. This gives the synthesized image of a bright square on a dark background. The square is bright where the pixels have a value around 40 brightness levels; the background is dark, and these pixels have a value near 0 brightness levels. This image is first given a label, pic, and then pic is allocated, :=, to the matrix defined by using the matrix dialogue box in Mathcad, specifying a matrix with eight rows and eight columns. The pixel values are then entered one by one until the matrix is complete (alternatively, the matrix can be specified by using a subroutine, but that comes later). Note that neither the background nor the square has a constant brightness. This is because noise has been added to the image. If we want to evaluate the performance of a computer vision technique on an image, but without the noise, we can simply remove it (one of the advantages of using synthetic images). The matrix becomes an image when it is viewed as a picture, in Figure 1.13(c). This is done either by presenting it as a surface plot, rotated by zerodegrees and viewed from above, or by using Mathcad’s picture facility. As a surface plot, Mathcad allows the user to select a greyscale image, and the patch plot option allows an image to be presented as point values.
1 2
3
4
1
1
2 1
2 2
3
2
1
2
2 1
3 1 38 39 37 36 3 1 pic
4 1 45 44 41 42 2 1 1 2 43 44 40 39 1 3 2 1 39 41 42 40 2 1 1 2
1
2
2
3
1 1
1 2
1
3
1
1
4 2
40 30 20 10 0 2 4 6
0
2
4
6
pic
(a) Matrix
(b) Surface plot
(c) Image
Figure 1.13 Synthesized image of a square
Mathcad stores matrices in row–column format. The coordinate system used throughout this text has x as the horizontal axis and y as the vertical axis (as conventional). Accordingly, x is the column count and y is the row count, so a point (in Mathcad) at coordinates x,y is actually accessed as picyx . The origin is at coordinates x = 0 and y = 0, so pic00 is the magnitude of the point at the origin and pic22 is the point at the third row and third column and pic32 is the point at the third column and fourth row, as shown in Code 1.1 (the points can be seen
pic2,2=38 pic3,2=45 rows(pic)=8 cols(pic)=8 Code 1.1 Accessing an image in Mathcad
Introduction
17
in Figure 1.13a). Since the origin is at (0,0) the bottom right-hand point, at the last column and row, has coordinates (7,7). The number of rows and the number of columns in a matrix, the dimensions of an image, can be obtained by using the Mathcad rows and cols functions, respectively, and again in Code 1.1. This synthetic image can be processed using the Mathcad programming language, which can be invoked by selecting the appropriate dialogue box. This allows for conventional for, while and if statements, and the earlier assignment operator which is := in non-code sections, is replaced by [back-arrow] in sections of code. A subroutine that inverts the brightness level at each point, by subtracting it from the maximum brightness level in the original image, is illustrated in Code 1.2. This uses for loops to index the rows and the columns, and then calculates a new pixel value by subtracting the value at that point from the maximum obtained by Mathcad’s max function. When the whole image has been processed, the new picture is returned to be assigned to the label newpic. The resulting matrix is shown in Figure 1.14(a). When this is viewed as a surface (Figure 1.14b), the inverted brightness levels mean that the square appears dark and its surroundings appear white, as in Figure 1.14(c).
new_pic:=
for x∈0..cols(pic)-1 for y∈0..rows(pic)-1 newpicturey,x←max(pic)-picy,x newpicture
Code 1.2 Processing image points in Mathcad
44 43 42 41 44 44 43 44 43 43 42 43 44 43 43 44
new_pic =
42 44 7
6
8
9
42 44
41 44 0
1
4
3
43 44
44 43 2
1
5
6
44 42
43 44 6
4
3
5
43 44
40 30 20 10 0 2 4 6
0
2
4
6
44 43 44 43 43 42 44 44 44 43 44 42 44 44 41 43
new_pic (a) Matrix
(b) Surface plot
(c) Image
Figure 1.14 Image of a square after division
Routines can be formulated as functions, so they can be invoked to process a chosen picture, rather than restricted to a specific image. Mathcad functions are conventional; we simply add two arguments (one is the image to be processed, the other is the brightness to be added) and use the arguments as local variables, to give the add function illustrated in Code 1.3. To add a value, we simply call the function and supply an image and the chosen brightness level as the arguments. 18
Feature Extraction and Image Processing
add_value(inpic,value):= for x∈0..cols(inpic)–1 for y∈0..rows(inpic)–1 newpicturey,x ← inpicy,x+value newpicture Code 1.3 Function to add a value to an image in Mathcad
Mathematically, for an image which is a matrix of N × N points, the brightness of the pixels in a new picture (matrix), N, are the result of adding b brightness values to the pixels in the old picture, O, given by: Nxy = Oxy + b
∀x y ∈ 1 N
(1.1)
Real images have many points. Unfortunately, the Mathcad matrix dialogue box only allows matrices that are 10 rows and 10 columns at most, i.e. a 10 × 10 matrix. Real images can be 512×512, but are often 256×256 or 128×128; this implies a storage requirement for 262 ×144, 65 × 536 and 16 × 384 pixels, respectively. Since Mathcad stores all points as high-precision, complex floating point numbers, 512 × 512 images require too much storage, but 256 × 256 and 128 × 128 images can be handled with ease. Since this cannot be achieved by the dialogue box, Mathcad has to be ‘tricked’ into accepting an image of this size. Figure 1.15 shows an image
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0 150 145 145 145 150 159 152 152 151 145
0 170 165 165 165 170 179 172 172 171 165
1 159 151 151 152 159 159 159 151 145 145
1 179 171 171 172 179 179 179 171 165 165
2 159 152 151 151 159 159 159 152 134 145
2 179 172 171 171 179 179 179 172 154 165
3 159 145 137 134 145 151 152 151 145 152
3 179 165 157 154 165 171 172 171 165 172
oldhippo = 4 145 142 128 128 134 145 145 151 150 159
newhippo = 4 165 162 148 148 154 165 165 171 170 179
5 134 145 142 137 134 145 145 145 151 152
5 154 165 162 157 154 165 165 165 171 172
6 142 145 151 142 145 151 151 145 159 159
6 162 165 171 162 165 171 171 165 179 179
7 145 151 152 145 134 145 152 159 170 170
7 165 171 172 165 154 165 172 179 190 190
8 152 159 158 151 145 142 151 152 170 152
8 172 179 178 171 165 162 171 172 190 172
9 158 158 152 152 142 134 145 159 159 151
9 178 178 172 172 162 154 165 179 179 171
(a) Part of original image as a matrix
(b) Part of processed image as a matrix
(c) Bitmap of original image
(d) Bitmap of processed image
Figure 1.15 Processing an image
Introduction
19
captured by a camera. This image has been stored in Windows bitmap (.BMP) format. This can be read into a Mathcad worksheet using the READBMP command (in capitals – Mathcad cannot handle readbmp), and is assigned to a variable. It is inadvisable to attempt to display this using the Mathcad surface plot facility, as it can be slow for images and requires a lot of memory. It is best to view an image using Mathcad’s picture facility or to store it using the WRITEBMP command, and then look at it using a bitmap viewer. So, we can make an image brighter, by addition, by the routine in Code 1.3, via the code in Code 1.4. The result is shown in Figure 1.15. The matrix listings in Figure 1.15(a) and (b) show that 20 has been added to each point (these only show the top left-hand section of the image where the bright points relate to the grass; the darker points on, say, the ear cannot be seen). The effect will be to make each point appear brighter, as seen by comparison of the (darker) original image (Figure 1.15c) with the (brighter) result of addition (Figure 1.15d). In Chapter 3 we will investigate techniques that can be used to manipulate the image brightness to show the face in a much better way. For the moment, we are just seeing how Mathcad can be used, in a simple way, to process pictures.
oldhippo:=READBMP(“hippo_orig”) newhippo:=add_value(hippo,20) WRITEBMP(“hippo_brighter.bmp”):=newhippo Code 1.4 Processing an image
Mathcad was used to generate the image used to demonstrate the Mach band effect; the code is given in Code 1.5. First, an image is defined by copying the hippo image (from Code 1.4) to an image labelled mach. Then, the floor function (which returns the nearest integer less than its argument) is used to create the bands, scaled by an amount appropriate to introduce sufficient contrast (the division by 21.5 gives six bands in the image of Figure 1.4a). The cross-section and the perceived cross-section of the image were both generated by Mathcad’s X–Y plot facility, using appropriate code for the perceived cross-section.
mach:=
for x∈0..cols(mach)–1 for y∈0..rows(mach)–1 ⎛ ⎞ x mach y,x ← brightness ·floor ⎜ ⎟ ⎝ bar_width ⎠ mach
Code 1.5 Creating the image of Figure 4(a)
The translation of the Mathcad code into application can be rather prolix when compared with the Mathcad version by the necessity to include low-level functions. Since these can obscure the basic image processing functionality, Mathcad is used throughout this book to show how the techniques work. The translation to application code is perhaps easier via Matlab, as it offers direct compilation of the code. There is also an electronic version of this book, which is a collection of worksheets to help you to learn the subject; and an example Mathcad worksheet 20
Feature Extraction and Image Processing
is given in Appendix 1 (Section 9.1 for Mathcad; 9.2 for Matlab). You can download these worksheets from this book’s website (http://www.ecs.soton.ac.uk/∼msn/book/) and there is a link to the old Mathcad Explorer there too. You can then use the algorithms as a basis for developing your own application code. This provides a good way to verify that your code actually works: you can compare the results of your final application code with those of the original mathematical description. If your final application code and the Mathcad implementation are both correct, the results should be the same. Your application code will be much faster than in Mathcad, and will benefit from the graphical user interface (GUI) that you have developed.
1.5.3 Hello Matlab! Matlab is rather different from Mathcad. It is not a WYSIWYG system, but instead it is more screen based. It was originally developed for matrix functions, hence the ‘Mat’ in the name. Like Mathcad, it offers a set of mathematical tools and visualization capabilities in a manner arranged to be very similar to conventional computer programs. In some users’ views, a WYSIWYG system like Mathcad is easier to start with, but there are a number of advantages to Matlab, not least the potential speed advantage in computation and the facility for debugging, together with a considerable amount of established support. Again, there is an image processing toolkit supporting Matlab, but it is rather limited compared with the range of techniques exposed in this text. Its popularity is reflected in a book dedicated to use of Matlab for image processing (Gonzalez et al., 2003), by perhaps one of the subject’s most popular authors. Essentially, Matlab is the set of instructions that process the data stored in a workspace, which can be extended by user-written commands. The workspace stores the different lists of data and these data can be stored in a MAT file; the user-written commands are functions that are stored in M-files (files with extension .M). The procedure operates by instructions at the command line to process the workspace data using either one of Matlab’s commands or your own commands. The results can be visualized as graphs, surfaces or images, as in Mathcad. Matlab provides powerful matrix manipulations to develop and test complex implementations. In this book, we avoid matrix implementations in favour of a more C++ algorithmic form. Thus, matrix expressions are transformed into loop sequences. This helps students without experience in matrix algebra to understand and implement the techniques without dependency on matrix manipulation software libraries. Implementations in this book only serve to gain understanding of the techniques’ performance and correctness, and favour clarity rather than speed. The system runs on Unix/Linux or Windows and on Macintosh systems. A student version is available at low cost. There is no viewer available for Matlab; you have to have access to a system for which it is installed. As the system is not based around worksheets, we shall use a script which is the simplest type of M-file, as illustrated in Code 1.6. To start the Matlab system, type MATLAB at the command line. At the Matlab prompt (>>) type chapter1 to load and run the script (given that the file chapter1.m is saved in the directory you are working in). Here, we can see that there are no text boxes and so comments are preceded by %. The first command is one that allocates data to our variable pic. There is a more sophisticated way to input this in the Matlab system, but that is not available here. The points are addressed in row–column format and the origin is at coordinates y = 1 and x = 1. So we access these points pic33 as the third column of the third row and pic43 as the point in the third column of the fourth row. Having set the display facility to black and white, we can view the array pic as a surface. When the surface (Figure 1.16a), is plotted, then Matlab has been made to pause until you press Return before moving on. Here, when you press Return, you will next see the image of the array (Figure 1.16b). Introduction
21
%Chapter 1 Introduction (Hello Matlab) CHAPTER1.M %Written by: Mark S. Nixon disp(‘Welcome to the Chapter1 script’) disp(‘This worksheet is the companion to Chapter 1 and is an introduction.’) disp(‘It is the source of Section 1.4.3 Hello Matlab.’) disp(‘The worksheet follows the text directly and allows you to process basic images.’) disp(‘Let us define a matrix, a synthetic computer image called pic.’) pic = [1 2 3 4 1 2 1 1
2 2 1 1 2 1 2 2
3 3 38 45 43 39 1 1
4 2 39 44 44 41 2 3
1 1 37 41 40 42 2 1
1 2 36 42 39 40 3 1
2 2 3 2 1 2 1 4
1; 1; 1; 1; 3; 1; 1; 2]
%Pixels are addressed in row-column format. %Using x for the horizontal axis (a column count), and y for the %vertical axis (a row count) then picture points are addressed as %pic(y,x). The origin is at coordinates (1,1), so the point %pic(3,3) is on the third row and third column; the point pic(4,3) %is on the fourth row, at the third column. Let’s print them: disp (‘The element pic(3,3) is’) pic(3,3) disp(‘The element pic(4,3)is’) pic(4,3) %We’ll set the output display to black and white colormap(gray); %We can view the matrix as a surface plot disp (‘We shall now view it as a surface plot (play with the controls to see it in relief)’) disp(‘When you are ready to move on, press RETURN’) surface(pic); %Let’s hold awhile so we can view it pause; %Or view it as an image disp (‘We shall now view the array as an image’) disp(‘When you are ready to move on, press RETURN’) imagesc(pic); %Let’s hold awhile so we can view it pause;
22 Feature Extraction and Image Processing
%Let’s look at the array’s dimensions disp(‘The dimensions of the array are’) size(pic) %now let’s invoke a routine that inverts the image inverted_pic=invert(pic); %Let’s print it out to check it disp(‘When we invert it by subtracting each point from the maximum, we get’) inverted_pic %And view it disp(‘And when viewed as an image, we see’) disp(‘When you are ready to move on, press RETURN’) imagesc(inverted_pic); %Let’s hold awhile so we can view it pause; disp(‘We shall now read in a bitmap image, and view it’) disp(‘When you are ready to move on, press RETURN’) face=imread(‘rhdark.bmp’,‘bmp’); imagesc(face); pause; %Change from unsigned integer(uint8) to double precision so we can process it face=double(face); disp(‘Now we shall invert it, and view the inverted image’) inverted_face=invert(face); imagesc(inverted_face); disp(‘So we now know how to process images in Matlab. We shall be using this later!’) Code 1.6 Matlab script for Chapter 1
1
50
2
40 30
3
20 4
10 0 8
5 6
6 4 2 0
1
2
3
4
5
6
7
8
7 8 1
(a) Matlab surface plot
2
3
4
5
6
7
8
(b) Matlab image
Figure 1.16 Matlab image visualization
Introduction
23
We can use Matlab’s own command to interrogate the data; these commands find use in the M-files that store subroutines. An example routine is called after this. This subroutine is stored in a file called invert.m and is a function that inverts brightness by subtracting the value of each point from the array’s maximum value. The code is illustrated in Code 1.7. Note that this code uses for loops, which are best avoided to improve speed, using Matlab’s vectorized operations (as in Mathcad). The whole procedure can be implemented by the command inverted=max(max(pic))-pic. One of Matlab’s assets is a ‘profiler’ which allows you to determine exactly how much time is spent on different parts of your programs. There is facility for importing graphics files, which is actually rather more extensive (i.e. it accepts a wider range of file formats) than available in Mathcad. When images are used, this reveals that unlike Mathcad, which stores all variables as full precision real numbers, Matlab has a range of datatypes. We must move from the unsigned integer datatype, used for images, to the double precision datatype to allow processing as a set of real numbers. In these ways Matlab can and will be used to process images throughout this book. As with the Mathcad worksheets, there are Matlab scripts available at the website for online tutorial support of the material in this book; an abbreviated example worksheet is given in Appendix 1 (Section 9.2).
function inverted=invert(image) %Subtract image point brightness from maximum % %Usage:[new image]=invert(image) % %Parameters: image-array of points % %Author: Mark S.Nixon %get dimensions [rows,cols]=size(image); %find the maximum maxi=max(max(image)); %subtract image points from maximum for x=1:cols %address all columns for y=1:rows %address all rows inverted(y,x)=maxi-image(y,x); end end Code 1.7 Matlab function (invert.m) to invert an image
1.6 Associated literature 1.6.1 Journals and magazines As in any academic subject, there are many sources of literature. The professional magazines include those that are more systems orientated, like Vision Systems Design and Advanced Imaging. These provide more general articles, and are often a good source of information about 24
Feature Extraction and Image Processing
new computer vision products. For example, they survey available equipment, such as cameras and monitors, and provide listings of those available, including some of the factors by which you might choose to purchase them. There is a wide selection of research journals, probably more than you can find in your nearest library unless it is particularly well stocked. These journals have different merits: some are targeted at short papers only, whereas some have short and long papers; some are more dedicated to the development of new theory, whereas others are more pragmatic and focus more on practical, working, image processing systems. But it is rather naive to classify journals in this way, since all journals welcome good research, with new ideas, which has been demonstrated to satisfy promising objectives. The main research journals include: IEEE Transactions on: Pattern Analysis and Machine Intelligence (in later references this will be abbreviated to IEEE Trans. on PAMI); Image Processing (IP); Systems, Man and Cybernetics (SMC); and Medical Imaging (there are many more IEEE transactions, some of which sometimes publish papers of interest in image processing and computer vision). The IEEE Transactions are usually found in (university) libraries since they are available at comparatively low cost; they are online to subscribers at the IEEE Explore site (http://ieeexplore.ieee.org/) and this includes conferences and IEE Proceedings (described soon). Computer Vision and Image Understanding and Graphical Models and Image Processing arose from the splitting of one of the subject’s earlier journals, Computer Vision, Graphics and Image Processing (CVGIP), into two parts. Do not confuse Pattern Recognition (Pattern Recog.) with Pattern Recognition Letters (Pattern Recog. Lett.), published under the aegis of the Pattern Recognition Society and the International Association of Pattern Recognition, respectively, since the latter contains shorter papers only. The International Journal of Computer Vision is a more recent journal, whereas Image and Vision Computing was established in the early 1980s. Finally, do not miss out on the IEE Proceedings – Vision, Image and Signal Processing (now called IET Computer Vision). Most journals are now online but usually to subscribers only; some go back a long way. Academic Press titles include Computer Vision and Image Understanding, Graphical Models and Image Processing and Real-Time Imaging (which will reappear as Springer’s Real-Time Image Processing). There are plenty of conferences too: the Proceedings of the IEEE conferences are held on the IEEE Explore site; Lecture Notes in Computer Science are hosted by Springer (http://www.springer.com/). Some conferences such as the British Machine Vision Conference series maintain their own site (http://www.bmva.ac.uk). The excellent Computer Vision Conferences page in Table 1.5 is brought to us by Keith Price and lists conferences in computer vision, image processing and pattern recognition.
1.6.2 Textbooks There are many textbooks in this area. Increasingly, there are web versions, or web support, as summarized in Table 1.4. The difficulty is one of access, as you need a subscription to be able to access the online book (and sometimes even to see that it is available online). For example, this book is available online to those subscribing to Referex in Engineering Village (http://www.engineeringvillage.org). The site given in Table 1.4, as this book is the support site which includes demonstrations, worksheets, errata and other information. The site given next, at Edinburgh University UK, is part of the excellent CVOnline site (many thanks to Bob Fisher there) and it lists current books as well pdfs, some of which are more dated, but still excellent Introduction
25
Table 1.4 Web textbooks and homepages This book’s homepage
Southampton University
http://www.ecs.soton.ac.uk/∼msn/book/
CVOnline: online book compendium
Edinburgh University
http://homepages.inf.ed.ac.uk/rbf/CVonline/books.htm
Image Processing Fundamentals
Delft University
http://www.ph.tn.tudelft.nl/Courses/FIP/noframes/fip.html
World of Mathematics
Wolfram Research
http://mathworld.wolfram.com
Numerical Recipes
Cambridge University Press
http://www.nr.com/
Digital Signal Processing
Steven W. Smith
http://www.dspguide.com/
The Joy of Visual Perception
York University
http://www.yorku.ca/research/vision/eye/thejoy.htm
(e.g. Ballard and Brown, 1982). There is also continuing debate on appropriate education in image processing and computer vision, for which review material is available (Bebis et al., 2003). The CVOnline site also describes a great deal of technique. Image Processing Fundamentals is an online textbook for image processing. The World of Mathematics comes from Wolfram research (the distributors of Mathematica) and gives an excellent web-based reference for mathematics. Numerical Recipes (Press et al., 2002) is one of the best established texts in signal processing. It is beautifully written, with examples and implementation, and is on the web too. Digital Signal Processing is an online site with focus on the more theoretical aspects which will be covered in Chapter 2. As previously mentioned, The Joy of Visual Perception is an online site on how the human vision system works. By way of context, for comparison with other textbooks, this text aims to start at the foundation of computer vision, and ends very close to a research level. Its content specifically addresses techniques for image analysis, considering shape analysis in particular. Mathcad and Matlab are used as a vehicle to demonstrate implementation, which is not always considered in other texts. But there are other texts, and these can help you to develop your interest in other areas of computer vision. This section includes only a selection of some of the texts. There are more than these, some of which will be referred to in later chapters; each offers a particular view or insight into computer vision and image processing. Some of the main textbooks include: Marr, Vision (1982), which concerns vision and visual perception (as mentioned previously); Jain, Fundamentals of Computer Vision (1989), which is stacked with theory and technique, but omits implementation and some image analysis, and Robot Vision (Horn, 1986); Sonka et al., Image Processing, Analysis and Computer Vision (1998), offers more modern coverage of computer vision including many more recent techniques, together with pseudocode implementation, but omitting some image processing theory (the support site http://css.engineering.uiowa.edu/ %7Edip/LECTURE/lecture.html has teaching material too); Jain et al., Machine Vision (1995), offers concise and modern coverage of 3D and motion (there is an online website at http://vision.cse.psu.edu/with code and images, together with corrections); Gonzalez and Wintz, Digital Image Processing (1987), has more tutorial element than many of the basically theoretical 26
Feature Extraction and Image Processing
texts and has a fantastic reputation for introducing people to the field; Rosenfeld and Kak, Digital Picture Processing (1982) is very dated now, but is a well-proven text for much of the basic material; and Pratt, Digital Image Processing (2001), which was originally one of the earliest books on image processing and, like Rosenfeld and Kak, is a well-proven text for much of the basic material, particularly image transforms. Despite its name, the recent text called Active Contours (Blake and Isard, 1998) concentrates rather more on models of motion and deformation and probabilistic treatment of shape and motion, than on the active contours which we shall find here. As such, it is more a research text, reviewing many of the advanced techniques to describe shapes and their motion. Image Processing – The Fundamentals (Petrou and Bosdogianni, 1999) surveys the subject (as its title implies) from an image processing viewpoint, covering not only image transforms, but also restoration and enhancement before edge detection. The latter of these is most appropriate for one of the major contributors to that subject. A newer text (Shapiro and Stockman, 2001) includes chapters on image databases and on virtual and augmented reality. Umbaugh’s Computer Imaging: Digital Image Analysis and Processing (2005) reflects recent interest in implementation by giving many programming examples. One of the most modern books is Forsyth and Ponce’s Computer Vision: A Modern Approach (2002), which offers much new, and needed, insight into this continually developing subject (two chapters that did not make the final text, on probability and on tracking, are available at the book’s website http://www.cs.berkeley.edu/%7Edaf/book.html) Kasturi and Jain’s Computer Vision: Principles (1991) and Computer Vision: Advances and Applications (1991) present a collection of seminal papers in computer vision, many of which are cited in their original form (rather than in this volume) in later chapters. There are other interesting edited collections (Chellappa, 1992), and one edition (Bowyer and Ahuja, 1996) honours Azriel Rosenfeld’s many contributions. Books that include a software implementation include Lindley, Practical Image Processing in C (1991), and Pitas, Digital Image Processing Algorithms (1993), which both cover basic image processing and computer vision algorithms. Parker, Practical Computer Vision Using C (1994), offers an excellent description and implementation of low-level image processing tasks within a well-developed framework, but again does not extend to some of the more recent and higher level processes in computer vision and includes little theory, although there is more in his later text Image Processing and Computer Vision (Parker, 1996). There is excellent coverage of practicality in Seul et al. (2000) and the book’s support site is at http://www.mlmsoftwaregroup.com/. As mentioned previously, a recent text, Computer Vision and Image Processing (Umbaugh, 2005), takes an applications-orientated approach to computer vision and image processing, offering a variety of techniques in an engineering format. One recent text concentrates on Java only, Image Processing in Java (Lyon, 1999), and concentrates more on image processing systems implementation than on feature extraction (giving basic methods only). As already mentioned, a newer textbook (Efford, 2000) offers Java implementation, although it omits much of the mathematical detail, making it a lighter (more enjoyable?) read. Masters, Signal and Image Processing with Neural Networks – A C++ Sourcebook (1994), offers good guidance in combining image processing technique with neural networks and gives code for basic image processing technique, such as frequency domain transformation. Other textbooks include Russ, The Image Processing Handbook (2002), which contains much basic technique with excellent visual support, but without any supporting theory, and has many practical details concerning image processing systems; Davies, Machine Vision: Theory, Algorithms and Practicalities (2005), which is targeted primarily at (industrial) machine vision systems, but covers much basic technique, with pseudo-code to describe their implementation; and
Introduction
27
the Handbook of Pattern Recognition and Computer Vision (Cheng, 2005). Last but by no means least, there is even an illustrated dictionary (Fisher, 2005) to guide you through the terms that are used.
1.6.3 The web The web entries continue to proliferate. A list of web pages is given in Table 1.5 and these give you a starting point from which to build up your own list of favourite bookmarks. All these links, and more, are available at this book’s homepage (http://www.ecs.soton.ac.uk/∼msn/book/). This will be checked regularly and kept up to date. The web entries in Table 1.5 start with the Carnegie Mellon homepage (called the computer vision homepage). The Computer Vision Online CVOnline homepage has been brought to us by Bob Fisher from the University of Edinburgh. There is a host of material there, including its description. Their group also proves the Hypermedia Image Processing Website and, in their words: ‘HIPR2 is a free www-based set of tutorial materials for the 50 most commonly used image processing operators. It contains tutorial text, sample results and JAVA demonstrations of individual operators and collections’. It covers a lot of basic material and shows you the results of various processing options. A big list of active groups can be found at the Computer Vision Homepage. If your University has access to the web-based indexes of published papers, the ISI index gives you journal papers (and allows for citation search), but unfortunately including medicine and science (where you can Table 1.5 Computer vision and image processing websites Name/scope
Host
Address Vision and its applications
The Computer Vision Homepage
Carnegie Mellon University
http://www.cs.cmu.edu/afs/cs/project/cil/ftp/ html/vision.html
Computer Vision Online
Edinburgh University
http://www.dai.ed.ac.uk/CVonline/
Hypermedia Image Processing Reference 2
Edinburgh University
http://www.dai.ed.ac.uk/HIPR2
Image Processing Archive
PEIPA
http://peipa.essex.ac.uk/
3D Reconstruction
Stanford University
http://biocomp.stanford.edu/3dreconstruction/ index.html
Face Recognition
Zagreb University
http://www.face-rec.org/
Conferences Computer Vision (and image processing)
Keith Price, USC
http://iris.usc.edu/Information/ Iris-Conferences.html
Newsgroups Computer Vision
Vision List
Image Processing
28 Feature Extraction and Image Processing
comp.ai.vision (http://www.vislist.com/) sci.image.processing
get papers with over 30 authors). Alternatively, Compendex and INSPEC include papers more related to engineering, together with papers in conferences, and hence vision, but without the ability to search citations. More recently, many turn to Citeseer and Google Scholar with direct ability to retrieve the papers, as well as to see where they have been used. Two newsgroups can be found at the addresses given in Table 1.5 to provide what is perhaps the most up-to-date information.
1.7 Conclusions This chapter has covered most of the prerequisites for feature extraction in image processing and computer vision. We need to know how we see, in some form, where we can find information and how to process data. More importantly, we need an image, or some form of spatial data. This is to be stored in a computer and processed by our new techniques. As it consists of data points stored in a computer, this data is sampled or discrete. Extra material on image formation, camera models and image geometry is to be found in Appendix 2, but we shall be considering images as a planar array of points from here on. We need to know some of the bounds on the sampling process, on how the image is formed. These are the subjects of the next chapter, which also introduces a new way of looking at the data, and how it is interpreted (and processed) in terms of frequency.
1.8 References Armstrong, T., Colour Perception – A Practical Approach to Colour Theory, Tarquin Publications, Diss, 1991 Ballard, D. H. and Brown, C. M., Computer Vision, Prentice-Hall, New Jersey, 1982 Bebis, G., Egbert, D. and Shah, M., Review of Computer Vision Education, IEEE Trans. Educ., 46(1), pp. 2–21, 2003 Blake, A. and Isard, M., Active Contours, Springer, London, 1998 Bowyer, K. and Ahuja, N. (Eds), Advances in Image Understanding, A Festschrift for Azriel Rosenfeld, IEEE Computer Society Press, Los Alamitos, CA, 1996 Bruce, V. and Green, P., Visual Perception: Physiology, Psychology and Ecology, 2nd edn, Lawrence Erlbaum Associates, Hove, 1990 Chellappa, R., Digital Image Processing, 2nd edn, IEEE Computer Society Press, Los Alamitos, CA, 1992 Cheng, C. H. and Wang, P. S. P., Handbook of Pattern Recognition and Computer Vision, 3rd edn, World Scientific, Singapore, 2005 Cornsweet, T. N., Visual Perception, Academic Press, New York, 1970 Davies, E. R., Machine Vision: Theory, Algorithms and Practicalities, 3rd edn, Morgan Kaufmann (Elsevier), Amsterdam, 2005 Efford, N., Digital Image Processing – A Practical Introduction Using JAVA, Pearson Education, Harlow, 2000 Fisher, R. B., Dawson-Howe, K., Fitzgibbon, A. and Robertson, C., Dictionary of Computer Vision and Image Processing, John Wiley & Sons, New York, 2005 Forsyth, D. and Ponce, J., Computer Vision: A Modern Approach, Prentice Hall, New Jersey, 2002 Introduction
29
Fossum, E. R., CMOS Image Sensors: Electronic Camera-On-A-Chip, IEEE Trans. Electron. Devices, 44(10), pp. 1689–1698, 1997 Gonzalez, R. C. and Wintz, P., Digital Image Processing, 2nd edn, Addison Wesley, Reading, MA, 1987 Gonzalez, R. C., Woods, R. E. and Eddins, S., Digital Image Processing using MATLAB, 1st edn, Prentice Hall, New Jersey, 2003 Horn, B. K. P., Robot Vision, MIT Press, Boston, MA, 1986 Jain, A. K., Fundamentals of Computer Vision, Prentice Hall International (UK), Hemel Hempstead, 1989 Jain, R. C., Kasturi, R. and Schunk, B. G., Machine Vision, McGraw-Hill Book Co., Singapore, 1995 Kasturi, R. and Jain, R. C., Computer Vision: Principles, IEEE Computer Society Press, Los Alamitos, CA, 1991 Kasturi, R. and Jain, R. C., Computer Vision: Advances and Applications, IEEE Computer Society Press, Los Alamitos, CA, 1991 Lindley, C. A., Practical Image Processing in C, Wiley & Sons, New York, 1991 Lyon, D. A., Image Processing in Java, Prentice Hall, New Jersey, 1999 Maple, Waterloo Maple Software, Ontario, Canada Marr, D., Vision, W. H. Freeman and Co., New York, 1982 Masters, T. Signal and Image Processing with Neural Networks – A C++ Sourcebook, Wiley and Sons, New York, 1994 MATLAB, The MathWorks, 24 Prime Way Park, Natick, MA, USA Mathcad, Mathsoft, 101 Main St., Cambridge, MA, USA Mathematica, Wolfram Research, 100 Trade Center Drive, Champaign, IL, USA Nakamura, J., Image Sensors and Signal Processing for Digital Still Cameras, CRC Press, Boca Raton, FL, 2005 Overington, I., Computer Vision – A Unified, Biologically-Inspired Approach, Elsevier Science Press, Amsterdam, 1992 Parker, J. R., Practical Computer Vision Using C, Wiley & Sons, New York, 1994 Parker, J. R., Algorithms for Image Processing and Computer Vision, Wiley & Sons, New York, 1996 Petrou, M. and Bosdogianni, P., Image Processing – The Fundamentals, John Wiley & Sons, London, 1999 Pitas, I., Digital Image Processing Algorithms, Prentice Hall International (UK), Hemel Hempstead, 1993 Pratt, W. K., Digital Image Processing: PIKS Inside, 3rd edn, Wiley, New York, 2001 Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P., Numerical Recipes in C++: The Art of Scientific Computing, 2nd edn, Cambridge University Press, Cambridge, 2002 Ratliff, F., Mach Bands: Quantitative Studies on Neural Networks in the Retina, Holden-Day Inc., San Francisco, USA, 1965 Rosenfeld, A. and Kak A. C., Digital Picture Processing, 2nd edn, Vols 1 and 2, Academic Press, Orlando, FL, 1982 Russ, J. C., The Image Processing Handbook, 4th edn, CRC Press (IEEE Press), Boca Raton, FL, 2002 Schwarz, S. H., Visual Perception, 3rd edn, McGraw-Hill, New York, 2004 Seul, M., O’Gorman, L. and Sammon, M. J., Practical Algorithms for Image Analysis: Descriptions, Examples, and Code, Cambridge University Press, Cambridge, 2000 30 Feature Extraction and Image Processing
Shapiro, L. G. and Stockman, G. C., Computer Vision, Prentice Hall, New Jersey, USA, 2001 Sonka, M., Hllavac, V. and Boyle, R, Image Processing, Analysis and Computer Vision, 2nd edn, Chapman Hall, London, 1998 Umbaugh, S. E., Computer Imaging: Digital Image Analysis and Processing, CRC Press, Boca Raton, FL, 2005
Introduction
31
This page intentionally left blank
2
.
.
Images, sampling and frequency domain processing
2.1 Overview In this chapter, we shall look at the basic theory which underlies image formation and processing. We shall start by investigating what makes up a picture and look at the consequences of having a different number of points in the image. We shall also look at images in a different representation, known as the frequency domain. In this, as the name implies, we consider an image as a collection of frequency components. We can operate on images in the frequency domain and we shall also consider different transformation processes. These allow us different insights into images and image processing which will be used in later chapters not only as a means to develop techniques, but also to give faster (computer) processing.
Table 2.1 Overview of Chapter 2 Main topic
Sub topics
Main points
Images
Effects of differing numbers of points and of number range for those points.
Greyscale, colour, resolution, dynamic range, storage.
Fourier transform theory
What is meant by the frequency domain, how it applies to discrete (sampled) images, how it allows us to interpret images and the sampling resolution (number of points).
Continuous Fourier transform and properties, sampling criterion, discrete Fourier transform and properties, image transformation, transform duals. Inverse Fourier transform.
Consequences of transform approach
Basic properties of Fourier transforms, other transforms, frequency domain operations.
Translation (shift), rotation and scaling. Principle of superposition and linearity. Walsh, Hartley, discrete cosine and wavelet transforms. Filtering and other operations.
33
2.2 Image formation A computer image is a matrix (a two-dimensional array) of pixels. The value of each pixel is proportional to the brightness of the corresponding point in the scene; its value is usually derived from the output of an analogue-to-digital (A/D) converter. The matrix of pixels, the image, is usually square and an image may be described as N × N m-bit pixels, where N is the number of points and m controls the number of brightness values. Using m bits gives a range of 2m values, ranging from 0 to 2m − 1. If m is 8 this gives brightness levels ranging between 0 and 255, which are usually displayed as black and white, respectively, with shades of grey in between, as they are for the greyscale image of a walking man in Figure 2.1(a). Smaller values of m give fewer available levels, reducing the available contrast in an image. The ideal value of m is related to the signal-to-noise ratio (bandwidth) of the camera. This is stated as approximately 45 dB for an analogue camera, and since there are 6 dB per bit, 8 bits will cover the available range. Choosing 8 bit pixels has further advantages in that it is very convenient to store pixel values as bytes, and 8 bit A/D converters are cheaper than those with a higher resolution. For these reasons images are nearly always stored as 8 bit bytes, although some applications use a different range. The relative influence of the 8 bits is shown in the image of the walking subject in Figure 2.1. Here, the least significant bit, bit 0 (Figure 2.1b), carries the least information (it changes most rapidly). As the order of the bits increases, they change less rapidly and carry more information. The most information is carried by the most significant bit, bit 7 (Figure 2.1i). Clearly, the fact that there is a walker in the original image can be recognized much more reliably from the high-order bits than it can from the other bits (notice too the odd effects in the bits, which would appear to come from lighting at the top of the image). Colour images follow a similar storage strategy to specify pixels’ intensities. However, instead of using just one image plane, colour images are represented by three intensity components. These components generally correspond to red, green and blue (the RGB model), although there are other colour schemes. For example, the CMYK colour model is defined by the components cyan, magenta, yellow and black. In any colour mode, the pixel’s colour can be specified in two main ways. First, you can associate an integer value with each pixel, which can be used as an index to a table that stores the intensity of each colour component. The index is used to recover the actual colour from the table when the pixel is going to be displayed, or processed. In this scheme, the table is known as the image’s palette and the display is said to be performed by colour mapping. The main reason for using this colour representation is to reduce memory requirements. That is, we only store a single image plane (i.e. the indices) and the palette. This is less than storing the red, green and blue components separately and so makes the hardware cheaper, and it can have other advantages, for example when the image is transmitted. The main disadvantage is that the quality of the image is reduced since only a reduced collection of colours is actually used. An alternative to represent colour is to use several image planes to store the colour components of each pixel. This scheme is known as true colour and it represents an image more accurately, essentially by considering more colours. The most common format uses 8 bits for each of the three RGB components. These images are known as 24 bit true colour and they can contain 16 777 216 different colours simultaneously. In spite of requiring significantly more memory, the image quality and the continuing reduction in cost of computer memory make this format a good alternative, even for storing the image frames from a video sequence. A good compression algorithm is always helpful in these cases, particularly if images
34
Feature Extraction and Image Processing
(a) Original image
(b) Bit 0 (LSB)
(c) Bit 1
(d) Bit 2
(e) Bit 3
(f) Bit 4
(g) Bit 5
(h) Bit 6
(i) Bit 7 (MSB)
Figure 2.1 Decomposing an image into its bits
need to be transmitted on a network. Here we will consider the processing of grey-level images only, since they contain enough information to perform feature extraction and image analysis. Should the image be originally colour, we will consider processing its luminance only, often computed in a standard way. In any case, the amount of memory used is always related to the image size. Images, sampling and frequency domain processing
35
Choosing an appropriate value for the image size, N , is far more complicated. We want N to be sufficiently large to resolve the required level of spatial detail in the image. If N is too small, the image will be coarsely quantized: lines will appear to be very ‘blocky’ and some of the detail will be lost. Larger values of N give more detail, but need more storage space and the images will take longer to process, since there are more pixels. For example, with reference to the image of the walking subject in Figure 2.1(a), Figure 2.2 shows the effect of taking the image at different resolutions. Figure 2.2(a) is a 64 × 64 image, which shows only the broad structure. It is impossible to see any detail in the subject’s face, or anywhere else. Figure 2.2(b) is a 128 × 128 image, which is starting to show more of the detail, but it would be hard to determine the subject’s identity. The original image, repeated in Figure 2.2(c), is a 256 × 256 image which shows a much greater level of detail, and the subject can be recognized from the image. (These images come from a research programme aimed to use computer vision techniques to recognize people by their gait; face recognition would have little potential for the low-resolution image, which is often the sort of image that security cameras provide.) If the image were a pure photographic image, some of the much finer detail like the hair would show up in much greater detail. This is because the grains in film are very much smaller than the pixels in a computer image. Note that the images in Figure 2.2 have been scaled to be the same size. As such, the pixels in Figure 2.2(a) are much larger than in Figure 2.2(c), which emphasizes its blocky structure. The most common choices are for 256 × 256 images or 512 × 512. These require 64 and 256 kbytes of storage, respectively. If we take a sequence of, say, 20 images for motion analysis, we will need more than 1 Mbyte to store the 20 256 × 256 images, and more than 5 Mbytes if the images were 512 × 512. Even though memory continues to become cheaper, this can still impose high cost. But it is not just cost which motivates an investigation of the appropriate image size, the appropriate value for N . The main question is: are there theoretical guidelines for choosing it? The short answer is ‘yes’; the long answer is to look at digital signal processing theory.
(a) 64 × 64
(b) 128 × 128
(c) 256 × 256
Figure 2.2 Effects of differing image resolution
The choice of sampling frequency is dictated by the sampling criterion. Presenting the sampling criterion requires understanding of how we interpret signals in the frequency domain. The way in is to look at the Fourier transform. This is a highly theoretical topic, but do not let that put you off (it leads to image coding, like the JPEG format, so it is very useful indeed). 36
Feature Extraction and Image Processing
The Fourier transform has found many uses in image processing and understanding; it might appear to be a complex topic (that’s actually a horrible pun!), but it is a very rewarding one to study. The particular concern is the appropriate sampling frequency of (essentially, the value for N ), or the rate at which pixel values are taken from, a camera’s video signal.
2.3 The Fourier transform The Fourier transform is a way of mapping a signal into its component frequencies. Frequency measures in Hertz (Hz) the rate of repetition with time, measured in seconds (s); time is the reciprocal of frequency and vice versa Hertz = 1/second s = 1/Hz. Consider a music centre: the sound comes from a CD player (or a tape, etc.) and is played on the speakers after it has been processed by the amplifier. On the amplifier, you can change the bass or the treble (or the loudness, which is a combination of bass and treble). Bass covers the low-frequency components and treble covers the high-frequency ones. The Fourier transform is a way of mapping the signal from the CD player, which is a signal varying continuously with time, into its frequency components. When we have transformed the signal, we know which frequencies made up the original sound. So why do we do this? We have not changed the signal, only its representation. We can now visualize it in terms of its frequencies, rather than as a voltage which changes with time. But we can now change the frequencies (because we can see them clearly) and this will change the sound. If, say, there is hiss on the original signal then since hiss is a high-frequency component, it will show up as a high-frequency component in the Fourier transform. So we can see how to remove it by looking at the Fourier transform. If you have ever used a graphic equalizer, you have done this before. The graphic equalizer is a way of changing a signal by interpreting its frequency domain representation; you can selectively control the frequency content by changing the positions of the controls of the graphic equalizer. The equation that defines the Fourier transform, Fp, of a signal p, is given by a complex integral: p = Fp = pte−jt dt (2.1) −
where Fp() is the Fourier transform; is the angular frequency, = 2f , measured in radians/s (where the frequency f is the reciprocal of time t f = 1/t); j is the complex variable (electronic engineers prefer j to i since they cannot confuse it with the symbol for current – perhaps they don’t want to be mistaken for mathematicians!); pt is a continuous signal (varying continuously with time); and e−jt = cost − j sint gives the frequency components in pt. We can derive the Fourier transform by applying Equation 2.1 to the signal of interest. We can see how it works by constraining our analysis to simple signals. (We can then say that complicated signals are just made up by adding up lots of simple signals.) If we take a pulse which is of amplitude (size) A between when it starts at time t = −T/2 and when it ends at t = T/2, and is zero elsewhere, the pulse is: A if − T/2 ≤ t ≤ T/2 pt = (2.2) 0 otherwise To obtain the Fourier transform, we substitute for pt in Equation 2.1. pt = A only for a specified time, so we choose the limits on the integral to be the start and end points of our pulse Images, sampling and frequency domain processing
37
(it is zero elsewhere) and set pt = A, its value in this time interval. The Fourier transform of this pulse is the result of computing: T/2 Ae−jt dt (2.3) Fp = −T/2
When we solve this we obtain an expression for Fp: Fp = −
Ae−jT/2 − AejT/2 j
(2.4)
By simplification, using the relation sin = ej − e−j /2j, the Fourier transform of the pulse is: 2A T sin if = 0 2 (2.5) Fp = AT if = 0 This is a version of the sinc function, sincx = sinx/x. The original pulse, and its transform are illustrated in Figure 2.3. Equation 2.5 (as plotted in Figure 2.3b) suggests that a pulse is made up of a lot of low frequencies (the main body of the pulse) and a few higher frequencies (which give the edges of the pulse). (The range of frequencies is symmetrical around zero frequency; negative frequency is a necessary mathematical abstraction.) The plot of the Fourier transform is called the spectrum of the signal, which can be considered akin to the spectrum of light.
Fp (ω )
p (t )
ω
t
(a) Pulse of amplitude A = 1
(b) Fourier transform
Figure 2.3 A pulse and its Fourier transform
So what actually is this Fourier transform? It tells us what frequencies make up a time domain signal. The magnitude of the transform at a particular frequency is the amount of that frequency in the original signal. If we collect together sinusoidal signals in amounts specified by the Fourier transform, we should obtain the originally transformed signal. This process is illustrated in Figure 2.4 for the signal and transform illustrated in Figure 2.3. Note that since the Fourier transform is a complex number it has real and imaginary parts, and we only plot the real part here. A low frequency, that for = 1, in Figure 2.4(a), contributes a large component of the original signal; a higher frequency, that for = 2, contributes less, as in Figure 2.4(b). This is because the transform coefficient is less for = 2 than it is for = 1. There is a very small contribution for = 3 (Figure 2.4c), although there is more for = 4 (Figure 2.4d). This is because there are frequencies for which there is no contribution, where the transform is zero. When these signals are integrated together, we achieve a signal that looks similar to the original pulse (Figure 2.4e). Here we have only considered frequencies from = −6 to = 6. If the 38
Feature Extraction and Image Processing
. .
.
Re(Fp (2).e j 2 t )
Re(Fp (1).e j t )
t
t
(a) Contribution for ω = 1
(b) Contribution for ω = 2
. .
. .
Re(Fp (4).e j 4 t )
Re(Fp (3).e j 3 t )
t
t
(c) Contribution for ω = 3
(d) Contribution for ω = 4
6
Fp (ω).e j.ω.t dω –6
t (e) Reconstruction by integration
Figure 2.4 Reconstructing a signal from its transform
frequency range in integration were larger, more high frequencies would be included, leading to a more faithful reconstruction of the original pulse. The result of the Fourier transform is a complex number. As such, it is usually represented in terms of its magnitude (or size, or modulus) and phase (or argument). The transform can be represented as: pte−jt dt = Re Fp + j Im Fp
(2.6) −
where Re and Im are the real and imaginary parts of the transform, respectively. The magnitude of the transform is then: pte−jt dt = Re Fp 2 + Im Fp 2 (2.7) −
and the phase is: Im Fp
pte−jt dt = tan−1 Re Fp
− Images, sampling and frequency domain processing
(2.8)
39
where the signs of the real and the imaginary components can be used to determine which quadrant the phase is in (since the phase can vary from 0 to 2 radians). The magnitude describes the amount of each frequency component, while the phase describes timing, when the frequency components occur. The magnitude and phase of the transform of a pulse are shown in Figure 2.5, where the magnitude returns a positive transform, and the phase is either 0 or 2 radians (consistent with the sine function).
arg (Fp (ω ))
⏐Fp (ω )⏐
ω (b) Phase
ω (a) Magnitude
Figure 2.5 Magnitude and phase of the Fourier transform of a pulse
To return to the time domain signal, from the frequency domain signal, we require the inverse Fourier transform. This is the process by which we reconstructed the pulse from its transform components. The inverse FT calculates pt from Fp according to: 1 pt = −1 Fp = Fpe−jt d (2.9) 2 − Together, Equations 2.1 and 2.9 form a relationship known as a transform pair that allows us to transform into the frequency domain, and back again. By this process, we can perform operations in the frequency domain or in the time domain, since we have a way of changing between them. One important process is known as convolution. The convolution of one signal p1 t with another signal p2 t, where the convolution process denoted by ∗, is given by the integral p1 t ∗ p2 t = p1 p2 t − d (2.10) −
This is the basis of systems theory, where the output of a system is the convolution of a stimulus, say p1 , and a system’s response, p2 , By inverting the time axis of the system response, to give p2 t − , we obtain a memory function. The convolution process then sums the effect of a stimulus multiplied by the memory function: the current output of the system is the cumulative response to a stimulus. By taking the Fourier transform of Equation 2.10, where the Fourier transformation is denoted by , the Fourier transform of the convolution of two signals is p1 t ∗ p2 t = p1 p2 t − d e−jt dt − − (2.11) =
40
−
−
p2 t − e−jt dt p1 d
Feature Extraction and Image Processing
Now, since p2 t − = e−jt Fp2 (to be considered later in Section 2.6.1), p1 t ∗ p2 t =
−
Fp2 p1 e−jt d
= Fp2
−
p1 e−jt d
(2.12)
= Fp2 × Fp1 As such, the frequency domain dual of convolution is multiplication; the convolution integral can be performed by inverse Fourier transformation of the product of the transforms of the two signals. A frequency domain representation essentially presents signals in a different way, but it also provides a different way of processing signals. Later, we shall use the duality of convolution to speed up the computation of vision algorithms considerably. Further, correlation is defined to be p1 t ⊗ p2 t =
−
p1 p2 t + d
(2.13)
where ⊗ denotes correlation ( is another symbol which is used sometimes, but there is not much consensus on this symbol). Correlation gives a measure of the match between the two signals p2 and p1 . When p2 = p1 we are correlating a signal with itself and the process is known as autocorrelation. We shall be using correlation later, to find things in images. Before proceeding further, we also need to define the delta function, which can be considered to be a function occurring at a particular time interval: 1 deltat − = 0
if t = otherwise
(2.14)
The relationship between a signal’s time domain representation and its frequency domain version is also known as a transform pair: the transform of a pulse (in the time domain) is a sinc function in the frequency domain. Since the transform is symmetrical, the Fourier transform of a sinc function is a pulse. There are other Fourier transform pairs, as illustrated in Figure 2.6. First, Figure 2.6(a) and (b) show that the Fourier transform of a cosine function is two points in the frequency domain (at the same value for positive and negative frequency); we expect this since there is only one frequency in the cosine function, the frequency shown by its transform. Figure 2.6(c) and (d) show that the transform of the Gaussian function is another Gaussian function; this illustrates linearity (for linear systems it is Gaussian in, Gaussian out, which is another version of GIGO). Figure 2.6(e) is a single point (the delta function) which has a transform that is an infinite set of frequencies (Figure 2.6f); an alternative interpretation is that a delta function contains an equal amount of all frequencies. This can be explained by using Equation 2.5, where if the pulse is of shorter duration (T tends to zero), the sinc function is wider; as the pulse becomes infinitely thin, the spectrum becomes infinitely flat. Finally, Figure 2.6(g) and (h) show that the transform of a set of uniformly spaced delta functions is another set of uniformly spaced delta functions, but with a different spacing. The spacing in the frequency domain is the reciprocal of the spacing in the time domain. By way of Images, sampling and frequency domain processing
41
Time domain signals
Frequency domain spectra
F cos (ω )
cos (t )
ω
t
(a) Cosine wave
(b) Fourier transform of cosine wave
Fg (ω )
g (t )
ω t
(c) Gaussian function
(d) Spectrum of Gaussian function
1
Delta (t, 0)
ω
t
(e) Delta function
(f) Frequency content of delta function
1⎛ manyd ⎛ω, ⎝ Ψ⎝
manyd (t , Ψ)
ω
t
(g) Sampling function in time domain
(h) Transform of sampling function
Figure 2.6 Fourier transform pairs
a (non-mathematical) explanation, let us consider that the Gaussian function in Figure 2.6(c) is actually made up by summing a set of closely spaced (and very thin) Gaussian functions. Then, since the spectrum for a delta function is infinite, as the Gaussian function is stretched in the time domain (eventually to be a set of pulses of uniform height) we obtain a set of pulses in the frequency domain, but spaced by the reciprocal of the time domain spacing. This transform pair is the basis of sampling theory (which we aim to use to find a criterion that guides us to an appropriate choice for the image size). 42 Feature Extraction and Image Processing
2.4 The sampling criterion The sampling criterion specifies the condition for the correct choice of sampling frequency. Sampling concerns taking instantaneous values of a continuous signal; physically, these are the outputs of an A/D converter sampling a camera signal. The samples are the values of the signal at sampling instants. This is illustrated in Figure 2.7, where Figure 2.7(a) concerns taking samples at a high frequency (the spacing between samples is low), compared with the amount of change seen in the signal of which the samples are taken. Here, the samples are taken sufficiently fast to notice the slight dip in the sampled signal. Figure 2.7(b) concerns taking samples at a low frequency, compared with the rate of change of (the maximum frequency in) the sampled signal. Here, the slight dip in the sampled signal is not seen in the samples taken from it.
Amplitude
Amplitude Signal
Signal
Sampling instants
Δt
Time
Sampling instants
Δt
(a) Sampling at high frequency
Time
(b) Sampling at low frequency
Figure 2.7 Sampling at different frequencies
We can understand the process better in the frequency domain. Let us consider a time-variant signal which has a range of frequencies between −fmax and fmax as illustrated in Figure 2.8(b). This range of frequencies is shown by the Fourier transform, where the signal’s spectrum exists only between these frequencies. This function is sampled every t s: this is a sampling function of spikes occurring every t s. The Fourier transform of the sampling function is a series of spikes separated by fsample = 1/ t Hz. The Fourier pair of this transform was illustrated earlier (Figure 2.6g and h). The sampled signal is the result of multiplying the time-variant signal by the sequence of spikes; this gives samples that occur every t s, and the sampled signal is shown in Figure 2.8(a). These are the outputs of the A/D converter at sampling instants. The frequency domain analogue of this sampling process is to convolve the spectrum of the time-variant signal with the spectrum of the sampling function. Convolving the signals, the convolution process, implies that we take the spectrum of one, flip it along the horizontal axis and then slide it across the other. Taking the spectrum of the time-variant signal and sliding it over the spectrum of the spikes results in a spectrum where the spectrum of the original signal is repeated every 1/ t Hz fsample in Figure 2.8(b–d). If the spacing between samples is t , the repetitions of the time-variant signal’s spectrum are spaced at intervals of 1/ t , as in Figure 2.8(b). If the sample spacing is small, then the time-variant signal’s spectrum is replicated close together and the spectra collide, or interfere, as in Figure 2.8(d). The spectra just touch when the sampling frequency is twice the maximum frequency in the signal. If the frequency domain spacing, fsample , is more than twice the maximum frequency, fmax , the spectra do not collide or interfere, as in Figure 2.8(c). If the Images, sampling and frequency domain processing
43
Signal
Time 1/Δt
(a) Sampled signal
Frequency response
Frequency –f sample = –1/Δt
–f max
f max
f sample = 1/Δt
(b) Oversampled spectra
Frequency response
Frequency –3f max
–2f max = –f sample
–f max
2f max = f sample
f max
3f max
(c) Sampling at the Nyquist rate
Frequency response
Frequency –f sample
–f max
f max
f sample
(d) Undersampled, aliased, spectra
Figure 2.8 Sampled spectra
sampling frequency exceeds twice the maximum frequency then the spectra cannot collide. This is the Nyquist sampling criterion: In order to reconstruct a signal from its samples, the sampling frequency must be at least twice the highest frequency of the sampled signal. 44 Feature Extraction and Image Processing
If we do not obey Nyquist’s sampling theorem the spectra collide. When we inspect the sampled signal, whose spectrum is within −fmax to fmax , wherein the spectra collided, the corrupt spectrum implies that by virtue of sampling, we have ruined some of the information. If we were to attempt to reconstruct a signal by inverse Fourier transformation of the sampled signal’s spectrum, processing Figure 2.8(d) would lead to the wrong signal, whereas inverse Fourier transformation of the frequencies between −fmax and fmax in Figure 2.8(b) and (c) would lead back to the original signal. This can be seen in computer images as illustrated in Figure 2.9, which show an image of a group of people (the computer vision research team at Southampton) displayed at different spatial resolutions (the contrast has been increased to the same level in each subimage, so that the effect we want to demonstrate should definitely show up in the print copy). Essentially, the people become less distinct in the lower resolution image (Figure 2.9b). Now, look closely at the two sets of window blinds behind the people. At higher resolution, in Figure 2.9(a), these appear as normal window blinds. In Figure 2.9(b), which is sampled at a much lower resolution, a new pattern appears: the pattern appears to be curved, and if you consider the blinds’ relative size the shapes actually appear to be much larger than normal window blinds. So by reducing the resolution, we are seeing something different, an alias of the true information: something that is not actually there at all, but appears to be there as a result of sampling. This is the result of sampling at too low a frequency: if we sample at high frequency, the interpolated result matches the original signal; if we sample at too low a frequency we can get the wrong signal. (For these reasons people on television tend not to wear chequered clothes, or should not!)
(a) High resolution
(b) Low resolution – aliased
Figure 2.9 Aliasing in sampled imagery
Obtaining the wrong signal is called aliasing: our interpolated signal is an alias of its proper form. Clearly, we want to avoid aliasing, so according to the sampling theorem we must sample at twice the maximum frequency of the signal coming out of the camera. The maximum frequency is defined to be 5.5 MHz, so we must sample the camera signal at 11 MHz. (For information, when using a computer to analyse speech we must sample the speech at a minimum frequency of 12 kHz, since the maximum speech frequency is 6 kHz.) Given the timing of a video signal, sampling at 11 MHz implies a minimum image resolution of 576 × 576 pixels. Images, sampling and frequency domain processing
45
This is unfortunate: 576 is not an integer power of two, which has poor implications for storage and processing. Accordingly, since many image processing systems have a maximum resolution of 512 × 512, they must anticipate aliasing. This is mitigated somewhat by the observations that: • globally, the lower frequencies carry more information, whereas locally the higher frequencies contain more information, so the corruption of high-frequency information is of less importance • there is limited depth of focus in imaging systems (reducing high frequency content). But aliasing can, and does, occur and we must remember this when interpreting images. A different form of this argument applies to the images derived from digital cameras. The basic argument that the precision of the estimates of the high-order frequency components is dictated by the relationship between the effective sampling frequency (the number of image points) and the imaged structure, still applies. The effects of sampling can often be seen in films, especially in the rotating wheels of cars, as illustrated in Figure 2.10. This shows a wheel with a single spoke, for simplicity. The film is a sequence of frames starting on the left. The sequence of frames plotted Figure 2.10(a) is for a wheel which rotates by 20 between frames, as illustrated in Figure 2.10(b). If the wheel is rotating much faster, by 340 between frames, as in Figure 2.10(c) and (d), the wheel will appear to rotate the other way. If the wheel rotates by 360 between frames, it will appear to be stationary. To perceive the wheel as rotating forwards, then the rotation between frames must be 180 at most. This is consistent with sampling at least twice the maximum frequency. Our eye can resolve this in films (when watching a film, I bet you haven’t thrown a wobbly because the car’s going forwards whereas the wheels say it’s going the other way) since we know that the direction of the car must be consistent with the motion of its wheels, and we expect to see the wheels appear to go the wrong way, sometimes.
20°
(a) Oversampled rotating wheel
(b) Slow rotation
340°
(c) Undersampled rotating wheel
Figure 2.10 Correct and incorrect apparent wheel motion
46 Feature Extraction and Image Processing
(d) Fast rotation
2.5 The discrete Fourier transform 2.5.1 One-dimensional transform Given that image processing concerns sampled data, we require a version of the Fourier transform which handles this. This is known as the discrete Fourier transform (DFT). The DFT of a set of N points px (sampled at a frequency which at least equals the Nyquist sampling rate) into sampled frequencies Fpu is: −1 −j 1 N
px e Fpu = √ N x=0
2 xu N
(2.15)
This is a discrete analogue of the continuous Fourier transform: the continuous signal is replaced by a set of samples, the continuous frequencies by sampled ones, and the integral is replaced by a summation. If the DFT is applied to samples of a pulse in a window from sample 0 to sample N/2 − 1 (when the pulse ceases), the equation becomes: 2 −1 −j 1
Fpu = √ Ae N x=0 N
2 xu N
(2.16)
Since the sum of a geometric progression can be evaluated according to: n
a0 r k =
k=0
a0 1 − r n+1 1−r
(2.17)
the discrete Fourier transform of a sampled pulse is given by: ⎞ ⎛ N 2 −j
u
A ⎜1−e N 2 ⎟ Fpu = √ ⎝ ⎠ 2 N −j u N 1−e
(2.18)
By rearrangement, we obtain:
2 1− A −j u 2 N sinu/2 Fpu = √ e sinu/N N The modulus of the transform is: A sinu/2 Fpu = √ N sinu/N
(2.19)
(2.20)
since the magnitude of the exponential function is 1. The original pulse is plotted Figure 2.11(a) and the magnitude of the Fourier transform plotted against frequency is given in Figure 2.11(b).
1 if x < 5
Fpu
0 otherwise
x
(a) Sampled pulse
u
(b) DFT of sampled pulse
Figure 2.11 Transform pair for sampled pulse
Images, sampling and frequency domain processing
47
This is clearly comparable with the result of the continuous Fourier transform of a pulse (Figure 2.3), since the transform involves a similar, sinusoidal, signal. The spectrum is equivalent to a set of sampled frequencies; we can build up the sampled pulse by adding up the frequencies according to the Fourier description. Consider a signal such as that shown in Figure 2.12(a). This has no explicit analytic definition, and as such it does not have a closed Fourier transform; the Fourier transform is generated by direct application of Equation 2.15. The result is a set of samples of frequency (Figure 2.12b).
Fpu
px
x
u
(a) Sampled signal
(b) Transform of sampled signal
Figure 2.12 A sampled signal and its discrete transform
The Fourier transform in Figure 2.12(b) can be used to reconstruct the original signal in Figure 2.12(a), as illustrated in Figure 2.13. Essentially, the coefficients of the Fourier transform tell us how much there is of each of a set of sinewaves (at different frequencies) in the original signal. The lowest frequency component Fp0 , for zero frequency, is called the d.c. component (it is constant and equivalent to a sinewave with no frequency) and it represents the average value of the samples. Adding the contribution of the first coefficient Fp0 (Figure 2.13b) to the contribution of the second coefficient Fp1 (Figure 2.13c), is shown in Figure 2.13(d). This shows how addition of the first two frequency components approaches the original sampled pulse. The approximation improves when the contribution due to the fourth component, Fp3 , is included, as shown in Figure 2.13(e). Finally, adding up all six frequency components gives a close approximation to the original signal, as shown in Figure 2.13(f). This process is the inverse DFT. This can be used to reconstruct a sampled signal from its frequency components by: px =
N −1
Fpu e
j
2 ux N
(2.21)
u=0
Note that there are several assumptions made before application of the DFT. The first is that the sampling criterion has been satisfied. The second is that the sampled function replicates to infinity. When generating the transform of a pulse, Fourier theory assumes that the pulse repeats outside the window of interest. (There are window operators that are designed specifically to handle difficulty at the ends of the sampling window.) Finally, the maximum frequency corresponds to half the sampling period. This is consistent with the assumption that the sampling criterion has not been violated, otherwise the high frequency spectral estimates will be corrupt. 48
Feature Extraction and Image Processing
px
Fp 0
t
x
(a) Original sampled signal
j.t .
Re Fp 1.e
(b) First coefficient Fp0
2⋅π
j.t .
10
Re Fp 0 + Fp 1.e
2⋅π 10
t
t
(d) Adding Fp1 and Fp0
(c) Second coefficient Fp1
j.t .
3 Fp u.e
Re
2⋅π . u 10
j.t .
5
Fp u.e
Re
u=0
2⋅π . u 10
u=0
t
t
(f) Adding all six frequency components
(e) Adding Fp0, Fp1, Fp2 and Fp3
Figure 2.13 Signal reconstruction from its transform components
2.5.2 Two-dimensional transform Equation 2.15 gives the DFT of a one-dimensional (1D) signal. We need to generate Fourier transforms of images, so we need a two-dimensional (2D) discrete Fourier transform. This is a transform of pixels (sampled picture points) with a 2D spatial location indexed by coordinates x and y. This implies that we have two dimensions of frequency, u and v, which are the horizontal and vertical spatial frequencies, respectively. Given an image of a set of vertical lines, the Fourier transform will show only horizontal spatial frequency. The vertical spatial frequencies are zero since there is no vertical variation along the y-axis. The 2D Fourier transform evaluates the frequency data, FPuv , from the N × N pixels Pxy as: −1 −1 N
−j 1 N
FPuv = P e N x=0 y=0 xy
2 ux+vy N
(2.22)
The Fourier transform of an image can be obtained optically by transmitting a laser through a photographic slide and forming an image using a lens. The Fourier transform of the image of the slide is formed in the front focal plane of the lens. This is still restricted to transmissive Images, sampling and frequency domain processing
49
systems, whereas reflective formation would widen its application potential considerably (since optical computation is just slightly faster than its digital counterpart). The magnitude of the 2D DFT to an image of vertical bars (Figure 2.14a) is shown in Figure 2.14(b). This shows that there are only horizontal spatial frequencies; the image is constant in the vertical axis and there are no vertical spatial frequencies.
(a) Image of vertical bars
(b) Fourier transform of bars
Figure 2.14 Applying the 2D discrete Fourier transform
The 2D inverse DFT transforms from the frequency domain back to the image domain. The 2D inverse DFT is given by: Pxy =
−1 N −1 N
FPuv e
j
2 ux+vy N
(2.23)
u=0 v=0
One of the important properties of the FT is replication, which implies that the transform repeats in frequency up to infinity, as indicated in Figure 2.8 for 1D signals. To show this for 2D signals, we need to investigate the Fourier transform, originally given by FPuv , at integer multiples of the number of sampled points FPu+mM v+nN (where m and n are integers). The Fourier transform FPu+mM v+nN is, by substitution in Equation 2.22: FPu+mNv+nN
−1 −1 N
−j 1 N
= Pxy e N x=0 y=0
FPu+mNv+nN
−1 −1 N
−j 1 N
= P e N x=0 y=0 xy
so,
2 u+mNx+v+nNy N
2 ux+vy N
× e−j2mx+ny
(2.24)
(2.25)
and since e−j2mx+ny = 1 (since the term in brackets is always an integer and then the exponent is always an integer multiple of 2), then FPu+mNv+nN = FPuv
(2.26)
which shows that the replication property does hold for the Fourier transform. However, Equations 2.22 and 2.23 are very slow for large image sizes. They are usually implemented by using the fast Fourier transform (FFT), which is a splendid rearrangement of the Fourier transform’s computation that improves speed dramatically. The FFT algorithm is beyond the 50
Feature Extraction and Image Processing
scope of this text, but is also a rewarding topic of study (particularly for computer scientists or software engineers). The FFT can only be applied to square images whose size is an integer power of 2 (without special effort). Calculation involves the separability property of the Fourier transform. Separability means that the Fourier transform is calculated in two stages: the rows are first transformed using a 1D FFT, then this data is transformed in columns, again using a 1D FFT. This process can be achieved since the sinusoidal basis functions are orthogonal. Analytically, this implies that the 2D DFT can be decomposed as in Equation 2.27: 2 2 2 −1 N −1 N −1 −1
ux+vy vy ux −j −j −j 1 N
1 N
N N Pxy e = Pxy e e N (2.27) N x=0 y=0 N x=0 y=0 showing how separability is achieved, since the inner term expresses transformation along one axis (the y-axis) and the outer term transforms this along the other (the x-axis). Since the computational cost of a 1D FFT of N points is ON logN, the cost (by separability) for the 2D FFT is ON 2 logN, whereas the computational cost of the 2D DFT is ON 3 . This implies a considerable saving since it suggests that the FFT requires much less time, particularly for large image sizes (so for a 128 × 128 image, if the FFT takes minutes, the DFT will take days). The 2D FFT is available in Mathcad using the icfft function, which gives a result equivalent to Equation 2.22. The inverse 2D FFT (Equation 2.23) can be implemented using the Mathcad cfft function. (The difference between many Fourier transform implementations essentially concerns the chosen scaling factor.) The Mathcad implementations of the 2D DFT and the inverse 2D DFT are given in Code 2.1(a) and (b), respectively. The implementations using the Mathcad functions using the FFT are given in Code 2.1(c) and (d), respectively.
FPu,v :=
rows(P)–1
1
.
rows(P)
–j.2.π.(u.y + v.x)
cols(P)–1
∑
∑
y=0
x=0
Py,x.e
rows(P)
(a) 2D DFT, Equation 2.22
IFPy,x :=
rows(FP)–1
cols(FP)–1
∑
∑
u=0
v=0
j.2.π.(u.y + v.x)
FPu,v·e
rows(FP)
(b) Inverse 2D DFT, Equation 2.23 Fourier(pic):=icfft(pic) (c) 2D FFT inv_Fourier(trans):=cfft(trans) (d) inverse 2D FFT Code 2.1 Implementing Fourier transforms
For reasons of speed, the 2D FFT is the algorithm commonly used in application. One (unfortunate) difficulty is that the nature of the Fourier transform produces an image which, at first, is difficult to interpret. The Fourier transform of an image gives the frequency components. The position of each component reflects its frequency: low-frequency components are near the origin and high-frequency components are further away. As before, the lowest frequency component, Images, sampling and frequency domain processing
51
for zero frequency, the d.c. component, represents the average value of the samples. Unfortunately, the arrangement of the 2D Fourier transform places the low-frequency components at the corners of the transform. The image of the square in Figure 2.15(a) shows this in its transform (Figure 2.15b). A spatial transform is easier to visualize if the d.c. (zero frequency) component is in the centre, with frequency increasing towards the edge of the image. This can be arranged either by rotating each of the four quadrants in the Fourier transform by 180 , or by reordering the original image to give a transform which shifts the transform to the centre. Both operations result in the image in Figure 2.15(c), wherein the transform is much more easily seen. Note that this is aimed to improve visualization and does not change any of the frequency domain information, only the way it is displayed.
(a) Image of square
(c) Rearranged DFT
(b) Original DFT
Figure 2.15 Rearranging the 2D DFT for display purposes
To rearrange the image so that the d.c. component is in the centre, the frequency components need to be reordered. This can be achieved simply by multiplying each image point Px,y by −1x+y . Since cos− = −1, then −1 = e−j (the minus sign is introduced just to keep the analysis neat), so we obtain the transform of the multiplied image as: −1 −1 N
−j 1 N
P e N x=0 y=0 xy
2 ux+vy N
× −1
x+y
−1 −1 N
−j 1 N
= P e N x=0 y=0 xy −1 −1 N
−j 1 N
= Pxy e N x=0 y=0
2 ux+vy N
2 N
u+
× e−jx+y
N N x+ v+ y 2 2
(2.28)
= FPu+ N v+ N 2
2
According to Equation 2.28, when pixel values are multiplied by −1x+y , the Fourier transform becomes shifted along each axis by half the number of samples. According to the replication theorem (Equation 2.26), the transform replicates along the frequency axes. This implies that the centre of a transform image will now be the d.c. component. (Another way of interpreting this is that rather than look at the frequencies centred on where the image is, our viewpoint has 52
Feature Extraction and Image Processing
been shifted so as to be centred on one of its corners, thus invoking the replication property.) The operator rearrange, in Code 2.2, is used before transform calculation, and results in the image of Figure 2.15(c) and all later transform images.
rearrange(picture):=
for y∈0..rows(picture)–1 for x∈0..cols(picture)–1 rearranged_picy,x ← picturey,x·(–1)(y+x) rearranged_pic
Code 2.2 Reordering for transform calculation
The full effect of the Fourier transform is shown by application to an image of much higher resolution. Figure 2.16(a) shows the image of a face and Figure 2.16(b) shows its transform. The transform reveals that much of the information is carried in the lower frequencies, since this is where most of the spectral components concentrate. This is because the face image has many regions where the brightness does not change a lot, such as the cheeks and forehead. The high-frequency components reflect change in intensity. Accordingly, the higher frequency components arise from the hair (and that awful feather!) and from the borders of features of the human face, such as the nose and eyes.
(a) Image of face
(b) Transform of face image
Figure 2.16 Applying the Fourier transform to the image of a face
As with the 1D Fourier transform, there are 2D Fourier transform pairs, illustrated in Figure 2.17. The 2D Fourier transform of a 2D pulse (Figure 2.17a) is a 2D sinc function (Figure 2.17b). The 2D Fourier transform of a Gaussian function (Figure 2.17c) is again a 2D Gaussian function in the frequency domain (Figure 2.17d). Images, sampling and frequency domain processing
53
Image domain
1 0.8 0.6 0.4 0.2 0 10 20 30
Transform domain
2
0
1 10
20
30
10
20
30
0 10 20 30
square
ft_square (b) 2D sinc function
(a) Square
1 0.8 0.6 0.4 0.2 0 10 20 30
0
1.5 1 0
10
20
30
0.5
0
10
20
30
0 10 20 30
Gauss
ft_Gauss
(c) Gaussian
(d) Gaussian
Figure 2.17 2D Fourier transform pairs
2.6 Other properties of the Fourier transform 2.6.1 Shift invariance The decomposition into spatial frequency does not depend on the position of features within the image. If we shift all the features by a fixed amount, or acquire the image from a different position, the magnitude of its Fourier transform does not change. This property is known as shift invariance. By denoting the delayed version of pt as pt − , where is the delay, and the Fourier transform of the shifted version as pt − , we obtain the relationship between a time domain shift in the time and frequency domains as: pt − = e−jt P Accordingly, the magnitude of the Fourier transform is: p t − = e−jt P = e−jt P = P
(2.29)
(2.30)
and since the magnitude of the exponential function is 1.0, the magnitude of the Fourier transform of the shifted image equals that of the original (unshifted) version. We shall use this property later in Chapter 7, when we use Fourier theory to describe shapes. There, it will allow us to give the same description to different instances of the same shape, but a different description 54
Feature Extraction and Image Processing
to a different shape. You do not get something for nothing: even though the magnitude of the Fourier transform remains constant, its phase does not. The phase of the shifted transform is: (2.31) p t − = e−jt P The Mathcad implementation of a shift operator (Code 2.3) uses the modulus operation to enforce the cyclic shift. The arguments fed to the function are: the image to be shifted (pic), the horizontal shift along the x-axis (x_value) and the vertical shift along the y-axis (y_value).
shift(pic,y_val,x_val):=
NC←cols(pic) NR←rows(pic) for y∈0..NR–1 for x∈0..NC–1 shiftedy,x ← picmod(y+y_val,NR),mod(x+x_val,NC) shifted
Code 2.3 Shifting an image
This process is illustrated in Figure 2.18. An original image (Figure 2.18a) is shifted along the x- and y-axes (Figure 2.18d). The shift is cyclical, so parts of the image wrap around; those parts at the top of the original image appear at the base of the shifted image. The Fourier transform of the original image and that of the shifted image are identical: Figure 2.18(b) appears the same as Figure 2.18(e). The phase differs: the phase of the original image (Figure 2.18c) is clearly different from the phase of the shifted image (Figure 2.18f).
(a) Original image
(b) Magnitude of Fourier transform of original image
(c) Phase of Fourier transform of original image
(d) Shifted image
(e) Magnitude of Fourier transform of shifted image
(f) Phase of Fourier transform of shifted image
Figure 2.18 Illustrating shift invariance
Images, sampling and frequency domain processing
55
The differing phase implies that, in application, the magnitude of the Fourier transform of a face, say, will be the same irrespective of the position of the face in the image (i.e. the camera or the subject can move up and down), assuming that the face is much larger than its image version. This implies that if the Fourier transform is used to analyse an image of a human face or one of cloth, to describe it by its spatial frequency, we do not need to control the position of the camera, or the object, precisely.
2.6.2 Rotation The Fourier transform of an image rotates when the source image rotates. This is to be expected since the decomposition into spatial frequency reflects the orientation of features within the image. As such, orientation dependency is built into the Fourier transform process. This implies that if the frequency domain properties are to be used in image analysis, via the Fourier transform, the orientation of the original image needs to be known, or fixed. It is often possible to fix orientation, or to estimate its value when a feature’s orientation cannot be fixed. Alternatively, there are techniques to impose invariance to rotation, say by translation to a polar representation, although this can prove to be complex. The effect of rotation is illustrated in Figure 2.19. An image (Figure 2.19a) is rotated by 90 to give the image in Figure 2.19(b). Comparison of the transform of the original image (Figure 2.19c) with the transform of the rotated image (Figure 2.19d) shows that the transform has been rotated by 90 , by the same amount as the image. In fact, close inspection of Figure 2.19(c) and (d) shows that the diagonal axis is consistent with the normal to the axis of the leaves (where the change mainly occurs), and this is the axis that rotates.
(a) Original image
(b) Rotated image
(c) Transform of original image
(d) Transform of rotated image
Figure 2.19 Illustrating rotation
2.6.3 Frequency scaling By definition, time is the reciprocal of frequency. So if an image is compressed, equivalent to reducing time, its frequency components will spread, corresponding to increasing frequency. Mathematically, the relationship is that the Fourier transform of a function of time multiplied by a scalar p t, gives a frequency domain function P/ , so: 1 (2.32) F p t = P
56
Feature Extraction and Image Processing
This is illustrated in Figure 2.20, where the texture image of a chain-link fence (Figure 2.20a) is reduced in scale (Figure 2.20b), thereby increasing the spatial frequency. The DFT of the original texture image is shown in Figure 2.20(c), which reveals that the large spatial frequencies in the original image are arranged in a star-like pattern. As a consequence of scaling the original image, the spectrum will spread from the origin consistent with an increase in spatial frequency, as shown in Figure 2.20(d). This retains the star-like pattern, but with points at a greater distance from the origin.
(a) Texture image
(b) Scaled texture image
(c) Transform of original texture
(d) Transform of scaled texture
Figure 2.20 Illustrating frequency scaling
The implications of this property are that if we reduce the scale of an image, say by imaging at a greater distance, we will alter the frequency components. The relationship is linear: the amount of reduction, say the proximity of the camera to the target, is directly proportional to the scaling in the frequency domain.
2.6.4 Superposition (linearity) The principle of superposition is very important in systems analysis. Essentially, it states that a system is linear if its response to two combined signals equals the sum of the responses to the individual signals. Given an output O which is a function of two inputs I1 and I2 , the response to signal I1 is OI1 , that to signal I2 is OI2 , and the response to I1 and I2 , when applied together, is OI1 + I2 , the superposition principle states: OI1 + I2 = OI1 + OI2
(2.33)
Any system which satisfies the principle of superposition is termed linear. The Fourier transform is a linear operation since, for two signals p1 and p2 : p1 + p2 = p1 + p2
(2.34)
In application this suggests that we can separate images by looking at their frequency domain components. This is illustrated for 1D signals in Figure 2.21. One signal is shown in Figure 2.21(a) and a second is shown in Figure 2.21(c). The Fourier transforms of these signals are shown in Figure 2.21(b) and (d). The addition of these signals is shown in Figure 2.21(e) and its transform in Figure 2.21(f). The Fourier transform of the added signals differs little from the addition of their transforms (Figure 2.21g). This is confirmed by subtraction of the two (Figure 2.21d) (some slight differences can be seen, but these are due to numerical error). Images, sampling and frequency domain processing
57
0
200
200
0
(b) ℑ (Signal 1)
(a) Signal 1
0
0
200
(e) Signal 1 + Signal 2
0
200
(f) ℑ (Signal 1 + Signal 2)
200
(c) Signal 2
0
200
(g) ℑ (Signal 1) + ℑ (Signal 2)
0
200
(d) ℑ (Signal 2)
0
200
(h) Difference: (f)–(g)
Figure 2.21 Illustrating superposition
By way of example, given the image of a fingerprint in blood on cloth it is very difficult to separate the fingerprint from the cloth by analysing the combined image. However, by translation to the frequency domain, the Fourier transform of the combined image shows strong components due to the texture (this is the spatial frequency of the cloth’s pattern) and weaker, more scattered, components due to the fingerprint. If we suppress the frequency components due to the cloth’s texture, and invoke the inverse Fourier transform, then the cloth will be removed from the original image. The fingerprint can now be seen in the resulting image.
2.7 Transforms other than Fourier 2.7.1 Discrete cosine transform The discrete cosine transform (DCT) (Ahmed et al., 1974) is a real transform that has great advantages in energy compaction. Its definition for spectral components DPuv is: N −1 N −1 1 N x=0 y=0 Pxy if u = 0 and v = 0 DPuv = −1 N −1 2 N 2x + 1u 2y + 1v P × cos × cos N x=0 y=0 xy 2N 2N
(2.35) otherwise
The inverse DCT is defined by Pxy =
58
−1 −1 N
2y + 1v 2 N
2x + 1u × cos DPuv × cos N u=0 v=0 2N 2N
Feature Extraction and Image Processing
(2.36)
A fast version of the DCT is available, like the FFT, and calculation can be based on the FFT. Both implementations offer about the same speed. The Fourier transform is not actually optimal for image coding, since the DCT can give a higher compression rate for the same image quality. This is because the cosine basis functions can afford for high-energy compaction. This can be seen by comparison of Figure 2.22(b) with Figure 2.22(a), which reveals that the DCT components are much more concentrated around the origin, than those for the Fourier transform. This is the compaction property associated with the DCT. The DCT has been considered as optimal for image coding, and this is why it is found in the JPEG and MPEG standards for coded image transmission.
(a) Fourier transform
(b) Discrete cosine transform
(c) Hartley transform
Figure 2.22 Comparing transforms of the Lena image
The DCT is actually shift variant, owing to its cosine basis functions. In other respects, its properties are very similar to the DFT, with one important exception: it has not yet proved possible to implement convolution with the DCT. It is possible to calculate the DCT via the FFT. This has been performed in Figure 2.22(b), since there is no fast DCT algorithm in Mathcad and, as shown earlier, fast implementations of transform calculation can take a fraction of the time of the conventional counterpart. The Fourier transform essentially decomposes, or decimates, a signal into sine and cosine components, so the natural partner to the DCT is the discrete sine transform (DST). However, the DST transform has odd basis functions (sine) rather than the even ones in the DCT. This lends the DST transform some less desirable properties, and it finds much less application than the DCT.
2.7.2 Discrete Hartley transform The Hartley transform (Hartley, 1942) is a form of the Fourier transform, but without complex arithmetic, with a result for the face image shown in Figure 2.22(c). Oddly, although it sounds like a very rational development, the Hartley transform was first invented in 1942, but not rediscovered and then formulated in discrete form until 1983 (Bracewell, 1983). One advantage of the Hartley transform is that the forward and inverse transform is the same operation; a disadvantage is that phase is built into the order of frequency components since it is not readily Images, sampling and frequency domain processing
59
available as the argument of a complex number. The definition of the discrete Hartley transform (DHT) is that transform components HPuv are: −1 −1 N
2 1 N
2 × ux + vy + sin × ux + vy (2.37) P × cos HPuv = N x=0 y=0 xy N N The inverse Hartley transform is the same process, but applied to the transformed image. −1 −1 N
1 N
2 2 Pxy = HPxy × cos × ux + vy + sin × ux + vy (2.38) N u=0 v=0 N N The implementation is then the same for both the forward and the inverse transforms, as given in Code 2.4.
Hartley(pic):=
NC←cols(pic) NR←row(pic) for v∈0.. NR – 1 for u∈0.. NC – 1 transv,u←
1 . NC
NR–1
NC–1
Σ
Σ
y=0
picy,x. cos
x=0
2.π.(u.x + v.y) NR
+ sin
2.π.(u.x + v.y) NC
trans
Code 2.4 Implementing the Hartley transform
Again, a fast implementation is available, the fast Hartley transform (Bracewell, 1984) (although some suggest that it should be called the Bracewell transform, eponymously). It is possible to calculate the DFT of a function, Fu, from its Hartley transform, Hu. The analysis here is based on 1D data, but only for simplicity since the argument extends readily to two dimensions. By splitting the Hartley transform into its odd and even parts, Ou and Eu, respectively, we obtain: Hu = Ou + Eu
(2.39)
where: Eu =
Hu + HN − u 2
(2.40)
and Hu − HN − u 2 The DFT can then be calculated from the DHT simply by Ou =
(2.41)
Fu = Eu − j × Ou
(2.42)
60 Feature Extraction and Image Processing
Conversely, the Hartley transform can be calculated from the Fourier transform by: Hu = Re Fu − Im Fu
(2.43)
where Re[ ] and Im[ ] denote the real and the imaginary parts, respectively. This emphasizes the natural relationship between the Fourier and the Hartley transform. The image of Figure 2.22(c) has been calculated via the 2D FFT using Equation 2.43. Note that the transform in Figure 2.22(c) is the complete transform, whereas the Fourier transform in Figure 2.22(a) shows magnitude only. As with the DCT, the properties of the Hartley transform mirror those of the Fourier transform. Unfortunately, the Hartley transform does not have shift invariance, but there are ways to handle this. Also, convolution requires manipulation of the odd and even parts.
2.7.3 Introductory wavelets: the Gabor wavelet Wavelets are a comparatively recent approach to signal processing, being introduced only in 1990 (Daubechies, 1990). Their main advantage is that they allow multiresolution analysis (analysis at different scales, or resolution). Furthermore, wavelets allow decimation in space and frequency, simultaneously. Earlier transforms actually allow decimation in frequency, in the forward transform, and in time (or position) in the inverse. In this way, the Fourier transform gives a measure of the frequency content of the whole image: the contribution of the image to a particular frequency component. Simultaneous decimation allows us to describe an image in terms of frequency which occurs at a position, as opposed to an ability to measure frequency content across the whole image. This gives us greater descriptional power, which can be used to good effect. First, though, we need a basis function, so that we can decompose a signal. The basis functions in the Fourier transform are sinusoidal waveforms at different frequencies. The function of the Fourier transform is to convolve these sinusoids with a signal to determine how much of each is present. The Gabor wavelet is well suited to introductory purposes, since it is essentially a sinewave modulated by a Gaussian envelope. The Gabor wavelet gw is given by
gwt 0 t0 a = e
−j0 t −
e
t−t0 a
2
(2.44)
where 0 = 2f0 is the modulating frequency, t0 dictates position and a controls the width of the Gaussian envelope which embraces the oscillating signal. An example Gabor wavelet is shown in Figure 2.23, which shows the real and the imaginary parts (the modulus is the Gaussian envelope). Increasing the value of 0 increases the frequency content within the envelope,
Re(gw (t ))
Im(gw (t ))
t (a) Real part
t (b) Imaginary part
Figure 2.23 An example Gabor wavelet
Images, sampling and frequency domain processing
61
whereas increasing the value of a spreads the envelope without affecting the frequency. So why does this allow simultaneous analysis of time and frequency? Given that this function is the one convolved with the test data, then we can compare it with the Fourier transform. In fact, if we remove the term on the right-hand side of Equation 2.44, we return to the sinusoidal basis function of the Fourier transform, the exponential in Equation 2.1. Accordingly, we can return to the Fourier transform by setting a to be very large. Alternatively, setting f0 to zero removes frequency information. Since we operate in between these extremes, we obtain position and frequency information simultaneously. An infinite class of wavelets exists which can be used as an expansion basis in signal decimation. One approach (Daugman, 1988) has generalized the Gabor function to a 2D form aimed to be optimal in terms of spatial and spectral resolution. These 2D Gabor wavelets are given by
1 − gw2Dx y = √ e
x−x0 2 +y−y0 2 2 2
e−j2f0 x−x0 cos+y−y0 sin
(2.45)
where x0 y0 control position, f0 controls the frequency of modulation along either axis, and controls the direction (orientation) of the wavelet (as implicit in a 2D system). The shape of the area imposed by the 2D Gaussian function could be elliptical if different variances were allowed along the x- and y-axes (the frequency can also be modulated differently along each axis). Figure 2.24, an example of a 2D Gabor wavelet, shows that the real and imaginary parts are even and odd functions, respectively; again, different values for f0 and control the frequency and envelope’s spread, respectively, the extra parameter controls rotation.
Re(Gabor_wavelet)
Im(Gabor_wavelet)
(a) Real part
(b) Imaginary part
Figure 2.24 Example 2D Gabor wavelet
The function of the wavelet transform is to determine where and how each wavelet specified by the range of values for each of the free parameters occurs in the image. Clearly, there is a wide choice which depends on application. An example transform is given in Figure 2.25. Here, the Gabor wavelet parameters have been chosen in such a way as to select face features: the eyes, nose and mouth have come out very well. These features are where there is local frequency content with orientation according to the head’s inclination. These are not the only 62
Feature Extraction and Image Processing
(a) Original image
(b) After Gabor wavelet transform
Figure 2.25 Example Gabor wavelet transform
features with these properties: the cuff of the sleeve is highlighted too! But this does show the Gabor wavelet’s ability to select and analyse localized variation in image intensity. The conditions under which a set of continuous Gabor wavelets will provide a complete representation of any image (i.e. that any image can be reconstructed) have only recently been developed. However, the theory is very powerful, since it accommodates frequency and position simultaneously, and further it facilitates multiresolution analysis. We shall find wavelets again, when processing images to find low-level features. Applications of Gabor wavelets include measurement of iris texture to give a very powerful security system (Daugman, 1993) and face feature extraction for automatic face recognition (Lades et al., 1993). Wavelets continue to develop (Daubechies, 1990) and have found applications in image texture analysis (Laine and Fan, 1993), coding (da Silva and Ghanbari, 1996) and image restoration (Banham and Katsaggelos, 1996). Unfortunately, the discrete wavelet transform is not shift invariant, although there are approaches aimed to remedy this (see, for example, Donoho, 1995). As such, we shall not study it further and just note that there is an important class of transforms that combine spatial and spectral sensitivity, and it is likely that this importance will continue to grow.
2.7.4 Other transforms Decomposing a signal into sinusoidal components was one of the first approaches to transform calculus, and this is why the Fourier transform is so important. The sinusoidal functions are called basis functions; the implicit assumption is that the basis functions map well to the signal components. There is (theoretically) an infinite range of basis functions. Discrete signals can map better into collections of binary components rather than sinusoidal ones. These collections (or sequences) of binary data are called sequency components and form the basis functions of the Walsh transform (Walsh, 1923). This has found wide application in the interpretation of digital signals, although it is less widely used in image processing. The Karhunen–Loéve transform (Karhunen, 1947; Loéve, 1948) (also called the Hotelling transform from which it was derived, or more popularly principal components analysis; see Chapter 12, Appendix 4) is a way of analysing (statistical) data to reduce it to those data which are informative, discarding those which are not. Images, sampling and frequency domain processing
63
2.8 Applications using frequency domain properties Filtering is a major use of Fourier transforms, particularly because we can understand an image, and how to process it, much better in the frequency domain. An analogy is the use of a graphic equalizer to control the way music sounds. In images, if we want to remove high-frequency information (like the hiss on sound) then we can filter, or remove, it by inspecting the Fourier transform. If we retain low-frequency components, we implement a low-pass filter. The low-pass filter describes the area in which we retain spectral components; the size of the area dictates the range of frequencies retained, and is known as the filter’s bandwidth. If we retain components within a circular region centred on the d.c. component, and inverse Fourier transform the filtered transform, then the resulting image will be blurred. Higher spatial frequencies exist at the sharp edges of features, so removing them causes blurring. But the amount of fluctuation is reduced too; any high-frequency noise will be removed in the filtered image. The implementation of a low-pass filter which retains frequency components within a circle of specified radius is the function low_filter, given in Code 2.5. This operator assumes that the radius and centre coordinates of the circle are specified before its use. Points within the circle remain unaltered, whereas those outside the circle are set to zero, black.
low_filter(pic):= for y∈0.. rows(pic)–1 for x∈0.. cols(pic)–1 filtered y,x ← picy,x if y –
rows(pic) 2
2
+
x–
cols(pic) 2
2
– radius2 ≤ 0 0
otherwise
filtered
Code 2.5 Implementing low-pass filtering
When applied to an image we obtain a low-pass filtered version. In application to an image of a face, the low spatial frequencies are the ones which change slowly, as reflected in the resulting, blurred image (Figure 2.26a). The high-frequency components have been removed as shown in the transform (Figure 2.26b). The radius of the circle controls how much of the original
(a) Low-pass filtered image
(b) Low-pass filtered transform
Figure 2.26 Illustrating low- and high-pass filtering
64 Feature Extraction and Image Processing
(c) High-pass filtered image
(d) High-pass filtered transform
image is retained. In this case, the radius is 10 pixels (and the image resolution is 256 × 256). If a larger circle were to be used, more of the high-frequency detail would be retained (and the image would look more like its original version); if the circle was very small, an even more blurred image would result, since only the lowest spatial frequencies would be retained. This differs from the earlier Gabor wavelet approach, which allows for localized spatial frequency analysis. Here, the analysis is global: we are filtering the frequency across the whole image. Alternatively, we can retain high-frequency components and remove low-frequency ones. This is a high-pass filter. If we remove components near the d.c. component and retain all the others, the result of applying the inverse Fourier transform to the filtered image will be to emphasize the features that were removed in low-pass filtering. This can lead to a popular application of the high-pass filter: to ‘crispen’ an image by emphasizing its high-frequency components. An implementation using a circular region merely requires selection of the set of points outside the circle, rather than inside as for the low-pass operator. The effect of highpass filtering can be observed in Figure 2.26(c), which shows removal of the low-frequency components: this emphasizes the hair and the borders of a face’s features, since these are where brightness varies rapidly. The retained components are those which were removed in low-pass filtering, as illustrated in the transform, Figure 2.26(d). It is also possible to retain a specified range of frequencies. This is known as band-pass filtering. It can be implemented by retaining frequency components within an annulus centred on the d.c. component. The width of the annulus represents the bandwidth of the band-pass filter. This leads to digital signal processing theory. There are many considerations to be made in the way you select, and the manner in which frequency components are retained or excluded. This is beyond a text on computer vision. For further study in this area, Rabiner and Gold (1975), or Oppenheim et al. (1999), although published (in their original form) a long time ago now, remain as popular introductions to digital signal processing theory and applications. It is possible to recognize the object within the low-pass filtered image. Intuitively, this implies that we could store only the frequency components selected from the transform data, rather than all the image points. In this manner a fraction of the information would be stored, and still provide a recognizable image, albeit slightly blurred. This concerns image coding, which is a popular target for image processing techniques. For further information see Clarke (1985) or a newer text, such as Woods (2006).
2.9 Further reading We shall meet the frequency domain throughout this book, since it allows for an alternative interpretation of operation, in the frequency domain as opposed to the time domain. This will occur in low- and high-level feature extraction, and in shape description. Further, it allows for some of the operations we shall cover. Because of the availability of the FFT, it is also used to speed up algorithms. Given these advantages, it is well worth looking more deeply. For introductory study, there is Who is Fourier (Lex, 1995), which offers a lighthearted and completely digestible overview of the Fourier transform, and is simply excellent for a starter view of the topic. For further study (and entertaining study too!) of the Fourier transform, try The Fourier Transform and its Applications by Bracewell (1986). Some of the standard image processing texts include much coverage of transform calculus, such as Jain (1989), Gonzalez and Wintz (1987) and Pratt (2007). For more coverage of the DCT try Jain (1989); for excellent coverage of the Walsh Images, sampling and frequency domain processing
65
transform try Beauchamp’s superb text (Beauchamp, 1975). For wavelets, the book by Wornell (1996) introduces wavelets from a signal processing standpoint, or there is Mallat’s classic text (Mallat, 1999). For general signal processing theory there are introductory texts (e.g. Meade and Dillon, 1986) or Ifeachor’s excellent book (Ifeachor and Jervis, 2002), for more complete coverage try Rabiner and Gold (1975) or Oppenheim and Schafer (1999) (as mentioned earlier). Finally, on the implementation side of the FFT (and for many other signal processing algorithms), Numerical Recipes in C (Press et al., 2002) is an excellent book. It is extremely readable, full of practical detail and well worth a look. Numerical Recipes is on the web too, together with other signal processing sites, as listed in Table 1.4.
2.10 References Ahmed, N., Natarajan, T. and Rao, K. R., Discrete Cosine Transform, IEEE Trans. Comput., pp. 90–93, 1974 Banham, M. R. and Katsaggelos, K., Spatially Adaptive Wavelet-Based Multiscale Image Restoration, IEEE Trans. Image Process., 5(4), pp. 619–634, 1996 Beauchamp, K. G., Walsh Functions and Their Applications, Academic Press, London, 1975 Bracewell, R. N., The Discrete Hartley Transform, J. Opt. Soc. Am., 73(12), pp. 1832–1835, 1983 Bracewell, R. N., The Fast Hartley Transform, Proc. IEEE, 72(8), pp. 1010–1018, 1984 Bracewell, R. N., The Fourier Transform and its Applications, Revised 2nd edn, McGraw-Hill Book Co., Singapore, 1986 Clarke, R. J., Transform Coding of Images, Addison Wesley, Reading, MA, 1985 Daubechies, I., The Wavelet Transform, Time Frequency Localization and Signal Analysis, IEEE Trans. Inform. Theory, 36(5), pp. 961–1004, 1990 Daugman, J. G., Complete Discrete 2D Gabor Transforms by Neural Networks for Image Analysis and Compression, IEEE Trans. Acoust. Speech Signal Process., 36(7), pp. 1169–1179, 1988 Daugman, J. G., High Confidence Visual Recognition of Persons by a Test of Statistical Independence, IEEE Trans. PAMI, 15(11), pp. 1148–1161, 1993 Donoho, D. L., Denoizing by Soft Thresholding, IEEE Trans. Inform. Theory, 41(3), pp. 613–627, 1995 Gonzalez, R. C. and Wintz P., Digital Image Processing, 2nd edn, Addison Wesley, Reading, MA, 1987 Hartley, R. L. V., A More Symmetrical Fourier Analysis Applied to Transmission Problems, Proc. IRE, 144, pp. 144–150, 1942 Ifeachor, E. C. and Jervis, B. W., Digital Signal Processing, 2nd edn, Prentice Hall, Hemel Hempstead, 2002 Jain, A. K. Fundamentals of Computer Vision, Prentice Hall International (UK), Hemel Hempstead, 1989 Karhunen, K., Über Lineare Methoden in der Wahrscheinlich-Keitsrechnung, Ann. Acad. Sci. Fennicae, Ser A.I.37, 1947 (Translation in I. Selin, On Linear Methods in Probability Theory, Doc. T-131, The RAND Corporation, Santa Monica, CA, 1960) Lades, M., Vorbruggen, J. C., Buhmann, J., Lange, J., Madsburg, C. V. D., Wurtz, R. P. and Konen, W., Distortion Invariant Object Recognition in the Dynamic Link Architecture, IEEE Trans. Comput., 42, pp. 300–311, 1993 Laine, A. and Fan, J., Texture Classification by Wavelet Packet Signatures, IEEE Trans. PAMI, 15, pp. 1186–1191, 1993 66 Feature Extraction and Image Processing
Lex, T. C. O. L. T. (!!), Who is Fourier, A Mathematical Adventure, Language Research Foundation, Boston, MA, US, 1995 Loéve, M., Fonctions Alétoires de Seconde Ordre, In: P. Levy (Ed.), Processus Stochastiques et Mouvement Brownien, Hermann, Paris, 1948 Mallat, S., A Wavelet Tour of Signal Processing, 2nd edn, Academic Press, New York, 1999 Meade, M. L. and Dillon, C. R., Signals and Systems, Models and Behaviour, Van Nostrand Reinhold (UK), Wokingham, 1986 Oppenheim, A. V., Schafer, R. W. and Buck, J. R. Digital Signal Processing, 2nd edn, Prentice Hall International (UK), Hemel Hempstead, 1999 Pratt, W. K., Digital Image Processing: PIKS Scientific Inside, 4th edn, Wiley, New York, 2007 Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P., Numerical Recipes in C++ The Art of Scientific Computing, 2nd edn, Cambridge University Press, Cambridge, 2002 Rabiner, L. R. and Gold, B., Theory and Application of Digital Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1975 da Silva, E. A. B. and Ghanbari, M., On the Performance of Linear Phase Wavelet Transforms in Low Bit-Rate Image Coding, IEEE Trans. Image Process., 5(5), pp. 689–704, 1996 Walsh, J. L., A Closed Set of Normal Orthogonal Functions, Am. J. Math., 45(1), pp. 5–24, 1923 Woods, J. W., Multidimensional Signal, Image, and Video Processing and Coding, Academic Press, New York, 2006 Wornell, G. W., Signal Processing with Fractals, A Wavelet-Based Approach, Prentice Hall, Upper Saddle River, NJ, 1996
Images, sampling and frequency domain processing
67
This page intentionally left blank
3
.
.
Basic image processing operations 3.1 Overview We shall now start to process digital images. First, we shall describe the brightness variation in an image using its histogram. We shall then look at operations that manipulate the image so as to change the histogram, processes that shift and scale the result (making the image brighter or dimmer, in different ways). We shall also consider thresholding techniques that turn an image from grey level to binary. These are called single point operations. After that, we shall move to group operations where the group is those points found inside a template. Some of the most common operations on the groups of points are statistical, providing images where each point is the result of, say, averaging the neighbourhood of each point in the original image. We shall
Table 3.1 Overview of Chapter 3 Main topic
Sub topics
Main points
Image description
Portray variation in image brightness content as a graph/histogram.
Histograms, image contrast.
Point operations
Calculate new image points as a function of the point at the same place in the original image. The functions can be mathematical, or can be computed from the image itself and will change the image’s histogram. Finally, thresholding turns an image from grey level to a binary (black and white) representation.
Histogram manipulation; intensity mapping: addition, inversion, scaling, logarithm, exponent. Intensity normalization; histogram equalization. Thresholding and optimal thresholding.
Group operations
Calculate new image points as a function of neighbourhood of the point at the same place in the original image. The functions can be statistical, including: mean (average); median and mode. Advanced filtering techniques, including feature preservation. Morphological operators process an image according to shape, starting with binary and moving to grey-level operations.
Template convolution (including frequency domain implementation). Statistical operators: direct averaging, median filter and mode filter. Anisotropic diffusion for image smoothing. Other operators: force field transform. Mathematical morphology: hit or miss transform, erosion, dilation (including grey-level operators) and Minkowski operators.
69
see how the statistical operations can reduce noise in the image, which is of benefit to the feature extraction techniques to be considered later. As such, these basic operations are usually for preprocessing for later feature extraction or to improve display quality.
3.2 Histograms The intensity histogram shows how individual brightness levels are occupied in an image; the image contrast is measured by the range of brightness levels. The histogram plots the number of pixels with a particular brightness level against the brightness level. For 8 bit pixels, the brightness ranges from zero (black) to 255 (white). Figure 3.1 shows an image of an eye and its histogram. The histogram (Figure 3.1b) shows that not all the grey levels are used and the lowest and highest intensity levels are close together, reflecting moderate contrast. The histogram has a region between 100 and 120 brightness values which contains the dark portions of the image, such as the hair (including the eyebrow) and the eye’s iris. The brighter points relate mainly to the skin. If the image was darker, overall, the histogram would be concentrated towards black. If the image was brighter, but with lower contrast, then the histogram would be thinner and concentrated near the whiter brightness levels.
400
p_histogrambright 200
0
(a) Image of eye
100
200
bright (b) Histogram of eye image
Figure 3.1 An image and its histogram
This histogram shows us that we have not used all available grey levels. Accordingly, we could stretch the image to use them all, and the image would become clearer. This is essentially cosmetic attention to make the image’s appearance better. Making the appearance better, especially in view of later processing, is the focus of many basic image processing operations, as will be covered in this chapter. The histogram can also reveal whether there is much noise in the image, if the ideal histogram is known. We might want to remove this noise, not only to improve the appearance of the image, but also to ease the task of (and to present the target better for) later feature extraction techniques. This chapter concerns these basic operations which can improve the appearance and quality of images. The histogram can be evaluated by the operator histogram, in Code 3.1. The operator first initializes the histogram to zero. Then, the operator works by counting up the number of image points that have an intensity at a particular value. These counts for the different values form the overall histogram. The counts are then returned as the two-dimensional (2D) histogram (a vector of the count values), which can be plotted as a graph (Figure 3.1b). 70
Feature Extraction and Image Processing
histogram(pic):= for bright∈0..255 pixels_at_levelbright←0 for x∈0..cols(pic)–1 for y∈0..rows(pic)–1 level←picy,x pixels_at_levellevel←pixels_at_levellevel+1 pixels_at_level Code 3.1 Evaluating the histogram
3.3 Point operators 3.3.1 Basic point operations The most basic operations in image processing are point operations where each pixel value is replaced with a new value obtained from the old one. If we want to increase the brightness to stretch the contrast we can simply multiply all pixel values by a scalar, say by 2 to double the range. Conversely, to reduce the contrast (although this is not usual) we can divide all point values by a scalar. If the overall brightness is controlled by a level, l, (e.g. the brightness of global light) and the range is controlled by a gain, k, the brightness of the points in a new picture, N, can be related to the brightness in old picture, O, by: Nxy = k × Oxy + l
∀x y ∈ 1 N
(3.1)
This is a point operator that replaces the brightness at points in the picture according to a linear brightness relation. The level controls overall brightness and is the minimum value of the output picture. The gain controls the contrast, or range, and if the gain is greater than unity, the output range will be increased. This process is illustrated in Figure 3.2. So the image of the eye, processed by k = 12 and l = 10, will become brighter (Figure 3.2a) and with better contrast, although in this case the brighter points are mostly set near to white (255). These factors can be seen in its histogram (Figure 3.2b).
400
b_eye_histbright
200
0
0
100
200
bright
(a) Image of brighter eye
(b) Histogram of brighter eye
Figure 3.2 Brightening an image
Basic image processing operations
71
The basis of the implementation of point operators was given earlier, for addition in Code 1.3. The stretching process can be displayed as a mapping between the input and output ranges, according to the specified relationship, as in Figure 3.3. Figure 3.3(a) is a mapping where the output is a direct copy of the input (this relationship is the dotted line in Figure 3.3c and d); Figure 3.3(b) is the mapping for brightness inversion where dark parts in an image become bright and vice versa. Figure 3.3(c) is the mapping for addition and Figure 3.3(d) is the mapping for multiplication (or division, if the slope was less than that of the input). In these mappings, if the mapping produces values that are smaller than the expected minimum (say negative when zero represents black), or larger than a specified maximum, then a clipping process can be used to set the output values to a chosen level. For example, if the relationship between input and output aims to produce output points with intensity value greater than 255, as used for white, the output value can be set to white for these points, as it is in Figure 3.3(c).
Output brightness
Output brightness
White
White
Black
Black
Black
White
Input brightness
Black
(a) Copy
White
(b) Brightness inversion
Output brightness
Output brightness
White
White
Black Black
Input brightness
Black White
Input brightness
(c) Brightness addition
Black
White
Input brightness
(d) Brightness scaling by multiplication
Figure 3.3 Intensity mappings
The sawtooth operator is an alternative form of the linear operator and uses a repeated form of the linear operator for chosen intervals in the brightness range. The sawtooth operator is used to emphasize local contrast change (as in images where regions of interest can be light or dark). This is illustrated in Figure 3.4, where the range of brightness levels is mapped into four linear regions by the sawtooth operator (Figure 3.4b). This remaps the intensity in the eye image to 72
Feature Extraction and Image Processing
50 saw_toothbright
0
100
200
bright (a) Image of ‘sawn’ eye
(b) Sawtooth operator
Figure 3.4 Applying the sawtooth operator
highlight local intensity variation, as opposed to global variation, in Figure 3.4(a). The image is now presented in regions, where the region selection is controlled by the intensity of its pixels. Finally, rather than simple multiplication we can use arithmetic functions such as logarithm to reduce the range or exponent to increase it. This can be used, say, to equalize the response of a camera, or to compress the range of displayed brightness levels. If the camera has a known exponential performance, and outputs a value for brightness which is proportional to the exponential of the brightness of the corresponding point in the scene of view, the application of a logarithmic point operator will restore the original range of brightness levels. The effect of replacing brightness by a scaled version of its natural logarithm (implemented as Nxy = 20 ln100Oxy ) is shown in Figure 3.5(a); the effect of a scaled version of the exponent (implemented as Nxy = 20 expOxy /100) is shown in Figure 3.5(b). The scaling factors were chosen to ensure that the resulting image can be displayed since the logarithm or exponent greatly reduces or magnifies pixel values, respectively. This can be seen in the results: Figure 3.5(a) is dark with a small range of brightness levels, whereas Figure 3.5(b) is much brighter, with greater contrast. Naturally, application of the logarithmic point operator will change any multiplicative changes in brightness to become additive. As such, the logarithmic operator can find application in reducing the effects of multiplicative intensity change. The logarithm operator is often used to
(a) Logarithmic compression
(b) Exponential expansion
Figure 3.5 Applying exponential and logarithmic point operators
Basic image processing operations
73
compress Fourier transforms, for display purposes. This is because the d.c. component can be very large with contrast, too large to allow the other points to be seen. In hardware, point operators can be implemented using look-up tables (LUTs), which exist in some framegrabber units. LUTs give an output that is programmed, and stored, in a table entry that corresponds to a particular input value. If the brightness response of the camera is known, it is possible to preprogram a LUT to make the camera response equivalent to a uniform or flat response across the range of brightness levels.
3.3.2 Histogram normalization Popular techniques to stretch the range of intensities include histogram (intensity) normalization. Here, the original histogram is stretched, and shifted, to cover all the 256 available levels. If the original histogram of old picture O starts at Omin and extends up to Omax brightness levels, then we can scale up the image so that the pixels in the new picture N lie between a minimum output level Nmin and a maximum level Nmax , simply by scaling up the input intensity levels according to: Nxy =
Nmax − Nmin × Oxy − Omin + Nmin Omax − Omin
∀x y ∈ 1 N
(3.2)
A Matlab implementation of intensity normalization, appearing to mimic Matlab’s imagesc function, the normalize function in Code 3.2, uses an output ranging from Nmin = 0 to Nmax = 255. This is scaled by the input range that is determined by applying the max and the min operators to the input picture. Note that in Matlab, a 2D array needs double application of the max and min operators, whereas in Mathcad max(image) delivers the maximum. Each point
function normalized=normalize(image) %Histogram normalization to stretch from black to white %Usage: [new image]=normalize(image) %Parameters: image-array of integers %Author: Mark S. Nixon %get dimensions [rows,cols]=size(image); %set minimum minim=min(min(image)); %work out range of input levels range=max(max(image))-minim; %normalize the image for x=1:cols %address all columns for y=1:rows %address all rows normalized(y,x)=floor((image(y,x)-minim)*255/range); end end Code 3.2 Intensity normalization
74 Feature Extraction and Image Processing
in the picture is then scaled as in Equation 3.2 and the floor function is used to ensure an integer output. The process is illustrated in Figure 3.6, and can be compared with the original image and histogram in Figure 3.1. An intensity normalized version of the eye image is shown in Figure 3.6(a), which now has better contrast and appears better to the human eye. Its histogram (Figure 3.6b) shows that the intensity now ranges across all available levels (there is actually one black pixel!).
400 n_histbright
200
0
50
100
150
200
250
bright (b) Histogram of intensity normalized eye
(a) Intensity normalized eye
400 e_histbright
200
0
50
100
150
200
250
bright
(c) Histogram of equalized eye
(d) Histogram of histogram equalized eye
Figure 3.6 Illustrating intensity normalization and histogram equalization
3.3.3 Histogram equalization Histogram equalization is a non-linear process aimed to highlight image brightness in a way particularly suited to human visual analysis. Histogram equalization aims to change a picture in such a way as to produce a picture with a flatter histogram, where all levels are equiprobable. In order to develop the operator, we can first inspect the histograms. For a range of M levels then the histogram plots the points per level against level. For the input (old) and the output (new) image, the number of points per level is denoted as Ol and Nl (for 0 ≤ l ≤ M), respectively. For square images, there are N 2 points in the input and the output image, so the sum of points per level in each should be equal: M l=0
Ol =
M
(3.3)
Nl
l=0
Basic image processing operations
75
Also, this should be the same for an arbitrarily chosen level p, since we are aiming for an output picture with a uniformly flat histogram. So the cumulative histogram up to level p should be transformed to cover up to the level q in the new histogram: p
Ol =
l=0
q
Nl
(3.4)
l=0
Since the output histogram is uniformly flat, the cumulative histogram up to level p should be a fraction of the overall sum. So the number of points per level in the output picture is the ratio of the number of points to the range of levels in the output image: Nl =
N2 Nmax − Nmin
(3.5)
So the cumulative histogram of the output picture is: q
Nl = q ×
l=0
N2 Nmax − Nmin
(3.6)
By Equation 3.4 this is equal to the cumulative histogram of the input image, so: q×
p N2 = Ol Nmax − Nmin l=0
(3.7)
This gives a mapping for the output pixels at level q, from the input pixels at level p as: q=
p Nmax − Nmin × Ol N2 l=0
(3.8)
This gives a mapping function that provides an output image that has an approximately flat histogram. The mapping function is given by phrasing Equation 3.8 as an equalizing function E of the level q and the image (O) as Eq O =
p Nmax − Nmin × Ol N2 l=0
(3.9)
The output image is then Nxy = EOxy O
(3.10)
The result of equalizing the eye image is shown in Figure 3.6. The intensity equalized image, Figure 3.6(c) has much better defined features (especially around the eyes) than in the original version (Figure 3.1). The histogram (Figure 3.6d) reveals the non-linear mapping process whereby white and black are not assigned equal weight, as they were in intensity normalization. Accordingly, more pixels are mapped into the darker region and the brighter intensities become better spread, consistent with the aims of histogram equalization. Its performance can be very convincing since it is well mapped to the properties of human vision. If a linear brightness transformation is applied to the original image then the equalized histogram will be the same. If we replace pixel values with ones computed according to Equation 3.1, the result of histogram equalization will not change. An alternative interpretation is that if we equalize images (before further processing) then we need not worry about any brightness transformation in the original image. This is to be expected, since the linear operation of the brightness change in Equation 3.2 does not change the overall shape of the histogram, only its size and position. However, noise in the image acquisition process will affect the shape of the original histogram, and hence the equalized version. So the equalized histogram of a 76
Feature Extraction and Image Processing
picture will not be the same as the equalized histogram of a picture with some noise added to it. You cannot avoid noise in electrical systems, however well you design a system to reduce its effect. Accordingly, histogram equalization finds little use in generic image processing systems, although it can be potent in specialized applications. For these reasons, intensity normalization is often preferred when a picture’s histogram requires manipulation. In implementation, the function equalize in Code 3.3, we shall use an output range where Nmin = 0 and Nmax = 255. The implementation first determines the cumulative histogram for each level of the brightness histogram. This is then used as a LUT for the new output brightness at that level. The LUT is used to speed implementation of Equation 3.9, since it can be precomputed from the image to be equalized.
equalize(pic):=
range ← 255 number ← rows(pic).cols(pic) for bright∈ 0..255 pixels_at_levelbright ← 0 for x∈0..cols(pic)–1 for y∈0..rows(pic)–1 pixels_at_levelpicy,x ← pixels_at_levelpicy,x+1 sum ← 0 for level∈0..255 sum ← sum+pixels_at_levellevel histlevel ← floor
⎛ range ⎞ ·sum+0.00001 ⎝ number⎠
for x∈0..cols(pic)–1 for y∈0..rows(pic)–1 newpicy,x ← histpicy,x newpic Code 3.3 Histogram equalization
An alternative argument against use of histogram equalization is that it is a non-linear process and is irreversible. We cannot return to the original picture after equalization, and we cannot separate the histogram of an unwanted picture. In contrast, intensity normalization is a linear process and we can return to the original image, should we need to, or separate pictures, if required.
3.3.4 Thresholding The last point operator of major interest is called thresholding. This operator selects pixels that have a particular value, or are within a specified range. It can be used to find objects within a Basic image processing operations
77
picture if their brightness level (or range) is known. This implies that the object’s brightness must be known as well. There are two main forms: uniform and adaptive thresholding. In uniform thresholding, pixels above a specified level are set to white, those below the specified level are set to black. Given the original eye image, Figure 3.7 shows a thresholded image where all pixels above 160 brightness levels are set to white, and those below 160 brightness levels are set to black. By this process, the parts pertaining to the facial skin are separated from the background; the cheeks, forehead and other bright areas are separated from the hair and eyes. This can therefore provide a way of isolating points of interest.
Figure 3.7 Thresholding the eye image
Uniform thresholding clearly requires knowledge of the grey level, or the target features might not be selected in the thresholding process. If the level is not known, histogram equalization or intensity normalization can be used, but with the restrictions on performance stated earlier. This is, of course, a problem of image interpretation. These problems can only be solved by simple approaches, such as thresholding, for very special cases. In general, it is often prudent to investigate the more sophisticated techniques of feature selection and extraction, to be covered later. Before that, we shall investigate group operators, which are a natural counterpart to point operators. There are more advanced techniques, known as optimal thresholding. These usually seek to select a value for the threshold that separates an object from its background. This suggests that the object has a different range of intensities to the background, in order that an appropriate threshold can be chosen, as illustrated in Figure 3.8. Otsu’s method (Otsu, 1979) is one of the most popular techniques of optimal thresholding; there have been surveys (Sahoo et al., 1988; Lee et al., 1990; Glasbey, 1993) which compare the performance different methods can achieve. Essentially, Otsu’s technique maximizes the likelihood that the threshold is chosen so as to split the image between an object and its background. This is achieved by selecting a threshold that gives the best separation of classes, for all pixels in an image. The theory is beyond the scope of this section and we shall merely survey its results and give their implementation. The basis is use of the normalized histogram where the number of points at each level is divided by the total number of points in the image. As such, this represents a probability distribution for the intensity levels as pl = 78
Nl N2
Feature Extraction and Image Processing
(3.11)
No. of points Background Object
Brightness
Optimal threshold value
Figure 3.8 Optimal thresholding
This can be used to compute the zero- and first-order cumulative moments of the normalized histogram up to the kth level as k =
k
pl
(3.12)
l · pl
(3.13)
l=1
and k =
k l=1
The total mean level of the image is given by T =
Nmax
l · pl
(3.14)
l=1
The variance of the class separability is then the ratio B2 k =
T · k − k2 k1 − k
∀k ∈ 1 Nmax
(3.15)
The optimal threshold is the level for which the variance of class separability is at its maximum, namely the optimal threshold Topt is that for which the variance B2 Topt = max B2 k (3.16) 1≤k
A comparison of uniform thresholding with optimal thresholding is given in Figure 3.9 for the eye image. The threshold selected by Otsu’s operator is actually slightly lower than the value selected manually, and so the thresholded image does omit some detail around the eye, especially in the eyelids. However, the selection by Otsu is automatic, as opposed to manual, and this can be to application advantage in automated vision. Consider for example the need to isolate the human figure in Figure 3.10(a). This can be performed automatically by Otsu as shown in Figure 3.10(b). Note, however, that there are some extra points, due to illumination, which have appeared in the resulting image together with the human subject. It is easy to remove the isolated points, as we will see later, but more difficult to remove the connected ones. In this instance, the size of the human shape could be used as information to remove the extra points, although you might like to suggest other factors that could lead to their removal. Basic image processing operations
79
(a) Thresholding at level 160
(b) Thresholding by Otsu (level = 127)
Figure 3.9 Thresholding the eye image: manual and automatic
The code implementing Otsu’s technique is given in Code 3.4, which follows Equations 3.11–3.16 to provide the results in Figures 3.9 and 3.10. Here, the histogram function of Code 3.1 is used to give the normalized histogram. The remaining code refers directly to the earlier description of Otsu’s technique.
k
ω(k,histogram):= Σ
histograml – 1
l=1 k
μ (k,histogram):= Σ
l·histograml – 1
l=1 256
μ T(histogram):= Σ
l·histograml – 1
l=1
Otsu(image):=
image_hist ←
histogram(image) rows(image)·cols(image)
for k∈1..255 (μ T(image_hist)·ω(k,image_hist) – μ (k,image_hist))2 valuesk ← ω(k,image_hist)·(1 – ω(k,image_hist)) find_value(max(values),values)
Code 3.4 Optimal thresholding by Otsu’s technique
So far, we have considered global techniques, methods that operate on the entire image. There are also locally adaptive techniques that are often used to binarize document images before character recognition. As mentioned before, surveys of thresholding are available, and one (more recent) approach (Rosin, 2001) targets thresholding of images whose histogram is unimodal (has a single peak). One survey (Trier and Jain, 1995) compares global and local techniques with reference to document image analysis. These techniques are often used in statistical pattern recognition: the thresholded object is classified according to its statistical properties. However, these techniques find less use in image interpretation, where a common paradigm is that there is more than one object in the scene, such as Figure 3.7 where the thresholding operator has 80
Feature Extraction and Image Processing
(a) Walking subject
(b) Automatic thresholding by Otsu
Figure 3.10 Thresholding an image of a walking subject
selected many objects of potential interest. As such, only uniform thresholding is used in many vision applications, since objects are often occluded (hidden), and many objects have similar ranges of pixel intensity. Accordingly, more sophisticated metrics are required to separate them, by using the uniformly thresholded image, as discussed in later chapters. Further, the operation to process the thresholded image, say to fill in the holes in the silhouette or to remove the noise on its boundary or outside, is morphology, which is covered in Section 3.6.
3.4 Group operations 3.4.1 Template convolution Group operations calculate new pixel values from a pixel’s neighbourhood by using a ‘grouping’ process. The group operation is usually expressed in terms of template convolution, where the template is a set of weighting coefficients. The template is usually square, and its size is usually odd to ensure that it can be positioned appropriately. The size is usually used to describe the template; a 3 × 3 template is 3 pixels wide by 3 pixels long. New pixel values are calculated by placing the template at the point of interest. Pixel values are multiplied by the corresponding weighting coefficient and added to an overall sum. The sum (usually) evaluates a new value for the centre pixel (where the template is centred) and this becomes the pixel in a new output image. If the template’s position has not yet reached the end of a line, the template is then moved horizontally by one pixel and the process repeats. This is illustrated in Figure 3.11, where a new image is calculated from an original one, by template convolution. The calculation obtained by template convolution for the centre pixel of the template in the original image becomes the point in the output image. Since the template cannot extend beyond the image, the new image is smaller than the original image because a new value cannot be computed for points in the border of the new image. When the template reaches the end of a line, it is repositioned at the start of the next line. For a 3 × 3 neighbourhood, nine weighting coefficients wt are applied to points in the original image to calculate a point in the new image. The position of the new point (at the centre) is shaded in the template. Basic image processing operations
81
X
X
Original image
New image
Figure 3.11 Template convolution process
To calculate the value in new image, N, at point with coordinates x,y, the template in Figure 3.12 operates on an original image O according to: w0 × Ox−1y−1 Nxy = w3 × Ox−1y w6 × Ox−1y+1
+ + +
w1 × Oxy−1 w4 × Oxy w7 × Oxy+1
+ + +
w2 × Ox+1y−1 w5 × Ox+1y w8 × Ox+1y+1
+ +
∀x y ∈ 2 N − 1 (3.17)
w0
w1
w2
w3
w4
w5
w6
w7
w8
Figure 3.12 3 × 3 Template and weighting coefficients
Note that we cannot ascribe values to the picture’s borders. This is because when we place the template at the border, parts of the template fall outside the image and have no information from which to calculate the new pixel value. The width of the border equals half the size of the template. To calculate values for the border pixels, we now have three choices: • set the border to black (or deliver a smaller picture) • assume (as in Fourier) that the image replicates to infinity along both dimensions and calculate new values by cyclic shift from the far border • calculate the pixel value from a smaller area. None of these approaches is optimal. The results here use the first option and set border pixels to black. Note that in many applications the object of interest is imaged centrally or, at least, imaged within the picture. As such, the border information is of little consequence to the remainder of the process. Here, the border points are set to black, by starting functions with a zero function which sets all the points in the picture initially to black (0). 82
Feature Extraction and Image Processing
An alternative representation for this process is given by using the convolution notation as N = W∗O
(3.18)
where N is the new image which results from convolving the template W (of weighting coefficients) with the image O. The Matlab implementation of a general template convolution operator convolve is given in Code 3.5. This function accepts, as arguments, the picture image and the template to be convolved with it, template. The result of template convolution is a picture convolved. The operator first initializes the temporary image temp to black (zero brightness levels). Then
function convolved=convolve(image,template) %New image point brightness convolution of template with image %Usage:[new image]=convolve(image,template of point values) %Parameters:image-array of points % template-array of weighting coefficients %Author: Mark S. Nixon %get image dimensions [irows,icols]=size(image); %get template dimensions [trows,tcols]=size(template); %set a temporary image to black temp(1:irows,1:icols)=0; %half of template rows is trhalf=floor(trows/2); %half of template cols is tchalf=floor(tcols/2); %then convolve the template for x=trhalf+1:icols-trhalf %address all columns except border for y=tchalf+1:irows-tchalf %address all rows except border sum=0; for iwin=1:trows %address template columns for jwin=1:tcols %address template rows sum=sum+image(y+jwin-tchalf-1,x+iwin-trhalf-1)* template(jwin,iwin); end end temp(y,x)=sum; end end %finally, normalize the image convolved=normalize(temp); Code 3.5 Template convolution operator
Basic image processing operations
83
the size of the template is evaluated. These give the range of picture points to be processed in the outer for loops that give the coordinates of all points resulting from template convolution. The template is convolved at each picture point by generating a running summation of the pixel values within the template’s window multiplied by the respective template weighting coefficient. Finally, the resulting image is normalized to ensure that the brightness levels are occupied appropriately. Template convolution is usually implemented in software. It can also be implemented in hardware and requires a two-line store, together with some further latches, for the (input) video data. The output is the result of template convolution, summing the result of multiplying weighting coefficients by pixel values. This is called pipelining, since the pixels essentially move along a pipeline of information. Note that two line stores can be used if the video fields only are processed. To process a full frame, one of the fields must be stored since it is presented in interlaced format. Processing can be analogue, using operational amplifier circuits and charge coupled device (CCD) for storage along bucket brigade delay lines. Finally, an alternative implementation is to use a parallel architecture: for multiple instruction multiple data (MIMD) architectures, the picture can be split into blocks (spatial partitioning); single instruction multiple data (SIMD) architectures can implement template convolution as a combination of shift and add instructions.
3.4.2 Averaging operator For an averaging operator, the template weighting functions are unity (or 1/9 to ensure that the result of averaging nine white pixels is white, not more than white!). The template for a 3 × 3 averaging operator, implementing Equation 3.17, is given by the template in Figure 3.13, where
1/9
1/9
1/9
1/9
1/9
1/9
1/9
1/9
1/9
Figure 3.13 3 × 3 averaging operator template coefficients
Figure 3.14 Applying direct averaging
84 Feature Extraction and Image Processing
the location of the point of interest is again shaded. The result of averaging the eye image with a 3 × 3 operator is shown in Figure 3.14. This shows that much of the detail has now disappeared, revealing the broad image structure. The eyes and eyebrows are now much clearer from the background, but the fine detail in their structure has been removed. For a general implementation, Code 3.6, we can define the width of the operator as winsize, the template size is winsize × winsize. We then form the average of all points within the area covered by the template. This is normalized (divided) by the number of points in the template’s window. This is a direct implementation of a general averaging operator (i.e. without using the template convolution operator in Code 3.5).
ave(pic,winsize):=
new ← zero(pic) winsize ⎛ ⎝ 2 for x∈half..cols(pic)–half–1 for y∈half..rows(pic)–half–1 half ← floor ⎛ ⎝
winsize–1
winsize–1
iwin=0
jwin=0
Σ
newy,x ← floor
Σ
picy+iwin–half,x+jwin–half
(winsize·winsize)
new
Code 3.6 Direct averaging
To implement averaging by using the template convolution operator, we need to define a template. This is illustrated for direct averaging in Code 3.7, even though the simplicity of the direct averaging template usually precludes such implementation. The application of this template is also shown in Code 3.7. (Note that there are averaging operators in Mathcad and Matlab that can also be used for this purpose.)
averaging_template(winsize):= sum ← winsize.winsize for y∈0..winsize–1 for x∈0..winsize–1 templatey,x ← 1 template sum smoothed:=tm_conv(p,averaging_template(3)) Code 3.7 Direct averaging by template convolution
The effect of averaging is to reduce noise; this is its advantage. An associated disadvantage is that averaging causes blurring which reduces detail in an image. It is also a low-pass filter Basic image processing operations
85
since its effect is to allow low spatial frequencies to be retained, and to suppress high-frequency components. A larger template, say 3 × 3 or 5 × 5, will remove more noise (high frequencies) but reduce the level of detail. The size of an averaging operator is then equivalent to the reciprocal of the bandwidth of a low-pass filter that it implements Smoothing was earlier achieved by low-pass filtering via the Fourier transform (Section 2.8). The Fourier transform gives an alternative method to implement template convolution and to speed it up, for larger templates. In Fourier transforms, the process that is dual to convolution is multiplication (as in Section 2.3). So template convolution (denoted∗ ) can be implemented by multiplying the Fourier transform of the template T by the Fourier transform of the picture, P, to which the template is to be applied. The result needs to be inverse transformed to return to the picture domain. P ∗ T = −1 P × T
(3.19)
The transform of the template and the picture need to be the same size before we can perform the point-by-point multiplication. Accordingly, the image containing the template is zero padded before its transform, which simply means that zeros are added to the template in positions which lead to a template of the same size as the image. The process is illustrated in Code 3.8 and starts by calculation of the transform of the zero-padded template. The convolution routine then multiplies the transform of the template by the transform of the picture point by point (using the vectorize operator, symbolized by the arrow above the operation). When the routine is invoked, it is supplied with a transformed picture. The resulting transform is reordered before inverse transformation, to ensure that the image is presented correctly. (Theoretical study of this process is presented in Section 5.3.2, where we show how the same process can be used to find shapes in images.)
conv(pic,temp):= pic_spectrum ← Fourier(pic) temp_spectrum ← Fourier(temp) convolved_spectrum ←(pic_spectrum.temp_spectrum) result ← inv_Fourier(rearrange(convolved_spectrum)) result
new_smooth:=conv(p,square) Code 3.8 Template convolution by the Fourier transform
Code 3.8 is simply a different implementation of direct averaging. It achieves the same result, but by transform domain calculus. It can be faster to use the transform rather than the direct implementation. The computational cost of a 2D fast Fourier transform (FFT) is of the order of N 2 logN. If the transform of the template is precomputed, there are two transforms required and there is one multiplication for each of the N 2 transformed points. The total cost of the Fourier implementation of template convolution is then of the order of CFFT = 4N 2 logN + N 2
(3.20)
The cost of the direct implementation for an m × m template is then m multiplications for each image point, so the cost of the direct implementation is of the order of 2
Cdir = N 2 m2 86
Feature Extraction and Image Processing
(3.21)
For Cdir < CFFT , we require: N 2 m2 < 4N 2 logN + N 2
(3.22)
If the direct implementation of template matching is faster than its Fourier implementation, we need to choose m so that m2 < 4 logN + 1
(3.23)
This implies that, for a 256 × 256 image, a direct implementation is fastest for 3 × 3 and 5 × 5 templates, whereas a transform calculation is faster for larger ones. An alternative analysis (Campbell, 1969) has suggested that (Gonzalez and Wintz, 1987) ‘if the number of non-zero terms in (the template) is less than 132 then a direct implementation is more efficient than using the FFT approach’. This implies a considerably larger template than our analysis suggests. This is in part due to higher considerations of complexity than our analysis has included. There are further considerations in the use of transform calculus, the most important being the use of windowing (such as Hamming or Hanning) operators to reduce variance in high-order spectral estimates. This implies that template convolution by transform calculus should perhaps be used when large templates are involved, and then only when speed is critical. If speed is indeed critical, it might be better to implement the operator in dedicated hardware, as described earlier.
3.4.3 On different template size Templates can be larger than 3 × 3. Since they are usually centred on a point of interest, to produce a new output value at that point, they are usually of odd dimension. For reasons of speed, the most common sizes are 3 × 3 5 × 5 and 7 × 7. Beyond this, say 9 × 9, many template points are used to calculate a single value for a new point, and this imposes high computational cost, especially for large images. (For example, a 9 × 9 operator covers nine times more points than a 3 × 3 operator.) Square templates have the same properties along both image axes. Some implementations use vector templates (a line), either because their properties are desirable in a particular application, or for reasons of speed. The effect of larger averaging operators is to smooth the image more, to remove more detail while giving greater emphasis to the large structures. This is illustrated in Figure 3.15. A 5 × 5 operator (Figure 3.15a) retains more detail than a 7 × 7 operator (Figure 3.15b), and much more than a 9 × 9 operator (Figure 3.15c). Conversely, the 9 × 9 operator retains only the largest structures such as the eye region (and virtually removing the iris), whereas this is retained more
(a) 5 × 5
(b) 7 × 7
(c) 9 × 9
Figure 3.15 Illustrating the effect of window size
Basic image processing operations
87
by the operators of smaller size. Note that the larger operators leave a larger border (since new values cannot be computed in that region) and this can be seen in the increase in border size for the larger operators, in Figure 3.15(b) and (c).
3.4.4 Gaussian averaging operator The Gaussian averaging operator has been considered to be optimal for image smoothing. The template for the Gaussian operator has values set by the Gaussian relationship. The Gaussian function g at coordinates x y is controlled by the variance 2 according to:
1 − x2+y2 gx y = e (3.24) 2 2 Equation 3.24 gives a way to calculate coefficients for a Gaussian template which is then convolved with an image. The effects of selection of Gaussian templates of differing size are shown in Figure 3.16. The Gaussian function essentially removes the influence of points greater than 3 in (radial) distance from the centre of the template. The 3 × 3 operator (Figure 3.16a) retains many more of the features than those retained by direct averaging (Figure 3.14). The effect of larger size is to remove more detail (and noise) at the expense of losing features. This is reflected in the loss of internal eye component by the 5 × 5 and the 7 × 7 operators in Figure 3.16(b) and (c), respectively.
(a) 3 × 3
2
2
(b) 5 × 5
(c) 7 × 7
Figure 3.16 Applying Gaussian averaging
A surface plot of the 2D Gaussian function of Equation 3.24 has the famous bell shape, as shown in Figure 3.17. The values of the function at discrete points are the values of a Gaussian template. Convolving this template with an image gives Gaussian averaging: the point in the averaged picture is calculated from the sum of a region where the central parts of the picture are weighted to contribute more than the peripheral points. The size of the template essentially dictates appropriate choice of the variance. The variance is chosen to ensure that template coefficients drop to near zero at the template’s edge. A common choice for the template size is 5 × 5 with variance unity, giving the template shown in Figure 3.18. This template is then convolved with the image to give the Gaussian blurring function. It is possible to give the Gaussian blurring function antisymmetric properties by scaling the x and y 88
Feature Extraction and Image Processing
Gaussian_template(19, 4)
Figure 3.17 Gaussian function
0.002
0.013
0.220
0.013
0.002
0.013
0.060
0.098
0.060
0.013
0.220
0.098
0.162
0.098
0.220
0.013
0.060
0.098
0.060
0.013
0.002
0.013
0.220
0.013
0.002
Figure 3.18 Template for the 5 × 5 Gaussian averaging operator = 10
coordinates. This can find application when an object’s shape, and orientation, is known before image analysis. By reference to Figure 3.16 it is clear that the Gaussian filter can offer improved performance compared with direct averaging: more features are retained while the noise is removed. This can be understood by Fourier transform theory. In Section 2.4.2 (Chapter 2) we found that the Fourier transform of a square is a 2D sinc function. This has a non-even frequency response (the magnitude of the transform does not reduce in a smooth manner) and has regions where the transform becomes negative, called sidelobes. These can have undesirable effects since there are high frequencies that contribute more than some lower ones, which is a bit paradoxical in lowpass filtering to remove noise. In contrast, the Fourier transform of a Gaussian function is another Gaussian function, which decreases smoothly without these sidelobes. This can lead to better performance since the contributions of the frequency components reduce in a controlled manner. In a software implementation of the Gaussian operator, we need a function implementing Equation 3.24, the Gaussian_template function in Code 3.9. This is used to calculate the coefficients of a template to be centred on an image point. The two arguments are winsize, the (square) operator’s size, and the standard deviation that controls its width, as discussed earlier. The operator coefficients are normalized by the sum of template values, as before. This summation is stored in sum, which is initialized to zero. The centre of the square template is then evaluated as half the size of the operator. Then, all template coefficients are calculated by Basic image processing operations
89
a version of Equation 3.24 which specifies a weight relative to the centre coordinates. Finally, the normalized template coefficients are returned as the Gaussian template. The operator is used in template convolution, via convolve, as in direct averaging (Code 3.5).
function template=gaussian_template(winsize,sigma) %Template for Gaussian averaging %Usage:[template]=gaussian_template(number, number) %Parameters: winsize-size of template (odd, integer) % sigma-variance of Gaussian function %Author: Mark S. Nixon %centre is half of window size centre=floor(winsize/2)+1; %we'll normalize by the total sum sum=0; %so work out the coefficients and the running total for i=1:winsize for j=1:winsize template(j,i)=exp(-(((j-centre)*(j-centre))+((i-centre) *(i-centre)))/(2*sigma*sigma)) sum=sum+template(j,i); end end %and then normalize template=template/sum; Code 3.9 Gaussian template specification
3.5 Other statistical operators 3.5.1 More on averaging The averaging process is actually a statistical operator since it aims to estimate the mean of a local neighbourhood. The error in the process is high; for a population of N samples, the statistical error is of the order of: mean error = √ (3.25) N Increasing the averaging operator’s size improves the error in the estimate of the mean, but at the expense of fine detail in the image. The average is an estimate optimal for a signal corrupted by additive Gaussian noise (see Appendix 3, Section 11.1). The estimate of the mean maximizes the probability that the noise has its mean value, namely zero. According to the central limit theorem, the result of adding many noise sources together is a Gaussian distributed noise source. In images, noise arises in sampling, in quantization, in transmission and in processing. By the 90
Feature Extraction and Image Processing
central limit theorem, the result of these (independent) noise sources is that image noise can be assumed to be Gaussian. In fact, image noise is not necessarily Gaussian distributed, giving rise to more statistical operators. One of these is the median operator, which has demonstrated capability to reduce noise while retaining feature boundaries (in contrast to smoothing, which blurs both noise and the boundaries), and the mode operator, which can be viewed as optimal for a number of noise sources, including Rayleigh noise, but is very difficult to determine for small, discrete, populations.
3.5.2 Median filter The median is another frequently used statistic; the median is the centre of a rank-ordered distribution. The median is usually taken from a template centred on the point of interest. Given the arrangement of pixels in Figure 3.19(a), the pixel values are arranged into a vector format (Figure 3.19b). The vector is then sorted into ascending order (Figure 3.19c). The median is the central component of the sorted vector; this is the fifth component since we have nine values.
2
8
7
4
0
6
3
5
7
2
4
3
(a) 3 × 3 template
8
0
5
7
6
7
7
8
(b) Unsorted vector
0
2
3
4
5
6
7
↑ median
(c) Sorted vector, giving median
Figure 3.19 Finding the median from a 3 × 3 template
The median operator is usually implemented using a template. Here, we shall consider a 3 × 3 template. Accordingly, we need to process the nine pixels in a template centred on a point with coordinates x y. In a Mathcad implementation, these nine points can be extracted into vector format using the operator unsorted in Code 3.10. This requires a integer pointer to nine values, x1. The modulus operator is then used to ensure that the correct nine values are extracted.
x1:=0..8 unsortedx1:=p
x+mod(x1,3)–1,x+floor
⎛x1 ⎛ –1 ⎝3 ⎝
Code 3.10 Reformatting a neighbourhood into a vector
Basic image processing operations
91
We then arrange the nine pixels, within the template, in ascending order using the Mathcad sort function (Code 3.11). This gives the rank ordered list and the median is the central component of the sorted vector, in this case the fifth component (Code 3.12). These functions can then be grouped to give the full median operator as in Code 3.13.
sorted:=sort(unsorted) Code 3.11 Using the Mathcad sort function
our_median:=sorted4 Code 3.12 Determining the median
med(pic):= newpic ← zero(pic) for x∈1..cols(pic)–2 for y∈1..rows(pic)–2 for x1∈ 0..8 unsortedx1 ← pic
x1 y+mod(x1,3)–1,x+floor ⎛ ⎛ –1 ⎝3⎝
sorted ← sort(unsorted) newpicy,x ← sorted4 newpic
Code 3.13 Determining the median
The median can be taken from larger template sizes. It is available as the median operator in Mathcad, but only for square matrices. The development here has aimed not only to demonstrate how the median operator works, but also to provide a basis for further development. The rank ordering process is computationally demanding (slow) and motivates study into the deployment of fast algorithms, such as Quicksort (e.g. Huang et al., 1979, is an early approach). The computational demand has also motivated use of template shapes, other than a square. A selection of alternative shapes is shown in Figure 3.20. Common alternative shapes include a cross or a line (horizontal or vertical), centred on the point of interest; these shapes can afford much faster operation since they cover fewer pixels. The basis of the arrangement presented here could be used for these alternative shapes, if required. The median has a well-known ability to remove salt and pepper noise. This form of noise, arising from say decoding errors in picture transmission systems, can cause isolated white and black points to appear within an image. It can also arise when rotating an image, when points 92
Feature Extraction and Image Processing
(a) Cross
(b) Horizontal line
(c) Vertical line
Figure 3.20 Alternative template shapes for median operator
remain unspecified by a standard rotation operator (Appendix 2), as in a texture image, rotated by 10 in Figure 3.21(a). When a median operator is applied, the salt and pepper noise points will appear at either end of the rank-ordered list and are removed by the median process, as shown in Figure 3.21(b). The median operator has practical advantage, owing to its ability to retain edges (the boundaries of shapes in images) while suppressing the noise contamination. As such, like direct averaging, it remains a worthwhile member of the stock of standard image processing tools. For further details concerning properties and implementation, see Hodgson et al. (1985). (Note that practical implementation of image rotation is a computer graphics issue, and is usually done by texture mapping; further details can be found in Hearn and Baker, 1997.)
(a) Rotated fence
(b) Median filtered
Figure 3.21 Illustrating median filtering
Finding the background to an image is an example application of statistical operators. Say we have a sequence of images of a walking subject, and we want to be able to find the background (so that we can separate the walking subject from it), such as the sequence of images shown in Figure 3.22(a)–(f) where a subject is walking from left to right. We can average the images so as to find the background. If we form a temporal average, an image where each point is the average of the points in the same position in each of the six images, we achieve a result where the walking subject appears to be in the background, but very faintly, as in Figure 3.22(g). The shadow occurs since the walking subject’s influence on image brightness is reduced by onesixth, but it is still there. We could use more images, the ones in between the ones we already Basic image processing operations
93
have, and then the shadow will become much fainter. We can also include spatial averaging as in Section 3.3.2, to reduce further the effect of the walking subject, as shown in Figure 3.22(h). This gives spatiotemporal averaging. For this, we have not required any more images, but the penalty paid for the better improvement in the estimate of the background is lack of detail. We cannot see the numbers in the clock, because of the nature of spatial averaging. However, if we form the background image by taking the median of the six images, a temporal median, we get a much better estimate of the background, as shown in Figure 3.22(i). A lot of the image detail is retained, while the walking subject disappears. In this case, for a sequence of images where the target walks in front of a static background, the median is the most appropriate operator. If we did not have a sequence, we could just average the single image with a large operator and that could provide some estimate of the background.
(a) Image 1
(b) Image 2
(g) Temporal averaging
(c) Image 3
(d) Image 4
(h) Spatiotemporal averaging
(e) Image 5
(f) Image 6
(i) Temporal median
Figure 3.22 Background estimation by mean and median filtering
3.5.3 Mode filter The mode is the final statistic of interest, although there are more advanced filtering operators to come. The mode is very difficult to determine for small populations and theoretically does not even exist for a continuous distribution. Consider, for example, determining the mode of the pixels within a square 5 × 5 template. It is possible for all 25 pixels to be different, so each could be considered to be the mode. As such, we are forced to estimate the mode: the truncated median filter, as introduced by Davies (1988), aims to achieve this. The truncated median filter is based on the premise that for many non-Gaussian distributions, the order of the mean, the median and the mode is the same for many images, as illustrated in Figure 3.23. Accordingly, if we truncate the distribution (i.e. remove part of it, where the part selected to be removed in Figure 3.23 is from the region beyond the mean) then the median of the truncated distribution will approach the mode of the original distribution. The implementation of the truncated median, trun_med, operator is given in Code 3.14. The operator first finds the mean and the median of the current window. The distribution of intensity of points within the current window is truncated on the side of the mean so that the median now bisects the distribution of the remaining points (as such, not affecting symmetrical 94 Feature Extraction and Image Processing
No. of points
Mode Median Mean
Brightness
Figure 3.23 Arrangement of mode, median and mean
trun_med(p,wsze):=
newpic←zero(p)
⎛ wsze⎞ ⎟ ⎝ 2 ⎠
ha←floor ⎟
for x∈ha..cols(p)−ha−1 for y∈ha..rows(p)−ha−1 win←submatrix(p,y−ha,y+ha,x−ha,x+ha) med←median(win) ave←mean(win) upper←2⋅med−min(win) lower←2⋅med−max(win) cc←0 for i∈0..wsze−1 for j∈0..wsze−1 if (winj,i
(winj,i>lower)⋅(med>ave) truncc←winj,i
cc←cc+1 newpicy,x←median(trun) if cc>0 newpicy,x←med otherwise newpic Code 3.14 The truncated median operator
distributions). So that the median bisects the remaining distribution, if the median is less than the mean, the point at which the distribution is truncated, upper, is upper = median + median − mindistribution = 2 · median − mindistribution
(3.26)
Basic image processing operations
95
If the median is greater than the mean, then we need to truncate at a lower point (before the mean), lower, given by lower = 2 · median − maxdistribution
(3.27)
The median of the remaining distribution then approaches the mode. The truncation is performed by storing pixels values in a vector trun. A pointer, cc, is incremented each time a new point is stored. The median of the truncated vector is then the output of the truncated median filter at that point. The window is placed at each possible image point, as in template convolution. However, there can be several iterations at each position to ensure that the mode is approached. In practice, only a few iterations are usually required for the median to converge to the mode. The window size is usually large, say 7 × 7 or 9 × 9 or even more. The action of the operator is illustrated in Figure 3.24 when applied to a 128 × 128 part of the ultrasound image (Figure 1.1c), from the centre of the image and containing a crosssectional view of an artery. Ultrasound results in particularly noisy images, in part because the scanner is usually external to the body. The noise is multiplicative Rayleigh noise, for which the mode is the optimal estimate. This noise obscures the artery which appears in cross-section in Figure 3.24(a); the artery is basically elliptical in shape. The action of the 9 × 9 truncated median operator (Figure 3.24b) is to remove noise while retaining feature boundaries, whereas a larger operator shows better effect (Figure 3.24c).
(a) Part of ultrasound image
(b) 9 × 9 operator
(c) 13 × 13 operator
Figure 3.24 Applying truncated median filtering
Close examination of the result of the truncated median filter shows that a selection of boundaries are preserved which are not readily apparent in the original ultrasound image. This is one of the known properties of median filtering: an ability to reduce noise while retaining feature boundaries. There have been many other approaches to speckle filtering; the most popular include direct averaging (Shankar, 1986), median filtering, adaptive (weighted) median filtering (Loupas and McDicken, 1987) and unsharp masking (Bamber and Daft, 1986).
3.5.4 Anisotropic diffusion The most advanced form of smoothing is achieved by preserving the boundaries of the image features in the smoothing process (Perona and Malik, 1990). This is one of the advantages of the 96 Feature Extraction and Image Processing
median operator and a disadvantage of the Gaussian smoothing operator. The process is called anisotropic diffusion, by virtue of its basis. Its result is illustrated in Figure 3.25(b), where the feature boundaries (such as those of the eyebrows or the eyes) in the smoothed image are crisp and the skin is more matt in appearance. This implies that we are filtering within the features and not at their edges. By way of contrast, the Gaussian operator result in Figure 3.25(c) smooths not just the skin but also the boundaries (the eyebrows in particular seem quite blurred), giving a less pleasing and less useful result. Since we shall later use the boundary information to interpret the image, its preservation is of much interest.
(a) Original image
(b) Anisotropic diffusion
(c) Gaussian smoothing
Figure 3.25 Filtering by anisotropic diffusion and the Gaussian operator
As ever, there are some parameters to select to control the operation, so we shall consider the technique’s basis so as to guide their selection. Further, it is computationally more complex than Gaussian filtering. The basis of anisotropic diffusion is, however, rather complex, especially here, and invokes concepts of low-level feature extraction which are covered in the next chapter. One strategy you might use is to mark this page, then go ahead and read Sections 4.1 and 4.2, and then return here. Alternatively, you could just plough on, since that is exactly what we shall do. The complexity is due to the process not only invoking low-level feature extraction (to preserve feature boundaries) but also, as its basis, invoking concepts of heat flow, as well as introducing the concept of scale space. So it will certainly be a hard read for many, but comparison of Figure 3.25(b) with Figure 3.25(c) shows that it is well worth the effort. The essential idea of scale space is that there is a multiscale representation of images, from low resolution (a coarsely sampled image) to high resolution (a finely sampled image). This is inherent in the sampling process, where the coarse image is the structure and the higher resolution increases the level of detail. As such, we can derive a multiscale set of images by convolving an original image with a Gaussian function, by Equation 3.24 Ixy = Ixy 0 ∗ gx y
(3.28)
where Ixy 0 is the original image, gx y is the Gaussian template derived from Equation 3.24, and Ixy is the image at level . The coarser level corresponds to larger values of the standard deviation ; conversely, the finer detail is given by smaller values. We have already seen that the larger values of reduce the detail and are then equivalent to an image at a coarser scale, so this is a different view of the same process. The difficult part is that the family of images derived this way can equivalently be viewed as the solution of the heat equation
I t = Ixy t (3.29) Basic image processing operations
97
where denotes del, the (directional) gradient operator from vector algebra, and with the initial condition that I0 = Ixy 0. The heat equation itself describes the temperature T changing with time t as a function of the thermal diffusivity (related to conduction) as
T t = 2 T
(3.30)
and in one-dimensional (1D) form this is
2 T
T t = 2 x
(3.31)
so the temperature measured along a line is a function of time, distance, the initial and boundary conditions, and the properties of a material. The relation of this with image processing is clearly an enormous ouch! There are clear similarities between Equations 3.31 and 3.29. They have the same functional form and this allows for insight, analysis and parameter selection. The heat equation (Equation 3.29) is the anisotropic diffusion equation
I t = · cxy tIxy t
(3.32)
where · is the divergence operator (which essentially measures how the density within a region changes), with diffusion coefficient cxy . The diffusion coefficient applies to the local change in the image Ixy t in different directions. If we have a lot of local change, we seek to retain it since the amount of change is the amount of boundary information. The diffusion coefficient indicates how much importance we give to local change: how much of it is retained. (The equation reduces to isotropic diffusion – Gaussian filtering – if the diffusivity is constant, since c = 0.) There is no explicit solution to this equation. By approximating differentiation by differencing (this is explored more in Section 4.2), the rate of change of the image between time step t and time step t + 1, we have
I t = It + 1 − It
(3.33)
This implies that we have an iterative solution, and for later consistency we shall denote the image I at time step t + 1 as I = It + 1, so we then have I − I = · cxy t Ixy
(3.34)
and again by approximation, using differences evaluated this time over the four compass directions north, south, east and west, we have N Ixy = Ixy−1 − Ixy
(3.35)
S Ixy = Ixy+1 − Ixy
(3.36)
E Ixy = Ix−1y − Ixy
(3.37)
W Ixy = Ix+1y − Ixy
(3.38)
The template and weighting coefficients for these are shown in Figure 3.26. 98
Feature Extraction and Image Processing
1 1
–4
1
1
Figure 3.26 Approximations by spatial difference in anisotropic diffusion
When we use these as an approximation to the right-hand side in Equation 3.34, we then have · cxy tIxy = cNxy N I + cSxy S I + cExy E I + cWxy W I , which gives (3.39) I − I = cNxy N I + cSxy S I + cExy E I + cWxy W I I = Ixy where 0 ≤ ≤ 1/4 and where cNxy cSxy cExy and cWxy denote the conduction coefficients in the four compass directions. By rearrangement of this we obtain the equation that we shall use for the anisotropic diffusion operator (3.40) I = I + cNxy N I + cSxy S I + cExy E I + cWxy W I I = Ixy This shows that the solution is iterative: images at one time step (denoted by ) are computed from images at the previous time step (denoted ), given the initial condition that the first image is the original (noisy) image. Change (in time and in space) has been approximated as the difference between two adjacent points, which gives the iterative equation and shows that the new image is formed by adding a controlled amount of the local change consistent with the main idea: that the smoothing process retains some of the boundary information. We are not finished yet, though, since we need to find values for cNxy cSxy cExy and cWxy . These are chosen to be a function of the difference along the compass directions, so that the boundary (edge) information is preserved. In this way we seek a function that tends to zero with increase in the difference (an edge or boundary with greater contrast) so that diffusion does not take place across the boundaries, keeping the edge information. As such, we seek cNxy = g N I cSxy = g S I cExy = g E I
(3.41)
cWxy = g W I and one function that can achieve this is −x2 k2 g x k = e
(3.42)
[There is potential confusion with using the same symbol as for the Gaussian function (Equation 3.24), but we have followed the original authors’ presentation.] This function clearly has the desired properties since when the values of the differences are large the function g is very small; conversely, when is small then g tends to unity. k is another parameter whose Basic image processing operations
99
value we have to choose: it controls the rate at which the conduction coefficient decreases with increasing difference magnitude. The effect of this parameter is shown in Figure 3.27. Here, the solid line is for the smaller value of k and the dotted one is for a larger value. Evidently, a larger value of k means that the contribution of the difference reduces less than for a smaller value of k. In both cases, the resulting function is near unity for small differences and near zero for large differences, as required. An alternative to this is to use the function g2x k =
1 1 + x 2 k2
(3.43)
which has similar properties to the function in Equation 3.42.
1
g (Δ, 10) 0.5
g (Δ, 30)
0
20
40
Δ
Figure 3.27 Controlling the conduction coefficient in anisotropic diffusion
This all looks rather complicated, so let’s recap. First, we want to filter an image by retaining boundary points. These are retained according to the value of k chosen in Equation 3.42. This function is operated in the four compass directions, to weight the brightness difference in each direction (Equation 3. 41). These contribute to an iterative equation which calculates a new value for an image point by considering the contribution from its four neighbouring points (Equation 3.40). This needs the choice of one parameter, . Further, we need to choose the number of iterations for which calculation proceeds. For information, Figure 3.25(b) was calculated over 20 iterations and we need to use sufficient iterations to ensure that convergence has been achieved. We also need to choose values for k and . By analogy, k is the conduction coefficient and low values preserve edges and high values allow diffusion (conduction) to occur; and how much smoothing can take place. The two parameters are interrelated, although largely controls the amount of smoothing. Given that low values of either parameter means that no filtering effect is observed, we can investigate their effect by setting one parameter to a high value and varying the other. In Figure 3.28(a)–(c) we use a high value of k, which means that edges are not preserved, and we can observe that different values of control the amount of smoothing. (A discussion of how this Gaussian filtering process is achieved can be inferred from Section 4.2.4.) Conversely, we can see how different values for k control the level of edge preservation in Figure 3.28(d)–(f), where some structures around the eye are not preserved for larger values of k. 100
Feature Extraction and Image Processing
(a) k = 100 and λ = 0.05
(b) k = 100 and λ = 0.15
(c) k = 100 and λ = 0.25
(d) k = 5 and λ = 0.25
(e) k = 15 and λ = 0.25
(f) k = 25 and λ = 0.25
Figure 3.28 Applying anisotropic diffusion
The original presentation of anisotropic diffusion (Perona and Malik, 1990) is extremely lucid and well worth a read if you consider selecting this technique. It has greater detail on formulation and on analysis of results than space here allows for (and is suitable at this stage). Among other papers on this topic, one (Black et al., 1998) studied the choice of conduction coefficient leading to a function which preserves sharper edges and improves automatic termination. As ever, with techniques that require much computation there have been approaches that speed implementation or achieve similar performance more rapidly (e.g. Fischl and Schwartz, 1999).
3.5.5 Force field transform There are many more image filtering operators; we have so far covered those that are among the most popular. Others offer alternative insight, sometimes developed in the context of a specific application. By way of example, Hurley developed a transform called the force field transform (Hurley et al., 2002, 2005) which uses an analogy to gravitational force. The transform pretends that each pixel exerts a force on its neighbours which is inversely proportional to the square of the distance between them. This generates a force field where the net force at each point is the aggregate of the forces exerted by all the other pixels on a ‘unit test pixel’ at that point. This very large-scale summation affords very powerful averaging which reduces the effect of noise. The approach was developed in the context of ear biometrics, recognizing people by their ears, which has unique advantage as a biometric in that the shape of people’s ears does not change with age, and of course, unlike a face, ears do not smile! The force field transform of an ear (Figure 3.29a) is shown in Figure 3.29(b). Here, the averaging process is reflected in the reduction of the effects of hair. The transform itself has highlighted ear structures, especially the top of the ear and the lower ‘keyhole’ (the notch). Basic image processing operations
101
(a) Image of ear
(b) Magnitude of force field transform
Figure 3.29 Illustrating the force field transform
The image shown is the magnitude of the force field. The transform itself is a vector operation, and includes direction (Hurley, 2002). The transform is expressed as the calculation of the force F between two points at positions ri and rj , which is dependent on the value of a pixel at point ri as ri − rj Fi rj = P ri (3.44) ri − rj 3 which assumes that the point rj is of unit ‘mass’. This is a directional force (which is why the inverse square law is expressed as the ratio of the difference to its magnitude cubed) and the magnitude and directional information has been exploited to determine an ear ‘signature’ by which people can be recognized. In application, Equation 3.44 can be used to define the coefficients of a template that is convolved with an image (implemented by the FFT to improve speed); as with many of the techniques that have been covered in this chapter; a Mathcad implementation is also given (Hurley et al., 2002). Note that this transform exposes low-level features (the boundaries of the ears), which is the focus of the next chapter. How we can determine shapes is a higher level process, and the processes by which we infer or recognize identity from the low- and the high-level features will be covered in Chapter 8.
3.5.6 Comparison of statistical operators The different image filtering operators are shown by way of comparison in Figure 3.30. All operators are 5 × 5 and are applied to the earlier ultrasound image (Figure 3.24a). Figure 3.30(a)–(d) are the result of the mean (direct averaging), Gaussian averaging, median and truncated median, respectively. We have just shown the advantages of anisotropic diffusion compared with Gaussian smoothing, so we will not repeat them here. Each operator shows a different performance: the mean operator removes much noise, but blurs feature boundaries; Gaussian averaging retains more features, but shows little advantage over direct averaging (it is not Gaussian-distributed noise anyway); the median operator retains some noise, but with clear feature boundaries; and the truncated median removes more noise, but along with picture detail. Clearly, the increased size of the truncated median template, by the results in Figure 3.24(b) 102
Feature Extraction and Image Processing
(a) Mean
(b) Gaussian average
(c) Median
(d) Truncated median
Figure 3.30 Comparison of filtering operators
and (c), can offer improved performance. This is to be expected since, by increasing the size of the truncated median template, we are essentially increasing the size of the distribution from which the mode is found. As yet, however, we have not yet studied any quantitative means to evaluate this comparison. We can only perform subjective appraisal of the images in Figure 3.30. This appraisal has been phrased in terms of the contrast boundaries perceived in the image, and on the basic shape that the image presents. Accordingly, better appraisal is based on the use of feature extraction. Boundaries are the low-level features studied in the next chapter; shape is a high-level feature studied in Chapter 5.
3.6 Mathematical morphology Mathematical morphology analyses images by using operators developed using set theory (Serra, 1986; Serra and Soile, 1994). It was originally developed for binary images and was extended to include grey-level data. The word morphology concerns shapes: in mathematical morphology we process images according to shape, by treating both as sets of points. In this way, morphological operators define local transformations that change pixel values that are represented as sets. The ways in which pixel values are changed is formalized by the definition of the hit or miss transformation. In the hit and miss transformation, an object represented by a set X is examined through a structural element represented by a set B. Different structuring elements are used to change the operations on the set X. The hit or miss transformation is defined as the point operator
X ⊗ B = x Bx1 ⊂ X ∩ Bx2 ⊂ X c (3.45) In this equation, x represents one element of X, that is a pixel in an image. The symbol X c denotes the complement of X (the set of image pixels that is not in the set X) and the structuring element B is represented by two parts, B1 and B2 , that are applied to the set X or to its complement X c . The structuring element is a shape and this is how mathematical morphology operations process images according to shape properties. The operation of B1 on X is a hit; the operation of B2 on X c is a miss. The subindex x in the structural element indicates that it is moved to the position of the element x. That is, in a manner similar to other group operators, B defines a window that is moved through the image. Basic image processing operations
103
Figure 3.31 illustrates a binary image and a structuring element. Image pixels are divided into those belonging to X and those belonging to its complement X c . The figure shows a structural element and it decomposition into the two sets B1 and B2 . Each subset is used to analyse the set X and its complement. Here, we use black for the elements of B1 and white for B2 to indicate that they are applied to X and X c , respectively.
B
X=
B1
B2
Xc =
Figure 3.31 Image and structural element
Equation 3.45 defines a process that moves the structural element B to be placed at each pixel in the image and it performs a pixel-by-pixel comparison against the template B. If the value of the image is the same as that of the structuring element, then the image’s pixel forms part of the resulting set X ⊗ B. An important feature of this process is that is not invertible. That is, information is removed to suppress or enhance geometrical features in an image.
3.6.1 Morphological operators The simplest forms of morphological operators are defined when either B1 or B2 are empty. When B1 is empty Equation 3.45 defines an erosion (reduction) and when B2 is empty it defines a dilation (increase). That is, an erosion operation is given by
X B = x Bx1 ⊂ X (3.46) and a dilation is given by
X ⊕ B = x Bx2 ⊂ X c
(3.47)
In the erosion operator, the hit or miss transformation establishes that a pixel x belongs to the eroded set if each point of the element B1 translated to x is on X. Since all the points in B1 need to be in X, this operator removes the pixels at the borders of objects in the set X. Thus, it erodes or shrinks the set. One of the most common applications of this is to remove noise in thresholded images. This is illustrated in Figure 3.32, where in (a) we have a noisy binary image, the image is eroded in (b), removing noise but making the letters smaller, and this is corrected by dilation in (c). We shall show how we can use shape to improve this filtering process: put the morph into morphology. Figure 3.33 illustrates the operation of the erosion operator. Figure 3.33(a) contains a 3 × 3 template that defines the structural element B1 . The centre pixel is the origin of the set. 104
Feature Extraction and Image Processing
(a) Original image
(b) Erosion
(c) Dilation
Figure 3.32 Filtering by morphology
(a) Structural element
(b) Image
(c) Erosion
Figure 3.33 Example of the erosion operator
Figure 3.33(b) shows an image containing a region of black pixels that defines the set X. Figure 3.33(c) shows the result of the erosion. The eroded set is formed from black pixels only and we use grey to highlight the pixels that were removed from X by the erosion operator. For example, when the structural element is moved to the position shown as a grid in Figure 3.33(c), the central pixel is removed since only five pixels of the structural element are in X. The dilation operator defined in Equation 3.47 establishes that a point belongs to the dilated set when all the points in B2 are in the complement. This operator erodes or shrinks the complement and when the complement is eroded, the set X is dilated. Figure 3.34 illustrates a dilation process. The structural element shown in Figure 3.34(a) defines the set B2 . We indicate its elements in white since it should be applied to the complement of X. Figure 3.34(b) shows an image example and Figure 3.34(c) the result of the dilation. The black and grey pixels belong to the dilation of X. We use grey to highlight the pixels that are added to the set. During the dilation, we place the structural element on each pixel in the complement. These are the white pixels in Figure 3.34(b). When the structural element is not fully contained, it is removed form the complement, so it becomes part of X. For example, when the structural element is moved to the position shown as a grid in Figure 3.34(c), the central pixel is removed from the complement since one of the pixels in the template is in X. There is an alternative formulation for the dilation operator that defines the transformation over the set X instead to its complement. This definition is obtained by observing that when all Basic image processing operations
105
(a) Structural element
(b) Image
(c) Dilation
Figure 3.34 Example of the dilation operator
elements of B2 are in X c , this is equivalent to none of the elements in the negation of B2 being in X. That is, dilation can also be written as the intersection of translated sets, as
X ⊕ B = x x ∈ ¬Bx2 (3.48) Here, the symbol ¬ denotes negation and it changes the structural element from being applied to the complement to the set. For example, the negation of the structural element in Figure 3.34(a) is the set in Figure 3.33(a). Thus, Equation 3.48 defines a process where a point is added to the dilated set when at least one element of ¬B2 is in X. For example, when the structural element is at the position shown in Figure 3.34(c), one element in X is in the template, thus the central point is added to the dilation. Neither dilation nor erosion specifies a required shape for the structuring element. In general, it is defined to be square or circular, but other shapes such as a cross or a triangle can be used. Changes in the shape will produce subtle changes in the results, but the main feature of the structural element is given by its size, since this determines the strength of the transformation. In general, applications prefer to use small structural elements (for speed) and perform a succession of transformations until a desirable result is obtained. Other operators can be defined by sequences of erosions and dilations. For example, the opening operator is defined by an erosion followed by a dilation. That is, X B = X B ⊕ B
(3.49)
Similarly, a closing operator is defined by a dilation followed of an erosion. That is, X • B = X ⊕ B B
(3.50)
Closing and opening operators are generally used as filters that remove dots characteristic of pepper noise and to smooth the surface of shapes in images. These operators are generally applied in succession and the number of times they are applied depends on the structural element size and image structure. In addition to filtering, morphological operators can be used to develop other image processing techniques. For example, edges can be detected by subtracting the original image and the one obtained by an erosion or dilation. Other example is the computation of skeletons that are thin representations of a shape. A skeleton can be computed as the union of subtracting images obtained by applying erosions and openings with structural elements of increasing sizes. 106
Feature Extraction and Image Processing
3.6.2 Grey-level morphology In the definition in Equation 3.45 pixels belong to either the set X or its complement. Thus, it only applies to binary images. Greyscale or grey-level morphology extends Equation 3.45 to represent functions as sets, thus morphology operators can be applied to grey-level images. There are two alternative representations of functions as sets: the cross-section (Serra, 1986; Serra and Soile, 1994) and the umbra (Sternberg, 1986). The cross-section representation uses multiple thresholds to obtain a pile of binary images. Thus, the definition of Equation 3.45 can be applied to grey-level images by considering a collection of binary images as a stack of binary images formed at each threshold level. The formulation and implementation of this approach is cumbersome since it requires multiple structural elements and operators over the stack. The umbra approach is more intuitive and it defines sets as the points contained below functions. The umbra of a function fx consists of all points that satisfy fx. That is, U X = x z z < f x
(3.51)
Here, x represents a pixel and fx its grey level. Thus, the space x z is formed by the combination of all pixels and grey levels. For images, x is defined in two dimensions, thus all the points of the form x z define a cube in 3D space. An umbra is a collection of points in this 3D space. Notice that morphological definitions are for discrete sets, thus the function is defined at discrete points and for discrete grey levels. Figure 3.35 illustrates the concept of an umbra. For simplicity we show fx as a 1D function. In Figure 3.35(a), the umbra is drawn as a collection of points below the curve. The complement of the umbra is denoted as U c X and it is given by the points on and above the curve. The union of UX and U c X defines all the image points and grey-level values x z. In grey-level morphology, images and structural elements are represented by umbrae. Figure 3.35(b) illustrates the definition of two structural elements. The first example defines a structural element for the umbra, that is B1 . Similar to an image function, the umbra of the structural elements is defined by the points under the curve. The second example in Figure 3.35(b) defines a structural element for the complement, that is B2 . Similar to the complement of the umbra, this operator defines the points on and over the curve.
z
U c(x ) f (x )
z
B1
U (x ) x (a) Umbra
z
U (B 2)
B2
U (B 1) x
x
(b) Structural elements
Figure 3.35 Grey-level morphology
Basic image processing operations
107
The hit or miss transformation in Equation 3.45 is extended to grey-level functions by considering the inclusion operator in the umbrae. That is, 1
2 U X ⊗ B = x z U Bxz (3.52) ⊂ U X ∩ U Bxz ⊂ U c X Similar to the binary case, this equation defines a process that evaluates the inclusion of the translated structural element B. At difference of the binary definition, the structural element is translated along the pixels and grey-level values; that is, to the points x z. Thus, a point x z belongs to the umbra of the hit or miss transformation, if the umbrae of the elements B1 and B2 translated to x z are included in the umbra and its complement, respectively. The inclusion operator is defined for the umbra and its complement in different ways. An umbra is contained in other umbra if corresponding values of its function are equal or lower. For the complement, an umbra is contained if corresponding values of its function are equal or greater. We can visualize the process in Equation 3.52 by translating the structural element in the example in Figure 3.35. To determine whether a point x z is in the transformed set, we move the structural element B1 to the point and see whether its umbra fully intersects UX. If that is the case, the umbra of the structural element is contained in the umbra of the function and 1 2 UBxt ⊂ UX is true. Similarly, to test for UBxt ⊂ U c X, we move the structural element 2 B and see whether it is contained in the upper region of the curve. If both conditions are true, then the point where the operator is translated belongs to the umbra of the hit or miss transformation.
3.6.3 Grey-level erosion and dilation Based on the generalization in Equation 3.52, it is possible to reformulate operators developed for binary morphology so they can be applied to grey-level data. The erosion and dilation defined in Equations 3.46 and 3.47 are generalized to grey-level morphology as 1
U X B = x z U Bxz ⊂ U X (3.53) and
2
U X ⊕ B = x z U Bxz ⊂ U c X
(3.54)
The erosion operator establishes that the point x z belongs to the umbra of the eroded set if each point of the umbra of the element B1 translated to the point x z is under the umbra of X. A common way to visualize this process is to think that we move the structural element upwards in the grey-level axis. The erosion border is the highest point we can reach without going out of the umbra. Similar to the binary case, this operator removes the borders of the set X by increasing the separation in holes. Thus, it actually erodes or shrinks the structures in an image. Figure 3.36(a) illustrates the erosion operator for the image in Figure 3.35(a). Figure 3.36(a) shows the result of the erosion for the structural element shown on the right. For clarity we have marked the origin of the structure element with a black spot. In the result, only the black pixels form the eroded set, and we use grey to highlight the pixels that were removed from the umbra of X. It is easy to see that when the structural element is translated to a point that is removed, its umbra intersects U c X. Analogous to binary morphology, the dilation operator can be seen as an erosion of the complement of the umbra of X. That is, a point belongs to the dilated set when all the points in the umbra of B2 are in U c X. This operator erodes or shrinks the set U c X. When the complement is eroded, the umbra of X is dilated. The dilation operator fills holes decreasing the separation between prominent structures. This process is illustrated in Figure 3.36(b) for the 108
Feature Extraction and Image Processing
z
z
x (a) Erosion
x (b) Dilation
Figure 3.36 Grey-level operators
example in Figure 3.36(a). The structural element used is shown to the right in Figure 3.36(b). In the results, the black and grey pixels belong to the dilation. We use grey to highlight points that are added to the set. Points are removed from the complement and added to UX by translating the structural element looking for points where the structural element is not fully included in U c X. It is easy to see that when the structural element is translated to a point that is added to the dilation, its umbra intersects UX. Similar to Equation 3.48, dilation can be written as intersection of translated sets, thus it can be defined as an operator on the umbra of an image. That is, 2
(3.55) U X ⊕ B = x z x z ∈ U ¬Bxz The negation changes the structural element from being applied to the complement of the umbra to the umbra. That is, it changes the sing of the umbra to be defined below the curve. For the example in Figure 3.36(b), it easy to see that if the structural element ¬B2 is translated to any point added during the dilation, it intersects the umbra at least in one point.
3.6.4 Minkowski operators Equations 3.53, 3.54 and 3.55 require the computation of intersections of the pixels of a structural element that is translated to all the points in the image and for each grey-level value. Thus, its computation involves significant processing. However, some simplifications can be made. For the erosion process in Equation 3.53, the value of a pixel can be simply computed by comparing the grey-level values of the structural element and corresponding image pixels. The highest position that we can translate the structural element without intersecting the complement is given by the minimum value of the difference between the grey level of the image pixel and the corresponding pixel in the structural element. That is,
x = mini f x − i − B i
(3.56)
Here, Bi denotes the value of the ith pixel of the structural element. Figure 3.37(a) illustrates a numerical example for this equation. The structural element has three pixels with values 0, 1 and 0, respectively. The subtractions for the position shown in Figure 3.37(a) are 4 − 0 = 4 6 − 1 = 5 and 7 − 0 = 7. Thus, the minimum value is 4. As shown in Figure 3.37(a), this corresponds to the highest grey-level value that we can move up to the structural element and it is still fully contained in the umbra of the function. Basic image processing operations
109
z
z
11 10 9 8 7 6 5 4 3 2 1 0
11 10 9 8 7 6 5 4 3 2 1 0
x
(a) Erosion
x
(b) Dilation
Figure 3.37 Example of Minkowski difference and addition
Similar to Equation 3.56, the dilation can be obtained by comparing the grey-level values of the image and the structural element. For the dilation we have ⊕ x = maxi f x − i + B i
(3.57)
Figure 3.37(b) illustrates a numerical example of this equation. For the position of the structural element in Figure 3.37(b), the summation gives the values 8 + 0 = 8 8 + 1 = 9 and 4 + 0 = 4. As shown in the figure, the maximum value of 9 corresponds to the point where the structural element still intersects the umbra; therefore, this point should be added to the dilation. Equations 3.56 and 3.57 are known as the Minkowski operators and they formalize set operations as summations and differences. Thus, they provide definitions very useful for computer implementations. Code 3.15 shows the implement of the erosion operator based on Equation 3.56. Similar to Code 3.5, the value pixels in the output image are obtained by translating the operator function eroded = Erosion(image,template) %Implementation of erosion operator %Parameters: Template and image array of points %get the image and template dimensions [irows,icols]=size(image); [trows,tcols]=size(template); %create result image eroded(1:irows,1:icols)=uint8(0); %half of template trhalf=floor(trows/2); tchalf=floor(tcols/2); %Erosion for x=trhalf+1:icols-trhalf %columns in the image except border for y=tchalf+1:irows-tchalf %rows in the image except border min=256; for iwin=1:tcols %template columns for jwin=1:trows %template rows xi=x-trhalf-1+iwin; yi=y-tchalf-1+jwin;
110
Feature Extraction and Image Processing
sub=double(image(xi,yi))-double(template(iwin,jwin)); if sub<min & sub>0 min=sub; end end end eroded(x,y)=uint8(min); end end
Code 3.15 Erosion implementation
along the image pixels. The code subtracts the value of corresponding image and template pixels and it sets the value of the pixel in the output image to the minima. Code 3.16 shows the implement of the dilation operator based on Equation 3.57. This code is similar to Code 3.15, but corresponding values of the image and the structural element are added, and the maximum value is set as the result of the dilation.
function dilated = Dilation(image,template) %Implementation of dilation operator %Parameters: Template and image array of points %get the image and template dimensions [irows,icols]=size(image); [trows,tcols]=size(template); %create result image dilated(1:irows,1:icols)=uint8(0); %half of template trhalf=floor(trows/2); tchalf=floor(tcols/2); %Dilation for x=trhalf+1:icols-trhalf %columns in the image except border for y=tchalf+1:irows-tchalf %rows in the image except border max=0; for iwin=1:tcols %template columns for jwin=1:trows %template rows xi=x-trhalf-1+iwin; yi=y-tchalf-1+jwin; sub=double(image(xi,yi))+double(template(iwin,jwin)); if sub>max & sub>0 max=sub; end end end dilated(x,y)=uint8(max); end end
Code 3.16 Dilation implementation
Basic image processing operations
111
(a) Original image
(b) Erosion
(c) Dilation
(d) Opening
Figure 3.38 Examples of morphology operators
Figure 3.38 shows an example of the results obtained from the erosion and dilation using Codes 3.15 and 3.16. The original image shown in Figure 3.38(a) has 128 × 128 pixels and we used a flat structural element defined by an image with 9 × 9 pixels set to zero. For its simplicity, flat structural elements are very common in applications and they are generally set to zero to avoid creating offsets in the grey levels. In Figure 3.38, we can see that the erosion operation reduces the objects in the image while dilation expands white regions. We also used the erosion and dilation in succession to perform the opening show in Figure 3.38(d). The opening operation has a tendency to form regular regions of similar size to the original image while removing peaks and small regions. The strength of the operators is defined by the size of the structural elements. In these examples we use a fixed size and we can see that it strongly modifies regions during dilation and erosion. Elaborate techniques have combined multiresolution structures and morphological operators to analyse an image with operators of different sizes (Montiel et al., 1995).
3.7 Further reading Many texts cover basic point and group operators in much detail; in particular, some texts give many more examples, such as Russ (1995) and Seul (2000). Books with a C implementation 112
Feature Extraction and Image Processing
often concentrate on more basic techniques, including low-level image processing (Lindley, 1991; Parker, 1994). Some of the more advanced texts include more coverage of low-level operators (Rosenfeld and Kak, 1982; Castleman,1996). Parker (1994) includes C code for nearly all the low-level operations in this chapter and Seul (2000) has code too, and there is Matlab code in Gonzalez (2003). For study of the effect of the median operator on image data, see Bovik et al. (1987). Some of the newer techniques receive little treatment in the established literature, except for Chan and Shen (2005; with extensive coverage of noise filtering too). The truncated median filter is covered again in Davies (2005). For further study of the effects of different statistical operators on ultrasound images, see Evans and Nixon (1995, 1996). The concept of scale-space allows for considerably more refined analysis than is given here and while we shall revisit it later, it is rather unsuited to an introductory text. It was originally introduced by Witkin (1983) and further developed by others, including Koenderink (1984) who also considers the heat equation. There is even a series of conferences devoted to scale-space and morphology.
3.8 References Bamber, J. C. and Daft, C., Adaptive Filtering for Reduction of Speckle in Ultrasonic Pulse-Echo Images, Ultrasonics, 24(3), pp. 41–44, 1986 Black, M. J., Sapiro, G., Marimont, D. H. and Meeger, D., Robust Anisotropic Diffusion, IEEE Trans. Image Process., 7(3), pp. 421–432, 1998 Bovik, A. C., Huang, T. S. and Munson, D. C., The Effect of Median Filtering on Edge Estimation and Detection, IEEE Trans. PAMI, 9(2), pp. 181–194, 1987 Campbell, J. D., Edge Structure and the Representation of Pictures, PhD Thesis, University of Missouri, Columbia, MO, 1969 Castleman, K. R., Digital Image Processing, Prentice Hall, Englewood Cliffs, NJ, 1996 Chan, T. and Shen, J., Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2005 Davies, E. R., On the Noise Suppression Characteristics of the Median, Truncated Median and Mode Filters, Pattern Recog. Lett., 7(2), pp. 87–97, 1988 Davies, E. R., Machine Vision: Theory, Algorithms and Practicalities, 3rd edn, Morgan Kaufmann (Elsevier), Amsterdam, Netherlands, 2005 Evans, A. N. and Nixon M. S., Mode Filtering to Reduce Ultrasound Speckle for Feature Extraction, Proc. IEE – Vision, Image and Signal Processing, 142(2), pp. 87–94, 1995 Evans, A. N. and Nixon M. S., Biased Motion-Adaptive Temporal Filtering for Speckle Reduction in Echocardiography, IEEE Trans. Med. Imaging, 15(1), pp. 39–50, 1996 Fischl, B. and Schwartz, E. L., Adaptive Nonlocal Filtering: A Fast Alternative to Anisotropic Diffusion for Image Enhancement, IEEE Trans. PAMI, 21(1), pp. 42–48, 1999 Glasbey, C. A., An Analysis of Histogram-Based Thresholding Algorithms, CVGIP–Graphical Models Image Process., 55(6), pp. 532–537, 1993 Gonzalez, R. C. and Wintz P., Digital Image Processing, 2nd edn, Addison Wesley, Reading, MA, 1987 Gonzalez, R. C., Woods, R. E. and Eddins, S., Digital Image Processing using MATLAB, 1st edn, Prentice Hall, 2003 Hearn, D. and Baker, M. P., Computer Graphics C Version, 2nd edn, Prentice Hall, Upper Saddle River, NJ, 1997 Hodgson, R. M., Bailey, D. G., Naylor, M. J., Ng, A. and McNeill, S. J., Properties, Implementations and Applications of Rank Filters, Image Vision Comput., 3(1), pp. 3–14, 1985 Basic image processing operations
113
Huang, T., Yang, G. and Tang, G., A Fast Two-Dimensional Median Filtering Algorithm, IEEE Trans. ASSP, 27(1), pp. 13–18, 1979 Hurley, D. J., Nixon, M. S. and Carter, J. N., Force Field Energy Functionals for Image Feature Extraction, Image Vision Comput., 20, pp. 311–317, 2002 Hurley, D. J., Nixon, M. S. and Carter, J. N., Force Field Feature Extraction for Ear Biometrics, Comput. Vision Image Understand., 98(3), pp. 491–512, 2005 Koenderink, J., The Structure of Images, Biol.Cybern., 50, pp. 363–370, 1984 Lee, S. A., Chung, S. Y. and Park, R. H., A Comparative Performance Study of Several Global Thresholding Techniques for Segmentation, CVGIP, 52, pp. 171–190, 1990 Lindley, C. A., Practical Image Processing in C, Wiley & Sons, New York, 1991 Loupas, T. and McDicken, W. N., Noise Reduction in Ultrasound Images by Digital Filtering, Br. J. Radiol., 60, pp. 389–392, 1987 Montiel, M. E., Aguado, A. S., Garza, M. and Alarcón, J., Image Manipulation using M-filters in a Pyramidal Computer Model, IEEE Trans. PAMI, 17(11), pp. 1110–1115, 1995 Otsu, N., A Threshold Selection Method from Gray-Level Histograms, IEEE Trans. SMC, 9(1), pp. 62–66, 1979 Parker, J. R., Practical Computer Vision using C, Wiley & Sons, New York, 1994 Perona, P. and Malik, J., Scale-Space and Edge Detection using Anisotropic Diffusion, IEEE Trans. PAMI, 17(7), pp. 629–639, 1990 Rosenfeld, A. and Kak, A. C., Digital Picture Processing, 2nd edn, Vols 1 and 2, Academic Press, Orlando, FL, 1982 Rosin, P. L., Unimodal Thresholding, Pattern Recog., 34(11), pp. 2083–2096, 2001 Russ, J. C., The Image Processing Handbook, 4th edn, CRC Press (IEEE Press), Boca Raton, FL, 1995 Sahoo, P. K., Soltani, S., Wong, A. K. C. and Chen, Y. C., Survey of Thresholding Techniques, CVGIP, 41(2), pp. 233–260, 1988 Serra J., Introduction to Mathematical Morphology, CVGIP, 35, pp. 283–305, 1986 Serra, J. P. and Soille, P. (eds), Mathematical Morphology and its Applications to Image Processing, Kluwer Academic,Springer, NY, USA, 1994 Seul, M., O’Gorman, L. and Sammon, M. J., Practical Algorithms for Image Analysis: Descriptions, Examples, and Code, Cambridge University Press, Cambridge, 2000 Shankar, P. M., Speckle Reduction in Ultrasound B Scans using Weighted Averaging in Spatial Compounding, IEEE Trans. Ultrasonics Ferroelectrics Frequency Control, 33(6), pp. 754–758, 1986 Sternberg, S. R., Gray Scale Morphology, CVGIP, 35, pp. 333–355, 1986 Trier, O. D. and Jain, A. K., Goal-Directed Evaluation of Image Binarization Methods, IEEE Trans. PAMI, 17(12), pp. 1191–1201, 1995 Witkin, A., Scale-Space Filtering: A New Approach to Multi-Scale Description, Proc. Int. Joint Conf. Artificial Intelligence, pp. 1019–1021, 1983
114
Feature Extraction and Image Processing
4
.
.
Low-level feature extraction (including edge detection)
4.1 Overview We shall define low-level features to be those basic features that can be extracted automatically from an image without any shape information (information about spatial relationships). As such, thresholding is a form of low-level feature extraction performed as a point operation. All of these approaches can be used in high-level feature extraction, where we find shapes in images. It is well known that we can recognize people from caricaturists’ portraits. That is the first low-level feature we shall encounter. It is called edge detection and it aims to produce a line drawing, like one of a face in Figure 4.1(a) and (d), something akin to a caricaturist’s sketch, although without the exaggeration a caricaturist would imbue. There are very basic techniques and more advanced ones and we shall look at some of the most popular approaches. The first order detectors are equivalent to first order differentiation, and the second order edge detection operators are equivalent to a one-higher level of differentiation. An alternative form of edge detection is called phase congruency and we shall again see the frequency domain used to aid analysis, this time for low-level feature extraction. We shall also consider corner detection, which can be thought of as detecting those points where lines bend very sharply with high curvature, as for the aeroplane in Figure 4.1(b) and (e). These are another low-level feature that again can be extracted automatically from the image. These are largely techniques for localized feature extraction, in this case the curvature, and the more modern approaches extend to the detection of localized regions or patches of interest. Finally, we shall investigate a technique that describes motion, called optical flow. This is illustrated in Figure 4.1(c) and (f) with the optical flow from images of a walking man: the bits that are moving fastest are the brightest points, like the hands and the feet. All of these can provide a set of points, albeit points with different properties, but all are suitable for grouping for shape extraction. Consider a square box moving through a sequence of images. The edges are the perimeter of the box; the corners are the apices; the flow is how the box moves. All these can be collected together to find the moving box. We shall start with the edge detection techniques, with the first order operators, which accords with the chronology of development. The first order techniques date back more than 30 years. 115
(a) Face image
(b) Plane silhouette
(c) Consecutive images of walking subject
(d) Edge detection
(e) Corner detection
(f) Motion detection
Figure 4.1 Low-level feature detection
Table 4.1 Overview of Chapter 4 Main topic
Sub topics
Main points
First order edge detection
What is an edge and how we detect it. The equivalence of operators to first order differentiation and the insight this brings. The need for filtering and more sophisticated first order operators.
Difference operation; Roberts Cross, smoothing, Prewitt, Sobel, Canny. Basis of the operators and frequency domain analysis.
Second order edge detection
Relationship between first and second order differencing operations. The basis of a second order operator. The need to include filtering and better operations.
Second order differencing; Laplacian, zero-crossing detection; Marr–Hildreth, Laplacian of Gaussian, difference of Gaussian. Scale space.
Other edge operators
Alternative approaches and performance aspects. Comparing different operators.
Other noise models; Spacek. Other edge models; Petrou.
Phase congruency
Inverse Fourier transform; phase for feature extraction. Alternative form of edge and feature detection
Frequency domain analysis; detecting a range of features; photometric invariance, wavelets. (Continued)
116
Feature Extraction and Image Processing
Table 4.1 (Continued) Main topic
Sub topics
Main points
Localized feature extraction
Finding localized low-level features; extension from curvature to patches. Nature of curvature and computation from: edge information; by change in intensity; and by correlation. Motivation of patch detection and principles of modern approaches.
Planar curvature; corners. Curvature estimation by: change in edge direction; intensity change; Harris corner detector. Modern feature detectors: SIFT operator and saliency.
Optical flow Estimation
Movement and the nature of optical flow. Estimating the optical flow by differential approach. Need for other approaches (including matching regions).
Detection by differencing. Optical flow; aperture problem; smoothness constraint. Differential approach; Horn and Schunk method; correlation.
4.2 First order edge detection operators 4.2.1 Basic operators Many approaches to image interpretation are based on edges, since analysis based on edge detection is insensitive to change in the overall illumination level. Edge detection highlights image contrast. Detecting contrast, which is difference in intensity, can emphasize the boundaries of features within an image, since this is where image contrast occurs. This is how human vision can perceive the perimeter of an object, since the object is of different intensity to its surroundings. Essentially, the boundary of an object is a step change in the intensity levels. The edge is at the position of the step change. To detect the edge position we can use first order differentiation, since this emphasizes change; first order differentiation gives no response when applied to signals that do not change. The first edge detection operators to be studied here are group operators which aim to deliver an output that approximates the result of first order differentiation. A change in intensity can be revealed by differencing adjacent points. Differencing horizontally adjacent points will detect vertical changes in intensity and is often called a horizontal edge-detector by virtue of its action. A horizontal operator will not show up horizontal changes in intensity since the difference is zero. (This is the form of edge detection used within the anisotropic diffusion smoothing operator in the previous chapter.) When applied to an image P the action of the horizontal edge-detector forms the difference between two horizontally adjacent points, as such detecting the vertical edges, Ex, as: Exxy = Pxy − Px+1y ∀x ∈ 1 N − 1 y ∈ 1 N (4.1) To detect horizontal edges we need a vertical edge-detector which differences vertically adjacent points. This will determine horizontal intensity changes, but not vertical ones, so the vertical edge-detector detects the horizontal edges, Ey, according to: Eyxy = Pxy − Pxy+1 ∀x ∈ 1 N y ∈ 1 N − 1 (4.2) Figure 4.2(b) and (c) show the application of the vertical and horizontal operators to the synthesized image of the square in Figure 4.2(a). The left-hand vertical edge in Figure 4.2(b) Low-level feature extraction (including edge detection)
117
(a) Original image
(b) Vertical edges, Equation 4.1
(c) Horizontal edges, Equation 4.2
(d) All edges, Equation 4.4
Figure 4.2 First order edge detection
appears to be beside the square by virtue of the forward differencing process. Likewise, the upper edge in Figure 4.2(c) appears above the original square. Combining the two gives an operator E that can detect vertical and horizontal edges together. That is, Exy = Pxy − Px+1y + Pxy − Pxy+1
∀x y ∈ 1 N − 1
(4.3)
which gives: Exy = 2 × Pxy − Px+1y − Pxy+1
∀x y ∈ 1 N − 1
(4.4)
Equation 4.4 gives the coefficients of a differencing template which can be convolved with an image to detect all the edge points, such as those shown in Figure 4.2(d). As in the previous chapter, the current point of operation (the position of the point we are computing a new value for) is shaded. The template shows only the weighting coefficients and not the modulus operation. Note that the bright point in the lower right corner of the edges of the square in Figure 4.2(d) is much brighter than the other points. This is because it is the only point to be detected as an edge by both the vertical and the horizontal operators and is therefore much brighter than the other edge points. In contrast, the top left-hand corner point is detected by neither operator and so does not appear in the final image. The template in Figure 4.3 is convolved with the image to detect edges. The direct implementation of this operator, i.e. using Equation 4.4 rather than template convolution, is given in Code 4.1. Template convolution could be used, but it is unnecessarily complex in this case. 118
Feature Extraction and Image Processing
2
–1
–1
0
Figure 4.3 Template for first order difference
edge(pic):= newpic←zero(pic) for x∈0.. cols(pic)–2 for y∈0.. rows(pic)–2 newpicy,x← 2.picy,x–picy,x+1–picy+1,x newpic Code 4.1 First order edge detection
Uniform thresholding (Section 3.3.4) is often used to select the brightest points, following application of an edge detection operator. The threshold level controls the number of selected points; too high a level can select too few points, whereas too low a level can select too much noise. Often, the threshold level is chosen by experience or by experiment, but it can be determined automatically by considering edge data (Venkatesh and Rosin, 1995), or empirically (Haddon, 1988). For the moment, let us concentrate on the development of edge detection operators, rather than on their application.
4.2.2 Analysis of the basic operators Taylor series analysis reveals that differencing adjacent points provides an estimate of the first order derivative at a point. If the difference is taken between points separated by x then by Taylor expansion for fx + x we obtain: x2 × f x + Ox3 2! By rearrangement, the first order derivative f x is: fx + x = fx + x × f x +
(4.5)
fx + x − fx − Ox (4.6) x This shows that the difference between adjacent points is an estimate of the first order derivative, with error Ox. This error depends on the size of the interval x and on the complexity of the curve. When x is large this error can be significant. The error is also large when the high-order derivatives take large values. In practice, the short sampling of image pixels and the reduced high-frequency content make this approximation adequate. However, the error can be reduced by spacing the differenced points by one pixel. This is equivalent to computing the first order difference delivered by Equation 4.1 at two adjacent points, as a new horizontal difference Exx, where f x =
Exxxy = Exx+1y + Exxy = Px+1y − Pxy + Pxy − Px−1y = Px+1y − Px−1y Low-level feature extraction (including edge detection)
(4.7) 119
This is equivalent to incorporating spacing to detect the edges Exx by: Exxxy = Px+1y − Px−1y ∀x ∈ 2 N − 1 y ∈ 1 N
(4.8)
To analyse this, again by Taylor series, we expand fx − x as: x2 × f x − Ox3 2! By differencing Equation 4.9 from Equation 4.5, we obtain the first order derivative as: fx − x = fx − x × f x +
(4.9)
fx + x − fx − x (4.10) − Ox2 2x Equation 4.10 suggests that the estimate of the first order difference is now the difference between points separated by one pixel, with error Ox2 . If x < 1, this error is clearly smaller than the error associated with differencing adjacent pixels, in Equation 4.6. Again, averaging has reduced noise, or error. The template for a horizontal edge detection operator is given in Figure 4.4(a). This template gives the vertical edges detected at its centre pixel. A transposed version of the template gives a vertical edge detection operator (Figure 4.4b). f x =
1
0
1 0 –1
–1
(a) Mx
(b) My
Figure 4.4 Templates for improved first order difference
The Roberts cross operator (Roberts, 1965) was one of the earliest edge detection operators. It implements a version of basic first order edge detection and uses two templates which difference pixel values in a diagonal manner, as opposed to along the axes’ directions. The two templates are called M + and M − and are given in Figure 4.5.
0 –1
+1 0 (a) M –
0 –1
+1 0
(b) M +
Figure 4.5 Templates for Roberts cross operator
In implementation, the maximum value delivered by application of these templates is stored as the value of the edge at that point. The edge point Exy is then the maximum of the two values derived by convolving the two templates at an image point Pxy : Exy = max M + ∗ Pxy M − ∗ Pxy ∀x y ∈ 1 N − 1 (4.11) 120
Feature Extraction and Image Processing
The application of the Roberts cross operator to the image of the square is shown in Figure 4.6. The two templates provide the results in Figure 4.6(a) and (b) and the result delivered by the Roberts operator is shown in Figure 4.6(c). Note that the corners of the square now appear in the edge image, by virtue of the diagonal differencing action, whereas they were less apparent in Figure 4.2(d) (where the top left corner did not appear).
(a) M –
(b) M +
(c) M
Figure 4.6 Applying the Roberts cross operator
An alternative to taking the maximum is simply to add the results of the two templates together to combine horizontal and vertical edges. There are of course more varieties of edges and it is often better to consider the two templates as providing components of an edge vector: the strength of the edge along the horizontal and vertical axes. These give components of a vector and can be added in a vectorial manner (which is perhaps more usual for the Roberts operator). The edge magnitude is the length of the vector, and the edge direction is the vector’s orientation, as shown in Figure 4.7.
M
My
θ Mx
Figure 4.7 Edge detection in vectorial format
4.2.3 Prewitt edge detection operator Edge detection is akin to differentiation. Since it detects change it is bound to respond to noise, as well as to step-like changes in image intensity (its frequency domain analogue is high-pass filtering, as illustrated in Figure 2.26c). It is therefore prudent to incorporate averaging within the edge detection process. We can then extend the vertical template, Mx, along three rows, Low-level feature extraction (including edge detection)
121
1 1 1
0 0 0
–1 –1 –1
1 0 –1
(a) Mx
1 0 –1
1 0 –1
(b) My
Figure 4.8 Templates for Prewitt operator
and the horizontal template, My, along three columns. These give the Prewitt edge detection operator (Prewitt and Mendelsohn, 1966), which consists of two templates (Figure 4.8). This gives two results: the rate of change of brightness along each axis. As such, this is the vector illustrated in Figure 4.7: the edge magnitude, M, is the length of the vector and the edge direction, , is the angle of the vector: Mx y =
Mxx y2 + Myx y2
x y = tan−1
Myx y Mxx y
(4.12)
(4.13)
Again, the signs of Mx and My can be used to determine the appropriate quadrant for the edge direction. A Mathcad implementation of the two templates of Figure 4.8 is given in Code 4.2. In this code, both templates operate on a 3 × 3 subpicture (which can be supplied, in Mathcad, using the submatrix function). Again, template convolution could be used to implement this operator, but (as with direct averaging and basic first order edge detection) it is less suited to simple templates. Also, the provision of edge magnitude and direction would require extension of the template convolution operator given earlier (Code 3.5).
2
Prewitt33_x(pic):= ∑ picy,0 – y=0
2
∑
2
picy,2
y=0
(a)Mx
Prewitt33_ y(pic):= ∑ pic0,x – x=0
2
∑
pic2,x
x=0
(b)My
Code 4.2 Implementing the Prewitt operator
When applied to the image of the square (Figure 4.9a), we obtain the edge magnitude and direction (Figure 4.9b and d, respectively, where part d does not include the border points, only the edge direction at processed points). The edge direction in Figure 4.9(d) is shown measured in degrees, where 0 and 360 are horizontal, to the right, and 90 is vertical, upwards. Although the regions of edge points are wider owing to the operator’s averaging properties, the edge data is clearer than the earlier first order operator, highlighting the regions where intensity changed in a more reliable fashion (compare, for example, the upper left corner of the square which was not revealed earlier). The direction is less clear in an image format and is better exposed by Mathcad’s vector format in Figure 4.9(c). In vector format, the edge direction data is clearly 122
Feature Extraction and Image Processing
(a) Original image
(b) Edge magnitude
dir =
313 331 3
3
24
47
298 315 1
2
42
63
273 276 13
43
88
88
269 268 199 117 91
92
242 225 181 178 133 116 225 210 183 179 155 132 prewitt_vec0, 1, prewitt_vec0, 0
(c) Vector format
(d) Edge direction
Figure 4.9 Applying the Prewitt operator
less well defined at the corners of the square (as expected, since the first order derivative is discontinuous at these points).
4.2.4 Sobel edge detection operator When the weight at the central pixels, for both Prewitt templates, is doubled, this gives the famous Sobel edge detection operator which, again, consists of two masks to determine the edge in vector form. The Sobel operator was the most popular edge detection operator until the development of edge detection techniques with a theoretical basis. It proved popular because it gave, overall, a better performance than other contemporaneous edge detection operators, such as the Prewitt operator. The templates for the Sobel operator can be found in Figure 4.10. The Mathcad implementation of these masks is very similar to the implementation of the Prewitt operator (Code 4.2), again operating on a 3 × 3 subpicture. This is the standard formulation of the Sobel templates, but how do we form larger templates, say for 5 × 5 or 7 × 7? Few textbooks state its original derivation, but it has been attributed (Heath et al., 1997) as originating from a PhD thesis (Sobel, 1970). Unfortunately, a theoretical basis that can be used to calculate the coefficients of larger templates is rarely given. One approach to a theoretical basis is to consider the optimal forms of averaging and of differencing. Gaussian averaging has already been stated to give optimal averaging. The binomial expansion gives the integer coefficients of a series that, in the limit, approximates the normal distribution. Pascal’s triangle Low-level feature extraction (including edge detection)
123
gives sets of coefficients for a smoothing operator which, in the limit, approach the coefficients of a Gaussian smoothing operator. Pascal’s triangle is then: Window size 2 3 4 5
1 1 1 1
1 2
3 4
1 3
1
6
4
1
This gives the (unnormalized) coefficients of an optimal discrete smoothing operator (it is essentially a Gaussian operator with integer coefficients). The rows give the coefficients for increasing template, or window, size. The coefficients of smoothing within the Sobel operator (Figure 4.10) are those for a window size of 3. In Mathcad, by specifying the size of the smoothing window as winsize, the template coefficients smoothx_win can be calculated at each window point x_win according to Code 4.3.
1 2 1
0 0 0
–1 –2 –1
1 0 –1
2 0 –2
(a) Mx
1 0 –1
(b) My
Figure 4.10 Templates for Sobel operator
smoothx_win:=
(winsize–1)! (winsize–1–x_win)!.x_win!
Code 4.3 Smoothing function
The differencing coefficients are given by Pascal’s triangle for subtraction: Window size 2 3 4 5
−1
1 1 1 1
0 −1
1 2
0
−1 −2
−1
−1
This can be implemented by subtracting the templates derived from two adjacent expansions for a smaller window size. Accordingly, we require an operator that can provide the coefficients of Pascal’s triangle for arguments which are a window size n and a position k. The operator is the Pascal(k,n) operator in Code 4.4. 124
Feature Extraction and Image Processing
Pascal(k,n):=
n! (n–k)!⋅k!
if(k≥0)⋅(k≤n)
0 otherwise Code 4.4 Pascal’s triangle
The differencing template, diffx_win , is then given by the difference between two Pascal expansions, as given in Code 4.5.
diffx_win:= Pascal(x_win, winsize–2)–Pascal(x_win–1, winsize–2) Code 4.5 Differencing function
These give the coefficients of optimal differencing and optimal smoothing. This general form of the Sobel operator combines optimal smoothing along one axis, with optimal differencing along the other. This general form of the Sobel operator is then given in Code 4.6, which combines the differencing function along one axis, with smoothing along the other.
Sobel_x(pic):=
winsize–1
winsize–1
x_win=0
y_win=0
∑
∑
smoothy_win.diffx_win.picy_win,x_win (a)Mx
winsize–1
winsize–1
x_win=0
y_win=0
∑
Sobel_ y(pic):=
∑
smoothx_win.diffy_win.picy_win,x_win (b)My
Code 4.6 Generalized Sobel templates
This generates a template for the Mx template for a Sobel operator, given for 5 × 5 in Code 4.7.
Sobel_template_x =
1 2
0 –2
–1
4 8
0 –8
–4
6 12 0 –12 –6 4 8
0 –8
–4
1 2
0 –2
–1
Code 4.7 5 × 5 Sobel template Mx
Low-level feature extraction (including edge detection)
125
All template-based techniques can be larger than 5 × 5 so, as with any group operator, there is a 7 × 7 Sobel, and so on. The virtue of a larger edge detection template is that it involves more smoothing to reduce noise, but edge blurring becomes a great problem. The estimate of edge direction can be improved with more smoothing since it is particularly sensitive to noise. There are circular edge operators designed specifically to provide accurate edge direction data. The Sobel templates can be invoked by operating on a matrix of dimension equal to the window size, from which edge magnitude and gradient are calculated. The Sobel function (Code 4.8) convolves the generalized Sobel template (of size chosen to be winsize) with the picture supplied as argument, to give outputs which are the images of edge magnitude and direction, in vector form.
Sobel(pic,winsize):= winsize⎞ w2←floor ⎛⎜ ⎟ ⎝ ⎠ 2 edge_mag←zero(pic) edge_dir←zero(pic) for x∈w2.. cols(pic)–1–w2 for y∈w2.. rows(pic)–1–w2 x_mag←Sobel_x(submatrix(pic,y–w2,y+w2,x–w2,x+w2)) y_mag←Sobel_y(submatrix(pic,y–w2,y+w2,x–w2,x+w2)) ⎛magnitude(x_mag,y_mag)⎞ ⎟ edge_magy,x←floor ⎜⎝ ⎠ mag_normalise edge_diry,x←direction(x_mag,y_mag) (edge_mag edge_dir) Code 4.8 Generalized Sobel operator
The results of applying the 3 × 3 Sobel operator can be seen in Figure 4.11. The original face image (Figure 4.11a) has many edges in the hair and in the region of the eyes. This is shown in the edge magnitude image (Figure 4.11b). When this is thresholded at a suitable value, many edge points are found (Figure 4.11c). Note that in areas of the image where the brightness
(a) Original image
(b) Sobel edge magnitude
Figure 4.11 Applying the Sobel operator
126
Feature Extraction and Image Processing
(c) Thresholded magnitude
remains fairly constant, such as the cheek and shoulder, there is little change, which is reflected by low edge magnitude and few points in the thresholded data. The Sobel edge direction data can be arranged to point in different ways, as can the direction provided by the Prewitt operator. If the templates are inverted to be of the form shown in Figure 4.12, the edge direction will be inverted around both axes. If only one of the templates is inverted, the measured edge direction will be inverted around the chosen axis.
–1 –2 –1
0 0 0
1 2 1
(a) –Mx
–1 0 1
–2 0 2
–1 0 1
(b) –My
Figure 4.12 Inverted templates for Sobel operator
This gives four possible directions for measurement of the edge direction provided by the Sobel operator, two of which (for the templates of Figures 4.10 and 4.12) are illustrated in Figure 4.13(a) and (b), respectively, where inverting the Mx template does not highlight
sobel_vec0,0, sobel_vec0,1
(a) Mx, My
sobel_vec0,0T, sobel_vec0,1T
(c) My, Mx
–sobel_vec0,0, sobel_vec0,1
(b) –Mx, My
–sobel_vec0,0T, sobel_vec0,1T
(d) – My, – Mx
Figure 4.13 Alternative arrangements of edge direction
Low-level feature extraction (including edge detection)
127
discontinuity at the corners. (The edge magnitude of the Sobel applied to the square is not shown, but is similar to that derived by application of the Prewitt operator; Figure 4.9b). By swapping the Sobel templates, the measured edge direction can be arranged to be normal to the edge itself (as opposed to tangential data along the edge). This is illustrated in Figure 4.13(c) and (d) for swapped versions of the templates given in Figures 4.10 and 4.12, respectively. The rearrangement can lead to simplicity in algorithm construction when finding shapes, as to be shown later. Any algorithm that uses edge direction for finding shapes must know precisely which arrangement has been used, since the edge direction can be used to speed algorithm performance, but it must map precisely to the expected image data if used in that way. Detecting edges by template convolution again has a frequency domain interpretation. The magnitude of the Fourier transform of a 5 × 5 Sobel template of Code 4.7 is given in Figure 4.14. The Fourier transform is given in relief in Figure 4.14(a) and as a contour plot in Figure 4.14(b). The template is for horizontal differencing action, My, which highlights vertical change. Accordingly, its transform reveals that it selects vertical spatial frequencies, while smoothing the horizontal ones. The horizontal frequencies are selected from a region near the origin (low-pass filtering) , whereas the vertical frequencies are selected away from the origin (high-pass) . This highlights the action of the Sobel operator; combining smoothing of the spatial frequencies along one axis with differencing of the other. In Figure 4.14, the smoothing is of horizontal spatial frequencies, while the differencing is of vertical spatial frequencies.
0
2
4
6
0 2 4 6
⎛ ⎜ ⎝
Fourier_of_Sobel
⎞T ⎟ ⎠
(a) Relief plot
Fourier_of_Sobel
(b) Contour plot
Figure 4.14 Fourier transform of the Sobel operator
An alternative frequency domain analysis of the Sobel can be derived via the z-transform operator. This is more the domain of signal processing courses in electronic and electrical engineering, and is included here for completeness and for linkage with signal processing. Essentially, z−1 is a unit time-step delay operator, so z can be thought of a unit (time-step) advance, so ft − = z−1 ft and ft + = zft, where is the sampling interval. Given that we have two spatial axes x and y, we can express the Sobel operator of Figure 4.12(a) using delay and advance via the z-transform notation along the two axes as −1 −1 −z−1 x zy + 0 + zx zy −1 Sx y = −2zx + 0 + 2zx −z−1 x z y + 0 + z x zy
128
Feature Extraction and Image Processing
(4.14)
including zeros for the null template elements. Given that there is a standard substitution (by conformal mapping, evaluated along the frequency axis) z−1 = e−j t to transform from the time z domain to the frequency domain , we have Sobel x y = −e−j x t e−j y t + ej x t e−j y t − 2e−j x t + 2ej x t − e−j x t ej y t + ej x t ej y t = e−j y t + 2 + ej y t −e−j x t + ej x t
−j y t j y t 2 −e−j x t + ej x t = e 2 +e 2 = 8j cos2
y t 2
(4.15)
sin x t
where the transform Sobel is a function of spatial frequency, x y , along the x and the y axes. This conforms rather well to the separation between smoothing along one axis (the first part of Equation 4.15) and differencing along the other; here by differencing (high-pass) along the x-axis and averaging (low-pass) along the y-axis. This provides an analytic form of the function shown in Figure 4.14; the relationship between the discrete Fourier transform (DFT), and this approach is evident by applying the DFT relationship (Equation 2.15) to the components of the Sobel operator.
4.2.5 Canny edge detection operator The Canny edge detection operator (Canny, 1986) is perhaps the most popular edge detection technique at present. It was formulated with three main objectives: • optimal detection with no spurious responses • good localization with minimal distance between detected and true edge position • single response to eliminate multiple responses to a single edge.
The first requirement aims to reduce the response to noise. This can be effected by optimal smoothing; Canny was the first to demonstrate that Gaussian filtering is optimal for edge detection (within his criteria). The second criterion aims for accuracy: edges are to be detected, in the right place. This can be achieved by a process of non-maximum suppression (which is equivalent to peak detection). Non-maximum suppression retains only those points at the top of a ridge of edge data, while suppressing all others. This results in thinning: the output of nonmaximum suppression is thin lines of edge points, in the right place. The third constraint concerns location of a single edge point in response to a change in brightness. This is because more than one edge can be denoted to be present, consistent with the output obtained by earlier edge operators. Canny showed that the Gaussian operator was optimal for image smoothing. Recalling that the Gaussian operator gx y is given by: g x y = e
− x2 +y2
(4.16)
2 2
Low-level feature extraction (including edge detection)
129
by differentiation, for unit vectors Ux = 1 0 and Uy = 0 1 along the coordinate axes, we obtain:
g x y =
g x y g x y Ux + Uy x y (4.17)
2 2 2 2 x −x +y y −x +y = − 2 e 2 2 Ux − 2 e 2 2 Uy
Equation 4.17 gives a way to calculate the coefficients of a derivative of Gaussian template that combines first order differentiation with Gaussian smoothing. This is a smoothed image, and so the edge will be a ridge of data. To mark an edge at the correct point (and to reduce multiple response), we can convolve an image with an operator which gives the first derivative in a direction normal to the edge. The maximum of this function should be the peak of the edge data, where the gradient in the original image is sharpest, and hence the location of the edge. Accordingly, we seek an operator, Gn , which is a first derivative of a Gaussian function g in the direction of the normal, n⊥ : Gn =
g n⊥
(4.18)
where n⊥ can be estimated from the first order derivative of the Gaussian function g convolved with the image P, and scaled appropriately as: n⊥ =
P ∗ g P ∗ g
(4.19)
The location of the true edge point is then at the maximum point of Gn convolved with the image. This maximum is when the differential (along n⊥ ) is zero: Gn ∗ P =0 n⊥
(4.20)
By substitution of Equation 4.18 in Equation 4.20, 2 G ∗ P =0 n⊥ 2
(4.21)
Equation 4.21 provides the basis for an operator which meets one of Canny’s criteria, namely that edges should be detected in the correct place. This is non-maximum suppression, which is equivalent to retaining peaks (and thus equivalent to differentiation perpendicular to the edge), which thins the response of the edge detection operator to give edge points that are in the right place, without multiple response and with minimal response to noise. However, it is virtually impossible to achieve an exact implementation of Canny given the requirement to estimate the normal direction. A common approximation is, as illustrated in Figure 4.15: 1. 2. 3. 4. 130
Use Gaussian smoothing (as in Section 3.4.4) (Figure 4.15a). Use the Sobel operator (Figure 4.15b). Use non-maximal suppression (Figure 4.15c). Threshold with hysteresis to connect edge points (Figure 4.15d). Feature Extraction and Image Processing
(a) Gaussian smoothing
(b) Sobel edge detection
(c) Non-maximum suppression
(d) Hysteresis thresholding
Figure 4.15 Stages in Canny edge detection
Note that the first two stages can be combined using a version of Equation 4.17, but are separated here so that all stages in the edge detection process can be shown clearly. An alternative implementation of Canny’s approach (Deriche, 1987) used Canny’s criteria to develop twodimensional (2D) recursive filters, claiming performance and implementation advantage over the approximation here. Non-maximum suppression essentially locates the highest points in the edge magnitude data. This is performed by using edge direction information, to check that points are at the peak of a ridge. Given a 3 × 3 region, a point is at a maximum if the gradient at either side of it is less than the gradient at the point. This implies that we need values of gradient along a line that is normal to the edge at a point. This is illustrated in Figure 4.16, which shows the neighbouring points to the point of interest, Pxy , the edge direction at Pxy and the normal to the edge direction at Pxy . The point Pxy is to be marked as maximum if its gradient, Mx y, exceeds the gradient at points 1 and 2, M1 and M2 , respectively. Since we have a discrete neighbourhood, M1 and
Edge direction at Px,y
Px ,y –1
Px –1,y –1
Px +1,y –1 M1
Px –1,y
Px,y
My
Px +1,y
Mx
Px,y +1
Px –1,y +1 Normal to edge direction
Px +1,y +1
M2
Figure 4.16 Interpolation in non-maximum suppression
Low-level feature extraction (including edge detection)
131
M2 need to be interpolated, First order interpolation using Mx and My at Pxy , and the values of Mx and My for the neighbours gives: M1 =
Mx − My My Mx + 1 y − 1 + Mx y − 1 Mx Mx
(4.22)
and Mx − My My Mx − 1 y + 1 + Mx y + 1 (4.23) Mx Mx The point Pxy is then marked as a maximum if Mx y exceeds both M1 and M2 , otherwise it is set to zero. In this manner the peaks of the ridges of edge magnitude data are retained, while those not at the peak are set to zero. The implementation of non-maximum suppression first requires a function that generates the coordinates of the points between which the edge magnitude is interpolated. This is the function get_coords in Code 4.9, which requires the angle of the normal to the edge direction, returning the coordinates of the points beyond and behind the normal. M2 =
get_coords(angle):= δ←0.000000000000001 ⎛ ⎛ π⎞ x1←ceil ⎜⎝cos ⎜⎝ angle+ ⎟⎠ . 8 ⎛ ⎛ π⎞ y1←ceil ⎜⎝–sin ⎜⎝ angle– ⎟⎠ . 8 ⎛ ⎛ π⎞ x2←ceil ⎜⎝cos ⎜⎝ angle– ⎟⎠ . 8 ⎛ ⎛ π⎞ y2←ceil ⎜⎝–sin ⎜⎝ angle– ⎟⎠ . 8
⎞
2⎟⎠ –0.5–δ ⎞
2⎟⎠ –0.5–δ ⎞
2⎟⎠ –0.5–δ ⎞
2⎟⎠–0.5–δ
(x1 y1 x2 y2)
Code 4.9 Generating coordinates for interpolation
The non-maximum suppression operator, non_max in Code 4.10, then interpolates the edge magnitude at the two points either side of the normal to the edge direction. If the edge magnitude at the point of interest exceeds these two then it is retained, otherwise it is discarded. Note that the potential singularity in Equations 4.22 and 4.23 can be avoided by use of multiplication in the magnitude comparison, as opposed to division in interpolation, as it is in Code 4.10. In practice, however, this implementation, Codes 4.9 and 4.10, can suffer from numerical imprecision and ill-conditioning. Accordingly, it is better to implement a hand-crafted interpretation of Equations 4.22 and 4.23 applied separately to the four quadrants. This is too lengthy to be included here, but a version is included with the Worksheets for Chapter 4. The transfer function associated with hysteresis thresholding is shown in Figure 4.17. Points are set to white once the upper threshold is exceeded and set to black when the lower threshold is reached. The arrows reflect possible movement: there is only one way to change from black to white and vice versa. 132
Feature Extraction and Image Processing
non_max(edges):=
for i∈1..cols(edges0,0)–2 for j∈1..rows(edges0,0)–2 Mx←(edges0,0)j,i My← (edges0,1)j,i ⎛ Mx ⎞ o←atan ⎜ ⎟ if My ≠ 0 ⎝ My ⎠ ⎛o← π ⎞ if(My=0).(Mx>0) ⎜ ⎟ ⎝ 2⎠
–π
otherwise 2 adds←get_coords(o)
o←
M1← My.(edges0,2)j+adds0,1,i+adds0,0... +(Mx–My).(edges0,2)j+adds0,3,i+adds0,2 adds←get_coords(o+π) M2← My.(edges0,2)j+adds0,1,i+adds0,0... +(Mx–My).(edges0,2)j+adds0,3,i+adds0,2 Mx.(edges0,2)j,i>M1
isbigger← +
. Mx.(edges0,2) ≥M2 j,i
Mx.(edges0,2)j,i<M1
...
. Mx.(edges0,2) ≤M2 j,i
new_edgej,i←(edges0,2)j,i if isbigger new_edgej,i←0 otherwise new_edge
Code 4.10 Non-maximum suppression
Thresholded data White
Black
Brightness
Upper switching threshold Lower switching threshold
Figure 4.17 Hysteresis thresholding transfer function
The application of non-maximum suppression and hysteresis thresholding is illustrated in Figure 4.18. This contains a ridge of edge data, the edge magnitude. The action of non-maximum suppression is to select the points along the top of the ridge. Given that the top of the ridge initially exceeds the upper threshold, the thresholded output is set to white until the peak of Low-level feature extraction (including edge detection)
133
Hysteresis thresholded edge data
Upper switching threshold Lower switching threshold
Non-maximum suppression
Figure 4.18 Action of non-maximum suppression and hysteresis thresholding
the ridge falls beneath the lower threshold. The thresholded output is then set to black until the peak of the ridge exceeds the upper switching threshold. Hysteresis thresholding requires two thresholds, an upper and a lower threshold. The process starts when an edge point from non-maximum suppression is found to exceed the upper threshold. This is labelled as an edge point (usually white, with a value of 255) and forms the first point of a line of edge points. The neighbours of the point are then searched to determine whether or not they exceed the lower threshold, as in Figure 4.19. Any neighbour that exceeds the lower threshold is labelled as an edge point and its neighbours are then searched to determine whether or not they exceed the lower threshold. In this manner, the first edge point found (the one that exceeded the upper threshold) becomes a seed point for a search. Its neighbours, in turn, become seed points if they exceed the lower threshold, and so the search extends, along branches arising from neighbours that exceeded the lower threshold. For each branch, the search terminates at points that have no neighbours above the lower threshold.
≥ lower ≥ lower ≥ lower
≥ lower seed ≥ upper ≥ lower
≥ lower ≥ lower ≥ lower
Figure 4.19 Neighbourhood search for hysteresis thresholding
In implementation, hysteresis thresholding clearly requires recursion, since the length of any branch is unknown. Having found the initial seed point, the seed point is set to white and its neighbours are searched. The coordinates of each point are checked to see whether it is within the picture size, according to the operator check, given in Code 4.11.
check(xc,yc,pic):= 1 if (xc≥1)⋅(xc≤cols(pic)–2)⋅(yc≥1)⋅(yc≤rows(pic)–2) 0 otherwise Code 4.11 Checking points are within an image
134
Feature Extraction and Image Processing
The neighbourhood (as in Figure 4.19) is then searched by a function connect (Code 4.12) which is fed with the non-maximum suppressed edge image, the coordinates of the seed point whose connectivity is under analysis and the lower switching threshold. Each of the neighbours is searched if its value exceeds the lower threshold, and the point has not already been labelled as white (otherwise the function would become an infinite loop). If both conditions are satisfied (and the point is within the picture) then the point is set to white and becomes a seed point for further analysis. This implementation tries to check the seed point as well, even though it has already been set to white. The operator could be arranged not to check the current seed point, by direct calculation without the for loops, and this would be marginally faster. Including an extra Boolean constraint to inhibit check of the seed point would only slow the operation. The connect routine is recursive: it is called again by the new seed point.
connect(x,y,nedg,low):= for x1∈x−1.. x+1 for y1∈y−1.. y+1 if(nedgy1,x1≥low)⋅(nedgy1,x1≠255)⋅check (x1,y1,nedg) nedgy1,x1←255 nedg←connect(x1,y1,nedg,low) nedg Code 4.12 Connectivity analysis after seed point location
The process starts with the point that exceeds the upper threshold. When such a point is found, it is set to white and it becomes a seed point where connectivity analysis starts. The calling operator for the connectivity analysis, hyst_thr, which starts the whole process, is given in Code 4.13. When hyst_thr is invoked, its arguments are the coordinates of the point of current interest, the non-maximum suppressed edge image, n_edg (which is eventually delivered as the hysteresis thresholded image), and the upper and lower switching thresholds, upp and low, respectively. For display purposes, this operator requires a later operation to remove points which have not been set to white (to remove those points which are below the upper threshold and which are not connected to points above the lower threshold). This is rarely used in application since the points set to white are the only ones of interest in later processing.
hyst_thr(n_edg,upp,low):= for x∈1.. cols(n_edg)–2 for y∈1.. rows(n_edg)–2 if[(n_edgy,x ≥ upp)·(n_edgy,x ≠255)] n_edgy,x ←255 n_edg←connect(x,y,n_edg,low) n_edg Code 4.13 Hysteresis thresholding operator
Low-level feature extraction (including edge detection)
135
A comparison with the results of uniform thresholding is shown in Figure 4.20. Figure 4.20(a) shows the result of hysteresis thresholding of a Sobel edge detected image of the eye with an upper threshold set to 40 pixels, and a lower threshold of 10 pixels. Figure 4.20(b) and (c) show the result of uniform thresholding applied to the image with thresholds of 40 pixels and 10 pixels, respectively. Uniform thresholding can select too few points if the threshold is too high, and too many if it is too low. Hysteresis thresholding selects all the points in Figure 4.20(b), and some of those in Figure 4.20(c), those connected to the points in (b). In particular, part of the nose is partly present in Figure 4.20(a), whereas it is absent in Figure 4.20(b) and masked by too many edge points in Figure 4.20(c). Also, the eyebrow is more complete in (a), whereas it is only partial in (b) and complete (but obscured) in (c). Hysteresis thresholding therefore has an ability to detect major features of interest in the edge image, in an improved manner to uniform thresholding.
(a) Hysteresis thresholding, upper level = 40, lower level = 10
(b) Uniform thresholding, level = 40
(c) Uniform thresholding, level = 10
Figure 4.20 Comparing hysteresis thresholding with uniform thresholding
The action of the Canny operator on a larger image is shown in Figure 4.21, in comparison with the result of the Sobel operator. Figure 4.21(a) is the original image of a face, Figure 4.21(b) is the result of the Canny operator (using a 5 × 5 Gaussian operator with = 10 and with upper and lower thresholds set appropriately) and Figure 4.21(c) is the result of a 3 × 3 Sobel operator with uniform thresholding. The retention of major detail by the Canny operator is very clear; the face is virtually recognizable in Figure 4.21(b), whereas it is less clear in Figure 4.21(c).
(a) Original image
(b) Canny
Figure 4.21 Comparing Canny with Sobel
136
Feature Extraction and Image Processing
(c) Sobel
4.3 Second order edge detection operators 4.3.1 Motivation First order edge detection is based on the premise that differentiation highlights change; image intensity changes in the region of a feature boundary. The process is illustrated in Figure 4.22, where Figure 4.22(a) is a cross-section through image data. The result of first order edge detection, f x = df dx in Figure 4.22(b), is a peak where the rate of change of the original signal, fx in Figure 4.22(a), is greatest. There are higher order derivatives; applied to the same cross-section of data, the second order derivative, f x = d2 f dx2 in Figure 4.22(c), is greatest where the rate of change of the signal is greatest and zero when the rate of change is constant. The rate of change is constant at the peak of the first order derivative. This is where there is a zero-crossing in the second order derivative, where it changes sign. Accordingly, an alternative to first order differentiation is to apply second order differentiation and then find zero-crossings in the second order information.
2
2
1 d f (x ) dx
f (x ) 0
2
4
6
1
–1 –2
0
2
4
6
x
x
(a) Cross-section through image data
(b) First order edge detection
1
d2 dx 2
f (x )
0
2
4
6
–1 x
(c) Second order edge detection
Figure 4.22 First and second order edge detection
4.3.2 Basic operators: the Laplacian The Laplacian operator is a template which implements second order differencing. The second order differential can be approximated by the difference between two adjacent first order differences: f x f x − f x + 1 Low-level feature extraction (including edge detection)
(4.24) 137
which, by Equation 4.6, gives f x −fx + 2fx + 1 − fx + 2
(4.25)
This gives a horizontal second order template, as given in Figure 4.23.
–1
2
–1
Figure 4.23 Horizontal second order template
When the horizontal second order operator is combined with a vertical second order difference we obtain the full Laplacian template, given in Figure 4.24. Essentially, this computes the difference between a point and the average of its four direct neighbours. This was the operator used earlier in anisotropic diffusion (Section 3.5.4), where it is an approximate solution to the heat equation.
0 –1 0
–1 4 –1
0 –1 0
Figure 4.24 Laplacian edge detection operator
Application of the Laplacian operator to the image of the square is given in Figure 4.25. The original image is provided in numeric form in Figure 4.25(a). The detected edges are the zero-crossings in Figure 4.25(b) and can be seen to lie between the edge of the square and its background.
p=
1 2 3
4
1
1
2 1
0
0
2 2 3
0
1
2
2 1
0
1
3 0 38 39 37 36 3 0
0
4 1 40 44 41 42 2 1
–
0 –
31
47
44 70
37
0
–
42 34
12
0
–
37 47
8
2 0 39 41 42 40 2 0
0
–
45 72
37
1 1
0
1 2 43 44 40 39 1 3 1 2 0
2
2
3
0 2 1 3 1 0 4 2 (a) Image data
L=
0 –
36
–
0
0
0
32
0
0
31
60
–
28 0 39 0
1
50
–
6
33
–
42 0
45
74
–
34 0
31
–
6
0
0 0 0 0 0 0 0 (b) After Laplacian operator
0
5
–
44
Figure 4.25 Edge detection via the Laplacian operator
138
0 –
Feature Extraction and Image Processing
–
38
–
–
40
–
An alternative structure to the template in Figure 4.24 is one where the central weighting is 8 and the neighbours are all weighted as −1. This includes a different form of image information, so the effects are slightly different. (Essentially, this now computes the difference between a pixel and the average of its neighbouring points, including the corners.) In both structures, the central weighting can be negative and that of the four or the eight neighbours can be positive, without loss of generality. It is important to ensure that the sum of template coefficients is zero, so that edges are not detected in areas of uniform brightness. One advantage of the Laplacian operator is that it is isotropic (like the Gaussian operator): it has the same properties in each direction. However, as yet it contains no smoothing and will again respond to noise, more so than a first order operator since it is differentiation of a higher order. As such, the Laplacian operator is rarely used in its basic form. Smoothing can use the averaging operator described earlier, but a more optimal form is Gaussian smoothing. When this is incorporated with the Laplacian we obtain a Laplacian of Gaussian (LoG) operator, which is the basis of the Marr–Hildreth approach, to be considered next. A clear disadvantage with the Laplacian operator is that edge direction is not available. It does, however, impose low computational cost, which is its main advantage. Although interest in the Laplacian operator abated with rising interest in the Marr–Hildreth approach, a non-linear Laplacian operator was developed (Vliet and Young, 1989) and shown to have good performance, especially in low-noise situations.
4.3.3 Marr–Hildreth operator The Marr–Hildreth approach (Marr and Hildreth, 1980) again uses Gaussian filtering. In principle, we require an image which is the second differential 2 of a Gaussian operator gx y convolved with an image P. This convolution process can be separated as:
2 gx y ∗ P = 2 gx y ∗ P
(4.26)
Accordingly, we need to compute a template for 2 gx y and convolve this with the image. By further differentiation of Equation 4.17, we achieve a LoG operator:
2 g x y = =
2 g x y 2 g x y U + Uy x x2 y2 g x y g x y Ux + Uy x y
2 2 2 2 −x +y2 2 −x +y2 2
2
x e y e = −1 + −1
2
2
2
2
−x2 +y2 x2 + y 2 1 2 2 = 2 − 2 e
2
(4.27)
2
This is the basis of the Marr–Hildreth operator. Equation 4.27 can be used to calculate the coefficients of a template which, when convolved with an image, combines Gaussian smoothing with second order differentiation. The operator is sometimes called a ‘Mexican hat’ operator, since its surface plot is the shape of a sombrero, as illustrated in Figure 4.26. Low-level feature extraction (including edge detection)
139
LoG (4,31)
Figure 4.26 Shape of Laplacian of Gaussian operator
The calculation of the Laplacian of Gaussian can be approximated by the difference of Gaussian, where the difference is formed from the result of convolving two Gaussian filters with differing variance (Marr, 1982; Lindeberg, 1994).
2 g x y =
g x y k − g x y g ≈
k −
(4.28)
where gx y is the Gaussian function and k is a constant. Although similarly named, the derivative of Gaussian (Equation 4.17) is a first order operator including Gaussian smoothing,
gx y. It does seem counter-intuitive that the difference of two smoothing operators should lead to second order edge detection. The approximation is illustrated in Figure 4.27, where in one dimension, two Gaussian distributions of different variance are subtracted to form a onedimensional (1D) operator whose cross-section is equivalent to the shape of the LoG operator (a cross-section of Figure 4.26).
⎛x − 50⎞ −⎜ 8 ⎟ ⎠
2
e⎝
⎛x − 50⎞ −⎜ ⎟ ⎝ 8 ⎠
⎛x − 50⎞ 2 −⎜ ⎟ ⎝ 10 ⎠
e
⎛x − 50⎞ −⎜ ⎟ ⎝ 10 ⎠
2
2
− 0.8⋅e
0
0.8⋅e
50
100
x 0
50
100
(a) Two Gaussian distributions
(b) After differencing
Figure 4.27 Approximating the Laplacian of Gaussian by difference of Gaussian
140
Feature Extraction and Image Processing
The implementation of Equation 4.27 to calculate template coefficients for the LoG operator is given in Code 4.14. The function includes a normalization function which ensures that the sum of the template coefficients is unity, so that edges are not detected in area of uniform brightness. This is in contrast with the earlier Laplacian operator (where the template coefficients summed to zero), since the LoG operator includes smoothing within the differencing action, whereas the Laplacian is pure differencing. The template generated by this function can then be used within template convolution. The Gaussian operator again suppresses the influence of points away from the centre of the template, basing differentiation on those points nearer the centre; the standard deviation, , is chosen to ensure this action. Again, it is isotropic consistent with Gaussian smoothing.
size–1 2 size–1 cy ← 2 for x∈0.. size–1 for y∈0.. size–1
LoG(σ,size):= cx ←
nx ← x–cx ny ← y–cy ⎛ nx2+ny2 ⎞ ⎟ 2 ⎠ 2.σ
–⎜ ⎝ nx2+ny2 ⎞ templatey,x ← 1 .⎛⎜ – 2⎟ .e σ2 ⎝
σ2
⎠
template ←normalize(template) template
Code 4.14 Implementation of the Laplacian of Gaussian operator
Determining the zero-crossing points is a major difficulty with this approach. A variety of techniques can be used, including manual determination of zero-crossing or a least squares fit of a plane to local image data, which is followed by determination of the point at which the plane crosses zero, if it does. The former is too simplistic, whereas the latter is quite complex (see Section 11.2, Appendix 3). The approach here is much simpler: given a local 3 × 3 area of an image, this is split into quadrants. These are shown in Figure 4.28, where each quadrant contains the centre pixel. The first quadrant contains the four points in the upper left corner and the third quadrant contains the four points in the upper right. If the average of the points in any quadrant differs in sign from the average in any other quadrant, there must be a zero-crossing at the centre point. In zerox, Code 4.15, the average intensity in each quadrant is then evaluated, giving four values and int0, int1, int2 and int3. If the maximum value of these points is positive, and the minimum value is negative, there must be a zero-crossing within the neighbourhood. If one exists, the output image at that point is marked as white, otherwise it is set to black. Low-level feature extraction (including edge detection)
141
1
•
•
•
•
•
•
•
•
•
3
4
2
Figure 4.28 Regions for zero crossing detection
zerox(pic):= newpic←zero(pic) for x∈1.. cols(pic)–2 for y∈1.. rows(pic)–2
x
Σ x1=x–1
int1← int2← int3←
y
x
Σ x1=x–1
int0←
x+1
Σ x1=x
Σ
y+1
Σ
y1=y
picy1,x1
y
Σ picy1,x1 y1=y–1
x+1
y+1
x1=x
y1=y
Σ
picy1,x1
y1= y–1
Σ
picy1,x1
maxval←max(int) minval←min(int) newpicy,x ←255 if (maxval>0)⋅(minval<0) newpic
Code 4.15 Zero crossing detector
The action of the Marr–Hildreth operator is given in Figure 4.29, applied to the face image in Figure 4.21(a). The output of the LoG operator is hard to interpret visually and is not shown here (remember that it is the zero-crossings which mark the edge points and it is hard to see them). The detected zero-crossings (for a 3 × 3 neighbourhood) are shown in Figure 4.29(b) and (c) for LoG operators of size and variance 11 × 11 with = 08 and 15 × 15 with = 18, respectively. These show that the selection of window size and variance can be used to provide edges at differing scales. Some of the smaller regions in Figure 4.29(b) join to form larger regions in Figure 4.29(c). Note that one virtue of the Marr–Hildreth operator is its ability to provide closed edge borders, which the Canny operator cannot. Another virtue is that it avoids the recursion associated with hysteresis thresholding that can require a massive stack size for large images. The Fourier transform of a LoG operator is shown in relief in Figure 4.30(a) and as a contour plot in Figure 4.30(b). The transform is circular–symmetric, as expected. Since the transform reveals that the LoG operator omits low and high frequencies (those close to the origin, and 142
Feature Extraction and Image Processing
(b) 11 × 11 LoG
(a) Face image
(c) 15 × 15 LoG
Figure 4.29 Marr–Hildreth edge detection
0 5
10
15
0 5 10 15
Fourier_of_LoG (a) Relief plot
Fourier_of_LoG (b) Contour plot
Figure 4.30 Fourier transform of LoG operator
those far away from the origin) it is equivalent to a band-pass filter. Choice of the value of
controls the spread of the operator in the spatial domain and the ‘width’ of the band in the frequency domain: setting to a high value gives low-pass filtering, as expected. This differs from first order edge detection templates which offer a high-pass (differencing) filter along one axis with a low-pass (smoothing) action along the other axis. The Marr–Hildreth operator has stimulated much attention, perhaps in part because it has an appealing relationship to human vision, and its ability for multiresolution analysis (the ability to detect edges at differing scales). In fact, it has been suggested that the original image can be reconstructed from the zero-crossings at different scales. One early study (Haralick, 1984) concluded that the Marr–Hildreth could give good performance. Unfortunately, the implementation appeared to be different from the original LoG operator (and has appeared in some texts in this form), as noted by one of the Marr–Hildreth study’s originators (Grimson and Hildreth, 1985). This lead to a somewhat spirited reply (Haralick, 1985) clarifying concern, but also Low-level feature extraction (including edge detection)
143
raising issues about the nature and operation of edge detection schemes which remain relevant today. Given the requirement for convolution of large templates, attention quickly focused on frequency domain implementation (Huertas and Medioni, 1986), and speed improvement was later considered in some detail (Forshaw, 1988). Later, schemes were developed to refine the edges produced via the LoG approach (Ulupinar and Medioni, 1990). Although speed and accuracy are major concerns with the Marr–Hildreth approach, it is also possible for zero-crossing detectors to mark as edge points ones that have no significant contrast, motivating study of their authentication (Clark, 1989). Gunn (1999) studied the relationship between mask size of the LoG operator and its error rate. Essentially, an acceptable error rate defines a truncation error, which in turn gives an appropriate mask size. Gunn also observed the paucity of studies on zero-crossing detection and offered a detector slightly more sophisticated than the one here (as it includes the case where a zero-crossing occurs at a boundary, whereas the one here assumes that the zero-crossing can only occur at the centre). The similarity is not coincidental: Mark developed the one here after conversations with Steve Gunn, who he works with!
4.4 Other edge detection operators There have been many approaches to edge detection. This is not surprising, since it is often the first stage in a vision process. The most popular operators are the Sobel, Canny and Marr– Hildreth operators. Clearly, in any implementation, there is a compromise between (computational) cost and efficiency. In some cases, it is difficult to justify the extra complexity associated with the Canny and the Marr–Hildreth operators. This is in part due to the images: few images contain the adverse noisy situations that complex edge operators are designed to handle. Also, when finding shapes, it is often prudent to extract more than enough low-level information, and to let the more sophisticated shape detection process use, or discard, the information as appropriate. For these reasons, we will study only two more edge detection approaches, and only briefly. These operators are the Spacek and the Petrou operators: both are designed to be optimal and both have different properties and a different basis (the smoothing functional in particular) to the Canny and Marr–Hildreth approaches. The Spacek and Petrou Operators will be reviewed briefly, by virtue of their optimality. Essentially, while Canny maximized the ratio of the signal-to-noise ratio with the localization, Spacek (1986) maximized the ratio of the product of the signal-to-noise ratio and the peak separation with the localization. In Spacek’s work, since the edge was again modelled as a step function, the ideal filter appeared to be of the same form as Canny’s. Spacek’s operator can give better performance than Canny’s formulation (Jia and Nixon, 1985), as such challenging the optimality of the Gaussian operator for noise smoothing (in step edge detection), although such advantage should be explored in application. Petrou questioned the validity of the step-edge model for real images (Petrou and Kittler, 1991). Given that the composite performance of an image acquisition system can be considered to be that of a low-pass filter, any step changes in the image will be smoothed to become a ramp. As such, a more plausible model of the edge is a ramp, rather than a step. Since the process is based on ramp edges, and because of limits imposed by its formulation, the Petrou operator uses templates that much wider to preserve optimal properties. As such, the operator can impose greater computational complexity, but is a natural candidate for applications with the conditions for which its properties were formulated. 144
Feature Extraction and Image Processing
Of the other approaches, Korn (1988) developed a unifying operator for symbolic representation of grey level change. The Susan operator (Smith and Brady, 1997) derives from an approach aimed to find more that just edges, since it can also be used to derive corners (where feature boundaries change direction sharply, as in curvature detection in Section 4.8) and structure-preserving image noise reduction. Essentially, SUSAN derives from smallest univalue segment assimilating nucleus, which concerns aggregating the difference between elements in a (circular) template centred on the nucleus. The USAN is essentially the number of pixels within the circular mask that have similar brightness to the nucleus. The edge strength is then derived by subtracting the USAN size from a geometric threshold, which is, say, three-quarters of the maximum USAN size. The method includes a way of calculating edge direction, which is essential if non-maximum suppression is to be applied. The advantages are in simplicity (and hence speed), since it is based on simple operations, and the possibility of extension to find other feature types.
4.5 Comparison of edge detection operators The selection of an edge operator for a particular application depends on the application itself. As has been suggested, it is not usual to require the sophistication of the advanced operators in many applications. This is reflected in analysis of the performance of the edge operators on the eye image. To provide a different basis for comparison, we shall consider the difficulty of low-level feature extraction in ultrasound images. As has been seen earlier (Section 3.5.4), ultrasound images are very noisy and require filtering before analysis. Figure 4.31(a) is part of the ultrasound image which could have been filtered using the truncated median operator (Section 3.5.3). The image contains a feature called the pitus (the ‘splodge’ in the middle) and we shall see how different edge operators can be used to detect its perimeter, although without noise filtering. Earlier, in Section 3.5.4, we considered a comparison of statistical operators on an ultrasound image. The median is perhaps the most popular of these processes for general (i.e. non-ultrasound) applications. Accordingly, it is of interest that one study (Bovik et al., 1987) has suggested that the known advantages of median filtering (the removal of noise with the preservation of edges, especially for salt and pepper noise) are shown to good effect if it is used as a prefilter to first and second order approaches, although with the cost of the median filter. However, we will not consider median filtering here: its choice depends more on suitability to a particular application. The results for all edge operators have been generated using hysteresis thresholding, where the thresholds were selected manually for best performance. The basic first order operator (Figure 4.31b) responds rather nicely to the noise and it is difficult to select a threshold that reveals a major part of the pitus border. Some is present in the Prewitt and Sobel operators’ results (Figure 4.31c and d, respectively), but there is still much noise in the processed image, although there is less in the Sobel. The Laplacian operator (Figure 4.31e) gives very little information indeed, as to be expected with such noisy imagery. However, the more advanced operators can be used to good effect. The Marr–Hildreth approach improves matters (Figure 4.31f), but suggests that it is difficult to choose a LoG operator of appropriate size to detect a feature of these dimensions in such noisy imagery, illustrating the compromise between the size of operator needed for noise filtering and the size needed for the target feature. However, the Canny and Spacek operators can be used to good effect, as shown in Figure 4.31(g) and (h), respectively. These reveal much of the required information, together with data away from the Low-level feature extraction (including edge detection)
145
(a) Original image
(b) First order
(c) Prewitt
(d) Sobel
(e) Laplacian
(f) Marr-Hildreth
(g) Canny
(h) Spacek
Figure 4.31 Comparison of edge detection operators
pitus itself. In an automated analysis system, for this application, the extra complexity of the more sophisticated operators would clearly be warranted.
4.6 Further reading on edge detection Few computer vision and image processing texts omit detail concerning edge detection operators, although few give explicit details concerning implementation. Many of the earlier texts omit the more recent techniques. Parker (1994) only includes C code for some of the most basic edge detection techniques. Further information can be found in journal papers; Petrou’s excellent study of edge detection (Petrou, 1994) highlights study of the performance factors involved in the optimality of the Canny, Spacek and Petrou operators with extensive tutorial support (although I suspect that Petrou junior may one day be embarrassed by the frequency with which his youthful mugshot is used: his teeth show up very well!). There have been a number of surveys of edge detection highlighting performance attributes in comparison. See, for example, Torre and Poggio (1986), which gives a theoretical study of edge detection and considers some popular edge detection techniques in light of this analysis. One survey (Heath et al., 1997) surveys many approaches, comparing them in particular with the Canny operator (and states where code for some of the techniques they compared can be found). This showed that best results can be achieved by tuning an edge detector for a particular application and highlighted good results by the Bergholm operator (Bergholm, 1987). Marr (1982) considers the Marr–Hildreth approach to edge detection in the light of human vision (and its influence on perception), with particular reference to scale in edge detection. More recently, Yitzhaky and Peli (2003) suggest ‘a general tool to assist in practical implementations of parametric edge detectors where an automatic process is required’ and use statistical tests to evaluate edge detector performance. Since edge 146
Feature Extraction and Image Processing
detection is one of the most important vision techniques, it continues to be a focus of research interest. Accordingly, it is always worth looking at recent conference proceedings to see any new techniques, or perhaps more likely performance comparison or improvement, that may help you to solve a problem.
4.7 Phase congruency The comparison of edge detectors highlights some of their innate problems: incomplete contours, the need for selective thresholding, and their response to noise. Further, the selection of a threshold is often inadequate for all the regions in an image, since there are many changes in local illumination. We shall find that some of these problems can be handled at a higher level, when shape extraction can be arranged to accommodate partial data and to reject spurious information. There is, however, interest in refining the low-level feature extraction techniques further. Phase congruency is a feature detector with two main advantages: it can detect a broad range of features; and it is invariant to local (and smooth) change in illumination. As the name suggests, it is derived by frequency domain considerations operating on the considerations of phase (i.e. time). It is illustrated detecting some 1D features in Figure 4.32, where the features are the solid lines: a (noisy) step function in Figure 4.32(a), and a peak (or impulse) in Figure 4.32(b). By Fourier transform analysis, any function is made up from the controlled addition of sinewaves of differing frequencies. For the step function to occur (the solid line in Figure 4.32a), the constituent frequencies (the dotted lines in Figure 4.32a) must all change at the same time, so they add up to give the edge. Similarly, for the peak to occur, the constituent frequencies must all peak at the same time; in Figure 4.32(b) the solid line is the peak and the dotted lines are some of its constituent frequencies. This means that to find the feature in which we are interested, we can determine points where events happen at the same time: this is phase congruency. By way of generalization, a triangle wave is made of peaks and troughs: phase congruency implies that the peaks and troughs of the constituent signals should coincide.
(a) Step edge
(b) Peak
Figure 4.32 Low-level feature extraction by phase congruency
Low-level feature extraction (including edge detection)
147
The constituent sinewaves plotted in Figure 4.32(a) were derived by taking the Fourier transform of a step and then determining the sinewaves according to their magnitude and phase. The Fourier transform in Equation 2.15 delivers the complex Fourier components Fp. These can be used to show the constituent signals xc by 2
xc t = Fpu ej N ut+Fpu
(4.29)
where Fpu is again the magnitude of the uth Fourier component (Equation 2.7) and Fpu = Fpu is the argument, the phase in Equation 2.8. The (dotted) frequencies displayed in Figure 4.32 are the first four odd components (the even components for this function are zero, as shown in the Fourier transform of the step in Figure 2.11). The addition of these components is indeed the inverse Fourier transform which reconstructs the step feature. The advantages are that detection of congruency is invariant with local contrast: the sinewaves still add up so the changes are still in the same place, even if the magnitude of the step edge is much smaller. In images, this implies that we can change the contrast and still detect edges. This is illustrated in Figure 4.33. Here, a standard image processing image, the ‘cameraman’ image from the early UCSD dataset, has been changed between the left and right sides so that the contrast changes in the two halves of the image (Figure 4.33a). Edges detected by Canny are shown in Figure 4.33(b) and by phase congruency in 4.33(c). The basic structure of the edges detected by phase congruency is very similar to that structure detected by Canny, and the phase congruency edges appear somewhat cleaner (there is a single line associated with the tripod control in phase congruency); both detect the change in brightness between the two halves. There is a major difference though: the building in the lower right side of the image is barely detected in the Canny image, whereas it can clearly be seen by phase congruency. Its absence is due to the parameter settings used in the Canny operator. These can be changed, but if the contrast were to change again, then the parameters would need to be reoptimized for the new arrangement. This is not the case for phase congruency.
(a) Modified cameraman image
(b) Edges by the Canny operator
(c) Phase congruency
Figure 4.33 Edge detection by Canny and by phase congruency
Such a change in brightness might appear unlikely in practical application, but this is not the case with moving objects which interact with illumination or in fixed applications where illumination changes. In studies aimed to extract spinal information from digital videofluoroscopic X-ray images to provide guidance for surgeons (Zheng et al., 2004), phase congruency was 148
Feature Extraction and Image Processing
found to be immune to the changes in contrast caused by slippage of the shield used to protect the patient while acquiring the image information. One such image is shown in Figure 4.34. The lack of shielding is apparent in the bloom at the side of the images. This changes as the subject is moved, so it proved difficult to optimize the parameters for Canny over the whole sequence (Figure 4.34b), but the detail of a section of the phase congruency result (Figure 4.34c) shows that the vertebrae information is readily available for later high-level feature extraction.
(a) Digital videofluoroscopic image of lower spine showing vertebrae
(b) Edges by the Canny operator
(c) Features by phase congruency
Figure 4.34 Spinal contour by phase congruency (Zheng et al., 2004)
The original notions of phase congruency are the concepts of local energy (Morrone and Owens, 1987), with links to the human visual system (Morrone and Burr, 1988). One of the most sophisticated implementations was by Kovesi (1999), with added advantage that his Matlab implementation is available on the web (http://www.csse.uwa.edu.au/∼pk/ Research/research.html), as well as much more information. Essentially, we seek to determine features by detection of points at which Fourier components are maximally in phase. By extension of the Fourier reconstruction functions in Equation 4.29, Morrone and Owens defined a measure of phase congruency PC as ⎛ ¯ x ⎞ Fpu cos u x − ⎝ u ⎠ PC x = maxx∈02 (4.30) ¯ Fpu u
where u x represents the local phase of the component Fpu at position x. Essentially, this computes the ratio of the sum of projections onto a vector (the sum in the numerator) to the total ¯ x that maximizes this equation vector length (the sum in the denominator). The value of is the amplitude weighted mean local phase angle of all the Fourier terms at the point being considered. In Figure 4.35 the resulting vector is made up of four components, illustrating the projection of the second onto the resulting vector. Clearly, the value of PC ranges from 0 to 1, the maximum occurring when all elements point along the resulting vector. As such, the resulting phase congruency is a dimensionless normalized measure which is thresholded for image analysis. Low-level feature extraction (including edge detection)
149
Imaginary
φ2(x )
F p3
Fp 4
Fp 1
|F p2 |
|Fp2| cos(φ2(x ) – φ(x ))
φ(x ) Real
Figure 4.35 Summation in phase congruency
In this way, we have calculated the phase congruency for the step function in Figure 4.36(a), which is shown in Figure 4.36(b). Here, the position of the step is at time step 40; this is the position of the peak in phase congruency, as required. Note that the noise can be seen to affect the result, although the phase congruency is largest at the right place.
0
50
100
0
(a) (Noisy) step function
50
100
(b) Phase congruency of step function
Figure 4.36 One-dimensional phase congruency
One interpretation of the measure is that since for small angles, cos = 1 − 2 , then Equation 4.30 expresses the ratio of the magnitudes weighted by the variance of the difference to the summed magnitude of the components. There is certainly difficulty with this measure, apart from difficulty in implementation: it is sensitive to noise, as is any phase measure; it is not conditioned by the magnitude of a response (small responses are not discounted); and it is not well localized (the measure varies with the cosine of the difference in phase, not with the difference itself, although it does avoid discontinuity problems with direct use of angles). In fact, the phase congruency is directly proportional to the local energy (Venkatesh and Owens, 1989), 150
Feature Extraction and Image Processing
so an alternative approach is to search for maxima in the local energy. The notion of local energy allows us to compensate for the sensitivity to the detection of phase in noisy situations. For these reasons, Kovesi developed a wavelet-based measure which improved performance, while accommodating noise. In basic form, phase congruency can be determined by convolving a set of wavelet filters with an image, and calculating the difference between the average filter response and the individual filter responses. The response of a (1D) signal I to a set of wavelets at scale n is derived from the convolution of the cosine and sine wavelets (discussed in Section 2.7.3), denoted Mne and Mno , respectively en x on x = I x ∗ Mne I x ∗ Mno
(4.31)
to deliver the even and odd components at the nth scale en x and on x, respectively. The amplitude of the transform result at this scale is the local energy An x = en x2 + on x2 (4.32) At each point x we will have an array of vectors which correspond to each scale of the filter. Given that we are only interested in phase congruency that occurs over a wide range of frequencies (rather than just at a couple of scales), the set of wavelet filters needs to be designed so that adjacent components overlap. By summing the even and odd components we obtain F x = en x n
H x =
on x
(4.33)
n
and a measure of the total energy A as An x ≈ en x2 + on x2 n
(4.34)
n
Then a measure of phase congruency is F x2 + H x2 PC x = An x +
(4.35)
n
where the addition of a small factor in the denominator avoids division by zero and any potential result when values of the numerator are very small. This gives a measure of phase congruency, which is essentially a measure of the local energy. Kovesi improved on this, improving on the response to noise, developing a measure which reflects the confidence that the signal is significant relative to the noise. Further, he considers in detail the frequency domain considerations, and its extension to two dimensions (Kovesi, 1999). For 2D (image) analysis, phase congruency can be determined by convolving a set of wavelet filters with an image, and calculating the difference between the average filter response and the individual filter responses. The filters are constructed in the frequency domain by using complementary spreading functions; the filters must be constructed in the Fourier domain because the log-Gabor function has a singularity at = 0. To construct a filter with appropriate properties, a filter is constructed in a manner similar to the Gabor wavelet, but here in the frequency domain and using different functions. Following Kovesi’s implementation, the first filter is a low-pass filter, here a Gaussian filter g with L different orientations g l = √
1 2 s
e
−
−l 2 2 s2
(4.36)
Low-level feature extraction (including edge detection)
151
where is the orientation, s controls the spread about that orientation and l is the angle is local orientation focus. The other spreading function is a band-pass filter, here a log-Gabor filter lg with M different scales. ⎧ =0 ⎪ ⎨0
2 log m (4.37) lg m = ⎪ ⎩ √ 1 e− 2log2 = 0 2
where is the scale, controls bandwidth at that scale and m is the centre frequency at that scale. The combination of these functions provides a 2D filter l2Dg which can act at different scales and orientations. l2Dg m l = g l × lg m
(4.38)
One measure of phase congruency based on the convolution of this filter with the image P is derived by inverse Fourier transformation −1 of the filter l2Dg (to yield a spatial domain operator) which is convolved as S mxy = −1 l2Dg m l xy ∗ Pxy
(4.39)
to deliver the convolution result S at the mth scale. The measure of phase congruency over the M scales is then M S mxy m=1 (4.40) PCxy = M S mxy + m=1
where the addition of a small factor numerator again avoids division by zero and any potential result when values of S are very small. This gives a measure of phase congruency, but is certainly a bit of an ouch, especially as it still needs refinement. Note that keywords recur within phase congruency: frequency domain, wavelets and convolution. By its nature, we are operating in the frequency domain and there is not enough room in this text, and it is inappropriate to the scope here, to expand further. Despite this, the performance of phase congruency certainly encourages its consideration, especially if local illumination is likely to vary and if a range of features is to be considered. It is derived by an alternative conceptual basis, and this gives different insight, as well as performance. Even better, there is a Matlab implementation available, for application to images, allowing you to replicate its excellent results. There has been further research, noting especially its extension in ultrasound image analysis (Mulet-Parada and Noble, 2000) and its extension to spatiotemporal form (Myerscough and Nixon, 2004).
4.8 Localized feature extraction Two main areas are covered here. The traditional approaches aim to derive local features by measuring specific image properties. The main target has been to estimate curvature: peaks of local curvature are corners, and analysing an image by its corners is especially suited to images of artificial objects. The second area includes more modern approaches that improve performance by using region or patch-based analysis. We shall start with the more established curvature-based operators, before moving to the patch or region-based analysis. 152
Feature Extraction and Image Processing
4.8.1 Detecting image curvature (corner extraction) 4.8.1.1 Definition of curvature Edges are perhaps the low-level image features that are most obvious to human vision. They preserve significant features, so we can usually recognize what an image contains from its edge-detected version. However, there are other low-level features that can be used in computer vision. One important feature is curvature. Intuitively, we can consider curvature as the rate of change in edge direction. This rate of change characterizes the points in a curve; points where the edge direction changes rapidly are corners, whereas points where there is little change in edge direction correspond to straight lines. Such extreme points are very useful for shape description and matching, since they represent significant information with reduced data. Curvature is normally defined by considering a parametric form of a planar curve. The parametric contour v t = x t Ux + y t Uy describes the points in a continuous curve as the endpoints of the position vector. Here, the values of t define an arbitrary parameterization, the unit vectors are again Ux = 1 0 and Uy = 0 1 . Changes in the position vector are given by the tangent vector function of the curve vt. That is, v˙ t = x˙ t Ux + y˙ t Uy . This vectorial expression has a simple intuitive meaning. If we think of the trace of the curve as the motion of a point and t is related to time, the tangent vector definesthe instantaneous motion. At any moment, the point moves with a speed given by ˙v t = x˙ 2 t + y˙ 2 t in the direction −1 t = tan y˙ t x˙ t . The curvature at a point vt describes the changes in the direction t with respect to changes in arc length. That is, d t (4.41) t = ds where s is arc length, along the edge itself. Here is the angle of the tangent to the curve. That is, = ± 90 , where is the gradient direction defined in Equation 4.13. That is, if we apply an edge detector operator to an image, we have for each pixel a gradient direction value that represents the normal direction to each point in a curve. The tangent to a curve is given by an orthogonal vector. Curvature is given with respect to arc length because a curve parameterized by arc length maintains a constant speed of motion. Thus, curvature represents changes in direction for constant displacements along the curve. By considering the chain rule, we have d t dt t = (4.42) dt ds The differential ds/dt defines the change in arc length with respect to the parameter t. If we again consider the curve as the motion of a point, this differential defines the instantaneous change in distance with respect to time. That is, the instantaneous speed. Thus, (4.43) ds dt = ˙v t = x˙ 2 t + y˙ 2 t and
(4.44) x˙ 2 t + y˙ 2 t dt ds = 1 By considering that t = tan−1 y˙ t x˙ t , then the curvature at a point vt in Equation 4.42 is given by x˙ t y¨ t − y˙ t x¨ t (4.45) t = ˙x2 t + y˙ 2 t 3/ 2 This relationship is called the curvature function and it is the standard measure of curvature for planar curves (Apostol, 1966). An important feature of curvature is that it relates the derivative Low-level feature extraction (including edge detection)
153
of a tangential vector to a normal vector. This can be explained by the simplified Serret–Frenet equations (Goetz, 1970) as follows. We can express the tangential vector in polar form as v˙ t = ˙vt cos t + j sin t
(4.46)
If the curve is parameterized by arc length, then ˙vt is constant. Thus, the derivative of a tangential vector is simply given by v¨ t = ˙v t − sin t + j cos t d t dt (4.47) Since we are using a normal parameterization, then d t dt = d t ds. Thus, the tangential vector can be written as v¨ t = tnt
(4.48)
where n t = vt − sin t + j cos t defines the direction of v¨ t, while the cur˙ vature t defines its modulus. The derivative of the normal vector is given by nt = ˙v t − cos t − i sin t d t /ds, which can be written as ˙ = −t˙vt nt
(4.49)
Clearly, nt is normal to v˙ t. Therefore, for each point in the curve, there is a pair of orthogonal vectors v˙ t and nt whose moduli are proportionally related by the curvature. In general, the curvature of a parametric curve is computed by evaluating Equation 4.45. For a straight line, for example, the second derivatives x¨ t and y¨ t are zero, so the curvature function is nil. For a circle of radius r, we have that x˙ t = r cos t and y˙ t = −r sin t. Thus, y¨ t = −r cos t, x¨ t = −r sin t and t = 1 r. However, for curves in digital images, the derivatives must be computed from discrete data. This can be done in three main ways. The most obvious approach is to calculate curvature by directly computing the difference between angular direction of successive edge pixels in a curve. A second approach is to derive a measure of curvature changes in image intensity. Finally, a measure of curvature can be obtained by correlation.
4.8.1.2 Computing differences in edge direction Perhaps the easier way to compute curvature in digital images is to measure the angular change along the curve’s path. This approach was considered in early corner detection techniques (Bennett and MacDonald, 1975; Groan and Verbeek, 1978; Kitchen and Rosenfeld, 1982) and it merely computes the difference in edge direction between connected pixels forming a discrete curve. That is, it approximates the derivative in Equation 4.41 as the difference between neighbouring pixels. As such, curvature is simply given by k t = t+1 − t−1
(4.50)
where the sequence t−1 t t+1 t+2 represents the gradient direction of a sequence of pixels defining a curve segment. Gradient direction can be obtained as the angle given by an edge detector operator. Alternatively, it can be computed by considering the position of pixels in the sequence. That is, by defining t = yt−1 − yt+1 xt−1 − xt+1 , where xt yt denotes pixel t in the sequence. Since edge points are only defined at discrete points, this angle can only take eight 154
Feature Extraction and Image Processing
values, so the computed curvature is very ragged. This can be smoothed out by considering the difference in mean angular direction of n pixels on the leading and trailing curve segment. That is, kn t =
n −1 1 1 t+i − n i=1 n i=−n t+i
(4.51)
The average also gives some immunity to noise and it can be replaced by a weighted average if Gaussian smoothing is required. The number of pixels considered, the value of n, defines a compromise between accuracy and noise sensitivity. Notice that filtering techniques may also be used to reduce the quantization effect when angles are obtained by an edge detection operator. As we have already discussed, the level of filtering the filtering is related to the size of the template (as in Section 3.4.3). To compute angular differences, we need to determine connected edges. This can easily be implemented with the code already developed for hysteresis thresholding in the Canny edge operator. To compute the difference of points in a curve, the connect routine (Code 4.12) only needs to be arranged to store the difference in edge direction between connected points. Code 4.16 shows an implementation for curvature detection. First, edges and magnitudes are determined. Curvature is only detected at edge points. As such, we apply maximal suppression. The function Cont returns a matrix containing the connected neighbour pixels of each edge. Each edge pixel is connected to one or two neighbours. The matrix Next stores only the direction of consecutive pixels in an edge. We use a value of −1 to indicate that there is no connected neighbour. The function NextPixel obtains the position of a neighbouring pixel
%Curvature detection function outputimage=CurvConnect(inputimage) [rows,columns]=size(inputimage); outputimage=zeros(rows,columns); [Mag,Ang]=Edges(inputimage); Mag=MaxSupr(Mag,Ang); Next=Cont(Mag,Ang);
%Image size %Result image %Edge Detection %Maximal Suppression %Next connected pixels
%Compute curvature in each pixel for x=1:columns-1 for y=1:rows-1 if Mag(y,x)~=0 n=Next(y,x,1); m=Next(y,x,2); if(n~=-1 & m~=-1) [px,py]=NextPixel(x,y,n); [qx,qy]=NextPixel(x,y,m); outputimage(y,x)=abs(Ang(py,px)-Ang(qy,qx)); end end end end Code 4.16 Curvature by differences
Low-level feature extraction (including edge detection)
155
by taking the position of a pixel and the direction of its neighbour. The curvature is computed as the difference in gradient direction of connected neighbour pixels. The result of applying this form of curvature detection to an image is shown in Figure 4.37. Figure 4.37(a) contains the silhouette of an object; Figure 4.39(b) is the curvature obtained by computing the rate of change of edge direction. In this figure, curvature is defined only at the edge points. Here, by its formulation the measurement of curvature gives just a thin line of differences in edge direction which can be seen to track the perimeter points of the shapes (at points where there is measured curvature). The brightest points are those with greatest curvature. To show the results, we have scaled the curvature values to use 256 intensity values. The estimates of corner points could be obtained by a uniformly thresholded version of Figure 4.37(b), well in theory anyway!
(a) Image
(b) Detected corners
Figure 4.37 Curvature detection by difference
Unfortunately, as can be seen, this approach does not provide reliable results. It is essentially a reformulation of a first order edge detection process and presupposes that the corner information lies within the threshold data (and uses no corner structure in detection). One of the major difficulties with this approach is that measurements of angle can be severely affected by quantization error and accuracy is limited (Bennett and MacDonald, 1975), a factor which will return to plague us later when we study methods for describing shapes.
4.8.1.3 Measuring curvature by changes in intensity (differentiation) As an alternative way of measuring curvature, we can derive the curvature as a function of changes in image intensity. This derivation can be based on the measure of angular changes in the discrete image. We can represent the direction at each image point as the function x y. Thus, according to the definition of curvature, we should compute the change in these direction values normal to the image edge (i.e. along the curves in an image). The curve at an edge can be locally approximated by the points given by the parametric line defined by x t = x + t cos x y and y t = y + t sin x y. Thus, the curvature is given by the change in the function x y with respect to t. That is, x y = 156
x y x y x t x y y t = + t x t y t
Feature Extraction and Image Processing
(4.52)
where x t t = cos and y t t = sin . By considering the definition of the gradient angle, we have that the normal tangent direction at a point in a line is given by x y = tan−1 Mx −My . From this geometry we can observe that cos = −My
Mx2 + My2
and
sin = Mx
Mx2 + My2
(4.53)
By differentiation of x y and by considering these definitions we obtain x y =
1 3
Mx2 + My2 2
2 Mx
My My Mx My − MxMy + Mx2 − MxMy x x y y
(4.54)
This defines a forward measure of curvature along the edge direction. We can use an alternative direction to measure of curvature. We can differentiate backwards (in the direction of − x y giving − x y. In this case we consider that the curve is given by x t = x + t cos − x y and y t = y + t sin − x y. Thus, − x y =
1
My2
3
Mx2 + My2 2
Mx My My Mx − MxMy − Mx2 + MxMy x x y y
(4.55)
Two further measures can be obtained by considering the forward and a backward differential along the normal. These differentials cannot be related to the actual definition of curvature, but can be explained intuitively. If we consider that curves are more than one pixel wide, differentiation along the edge will measure the difference between the gradient angle between interior and exterior borders of a wide curve. In theory, the tangent angle should be the same. However, in discrete images there is a change due to the measures in a window. If the curve is a straight line, then the interior and exterior borders are the same. Thus, gradient direction normal to the edge does not change locally. As we bend a straight line, we increase the difference between the curves defining the interior and exterior borders. Thus, we expect the measure of gradient direction to change. That is, if we differentiate along the normal direction, we maximize detection of gross curvature. The value ⊥ x y is obtained when x t = x + t sin x y and y t = y + t cos x y. In this case, ⊥ x y =
1 3
Mx2 + My2 2
Mx2
My My My Mx − MxMy − MxMy + MyMy x x y y
(4.56)
In a backward formulation along a normal direction to the edge, we obtain: −⊥ x y =
1 3
Mx2 + My2 2
−Mx2
Mx My My Mx + MxMy − MxMy + My2 x x y y
(4.57)
This was originally used by Kass et al. (1988) as a means to detect line terminations, as part of a feature extraction scheme called snakes (active contours), which are covered in Chapter 6. Code 4.17 shows an implementation of the four measures of curvature. The function Gradient is used to obtain the gradient of the image and to obtain its derivatives. The output image is obtained by applying the function according to the selection of parameter op. Low-level feature extraction (including edge detection)
157
%Gradient Corner Detector %op=T tangent direction %op=TI tangent inverse %op=N normal direction %op=NI normal inverse function outputimage=GradCorner(inputimage,op) [rows,columns]=size(inputimage); %Image size outputimage=zeros(rows,columns); %Result image [Mx,My]=Gradient(inputimage); %Gradient images [M,A]=Edges(inputimage); %Edge Suppression M=MaxSupr(M,A); %Derivatives of the [Mxx,Mxy]=Gradient(Mx); [Myx,Myy]=Gradient(My); %gradient image
%compute curvature for x=1:columns for y=1:rows if(M(y,x)~=0) My2=My(y,x)^2; Mx2=Mx(y,x)^2; MxMy=Mx(y,x)*My(y,x); if((Mx2+My2)~=0) if(op=='TI') outputimage(y,x)=(1/(Mx2+My2)^1.5)*(My2*Mxx(y,x) -MxMy*Myx(y,x)-Mx2*Myy(y,x) +MxMy*Mxy(y,x)); elseif (op=='N') outputimage(y,x)=(1/(Mx2+My2)^1.5)*(Mx2*Myx(y,x) -MxMy*Mxx(y,x)-MxMy*Myy(y,x) +My2*Mxy(y,x)); elseif (op=='NI') outputimage(y,x)=(1/(Mx2+My2)^1.5)*(-Mx2*Myx(y,x) +MxMy*Mxx(y,x)-MxMy*Myy(y,x) +My2*Mxy(y,x)); else %tangential as default outputimage(y,x)=(1/(Mx2+My2)^1.5)*(My2*Mxx(y,x) -MxMy*Myx(y,x)+Mx2*Myy(y,x) -MxMy*Mxy(y,x)); end end end end end
Code 4.17 Curvature by measuring changes in intensity
Let us see how the four functions for estimating curvature from image intensity perform for the image given in Figure 4.37(a). In general, points where the curvature is large are highlighted by each function. Different measures of curvature (Figure 4.38) highlight differing points on the feature boundary. All measures appear to offer better performance than that derived by reformulating hysteresis thresholding (Figure 4.37b), although there is little discernible performance advantage between the direction of differentiation. As the results in Figure 4.38 suggest, detecting curvature directly from an image is not a totally reliable way of determining curvature, and 158
Feature Extraction and Image Processing
(a) κϕ
(b) κ –ϕ
(c) κ⊥ϕ
(d) κ−⊥ϕ
Figure 4.38 Comparing image curvature detection operators
hence corner information. This is in part due to the higher order of the differentiation process. (Also, scale has not been included within the analysis.)
4.8.1.4 Moravec and Harris detectors In the previous section, we measured curvature as the derivative of the function x y along a particular direction. Alternatively, a measure of curvature can be obtained by considering changes along a particular direction in the image P itself. This is the basic idea of Moravec’s corner detection operator. This operator computes the average change in image intensity when a window is shifted in several directions. That is, for a pixel with coordinates x y, and a window size of 2w + 1 we have: Euv x y =
w w 2 Px+iy+j − Px+i+uy+j+v
(4.58)
i=−w j=−w
This equation approximates the autocorrelation function in the direction (u, v). A measure of curvature is given by the minimum value of Euv x y obtained by considering the shifts (u, v) in the four main directions. That is, by (1,0), (0,−1), (0,1) and (−1,0). The minimum is chosen because it agrees with the following two observations. First, if the pixel is in an edge defining a straight line, Euv x y is small for a shift along the edge and large for a shift perpendicular to the edge. In this case, we should choose the small value since the curvature of the edge is small. Secondly, if the edge defines a corner, then all the shifts produce a large value. Thus, if we also chose the minimum, this value indicates high curvature. The main problem with this approach Low-level feature extraction (including edge detection)
159
is that it considers only a small set of possible shifts. This problem is solved in the Harris corner detector (Harris and Stephens, 1988) by defining an analytic expression for the autocorrelation. This expression can be obtained by considering the local approximation of intensity changes. We can consider that the points Px+iy+j and Px+i+uy+j+v define a vector (u, v) in the image. Thus, in a similar fashion to the development given in Equation 4.58, the increment in the image the points can be approximated by the directional derivative function between uPx+iy+j x + vPx+iy+j y. Thus, the intensity at Px+i+uy+j+v can be approximated as Px+i+uy+j+v = Px+iy+j +
Px+iy+j Px+iy+j u+ v x y
(4.59)
This expression corresponds to the three first terms of the Taylor expansion around Px+iy+j (an expansion to first order). If we consider the approximation in Equation 4.58 we have: Euv x y =
w w i=−w j=−w
Px+iy+j Px+iy+j u+ v x y
2
(4.60)
By expansion of the squared term (and since u and v are independent of the summations), we obtain: Euv x y = A x y u2 + 2C x y uv + B x y v2
(4.61)
where A x y =
w w
i=−w j=−w
C x y =
w
w
i=−w j=−w
Px+iy+j x Px+iy+j x
2 B x y =
w w i=−w j=−w
Px+iy+j y
Px+iy+j y
2 (4.62)
That is, the summation of the squared components of the gradient direction for all the pixels in the window. In practice, this average can be weighted by a Gaussian function to make the measure less sensitive to noise (i.e. by filtering the image data). To measure the curvature at a point (x, y), it is necessary to find the vector (u, v) that minimizes Euv x y given in Equation 4.61. In a basic approach, we can recall that the minimum is obtained when the window is displaced in the direction of the edge. Thus, we can consider that u = cos x y and v = sin x y. These values were defined in Equation 4.53. Accordingly, the minima values that define curvature are given by uv x y = min Euv x y =
A x y My 2 + 2C x y Mx My + B x y Mx 2 Mx 2 + My 2
(4.63)
In a more sophisticated approach, we can consider the form of the function Euv x y. We can observe that this is a quadratic function, so it has two principal axes. We can rotate the function such that its axes have the same direction as the axes of the coordinate system. That is, we rotate the function Euv x y to obtain Fuv x y = x y2 u2 + x y2 v2 160
Feature Extraction and Image Processing
(4.64)
The values of and are proportional to the autocorrelation function along the principal axes. Accordingly, if the point (x, y) is in a region of constant intensity, both values are small. If the point defines a straight border in the image, then one value is large and the other is small. If the point defines an edge with high curvature, both values are large. Based on these observations a measure of curvature is defined as k x y = − k + 2
(4.65)
The first term in this equation makes the measure large when the values of and increase. The second term is included to decrease the values in flat borders. The parameter k must be selected to control the sensitivity of the detector. The higher the value, the more sensitive the computed curvature will be to changes in the image (and therefore to noise). In practice, to compute k x y it is not necessary to compute explicitly the values of and , but the curvature can be measured from the coefficient of the quadratic expression in Equation 4.61. This can be derived by considering the matrix forms of Equations 4.61 and 4.64. If we define the vector DT = u v , then Equations 4.60 and 4.63 can be written as Euv x y = DT MD and where
T
Fuv x y = DT QD
denotes the transpose and where A x y C x y M= and C x y B x y
Q=
0
0
(4.66)
(4.67)
To relate Equations 4.60 and 4.63, we consider that Fuv x y is obtained by rotating Euv x y by a transformation R that rotates the axis defined by D. That is, Fuv x y = RDT MRD
(4.68)
This can be arranged as Fuv x y = DT RT MRD
(4.69)
By comparison with Equation 4.66, we have: Q = RT MR
(4.70)
This defines a well-known equation of linear algebra and it means that Q is an orthogonal decomposition of M. The diagonal elements of Q are called the eigenvalues. We can use Equation 4.70 to obtain the value of , which defines the firstterm in Equation 4.65 by T Q considering the determinant of the matrices. That is, det = det R det M det R. Since R is a rotation matrix det RT det R = 1, thus = A x y B x y − C x y2
(4.71)
which defines the first term in Equation 4.65. The second term can be obtained by taking the trace of the matrices on each side of this equation. Thus, we have: + = A x y + B x y
(4.72)
We can also use Equation 4.70 to obtain the value of + , which defines the first term in Equation 4.65. By taking the trace of the matrices in each side of this equation, we have: k x y = A x y B x y − C x y2 − k A x y + B x y2 Low-level feature extraction (including edge detection)
(4.73) 161
Code 4.18 shows an implementation for Equations 4.64 and 4.73. The equation to be used is selected by the op parameter. Curvature is only computed at edge points; that is, at pixels whose edge magnitude is different of zero after applying maximal suppression. The first part of the code computes the coefficients of the matrix M. Then, these values are used in the curvature computation.
%Harris Corner Detector %op=H Harris %op=M Minimum direction function outputimage=Harris(inputimage,op) w=4; k=0.1; [rows,columns]=size(inputimage); outputimage=zeros(rows,columns); [difx,dify]=Gradient(inputimage); [M,A]=Edges(inputimage); M=MaxSupr(M,A);
%Window size=2w+1 %Second term constant %Image size %Result image %Differential %Edge Suppression
%compute correlation for x=w+1:columns-w %pixel (x,y) for y=w+1:rows-w if M(y,x)~=0 %compute window average A=0;B=0;C=0; for i=-w:w for j=-w:w A=A+difx(y+i,x+j)^2; B=B+dify(y+i,x+j)^2; C=C+difx(y+i,x+j)*dify(y+i,x+j); end end if(op=='H') outputimage(y,x)=A*B-C^2-k*((A+B)^2);; else dx=difx(y,x); dy=dify(y,x); if dx*dx+dy*dy~=0 outputimage(y,x)=((A*dy*dy2*C*dx*dy+B*dx*dx)/(dx*dx+dy*dy)); end end end end end
Code 4.18 Harris corner detector
Figure 4.39 shows the results of computing curvature using this implementation. The results are capable of showing the different curvature in the border. We can observe that k x y 162
Feature Extraction and Image Processing
(a) κu,v (x, y )
(b) κk (x, y )
Figure 4.39 Curvature via the Harris operator
produces more contrast between lines with low and high curvature than uv x y. The reason is the inclusion of the second term in Equation 4.73. In general, the measure of correlation is not only useful to compute curvature; this technique has much wider application in finding points for matching pairs of images.
4.8.1.5 Further reading on curvature Many of the arguments earlier advanced on extensions to edge detection in Section 4.4 apply to corner detection as well, so the same advice applies. There is much less attention paid by established textbooks to corner detection through Davis (2005) devotes a chapter to the topic. Van Otterloo’s fine book on shape analysis (van Otterloo, 1991) contains a detailed analysis of measurement of (planar) curvature. There are other important issues in corner detection. It has been suggested that corner extraction can be augmented by local knowledge to improve performance (Rosin, 1996). There are many other corner detection schemes, each offering different attributes, although with differing penalties. Important work has focused on characterizing shapes using corners. In a scheme analogous to the primal sketch introduced earlier, there is a curvature primal sketch (Asada and Brady, 1986), which includes a set of primitive parameterized curvature discontinuities (such as termination and joining points). There are many other approaches: one suggestion is to define a corner as the intersection between two lines; this requires a process to find the lines. Other techniques use methods that describe shape variation to find corners. We commented that filtering techniques can be included to improve the detection process; however, filtering can also be used to obtain a multiple detail representation. This representation is very useful to shape characterization. A curvature scale space has been developed (Mokhtarian and Mackworth, 1986; Mokhtarian and Bober, 2003) to give a compact way of representing shapes, and at different scales, from coarse (low-level) to fine (detail), and with the ability to handle appearance transformations.
4.8.2 Modern approaches: region/patch analysis 4.8.2.1 Scale invariant feature transform The scale invariant feature transform (SIFT) (Lowe, 1999, 2004) aims to resolve many of the practical problems in low-level feature extraction and their use in matching images. The earlier Low-level feature extraction (including edge detection)
163
Harris operator is sensitive to changes in image scale and as such is unsuited to matching images of differing size. SIFT involves two stages: feature extraction and description. The description stage concerns use of the low-level features in object matching, and this will be considered later. Low-level feature extraction within the SIFT approach selects salient features in a manner invariant to image scale (feature size) and rotation, and with partial invariance to change in illumination. Further, the formulation reduces the probability of poor extraction due to occlusion clutter and noise. It also shows how many of the techniques considered previously can be combined and capitalized on, to good effect. First, the difference of Gaussians operator is applied to an image to identify features of potential interest. The formulation aims to ensure that feature selection does not depend on feature size (scale) or orientation. The features are then analysed to determine location and scale before the orientation is determined by local gradient direction. Finally, the features are transformed into a representation that can handle variation in illumination and local shape distortion. Essentially, the operator uses local information to refine the information delivered by standard operators. The detail of the operations is best left to the source material (Lowe, 1999, 2004), for it is beyond the level or purpose here. As such, we shall concentrate on principle only. The features detected for the Lena image are illustrated in Figure 4.40. Here, the major features detected are shown by white lines, where the length reflects magnitude and the direction reflects the feature’s orientation. These are the major features, which include the rim of the hat, face features and the boa. The minor features are the smaller white lines: the ones shown here are concentrated around a background feature. In the full set of features detected at all scales in this image, there are many more of the minor features, concentrated particularly in the textured regions of the image (Figure 4.43). Later, we shall see how this can be used within shape extraction, but the purpose here is the basic low-level features extracted by this new technique.
(a) Original image
(b) Output points with magnitude and direction
Figure 4.40 Detecting features with the SIFT operator
164
Feature Extraction and Image Processing
In the first stage, the difference of Gaussians for an image P is computed in the manner of Equation 4.28 as D x y =
g x y k − g x y ∗ P k −
(4.74)
= L x y k − L x y k The function L is a scale-space function which can be used to define smoothed images at different scales. Note again the influence of scale-space in the more modern techniques. Rather than any difficulty in locating zero-crossing points, the features are the maxima and minima of the function. Candidate keypoints are then determined by comparing each point in the function with its immediate neighbours. The process then proceeds to analysis between the levels of scale, given appropriate sampling of the scale-space. This then implies comparing a point with its eight neighbours at that scale and with the nine neighbours in each of the adjacent scales, to determine whether it is a minimum or a maximum, as well as image resampling to ensure comparison between the different scales. To filter the candidate points to reject those which are the result of low local contrast (low edge strength) or which are poorly localized along an edge, a function is derived by local curve fitting, which indicates local edge strength and stability as well as location. Uniform thresholding then removes the keypoints with low contrast. Those that have poor localization, i.e. their position is likely to be influenced by noise, can be filtered by considering the ratio of curvature along an edge to that perpendicular to it, in a manner following the Harris operator in Section 4.8.1.4, by thresholding the ratio of Equations 4.71 and 4.72. To characterize the filtered keypoint features at each scale, the gradient magnitude is calculated in exactly the manner of Equations 4.12 and 4.13 as MSIFT x y =
L x + 1 y − L x − 1 y2 + L x y + 1 − L x y − 12 L x y + 1 − L x y − 1 −1 SIFT x y = tan L x + 1 y − L x − 1 y
(4.75) (4.76)
The peak of the histogram of the orientations around a keypoint is then selected as the local direction of the feature. This can be used to derive a canonical orientation, so that the resulting descriptors are invariant with rotation. As such, this contributes to the process which aims to reduce sensitivity to camera viewpoint and to non-linear change in image brightness (linear changes are removed by the gradient operations) by analysing regions in the locality of the selected viewpoint. The main description (Lowe, 2004) considers the technique’s basis in much greater detail, and outlines factors important to its performance, such as the need for sampling and performance in noise. As shown in Figure 4.41, the technique can certainly operate well, and scale is illustrated by applying the operator to the original image and to one at half the resolution. In all, 601 keypoints are determined in the original resolution image and 320 keypoints at half the resolution. By inspection, the major features are retained across scales (a lot of minor regions in the leaves disappear at lower resolution), as expected. Alternatively, the features can be filtered further by magnitude, or even direction (if appropriate). If you want more than results to convince you, implementations are available for Windows and Linux (http://www.cs.ubc.ca/spider/lowe/ research.html): a feast for a developer. These images were derived using siftWin32, version 4. Low-level feature extraction (including edge detection)
165
(a) Original image
(b) Key points at full resolution
(c) Key points at half resolution
Figure 4.41 SIFT feature detection at different scales
4.8.2.2 Saliency The new saliency operator (Kadir and Brady, 2001) was also motivated by the need to extract robust and relevant features. In the approach, regions are considered salient if they are simultaneously unpredictable both in some feature and scale–space. Unpredictability (rarity) is determined in a statistical sense, generating a space of saliency values over position and scale, as a basis for later understanding. The technique aims to be a generic approach to scale and saliency compared to conventional methods, because both are defined independent of a particular basis morphology–meaning that it is not based on a particular geometric feature like a blob, edge or corner. The technique operates by determining the entropy (a measure of rarity) within patches at scales of interest and the saliency is a weighted summation of where the entropy peaks. The new method has practical capability in that it can be made invariant to rotation, translation, non-uniform scaling and uniform intensity variations and robust to small changes in viewpoint. An example result of processing the image in Fig. 4.42(a) is shown in Figure 4.42(b) where the 200 most salient points are shown circled, and the radius of the circle is indicative of the scale. Many of the points are around the walking subject and others highlight significant features in the background, such as the waste bins, the tree or the time index. An example use of saliency was within an approach to learn and recognize object class models (such as faces, cars or animals)
(a) Original image
Figure 4.42 Detecting features by saliency
166
Feature Extraction and Image Processing
(b) Top 200 saliency matches circled
from unlabelled and unsegmented cluttered scenes, irrespective of their overall size (Fergus et al., 2003). For further study and application, descriptions and Matlab binaries are available from Kadir’s website (http://www.robots.ox.ac.uk/∼timork/).
4.8.2.3 Other techniques and performance issues There has been a recent comprehensive performance review (Mikolajczyk and Schmid, 2005), comparing established and new patch-based operators. The techniques that were compared included SIFT, differential derivatives by differentiation, cross-correlation for matching, and a gradient location and orientation-based histogram (an extension to SIFT, which performed well); the saliency approach was not included. The criterion used for evaluation concerned the number of correct matches, and the number of false matches, between feature points selected by the techniques. The matching process was between an original image and one of the same scene when subject to one of six image transformations. The image transformations covered practical effects that can change image appearance, and were: rotation, scale change, viewpoint change, image blur, JPEG compression, and illumination. For some of these there were two scene types available, which allowed for separation of understanding of scene type and transformation. The study observed that, within its analysis, ‘the SIFT-based descriptors perform best’, but this is a complex topic and selection of technique is often application dependent. Note that there is further interest in performance evaluation, and in invariance to higher order changes in viewing geometry, such as invariance to affine and projective transformation.
4.9 Describing image motion We have looked at the main low-level features that we can extract from a single image. In the case of motion, we must consider more than one image. If we have two images obtained at different times, the simplest way in which we can detect motion is by image differencing. That is, changes or motion can be located by subtracting the intensity values; when there is not motion, the subtraction will give a zero value and when an object in the image moves their pixel’s intensity changes, so the subtraction will give a value different of zero. To denote a sequence of images, we include a time index in our previous notation. That is, Ptxy . Thus, the image at the origin of our time is P0xy and the next image is P1xy . As such, the image differencing operation which delivered the difference image D is given by Dt = Pt − Pt − 1
(a) Difference image D
(4.77)
(b) First image
(c) Second image
Figure 4.43 Detecting motion by differencing
Low-level feature extraction (including edge detection)
167
Figure 4.43 shows an example of this operation. The image in 4.43(a) is the result of subtracting the image in Figure 4.43(b) from the one in Figure 4.43(c). This shows rather more than just the bits that are moving; we have not just highlighted the moving subject, we have also highlighted bits above the subject’s head and around his feet. This is due mainly to change in the lighting (the shadows around the feet are to do with the subject’s interaction with the lighting). However, perceived change can also be due to motion of the camera and to the motion of other objects in the field of view. In addition to these inaccuracies, perhaps the most important limitation of differencing is the lack of information about the movement itself. That is, we cannot see exactly how image points have moved. To describe the way in which the points in an image move, we should study how the pixels’ position changes in each image frame.
4.9.1 Area-based approach When a scene is captured at different times, 3D elements are mapped into corresponding pixels in the images. Thus, if image features are not occluded, they can be related to each other and motion can be characterized as a collection of displacements in the image plane. The displacement corresponds to the project movement of the objects in the scene and it is referred to as the optical flow. If you were to take an image, and its optical flow, you should be able to construct the next frame in the image sequence. So optical flow is like a measurement of velocity, the movement in pixels/unit of time, more simply pixels/frame. Optical flow can be found by looking for corresponding features in images. We can consider alternative features such as points, pixels, curves or complex descriptions of objects. The problem of finding correspondences in images has motivated the development of many techniques that can be distinguished by the features, the constraints imposed and the optimization or searching strategy (Dhond and Aggarwal, 1989). When features are pixels, the correspondence can be found by observing the similarities between intensities in image regions (local neighbourhood). This approach is known as area-based matching and it is one of the most common techniques used in computer vision (Barnard and Fichler, 1987). In general, pixels in non-occluded regions can be related to each other by means of a general transformation of the form P t + 1x+xy+y = P txy + H txy
(4.78)
where the function Htxy compensates for intensity differences between the images, and x y defines the displacement vector of the pixel at time t + 1. That is, the intensity of the pixel in the frame at time t + 1 is equal to the intensity of the pixel in the position (x, y) in the previous frame plus some small change due to physical factors and temporal differences that induce the photometric changes in images. These factors can be due, for example, to shadows, specular reflections, differences in illumination or changes in observation angles. In a general case, it is extremely difficult to account for the photometric differences, thus the model in Equation 4.78 is generally simplified by assuming that • the brightness of a point in an image is constant • that neighbouring points move with similar velocity. According to the first assumption, we have H x ≈ 0. Thus, Pt + 1x+xy+y = Ptxy 168
Feature Extraction and Image Processing
(4.79)
Many techniques have used this relationship to express the matching process as an optimization or variational problem (Jordan and Bovik, 1992). The objective is to find the vector x y that minimizes the error given by (4.80) exy = S Pt + 1x+xy+y Ptxy where S() represents a function that measures the similarity between pixels. As such, the optimum is given by the displacements that minimizes the image differences. Alternative measures of similarity can be used to define the matching cost (Jordan and Bovik, 1992). For example, we can measure the difference by taking the absolute of the arithmetic difference. Alternatively, we can consider the correlation or the squared values of the difference or an equivalent normalized form. In practice, it is difficult to try to establish a conclusive advantage of a particular measure, since they will perform differently depending on the kind of image, the kind of noise and the nature of the motion we are observing. As such, one is free to use any measure as long as it can be justified based on particular practical or theoretical observations. The correlation and the squared difference will be explained in more detail in the next chapter when we consider how a template can be located in an image. We shall see that if we want to make the estimation problem in Equation 4.80 equivalent to maximum likelihood estimation then we should minimize the squared error. That is, 2 exy = Pt + 1x+xy+y Ptxy (4.81) In practice, the implementation of the minimization is extremely prone to error since the displacement is obtained by comparing intensities of single pixels; it is very likely that the intensity changes or that a pixel can be confused with other pixels. To improve the performance, the optimization includes the second assumption presented above. If neighbouring points move with similar velocity, we can determine the displacement by considering not just a single pixel, but pixels in a neighbourhood. Thus, 2 Pt + 1x +xy +y Ptx y (4.82) exy = x y ∈W
That is, the error in the pixel at position (x, y) is measured by comparing all the pixels x y in a window W . This makes the measure more stable by introducing an implicit smoothing factor. The size of the window is a compromise between noise and accuracy. The automatic selection of the window parameter has attracted some interest (Kanade and Okutomi, 1994). Another important problem is the amount of computation involved in the minimization when the displacement between frames is large. This has motivated the development of hierarchical implementations. Other extensions have considered more elaborate assumptions about the speed of neighbouring pixels. A straightforward implementation of the minimization of the square error is presented in Code 4.19. This function has a pair of parameters that define the maximum displacement and the window size. The optimum displacement for each pixel is obtained by comparing the error for all the potential integer displacements. In a more complex implementation, it is possible to obtain displacements with subpixel accuracy (Lawton, 1983). This is normally achieved by a postprocessing step based on subpixel interpolation or by matching surfaces obtained by fitting the data at the integer positions. The effect of the selection of different window parameters can be seen in the example shown in Figure 4.44. Figure 4.44(a) and (b) show an object moving up into a static background (at least for the two frames we are considering). Figure 4.44(c)–(e) shows the displacements obtained by considering windows of increasing size. Here, we can observe that as the size of the window increases, the result is smoother, but detail has been about Low-level feature extraction (including edge detection)
169
%Optical flow by correlation %d: max displacement., w:window size 2w+1 function FlowCorr(inputimage1,inputimage2,d,w) %Load images L1=double(imread(inputimage1, 'bmp')); L2=double(imread(inputimage2,'bmp')); %image size [rows,columns]=size(L1); %L2 must have the same size %result image u=zeros(rows,columns); v=zeros(rows,columns); %correlation for each pixel for x1=w+d+1:columns-w-d for y1=w+d+1:rows-w-d min=99999; dx=0; dy=0; %displacement position for x2=x1-d:x1+d for y2=y1-d:y1+d sum=0; for i=-w:w% window for j=-w:w sum=sum+(double(L1(y1+j,x1+i))double(L2(y2+j,x2+i)))^2; end end if (sum<min) min=sum; dx=x2-x1; dy=y2-y1; end end end u(y1,x1)=dx; v(y1,x1)=dy; end end %display result quiver(u,v,.1);
Code 4.19 Implementation of area-based motion computation
the boundary of the object. We can also observe that when the window is small, the are noisy displacements near the object’s border. This can be explained by considering that Equation 4.78 supposes that pixels appear in both images, but this is not true near the border since pixels appear and disappear (i.e. occlusion) from and behind the moving object. In addition, there are problems in regions that lack of intensity variations (texture). This is because the minimization 170
Feature Extraction and Image Processing
(a) First image
(b) Second image
(c) Window size 3
(d) Window size 5
(e) Window size 11
Figure 4.44 Example of area-based motion computation
function in Equation 4.81 is almost flat and there is no clear evidence of the motion. In general, there is not a very effective way of handling these problems since they are caused by the lack of information in the image.
4.9.2 Differential approach Another popular way to estimate motion focuses on the observation of the differential changes in the pixel values. There are many ways of calculating the optical flow by this approach (Nagel, 1987; Barron et al., 1994). We shall discuss one of the more popular techniques (Horn and Schunk, 1981). We start by considering the intensity equity in Equation 4.79. According to this, the brightness at the point in the new position should be the same as the brightness at the old position. Like Equation 4.5, we can expand Pt + tx+xy+y by using a Taylor series as Pt + tx+xy+y = Ptxy + x
Ptxy Ptxy Ptxy + y + t + x y t
(4.83)
where contains higher order terms. If we take the limit as t → 0 then we can ignore as it also tends to zero, which leaves Pt + tx+xy+y = Ptxy + x
Ptxy Ptxy Ptxy + y + t x y t
Low-level feature extraction (including edge detection)
(4.84) 171
Now by Equation 4.79 we can substitute for Pt + tx+xy+y to give Ptxy = Ptxy + x
Ptxy Ptxy Ptxy + y + t x y t
(4.85)
which with some rearrangement gives the motion constraint equation x P y P P + =− t x t y t
(4.86)
We can recognize some terms in this equation. P x and P y are the first order differentials of the image intensity along the two image axes. P t is the rate of change of image intensity with time. The other two factors are the ones concerned with optical flow, as they describe movement along the two image axes. Let us call u = x t and v = y t. These are the optical flow components: u is the horizontal optical flow and v is the vertical optical flow. We can write these into our equation to give P P P =− (4.87) u +v x y t This equation suggests that the optical flow and the spatial rate of intensity change together describe how an image changes with time. The equation can be expressed more simply in vector form in terms of the intensity change P = x y = P x P y and the optical flow v = u v T , as the dot product •
P · v = − P
(4.88)
We already have operators that can estimate the spatial intensity change, x = P x and
y = P y, by using one of the edge detection operators described earlier. We also have an operator which can estimate the rate of change of image intensity, t = P t, as given by Equation 4.77. Unfortunately, we cannot determine the optical flow components from Equation 4.87 since we have one equation in two unknowns (there are many possible pairs of values for u and v that satisfy the equation). This is called the aperture problem and makes the problem ill-posed. Essentially, we seek estimates of u and v that minimize the error in Equation 4.90 over the entire image. By expressing Equation 4.87 as u x + v y + t = 0
(4.89)
we then seek estimates of u and v that minimize the error ec for all the pixels in an image !! u x + v y + t2 dxdy (4.90) ec = We can approach the solution (equations to determine u and v) by considering the second assumption we made earlier, namely that neighbouring points move with similar velocity. This is called the smoothness constraint as it suggests that the velocity field of the brightness varies in a smooth manner without abrupt change (or discontinuity). If we add this in to the formulation, we turn a problem that is ill-posed, without unique solution, to one that is well-posed. Properly, we define the smoothness constraint as an integral over the area of interest, as in Equation 4.90. Since we want to maximize smoothness, we seek to minimize the rate of change of the optical flow. Accordingly, we seek to minimize an integral of the rate of change of flow along both axes. This is an error es, as
! ! u 2 u 2 v 2 v 2 es = dxdy (4.91) + + + x y x y 172
Feature Extraction and Image Processing
The total error is the compromise between the importance of the assumption of constant brightness and the assumption of smooth velocity. If this compromise is controlled by a regularization parameter , then the total error e is e = × ec + es
2 2 2 2 !! P P P 2 u u v v = × u +v + dxdy + + + + x y t x y x y (4.92) There are several ways to approach the solution (Horn, 1986), but the most appealing is perhaps also the most direct. We are concerned with providing estimates of optical flow at image points. So we are interested in computing the values for uxy and vxy . We can form the error at image points, like esxy . Since we are concerned with image points, we can form esxy by using first order differences, just like Equation 4.1 at the start of this chapter. Equation 4.90 can be implemented in discrete form as 1 2 2 2 2 esxy = (4.93) ux+1y − uxy + uxy+1 − uxy + vx+1y − vxy + vxy+1 − vxy x y 4 The discrete form of the smoothness constraint is then that the average rate of change of flow should be minimized. To obtain the discrete form of Equation 4.92 we then add in the discrete of ec (the discrete form of Equation 4.90) to give 2 ecxy = uxy xxy + vxy yxy + txy (4.94) x
y
where xxy = Pxy x, yxy = Pxy y and txy = Pxy t are local estimates, at the point with coordinates x,y of the rate of change of the picture with horizontal direction, vertical direction and time, respectively. Accordingly, we seek values for uxy and vxy that minimize the total error e as given by exy = ×ecxy +esxy x
y
⎞ ⎛ 2 uxy xxy +vxy yxy + txy + × ⎝ 1 2 2 2 2 ⎠ = −u + u −u + v −v + v −v u x y x+1y xy xy+1 xy x+1y xy xy+1 xy 4 (4.95) Since we seek to minimize this equation with respect to uxy and vxy we differentiate it separately, with respect to the two parameters of interest, and the resulting equations when equated to zero should yield the equations we seek. As such, exy (4.96) = × 2 uxy xxy + vxy yxy + txy xxy + 2 uxy − uxy = 0 uxy and
exy = × 2 uxy xxy + vxy yxy + txy yxy + 2 vxy − vxy = 0 vxy
This gives a pair of equations in uxy and vxy
2 1 + xxy uxy + xxy yxy vxy = uxy − xxy txy
2 xxy yxy uxy + 1 + yxy vxy = vxy − xxy txy Low-level feature extraction (including edge detection)
(4.97)
(4.98)
173
This is a pair of equations in u and v with solution
2 2 2 uxy = 1+ yxy uxy − xxy yxy vxy − xxy txy 1+ xxy + yxy
2 2 2 1+ xxy + yxy vxy = − xxy yxy uxy + 1+ xxy vxy − yxy txy (4.99) The solution to these equations is in iterative form, where we shall denote the estimate of u at iteration n as u , so each iteration calculates new values for the flow at each point according to
u +
y v +
t
x xy xy xy xy xy
xxy = u u xy xy − 2 2 1 + xxy + yxy ⎞ ⎛ (4.100)
x u +
y v +
t xy xy xy ⎠ yxy ⎝ xy xy
vxy = v xy − 2 2 1 + xxy + yxy Now we have it, the pair of equations gives iterative means for calculating the images of optical flow based on differentials. To estimate the first order differentials, rather than use our earlier equations, we can consider neighbouring points in quadrants in successive images. This gives approximate estimates of the gradient based on the two frames. That is, P0x+1y + P1x+1y + P0x+1y+1 + P1x+1y+1 − P0xy + P1xy + P0xy+1 + P1xy+1
xxy = 8 (4.101) P0xy+1 + P1xy+1 + P0x+1y+1 + P1x+1y+1 − P0xy + P1xy + P0x+1y + P1x+1y
yxy = 8 In fact, in a later reflection (Horn and Schunk, 1993) on the earlier presentation, Horn noted with rancour that some difficulty experienced with the original technique had been caused by use of simpler methods of edge detection which are not appropriate here, as the simpler versions do not deliver a correctly positioned result between two images. The time differential is given by the difference between the two pixels along the two faces of the cube, as P1 xy + P1x+1y + P1xy+1 + P1x+1y+1 − P0xy + P0x+1y + P0xy+1 + P0x+1y+1 (4.102)
txy = 8 Note that if the spacing between the images is other than one unit, this will change the denominator in Equations 4.101 and 4.102, but this is a constant scale factor. We also need means to calculate the averages. These can be computed as ux−1y + uxy−1 + ux+1y + uxy+1 ux−1y−1 + ux−1y+1 + ux+1y−1 + ux+1y+1 u¯ xy = + 2 4 vx−1y + vxy−1 + vx+1y + vxy+1 vx−1y−1 + vx−1y+1 + vx+1y−1 + vx+1y+1 v¯ xy = + 2 4 (4.103) The implementation of the computation of optical flow by the iterative solution in Equation 4.100 is presented in Code 4.20. This function has two parameters that define the smoothing parameter 174
Feature Extraction and Image Processing
%Optical flow by gradient method %s = smoothing parameter %n = number of iterations function OpticalFlow(inputimage1,inputimage2,s,n) %Load images L1=double(imread(inputimage1, 'bmp')); L2=double(imread(inputimage2, 'bmp')); %Image size [rows,columns]=size(I1); %I2 must have the same size %Result flow u=zeros(rows,columns); v=zeros(rows,columns); %Temporal flow tu=zeros(rows,columns); tv=zeros(rows,columns); %Flow computation for k=1:n %iterations for x=2:columns-1 for y=2:rows-1 %derivatives Ex=(L1(y,x+1)-L1(y,x)+L2(y,x+1)-L2(y,x)+L1(y+1,x+1) -L1(y+1,x)+L2(y+1,x+1)-L2(y+1,x))/4; Ey=(L1(y+1,x)-L1(y,x)+L2(y+1,x)-L2(y,x)+L1(y+1,x+1) -L1(y,x+1)+L2(y+1,x+1)-L2(y,x+1))/4; Et=(L2(y,x)-L1(y,x)+L2(y+1,x)-L1(y+1,x)+L2(y,x+1) L1(y,x+1)+L2(y+1,x+1)-L1(y+1,x+1))/4; %average AU=(u(y,x-1)+u(y,x+1)+u(y-1,x)+u(y+1,x))/4; AV=(v(y,x-1)+v(y,x+1)+v(y-1,x)+v(y+1,x))/4; %update estimates A=(Ex*AU+Ey*AV+Et); B=(1+s*(Ex*Ex+Ey*Ey)); tu(y,x)= AU-(Ex*s*A/B); tv(y,x)= AV-(Ey*s*A/B); end%for (x,y) end %update for x=2:columns-1 for y=2:rows-1 u(y,x)=tu(y,x); v(y,x)=tv(y,x); end %for (x,y) end end %iterations %display result quiver(u,v,1); Code 4.20 Implementation of gradient-based motion
Low-level feature extraction (including edge detection)
175
and the number of iterations. In the implementation, we use the matrices u, v, tu and tv to store the old and new estimates in each iteration. The values are updated according to Equation 4.100. Derivatives and averages are computed by using simplified forms of Equations 4.101–4.103. In a more elaborate implementation, it is convenient to include averages, as we discussed in the case of single image feature operators. This will improve the accuracy and reduce noise. In addition, since derivatives can only be computed for small displacements, generally, gradient algorithms are implemented with a hierarchical structure. This will enable the computation of displacements larger than one pixel. Figure 4.45 shows some examples of optical flow computation. In these examples, we used the same images as in Figure 4.44. The first row in the figure shows three results obtained by different number of iterations and fixed smoothing parameter. In this case, the estimates converged quite quickly. Note that at the start, the estimates of flow in are quite noisy, but they quickly improve; as the algorithm progresses the results are refined and a more smooth and accurate motion is obtained. The second row in Figure 4.45 shows the results for a fixed number of iterations and a variable smoothing parameter. The regularization parameter controls the compromise between the detail and the smoothness. A large value of will enforce the smoothness constraint, whereas a small value will make the brightness constraint dominate the result. In the results we can observe that the largest vectors point in the expected direction, upwards, while some of the smaller vectors are not exactly correct. This is because there are occlusions and some regions have similar textures. We could select the brightest of these points
(a) Two iterations
(b) Four iterations
(c) Ten iterations
(d) λ = 0.001
(e) λ = 0.1
(f) λ = 10.0
Figure 4.45 Example of differential-based motion computation
176
Feature Extraction and Image Processing
by thresholding according to magnitude. That would leave the largest vectors (the ones that point in exactly the right direction). Optical flow has been used in automatic gait recognition (Little and Boyd, 1998; Huang et al., 1999), among other applications, partly because the displacements can be large between successive images of a walking subject, which makes the correlation approach suitable (note that fast versions of area-based correspondence are possible; Zabir and Woodfill, 1994). Figure 4.46 shows the resu