ADVANCES IN IMAGING AND ELECTRON PHYSICS VOLUME 151
EDITOR-IN-CHIEF
PETER W. HAWKES CEMES-CNRS Toulouse, France
HONORARY ASSOCIATE EDITORS
TOM MULVEY BENJAMIN KAZAN
Advances in
Imaging and Electron Physics
E DITED BY
PETER W. HAWKES CEMES-CNRS Toulouse, France
VOLUME 151
AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Academic Press is an imprint of Elsevier
Academic Press is an imprint of Elsevier 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK ∞ This book is printed on acid-free paper.
Copyright © 2008, Elsevier Inc. All Rights Reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the Publisher. The appearance of the code at the bottom of the first page of a chapter in this book indicates the Publisher’s consent that copies of the chapter may be made for personal or internal use of specific clients. This consent is given on the condition, however, that the copier pay the stated per copy fee through the Copyright Clearance Center, Inc. (www.copyright.com), for copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. Copy fees for pre-2008 chapters are as shown on the title pages. If no fee code appears on the title page, the copy fee is the same as for current chapters. 1076-5670/2008 $35.00 Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then “Obtaining Permissions.” For information on all Elsevier Academic Press publications visit our Web site at www.books.elsevier.com. ISBN-13: 978-0-12-374218-6
PRINTED IN THE UNITED STATES OF AMERICA 08 09 10 11 9 8 7 6 5 4 3 2 1
CONTENTS
C ONTRIBUTORS . . . . . . . . . . . . . . . . . . . . . . . . . . . P REFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F UTURE C ONTRIBUTIONS . . . . . . . . . . . . . . . . . . . . . .
vii ix xi
Reconstruction Algorithms for Computed Tomography C LAAS B ONTUS AND T HOMAS KÖHLER I. II. III. IV.
Introduction . . . . . . . . . . . . Principles of Computed Tomography CT Reconstruction . . . . . . . . . Outlook . . . . . . . . . . . . . . References . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
2 4 15 59 61
Color Spaces and Image Segmentation L AURENT B USIN , N ICOLAS VANDENBROUCKE AND L UDOVIC M ACAIRE I. II. III. IV. V.
Introduction . . . . . . . . . . . . . . . . . . . . . Color Spaces . . . . . . . . . . . . . . . . . . . . . Color Image Segmentation . . . . . . . . . . . . . . Relationships between Segmentation and Color Spaces Conclusion . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
66 66 110 140 161 162
Generalized Discrete Radon Transforms and Applications to Image Processing G LENN R. E ASLEY AND F LAVIA C OLONNA I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 170 II. Background on Wavelets . . . . . . . . . . . . . . . . . . . . 179 v
vi III. IV. V. VI. VII. VIII. IX.
CONTENTS
Beyond Wavelets . . . . . . . . . . . The Discrete p-Adic Radon Transform Generalized Discrete Radon Transform Noise Removal Experiments . . . . . Applications to Image Recognition . . Recognition Experiments . . . . . . . Conclusion . . . . . . . . . . . . . . References . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
190 197 204 217 221 230 231 235
. . . . . . . .
. . . . . . . .
. . . . . . . .
242 245 251 257 273 310 338 360
. . . . . . . .
. . . . . . . .
363 364 374 380 382 388 410 411
Lie Algebraic Methods in Charged Particle Optics ˇ T OMÁŠ R ADLI CKA
I. II. III. IV. V. VI. VII.
Introduction . . . . . . . . . . . . . . . . . . . . . . Trajectory Equations . . . . . . . . . . . . . . . . . . The Field Computation . . . . . . . . . . . . . . . . . Trajectory Equations: Solution Methods . . . . . . . . . The Analytic Perturbation Method . . . . . . . . . . . The Symplectic Classification of Geometric Aberrations Axial Symmetric Aberrations of the Fifth Order . . . . . References . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
Recent Developments in Electron Backscatter Diffraction VALERIE R ANDLE I. II. III. IV. V. VI. VII.
Introduction . . . . . . . . . . . . . . . Fundamental Aspects of EBSD . . . . . . The Orientation Map and Data Processing . Established Applications of EBSD . . . . Recent Advances in EBSD . . . . . . . . Advances in EBSD Technology . . . . . . Trends in EBSD Usage . . . . . . . . . . References . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
I NDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
CONTRIBUTORS
Numbers in parentheses indicate the pages on which the authors’ contributions begin.
C LAAS B ONTUS (1), Philips Research Europe–Hamburg, Sector Medical Imaging Systems, Röntgenstrasse 24–26, D-22 335 Hamburg, Germany L AURENT B USIN (65), Laboratoire LAGIS UMR CNRS 8146 – Bâtiment P2, Université des Sciences et Technologies de Lille, 59655 Villeneuve d’Ascq, France F LAVIA C OLONNA (169), Department of Mathematical Sciences, George Mason University, Fairfax, VA 22030, USA G LENN R. E ASLEY (169), System Planning Corporation, Arlington, VA 22209, USA T HOMAS KÖHLER (1), Philips Research Europe–Hamburg, Sector Medical Imaging Systems, Röntgenstrasse 24–26, D-22 335 Hamburg, Germany L UDOVIC M ACAIRE (65), Laboratoire LAGIS UMR CNRS 8146 – Bâtiment P2, Université des Sciences et Technologies de Lille, 59655 Villeneuve d’Ascq, France ˇ (241), Institute of Scientific Instruments AS CR, T OMÁŠ R ADLI CKA Královopolská 147, 612 64 Brno, Czech Republic
VALERIE R ANDLE (363), Materials Research Centre, School of Engineering, University of Wales Swansea, Swansea, UK N ICOLAS VANDENBROUCKE (65), Laboratoire LAGIS UMR CNRS 8146 – Bâtiment P2, Université des Sciences et Technologies de Lille, 59655 Villeneuve d’Ascq, France Ecole d’Ingénieurs du Pas-de-Calais (EIPC), Campus de la Malasisse – BP39, 62967 Longuenesse Cedex, France
vii
This page intentionally left blank
PREFACE The five chapters that make up this latest volume range over tomography, color image processing, charged particle optics and electron microscopy. We begin with a full and scholarly account of many of the algorithms used for computed tomography, by C. Bontus and T. Köhler. They take the reader systematically through the mathematical foundations and in particular, devote considerable space to the reconstruction algorithm developed by A. Katsevich for helical computed tomography and the various developments engendered by that work. In a second contribution from the same area, G.R. Easley and F. Colonna provide a very learned account of the generalized discrete Radon transform. After a reminder of the classical Radon transform, they introduce wavelets and their descendants—ridgelets, curvelets and shearlets. All these tools are then applied to the Radon transform and various applications are included. The introduction of color is a major complication of image processing and few of the traditional tools can be transferred immediately from black-andwhite images to color images. L. Busin, N. Vandenbroucke and L. Macaire describe the notion of color spaces and then examine in considerable detail the problem of segmentation of color images. Charged particle optics is still very much a developing subject, as the chapter by T. Radliˇcka shows. He reconsiders the various methods of calculating aberration coefficients, notably using differential algebra, a relatively new addition to the armoury, and examines their relative merits. A very interesting section is devoted to the symplectic classification. Finally, V. Randle provides an account of recent developments in electron backscatter diffraction, a subject on which she is an acknowledged expert. Although the technique is well established, there have been significant advances in the past five years and it is these that are presented here, with a wealth of illustrations. All the authors have taken great pains to ensure that their subjects can be understood by readers from other specialities, which is a feature of these Advances that I like to emphasize. I thank them here for all the trouble they have taken and conclude as always with a list of forthcoming contributions. Finally, just a reminder that these Advances are now available via ScienceDirect; the whole series can be consulted, Advances in Electronics and Electron Physics as well as the present title. Peter Hawkes ix
This page intentionally left blank
FUTURE CONTRIBUTIONS
S. Ando Gradient operators and edge and corner detection P. Batson (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy C. Beeli Structure and microscopy of quasicrystals A.B. Bleloch (special volume on aberration-corrected electron microscopy) Aberration correction and the SuperSTEM project C. Bobisch and R. Möller Ballistic electron microscopy G. Borgefors Distance transforms Z. Bouchal Non-diffracting optical beams F. Brackx, N. de Schepper and F. Sommen The Fourier transform in Clifford analysis A. Buchau Boundary element or integral equation methods for static and time-dependent problems B. Buchberger Gröbner bases T. Cremer Neutron microscopy xi
xii
FUTURE CONTRIBUTIONS
N. de Jonge and E.C. Heeres Electron emission from carbon nanotubes A.X. Falcão The image foresting transform R.G. Forbes Liquid metal ion sources C. Fredembach Eigenregions for image classification A. Gölzhäuser Recent advances in electron holography with point sources D. Greenfield and M. Monastyrskii Selected problems of computational charged particle optics M. Haider (special volume on aberration-corrected electron microscopy) Aberration correction in electron microscopy M.I. Herrera The development of electron microscopy in Spain N.S.T. Hirata (vol. 152) Stack filter design M. Hÿtch, E. Snoeck and F. Houdellier (special volume on aberrationcorrected electron microscopy) Aberration correction in practice J. Isenberg Imaging IR-techniques for the characterization of solar cells K. Ishizuka Contrast transfer and crystal images A. Jacobo Intracavity type II second-harmonic generation for image processing B. Kabius (special volume on aberration-corrected electron microscopy) Aberration-corrected electron microscopes and the TEAM project
FUTURE CONTRIBUTIONS
xiii
L. Kipp Photon sieves A. Kirkland and P.D. Nellist (special volume on aberration-corrected electron microscopy) Aberration-corrected electron microscopy G. Kögel Positron microscopy T. Kohashi Spin-polarized scanning electron microscopy O.L. Krivanek (special volume on aberration-corrected electron microscopy) Aberration correction and STEM R. Leitgeb Fourier domain and time domain optical coherence tomography B. Lencová Modern developments in electron optical calculations H. Lichte New developments in electron holography M. Matsuya Calculation of aberration coefficients using Lie algebra S. McVitie Microscopy of magnetic specimens P.G. Merli Scanning electron microscopy of thin films S. Morfu, P. Marquié (vol. 152) Nonlinear systems for image processing T. Nitta (vol. 152) Back-propagation and complex-valued neurons M.A. O’Keefe Electron image simulation
xiv
FUTURE CONTRIBUTIONS
D. Oulton and H. Owens Colorimetric imaging N. Papamarkos and A. Kesidis The inverse Hough transform K.S. Pedersen, A. Lee and M. Nielsen The scale-space properties of natural images S.J. Pennycook (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy E. Rau Energy analysers for electron microscopes E. Recami Superluminal solutions to wave equations H. Rose (special volume on aberration-corrected electron microscopy) The history of aberration correction in electron microscopy G. Schmahl X-ray microscopy R. Shimizu, T. Ikuta and Y. Takai Defocus image modulation processing in real time S. Shirai CRT gun design methods T. Soma Focus-deflection systems and their applications J.-L. Starck (vol. 152) Independent component analysis: the sparsity revolution I. Talmon Study of complex fluids by transmission electron microscopy N. Tanaka (special volume on aberration-corrected electron microscopy) Aberration-corrected microscopy in Japan
FUTURE CONTRIBUTIONS
xv
M.E. Testorf and M. Fiddy Imaging from scattered electromagnetic fields, investigations into an unsolved problem N.M. Towghi Ip norm optimal filters E. Twerdowski Defocused acoustic transmission microscopy Y. Uchikawa Electron gun optics K. Urban and J. Mayer (special volume on aberration-corrected electron microscopy) Aberration correction in practice K. Vaeth and G. Rajeswaran Organic light-emitting arrays M. van Droogenbroeck and M. Buckley Anchors in mathematical morphology R. Withers (vol. 152) Disorder, structured diffuse scattering and local crystal chemistry M. Yavor Optics of charged particle analysers Y. Zhu (special volume on aberration-corrected electron microscopy) Some applications of aberration-corrected electron microscopy
This page intentionally left blank
ADVANCES IN IMAGING AND ELECTRON PHYSICS, VOL. 151
Reconstruction Algorithms for Computed Tomography CLAAS BONTUS AND THOMAS KÖHLER Philips Research Europe–Hamburg, Sector Medical Imaging Systems, Röntgenstrasse 24–26, D-22 335 Hamburg, Germany
I. Introduction . . . . . . . . . . . . . II. Principles of Computed Tomography . . . . . A. X-Ray Attenuation . . . . . . . . . . B. Parameterization of the Measurements . . . . 1. Field of View and Region of Interest . . . . C. Detector Shapes . . . . . . . . . . 1. Basis Vectors . . . . . . . . . . D. Trajectories . . . . . . . . . . . . 1. Circular Trajectory . . . . . . . . . 2. Helical Trajectory . . . . . . . . . III. CT Reconstruction . . . . . . . . . . . A. Radon Transform . . . . . . . . . . 1. Inverse Radon Transform . . . . . . . 2. Fourier Slice Theorem . . . . . . . . B. Exact Filtered Backprojection . . . . . . 1. Derivative of the Projection Data . . . . . 2. Reconstruction Formula . . . . . . . 3. Mathematically Complete Trajectories . . . 4. Reconstruction Algorithm . . . . . . . 5. Filtering . . . . . . . . . . . . 6. Efficient Filtering . . . . . . . . . 7. Backprojection Using Wedge Detector Geometry C. Mathematics on the Planar Detector . . . . . D. Circle Plus Line . . . . . . . . . . 1. Geometrical Properties . . . . . . . 2. Filter Lines . . . . . . . . . . . 3. Proof of Exactness . . . . . . . . . 4. Reconstruction Results . . . . . . . E. The Katsevich Algorithm for Helical Pi Acquisition 1. Geometrical Properties . . . . . . . 2. Filter Lines . . . . . . . . . . . 3. Proof of Exactness . . . . . . . . . F. EnPiT: Helical n-Pi Reconstruction . . . . . 1. Reconstruction Algorithm . . . . . . . 2. Filter Lines . . . . . . . . . . . G. CEnPiT: Helical Cardiac CT . . . . . . . 1. Cardiac CT . . . . . . . . . . . 2. Filter Lines . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 4 4 5 6 7 10 10 10 12 15 15 16 18 19 19 20 23 24 25 27 31 32 34 35 36 38 40 42 42 43 46 49 50 51 52 52 56
1 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00401-6
Copyright 2008, Elsevier Inc. All rights reserved.
2
BONTUS AND KÖHLER
3. Gated and Ungated Contributions IV. Outlook . . . . . . . . . . References . . . . . . . . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
58 59 61
I. I NTRODUCTION Computed tomography (CT) is of vital importance for medical diagnosis, intervention planning, and treatment evaluation. CT yields three-dimensional (3D) tomographic images of large parts of the human body within a few seconds. The rather wide bore allows use of CT scans for obese patients and claustrophobia is rarely an issue. At the same time, the current pace of technological progress is unprecedented. Two major advances are important. First, X-ray detectors are larger, allowing scanning larger volumes in shorter times. Second, rotation speeds of the tube-detector system are steadily increasing. Although the technological progress is of great benefit for patients and medical staff, the engineering challenges are great. X-ray tubes must be able to withstand the centrifugal forces increasing with the rotation speed. These tubes also must be able to illuminate detectors of increasing sizes. Integration times of the detector elements become shorter. The amount of data measured per time unit increases with increasing rotation speed and detector sizes. All these data must be transmitted and processed in an appropriate amount of time. Data processing is separated into different steps: preprocessing, reconstruction and image processing. Preprocessing covers correction methods to treat disturbing effects and system imperfections, such as beam hardening, cross-talk, and afterglow. For example, beam hardening is a nonlinear effect resulting from the fact that X-rays experience weaker absorption with increasing photon energy. Image-processing methods cover all algorithms that are applied to the image data to assist radiologists in image analysis. Examples for image-processing methods are tools that automatically segment different organs, as well as computer-aided diagnosis (CAD) tools. This chapter covers the field of CT reconstruction, i.e., the mathematics to obtain the image data from the measurements. In other words, reconstruction deals with an inverse problem. Reconstruction algorithms must fulfill certain criteria to be useful in practice. They must be numerically stable and yield images of sufficient quality with respect to artefacts, resolution, and signal to noise ratio (SNR). Furthermore, reconstruction algorithms should be sufficiently fast so that images can be obtained in an adequate amount of time. Historically, most CT scanners were so-called single-row scanners. These scanners yield two-dimensional (2D) images of cross-sectional slices. The
RECONSTRUCTION ALGORITHMS
3
mathematics for single-row scanners is well understood. Modern cone-beam CT scanners, in which the detector consists of many rows, require completely new reconstruction algorithms. The first reconstruction algorithms proposed for cone-beam scanners were extensions of 2D methods. These algorithms, which are of approximative nature, yield good image quality as long as the number of detector rows does not become too large. Once the number of rows becomes large, more effort is necessary and the underlying mathematics must be analyzed to develop more sophisticated algorithms. In 2002 Alexander Katsevich (2002) published the first reconstruction algorithm for helical CT, which is mathematically exact, of the filtered backprojection (FBP) type, and in which the filtering is shift invariant. In particular, the exactness guarantees that remaining artefacts originate from the discrete reformulation of the inversion formulas. This discretization is required by the discrete sampling of the projection data. FBP, in combination with the shift invariance of the underlying filter, ensures that the algorithm can be implemented in an efficient way. Katsevich’s work resulted in many publications analyzing the underlying mathematics or extending the algorithm to different settings. The chapter is devoted to reconstruction algorithms of the FBP type that are derived from an exact method. The underlying mathematics is discussed on a broad basis. Nevertheless, more effort is placed on descriptive discussions than on waterproof mathematical analyses. Section II provides a short summary of the principles. The measurement process as well as different kinds of detectors and trajectories are discussed. Section III covers the field of reconstruction in several subsections, beginning with the Radon transform in Section III.A. As it turns out, the mathematics associated with the discussed algorithms can be separated into two parts. While Sections III.B and III.C deal with the formulas that are independent of the particular system settings, the subsequent sections specify the parameters for different types of acquisitions. The mathematics of the circle and line trajectory (CLT) can be understood readily. This trajectory, therefore, serves as the first example, covered in Section III.D. The Katsevich algorithm for a socalled Pi acquisition is covered in Section III.E, while Section III.F deals with the so-called EnPiT algorithm. The EnPiT algorithm can be considered an extension of the Katsevich method useful for more freely choosing the patient table feed. Cardiac CT is one of the most important applications for modern CT scanners. The CEnPiT algorithm, which was developed for cardiac cone-beam CT based on a helical acquisition, is briefly summarized in Section III.G.
4
BONTUS AND KÖHLER
II. P RINCIPLES OF C OMPUTED T OMOGRAPHY A. X-Ray Attenuation Figure 1 shows a modern CT scanner. Within such a CT system a tube emits X-rays in the direction of a detector. The X-rays penetrate a certain part of the patient’s body before they enter the detector. During the scan the tube-detector system (termed as the gantry) rotates around the patient (see Figure 2) so that projections from a large number of directions are taken. The patient lies on a patient table, that can be moved during the scan. In this manner, a relative motion of the patient with respect to the rotation plane can be realized. In practice, this is used, for example, to obtain a relative trajectory in which the X-ray tube moves on a helix around the patient. Consider the X-ray tube itself. Let us denote the flux density of X-ray photons with energy E emitted into a certain direction by I0 (E). Typical units for the photon flux density are [I ] = 1/s sr keV; that is I (E) is the number of photons emitted within each second per solid angle and per photon energy. Some X-ray photons are scattered or absorbed when penetrating the patient. The number of photons, that pass through the patient (not scattered and not absorbed) is given by the Beer–Lambert law: (1) I (E) = I0 (E) exp − d μ x(), E .
F IGURE 1.
The Philips Brilliance 64 is a modern CT scanner with 64 detector rows.
RECONSTRUCTION ALGORITHMS
5
F IGURE 2. (a) The tube-detector system rotates around the patient during a CT scan. (b) The patient table can be moved so that a helical trajectory can be realized.
Here, x() parameterizes the path along which the X-rays travel. The scalar function μ(x) are the so-called absorption coefficients. Function μ(x) is also known as the object function. Certainly, the values of μ are different for different kinds of tissue. The aim of reconstruction is to recover the μ-values from the measurements. From Eq. (1) we easily obtain d μ x(), E = ln I0 (E)/I (E) .
(2)
In other words, a monochromatic CT system measures line integrals. Unfortunately, X-rays emitted from today’s tubes have a wide spectrum and the detectors average over the photon energies. Therefore, some preprocessing steps are necessary before the measurements can approximately be interpreted as line integrals. B. Parameterization of the Measurements The following sections assume that the preprocessing steps yield data, which can be interpreted as line integrals in the sense of Eq. (2) with E = E0 , where E0 is some mean energy. We choose a coordinate system, which is fixed to the patient. At a particular time the X-ray source will be at a point y with respect to this coordinate system. The line integrals associated with the measurements at this point in time all originate from y. Terms source position and focal spot are used synonymously throughout this chapter and correspond to vectors y.
6
BONTUS AND KÖHLER
Each line integral can be parameterized as ∞ d μ(y + θ),
D(y, θ) =
(3)
0
where θ is a unit vector pointing into the direction of the line integral. The different values of θ are determined by the locations of the different detector elements. If the patient table is fixed during the scan (see Figure 2), the source moves on a circular trajectory with respect to the coordinate system defined above. This trajectory can be parameterized as R cos s (4) y ◦ (s) = R sin s , z0 where R corresponds to the distance from the source to the rotation axis. We use the angular variable s to parameterize points on the trajectory. Each measurement belongs to a certain s. The set of measured values, i.e., all detector values, belonging to a certain s is denoted as a projection. For a helical trajectory, the positions of the source can be described with the help of the vector R cos s (5) y H (s) = R sin s , s h¯ where h = 2π h¯ corresponds to the pitch—the table feed per rotation. Finally, a line parallel to the z-axis is given by R (6) y | (z) = y | (hs) = 0 , z where h is an arbitrary constant, which was introduced to make y depend on the angular variable s. A linear trajectory according to Eq. (6) can be realized if the tube-detector system does not rotate but the patient table moves (compare with Figure 2). 1. Field of View and Region of Interest As discussed above, the aim of CT reconstruction is to recover the absorption coefficients μ from the measurements. The question of whether a certain object point x is reconstructible—if its μ value can be determined—is of particular interest. This question is related to the set of directions from which x is illuminated by X-rays. The set of all object points, which are
RECONSTRUCTION ALGORITHMS
7
reconstructible, is denoted as the field of view (FOV). The region of interest (ROI) is the set of all object points the user wants to be reconstructed. Some approximate algorithms yield absorption coefficients even for object points that do not belong to the FOV. These algorithms provide good approximation for object points that are not reconstructible in a strict sense. In other words, the ROI can be a subset or a superset of the FOV. C. Detector Shapes Detectors in conventional CT scanners are so-called focus-centered detectors. These detectors have a cylindrical shape such that the symmetry axis of the cylinder is parallel to the z-axis and contains the focal spot. The focal spot is the point within the X-ray tube from which the X-rays originate. Points on the focus detector are parameterized using the fan-angle variable ϕ and the coordinate vF as shown in Figure 3. Our conventions are such that the point (ϕ = 0, vF = 0) corresponds to the center of the detector. Physical detectors consist of detector elements, such that the detector coordinates used to describe the measurements take only discrete values. In particular, these values are ϕk = ϕ0 + kϕ, vFp = vF0 + pv,
k = 0, . . . , #columns − 1,
(7)
p = 0, . . . , #rows − 1,
(8)
which defines the notion of detector columns and rows. For convenience we introduce two virtual detectors—the planar and the parallel (wedge) detector. Data on these virtual detectors can be obtained by rebinning steps using the focus-detector data. In our convention, the planar
F IGURE 3. Points on the focus-centered detector are parameterized by the fan-angle variable ϕ and coordinate vF .
8
BONTUS AND KÖHLER
F IGURE 4.
Points on the planar detector are parameterized by the variables uP and vP .
detector contains part of the rotation axis. Points on the planar detector are parameterized using the coordinates uP and vP as shown in Figure 4. Focusand planar-detector coordinates are related by uP = R tan ϕ and vP =
R 2 + u2P tan λ = R
vF R tan λ = , cos ϕ cos ϕ D
(9)
tan λ =
vF , D
(10)
where D is the distance from the source to the center of the focus detector, and λ is the cone angle. The wedge detector is illustrated in Figure 5. It has an extended source. In particular, from each source position along the circle or helix, only data from one virtual focus-detector column enter into a wedge-detector projection. A virtual column is a column located somewhere between physical columns. Data along the virtual column can be obtained by interpolation of data from physical columns. For a helical trajectory the wedge detector is bent along the z-direction; that is its lower and upper boundaries have the slope of the helix. Details are provided in the following text. The wedge detector coordinates are designated as u and v , as shown in Figure 6. Furthermore, we parameterize different wedge-detector projections by the angular variable t similar to s, which parameterizes different focusdetector projections [compare with Eqs. (4) and (5)]. In particular, t is the angle enclosed by the parallel fans shown in Figure 5 and the x-axis. The angle ϕ is measured in a plane parallel to the xy-plane. Therefore, the line
RECONSTRUCTION ALGORITHMS
9
F IGURE 5. For the parallel (wedge) detector data from different source positions must be combined. Data belonging to a particular column of the wedge detector originate from one column of the focus detector at a specific source position. The right image shows the wedge detector on the left seen from above.
F IGURE 6.
Points on the wedge detector are parameterized by the variables u and v .
(intersecting with the rotation axis) used to measure that angle in Figure 6 does not intersect with the center of the wedge detector. The relationships between focus- and wedge-detector coordinates are s = t + ϕ,
(11)
u = R sin ϕ
(12)
and v vF = . (13) D R These equations should be interpreted as follows. For given coordinates (t, u , v ) compute ϕ according to Eq. (12). Now, Eq. (11) indicates from which source positions the data must be taken. Finally, the required row on the focus detector can be computed using Eq. (13). Note that our conventions are such that constant v means constant vF . In other words, data belonging tan λ =
10
BONTUS AND KÖHLER
to a row of the wedge detector are taken from the same row but different projections of the focus detector. Therefore, when performing the rebinning from focus to wedge data, interpolations are necessary in ϕ and s but not in vF . This explains the bending of the wedge detector in the helical case above. 1. Basis Vectors For the subsequent discussion it is useful to introduce an orthonormal basis, which is stationary on the planar detector. We define basis vectors u(s), v(s), and w(s) such that (14) x = uP u + vP + yz (s) v gives the world coordinates of point (uP , vP ). The term proportional to yz (s) (which is the z-component of vector y) in Eq. (14) is necessary, because the center of the planar detector moves with the source along z. A point x is located on the planar detector (at least if we consider the detector to have infinite extents), if x · w = 0. In this case, (uP = x · u, vP = x · v − yz (s)) gives the planar-detector coordinates of x. For the cylindrical trajectories given by Eqs. (4) and (5), the explicit form of the basis vectors are − sin s 0 cos s u(s) = v(s) = 0 , w(s) = sin s . cos s , 0 1 0 (15) D. Trajectories 1. Circular Trajectory Let us have a closer look at the circular trajectory, Eq. (4). For later reference we consider the stereographic projection of the circle onto the planar detector seen from a source at y | (z1 ), which is not necessarily in the circle plane [compare with Eq. (6)]. The following derivation aims at obtaining Eq. (22). We describe the line containing the focal spot and the point y ◦ (s) via R cos s R (s, λ) = (1 − λ)y | (z1 ) + λy ◦ (s) = (1 − λ) 0 + λ R sin s , z0 z1 (16) where λ parameterizes points on the line. See Figure 7 for an illustration. The just-defined line intersects with the planar detector, if 0 = · w = (1 − λ0 )R + λ0 R cos s,
(17)
RECONSTRUCTION ALGORITHMS
F IGURE 7. planar detector.
11
The line connecting the focal spot at y and a point on the circle intersects with the
where λ0 parameterizes the point of intersection. In the latter equation w is the basis vector defined in Section II.C normal to the planar detector. Therefore, w = (1, 0, 0) in our case. Solving for λ0 , we obtain λ0 =
1 1 = . 1 − cos s 2 sin2 s/2
(18)
The planar-detector coordinates of the point of intersection are obtained using basis vectors u = (0, 1, 0) and v = (0, 0, 1) sin s
s = R cot , 2 2 sin s/2 z0 − z1 vP (s) = (s, λ0 ) · v − z1 = λ0 (z0 − z1 ) = . 2 sin2 s/2 uP (s) = (s, λ0 ) · u = λ0 R sin s = R
2
(19) (20)
These equations are still parameterized by s. We solve Eq. (19) for s. Inserting the result into Eq. (20) and using sin2 arctan
R R2 = 2 uP uP + R 2
(21)
provides vP (uP ) =
u2 z0 − z 1 1 + P2 . 2 R
(22)
The latter equation parameterizes the projection of the circle at z = z0 onto the planar detector as a function of uP . For symmetry reasons this equation is valid for all source positions at z = z1 at the surface of the cylinder with radius R.
12
BONTUS AND KÖHLER
2. Helical Trajectory The key for exact reconstruction of helical CT data is the Pi window or Tam– Danielsson window. It is related to the minimum amount of data, that need to be acquired to perform an exact reconstruction (Danielsson et al., 1997; Tam, 1995). The generalization of the Pi window is the so-called n-Pi window, where n is an odd integer (Proksa et al., 2000). Figure 8 depicts the surfaces enclosing the points belonging to the Pi and 3-Pi window associated with a certain source position. These surfaces contain the focal spot and points on the helix. In particular, for the n-Pi window the points on the helix defining the lower surface fulfill the relation s0 − (n + 1)π < s < s0 − (n − 1)π,
(23)
where s0 parameterizes the current source position [see Eq. (5)]. Similarly, for the upper surface s0 + (n − 1)π < s < s0 + (n + 1)π.
(24)
We will now compute the projection of the n-Pi window boundaries onto the planar detector. The derivation is very similar to the treatment of the circular trajectory above. It results in Eq. (32). In particular, the line containing the current source position at y H (s0 ) and another point on the helix at y H (s) is parameterized by R cos s R cos s0 (25) (s, λ) = (1 − λ) R sin s0 + λ R sin s . h¯ s0 h¯ s
F IGURE 8. Left: The Pi window contains object points enclosed by the shown surfaces. These surfaces contain the focal spot and parts of the helical trajectory. Right: For the 3-Pi window the next-to-next turns of the helix determine the enclosing surfaces.
13
RECONSTRUCTION ALGORITHMS
This line intersects with the planar detector, if 0 = (s, λ0 ) · w(s0 ) = R + λ0 R(cos s − 1),
s = s − s0 ,
(26)
where we used the expression for w in Eq. (15). Solving for λ0 , we obtain λ0 =
1 1 = . 2 1 − cos s 2 sin (s/2)
(27)
Coordinates uP and vP of the point of intersection are obtained via uP (s) = (s, λ0 ) · u(s0 ) = R
sin s 2
= R cot
2 sin (s/2) hs ¯ vP (s) = (s, λ0 ) · v(s0 ) − h¯ s0 = . 2 sin2 (s/2)
s , 2
(28) (29)
Now, we introduce angle ¯s , such that ¯s = s + (n + 1)π for the lower n-Pi window boundary, and ¯s = s − (n − 1)π for the upper boundary (compare with Figure 8). Comparing with Eqs. (23) and (24), we realize that 0 < ¯s < 2π. Using ¯s in Eq. (28) and making use of the symmetry of the tan function, we obtain uP ¯s = arccot . 2 R From Eq. (29) we determine
uP 2 ¯s − (n + 1)π, lower boundary, vP = h¯ 1 + ¯s + (n − 1)π, upper boundary R
(30)
(31)
where we inserted Eq. (30) into the denominator of Eq. (29) and used Eq. (21). Now, using Eq. (30) again and making use of arccot uP /R = π/2 − arctan uP /R, we derive uP π uP 2 up,low . n ∓ arctan vP,n (uP ) = ±h¯ 1 + R 2 R
(32)
The latter equation gives the upper and lower boundaries of the n-Pi window on the planar detector. Figure 9 shows the corresponding curves for cases n = 1, 3, and 5. For later reference, the projections of the lower and upper boundaries of the focus detector result in 2 2 H uP + R F , (33) (uP ) = ± vP± 2 D where H is the height of the focus detector.
14
BONTUS AND KÖHLER
F IGURE 9. The Pi window (solid), 3-Pi window (dashed), and 5-Pi window (dash-dotted) boundaries on the planar detector.
The introduction of the Pi and n-Pi window gives rise to the definition of the Pi and n-Pi segments. These segments are parts of the helix associated with each object point. We start with the Pi segment and consider the projection of a certain object point x onto the planar detector. From each source position y H (s), x is either projected inside or outside the Pi window. Now, the Pi segment is the union of all points on the helix from which x is projected into the Pi window. In other words, y H (s) belongs to the Pi segment of x, if up
low (uP ) vP vP,1 (uP ). vP,1 low Here, (uP , vP ) are the coordinates of the projected object point x, and vP,1 up and vP,1 are the Pi-window boundaries according to Eq. (32). For each object point two angles s1 and s2 can be found, such that y H (s) belongs to the Pi segment of x, if and only if s1 s s2 (Defrise, Noo and Kudo, 2000). Points y H (s1 ) and y H (s2 ) are sometimes denoted as sunrise and sunset. The three points x, y H (s1 ), and y H (s2 ) are located on a straight line, since x is projected onto the Pi-window boundary, i.e., onto the helix, from the points of sunrise and sunset (Figure 10). Interval [s1 , s2 ] is denoted as the Pi interval of object point x. The line containing y H (s1 ) and y H (s2 ) is called the Pi-line of x. Obviously, all points located on that line share the same Pi-line and Pi segment. The situation is similar but slightly more complicated for the n-Pi window. Every point y H (s) belongs to the n-Pi segment of x, if x is projected between the lower and upper n-Pi window boundaries. Now, for some object points the n-Pi segment is not a single segment. In fact, the n-Pi segment can consist of up to n subsegments. Therefore, some object points experience a sunrise and a sunset up to n times along the helix, a phenomenon known as interrupted illumination. See Figure 11 for an example. Similar to the definition of the
RECONSTRUCTION ALGORITHMS
15
F IGURE 10. The Pi segment of the exemplified object point x along the helix is printed bold. From each point y H (s) within the Pi segment, x is located between the Pi-window surfaces shown in Figure 8.
F IGURE 11. Left: The bold points denote the 3-Pi segment of the exemplified object point. Right: Projection of the object point onto the planar detector from different source positions results in the shown trace on the detector. Sunrise or sunset occur if the object point is projected onto the boundaries of the 3-Pi window.
Pi-line, an n-Pi line contains a point of sunrise, a point of sunset, and x. The number of n-Pi lines can reach the value of n if interrupted illumination occurs.
III. CT R ECONSTRUCTION A. Radon Transform The Radon1 transform is an important tool for the analysis and development of reconstruction algorithms. The 3D Radon transform is the set of all plane integrals through a given object function. We parameterize Radon planes by normal vectors ω and scalars ρ, where ρ is the shortest distance of the plane to the origin (Figure 12). With these conventions the Radon transform of object 1 Johann Radon, Austrian mathematician, 1887–1956.
16
BONTUS AND KÖHLER
F IGURE 12. An exemplary Radon plane with normal vector ω. The dashed line has the length ρ, which is the shortest distance from the origin to the plane. Points x on the plane fulfill the relation ω · x = ρ.
function μ(x) is given by ∞
Rμ(ρ, ω) =
d3 x μ(x)δ(ρ − ω · x).
(34)
−∞
1. Inverse Radon Transform The inverse of the Radon transform can be obtained via the following formula (Natterer, 1986) −1 μ(x) = 8π 2
π
2π dϑ
0
dα sin α R μ(ω · x, ω),
(35)
0
where ω = (cos α sin ϑ, sin α sin ϑ, cos ϑ)
(36)
and
R μ(ρ, ω) =
∂ 2 Rμ(ρ, ω) ∂ρ 2
(37)
is the second derivative of the Radon transform with respect to ρ. The interpretation of Eq. (35) is as follows. In standard convention the (scalar) value associated with one particular Radon plane enters Radon space at point ρω. Now, for the evaluation of Eq. (35) all Radon values are taken into account for which ρ = ω · x, i.e., all Radon planes are considered, which contain x.
RECONSTRUCTION ALGORITHMS
17
F IGURE 13. Two-dimensional cut parallel to ω and containing x and the origin O. The Radon plane R is a straight line in this view, since ω is normal to R.
We show that all Radon values that must have to be taken into account are located on the surface of a sphere in Radon space (Figure 13). This figure shows a 2D cut parallel to ω containing x and the origin O. The Radon plane R defined by ρ and ω is perpendicular to this view since ω is orthogonal to R. The Radon plane contains x since ρ = ω · x for the evaluation of Eq. (35). Obviously |x| cos ε = ω · x = ρ.
(38)
We compute the distance r measured from x/2 to ρω with the help of the law of cosine: 2 1 1 |x| + ρ 2 − 2 |x|ρ cos ε. (39) r2 = 2 2 Exchanging |x| cos ε by ρ yields r = |x|/2, which is independent of ρ and ω. In other words, the Radon values required in Eq. (35) are located on the surface of a sphere with radius r. The center of this Radon sphere is located at x/2 as illustrated in Figure 14. As discussed previously, a CT scanner measures line integrals, such that Eq. (35) cannot be applied directly. Grangeat (1991) found a relation between the first derivative of the Radon transform and cone-beam projections. Based on this relation, different authors proposed reconstruction algorithms for the helical trajectory, which can also be applied to CT data with projections truncated along z (Kudo, Noo and Defrise, 1998; Schaller et al., 2000; Tam et al., 1999). Nevertheless, reconstruction algorithms based on direct Radon inversion tend to be inefficient and numerically unstable. The theory of Radon inversion will be used in the sequel for the derivation of filtered backprojection (FBP) algorithms. These FBP algorithms are numerically very efficient and yield results in which artefacts result only from the discrete implementation.
18
BONTUS AND KÖHLER
F IGURE 14. Radon values located on the surface of a sphere are required for the evaluation of the inverse Radon transform at x according to Eq. (35). The center of the Radon sphere is located at x/2 and the radius of the sphere is equal to |x/2|.
2. Fourier Slice Theorem The Fourier slice theorem gives an important relation between the Radon transform and the object function in the Fourier domain. For its derivation we consider Eq. (34) and perform the Fourier transform along ρ: ∞
FRμ(ρ, ˜ ω) =
dρ exp(−2π iρρ) ˜ Rμ(ρ, ω) −∞ ∞
=
∞ dρ exp(−2π iρρ) ˜
−∞ ∞
=
d3 x μ(x)δ(ρ − ω · x)
−∞
d3 x μ(x) exp(−2πiρω ˜ · x),
(40)
−∞
where in the last step the δ-function was evaluated for the integration over ρ. Now, introducing the 3D Fourier transform of the object function ∞
F μ(x, ˜ y, ˜ z˜ ) =
d3 x exp −2πi(x x˜ + y y˜ + z˜z) μ(x, y, z)
(41)
−∞
and using Eq. (36) yields
FRμ(ρ, ˜ ω) = F μ(x, ˜ y, ˜ z˜ )| x= ˜ ρ˜ cos α sin ϑ . y= ˜ ρ˜ sin α sin ϑ z˜ =ρ˜ cos ϑ
(42)
RECONSTRUCTION ALGORITHMS
19
The latter equation is known as the Fourier slice theorem. It builds a relationship between the Fourier transform of the object function and the Radon transform. Consider a Radon plane, which is perpendicular to the z-axis, i.e., ϑ = 0. According to Eq. (42) we have to set x˜ = y˜ = 0 for these planes. In other words, planes perpendicular to the z-axis contribute only to vanishing frequency components along xy. Next, consider a Radon plane, that is nearly perpendicular to the z-axis, i.e., ϑ is small. The maximum value of ρ˜ is of the order of the largest frequency necessary to describe the object function (or the Nyquist frequency). Therefore, ρ˜ sin ϑ is much smaller than this frequency and x˜ and y˜ are small. In conclusion, Radon planes, which are nearly perpendicular to the z-axis, contribute to low-frequency components of trans-axial slices only.
B. Exact Filtered Backprojection Reconstruction algorithms discussed in this chapter share the basic principles. The formulas describing these principles are described in this section before discussion in subsequent sections of how these equations can be applied to different trajectories. 1. Derivative of the Projection Data The derivative of the projection data is of particular interest. Specifically, we consider the derivative of the line integral data [compare with Eq. (3)] with respect to s: ∂D(y(q), θ) . D y(s), θ = (43) ∂q q=s Notice that (since θ is constant) the evaluation of the differentiation requires the consideration of parallel rays from different source positions. Therefore, the differentiation can be computed by considering data along the rows of the wedge detector. Using Fourier filtering techniques, differentiated data along a complete wedge-detector row can be obtained in a single processing step. Furthermore, Fourier techniques allow a modification of the filter in Fourier space. The latter can, for example, be used to suppress the high and emphasize the low frequencies. On the other hand, the Fourier filtering approach requires two rebinning steps. Data must be rebinned from focus to wedge geometry before filtering
20
BONTUS AND KÖHLER
and back to focus geometry afterward. The determination of the derivative via discrete differences is computationally more efficient. A smart way to compute the derivative using discrete differences was published by Noo, Pack and Heuscher (2003). 2. Reconstruction Formula We assume that the trajectory is described by a vector y(s) similar to Eqs. (4), (5), or (6). Let us introduce the unit vector b(s, x) pointing from y(s) to the object point at x: b(s, x) =
x − y(s) . |x − y(s)|
(44)
Furthermore we consider a set of unit vectors eν (s, x) and a set of corresponding constants (weights) μν , ν = 1, . . . , Ne . Vectors eν and weights μν are trajectory-specific (or algorithm specific) reconstruction parameters. They must be chosen properly, as will become evident below. Generally, the number of e-vectors, Ne , can depend on s. Each e-vector is supposed to be orthogonal to b(s, x), such that each pair b(s, x) and eν (s, x) spans a plane containing x and y. We denote these planes as κ-planes. See Figure 15 for an illustration. We introduce the backprojection interval IBP (x) and the backprojection segment CBP (x). For instance, IBP (x) can correspond to the Pi interval introduced in Section II.D. The backprojection interval defines projections, i.e., s-values, which are taken into account for the reconstruction of μ(x). Similarly, the backprojection segment is the set of all points along the trajectory associated with IBP (x). Therefore,
CBP (x) = y(s) | s ∈ IBP (x) . (45)
F IGURE 15. Vectors b and e span a plane containing the object point x and the focal spot y. We denote these planes as κ-planes. κ-Planes must not be confused with Radon planes.
RECONSTRUCTION ALGORITHMS
The reconstruction formula derived in this section is I (s, x) −1 , ds μ(x) ˜ = |x − y(s)| 2π 2
21
(46)
IBP (x)
where I (s, x) =
Ne
π μν
ν=1
−π
dγ D y(s), cos γ b + sin γ eν sin γ
(47)
D
is given by Eq. (43). Eq. (46) corresponds to an exact reconstruction and algorithm, if the reconstruction parameters eν (s, x) and μν have been chosen properly. For the derivation of Eq. (46), we evaluate Eq. (3) at θ(γ ) = cos γ b(s, x) + sin γ eν (s, x),
(48)
where −π γ π. Inserting Eq. (35) we obtain D(y, cos γ b + sin γ eν ) −1 d dΩ = R μ ω · y + (cos γ b + sin γ e ) ,ω , ν 8π 2 where dΩ = dϑ sin α dα. Eq. (47) becomes
(49)
Ne dγ −1 d dΩ μν I (s, x) = sin γ 8π 2 ν=1
∂ R μ ω · y(q) + (cos γ b + sin γ eν ) , ω q=s ∂q by insertion of Eq. (49). Inserting a δ-function δ ω · y(q) + (cos γ b + sin γ eν ) − ξ ∞ = dρe2πiρ(ω·[y(q)+(cos γ b+sin γ eν )]−ξ )
(50)
(51)
−∞
we make
R μ
depend on ξ
I (s, x) =
Ne dγ −1 d dΩ dρ dξ μ ν sin γ 8π 2 ν=1 ∂ 2πiρ(ω·[y(q)+(cos γ b+sin γ eν )]−ξ ) e R μ(ξ, ω). ∂q q=s
(52)
22
BONTUS AND KÖHLER
Next, we make a change of variables u1 = cos γ , u2 = sin γ , such that ddγ du1 du2 ddγ = = . (53) sin γ u2 u2 Using this, the integrations over and γ yield (Jeffrey, Gradshteyn and Ryzhik, 1994) du2 du1 exp 2π iρω · (u1 b + u2 eν ) u2 = iπ sgn(ρω · eν )δ(ρω · b) = iπ sgn(ω · eν )δ(ω · b)/ρ = iπ x − y(s) sgn(ω · eν )δ ω · x − y(s) /ρ,
(54)
where the last step follows directly from Eq. (44). The derivative with respect ˙ to q in Eq. (52) yields a factor 2π iρω · y(s), and together with Eq. (54) we obtain Ne −1 ˙ dΩ dρ dξ 2π iρω · y(s) iπ x − y(s) μ I (s, x) = ν 8π 2 ν=1 sgn(ω · eν )δ ω · x − y(s) /ρe2πiρ(ω·y(s)−ξ ) R μ(ξ, ω) Ne 1 ˙ sgn(ω · eν ) = x − y(s) μν dΩω · y(s) 4 ν=1 δ ω · x − y(s) R μ(ω · x, ω), (55) where in the last step the integrations over ρ and ξ were performed. We made R μ depend on ω·x using the δ-function under the integral. For the evaluation of the δ-function, remember that ω is the normal vector of a Radon plane, which contains the object point x. Vector (x − y(s)) points from the focal spot to x. Therefore, the argument of the δ-function vanishes exactly at those points y(sk˜ ), at which the Radon plane intersects with the trajectory. Using the scaling property of the δ-function, we obtain δ(s − sk˜ ) , (56) δ ω · x − y(s) = ˙ |ω · y(s)| k˜
where the sum runs over all intersection points sk˜ of the trajectory with the Radon plane defined by x and normal vector ω. The main result of this section is obtained by integration of I (s, x) over s as in Eq. (46). It yields −1 dΩ R μ(ω · x, ω)w(x, ω), (57) μ(x) ˜ = 8π 2
RECONSTRUCTION ALGORITHMS
23
F IGURE 16. The backprojection segment CBP (x) associated with IBP (x) for the object point x is drawn in bold along the helix. The Radon plane defined by x and ω has three intersection points with the helix: y(s1 ), y(s2 ), and y(s3 ). It has only one intersection point, y(s2 ), with CBP (x).
where w(x, ω) =
NI N e (k) k
˙ k ) sgn(ω · eν ). μν sgn ω · y(s
(58)
ν=1
The sum over k is different from the sum over k˜ in Eq. (56). The reason for this is that the integration over s was performed over IBP (x). Therefore, all sk fulfill sk ∈ IBP (x). In other words, k counts all intersection points (IPs) of the Radon plane (associated with ω) with the backprojection segment CBP (x) along the trajectory, whereas k˜ counts all IPs of the Radon plane with the complete trajectory (Figure 16). As mentioned, the number of eν -vectors, which we define, can depend on s. Therefore, Ne depends on k in Eq. (58). In the following we denote Radon planes as m-planes, if they have m IPs with CBP (x). Comparing Eqs. (35) and (57), we realize that Eq. (57) is the inverse Radon transform if and only if w(x, ω) is identical to one independent of x and ω. Therefore, if we define all eν and μν properly, such that this requirement is fulfilled, we derive a reconstruction algorithm that is mathematically exact. Subsequent sections are devoted to adequate definitions of these variables for different cases. First, we describe the general scheme of the reconstruction algorithm and show how it can be implemented efficiently. Section III.C shows how Eq. (58) can be evaluated using projections onto the planar detector. 3. Mathematically Complete Trajectories The foregoing discussion showed that we want to define the parameters of our reconstruction algorithms in such a way that Eq. (58) results in a constant value of 1. Here we elaborate on the requirements that the trajectory must fulfill. The result is known as the Tuy condition (Tuy, 1983).
24
BONTUS AND KÖHLER
F IGURE 17. Every Radon plane intersects at least once with the helix (left). The circular trajectory is incomplete for object points located outside of the circle plane (right).
Remember that k in Eq. (58) counts the number of IPs of the Radon plane with the backprojection segment CBP (x). Therefore, if Radon planes exist, which do not intersect with CBP (x), Eq. (58) will always give a vanishing result for these planes. In other words, an object point x is reconstructible only if every Radon plane containing x intersects with CBP (x) at least once. Trajectories fulfilling this requirement for all x in the ROI are denoted complete. Figure 17 illustrates completeness and incompleteness using the helical and circular trajectories as examples. In practice, an even a stronger restriction exists. The argumentation so far neglects the finite size of the detector. Taking the detector dimensions into account, a certain trajectory is complete, if every Radon plane that contains x has at least one IP from which x is projected onto the detector. In the sequel, we tacitly assume that IBP (x) is defined such that x is projected onto the detector from every point within CBP (x). This ensures that a certain trajectory is complete if every Radon plane that contains x intersects with CBP (x). 4. Reconstruction Algorithm In the following we assume that vectors eν and weights μν have been defined such that Eq. (58) results in a constant value of 1. With this assumption, Eqs. (47) and (46) define a reconstruction algorithm, which recovers the μvalues in a mathematically exact way. The reconstruction steps are as follows: 1. Compute the derivative in the sense of Eq. (43). 2. Filter the differentiated data, i.e., filter D (y(s), θ). 3. Perform the backprojection along IBP (x)—evaluate Eq. (46). Steps 1 and 2 follow from Eq. (47): the filtering is performed along γ with kernel 1/ sin γ . The filtering depends on x only via b(s, x). Now, all object points on one particular line from the focal spot to the detector share the same b. Consider two vectors b1 and b2 , where b2 is located in the κ-plane of b1 . In many cases, b1 is also located in the κ-plane of b2 . Now the filtering step
RECONSTRUCTION ALGORITHMS
25
is shift invariant. Therefore, using Fourier techniques, filtered values for all object points in the κ-plane spanned by b and eν (see Figure 15) can be obtained in one single processing step. Independent of this observation, this processing step must be repeated for each ν, i.e., the sum over ν must be calculated. The filtering step is further discussed in Section III.B.5. The backprojection in step 3 corresponds to the integration over s in Eq. (46). It incorporates the sum over ν in Eq. (47) using the weights μν . Section III.B.7 shows how the backprojection can be implemented most efficiently using wedge-detector geometry. 5. Filtering The filtering, i.e., the integration over γ in Eq. (47), is one of the central steps of the FBP algorithms discussed here. The vectors eν define which data must be involved in the filtering step. In practice, vectors eν are not the most useful quantities for parameterization of the filtering. Instead, we consider the intersection of the κ-planes with the planar detector (Figure 18). The line of intersection defines the filter line, which is sometimes called a κ-line. Any κ-line can be parameterized by two constants, which we denote as v0 and σ . With these, points on the κ-line are given by vP (uP ) = v0 + σ uP
(59)
in planar-detector coordinates. We derive some relationships between the quantities of a κ-plane and the corresponding κ-line in the following. Consider the line containing the focal spot at y(s) and being parallel to θ(γ ) defined in Eq. (48). Points on this line are given by (λ, γ ) = y(s) + λθ(γ ),
(60)
where λ parameterizes the points. Obviously, this line lies in the κ-plane. It intersects with the planar detector, if 0 = (λ0 , γ ) · w(s) = R + λ0 [cos γ b · w + sin γ e · w],
F IGURE 18.
The intersection of the κ-plane with the planar detector defines the κ-line.
(61)
26
BONTUS AND KÖHLER
where w(s) corresponds to the basis vector defined in Section II.C [see Eq. (15)]. Here, we have used the fact that y · w = R for the trajectories considered in this chapter. The point of intersection is parameterized by λ0 . From the latter equation we derive λ0 =
−R . cos γ b · w + sin γ e · w
(62)
With these results points on the κ-line can be parameterized by angle γ as cos γ b · u + sin γ e · u , (63) cos γ b · w + sin γ e · w cos γ b · v + sin γ e · v vP (γ ) = (λ0 , γ ) · v(s) − yz (s) = −R , (64) cos γ b · w + sin γ e · w
uP (γ ) = (λ0 , γ ) · u(s)
= −R
where we have used y · v − yz = 0. Finally, we compute the parameters of Eq. (59). Since σ corresponds to the gradient of the filter line, it can be computed from the last two equations by σ =
vP (γ2 ) − vP (γ1 ) uP (γ2 ) − uP (γ1 )
(65)
for arbitrary but different γ1 and γ2 . Parameter v0 corresponds to the vP coordinate of the κ-line at uP = 0. It is given by v0 = vP (γ0 ),
tan γ0 = −
b·u , e·u
(66)
where γ0 is defined by the condition uP (γ0 ) = 0. So far we have considered the case that vectors b and e define the filtering plane. Based on this, we have computed the parameters in Eq. (59). For completeness, we now follow the opposite approach: we assume that the κline is given by Eq. (59) and compute vectors b and e. Since Eq. (14) gives points in world coordinates, vector b associated with coordinate uP and the filter line under consideration can be computed via b(uP ) =
uP u(s) + (vP (uP ) + yz (s))v(s) − y(s) . |uP u(s) + (vP (uP ) + yz (s))v(s) − y(s)|
(67)
For convenience we introduce vector w1 (uP ) = b(uP + u) − b(uP ),
(68)
which is parallel to the filter line. Since vector e is parallel to the κ-plane, it is a linear combination of b(uP ) and w 1 (uP ). Furthermore, e is orthogonal to b. A vector that fulfills both criteria is given by wT (uP ) = w 1 (uP ) − (w 1 · b)b(uP ).
(69)
RECONSTRUCTION ALGORITHMS
27
Normalization gives e(uP ) =
wT . |w T |
(70)
6. Efficient Filtering So far we have considered reconstruction as computing the μ-value of one particular object point. In practice, reconstruction aims at computing the absorption coefficients in the ROI. Therefore, we now change our paradigm from an object-point–driven reconstruction to a projection-driven reconstruction. For this we assume that we have a set of filter lines distributed over the detector. Figure 19 shows an example. Each of the filter lines shown corresponds to a κ-line in the sense of Figure 18. Each filter line intersects with the uP -axis at a certain v0 . The gradient in the sense of Eq. (59) is given by v0 v0 (71) cot σ = h¯ R for each of the filter lines in our example. The number of filter lines shown in Figure 19 must not be confused with the sum over ν in Eq. (47). The example in Figure 19 has only one set of filter lines: the sum takes only the value ν = 1. As discussed in further detail in Section III.E the described filter lines suffice for exact reconstruction using the Pi segment. Within the Pi segment,
F IGURE 19. the Pi segment.
The Pi window on the planar detector and filter lines for exact reconstruction along
28
BONTUS AND KÖHLER
any object point will be projected into the Pi window. From each source position it will be projected onto a certain filter line. Now, the filter lines are defined such that the filtered value at the point of projection should be used for the backprojection. With this a projection-driven reconstruction can be phrased as follows: 1. Transform each projection into one or more filtered projections. 2. During the backprojection use the filtered values associated with the points onto which the object point is projected from each source position. Step 1 indicates that we perform the filtering along each of the filter lines, by which we obtain filtered values along the filter lines. For the backprojection in step 2, we might need to take different sets of filter lines into account and sum these using the correct weights. In other words, we must perform the sum over ν in Eq. (47) and use the weights μν . Using the results of Section III.B.5 points on the filter lines can be parameterized by angle γ . With this, the filtering along each line can be performed based on Eq. (47); that is, we must perform the integration over γ utilizing the shift invariance of the filtering kernel. In practice, the measured data are associated with the focus detector, whereas the filter lines have been defined on the planar detector. The relationship between focus- and planardetector variables is given by the formulas in Section II.C. We can proceed from one geometry to the other using interpolations. Nevertheless, it is most convenient if we can sample the data on the filter lines equidistantly in the focus-detector variable ϕ. Sampling the data equidistantly in ϕ requires fewer interpolations and therefore results in a better resolution. Angle γ is different from angle ϕ (Figure 20). Therefore, sampling points equidistantly in γ on the filter lines requires interpolations along vF and along ϕ. In the following we show how the formula for filtering can be
F IGURE 20.
Angles γ and ϕ are measured in different planes.
RECONSTRUCTION ALGORITHMS
29
) on the κ-line are separated by F IGURE 21. The two points P = (uP , vP ) and P = (uP , vP angle γ . The dashed line is orthogonal to the κ-line and has length . The three solid lines originating from the focal spot have lengths r, r , and r .
transformed so that the points on the filter lines can be sampled equidistantly in ϕ. With the subsequent derivation we follow Noo, Pack and Heuscher (2003) but aim for a slightly different result. Consider Figure 21. It shows the planar detector and one particular κ-line. Two specific points on the κ-line are denoted as P = (uP , vP ) and P = ). They are separated by an angle γ seen from the focal spot. The (uP , vP dashed line is orthogonal to the κ-line and has length . Using the formulas, which compute the area of a triangle, we observe that L = rr sin γ and L = r r sin γ . Hence, r sin γ L = . L r sin γ
(72)
At the same time L/L = uP /(uP − uP ), such that uP r sin γ . = uP − uP r sin γ
(73)
Since we are interested in infinitesimal quantities, this yields duP r dγ . = uP − uP r sin γ
(74)
For the transition to focus-detector coordinates, we realize by comparison with Eq. (9) that uP − uP = R(tan ϕ − tan ϕ) = R
sin(ϕ − ϕ) , cos ϕ cos ϕ
(75)
30
BONTUS AND KÖHLER
such that duP R = . dϕ cos2 ϕ
(76)
Using this in Eq. (74), we obtain r cos ϕ dϕ dγ = . sin γ r cos ϕ sin(ϕ − ϕ)
(77)
The latter equation still contains quantities r and r . From Figure 21 we can establish 2 r 2 cos2 ϕ = R 2 + u2P + vP cos2 ϕ tan2 λ R2 2 cos , (78) ϕ = = R 2 1 + tan2 ϕ + cos2 ϕ cos2 λ where we have used Eqs. (9) and (10). Now the integration measure can be written as dγ cos λ dϕ = . sin γ cos λ sin(ϕ − ϕ)
(79)
Comparing this result with Eq. (47) allows us to write the filtering step as π Pν (s, uP , vP ) = −π
dϕ cos λ D y(s), uP (ϕ ), vP (ϕ ) , sin(ϕ − ϕ) (80)
where
vP (ϕ ) = vP uP (ϕ ) ,
(81)
vP (uP ) is defined in Eq. (59), and uP (ϕ ) is given by Eq. (9). The factor 1/ cos λ in Eq. (79) was not used in Eq. (80). It will be taken into account in the backprojection formula below. Variable ϕ in Eq. (80) is associated with uP via Eq. (9). In other words, Eq. (80) parameterizes the filtered values in coordinates of the planar detector. The relationship with focus-detector coordinates can easily be established with the equations in Section II.C. The index ν indicates that the formula in Eq. (80) gives filtered values associated with a specific set of filter lines. Within this set the κ-line containing point (uP , vP ) must be used to obtain the desired filtered value. The integration, i.e., the filtering, is now performed over the angular variable ϕ . The filtering kernel is identical to the one in Eq. (47). Therefore
RECONSTRUCTION ALGORITHMS
31
the shift invariance is preserved. Notice the extra factor cos λ in Eq. (80). According to Eq. (10) this factor depends only on the focus-detector row. Therefore, the factor cos λ can easily be taken into account by a simple modification of the projection data. This can be done before or after the derivative D is computed. With the applied modifications the backprojection [compare with Eq. (46)] can now be written as Ne −1 1 ds μ(x) = μν Pν s, uP (x, s), vP (x, s) . 2 |x − y(s)| cos λ 2π IBP (x)
ν=1
(82) Here the factor 1/ cos λ stems from Eq. (79), since it was not absorbed into Eq. (80). This factor corresponds to a postprocessing step after the filtering. Similar to the factor 1/ cos λ in Eq. (80), it depends only on the detector position. This factor can therefore be taken into account by an object-point– independent modification of the filtered projection data. Functions uP (x, s) and vP (x, s) correspond to the coordinates of the object point x projected onto the planar detector from y(s). These coordinates are given by Eqs. (63) and (64) for γ = 0. 7. Backprojection Using Wedge Detector Geometry The backprojection formula [Eq. (82)] contains a factor 1/|x − y(s)|. This factor depends on the object point x and on the focal-spot position y(s). Due to this factor the backprojection becomes rather inefficient. Here we show how we can eliminate this factor by a transition to parallel (wedge) detector coordinates. For this consider Figure 22. It shows a plane containing x and parallel to the x- and y-axes. The source path y(s) is projected into this plane. Variable ϕ corresponds to the fan angle under which the object point appears from the projected y(s). While s corresponds to the angular variation of the source position, variable t is the angle under which the source is seen from x. From Figure 22 we realize that L dt R ds = . (83) cos ϕ Here, L is the distance from the projected object point to the projected focalspot position. This distance is related to λ via L , (84) cos λ = |x − y(s)| such that we derive from Eq. (83) ds cos λ = dt. (85) |x − y(s)| R cos ϕ
32
BONTUS AND KÖHLER
F IGURE 22. Projection of the object point x into the xy-plane. While angle s is measured from the rotation axis, angle t is measured from the object point.
Using this result, the backprojection formula Eq. (82) becomes μ(x) =
−1 2π 2
I˜BP (x)
Ne dt μν P˜ν t, u (x, t), v (x, t) . R cos ϕ ν=1
(86)
Here, P˜ν (t, u , v ) corresponds to the filtered projection data rebinned into wedge geometry. Functions u (x, t) and v (x, t) yield the coordinates of the object point projected onto the wedge detector, and I˜BP (x) is the backprojection interval of x in parallel geometry. The factor 1/R cos ϕ in Eq. (86) is independent of the object point x. This factor varies only with the position on the detector. Again, this factor is taken into account by a modification of the projection data right after filtering. C. Mathematics on the Planar Detector The preceding sections describe a certain type of a filtered backprojection algorithm. So far, the algorithm has been presented in its most general way. The subsequent sections are devoted to specific trajectories and acquisition types. The concept of projecting the trajectory and the vectors involved onto
RECONSTRUCTION ALGORITHMS
33
F IGURE 23. The intersection of the κ-plane defined by e and the planar detector results in the shown κ-line. The Radon plane defined by ω intersects with the detector, resulting in line Lω . Projecting e onto the detector gives vector eˆ , which is parallel to the κ-line. The projection of ω, ˆ is orthogonal to Lω . denoted as ω,
the planar detector proves very useful (Katsevich, 2004b). Some general formulas helpful for the mathematical discussions that follow are presented here. The projections of different trajectories were discussed in Section II.D. ˙ Remember that vectors e and ω define Now we consider vectors e, ω, and y. ˙ κ- and Radon planes, respectively. Notice that y(s), i.e., the tangent on the trajectory, is parallel to the planar detector for the trajectories considered. Figure 23 shows exemplary vectors e and ω. The projections of these vectors ˆ In particular, the κ-line is onto the planar detector are denoted as eˆ and ω. parallel to eˆ , while ωˆ is orthogonal to Lω . Here, Lω is the line that results from the intersection of the Radon plane with the detector. Explicit expressions for eˆ and ωˆ can be obtained via eˆ = (e × b) × w
(87)
ωˆ = (ω × w) × w = ω + w(ω · w).
(88)
and
Here, w is the basis vector defined in Section II.C and b is defined in Eq. (44). Furthermore, we used the relation a × (b × c) = b(a · c) − c(a − b) above. Now, using the general formula (a × b) · (c × d) = (a · c)(b · d) − (b · c)(a · d)
(89)
twice and applying b · ω = 0, we derive eˆ · ωˆ = (w × ω) · (e × b) = −(b · w)(e · ω).
(90)
34
BONTUS AND KÖHLER
F IGURE 24. The planar detector seen from y(s2 ) in Figure 16. The intersection of the Radon ˆ The backprojection segment is drawn plane results in line Lω . Object point x is projected onto point x. bold.
Similarly, using w · y˙ = 0, we obtain from Eq. (88) ˙ ˙ ωˆ · y(s) = ω · y(s).
(91)
Considering Eqs. (90) and (91) and keeping Eq. (58) in mind yields sgn(ω · e) = sgn(ωˆ · eˆ )
and
˙ = sgn(ωˆ · y), ˙ sgn(ω · y)
(92)
where we have used b · w < 0. Comparing Eqs. (58) and (92) shows that for the evaluation of Eq. (58) it is sufficient to consider the projections of the involved vectors onto the planar detector. We further argue that projections onto the planar detector also are useful for answering certain questions (such as, how often does a certain Radon plane intersect with the trajectory?). For an illustration consider Figure 24. It shows the planar detector seen from y(s2 ) in Figure 16. The backprojection segment along the helix is drawn bold. As seen, the other two IPs of the Radon plane (besides y(s2 )) do not intersect with the backprojection segment. Eq. (58) is invariant under ω → −ω. We will use this by ensuring that ˙ = 1 in the examples given throughout this chapter. sgn(ω · y) D. Circle Plus Line This section is devoted to a trajectory consisting of a circle and a line, where the line is parallel to the z-axis. The circle and line trajectory (CLT) is best suited to demonstrate the methods used in this chapter. It is relatively easy to understand, while it requires applications to the typical case differentiations. The resulting reconstruction algorithm was first published by Katsevich (2004a).
RECONSTRUCTION ALGORITHMS
35
1. Geometrical Properties The parameterization of the trajectory under consideration is shown by Eqs. (4) and (6). The circle and the line intersect at one point, denoted as Pc . For every object point x a line can be found that contains x and intersects with the circle and the line. Figure 25 illustrates this concept. In analogy to helical CT, such a line is called a Pi-line. Furthermore, the IPs of the Pi-line with the circle and the line are denoted as Pπc and Pπ , respectively. Now, we choose the backprojection segment CBP (x) for the CLT to extend from Pπ c to Pc along the circle and from Pc to Pπ along the line. This gives two possibilities along the circle (Figure 25). Using one of these backprojection segments, the combination of the Pi-line and CBP (x) gives a closed contour. Therefore, every Radon plane containing x intersects with CBP (x) and the trajectory is complete. Moreover, every Radon plane has either one or three IPs with CBP (x). We neglect planes that are tangential on the circle or contain the line since these have a Lebesgue measure zero. As mentioned in Section III.B.2, Radon planes with one IP are denoted as 1planes, while Radon planes with three IPs are called 3-planes. Radon planes with three IPs intersect twice with the circle and once with the line, whereas 1-planes either have one IP with the line or one IP with the backprojection segment along the circle. Figure 26 illustrates the three different cases. In the sequel we define filter lines separately for the circle and for the line part of CBP (x). Based on Eq. (58) we show that the filter lines are defined such that every 1-plane receives a weight of 1 from the single IP. Every 3plane gets a weight of 1 from each of the two IP along the circle. It gets a weight of −1 from the IP on the line, such that the sum over all three weights results in +1, as desired.
F IGURE 25. The circle and the line intersect at point Pc . For each object point x a unique Pi-line can be found, which contains x and intersects with the circle and the line. The points of intersection are denoted as Pπ c and Pπ , respectively. For the backprojection segment, which is drawn in bold, the two possibilities are shown in the left and right images.
36
BONTUS AND KÖHLER
F IGURE 26. Left: The two Radon planes each have one IP with the backprojection segment. Right: The Radon plane has three IPs. For the other possibility of CBP (x), this plane corresponds to a 1-plane.
2. Filter Lines The methods introduced in Section III.B can be used to parameterize each filter line using planar-detector coordinates. In particular, we specify pairs of constants v0 and σ , which define a filter line in the sense of Eq. (59). For the circle part of CBP (x), the filter lines used are all parallel to the uP -axis, such that σ = 0. In other words, specifying a complete set of filter lines for the circle part, parameter v0 differentiates between different lines of this set. We need to define only one set of filter lines for the circle, such that Ne = 1 [compare with Eqs. (47), (58), and (86)]. Moreover, the weights belonging to this filter-line set must be set to μν = 1. Figure 27 depicts the filter lines. We also need one set of filter lines for the line part of the CLT, i.e., Ne = 1. The filter lines are tangents on the projected circle. The projection of the circle can be computed with Eq. (22), where |z0 − z1 | is the distance between the
F IGURE 27.
The filter lines for the circle part of the CLT are all parallel to the uP -axis.
RECONSTRUCTION ALGORITHMS
37
F IGURE 28. Projection of the circle onto the planar detector from a point on the line for the two cases shown in Figure 25. The backprojection segment is drawn bold. The dashed line is the projection of the Pi-line. This projection is parallel to the vP -axis.
F IGURE 29. Figure 25.
Sets of filter lines for a particular point on the line for the two cases shown in
point on the line and the circle plane. Since Eq. (22) depends on z, the sets of filter lines vary for different positions on the line. Figure 28 considers the two different cases for the choice of CBP (x). The backprojection segment is drawn bold, and the projection of the Pi-line is dashed. The specific definition of the filter lines requires that each filter line is tangential on the projected backprojection segment, i.e., on the segments drawn bold in Figure 28. This results in the filter-line sets depicted in Figure 29 for the two cases. Obviously, these filter lines are tangential on the projected circle, and the point of tangency is either right or left of the point onto which x is projected. The weights must be set such that μν = 1 for the case shown in the left image of Figure 29, while μν = −1 for the case shown in the right image of Figure 29. a. Circular Full Scan. For the two possibilities of CBP (x) shown in Figure 25, the two segments on the circle together cover the complete circle. This provides an easy “recipe” for defining a full-scan algorithm, which uses data from the complete circle. We could apply just the algorithm defined above for both cases and average the results. More efficiently, we obtain the same output by performing the backprojection over the complete circle using the filter lines defined above. For the line, we filter the data using both sets of filter lines and add the data before backprojection. In other words, we need to
38
BONTUS AND KÖHLER
set Ne = 2 in Eqs. (82) and (86). Weights μν must be multiplied by 1/2 for the line part. 3. Proof of Exactness We now prove that the reconstruction algorithm is mathematically exact. As discussed in Section III.B, we need to show that Eq. (58) results in a constant value of 1 using the defined filter lines and weights. The result of Eq. (58) must be equal to 1 for every object point in the FOV and for every Radon plane containing the object point. We begin with analyzing the filter lines of the circle part. Remember that every Radon plane either has zero, one or two IPs with the circle part of CBP (x). If it has two IPs, the Radon plane is a 3-plane, i.e., it has one additional IP with the line. According to Section III.C it is sufficient to consider the projection of vectors ω and e onto the planar detector. The projected vectors are denoted as ωˆ and eˆ , respectively. Moreover, the tangent ˙ on the trajectory y(s) is parallel to the planar detector. We defined the filter lines for the circle part to be parallel to the uP -axis. Therefore, eˆ is also parallel to this axis for each filter line. Realizing that y˙ ◦ (s) is also parallel to the uP -axis, we conclude that the two sgn-functions in Eq. (58) result in ˙ sgn(ωˆ · eˆ ν ) = sgn2 (ωˆ · eˆ ν ) = 1. Now, μν = 1, Ne = 1, and k sgn(ωˆ · y) counts the number of IPs. Therefore, evaluating Eq. (58) only for the circle part results in a value of 0, 1, or 2. In other words, for a 1-plane with IP along the circle, Eq. (58) gives the desired result of 1. For a 3-plane the two IP along the circle contribute with a weight of 2. In the following we therefore must show that the line contributes with a weight of −1 for 3-planes, while 1-planes with IP on the line get a weight of 1. For the discussion of the filter lines used along the line consider Figure 30. It corresponds to the left image in Figure 28. We assume that the object point ˆ Remember that x is located under consideration is projected onto point x. on the Pi-line, which is drawn as a dashed line. The corresponding filter line, which is parallel to vector eˆ , is tangential on the projected circle, with the point ˆ The tangent on the trajectory, y˙ | , is parallel to of tangency located right of x. the vP -axis, i.e., it is parallel to the z-axis. The two dotted lines correspond to exemplary Radon planes with normal vectors ω1 and ω2 . Obviously the Radon plane associated with ω1 is a 1-plane. The sole IP is the current source position. The Radon plane associated with ω2 is a 3-plane, for which only one IP on the circle is shown in Figure 30. The second IP is located farther to the right. As discussed in Section III.C, Eq. (58) is invariant under ω → −ω. For ˙ = 1 in our examples. The angle convenience, we ensure that sgn(ω · y) between eˆ and ωˆ 1 is less than π/2. Therefore, sgn(ωˆ 1 · eˆ ) = 1. Evaluating
RECONSTRUCTION ALGORITHMS
39
F IGURE 30. The planar detector seen from one point on the line. The two dotted lines ˆ correspond to Radon planes with normal vectors ω1 and ω2 . The object point is projected onto point x. The filter line, which is parallel to eˆ , is tangential on the projected circle with the point of tangency ˆ located to the right of x.
Eq. (58) with Ne = NI = 1 and μν = 1, we realize that the Radon plane associated with ω1 receives a weight of 1, as desired for a 1-plane. The angle between eˆ and ωˆ 2 is larger than π/2, such that sgn(ωˆ 2 · eˆ ) = −1. Now, with NI = 3 we can evaluate Eq. (58) and obtain 1 + 1 − 1 = 1, where the first two summands stem from the two IPs on the circle. In other words, the 3-plane associated with ω2 derives the desired weight of 1 from the complete CLT. Figure 31 shows the situation of Figure 30 for the second possibility of CBP (x). Both Radon planes now correspond to 1-planes. The filter line is ˆ Obviously, tangential on the circle with point of tangency located left of x.
F IGURE 31. The same scenario as in Figure 30 but now for the second case shown in Figure 25. The filter line, which is parallel to eˆ , is tangential on the projected circle with point of tangency located ˆ left of x.
40
BONTUS AND KÖHLER
sgn(ωˆ 1 · eˆ ) = sgn(ωˆ 2 · eˆ ) = −1, and with μν = −1 Eq. (58) results in a value 1 for the two Radon planes shown. All Radon planes that can be covered from the current focal spot position y | (z) must contain y | (z) and x. Therefore, all these Radon planes can be ˆ Starting considered by rotating one of the dashed lines in Figure 30 around x. with the line indicated by ωˆ 1 and rotating counterclockwise, the associated Radon plane becomes a 3-plane as soon as the line intersects with the circle. Now, since the filter line is tangential on the circle, ωˆ · eˆ changes sign exactly at the transition from a 1-plane to a 3-plane. The next sign change occurs when the dashed line becomes parallel to the vP -axis—remember that we ˙ = 1. Therefore, we must change ωˆ from pointing to the left to want sgn(ωˆ · y) pointing to the right, once the dashed line becomes parallel to vP . Again, the sign change occurs exactly at the transition from a 3-plane to a 1-plane, since the projected Pi-line is parallel to the vP -axis. Continuing with the rotation until the dashed line becomes parallel to the line with which we started, no further sign change occurs. In other words, for the current source position and object point x, all Radon planes can be treated in the same way as above for the exemplary Radon planes with normal vectors ω1 and ω2 . The discussion so far has not been specific in the choice of the focal spot position or the object point. Choosing different x and/or y | (z) all arguments can be repeated. In other words, 1- and 3-planes get the desired weights from all source positions, such that the proof is finished. 4. Reconstruction Results For the evaluation of the CLT reconstruction algorithm we simulated a CT scanner with 256 detector rows. Table 1 summarizes the simulation parameters of the circle part. The circle plane was located at a distance TABLE 1 S IMULATION PARAMETERS Projections per turn Detector columns Detector rows Fan angle Detector height Focal-spot size Distance source-rotation axis Distance source-detector center Detector oversampling Focal-spot oversampling Angular oversampling Reconstructed voxel size
1160 512 256 26.5 291.9 0.91 × 1.37 570 1040 3×3 3×3 3 (0.5)3
degrees mm mm2 mm mm
mm3
41
RECONSTRUCTION ALGORITHMS
(a)
(b)
(c)
F IGURE 32. Reconstruction results for the CLT. The three images show (a) the result of the complete CLT, (b) the contribution of the circle, and (c) the contribution of the line, respectively. L/W: 40 HU/200 HU for the first two images, L/W: −950 HU/200 HU for the line image. L, level; W, window.
z = 40 mm from the origin of the phantom. For the line part, we simulated 1025 projections over a distance of 320 mm along z. We used the Forbild head phantom (www.imp.uni-erlangen.de/forbild/english/results/head/head.html), which contains some high-frequency components as well as some lowcontrast parts. The head phantom therefore is a challenge for every reconstruction algorithm. Throughout this chapter we use Hounsfield2 units (HU) for the presentation of reconstruction results. Hounsfield units correspond to a shift and scaling of the absorption coefficients, such that HU(μ) = 1000
μ − μ H2 O . μ H2 O
(93)
Here, μH2 O corresponds to the absorption coefficient of water. We have set μH2 O = 0.0183/mm. Notice that the shift used in the transition from μ-values to HU must be taken into account when adding images. Figure 32 shows some reconstruction results. The specification of the level and window (L/W) 40 HU/200 HU means that the available gray-level range is assigned to object points between −60 HU and 140 HU. Object points below −60 HU are drawn as black, while object points above 140 HU are white. Figure 32 shows the result for the complete CLT, as well as the contributions of the circle and the line. The (transaxial) slice shown is perpendicular to the z-axis and has a distance of 45 mm to the circle plane. Obviously, the addition of the line significantly reduces the artefacts visible in the image of the pure circle. 2 Sir Godfrey Newbold Hounsfield, English electrical engineer, 1919–2004.
42
BONTUS AND KÖHLER
E. The Katsevich Algorithm for Helical Pi Acquisition The reconstruction algorithms described next deal with helical CT, beginning with the Pi mode. An exact shift invariant FBP algorithm for a helical Pi acquisition was first found by Katsevich (2002). The following sections describe the further improvement of the algorithm published in Katsevich (2004b). 1. Geometrical Properties The helical trajectory is parameterized by Eq. (5). As discussed in Section II.D, a Pi-line can be found for every object point x in the FOV. A Pi-line contains x and two points y H (sπ1 ), y H (sπ2 ) on the helix with sπ2 −sπ1 < 2π. All points y H (s) on the helix with sπ1 s sπ2 belong to the Pi segment of x. The lower and upper Pi-window boundaries are given by Eq. (32) setting n = 1. From every point y H (s) within the Pi segment, x is projected onto the planar detector between these boundaries. For the reconstruction algorithm discussed here, the Pi segment corresponds to the backprojection segment CBP (x). Since the combination of the Pi-line and the Pi segment gives a closed contour, every Radon plane containing x intersects with the Pi segment. The trajectory is complete. In particular, every Radon plane has either one or three IPs with CBP (x) (again we neglect planes tangential on the helix, since these have a measure zero). Figure 33 depicts one particular object point x and its Pi-line. The shown Radon plane intersects three times with the Pi segment, and the IPs are y H (s1 ), y H (s2 ), and y H (s3 ). Figure 34 shows the planar detector seen from the three points y H (s1 ) through y H (s3 ). The object ˆ The dotted lines in point is projected onto different positions indicated by x. Figure 34 correspond to the intersections of the Radon plane with the detector.
F IGURE 33. point x.
The three IPs of the Radon plane are located within the Pi segment of object
RECONSTRUCTION ALGORITHMS
43
F IGURE 34. The planar detector seen from the three IPs y H (s1 ) through y H (s3 ) in Figure 33. ˆ The dotted lines indicated by R correspond to the The object point is projected onto points x. intersections of the Radon plane. Projections of the Pi-line result in the solid lines indicated by π . The other solid line L1 corresponds to the asymptote on the helix.
The asymptote on the projected helix plays a crucial role. The asymptote is indicated by the solid line L1 in Figure 34. It is parallel to y˙ H (s). Line L1 contains the origin of the planar detector and its gradient is equal to h/R. The ¯ latter follows from Eq. (32) using arctan x → (π/2 − 1/x) for x → ∞. The solid lines indicated by π in Figure 34 are the projections of the Piline. Remember that the points on the Pi-window boundaries are associated with points on the helix. Traversing these boundaries from right to left the corresponding s-value increases. In particular, two points on the lower and upper Pi-window boundaries at the same uP -coordinate are separated by an angular distance of 2π [compare with Eq. (28)]. Since the points of sunrise and sunset now are separated by less than 2π , the Pi-line projected onto the planar detector must have positive slope. Point y H (s1 ) is the first IP of the exemplary Radon plane in Figure 33, which is reached when traversing the helix. Seen from that point (the first image in Figure 34), the projection of the Radon plane (dotted line) has two IPs with the upper Pi-window boundary. Moreover, these two IPs belong to the Pi segment, since the intersection of the Pi-line with the Pi-window boundary is farther to the left. In summary, from the first image in Figure 34 it can be deduced that the indicated Radon plane is a 3-plane with two further IPs in the future. Similarly, in the second image of Figure 34 the dotted line has a gradient larger than the gradient of L1 . Therefore, the dotted line has one IP with the lower Pi-window boundary and one IP with the upper one. This indicates that the current source position corresponds to the central IP of the 3-plane. 2. Filter Lines As discussed in Section III.B, the filtering can be defined using κ-planes or κ-lines, where the latter are the intersection of the κ-planes with the
44
BONTUS AND KÖHLER
F IGURE 35. The κ-planes used for Pi reconstruction contain the object point x, the current source position y(s), and two further points on the helix: y(sκ1 ) and y(sκ2 ). The angular distance sκ2 − s is twice as large as the distance sκ1 − s.
planar detector. For the reconstruction algorithm described here, the κ-planes correspond to planes that contain the object point, the current source position, and two additional points on the helix denoted as y H (sκ1 ) and y H (sκ2 ) in the following. Both points y H (sκ1 ) and y H (sκ2 ) must belong to the Pi segment of x, and they must fulfill sκ2 − sκ1 = sκ1 − s, where y H (s) corresponds to the current source position (Figure 35). If x is projected above line L1 (compare with Figure 34), s < sκ1 < sκ2 , while sκ2 < sκ1 < s, if x is projected below L1 . If x is projected onto the asymptote L1 , the κ-line is identical to L1 , i.e., the κ-plane is parallel to the tangent on the helix. For the method specified here, the weights must be set to μν = 1. We show in the following that the definition of the κ-planes results in filter lines with gradients given by Eq. (71). For symmetry reasons we can choose the source to be located at y H (0). We consider the κ-plane, which contains the focal spot and points y H (s) and y H (2s). A parameterization of this plane is given by p(α, β) = y H (0) + α y H (s) − y H (0) + β y H (2s) − y H (0) R R(cos s − 1) R(cos 2s − 1) = 0 +α +β . (94) R sin s R sin 2s 0 hs 2hs ¯ ¯ Due to our choice of parameters the plane intersects with the planar detector at x = 0, such that 0 = 1 + α(cos s − 1) + β(cos 2s − 1)
(95)
RECONSTRUCTION ALGORITHMS
45
gives the condition for the points of intersection. Solving the latter equation for β results in 1 + α(cos s − 1) 1 + α(cos s − 1) = . (96) 1 − cos 2s 2 sin2 s Inserting this equation into Eq. (94), the y-component corresponds to the uP coordinate of the filter line: sin 2s sin 2s(cos s − 1) uP (α) = R + αR sin s + 2 sin2 s 2 sin2 s (1 − cos s) , (97) = R cot s + αR sin s or uP sin s − R cos s uP − R cot s α(uP ) = sin s = . (98) R(1 − cos s) R(1 − cos s) Similarly, for the parameterization of the vP -coordinate of the κ-line, we obtain h¯ s cos s − 1 vP (α) = + α hs ¯ 1+ sin2 s sin2 s h¯ s cos s(1 − cos s) = + α hs . (99) ¯ 2 sin s sin2 s Inserting Eq. (98) gives β=
h¯ s hs cos2 s ¯ cos s − hs + uP ¯ 2 2 R sin s sin s sin s hs ¯ = hs ¯ + uP cot s R v0 v0 (100) = v0 + uP cot , h¯ R which is the parameterization of the filter line in the sense of Eqs. (59) and (71) with v0 = hs. ¯ Considering Figure 19 we observe that the defined filter lines cover the complete Pi window, such that a filter line can be found for every object point and for every source position within CBP (x). Nevertheless, the filter lines have different gradients, such that intersection points exist. For illustration consider Figure 36. The two κ-lines intersect at point P1 . The definition given above specifies that κ-planes must have three IPs with the Pi segment. Line κ1 has two IPs outside the Pi segment for object points projected onto P1 . Remember that the projection of the Pi-line has a positive slope on the planar detector (compare with Figure 34). Therefore, we must use filtered values associated vP (uP ) =
46
BONTUS AND KÖHLER
F IGURE 36. The two filter lines κ1 and κ2 intersect at point P1 . For line κ2 only the corresponding κ-plane has intersection points within the Pi segment of object points projected onto P1 .
with line κ2 for points projected onto P1 . Line κ1 serves as a filter line for object points projected, for example, onto point P2 . 3. Proof of Exactness We now prove that the defined filter lines specify an exact reconstruction algorithm by showing that Eq. (58) results in a constant value of 1. Consider Figure 37. An exemplary object point is projected onto point xˆ of the planar detector. The corresponding Pi-line is indicated by π . The line indicated by T contains xˆ and is tangential on the upper Pi-window boundary. The Pi-line and line T divide the detector plane into areas A and B, such that the dotted line belongs to area B. The Radon planes with lines of intersection in area A are 1planes. These planes either have only one IP with the helix (the current source position) or the remaining IPs do not belong to the Pi segment, i.e., to CBP (x). Radon planes with intersection lines in area B are 3-planes. The proof given in the sequel consists of different steps. First, we show that 1-planes receive a weight of 1 from the single IP. Second, we argue that for 3-planes, the central IP always contributes with a weight of 1. Finally, we prove that for 3-planes the contributions of the first and third IP cancel. We begin with some general observations. Remember that line L1 is the asymptote on the helix. Line L1 is parallel to y˙ H (s). We denote the line of intersection of the Radon plane and the planar detector as R. Now, we argue that projecting x into the Pi window from the first point of a 3-plane, the point of projection must be above L1 . Line R must intersect twice with the CBP (x)part of the upper Pi-window boundary seen from the first IP (compare with Figure 34). If x is projected below L1 , line R must be steeper than line L1 in order to have an intersection with the upper Pi-window boundary. Now,
RECONSTRUCTION ALGORITHMS
47
ˆ The Pi-line (indicated by π ) and the tangent F IGURE 37. The object point is projected onto x. on the upper Pi-window boundary (indicated by T ) separate the detector plane into areas A and B. The dotted line corresponds to a Radon plane with three IPs with CBP (x).
line R can have at most one IP with the CBP (x)-part of the upper Pi-window boundary, such that the source position cannot be the first IP, if x is projected below L1 . Similarly, x must be projected below L1 from the third IP of a 3plane. With these observations in mind we realize that the tangential line T in Figure 37 must have a gradient smaller than the gradient of line L1 . The arrow in Figure 38 originating from xˆ is parallel to y˙ H (s) such that the line that contains xˆ and is parallel to the arrow is parallel to L1 . This line divides area B into areas B1 and B2 . Obviously, for Radon planes with intersection lines R in B1 , the current source position corresponds to the first IP, whereas for 3-planes with R in B2 the current source position is the ˆ is central IP. We argue that the filter line, which must be used for point x, located in area B1 . Remember that the corresponding κ-plane has three IPs with CBP (x). If xˆ is located above L1 , the κ-plane has two IPs in the future. In other words, the κ-line must have two IPs with the upper Pi-window boundary. This is only possible if the filter line is located in area B1 . Consider the exemplary line R in Figure 38 located in area B2 . Once again, ˆ i.e., the projection of the normal vector of the Radon plane, we draw ω, such that sgn(ωˆ · y˙ H ) = 1. Since the filter line is located in B1 , we realize that also sgn(ωˆ · eˆ ν ) = 1. This remains true if we rotate the dotted line R counterclockwise into area A (remember that the gradient of line T is smaller than the gradient of L1 ). In other words, the two sgn-functions in Eq. (58) contribute with a factor 1, both for 1-planes (area A) and from the central IP of 3-planes (area B1 ). Therefore, all that remains to finish the proof is to show that the contributions of the first and third IPs of a 3-plane cancel when summing over k in Eq. (58).
48
BONTUS AND KÖHLER
F IGURE 38. The arrow originating from xˆ is parallel to y˙ H (s). The line containing xˆ and the arrow is parallel to the asymptote on the helix. This line divides area B defined in Figure 37 into areas B1 and B2 . The dotted line R is located in area B2 . For the 3-plane associated with R the current focal-spot position is the central IP.
We consider a 3-plane and denote the three IPs as y H (s1 ), y H (s2 ), and y H (s3 ), s1 < s2 < s3 . Seen from y H (s1 ), the line R associated with the 3-plane must be located in area B1 . Moreover, the filter line to be used from y H (s1 ) is also located in area B1 . We denote the remaining two IPs of the corresponding κ-plane as y H (sκ1 ) and y H (sκ2 ). To be precise, we write sκ1 (s1 ), sκ2 (s1 ) for the specification of the filter line to be used from y H (s1 ). There are two possibilities for the ordering of the IP, illustrated in Figure 39. The solid lines denoted by κ indicate the filter lines, while the dotted lines correspond to the Radon plane. The two possibilities are treated separately in the following two cases. Case 1. s1 < sκ1 (s1 ) < s2 < s3 < sκ2 (s1 ). This case corresponds to the first image in Figure 39. Obviously, the angles between ωˆ and y˙ H and between ωˆ and eˆ ν are less than 90 degrees. Therefore, the two sgn-functions in Eq. (58) contribute with a factor of 1 for the first IP of the Radon plane in this case. With the definition of the κ-planes given above, the central IP of the κ-plane must be located right in the (angular) middle of the other two IPs. Therefore, sκ2 (s1 ) + s1 < s2 < s3 < sκ2 (s1 ). (101) 2 We now argue that, seen from y H (s3 ), the situation shown in Figure 40 follows, such that s1 < sκ2 (s3 ) < sκ1 (s3 ) < s2 < s3 . Otherwise, sκ2 (s3 ) < s1 < s2 < sκ1 (s3 ) = (sκ2 (s3 ) + s3 )/2 must be fulfilled. Together with Eq. (101) this leads to (sκ2 (s1 ) + s1 )/2 < (sκ2 (s3 ) + s3 )/2 and sκ2 (s3 ) < s1 < s3 < sκ2 (s1 ), which is clearly a contradiction. sκ1 (s1 ) =
RECONSTRUCTION ALGORITHMS
49
F IGURE 39. Upper half of the planar detector. Considering 3-planes with two IPs in the future, two possibilities exist for the order of IPs. Either s1 < sκ1 < s2 < s3 < sκ2 or s1 < s2 < sκ1 < sκ2 < s3 . For illustration purposes, the shown filter lines κ deviate from the filter lines of the specified algorithm. Nevertheless, the given arguments remain valid.
F IGURE 40. The lower half of the planar detector; similar to the first image in Figure 39, seen from the third IP of the 3-plane. Here s1 < sκ2 < sκ1 < s2 < s3 .
The angle between ωˆ and eˆ ν in Figure 40 is larger than 90 degrees, such that the third IP contributes with a weight of −1 in Eq. (58). In other words, the contributions of the first and the third IPs cancel as desired. Case 2. s1 < s2 < sκ1 (s1 ) < sκ2 (s1 ) < s3 . This case corresponds to the second image in Figure 39. Now, the first IP contributes with a weight of −1 in Eq. (58). The proof that the third IP now contributes with a positive weight is identical to Case 1. F. EnPiT: Helical n-Pi Reconstruction The Pi-method discussed in the previous section suffices for demonstrating exact reconstruction using helical data. Nevertheless, data obtained with a Piacquisition have only few redundancies. Therefore, Pi-methods are sensitive to patient motion and might result in noisy data if the tube current is not sufficiently high. Moreover, for given gantry rotation speed and detector
50
BONTUS AND KÖHLER
height, the Pi-mode determines the table feed. The latter can reach huge values for modern cone-beam scanners. The n-Pi mode introduced in Section II.D reduces some of these problems. For an n-Pi acquisition, the table feed is much smaller than for the Pi-mode, and more redundant data are acquired. Redundancies reflect themselves in the type of Radon planes that occur. In the n-Pi case, Radon planes can have 1, 3, . . . , n + 2 IPs with CBP (x). The backprojection segment, CBP (x), contains all points along the helix from which x is projected into the n-Pi window. The vast majority of Radon planes has n IPs for typical system parameters. Here we summarize a reconstruction algorithm denoted as EnPiT, which was published by Bontus et al. (2003, 2005). EnPiT is a quasi-exact reconstruction algorithm in the sense that some n- and (n + 2)-planes receive the incorrect weighting. Nevertheless, reconstruction results show that the approximation is rather good. An exact algorithm for the 3-Pi case was published by Katsevich (2006). 1. Reconstruction Algorithm (L) The filter-line sets defined below are denoted as Lp (p odd, 1 p < n) (R) and Lp (p odd, 1 p n). As shown in Bontus et al. (2005), these sets provide the following features at least in cases in which no interrupted illumination occurs:
1. A 1-plane gets a positive contribution from each filter-line set. (R) 2. An n-plane gets a positive contribution from set Ln for each of the n IPs (intersection with CBP (x)). The contributions of Lp(L) and Lp(R) with p < n cancel. (R) 3. An m-plane, 3 m < n, gets a positive contribution from Ln at each of (L) (R) the m IPs, while the contributions of Lp and Lp with p < m cancel. (L) (R) For m p < n, Lp and Lp contribute with a positive weight only to the first and last IPs while the contributions of these sets cancel for the inner IP. Filter lines of each set Lp(L) or Lp(R) are located completely in the pPi window. In order to consider all filter lines for an n-Pi reconstruction, (L) (L) (L) (R) (R) (R) we must use all sets L1 , L3 , . . . , Ln−2 and L1 , L3 , . . . , Ln . From every position within the backprojection interval the object point under consideration is projected onto the detector. For the reconstruction, the values of all filter lines, which contain the projection point, must be added using the correct weighting factors μν . This means that up to n values must be added, depending on the position of the projection point. We denote the weights for (R) (L) (R) (L) sets Lm and Lm as μm and μm , respectively. For n = 3 all weights are
RECONSTRUCTION ALGORITHMS
equal to 1. For n > 3 the corresponding values are given by ⎧ ⎨ n+1 4 , m = 1, (R) nμm = 1 , 3 m n − 2, ⎩2 1, m = n,
n+1 , m = 1, (L) −nμm = 1 4 , 3 m n − 2. 2
51
(102)
These weights have been chosen such that (keeping the features enumerated above in mind) the majority of Radon planes receive the correct weighting. For instance, the second point in the enumeration above tells us that n-planes (R) receive positive contributions only from set Ln . From each of the n IPs the n-plane gets a weight of 1/n. 2. Filter Lines To define the different filter lines that are necessary to perform the n-Pi reconstruction we first introduce certain lines, which separate the planar detector into different regions. As above, L1 is the asymptote of the Pi window. In the same manner, we define the lines Lm and L−m as the lines parallel to L1 but tangential on the upper and lower m-Pi window boundaries, respectively. Finally, we define the set of lines Lm −p as the lines that have a negative gradient and are both tangential on the upper m-Pi window boundary and on the lower p-Pi window boundary. Figure 41 shows some of these lines within the 5-Pi window. (L) We define different sets of filter lines and denote these sets as Lm (R) and Lm for different values of m. In particular, the corresponding filter (L) line within Lm for a point (uP , vP ) within the m-Pi window fulfills the following conditions: IF (uP , vP ) is above Lm −1 , the filter line is tangential on the upper m-Pi window boundary ELSE IF (uP , vP ) is above L1−1 , the filter line is tangential on the lower Pi-window boundary ELSE IF (uP , vP ) is above L1−m , the filter line is tangential on the upper Pi-window boundary ELSE the filter line is tangential on the lower m-Pi window boundary. (R)
The corresponding filter line within Lm for a point (uP , vP ) within the m-Pi window fulfills the following conditions: IF (uP , vP ) is above Lm , the filter line is tangential on the upper m-Pi window boundary
52
BONTUS AND KÖHLER
F IGURE 41.
Different lines of the sets Lm and Lm −p within the 5-Pi window.
ELSE IF (uP , vP ) is above L−m , the filter line is parallel to L1 ELSE the filter line is tangential on the lower m-Pi window boundary. Obviously, any filter line is either tangential on one of the p-Pi window boundaries or parallel to y˙ H (s). For the definitions of the tangents, we still have to say, whether the point of tangency is on the left-hand side (LHS) or the right-hand side (RHS) of (uP , vP ). In general, for filter lines within Lm(L) , the point of tangency is on the LHS of (uP , vP ) if the line is tangential on one of the upper p-Pi window boundaries, while it is on the RHS, if the line is tangential on one of the lower p-Pi window boundaries. For those lines within (R) Lm that are tangential on the m-Pi window boundary, we must consider the point at which Lm is tangential on the m-Pi window boundary. If (uP , vP ) is located left of this point, the point of tangency is on the RHS of (uP , vP ); if (uP , vP ) is located right of this point, the point of tangency is on the LHS of (uP , vP ). Figures 42 and 43 show the filter lines for the 5-Pi case. G. CEnPiT: Helical Cardiac CT 1. Cardiac CT Cardiac CT is one important application of CT. Due to the beating heart cardiac CT is challenging with respect to image quality and temporal resolution. We restrict this application to a discussion of cardiac CT based on a helical
53
RECONSTRUCTION ALGORITHMS
F IGURE 42.
(L)
5-Pi filter lines; from top to bottom: L1
(L)
and L3
.
acquisition. In particular, we assume that the data were obtained with a lowpitch helix, which can be associated with an n-Pi acquisition. Typical values of n are 9 or 11. Cardiac CT data usually are acquired simultaneously with the electrocardiogram (ECG). The aim of reconstruction is to obtain images associated with a certain motion state of the heart, and the ECG data are related to the motion states. R peaks are the canonical markers to describe the cardiac cycle based on the ECG. The cardiac cycle is considered to reach from one R peak to the next. States of the cardiac cycle are described based on these intervals. For instance, a phase point of 70% specifies that all time points are considered that are located at t = t0 + 0.7t. Here t is the time period between the R peak at t0 and its successor (see Figure 44). The situation becomes more difficult since the phase point in which the heart is in the state of least motion varies from patient to patient. Motion maps (Manzke et al., 2004) yield good results for finding the phase points of least motion. In the following discussion, we assume that the phase point at which the reconstruction shall be performed is known.
54
BONTUS AND KÖHLER
F IGURE 43.
(R)
5-Pi filter lines. From top to bottom: L1
(R)
, L3
(R)
, and L5
.
RECONSTRUCTION ALGORITHMS
55
F IGURE 44. The highlighted points on the ECG correspond to the phase points. The phase points are specified relative to the R peaks. Each phase point is associated with a certain point on the helix. The amount of data used for reconstruction around each phase point define the width of the gating window.
A simple EnPiT reconstruction cannot yield good cardiac images. Cardiac CT requires to use of data acquired at time points close to the phase points, while data at a large temporal distance to the phase points should be neglected. A good reconstruction algorithm therefore must exploit the given redundancies. Redundancies reflect themselves in the different types of Radon planes for the kinds of reconstruction algorithms discussed here. Certainly for 1-planes there is no redundancy. Fortunately, the majority of Radon planes has n IPs and Radon planes with fewer IPs contribute mainly to the low-frequency components of the images (compare with Section III.A.2). The CEnPiT algorithm, published first in Bontus et al. (2006), exploits these facts. Moreover, CEnPiT uses data of the complete focus detector—it is not restricted to the n-Pi window. The latter fact is important for obtaining the optimal temporal resolution and the best SNR in all cases. Reconstruction algorithms are called gated algorithms if they assign weights to the projections and if the weights depend on the distance of the projection to the phase point. The CEnPiT approach separates the data into two contributions, μug and μg , which are added to result in the final image. In particular, μug contains contributions of Radon planes with up to n IPs, while μg contains contributions of Radon planes with more than n IPs. With discussion in mind, we realize that μug yields mainly low-frequency components. Together with the fact that μug contains Radon planes with only few redundancies, we are motivated to use an ungated backprojection when computing μug . Gating is applied only to obtain μg . For the possibilities to choose an adequate gating function we refer to Bontus et al. (2006). Here we summarize the filter-line sets, which can be used to separate the data into μug and μg .
56
BONTUS AND KÖHLER
2. Filter Lines The filter lines summarized here were first published by Köhler, Bontus and Koken (2006). The topic was discussed on an even wider basis in Köhler, Bontus and Koken (2006), where the results were used to incorporate overscan data. Overscan data correspond to measured data outside of the n-Pi window. (L) (R) , Pm , and The different kinds of filter-line sets are denoted as Lm(R) , Pm (R) T , where m can be any positive odd integer. The definition for sets Lm(R) has been given in Section III.F. For the definition of the remaining sets, we m F introduce helper lines Lm p and L± in addition to lines Lm , L−m , and L−p introduced in Section III.F (compare with Figure 41). In particular, line Lm p has a positive gradient and is tangential on the upper m-Pi window and on the lower p-Pi window. Lines LF± are parallel to L1 and tangential on the projected boundaries of the focus detector [compare with Eq. (33)]. Figure 45 shows some of these helper lines. The definition of filter-line set T (R) is similar to that of sets Lm(R) . Filter lines within T (R) are parallel to L1 , if (uP , vP ) is located between LF− and LF+ . Otherwise, the filter lines are tangential on the projected focusdetector borders [cf. Eq. (33)]. Figure 46 shows the filter-line set T (R) . The (L) (R) filter lines Pm and Pm are completely contained within the m-Pi window. (R) In particular, the corresponding filter line within Pm for a point (uP , vP ) within the m-Pi window fulfills the following conditions:
F F IGURE 45. Different lines of set Lm p and lines L± within the 7-Pi window. The dash-dotted lines correspond to the projections of the focus-detector boundaries [Eq. (33)].
RECONSTRUCTION ALGORITHMS
F IGURE 46.
57
Filter-line set T (R) .
IF (uP , vP ) is above Lm 3 , the filter line is tangential on the upper m-Pi window boundary , the filter line is tangential on the lower ELSE IF (uP , vP ) is above Lm−2 3 3-Pi window boundary , the filter line is tangential on the upper ELSE IF (uP , vP ) is above Lm−2 5 (m − 2)-Pi window boundary , the filter line is tangential on the lower ELSE IF (uP , vP ) is above Lm−4 5 5-Pi window boundary ··· ELSE IF (uP , vP ) is above L3m , the filter line is tangential on the upper 3-Pi window boundary ELSE the filter line is tangential on the lower m-Pi window boundary. (L) for a point (uP , vP ) within the The corresponding filter line within Pm m-Pi window fulfills the following conditions:
IF (uP , vP ) is above Lm −1 , the filter line is tangential on the upper m-Pi window boundary ELSE IF (uP , vP ) is above Lm−2 −1 , the filter line is tangential on the lower Pi-window boundary ELSE IF (uP , vP ) is above Lm−2 −3 , the filter line is tangential on the upper (m − 2)-Pi window boundary ELSE IF (uP , vP ) is above Lm−4 −3 , the filter line is tangential on the lower 3-Pi window boundary ··· ELSE IF (uP , vP ) is above L1−m , the filter line is tangential on the upper Pi-window boundary ELSE the filter line is tangential on the lower m-Pi window boundary.
58
BONTUS AND KÖHLER
F IGURE 47.
(R)
Filter-line sets P3
(R)
and P7 .
We still must specify whether the point of tangency is on the RHS or LHS (L) of (uP , vP ). In general, for filter lines within Pm , the point of tangency is on the LHS of (uP , vP ), if the line is tangential on one of the upper p-Pi window boundaries, while it is on the RHS, if the line is tangential on one of (R) the lower p-Pi-window boundaries. For filter lines within Pm , the point of tangency is on the RHS of (uP , vP ), if the line is tangential on one of the upper p-Pi window boundaries, while it is on the LHS, if the line is tangential (R) on one of the lower p-Pi window boundaries. Figure 47 exemplifies sets P3 and P7(R) , while Figure 48 illustrates sets P1(L) and P5(L) . 3. Gated and Ungated Contributions With the definitions of the filter lines we can now specify how the gated and ungated contributions μg and μug are obtained. We assume that n specifies the maximum Pi-mode; i.e. the n-Pi window can be projected completely onto the focus detector, while the (n + 2)-Pi window cannot. With this convention the filter-line sets and weights μν for μug are summarized in Table 2. The
RECONSTRUCTION ALGORITHMS
F IGURE 48.
(L)
Filter-line sets P1
59
(L)
and P5 .
contributions of all those filter-line sets must be added. For instance, line 2 in (R) (R) (R) Table 2 shows that sets L3 , L5 , . . . , Ln−2 must be incorporated using the specified weights. Table 3 summarizes the filter-line sets and weights for μg .
IV. O UTLOOK The publication of Katsevich’s Pi algorithm initiated many studies. It even resulted in new insights on fan-beam CT (Noo, Clackdoyle and Pack, 2004). We have summarized the principles and showed some applications of the methods. A more detailed analysis of the underlying mathematics was given by Chen (2003), Katsevich (2003). In practice, a utilization of all measured data is desired. A restriction to the n-Pi window in the case of helical CT is not acceptable for clinical applications. Köhler, Bontus and Koken (2006) showed how overscan data can be incorporated in the reconstruction. A nice FBP algorithm for the saddle trajectory was presented by Yang et al. (2006). Future applications could require algorithms for helical CT with
60
BONTUS AND KÖHLER TABLE 2 F ILTER -L INE S ETS AND THE C ORRESPONDING W EIGHTS FOR μug
TABLE 3 F ILTER -L INE S ETS AND THE C ORRESPONDING W EIGHTS FOR μg
nμν
Filter-line sets
(R)
n 3
Ln
(R)
2n p(p+2)
Pn
1
(R) Pn T (R)
Filter-line sets L1
Lp , 3 p < n (R) Ln (L) P1 (L) Pp , 3 p < n (L) Pn (R) Pp , 3 p < n (R) Pn
− n3
nμν
(R)
n − n+2
(L)
n 2(n+2) n 2(n+2) n n+2
−n p(p+2) − 12 −n p(p+2) 1 −2
a tilted gantry (Noo, Defrise and Kudo, 2004) or for acquisitions in which the table feed (pitch) is not constant (Katsevich, Basu and Hsieh, 2004; Ye, Zhu and Wang, 2004). Circular CT becomes increasingly attractive with greater numbers of detector rows. This is especially true if the organ under examination is completely covered by the X-ray cone. As discussed previously, a circular cone-beam acquisition yields an incomplete data set. The combination of the circle with a line provides one way to restore completeness as shown in Section III.D. Alternatively, the circle can also be combined with a helical segment (Bontus et al., 2007a, 2007b). The advantage of this approach is that the gantry need not be decelerated or accelerated, while the helical segment contributes only to low-frequency components in the same sense as the line does in the CLT. A completely new class of reconstruction algorithms appeared by exploitation of a formula derived by Zou and Pan (2004a). The authors derived two reconstruction algorithms for which the data must be filtered either in the image domain (Zou and Pan, 2004a) (backprojection filtering, BPF) or along projected Pi-lines (Zou and Pan, 2004b) (filtered backprojection, FBP). Another BPF formula was derived by Zhuang et al. (2004), which can be applied to different kinds of source trajectories. BPF algorithms apply a backprojection to data, that are only differentiated along the tangent of the trajectory [as in Eq. (43)]. The result corresponds to image data modified by a Hilbert transform. Therefore, an inverse Hilbert transform must be applied as one of the final processing steps. Pack, Noo and Clackdoyle (2005) presented a BPF method that can be used for different source trajectories and allows use of an arbitrary amount of overscan data during the backprojection.
RECONSTRUCTION ALGORITHMS
61
Nevertheless, with this method a large number of processing steps can become necessary to obtain an adequate SNR. CT reconstruction is an interesting and agile field of research. The future will show which of the proposed algorithms fulfill the requirements imposed by clinical practice.
R EFERENCES Bontus, C., Köhler, T., Proksa, R. (2003). A quasiexact reconstruction algorithm for helical CT using a 3-Pi acquisition. Med. Phys. 30 (9), 2493– 2502. Bontus, C., Köhler, T., Proksa, R. (2005). EnPiT: Filtered back-projection algorithm for helical CT using an n-Pi acquisition. IEEE Trans. Med. Imaging 24 (8), 977–986. Bontus, C., Koken, P., Köhler, T., Grass, M. (2006). CEnPiT: Helical cardiac CT reconstruction. Med. Phys. 33 (8), 2792–2799. Bontus, C., Grass, M., Koken, P., Köhler T. (2007a). Exact reconstruction algorithm for circular short-scan CT combined with a helical segment. In: Proceedings of the Fully 3D Meeting. Lindau, Germany, pp. 88–91. Bontus, C., Koken, P., Köhler, T., Proksa, R. (2007b). Circular CT in combination with a helical segment. Phys. Med. Biol. 52, 107–120. Chen, G.-H. (2003). An alternative derivation of Katsevich’s cone-beam reconstruction formula. Med. Phys. 30 (12), 3217–3225. Danielsson, P.E., Edholm, P., Eriksson, J., Magnusson-Seger, M. (1997). Towards exact 3D-reconstruction for helical cone-beam scanning of long objects. In: Proceedings of the 3D’97 Conference. Nemacolin, Pennsylvania, pp. 141–144. Defrise, M., Noo, F., Kudo, H. (2000). A solution to the long object problem in helical cone-beam tomography. Phys. Med. Biol. 45, 623–643. Grangeat, P. (1991). Mathematical Framework of Cone-Beam 3D-Reconstruction via the First Derivative of the Radon Transformation. Mathematical Methods in Tomography. Springer, Berlin, Germany. Jeffrey, A., Gradshteyn, I.S., Ryzhik, I.M. (Eds.) (1994). Table of Integrals, Series, and Products, fifth edition. Academic Press, San Diego. Katsevich, A. (2002). Theoretically exact FBP-type inversion algorithm for spiral CT. SIAM J. Appl. Math. 62, 2012–2026. Katsevich, A. (2003). A general scheme for constructing inversion algorithms for cone beam CT. Int. J. Math. Math. Sci. 21, 1305–1321. Katsevich, A. (2004a). Image reconstruction for the circle and line trajectory. Phys. Med. Biol. 49, 5059–5072. Katsevich, A. (2004b). An improved exact FBP algorithm for spiral CT. Adv. Appl. Math. 32 (4), 681–697.
62
BONTUS AND KÖHLER
Katsevich, A. (2006). 3PI algorithms for helical computed tomography. Adv. Appl. Math. 36, 213–250. Katsevich, A., Basu, S., Hsieh, J. (2004). Exact filtered backprojection reconstruction for dynamic pitch helical cone beam computed tomography. Phys. Med. Biol. 49, 3089–3103. Köhler, T., Bontus, C., Koken, P. (2006). The Radon-split method for helical cone-beam CT and its application to nongated reconstruction. IEEE Trans. Med. Imaging 25 (7), 882–897. Kudo, H., Noo, F., Defrise, M. (1998). Cone-beam filtered-backprojection algorithm for truncated helical data. Phys. Med. Biol. 43, 2885–2909. Manzke, R., Köhler, T., Nielsen, T., Hawkes, D., Grass, M. (2004). Automatic phase determination for retrospectively gated cardiac CT. Med. Phys. 31 (12), 3345–3362. Natterer, F. (1986). The Mathematics of Computerized Tomography. Wiley, New York. Noo, F., Clackdoyle, R., Pack, J. (2004). A two-step Hilbert transform method for 2D image reconstruction. Phys. Med. Biol. 49, 3903–3923. Noo, F., Defrise, M., Kudo, H. (2004). General reconstruction theory for multi-slice x-ray computed tomography with a gantry tilt. IEEE Trans. Med. Imaging 43 (9), 1109–1116. Noo, F., Pack, J., Heuscher, D. (2003). Exact helical reconstruction using native cone-beam geometries. Phys. Med. Biol. 48, 3787–3818. Pack, J.D., Noo, F., Clackdoyle, R. (2005). Cone-beam reconstruction using the backprojection of locally filtered projections. IEEE Trans. Med. Imaging 24 (1), 70–85. Proksa, R., Köhler, T., Grass, M., Timmer, J. (2000). The n-PI-method for helical cone-beam CT. IEEE Trans. Med. Imaging 19 (9), 848–863. Schaller, S., Noo, F., Sauer, F., Tam, K.C., Lauritsch, G., Flohr, T. (2000). Exact Radon rebinning algorithm for the long object problem in helical cone-beam CT. IEEE Trans. Med. Imaging 19 (5), 361–375. Tam, K.C. (1995). Three-dimensional computerized tomography scanning method and system for large objects with smaller area detectors. US Patent 5,390,112. Tam, K.C., Sauer, F., Lauritsch, G., Steinmetz, A. (1999). Backprojection spiral scan region-of-interest cone beam CT. Proc. SPIE Med. Imaging Conf. 3661, 433–441. Tuy, H.K. (1983). An inversion formula for cone-beam reconstructions. SIAM J. Appl. Math. 43 (3), 546–552. Yang, H., Li, M., Koizumi, K., Kudo, H. (2006). View-independent reconstruction algorithms for cone beam CT with general saddle trajectory. Phys. Med. Biol. 51, 3865–3884.
RECONSTRUCTION ALGORITHMS
63
Ye, Y., Zhu, J., Wang, G. (2004). Minimum detection windows, PI-line existence and uniqueness for helical cone-beam scanning of variable pitch. Med. Phys. 31 (3), 566–572. Zhuang, T., Leng, S., Nett, B.E., Chen, G. (2004). Fan-beam and cone-beam image reconstruction via filtering the backprojection image of differentiated projection data. Phys. Med. Biol. 49, 5489–5503. Zou, Y., Pan, X. (2004a). Exact image reconstruction on PI-lines from minimum data in helical cone-beam CT. Phys. Med. Biol. 49, 941–959. Zou, Y., Pan, X. (2004b). Image reconstruction on PI-lines by use of filtered backprojection in helical cone-beam CT. Phys. Med. Biol. 49, 2717–2731.
This page intentionally left blank
ADVANCES IN IMAGING AND ELECTRON PHYSICS, VOL. 151
Color Spaces and Image Segmentation LAURENT BUSIN1 , NICOLAS VANDENBROUCKE1,2 , AND LUDOVIC MACAIRE1 1 Laboratoire LAGIS UMR CNRS 8146 – Bâtiment P2,
Université des Sciences et Technologies de Lille, 59655 Villeneuve d’Ascq, France 2 Ecole d’Ingénieurs du Pas-de-Calais (EIPC), Campus de la Malasisse – BP39,
62967 Longuenesse Cedex, France
I. Introduction . . . . . . . . . . . . . . . . II. Color Spaces . . . . . . . . . . . . . . . A. Digital Color Image . . . . . . . . . . . . 1. Human Color Vision . . . . . . . . . . . 2. Color Image Acquisition . . . . . . . . . . 3. Color Image Visualization . . . . . . . . . . B. Color Spaces . . . . . . . . . . . . . . 1. Primary Spaces . . . . . . . . . . . . . 2. Luminance-Chrominance Spaces . . . . . . . . 3. Perceptual Spaces . . . . . . . . . . . . 4. Independent Axis Spaces . . . . . . . . . . 5. Hybrid Color Space . . . . . . . . . . . C. Digital Color Images and Color Spaces . . . . . . . 1. Color Space Coding for Image Analysis . . . . . 2. Application to Color Spaces . . . . . . . . . D. Summary . . . . . . . . . . . . . . . III. Color Image Segmentation . . . . . . . . . . . A. Introduction . . . . . . . . . . . . . . . B. Edge Detection . . . . . . . . . . . . . . 1. Overview . . . . . . . . . . . . . . 2. Edge Detection by the Analysis of a Color Gradient Vector C. Region Construction Based on a Spatial Analysis . . . . 1. Segmentation by Region Growing . . . . . . . 2. Segmentation by Region Merging . . . . . . . D. Region Construction Based on a Color Space Analysis . . 1. Introduction . . . . . . . . . . . . . . 2. Analysis of One-Dimensional Histograms . . . . . 3. Analysis of the Three-Dimensional Color Histogram . . 4. Clustering-Based Segmentation . . . . . . . . 5. Spatial-Color Classification . . . . . . . . . E. Summary . . . . . . . . . . . . . . . IV. Relationships between Segmentation and Color Spaces . . . A. Introduction . . . . . . . . . . . . . . . B. Edge Detection . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
66 66 67 67 71 74 75 76 84 92 100 102 104 104 106 109 110 110 111 111 114 115 115 117 119 119 123 129 133 136 139 140 140 141
65 ISSN 1076-5670 DOI: 10.1016/S1076-5670(07)00402-8
Copyright 2008, Elsevier Inc. All rights reserved.
66
BUSIN , VANDENBROUCKE AND MACAIRE
1. Quantitative Evaluation Methods with Ground Truth . . . . . . . 2. Evaluation Methods for Color Image Segmentation by Edge Detection . . C. Region Construction . . . . . . . . . . . . . . . . . 1. Quantitative Evaluation Methods with Ground Truth . . . . . . . 2. Quantitative Evaluation Methods without Ground Truth . . . . . . 3. Evaluation Methods for Color Image Segmentation by Region Construction D. Selection of the Most Discriminating Color Space . . . . . . . . 1. Pixel Classification in an Adapted Hybrid Color Space . . . . . . 2. Unsupervised Selection of the Most Discriminating Color Space . . . E. Conclusion . . . . . . . . . . . . . . . . . . . . V. Conclusion . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
141 143 143 143 146 147 147 149 151 161 161 162
I. I NTRODUCTION One of the most important problems in color image analysis is segmentation. This chapter considers color uniformity as a relevant criterion to partition an image into significant regions. For this purpose, the color of each pixel is initially represented in the (R, G, B) color space where the color component levels of the corresponding pixel are namely red (R), green (G), and blue (B). Other color spaces can be used, and the performance of an image segmentation procedure depends on the choice of the color space. Our goal is to describe the relationships between the color spaces and the color image segmentation. The different color spaces used to characterize the colors in the digital images are detailed in Section I. Section II is devoted to the segmentation methods designed for color image analysis. In Section III, we present several studies about the impact of the choice of the color space on the quality of results provided by segmentation methods.
II. C OLOR S PACES This section provides the details of how the colors of the pixels of digital color images can be represented in different color spaces for color image analysis applications. The first part of this section shows how color is quantized to produce a digital color image. The basics of human color vision are first summarized to facilitate understanding of this scheme. Then the problem of color image acquisition is discussed. In the second part, the most classical color spaces are regrouped into different families. In the framework of the use of these color spaces by color image analysis applications, the last part highlights the need to efficiently code the components of these color spaces to preserve their intrinsic properties.
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 1.
67
Digital color image acquisition.
A. Digital Color Image A digital color image acquisition device is composed of different elements (Figure 1). First, the characteristics of the light and the material that constitute the observed object determine the physical properties of its color. Then, color images are acquired by an image sensor with an optical device and finally digitized thanks to a digital processor (Sharma and Trussell, 1997; Trussell, Saber and Vrhel, 2005; Vrhel, Saber and Trussell, 2005) before being displayed on a monitor or printed by a printer. To understand how color is represented in digital images, it is first necessary to explain how it is perceived by a human observer. Thus, the basics of the color human vision are presented first. Then, the color image acquisition device is briefly described, and finally, the color image display device is presented. 1. Human Color Vision The human perception of color is the response of the human receptor (the eye) and the human interpretation system (the brain) to a color stimulus— the reflection or the transmission of a light source by a material. In addition to the light and the material characterized by their physical properties, the eye and the brain, which are specific to each observer, provide physiologic and psychological features to the color sensation (Wyszecki and Stiles, 1982). This section presents these different properties of the color, which are very complex notions. a. Physical Properties. First, there is no color without any light. The light, at the beginning of the color sensation, is defined as a electromagnetic radiation—a set of electromagnetic waves produced by the propagation of luminous particles, the photons. An electromagnetic radiation is characterized
68
BUSIN , VANDENBROUCKE AND MACAIRE
by its wavelength λ expressed in meters (m). Visible light is the region of the electromagnetic radiation emitted by the sun for which our eyes are sensitive (between ∼ 380 and 780 nm). This light can be scattered in a spectrum of luminous rays thanks to a prism. Each ray is composed of radiations with the same wavelength, the monochromatic waves. Thus, each light source is characterized by its spectral power distribution (SPD), the quantity of energy per unit of wavelength (usually 1 nm). Certain sources corresponding with classical observation conditions have been normalized by the Commission Internationale de l’Éclairage (CIE) under the name of illuminant (Commission Internationale de l’Éclairage, 1986). CIE is an international standardization organization devoted to establishing recommendations for all matters relating to the science and art of lighting. Another characteristic of an illuminant is its correlated color temperature—the temperature for which it would be necessary to carry a black body radiator to obtain the visual impression nearest to that produced by the corresponding source of light. Illuminants A (light emitted by a black body radiator at the temperature of 2856 ◦ K or incandescence lamp), C (direct light of the sun at midday), D (daylight), F (fluorescent lamp), and E (light of equal energy) are the most widely used illuminants. When the light source lights a material, the luminous rays emitted by this source are reflected or transmitted by this material. Thus, the material transforms the properties of the light to produce a color stimulus. When the light comes into contact with a material, two phenomena occur: a surface reflection of the incident radiations and a penetration of the incident light into the material. When the light penetrates into the material, it meets the pigments. Pigments are particles that determine the color of material while absorbing, diffusing, or transmitting the light that reaches them. The pigments modify the SPD of the light by selectively absorbing a part of the corresponding electromagnetic waves. The light that is not absorbed by the pigments is diffused or transmitted outside the material and carries the color information of this material. According to its nature, a material can be characterized by its capacity to reflect (reflectance) or to transmit (transmittance) incident light energy. Conversely, it also can be characterized by its capacity to absorb (absorption) the incident light energy. Finally, the color of a material depends not only on its characteristics but also on the conditions of lighting and viewing (position, orientation, distance, and so on). b. Physiologic and Psychological Properties. After crossing several elements of the eye, the color stimulus reaches a photosensitive zone at the back of the eye, the retina, where the images of the observed scene are projected. The retina contains two kinds of photosensitive cells: cones and rods. The rods allow night vision (scotopic vision) while the cones allow diurnal vision (photopic vision). There are three kinds of cones: the S cones, which are
COLOR SPACES AND IMAGE SEGMENTATION
69
sensitive to short wavelengths close to blue; the M cones, which are sensitive to medium wavelengths close to green; and the L cones, which are sensitive to long wavelengths close to red. The eye converts a color stimulus into a color signal at the entry of the optic nerve. The optic nerve transmits this electric signal to the external (or lateral) geniculate body, a relay charged to establish connections with fibers going to the brain. A first analysis of the data is achieved there. According to the opposite colors theory of Hering (1875), it seems that the color signal is coded in an antagonistic fashion—by an achromatic signal (black-white opposition) and by red-green and blue-yellow opposition signals (Hering, 1875). These signals finally are transmitted to another area of the brain, the visual cortex, where color interpretation is performed. c. Trichromacy. Color human perception is characterized by its threedimensional (3D) aspect. The works of Young at the beginning of the nineteenth century and Helmholtz in 1866 showed that any color stimulus can be reproduced by the mixture of three other stimuli: the red, the green and the blue stimuli, termed primaries or reference stimuli (Helmholtz, 1866; Young, 1807). This principle is known as trichromacy, trichromatic theory, or color mixture. Three primaries are therefore necessary and sufficient to match any color by mixture; colorimetry, the science of color measurement, is based on this theory. The amounts of each of the primaries necessary to match a color are called the tristimulus values. There are two kinds of color mixture: the additive and subtractive color mixtures (Figure 2). The
(a) F IGURE 2. Insert.)
(b)
Color mixture. (a) Additive color mixture. (b) Subtractive color mixture. (See Color
70
BUSIN , VANDENBROUCKE AND MACAIRE
additive color mixture is the result of the juxtaposition of colored lights corresponding to each of the three primaries. The additive mixture in equal amount of the three primaries creates white. Additive color mixture is used for constituting the image of a television, a monitor, or the image acquired by a color video or still camera. The subtractive color mixture is the result of the selective absorption principle of the light by a material with respect to different wavelengths. Subtractive color mixture is used in printing or painting. The primaries used differ according to the type of color mixture. The primaries used by additive color mixture are the red, the green and the blue colors, whereas the magenta, the cyan, and the yellow colors are used by subtractive color mixture (the color black is often added as a primary in printing applications). The primaries of these two types of color mixture are complementary. The additive color mixture of two complementary colors creates the color white. On the basis of the trichromatic theory established by Young, in 1853 Grassman proposed the laws consigning the fundamental properties of the color mixtures that were supplemented by Abney in 1913 (Abney, 1913; Grassman, 1853). These laws make it possible to apply the additive, associative, multiplicative, and transitive properties of the algebraic equalities to the colorimetric equalities. Today Grassman’s laws are the mathematical basis of colorimetry. d. Perceptual Attributes. The human perception of color is a subjective reaction to the stimulation of the eye, and it seems to be more adapted to characterize a color in terms of brightness, hue, and saturation. Brightness is the attribute of a visual sensation according to which an area appears to emit more or less light. Thus, it corresponds to a feeling in terms of light (like dark or luminous) and characterizes the luminous level of a color stimulus. The terms brightness, lightness, luminance, luminosity, radiant intensity, luminous intensity, illumination, luma, and so on are often used in the literature to indicate the concept of luminosity. Poynton (1993) highlights the confusion between these terms. Hue corresponds to the main colors: red, green, blue and yellow. It is the attribute of a visual sensation according to which an area appears to be similar to one of these perceived colors. It corresponds to the dominant wavelength of a SPD—the wavelength with the highest energy. Hue can be defined by an angle termed hue angle. Since black, white, or gray are colors without any hue, they are called neutral colors or achromatic colors. Saturation is an attribute that allows the colorfulness of an area to be estimated with respect to its brightness. The chroma is a brightness-dependent attribute that also estimates the colorfulness of the hue. Saturation represents the purity of a perceived color, such as bright, pale, dusty, and so on. It
COLOR SPACES AND IMAGE SEGMENTATION
71
corresponds to the ratio between the energy of the dominant wavelength of an SPD and the sum of the energies of the other wavelengths. 2. Color Image Acquisition The previous section explained that, in the framework of human vision, the eye and the brain are the receptor and the interpretation systems of the color stimuli, respectively. In the framework of computer vision, the receptor is a color camera and the interpretation system is a computer (see Figure 1). This section describes how a color image can be acquired by a color camera and be digitized by an image acquisition device. a. Color Camera. The main element of a camera is the image sensor (Trussell, Saber and Vrhel, 2005; Vrhel, Saber and Trussell, 2005). The image sensor is composed of a set of photosensitive elements that convert the luminous flux (photon) into an electric information (electron) to provide one or several video signals to the digitization system. Each photosensitive receptor produces an increasing voltage in function of the received light intensity. They can be arranged either as a row (line scan camera) or as an array (area array camera). The obtained image is made up of a set of points called pixels,1 which correspond to the photosensitive receptor. Two types of technology exist to make a sensor array: charged coupled device (CCD) technology and complimentary metal-oxide–semiconductor (CMOS) technology. The CCD sensors provide higher signal-to-noise ratio (SNR) levels, whereas the CMOS sensors have a lower SNR but allow the integration of other components into each photosensitive receptor of the sensor. By analogy with the human vision system, color information is created by means of three color filters. Each is sensitive to the short (red), the medium (blue), or the long (green) wavelengths. Two kinds of color camera exist (Figure 3): • One-chip color cameras are designed with one single-sensor array and three interlaced color filters, termed a color filter array. A Bayer color filter array is the most widely used device. Thus, the photosensitive receptors that are laid sequentially on a same line of the sensor are fitted with red, green, and blue filters in succession. The color information is obtained by receptors located at different places. This technology generates a loss of resolution and chromatic aberration (false colors). Therefore, it requires interpolation techniques (termed demosaicking) so that the color of each pixel of the image is defined by three components (Gunturk et al., 2005). Other 1 PICture ELement = pixel.
72
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b) F IGURE 3. Insert.)
Color camera. (a) One-chip color camera. (b) Three-chip color camera. (See Color
technologies, based on only one sensor array, resolve these problems (Lyon and Hubel, 2002). • The three-chip color cameras are fitted out with three sensor arrays associated with an optical system based on prisms. Each of the three sensors receives simultaneously either the red, or the green, or the blue stimulus via dichroic filters fixed on prisms. The color of a pixel is provided by the response of three receptors and therefore there is not loss of resolution. Demosaicking is not necessary, and the quality of the image is better than the quality of an image acquired by a one-chip color camera. However, the three-chip technology can generate the phenomenon of shading, which creates a color scale on the image of a white background when luminous rays reaching the prisms are not perfectly parallel.
COLOR SPACES AND IMAGE SEGMENTATION
73
Color cameras are fitted with infrared filters because sensors are sensitive to wavelengths located over the visible domain. The spectral response of an image sensor is not the same as that of the spectral response of the human eye. b. Color Image Digitization. A digitization system achieves the conversion (amplification, sampling, and so on) of one or more video signals derived from an image sensor into a digital signal where each element of a color image is represented by three digital values that can be stored in the computer memory for further processing and display application. This stage of acquisition is called color quantization. Color quantization consists of associating, for each pixel of a color image, three digital values—a red level (R), a green level (G), and a blue level (B)— which correspond to the tristimulus values of this color. Thus, a digital color image is an array of pixels that are located by their spatial coordinates in the image plane and whose color is defined by the three components R, G, and B. Generally, each of these three components is coded on 8 bits and are quantized with 256 different unsigned integer values belonging to the interval [0, 255]. A color is so coded with 3 × 8 (24) bits and it is then possible to represent 224 (16,777,216) colors by additive mixture, whereas the human visual system allows differentiation of approximatively 350,000 colors. However, even though this quantization represents more colors than the human observer can see, the human visual system is not uniform; for specific intervals, it is necessary to code the color component on 12 bits to discriminate all the colors that the human visual system can perceive on this interval. Digital values, which are provided by the digitization system, depend on the characteristics of the device used, the choice of lighting system, and the setup of the acquisition system. Indeed, the SPD of a light depends on the lighting system, and the spectral responses of an image sensor depend on the chosen color camera. Thus, the R, G, and B components of a given color are not the same with different equipments and parameter setups; therefore, the color components of an acquired image are device dependent (or rendered) (Trussell, Saber and Vrhel, 2005). To ensure that the R, G, and B color components are invariant to acquisition system, it is necessary to apply a calibration step. Calibration is achieved by using a chart (Macbeth Color Checker, IT8 chart, white target) with different colors whose theoretical tristimulus values are known a priori. The image of this chart is acquired by the acquisition system and the parameters of this system are adjusted so that the R, G, and B components of each color on the chart in the image are the same as their theoretical values. This adjustment can be achieved via simple white balance or by more precise numerical methods (Bala and Sharma, 2005; Ramanath et al., 2005).
74
BUSIN , VANDENBROUCKE AND MACAIRE
3. Color Image Visualization In most applications, color images are designed to be displayed on a screen or printed on paper. For color image analysis applications, it is not always necessary to display the color when the result of the analysis is not an image but instead some data. However, many color spaces have been developed to display color images on televisions or monitors and depend on their technologies. In order to understand these color spaces, which are presented later, we describe how the colors are displayed. a. Display. The display of acquired color image is usually achieved by a cathode ray tube (CRT) monitor or by a liquid crystal display (LCD) flat panel monitor (Vrhel, Saber and Trussell, 2005). The screen of the CRT monitor consists of a thin layer of a mosaic of red, green, and blue phosphor cells. The CRT is to bombard the screen with electron beams that stimulate each of the three types of phosphor with varying intensities. Three colored lights (whose spectral characteristics depend on the chemical nature of the phosphor) are produced and display a color on the screen according to the additive mixture principle. For an LCD flat panel monitor, color display is produced by a mosaic of red, green, and blue color filters that are backlit by a fluorescent light source. These backlight and color filters were designed to produce chromaticities identical to those of the CRT phosphors to ensure compatibility between the two devices. To achieve the display of a color image on a screen, the digital values of the color components of each pixel of the image are loaded into the memory of the video card. They then are converted to analog data to specify the intensity of the beam (the amount of light that reaches the screen). This intensity, denoted x, is a nonlinear function, denoted Γ , of the voltage generated by the video card. For a CRT monitor, it follows a power law defined by Γ (x) = x γ , where the value of γ generally ranges between 2 and 3. For an LCD monitor, the response of an LCD pixel cell is quite different from that of a CRT and tends to follow a sigmoidal law. To correct this nonlinearity, an inverse law must be applied when the image is displayed. This operation is known as gamma correction. b. Gamma Correction. Generally, the acquisition system performs the gamma correction, which consists of compensating for the nonlinearity of display devices by transforming the acquired video signals so that the R, G, and B trichromatic components of the pixels of the acquired image are corrected. For example, a television channel transmits the signals that can be displayed on a CRT TV screen and are gamma corrected. This correction depends on
COLOR SPACES AND IMAGE SEGMENTATION
75
the technology used, which defines the primaries necessary to achieve the additive mixture of the colors of the image. More precisely, this technology depends on the country. Thus, North American television receivers accept both National Television Standards Committee (NTSC) old standard, which uses the primaries defined by the Federal Communications Commission (FCC), and the Society of Motion Picture and Television Engineers (SMPTEC) new standard. For the NTSC standard, γ is set to 2.2. European television receivers satisfy the phase alternation by line (PAL) German standard, which uses the primaries defined by the European Broadcasting Union (EBU) or can satisfy the SÉquentiel Couleur À Mémoire (SECAM) French standard. For the PAL standard, γ is set to 2.8. With most color cameras, it is possible to adjust the gamma correction directly on the red, green, and blue acquired signals. This camera setup allows correction of the display problems only on monitors and so, it is not necessary to use this setup for color image analysis application. On the other hand, to render color applications or for color measure applications on a screen, it is necessary to apply gamma correction on the displayed color image. When color images whose acquisition conditions are not specified (web images) are considered, it is very difficult to know whether these images are gamma corrected. Poynton (1996) noticed an amazing coincidence between the gamma correction law and the transfer function of the human eye. Indeed, the human eye response to a stimulus is nonlinear but corresponds to a logarithmic law very close to the gamma correction law. Thus the application of a gamma correction on the acquired color component produces a color representation close to the human color perception. According to Poynton (1993), even if the gamma correction is not necessary for technological reasons, it is useful for perceptual reasons. B. Color Spaces A color space is a geometrical representation of colors in a space and allows specification of colors by means of (generally) three components whose numerical values define a specific color. The first proposed color spaces to specify colors were based on color human perception experiments designed to develop classifications of colors. The ordered color systems are collections of color samples (color charts, color atlas). These spaces are generally 3D spaces where the units of the components are merely conventional and help to specify colors within this representation. These spaces (e.g., Munsell system, Natural Color System [NCS], Optical Society of America [OSA] system, or Deutsches Institut für Normung [DIN] system) are rarely used for color image analysis applications.
76
BUSIN , VANDENBROUCKE AND MACAIRE
The most frequently used spaces propose a metric to measure the distance between colors. The goal of this section of the text is to present the most frequently used classical color spaces in the framework of color image analysis. According to their characteristics, they are classified into four families: 1. The primary spaces are based on the trichromatic theory, assuming that it is possible to match any color by mixing appropriate amounts of three primary colors. 2. The luminance-chrominance spaces where one component represents the luminosity and the two others the chromaticity. 3. The perceptual spaces attempt to quantify the subjective human color perception by using the intensity, the hue, and the saturation components. 4. The independent axis spaces resulting from different statistical methods, which provide the least correlated components as possible. Since many color spaces for different applications and several definitions for the same color space exist, it is difficult to select only one for a color image analysis application; therefore it is important to provide an overview of their specificities and their differences. 1. Primary Spaces The primary spaces are based on the trichromatic theory, which assumes that we are able to match any color by mixing appropriate amounts of three primary colors. By analogy to physiology of the human eye, the primary colors are close to red (R), green (G), and blue (B). Therefore, a primary space depends on the choice of a set of primary colors denoted here [R], [G], and [B]. Ratios fixed for each primary are defined in order to reproduce a reference white (or white point), denoted [W ], by the additive mixing of the same amount of each primary. The choice of the primary depends on the device used, and the reference white is generally a CIE illuminant. The next sections of the text present the most useful primary spaces. They can be separated into the real primary spaces, for which the primary colors are physically feasible, and the imaginary primary spaces, whose primaries do not physically exist. a. Real Primary Spaces. The CIE (R, G, B) color space is derived from color-matching experiments led in 1931 by Wright and Guild (Commission Internationale de l’Éclairage, 1986) that used the primaries, denoted here [RC ], [GC ] and [BC ] (where C indicates CIE), respectively, to match all the available colors of the visible spectrum. These primaries are red, green, and blue monochromatic color stimuli with wavelengths of 700.0, 546.1, and
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 4.
77
RGB color cube. (See Color Insert.)
435.8 nm, respectively, and the reference white is the equal-energy illuminant E (Commission Internationale de l’Éclairage, 1986). The CIE (R, G, B) color space can be considered as the reference (R, G, B) color space because it defines a standard observer whose eye spectral response approximates the average eye spectral response of a representative set of human observers. For each of the three [RC ], [GC ], and [BC ] primaries there correspond three −−→ −−→ −−→ normalized vectors RC , GC , and BC , respectively, that constitute the reference of a vectorial space whose original point is denoted O. In this space, each color stimulus [C] is represented by a point C that defines the color vector −−→ OC. The coordinates of this vector are the tristimulus values RC , GC , and BC . The coordinates of specific color vectors are negative because they correspond to color stimuli that are not reproducible (matched) by additive mixture. Points that correspond to color stimuli with positive tristimulus values are inside a cube, named the RGB color cube (Figure 4). The original point O corresponds to the black color (RC = GC = BC = 0), whereas the reference white is defined by the additive mixture of equal quantities of the three primaries (RC = GC = BC = 1).
78
BUSIN , VANDENBROUCKE AND MACAIRE
The straight line joining the points Black and White in Figure 4 is called the gray axis, the neutral color axis, or the achromatic axis. Indeed, the points of this line represent gray nuances from black to white. The tristimulus values of a color stimulus depend on the color’s luminance. Two different color stimuli can be described by the same chromatic characteristics (here called chrominance), but their tristimulus values can be different due to their luminance. In order to obtain color components that do not depend on the luminance, it is necessary to normalize their values. For this purpose, it is possible to divide each color component value by the sum of the three ones. The three thus-obtained color components, called chromaticity coordinates or normalized coordinates, are denoted rC , gC , and bC , respectively, and are defined by: ⎧ RC ⎪ r = RC +G , ⎪ C +BC ⎨ C GC gC = RC +G , (1) C +BC ⎪ ⎪ ⎩b = BC . C
RC +GC +BC
The transformation defined by Eq. (1) corresponds to the projection of the C point on the plane that is normal to the achromatic axis. This plane is defined by the equation: RC + GC + BC = 1, and the intersection between this plane and the RGB color cube constitutes an equilateral triangle whose summits are the three primaries [RC ], [GC ], and [BC ]. This triangle, termed the Maxwell triangle, is represented by a dotted line in Figure 4. The color space that can be associated with the chromaticity coordinates is called the normalized (RC , GC , BC ) color space and is denoted (rC , gC , bC ). As rC + gC + bC = 1, two components are sufficient to represent the chrominance of a color stimulus. Thus, Wright and Guild have proposed a diagram called the chromaticity diagram (rC , gC ). Figure 5 represents this diagram, which contains a curve (the spectrum locus), joining the points corresponding to monochromatic color stimuli whose wavelengths range between 380 and 780 nm. The two extremities of this curve are connected by a straight line known as the nonspectral line of purples. All the colors of the visible spectrum are contained in the area defined by the spectrum locus and the purple line. In this figure, the Maxwell triangle does not contain all the colors because several colors cannot be matched by additive mixture and are defined by negative chromaticity coordinates. As mentioned previously, the CIE (R, G, B) color space can be considered as the reference (R, G, B) color space because it defines the standard observer. In many application fields, however, it is not possible to use this color space and other primary colors must be used. These other (R, G, B) color spaces differ by the primaries used to match colors by additive mixture and by the used reference white. Thus, they are device dependent
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 5.
79
CIE chromaticity diagram (rC , gC ).
color spaces (Süsstrunk, Buckley and Swen, 1999). The most widely known (R, G, B) color spaces used by the following application fields are as follows: • Video and image acquisition. In color image analysis applications, the images can be acquired by analog or digital video camera, by digital still camera, by scanner, and so on. The color of each pixel of an image is defined by three numerical components that depend on the acquisition device and its setup to obtain a reference white. Thus, an image acquired with the same lighting and observation conditions by two different cameras produces different colors if the primaries’ colors associated with the two cameras are not the same. In this chapter, the (R, G, B) color space used for acquiring images is called the image acquisition space. • Television image display. For TV analog signal transmission, it is assumed that the chromaticity coordinates of the primaries are different from those of
80
BUSIN , VANDENBROUCKE AND MACAIRE
the CIE primaries because they correspond to phosphors or color filters with different wavelengths. The different TV signal transmission standards use (R, G, B) color spaces that are different. For example, the NTSC standard uses the (RF , GF , BF ) color space based on the FCC primaries; the PAL and SECAM standards use the (RE , GE , BE ) color space based on the EBU primaries, and SMPTE-C standard uses the (RS , GS , BS ) color space based on the SMPTE primaries. In the same way, the reference white to reproduce depends on the display technology and its setup. For example, we generally assume that the reference white used by the NTSC standard is the C illuminant, whereas the D65 illuminant is used by the PAL/SECAM or SMPTE-C standard. • Computer graphics. Primaries and reference white used by the monitors of computers are defined by their manufacturers and differ from those used in television or defined by CIE. In the same way, when an image is generated by computer, the color of the pixel depends on the (R, G, B) color space that uses the software for image coding. • Image printing. Printing systems usually use color spaces based on a subtractive color mixture. The primaries used are cyan, magenta, and yellow, and the primary color space is the (C, M, Y ) color space that can be derived from an (R, G, B) color space by the relation: C R (2) M =1− G . Y B The reference white used in printing systems is often the D50 illuminant. This multitude of primary color spaces leads to many errors in color management. It is necessary to dispose of an (R, G, B) color space well suited to CRT and LCD monitors, TVs, scanners, cameras, and printers to ensure compatibility between these different devices and so that the colors are rendered identically (Bala and Sharma, 2005; Ramanath et al., 2005). This is why the sRGB color space has been proposed as a standard default (R, G, B) color space that characterizes an average monitor (IEC 61966-2-1/FDIS, 1999). This color space can be considered an independent device (or unrendered) color space for color management applications. The reference white and the primaries are defined by the International Telecommunication Union (ITU) standard according to the ITU-R BT.709 recommendation (ITU-R BT.709-5, 2002). Thus, the sRGB color space is denoted here as (RI , GI , BI ). Regardless of which primary space is used, it is possible to achieve a primary conversion as a simple linear transformation of the component values by means of a matrix equation. So there exists a transformation matrix P that allows (R, G, B) color space conversions (Pratt, 1978, 1991).
COLOR SPACES AND IMAGE SEGMENTATION
81
For each primary space, it is possible to define a RGB color cube and chromaticity coordinates. b. Imaginary Primary Space. The (R, G, B) color spaces present some major drawbacks: • Because it is not possible to match all the colors by additive mixture with a real primary space, the tristimulus values and chromaticity coordinates can be negative. • The tristimulus values depend on the luminance, which is a linear transformation of the R, G, B color components. • Because (R, G, B) color spaces are device dependent, there is a multitude of (R, G, B) color spaces with different characteristics. The CIE therefore defines the imaginary primary space, named the CIE (X, Y, Z) color space, where the primary colors are imaginary (virtual or artificial) in order to overcome the problems of the primary spaces. In this space, the [X], [Y ], and [Z] primaries are not physically realizable, but they have been defined so that all the color stimuli are expressed by positive tristimulus values and so that one of these primaries, the [Y ] primary, represents the luminance component. Because all the (R, G, B) color spaces can be converted to the (X, Y, Z) color space by linear transforms, this space is an independent device color space. It defines an ideal observer, the CIE 1931 standard colorimetric observer, and all the colorimetry applications are based on it. Chromaticity coordinates can be deduced from the (X, Y, Z) color space to obtain a normalized (X, Y, Z) color space denoted (x, y, z). Thus, the chromaticity coordinates x, y, and z are derived from the X, Y , and Z tristimulus values by: ⎧ X ⎪ ⎨ x = X+Y +Z , y = X+YY +Z , ⎪ ⎩ z = X+YZ +Z .
(3)
As x + y + z = 1, z can be deduced from x and y. Thus, the colors can be represented in a plane called the (x, y) chromaticity diagram (Figure 6). In this diagram, all the colors are inside the area delimited by the spectrum locus and the purple line. Since the chromaticity coordinates of these colors are positive, the CIE primaries [X], [Y ], and [Z] allow matching of any color by additive color mixture.
82
BUSIN , VANDENBROUCKE AND MACAIRE
F IGURE 6.
(x, y) Chromaticity diagram.
The conversion from an (R, G, B) color space to the (X, Y, Z) color space is defined by the following equation: X R X R XG XB (4) Y = P × G , with P = YR YG YB . ZR ZG ZB Z B The coefficients of the P matrix are defined with respect to the [R], [G], and [B] primaries and the reference white used to reproduce the equal mixing of the three [X], [Y ], [Z] primaries. Generally, the primaries and the reference white are characterized by their x and y chromaticity coordinates. Table 1 lists the x and y chromaticity coordinates of the primaries used by different (R, G, B) color spaces, and Table 2 lists those of the A, C, D65, D50, F2, and E illuminants.
COLOR SPACES AND IMAGE SEGMENTATION
83
TABLE 1 x, y AND z C HROMATICITY C OORDINATES OF D IFFERENT P RIMARIES Standard
Primaries
x
y
z
CIE
[RC ] [GC ] [BC ]
0.735 0.274 0.167
0.265 0.717 0.009
0.000 0.009 0.824
FCC
[RF ] [GF ] [BF ]
0.670 0.210 0.140
0.330 0.710 0.080
0.000 0.080 0.780
SMPTE
[RS ] [GS ] [BS ]
0.630 0.310 0.155
0.340 0.595 0.070
0.030 0.095 0.775
EBU
[RE ] [GE ] [BE ]
0.640 0.290 0.150
0.330 0.600 0.060
0.030 0.011 0.790
ITU
[RI ] [GI ] [BI ]
0.640 0.300 0.150
0.330 0.600 0.060
0.030 0.010 0.790
TABLE 2 x, y AND z C HROMATICITY C OORDINATES OF D IFFERENT I LLUMINANTS AND T HEIR C ORRELATED C OLOR T EMPERATURES (Tp ) Illuminant
Tp (◦ K)
x
y
z
A C D50 D65 E F2
2856 6774 5000 6504 5400 4200
0.448 0.310 0.346 0.313 0.333 0.372
0.407 0.316 0.358 0.329 0.333 0.375
0.145 0.374 0.296 0.358 0.333 0.253
A set of primaries defines a triangle in the (x, y) chromaticity diagram whose summits are the chromaticity coordinates of these primaries. Examples of this triangle and white points have shown in the (x, y) chromaticity diagram in Figure 7. This figure shows that none of these triangles contains all the visible colors; this means that all the colors cannot be matched by additive mixture with an (R, G, B) color space. Each (R, G, B) color space defines a set of matched colors called a gamut.
84
BUSIN , VANDENBROUCKE AND MACAIRE
F IGURE 7.
Illuminants and gamuts in the (x, y) chromaticity diagram.
Table 3 provides examples of P transformation matrices used to convert (R, G, B) color spaces to the (X, Y, Z) color space (Pascale, 2003). 2. Luminance-Chrominance Spaces Because a color can be expressed by its luminosity and chromaticity, several color spaces separate the luminance from the chrominance information. These color spaces can be categorized in the family of luminance-chrominance spaces. The components of a luminance-chrominance space are derived from the component of an (R, G, B) color space by linear or nonlinear transformations. The type of transformation depends on the type of luminance-chrominance spaces that can be classified in the following spaces:
COLOR SPACES AND IMAGE SEGMENTATION
85
TABLE 3 (R, G, B) TO (X, Y, Z) C OLOR S PACE C ONVERSION (PASCALE , 2003) Primary system
Color space
Reference white
CIE
(RC , GC , BC )
E
FCC
(RF , GF , BF )
C
SMPTE
(RS , GS , BS )
D65
EBU
(RE , GE , BE )
D65
ITU
(RI , GI , BI )
D65
P transformation matrix ⎡ 0.489 0.311 ⎣ 0.176 0.813 ⎡ 0.000 0.010 0.607 0.174 ⎣ 0.299 0.587 ⎡ 0.000 0.066 0.394 0.365 ⎣ 0.212 0.701 ⎡ 0.019 0.112 0.431 0.342 ⎣ 0.222 0.707 ⎡ 0.020 0.130 0.412 0.358 ⎣ 0.213 0.715 0.019 0.119
⎤ 0.200 0.011 ⎦ 0.990 ⎤ 0.200 0.114 ⎦ 1.116 ⎤ 0.192 0.087 ⎦ 0.958 ⎤ 0.178 0.071 ⎦ 0.939 ⎤ 0.180 0.072 ⎦ 0.950
• The perceptually uniform spaces, which propose a metric to establish a correspondence between a color difference perceived by a human observer and a distance measured in the color space. • The television spaces, which separate luminosity signal to chromaticity signals for the television signal transmission. • The antagonist (or opponent color) spaces, which aim to reproduce the model of the opponent color theory proposed by Hering. Let L be the luminance component of a luminance-chrominance space. In function of the specified luminance-chrominance space, the luminance component can represent the lightness, brightness, luminous intensity, luma, or luminance. The term luminance used in this chapter is a general terminology of one of these magnitudes. Let Chr1 and Chr2 be the two chrominance components of a luminance-chrominance space. A luminance-chrominance space is there denoted as (L, Chr1 , Chr2 ). a. Perceptually Uniform Spaces. Nonuniformity is one drawback of the (X, Y, Z) color space. Indeed, in the (x, y) chromaticity diagram, an equal distance between two color points does not correspond to the same color difference perceived by a human observer according to the considered area of the diagram. Thus there are areas in the (x, y) chromaticity diagram for which color differences are not perceptible by a human observer. The size and orientation of these regions, known as the MacAdam ellipses, depend
86
BUSIN , VANDENBROUCKE AND MACAIRE
on their positions in the diagram (MacAdam, 1985). This nonuniformity is a problem for applications where it is necessary to measure color difference. Indeed, colors that are perceptually close can be separated by longer distances, whereas colors that are perceptually different can be close in the (x, y) chromaticity diagram. Because the Euclidean distances evaluated in the (R, G, B) or (X, Y, Z) color spaces do not correspond to the color differences that are actually perceived by an human observer, the CIE recommends two perceptually uniform spaces—the (L∗ , u∗ , v ∗ ) and (L∗ , a ∗ , b∗ ) color spaces—where L∗ represents the lightness (luminance component) and where u∗ , v ∗ and a ∗ , b∗ are chromaticity coordinates (chrominance components) (Commission Internationale de l’Éclairage, 1986). The first perceptually uniform color space proposed by the CIE in 1960 was the (U, V , W ) color space, which is derived from the (X, Y, Z) color space. In this color space, V is a luminance component and it is possible to define a uniform chromaticity diagram (called the 1960 Uniform Chromaticity Scale [UCS] diagram or (u, v) chromaticity diagram). Because the (U, V , W ) color space does not satisfy the nonuniformity problem, the CIE proposed the (U ∗ , V ∗ , W ∗ ) in 1964 (Pratt, 1978, 1991). This color space finally underwent significant modification in 1976 to become the (L∗ , u∗ , v ∗ ) color space (also named the CIELUV color space). The luminance component of the (L∗ , u∗ , v ∗ ) color space is expressed as: ⎧ ⎨ 116 × 3 Y − 16 if Y > 0.008856, YW YW L∗ = (5) ⎩ 903.3 × Y if Y 0.008856, YW
YW
where X W , Y W , and Z W are the tristimulus values of the reference white. The chrominance components are (6) u∗ = 13 × L∗ × u − u W , (7) v ∗ = 13 × L∗ × v − v W , with 4X , X + 15Y + 3Z 9Y , v = X + 15Y + 3Z
u =
(8) (9)
where u W and v W are the chrominance components of u and v for the reference white, respectively. Figure 8 shows the CIE 1976 (u , v ) chromaticity diagram derived from the (L∗ , u∗ , v ∗ ) color space.
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 8.
87
(u , v ) Chromaticity diagram.
The (L∗ , a ∗ , b∗ ) color space proposed by the CIE (also named the CIELAB color space) is derived from the (X, Y, Z) color space by nonlinear relations. The luminance component is defined by Eq. (5) and the chrominance components are expressed as Y X ∗ −f , (10) a = 500 × f XW YW Z Y b∗ = 200 × f − f , (11) YW ZW with
√ 3 x f (x) = 7.787x +
16 116
if x > 0.008856, if x 0.008856.
(12)
88
BUSIN , VANDENBROUCKE AND MACAIRE
In these two last uniform color spaces, the luminance component L corresponds to the lightness (or brightness) and represents the human eye response to a level of luminance. The CIE models this nonlinear response by a cubic root relation. The first chrominance component (a ∗ or u∗ ) of these two color spaces corresponds to a green-red color opposition, whereas the second chrominance component (b∗ or v ∗ ) corresponds to a blue-yellow color opposition. b. Television Spaces. The transmission of TV signals requires the separation between the luminance and chrominance components. This separation can be achieved by a linear transformation of the component values of an (R, G, B) color space. The luminance component corresponds to the Y color component of the (X, Y, Z) color space. The Chr1 and Chr2 chrominance components are as follows:
Chr1 = a1 (R − Y ) + b1 (B − Y ), (13) Chr2 = a2 (R − Y ) + b2 (B − Y ). The coefficients a1 , b1 , a2 , and b2 are specific to the norms used, standards, or commissions (e.g., NTSC, PAL, SECAM, SMPTE). Because the Y color component is evaluated by means of a linear transformation of the R, G, and B components, it is possible to express the conversion of an (R, G, B) color space [or (X, Y, Z) color space] to (Y, Chr1 , Chr2 ) color space by using a transformation matrix Q: R Y X (14) Chr1 = Q × G = Q × P × Y . Chr2 B Z Notice that the signal transmitted by TV channels are gamma corrected for different reasons (display, bandwidth, human eye response and so forth). Therefore, the conversion of an (R, G, B) color space to a television space is applied to gamma-corrected R, G, and B components, denoted R , G , and B . The use of the prime ( ) notation is extended to all color components that are gamma corrected. The main television spaces are as follows: • (Y , I , Q ) color space used by the NTSC television standard, where Y is the luminance component and I and Q are the chrominance components. The Q transformation matrix (listed in Table 3) is used to define the Y color component as: Y = 0.299 × R + 0.587 × G + 0.114 × B .
(15)
The chrominance components are defined by Eq. (13) with a1 = 0.74, b1 = −0.27, a2 = 0.48 and b2 = 0.41.
COLOR SPACES AND IMAGE SEGMENTATION
89
• (Y , U , V ) color space used by the EBU broadcasting standard where the chrominance components are U and V . To ensure compatibility among the different TV standards, the luminance component of the EBU standard is also defined by Eq. (15). The chrominance components are defined by Eq. (13) with a1 = 0, b1 = 0.493, a2 = 0.877 and b2 = 0. • (Y , Dr , Db ) color space used by the SECAM color TV broadcasting standard, where the chrominance components are Dr and Dr . The luminance component is also defined by Eq. (15), and the chrominance components are defined by Eq. (13) with a1 = −1.9, b1 = 0, a2 = 0 and b2 = 1.5. • (Y , Cb , Cr ) color space is an international standard for digital image and video coding. ITU proposes two recommendations: ◦ The ITU-R BT.601 recommendation for digital coding of standarddefinition television (SDTV) signals. According to this recommendation, the (Y , Cb , Cr ) color space is independent of the primaries and the reference white (ITU-R BT.601-7, 2007) and it is also used by video and image-compression schemes such as MPEG and JPEG. The luminance component is also defined by Eq. (15), and the chrominance components are defined by Eq. (13) with a1 = 0, b1 = 0.564, a2 = 0.713 and b2 = 0. ◦ The ITU-R BT.709 recommendation for digital coding of high-definition television (HDTV) signals (ITU-R BT.709-5, 2002) with a1 = 0, b1 = 0.534, a2 = 0.635 and b2 = 0. Table 4 lists some transformation matrices to convert a primary space to one of the main TV spaces. c. Antagonist Spaces. The antagonist spaces are based on the Hering color opponent theory in order to model the human visual system. According to this theory, the color information acquired by the eye is transmitted to the brain in three components: an achromatic component A and two chrominance components C1 and C2 . The A color component integrates the signals derived from the three types of cones of the human retina and represents a blackwhite opposition signal, whereas the C1 and C2 components integrate only the signals provided by different types of cones and correspond to green-red and yellow-blue oppositions, respectively. The (L∗ , u∗ , v ∗ ) and (L∗ , a ∗ , b∗ ) color spaces previously presented also can be considered as antagonist spaces because they share these same properties. Different antagonist spaces have been proposed for color image analysis. In 1976, Faugeras proposed a human visual system model in which A, C1 , and C2 color components are evaluated from three primaries, denoted [L], [M], and [S], which correspond to the three types of cones of the human retina (Faugeras, 1979). He proposed transformation matrices to convert an (R, G, B) color space to the (L, M, S) color space. For example,
90
BUSIN , VANDENBROUCKE AND MACAIRE TABLE 4 P RIMARY S PACE TO T ELEVISION S PACE C ONVERSION
Norm– Standard
Television space
Primary space
Reference white
NTSC
(Y , I , Q )
(RF , GF , BF )
C
PAL
(Y , U , V )
(RE , GE , BE )
D65
SECAM
(Y , Dr , Db )
(RE , GE , BE )
D65
ITU-R BT.601
(Y , Cb , Cr )
None
None
ITU-R BT.709
(Y , Cb , Cr )
(RI , GI , BI )
D65
Q transformation matrix ⎡ ⎤ 0.299 0.587 0.114 ⎣ 0.596 −0.274 −0.322 ⎦ ⎡ 0.212 −0.523 0.311 ⎤ 0.299 0.587 0.114 ⎣ −0.148 −0.289 0.437 ⎦ ⎤ ⎡ 0.615 −0.515 −0.100 0.299 0.587 0.114 ⎣ −0.450 −0.883 1.333 ⎦ ⎡ −1.333 1.116 0.217 ⎤ 0.299 0.587 0.114 ⎣ −0.169 −0.331 0.500 ⎦ ⎡ 0.500 −0.419 −0.081 ⎤ 0.222 0.715 0.072 ⎣ −0.115 −0.386 0.500 ⎦ 0.501 −0.455 −0.046
the conversion of an (R, G, B) color space used by a CRT monitor to the (L, M, S) color space with the C illuminant as reference white can be achieved by the following transformation matrix: 0.3634 0.6102 0.0264 P = 0.1246 0.8138 0.0616 . (16) 0.0009 0.0602 0.9389 A, C1 , and C2 color components are defined by: A = a α log(L) + β log(M) + γ log(S) , C1 = u1 log(L) − log(M) , C2 = u2 log(L) − log(S) .
(17) (18) (19)
These equations indicate that the cone response to a color stimulus is nonlinear. Therefore, Faugeras proposed to model this nonlinearity by using the logarithmic function, while the CIE method uses the cube root function [see Eq. (5)]. By adjusting the a, α, β, γ , u1 , and u2 parameters, Faugeras proposes different applications of the model. For example, he uses the following coefficients for color image analysis: a = 22.6, α = 0.612, β = 0.369, γ = 0.019, u1 = 64, and u2 = 10. In the framework of artificial vision, Garbay propose application of this space directly on the R, G, and B color components of an image acquisition
COLOR SPACES AND IMAGE SEGMENTATION
91
system (Chassery and Garbay, 1984; Garbay, Brugal and Choquet, 1981). The a, α, β, γ , u1 , and u2 parameters are defined by: 1 × log(R) + log(G) + log(B) , (20) 3 √ 3 C1 = × log(R) − log(G) , (21) 2 log(R) + log(G) . (22) C2 = log(B) − 2 Ballard and colleagues proposed the use of an antagonist space that does not take into account the nonlinearity of the human eye response (Ballard and Brown, 1982; Swain and Ballard, 1991). Thus, the equation of this space, denoted as (wb, rg, by), can be written with a transformation matrix: ⎡ 1 1 1⎤ 3 3 3 wb R √ ⎥ ⎢ √3 3 (23) rg = P × G , with P = ⎣ 2 − 2 0 ⎦. by B 1 1 −2 −2 1 A=
Other human visual system models have been applied in the color image analysis field, such as the (G1 , G2 , G3 ), (G∗1 , G∗2 , G∗3 ) or (H1 , H2 , H3 ) color spaces (Braquelaire and Brun, 1997; Robinson, 1977; Pratt, 1978). d. Other Luminance-Chrominance Spaces. Other luminance-chrominance spaces, which cannot be directly classified in the previous subfamilies of luminance-chrominance spaces, are applied in color image analysis. By studying the properties of different luminance-chrominance spaces, Lambert and Carron (1999) proposed a transformation matrix to convert an (R, G, B) color space to the luminance-chrominance space denoted (Y, Ch1 , Ch2 ) in which the luminance component is defined by the wb color component of Eq. (23), whereas the chrominance components are defined by the following: ⎡1 1 1 ⎤ 3 3 3 R Y ⎢ ⎥ (24) Ch1 = P × G , with P = ⎣ 1 − 12 − 12 ⎦ . √ √ Ch2 B 3 3 0 − 2 2 The Ch1 and Ch2 color components correspond to a cyan-red and green-blue color oppositions, respectively. By associating the r and g chromaticity coordinates with the luminance component defined by Eq. (23) and denoted here I , the (I, r, g) luminancechrominance space is defined. This space is often used in color image
92
BUSIN , VANDENBROUCKE AND MACAIRE
analysis (Nevatia, 1977; Ohta, Kanade and Sakai, 1980; Sarabi and Aggarwal, 1981). CIE proposed use of the (Y, x, y) color space, where Y represents the luminance component [see Eq. (15)] and x and y, which are the chromaticity coordinates of the (X, Y, Z) color space [see Eq. (3)], represent the chrominance. 3. Perceptual Spaces Humans do not directly perceive color as a combination of tristimulus values related to primary colors but according to more subjective entities related to luminosity, hue, and saturation. Therefore, it is natural that many color spaces quantify the color according to these perceptual data and are grouped here into the perceptual space family. This perceptual approach allows an adequate communication between human vision and machinery for describing colors. There exist many such color spaces presented with different notations, such ISH, HSB, HSL, HSV, LCH, LSH, and so on. These different notations correspond to the same color components, but the equations to determine them are different. Two kinds of perceptual spaces can be distinguished: • The polar (or cylindrical) coordinate spaces that correspond to expressions in polar coordinates of the luminance-chrominance components2 • The perceptual coordinate spaces that are directly evaluated from primary spaces. Perceptual spaces also can be considered as luminance-chrominance ones because they are consist of a luminance component and two chrominance components. a. Polar Coordinates Spaces. This family of color spaces is derived from the color spaces that separate color information into a luminance axis and a chrominance plane by transposition of Cartesian coordinates to polar coordinates. Let P be a point that represents a color in an (L, Chr1 , Chr2 ) luminance-chrominance space. This point is− defined by the coordinates of −−→ → −−−→ −−−→ the color vector OP in the reference (O, L, Chr1 , Chr2 ). Let P be the projection of P on the (O, Chr1 , Chr2 ) plane along the L axis and let O be the orthogonal projection P−→on the L axis. By construction, the norm −−−→ of−− −−−→ and the −−−→orientation of O P and OP vectors are equal as well as for the OO and P P vectors. Thus it is possible to locate the P point via the norm of the −−−→ −−−→ −−−→ OP vector, the angle between the OP and Chr1 vectors, −−−→ and its coordinate along the L axis, which is equal to the norm of the OO vector. The three components so obtained are denoted L, C, and H and constitute the (L, C, H ) polar coordinate space (see Figure 9). 2 These color spaces are also named spherical color transform (SCT).
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 9.
93
Polar coordinate space.
The first component of a polar coordinate space represents the luminance component, L, which is identical to the first color component of the corresponding luminance-chrominance space. −−−→ The norm of the OP vector represents the chroma, C, which is defined by −−−→ C = OP = Chr21 + Chr22 . (25) The chroma corresponds to the distance between the P point and the luminance axis. −−−→ −−−→ The angle of the OP vector with the Chr1 vector represents the hue, H . In order to obtain hue values ranging between 0 and 2π , the evaluation of H must satisfy these conditions: −−− → −−−→ with H = Chr 1 , OP ⎧ if Chr1 > 0 and Chr2 0 then 0 H < π , ⎪ ⎪ ⎨ if Chr1 0 and Chr2 > 0 then π H < 2π, 2 (26) if Chr1 < 0 and Chr2 0 then π H < 3π ⎪ 2 , ⎪ ⎩ if Chr1 0 and Chr2 < 0 then 3π 2 H < 2π.
94
BUSIN , VANDENBROUCKE AND MACAIRE
The trigonometric functions are used to evaluated H . For example, H can be defined by: Chr2 . (27) H = arctan Chr1 This equation gives an angle belonging in the interval [−π/2, π/2]. In order to obtain an angle belonging in the interval [0, 2π ] and satisfying Eq. (26), it is necessary to first apply an operation on Chr2 when Chr1 < 0: if Chr2 0 then H = π + H , else H = π − H . This first operation gives an angle belonging in the interval [−π, π ]. So the following second operation is applied: H = H + 2π , if H < 0. By definition, it is possible to construct an (L, C, H ) color space from any of the luminance-chrominance spaces. For example, by using Eqs. (25) and (27), CIE defines the components of two (L, C, H ) polar coordinate spaces from the perceptually uniform spaces (L∗ , u∗ , v ∗ ) and (L∗ , a ∗ , b∗ ), ∗ , h ) and (L∗ , C ∗ , h ), respectively. These spaces are denoted (L∗uv , Cuv uv ab ab ab respectively (Commission Internationale de l’Éclairage, 1986). The components of these color spaces are used to evaluate color differences (Commission Internationale de l’Éclairage, 1986, 1995). b. Perceptual Coordinate Spaces. The perceptual coordinate spaces are evaluated directly from a primary space and represent the subjective entities of the color human perception in terms of intensity (I ), saturation (S), and hue (T ). The intensity I corresponds to a luminance component, and the saturation S is related to a chroma component expressed by the relation S = C / L. The perceptual coordinate spaces are denoted here as (I, S, T ) to differentiate them from the polar coordinate spaces. This part presents the most widely used perceptual coordinate spaces (Shih, 1995): • The triangle model • The hexcone model • The double hexcone model. To propose a perceptual coordinate space that takes into account most of the models, Levkowitz and Herman (1993) proposed a generalized lightness, hue, and saturation (GLHS) color model. i. Triangle Model. The triangle model corresponds to the expression of the I , S, and T color components in the RGB color cube (Kender, 1976). In the representation of the RGB color cube (see Figure 4), the achromatic axis corresponds to the intensity axis in the (I, S, T ) color space. A point P , whose coordinates are the values of the R, G, and B color components, is located on a plane perpendicular to the achromatic component. The intersections of this
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 10.
95
Triangle model. (See Color Insert.)
plane with the red, green, and blue axes are the summits of a triangle (α, β, γ ), which is proportional to the Maxwell triangle (Figure 10). On the (α, β, γ ) triangle, it is possible to evaluate the saturation and hue components. Different methods have been proposed, and many formulations of the (I, S, T ) color spaces exist. Some of them are presented here. the achromatic axis. The I Let O be the orthogonal projection of P on−− −→ color component corresponds to the norm of the OO vector. To maximize its value to the unity when R = G = B = 1, the I color component is evaluated by the equation: 1 −−−→ R + G + B . (28) I = √ OO = 3 3 To simplify this calculation, a nonweighted formulation of the previous equation is often used: I = R + G + B.
(29)
−−−→ The saturation corresponds to the ratio between the norm of the O P vector
(distance between the P point and the achromatic axis) and the norm of the
96
BUSIN , VANDENBROUCKE AND MACAIRE
−−−→ OO
vector, which represents the intensity component. The calculation of the S color component is by √ −−−→ R 2 + G2 + B 2 − RG − GB − RB O P √ . (30) S = −−−→ = 2 × R+G+B OO
In this formulation, colors with equal saturation values are located on a circular base cone centered on the achromatic axis and with O as summit. The saturation value reaches its maximal value only for the three primary colors. In order to maximize the saturation value for all points belonging to the boundaries of the (α, β, γ ) triangle (like those corresponding to complementary colors), the saturation component is evaluated by the minimum distance between the P point and one boundary of this triangle: 3 × min(R, G, B) . (31) R+G+B Different formulations of this equation exist and it is important to show their relations. Thus, another similar equation can be used to define the S color component: S =1−
S = 1 − 3 × min(r, g, b).
(32)
This equation uses the r, g, and b chromaticity coordinates and corresponds to the evaluation of the saturation component directly in the Maxwell triangle. The saturation component can be expressed in function of the intensity component of Eq. (29): 3 × min(R, G, B) . (33) I −−−→ The hue component corresponds to the orientation of the O P vector. Let M be the intersection point between the achromatic axis and the Maxwell triangle, and let P be the projection of the P point on this triangle along the achromatic axis. Generally, the axis defined by the M point and the Red point [coordinates (1, 0, 0)] − in−−→ the RGB color cube is the reference axis to define the orientation of the O P vector and to specify the value of the hue component. So −−−−− −−→ −−−→ MP T = M Rouge, S =1−
and the value of the red color is set to 0. The hue component can be evaluated via trigonometric relations. For example, the following equation defines the T color component: √ 3(G − B) T = arctan . (34) 2R − G − B
COLOR SPACES AND IMAGE SEGMENTATION
97
To obtain hue values included between 0 and 2π , it is necessary to test the sign of the numerator and the denominator. Another analog trigonometric equation can be used to define the T color component: 1 ⎧ [(R−G)+(R−B)] ⎪ if B G, ⎨ arccos √ 2 (R−G)2 +(R−B)(G−B) T = (35) 1 ⎪ ⎩ 2π − arccos √ 2 [(R−G)+(R−B)] if B > G. 2 (R−G) +(R−B)(G−B)
With this formulation, it is necessary to test if B > G in order to consider only those angles included between π and 2π . By using the chromaticity coordinates, Eq. (35) can be written as ⎧ 2r−g−b ⎪ arccos if b g, ⎪ 2 2 2 ⎨ T =
6×[(r− 13 ) +(g− 31 ) +(b− 31 ) ]
⎪ ⎪ ⎩ 2π − arccos
2r−g−b 2
2
2
6×[(r− 13 ) +(g− 31 ) +(b− 31 ) ]
(36) if b > g.
All these formulas correspond to the triangle model and can be associated to constitute the (I, S, T ) color space. ii. Hexcone Model. Each point P , whose coordinates are the values of the R, G, and B color components, belongs to a face of a color cube whose summit O corresponds to the maximum level of the R, G, and B components. The projection of the points of this color cube along the achromatic axis on the plane perpendicular to this axis and joining the O point constitutes a hexagonal closed area whose summits are the projections of the summits of the so defined color cube and whose center is the O point. In this model, which is represented by Figure 11, it is thus possible to define the I , S, and T color components. The intensity component, known by the term value and denoted V , is represented by this achromatic point and is expressed as 1 −−−→ I = V = √ OO = max(R, G, B). (37) 3 The projection on the plane perpendicular to the achromatic axis and joining the summit O of the so constructed color cube defines a hexagon in which is located the P point, projection of P . The saturation component corresponds to the length of the O P segment divided by the maximal length for a same hue. It is expressed by Eq. (33): S=
V − min(R, G, B) . V
(38)
98
BUSIN , VANDENBROUCKE AND MACAIRE
F IGURE 11.
Hexcone model. (See Color Insert.)
In the case when V = 0, the saturation components cannot be defined (hence S = 0). By construction, the White point is also the projection of the Black point. −−−−−−−−−→ −−−−−−−−→ The projection of the Black Red vector is thus the White Red vector. Let M be the projection of P on the plane perpendicular to the achromatic axis and joining the White point. The hue component is defined as the angle between −−−−−−−−−→ −−−−−−−→ the White M vector and the White Red vector. Therefore, −−−−−−−− −→ −−−−−−−→ , White M T = White Red and the T color component is evaluated, for all S = 0, by ⎧ G−B ⎪ if V = R, ⎪ ⎨ V −min(R,G,B) B−R if V = G, T = 2 + V −min(R,G,B) ⎪ ⎪ R−G ⎩4 + if V = B.
(39)
V −min(R,G,B)
If S = 0 (when R = G = B), then the hue component is not defined. Moreover, if V = R and min(R, G, B) = G, then T is negative. The hexcone model is defined by Eqs. (37), (38), and (39) (Foley, Dam and Feiner, 1990; Marcu and Abe, 1995; Shih, 1995).
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 12.
99
Double hexcone model. (See Color Insert.)
iii. Double Hexcone Model. This model is based on the same principle as the previous model, except that the projections of the subcube are achieved on each side of the plane perpendicular to achromatic axis and joining the middle of this axis (Foley, Dam and Feiner, 1990; Marcu and Abe, 1995; Shih, 1995). Figure 12 shows a representation of the double hexcone model. If min = min(R, G, B) and max = max(R, G, B), the intensity component is expressed as max + min . (40) 2 Let Imax be the maximum value of the I color component. The saturation component is defined, for all I = 0, by I=
S=
max−min max+min max−min 2×Imax −max−min
if I if I >
Imax 2 , Imax 2 .
(41)
100
BUSIN , VANDENBROUCKE AND MACAIRE
The saturation component is equal to 0 (S = 0) if I = 0 and the hue component is not defined if S = 0 (when R = G = B). Otherwise, the relations that allow processing the T color component are the same as those of the above hexcone color model [see Eq. (39)]. ∗ , h ) Color Space. This space was defined by the CIE at iv. (L∗uv , Suv uv ∗ , h ) and (L∗ , C ∗ , h ) color spaces were the same time that the (L∗uv , Cuv uv ab ab ab ∗ ∗ ∗ defined. In the (L , u , v ) color space, CIE defines the saturation component as the ratio ∗ Suv =
∗ Cuv . L∗uv
(42)
The components L∗uv and huv are defined by Eqs. (5) and (25), respectively (Commission Internationale de l’Éclairage, 1986). 4. Independent Axis Spaces Because the color components of an (R, G, B) color space are strongly correlated, they share common luminance information (Lee, Chang and Kim, 1994; Ohta, Kanade and Sakai, 1980). Therefore, many authors attempt to determine color spaces whose components are independent (i.e., components that share different noncorrelated and nonredundant informations). A general solution consists in applying a Karhunen–Loeve transformation or a principal component analysis (PCA) to the components of a color space. a. Principal Component Analysis. Principal component analysis is a data analysis method. Its goal is to analyze a set of quantitative data represented in a multidimensional space to obtain a representation space with (eventually) a reduced dimension whose components, named principal components, are uncorrelated and share different informations. When the data are the values of the R, G, and B color components of pixels of an image, the principal components analysis provides a color space whose color components are as uncorrelated as possible and can be independently considered. For this purpose, the set of color vectors within a color image is characterized by its diagonalized covariance matrix. The eigenvectors (denoted wi of this matrix) are determined and the principal components (denoted Xi ) are processed by the relation: Xi = wi [R G B]T . The Karhunen–Loeve transformation is used to apply this relation to each of the Xi components. Thus, this linear transformation consists of projecting the data into another space. The maximal eigenvalue λi corresponds to the first principal component X1 —the component that mainly represents the image. The components
COLOR SPACES AND IMAGE SEGMENTATION
101
are thus ordered with respect to their decreasing discriminant powers (their eigenvalues λi ). This transformation, which also allows reduction of the dimension of the color space, is principally applied from an (R, G, B) color space but can be derived from any of color spaces (Savoji and Burge, 1985; Tominaga, 1992). The drawback of the PCA-based methods is that it depends on the statistical properties of a data set. In color image analysis, the application of PCA on each analyzed color image is time consuming. To reduce computation time, several authors determined independent axis spaces that approximate the Karhunen–Loeve transformation by applying PCA on several different sets of images, such as (I 1, I 2, I 3), (P 1, P 2, P 3), (I, J, K), or (i1new , i2new , i3new ) color spaces (Ohta, Kanade and Sakai, 1980; Philipp and Rath, 2002; Robinson, 1977; Verikas, Malmqvist and Bergman, 1997). In the framework of natural color image analysis, an experiment led by Ohta et al. in 1980 on eight different images allows determination of a color space based on the Karhunen–Loeve transformation (Ohta, Kanade and Sakai, 1980). These authors proposed segmenting color images into regions by a recursive thresholding method and applying a PCA at each iteration step of the algorithm. They showed that there exists a single transformation that allows efficient conversion of the (R, G, B) color space to a space denoted (I 1, I 2, I 3). This transformation can be defined by this transformation matrix: ⎡ 1 1 1 ⎤ 3 3 3 I1 R (43) I 2 = P × G , with P = ⎣ 21 0 − 12 ⎦ . I3 B 1 1 1 − − 4
2
4
The first component of the (I 1, I 2, I 3) color space is the most discriminating one and represents a luminance component because it verifies Eq. (28). The two other color components represent the blue-red and magenta-green color oppositions, respectively. These two color components are less discriminating than the first one because their corresponding eigenvalues are low and the third component is not always used by color image analysis. Thus, the (I 1, I 2, I 3) color space can also be considered a luminance-chrominance space. b. Independent Component Analysis. Another solution to obtain an independent axis space consists of transforming the R, G, and B color components of the pixels to independent components via an independent component analysis method. This method allows determination of color spaces whose components are statistically independent but where information is dispersed on all components, whereas the first component deduced by PCA contains the maximum information (Lee and Lewicki, 2002).
102
BUSIN , VANDENBROUCKE AND MACAIRE
5. Hybrid Color Space The analysis of the colors of the pixels in a color space is not restricted to the (R, G, B) color space. Indeed, the previous text shows that there exist a large number of color spaces that can be used to represent the colors of the pixels (Sharma and Trussell, 1997). They are supported by specific physical, physiologic, and psychovisual properties (Sangwine and Horne, 1998). The multitude and the diversity of available color spaces requires classifying them into categories according to their definitions (Poynton, 1995; Tkalcic and Tasic, 2003). We propose grouping the most classical color spaces into four principal families that are further divided into subfamilies. Figure 13 distinguishes these four main families by four main rectangles: • • • •
Primary spaces Luminance-chrominance spaces Perceptual space Independent axis spaces.
The dotted rectangles within the main rectangles correspond to subfamilies of color spaces. In general, color images are acquired through the (R, G, B) color image acquisition space. Therefore, all color spaces are defined by an equation whose inputs are the R, G, and B color components. Figure 13 shows how a color space is determined by following the arrows starting from the (R, G, B) color space. We have compiled a nonexhaustive list of the most widely used color spaces by image analysis applications. Authors sometimes propose color spaces that are specific to their applications. For example, CIE presently proposes a color appearance model (CIECAM) that attempts to consider all the visual phenomena interfering on the color human perception (Commission Internationale de l’Éclairage, 2004). This empirical model defines the color of a surface by taking into account its neighborhood. In the color image analysis framework, the use of such a color model leads to considering the neighborhood of the pixels to evaluate color components. On the one hand, when considering the multitude of available color spaces, it is difficult to select a specific color space adapted to all the color image analysis applications. On the other hand, the performance of an image-processing procedure depends on the choice of the color space. Many authors have tried to determine the color spaces that are well suited for their specific color image segmentation problems. Authors provide contradictory conclusions about the relevance of the available color spaces in the context of color image analysis; it is thus easy to conclude that, there currently does not exist any classical color space that provides satisfying results for the analysis of all types of color images.
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 13.
Color space families. (See Color Insert.)
103
104
BUSIN , VANDENBROUCKE AND MACAIRE
Instead of searching for the best classical color space for color image analysis, Vandenbroucke, Macaire, and Postaire proposed an original approach to improve the results of image processing. They defined a new kind of color space by selecting a set of color components that can belong to any of the different classical color spaces listed in Figure 13 (Vandenbroucke, Macaire and Postaire, 1998). These spaces, which have neither psychovisual nor physical color significance, are called hybrid color spaces and can be automatically selected by means of an iterative feature selection procedure operating with supervised learning scheme (Vandenbroucke, Macaire and Postaire, 2003). C. Digital Color Images and Color Spaces In the previous section, the most classical color spaces were presented and classified into different families. However, the use of these color spaces in color image analysis requires some precautions. Indeed, when a color image is converted to an image where colors are represented in another color space, their values belong to different value domains in function of the applied transformation. In analyzing the color distribution of an image in a color space (e.g., Euclidean distance comparison, histogram evaluation, memory storage, image coding, color display), the values of the transformed color components often must belong to the same value domain as that in the original color space. To satisfy this constraint, it is necessary to code each color space without changing its intrinsic properties (Vandenbroucke, Macaire and Postaire, 2000a). This last condition proposes a color space-coding scheme applied to the different families of color spaces as presented by Figure 13. 1. Color Space Coding for Image Analysis The coding of the color spaces consists of rounding, scaling, and normalizing the values of their components in order to process values that range between the unsigned integer values 0 and 255. We propose a color space-coding scheme that preserves the properties of each of the color spaces. a. Independent and Dependent Coding. The components of all color spaces are defined by one or several successive color transformations of the R, G, and B components. Generally, each of these three components is coded on 8 bits and are quantized with 256 different unsigned integer values belonging to the interval [0, 255]. Let us denote T1 , T2 , and T3 , the transformed components of any color spaces obtained from the R, G, and B components by a set of color
COLOR SPACES AND IMAGE SEGMENTATION
F IGURE 14.
105
Color space coding.
transformations T. In most of cases, transformed color components assume signed floating values that do not range between 0 and 255. For several color image analysis applications, the transformed values of the color components must be comparable to those represented by the original color space. That means that a specific coding scheme is necessary. Let T1 , T2 and T3 , be the coded components provided by a transformation, denoted C (Figure 14). The space-coding scheme of a transformed color space (T1 , T2 , T3 ) is divided into different successive steps as follows: • Shifting the color component values so that they are unsigned. This operation requires knowing the minimal value of each component of the considered color space. • Normalizing the shifted component values so that they range between 0 and 255 (for 8 bit coding). This operation requires knowing the range of each component of the considered color space (i.e., their minimal and maximal values). • Rounding the normalized shifted component values to obtain integer values. These values are rounded to the nearest integer. In order to achieve the coding of the transformed color components T1 , T2 , and T3 , it is necessary to determine the extreme values of their ranges. Let us denote mk and Mk (k ∈ {1, 2, 3}), the minimum and maximum values of the transformed color component Tk , respectively, so that k = Mk − mk represents the range of the transformed color component Tk . In order to adjust the width of this range to 255, the transformed color component Tk can be coded independently of the two other components as 255 × (Tk − mk ). (44) k According to the properties of a color space, it is sometimes necessary to use a dependent coding scheme so that the range of at least one color component is equal to 255. Let max denote the larger range of the three components of a color space, defined as Tk =
max = max (k ). Tk
(45)
106
BUSIN , VANDENBROUCKE AND MACAIRE
The dependent coding of the components of a color space is expressed as 255 × (Tk − mk ). (46) max The dependent coding scheme achieves an equal scaling process for each component of a color space. Therefore, the relative position of colors in the space defined by this coding scheme is not modified. Furthermore, the Euclidean distances between colors are preserved by such a coding scheme. Tk =
2. Application to Color Spaces In order to preserve their properties, we apply one of the two above-defined coding schemes to the color spaces of the four families described by Figure 13. a. Primary Spaces. The transformation of an (R, G, B) color space to any other primary space is a linear transformation that depends on the choice of the primaries and the reference white. For instance, the transformation from the NTSC (RF , GF , BF ) color space to the CIE (X, Y, Z) color space with the C illuminant is defined via a transformation matrix [X Y Z]T = T [RF GF BF ]T (see Table 3), where 0.607 0.174 0.200 T = 0.299 0.587 0.114 . (47) 0.000 0.066 1.116 The chromaticity coordinates of the C illuminant are xn = 0.310 and yn = 0.316, and those of the primaries RF , GF , BF are xr = 0.670, yr = 0.330, xg = 0.210, yg = 0.710, and xb = 0.140 and yb = 0.080, respectively. Table 5 lists the coding parameters of the (X, Y, Z) color space. Ohta, Kanade and Sakai (1980) proposed coding this color space by means of an independent coding scheme associated with a matrix Cind so that [T1 T2 T3 ]T = Cind T [RF GF BF ]T with 0.618 0.177 0.205 Cind T = 0.299 0.587 0.114 . (48) 0.000 0.056 0.944 TABLE 5 X, Y AND Z C OMPONENTS E XTREMES VALUES
mk Mk
X
Y
Z
0 250.16
0 255.00
0 301.41
COLOR SPACES AND IMAGE SEGMENTATION
107
With such a coding scheme, the evaluated chromaticity coordinates are xn = 0.333, yn = 0.333, xr = 0.674, yr = 0.326, xg = 0.224, yg = 0.743, and xb = 0.162 and yb = 0.090. They do not correspond to the coordinates that are computed by the matrix T. Since the position of the C illuminant in the chromaticity diagram has changed with the independent coding scheme, we can conclude that the definition of the (X, Y, Z) color space from the primary space NTSC is modified by such a coding scheme. By extending the case of this illuminant to all the available colors, we conclude that the spectrum locus contained by the (x, y) chromaticity diagram is distorted by the independent coding scheme. Conversely, the dependent coding scheme provides a matrix Cdep so that: 0.514 0.147 0.169 Cdep T = 0.253 0.497 0.096 . (49) 0.000 0.056 0.944 The dependent coding scheme does not modify the actual chromaticity coordinates. This example can be generalized to the other primary spaces. Thus, in order to preserve the colorimetric properties of a primary space, we propose to achieve a dependent coding for each of its components. b. Luminance-Chrominance Spaces. The color spaces of this family are defined by linear or nonlinear transformations. The color spaces obtained through a linear transformation can be considered as primary spaces and must be coded with a dependent coding scheme for the same reasons as the primary spaces. The color components of the (L∗ , u∗ , v ∗ ) color space are defined by nonlinear transformations [see Eqs. (5)–(7)]. By applying an independent coding to the components of this space, the ellipses of MacAdam are distorted so that the color distances that are evaluated in this color space do not correspond to the color differences perceived by a human observer. To preserve the shape of MacAdam ellipses and to preserve an adequacy between the Euclidean distance and the visual perception, we propose application of a dependent coding scheme to the perceptually uniform spaces. To illustrate this phenomenon, an ellipse is represented in the (u∗ , v ∗ ) chromaticity diagram of Figure 15a before applying a coding scheme. In this figure, the two distances denoted D1 and D2 are equal. MacAdam rules assume that there are no perceptible differences between the corresponding colors for the human eye. By applying an independent coding scheme to the (L∗ , u∗ , v ∗ ) color space (Figure 15b), these properties are modified because D1 = D2 , whereas with a dependent coding scheme, these properties are always satisfied because D1 = D2 (Figure 15c). We extend this dependent coding scheme to all the luminance-chrominance spaces obtained through nonlinear transformations.
108
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c)
F IGURE 15. Application of the independent and dependent codings to an ellipse represented in the (u∗ , v ∗ ) chromaticity diagram. (a) Ellipse in the (u∗ , v ∗ ) chromaticity diagram. (b) Independent coding. (c) Dependent coding.
COLOR SPACES AND IMAGE SEGMENTATION
109
c. Perceptual Spaces. Since the components of the perceptual spaces represent three subjective entities to qualify a color, they can be considered independently. Furthermore, in order to compare two colors, the Euclidean distance is meaningless since the hue component is periodical. Therefore, we propose applying an independent coding scheme to the perceptual spaces. d. Independent Axis Spaces. Because the components of the independent axis spaces are determined by linear transformations, they can be considered primary spaces so they must be coded with a dependent coding scheme. e. Hybrid Color Spaces. The components of a hybrid color space are selected among all the components of several color spaces. Thus, this color space has neither psychovisual nor physical color significance and dependent color coding is not justified. That means that the hybrid color spaces must be coded by means of an independent color-coding scheme. D. Summary This first text section which dealt with the representation of the colors characterizing the pixels in a color space, underscoring two important points: • In order to use a color space, it is important to know the acquisition conditions (i.e., to specify the reference white, the lighting system, and the camera or scanner parameters, such as gain, offset, and gamma corrections). • A color calibration of the color image acquisition device is recommended so that the colors of the objects observed by an image sensor are represented correctly in the acquired color images. The second part of this section presented the most commonly used color spaces for color image analysis. Each color space is characterized by its physical, physiologic or psychological properties, but most of these color spaces are not initially developed for color image analysis applications. Therefore, their exploitation by a digital color image acquisition device require that some conditions be satisfied. The last part of this section shows that a specific coding scheme must be applied on these color spaces to correctly use them for color image analysis without modifying their intrinsic properties. The next section of this chapter shows how the color spaces can be exploited by color image segmentation algorithms, and the last section studies the impact of a choice of a color space on the results provided by these algorithms.
110
BUSIN , VANDENBROUCKE AND MACAIRE
III. C OLOR I MAGE S EGMENTATION A. Introduction Generally, we assume that the different colors that are present in an image correspond mainly to different properties of the surfaces of the observed objects. The segmentation procedures analyze the colors of pixels in order to distinguish the different objects that constitute the scene observed by a color sensor or camera. It is a process of partitioning an image into disjoint regions (i.e., into subsets of connected pixels that share similar color properties). The segmentation of the image denoted I regroups connected pixels with similar colors into NR regions Ri , i = 1, . . . , NR (Zucker, 1976). The pixels of each region must respect homogeneity and connectedness conditions. The homogeneity of a region Ri is defined by a uniformity-based predicate, denoted Pred(Ri ), which is true if the colors of Ri are homogeneous and false on the opposite. The regions must respect the following conditions: ! • I = i=1,...,NR Ri : each pixel must be assigned to one single region and the set of all the regions must correspond to the image. • Ri contains only connected pixels ∀i = 1, . . . , NR : a region is defined as a subset of connected pixels. • Pred(Ri ) is true ∀i = 1, . . . , NR : each region must respect the uniformitybased predicate. • Pred(Ri ∪ Rj ) = false ∀i = j , Ri and Rj being adjacent in I: two adjacent regions do not respect the predicate. The result of the segmentation is an image in which each pixel is associated with a label corresponding to a region. Segmentation schemes can be divided into two primary approaches with respect to the used predicate (Cheng et al., 2001). The first approach assumes that adjacent regions representing different objects present local discontinuities of colors at their boundaries. (Section III.B describes the edge detection methods deduced from this assumption.) The second approach assumes that a region is a subset of connected pixels that share similar color properties. The methods associated with this assumption are called region construction methods and look for subsets of connected pixels whose colors are homogeneous. These techniques can be categorized into two main classes—whether the distribution of the pixel colors is analyzed in the image plane or in the color space. The spatial analysis described in Section III.C is based on a region growing or merging process.
COLOR SPACES AND IMAGE SEGMENTATION
111
The analysis in the color space takes advantage of the characterization of each pixel P of an image I by its color point I(P ) in this space. Since the pixels are represented by points in a 3D color space, this approach assumes that homogeneous regions in the image plane give rise to clusters of color points in the color space. Each cluster corresponds to a class of pixels that share similar color properties. These clusters generally are identified by means of an analysis of the color histogram or a cluster analysis procedure and are mapped back to the original image plane to produce the segmentation (see Section III.D). In this chapter, we do not consider the physically based segmentation techniques that use the explicit assumptions about the physics that create the images. Klinker, Shafer and Kanade (1990) use a model of the formation of the color, called the dichromatic reflection model order to segment a color image. This model contains several rigid assumptions, such as the illumination conditions and the type of observed materials. For most real scenes, these assumptions cannot always be justified. Therefore, the proposed model can be used only for a restricted number of images, such as the scenes observed within a controlled environment. For illustration purposes, we apply segmentation schemes to the synthetic image of Figure 16a, which contains six different regions with different colors and shapes (a brown background, a small yellow square, a orange large square, a purple patch, and two concentric green disks). This image, with color coded in the (R, G, B) color space, is corrupted by an uncorrelated Gaussian noise with a standard deviation σ = 5, which is independently added to each of the three color components. To show the influence of the chosen color space on the segmentation results, the colors of the pixels are represented by some of the most widely known color spaces: the (R, G, B) color space (Figure 16a), the (Y , U , V ) color space defined by Section II.B.2.b (Figure 16b), and the triangular (I, S, T ) color space defined by Section II.B.3.b (Figure 16c). B. Edge Detection 1. Overview Edge detection consists of detecting local discontinuities of colors. The result of the edge pixel detection procedure is a binary image composed of edge and nonedge pixels. Cumani proposed detecting the zero-crossings of second-order derivative filters (Cumani, 1991). However, this approach is very sensitive to the presence of noise in the images, why is the reason several authors apply first-order derivative filters (gradient) to detect edges. Locations of gradient maxima that are higher than a threshold generate edge pixels.
112
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 16. Synthetic image whose color is coded in different color spaces. (a) Color synthetic (R, G, B) image. (b) Image of (a) coded in the (Y , U , V ) color space. (c) Image of (a) coded in the (I, S, T ) color space. (See Color Insert.)
The color edge detection methods based on gradient analysis can be divided into three families according to the analyses of the color component images denoted Ik , k = R, G, B (i.e., the monochromatic images where pixels are characterized by the levels of one single color component): 1. The methods that fuse edge binary images obtained by the analyses of different component images (Rosenfeld and Kak, 1981) (Figure 17a). 2. The methods that independently analyze each color component image in order to process three marginal gradient vectors. These three vectors are combined to provide the norm of a gradient (Lambert and Carron, 1999) (Figure 17b). 3. The methods that process a color gradient vector from three gradient vectors computed in each of the three color component images. The edge pixels are detected by an analysis of a color vector gradient (Di Zenzo, 1986; Lee and Cok, 1991) (Figure 17c). Since these methods provide the
COLOR SPACES AND IMAGE SEGMENTATION
113
(a)
(b)
(c) F IGURE 17. Different approaches for color edge detection. (a) Analysis of edge binary images. (b) Analysis of the norm of a color gradient. (c) Analysis of a color gradient vector. (See Color Insert.)
114
BUSIN , VANDENBROUCKE AND MACAIRE
best results in terms of edge detection (Cheng et al., 2001), we describe this method, which is the most widely used method. 2. Edge Detection by the Analysis of a Color Gradient Vector In order to detect color edge pixels, the color gradient magnitude and direction can be determined by Di Zenzo’s algorithm, which computes the first-order differential operators (Di Zenzo, 1986). The color gradient detection consists in determining, at each pixel, the spatial direction denoted θmax along which the local variations of the color components are the highest. The absolute value of this maximum variation corresponds to the gradient module, which is evaluated from the horizontal and vertical first derivatives of each color component. Let us denote IR (x, y), IG (x, y), and IB (x, y) as the three (R, G, B) color components of a pixel P (x, y) with spatial coordinates (x, y) in the color image I and θ the gradient direction. Let F be a variational function expressed as F (θ) = p cos2 θ + q sin2 θ + 2t sin θ cos θ,
(50)
where p is the squared module of the horizontal first-order partial derivatives, q corresponds to the squared module of the vertical first-order partial derivatives, and t is the mixed-squared module of the horizontal and vertical first-order partial derivatives: R G B ∂I (x, y) 2 ∂I (x, y) 2 ∂I (x, y) 2 p= + + , (51) ∂x ∂x ∂x R G B ∂I (x, y) 2 ∂I (x, y) 2 ∂I (x, y) 2 q= + + , (52) ∂y ∂y ∂y and ∂IR (x, y) ∂IR (x, y) ∂IG (x, y) ∂IG (x, y) ∂IB (x, y) ∂IB (x, y) + + . ∂x ∂y ∂x ∂y ∂x ∂y (53) The direction of the color gradient that maximizes the function F (θ) is given by: t=
2t 1 arctan , (54) 2 p−q √ and the gradient magnitude is therefore equal to F (θmax ). This maximum variation is evaluated from the first horizontal and vertical derivatives of each color component [see Eq. (50)]. For this purpose, Deriche proposes to apply a recursive filter as a differential operator (Deriche, 1990). θmax =
COLOR SPACES AND IMAGE SEGMENTATION
115
This optimal filter is well suited for edge detection in noisy images (Stoclin, Duvieubourg and Cabestaing, 1997). The behavior of this filter is governed by a parameter α. It is adjusted according to the expected filter performance, which is evaluated in terms of detection and localization. In order to extract well-connected edge pixels, detection of all the spatial color variations is more important than the accuracy of their localizations. The parameter α is then adjusted by the analyst so that the trade-off between detection and localization favors detection. To extract thin edges from the gradient image obtained by this process, the edge pixels whose magnitude are local maxima along the gradient direction are retained. Then, an hysteresis thresholding scheme, with a low threshold Thl and a high threshold Thh , provides a binary edge image. These parameters are adjusted by the analyst so that a maximum of pertinent edge pixels are detected. To illustrate the behavior of Di Zenzo’s method, we propose its applications to the synthetic images in Figure 16. In the images of Figure 18, the detected edge pixels are represented by a white overlay. Different tests show that this approach is not very sensitive to the adjustment of the thresholds used by the hysteresis scheme on these images. The three images in Figure 18 show that the edge detection using the same parameter values provides different results with respect to the color space used. Furthermore, one limitation of such an approach is the connectivity between the detected edge pixels (Ultré, Macaire and Postaire, 1996). Indeed, the region extraction is performed by a chaining scheme of edge pixels. This scheme assumes that the detected edge pixels that represent the boundary of a region are well connected (Rosenfeld and Kak, 1981). A postprocessing edge closing step has to be achieved when the edge pixels are not connected, which is why many authors prefer approaches based on region construction in order to segment color images. C. Region Construction Based on a Spatial Analysis These methods, which analyze the image plane to construct the regions in the image, can be divided into two approaches: the region growing method and the region merging method. 1. Segmentation by Region Growing A region growing method consists of sequentially merging neighboring pixels of similar colors starting from initial seeds (Trémeau and Borel, 1997). Regions are expanded as far as possible with respective aggregating conditions.
116
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 18. Edge pixels of the images in Figure 16 detected by Di Zenzo’s method. The parameters α, Thh , and Thl are set to 1.0, 10, and 5, respectively. (a) (R, G, B) color space. (b) (Y , U , V ) color space. (c) Triangular (I, S, T ) color space. (See Color Insert.)
The region growing method scans the image in some predetermined manner such as left-right top-bottom. The first analyzed pixel constitutes an initial region, named seed, and a next-neighboring pixel is examined. The color of this pixel is compared with the color of the already (but not necessarily) completed neighboring regions. If its color and the color of a neighboring region in progress are close enough (i.e., the Euclidean distance separating the colors is lower than a threshold), then the pixel is added to this region and the color of this region is updated. If there are several regions in progress whose colors are close enough, then the pixel is added to the region in progress whose color is the closest. However, when the colors of the two adjacent regions in progress are close enough, the two regions are merged and the pixel is added to the merged region. When the color of the examined pixel and that of any neighboring region are not close enough, a new region in progress is created (Moghaddamzadeh and Bourbakis, 1997).
COLOR SPACES AND IMAGE SEGMENTATION
117
Trémeau and Borel proposed three criteria to compare the color of a pixel with that of the region in progress (Trémeau and Borel, 1997). Let us denote Ri , the subset of pixels that constitutes the region in progress, P the analyzed pixel, N(P ) the set of neighboring pixels of P , and I(P ) the color of the pixel P . We denote with the same manner the color of a pixel P with (x, y) spatial coordinates by I(P ) and I(x, y). The pixel P merges with one of its neighboring pixels Q that also belongs to Ri : (Q ∈ N(P ) and Q ∈ Ri ), if the three following aggregating conditions are respected: • I(P ) − I(Q) < T1 • I(P ) − μN (P ) < T2 • I(P ) − μRi < T3 , where the thresholds Tj are adjusted by the analyst, is the Euclidean distance, μN (P ) represents the mean color of the pixels Q that belong to N(P ), and Ri , and μRi represents the mean color of the region Ri in progress. This scheme grows the region in progress until the maximum of the differences between one color component of the region and that of the examined pixel is higher than 10 (Figure 19). The first pixel to be analyzed is located in the left-top side of the image, and the image is scanned in the leftright top-bottom order. In the images in Figure 19, the constructed regions are labeled with false colors. To show the influence of the color space on the quality of the region construction, the colors of the pixels are coded in the (R, G, B) color space (see Figure 19a), in the (Y , U , V ) color space (see Figure 19b) and in the triangular (I, S, T ) color space (see Figure 19c). These three images show that the growing scheme using the same parameter values provides different results with respect to the used color space. This scheme suffers from the difficulty of adjusting relevant thresholds for all the color spaces. Furthermore, one of the main drawbacks of these neighborhoodbased segmentation methods is their sequential nature: the resulting regions depend on the order in which pixels are merged and on the selection of the initial seeds. 2. Segmentation by Region Merging In order to avoid a posteriori the oversegmentation problem, Trémeau and Colantoni proposed that the segmentation is achieved with two successive steps: a low-level step provides an oversegmented image by means of a region growing scheme (see Section III.C.1) and a high-level step merges adjacent regions with similar colors (Trémeau and Colantoni, 2000). He proposes that the high-level scheme is applied to a region adjacency graph that models a segmented image. A node of this graph represents a region, and an edge between two nodes corresponds to a pair of adjacent regions in the image.
118
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 19. Segmentation of the images in Figure 16 by region growing. The region grows until the max of the differences between one color component of the region and that of the examined pixel is higher than 10. (a) (R, G, B) color space. (b) (Y , U , V ) color space. (c) Triangular (I, S, T ) color space. (See Color Insert.)
Figure 20b shows the adjacency graph obtained from the presegmented image of Figure 20a. The iterative analysis of the graph consists of merging two nodes relied by an edge when they respect a fusion criterion at each iteration step. More precisely, this procedure merges adjacent regions whose colors are close enough (Lozano, Colantoni and Laget, 1996). In addition to the color homogeneity, Schettini proposes another criterion that considers the perimeters of two adjacent regions that are candidate for merging (Schettini, 1993). These criteria are adapted in case of uniform color regions but not in case of textured regions. This led Panjwani and Healey to model the regions by color Markov fields, which are designed for characterizing color textures (Panjwani and Healey, 1995).
COLOR SPACES AND IMAGE SEGMENTATION
(a)
119
(b)
F IGURE 20. Adjacency graph of regions. (a) Presegmented image. (b) Schematic adjacency graph of the image of figure (a). (See Color Insert.)
These methods favor the spatial interactions between pixels and analyze the colors of the pixels only with respect to grow or merge regions. Since these methods require an a priori knowledge of the images in order to adjust the used parameters, many authors prefer to achieve a global analysis of the color distribution in the color space. D. Region Construction Based on a Color Space Analysis 1. Introduction Because the color of each pixel can be represented in a color space, it is also possible to analyze the distribution of pixel colors rather than examining the image plane. In the (R, G, B) color space, a color point is defined by the color component levels of the corresponding pixel, namely, red (R), green (G), and blue (B). It is generally assumed that homogeneous regions in the image plane create clusters of color points in the color space, each cluster corresponding to a class of pixels that share similar color properties. Let us consider the synthetic image of Figure 21a which is composed of 6 regions, denoted Ri , i = 1, . . . , 6, and is identical to that of Figure 16a. The color points representing the pixels in the (R, G, B) color space are shown in Figure 21b. To estimate the separability of the clusters, let us also examine Figures 21c–e, which show the clusters of color points of the image of Figure 21a projected onto the different chromatic planes (R, G), (G, B), and (R, B), respectively. The regions R1 , R2 , R3 , and R4 give rise to wellseparated clusters of color points, whereas those of the regions R5 and R6 form two overlapping clusters. It is difficult to identify these two clusters by the analysis of the color point distribution in the (R, G, B) color space.
120
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c)
(d)
(e) F IGURE 21. Clusters of color points of the image of figure (a) in the (R, G, B) color space. (a) Color synthetic image. (b) Clusters of color points in the (R, G, B) color space. (c) Clusters of color points in the (R, G) chromatic plane. (d) Clusters of color points in the (G, B) chromatic plane. (e) Clusters of color points in the (R, B) chromatic plane. (See Color Insert.)
COLOR SPACES AND IMAGE SEGMENTATION
(a)
121
(b)
F IGURE 22. Clusters of color points of the image of Figure 21a in different color spaces. (a) Clusters of color points in the (Y , U , V ) color space. (b) Clusters of color points in the (I, S, T ) color space. (See Color Insert.)
The classes of pixels are constructed by means of a cluster identification scheme that is performed either by an analysis of the color histogram or by a cluster analysis procedure. When the classes are constructed, the pixels are assigned to one of them by means of a decision rule and are mapped back to the original image plane to produce the segmentation. The regions of the segmented image are composed of connected pixels, which are assigned to the same classes. When the distribution of color points is analyzed in the color space, the procedures generally lead to a noisy segmentation with small regions scattered through the image. Usually, a spatial-based postprocessing is performed to reconstruct the actual regions in the image (Cheng, Jiang and Wang, 2002; Nikolaev and Nikolayev, 2004). Several color spaces can be used to represent the colors of the pixels. Figure 22 shows the color points of the image in Figure 21a represented in the (Y , U , V ) and (I, S, T ) color spaces. By visually comparing this figure with Figure 21e, we see that the clusters representing the colors of the regions R5 and R6 form two overlapping clusters in the (R, G, B) color space—one single cluster in the (Y , U , V ) color space and well-separated clusters in the (I, S, T ) color space. These two figures show that the color distribution and the cluster separability depend on the choice of the color space. To illustrate this problem with a real example (soccer image segmentation), let us examine Figure 23a extracted from Vandenbroucke, Macaire and Postaire (2000b) where each of the images contains one soccer player. The players are regrouped into four classes ωj , j = 1, . . . , 4. Figure 23b shows
122
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c)
F IGURE 23. Clusters corresponding to different classes projected onto different color spaces. (a) Classes of pixels to be considered. (b) Color points in the (R, G, B) color space. (c) Color points in Carron’s (Y, Ch1 , Ch2 ) color space. (d) Color points in the hybrid (x, Ch2 , I 3) color space. (See Color Insert.)
COLOR SPACES AND IMAGE SEGMENTATION
123
(d) F IGURE 23.
(continued)
the colors of pixels of each class projected onto the (R, G, B) color space. Clusters corresponding to the classes ω2 , ω3 , and ω4 are not well separated in the (R, G, B) color space while they are more compact in the (Y, Ch1 , Ch2 ) Carron’s color space (defined in Section II.B.2.d; see Figure 23c) and well separated in the (x, Ch2 , I 3) hybrid color space (defined in Section II.B.2.d; see Figure 23d). This example shows that the selection of the color space is crucial for the construction of pixel classes. The key of the segmentation problem based on color-spaced analysis consists of the construction of the pixel classes. The segmentation of color image based on pixel classification can be divided into four groups: (1) the analysis of histograms (see Section III.D.2), (2) the analysis of the 3D color histogram (see Section III.D.3), (3) the clustering approaches, which take into account only color distribution (see Section III.D.4), and (4) the spatial-color classification methods, which simultaneously consider the spatial interactions between the pixels and the color distribution (see Section III.D.5). 2. Analysis of One-Dimensional Histograms Many statistical procedures seek to accomplish class construction by detecting modes of the color space. The modes are characterized by high local concentration of color points in the color space separated by valleys with a low local concentration of color points. For this purpose, they estimate the probability density function (pdf) underlying the distribution of the color points (Devijver and Kittler, 1982). Assuming that each mode of the color space corresponds to the definition domain of a pixel class, the color image segmentation can be viewed as a pdf mode detection problem. The most widely used tools for the approximation of the pdf are the 1D histograms denoted H k [I], of the color component images Ik , k = R, G, B,
124
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 24. One-dimensional histograms of the image in Figure 16a. (a) One-dimensional histogram of the color component image IR . (b) One-dimensional histogram of the color component image IG . (c) One-dimensional histogram of the color component image IB . (See Color Insert.)
COLOR SPACES AND IMAGE SEGMENTATION
125
F IGURE 25. Mode detection in the chromatic plane (G, B) by the analysis of the 1D histograms H G [I] and H B [I] of the image in Figure 21a. (See Color Insert.)
when the (R, G, B) color space is used. The bin H k [I](n) contains the number of pixels P whose color component level Ik (P ) is equal to n. By examining Figure 24 (which contains three 1D histograms H k [I], k = R, G, B, in the image of Figure 21a), we see that the color component distributions of the different regions create peaks of the 1D histograms. Furthermore, the histograms show that the pixel classes may not be equiprobable. The peaks of the corresponding 1D histogram are determined by means of searching thresholds that delimit them (Schettini, 1993). When the thresholds are determined, the color space is partitioned into parallelepipedic boxes by means of the Cartesian product of the intervals delimited by these thresholds. An analysis of the populations of pixels whose color points fall into these boxes allows us to identify the modes of the color space (Figure 25).
126
BUSIN , VANDENBROUCKE AND MACAIRE
A peak of a 1D histogram may contain the color component levels of pixels that constitute different regions of the image (see Figure 24a). Pixels of each region must be regrouped into a specific class of pixels defined by a mode of the color space. Therefore, the methods of mode detection analyzes the case when one peak of the 1D histogram corresponds to several modes of the color space (Ohlander, Price and Reddy, 1978). For this purpose, an iterative analysis of the 1D histograms is achieved (Busin et al., 2005). At each iteration step, this procedure constructs one class of pixels. The procedure looks for the most significant mode to construct one class, according to different criteria, such as the population size. The pixels assigned to this class and which are connected in the image constitute one of the reconstructed regions in the segmented image. The pixels assigned to the so-constructed class are extracted from the color image so that they are not taken into account at the next iteration steps of the procedure. The iterative procedure stops when only a few pixels that are not assigned to any constructed classes remain. When the iterative procedure stops, the pixels have not been assigned to any class, that can be assigned to one of the constructed classes by means of a specific decision rule. Figure 26 illustrates the iterative segmentation of the Hand image in Figure 26a (Busin et al., 2004). The image in Figure 26b shows pixels assigned at the first class built by an analysis of the 1D color histograms. White-labeled pixels in the image of Figure 26c are not assigned to this class and thus are considered by the second iteration step in order to build the second class. Images of Figures 26d, f, h, and j show the successively built classes, whereas images of Figures 26e, g, i, and k contain the pixels to be analyzed by the successive iteration steps. The final segmented image of Figure 26l with pixels labeled to the five built classes shows that this scheme provides satisfying results. Actually, the performance of the mode extraction scheme is strongly affected by the choice of color space, implicitly indicating that the performance depends on the color distribution of the analyzed image (Lim and Lee, 1990). That finding led Tominaga to perform the Karhunen–Loeve transformation of the (R, G, B) color components (i.e., the PCA) in order to construct the classes (Tominaga, 1992). This procedure transforms the (R, G, B) color space into the (X1 , X2 , X3 ) principal component space. The histogram H [I]X1 of the most discriminating component X1 is first analyzed. If this histogram is a multimodal one, the most significant peak is detected. Then a new Karhunen–Loeve transformation is achieved with the pixels that do not belong to the built class. If the histogram of the first component contains only one single peak, then Tominaga proposes analysis of the second component X2 . This iterative scheme stops when the histograms
127
COLOR SPACES AND IMAGE SEGMENTATION
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
F IGURE 26. Iterative segmentation of the Hand image. (a) Hand image. (b) Pixels assigned to the class built at the first iteration. (c) Pixels to be analyzed after the first iteration. (d) Pixels assigned to the class built at the second iteration. (e) Pixels to be analyzed after the second iteration. (f) Pixels assigned to the class built at the third iteration. (g) Pixels to be analyzed after the third iteration. (h) Pixels assigned to the class built at the fourth iteration. (i) Pixels to be analyzed after the fourth iteration. (j) Pixels assigned to the class built at the fifth iteration. (k) Pixels to be analyzed after the fifth iteration. (l) Segmented image. (See Color Insert.)
128
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 27. Color component images resulting from the Karhunen–Loeve transformation of the image in Figure 21a. (a) Color component image IX1 . (b) Color component image IX2 . (c) Color component image IX3 .
are empty or when the histogram of the third component X3 contains one single peak. Figure 27 shows the three color component images IX1 , IX2 , and IX3 resulting from the Karhunen–Loeve transformation of the image in Figure 21a. The IX1 color component image contains the greatest amount of information. Tominaga proposes to first extract the most significant peak from the histogram H X1 [I] (Figure 28a). Pixels of the image IX1 whose X1 levels fall into the interval delimiting this peak are shown with a false color in the image of Figure 29a, where unassigned pixels are labeled as black. By examining this image, we see that the pixels of the background constitute the first built class. The scheme is iterated to yield the final satisfying segmentation shown by Figure 29b.
COLOR SPACES AND IMAGE SEGMENTATION
(a)
129
(b)
(c) F IGURE 28. One-dimensional histograms of the IX1 , IX2 , and IX3 color component images in Figure 27. (a) One-dimensional histogram of the image in Figure 27a. (b) One-dimensional histogram of the image in Figure 27b. (c) One-dimensional histogram of the image in Figure 27c. (See Color Insert.)
3. Analysis of the Three-Dimensional Color Histogram Figure 24 shows that a peak of each 1D histogram of a color image can reflect several classes. Even when the mode detection is separately performed on the three 1D histograms, the appropriate number of classes may not be detectable because of this superposition. Moreover, since the thresholds obtained from the analysis of the 1D histograms are used to partition the color space into parallelepipedic boxes, the boundaries between classes cannot be adequately determined, sometimes yielding false segmentation results. Because color is 3D information, several authors use the 3D color histogram for approximating the pdf. The color histogram is composed of bins whose coordinates are the three color components. Each bin H [I](C) indicates the number of pixels P whose color I(P ) is equal to the color C.
130
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
F IGURE 29. Image of Figure 21a segmented by Tominaga’s scheme. (a) Pixels assigned to the class built at the first iteration step. (b) Segmented image segmented after the last iteration step. (See Color Insert.)
Since the colors of pixels are usually quantified with 256 levels, a 3D histogram requires a considerable amount of computer memory. That explains why only a few authors have analyzed the 3D histogram for color image segmentation (Lambert and Macaire, 2000). A simple solution consists of color quantization by considering only the highest bits for each component in order to reduce the computational complexity. This solution yields poor results, because this quantization phase leads to a very crude segmentation. Another approach consists of mode detection by thresholding the bins. Nevertheless, the determination of the threshold is very delicate, which is why several mode detection methods use specific morphological transformations of the color histogram to enhance the modes (Park, Yun and Lee, 1998; Shaffarenko, Petrou and Kittler, 1998). Let us consider the image of Figure 30a, which is made of two concentric disks lying on a distinct background. This image is corrupted by a noncorrelated Gaussian noise, with a standard deviation equal to 9, that is independently added to the color components. We can visually distinguish three regions, each being characterized by a specific shade of green. Therefore, pixels of this image must be grouped into three classes. In order to simplify the illustration, the blue component has been set to 0 throughout the image. The color histogram of this image is plotted in Figure 30b over a restricted domain of the chromatic plane (R, G) where its values are non-zero. Since the distributions of those colors overlap, the three color modes are barely detected by an automatic processing of this histogram. Gillet et al. define a mode as a parallelepipedic box of the color space where the histogram function is locally concave while reaching values high
COLOR SPACES AND IMAGE SEGMENTATION
(a)
131
(b)
(c)
(d)
(e)
F IGURE 30. Mode detection by convexity analysis of the color histogram. (a) Image. (b) Histogram. (c) Modes of the histogram detected by the convexity test; their identification is illustrated by false colors. (d) Prototypes of the three built pixel classes. (e) Pixels assigned to the three classes. (See Color Insert.)
132
BUSIN , VANDENBROUCKE AND MACAIRE
enough to be relevant (Gillet et al., 2002). In order to assign a “concave” or “convex” label to each color C of the color space, a local convexity test is performed on the histogram. Let D(C, l) be a cubic box, centered at the color C = [C R , C G , C B ]T in the (R, G, B) color space, whose edges of length l (an odd integer) are parallel to the color component axes. The box D(C, l) therefore includes all the color points c = [cR , cG , cB ]T such that l−1 k k C k − l−1 2 c C + 2 , k = R, G, B. The mode detection is based on the property stating that, at each color point C where a trivariate function f is locally concave, the average value of f processed over a box D(C, l) is a decreasing function of l (Postaire and Vasseur, 1980). The histogram local convexity is therefore evaluated at each color point C by examining the variation of the average histogram, denoted μ, according to the size of the cubic box centered at C. First, we average the histogram over the box D(C, l1 ): " cD(C,l1 ) H (c) μ1 (C) = . (55) (l1 )3 We then compute the second average value μ2 using a slightly larger box D(C, l2 ): " cD(C,l2 ) H (c, l) μ2 (C) = , with l2 > l1 . (56) (l2 )3 Empirically, length l1 is set to 5, while length l2 is set to l1 + 2. In these conditions, if μ1 (C) > μ2 (C), the histogram is considered as locally concave. If this condition is satisfied and if H [I](C) is high enough (i.e., higher than a threshold adjusted by the analyst), we decide that the color point C belongs to a mode. This mode detection algorithm is applied to the 2D histogram of the synthetic image (see Figure 30b). Because small modes may be detected by convexity analysis, a binary morphological opening is applied to the modes to preserve only the significant ones—modes whose volume is larger than that of the structuring element that is used. Figure 30c shows the three detected modes in the chromatic plane (R, G). The detected modes then are identified using a connected component analysis procedure in the color space. Each mode identified in this manner is the definition domain of a pixel class. All the pixels whose colors are located in an identified mode define the prototypes of the corresponding class and are assigned to it. Figure 30d shows the prototypes of the three pixel classes labeled as false colors, constructed by the analysis of the histogram in Figure 30b. The black-marked pixels are the ones that are not yet assigned. This image shows that the prototypes correctly represent the background and
COLOR SPACES AND IMAGE SEGMENTATION
133
the two concentric shapes. This example illustrates that the convexity analysis of the histogram efficiently constructs the pixel classes even when the color distributions of the different regions strongly overlap. The next classification procedure step consists of assigning all the pixels that are not yet labeled by use of a specific decision rule in order to produce the final segmented image in Figure 30e (Vannoorenberghe and Flouzat, 2006). 4. Clustering-Based Segmentation The multithresholding methods based on histogram analysis require the computation of the histogram, which is both time and memory space consuming. For this reason, many authors have proposed a looking for clusters of color points in the color space. Among the clustering techniques based on the least sum of squares criterion, the c-means algorithm is one of the most widely used methods for clustering multidimensional data (Ismail and Kamel, 1989). This iterative approach requires the analyst to adjust the desired number Nω of pixel classes. Let Cj denote the gravity center of the cluster that is associated with the class ωj . This approach tends to minimize the global within-class dispersion, which is defined by SW =
Nω I(P ) − Cj .
(57)
j =1 P ∈ωj
At the initial iteration step, the locations of the Nω gravity centers are randomly fixed in the color space. At each iteration step denoted t of the process, the Nω gravity centers Cj (t) are updated via this construction scheme: • Each pixel P is assigned to the class ωj 0 whose gravity center Cj 0 (t) is the nearest one to the color I(P ) in the color space: Cj 0 (t) − I(P) = min Cj (t) − I(P) . j =1,...,Nω
• The gravity center Cj (t) of each class ωj is updated by taking into account the pixels assigned to ωj : I(P ). Cj (t) = P ∈ωj
• The variation denoted ε of the gravity centers between the preceding (t −1) and current (t) steps is determined by: Cj (t) − Cj (t − 1) . ε= j =1,...,Nω
134
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c) F IGURE 31. Clustering-based segmentation of the image of Figure 21a. (a) Image segmented thanks to the c-means scheme with a random initialization of the gravity centers. (b) Image segmented thanks to the c-means scheme with an interactive initialization of the gravity centers. (c) Image segmented thanks to the competitive learning scheme. (See Color Insert.)
• If ε is higher than a threshold, then the process achieves a new iteration step. Wherever the initialization of the gravity centers, the scheme converges to a local minimum of SW . In this way, the algorithm stops at a local optimum, which can be far away from the real global optimum, especially when large numbers of pixels and clusters are involved. Since the convergence and the classification results depend on the initialization step, different approaches are proposed as follows. • The initial locations of the gravity centers are randomly selected (see Figure 31a)
135
COLOR SPACES AND IMAGE SEGMENTATION
• During a supervised learning, the initial locations of the gravity centers are interactively selected so that they are uniformly scattered in the color space (see Figure 31b). The isodata method adds two rules to determine the number Nω of classes: if the within-class dispersion of a class is greater than a threshold, then the class is split into two different classes. If the distance between two gravity centers is too low, then two corresponding classes are merged into one class (Takahashi, Nakatani and Abe, 1995). In order to minimize the within-class dispersion SW , Uchiyama and Arbib constructed the Nω classes by means of an iterative process based on competitive learning (Uchiyama and Arbib, 1994). This process, which does not require an initialization step, converges to approximate the optimum solution. We apply the competitive learning scheme to the image in Figure 21a (see Figure 31c). By examining the segmented image, we notice that the regions R5 and R6 are merged into one class, because the color component distributions overlap. The regions R4 and R2 are represented by one class, because the regions with the lowest population (in our case R4 ) are not taken in account by the competitive learning scheme. Liew et al. apply the fuzzy c-means, a fuzzy classification scheme that considers that each pixel P belongs to each class ωj according to a membership degree denoted Uj (P ) ranging between 0 and 1 (Liew, Leung and Lau, 2000). This method constructs the classes by minimizing the m-fuzzy within-class dispersion J (m): J (m) =
Nω
Uj (P )
P
m Cj − I(P ) ,
(58)
j =1
where
Uj (P )
m
=
−1 Nω I(P ) − Ci 2/(m−1) i=1
I(P ) − Cj
.
(59)
The color components of the gravity center Cj associated with the class ωj are defined by: " (Uj (P ))m .Ik (P ) k , with k = R, G, B. (60) Cj = P" m P (Uj (P )) When the gravity centers of the classes are estimated, the membership degrees of each pixel to the classes are determined. The pixel P is assigned to the class for which the membership degree is the highest (Scheunders, 1997).
136
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
F IGURE 32. Relationship between the regions in the image and the clusters of color points in a color space. (a) Original synthetic image. (b) Clusters of color points projected onto the (R, G) chromatic plane. (See Color Insert.)
5. Spatial-Color Classification Let us consider the synthetic image of Figure 32a, which is composed of six regions, denoted Ri , i = 1, . . . , 6, with different sizes and shapes. The color distribution in this synthetic image is different from that of Figure 16a. For the sake of illustration simplicity, the blue level of the pixels is set to 0. The projections of the color points representing the regions onto the (R, G) chromatic plane are displayed in Figure 32b. The regions R1 and R2 give rise to well-separated clusters of color points, whereas those of regions R3 and R4 form two overlapping clusters. Note that as the color points of regions R5 and R6 constitute a single cluster, they cannot be discriminated by the analysis of the color point distribution in the (R, G) chromatic plane. This image shows that there is not always a one-to-one correspondence between the regions in the image and the clusters of color points in the color space. Classical classification schemes cannot identify clusters of color points in the color space that correspond to several different regions in the image. A class construction procedure that would consider simultaneously the color properties of pixels as well as their spatial arrangement in the image could be appealing to identify pixel classes corresponding to the actual regions (Balasubramanian, Allebach and Bouman, 1994; Orchard and Bouman, 1991). The JSEG algorithm proposed by Deng and Manjunath (2001) separates the segmentation procedure into two successive steps. In the first step, pixels are classified in the color space without considering their spatial distribution. Then the colors of the pixels are replaced by their corresponding color class
COLOR SPACES AND IMAGE SEGMENTATION
137
labels, thus forming a class-map of the image. At each pixel the denoted JSEG criterion, which depends on the dispersion of the labels associated with its neighboring pixels, is measured. In the second step, the application of this criterion in the class-map yields an image in which high and low values correspond to possible boundaries and interiors of regions, respectively. A region growing method (see Section III.C.1) using this criterion provides the final segmentation. Cheng, Jiang and Wang (2002) proposes a fuzzy homogeneity approach to take into account simultaneously the color and spatial properties of the pixels. Their segmentation scheme also is divided into two steps. In the first step, each of the three color component images Ik , k = R, G, B, is analyzed to aggregate its pixels into classes. For this purpose, the authors introduce the homogram that is defined for each color component level as the mean measure of homogeneity degrees described in Cheng et al. (1998) between pixels with this level and their neighbors. A fuzzy analysis detects the peaks of each homogram in order to identify the prominent classes of pixels for each color component image. The prominent classes built by the analyses of the three color component images are combined to form the classes of pixels of the color image. When some of these classes contain too few pixels, the procedure may lead to oversegmentation. This problem is partially addressed in the second step by merging neighboring classes. This interesting approach is very sensitive to the adjustment of the parameters required for determining the peaks of each homogram in the first step and for merging classes in the second step. Another solution consists of estimating the gradient of the probability density of color occurring jointly in spatial and color domains of the color image (Comaniciu and Meer, 2002). Each pixel is associated with the closest local mode of the density distribution determined by means of the nonparametric “mean-shift” estimation. The quality of the segmentation depends on the precise adjustment of two parameters that control the resolutions in the spatial and in the color domains. We propose to segment the image of Figure 32a by use of the EDISON software developed by Comaniciu and Meer (available at the web address: http://www.caip.rutgers.edu/riul/research/code.html). For this purpose, the pair of parameters hs and hr , which are the side lengths of the windows used to compute the gradient in the spatial and color domains, are set to the values suggested by the authors for segmenting most color images. The parameter M, which corresponds to the minimal population sizes of the classes, is set to 1 so that all the classes, even those with small population sizes, are selected. Figures 33a and b yield the best results obtained by the “mean-shift” analysis. Figure 33a shows that when the side length of the color domains is
138
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
F IGURE 33. Image in Figure 32a segmented by the “mean-shift” analysis. The labels of pixels are false colors. (a) hs = 8, hr = 4, M = 1. (b) hs = 8, hr = 8, M = 1. (See Color Insert.)
low (hr is set to 4), the procedure based on the “mean-shift” method succeeds in separating the two clusters of color points representing the two green concentric disks. However, it also splits the cluster of color points representing the background of the image, which is segmented into several different regions. When the side length of the color domains increases (hr is set to 8), the procedure fails to separate the two clusters of color points corresponding to the two green concentric disks (see Figure 33b). It is difficult, or even impossible, to adjust the parameters required by this procedure in order to identify the two classes of pixels representing the two concentric green disks without splitting the other clusters corresponding to the other regions of the image. However, the mean-shift method does not require adjustment of the desired number of pixel classes. When the number of pixel classes is given, a postprocessing step to the mean-shift method would lead to good segmentation results. Macaire, Vandenbroucke and Postaire (2006) assume that each region can be considered as a subset of strongly connected pixels with homogeneous colors. Hence, they propose selecting parallelepipedic boxes in the color space that define subsets (named color subsets) of strongly connected pixels in the image with as homogeneous colors as possible. In other words, pixels in the image whose color points fall into such boxes (named color domains) of the color space constitute color subsets, so that each color domain is associated with a color subset. The color and connectedness properties of the color subsets associated with the considered color domains are simultaneously taken into account by the pixel class construction scheme. To measure these properties, Fontaine, Macaire and Postaire (2000) introduced the concept of spatial-color compactness degree (SCD) of a color subset, which measures
COLOR SPACES AND IMAGE SEGMENTATION
(a)
139
(b)
F IGURE 34. Image in Figure 32a segmented by the SCD analysis. (a) Color domains of the image in Figure 32a automatically selected by the proposed procedure. (b) Segmented color image from the analysis of the image of Figure 32a. The boundaries of the reconstructed regions are displayed in black. (See Color Insert.)
the spatial arrangement of its pixels in the image plane and the dispersion of the color points representing its pixels in the color space. In order to select the color domains that define color subsets corresponding to the actual regions in the image, the pixel class construction procedure looks for maxima of the SCDS in the color space. The labeled image in Figure 34b shows how the regions of the image in Figure 32a are reconstructed from the identified class color domains displayed in Figure 34a according to specific rules (Macaire, Vandenbroucke and Postaire, 2006). In Figure 34b the boundaries between the reconstructed regions are overlaid in black. This image shows that the six regions are well reconstructed by the SCD analysis. The overlapping of the color point distributions of the regions R3 and R4 explains why a few pixels of the region R3 are misclassified. Note that even with the strong overlapping of the color point distributions of the two regions R5 and R6 , most pixels of the regions corresponding to the two green concentric disks are correctly classified. This result shows that this method is able to handle unequiprobable and overlapping classes of pixels. E. Summary In this section, we have attempted to present classical color segmentation methods. Each method is adapted to a type of color image, so a method that provides a satisfying segmentation of all images does not exist. Several
140
BUSIN , VANDENBROUCKE AND MACAIRE
approaches combine the analysis of the spatial interaction between pixels and of the color distribution to improve segmentation results. Moreover, we have shown with different examples that the color distribution strongly depends on the selection of the color space. That explains why the results provided by segmentation procedures are affected by the chosen color space. The next text section describes studies about the relationships between color spaces and image segmentation.
IV. R ELATIONSHIPS BETWEEN S EGMENTATION S PACES
AND
C OLOR
A. Introduction The first part of this chapter has shown that the information of color can be coded by several color spaces that respect their own specific properties. The second part of the chapter has shown that color image segmentation could be divided into two main approaches according to whether the pixels are analyzed in the image plane or in a color space. If the pixels are analyzed in the image plane, then two dual types of methods are available. On the one hand, the segmentation can be produced by an edge detection scheme. On the other hand, the segmentation can be produced via region construction analysis. Several authors have studied the influence of color spaces for color image segmentation with the objective to determine if there exists a color space that improves the quality reached by color image segmentation methods. To determine whether this “best” color space exists, they estimated the results produced by color image segmentation methods via evaluation methods. First, Ohlander, Price and Reddy (1978) and Ohta, Kanade and Sakai (1980) visually estimated the quality of color segmentation. Because it is very difficult to evaluate the segmentation of a natural scene, several authors adopted the qualitative evaluation method as the most reliable one. Consequently, this part of text is divided as follow. Section IV.B presents the primary evaluation methods used for color image segmentation by edge detection, and Section IV.C presents the main evaluation methods used for color image segmentation by region construction. For both sections, quantitative evaluation methods with ground truth, which correspond to unsupervised evaluations, and quantitative evaluation methods without ground truth, which correspond to supervised evaluations, are presented in the subsections. To illustrate the relationships between the color spaces and segmentation, Section IV.D details two pixel classification methods that determine the most discriminating color space (hybrid color space or classic color spaces).
COLOR SPACES AND IMAGE SEGMENTATION
141
B. Edge Detection In order to evaluate the result provided by an edge detection segmentation, the most common method used in the literature consists of comparing the result of the segmentation I with the ground truth Iref (also called the reference segmentation or gold standard). These comparison methods are presented in Section IV.B.1. Since very few methods based on an evaluation method without ground truth are available and used in the literature for color edge detection segmentation methods, none of them are presented here. Finally, a table summarizing the selected color spaces by means of these criteria that provide the best segmentation results is presented in Section IV.B.2. 1. Quantitative Evaluation Methods with Ground Truth The most common way to evaluate the segmented image computed via an edge detection segmentation algorithm consists of evaluating the extracted edges of the segmented image I with the edges of the ground truth image Iref . For this type of evaluation, two primary methods are available: a probabilistic error approach and a spatial approach. For both approaches, the discrepancy between a binary image computed by means of an edge detection segmentation algorithm consists of evaluating the extracted edges of the segmented image with the edges of the ground truth (Román-Roldán et al., 2001). To detail these both approaches, we define the following: • Mistake—The discrepancy between the detected edge and the real edge is due to individual discrepancies arising from pixel to pixel, which are referred to as mistakes. These may be of two kinds: 1. Bit or overdetected edge pixel—A mistake due to excess, when a pixel is erroneously defined as an edge pixel. 2. Hole or subdetected edge pixel—A mistake due to failure of the method to identify an edge pixel as such. • • • •
Nb —The number of bits in the segmented image. Nh —The number of holes in the segmented image. Ne —The number of edge pixels in the ground truth. N—The number of pixels in the segmented image.
a. Probabilistic Error Approach. The first discrepancy measure established by Peli and Malah (1982) is called error probability and is denoted Pe (I, Iref ) hereafter. This rate is based on a statistical comparison between the number Nb of bits and the number Ne of edge pixels in the ground truth image Iref
142
BUSIN , VANDENBROUCKE AND MACAIRE
and can be computed as: Pe (I, Iref ) =
Nb . Ne
(61)
Another probabilistic discrepancy measure introduced by Lee, Chung and Park (1990)—denoted PE (I, Iref ) hereafter—is based on a two-class problem (denoted ω1 and ω2 ). In the edge detection segmentation methods, the class of pixels ω1 corresponds to those that are assigned as edge pixels, whereas the class of pixel ω2 corresponds to those that are assigned as nonedge pixels. This discrepancy measure is better than the measure proposed by Peli and Malah (1982) because both types of errors (holes and bits) are taken into account. The PE (I, Iref ) probabilistic discrepancy measure based on the classical classification error measure can be expressed as: PE (I, Iref ) = P (ω1 ) × P (ω2 |ω1 ) + P (ω2 ) × P (ω1 |ω2 ) =
Nh + Nb , N (62)
where: Ne N N−Ne P (ω2 ) = N h P (ω2 |ω1 ) = N Ne Nb P (ω1 |ω2 ) = N−N . e
• P (ω1 ) = • • •
b. Spatial Approach. Because the Peli and Lee discrepancy measures do not consider the spatial distance between the edges detected in the segmented image and the edges from the ground truth, most authors prefer Pratt’s discrepancy measure based on the mean-square distance called figure of merit (FOM) (Pratt, 1978) and defined as: FOM(I, Iref ) =
1 M
Ne −N h +Nb i=1
1 , 1 + αd(i)2
(63)
where α is a scaling parameter, M = max(Ne , Ne − Nh + Nb ), and d(i) is the distance from the ith edge pixel in the segmented image to its exact location in the ground truth. The measure is a normalized one so that FOM = 1 indicates a perfect segmentation. Pratt’s measure is the most frequently used discrepancy measure, even though it has been modified by Strastersa and Gerbrands (1991) to improve its accuracy with respect to bits and holes. Another interesting discrepancy measure proposed by Román-Roldán et al. (2001) requires a training step to fix the parameters used.
COLOR SPACES AND IMAGE SEGMENTATION
143
TABLE 6 M ETHODS U SED TO E VALUATE C OLOR I MAGE S EGMENTATION BY E DGE D ETECTION Reference
Evaluation method
Candidate color spaces
Selected color spaces
Rakotomalala et al. (1998)
Pratt’s discrepancy measure and probability error
(R, G, B), (X, Y, Z), (I 1, I 2, I 3), (A, C1 , C2 ), (I, S, T ), (L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ )
(L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ )
Wesolkowski, Jernigan and Dony (2000)
Pratt’s discrepancy measure
(R, G, B), (X, Y, Z), (L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ ), (r, g, b), (l1 , l2 , l3 ), (h1 , h2 , h3 )
(h1 , h2 , h3 )
2. Evaluation Methods for Color Image Segmentation by Edge Detection Several authors apply these methods to evaluate the accuracy of the color image segmentation by edge detection for several color spaces. For the sake of simplicity, the results of these works are presented in Table 6. Table 6 indicates that no color space is actually adapted to edge detection. Moreover, the conclusions of the authors cited are contradictory— Rakotomalala et al. (1998) select (L∗ , a ∗ , b∗ ) or (L∗ , u∗ , v ∗ ) spaces, whereas Wesolkowski, Jernigan and Dony (2000) do not select this as a color space in which the edge detection is efficient. The next text section details the primary evaluation methods for region construction segmentation methods to determine whether an efficient color space exists for such methods. C. Region Construction Evaluation methods for region construction segmentation methods can be divided into two main approaches according to whether a ground truth is available or not. The primary evaluation methods based on discrepancy measure with a ground truth are presented in Section IV.C.1. Without ground truth, it is impossible to compare the segmented images. Therefore, the unsupervised evaluation methods that attempt to mimic human sensitivity are presented in Section IV.C.2. Finally, the selected color spaces by means of these criteria that provide the best segmentation results are summarized in a table in Section IV.C.3. 1. Quantitative Evaluation Methods with Ground Truth Supervised evaluation methods of color image segmentation by region construction can be divided into two families. The first family is based on the
144
BUSIN , VANDENBROUCKE AND MACAIRE
discrepancy area of the regions between the segmented image I and the ground truth Iref . For this method, correct classification rate is the most often used (Sonka, Hlavac and Boyle, 1994). The second family consists of evaluating the color discrepancy between the pixels of the segmented image I and the ground truth Iref . For this method, two approaches are available. The first consists of analyzing color discrepancies between the segmented image I and the ground truth Iref . For this evaluation methods, the color of each pixel in both the segmented image I and the ground truth Iref must be the mean color of the pixels that belong to the same region. The most current evaluation methods for this approach are the mean square error (MSE) and the peak SNR. The second approach consists of evaluating the probability error of mislabeling pixels between the segmented image I and the ground truth Iref . Receiver operator characteristic (ROC) curves and probabilistic error approach often are used for a such analysis. a. Correct Classification Rate. Generally, a similarity measure is used based on the correct classification rate between a segmented image I and a ground truth Iref (Sonka, Hlavac and Boyle, 1994). According to the notation by Chabrier et al. (2006), the first step of the method is computing the superposition table T (I, Iref ) as follows: ref ref T (I, Iref ) = card Ri ∩ Rj , i = 1, . . . , NR , j = 1, . . . , NR , (64) ref
where card{Ri ∩ Rj } is the number of pixels that belong to both the region ref Ri in the segmented image I and to the region Rj in the ground truth ref Iref while NR and NR are the number of regions in the image I and Iref , respectively. From this superposition table, only the C couples that maximize ref card(Ri ∩ Rj ) are selected to compute correct classification rate. Thus, the dissimilarity measure that is a normalized one is computed, so that a measure close to 0 means that the images I and Iref strongly share similar regions, whereas a measure close to 1 means that the images I and Iref are very dissimilar. This measure, denoted CCR(I, Iref ), is computed as follows: CCR(I, Iref ) =
card(I) −
"
C
ref
card(Ri ∩ Rj )
card(I)
.
(65)
b. Mean Square Error. The mean square error, denoted as MSE(I, Iref ), expresses the delineation accuracy and region homogeneity of the final partitioning between the segmented image I and the ground truth Iref . The lower the values of the MSE(I, Iref ), the better the segmentation results.
COLOR SPACES AND IMAGE SEGMENTATION
The MSE(I, Iref ) can be computed as Iref (P ) − I(P ) , MSE(I, Iref ) =
145
(66)
P ∈I
where Iref (P ) is the color mean of the region to which the pixel P belong. c. Peak SNR. Peak signal to noise ratio, denoted as PSNR(I, Iref ), is close to the MSE(I, Iref ) evaluation method and expresses the same concept. The main difference lies in the interpretation of the result provided by the PSNR(I, Iref ). Indeed, the higher the PSNR(I, Iref ), the better the segmentation result. The PSNR of an X × Y image (measured in decibels) can be computed as 2552 × X × Y × card(k) . (67) PSNR(I, Iref ) = 10 × lg " P ∈I Iref (P ) − I(P ) d. Probabilistic Error Approach. The discrepancy measure introduced in Section IV.B.1 has been extended for the ωn classes problem. In the case of an ωn class problem, each ωn class corresponds to a specific region, usually determined by a pixel classification, as in Lim and Lee (1990) and Park, Yun and Lee (1998), for example. The PE (I, Iref ) probabilistic discrepancy measure can be calculated for an ωn class problem as follows: PE (I, Iref ) =
Rn Rn
P (Ri |Rj ) × P (Rj ).
(68)
j =1 i=1 i =j
e. ROC Curves. ROC curves are commonly used to present results for binary decision problems. By comparison of the labeled pixels provided by a classification method, four cases are possible as shown in the confusion matrix shown in Table 7. The ROC curves can be represented by plotting in a 2D space where the xaxis corresponds to the fraction of false positives (false-positive rate, or FPR) and the y-axis corresponds to the fraction of true positives (true positive rate, TABLE 7 C ONFUSION M ATRIX
Pixels assigned to the class ωi (i = 1, 2) in I
ω1
Pixels assigned to the class ωi (i = 1, 2) in Iref ω1 ω2 True positive (TP) False positive (FP)
ω2
False negative(FN)
True negative (TN)
146
BUSIN , VANDENBROUCKE AND MACAIRE
TPR) (Liu and Shriberg, 2007). The TPR and FPR are expressed as: TP , (69) TPR = TP + FN FP FPR = . (70) FP + TN Analog curves of ROC curves, denoted precision recall (PR) curves, exist. Davis and Goadrich (2006) have shown that a deep relationship exists between ROC space and PR space, such that a curve dominates in ROC space if and only if it dominates in PR space. Precision and Recall are expressed as TP Precision = , (71) TP + FP TP Recall = . (72) TP + FN Martin, Fowlkes and Malik (2004) emphasize that PR curves are better adapted for color image segmentation by edge detection than ROC curves. 2. Quantitative Evaluation Methods without Ground Truth The mislabeling rate often is used to evaluate the quality of the segmentation. However, a segmented image Ia with a mislabeling rate lower than a segmented image Ib may not correspond to a better segmentation with a visual estimation, as illustrated by Liu and Yang (1994). They explain this phenomenon by the variable sensitivity of human perception according to the inspected scene. Moreover, it is demanding to construct ground truth segmentation of real scene images. Martin et al. (2001) have built a benchmark image database that contains human segmentation of real scene images. Global measures of segmentation quality were proposed first by Liu and Yang (1994) and improved by Borsotti, Campadelli and Schettini (1998). Both criteria do not require evaluation parameters. a. Criterion of Liu and Yang. To evaluate segmentation results both locally and globally, the evaluation function F (I) is defined by Liu and Yang (1994) as F (I) =
#
NR ×
NR e2 √i , Ai i=1
(73)
where NR is the number of regions in the segmented image, Ai is the√area of the ith region, and ei its mean error color. In this equation, the term NR penalizes√an oversegmentation while the local measure is estimated by the term ei2 / Ai and penalizes small regions or regions with a large color error. The lower the value of the criterion F (I), the better the segmentation result.
COLOR SPACES AND IMAGE SEGMENTATION
147
b. Criterion of Borsotti, Campadelli and Schettir. Liu and Yang’s criterion and Borsotti et al.’s criterion are empirically defined as Zhang (1996) suggests. Borsotti et al. improve Liu and Yang’s criterion by a heavier penalization of the segmented image with too many small regions and inhomogeneous color regions. For this task, the sum of Eq. (73) is split into two terms rather than only one. The criterion of Borsotti et al. Q(I) of an X × Y segmented image I can be computed as √
Q(I) =
R ei2 NR NR (Ai )2 × + , X×Y 1 + log Ai A2i i=1
N
(74)
where NR (Ai ) is the number of regions having an area equal to Ai . 3. Evaluation Methods for Color Image Segmentation by Region Construction This section provides an overview of several works that used the previous evaluation methods to determine whether a single color space that provides best results for color image segmentation by region construction exists. As in Section IV.B.2, these studies are summarized in a table (see Table 8). As shown in Tables 6 and 8 there is not single color space that is recommended by all studies. Thus, we can conclude that there does not exist a single color space that provides the best results for color image segmentation. In fact, the choice of the color space varies according to the type of image to segment and the segmentation method used. D. Selection of the Most Discriminating Color Space Sections IV.B and IV.C have shown contradictory conclusions about the pertinence of the available color spaces in the context of image segmentation. Instead of searching for the best classical color space to improve the results of image segmentation by pixel classification, Vandenbroucke, Macaire and Postaire (2003) defined a new type of color space by selecting a set of color components that can belong to any of several color spaces. Such spaces, which have neither psychovisual nor physical color significance, are called hybrid color spaces (HCS) by the authors and have been used to extract meaningful regions representing the soccer players and to recognize their teams. More details about this original approach are presented in Section IV.D.1. Another interesting approach proposed by Busin et al. (2005) consists of selecting a specific color space for the construction of each extracted class of pixels by using to an iterative procedure. We explain this method in Section IV.D.2.
148
BUSIN , VANDENBROUCKE AND MACAIRE
TABLE 8 M ETHODS U SED TO E VALUATE C OLOR I MAGE S EGMENTATION BY R EGION C ONSTRUCTION Reference
Evaluation method
Candidate color spaces
Selected color spaces
Meas-Yedid et al. (2004)
Liu’s criterion
(R, G, B), (r, g, b), (H 1, H 2, H 3) (I 1, I 2, I 3), (X, Y, Z), (Y, I, Q) (L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ )
(I 1, I 2, I 3)
Borsotti’s criterion specific criterion
(Y , I , Q ) (H 1, H 2, H 3)
Makrogiannis, Economou and Fotopoulos (2005)
PSNR
(R, G, B), (Y, Cb , Cr ), (Y, U, V ), (L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ )
(L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ )
Liu and Yang (1994)
Liu’s criterion
(R, G, B), (I, S, T ), (I 1, I 2, I 3), (L∗ , a ∗ , b∗ ), (X, Y, Z), (Y , I , Q )
Varies according to the image type
Littmann and Ritter (1997)
ROC curves
(R, G, B), (Y, u, v), (Y, QRG , QRB )
(R, G, B)
Phung, Bouzerdoum and Chai (2005)
ROC curves
(R, G, B), (H, S, V ), (Y, Cb , Cr ), (L∗ , a ∗ , b∗ ), (r, g), (H, S), (Cb , Cr ), (a ∗ , b∗ )
(Cb , Cr )
Lezoray (2003)
MSE
(R, G, B), (X, Y, Z), (H, S, L), (I 1, I 2, I 3), (Y , I , Q ), (Y , U , V ), (Y, Ch1 , Ch2 ), (Y, Cb , Cr ), (L∗ , u∗ , v ∗ ), (L∗ , a ∗ , b∗ )
Color spaces ordered
Lim and Lee (1990)
Probability error
(R, G, B), (X, Y, Z), (Y , I , Q ), (U ∗ , V ∗ , W ∗ ), (I 1, I 2, I 3)
(I 1, I 2, I 3)
Park, Yun and Lee (1998)
Probability error
(R, G, B), (X, Y, Z), (Y , I , Q ), (U ∗ , V ∗ , W ∗ ), (I 1, I 2, I 3)
Segmented images are unaffected by the choice of the color space
COLOR SPACES AND IMAGE SEGMENTATION
149
1. Pixel Classification in an Adapted Hybrid Color Space Instead of searching for the best classical color space to improve the results of image segmentation by pixel classification, Vandenbroucke, Macaire and Postaire (2003) defined an HCS by selecting a set of color components that can belong to any of the different color spaces listed in Figure 13. The HCSs have been used to extract meaningful regions representing the soccer players and to recognize their teams. The goal of the analysis of these images is to extract the players and the referee (Figure 35). In this context, there are three pixel classes of interest—the pixels representing the players of the two opposing teams and the referee—which are identified by the colors of their uniforms. This section is divided as follow. First, we detail the determination of an HCS. Second, we present the color pixels classification algorithm. Finally, we show several results obtained with this method. a. Adapted HCS Construction. For this study, the color components of the HCS are chosen among a set of color components Π K = (R, G, B, r, g, b, I, H, S, X, Y, Z, x, y, z, L∗ , a ∗ , b∗ , u∗ , v ∗ , I 1, I 2, I 3, A, C1, C2, Y , I , Q , ∗ , ho , S ∗ , C ∗ , ho ), where K is the number of available color U , V , Cuv uv uv ab ab components. Each player pixel P (x, y) in Figure 35 is represented by a point in the space Π K whose kth coordinate is denoted π k (x, y). ωj is a class of player pixels (j = 1, . . . , Nω where Nω is the number of classes). They also denote Nωj , the number of player pixels in the class of player pixels ωj . For each color feature of this space, the average mkj of the player pixels values in class of player pixels ωj is computed as: mkj =
1 × Nωj
π k (x, y).
(75)
P (x,y)∈ωj
F IGURE 35. Example of three classes of pixels representing the players of the two opposing teams and the referee. (See Color Insert.)
150
BUSIN , VANDENBROUCKE AND MACAIRE
Thus, they define a K-dimensional color feature vector T Mj = m1j , . . . , mkj , . . . , mK j for each class of player pixels ωj . In order to select the set of most discriminating color features among K available color features, they propose use of a specific informational criterion. They assume that the better the classes are separated and compact in the HCS, the higher the discriminating power of the selected feature. That assumption leads to their choice of the measures of separability and compactness. The measure of compactness of each class ωj (j = 1, . . . , Nω ) is defined by the within-class dispersion matrix Σj Σj =
1 × Nωj
(Xj − Mj )(Xj − Mj )T ,
(76)
P (x,y)∈ωj
where Xj = [π 1 (x, y), . . . , π k (x, y), . . . , π K (x, y)]T is the color point of the pixel P (x, y) that belongs to the class ωj . They define the total withinclass dispersion matrix ΣC as ω 1 × Σj . Nω
N
ΣC =
(77)
j =1
The measure of the class separability is defined by the between-class dispersion matrix ω 1 × (Mj − M)(Mj − M)T , Nω
N
ΣS =
(78)
j =1
where M = [m1 , . . . , mk , . . . , mK ]T is the mean vector of all the classes ω 1 × Mj . Nω
N
M=
(79)
j =1
The most discriminating set of color features maximizes the criterion J = trace(ΣC−1 ΣS ). The HCS is constructed by means of the “knock-out” algorithm (Firmin et al., 1996), and its dimension d is based on a correlation measure. The dimensionality of the HCS increases while this correlation measure is lower than a given threshold. With this procedure, they select the most discriminating color features among the K available ones. The player pixels are classified in this most discriminating HCS.
COLOR SPACES AND IMAGE SEGMENTATION
151
b. Color Pixels Classification Algorithm. Before classifying the player pixels of a current color image, the player pixels are extracted. Then the R, G, and B features of each player pixel P (x, y) are transformed into HCS features. To classify a player pixel P (x, y) of a color image, Vandenbroucke, Macaire and Postaire (2003) consider the set of player pixels falling into a neighborhood of P (x, y). The size of this neighborhood depends on the mean player size in the image. Then for each player pixel, they evaluate a mean vector MP = [m1p , . . . , mdp ]T of the HCS features of the player pixels belonging to neighborhood. For each class ωj , they evaluate the Euclidean distance Dj (x, y) between the mean vector Mj of the class ωj and the mean vector MP in the HCS $ % d % 2 mk − mk . (80) D (x, y) = M − M = & j
j
P
j
p
k=1
Finally, a minimum decision rule is used to assign P (x, y) to the class ωj (j = 1, . . . , Nω ) for which Dj (x, y) is minimum. c. Results. Vandenbroucke, Macaire and Postaire (2003) illustrate the effectiveness of the HCS for color image segmentation by pixels classification to extract meaningful regions representing the soccer players and to recognize their teams. The images in Figure 36 constitute a test sample extracted from the same sequence. These images are not easy to segment because each of them contains at least two adjacent players. The player pixels extracted from the images of Figure 36 by means of the multithresholding scheme of Ohlander, Price and Reddy (1978) are shown in Figure 37. The images in Figure 38 show how the extracted player pixels have been classified in the (a ∗ , CU V ) adapted HCS. The player pixels assigned to the same class are labeled with the same color. The adjacent player pixels with the same label constitute regions that correspond to the different soccer players. Vandenbroucke, Macaire and Postaire (2003) compare the results (see Figure 38) with the ground truth of Figure 39 by means of the error classification rate (see Table 9). Since the mean classification error rate associated with the adapted hybrid color space (a ∗ , CU V ) is the lowest one, they conclude that the classification in the HCS provides better results than the classification in the classical color spaces. 2. Unsupervised Selection of the Most Discriminating Color Space For several applications such as image retrieval, color space must be selected without any learning. Busin et al. (2005) proposed a selection scheme based on an unsupervised learning analysis of color histogram.
152
BUSIN , VANDENBROUCKE AND MACAIRE
F IGURE 36.
F IGURE 37.
F IGURE 38.
Color soccer images (125 × 125 pixels). (See Color Insert.)
Player pixels extracted from the images in Figure 36. (See Color Insert.)
Player pixels of Figure 37 classified in the hybrid color space. (See Color Insert.)
F IGURE 39.
Ground truth of the player pixels of Figure 37. (See Color Insert.)
COLOR SPACES AND IMAGE SEGMENTATION
153
TABLE 9 C LASSIFICATION E RROR R ATES OF P LAYER P IXELS IN F IGURE 37 Color space
(r, g, b) (H, S, I ) (Y , CU V , hU V ) (a ∗ , CU V ) (R, G, B)
Classification error rates image (a)
image (b)
image (c)
image (d)
Mean error rate
32.63 8.63 12.28 11.5 24.12
10.01 27.77 17.15 7.35 29.25
5.19 11.63 23.49 1.86 44.57
0.65 1.81 6.14 0.29 48.23
10.24 14.39 15.08 4.98 36.73
Because the classes constructed by a classification procedure depend on the color space used, it would be prudent to select the color space that is the most relevant for detecting the modes that correspond to regions. For this purpose, Busin et al. (2005) assumed that the higher the discriminating power of the 1D histogram, the more probable the detected modes correspond to regions with the same colors. Thus from among several color spaces, the proposed procedure selects those for which the discriminating powers of the 1D histograms are the highest. The discriminating power of a 1D histogram depends on the number of detected modes and on the connectedness properties of pixels whose color component levels fall into these detected modes. They consider that the higher the number of detected modes and the greater the connectivity in the image plane of the pixels belonging to the detected modes, the more discriminating the considered 1D histogram. The proposed unsupervised procedure selects the most relevant color space for constructing each class of pixels at each iteration step. The selection simultaneously takes into account the color and spatial connectedness properties of the pixels in the image. Each stage of the proposed iterative method is shown by Figure 40 and is detailed in the following subsections. a. One-Dimensional Histograms Determination. At each iteration step, the color vectors of the pixels submitted to the analysis are represented into the NS = 11 color spaces ((R, G, B), (r, g, b), (X, Y, Z), (x, y, z), (Y , I , Q), (Y , U , V ), (bw, rg, by), (Y, Ch1 , Ch2 ), (L∗ , a ∗ , b∗ ), (L∗ , u∗ , v ∗ ), and (I 1, I 2, I 3)). In the ith color space, each of the three 1D histograms H i,j (x) of each color component numbered j (j = 1, 2, 3), where x is the color component level, is determined. The 1D histograms of the red, green, and blue components of the House image in Figure 41a are represented in Figures 41b, c, and d, respectively.
154
BUSIN , VANDENBROUCKE AND MACAIRE
F IGURE 40.
Color image segmentation flowchart.
b. One-Dimensional Histograms Smoothing. Because the 1D histograms are corrupted by noise, it is difficult to detect their modes. Thus, Busin et al. (2005) propose to smooth them by means of adaptive filtering. A smoothed i,j histogram Hσ (x) is computed by the convolution between the 1D histogram i,j H (x) and a Gaussian kernel gσ (x), where σ is the standard deviation i,j hi,j σ (x) = h (x) ∗ gσ (x),
(81)
where “∗” denotes the convolution operator and 2 1 −x . gσ (x) = √ exp 2σ 2 2π σ The effect of the smoothing depends on the standard deviation σ used to define the Gaussian kernel. For each 1D histogram, σ is automatically determined by means of the procedure proposed by Lin, Wang and Yang (1996), so that the smoothed 1D histogram reveals its modes. Figure 42 shows the smoothed 1D histograms of the House image of Figure 42a. These smoothed 1D histograms can be easily analyzed for mode detection. c. Most Relevant Color Space Selection. The thresholds that delimit the i,j modes of each smoothed histogram Hσ (x) are determined by the analysis of the zero-crossing of its first-derivative function. A threshold is detected by i,j a zero-crossing of the first-derivative function of Hσ (x) whose sign changes from minus to plus (local minimum). A mode is detected by a zero-crossing of i,j the first-derivative function of Hσ (x) whose sign changes from plus to minus (local maximum). The number of these detected modes is denoted N i,j .
COLOR SPACES AND IMAGE SEGMENTATION
(a)
(b)
(c)
(d)
155
F IGURE 41. One-dimensional histograms H i,j of the House image in the (R, G, B) color space (i = 1). (a) Original image. (b) Red one-dimensional histogram H 1,1 . (c) Green onedimensional histogram H 1,2 . (d) Blue one-dimensional histogram H 1,3 . (See Color Insert.)
Three features characterize the kth (k = 1, . . . , N i,j ) detected mode of the i,j 1D histogram Hσ (x): i,j,k
• The left and right detected thresholds Tleft • The amplitude
i,j,k
and Tright
i,j,k
lTright
Ai,j,k = max hi,j σ (l). i,j,k
l=Tleft i,j
For each 1D histogram Hσ (x), K(i, j ) denotes the rank order of the detected mode with the highest amplitude.
156
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c)
(d) i,j
F IGURE 42. Smoothed one-dimensional histograms Hσ by Lin et al.’s method processed from one-dimensional histograms of Figure 41. (a) Original image. (b) Smoothed one-dimensional histogram Hσ1,1 . (c) Smoothed one-dimensional histogram Hσ1,2 . (d) Smoothed one-dimensional histogram Hσ1,3 . (See Color Insert.)
To illustrate the procedure, Table 10 shows the features of the detected modes in the smoothed histograms of Figures 42b–d with only the (R, G, B) color space (i = 1). Figure 43 shows the features of the detected modes from the smoothed histogram Hσ1,3 (x) in Figure 42c. The main problem of analysis for schemes 1D histograms is that they analyze only the similarities between the colors of the pixels and ignore their spatial arrangement in the image. However, a region is a subset of pixels that shares similar color properties and is strongly connected in the image plane. To select the most relevant color space, Busin et al. (2005) measure the connectedness degrees of subsets of pixels S by means of connected degrees
157
COLOR SPACES AND IMAGE SEGMENTATION
TABLE 10 F EATURES OF THE D ETECTED M ODES IN THE 1D H ISTOGRAMS OF F IGURES 42 B – D IN THE (R, G, B) C OLOR S PACE j
1
N i,j k 1,j,k Tleft 1,j,k Tright A1,j,k
CD(S) R 1,j
2
2
3
3
2
1
2
1
2
3
1
2
135
20
145
75
0
140
30
205
134
225
144
74
255
139
1556
373
1051
894
273
1077
929
0.93
0.94
0.94
0.92
0.96
0.96 1.89
2.80
0.97 1.93
j is the number of the color component in the considered color space (j ∈ {1, 2, 3}).
F IGURE 43. Features of the detected modes from the smoothed 1D histogram Hσ1,3 of the blue component. (See Color Insert.)
CD(S) introduced by Macaire, Vandenbroucke and Postaire (2006). The connectedness degree CD(S) is a normalized measure, so that a connectedness degree close to 1 means that the pixels belonging to the subset S are strongly connected in the image, whereas a connectedness degree close to 0 means that the pixels are sparsely scattered throughout the image. Table 10 shows the connectedness degrees CD(S) of the detected modes of the smoothed histograms shown in Figures 42b–d. Among the three smoothed 1D histograms of the ith color space, i = 1, . . . , NS , the most discriminating
158
BUSIN , VANDENBROUCKE AND MACAIRE
1D histogram is determined. For this purpose the discriminating power, dei,j noted R i,j , is evaluated for each smoothed 1D histogram Hσ (x). R i,j is the i,j,k i,j,k sum of the connectedness degrees of the subsets S[Tleft , Tright ] associated with the detected modes of the histogram i,j
R
i,j
=
N
i,j,k i,j,k CD S Tleft , Tright .
(82)
k=1
Because the Gaussian smoothing eliminates the nonsignificant modes, the 1D histogram allows discrimination of the classes if the number of the detected modes is high and if the sum of their connectedness degrees is high. Thus, the higher the discriminating power R i,j , the more probable the modes correspond to regions with the same colors in the image. Therefore, the most discriminating 1D histogram of the ith color space is the 1D histogram with the highest discriminating power R i,j . They denote J (i) as the rank order of the color component that corresponds to the most discriminating 1D histogram of the ith color space—that is, the 1D histogram with the highest discriminating power R i,j . Table 10 shows that, in our example, among the 3D histograms of Figure 42, the most discriminating histogram of the (R, G, B) color space (i = 1) is the histogram of the component G, so J (1) is set to 2. The most relevant color space is selected among the NS color spaces as that with the highest discriminating power. If the discriminating powers of several color spaces are equal, the most relevant color space among those spaces is selected as that which contains the 1D histogram with the second highest value of discriminating power R i,j . They denote I the rank order of the color space that is selected as the most relevant one. For the House image in Figure 41a, the most relevant color space that is selected at the first step is the (Y , U , V ) color space (I = 6). The most discriminating 1D histogram of the selected color space is the V component (J (6) = 3). d. One Class Construction. One class of pixels is constructed by analyzing the most relevant color space with rank order I , which has been determined at the current iteration step of the algorithm. This class of pixels is defined by a parallelepipedic box in the most relevant color space. Only one parallelepipedic box is selected to build the class of pixels. This box is delimited by two thresholds defined along each color component of the most relevant color space. Along the color component with rank order J (I ) which corresponds to the I,J (I ),K(I,J (I )) most discriminating 1D histogram, the two thresholds Tleft and I,J (I ),K(I,J (I ))
Tright are those that delimit the mode with the highest amplitude. The thresholds along the two other color components are selected among the
COLOR SPACES AND IMAGE SEGMENTATION I,j,k
159
I,j,k
thresholds Tleft and Tright , j = J (I ), determined by the mode detection stage. The selected thresholds delimit the box into which fall the color vectors of the highest population of pixels. The pixels whose color vectors fall into this box constitute the class of pixels constructed at the current iteration step. e. Pixels of the Class Extraction. The pixels assigned to the constructed class are extracted from the color image so that they are not taken into account at the next iteration steps of the procedure. The pixels that are assigned to this class and which are connected in the image constitute one of the reconstructed regions in the segmented image. f. Stopping Criterion. The iterative procedure stops when a percentage p of pixels of the image have not been assigned to any of the previously constructed classes. The parameter p, adjusted by the analyst, allows the desired coarseness of the segmentation be tuned. When the iterative procedure stops, the pixels that have not been assigned to any class could be assigned to one of the constructed classes by means of a specific decision rule. g. Results. This class construction scheme is based on the analysis of both the connectedness and the color properties of the subsets of pixels. To demonstrate the interest of this approach, Busin et al. (2005) propose segmenting the benchmark image named House (see Figure 44a) by means of the presented procedure (see Figure 44b). The extracted pixels of a constructed class at each iteration step of the algorithm are labeled with a false color in the segmented image of Figure 44b. These labels are represented in the first column of Table 11. As these false colors are ordered, the iteration steps at which the classes of pixels are constructed by examining the proposed segmented images results can be determined. Table 11 indicates the discriminating powers of the color spaces selected at each step of the procedure applied to the image in Figure 44a. It shows that there does not exist one single color space that is the most relevant at all iteration steps of the procedure. Figure 44b shows that this procedure provides satisfying segmentation results in terms of pixel classification. Indeed, the regions representing the different objects in the images are well reconstructed. The result obtained with the proposed method is compared with the one obtained when only the (R, G, B) color space is taken into account by each iteration step (see Figure 44c), and with the one obtained when only the most relevant color space selected at the first iteration step is taken into account by the other iteration steps (see Figure 44d). By examining the segmented images, Busin et al. (2005) conclude that the selection of the most relevant
160
BUSIN , VANDENBROUCKE AND MACAIRE
(a)
(b)
(c)
(d)
F IGURE 44. Segmentation of the House image (255 × 255 pixels) by the proposed method. (a) Original image House. (b) Segmented image House by the proposed method. (c) Segmented image House when the (R, G, B) color space is selected at each iteration. (d) Segmented image House when the (Y , U , V ) color space is selected at each iteration. (See Color Insert.) TABLE 11 N UMBER OF M ODES N I,J (I ) AND D ISCRIMINATING P OWER R I,J (I ) OF THE M OST D ISCRIMINATING 1D-H ISTOGRAM J (I ) OF THE C OLOR S PACE I S ELECTED AT E ACH S TEP OF THE P ROCEDURE A PPLIED TO THE H OUSE I MAGE Iteration step
(See Color Insert.)
I
J (I )
N I,J (I )
R I,J (I )
(Y , U , V ) (Y , U , V ) (R, G, B) (R, G, B) (bw, rg, by) (I 1, I 2, I 3) (r, g, b) (Y , U , V )
3 3 2 2 1 1 1 1
4 3 5 4 3 3 2 2
3.42 2.54 3.48 2.52 1.75 1.94 1.20 1.49
COLOR SPACES AND IMAGE SEGMENTATION
161
color space provides results that are more acceptable in terms of segmentation quality than the class construction achieved in one single color space. These results show that the selection of different color spaces at the iteration steps of the procedure, which is designed to discriminate the considered pixel classes, is a relevant solution for the construction of the regions. E. Conclusion This section has presented several evaluation methods of color image segmentation to highlight the relationships between segmentation and color spaces. These evaluation methods have been used to compare the segmentation results according to the chosen color space used to represent the colors of the pixels. We have shown that the choice of the color space depends on the kind of image to be segmented and the segmentation method. Because no single “best” color space exists for color image segmentation, several studies select them by means of decision rules. The results provided by these color image segmentation methods allow improved image segmentation results.
V. C ONCLUSION In this chapter we have first described the most used color spaces for digital color image analysis. We have pointed out that most transformations from (R, G, B) to another color space require the prior knowledge of the acquisition conditions (reference white, illuminating properties). Moreover, the coding of the color components must be adapted to the used color space. In the second section we presented classical segmentation schemes designed for exploiting colors, which can be divided in two families. Since edge detection requires postprocessing steps to yield closed boundaries, most schemes construct the regions in the image via either image plane analysis or color space analysis. Recent approaches tend to combine the spatial and the colorimetric analysis to improve the quality of the segmentation. The third section deals with the relationships between color spaces and segmentation. Regardless of the criteria used to evaluate the quality of a segmentation, no color spaces effect in which segmentation schemes provide efficient results of all images, which prompted our development of schemes to determine the color space that is the most adapted to a specific set of images. These schemes are based on multidimensional statistical criteria but not on psychovisual criteria. One improvement in our approaches could be the integration of these criteria to the selection of the color space for image segmentation.
162
BUSIN , VANDENBROUCKE AND MACAIRE
R EFERENCES Abney, W.W. (1913). Researches in Color Vision. Longmans, Green, London. Bala, R., Sharma, G. (2005). System optimization in digital color imaging. IEEE Signal Processing Magazine 22 (1), 55–63. Balasubramanian, R., Allebach, J.P., Bouman, C.A. (1994). Color-image quantization with use of a fast binary splitting technique. J. Opt. Soc. Am. 11 (11), 2777–2786. Ballard, D.H., Brown, C.M. (1982). Computer Vision. Prentice-Hall, New Jersey. Borsotti, M., Campadelli, P., Schettini, R. (1998). Quantitative evaluation of color image segmentation results. Pattern Recogn. Lett. 19 (8), 741–747. Braquelaire, J.-P., Brun, L. (1997). Comparison and optimization of methods of color image quantization. IEEE Trans. Image Proc. 6 (7), 1048–1052. Busin, L., Vandenbroucke, N., Macaire, L., Postaire, J.-G. (2004). Color space selection for unsupervised color image segmentation by histogram multithresholding. In: IEEE International Conference on Computer Vision. Singapore, pp. 203–206. Busin, L., Vandenbroucke, N., Macaire, L., Postaire, J.-G. (2005). Color space selection for unsupervised color image segmentation by analysis of connectedness properties. Int. J. Robot. Autom. 20 (2), 70–77. Chabrier, S., Emile, B., Rosenberger, C., Laurent, H. (2006). Unsupervised performance evaluation of image segmentation. In: Special Issue on Performance Evaluation in Image Processing. EURASIP J. Appl. Signal Proc. 5, 1–12. Chassery, J.M., Garbay, C. (1984). An iterative method based on a contextual color and shape criterion. IEEE Trans. Pattern Anal. Mach. Intell. 6 (6), 794–800. Cheng, H., Jiang, X., Wang, J. (2002). Color image segmentation based on homogram thresholding and region merging. Patt. Recogn. 35 (2), 373– 393. Cheng, H.D., Chen, C.H., Chiu, H.H., Xu, H.J. (1998). Fuzzy homogeneity approach to multilevel thresholding. IEEE. Trans. Image Proc. 7 (7), 1084– 1088. Cheng, H.D., Jiang, X.H., Sun, Y., Wang, J. (2001). Color image segmentation: advances and prospects. Pattern Recogn. 34 (12), 2259–2281. Comaniciu, D., Meer, P. (2002). Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24 (5), 603–619. Commission Internationale de l’Éclairage (1986). Colorimetry. Technical report 15.2. Bureau central de la CIE, Vienna. Commission Internationale de l’Éclairage (1995). Industrial color-difference evaluation. Technical report 116. Bureau central de la CIE.
COLOR SPACES AND IMAGE SEGMENTATION
163
Commission Internationale de l’Éclairage (2004). A color appearance model for color management systems: CIECAM02. Technical report. 159. Bureau central de la CIE, Vienna. Cumani, A. (1991). A note on the gradient of multi-image edge detection in multispectral images. Comput. Vis. Graph. Image Proc. 53, 40–51. Davis, J., Goadrich, M. (2006). The relationship between precision-recall and ROC curves. In: International Conference on Machine Learning, pp. 233– 240. Deng, Y., Manjunath, B. (2001). Unsupervised segmentation of color-texture regions in images and video. IEEE Trans. Pattern Anal. Mach. Intell. 23 (8), 800–810. Deriche, R. (1990). Fast algorithms for low level vision. IEEE Trans. Pattern Anal. Mach. Intell. 12 (1), 78–87. Devijver, P., Kittler, J. (1982). Pattern Recognition: A Statistical Approach. Prentice-Hall, New York. Di Zenzo, S. (1986). A note on the gradient of multi-image. Comput. Vis. Graph. Image Proc. 33, 116–125. Faugeras, O.D. (1979). Digital color image processing within the framework of a human visual model. IEEE Trans. Acoust. Speech Signal Process. 27 (4), 380–393. Firmin, C., Hamad, D., Postaire, J.-G., Zhang, R.D. (1996). Feature extraction and selection for fault detection in production of glass bottles. Machine Graphics & Vision 5 (1), 77–86. Foley, J.D., Dam, A.V., Feiner, S.K. (1990). Computer Graphics, Principles and Practice, 2nd ed. Addison-Wesley, New York. Fontaine, M., Macaire, L., Postaire, J.-G. (2000). Image segmentation based on an original multiscale analysis of the pixel connectivity properties. In: IEEE International Conference on Image Processing, vol. 1. Vancouver, BC, pp. 804–807. Garbay, C., Brugal, G., Choquet, C. (1981). Application of colored image analysis to bone marrow cell recognition. Anal. Quant. Cytol. 3 (4), 272– 280. Gillet, A., Macaire, L., Botte-Lecocq, C., Postaire, J.-G. (2002). Fuzzy filters for image processing. In: Nachtegael, M., Van der Weken, D., Van De Ville, D., Kerre, E.E. (Eds.), Color Image Segmentation by Analysis of 3D Histogram With Fuzzy Morphological Filters. In: Studies in Fuzziness and Soft Computing. Springer, pp. 154–177. Grassman, H. (1853). On the theory of compound colors. Phil. Mag. Series 4 (7), 254–264. Gunturk, B.K., Glotzbach, J., Altunbasak, Y., Schafer, R.W., Mersereau, R.M. (2005). Demosaicking: Color filter array interpolation. IEEE Signal Processing Magazine 22 (1), 44–54.
164
BUSIN , VANDENBROUCKE AND MACAIRE
Helmholtz, H. (1866). Handbuch der physiologishen Optik. Voss, Hambourg, Leipzig. Hering, E. (1875). Zur lehre vom lichtsinne. Wien. Math. Nat. Kl. 70, 169– 204. IEC 61966-2-1/FDIS (1999). Multimedia systems and equipment—Color measurement and management—part 2-1: Color management—Default RGB color space—SRGB. Technical report. International Electrotechnical Commission. Ismail, M.A., Kamel, M.S. (1989). Multidimensional data clustering utilizing hybrid search strategies. Patt. Recogn. 22 (1), 75–89. ITU-R BT.601-7 (2007). Studio encoding parameters of digital television for standard 4:3 and wide screen 16:9 aspect ratios. Technical report. International Telecommunication Union. ITU-R BT.709-5 (2002). Parameter values for the HDTV standards for production and international programe exchange. Technical report. International Telecommunication Union. Kender, J.R. (1976). Saturation, hue, and normalized color: Calculation, digitization effects, and use. Technical report. Department of Computer Science, Carnegie-Mellon University, Pittsburgh. Klinker, G.J., Shafer, S.A., Kanade, T. (1990). A physical approach to color image understanding. Int. J. Comput. Vis. 4 (1), 7–30. Lambert, P., Carron, T. (1999). Symbolic fusion of luminance-hue-chroma features for region segmentation. Patt. Recogn. 32 (11), 1857–1872. Lambert, P., Macaire, L. (2000). Filtering and segmentation: The specifity of color images. In: International Conference on Color in Graphics and Image Processing. Saint-Etienne, France, pp. 57–71. Lee, H.C., Cok, D. (1991). Detection boundaries in a vector field. IEEE Trans. Signal Proc. 39 (5), 1181–1194. Lee, J.H., Chang, B.H., Kim, S.D. (1994). Comparison of color transformations for image segmentation. Electron. Lett. 30 (20), 1660–1661. Lee, S.U., Chung, S.Y., Park, R.-H. (1990). A comparative performance study of several global thresholding techniques for segmentation. Comput. Vis. Graphics Image Proc. 52 (2), 171–190. Lee, T., Lewicki, M.S. (2002). Unsupervised image classification, segmentation, and enhancement using ICA mixture model. IEEE Trans. Image Proc. 11 (3), 271–279. Levkowitz, H., Herman, G.T. (1993). GLHS: A generalized lightness, hue, and saturation color model. Graph. Model Image Proc. 55 (4), 271–285. Lezoray, O. (2003). Supervised automatic histogram clustering and watershed segmentation. Application to microscopic medical images. Image Anal. Stereol. 22 (2), 113–120. Liew, A., Leung, S., Lau, W. (2000). Fuzzy image clustering incorporating spatial continuity. IEEE Proc. Vis. Image Sign. Proc. 147 (2), 185–192.
COLOR SPACES AND IMAGE SEGMENTATION
165
Lim, Y.W., Lee, S.U. (1990). On the color image segmentation algorithm based on the thresholding and the fuzzy c-means techniques. Patt. Recogn. 23 (9), 935–952. Lin, H.-C., Wang, L.-L., Yang, S.-N. (1996). Automatic determination of the spread parameter in Gaussian smoothing. Patt. Recogn. Lett. 17, 1247– 1252. Littmann, E.
