Computer Vision and Graphics: International Conference, ICCVG 2008, Warsaw, Poland, November 10-12, 2008 Revised Papers (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Leonard Bolc | Juliusz Lech Kulikowski | Konrad Wojciechowski

89 downloads 1538 Views 14MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5337

Leonard Bolc Juliusz L. Kulikowski Konrad Wojciechowski (Eds.)

Computer Vision and Graphics International Conference, ICCVG 2008 Warsaw, Poland, November 10-12, 2008 Revised Papers

13

Volume Editors Leonard Bolc Konrad Wojciechowski Polish-Japanese Institute of Information Technology Koszykowa 86, 02-008 Warsaw, Poland E-mail: {pjwstk, konradw}@pjwstk.edu.pl Juliusz L. Kulikowski Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences Ksie˛cia Trojdena 4, 02-109 Warsaw, Poland E-mail: [email protected] Konrad Wojciechowski The Silesian University of Technology Institute of Computer Science Akademicka 16, 44-100 Gliwice, Poland E-mail: [email protected]

Library of Congress Control Number: Applied for CR Subject Classiﬁcation (1998): I.4, I.3.5, I.5, I.2.8-10, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13

0302-9743 3-642-02344-4 Springer Berlin Heidelberg New York 978-3-642-02344-6 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microﬁlms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientiﬁc Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12695829 06/3180 543210

Preface

The need for eﬃcient information processing, especially in computer vision and graphics, is dramatically increasing. Growth in these ﬁelds stimulated by emerging applications has been both in concepts and techniques. New ideas, concepts and techniques are developed, presented, discussed and evaluated, subsequently being expanded or abandoned. The objectives of the ICCVG conference are: presentation of current research topics and discussions leading to the integration of the community engaged in machine vision and computer graphics, carrying out and supporting research in the ﬁeld and ﬁnally promotion of new applications. The ICCVG is a continuation of the former International Conference on Computer Graphics and Image Processing called GKPO, held in Poland every second year in May since 1990. ICCVG 2008 gathered about 100 authors. The ICCVG 2008 proceedings contain 49 papers, each accepted on the basis of reviews by three independent referees. Contributions are organized into the following sessions corresponding to the scope of the conference: image processing, image quality assessment, geometrical models of objects and scenes, motion analysis, visual navigation and active vision, image and video coding, virtual reality and multimedia applications, biomedical applications, practical applications of pattern recognition, computer animation, visualization and graphical data presentation. ICCVG 2008 was organized by the Polish–Japanese Institute of Information Technology and Association for Image Processing. The Polish–Japanese Institute of Information Technology was founded in 1994 by the Computer Techniques Development Foundation under the agreement of the Polish and Japanese governments and is one of the leading, nonstate (private) Polish universities and cooperates with a number of EU, US and Japanese universities. The Association for Image Processing integrates the Polish community working on the theory and applications of computer vision and graphics. This is done through the organization of scientiﬁc and technical meetings, publishing activities, establishing relations with other organizations having similar objectives, and promoting image processing-based methods in the industrial community. We would like to thank all members of the Program Committee, as well as the additional reviewers, for their help in selecting and ensuring the high quality of the papers. We would also like to thank Bernadeta Bonio and Pawel Wieman and for their engagement in the conference organization and administration. We are highly grateful to the Polish–Japanese Institute of Information Technology for hosting the conference and for the help in the conference organization. Finally, we would like to invite everyone to the next conference, ICCVG 2010,

VI

Preface

which will take place in the Polish–Japanese Institute of Information Technology, Warsaw, Poland. November 2008

Leonard Bolc Juliusz Lech Kulikowski Konrad Wojciechowski

Organization

• Association for Image Processing (TPO) • Polish–Japanese Institute of Information Technology (PJIIT)

Conference General Chairs L. Bolc (Poland) J.L. Kulikowski (Poland) K. Wojciechowski (Poland)

Conference Committee Members: E. Bengtsson (Sweden) P. Bhattacharya (USA) L. Bolc (Poland) A. Borkowski (Poland) D. Chetverikov (Hungary) L. Chmielewski (Poland) R. Choras (Poland) A. Datta (Australia) S. Dellepiane (Italy) M. Domanski (Poland) U. Eckhardt (Germany) A. Gagalowicz (France) E. Grabska (Poland) H. Heijmans (The Netherlands) J.M. Jolion (France) A. Kasinski (Poland) R. Klette (New Zealand) J. Kohout (Czech Republic) I. Koligerova (Czech Republic) W. Kosinski (Poland) P. Kovesi (Australia) R. Kozera (Australia) M. Kurzynski (Poland) G. Levina (Russia) R. Lukac (Canada) V. Lukin (Ukraine)

A. Polanski (Poland) K. Marasek(Poland) A. Materka (Poland) K. Myszkowski (Poland) H. Niemann (Germany) M. Nieniewski (Poland) L. Noakes (Australia) M. Orkisz (France) H. Palus (Poland) D. Paulus (Germany) L. Polkowski (Poland) J. Roerdink (The Netherlands) P. Rokita (Poland) R. Sara (Czech Republic) V. Skala (Czech Republic) B. Smolka (Poland) J. Soldek (Poland) G. Stanke (Germany) K. Subieta (Poland) M. Syslo (Poland) R. Tadeusiewicz (Poland) T. Vintsiuk (Ukraine) K. Wiatr (Poland) J. Zabrodzki (Poland) M. Zaremba (Canada)

Table of Contents

Image Processing Architecture of an Integrated Software-Hardware System for Accelerated Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Boguslaw Cyganek

1

A Fast Logical-Morphological Method to Segment Scratch - Type Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas Kuleschow, Christian M¨ unzenmayer, and Klaus Spinnler

14

Spatio–Temporal Track–Before–Detect Algorithm for Interlaced and Progressive Scan Imaging Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Mazurek

24

Road Lane Detection with Elimination of High-Curvature Edges . . . . . . . ´ Krzysztof Slot, Michal Strzelecki, Agnieszka Krawczy´ nska, and Maciej Pola´ nczyk

33

Image Quality Assessment A Statistical Reduced-Reference Approach to Digital Image Quality Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Okarma and Piotr Lech

43

Colour Image Quality Assessment Using Structural Similarity Index and Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Okarma

55

Top-Down Approach to Image Similarity Measures . . . . . . . . . . . . . . . . . . . Jacek Piotrowski The Inﬂuence of Picture Quality Scale Measure on Choosing the Wavelet Transform during Image Compression . . . . . . . . . . . . . . . . . . . . . . . Maria Skublewska-Paszkowska and Jakub Smolka Image Quality Assessment Using Phase Spectrum Correlation . . . . . . . . . Przemyslaw Skurowski and Aleksandra Gruca

66

70 80

Geometrical Models of Object and Scenes A New Image Fusion Method for Estimating 3D Surface Depth . . . . . . . . Marcin Denkowski, Michal Chlebiej, and Pawel Mikolajczak

90

X

Table of Contents

Collecting 3D Content: Examples from Art and Medicine . . . . . . . . . . . . . Leszek Luchowski

100

3D Object Reconstruction from Parallel Cross-Sections . . . . . . . . . . . . . . . Karolina Nurzy` nska

111

Implementation of Progressive Meshes for Hierarchical Representation of Cultural Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Skabek and L ukasz Z¸abik

123

Motion Analysis, Visual Navigation and Active Vision Multi-Object Tracking Based on Particle Filter and Data Association in Color Image Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ayoub Al-Hamadi, Saira Saleem Pathan, and Bernd Michaelis Residual of Resonant SVD as Salient Feature . . . . . . . . . . . . . . . . . . . . . . . . Dmitry Chetverikov

133 143

Building Pedestrian Contour Hierarchies for Improving Detection in Traﬃc Scenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ion Giosan and Sergiu Nedevschi

154

Potential Field Based Camera Collisions Detection within Translating 3D Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Wojciechowski

164

Image and Video Coding A Simple Quantitative Model of AVC/H.264 Video Coders . . . . . . . . . . . . Tomasz Grajek and Marek Doma´ nski

174

Approximation of Signals by Predict Wavelet Transform . . . . . . . . . . . . . . Marcin Jaromin

184

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jaroslaw Marek

198

Lossless and Near-Lossless Image Compression Scheme Utilizing Blending-Prediction-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grzegorz Ulacha and Piotr Dziurzanski

208

Table of Contents

XI

Virtual Reality and Multimedia Applications Marker Less Vision-Based Tracking of Partially Known 3D Scenes for Outdoor Augmented Reality Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . Fakhreddine Ababsa, Jean-Yves Didier, Imane Zendjebil, and Malik Mallem Geometric and Optical Flow Based Method for Facial Expression Recognition in Color Image Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ayoub Al-Hamadi, Robert Niese, Saira S. Pathan, and Bernd Michaelis

218

228

Local Rank Patterns – Novel Features for Rapid Object Detection . . . . . Michal Hradis, Adam Herout, and Pavel Zemcik

239

Detection of Dogs in Video Using Statistical Classiﬁers . . . . . . . . . . . . . . . Roman Jur´ anek

249

Automatic Video Editing for Multimodal Meetings . . . . . . . . . . . . . . . . . . . Radek Kubicek, Pavel Zak, Pavel Zemcik, and Adam Herout

260

Foreground Segmentation via Segments Tracking . . . . . . . . . . . . . . . . . . . . . Bogdan Kwolek

270

Multi-layer Background Change Detection Based on Spatiotemporal Texture Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Roland Miezianko and Dragoljub Pokrajac

282

Biomedical Applications The Development and Validation of a Method for 4D Motion Reconstruction of a Left Ventricle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michal Chlebiej, Marcin Denkowski, and Krzysztof Nowi´ nski

292

Estimation of Eye Blinking Using Biopotentials Measurements for Computer Animation Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert Krupi´ nski and Przemyslaw Mazurek

302

Approximation of Subtle Pathology Signs in Multiscale Domain for Computer-Aided Ischemic Stroke Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . Artur Przelaskowski, Rafal J´ o´zwiak, Grzegorz Ostrek, and Katarzyna Sklinda Iris Identiﬁcation Using Geometrical Wavelets . . . . . . . . . . . . . . . . . . . . . . . Mariusz Rachubi´ nski Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrzej Zacher

311

322

333

XII

Table of Contents

Practical Applications of Pattern Recognition Mixtures of Classiﬁers for Recognizing Standing and Running Pedestrians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Raluca Borca-Mure¸san, Sergiu Nedevschi, and Florin M˘ aguran

345

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Irene Y.H. Gu, Henrik Andersson, and Ra´ ul Vicen

356

Automatic Surveillance and Analysis of Snow and Ice Coverage on Electrical Insulators of Power Transmission Lines . . . . . . . . . . . . . . . . . . . . Irene Y.H. Gu, Unai Sistiaga, Sonja M. Berlijn, and Anders Fahlstr¨ om GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Herout, Radovan Josth, Pavel Zemcik, and Michal Hradis Image Recognition Technique for Unmanned Aerial Vehicles . . . . . . . . . . . Karol J¸edrasiak and Aleksander Nawrat The Performance of Two Deformable Shape Models in the Context of the Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Schmidt and Andrzej Kasinski A Hierarchical Model for the Recognition of Deformable Objects . . . . . . . Martin Stommel and Klaus-Dieter Kuhnert

368

380 391

400 410

Computer Animation Exploiting Quaternion PCA in Virtual Character Motion Analysis . . . . . Tomasz Grudzinski

420

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics . . . Jakub Grudzi´ nski

430

Debugging, Object and State Management with OpenGL 1.x and 2.x . . . Jaroslaw Konrad Lipowski

441

Estimation of State–Space Spatial Component for Cuboid Track–Before–Detect Motion Capture Systems . . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Mazurek

451

Visualization and Graphical Data Presentation Automatic Extraction of Graph-Like Structures from Binary Images . . . . Marcin Iwanowski

461

Table of Contents

XIII

‘SNN3DViewer’ - 3D Visualization Tool for Spiking Neural Network Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrzej Kasi´ nski, Juliusz Pawlowski, and Filip Ponulak

469

Using Graph Transformations in Distributed Adaptive Design System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leszek Kotulski and Barbara Strug

477

The Lifting Scheme for Multiresolution Wavelet-Based Transformation of Surface Meshes with Additional Attributes . . . . . . . . . . . . . . . . . . . . . . . . Agnieszka Szcz¸esna

487

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

497

Architecture of an Integrated Software-Hardware System for Accelerated Image Processing Boguslaw Cyganek AGH - University of Science and Technology Al. Mickiewicza 30, 30-059 Krak´ ow, Poland [email protected]

Abstract. The paper presents an architecture and features of a hybrid software/hardware system aimed at acceleration of image processing. Special design methodology of the software layers has been undertaken to meet the requirements of hardware and software cooperation. The front end consists of a ﬁxed software interface. On the other hand, implementation variability is embedded into the complementary software and hardware layers, from which the best implementation can be selected in run time. The basic properties of the system are presented with some applications and discussion of further steps.

1

Introduction

Image processing (IP) is concerned with large amounts of data. Generally there are two, in many aspects complementary, ways of design and implementation of such systems. On the one hand, sequential algorithms allow considerably fast development and implementation, however at a cost of run-time performance. On the other hand, the parallel processing usually allows much faster performance but at a cost of hardware resources and expensive implementation. The former approach is inherent to software platforms while the latter relates to the hardware or multi-processor based solutions. For years there is a continuous research to improve both domains for better throughput of the systems. However, despite growing computational power of contemporary computers and development platforms, there are always new applications that require more processing power than is available. In this context the new possibility arose throughout recent years in a form of the programmable hardware. In this area the leading technologies are ASICs and FPGAs, with the latter oﬀering much more plausible cost/performance ratio for retail enterprises. However, the design and implementation costs are much higher than for software since there is no uniform way of porting serial algorithms to parallel counterparts, as well as the processing tasks need to be designed for a speciﬁc hardware conﬁgurations and available resources. Change of either factor results usually with a costly redesign. There were many attempts to build versatile libraries for IP. The best known examples of these are Intels OpenCV [10] and Microsoft Vision SDK [12]. At L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 1–13, 2009. c Springer-Verlag Berlin Heidelberg 2009

2

B. Cyganek

the other end goes research for IP hardware based solutions which are focused mostly on development of the intermediate level languages that can be directly compiled into VHDL or FPGA binaries. Recently Crookes et al. [2] devised a high level Image Processing Coprocessor (IPC) with a core instruction set based on the operations of Image Algebra (IA) [11]. Their system includes a generator which produces optimized architectures for speciﬁc user-deﬁned operations. The similar approach was undertaken by Draper et al. in the Cameron project [6]. They propose a Single Assignment C (SA-C) language that generates code for FPGA. There are many other projects in this area. However they are either oriented towards speciﬁc applications or the developed intermediate languages lack enough ﬂexibility for other hardware boards and usually they do not operate automatically with new families of programmable devices, such as Xilinx [13], neither eﬃciently use their resources [2]. In this paper we discuss the architecture, design process and implementation of the hybrid software/hardware (SH) system for IP. To the best of our knowledge there are not many papers devoted to these problems and especially to the codesign of software and hardware for IP. We present obtained results as well as we share our expertise. Our solution joins the software platform with the FPGAs based accelerating board. Many desirable features were achieved with the layered architecture and the object-oriented (OO) strategy for software design [7][1]. For example, the front end of the software interface allows easy integration of our system with existing software projects. For instance, this layer can cooperate with the already mentioned OpenCV library [10]. The handle/body and proxy design patterns allow separation of the user interface from the implementation of the IP algorithms. A great degree of ﬂexibility was achieved thanks to this strategy: implementation can be seamlessly changed from software to hardware if the latter is available in the system. Due to space limitations we do not provide implementation details, focusing rather on the architecture and design strategies that can be followed in similar projects. However, details can be found in the provided software platform of our system [4][3][9].

2

Functionality of the System

The basic assumptions on the functionality of the system are as follows [5]: 1. Operations of the library are deﬁned in terms of the processing objects: (a) A hardware processing object a module implementing an IP operation. (b) A software processing objects software component for a given operation. 2. The library allows many image operations to be performed in a hardware pipeline. The initial pipeline can be fed by a library user by constructing a composition of processing objects and run as a batch-job in hardware. 3. The input images allow multiple types of pixels. 4. Hardware processing modules can operate in two of the arithmetic modes: (a) Integer format in this representation all operations are performed only with an integer arithmetic.

Architecture of an Integrated Software-Hardware System

3

(b) Fixed point format in this representation the 16 bits per pixel is assumed for input and output images. However, the single 16 bits/pixel channels can be further joined to form an extended precision ﬁxed point representation. A detailed list of image operations of the system is provided in [5].

3

Architecture of the System

Fig. 1 depicts architecture of the library. It follows the handle-body pattern which allows decoupling of the interface deﬁnition from its implementation [7]. A handle constitutes an unchanged interface. However, there are two bodies. The ﬁrst one is a pure software implementation of all the functions. The second is a hardware layer implementing a sub-set of operations, depending on available hardware resources. Hardware is connected through a driver layer which implementation depends on the operating system (4). Such an architecture which separates a front end interface from implementation allows easy updates and modiﬁcations, as well as code reusability. This is important in an environment of fast changing hardware modules. Especially changes to the hardware can cause signiﬁcant delays in the system release. This is caused by much slower implementation of hardware components compared to the same functionality realized in software. The main problems are incurred by higher complexity of the parallel implementation, insuﬃcient hardware resources, lower level of the implementation languages, limited external libraries, much harder debugging means, limited human resources, etc. For instance in the presented system, at the beginning stage of design and implementation, only a hardware board with Virtex 2 by Xilinx was available.

Fig. 1. Architecture of the image processing library which follows the handle-body pattern. A handle constitutes a ﬁxed interface. There are two bodies: The ﬁrst one is a software implementation. The second is a hardware library.

4

B. Cyganek

However, during implementation new Virtex 4, then Virtex 5 devices started on the market. Finally, a new board with the latter devices and much larger and faster memory buﬀers was manufactured. This made possible implementation of advanced processing modules for real-time warping of the HDTV format video streams (5). 3.1

Representation of Pixels

A pixel can be characterized by the following parameters [8]: 1. A scalar (monochrome) or multidimensional value (color) coeﬃcient; 2. A position in an image. The ﬁrst attribute associates a group of allowable operations. It can be set of integers, positive integers, real or complex values, etc., but also a vector of integers or real values. In practice, however, pixel value must be modeled by a best representation available on a given computer platform. The second parameter, a position of a pixel, is a vector which dimension follows dimension of an image. Naturally, in still images this is 2D and 3D in movies. However, there is no obstacle in deﬁning higher dimensional images. Because images are digitized it is a common practice to locate pixels on an integer grid thus coordinate values of position vectors belong to the set of integers. However, there are image processing techniques that operate on fractional displacements or pixels positions. In such cases pixel positions need not be integer values anymore and should belong to the real domain. In practical computer realization, the chosen solution for a pixel model should be able to ﬁt diﬀerent types of pixels for data representations encountered in systems, such as: monochrome pixels (8 bits), color pixels (3x8 bits), but also ﬁxed and ﬂoating point pixels, etc. On the other hand, choice of a data structure representing pixels inﬂuences robustness of the system. Based on many experiments and code proﬁling, the following data models are proposed to represent pixels in the library: 1. For scalar pixels the built in type (e.g. unsigned char, int, long, etc.). 2. For vector valued pixels the template class parameterized by single coeﬃcient type and number of coeﬃcients for a pixel (the MMultiPixelFor class [4]). The second representation is a super-set model of a pixel. It supports also basic semantics assumed for every type of a pixel (such as initialization, simple arithmetic, etc.). In the MMultiPixelFor class pixel values are stored in the C++ arrays [4]. This showed to be faster or equal in performance compared with simple structure with separate data member for each pixel (such as struct { char a; char b; char c; };). At the same time, an array allows uniform algorithmic access when extending the class to higher dimensions. There are basically three main groups of function-members in the MMultiPixelFor [4]: 1. Construction and assignment. 2. Arithmetic operations among pixel objects. 3. Arithmetic operations among pixel object and scalar values. The above deﬁne basic functionality of the pixel objects. This can be interpreted also in terms of vector operations [4].

Architecture of an Integrated Software-Hardware System

3.2

5

Representation of Image Objects

Due to system requirements it was necessary to deﬁne a set of special objects for representation of images both for software and also for hardware representations. In the latter case the crucial is a compact memory location that allows eﬃcient DMA transfers. The heart of the hierarchy presented in Fig. 2 is the base template class TImageFor. It deﬁnes the interface for all images used in the system. The class is parameterized by the pixel type. Scalar pixels can be represented by built-in types or by system deﬁned classes, as shown in Fig. 2. For special pixels separate classes have to be deﬁned. These is the already discussed MMultiPixelFor, as well as the FixedFor class which models ﬁxed-point arithmetic in C++ [3][5]. From the TImageFor three classes are derived: 1. The TDanglingImageFor which models a special images with vectors pixels, which possibly are of diﬀerent length. 2. The TProxyImageFor which implements the proxy design pattern. Objects of this class are used to model region-of-interest in images. Those objects allow to treat part of an image as a separate image with its own local coordinates system (a local discrete manifold), although there is only one set of pixels. 3. The TMultiChannelImageFor for multi-channel images, i.e. images which are composed of number of simpler type image (non-interlaced images). There are three distinct groups of members in the TImageFor [4]: 1. Constructors; 2. Pixel access routines (GetPixel, SetPixel, and their reference based counterparts); 3. A set of overloaded arithmetical and logical operations on images. Multi-channel pixels are elements of the multi-channel images. These, in turn, are examples of a more general group of template-images which are images with pixels also being images. However, the multi-channel images are not just any collection of channels. The important feature is that all images must be of the same size. Moreover, the elements located at the same position in each channel

Fig. 2. The class hierarchy for representation of image objects

6

B. Cyganek

compose a multi-channel pixel which can be accessed at once. Each multi-channel pixel can be accessed providing its three coordinates (details in [4]). 3.3

Representation of Image Operations

Image operations are also objects. They come from the class hierarchy which was presented and discussed in [4][3]. TImageOperation is a pure virtual base class for all of them. Other operations are derived from this class. The general ideas for the image operations can be summarized as follows: 1. The base class TImageOperation is a pure virtual class, so all operators should be objects of the derived classes. 2. The base TImageOperation accepts and stores references to the two external objects: (a) Thread security object that in run-time controls access to the processing resources; (b) A callback object which is called upon completion of an operation. Each operation should be decorated with operation begin and operation end sequences. To help this action an automatic variable of the inner MImageOperationRetinue class has to be deﬁned at the beginning of each operator() in a derived class. Then, when this variable gets out of a scope of the operator(), its destructor is automatically invoked which, in turn, calls operator end() member. All parameters passed to image operators are treated as images. So, an image is a more ample notion than a classical ”visible” image. For instance an image can store in its pixels a value of a just found maximum pixel in another image, as well as x and y coordinates of that pixel. This is analogous to a matrixprocessing context where each entity is treated as a matrix. The core of the library is composed of the following classes that model operations on images [4][3]: 1. The TImageOperation class - its responsibility is to deﬁne a common interface for all operations. The most important part of its interface consists of the pure virtual overloaded functional operator virtual void * operator()( void ) = 0; This is a common operator for all image operations. For the virtual functions it is required that its declaration is the same throughout the whole hierarchy. Therefore all the necessary parameters of speciﬁc operations are supplied to its constructor. This is a common strategy assumed in this library. 2. The TUnaryImageOperationFor class - its responsibility is to compose a branch of speciﬁc unary image operations, i.e. such operations that need only one image as its input. 3. The TBinaryImageOperationFor class - its responsibility is to start a branch of speciﬁc binary image operations, i.e. operations that take two images of the same size as its input. 4. The TImageTemplateOperationFor class - a specialization of the TImageOperation, its second argument is assumed to be an image template.

Architecture of an Integrated Software-Hardware System

7

Fig. 3. Steps of an operation. There are three stages of execution: a preamble which consists of the resources acquisition, the main image operation, and the operation ﬁnishing sequence which consists of resource release and callback notiﬁcation.

5. The TComposedImageOperationFor class - this class creates a speciﬁc composite, i.e. an operation that is built up from the other image operations, i.e. those that are derived from the base TImageOperation class. This multi parameter template technique allows for ﬂexible creation of speciﬁc operations with strict type checking of its input arguments. This way, an operation deﬁned for a given type of input images constitutes a diﬀerent type than the same operation but instantiated for diﬀerent input images. The other classes in the hierarchy are speciﬁc derivatives of the already described base classes. For example, the FormatConvert OperationFor class deﬁnes objects for image format conversion [4][3]. Fig. 3 presents a ﬂow chart with steps of execution of each image operation: 1. An operation preamble which consists of acquisition of processing resources; 2. The main image operation; 3. The ﬁnishing sequence - resource release and callback notiﬁcation. Fig. 4 depicts a scenario of compound image operations. Operations are provided with many input and output images which can be of diﬀerent size and diﬀerent type of pixels. An image can constitute an input for some operations and output for the other. Each operation has associated number of input images and a single output image, however. The operations can be further grouped

8

B. Cyganek

Fig. 4. A diagram of compound image operations. Operations accept many images and are grouped in certain compositions. Each operation has associated number of input images and a single output image. An image can be an input for some operations and output for the other. Order of execution is set by position of that operation in enclosing composition object. Some operations can launch callback notiﬁcation upon completion. Each operation can be supplied with an object responsible for resource acquisition.

in certain compositions. Order of execution is determined by position of that operation in enclosing composition object. The composition objects can be recursively composed in clusters, and so on. As alluded to previously, the operations can launch callback notiﬁcation upon completion. Each operation can be supplied with resource access object. For this reason the automatic variable of class TImageOperation::MImageOperationRetinue should be deﬁned in each function operator(). An important feature of the presented mechanism is that the compound image operations are constructed recursively by building simple operations and adding them to composite objects. The latter can be also grouped in bigger composites, since composites are image operations by themselves because they are also derived from the TImageOperation base, and so on. The resource acquisition and access can pose a serious problem in a multithreaded environment. Each thread should check if a resource it tries to access is available and if not it should be blocked waiting for a notiﬁcation signal. The presented system is prepared for such situations: each operation at its construction time can be supplied with an external object, of a class TThreadSecurity or derived, that controls an access to the computer resources (such as the

Architecture of an Integrated Software-Hardware System

9

hardware devices, etc.). Each operation has been endowed with a special calling sequence - see Fig. 3. Before the main computation part commences, the resource acquisition preamble is called. Actually this is done by action delegation to the resource control object (supplied in a constructor). If such a resource is not available at a time, the whole thread is blocked. Thus, a run-time system reconﬁguration was achieved. However, the actual action is entirely up to the external object. After the computations the resource release procedure is invoked.

4

Integration of System Components

Communication with the hardware components can be achieved in number of communication channels. In the system the highest data transmission is achieved through the PCI Express link. However, the board can be also communicated through the Internet. The video stream can be transferred directly through the SDI link. In this conﬁguration the PC computer with software front end plays a role of a control unit. However, communication of the higher layers of software with hardware requires an interface and functionality of the device driver which is speciﬁc to an operating system of the host computer. Our main target is Windows OS. However, the speciﬁcation presented in this section can be used in other systems as well. The Windows OS provides a standard interface to the system resources through its Hardware Abstraction Layers (HAL). All actions of hardware access are controlled by HAL. This includes device addressing, registers access, interrupt management, Direct Memory Access (DMA) transfers, etc. Especially the latter feature is important for robustness of the whole system. For this purpose the TImageFor class has been endowed with the memory allocation policy which allows operation of a specially designed memory allocator. This is designed to facilitate DMA transfers. The used scatter/gather features supported for the DMA devices (i.e. DMA chaining) allow devices to be programmed with multiple pairs of base addresses and lengths simultaneously. In this way, even a logically fragmented memory could be processed with minimal performance penalty. There are three possible architectures of I/O data exchange: 1. Programmed I/O devices - they move data under program control between the device and host memory. 2. Busmaster DMA devices - they autonomously transfer data between themselves and host memory using intelligent built into the device. 3. System DMA devices (obsolete) - they utilize a system DMA controller to transfer data between themselves and host memory. In the system the busmaster DMA access is assumed. In this case the appropriate driver supports a device (This speciﬁcation concerns Windows OS): 1. The logical address of data to be transferred (DMA transfers in Windows are always performed by using logical addresses); 2. Length of data;

10

B. Cyganek

3. Direction of the transfer. Data are transferred without help of the host CPU thus leaving microprocessor for other tasks. Further, there are two types of DMA devices: 4. The Packet-Based DMA devices (most common) data transfers are to/from diﬀerent logical addresses; The device interrupts when transfer is ready. 5. The Common-Buﬀer devices - the same buﬀer is used for all transfer operations. The driver and the device have a common buﬀer in host memory. In the presented system the packet-based transmission is assumed. However, this can be changed if necessary. Prior to the DMA data transfer some registers have to be set. This can be done for example by setting data address and data registers (see Fig. 5b). The calling sequence is deﬁned as follows: 1. Write logical address of an image from the operating memory (i.e. in a process address space) to the Data Address Register (DAR). 2. Write physical address of the frame buﬀer on the board to the DAR. 3. Write data length in bytes to the Data Length Register. 4. Set direction bit in the Memory Frame Control Register (actually it is one control register and the hardware object (HO) it refers to is set via the Hardware Object Address Register). 5. Initiate data transfer through the Control Register. 6. Read status of the data transfer from the Memory Frame Status Register. If necessary other transfer scenarios are also possible depending on a system platform, however. DMA data transfers from the systems operational memory (DRAM) to the image VSRAM memory frames on the board depicts Fig. 5a. Each separate HO that resides in the FPGA (i.e. logic for image operations) can have number of input/output control and status registers. Precise deﬁnition will be diﬀerent for each object. However, it is possible to deﬁne a common functionality and set of bits. Fig. 5b depicts the registers of the hardware board. All registers are accessed by a common register interface. The particular hardware object is addressed by setting its internal address in the HO Address Register

a)

b)

Fig. 5. DMA data transfers from the systems operational memory (DRAM) to the image VSRAM memory frames on the board (a). Register architecture of the PHB (b).

Architecture of an Integrated Software-Hardware System

11

(red in Fig. 5b). The common register interface is visible to an external devices and software. This is a mirror of a set of these registers in each of the HOs. The currently addressed HO is set in only one register from the interface - the HO Address Register. Finally, it is interesting to observe that HO are not only image processing objects but also board housekeeping objects, such as the frames memory controller or programming module.

5

Experimental Platform and Results

Fig. 6 presents a cooperation diagram of the two hardware boards and controlling software front end. The ﬁrst board is endowed with the four Virtex 2 devices, whereas the board no. 2 with two Virtex 5. Both platforms are controlled by software layers of our system. The video stream can be processed independently by hardware. Communication is achieved through PCI Ex and through the Internet link. Thanks to great ﬂexibility of the design and high computational power of the system it was possible to design a real-time processing system for multi-projector displaying system for digital cinema. This required competing a very stringent real-time HDTV speciﬁcation. The warping was implemented in hardware and operates with a pre-loaded look-up table. This requires a very fast memory buﬀer of a size of a single HDTV frame, i.e. 1k×2k of 64 bits for warping parameters. Its parameters, however, are computed oﬀ-line by a software front end and loaded into the memory on the board. Fig. 7 depicts some experimental results of the real-time HDTV non-linear warping system performed by our system. Apart from the warping application all operations from the functional speciﬁcation were implemented in hardware. Thanks to this the basic image operations, such as convolution or morphological operations can be performed in real-time. However, contrary to the software counterparts, these functions

Fig. 6. Cooperation diagram (ﬂexibility diagram) of the hardware board no. 1 (Virtex 2) and no. 2 (Virtex 5) and software monitor. Both platforms are controlled by the software front end. The HDTV video stream is processed in real-time and independently by hardware. Communication is achieved through Internet and PCI Ex.

12

B. Cyganek

Fig. 7. Experimental results of the real-time HDTV non-linear warping system

has some limitations such as allowable size of the ﬁlter masks and only ﬁxed point arithmetics. The best functions can be chosen by the system. If a hardware implementation is available for a request then it is chosen due to much faster execution (in practice up to few orders of magnitude). However, in all other cases the software counterparts are used. The available (and registered) hardware functions can be checked by a caller.

6

Conclusions

The paper presents architecture, design process, and experience gained during implementation of the hybrid software/hardware system for image processing. One of the goals of this project was to develop a fast and versatile IP system in a form of a universal library that can be linked to other projects. The system was designed to achieve ﬂexibility in implementation that allows easy changes of the hardware and extensions to the implementation, with unchanged interface at the same time. These objectives were met by the modular architecture and separation of interfaces from implementations. The parallel algorithms ported into hardware allowed speed-up factors from 10 to over 500 times compared to software. However, these has some limitations in calling format, as well as size and precision data. Thus, both platforms, software and hardware, are complementary to each other. The main lessons learned from design and implementation of the system can be summarized as follows: 1. Systems that facilitate software and hardware interactions are not widely discussed. Adding the hardware factor early in the design process is a must for a success. 2. There are signiﬁcant diﬀerences in functions implementation in software (serial) and hardware (parallel). Generally there are no methods of automatic conversion from serial to parallel processing. 3. The highly opened and object-oriented approach allows easy extensions, code reusability, and reliability of the system. 4. Fixed-point arithmetic should be used if higher precision is required for data with relatively low dynamic range. This data type (there is not a built-in type in C++) allows precise computations with smaller memory requirements. The experiments showed that in IP applications that require signiﬁcant computations per pixel and are not I/O bounded, the hardware can spectacularly

Architecture of an Integrated Software-Hardware System

13

outperform software implementation. They showed also that data exchange between processes and operating memory leads to growing I/O bound for large images (e.g. HDTV). However, it should be remembered that time and costs of hardware development can be signiﬁcant and sometimes can even pose an overkill to a system if not foreseen in an early stage of a design. With this paper we provide the software layers of the presented system [9].

Acknowlegement This work was supported from the Polish funds for the scientiﬁc research in 2008.

References 1. Alexandrescu, A.: Modern C++ Design. Generic Programming and Design Patterns Applied. Addison-Wesley, Reading (2001) 2. Crookes, D., Benkrid, K., Bouridane, A., Alotaibi, K., Benkrid, A.: High Level Programming for Real Time FPGA Based Image Processing. Tech. Report, School of Computer Science, The Queen’s University of Belfast, BT7 1NN, UK (2003) 3. Cyganek, B.: Introduction and Tutorial to the Image Library. Technical Report (2008), http://www.wiley.com/go/cyganek3dcomputer 4. Cyganek, B., Siebert, J.P.: An Introduction to 3D Computer Vision Techniques and Algorithms. Wiley, Chichester (2009) 5. Cyganek, B.: Hardware-Software System for Acceleration of Image Processing Operations. Machine Graphics & Vision 15(3/4), 329–337 (2006) 6. Draper, B.A., Beveridge, R., Bhm, A.P.W., Ross, C., Chawathe, M.: Accelerated Image Processing on FPGAs. IEEE Tr. on Image Proc. 12(12), 1543–1551 (2003) 7. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns. Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading (1995) 8. Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Prentice-Hall, Englewood Cliﬀs (2002) 9. http://www.wiley.com/go/cyganek3dcomputer (2008) 10. INTEL: Intel Open Source Computer Vision Library (2004), http://www.sourceforge.net 11. Ritter, G.: Image Algebra (1994), ftp://ftp.cise.ufl.edu/pub/src/ia/documents 12. http://www.research.microsoft.com/research/vision/ 13. http://www.xilinx.com

A Fast Logical-Morphological Method to Segment Scratch - Type Objects Andreas Kuleschow, Christian M¨ unzenmayer, and Klaus Spinnler Fraunhofer Institut f¨ ur Integrierte Schaltungen, Am Wolfsmantel 33, D-91058 Erlangen {andreas.kuleschow, christian.muenzenmayer, klaus.spinnler}@iis.fraunhofer.de

Abstract. In spite of the fast progress of computer technologies there are many tasks, which need accelerated software methods to obtain the results in real time. We propose one of such accelerated logicalmorphological methods to detect scratch-type objects on noisy surfaces. The method is based on the principles of human vision and includes an adaptive multithresholding and logical-morphological operations for the fusion of the object’s fragments. Keywords: image processing, segmentation, morphological operations, adaptive thresholding, surface inspection.

1

Introduction and Related Work

A search for scratch-type defects on surfaces is a common task in industrial image processing. Scratch-type objects (scratches, cracks and so on) have frequently no solid appearance and disintegrate into several fragments with a diﬀerent width and contrast. In order to detect those objects and to calculate their real dimensions it is necessary to ﬁnd and collect their fragments. Usually this task can be performed after binarization with the well-known morphological closing operation [1]. But on noisy surfaces such methods can be too slow and imprecise due to many false detections of the segmentation. Another common problem of industrial image processing is nonuniform lightning or nonuniform features of the surface itself. The diﬀerent methods are proposed to avoid the eﬀect of those factors by segmentation; one of the fast and mighty methods is the adaptive thresholding. The threshold for segmentation by adaptive thresholding is calculated at the point of interest from the local features of the surrounding image [2]. A new method for the adaptive image segmentation was proposed in [3]. This method takes into account the principles of human vision; the most important rules are: – The outline of an object is recognised by the brightness diﬀerence, and not by the gradient of brightness only. – This diﬀerence has to be normalized by the brightness of the surrounding. – The pixels have to be sorted by a logical operation; all pixels inside the closed boundary belong to the same object independent of their brightness. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 14–23, 2009. c Springer-Verlag Berlin Heidelberg 2009

A Fast Logical-Morphological Method to Segment Scratch - Type Objects

15

In accordance with this rules a very easy procedure of segmentation, which includes two steps, was proposed. The initial segmentation, or step one of the procedure, runs very fast and includes the following calculations: – Averaging of brightness of image in a small neighbourhood (blurring); – Comparison of the current brightness of image with averaged value; marking the pixels, where the normalized diﬀerence of current and averaged brightness exceeds a pre-selected threshold; – Collection of all marked pixels into blobs (by four - neighbourhood); The ﬁrst two operations form an initial adaptive thresholding and can be described as: ¯ y) T (x, y) = C I(x,

(1)

where T (x, y) is the adaptive threshold, C is a pre-deﬁned contrast coeﬃcient ¯ y) is the averaged image brightness in the surrounding: and I(x,

¯ y) = I(x,

1 NMN

m=M

n=N

I(x − m, y − n)

(2)

m=−M n=−N

and NMN is the area of the averaging ﬁlter. Similar rules based on human brightness perception and foveal adaptation were considered in [4], but the procedure proposed in this work seems to be more diﬃcult and time-consuming as the step one of our method. The results of the segmentation by method [4] and by step one of [3] are very similar. The step two of method [3] is based on the local histogram - analysis and has to trace and to complete the boundaries of large objects, but the step one is usually enough to ﬁnd small objects and scratches. A lot of other methods to ﬁnd elongated objects were developed for vessel segmentation in retinal images [5,6,7]. But the tasks in industrial image processing and in retinal image processing are diﬀerent; most important feature of industrial image processing is the processing speed, whereas the most important feature of medical image processing is its accuracy. The methods developed in [5,6,7] are very complex and too slow for industrial applications. For example, the average computation time is given in [5]; for the image size 605 x 700 pixels this method needs from 8 up to 36 seconds per image, depending on the selected thresholding step (Pentium III by 600 MHz). Our procedure [3] step one needs for initial segmentation of image 700 x 900 pixels 0.015 second (Pentium 4 at 2.8 GHz). Consequently we used the procedure described in [3] for the initial thresholding of images. For the collection of object fragments we use the well known morphological closing procedure on the one hand and the new fast logical - morphological procedure of object’s growth on the other hand.

16

2

A. Kuleschow, C. M¨ unzenmayer, and K. Spinnler

Scratch Segmentation Methods

As a common procedure for comparison with the developed method we use the following 5 - steps technique (see Fig. 1a). There are many libraries, which provide accelerated calculations for diﬀerent morphological operations; but our method can be accelerated too. On the other hand, the usual closing operation was not completely satisfactory for us. We achieved best connection of fragments by a modiﬁed erosion operation, where the pixels with only one contact to free space do not have to be deleted. Also we implemented both logical-morphological dilation and special closing with the modiﬁed erosion step. Retrieval and classiﬁcation of connected marked pixels is performed in both cases by a special function (blobﬁnder), which calculates the bounding box of every blob and its parameters. This function checks the processed image for the initial marked pixels from the upper left corner to the right bottom one. If the ﬁrst marked pixel is found, connected pixels after 4 - neighbourhoods will be checked and all connected marked pixels will be located and added to the blob. This checking procedure repeats for every encountered connected pixel until no new connected marked pixel can be found. The located blob has to be added to the blob list and all pixels of this blob have to be relabeled during the checking procedure corresponding to the number of the blob in the blob list. In this manner we can discriminate pixels, which are situated inside of overlapping bounding boxes and belonging to diﬀerent blobs.

(a)

(b)

Fig. 1. Block diagram of common (a) and proposed (b) methods for scratch type objects segmentation

A Fast Logical-Morphological Method to Segment Scratch - Type Objects

17

We continue the search procedure until all pixels with initial marking will be encountered and all blobs will be completed. While doing so we can calculate diﬀerent parameters for every blob, such as the bounding box, number of marked pixel, contrast to surrounding and so on. In common case after the blobs retrieval we apply the step 2 of our method [3] to trace the boundaries of large objects, which can not be marked completely, to complete them. But in the case of in fact one-dimensional objects such as cracks and scratches this step is unnecessary. Object checking itself is a simply procedure, which veriﬁes if the size of the object exceeds a pre-deﬁned threshold. We used a size threshold of 140 pixels to deﬁne signiﬁcant cracks. But we inserted an additional procedure before the last step, namely we carried out the additional geometrical aggregation (see below) of detected fragments before the size checking.

3

Logical-Morphological Dilation (LMD)

The usual dilation regards no features of surrounding pixels. The only feature under consideration is the geometrical contact to the marked pixel. On noisy surfaces with lots of objects and short distances between them this can lead to errors. Physically diﬀerent objects may be connected. By high numbers of closing steps we can even connect the whole surface. For example in Fig. 2 a fragment of

(a)

(b)

(c) (d) Fig. 2. (a) Original image; (b) initial segmentation by the adaptive thresholding; (c) ﬁnal segmentation after closing with the modiﬁed erosion; (d) ﬁnal segmentation after standard closing from IPL library. In c) and d) all detected objects more than 40 pixel size are shown.

18

A. Kuleschow, C. M¨ unzenmayer, and K. Spinnler

Fig. 3. Final segmentation by the logical-morphological dilation for the same original image. All detected objects (1 object) more than 40 pixel size are shown.

the original image(a), its initial thresholdig by a contrast coeﬃcient of C = 0.8 (size of averaging ﬁlter is 11 x 11 pixels, b), and two images after diﬀerent closing procedures (c,d) are shown. The main idea of the logical-morphological dilation (LMD) is to join the fragments of an object, which are joined for a human being. If the object is decomposed into any fragments by a certain (strong) threshold, it can be segmented as a connected one by any other (weak) threshold. But if we apply this weak threshold to the whole area of the image, we obtain a lot of other objects (artefacts, errors). Therefore we have to ﬁnd and classify (elongated or not) fragments of objects by the strong threshold and try to dilate elongated objects with the information given by the weak threshold. In fact we temporally dilated the initial elongated objects with pixels of weak threshold after the 8 - neighbourhood. If the outline of this dilated object reaches another ”strong” marked object, this object has to be added to the initial one. The ”weak” marking itself is not shown. That way we can connect fragments, which are separated by a large distance and can not be connected by the common closing procedure; on the other hand we do not connect two neighbouring fragments, if there is no ”natural” connection between them. In Fig. 3 the result of such connection for the same original image is shown; ”strong” contrast quotient for the initial thresholding is 0.8 as above and for the second threshold a contrast coeﬃcient of 0.9 is used. The size of the averaging ﬁlter is the same.

4

Results of the Fine Cracks Detection and Fragments Geometrical Aggregation

Not all scratches and cracks are as broad and evident as in Fig. 2. In many cases the object is really divided into separate fragments as in the original image Fig. 4a, and we obtain fragments independent from the method of image processing. In Fig. 4b the result of an image processed by the logical-morphological dilation procedure is shown. In Fig. 5a,b the processing results of the same original

A Fast Logical-Morphological Method to Segment Scratch - Type Objects

19

(a)

(b) Fig. 4. (a) Original image, (b) Crack, as it is detected by logical-morphological dilation

(a)

(b) Fig. 5. The same original image, processed (a) by closing with modiﬁed erosion and (b) by standard closing from IPL library

image by the closing with the modiﬁed erosion (a) and the standard closing procedure (b) are shown. In both Fig. 4b and Fig. 5a,b only big fragments (more than 40 pixel size) are displayed.

20

A. Kuleschow, C. M¨ unzenmayer, and K. Spinnler

Table 1. Number of objects, detected with the logical-morphological dilation (LMD) and diﬀerent closing procedures Method LMD Modiﬁed Closing Standard Closing

Number of detected objects Number of crack’s fragments 125 (27) 4 842 4 1230 17

The LMD method analyses the blobs before the dilation, therefore we can sort the objects and reduce their number considerably. Anyway we process the elongated objects only; in addition to that and without aﬀecting the results of processing we can ignore the small objects up to N pixels size, where N depends on the image and the task. Here we set N to 4 pixels. The number of detected objects and number of the crack fragments in the original image Fig. 4a are shown in Table 1. The number of detected objects in the original image by LMD method was 125, where only 27 elongated objects were detected. Considerably more objects were detected by modiﬁed closing and a really huge number of objects was detected by the standard closing procedure. Here the number of crack’s fragments was enormous too. But the area displayed in Fig. 4a is really only 20% of the whole surface of the workpiece. Consequently a signiﬁcant part of the processing time by the standard closing procedure was spent to blobs retrieval and sorting. Anyway human beings perceive this physically fragmented crack as a single object. To reproduce this human perception feature we applied a geometrical aggregation, i.e. collected few fragments, which are located as a prolongation of each other. We applied an easy method, which operated with the direction of cracks and the gap between marked pixels of both fragments. It is obvious, that the more little fragments are detected, the more diﬃculties we have by geometrical aggregation, therefore the standard closing has a deﬁnite disadvantage here. To compensate the acceleration for the new method due the suppression of the small objects, we suppressed the small objects after the modiﬁed closing and after the standard closing too. The suppression threshold was set the same, namely 4 pixels.

5

Comparison of the Processing Speed and Eﬃciency for the Diﬀerent Closing Procedures and the Proposed Logical-Morphological Dilation

To compare the two techniques we used 20 images of real work pieces with complex noisy surfaces, among them 10 images without defects (cracks) and 10 images of cracked surfaces. Those work pieces were selected from about 350 production samples and represented all possible surface types and cracks features. Any examples of those samples are shown in Fig. 6. The whole surface to be inspected had the size 700 x 900 pixels by the resolution about 5 micrometer per pixel. The initial segmentation was completed with

A Fast Logical-Morphological Method to Segment Scratch - Type Objects

(a)

(b)

(c)

(d)

(e)

(f)

21

Fig. 6. Diﬀerent examples of inspected surfaces: a - c without defects, d - f with diﬀerent cracks

the contrast coeﬃcient of 0.78 for both closing procedures and with contrasts 0.78 and 0.89 for the LMD procedure. Averaging parameters were the same for all procedures. The results of the comparison are shown in Table 2. The averaged processing time is considerably shorter by the proposed new LMD method. All cracks were detected by both the LMD method and modiﬁed closing procedure, but the modiﬁed closing procedure detected too many ”cracks” on the good work pieces. ”Other artefacts” means here that additionally artefacts were detected on the cracked surfaces, what does not change the estimation of the work piece, but is not correct. By the contrast coeﬃcient of 0.75 for the initial segmentation we obtained few false negative results by modiﬁed closing too, but the LMD method still got the right results. We tested the standard closing procedure with IPL library as well as modiﬁed one. The averaged processing time exceeds considerably the averaged processing time for LMD, and results of cracks detection are not as good as by modiﬁed closing. Many cracks were strongly fragmented, and already by the contrast of 0.78 we obtained one false negative result. Table 2. Comparison between the logical-morphological dilation (LDM) and closing with modiﬁed erosion step. Processing time is given in milliseconds. Single processor, Pentium 4 at 2.8 GHz. Method

Averaged False processing time positive LMD 64.8 0 Modiﬁed Closing 276.9 6 Standard Closing 215.1 6

False negative 0 0 1

Number of other artefacts 0 3 0

22

A. Kuleschow, C. M¨ unzenmayer, and K. Spinnler

Table 3. Processing time for the single steps of diﬀerent procedures (milliseconds). For the LMD-based segmentation method the object growth step is included in the blobs retrieval and analysis one. By guess the object growth per LMD consumed averaged about 10% of the step processing time. Operation

LMD-based method Initial thresholding 31.3 Growth (closing) Blobs retrieval and analysis 32.0 Aggregation 1.5 Total 64.8

Modified Closing-based 16.5 121.3 74.3 64.8 276.9

Standard Closing-based 18.1 16.6 115.7 64.7 215.1

The processing time for single steps of those methods are shown more closely in Table 3. Initial thresholding for the LMD-based method consumed signiﬁcantly more time as for two other methods because of more calculations for two thresholds. The initial thresholding was accelerated partially (averaging step) by the use of the IPL library for all methods. The processing time for the initial thresholding by both closing methods actually has to be the same. The observed diﬀerence between those two values indicates the statistical inaccuracy by the calculations due to the interference with the operating system (Windows 2000). By the blobs retrieval and analysis and especially by the geometrical aggregation we can see advantages of the proposed method through the considerably reduced number of detected objects and through the wilful exclusion of nonelongated objects from the objects list.

6

Conclusion

A new method for fast detection of scratch type objects on the noisy surfaces was proposed, implemented and tested. The quality of crack detection on heavily disturbed surfaces and the robustness of the new method are better then by diﬀerent closing procedures. The processing speed by LMD is considerably higher than for both the usual closing and the closing with the modiﬁed erosion. We have applied this new method for current tasks in the ﬁeld of industrial image processing.

References 1. Castleman, K.R.: Digital image processing, p. 473. Prentice-Hall, Englewood Cliﬀs (1996) 2. Chow, C.K., Kaneko, T.: Automatic Boundary Detection of the Left - Ventricle from Cineangiograms. Comput. Biomed. Res. 5, 388–410 (1972) 3. Kuleschow, A., Spinnler, K.: New Methods for Segmentation of Images Considering the Human Vision Principles. In: Proc. ICCVG 2004, Warsaw, pp. 1037–1042 (2004)

A Fast Logical-Morphological Method to Segment Scratch - Type Objects

23

4. Heucke, L., Knaak, M., Orglmeister, R.: A New Image Segmentation Method Based on Human Brightness Perception and Foveal Adaptation. IEEE Signal Processing Letter 7(6), 129–131 (2000) 5. Jiang, X., Mojon, D.: Adaptive Local Thresholding by Veriﬁcation - Based Multithreshold Probing with Application to Vessel Detection in Retinal Images. IEEE Trans. on Pattern Analysis and Machine Intelligence 25(1), 131–137 (2003) 6. Ayala, G., Leon, T., Zapater, V.: Diﬀerent Averages of Fuzzy Set With an Application to Vessel Segmentation. IEEE Trans. On Fuzzy Systems 13(3), 384–393 (2005) 7. Lam, B.S.Y., Yan, H.: A Novel Vessel Segmentation Algorithm for Pathological Retina Images Based on the Divergence of Vector Fields. IEEE Trans. On Medical Imaging 27(2), 237–246 (2008)

Spatio–Temporal Track–Before–Detect Algorithm for Interlaced and Progressive Scan Imaging Sensors Przemyslaw Mazurek Szczecin University of Technology Chair of Signal Processing and Multimedia Engineering 26. Kwietnia 10, 71126 Szczecin, Poland [email protected] http://www.media.ps.pl

Abstract. This paper consider Spatio–Temporal Track–Before–Detect algorithm for tracking of moving objects using interlaced and progressive scan imaging sensors. Using the same image size and pixels’ readout rate interlaced and progressive scan approaches are compared. Especially the most important case of dim and pixel size target is analyzed due to spatial sampling theory violation for interlaced sensors applications. Keywords: Track–Before–Detect, Imaging Sensors, Image Processing, Interlaced Sensors.

1

Introduction

The Track–Before–Detect (TBD) algorithms are state of the art tracking algorithms especially used for kinematics parameters estimation of dim targets [1,2,3,5,6]. The TBD approach give ability of detection target for signals that are at noise ﬂoor level or even below (SN R < 1) what is especially important for ’stealth’ targets, like warplanes and submarines tracking. This approach is diﬀerent from classical tracking algorithms based on following processing phases: detection, tracking and assignment, because two ﬁrst phases are swapped. There are many methods used for improving detection and sensors replacement by more recent designs is most popular one for contemporary available applications. Progress in development of better signal processing algorithm is also important for update of available system. Another method for signal measurement improvement is new whole system design. The TBD algorithms can be used for new systems and as an update of today used if suﬃcient computation power is available. Contemporary high– performance computing devices are available like multi–core processors, specialized integrated circuits, reprogrammable devices (e.g. FPGA), DSP (Digital Signal Processors) and GPGPU (General–Purpose Graphics Processing Unit) processors suitable for real–time TBD processing. The TBD algorithm incorporates additional a–priori knowledge for improving overall tracking performance. Sensors parameters and target motion models are L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 24–32, 2009. c Springer-Verlag Berlin Heidelberg 2009

Spatio–Temporal TBD Algorithm for Interlaced and Progressive Sensors

25

very often available and gives ability of using constraints useful for reduction of computations or improving results. Such knowledge can be incorporated using e.g. measurement likelihood function [3] or sensor spatial model [2]. Some of the TBD algorithms uses Bayesian methodology or even directly uses Bayesian computational approach [3]. The TBD algorithms if they are well designed allows tracking of very small (dim) targets and Nyquist theorem virtually overrides using accumulative approach. For example sub–pixel position and motion can be estimated for sub–pixel size targets [4]. There are two kinds of two–dimensional (2D) sensors available today. The ﬁrst one sensors are non–interlaced (progressive) scan shown in ﬁgure 1 that gives measurement for all pixels at one time. The second one are interlaced scan sensors shown in ﬁgure 2 that give measurement for all odd or even sensor rows at one time [8] so sampling theorem is violated.

r r+1 r+2 r+3 r+4 r+5 r+6 r+7 Fig. 1. Non–interlaced sensor model

r r+1 r+2 r+3 r+4 r+5 r+6 r+7

r r+1 r+2 r+3 r+4 r+5 r+6 r+7

Fig. 2. Interlaced sensor model (odd and even time moment scans). Gray area are inactive pixels.

From tracking point–of–view it is very interesting question about non–interlaced and progressive sensors inﬂuence on the tracking results. Assuming two comparable sensors (equality of spatial resolution and pixel readout rate), one of them should be better for tracking application. Progressive sensor gives full frame scan at ﬁxed frame rate (e.g. 25 fps) and full resolution. Interlaced sensors work at ﬁxed double frame rate (e.g. 50 fps). For interlaced sensors only odd and even rows are available at some time moment

26

P. Mazurek

and if missing rows values are de–interlacing is necessary [9,10]. For peoples the interlaced scan is acceptable and gives high quality and much more dynamic video image. The TBD algorithms are designed for progressive scan data but interlaced sensors gives doubles frame rate what is interesting for improving tracking. If there is available double frame rate linearization of motion target is possible and Markov matrix is smaller what occurs for interlaced sensors. Unfortunately there are special target trajectories non–observable for interlaced sensors due loss spatial sampling. Monte Carlo approach should be used for tracking performance tests because motion model can be non–linear and there are a lot of nonlinear errors due to target size that is comparable to the pixel size.

2

Spatio–Temporal Track–Before–Detect Algorithm

There are a few groups of TBD algorithms [1,2,3]. In this paper is considered recurrent version of spatio–temporal TBD that is based on ﬁrst order fading memory ﬁlter. Motion model is described directly using state space decomposition (four–dimensional) to set of subspaces (two–dimensional). Every subspace corresponds to speciﬁc motion vector. This TBD algorithm can be described using following pseudoalgorithm: Start // Initial Likelihood Ratio: P (k = 0, s) = 0

(1)

qk (s|sk−1 )P (k − 1, sk−1 )dsk−1

(2)

For k ≥ 1 and s ∈ S // Motion Update:

−

P (k, s) =

S

// Information Update: P (k, s) = αP − (k, s) + (1 − α)Xk (k) EndFor Stop where: S – state space: 2D position and motion vectors V x, V y, s – state (spatial and velocity components), k – step number or time moment, α – smoothing coeﬃcient α ∈ (0, 1), Xk – measurements (input image), P (k, s) – estimated value of targets, P − (k, s) – predicted value of targets, qk (s|sk−1 ) – state transitions (Markov matrix).

(3)

Spatio–Temporal TBD Algorithm for Interlaced and Progressive Sensors

27

Estimated value has all hypotheses for overall state space. If there is single target the maximal value should be related to target so position and velocity can be obtained directly. In real systems the TBD algorithm is used for image preprocessing and additional conventional tracking algorithms are used for improving system performance.

3

Performance Comparison of Interlaced and Progressive Scan TBD

Performance comparisons for the TBD algorithm and diﬀerent sensor types are usually quite complicated because nonlinear eﬀect occurs and most convenient are Monte Carlo tests. In this paper selected results are presented for both sensors for intentionally selected objects trajectories due to expected behaviors of Markov matrix transitions. Assumed motion vectors in this algorithm are shown in ﬁgure 3 and are represented as an unidirectional graph.

Vy

Vx

Fig. 3. Assumed Markov matrix motion vectors used in experiments

Central looped vector is responsible for lack of movement of target. This graph is incorporated into Markov matrix that is quite large and in this paper is omitted. For real applications if there are available motion constraints they are incorporated into Markov matrix for best performance and computational cost reductions. Algorithm performance is best if motion vector of real target is ideally ﬁtted to the motion vectors described by Markov matrix and state space. In more realistic cases target motion vector is not well ﬁtted and it is approximated by transition between neighborhood state spaces only. This is result of discreet (grid based) state space implementation.

28

P. Mazurek

Fig. 4. Estimated value of target signal (TBD output) for progressive and interlaced sensor example

For constant velocity and ﬁxed direction oscillation are observed. Additionally this eﬀect is inducted by object size that is a few times smaller in comparison to the pixel size. An illustration of results for single–pass for non–interlaced and progressive scan sensors is show in ﬁgure 4. For both cases image is Gaussian noised so there are additional disturbances. The TBD processing for progressive sensor is smoother in comparison to the interlaced sensor and result are show in ﬁgure 4. For the second case high frequency impulses are a result of modulation introduced by interlaced acquisition of image. Spatial downsampling without ﬁltering give such eﬀect that is visible as a modulation of tracked target values. This frequency is for non–moving target equal to the Nyquist frequency what is shown in ﬁgure 11. If target is moving this frequency is little lower then Nyquist (or higher) but due to sampling only modulation below Nyquist frequency (as alias) is observed what is shown in ﬁgures 9 and 10. Additionally two time lower magnitude of output is for interlaced sensor. A few results for three motion vectors are shown on following images for both sensors. Frequency axis is normalized so Nyquist frequency correspond to the value 0.5. In all tests is generated image at double frame rate for fulﬁlling requirements of interlaced image generation and progressive scan resolution. High

Fig. 5. Maximal value of noise (TBD output) for progressive and interlaced sensor example for selected time period

Spatio–Temporal TBD Algorithm for Interlaced and Progressive Sensors

29

Fig. 6. Histogram and spectrum for estimated values obtained by TBD for progressive sensor (seven examples)

Fig. 7. Histogram and spectrum for estimated values obtained by TBD for interlaced sensor (seven examples)

α = 0.9 value is used for reduction of noise eﬀects from incoming new data and state space is signiﬁcantly blurred by Markov matrix using motion update formula. Such case is typical for real applications so noise level is quite low and modulation eﬀects are better visible also. Target has point size so only one pixel is active (higher value but still near to the noise level). Using point spread function optic and measurement eﬀect can be simulated but highest value is most important due to state space mentioned smoothing eﬀect. The ﬁrst one case is for target motion ﬁtted to the selected direction but not to the length of assumed TBD motion vector. Results are show in ﬁgures 6 and 7 and for interlaced sensor is well visible high peak below 0.5 normalized frequency responsible for modulation. Due to Gaussian noise histogram shapes are also Gaussian. Set of the peaks at low frequencies (near to the 0.0) is result of discreet motion vectors responsible for low–frequency modulations. The second one case is for target not ﬁtted to available TBD motion vectors show in ﬁgures 8 and 9. This example shows lack of ﬁtting to the motion vectors and has similar description. There much more peaks at low frequencies due to the ﬁtting problems to the assumed motion vectors.

30

P. Mazurek

Fig. 8. Histogram and spectrum for estimated values obtained by TBD for progressive sensor (seven examples)

The third one is for non–moving target (it is ideally ﬁtted to looped vector) and only noise inﬂuences are important for this case show in ﬁgures 10 and 11. For interlaced sensor there is 0.5 (Nyquist) modulation due to sampling. Because target is non–moving most of the spectrum is ﬁlled by signal noise. Noise ﬂoor for estimated values from TBD is diﬀerent for both sensors show in ﬁgure 6. Comparing noise ﬂoor and estimated values if target is present show in ﬁgure 5 it is visible that for interlaced sensors signal to noise ratio is about twice smaller what is not so good in comparison to the progressive sensor. For interlaced sensor double frame rate is used but it can not help for tested cases. Spectrum of estimated values for interlaced sensor has modulation eﬀects near to the Nyquist frequency that can be removed using digital signal ﬁltering, but they are not removed by ﬁltering properties of TBD (motion update component). Progressive and interlaced sensors have modulation eﬀects (low frequency) if target motion vector is not ﬁtted to the available motion vector set assumed in the TBD Markov matrix and state space. Using much more motion vectors needs appropriately more computation power.

Fig. 9. Histogram and spectrum for estimated values obtained by TBD for interlaced sensor (seven examples)

Spatio–Temporal TBD Algorithm for Interlaced and Progressive Sensors

31

Fig. 10. Histogram and spectrum for estimated values obtained by TBD for progressive sensor (seven examples)

Fig. 11. Histogram and spectrum for estimated values obtained by TBD for interlaced sensor (seven examples)

4

Conclusions

In this paper behaviors of progressive (non–interlaced) and interlaced sensors are compared from tracking perspective using Track–Before–Detect algorithm. It is shown using numerical experiments that progressive sensor is better (about two times) in comparison to the interlaced sensor working at double frame rate what is important for many applications and there are no modulation eﬀects. There are also bad trajectories for interlaced sensors what is well known fact and if they occurs tracking of target is not possible. If image is blurred there is possible to track them but corresponding signal value is low. It is also shown that TBD algorithm can be used for interlaced sensors what is important for today available acquisition systems. Most eﬀects described in this paper can be generalized to another TBD algorithm. The Track–Before–Detect algorithms are very powerful because they are independent on sensor type and can be used for both types. In future research

32

P. Mazurek

will be considered performance of sensors due to diﬀerent noise levels and noise characteristics. Acknowledgments. This work is supported by the MNiSW grant N514 004 32/0434 (Poland). This work is supported by the UE EFRR ZPORR project Z/2.32/I/1.3.1/267 /05 ”Szczecin University of Technology - Research and Education Center of Modern Multimedia Technologies” (Poland).

References 1. Blackman, S.S., Popoli, R.: Design and Analysis of Modern Tracking Systems. Artech House, Norwood (1999) 2. Ristic, B., Arulampalam, S., Gordon, N.: Bayesian Beyound the Kalman Filter: Particle Filters for Tracking Applications. Artech House, Norwood (2004) 3. Stone, L.D., Barlow, C.A., Corwin, T.L.: Bayesian Multiple Target Tracking. Artech House, Norwood (1999) 4. Daca, W., Mazurek, P.: Subpixel Track–Before–Detect Algorithm for Matrix Sensors. In: VI National Polish Conference on Electronic KKE 2007, Darlowko Wschodnie 2007 (article in polish), pp. 429–434 (2007) 5. Mazurek, P.: Likelihood Ratio Tracker as a Track–Before–Detect Method for dim object 2–D Tracking. In: XXX International Conference on Fundamentals of Electrotechnics and Circuit Theory IC-SPETO 2007, Gliwice–Ustron, pp. 211–212 (2007) 6. Mazurek, P.: Track–Before–Detect Algorithm for Bayer Pattern Sensors. In: XXXI International Conference on Fundamentals of Electrotechnics and Circuit Theory IC-SPETO 2008, pp. 133–134 +CD (2008) 7. Bayer, B.E.: Color imaging array. U.S. patent no. 3,971,065 (1976) 8. Ballard, R.C.: Television system. U.S. patent no. 2,152,234 (1932) 9. Amanatides, J., Mitchell, D.P.: Antialiasing of Interlaced Video Animation. In: SIGGRAPH 1990. Computer Graphics, Dallas, August 6–10, 1990, vol. 24(4) (1990) 10. Bellers, E.B., de Haan, G.: Advanced de–interlacing techniques. In: Proc. ProRISC/IEEE Workshop on Circuits, Systems and Signal Processing, Mierlo, The Netherlands, November 1996, pp. 7–17 (1996)

Road Lane Detection with Elimination of High-Curvature Edges ´ Krzysztof Slot, Michal Strzelecki, Agnieszka Krawczy´ nska, and Maciej Pola´ nczyk Institue of Electronics, Technical University of L ´ od´z 211/215 Wolczanska Str., 90-924 L ´ od´z, Poland {kslot,Michal.Strzelecki,Agnieszka.Krawczynska, Maciej.Polanczyk}@p.lodz.pl

Abstract. The following paper proposes a procedure for an automatic detection of road-lanes, which involves three main steps: two-category scene segmentation, a detection of lane segment candidates and decision making. The main contribution of the paper is an introduction of a new step high-curvature edge ﬁltering into a typical road image processing. This step substantially reduces an amount of noise and yields an improved lane detection performance. The remaining two steps of lane detection are dynamic programming-based selection of line segments, and Hough transform-based selection of the most consistent segments. The proposed approach proved to produce good results for a variety of real-traﬃc conditions, including correct detection in heavily shadowed road images. Keywords: road line detection, dynamic programming.

1

Introduction

A task of road line detection in road images is an essential problem for a development of a smart vehicle technology. There exist numerous problems that make the task of road-lane detection non-trivial. The ﬁrst group of diﬃculties includes factors that impair line visibility, such as shading (caused by trees or other vehicles) or complete line occlusions. Another set of problems is related to line appearance variability, which involves line color diﬀerences (e.g. for yellow lines) and line intensity variations. Finally, an automatic detection of line candidates becomes diﬃcult in heavily cluttered images, taken at heavy traﬃc conditions with a presence of other road markings. Several approaches to road line detection, which feature varying complexity and performance, have been proposed so far. A framework of majority of road lane detection algorithms involves a set of typical tasks: a deﬁnition of a road model, extraction of road-line relevant features from predominantly visual input, and an analysis of collected information, followed by optional, post-processing operations [1]. Most road lane models are built based on ﬁrst [2] or second order polynomials [3, 4,5] some of them also use piece-wise linear road modeling [6]. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 33–42, 2009. c Springer-Verlag Berlin Heidelberg 2009

34

´ K. Slot et al.

Majority of existing approaches to lane detection use edges as line representative features [4,5,7,8]. As in case of road-lane feature selection, a variety of diﬀerent approaches to an analysis of extracted feature images and to subsequent postprocessing, have also been exercised. Common approach to road line detection in edge images is to apply Hough transform [4,5,7]. Edge images are used as an input for quadratic approximations in [4]. Detailed survey on lane detecting and lane tracking algorithms can be found for example in [1] and [9]. An objective of the following paper is to present a method for detecting road lanes, which is expected to perform well in a variety of typical road traﬃc and road lighting conditions. The proposed procedure is composed of three main steps. The ﬁrst part is concerned with an extraction of road line-relevant information from input images. The main goal of the step is to produce a binary image, which is composed of edges that are likely to belong to road line markings. To ﬁlter out irrelevant scene elements, edge curvature is used as a criterion of their further acceptance. An objective of the second part of the procedure is to select only the edges that are feasible segments of road lines. This is performed by expressing a line-search task within a framework of dynamic programming [10]. A derivation of an appropriate cost function and a consecutive backtracking procedure are organized in a way, which is expected to favor appropriately distributed components. Finally, the third phase of the algorithm attempts to derive a linear approximation of road lane boundaries, by searching for the dominant peaks in a Hough space, constructed using dynamic programming results. Although linear approximations are not well-suited for road curves, they work suﬃciently well for moderate vehicle velocities, while oﬀering substantial computational advantages.

2

Line Detection Algorithm

The proposed road-line detection algorithm conforms to a typical scheme adopted for the task realization and involves three steps: preprocessing, analysis and postprocessing. To provide algorithms robustness, only some general assumptions are made with respect to objects that are to be detected. In particular, lines are expected to originate anywhere in the lower region of an image (that includes line detection during lane changes). Also, they are assumed to have a limited curvature and sharp edges. Finally, they are expected to have either a dominant local intensity (in case of white lines) or have a color that falls within a coarse deﬁnition of a yellow. These constraints are implemented at consecutive stages of the procedure. Image preprocessing step is aimed at extraction of these image object edges, which have a low curvature, with a special emphasis at edges originating from yellow image regions. Subsequent image analysis, performed by means of dynamic programming, attempts to ﬁnd several paths that terminate at the lower part of an image and presumably contain road-line segments. Finally, Hough transform is used to extract a linear approximation to near-ﬁeld lane boundaries. An objective of image preprocessing phase of the procedure is to extract from input images only the information, which is relevant for road line detection. The

Road Lane Detection with Elimination of High-Curvature Edges

35

Fig. 1. Image preprocessing: input image (a), edge detection results (b), extraction of regions with a desired hue value range and their boundaries (c and d), combined edges (e) and a result of high curvature edge removal (f)

simplest properties, which could be considered as indicators of a line membership of a given image pixel are its color (usually white or yellow) and an existence of strong intensity gradients (that occur at line boundaries). As in case of majority of proposed methods, we assumed that both of the properties will be used as features in further processing. Image preprocessing step begins with detection of input image edges. A Laplacianof- Gaussian (LoG) [11] image ﬁltering and consecutive detection of zerocrossings is used for edge extraction. LoG-based edge extraction method has been selected among many other possible approaches to avoid an excessive amount of detected pixels, and consequently, excessive amount of road-line candidate paths that could appear in subsequent processing. Sample road image and corresponding edge detection results have been shown in Fig. 1. The second step of image preprocessing is aimed at emphasizing road lanes that are painted in yellow. Yellow lines typically feature poorer contrast and are likely to get undetected during the ﬁrst part of the procedure. Unfortunately, a range of colors that are typically used for making yellow road markings is quite wide, so that a detection of any particular value of a color attribute (such as hue from HSV color space) is likely to fail. Therefore, we propose another approach for yellow line detection. First, we ﬁlter from HSV representation of an image (Fig. 1c) all elements with a hue components located within a range of 300 to 700 (the interval includes colors ranging from an orange to yellow-green on a hue scale). Next, we detect edges of the extracted regions and we insert them into the edge map that has been constructed in the ﬁrst step of the preprocessing. The third step of the preprocessing is aimed at removal of all of the edges with large curvature, i.e. edges, which cannot belong to line boundaries. From

36

´ K. Slot et al.

diﬀerential geometry it is known that eigenvalues of a Hessian matrix of image intensity can be used for estimating a local curvature at any image point. Large eigenvalue ratios indicate regions with substantially diﬀerent gradient changes along diﬀerent principal directions, i.e. edge-like image structures. As suggested in [12], we used computationally eﬃcient assessment of an eigenvalue ratio. Since a determinant of a Hessian matrix and its trace can be expressed using its eigenvalues λ1 and λ2 = kλ1 (det(H) = λ2 λ1 , TR(H) = λ2 + λ1 ), one can construct computationally convenient criterion function C(k) for local curvature evaluation: (T r(H))2 (λ1 + kλ1 )2 (1 + k)2 C(k) = = = (1) det(H) λ1 kλ1 k A decision for an acceptance or a rejection of an edge point can be based on testing a value of (1) against a predeﬁned threshold T . Observe that for eigenvalues of the same magnitudes and of the same sign C(k) = 4, whereas for eigenvalues of the same magnitude and opposite signs C(k) = 0. Therefore, an adopted edge-acceptance criterion is the following: C(k) ↔ T :

C < −T or C > 4 + T accept otherwise reject

(2)

The test condition (2) accounts for possibly diﬀerent signs of eigenvalues (k can be either positive or negative) and can be easily derived from a function (1). The best performance over a database of road images used in our experiments (as shown in Section 3) was achieved for T = 2.25, which translates into rejecting all these edge points, where an absolute ratio of eigenvalues (k) was lower than four. A result of edge-ﬁltering, shown in Fig. 2f forms a basis for a derivation of a cost map for the dynamic programming step of the procedure. Dynamic programming is a strategy for searching for a minimum-cost path along a two-dimensional oriented graph [10]. Given an image, a corresponding search graph can be constructed by letting each of its pixels to be a graph node. A pattern of available inter-node transitions is homogenous for the whole graph, excluding its boundary elements. Dynamic programming is composed of two phases. The ﬁrst one is a computation of minimum costs of visiting of any of graphs nodes. A cost assigned to any node, located in i-th row and j-th column of a graph (a node ij) is evaluated using a recurrent formula of the general form: c∗ij = min(c∗ + dkl→ij + dij ) (3) kl

The symbols dkl→ij and dij denote a cost of a transition from a node kl to a node ij and a cost of staying at a node ij respectively. The cost c∗ij is the minimum value taken for all nodes kl, which are legal predecessors of ij, i.e. for nodes determined by an adopted transition pattern. Concurrently with computing minimal costs, also locations of nodes predecessors, which score the

Road Lane Detection with Elimination of High-Curvature Edges

37

Fig. 2. A scheme of inter-node transitions used in cost map derivation (a), imageintensity related transition cost (b) and additional vertical transition-preference costs (c)

best, are recorded. This information is used in the second part of dynamic programming (backtracking) for a quick recovery of the best path (if a single starting point and a single end point are assumed) or best paths (if multiple terminal points are allowed). In the proposed method, dynamic programming is used to create several paths that are expected to link upper and lower portions of an image, which is produced during the preprocessing phase. Throughout our research, we are concerned with a detection of near-ﬁeld portions of road lanes, as this is of primary interest for vehicle navigation. Therefore, we restrict a range of allowed line orientations, by excluding candidates with horizontal and near-horizontal slopes. This goal can be attained by setting an appropriate pattern of admissible node transitions. A pattern that has been adopted in the proposed method, depicted in Fig. 2, allows for only downward transitions, within a limited neighborhood. Any node can be accessed only from its ﬁve upper neighbors, located no farther than plus or minus two columns away. This is equivalent to ﬁltering out of all edges with slopes that fall below 260 or above 1540 with respect to image horizontal axis. Cost computation is performed row-wise, originating at the top row of the produced edge image. To prevent paths from ending at boundaries of an upper part of an image, an additional cost component, which favors initial transitions along central columns, is imposed on elements of an upper image strip. Since an expected outcome of dynamic programming is to extract a set of paths that are composed of the extracted road image edges, the cost-assignment rule (3) needs to be appropriately speciﬁed. We assume that only the

38

´ K. Slot et al.

transition-related cost components (dkl→ij ) are relevant for a path search, so that dij elements are set to zero: c∗ij = min(c∗kl + dkl→ij ) kl

(4)

To prefer edge-point following, we set a cost for transitions between edge elements to zero, transitions between background elements to one and transitions between mixed elements to a half (Fig. 2b). The last element of the cost-assignment rule is to favor paths that evolve directly towards a bottom of an image. This measure is introduced to account for cases, when several obstacles (cars), generating strong, competing edges, are present in neighboring road lanes. The constraint is implemented by additional multiplicative transition weights, as depicted in Fig. 2c. As a result, a cost map is derived by evaluating the equation (4) for every image pixel, using the adopted node-transition cost values. In addition, every image pixel is labeled with a location of its predecessor for the purpose of fast recovery of the optimum path. A cost map derived for an image by means of the presented rules is subsequently used as input to the second phase of dynamic programming, which is backtracking. Backtracking is a recovery of paths that yield the lowest costs and which originate at a set of selected terminal points.

Fig. 3. Dynamic Programming-generated paths (a) and actual edge segments along these paths (b)

A result of dynamic programming is a set of paths, which link upper and lower image regions. These paths are composed of edge segments as well as links that were introduced by backtracking in background-only regions. Since only edges of an original image are of interest for subsequent analysis, a simple logical pixelwise AND operation is performed on edge-image and dynamic-programming generated paths (Fig. 3) to extract these edges. The result of this step becomes an input to the last element of the algorithm, which is a Hough transform-based detection of lines. This step of the procedure can be replaced by any other technique of road-lane approximation. In addition, it can be preceded by some additional procedures that can validate, whether extracted edges are likely to be a boundary of actual road lane markings.

3

Experimental Evaluation

An objective of an experimental evaluation of the proposed method was to verify its performance in a variety of realistic road conditions. The main goal of the

Road Lane Detection with Elimination of High-Curvature Edges

39

Table 1. Results of experimental veriﬁcation of the proposed procedure Event Number of instances Percentage Correct line detection 1811 75% DP failure 87 24% Detection errors 25 1%

Fig. 4. Dynamic programming-produced paths in a set of sample road-images

proposed method is to provide high-conﬁdence detection of a road line. Therefore, we were trying to optimize methods parameters to get the minimum rate of erroneous line segment detections. Since the proposed algorithm has no mechanisms to prevent from errors originating from road markings, frames, which include these elements, were under represented during testing. However, since it is possible to detect such scene elements, we consider this limitation to be of minor importance. A performance of the method was evaluated using video recordings, taken at varying traﬃc conditions and at varying road daylight illumination. Color frames taken at a resolution of 640 × 480 from several video sequences were subject to an independent analysis. As input images contain irrelevant information in their upper portions, all of the frames were cropped to cover a range from 3/8 to 7/8 of the original height. Sample images and results of their analysis, are presented in Fig. 5, where input images, dynamic programming-generated paths and ﬁnal processing results are shown. As it can be seen, the algorithm is able to correctly identify line segments in typical road conditions. An overall performance of the proposed procedure has been summarized in Table 1. Three types of events have been considered for the algorithm evaluation. Correct detection occurs if an extracted edge is a boundary of an actual line segment (this can be considered as a true positive case). All the cases, when lines are missed, are labeled as DP failure. These false negative detection results refer to dynamic programming-generated paths that contain no line-edge segments, and consequently, cannot yield a correct result. The last category of events Detection errors refers to extraction of edges, which are not line segment boundaries, i.e. to false positive detection results. Sample results of the proposed algorithm operation are presented in Fig. 4 and Fig. 5. The former ﬁgure shows an outcome of the dynamic programming

40

´ K. Slot et al.

Fig. 5. Illustration of basic steps of the proposed road-lane detection algorithm: input image (a), edge-ﬁltering result (b), curved-edge ﬁltering (c), dynamic programming generated paths (d), edges extracted from the optimal paths (e), line approximations derived using Hough transform (f) and dominant peaks in the Hough space, which yield the approximations (g)

procedure. It can be seen that the proposed rules for optimal path selection result in extracting edge collections, which contain the target objects, i.e. boundaries of road-lane markings. A complete road-lane extraction procedure has been shown in Fig. 5. It can be seen that information produced by subsequent steps of the proposed algorithm: edge extraction and ﬁltering, dynamic programming and Hough transform-based line approximation, yields good estimates of a road-lane. A complexity of the presented procedure is moderate dynamic programming part of the procedure has a complexity of O(N M ), where N and M denote width and a height of an image. A computational cost of image preprocessing depends on image contents, as only extracted edges are subject to curvature estimation. For the presented experiment, a total time of image preprocessing did not exceed

Road Lane Detection with Elimination of High-Curvature Edges

41

a dynamic programming execution time in any of analyzed cases, and on average it fell below a quarter of the latter value.

4

Conclusion

The proposed procedure focuses on robust detection of road line segments in several realistic road conditions. It has been shown that this task can be done at low error rates by combining dynamic programming with additional pre-processing. One of the advantages of the proposed algorithm is its relative computational simplicity. A bulk of computations is done during its dynamic programming phase; however, these are simple local operations, which can be transferred into some specialized hardware. There exist numerous challenges that have not been addressed in the reported research and that aﬀect line detection performance. A poor line-to-road surface contrast might require either an application of more sophisticated edge extraction methods than the adopted LoG ﬁltering or could result in an adoption of strategies that do not rely on edges. Another diﬃculty relates to the aforementioned variability in colors of non-white lines. Road markings and crosswalk detection are additional separate problems that need to be solved to enhance a performance of the presented algorithm. As it has been also mentioned, several additional tests can be applied to an outcome of the presented algorithm, before a ﬁnal decision regarding a location of a road line is made. Taking into account a low detection error rate, the proposed approach may become an attractive strategy for the purpose of automatic vehicle navigation.

References 1. McCall, J.C., Trivedi, M.: Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation. IEEE Trans. Intell. Transp. Syst. 7(1), 20–37 (2006) 2. Saha, B., Mazumdar, A., Pal, N.: Bidirectional Fuzzy-Regression Model for Roadlines Detection. In: Proc. IEEE Int. Conf. on Engineering of Intelligent Systems, pp. 1–6 (2006) 3. Kreucher, C., Lakshmanan, S.: LANA: A Lane Extraction Algorithm that Uses Frequency Domain Features. IEEE Trans. Robot. Autom. 15(2), 343–350 (1999) 4. Jung, C., Kelber, C.: Lane following and lane departure using a linear-parabolic model. Image Vision Comput. 23(13), 1192–1202 (2005) 5. Li, Q., Zheng, N., Cheng, H.: Springrobot: A prototype autonomous vehicle and its algorithms for lane detection. IEEE Trans. Intell. Transp. Syst. 5(4), 300–308 (2004) 6. Cheng, H., Jeng, B., Tseng, P., Fan, K.: Lane Detection With Moving Vehicles in the Traﬃc Scenes. IEEE Trans. Intell. Transp. Syst. 7(4), 571–582 (2006)

42

´ K. Slot et al.

7. Kang, D., Jung, M.: Road lane segmentation using dynamic programming for active safety vehicles. Pattern Recognition Letters 24(16), 3177–3185 (2003) 8. Yim, Y., Oh, S.: Three-Feature Based Automatic Lane Detection Algorithm (TFALDA) for Autonomous Driving. IEEE Trans. Intell. Transp. Syst. 4(4), 219– 225 (2003) 9. Kastrinaki, V., Zervakis, M., Kalaitzakis, K.: A survey of video processing techniques for traﬃc applications. Image and Vision Computing 21(4), 359–381 (2003) 10. Bellman, R.E.: Dynamic Programming. Dover Publications, New York (2003) 11. Gonzales, R.C., Woods, R.E., Eddins, S.L.: Digital image processing using Matlab. Pearson Education Inc., London (2004) 12. Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)

A Statistical Reduced-Reference Approach to Digital Image Quality Assessment Krzysztof Okarma and Piotr Lech Szczecin University of Technology, Faculty of Electrical Engineering Chair of Signal Processing and Multimedia Engineering 26. Kwietnia 10, 71-126 Szczecin, Poland {krzysztof.okarma,piotr.lech}@ps.pl

Abstract. In the paper a fast method of the digital image quality estimation is proposed. Our approach is based on the Monte Carlo method applied for some classical and modern full-reference image quality assessment methods, such as Structural Similarity and SVD-based measure. Obtained results are compared to the eﬀects achieved using the full analysis techniques. Signiﬁcant reduction of the number of analysed pixels or blocks leads to fast and eﬃcient estimation of image quality especially in low performance systems where the processing speed is much more important than the accuracy of the quality assessment. Keywords: image quality assessment, Monte Carlo method, statistical image analysis.

1

Introduction

Automatic image quality assessment can be considered as one of the most relevant ﬁelds of contemporary image processing and analysis. Because of wide potential area of applications of image quality estimation algorithms the dynamic progress in this ﬁeld of research can be observed in recent years. The most common applications of objective image quality assessment techniques can be the development of some new image processing (e.g. ﬁltering) algorithms or lossy compression standards, as well as their optimisation. In fact, the development of any new image processing method requires a truthful comparison of obtained results to previously used techniques. Digital image quality assessment methods can be divided into two major groups. The ﬁrst one is related to the subjective evaluation of the image by human observers, ﬁlling up some questionnaires and the statistical analysis of obtained results e.g. using Mean Opinion Score approach. Subjective image quality assessment may be helpful in the development of some new objective metrics based on the Human Visual System (HVS) model. However, such approach is impractical in most computer systems mainly because of the necessity of human interaction during the whole process. The most desired solution for the development of some new image processing algorithms is the numerical evaluation of the image quality, preferably using a single scalar value, although some vector L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 43–54, 2009. c Springer-Verlag Berlin Heidelberg 2009

44

K. Okarma and P. Lech

metrics can also be used for that purpose. An automatic calculation of such metrics corresponds to the objective image quality assessment, regardless of the diﬀerences between subjective evaluations performed by a number of observers. Three general types of them can be distinguished: full-reference, reducedreference and no-reference (blind) ones. The ﬁrst group of metrics requires the access to the original image before processing and the quality metric is calculated as a measure of the diﬀerences between the original and processed image. Such methods are treated as the most universal ones and for that reason only the full-reference approach is investigated further in the paper. Existing blind algorithms [9] are usually sensitive only to a speciﬁed type of distortions (e.g. block eﬀects in JPEG compression [2,11,19,21] or blurring [10,12]) so their universality is at least doubtful. Reduced-reference methods [3,22] utilise only the partial information about the original image but their computational complexity is usually similar to the full-reference ones. As the side information used instead of the original image, the partial wavelet data, edge locations, local statistics etc. may be used with the speciﬁed assumptions related to the application or domain. Both blind and reduced-reference metric families seem to be promising but little progress has been made in this ﬁeld during recent years. The solution presented in the paper can be treated as a kind of reduced-reference method but in fact it is based on well known full-reference approach. The rest of the paper is organised as follows: in the section 2 some traditional and modern image quality metrics are brieﬂy described, then the idea of the statistical approach for fast image quality estimation is discussed in the section 3 and ﬁnally the results of the quality assessment of some images with some typical distortions are presented in the fourth section.

2

Full-Reference Image Quality Metrics

Classical approach to image quality assessment, mainly based on the Mean Square Error (MSE) and many similar metrics, such as Peak Signal to Noise Ratio (PSNR), Mean Absolute Error (MAE) etc., has a lot of disadvantages. Although it has been used for many years [6], the correlation of such metrics with the way human perceive digital images is very poor. It is well known that it is not a diﬃcult task to prepare such a pair of images (the original and assessed one) that such traditional metrics indicate relatively poor quality while the assessed image is perfect but e.g. translated by several pixels. Nevertheless, many researchers still use them regardless of their poor correlation to the Human Visual System (HVS). An overview of the most relevant classical image quality metrics can be found e.g. in the paper [5]. In the further part of the paper only the PSNR and MAE will be used from the variety of the classical metrics. Because of many problems with such measures, related e.g. to their strong sensitivity to the image content, various dynamic ranges and results not necessarily corresponding with the subjective evaluation, some diﬀerent approaches to the objective image quality assessment have been investigated during the recent years in purpose to improve

A Statistical Approach to Digital Image Quality Assessment

45

the evaluation correctness and the correlation to the HVS. Nevertheless, some of them are designed for the sensitivity to only two or three speciﬁc types of distortions like Gaussian noise and JPEG compression [1] or salt and pepper noise and changes of luminance [17]. The ﬁrst measure, intended for elimination of the most typical disadvantages of traditional measures, is the Universal Image Quality Index (UIQI) deﬁned by Wang and Bovik [18] as: Q=

(σx2

+

4σxy x ¯y¯ σxy 2¯ xy¯ 2σx σy = · · 2 , 2 2 2 2 · [(¯ x) + (¯ y) ] σx σy (¯ x) + (¯ y ) σx + σy2

σy2 )

(1)

where x and y denote the original and distorted image respectively and x¯ =

M N 1 xij M · N i=1 j=1

1 (xij − x ¯)2 M · N − 1 i=1 j=1 M

σx2 =

y¯ =

N

1 (yij − y¯)2 M · N − 1 i=1 j=1 M

σy2 =

N

(2)

1 = (xij − x ¯)(yij − y¯) . M · N − 1 i=1 j=1 M

σxy

M N 1 yij M · N i=1 j=1

N

where xij and yij are the pixels of two compared images. Such approach leads to the quality map of the image of M × N pixels and the overall quality index is deﬁned as the mean value of metrics obtained from (1) calculated for all image fragments (usually of 8 × 8 pixels as suggested in the paper [18]) using the sliding window approach. Nevertheless, because of the usage of the sliding window approach, the resolution of the image quality map is only a little smaller than the image size (depending on the window size), diﬀerently from the block approach as in the MSVD discussed later. It is also worth noticing that such measure is sensitive to three common types of distortions introduced by many image processing algorithms: the loss of correlation, luminance distortions and the loss of contrast. Because of the possibility of division by zero for large regions of the same colour, especially dark ones, the UIQI measure has been extended into Structural Similarity (SSIM) with additional possible choice of weighting coeﬃcients for each of three factors [20] using the formula (in shortened notation) SSIM =

(2¯ xy¯ + C1 ) · (2σxy + C2 ) , (σx2 + σy2 + C1 ) · [(¯ x)2 + (¯ y )2 + C2 ]

(3)

where C1 and C2 are chosen such that they do not introduce any signiﬁcant changes of the results (e.g. C1 = (0.01×255)2 and C2 = (0.03×255)2 as suggested by the authors of the paper [20]. The role of those coeﬃcients is related only to the prevention of the possible division by zero for ﬂat and dark areas of the image. Another diﬀerence introduced in comparison to the UIQI is the type and size of the sliding window used during calculations. Instead of the 8 × 8 pixels rectangular window, the Gaussian one is applied with the size of 11 × 11 pixels.

46

K. Okarma and P. Lech

In order to provide even better correlation with the subjective evaluation another image quality measure based on the Singular Value Decomposition (SVD) of the matrix corresponding to the image block (size of 8 × 8 pixels is suggested as well) has been proposed in the papers [15] and [16]. At ﬁrst the following value is computed for each block in that approach: N 2 Di = (sk − sˆk ) , (4) k=1

where sk and sˆk are the singular values for original and distorted image blocks. The quality measure is then deﬁned as the following expression: K

MSVD =

i=1

|Di − Dmid | K

,

(5)

where K denotes the total number of N × N pixels blocks and Dmid is the middle element of the sorted vector D. Such calculations also make it possible to create the quality map of the image (with much smaller resolution than for the SSIM because of using block approach instead of sliding window technique) and obtained values can also be averaged leading to the overall scalar metric. Another idea based on full image transforms (Discrete Fourier Transform - DFT, Discrete Wavelet Transform - DWT and Discrete Cosine Transform DCT) is presented in the paper [8]. The main idea is related to the calculations of the mean of four standard deviations of the diﬀerences between transform’s magnitude coeﬃcients for four bands in original and distorted image. Assuming the size of a speciﬁed image is N × N pixels each of three metrics denoted as MDWT , MDCT and MDFT can be computed as follows [8]: Apply the transform (DFT, DWT, or DCT) to the original image x. Apply the transform to the distorted image y. Divide the transform coeﬃcients into four bands. For each band: – Calculate the magnitudes M xi of original transform coeﬃcients. – Calculate the magnitudes M yi of distorted ones (i = 1..N 2 /4). – Calculate the absolute value of the diﬀerences |M xi − M yi |. – Calculate the standard deviation of the diﬀerences. 5. Compute the mean of four standard deviations as the result. 1. 2. 3. 4.

3

Monte Carlo Approach to Fast Image Quality Estimation

In many applications the exact image or video quality estimation is not as important as the high speed of such computations. The reason may be the necessity

A Statistical Approach to Digital Image Quality Assessment

47

of using the maximum possible computational power e.g. for compression, transmission or acquisition purposes. The same situation may take place according to the amount of memory required for such quality assessment, especially in low performance embedded systems with the limited amount of operating memory. For that reason an interesting direction of research seems to be the decrease of the amount of data needed for proper quality estimation of images and the reduction of total computational cost of the assessment. The price is the lower accuracy of computation but in fact practically all known quality metrics should be actually treated as estimators rather than the exact quality measures, so if the image quality estimation should be fast enough to avoid the introduction of additional delays, a reasonable estimation error can be allowed. In our method the decrease of the number of analysed pixels is based on the statistical experiment using the Monte Carlo method [14]. Because of its low computational cost it can be widely used in image analysis, especially for preliminary purposes, as well as for fast motion detection instead of full frame analysis used e.g. in well known MPEG algorithms. The analysis of the Monte Carlo technique with its applications can be found in [13]. Proposed technique of the image quality assessment is analogous to the object’s area estimation using the Monte Carlo method. In comparison to more sophisticated statistical methods [7] such as popular Markov Chain Monte Carlo or Sequential Monte Carlo [4], which can also be applied to image analysis, the presented approach is a simpliﬁed but very eﬃcient solution, especially in real-time applications. The main idea is based on drawing a speciﬁed number of pixels using the pseudo-random number generator with uniform distribution in order to use them for comparison of the assessed image to the original one. The necessity of using two independent generators for two coordinates x and y may be replaced by indexing all the pixels what is equivalent to copying them to a one-dimensional vector. Such approach can be successfully implemented for most classical metrics, such as PSNR or MAE, as well as for some modern ones - UIQI and SSIM. Considering the MSVD metric such approach cannot be implemented in the same way because of the necessity of analysing full blocks in order to perform the Singular Value Decomposition. Nevertheless, the application of the Monte Carlo approach is also possible and the proposed solution relies on the drawing of the given number of the whole 8 × 8 pixels blocks. It is worth noticing that the number of randomly chosen blocks is not restricted by the rectangular character of the image and can be whichever assuming that all blocks are copied into the auxiliary image of K × 8 pixels where K is the number of the drawn blocks. An identical approach can be applied for transform-based metrics assuming their calculation for the auxiliary images constructed from the drawn blocks.

4

Discussion of Results

The veriﬁcation of the proposed technique has been performed for 7 quality metrics: PSNR, MAE, SSIM, MSVD and three transform-based ones. The images chosen for the tests are some of the well known and widely used test images by the

48

K. Okarma and P. Lech

Fig. 1. Test images with some typical distortions: 15% impulse noise, median ﬁltration (5x5 pixels mask), low-pass ﬁltration (3x3 mask), JPEG compression, loss of contrast and palette reduction to 16 colours

image processing society. Some of the typical distortions used in our experiments can be observed in Fig. 1. The results of the image quality assessment using traditional full frame analysis are presented in Table 1. The application of the proposed pixel-based statistical technique for the MAE, PSNR and SSIM metrics leads to the results presented as the relative errors in the Fig. 3 while the relative errors obtained for the block-based approach using the MSVD and transformbased metrics are illustrated on Figs. 2 and 4 respectively. The more detailed results obtained for the proposed Monte Carlo based SSIM metric are presented in Table 2. Analysing the eﬀects of applying the proposed technique for two classical image quality measures (MAE and PSNR) it can be stated that the reduction from 262144 considered pixels to 50000 or even 5000 ones gives relatively good results as the average relative error does not reach 1%. However, further decrease of the number of drawn pixels causes the noticeable increase of the average relative error e.g. to about 3.5% for PSNR and about 5.9% for MAE using only 50 pixels with the error level for speciﬁc distortions reaching 18%. Application of the MSVD and transform-based metrics leads to some other conclusions, especially the results obtained using the MDFT measure where the relative errors are unacceptably high. Nevertheless, the dependence between the relative errors and the number of blocks used in calculations is clearly visible. Considering the results achieved for the MSVD measure it can be noticed that even for 1000 from 4096 blocks the average relative error for all distortions is

A Statistical Approach to Digital Image Quality Assessment

49

Table 1. Results of quality assessment for various distortions with full image analysis - mean values of 5 example images Quality metric PSNR MAE SSIM MSVD MDWT MDCT MDFT Quality metric PSNR MAE SSIM MSVD MDWT MDCT MDFT

low-pass 3x3 23.9937 0.0582 0.8948 37.2795 0.0337 0.0458 27.0228 20% noise 12.7295 0.0908 0.1257 69.1532 0.1780 0.1411 56.4413

low-pass 5x5 20.3765 0.0776 0.7641 43.8407 0.0538 0.0634 41.9517 16 colours 5.9054 0.4784 0.0775 256.2924 0.1199 0.2724 189.2957

median 3x3 32.9129 0.0140 0.9060 12.0719 0.0214 0.0165 7.7785 contrast 0.7 24.1660 0.0544 0.9551 42.8910 0.0241 0.0346 29.6573

median 5x5 28.7038 0.0216 0.8203 21.3933 0.0324 0.0246 14.0310 contrast 0.5 19.7148 0.0908 0.8786 71.1702 0.0401 0.0577 49.5194

5% noise 18.4126 0.0248 0.3740 63.6806 0.1103 0.0724 28.5594 JPEG 60% 35.2828 0.0130 0.9374 4.4632 0.0132 0.0114 5.0666

10% noise 15.5273 0.0479 0.2219 65.2958 0.1440 0.1015 40.1885 JPEG 40% 33.6571 0.0155 0.9175 6.0622 0.0162 0.0139 6.1745

15% noise 13.8699 0.0699 0.1594 67.4011 0.1641 0.1233 48.9829 JPEG 10% 28.6246 0.0272 0.8124 15.1838 0.0280 0.0239 12.1463

Fig. 2. Absolute relative errors of MSVD for various number of randomly chosen blocks and various image distortions

about 4.9% and is particularly high for JPEG compressed images (the average error without JPEG compression is about 1.8%). A slightly better situation takes place for the transform measures based on the DWT (using Daubechies wavelets) and the DCT with the relative errors from 2% to even 10% depending on the number of blocks as shown in Fig. 4.

50

K. Okarma and P. Lech

Fig. 3. Absolute relative errors of MAE, PSNR and SSIM metrics for various number of randomly chosen pixels and various image distortions

A Statistical Approach to Digital Image Quality Assessment

51

Fig. 4. Absolute relative errors of the transform-based metrics for various number of randomly chosen blocks and various image distortions

52

K. Okarma and P. Lech

Table 2. Relative errors of the SSIM metric various distortions obtained using various number of samples - mean values of 5 example images Number of samples 50 100 500 1000 5000 10000 50000 Number of samples 50 100 500 1000 5000 10000 50000

low-pass 3x3 0.879% 0.029% 0.452% −0.017% −0.012% 0.042% 0.027% 20% noise 0.504% −2.624% 4.751% 1.603% 0.304% −0.083% −0.244%

low-pass 5x5 −1.303% 0.795% 0.479% −0.329% 0.050% 0.144% −0.049% 16 colours −2.631% 1.320% 4.272% −1.141% 0.255% −0.088% −0.029%

median 3x3 0.808% −0.454% 0.433% −0.040% 0.009% −0.002% 0.019% contrast 0.7 0.332% 0.202% −0.066% 0.027% 0.020% −0.017% 0.007%

median 5x5 −0.167% 0.953% 0.989% −0.140% −0.096% 0.001% −0.013% contrast 0.5 0.358% 0.446% −0.149% 0.051% 0.012% −0.038% 0.019%

5% noise −8.529% 5.159% 0.402% 1.070% −0.797% 0.026% −0.099% JPEG 60% 0.100% −0.308% 0.056% 0.105% −0.017% −0.010% −0.002%

10% noise −2.515% −7.378% 4.365% 0.356% 0.049% 0.115% 0.268% JPEG 40% 0.130% −0.555% −0.020% 0.186% −0.046% −0.057% 0.008%

15% noise −6.613% −7.548% 3.207% 0.623% 0.637% 0.163% −0.050% JPEG 10% 0.132% −1.879% 0.712% 0.290% −0.071% −0.050% −0.007%

The most promising results have been obtained for the Structural Similarity index with the relative error not exceeding 0.5% even for only 1000 samples used for the calculations. For many types of distortions even using only the 50 ones the relative error does not exceed 1%. Analysing presented results, an impulse noise can be generally classiﬁed as the most diﬃcult type of distortions, causing the increase of the relative errors in our method for the SSIM index when a small number of samples is used for the quality estimation.

5

Summary

Presented method of fast quality estimation of the digital images based on the Monte Carlo approach and the Structural Similarity index with its low computational cost and high accuracy can be an eﬃcient solution especially for the real-time hardware and embedded systems with a low amount of memory. It can be particularly relevant for all the applications where a rough but fast image quality estimation is needed but their main task is related to image or video compression, transmission, classiﬁcation, recognition etc. Presented solution is much faster than some other popular statistic methods widely used in image and video analysis and this advantage can be treated as the most relevant one. As a part of our future work the application of proposed technique for the colour image quality assessment can be considered as well as the analysis of the inﬂuence of the pseudo-random generator’s quality on the results of estimation as only the Matlab’s built-in generator has been used in this paper. Another idea can be related to more specialised areas of applications and the analysis of possibilities of using some other distributions for some speciﬁc types of images.

A Statistical Approach to Digital Image Quality Assessment

53

References 1. Beghdadi, A., Pesquet-Popescu, B.: A New Image Distortion Measure Based on Wavelet Decomposition. In: Proc. 7th Int. Symp. Signal Processing and Its Applications, Paris, France, pp. 485–488 (2003) 2. Bovik, A., Liu, S.: DCT-domain Blind Measurement of Blocking Artifacts in DCTCoded Images. In: Proc. Int. Conf. Acoustics, Speech and Signal Processing, Salt Lake City, USA, pp. 1725–1728 (2001) 3. Carnec, M., Le Callet, P., Barba, P.: An Image Quality Assessment Method Based on Perception of Structural Information. In: Proc. Int. Conf. Image Processing, Barcelona, Spain, vol. 2, pp. 185–188 (2003) 4. Chen, D., Odobez, J.-M.: Sequential Monte Carlo Video Text Segmentation. In: International Conference on Image Processing ICIP 2003, vol. 3, pp. 21–24. IEEE Press, New York (2003) 5. Eskicioglu, A., Fisher, P., Chen, S.: Image Quality Measures and Their Performance. IEEE Trans. Comm. 43(12), 2959–2965 (1995) 6. Eskicioglu, A.: Quality Measurement for Monochrome Compressed Images in the Past 25 Years. In: Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Istanbul, Turkey, pp. 1907–1910 (2000) 7. Fearnhead, P.: Computational Methods for Complex Stochastic Systems: A Review of Some Alternatives to MCMC. Statistics and Computing 18(2), 151–171 (2008) 8. Girshtel, E., Slobodyan, V., Weissman, J., Eskicioglu, A.: Comparison of Three Full–Reference Color Image Quality Measures. In: Proc. SPIE of 18th IS&T/SPIE Annual Symposium on Electronic Imaging, Image Quality and System Performance, San Jose, CA, vol. 6059 (2006) doi:10.1117/12.644226 9. Li, X.: Blind Image Quality Assessment. In: Proc. IEEE Int. Conf. Image Proc., pp. 449–452 (2002) 10. Marziliano, P., Dufaux, F., Winkler, S., Ebrahimi, T.: A No-Reference Perceptual Blur Metric. In: Proc. IEEE Int. Conf. Image Processing, Rochester, USA, pp. 57–60 (2002) 11. Meesters, L., Martens, J.-B.: A Single-Ended Blockiness Measure for JPEG-Coded Images. Signal Processing 82(3), 369–387 (2002) 12. Ong, E.-P., Lin, L.W., Yang, Z., Yao, S., Pan, F., Jiang, L., Moschetti, F.: A NoReference Quality Metric for Measuring Image Blur. In: Proc. 7th Int. Symp. Signal Processing and Its Applications, Paris, France, pp. 469–472 (2003) 13. Okarma, K., Lech, P.: Monte Carlo Based Algorithm for Fast Preliminary Video Analysis. In: Bubak, M., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2008, Part I. LNCS, vol. 5101, pp. 790–799. Springer, Heidelberg (2008) 14. Rubinstein, R.Y.: Simulation and the Monte Carlo Method. Wiley, Chichester (1981) 15. Shnayderman, A., Gusev, A., Eskicioglu, A.: A Multidimensional Image Quality Measure Using Singular Value Decomposition. Proc. SPIE Image Quality and Syst. Perf. 5294(1), 82–92 (2003) 16. Shnayderman, A., Gusev, A., Eskicioglu, A.: An SVD-Based Gray-Scale Image Quality Measure for Local and Global Assessment. IEEE Trans. Image Processing 15(2), 422–429 (2006) 17. Van der Weken, D., Nachtegael, M., Kerre, E.: A New Similarity Measure for Image Processing. Journal of Computational Methods in Sciences and Engineering 3(2), 209–222 (2003)

54

K. Okarma and P. Lech

18. Wang, Z., Bovik, A.: A Universal Image Quality Index. IEEE Signal Processing Letters 9(3), 81–84 (2002) 19. Wang, Z., Bovik, A., Evans, B.: Blind Measurement of Blocking Artifacts in Images. In: Proc. IEEE Int. Conf. Image Processing, pp. 981–984 (2000) 20. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image Quality Assessment: From Error Measurement to Structural Similarity. IEEE Trans. Image Processing 13(4), 600–612 (2004) 21. Wang, Z., Sheikh, H., Bovik, A.: No-Reference Perceptual Quality Assessment of JPEG Compressed Images. In: Proc. IEEE Int. Conf. Image Processing, Rochester, USA, pp. 477–480 (2002) 22. Wang, Z., Simoncelli, E.: Reduced-Reference Image Quality Assessment using a Wavelet-Domain Natural Image Statistic Model. In: Proc. Human Vision and Electronic Imaging Conference, Proceedings of SPIE, San Jose, USA, vol. 5666, pp. 149–159 (2005)

Colour Image Quality Assessment Using Structural Similarity Index and Singular Value Decomposition Krzysztof Okarma Szczecin University of Technology, Faculty of Electrical Engineering Chair of Signal Processing and Multimedia Engineering 26. Kwietnia 10, 71-126 Szczecin, Poland [email protected]

Abstract. In the paper the analysis of the inﬂuence of the colour space on the results obtained during image quality assessment using the Structural Similarity index and the Singular Value Decomposition approach has been investigated. Obtained results have been compared to the ones achieved by widely used Normalised Colour Diﬀerence (NCD) metric. All the calculations have been performed using the LIVE Image Quality Assessment Database in order to compare the correlation of achieved results with the Diﬀerential Mean Opinion Score (DMOS) values obtained from the LIVE database. As a good solution for the further research, also with the use of some other image quality metrics, the application of the HSV colour space is proposed instead of commonly used YUV/YIQ luminance channel or the average of the RGB channels. Keywords: colour image quality assessment, colour spaces, Structural Similarity, SVD.

1

Introduction

Colour image quality assessment is currently one of the most relevant aspects of digital image analysis. The most important applications of digital image and video quality assessment methods are related not only to lossy compression but to development of some new image and video processing algorithms (e.g. nonlinear ﬁltration, restoration, reconstruction etc.) as well. Since many researchers still use some metrics developed for greyscale images, there is a need of developing some good colour image quality metrics, which can be fast calculated and should be well correlated with the subjective evaluation of images. It is well known that some traditional image metrics such as the Mean Square Error (MSE), the Peak Signal to Noise Ratio (PSNR) and many similar ones [4,5] are poorly correlated with the Human Visual System (HVS) and the perception of many types of distortions. Regardless of the problems with various dynamic ranges and results not necessarily corresponding with the subjective evaluation, many researchers still use them, probably mainly because of their low computational complexity. Another popular metric is the Normalised Colour Diﬀerence L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 55–65, 2009. c Springer-Verlag Berlin Heidelberg 2009

56

K. Okarma

(NCD) deﬁned in the CIE Lab or CIE Luv colour spaces [20] as the typical method of colour image quality assessment. Apart from the MSE-based metrics there are also some interesting blind (noreference) quality measures [7]. Unfortunately their universality is rather poor since they are specialised for the detection of one or two types of distortions e.g. block eﬀects on JPEG compression [2,9,16,18] or image blurring [8,10]. A similar situation takes place in the development of reduced-reference metrics [3,19] which use only a partial information from the original image. Although they seem to be very promising, the progress in that ﬁeld during last several years is unsatisfactory. An ideal image quality metric should be independent on the image content and its dynamic range should be from 0 to 1 (or from −1 to 1 as in the SSIM). The universal character of such measure can be ensured currently only by the full-reference techniques. Low computational complexity and a good correlation with subjective evaluations are also desired. The correlation of results obtained using such a measure with the results of subjective evaluations accomplished by human observers should be preferably linear.

2

Modern Full-Reference Image Quality Metrics

Considering all the problems with the traditional image quality assessment methods some other full-reference approaches have been proposed. Some of them are sensitive only to some chosen types of distortions e.g. JPEG compression and Gaussian noise [1] or changes of the luminance and the impulse noise [14]. Some other ones, based on the Singular Value Decomposition [12,13] or transforms, such as DWT, DCT and DFT [6], can be treated as more universal but their weak side is relatively high computational complexity. Nevertheless, the approach based on the SVD is also considered in this paper. A milestone in the modern digital image quality assessment has been the development of two new universal full-reference image quality metrics: the Universal Image Quality Index [15] and the Structural Similarity [17] as the further extension of that method. The local Universal Image Quality Index for M × N pixels window is deﬁned as: Q=

(σx2

+

4σxy x ¯y¯ σxy 2¯ xy¯ 2σx σy = · · 2 , 2 2 2 2 · [(¯ x) + (¯ y) ] σx σy (¯ x) + (¯ y ) σx + σy2

σy2 )

(1)

where x and y denote the original and distorted image respectively and 1 = (xij − x¯)2 M · N − 1 i=1 j=1 M

M N 1 x ¯= xij M · N i=1 j=1 M N 1 y¯ = yij M · N i=1 j=1

σx2

1 = (yij − y¯)2 M · N − 1 i=1 j=1 M

σy2

N

1 (xij − x ¯)(yij − y¯) . M · N − 1 i=1 j=1 M

σxy =

N

N

(2)

Colour Image Quality Assessment Using SSIM Index and SVD

57

The overall quality index is calculated as the mean value from the whole quality map being the result of using sliding 8 × 8 pixels rectangular window for the whole image. Such universal measure is sensitive to three common types of distortions: the loss of correlation, luminance distortions and the loss of contrast. The extension of the UIQI into the Structural Similarity has been caused by the possibility of division by zero especially for dark large regions of the same colour. It has been protected by the additional stability constants, so the formula (1) has changed into the following expression: SSIM =

(2 · x ¯ · y¯ + C1 ) · (2 · σxy + C2 ) , (σx2 + σy2 + C1 ) · [(¯ x)2 + (¯ y )2 + C2 ]

(3)

where C1 and C2 are chosen in the way that they do not introduce signiﬁcant changes of the results (C1 = (0.01 × 255)2 and C2 = (0.03 × 255)2 are the values suggested by the authors of the paper [17]). Another change is related to the sliding window shape and size - 11×11 pixels Gaussian window has been applied. The application of the Singular Value Decomposition in the image quality assessment is related to the calculation of the singular values of the matrix corresponding to the image block (size of 8 × 8 pixels is suggested in the papers [12,13]). For each block the following value is computed : N 2 Di = (sk − sˆk ) , (4) k=1

where sk and sˆk denote the singular values for original and distorted image blocks. The quality metric is deﬁned as : K

MSVD =

i=1

|Di − Dmid | K

,

(5)

where K stands for the total number of N × N pixels blocks and Dmid is the middle element of the sorted vector D. As the result the reduced resolution quality map of the image can be obtained and after averaging its values the overall scalar quality index is obtained. The reduction of the quality map’s resolution is related to the usage of blocks in contrast to the sliding window approach used during calculations of the SSIM index.

3

Colour Information in Digital Image Quality Assessment

Regardless of the advantages of many modern image quality assessment methods, their authors usually do not take care of colour information. Such an attitude

58

K. Okarma

is not based on any particular arguments so there is a need to investigate how can colour information inﬂuence on the image quality. Many researchers take advantage of the LIVE database [11] containing the Diﬀerential Mean Opinion Score values for nearly 1000 images with ﬁve types of distortions: JPEG2000 and JPEG compression with various compression ratios, images contaminated by white Gaussian noise, Gaussian blurred images and JPEG2000 compressed images transmitted over simulated fast fading Rayleigh channel with bit errors typical for the wireless transmission. Although all the images are 24-bit colour ones, many researchers treat them as greyscale ones analysing the luminance channel only or converting them before the analysis. Considering the fact that many colour image processing algorithms (e.g. median ﬁltering) are not the straightforward extensions of their greyscale equivalents, the quality assessment of their results should be performed in view of colour speciﬁc aspects of a subjective evaluation. 3.1

Details of the Calculations and Results

In this paper the evaluation of the SSIM, MSVD and the NCD metrics for the quality assessment of images from the LIVE database is performed. The NCD metric has been calculated in the CIE Lab colour space according to: M N 2 2 2 Lij − Lij + aij − aij + bij − bij

N CD =

i=1 j=1

M N 2 2 2 (Lij ) + (aij ) + (bij )

(6)

i=1 j=1

where L,a,b denote the CIE Lab channels of the original image and L ,a ,b mean the same for the distorted one. The measure is based on the perceptually uniform character of the CIE Lab colour space (some authors use also the CIE Luv colour space) so identical change of the NCD value should indicate the same perceptual impression. The Structural Similarity and the MSVD metrics have been applied for each of the RGB channels as well as for the HSV and the CIE Lab channels. Additionally, the average values for the RGB and the weighted average according to the luminance in the YUV and YIQ colour models has been used for the comparison. After the elimination of the results for the original images present in the database, the absolute values of the Pearson’s correlation coeﬃcients between the DMOS and each metric have been computed (using Matlab’s corrcoef function) for each type of distortion as well as for the whole database (excluding the original images). Obtained results are presented in the Table 1 and illustrated in Figs 1 – 3. 3.2

Analysis of Results

Analysing obtained results it can be stated that the SSIM metric outperforms the NCD approach in most cases, except for the hue and the CIE Lab chrominances.

Colour Image Quality Assessment Using SSIM Index and SVD

Fig. 1. The SSIM metrics in the RGB and HSV colour spaces vs. DMOS

59

60

K. Okarma

Fig. 2. The NCD, SSIM and MSVD metrics in the CIE Lab colour space vs. DMOS

Colour Image Quality Assessment Using SSIM Index and SVD

Fig. 3. The MSVD metrics in the RGB and HSV colour spaces vs. DMOS

61

62

K. Okarma

Table 1. The absolute values of the linear correlation coeﬃcients between the DMOS and SSIM for various channels compared to the NCD and MSVD Quality JPEG2000 metric NCD (CIE Lab) 0.6747 SSIM (Red) 0.9002 SSIM (Green) 0.8993 SSIM (Blue) 0.9020 SSIM (Hue) 0.3787 SSIM (Saturation) 0.8097 SSIM (Value) 0.9010 SSIM (CIE Lab L) 0.8070 SSIM (CIE Lab a) 0.6705 SSIM (CIE Lab b) 0.7316 SSIM (YUV/YIQ Y) 0.8974 SSIM (R+G+B)/3 0.8977 MSVD (Red) 0.8705 MSVD (Green) 0.8617 MSVD (Blue) 0.8654 MSVD (Hue) 0.4859 MSVD (Saturation) 0.7537 MSVD (Value) 0.8699 MSVD (CIE Lab L) 0.8014 MSVD (CIE Lab a) 0.6929 MSVD (CIE Lab b) 0.7691 MSVD (YUV/YIQ Y) 0.8643 MSVD (R+G+B)/3 0.8511

JPEG 0.6597 0.8596 0.8206 0.8696 0.6690 0.8551 0.8607 0.7782 0.5786 0.6683 0.8503 0.8532 0.8290 0.8492 0.8164 0.7070 0.7271 0.8288 0.7284 0.7129 0.7390 0.8476 0.7978

White Gaussian Fast fading noise blur Rayleigh 0.8462 0.5853 0.7444 0.9472 0.8498 0.9006 0.9472 0.8554 0.9043 0.9452 0.8518 0.8974 0.7548 0.5506 0.4944 0.8831 0.8025 0.8214 0.9569 0.8543 0.9009 0.9195 0.6520 0.8386 0.8935 0.4249 0.6623 0.9013 0.5421 0.7477 0.9644 0.8486 0.9008 0.9681 0.8457 0.9008 0.9190 0.8057 0.8721 0.9192 0.7997 0.8688 0.9214 0.7858 0.8307 0.1847 0.4239 0.3693 0.7514 0.5103 0.6582 0.8661 0.8096 0.8827 0.8477 0.7615 0.8518 0.3333 0.7289 0.8102 0.4741 0.7491 0.8454 0.8724 0.7975 0.8715 0.8645 0.7256 0.8545

All 0.5213 0.6874 0.6793 0.6951 0.3561 0.6108 0.7208 0.6131 0.4779 0.5352 0.7364 0.7575 0.7900 0.7676 0.7812 0.3759 0.5423 0.5946 0.7368 0.6146 0.6872 0.7457 0.7538

The best results have been obtained for the value (brightness) in the HSV (HSB) colour model, which seems to be the most universal channel for the quality assessment of colour images using the SSIM approach. Although for the whole database better results have been obtained for the luminance (Y) and the RGB average, it is worth noticing that for four from ﬁve types of distortions analysed separately the value (brightness) leads to a better correlation with the DMOS values. Relatively low values of the correlation coeﬃcients are caused by their linearity. Adding the nonlinear mapping e.g. according to the logistic function would increase the correlation to over 90% but the goal of the paper is to ﬁnd the metric with possibly high linear correlation with the DMOS values for various types of distortions. It can be easily observed in the presented ﬁgures that the NCD and both SSIM and MSVD using hue component are poorly correlated with the DMOS as the cloud of results is relatively widely distributed. An interesting phenomenon can be observed for the MSVD metric. The application of the CIE Lab luminance channel causes much better correlation with the DMOS than for the SSIM index but the value/brightness performs much worse for the whole database (without the original images). Nevertheless, it is worth mentioning that the usage of that channel leads to much better linear

Colour Image Quality Assessment Using SSIM Index and SVD

63

Fig. 4. Example images with the SSIM values calculated in the HSV colour space well and poor correlated with the DMOS

correlation with the DMOS values for the separate sets of images with a speciﬁed type of distortions regardless of the type. In such case only the nonlinear correlation would be high for the whole database (obtained coeﬃcients of the model diﬀer seriously for each type of distortion so their nonlinear combination is needed to obtain a good correlation for the whole set of images). Another interesting feature is much better correlation of the MSVD obtained for each of the RGB channels in comparison to the SSIM index but for the luminance (Y) and the RGB average the results are similar for both metrics. For a better illustration of the relevance of the correlation between quality measures and the subjective evaluations the example images compressed using JPEG algorithm are shown in Fig. 4 with the reference images on the left. For the top image the DMOS value from the LIVE database is 57.2497, indicating very poor quality, while for the bottom one it is equal to 32.4894. The diﬀerences in their quality can be easily observed but the SSIM values calculated in the HSV colour space for the hue channel are equal to 0.2266 and 0.2633 respectively, so they indicate similar quality. The usage of the value/brightness for the calculations leads to SSIM equal to 0.4653 and 0.8442. Actually such results are much better correlated with the DMOS values as it can also be observed for the other images from the LIVE database.

64

K. Okarma

4

Conclusions

Presented results can be a good impression for some other researchers for the possible improvement of some other image quality assessment methods towards better utilisation of colour information from the digital images. Applying some other distortions such as the contamination by an impulse noise, median and lowpass ﬁltering etc. requires the extension of the database of the DMOS values so some additional subjective evaluations would be necessary in that case. The cost of the usage of the multichannel analysis of the digital images is the increase of the overall computational complexity and the main advantage of proposed approach is the improvement of the quality assessment of the images with some colour distortions which are clearly visible in the full-colour image but can be hard to notice in the luminance channel.

References 1. Beghdadi, A., Pesquet-Popescu, B.: A New Image Distortion Measure Based on Wavelet Decomposition. In: Proc. 7th Int. Symp. Signal Processing and Its Applications, Paris, France, pp. 485–488 (2003) 2. Bovik, A., Liu, S.: DCT-domain Blind Measurement of Blocking Artifacts in DCTCoded Images. In: Proc. Int. Conf. Acoustics, Speech and Signal Processing, Salt Lake City, USA, pp. 1725–1728 (2001) 3. Carnec, M., Le Callet, P., Barba, P.: An Image Quality Assessment Method Based on Perception of Structural Information. In: Proc. Int. Conf. Image Processing, Barcelona, Spain, vol. 2, pp. 185–188 (2003) 4. Eskicioglu, A., Fisher, P., Chen, S.: Image Quality Measures and Their Performance. IEEE Trans. Comm. 43(12), 2959–2965 (1995) 5. Eskicioglu, A.: Quality Measurement for Monochrome Compressed Images in the Past 25 Years. In: Proc. IEEE Int. Conf. Acoust. Speech Signal Process, Istanbul, Turkey, pp. 1907–1910 (2000) 6. Girshtel, E., Slobodyan, V., Weissman, J., Eskicioglu, A.: Comparison of Three Full–Reference Color Image Quality Measures. In: Proc. SPIE of 18th IS&T/SPIE Annual Symposium on Electronic Imaging, Image Quality and System Performance, San Jose, CA, vol. 6059 (2006) doi:10.1117/12.644226 7. Li, X.: Blind Image Quality Assessment. In: Proc. IEEE Int. Conf. Image Proc., pp. 449–452 (2002) 8. Marziliano, P., Dufaux, F., Winkler, S., Ebrahimi, T.: A No-Reference Perceptual Blur Metric. In: Proc. IEEE Int. Conf. Image Processing, Rochester, USA, pp. 57–60 (2002) 9. Meesters, L., Martens, J.-B.: A Single-Ended Blockiness Measure for JPEG-Coded Images. Signal Processing 82(3), 369–387 (2002) 10. Ong, E.-P., Lin, L.W., Yang, Z., Yao, S., Pan, F., Jiang, L., Moschetti, F.: A NoReference Quality Metric for Measuring Image Blur. In: Proc. 7th Int. Symp. Signal Processing and Its Applications, Paris, France, pp. 469–472 (2003) 11. Sheikh, H.R., Wang, Z., Cormack, L., Bovik, A.C.: LIVE Image Quality Assessment Database Release 2, http://live.ece.utexas.edu/research/quality

Colour Image Quality Assessment Using SSIM Index and SVD

65

12. Shnayderman, A., Gusev, A., Eskicioglu, A.: A Multidimensional Image Quality Measure Using Singular Value Decomposition. Proc. SPIE Image Quality and Syst. Perf. 5294(1), 82–92 (2003) 13. Shnayderman, A., Gusev, A., Eskicioglu, A.: An SVD-Based Gray-Scale Image Quality Measure for Local and Global Assessment. IEEE Trans. Image Processing 15(2), 422–429 (2006) 14. Van der Weken, D., Nachtegael, M., Kerre, E.: A New Similarity Measure for Image Processing. Journal of Computational Methods in Sciences and Engineering 3(2), 209–222 (2003) 15. Wang, Z., Bovik, A.: A Universal Image Quality Index. IEEE Signal Processing Letters 9(3), 81–84 (2002) 16. Wang, Z., Bovik, A., Evans, B.: Blind Measurement of Blocking Artifacts in Images. In: Proc. IEEE Int. Conf. Image Processing, pp. 981–984 (2000) 17. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image Quality Assessment: From Error Measurement to Structural Similarity. IEEE Trans. Image Processing 13(4), 600–612 (2004) 18. Wang, Z., Sheikh, H., Bovik, A.: No-Reference Perceptual Quality Assessment of JPEG Compressed Images. In: Proc. IEEE Int. Conf. Image Processing, Rochester, USA, pp. 477–480 (2002) 19. Wang, Z., Simoncelli, E.: Reduced-Reference Image Quality Assessment using a Wavelet-Domain Natural Image Statistic Model. In: Proc. Human Vision and Electronic Imaging Conference, Proceedings of SPIE, San Jose, USA, vol. 5666, pp. 149–159 (2005) 20. CIE Recommendations on Uniform Colour Spaces, Colour-diﬀerence Equations, and Psychometric Colour Terms, Supplement No. 2 to Publication CIE No. 15, Colorimetry (E-1.3.1) 1971, Bureau Central de la CIE, Paris (1978)

Top-Down Approach to Image Similarity Measures Jacek Piotrowski Institute of Computer Science, Computer Graphics Laboratory Warsaw University of Technology Nowowiejska 15/19, 00-665 Warsaw, Poland [email protected]

Abstract. We propose a method of veriﬁcation whether it is possible to measure image similarity by constructing a vector of metrics, regardless of what low-level features were extracted from images. We also present an on-line system which will be used to gather a dataset required to conduct the proposed experiment. Keywords: image similarity, metrics.

1

Introduction

One of the popular approaches to Content-based Image Retrieval (CBIR) [1,2,3] is construction of various image similarity measures (or rather metrics, to be strict with mathematical terminology). Unfortunately these measures often do not have all of the properties required by the mathematical deﬁnition of a metric. In the classical approach such image similarity measures are constructed using bottom-up approach on top of arbitrarily chosen low-level image features and then they are evaluated against human perception. Such approach allows us to create measures that perform better and better, yet it does not produce any feedback whether the path we are following leads to the goal nor how close to the solution we are. Also it is not clear whether it is possible to construct the ideal image similarity measure at all. Examples of similarity measures can be found in [4] and [5]. Very good examples of problems in ﬁnding similarity measure that resembles human perception were presented in [6] and [7]. Once we can measure the similarity we can easily create a CBIR system [8]. Also we can use kernel based machine learning methods for image classiﬁcation. Finally such measures are also helpful in propagation of annotations between similar images.

2

The Ideal Image Similarity Measure

Lets assume we have a set of images I (we will denote its elements as Ix , Iy , Iz and so on). If we wanted to deﬁne the ideal image similarity measure d(Ix , Iy ) L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 66–69, 2009. c Springer-Verlag Berlin Heidelberg 2009

Top-Down Approach to Image Similarity Measures

67

(or rather a dissimilarity measure, as the more similar images are, the lower its value), we would describe it in mathematical terms as a metric on I. Therefore it would have to meet following conditions: d(Ix , Iy ) = 0 ⇐⇒ Ix = Iy ,

(1)

d(Ix , Iy ) = d(Iy , Ix ) , d(Ix , Iz ) ≤ d(Ix , Iy ) + d(Iy , Iz ) .

(2) (3)

That is, it would have to have value 0 only for identical images, be symmetrical and satisfy the triangle inequality. These conditions imply that for all pairs of diﬀerent images d(Ix , Iy ) would have a positive value: d(Ix , Iy ) > 0 ⇐⇒ Ix = Iy .

(4)

Additionally we can assume, without any impact on topology of I, that d(Ix , Iy ) < 1 .

(5)

We obtain a function d : I × I → [0, 1) that maps a pair of images to a real value, but humans use many diﬀerent criteria for image comparison and we can view these criteria as various image dimensions. Therefore our metric reduces high-dimensional space into a scalar. Experiences with kernel-based machine learning algorithms teach us that this will not aid the learning process. Instead we have to increase dimensionality or at least preserve it, so rather than a single metric we can deﬁne a vector of metrics d(Ix , Iy ) where each element would meet the conditions (1)–(5). Then each element of that vector would represent diﬀerent aspect of similarity between two images. The problem is that we do not know the length of d(Ix , Iy ) nor we do not know if it is possible at all to construct such vector of metrics. Balcan et al. show in [9] that for machine learning we can use less strict conditions to achieve satisfactory results. Nevertheless to answer these two questions we can perform an experiment.

3

Proposition of an Experiment

To see if can create a vector of metrics described in previous section, we can solve the following optimisation problem for diﬀerent d(Ix , Iy ) lengths: 2 min dij − dˆij , (6) dij

i,j∈{0,...,n}

subject to four sets of constraints (for brevity we have used the element-byelement style vector operators): dii = 0

∀i ∈ {0, . . . , n} ,

dij > 0 ∀i, j ∈ {0, . . . , n}, i =j , dij = dji ∀i, j ∈ {0, . . . , n} , dik ≤ dij + djk

∀i, j, k ∈ {0, . . . , n}

(7) (8) (9) (10)

68

J. Piotrowski

and one set of bounds: 0 ≤ dij < 1 .

(11)

where dij = d(Ii , Ij ), n = |I| and dˆij is the similarity score assigned by humans. The goal function measures the error. If we can ﬁnd a solution for given length of d(Ix , Iy ) where this error is close to 0, it means that it is possible to obtain such a vector of metrics for given set of images. The next step woulb be to ﬁnd a way to derive values of those metrics from images. For example we could use neural networks for that, but there is no guarantee that values generated by these networks will hold all metric properties for other sets of images. However if there is no solution with low error value, it means that whatever method we will use to construct the metrics, they will behave at least equally bad, so we will get the upper bound for image similarity measure performance. Of course before solving the above problem we can reduce the number of variables by over 50 % using constraints (7) and (9). Also for technical reasons we have to replace strict inequalities with non-strict ones in (8) and (11), but this will not aﬀect the results signiﬁcantly. Alternatively we can solve the dual problem, that is to ﬁnd a set of values that match the experimental data exactly and approximate metric properties as well as it possible. There are two elements that have not yet been deﬁned. The . norm used in goal function and dˆij values. The norm transforms a vector of distances into a single value that can be compared with human responses. Whether it should be an Euclidean norm or other one has to be checked in an experimental way. The dˆij values represent data gathered in an experiment where a number of people had a chance to compare two images and express their similarity with a numeric value.

4

Data for the Experiment

To gather enough data to conduct the experiment proposed in the previous section, we have designed a system that collects information about image similarity through a web page [10]. This way any person in the world can take part in building the data set. This is important because even for a mere 100 images we need nearly 5000 responses to cover each pair once and want more than one response, so the dataset represents typical human perception and is not biased towards a single person. The system was implemented using Ruby on Rails framework and currently it is being prepared for deployment at http://cbir.ii.pw.edu.pl. The images will be collected from the ﬂickr.com site, but only those with proper Creative Commons licences to ensure that the dataset can be freely distributed.

5

Conclusions and Future Work

We have proposed a method for veriﬁcation whether a set of similarity metrics can be found for a given set of images. Using this method we can conclude

Top-Down Approach to Image Similarity Measures

69

whether it is possible to ﬁnd an ideal way, from a mathematical point of view, for measuring image similarity. If results of the experiment show that it is not possible, we will have to establish a new goal by relaxing some of the metric properties, without making the similarity function inapplicable to machine learning algorithms. As soon as the dataset is completed we will conduct the proposed experiment and also explore the possibilities of approximation of obtained metric values.

References 1. Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1349–1380 (2000) 2. Kherﬁ, M.L., Ziou, D., Bernardi, A.: Image retrieval from the world wide web: Issues, techniques, and systems. ACM Computing Surveys 36(1), 35–67 (2004) 3. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, inﬂuences, and trends of the new age. ACM Computing Surveys 40(2), 1–60 (2008) 4. Balmashnova, E., Florack, L.M.J.: Novel similarity measures for diﬀerential invariant descriptors for generic object retrieval. Journal of Mathematical Imaging and Vision 31(2–3), 121–132 (2008) 5. Guo-Dong, G., Jain, A.K., Wei-Ying, M., Hong-Jiang, Z.: Learning similarity measure for natural image retrieval with relevance feedback. IEEE Transactions on Neural Networks 13(4), 811–820 (2002) 6. Santini, S., Jain, R.: The graphical speciﬁcation of similarity queries. Journal of Visual Languages and Computing 7(4), 403–421 (1996) 7. Jacobs, D.W., Weinshall, D., Gdalyahu, Y.: Classiﬁcation with nonmetric distances: Image retrieval and class representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(6), 583–600 (2000) 8. Vleugels, J., Veltkamp, R.C.: Eﬃcient image retrieval through vantage objects. Pattern Recognition 35(1), 69–80 (2002) 9. Balcan, M.F., Blum, A., Srebro, N.: A theory of learning with similarity functions. Machine Learning 72(1-2), 89–112 (2008) 10. Piotrowski, J.: On-line experimental setup for capturing image similarity data. In: III Konferencja Naukowo-Techniczna Doktorant´ ow i Mlodych Naukowc´ ow Mlodzi naukowcy wobec wyzwa´ n wsp´ olczesnej techniki, Warsaw University of Technology, September 22–24, 2008, pp. 363–367 (2008)

The Influence of Picture Quality Scale Measure on Choosing the Wavelet Transform during Image Compression Maria Skublewska-Paszkowska and Jakub Smolka Computer Science Institute Lublin University of Technology Nadbystrzycka 36 b, 20-618 Lublin, Poland [email protected]

Abstract. One important issue of the image compression is the quality of the resultant image. The Picture Quality Scale Measure (PQS) was used to improve this quality. This perceptual measure evaluates the visual distortions in color compressed image on the basis of the original and the resultant ones. This measure uses also a group of observers who are needed to evaluate the distortions. The implementation of PQS measure together with neural network enabled the independence of human observers. It was also possible to use this measure to inﬂuence the quality during the process of image compression. Each level of the wavelet transform was compressed with one wavelet function which caused the least visual distortions which were computed by Picture Quality Scale. The image was compressed with diﬀerent transforms. The choice of a particular transform was taken on the ground of the PQS value. There were ﬁve diﬀerent wavelet transforms to choose: Haar, Daubechies degree 4, Daubechies degree 6, Daubechies degree 8 and 5/3 transform. The image was coded with Embedded Zerotree Wavelet Coding (EZW). Keywords: wavelet compression, wavelet transform, Picture Quality Scale Measure.

1

Introduction

Nowadays the quality of the compressed image is a very important issue. The level of visual distortions and the image quality can be computed by Picture Quality Scale measure. However, it needs a group of observers to calculate the distortions value. The Picture Quality Scale was implemented together with neural network which opted out of the observers. That enabled both to evaluate visual distortions without humans and to use it during the process of image compression. The implementation ensured the good quality of the resultant image.

2

Wavelet Transforms

Wavelet transform is an operation that converts the original image into wavelet coeﬃcients in the way that almost the whole image energy is cumulated in only L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 70–79, 2009. c Springer-Verlag Berlin Heidelberg 2009

The Inﬂuence of PQS Measure on Choosing the Wavelet Transform

71

a few of them. The more coeﬃcients are near zero, the less data to decode and the better the compression rate is achieved. The choice of wavelet transform inﬂuences also the quality of the resultant image. The image transformation divides the original picture into the coarse subimage that reﬂect the averaged image information and three detailed subimages computed on the j th resolution level. The coarse subimage is decreased and smoothed version of the original image. ˜ and highpass (˜ In image wavelet compression the downpass (h) g) ﬁlters are used. Their coeﬃcients are tightly connected with wavelet and scaling functions (3, 4). The set of wavelet coeﬃcients, which are computed during the transformation process, are obtained by the discrete decomposition the original signal f (n) into (j) (j) the coarse (fn ) and detailed (dn ) images. It can be described as in equations (1) and (2) [1]. ˜ 2n−l f (j−1) fn(j) = h (1) l l

d(j) n

=

l

(j−1)

g˜2n−l fl

1 t √ φ = h˜n φ(t − n) 2 2 n∈Z 1 t √ ψ = g˜n ψ(t − n) 2 2

(2)

(3)

(4)

n∈Z

The inverse wavelet transform computes the image using the wavelet coeﬃcients (5) [1]. (j) (j) − fn(j−1) = h+ + gl−2n dl (5) l−2n fl l

l

The ﬁlters h+ and g − are deﬁned as (6, 4) [1]. √ + φ(t) = 2 hk φ(2t − k)

(6)

√ − 2 gk ψ(2t − k)

(7)

k

ψ(t) =

k

The image can be treated as a two dimensional signal. That is why the transformation is computed ﬁrstly on rows and then on columns of the original picture. In inverse transform this sequence is reversed. This solution is called Mallat’s algorithm which is presented in Fig.1. Mallat three level decomposition is presented in Fig.2. Four diﬀerent subimages are created through the application of various types of ﬁlters. The ↓ 2 parameter represents the reduction of the resolution. The type of used wavelet transform inﬂuences the quality of the compressed image.

72

M. Skublewska-Paszkowska and J. Smolka

Fig. 1. Mallat Algorithm [1]

Fig. 2. Three level wavelet transformation [6]

3

Picture Quality Scale

The Picture Quality Scale (PQS) [4] [5] [9] is an objective measure that is used to evaluate perceptual distortions in compressed image in comparison to the original one. The PQS for color images with neural network schema is presented in Fig. 3 [4] [5] [9] [10]. It involves the Human Visual System features, such as: Weber-Fechner Law or visual masking, and also subjective Mean Opinion Score (MOS) measure [9]. Very important issue in compressed color images is taken into the account the information about the color, how and if it has changed. The diﬀerence between these colors can be computed using SCIE Lab space (8) [10]. Matrix e stands for the color diﬀerence between the original and the compressed images, where ΔL, Δa, Δb mean the diﬀerence between the images in the appropriate color spaces. e[m, n] = (ΔL[m, n])2 + (Δa[m, n])2 + (Δb[m, n])2 (8) In PQS measure the diﬀerent local distortion maps f1 , f2 , f3 , f4 in compressed image are calculated on the ground of the color diﬀerence and the luminance of the compressed image. They are used to gain distortion vector F ={F1 , F2 , F3 , F4}.

The Inﬂuence of PQS Measure on Choosing the Wavelet Transform

73

Fig. 3. The PQS schema for color images with neural network

This vector is necessary to compute the covariance matrix and then both the eigenvalues and eigenvector of values are calculated. The distortion parameters Z = {Z1 , Z2 , Z3 , Z4 } are acquired from eigenvector and distortion vector. To point out the PQS value two images are necessary: the original one xo and the compressed one xr . The matrix f1 (9) [5] [9] reﬂects the distortions between the original and the compressed images which are adapted by Werner Fechner law [5] and spatial domain counterpart of the frequency response Sa (ω) [5]. Additionally, this distortion cuts the values which are below thresholds T by an indicator function IT [5] [9]. f1 [m, n] = IT (e[m, n] ∗ Sa [m, n])2 (9) The matrix f2 (10) [5] [9] illustrates the diﬀerences in linear elements of images, especially the end of block disturbances [5] [9]. This matrix is made of two components that are the errors in vertical and horizontal block discontinuities. The equations for vertical matrix are analogical to equations for horizontal matrix (10, 11) [5] [9]. The matrix ew represents the convolution between the matrix e (8) and the modelled contrast sensitivity function (Sa ) [5] [9]. f2h [m, n] = IH [m, n]Δ2h [m, n]

(10)

Δh [m, n] = ew [m, n] − ew [m, n + 1]

(11)

In compressed images the correlation distortions can often be seen. They can be present in texture, image features and blocking eﬀects [5] [9]. That is why the local distortion map f3 (12) [5] [9] is created with the use of local spatial correlations. f3 [m, n] = |r(m, n, k, l)|0.25 (12) (k,l)∈W,l=k=0

r(m, n, k, l)] =

1 W

i,j

ew [i, j]ew [i + k, j + l]−

1 ew [m, n ew [i + k, j + l] W i,j i,j

(13) Parameter W represents the number of elements in the window of ﬁve by ﬁve items. Additionally and must be fulﬁlled |l| ≤ 2 and |k| ≤ 2 [5].

74

M. Skublewska-Paszkowska and J. Smolka

The last distortion that is taken into the account in PQS measure is the vicinity of high contrast image transitions [5] [9]. The two Human Visual System features are used in that item: visual masking and enhanced visibility of misalignment. The f4 distortion is deﬁned as [5] [9]: f4 [m, n] = IM [m, n]|ew [m, n]|(Sh [m, n] + Sv [m, n])

(14)

The IM is an indicator function that selects only the items close to high intensity transitions. The Sh and Sv stand for horizontal and vertical masking factors respectively (15) [5] [9]. Sh [m, n] = e−0.04Vh [m,n] Vh [m, n] =

|xo [m, n − 1] − xo [m, n + 1]| 2

(15) (16)

The distortion factors are based on the local distortion maps (from f1 to f4 ). The ﬁrst distortion is computed as the sum of all matrix f1 elements divided by the sum of squared values of compressed image (17) [5] [9]. f1 [m, n] F1 = m n 2 (17) m n xr [m, n] The second factor is the square of two elements: the block error discontinuities at vertical F3v and horizontal F3h block boundaries (18) [5] [9]. 2 + F2 F2 = F2h (18) 2v F2h =

1 f2h [m, n] Nh m n

(19)

F2v =

1 f2v [m, n] Nv m n

(20)

The parameters Nh and Nv stand for the number of items pointed out by an indicator function. The third factor is the sum of items of matrix f3 divided by the size of images, where N represents the number of columns and the M - number of rows (21) [5] [9]. 1 F3 = f3 [m, n] (21) MN m n The last factor is the sum of all elements of matrix f4 divided by the parameter that represents the number of points, whose values after Kirsch operator, are greater or equal to 400 (22) [5] [9]. F4 =

1 f4 [m, n] Nk m n

(22)

The Inﬂuence of PQS Measure on Choosing the Wavelet Transform

75

On the base of the computing distortion factors the covariance matrix is computed and normalised (23) [2] [5] [9] so that the values belong to the [0, 1] interval. |fi [m, n] − fi | − m n |fj [m, n] − fj | COV [fi , fj ] = m n 2 2 m n (fi [m, n] − fi ) m n (fj [m, n] − fj )

(23)

The parameter fi represents the average value of matrix fi . On the base of covariance matrix the eigenvalues and eigenvector are computed [5]. The vector Z is calculated as the multiplication of covariance matrix by factor values. The PQS value should be similar to the Mean Opinion Score (MOS) [5] [9]. The last one measure is based on the assessments of visual distortions in images by the group of observers (24) where M represents the number of observers and the A(i, k) stands for the assessment of ith observers of k th image. The observers are obliged to adjust the assessment value to the image distortions so that they can be noticed in ﬁve scale measure from 1 to 5. If the distortions in compressed image are invisible in contrary to the original one, the assessment value should be 5. If the distortions are slightly visible, the value should be 4. If the distortions are visible - value 3. If they are very visible - value 2 and when the information in the image is lost - value 1. M OS =

4

1 A(i, k) M

(24)

Neural Network

In order to make easier the implementation of Picture Quality Scale measure without the observers, the feed-forward neural network was adapted [7]. The network consists of three layers: the input layer, the hidden layer and the output layer. The ﬁrst layer is made of four neurons which all have four inputs and one output. The second layer is made of ten neurons all of which possess four inputs and one output. The last one has got only one neuron with ten inputs and one output that generates the output of the whole network which is the PQS value. All neurons in all layers have got the sigmoid transfer function. Picture Quality Scale together with neural network was implemented to opt out of the necessity of involving human observers to evaluate visual distortions in compressed images. From compressed images a set of test pictures was chosen to evaluate visual distortions with MOS measure. Each pair of the original and the compressed images was evaluated by ﬁfteen observers and then the MOS values were calculated. These data were necessary for the process of learning of the neural network. From the test images the Z parameters were computed which were the inputs of neural network in the process of learning. The output of neural network was the set of MOS values obtained from the group of observers. The learning set consisted of 120 pairs of input-output values. The training after learning the back-propagation network was gained at a level of 0.0095 which

76

M. Skublewska-Paszkowska and J. Smolka

is satisfactory. The neural network was used to assess the PQS value from the images compressed with Daubechies wavelet transforms. This kind of Picture Quality Scale implementation can be used not only to evaluate the visual distortions in compressed images without observers but also as a tool to aﬀect the quality of the resultant image.

5

Wavelet Image Compression Using Picture Quality Scale

Wavelet compression consists of several parts. One of them is the transformation of the whole image into the wavelet coeﬃcients which can be easily coded and decoded. This coeﬃcients can be divided into the coarse ones and the detailed ones. The ﬁst should contain almost whole image energy while the other only the information about image details. The wavelet transform decomposes the image into coeﬃcients on levels. Firstly, the whole image is transformed and a coarse subimage and three detailed subimages are obtained. The coarse image is smoothed and twice smaller than the original one. Secondly, the coarse subimage is transformed and the algorithm is continued until the coarse subimage is only one coeﬃcient. That is how the whole transformation of the image is being done. The idea of applying the Picture Quality Scale in image compression is to choose the best wavelet transforms on each level of decomposition. The scheme of adaptive transform is shown in Fig. 4.

Fig. 4. The scheme of the transformation the original image

Firstly, original image is transformed to the YUV space. Then the level compression is done. During the ﬁrst level decomposition the whole image is transformed with Haar wavelet function. Then the quantization process is applied and the inverse transform is computed. In other words the compressed image after 1 level is obtained. Using the resultant image and the original one the visual distortions in compressed image are evaluated using Picture Quality Scale measure with neural network. PQS value represents the level of disturbances that can be possessed by human being. After this assessment, the same original image was compressed with second type of wavelet function, Daubechies 4 degree, and the distortions of the resultant image were evaluated by PQS value, as well. This process was repeated until the whole wavelet functions were used. The next step

The Inﬂuence of PQS Measure on Choosing the Wavelet Transform

77

was to choose that wavelet transform which causes the least visual distortions that is for which the higher PQS value is gained. Secondly, the subimage is cut from the original picture which size is twice smaller than the original and the process of level compression is applied. After that the best wavelet transform can be chosen. The level compression ends when the coarse image was size of one pixel. For square pictures the number of levels was 9. There was a set of wavelet transforms such as: Haar, Daubechies 4, 6, 8 degree and 5/3 transform which were implemented during the level compression process in a way that should ensure the good quality of the resultants images. After this kind of compression the transformed image was got which was coded with EZW (Embedded Zerotree Wavelet Transform).

6

Conclusions

Four various colour images 512x512 pixels (Fig. 5) were compressed with diﬀerent wavelet transforms and the results were evaluated and compared with the quality of the resultant images transformed with the particular one kind of wavelets. The

(a)

(b)

(c) (d) Fig. 5. Original image a) Lena, b) Baboon, c) Peppers, d) Airplane

78

M. Skublewska-Paszkowska and J. Smolka

(a)

(b)

(c)

(d)

Fig. 6. Point charts represent PQS evaluations for 4 resultant images for various wavelet transforms: a) Lena: compression rate 5 b) Baboon: compression rate 6 c) Peppers: compression rate 34 d) Airplane: compression rate 25 Table 1. Chosen wavelet transforms for Baboon picture Level Wavelet transform 0 DB8 1 DB8 2 DB8 3 DB4 4 DB8 5 DB8 6 DB8 7 DB4 8 DB8

results are presented in Fig.6. During the compression the Embedded Zerotree Wavelet coding was used [3] [8]. The quality of the pictures compressed with adaptive wavelets is much better or almost the same than the compressed ones with only one kind of wavelet. The quality depends on the kind of images and also on the compression rate. The

The Inﬂuence of PQS Measure on Choosing the Wavelet Transform

79

bigger it is, the higher diﬀerence in the resultant images can be noticed and the quality of the compressed images rises. The worst quality was gained for Haar and 5/3 wavelet transform while the least visual distortions resulted in adaptive and Daubechies transforms. In Tab. 1 the chosen transforms during nine-level adaptive compression for Baboon image were presented. It can be easily seen that only Daubechies function for diﬀerent degree were chosen. As the results show, there is an opportunity of improvement the quality of the compressed pictures during the level wavelet transformation process.

References 1. Bialasiewicz, J.: Falki i aproksymacje. Wydawnictwo Naukowo-Techniczne, Warszawa (2000) 2. Cyganek B.: Three-dimmentional computer processing, EXIT (2002) 3. EZW encoding, Valens 1999-2004, http://pagesperso-orange.fr/polyvalens/clemens/ezw/ezw.html 4. Kotani, K., Gan, Q., Miyahara, M., Algazi, V.: Objective Picture Quality Scale for Color Image Coding. IEEE Image Processing 3 (October 1995) 5. Miyahara, M., Kotani, K., Ralph Algazi, V.: Objective Picture Quality Scale (PQS) for Image Coding. IEEE Transactions on Communications 46(9) (September 1998) 6. Salomon, D.: Data Compression. The complite reﬀerence, 3rd edn. Springer, Heidelberg (2004) 7. Tadeusiewicz, R., G¸aciarz, T., Borowik, B., Leper, B.: Reveal the neural network features using C# programs, Krak´ ow (2007) 8. Shapiro, J.M.: Embedded Image Coding Using Zerotrees of Wavelet Coeﬃcients. IEEE Transactions on signal processing 41(12) (December 1991) 9. Wu, H.R., Rao, K.R.: Digital Video Image Quality and Perceptual Coding. CRC Press, Taylor & Francis Group, London, New York (2006) 10. Zhang, X.: Wandell Brian A.: A Spatial Extension Of CIELAB For Digital Color Image Reproduction. Department of Psychology, Stanford University Stanford

Image Quality Assessment Using Phase Spectrum Correlation Przemyslaw Skurowski and Aleksandra Gruca Institute of Informatics, Silesian University of Technology Akademicka 16, 44-100 Gliwice, Poland {przemyslaw.skurowski,aleksandra.gruca}@polsl.pl

Abstract. This paper presents the method of evaluating the image quality using similarity of images phase spectrum. The authors introduce phase correlation coeﬃcient as an objective measure of an image quality index which is compared to the subjective distortion evaluation using stimulus impairment scale. Artiﬁcially distorted images produced by proportional mixing of images phase spectra with a noise were used for testing purposes. The results are the mean correlation coeﬃcient values related to the mean opinion score grades. The obtained relation between human responses and phase correlation is linear. Keywords: image quality assessment, phase spectrum.

1

Introduction

Image quality estimation methods are crucial for image compression and reproduction algorithms. Unfortunately, classical quality measures derived from electrical engineering are not correlated with human visual perception [1]. On the other hand modern solutions which are supposed to be correlated with human perception are nonlinearly related to human responses [2]. In the classic paper [3] Oppenheim and Lim demonstrate that most of signal structural information is included in phase spectrum. That fact especially refers to human perceivable media like sound or image. However, most of image distortion or quality measures consider mainly amplitude spectra using human related weighting function [4,5,6], other solutions are based on multiresolution approach [7] or sophisticated statistics [8]. Therefore measuring image similarity or distortion by phase comparing still remains promising ﬁeld of research [9]. In this paper authors use phase correlation which diﬀers from standard methods and has a desirable linear relation to human perception. The distortion of image using both types of scale - objective correlation between phase spectra and subjective so-called MOS (mean opinion score) is measured. As a ﬁnal result we obtained mean correlation coeﬃcient values representing every visual category from MOS scale. That ﬁlls the gap between the objective and subjective approach and allows designing image similarity measure. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 80–89, 2009. c Springer-Verlag Berlin Heidelberg 2009

Image Quality Assessment Using Phase Spectrum Correlation

2

81

Background

Fourier transform is a canonical tool [10] widely used in image processing and analysis. In analysis of Fourier spectra usually only its amplitude part is considered despite the fact that signiﬁcant role of the phase part was proven [3]. As it is shown in Fig. 1 demonstrating spectral mixing of images, contents is rather stored in phase than in amplitude spectrum.

Fig. 1. Example of spectral mixing of images ’lake’ and ’lena’. Annotations as in eq. 3.

2.1

Phase Spectra Mixing

Formally spectral mixing is given as the following sequence of mathematical operations. Let us consider operator F( ) as a discrete two dimensional Fourier transform: O(u, v) = F (o) =

M−1 −1 N

o(m, n)e−j(2πu/M )m e−j(2πv/N )n ,

(1)

m=0 n=0

which returns as a result complex spectra consisting of real and imaginary parts (spectra). Using Euler’s formula these parts can be also considered as an amplitude (A) and phase (θ) spectra given with formulas: A(u, v) = |O(u, v)| = re(O(u, v))2 + im(O(u, v))2 , im(O(u, v)) −1 θ(u, v) = tan (2) re(o(u, v))

82

P. Skurowski and A. Gruca

where: F( ) is Fourier transform; o( ), O( ) are image forms respectively spatial and spectral; M, N are spatial sizes; m, n are spatial coordinates; u, v are frequency coordinates. Using the above deﬁnitions we can obtain spectral mixing of two images by swapping of images phase and amplitude spectra as follows: O1 = F (o1 ), O2 = F (o2 ), R1 = A1 ejθ2 , R2 = A2 ejθ1 , r1 = F −1 (R1 ), r2 = F −1 (R2 ),

(3)

where: oi , ri - source and resulting images; Oi , Ri - complex spectra of source and resulting images; Ai , θi are amplitude and phase spectra of corresponding complex spectra Oi ; F−1 ( ) is inverse Fourier transform. 2.2

Proportional Spectral Phase Mixing

Fundamental concept of this paper is proportional spectral phase mixing. Our idea is based on proportional mixing of image phase spectrum with given proportions. Such a method allows to produce series of images (see Fig. 2) with monotonically changing spectral phase of image from ﬁrst to second one, keeping amplitude spectrum of ﬁrst one intact. Generally such process can be described in similar way to the previous one: O1 = F (o1 ), O2 = F (o2 ), θm = (1 − α)θ1 + αθ2 , R1 = A1 e

jθm

, r1 = F

−1

(4)

(R1 ),

where: oi , r1 are source and resulting image; Oi , R1 are complex spectra of source and resulting images; Ai , θi are amplitude and phase spectra of corresponding complex spectra Oi ; θm is mixed phase spectrum; F−1 ( ) is inverse F. transform; α - mixing proportion coeﬃcient varying in range [0, 1]. During preparation of the numerical procedure based on the above assumptions we encountered two problems. The ﬁrst was related to spectral symmetry. To obtain real numbers as a result of inverse Fourier transform it is necessary for the spectra to be even for its amplitude part (symmetric) and odd for the phase part (counter symmetric). Therefore, a matrix representing an image has to be conjugate symmetric - these conditions can be noted as: A(u) = A(−u),

θ(u) = −θ(−u).

(5)

In original images Nyquist frequencies represented by single point values were self conjugate (0 or π) , unfortunately as a result of proportional mixing the values no longer satisﬁed counter symmetry condition. This problem was encountered when the image size was an even number of points. For images with odd number of point in size, Nyquist frequencies were represented by double values what ensures fulﬁlling the condition above. Our solution was to add small margins to enforce the image size to be odd number.

Image Quality Assessment Using Phase Spectrum Correlation

83

While performing image proportional mixing with white noise realizations, we encountered another problem - intensive, ﬂare-like phenomenon in image corners. We investigated the problem and our analysis revealed that it is caused by random resulting values of phase spectra in low frequencies where large amplitudes for natural images are located. To assure that the test images are unbiased we introduced 10-points wide margins.

Fig. 2. Ramp of images produced by proportional mix of phase between o1 and o2

2.3

Similarity Measurement of Phase Spectrum

For the measurement of spectral phase similarity we decided to use one of the most canonical tools Pearson correlation coeﬃcient [11]: MN

rxy =

i=1

(xi − x)(yi − y)

(M N − 1)σx σy

,

(6)

where in our case: M, N - sizes of the images; xi , yi are corresponding phase spectrum values of original and distorted image; x, y, σx , σy are mean values and variances of phase within image. To verify whether the phase correlation can be used as an image distortion and similarity measure we recorded the correlation coeﬃcient values obtained for diﬀerent values of mixing proportion coeﬃcient (α = 0, . . . , 1). In the Figure 3 we present the results of source image (’baboon’) phase mixing with diﬀerent distortion sources. As a distortion source we used: two diﬀerent images (’woman1’

84

P. Skurowski and A. Gruca

and ’lake’), an original image (’baboon’) where o2 is processed o1 (with blur, USM sharpening, JPEG compression at 0.16 and 0.05 bits per pixel and halftoning) and white noise (averaged correlation coeﬃcients for 20 realizations).

Fig. 3. Correlation coeﬃcients between phase spectra of original image and resulting image obtained by proportional phase mixing between ’baboon’ and another image

On the basis of the results presented in the Fig. 3 we conclude, that phase correlation is a potentially good measure of image similarity or diﬀerence in image processing and communication. However it cannot be used as a measure of image content similarity. Our assumptions are based on the following: – at α = 1 (and roughly for all other proportion values) correlation coeﬃcients are related to each other according to intuitive order of quality: • USM ﬁltered image supersedes all other images (phase correlation close to 1) as it might be considered as image quality improvement • Moderately high JPEG compression (0.16 bpp) surpasses high JPEG compression (0.05bpp) and poor quality (low resolution) halftoning • All above exceeds quality of strong blur of image – We do not observe diﬀerences of phase correlation values recorded for source image mixed with: completely diﬀerent image (B-L), somewhat similar image (B-W) and with white noise. This fact implies that similarity of image contents cannot be distinguished using phase correlation.

3

Experiment

We decided to collect human responses about quality of images. For selected pool of images from the SIPI USC database [12] (’baboon’, ’couple’, ’crowd’, ’lake’, ’lax’, ’lena’, ’man’, ’milkdrop’, ’plane’, ’woman1’, ’woman2’) we generated 264 variants of distorted images (24 per image) which phase spectra was mixed with white noise

Image Quality Assessment Using Phase Spectrum Correlation

85

image phase spectra at varying coeﬃcient α from 0.025 to 0.6 and 0.025 step. The white noise image was chosen as a source of distortion for its spectral properties - all values at all frequencies are equally probable so every spectral component of original image has equal probability of being distorted. Such an approach excludes any preferences or intents in measurement results and interpretation. The images were presented to respondents to gather their opinions on image quality. 3.1

Measurement Method

Subjects opinions were collected in double stimulus impairment scale (DSIS) fashion. Respondents were asked to asses image quality in scale sometimes called MOS scale as it is being used to obtain mean opinion score (it is in fact the ﬁnal result of using quality grade scale). In a quality scale, image is described with one of the adjectives: bad, poor, fair, good and excellent. Each adjective is also coded with a number - from 1 to 5 respectively. Additionally the respondents were supported with impairment scale (see Table 1). Every processed image was demonstrated to the respondent with its original just about on the uniform 50% gray background on the TFT computer screen. Table 1. Table of adjective quality and impairment scales according to ITU [11,12] Numerical value Quality 1 bad 2 poor 3 fair 4 good 5 excellent

Impairment Very annoying Annoying Slightly annoying Perceptible but not annoying Imperceptible

During the test, the image quality evaluation procedure was performed as follows: at ﬁrst subject was given short instruction, next he or she judged 50 random images drawn from overall pool of 264 images. The images were presented to the respondent in random order. Users viewing point was also ﬁxed in the middle of the screen by head support that the observation distance was 0.5 meter. The pictures were presented to 26 persons (19 males and 7 females). Subjects vision defect, if any, were corrected as ordered by physician. Respondents were not involved in image processing research or industry.

4

Results

Results of the subjective analysis of the image quality with reference to the values of the Pearson correlation coeﬃcient assigned to the pictures are presented in the Fig. 4. Each small dot represents one image that was presented to respondent. The values on x axis are subjective scores of the image quality, and the values on the y axis are values of the Pearson correlation coeﬃcients. For each image (represented by the dot) one may easily obtain its Pearson correlation coeﬃcient value and the score given by the individual respondent.

86

P. Skurowski and A. Gruca 1

0.9

responses linear function fit mean correlation coefficients

0.8

0.7

0.6

correlation coefficient

0.5

0.4

0.3

0.2

0.1

1

2

3

4

scores

5

Fig. 4. Results of the subjective analysis of the image quality with reference to the values of the Pearson correlation coeﬃcient

For each group of the images that were given the same score by the respondents we computed the mean of the Pearson correlation coeﬃcients values. One may interpret this mean value as the objective representation of the subjective opinion of the respondents. The obtained mean values are presented in the table 2. Table 2. Mean values of the Pearson correlation coeﬃcients obtained for each grade Numerical value 1 2 3 4 5

Quality bad poor fair good excellent

mean correlation coeﬃcient 0.245 0.46 0.599 0.75 0.895

To obtain general relation between phase correlation coeﬃcient and user response we applied curve ﬁtting procedure. After review of commonly used functions (exponential, power and sigmoid) we decided to use a linear function as it proved to be the best curve representing experimental data. Wide range of the correlation coeﬃcient values assigned to each score by the respondents may be the reason of some questions. Is this dispersion caused by human subjective perception or by properties of speciﬁc images? Below, we present the analysis of the mean correlation values computed separately for individual images and for respondents. To verify, whether the features of the individual image inﬂuence the obtained mean correlation values, we separately computed mean correlations for each of the 11 images presented to the respondents. Fig. 5 presents results of such comparative analysis. Similar to the Fig. 4, the values on x axis are subjective scores of the image quality, and the values on the y axis are values of the Pearson

Image Quality Assessment Using Phase Spectrum Correlation

87

correlation coeﬃcient. Each point represents a mean value of the individual picture correlation coeﬃcients computed for the corresponding score value. We also applied curve ﬁtting procedure to the results obtained for each image. Goodness of ﬁt was measured with coeﬃcient of determination R2 [11] (see Table 3). We observed that a goodnesses for sigmoid, line and power functions were very close to each other. We also noticed that within a range of interest, sigmoid and power functions parameters resulted in curve elongation very close to the line. Therefore, according to the Ockham’s razor principle, we decided to choose line function as it is the simplest possible solution. 1.1

1

0.9

0.8

0.7

0.6

0.5 mean correlationbaboon correlation coefficient

0.4

fitted linebaboon mean correlationcrowd fitted linecrowd

0.3

mean correlationlake fitted linelake

0.2

mean correlationplane fitted lineplane

0.1

1

2

3

4

scores

5

Fig. 5. Mean correlation coeﬃcients for scores, recorded for four representative images

Due to the clarity the results are presented only for four representative images: crowd represents the image most sensitive to the phase distortion, baboon represents the least sensitive image, lake, that may be considered as a median, and a plane, which is one of the two images that are less demanding for the low scores, but very sensitive to the phase distortion for high scores. For graph readability only mean values of correlation coeﬃcients for speciﬁed class are presented. Table 3. Coeﬃcients of determination of used curve ﬁts for representative images baboon crowd lake plane

sigmoidal 0.8443 0.7880 0.7624 0.7398

linear exponential power 0.8398 0.8182 0.8383 0.7654 0.7006 0.7819 0.7756 0.7378 0.7791 0.7256 0.6671 0.7262

Although the diﬀerence between mean values of coeﬃcients obtained for two most extreme pictures (crowd and baboon) is almost 0.1, the trends of mean coeﬃcients values obtained for the individual images are linear.

88

P. Skurowski and A. Gruca

The next step of analysis involved computation of the individual mean correlation coeﬃcients for each of the 26 respondents separately. The obtained results for three representative examples (AB, BM, LS) are presented in Fig. 6. Same as in previous ﬁgures, values on x axis are scores related to the MOS scale and values on y axis are correlation coeﬃcients. Each of the points represents a mean value of the correlation coeﬃcients for the images that were presented to the person and were given the same score value. For such obtained mean values we applied curve ﬁtting procedures to ﬁnd the functions that best model the characteristic of the respondents answers. 1.1

1

0.9

0.8

0.7

0.6

correlation coefficient

0.5

0.4 mean correlationAB fitted linear functionAB

0.3

mean correlationBM fitted power functionBM

0.2

0.1

mean correlationLS fitted exponential functionLS 1

2

3

4

score

5

Fig. 6. Mean correlation coeﬃcients for scores recorded for the individual respondents

Simple comparison just among the mean correlation coeﬃcients reveals, that there is a huge diﬀerence among respondents individual perception of the quality. Especially in the middle of the scale there is almost 50% diﬀerence between recorded mean values for respondents. For example, for the score 3 (fair on the MOS scale) the mean value of correlation coeﬃcient of the respondent BM is 0.854 while for respondent LS it is only 0.4296. The interesting fact however, is that the diﬀerences among the respondents answers are smaller on the extremes of the scale (scores 1, bad and 5, excellent). There are also diﬀerences among types of the functions ﬁtted to the mean values of the correlation coeﬃcients. In Fig. 6. we present several functions, ﬁtted for diﬀerent respondents: linear, power and exponential. The shape of the curve strongly depends of the individual respondent perception of the quality.

5

Conclusions

The paper is an attempt to construct image quality measure based on the well known but rarely used phenomenon. The results are ambiguous and require additional comments. Despite the fact, that we did not use any human visual system model, we obtained very elegant linear relation between human responses and

Image Quality Assessment Using Phase Spectrum Correlation

89

phase correlation used as a quality measure. On the other hand users answers for each score are widely dispersed along the correlation coeﬃcient axis so one can doubt if they can be useful. In our opinion it is the result of user individual preferences about image quality, rather than the properties of individual images. We also suspect that rough scale used in the experiment could be another reason for the dispersion of the results. In authors opinion the results demonstrated in this paper are promising and worth further research. They can be improved in two ways: using ﬁner quality judgment scale like DSCQS with scores 1-100, and applying weighting phase spectral coeﬃcients with its amplitude and its perceptual importance using a human visual system model. Further research can also include comparisons with other quality assessment methods against publicly available distorted images databases.

References 1. Wang, Z., Bovik, A., Liu, L.: Why is image quality assessment so diﬃcult? In: Proc. IEEE Int. Conf. Acoustics Speech Signal Proc., Orlando, vol. 4, pp. 3313– 3315 (2002) ´ ık, M., Slav´ık, P.: Evaluation of Two Principal Approaches to Objective Image 2. Ead´ Quality Assessment. In: Proc. of the 8th Int. Conf on the Information Visualisation (IV 2004), pp. 513–518. IEEE Computer Society Press, Los Alamitos (2004) 3. Oppenheim, A.V., Lim, J.S.: The importance of phase in signals. Proce. of the IEEE 69, 529–541 (1981) 4. Campbell, F.W.: The human eye as an optical ﬁlter. Proc. of the IEEE 56, 1009– 1014 (1968) 5. Zhang, X., Wandell, B.: A spatial extension of CIELAB for digital color image reproduction. In: Proc. Soc. Inform. Display 1996 Digest, S.Diego, pp. 731–734 (1996) 6. Mantiuk, R., et al.: Predicting Visible Diﬀerences in High Dynamic Range Images - Model and its Calibration. In: Proc. of Human Vision and Electronic Imaging X, IS&T/SPIE 17th Annual Symposium on Electr. Imag., S.Jose, pp. 204–214 (2005) 7. Wang, Z., Simoncelli, E.P.: Reduced-reference image quality assessment using a wavelet-domain natural image statistic model. In: Proc. of Human Vision and Electronic Imaging X, IS&T/SPIE 17th Annual Symposium on Electronic Imaging, San Jose, pp. 149–159 (2005) 8. Wang, Z., et al.: Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 600–612 (2004) 9. Liu, Z., Laganiere, R.: On the Use of Phase Congruency to Evaluate Image Similarity. In: Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, Toulouse, pp. 937–940 (2006) 10. Bracewell, R.: The Fourier Transform and Its Applications, 3rd edn. McGraw-Hill, New York (1999) 11. Rice, J.A.: Math. Stat. & Data Analysis, 2nd edn. Duxbury Press, Belmont (1995) 12. University of South California, Signal & Image Processing Institute (SIPI), Image Database, http://sipi.usc.edu/database/ 13. ITU-R Recommendation: P.800 : Methods for subjective determination of transmission quality. ITU, Geneva (1996) 14. ITU-R Recommendation BT.500-11: Methodology for the subjective assessment of the quality of television pictures. ITU, Geneva (2002)

A New Image Fusion Method for Estimating 3D Surface Depth Marcin Denkowski1 , Michal Chlebiej2 , and Pawel Mikolajczak1 1

2

Faculty of Computer Science, Maria Curie-Sklodowska University pl. Marii Curie-Sklodowskiej 5, 20-031 Lublin, Poland [email protected] Faculty of Mathematics and Computer Science, N. Copernicus University Chopina 12/18, 87-100 Toru´ n, Poland

Abstract. Creation of virtual reality models from photographs is very complex and time-consuming process, that requires special equipment like laser scanners, a large number of photographs and manual interaction. In this work we present a method for generating of surface geometry of photographed scene. Our approach is based on the phenomenon of shallow depth-of-ﬁeld in close-up photography. Representing such surface details is useful to increase the visual realism in a range of application areas, especially biological structures or microorganisms. For testing purposes a set of images of the same scene is taken from a typical digital camera with macro lenses with a diﬀerent depth-of-ﬁeld. Our new image fusion method employs discrete Fourier transform to designate sharp regions in this set of images, combine them together into a fully focused image and ﬁnally produce a height ﬁeld map. Further image processing algorithms approximate three dimensional surface using this height ﬁeld map and a fused image. Experimental results show that our method works for wide range of cases and gives a good tool for acquiring surfaces from a few photographs. Keywords: depth-of-ﬁeld reconstruction, image fusion, 3D scene generation.

1

Introduction

Image fusion is a process of combining a set of images of the same scene into one composite image. The main objective of this technique is to obtain an image that is more suitable for visual perception. This composite image has reduced uncertainty and minimal redundancy while the essential information is maximized. In other words, image fusion integrates redundant and complementary information from multiple images into a composite image but also decreases dimensionality. There are many methods discovered and discussed in literature that focus on image fusion. They vary with the aim of application used, but they can be mainly categorized due to algorithms used into pyramid techniques [1,2], morphological methods [3,4,5], discrete wavelet transform [6,7,8] and neural network fusion [9]. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 90–99, 2009. c Springer-Verlag Berlin Heidelberg 2009

A New Image Fusion Method for Estimating 3D Surface Depth

91

The diﬀerent classiﬁcation of image fusion involves pixel, feature and symbolic levels [10]. Pixel-level algorithms are low level methods and work either in the spatial or in transform domain. This kind of algorithms work as a local operation despite of transform used and can generate undesirable artifacts. These methods can be enhanced using multiresolution analysis [1] or by complex wavelet transform [8]. Feature-based methods use segmentation algorithms to divide images into relevant patterns and then combine them to create output image by using various properties [11]. High-level methods combine image descriptions, typically, in the form of relational graphs [12]. In this work we present our image fusion algorithm to achieve ﬁrst of our aim, i.e. to obtain the deepest possible depth-of-ﬁeld using standard digital camera images. The depth-of-ﬁeld can be deﬁned as the distance in front of and behind the subject appearing in focus. This factor is very important in macro photography where only a very short range of the photographed subject will appear in exact focus. For that reason it is essential to focus on the most important part of the subject. The most important factor that determines whether the subject appears in focus is how a single point is mapped onto the sensor area. If a given point is exactly at the focus distance it will be imaged as one point on the sensor, in other case it will produce a disk of which border is known as a ”circle of confusion”. These circles can be used to deﬁne the measure of focus and blurriness as they increase in diameter the further away they are from the focus point. For a speciﬁc ﬁlm format, the depth of ﬁeld is described as a function parametrized by: the focal length of the lens, the diameter of the lens opening (the aperture), and the distance between the subject and the camera. Let D be the distance at which the camera is focused, F the focal length (in millimeters) calculated for an aperture number f and k - the ”circle of confusion” for a given ﬁlm format (in millimeters), then depth of ﬁeld (DOF) [13] can be deﬁned as: DOF1,2 =

D 1±

1000×D×k×f F2

(1)

where DOF1 is distance from the camera to the far depth of ﬁeld limit, and DOF2 is the distance from the camera to the near depth of ﬁeld limit. The aperture controls the eﬀective diameter of the lens opening. Reducing the aperture size increases the depth of ﬁeld, however, it also reduces the amount of light transmitted. Lenses with a short focal length have a greater depth-of-ﬁeld than long lenses. Greater camera-to-subject distance results in a greater depthof-ﬁeld. We use this optical phenomenon to determine the distances from the camera to every point of the scene which gives as the height map ﬁeld of this scene. The height map ﬁeld allows us to achieve our second goal i.e. to create a three-dimensional model of the photographed scene. As an input we have created a series of macro photograph images of the same subject with diﬀerent focus lengths and registered them to each other to create a properly aligned stack of images. The next step is to fuse them into a one composite image. A few methods of image fusion were discussed in our earlier works [14,15], but there were still some problems that needed to be solved. In

92

M. Denkowski, M. Chlebiej, and P. Mikolajczak

this paper we present our new method based on discrete Fourier transform which cope with some of such problems. As an eﬀect of fusing algorithm we obtain a height map ﬁeld and the reconstructed focused image with a very deep depthof-ﬁeld. The height map ﬁeld is a label map which determines the height of each part of the scene. From this map, we can construct a 3D model of the scene. In this work we limit our method to macro photography only and we assume that images were taken perpendicularly or almost perpendicularly to the scene. There is also a strong limitation of our method to scenes that can be represented as a height ﬁeld. The whole method consists of several phases including: image segmentation, height map creation, image reconstruction and 3D scene generation.

2

Methodology

We capture our set of images using standard digital SLR camera mounted on a tripod with macro lenses attached. Our method works best when the photographed plan is perpendicular or almost perpendicular to the lens line. It is also a good idea to avoid specularities and shinning surfaces. For better results gray background can be used. All images are taken in RAW format and then manually matched to each other to equalize their illumination, white balans and exposure. After that, all images are aligned to each other and the reconstruction process combines the image stack into the height map ﬁeld and the fused image. We introduce a new method which employs discrete Fourier transform to designate sharp regions in the set of images and combine them together into an image where all regions are properly focused. From the created height map ﬁeld and the fused image we can generate a 3D surface model of the scene. After that the mesh is created and textured with a plane mapping using the fused image. The main diﬃculty is to obtain the height map ﬁeld without spikes or noise, generally smooth but with sharp edges. It is not essential from the point of view of the image fusion, but it may be crucial in three-dimensional reconstruction of the scene. Most of such peaks are generated in smooth regions, where noise in defocused region on one image from the stack is often more varied than in the corresponding region on sharp image. This leads to undesired deformations of reconstructed spatial surface. For that reason, we introduced a background plane. For now, we assumed that the background plane overlaps with the last image on the stack, but the user can choose it by hand. 2.1

Image Fusion

In our work we use discrete Fourier transform methods combined with Canny edge detector and inpainting techniques to distinguish homogeneous regions. Our fusion method is also capable to work with color images. Color image fusion has been discussed in [16]. At this stage we assume that images on the image stack are aligned to each other. The main objective is to create the focused image and the height map ﬁeld (HM F ).

A New Image Fusion Method for Estimating 3D Surface Depth

93

Fig. 1. Image Fusion scheme showing the steps in our method

First, the discrete Fourier transform for all images is calculated as follows: F (u, v) =

N −1 M−1 yv xu 1 f (x, y)e−2πi( N + M ) N M x=0 y=0

(2)

This transform is multiplied with normalized two dimensional Gaussian distribution: F (u, v) = F (u, v) ∗ G(u, v) (3) where

1 (u + v)2 exp (4) k 2σ 2 where k is normalization factor and σ is a free parameter determining degree of details preserved, which can be speciﬁed by the user. After that inverse transform f (x, y) = F −1 (u, v) is calculated. This gives us an image where pixels with high local gradients are emphasised, see Fig. 2b. Next, the height map ﬁeld is created according to: max(f (x, y)), f (x, y) > k HM F (x, y) = (5) −1, otherwise G(u, v) =

where k is a mean value of f (x, y) multiplied by a free threshold value T s. All values equal to −1 are treated as unresolved. In the next step Canny Edge Detector ﬁlter [17] ﬁnds sharp edges in all input images. Additionally, small islands are removed by hit and miss morphology operator. From those edge images we form one edge image with all the edges by simply applying bitwise OR operator for all Canny images. This edge image helps us to prevent appearance of halo eﬀects on fused image when there is a large diﬀerence in lightness in local area. With this edge image we just mark all pixels in a nearby space and on the edges in the height map ﬁeld HM F (x, y) as unresolved, see Fig. 2f. To classify all unresolved pixels in the height map ﬁeld HM F we distinguish two cases:

94

M. Denkowski, M. Chlebiej, and P. Mikolajczak

Fig. 2. Example of image fusion in steps; input images (a); high frequencies in images (b); unresolved pixels (c); edges pixels (d); unresolved + edges pixels (e); composed Height Map Field with unresolved pixels (f); inpainted unresolved pixels in the HMF (g), fused image (h)

1. Field of island formed by linked unresolved pixels in bigger than background factor Bf - these pixels are marked as backround. 2. Otherwise we employ image inpainting technique, described for example in [18] to ﬁll remaining gaps. Inpainting is a technique for reconstructing lost or broken parts of image, widely used for image restoration or noise removal. Generally, the idea is to ﬁll missing gaps using information from the surrounding area. In our work we use Bertalmo [18] algorithm which uses Navier-Stokes partial diﬀerential equations with boundary conditions for continuity. An example of inpainting technique is shown in Fig. 2g. Now, we have the height map ﬁeld prepared to fuse blurry images into a fused (z) one. A value of a fused image pixel If used (x, y) is equal to the pixel Ii (x, y) from z − th input image on the stack, where z is a value interpolated form the height map ﬁeld HM F (x, y). Separately, regions marked as a background in the HM F in fused image If used are taken from a speciﬁc image selected by the user, but generally they can be taken from any image from the stack due to smoothness and not signiﬁcant diﬀerences between corresponding images in background regions.

A New Image Fusion Method for Estimating 3D Surface Depth

95

Fig. 3. Example images ﬁltered by bilateral ﬁlter; original image (a); image ﬁltered with radius r = 24, σ = 20 (b); r = 24, σ = 96 (c); r = 12, σ = 12, 8 iterations (d); examples of the HMF from fusion algorithm (on left) and the HMF ﬁltered by bilateral ﬁlter (on right) (e, f)

2.2

Scene Visualization

Before three-dimensional visualization the HM F is ﬁltered by the median and bilateral ﬁlter [19] to smooth homogeneous regions while preserving edges beetwen objects. Bilateral ﬁltering is in details a simple, non-iterative scheme for edgepreserving smoothing, that works in spatial and intensity domain and uses shiftinvariant low pass Gauss ﬁlters. An output pixel’s value is calculated according to: h(x) = k f (x, i)C(x, i)(I(x, i) (6) i∈R

where:

1 C(x, i) = exp − 2

d(x, i) σd

2 (7)

is a closeness function, a typical Gaussian ﬁlter, where d(x, i) = d(x−i) =x−iis the Euclidian distance between x and i; 2 1 δ(f (x), f (i)) I(x, i) = exp − (8) 2 σδ is an intensity function, where δ(φ, θ) = δ(φ − θ) =φ − θis a suitable measure of distance between two intensity values φ and θ. Examples of diﬀerent σ parameters for both closeness and intensity functions are shown in Fig 3. Now, the HM F is prepared for creating three dimensional surface. Generally, spatial scene can be visualized by any rendering technique which is able to show information contained in the height map ﬁeld, where each pixel value

96

M. Denkowski, M. Chlebiej, and P. Mikolajczak

represents z coordinate of appropriate mesh vertex. In our work we decided to approximate a height ﬁeld with an irregular triangle mesh using algorithm similar to [20]. The input for this algorithm is a height ﬁeld map represented by an image of which scalar values are heights and the output is polygonal data consisting of triangles. Because of large and memory consuming height ﬁelds the number of triangles in the output is reduced in number as compared to a naive tessellation of the input height ﬁeld map. Generated mesh is smoothed and resulted surface is textured with a plane mapping by the fused image. Additionally, the scene is lit by a directional light which is able to cast shadows to make bumpy surfaces more visible.

3

Experimental Results

The proposed method has been implemented on Linux platform in C++ language using SemiVis framework [21] and Kitware VTK library for visualization purposes. To test whole reconstruction procedure we have prepared eight image stacks from macrophotography. Each stack contained six to twelve images taken with diﬀerent depth-of-ﬁeld. In all cases the procedure is performed in the following order. At ﬁrst, we have to equalize all images together, then the registration process aligns multifocus images to each other to minimize misregistration measure. Next, the reconstruction process combines image stack into the height map ﬁeld and fused image. Finally, 3D scene is generated. Reconstruction time strongly depends on the size of the images used in the fusion and the number of images on the stack. The fusion process takes about 45%, and generation of three dimensional mesh takes remaining 55%. For a typical set of images, containing ten images with resolution 512x512 the whole procedure lasts about 60 seconds. Examples of multifocus images with height map ﬁelds and reconstructed fused images are shown in Fig. 4. Table 1 contains widely used metric QAB/F that measures quality of image fusion. This measure was proposed by Xydeas and Petrovi´c in [22]. In this case, a per-pixel measure of information preservation is obtained between each input and the fused image which is aggregated into a single score QAB/F using a simple local importance assignment. This metric is based on the assumption that fusion algorithm transferring input gradient information into resulting image more accurately has a very good performance rate. QAB/F is in range [0, 1] where 0 means complete loss of information and 1 means perfect fusion. Table 1. The quality measure QAB/F for all eight cases S1 QAB/F

S2

S3

S4

S5

S6

S7

S8

0.43 0.38 0.53 0.61 0.46 0.39 0.44 0.52

A New Image Fusion Method for Estimating 3D Surface Depth

97

Fig. 4. Result fused images and 3D models for three cases. Reconstructed focus images (a, h, n); ﬁltered HMF (b, i); 3d mesh and surface (c, d); surface genarated on the basis of the HMF acquired just after DFT step - noisy spikes clearly visible (e, k); surface reconstracted on the basis of the HMF without bilateral ﬁltering (j); reconstructed 3d models (f, g, l, m, o, p).

Fig. 4 shows qualitative results of our method for eight tested image sets. The biggest problem in this 3d reconstruction is to obtain a surface which is smooth enough in uniform regions and simultaneously has sharp edges on the objects boundaries. The best results are received when the photographs are taken perpendicularly to the background, objects are within the scene, and they are rough without smooth regions. Our method often fails when there are large smooth regions which don’t belong to the background plane. The main diﬃculty in such cases is to distinguish between background and an object without any external spatial knowledge of the scene.

98

4

M. Denkowski, M. Chlebiej, and P. Mikolajczak

Conclusions

This paper presented a new attempt to the image fusion and estimation of surface depth based on multifocus images. We proposed the whole pipeline from raw photographs to the ﬁnal spatial surface. Input multifocus images were fused by DFT method and the height map ﬁeld was created. Based on the HM F the image with a greater depth-of-ﬁeld was composed. Finally, further algorithms reconstructed the 3d surface of the photographed scene. The presented results of generation of 3D models are better than in our previous work [14] and show that our method gives a good tool for acquiring surfaces from a few photographs. However, future work should include automatic detection of the background plane. Second, there should be more complex methods used to identify smooth regions of objects. We think that in both cases pattern recognition algorithms should improve eﬀectiveness of our method. Also Feature-based fusion methods such as [11] could generate more accurate height maps.

References 1. Burt, P.J.: The pyramid as a structure for eﬃcient computation, Multiresolution Image Processing and Analysis, pp. 6–35. Springer, Heidelberg (1984) 2. Toet, A.: Image fusion by rati of low-pass pyramid. Pattern Recognition Letters 9(4), 245–253 (1989) 3. Ishita, D., Bhabatosh, C., Buddhajyoti, C.: Enhancing eﬀective depth-of-ﬁeld by image fusion using mathematical morphology. Image and Vision Computing 24, 1278–1287 (2006) 4. Mukopadhyay, S., Chanda, B.: Fusion of 2d gray scale images using multiscale morphology. Pattern Recognition 34, 1939–1949 (2001) 5. Matsopoulos, G.K., Marshall, S., Brunt, J.N.M.: Multiresolution morphological fusion of mr and ct images of the human brain. IEEE Proceedings Vision, Image and Signal Processing 141(3), 137–142 (1994) 6. Li, H., Manjunath, H., Mitra, S.: Multisensor image fusion using the wavelet transform. Graphical Models and Image Processing 57(3), 235–245 (1995) 7. Chibani, Y., Houacine, A.: Redundant versus orthogonal wavelet decomposition for multisensor image fusion. Pattern Recognition 36, 879–887 (2003) 8. Lewis, L.J., O’Callaghan, R., Nikolov, S.G., Bull, D.R., Canagarajah, N.: Pixeland region-based image fusion with complex wavelets. Information Fusion 8, 119– 130 (2007) 9. Ajjimarangsee, P., Huntsberger, T.L.: Neural network model for fusion of visible and infrared sensor outputs. In: Sensor Fusion, Spatial Reasoning and Scene Interpretation, The International Society for Optical Engineering, SPIE, Bellingham, USA, vol. 1003, pp. 152–160 (1988) 10. Goshtasby, A.A.: Guest editorial: Image fusion: Advances in the state of the art. Information Fusion 8, 114–118 (2007) 11. Piella, G.: A general framework for multiresolution image fusion: from pixels to regions. Information Fusion 4, 259–280 (2003) 12. Wiliams, M.L., Wilson, R.C., Hancock, E.R.: Deterministic search for relational graph matching. Pattern Recognition 32, 1255–1516 (1999)

A New Image Fusion Method for Estimating 3D Surface Depth

99

13. Constant, A.: Close-up Photography. Butterworth-Heinemann, Butterworth (2000) 14. Denkowski, M., Chlebiej, M., Mikolajczak, P.: Modeling of 3D Scene Based on Series of Photographs Taken with Diﬀerent Depth-of-Field. In: Bubak, M., et al. (eds.) ICCS 2008, Part II. LNCS, vol. 5102, pp. 25–34. Springer, Heidelberg (2008) 15. Denkowski, M., Chlebiej, M., Mikolajczak, P.: Three-Dimensional Model Generation Based on Depth-of-Field Image Fusion. Polish Journal of Environmental Studies 17(3B), 78–82 (2008) 16. Bogoni, L., Hansen, M.: Pattern-selective color image fusion. Pattern Recognition 34, 1515–1526 (2001) 17. Gonzalez, R.C., Woods, R.E.: Digital image processing. Addison-Wesley Publishing Company, Inc, Reading (1992) 18. Bertalm´ıo, M., Bertozzi, A.L., Sapiro, G.: Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), vol. 1 (2001) 19. Tomasi, C., Manduchi, R.: Bilateral Filtering for Gray and Color Images. In: Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India (1998) 20. Garland, M., Heckbert, P.S.: Fast Polygonal Approximations of Terrain and Height Fields, Technical Report CMU-CS-95-181, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 (1995) 21. Denkowski, M., Chlebiej, M., Mikolajczak, P.: Development of the cross-platform framework for the medical image processing. Annales UMCS, Sectio AI Informatica III, 159–167 (2005) 22. Xydeas, C., Petrovi´c, V.: Objective image fusion performance measure. Electronics Letters 36(4), 308–309 (2000)

Collecting 3D Content: Examples from Art and Medicine Leszek Luchowski Institute for Theoretical and Applied Informatics Polish Academy of Sciences Gliwice, Poland [email protected] http://www.iitis.gliwice.pl/en/employee.php?ID=7

Abstract. The paper discusses some aspects of the acquisition and preparation of 3D imaging data in two application areas: orthodontics and cultural heritage. Scanning devices used by our team are presented. Examples are given of the imaging of museum exhibits and other historic objects. Also described are possible modalities of presenting the 3D data in various hardware contexts and for various categories of users. Keywords: 3D imaging, digital content, virtual museum.

Introduction This text presents some of the joint experience of the Computer Vision Systems research group at our Institute. In particular, most of the research in the medical area is the work of dr Agnieszka Tomaka, and dr Krzystof Skabek is in charge of the Virtual Museum project. Thanks are due to dr Przemysaw Kowalski who devoted a lot of eﬀort to the registration and fusion of the 3D church. Our team has been working on image processing since 1986. Several years ago, we extended our interests towards 3D imaging. We pursue two major application areas: – medical imaging of the human head and face for the purposes of orthodontic diagnosis and treatment planning – the ”Virtual Museum” project - digitization of objects of cultural heritage. These two very diﬀerent areas share some common characteristics: – they deal with objects which cannot be described by common geometric solids, and rich 3D measurement data (such as 3D scans) is needed to represent their shapes. – when placed in a database, the 3D imagery has to be accompanied by other data (medical or historic, respectively, as well as administrative) in the form of texts and numbers. – In a networking environment, access to the images and other data has to be controlled. This requirement is obvious for medical data. For the data representing objects of cultural signiﬁcance, access control is required for several reasons, which will be described below. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 100–110, 2009. c Springer-Verlag Berlin Heidelberg 2009

Collecting 3D Content: Examples from Art and Medicine

1

101

Types of 3D Scanning Devices and Data Used in Our Projects

In topological terms, 3D scanners and the data they yield fall into two categories: Surface scanners collect the coodinates of points on the surface of objects, by using light reﬂected oﬀ the surface or by touching the surface with a tactile sensor. The resulting data is a large set of coodinate triplets representing surface points. The points can be disconnected (a point cloud) or organized into a polygonal mesh (typically, of triangles or quadrangles).

Fig. 1. Left to right: a 3D point cloud representing a part of the surface of a statue; the same points organized into a triangular mesh; volumetric CT image of a human head

Fig. 2. Left to right: the Minolta Vi-9i, the 3DMD twin-module face scanner, the Faro Laser ScanArm

Our laboratory has three optical (light-based) scanners (Fig. 2): – an industrial-grade Minolta Vi-9i laser scanner, – a twin-module 3DMD structured-light face scanner, – a Faro Laser ScanArm.

102

L. Luchowski

The 3DMD scanner has two modules, each of which is a 3D scanner in itself. Two modules are used for a wider ﬁeld of view. The structure of a module is presented in Fig. 3. The ﬁve basic components which can be seen in the module are, top to bottom: – – – – –

a monochrome (black and white) camera a uniform white ﬂash light a color camera a structured ﬂash lamp another monochrome camera.

Fig. 3. One module of the 3DMD scanner and a fragment of its projected pattern

The color camera and the uniform ﬂash simply take a color photograph of the subject’s face. The structured-light ﬂash projects a complicated pattern of ﬁne dots and lines onto the face. This is photographed by the two monochrome cameras working as a stereo pair. The two lamps ﬁre almost simultaneously, but at a time interval suﬃcient for the two imaging modes not to interfere with each other. The Faro arm is a passive 6 degree-of-freedom mechanical arm with precise position transducers, ending with a spherical or conical tip. It can be used alone, to gather coordinates of individual points, or combined with a small laser scanner to collect dense 3D data by sweeping the scanner on the arm over the surface of the object. Volumetric imaging devices use various types of penetrating radiation to measure some physical parameter over the full range of X,Y, and Z coordinates in a certain volume of space. The parameter can be radiometric density (in Xray based CT), the concentration of hydrogen nuclei (in NMR) or other. The resulting data is a 3D array of voxels (volume elements), typically organized into rows, columns and layers.

Collecting 3D Content: Examples from Art and Medicine

103

While we do not have volumetric imaging devices in our institute, we are working with the Institute of Radiology of the Silesian Medical Academy in Zabrze, Poland, where this kind of imaging is regularly applied to patients. Thanks to this cooperation we have also been able to obtain volumetric images of objects such as dry human skulls and selected calibration bodies.

2

An Overview of Our Work for Orthodontics

Orthodontics is a branch of medicine which examines and treats irregularities in the shape of the craniofacial area, typically in children and adolescents. Planning surgical interventions requires distances and angles to be measured on the patient’s bone structures. This was traditionally done on 2D X-ray images, and more recently with CT scans. Both techniques expose the patient to harmful radiation and therefore cannot be used arbitrarily often. Our goal in this respect is to identify mathematical correlations between the bone structure and the outer surface of the face. When this is achieved, it will be possible to only do a CT scan once, at the beginning of a stage of treatment such as osteodistraction, and then use harmless optical scanning of the surface to estimate the changes occurring in the bone. One particular technique we have developed [2] to limit X-ray exposure is shown in Fig. 4.

Fig. 4. Registering a plaster model of teeth to the face

The shape of the teeth is obtained mechanically by taking a dental cast. The plaster model can then be digitized by scanning. However, a cast does not provide information about the position of the teeth relative to the face and head. In other words, scanning both the plaster model and the face yields two sets of

104

L. Luchowski

3D data in separate, unrelated coordinate systems. This does not allow us to see if the occlusal plane (where the upper and lowet teeth meet) is tilted sideways or has an incorrect front/back inclination. The technique we used to solve this problem consists in putting an auxillary object called a dental arch between the patient’s teeth. The rear part of the arch has a wax pad in which the teeth leave impressions. The outer part which has a shape which is easy to locate in a 3D scan. The patient’s face is scanned with the arch and without it. The arch, with the impressions of the patient’s teeth on its wax pad, is then placed between the plaster models of the two jaws and scanned with them. After cross-referencing a total of ﬁve 3D images, the teeth are virtually placed inside the face in the position they have in reality. No X-rays are used in the process. Another set of correlations we are interested in identifying is between the shapes of facial structures at various stages in life. When correcting the geometry of a child’s face, it is important to take future growth into account. To help investigate the patterns of growth, we are accumulating a corpus of data by scanning the faces of eight healthy children every six months. The children were taken from our own families, to ensure that they will be available for an extended period of time.

3

The Virtual Museum Project

The purpose of the project is to gather and develop techniques which can produce 3D images of objects of cultural heritage, store the images in an organized database, and deliver them to users at various levels of detail, depending on available bandwidth (or storage capacity) and the users’ authorized level of access. The objects scanned so far under the project come from two sources: local churches and the city museum of Gliwice. 3.1

Church Objects

The ﬁrst object of artistic value we scanned was the baptismal font of the Gliwice cathedral (Fig. 5). The scanner was placed on a tripod in 8 diﬀerent places around the font, with the tripod adjusted to four diﬀerent heights in each place. At each height, the scanner was tilted back and forth to a various number of diﬀerent inclinations. A total of 130 scans were made in this way. The right part of Fig. 6 shows the coordinate systems associated with a few of them. The scanning took three people a full day. Putting the partial images together took a whole month of very meticulous work. The ﬁnal result was the shape of the font represented by a closed mesh of triangles which formed a closed, topologically correct polyhedron. As such, it deﬁned a solid which could be printed on a rapid prototyping device (see below).

Collecting 3D Content: Examples from Art and Medicine

105

Fig. 5. Scanning the baptismal font

Fig. 6. 3D scans of an epitaph and a statue

The next artifact to be scanned was an epitaph to Jan Welczek in the Holy Cross church in Gliwice. We chose it because of its near-planar shape,which was suitable for our work on the registration of 3D images on the basis of a photograph [3]. We also scanned a statue of St John the Evangelist in the XIIIthcentury Cistercian abbey in Rudy Wielkie, and the entire Assumption of Our Lady church in Gliwice (originally built in Zebowice). The church was scanned with a Faro total station, temporarily in our posession. Its permanent assignment is at the Technology and Humanities Academy in Bielsko Biaa, our partner in a consortium which also includes the Technical University in Gliwice.

106

L. Luchowski

Fig. 7. Scanning a church with a Faro total station

3.2

Museum Exhibits

We are presently scanning objects from two sections of the museum: ceramics and archaeology. Cast metal objects from the historic Gliwice foundry will be scanned next.

Fig. 8. Scanning a bisque figure and a porcelain vase from the ceramics section, and two archeological artifacts from a Lusatian burial ground in Swibie

3.3

Access Control Issues for Cultural Artifacts

In our experience, gaining access to church objects proved easy. We were allowed to scan objects and buildings as long as it did not cause too much disruption. In some cases, the procedure was welcomed as a way of presenting the church assets, or of creating a digital backup of objects which might be stolen or destroyed. No reservations were made as to how the images may be made available to the public. The situation is very diﬀerent in museums. Museums are quite understandably protective of the copyright they hold over their exhibits. In particular, they usually wish high-quality images (both 2D or 3D) to only be accessible to authorized researchers. This leads to the creation of at least two categories of users: – the lower category, which we might call casual web surfers, who should have access to visually attractive but not precisely detailed images, and to general historic data such as the age and provenance of the exhibit;

Collecting 3D Content: Examples from Art and Medicine

107

– the authorized researchers, who should have full access to state-of-the-art imagery and rich historic and administrative information. An intermediate level of detail, both in images and other data, can be put on media such as DVDs issued by the museum.

4

Problems and Processing

Irregularities encountered in scanning include occlusions (parts of the object hiding behind each other), holes (gaps in the mesh due to imperfect scanning), spikes (extremely elongated triangles on inclined surfaces, or spanning depth discontinuities), registration fault lines (where two partial models are imperfectly brought into the same coordinate system) and lighting discrepancies (resulting in the same point having a diﬀerent hue in diﬀerent scans). Figure 9 presents examples of such defects, left to right: – a scan of the bisque ﬁgure, showing a large occlusion gap where a part of the man’s back is hidden by the woman’s arm; – a scan of another ﬁgure, showing large holes ﬁlled with spikes; – a model of the same ﬁgure composed of multiple scans, with a registration fault on the neck; – a similarly composed model of a face, with a texture showing bad lighting discrepancies.

Fig. 9. Examples of defects in scanned and merged images

5

Levels of 3D Modeling

Creating a 3D model out of scans is a tedious process. How much eﬀort should be spent on it depends on the purpose the model is intended for. Table 1 describes issues related to the geometry and topology of a 3D mesh. Color is a separate problem. We would like to have a model of the object which could be placed in a virtual scene and illuminated in arbitrary ways, always appearing as the physical original would under the same lighting. To achieve this fully, we would need to measure the spectral and directional characteristics of light reﬂection, absorption and dispersion at every point of the object (at

108

L. Luchowski Table 1. Summary of requirements on 3D models for various applications Purpose of model Create one rendered 2D ”photo” illustrating 3D shape Put a 3D model on the web, make an amateur 3D portrait etc

Requirements Just one 3D scan can be enough, with little or no processing Multiple scans required. Some holes and fault lines are acceptable. Low number of vertices, because fine detail is not required and bandwidth may be limited Model to be examined by professionals Registration must not introduce vis(such as medical practitioners or art ible artifacts or distort dimensions. historians) High resolution usually required. Mesh topology may be imperfect. 3D printing (rapid prototyping) Level of detail depends on scale and printer resolution. Mesh topology must be flawless, i.e. it must be an orientable closed surface with no self-intersections

some resolution). The equipment and labor cost of such measurements is, for our purposes, prohibitive. A more practical approach, and the one we follow, is to take color photographs of the object and apply its colors to the surface of the model. What is obtained in this way is not the inherent color of the object; it is an image of it taken under some lighting distribution. This distribution will usually not be uniform over the whole surface of the object, because, even under homogenous ambient light, parts of the object itself will aﬀect each other’s actual lighting. Some 3D scanners (such as our Minolta and 3DMD) take a color picture of the object while scanning, but the light distribution can be diﬀerent in each individual scan. This leads to unacceptable color discontinuities in merged models. The Minolta is particularly diﬃcult in this respect, as taking multiple pictures requires either the scanner or the object to be moved, which changes the distribution of light on the object - either because the position of the object has changed with respect to light sources, or because the large black bulk of the scanner is blocking a diﬀerent part of the ambient light. To avoid such unwelcome eﬀects, the color information taken by the scanner is often discarded, and high-resolution 2D pictures of the object are taken with a digital SLR camera, under controlled lighting conditions. The color data from the photos are then applied to the 3D mesh, manually, using a 3D graphics editor.

6

Means of Presentation of 3D Images

Computer graphics has techniques of creating suggestive 2D images of 3D objects, and sending them to an ordinary computer screen or printer. Many of the pictures in this paper (Fig. 4, Fig. 6, and the ﬁrst three parts of Fig. 9) were

Collecting 3D Content: Examples from Art and Medicine

109

prepared in this way. We can expect that the 3D data placed on the Web or published on DVD will typically be viewed via such 2D renditions. However, it is very desirable to view the objects in true 3D. In our medical applications, 3D viewing radically improves doctor’s perception of bone structures, which has been constrained to 2D since the invention of X rays. In the virtual museum context, the interest of viewing sculptures and architecture in full 3D is obvious. The technical means to deliver a 3D object to the user come in several variations: – 3D printing, also known as rapid prototyping (although the two terms are not exactly synonymous), producing a material scaled copy of the object;

Fig. 10. Producing a hard copy of the 3D model. Left to right: the inside of the printer, the model surrounded by support material, washing away the support, the finished model

The Dimension Elite 3D printer (Fig. 10) works by depositing molten plastic, layer after layer, on a tray. The head of the printer has two heated nozzles which extrude two types of plastic: the model material, which is white in this case, and the dark brown support material. The support material is needed wherever an upper part of the model is broader than the part underneath it. As it is impossible to deposit a drop of the model material on top of air, the space under every such part is ﬁlled with support material. The support material is then removed in a bath of hot (70 deg.C) NaOH solution. – 3D screens, which exist in desktop or auditorium size (Fig. 11). The latter typically require the user to wear special eyeglasses, either with polarization ﬁlters or liquid crystal shutters. Several large screens can be combined to form a cave, a large cube enclosing the viewer, where some or all of the six faces of the cube are screens. More sophisticated systems track the movements of the user’s head and adjust the images to the changed viewpoint; – Images printed on paper and combined with some viewing equipment, such as lenticular foil, anaglyphic (red-green) glasses, or stereoscopes;

110

L. Luchowski

Fig. 11. Viewing a projected 3D image

– 3D immersion goggles, delivering a stereoscopic pair of images directly to the eyes, in an active version of a stereoscope (can also use head tracking); – Naked-eye stereograms (mostly limited to entertainment purposes)

Conclusion 3D imagery is increasingly becoming a part of the ubiquitous digital representation of the world. Modalities of image acquisition and presentation have been reviewed, with indications as to the format and scope of data depending on the intended user.

Acknowledgement This work was supported in part by the Polish Ministry of Science and Higher Education, under the research grant N N516 1862 33.

References 1. Luchowski, L.: Stability of feature points defined by approximating quadric surfaces. In: Computer Recognition Systems: Proceedings of 5th International Conference on Computer Recognition Systems CORES 2007 (2007) 2. Tomaka, A., Tarnawski, M., Luchowski, L., Lisniewska-Machorowska, B.: Digital dental models and 3D patient photographs registration for orthodontic documentation and diagnostic purposes. In: Computer Recognition Systems: Proceedings of 5th International Conference on Computer Recognition Systems - CORES 2007 (2007) 3. Skabek, K., Tworzydlo, A., Luchowski, L., Winiarczyk, R.: Bringing into Register Incomplete Range Images of Cultural Artifacts. In: International Conference on Computer Vision and Graphics ICCVG (2006)

3D Object Reconstruction from Parallel Cross-Sections Karolina Nurzy` nska Institute of Informatics, The Silesian University of Technology Akademicka 16, 44-100 Gliwice, Poland [email protected]

Abstract. The paper presents a novel approach to the contour stitching method which improves the reconstruction of 3D objects from parallel contours. The innovation assumes the construction of outer and inner contour from many contours on one cross-section in case of branching. This exchange allows easy one to one connection, between contours on adjoining cross-sections, which has already been deﬁned. Additionally, such a deﬁnition permits from artiﬁcial holes generation. Moreover, the method takes care of smooth surface reconstruction in case of concave and convex shapes. The performance of the novel algorithm has been tested on the set of artiﬁcial data as well as on the set of data gathered from CT scans and proved it to work well. Keywords: 3D object reconstruction, contour stitching, CT scans, parallel contours.

1

Introduction

The development of technology in medicine provided many new facilities which aim to improve physicians work and the disease diagnosis. In case of the mediastinum tumour diagnosis nowadays not only the computed tomography (CT) scanners are utilised but they are backed by the information gathered from the positron emission tomography (PET). The fusion of the data from those sources allows for accurate speciﬁcation where the changed cells, which need therapy, are. Although, the access to such equipment has become wide in recent years and many patients have been healed thanks to proper diagnosis, there are still needs to improve the performance. Due to the determination of the location of a tumour in the mediastinum the physicians need to visualize in 3D the intrinsic body structure, which could be obtained from the CT scans and enrich the data with tumour regions visible on the PET scans. To make the fusion more diﬃcult, usually the scans are not only gathered in diﬀerent resolution on the plane but also the resolution, or at least the places of gathering the cross-sections scans in Z axes, may diﬀer. Since, so many troubles there have not been created many systems to visualise the fusion data. However, those which have been implemented proved to be in L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 111–122, 2009. c Springer-Verlag Berlin Heidelberg 2009

112

K. Nurzy` nska

Fig. 1. An example of a set of adjoining cross-sections of mediastinum scans from computed tomography

a poor quality. Therefore, the physicians mostly base their work on the sets of parallel cross-sections (see the Fig. 1) than its 3D visualization. In order to build a suitable system for mediastinum tumour diagnosis one need to pay attention to many problems. First of all, it should allow proper reconstruction of object from parallel cross-sections. The reconstruction procedure should manage any possible shapes, e.g. convex, concave or bifurcating. It is also very important that the resulting object reﬂects the original one, otherwise the medical diagnosis could not be based on such a system. Therefore, it is crucial that the exploited method concentrates on accuracy of reconstruction, however the plausible surface generation would also be an advantage. The reconstruction process should not create any artefacts, like for example artiﬁcial holes. It is signiﬁcant that one method should manage all this demands. On the other hand, this system should manage well the fusion of reconstructed organs from CT scans with data containing the information of tumour gathered from PET. This article presents the preliminary research for a method which allows for a 3D object reconstruction from parallel cross-sections and fulﬁls all the demands described above. According to the previous research [17] on the 3D object reconstruction methods from point cloud it was proven that the best method, which

3D Object Reconstruction from Parallel Cross-Sections

113

after some improvements should give satisfactory results, is the contour stitching algorithm. Therefore, in this paper the novel approach to the contour stitching algorithm is described. It reconstructs the objects of various shapes but preserves the original structure unchanged, as it is an interpolation method. The structure of the article is as follows. In the second section the overview of existing approaches for 3D object reconstruction from point cloud is given. The third section describes the novel assumptions and the algorithm itself. While results are described in the section four and the section ﬁve draws the conclusions.

2

Domain Overview

The 3D object reconstruction from point cloud is a widely researched domain of computer graphics. Therefore, many various techniques have been introduced which solve the reconstruction task diﬀerently. We could distinguish between surface- or volume-oriented methods, those which result in continuous surface or discrete surface represented by the irregular polygonal mesh, which are addressed in this work. Whereas, the other point of view, it could be divided into interpolation and approximation methods. The Marching Cubes algorithm was introduced by Lorensen and Cline [15]. In the original version it allows to achieve the irregular polygonal mesh which is an approximation of the surface described as a volumetric data. The method divides the volume into sub-cubes and triangulates each of them according to one of the given triangulation classes. Bourke [5], [6] deﬁnes this algorithm for input data given as a point cloud or a scalar ﬁeld. Moreover, Chernyaev [8] suggests additional triangulation classes to avoid ambiguity and holes creation. Unfortunately, in case of the reconstruction for parallel cross-sections, where the distance between slices is usually 10 times bigger than the resolution on the slice, it results in stair-case eﬀect. Therefore, in the initial stage the volumetric methods (named also ﬁeld-based or Eulerian methods) are utilized. They aim to recreate the lacking data by deﬁning the signed distance function [7], [10], [16]. Also the Marching Voxels method [14] addresses the problems which are troublesome for Marching Cubes: the ambiguity, holes and evaluation time. The authors suggest that all this problems are due to division of the space into cubes and therefore their solution uses the vertices for algorithm work. It processes the input data in three stages. Firstly, there are created triangles among all vertices, what protects from holes generation. Secondly, those triangles are joined in polygons by adjoining edges, what creates a polygonal mesh. Finally, for visualization, the data is projected on the 2D plane. Another group of methods might be represented by the Delaunay triangulation which is based on the observation of Delaunay [9], who stated that for a set of points P exists such a triangulation DT (P ) that no point in P is inside the circumference of any triangle in DT (P ). The Delaunay triangulation for a discrete point set is unequivocal with Voronoi diagram. On the other hand, for a given closed contour in space and a 3D trajectory it is possible to reconstruct a surface enclosed by the contour while its movement along the trajectory; it is called sweeping [1], [2], [19].

114

K. Nurzy` nska

Finally, the contour stitching [11], [12], [13], [18], known also as a Langrangian method, as an input data needs a set of contours and results in the irregular polygonal mesh which interpolates the object surface. The aim of this method is to create the connections between points in the adjacent contours. For achieving satisfactory results three issues should be addressed: the correspondence, tiling, and branching. The broad review of this method can be found in the work of Bajaj et al. [3]. In previous researches [17] all static methods which reconstruct discrete manifolds have been investigated. The algorithms were compared in respect of accuracy, various shape reconstruction as well as artefacts creation. It was proven that the contour stitching method is the most promising one for medical objects reconstruction, however the branching problem must be solved.

3

Contour Stitching — The Novel Approach

The presented approach is a modiﬁcation of the contour stitching algorithm which addresses all three problems: the correspondence, tiling, and branching. The method manages the reconstruction of 3D objects from parallel contours gathered from CT scans. There have been paid a special care to receive plausible resulting surface. 3.1

Correspondence Problem

The correspondence deﬁnes which vertices from a pair of contours should be connected (see Fig. 2). Wrong matching results in the eﬀect of surface twisting. Usually, the solution of correspondence is inﬂuenced by the solution chosen for tiling problem and vice versa.

Fig. 2. On the left side the example of correct correspondence creation. On the right side wrong correspondence which may lead to surface twisting.

In the literature there have already been suggested some solutions suﬃcient to solve the problem. For instance, Parker and Attia [18] aim to ﬁnd the global length of the edges needed for connections and then choose the shortest solution. On the other hand, Keppel [13] prefers to choose such point connections which correspond to the biggest object volume. Whereas, Ganapathy and Dennehy [11]

3D Object Reconstruction from Parallel Cross-Sections

115

solve this problem by deciding to choose one pair for start proper connecting while the others result from proper tiling. Therefore, they assume that the ﬁrst pair is created from vertices which have the smallest x coeﬃcient (see the left image in the Fig. 2.), which has been employed in the described solution. 3.2

Tiling Problem

The tiling is responsible for a creation of facets in the mesh build on two adjoining contours. In previous works the mesh was reconstructed basing on Delaunay triangulation, which was broadly described in [11] but ﬁrstly mentioned by Boissonnat. Diﬀerent solution, considering the mesh creation on the level of each triangle, was suggested in the one of ﬁrst method described by Gueziec et al. [12]. The resulting mesh from this method tends to have a minimal surface size and it is useful especially in cases where the minimal surface of reconstructed solid is necessary. In case of the same number of vertices in both contours, the solution requires equal facets spacing, however when the vertices count diﬀer (which is the most frequent case) additional rules must be speciﬁed. For example, see the Fig. 3 where on the right side one may see the correct mesh reconstruction while, on the right side the surface which twists itself. Moreover, it proved to be important to distinguish between connection of two convex contours and two concave or concave and convex. In the ﬁrst case the equal distribution of additional facets is suﬃcient rule to achieve nice looking surface, whereas in the other cases additional restriction must be imposed. The left part of Fig. 4 depicts the reconstruction result when only the rule of equal facets spacing was utilized, on the right side additional constraints have been added therefore its surface is plausible.

Fig. 3. On the left side of the image the correct tiling solution. On the right side tiling resulting in the twisted surface reconstruction.

3.3

Branching Problem

The branching problem appears when on two adjoining slices the contour number diﬀers. In this case, it must be decided which contours should be connected to each other and how to create the surface. The Fig. 5 shows all possibilities of

116

K. Nurzy` nska

Fig. 4. On the left side the example of surface tiling where only the equal facet addition rule was applied. On the right side example of surface with all constraints utilized for its reconstruction.

Fig. 5. All possible branching construction between cross-sections with one contour and two contours

surface reconstruction in case when there is one contour on one cross-section and two contours on the adjoining one. To decide which contours belong to one object many solutions can be discussed, however two most promising and used in previous implementations are: the shape resemblance parameters and overlapping percent. The shape resemblance parameters, allow to describe some characteristics of the shape, although good they are in case of bifurcating object the assumption that its outline preserves similar shape might be wrong. Therefore, the overlapping percent parameter proved to give better results. Since the reconstruction of the surface between adjoining contours is known, then to generate a connection in the case of branching substitute contours are introduced. The idea of substitute contours is described in following points.

3D Object Reconstruction from Parallel Cross-Sections

3.4

117

The Substitute Contour Concept

It is easy to notice that when considering two adjacent slices there may occur the following cases: both slices have the same number of contours or the number diﬀers (for clarity it is assumed that one to many contours relationship takes place, as it is easily resolved into other cases). To connect two adjoining cross-sections with diﬀerent contour number the substitute contours are generated which replace the original ones by the outer and inner contour. The outer contour is deﬁned as an outline of all contours on the slice and it consists of the vertices which would be common to this outline and the contours. However, the inner contour is deﬁned as a diﬀerence between the outer contour and all contours on the slice. The replacement is necessary to create one to one relation between a contour and the outer contour, which connection problem has been already solved and only the additional deﬁnition for creating a closing surface from the inner contour is necessary. Let assume that the multilayer contour set (j = 1, . . . , n) is given where each cross-section is denoted as Πj . Each slice Πj consists of contours Φj,i where i = 1, . . . , k. Let assume that the contour Φj,i is a sequence of points (p = 1, . . . , m) and that the ﬁrst point is the one which has the smallest x value (following Ganapthy) in order to assure the proper correspondence of points among contours. Additionally, let assume that the contours on one slice are ordered relatively to each other. 3.5

The Outer and Inner Contour Definition

Let assume that there is one contour Φj,i at layer Πj and many contours at layer Πj+1 . In the ﬁrst step of the method the outer and inner contour must be found for the slice Πj+1 . The outer contour Φj+1,OUT (see dark grey line on Fig. 6) consists of points which are used to generate the connection between the contour on cross-section Πj and Πj+1 , whereas the inner contour Φj+1,IN (see light grey line on Fig. 6) consists of points which are used to create a closing surface between branches. Due to ﬁnd the division of contour points the polygon Ωj+1 = {s1 , s2 , . . . , sk−1 , sk } is deﬁned (see dashed line on Fig. 6), which vertices si are the centroids of contours Φj+1,i on the cross-section Πj+1 . Let introduce the Oj+1 as a set of contour points limited by the polygon Ωj+1 . Then, the points which are not in the polygon belong to the outer contour, while those belonging to the polygon belong also to the inner contour as given in the formula: ⎛ ⎞ ⎛ ⎞ Φj+1,IN = ⎝ Φj+1,i ⎠ ∩ Oj+1 Φj+1,OUT = ⎝ Φj+1,i ⎠ \Oj+1 . (1) Πj+1

Πj+1

It is necessary to take note of the following problems. While moving the points from contours Φj+1,i on the slice Πj+1 adequately to outer and inner contour the proper order of points must be kept. Additionally, due to assure the continuity of the surface (that there are any holes in the area of outer and inner contour

118

K. Nurzy` nska

Fig. 6. The creation of an inner and outer contour from three contours on one crosssection. Ellipsis depicts original contours. Dashed line represents the polygon dividing points of original contours into inner and outer ones. Dark grey line represents the outer contour. Light grey line represents the inner contour.

connection) the points from the contour Φj+1,i on the layer Πj+1 which satisfy the formula: ∀

∃

(pi ,pi+1 ) pi ∈Ωj+1

∧

∃

∨

pi+1 ∈Ωj+1

∃

pi ∈Ωj+1

∧

∃

pi+1 ∈Ωj+1

(2)

⇒ (p1 , pi+1 ) ∈ Φj+1,IN ∧ (pi , pi+1 ) ∈ Φj+1,OUT should be inserted into both contours (outer and inner). 3.6

Filling Holes with Points

In the next step of the processing it is worth to add some points to both contours (inner and outer) on the adjacent slices connecting the contours Φj,i and Φj+1,i , it is due to assure natural look of the resulting mesh. The manner of choosing the points depends on the expected appearance of the object. The easiest solution is to take points from the edge of the polygon between the consecutive contours Φj,i and Φj,i+1 , but more complex solution may give better visual eﬀects. Some examples are depicted in Fig. 7. 3.7

Final Mesh Generation

Finally, the interpolation table of points from contours Φj,i and Φj+1,i , in slices Πj and Πj+1 adequately, is created which is used for mesh generation. The outer contour, as it was mentioned before, should be connected with the contour on the adjacent layer using the contour stitching in the same way as it would be done for two slices Πj and Πj+1 with the same number of contours. While in the case of inner contour the surface should be created which close the part of the object between branches. There are possible many solutions, yet the centre of area might be found and used as a ﬁrst vertex for all triangles, where two other vertices are successive vertices from the inner contour Φj+1,IN .

3D Object Reconstruction from Parallel Cross-Sections

119

Fig. 7. Examples of additional points added to ﬁll spaces on outer and inner contour in the places of connection of the original contour

4

Results

The described method for object reconstruction from parallel cross-sections was implemented and proved to manage well any possible shape of an object. Its performance test was conveyed on the artiﬁcial data set, which was created in order to check whether the branching problem is solved correctly. On the other hand, the original data set, containing data gathered from CT scans and representing lungs, has been exploit to see how the algorithm operates with real data, where objects changes their shape from convex structures to concave ones. Figure 8 shows the example of reconstruction of an object from artiﬁcial set. As it is easily seen the generated mesh is smooth as well as on the whole object and in the branching area. The result of the reconstruction of a lung example is depicted on the Fig. 9. It is worth to point out, that the mesh not only reﬂects the original organ without any artefacts but also presents a good quality mesh.

Fig. 8. The artiﬁcial object reconstruction. The contours and views from diﬀerent sides.

120

K. Nurzy` nska

Fig. 9. The lungs object reconstruction. The contours on the left side and the reconstruction on the right.

Fig. 10. The plot represents the dependency between the input vertex number and evaluation time

It is of high importance that the algorithm not only reconstructs objects correctly but also performs the operation in short time, to allow the reconstruction at least in the soft real time. In the research the data is processed on the IntelCore 2 Duo 2GHz processor with 2GB RAM, however only one core of the processor is utilized as the software was not yet optimised for parallel processing. The results of time performance test are depicted in the Fig. 10. Those tests proved that the dependence of time performance from input vertices is linear, which, taking under consideration the fact that this algorithm could be easily parallelized, mean that even for much larger input data its performance remain unchanged.

5

Conclusions

The paper presents a novel approach to the contour stitching method which improves the reconstruction of 3D objects from parallel contours. The innovation

3D Object Reconstruction from Parallel Cross-Sections

121

assumes the construction of outer and inner contour from many contours on one cross-section in case of branching. This exchange allows easy one to one connection between contours on adjoining cross-section, which have already been deﬁned. Additionally, such a deﬁnition permits from artiﬁcial holes generation. Moreover, the method takes care of smooth surface reconstruction in case of concave and convex shapes. The performance of the novel algorithm has been evaluated on the set of artiﬁcial data as well as on the set of data gathered from CT scans and proved it to work well.

References 1. Abdel-Malek, K., Blackmore, D., Joy, K.: Swept Volumes: Foundations, Perspectives and Applications. International Journal of Shape Modelling, http://www. engineering.uiowa.edu/~amalek/papers/swept-volume-review.pdf 2. Abdel-Malek, K., Yang, J., Blackmore, D.: Closed-form Swept Volume of Implicit Surfaces. In: Proc. of ASME Design Engineering and Technical Conferences (2000), http://www.engineering.uiowa.edu/amalek/papers/DAC-14232-2000.pdf 3. Bajaj, C., Coyle, E.J., Kun-Nan, L.: Arbitrary Topology Shape Reconstruction from Planar Cross Sections. Graphical Models and Image Processing, 524–543 (1996) 4. Barequet, G., Shapiro, D., Tal, A.: History Consideration in Reconstruction Polyhedral Surfaces from Parallel Slices. In: Proc. of Conference on Visualization, pp. 149–157 (1996) 5. Bourke, P.: Polygonising a Scalar Field (1994), http://local.wasp.uwa.edu.au/pbourke/geometry/polygonise 6. Bourke, P.: Polygonising a Scalar Field Usign Tetrahedrons (1997), http://local.wasp.uwa.edu.au/~pbourke/geometry/polygonise 7. Braude, I., Marker, J., Museth, K., Nissanov, J., Breen, D.: Contour-Based Surface Reconstruction using MPU Implicit Models. Elsevier Science Graphical Models, pp. 139–157 (2007) 8. Chernyaev, E.V.: Marching Cubes 33: Construction of Topologycally Correct Isosurfaces. Technical Report CERN CN 95-17 (1995), http://www.info.cern.ch/asdoc/./psdir/mc.ps.gz 9. Delaunay, B.: Sur la sph´ere vide in Otdelenie Matematicheskikh i Estestvennykh Nauk (1934) 10. Galin, E., Akkouche, S.: Fast Surface Reconstruction from Contours using Implicit Surfaces. In: Implicit Surface Conference (1998) 11. Ganapathy, S., Dennehy, T.G.: A New General Triangulation Method for Planar Contours. In: ACM SIGGRAPH Computer Graphics, pp. 69–75 (1982) 12. Gueziec, A., Taubin, G., Lazarus, F.: Cutting and Stitching: Converting Sets of Polygons to Manifold Surfaces. IEEE Trans. on Vizualization and Computer Graphics (2001) 13. Keppel, E.: Approximating Complex Surfaces by Triangulation of Contour Lines. IBM, Resource Development, pp., 2–11 (1975) 14. Lin, C.-F., Yang, D.-L., Chung, Y.-C.: A Marching Voxels Method for Surface Rendering of Volume Data. In: Proc. of International Conference on Computer Graphics, pp. 306–313 (2001)

122

K. Nurzy` nska

15. Lorensen, W.E., Cline, H.E.: Marching Cubes: A high resolution 3D surface construction algorithm. In: Proc. of 14th Conference on Computer Graphics and Interactive Technologies, pp. 163–169 (1987) 16. Nilsson, O., Breen, D., Museth, K.: Surface Reconstruction Via Contour Metamorphosis: an Eulerian Approach With Lagrangian Particle Tracking. In: Proc. of IEEE Visualization, pp. 407–414 (2005) 17. Nurzyska, K.: Static Methods for Object Reconstruction Overview: for Medical Diagnosis Use. In: 7th International Conference on Computer Science – Research and Application (in press) (2008) 18. Parker, J.R., Attia, E.N.: Object Reconstruction from Slices for Vision. In: Proc. of International Computer Graphics, pp. 58–64 (1999) 19. Schmidt, R., Wyvill, B.: Generalized Sweep Templates for Implicit Modeling. In: Proc. of 3rd International Conference on Computer Graphics and Interactive Techniques (2005), http://www.unknownroad.com/publications/SweepsGraphite05.pdf

Implementation of Progressive Meshes for Hierarchical Representation of Cultural Artifacts Krzysztof Skabek and L ukasz Z¸abik Institute of Theoretical and Applied Informatics, Polish Academy of Sciences ul. Baltycka 5, 44-100 Gliwice, Poland [email protected] http://www.iitis.gliwice.pl

Abstract. In the article we focused on progressive meshes which are the way to hierarchically represent 3D objects. We present the Me3d system developed to process and present digitalized objects of cultural heritage. The optimization of progressive meshes was described and the rendering performance of displaying the simpliﬁed meshes is analyzed. Keywords: progressive mesh, mesh processing and simpliﬁcation, hierarchical representation of 3D objects.

1

Introduction

Designing eﬃcient, lossless and continuous-resolution representation leads to many practical problems in computer graphics: mesh compression, smooth morphing of level-of-detail (LOD) approximations, progressive transmission, selective reﬁnement. Modern computer graphics require very realistic and fast 3D model representation. Detailed models are obtained by applying versatile modelling operations. For eﬃcient presentation, these models must be stored as approximations - meshes. Detailed meshes are also obtained by scanning physical objects using range scanning systems. In either case, the resulting complex meshes are expensive in storing, transmitting and rendering. There are some way to make the processing and presentation of 3D complex objects more eﬃcient. One of the improvements deﬁnes several levels of details, e.g. to display more detailed model when viewer is coming closer. Transmitting a mesh over communication line one may want to see a model with further approximations, next levels-of-details approximations. Mesh storing is very memory consuming. The problem may be solved in two ways: there may be used mesh simpliﬁcation or compression. There are many diﬀerent ways to represent graphical 3D models, in this article we focused on progressive meshes which were introduced by Hoppe [3]. We assume here that the progressive mesh complexity does not depend on viewer position, it is simpliﬁed on the whole surface. In the article the Me3d system developed in our Institute was described. The program is used to process and present digitalized objects of cultural heritage. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 123–132, 2009. c Springer-Verlag Berlin Heidelberg 2009

124

K. Skabek and L . Z¸abik

A module for building and presenting the progressive meshes is its component. Storing data in the progressive meshes is a way to hierarchically represent 3D objects. Such approach is being developed and extended in the research project N N516 1862 33 1 and its aim is to store and virtually distribute objects of cultural heritage. 1.1

Mesh Representations

A mesh is the basic representation of 3 dimensional graphical objects [6]. We focused on triangle meshes where each face is build of 3 vertices. The basis of the representation is a set of point coordinates (x, y, z). The orientation of faces as well as storing order of vertices is very important. Therefore the normal vectors for faces and the ordered vertices are also used to deﬁne the mesh properties. The mesh geometry can be denoted by a tuple (K, V ) [2], where K is a simplicial complex specifying the connectivity of the mesh simplices (the adjacency of the vertices, edges, and faces), and V = {v1 , . . . , vm } is the set of vertex positions deﬁning the shape of the mesh in R3 . More precisely, we construct a parametric domain |K| ⊂ Rm by identifying each vertex of K with a canonical basis vector of Rm , and deﬁne the mesh as the image φv (|K|) where φv : Rm → R3 is a linear map. Besides the geometric positions and topology of its vertices, the mesh structure has another appearance attributes used to render its surface. These attributes can be associated with faces of the mesh. A common attribute of this type is the material identiﬁer which determines the shader function used in rendering a face of the mesh. Many attributes are often associated with a mesh, including diﬀuse colour (r, g, b), normal (nx , ny , nz ) and texture coordinates (u, v). These attributes specify the local parameters of shader functions deﬁned on the mesh faces. They are associated with vertices of the mesh. We can further express a mesh as a tuple M = (K, V, D, S), where V speciﬁes its geometry, D is the set of discrete attributes df associated with the faces f = {j, k, l} ∈ K, and S is the set of scalar attributes s(v, f ) associated with the corners (v, f ) of K. As many vertices may be connected in one corner with the same attributes, the intermediate representation called wedge was introduced to save the memory [4]. Each vertex of the mesh is partitioned into a set of one or more wedges, and each wedge contains one or more face corners. Finally we can deﬁne the mesh structure that contains an array of vertices, an array of wedges, and an array of faces, where faces refer to wedges, and wedges refer to vertices. Face contains indices to vertices, additionally this structure contains array of face neighbours (fnei) in which indices of tree adjacent faces are stored, this information is necessary to build a progressive mesh. There is nothing said in original papers about order of vertices and indexes of adjacent faces in Face structure. In our implementation the counter is stored clockwise and additionally ﬁrst adjacent face is at ﬁrst position as ﬁrst vertex, so if we cross ﬁrst edge we ﬁnd the ﬁrst neighbour, if we cross second we ﬁnd the second, etc. 1

http://www.iitis.gliwice.pl/zksw/vm project/

Implementation of Progressive Meshes for Hierarchical Representation

125

In many places of this article we use the word edges. The edge is a connected pair of vertices or, in other words, it is a pair of adjacent vertices. There is no additional list of edges, but the ﬁrst vertex and the face to which this edge belongs are deﬁned instead. Using wedge we can access vertex, even if the adjacent face does not exist we can deﬁne edge. Deﬁnition of edges is necessary to simplify meshes, to create progressive meshes as well as to determine which edge (vertex) could be collapsed. 1.2

Manifold Surfaces

Surfaces are often, as well as in our approach, assumed to be manifolds. A manifold [6] is a surface, all of whose points have a neighbourhood which is topologically equivalent to a disk. A manifold with boundary is a surface all of whose points have a neighbourhood which is topologically equivalent to either a disk or a half-disk. A polygonal surface is a manifold (with boundary) if every edge has exactly two incident faces (except edges on the boundary which must have exactly one), and the neighbourhood of every vertex consists of a closed loop of faces (or a single fan of faces on the boundary). Many surfaces encountered in practice tend to be manifolds, and many surfacebased algorithms require manifold input. It is possible to apply such algorithms to non-manifold surfaces by cutting the surface into manifold components and subsequently stitching them back together. However, it can be advantageous for simpliﬁcation algorithms to explicitly allow non-manifold surfaces. 1.3

Progressive Mesh

Progressive mesh (PM) [3] is special case of the mesh or rather extension of the mesh, it makes it possible to build mesh of diﬀerent level-of-details (LOD) [5]. It also allows loading base mesh M 0 , the mesh of lower LOD (explained later in more details), and then progress loading of the remaining parts of the mesh. As an input source of loading we may use memory input stream. is stored as a much coarser mesh M 0 In PM form, an arbitrary mesh M together with a sequence of n detail records that indicate how to incrementally = M n . Each of these records reﬁne M 0 exactly back into the original mesh M stores the information about a vertex split, an elementary mesh transformation that adds an additional vertex to the mesh. Thus the PM representation of M 0 1 n deﬁnes a continuous sequence of meshes M , M , . . . M of increasing accuracy, from which LOD approximations of any desired complexity can be eﬃciently retrieved. Moreover, smooth visual transitions (geomorphs) [3] can be eﬃciently constructed between any two such meshes. In short, progressive meshes oﬀer an eﬃcient, lossless, continuous-resolution representation. Progressive meshes makes it possible not only to store the geometry of the mesh surface, but, what is more important, preserve its overall appearance, as deﬁned by the discrete and scalar attributes associated with the surface. There are three operations that make it possible to determine the base mesh of : edge collapse, vertex split and edge swap. Edge collapse operation is suﬃcient M

126

K. Skabek and L . Z¸abik

to successfully simpliﬁed meshes. An edge collapse operation ecol(vs , vt ) remove one edge and instead two vertices vs and vt insert new one vs . Additionally two faces (vt , vs , vl ) and (vt , vr , vs ) are removed. The initial mesh M0 can be obtained = M n: by applying a sequence of n edge collapse operations to M = M n) (M

ecoln−1

→

ecol

ecol

· · · →1 M 1 →0 M 0

The edge collapse operation is invertible. The inverse transformation is called vertex split. Vertex split operation adds in place of vertex vs two new vertices vs and vt and two new faces (vt , vs , vl ), (vt , vr , vs ) if edge {vs , vt } is boundary then adds only one face. Because edge collapse transformation is invertible our can be presented as a simple M 0 and sequence of n vsplits records: mesh M M0

vsplit0

→ M1

vsplit1

→ ···

vsplitn−1

→

= M n) (M

We call (M 0 , vsplit0 , ..., vsplitn−1 ) a progressive mesh (PM) representation of M .

2

KD-Trees

A KD-tree (k-dimensional tree) [7] is a space-partitioning data structure for organizing points in a k-dimensional space. KD-trees are useful data structures for several applications, such as searches involving a multidimensional search key (e.g. range searches and nearest neighbour searches). In fact, KD-trees are a special case of BSP trees. The KD-tree uses only splitting planes that are perpendicular to one of the coordinate system axes. Each splitting plane must go through one of the points in the KD-tree. We assumed the following rules to choose the splitting planes. As one value moves down the tree, another one cycles through the axes used to select the splitting planes. At each step, the point selected to create the splitting plane is the median of the points being put into the KD-tree, with respect to their coordinates in the axis being used. This method leads to a balanced kd-tree, in which each leaf node is about the same distance from the root. We use traversing the tree technique, starting from the root and moving to either the left or the right child depending on whether the point to be inserted is on the ”left” or ”right” side of the splitting plane to Add a new point to a KD-tree. The nearest neighbour (NN) algorithm, to ﬁnd the NN to a given target point not in the tree, relies on the ability to discard large portions of the tree by performing a simple test. This is done recursively for all the depth-searched branches of the tree until a ﬁnal minimum region containing the node is found. Finding the nearest point is an O(log N ) operation.

3

The Implemented Algorithms

Our software environmentmake it possible to read many diﬀerent ﬁle formats (stl, obj, ply, etc.). Some of the ﬁle formats store only information about triangles

Implementation of Progressive Meshes for Hierarchical Representation

127

alignment, i.e. vertices and normals (e.g. stl), the others store also information about the object topology like face neighbours or common points (obj). We also deﬁned our xml-like format to store data in the Me3d application. In order to unify all the formats after loading object into memory it is necessary to perform several operation on objects before we start work with them. After that we describe methods to create and process the progressive meshes. 3.1

Finding Adjacent Faces

There are many operations that need information about object topology, thus before we perform any operation of this type we have to calculate appropriate object attributes. Finding adjacent faces is not fast algorithm in simplest form, because it is necessary to compare in each face each edge with every edge in other faces, so its complexity is O(n2 ) taking as most important operation comparing two faces. For small objects this time is not signiﬁcant but for bigger and more complex objects time of performing this operation becomes very signiﬁcant. To shorten that time we decided to use a modiﬁed KD-tree; in this case algorithm complexity is O(n log(n)). We have to ﬁnd face neighbours for each face, in KDtree ﬁnding element complexity is O(log(n)). Because we assume that ﬁnding adjacent faces is performed before reducing points so we add to each node two additional ﬁelds: m setFaces and m setPoints. The both structures are lists. The ﬁrst one stores all points that have the same point coordinates and the second one stores point indices that have this value ( we can perform two operations simultaneously ﬁnd adjacent faces and redundant point reduction). We used ordinary KD-tree here, the point coordinates were compared as keys. In the implementation we used set data structure instead list or vector as it matched the program requirements and was easier in managing. The elements were sorted in the tree and as each key was unique, we decided not to take care about double keys. 3.2

Reduction of Points

Some ﬁle formats (e.g. stl) store triangles with point coordinates, not with indexes to the points, so it was necessary to implement function for reduction of redeundant points. During this operation we have to be very careful because we have to maintain all connections in the structure of the resultant object. The designed algorithm performs several operations: (1) making a copy of the original mesh, (2) removing all points from the base mesh, (3) operations on all avaliable faces (get point, ﬁnd all points of this value and replace by one with the same index, insert indices in correct place), (4) setting the reduced points, (5) cleaning all unnecessary data. The copy of original mesh is necessary to save all untouched data, because we work on copy of mesh and we save changes only in original mesh. Finding points of the same value is not a big problem because each face is ”aware” of its neighbours so we simply go through appropriate neighbour and we change indices in each face where it is necessary.

128

3.3

K. Skabek and L . Z¸abik

Construction and Processing the Progressive Meshes

is to determine a Our goal in construction of the PM for a given mesh M sequence of edge collapses transforming M = M n into a base mesh M 0 . There are many possible PM construction algorithms more or less eﬃcient and accurate. The easiest way for selecting edge collapses is to choose them completely at random. In each case we should preserved some local conditions, i.e. maintaining manifold structure. We can also distinguish methods that ﬁnd approximating meshes optimal with respect to some appearance metric, for instance the energy geometric metrics Edist proposed by Hoppe [3]. The Progressive Mesh construction is performed only once. Once we build base mesh M 0 we can save it and use any time we need, but we have to save it with sequence of vsplit operations. It is important and ambigious to choose the new point position in the collapse operation. There are many diﬀerent approaches to do this (see Fig. 1). The simplest one is to remove vt and leave vs without changing it position. The other is to determine medium point: vs = (vt + vs )/2; we decided to use this approach. Probably one of the best approaches is to calculate new point position using a spine. It gives the best results but is very time consuming. If we use one of two ﬁrst approaches i.e. medium calculation or laving vs as new point we may see object sinking.

Fig. 1. Diferrent methods of point placement: a) original one, b) using spline, c) medium point, d) drop vt

We determine base mesh M 0 and {vsplit0 , ..., vsplitn−1 } sequence. Before we start building our PM we have to build the cost table containing costs of each face removal. The algorithm is divided into two parts: building vsplit sequence and building base mesh M 0 . We used the energy function proposed by Hoppe [3] to optimize the mesh. This function has following form: E(M ) = Edist (M ) + Espring (M ) + Escalar (M ) + Edisc (M ) The basic operation during manipulating the Progressive Meshs is vertex split (Vsplit). It adds one vertex, one or two faces (depending on mesh structure) and up to 6 wedges. It traverses the old corners around the newly added vertex vt (using the face adjacencies in fnei set) and possibly updates the corners to

Implementation of Progressive Meshes for Hierarchical Representation

129

point the new wedge(s) associated with vt . It updates the local face adjacencies to reﬂect the introduction of the new faces. Finally, it updates the vertex and wedge attributes using the deltas stored in Vsplit. This operation is performed by method next() placed from PMesh Iterator structure (PMeshIter). Edge collapse (Ecol) is the reverse operation to vertex split. It removes one vertex, 1 or 2 faces and up to 6 wedges, depending on object structure. It also modiﬁes all connections in the mesh, i.e. neighbours of faces, wedges etc. The operation is performed in reverse order to vertex split. It updates vertex and wedge attributes, updates face adjacencies, updates corners around the old vertex vt , and ﬁnally removes 1 vertex, 1-6 wedges, and 1-2 faces from the ends of the arrays in Mesh. This operation is performed by method PMeshIter::prev(). 3.4

Hausdorﬀ Distance

Let X and Y be two compact subsets of a metric space M . The Hausdorﬀ distance dH (X, Y ) [6] is the minimal number r such that the closed r-neighborhood of X containing Y and the closed r-neighborhood of Y containing X. In other words, if d(x, y) denotes the distance in M , then: dH (X, Y ) = max{ sup inf d(x, y), sup inf d(x, y)} x∈X y∈Y

y∈Y x∈X

This distance function turns the set of all non-empty compact subsets of M into a metric space, say F (M ). The topology of F (M ) depends only on the topology of M . If M is compact, then so is F (M ). If M is complete, then so is F (M ). In Euclidean geometry, one often uses an analogy, Hausdorﬀ distance up to isometry. Namely, let X and Y be two compact ﬁgures in a Euclidean space; then DH (X, Y ) is the minimum of dH (I(X), Y ) along all isometries I of Euclidean space. This distance measures how far X and Y are from being isometric. Above deﬁnition precises the distance between any two sets of points and is often used to determine error between two meshes.

4

Me3d Framework Architecture

The PM was designed in software framework Me3d. The overview of program window is shown in Fig. 2. The framework make it possible to display and process 3D objects stored in several ﬁle formats (em stl, obj, ply), it is also possible to store objects in its native format as me3d or me3db ﬁles. The program enables catching a current view and saving it to bmp ﬁle. One can open more than one object at the same time. The opened objects are listed in the project panel. It is also possible to store the scene settings (rotation, shift, zoom). There are some functional panels connected to the program that operate on graphical 3D structures. First of all the Picking Panel to interactively select the region of the object or just pick the whole one. The Transform Panel is used to move, rotate and stretch objects in every direction (axis x,y,z). The object rotation can be performed around any arbitrary given line using the Rotate About

130

K. Skabek and L . Z¸abik

Fig. 2. Me3d Framework

Given Line Panel. Program enables displaying objects in three modes: faces, vertices and edges. It works in two modes normal mode and object picking mode. The ﬁrst mode is for viewing objects from any direction, the second one is for selecting areas, it is also possible to zoom the object then. The Me3d framework make it possible to build user plug-ins and implement methods operating on me3d structures in form of dynamically embedded libraries (dlls). A spacial plug-in manager is provided to control the user modules. The application was built in Microsoft Visual Studio. It consists of main window and many dockable panels. One of the modules is designed to operate on progressive meshes (see Fig. 2 on the right side). It is possible to degine the desired base mesh M 0 size (button Desired Base Mesh Size), however, ﬁnal mesh may be less. We also choose the method of cost calculation among items: smallest edge, smallest area and quadratic error measure (quad) [1]. To create the PM we have to perform two steps: ﬁrst we Build the Base Mesh (button Build Base Mesh) and, if the obtained mesh is satisfactory for us, we build the proper progressive mesh using the button Build Progressive Mesh. After we created the PM Progressive Mesh LOD Panel become active and we can interactively change the complexity of ﬁnal mesh. It can be done in two ways: using buttons Ecoll and Vsplit to move one step forward/backward or using a slight bar. At every step (i) it is possible to measure the Hausdorﬀ distance between the original and the partial progressive mesh M i (button Calculate Error). mesh M

5

Tests and Results

Tests were performed using processor Pentium IV 1.8GHz, 512 MB and graphic card Radeon 7500 with 32MB RAM. During tests the rendering speed of

Implementation of Progressive Meshes for Hierarchical Representation

131

Fig. 3. Object – screwdriver: a) 13574 faces – original object, b) 8884 faces, c) 4404 faces, d) estimation of coarse error

displaying objects with diﬀerent LODs was measured. Hausdorﬀ distance was measured (distances to points) to calculate the mesh interpolation error. Results, if given in percents, are calculated with regard to the longest dimension of an object. Frame rate value is the average for 10 frames. In pictures there are shown only boundary cases of tested objects i.e. original picture, the most coarse case and the middle case. The FPS rates shown on the charts are only theoretical, because program limit frame rate to 30 frames per second. That means that its maximum rendering speed that may be achieved, if we use program without limitation of frame rate. As we can see the results of ﬁrst test are probably the best is it caused by uniform (regular) faces (see Fig. 3. Almost all faces has the same shape. If we use objects with accumulation of small faces in some areas, these areas will be simpliﬁed ﬁrst. The plates of mashroom (see Fig. 4.mashroom) are removed ﬁrst because they are small; in this case the object can be simpliﬁed very much, we can remove more than half faces almost without losing quality. Having a look at the duck example (see Fig. 4.duck) the faces were removed from the most thick area, but in this case lost of quality is signiﬁcant. In examples shown above removing about 25% of faces does not inﬂuence very signiﬁcantly quality of objects. Quality of mesh depends very strongly on shape of object, if we look at Duck (see Fig. 4.duck) it almost does not loose quality if even half of faces is removed because there are many small faces located at one area this object coarse does not inﬂuence general shape very much. However if we coarse object with many sharp edges (elements) then even removing small numbers of faces can damage object very much, so there is no general idea how many faces we can remove, it strongly depend on object shape. Our system was designed to process the virtual objects of cultural heritage. In Fig. 5 we present the simpliﬁcation of the bowl from archaeological collection of Museum in Gliwice. In further work we are going to do more adjustments of the PM building process to make it smoother and intuitive. We also plan to implement the

132

K. Skabek and L . Z¸abik

Fig. 4. Rendering performance of diﬀerent progressive meshes

Fig. 5. Progressive representation of archaeological bowl: a) 14787 faces, b) 23560 faces, c) 35430 faces, d) 41400 faces – original object

view-dependent PM to represent the mesh structure in denser/coarser way depending on the region of interest.

Acknowledgement This work was supported by the Ministry of Science and Higher Education of Polish Government, the research project no N N516 1862 33.

References 1. Garland, M.: Quadric-Based Polygonal Surface Simpling. School of Computer Science Carnegie Mellon University, Pittsburgh (1999) 2. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W.: Mesh Optimization. In: Computer Graphics (SIGGRAPH 1993 Proceedings), pp. 19–26 (1993) 3. Hoppe, H.: Progressive Meshes. In: Computer Graphics (SIGGRAPH 1996 Proceedings), pp. 99–108 (1996) 4. Hoppe, H.: Eﬃcient Implementation of Progressive Meshes. Computer & Graphics 22(1), 27–36 (1998) 5. Luebke, D., Reddy, M., Cohen, J.D., Varshney, A., Watson, B., Huebner, R.: Level of Details for 3D Graphics. Morgan Kaufmann, San Francisco (2003) 6. Nielsen, F.: Visual Computing: Geometry, Graphics and Vision. Charles River Media (2005) 7. Samet, H.: Foundations of Multidimensional and Metric Data Structures. Morgan Kaufmann, San Francisco (2006)

Multi-Object Tracking Based on Particle Filter and Data Association in Color Image Sequences Ayoub Al-Hamadi, Saira Saleem Pathan, and Bernd Michaelis Institute for Electronics, Signal Processing and Communications (IESK) Otto-von-Guericke-University Magdeburg, Germany {Ayoub.Al-Hamadi,Saira.Pathan}@ovgu.de

Abstract. Robust tracking of multi-objects is still challenging in real scenarios such as crowed scenes. In this paper a novel method in color image sequences is proposed for tracking multiple objects in non-cooperative situations. A system of independent particle ﬁlters with an adaptive motion model is used which tracks the moving objects under complex situations. Besides, in order to handle the conﬂicted situations, an integrated data association technique is exploited which adjusts the particle ﬁlters accordingly. Results have shown the good performance of the proposed method on various complex-situation image sequences. Keywords: Particle Filter, Object Tracking, Pattern Recognition.

1

Introduction

Object tracking since its beginning is an attractive topic for researcher due its situation dependence behaviors. Many techniques have been developed but the eﬀectiveness and robustness of these proposed techniques are very sensitive to non-cooperative situations such as occlusion, deformation, and brightness variation. A very powerful approach used in tracking is particle ﬁlter which estimates of new state of the object through the previous state using motion model and measurement. In multi-object tracking besides the state of the object, history of the objects is also used for preceding images for data association. Therefore, it is required to update the object states with the correct measurement. Otherwise the particle propagation would be deﬂected and an object loss would be caused. The multi-object tracking (MOT) in real-time is a crucial problem. Here we brieﬂy discuss various approaches and issues that became the motivation of this research. One of the prominent techniques is the Condensation algorithm that creates a basis for many extensions [1]. The idea is based on considering particle with the highest probabilities and not the whole state space as it is used with the grid-based approaches. However, it exponentially increases complexity with every dimension of the state space. Sidenbladh [2] analyzes the whole scene with several objects using a particle ﬁlter. In doing so, every particle or state vector Xt contains the states of all objects. Thus, a particle describes the variation of all system-describing parameters by which a wide range of combinations arises. In this manner an extremely high number of particles become necessary to cover all probable states and also to L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 133–142, 2009. c Springer-Verlag Berlin Heidelberg 2009

134

A. Al-Hamadi, S.S. Pathan, and B. Michaelis

approximate the a-posteriori distribution. As a result identiﬁcation of the single objects in this installation of the particle ﬁlter is possible only at very high computational cost. Chang [3] suggested the Kernel-Particle Filter to support particle’s evolution. Doucet [4] and Murphy [5] put together both the particle ﬁlters and kalman ﬁlters. Thus, the non-linear part is approximated using particle ﬁlter and the linear part is solved by the kalman ﬁlter analytically. However, for high eﬃciency in high dimensional state spaces, the state space can be disassembled with the Rao-Blackwellisation-Method in the linear and non-linear for separate processing [6]. Vermaak [7] introduce the Mixture Particle Filters (MPF) in comparison to standard particle ﬁlter. The particle ﬁlter is composed from several components. Every component is assigned a certain number of particles with only the state of one object and in every time step it is developed independently. Vermaak [8] extends the MPF approach via the association using Joint Probabilistic Data Association (JPDA). However, with the insertion of the JPDA the use of particle ﬁlter on real scenes with ﬁxed and known number of objects remains delimited. In another approach, Cox [9] uses a multi-hypothesis process to the data association, which allows the use of a variable object number, but it is mathematically complex. In short here multi-object tracking (MOT) is investigated by exploiting diﬀerent methods in which the scene modeling takes place using simple particle ﬁlter. Therefore, the identiﬁcation of the single objects and determination their trajectories are missing [2], [7]. In other methods, the diﬀerentiation of the objects is analyzed by using data association [10], [11]. In these methods, the achieved abilities are demonstrated merely on simulated data or on radar data without considering the complex situations. To overcome these diﬃculties of MOT, a system of independent particle ﬁlters with an adaptive motion model is suggested here. Using an integrated data association step it is possible to recognize conﬂict situations and to adapt and adjust the particle ﬁlters to these situations. Added beneﬁts are the reduction of image data volumes, and robust determination of motion trajectories even in the presence of image-speciﬁc disturbances.

2

Multi-Object Tracking

There are three main steps in particle ﬁlter; sampling, prediction and updating. A modiﬁed approach is proposed where the above mentioned steps are integrated into the process ﬂow of suggested particle ﬁlter system (Fig. 1). In contrast to other works, an extended adaptive motion model is used for the prediction, which consists of standard model, experience model and standing model. The particle ﬁlter is linked with a data association step for recognizing conﬂicts. These conﬂicts are due to sudden visual cluttering and object overlaying. The data association step includes data exclusion, data allocation and data administration. The association step allows the MOT interconnects prediction and updating. For MOT, ﬁrst particle ﬁlters are initialized requiring moving objects segmentation and state description. The moving objects segmentation is done using fast modiﬁed diﬀerence-image-based approach (MDI) [12]. The resulting image regions present so-called Motion-Blobs (MB), which represent the basis for state

MOT Based on Particle Filter and Data Association

135

Fig. 1. suggested particle ﬁlter system for MOT in color image sequences

description. State parameters are image- and color-based parameters as follows: • Image parameter. The position (px , py ) is image parameter which is the center of gravity of the respective MB. We also consider the direction and the speed (dx , dy ). This is the ﬁrst part of the state vector of an MB (object) for the image parameters (XF,t ) at the time (t): XF,t = (px,t , py,t , dx,t , dy,t )

(1)

• Color parameter. The color information of MB is also considered to handle the moving objects under the inﬂuence of local transformations and partial occlusion. YUV color space is used for description of color cluster for each MB. The resulting color cluster is divided into small color sub-clusters. For cluster decomposition a region growing method is used based on the distances between the points. By using of the principle component analysis (PCA) parameters for the single color sub-cluster like center of gravity (s), extent (ex) and alignment r = (rx , ry , rz ) are determined. These are summarized into the second part of the state vector Xc,t at the time t (Eq. 2). Xc,t = (s(1) , ex(1) , r(1) , ..., s(c) , ex(c) , r(c) )

(2)

A sub-cluster c is selected according to the object’s description by its main regions from cluster set. We observed that an object with c = 3 clusters is showing the biggest among others sub-cluster. Hence, we ﬁnd that an object can be described as a composition from three point clouds by the cluster parameters. 2.1

Particle Filter System

Particle ﬁlter system used here is a composition of several particle ﬁlters. For each moving object a particle ﬁlter is used, which contains the state of the belonging object. The state spaces are divided into separate particle ﬁlters, so that every object is estimated and represented by its own particle tracker resulting eﬃcient computation by decreasing the number of the particles. The whole ﬁlter distribution p(Xt |Zt ) (so-called a-posteriori distribution) consists therefore of single distributions p(m) (Xt |Zt ) of M- particle ﬁlters. The current states of the particles are estimated based on the last state of the particle ﬁlter p(Xt−1 |Zt−1 )

136

A. Al-Hamadi, S.S. Pathan, and B. Michaelis

and the motion model p(Xt |Xt−1 ). The result of the prediction is the eﬀective a-priori probability p(Xt |Zt−1 ) (Eq. 3). On current measurement basis (Z) the predicted particles are weighted by observation model p(Zt |Xt ) according to their likelihood. The updating step takes place according to Bayes Rule (Eq. 4). The eﬀective a-priori probability p(Xt |Zt−1 ) is weighted by a similarity function and produces the a-posteriori distribution which is now again the basis for the new a-priori distribution. In the sampling step a corrected concentrating particle set is generated around the measured state (Fig. 1). The denominator in (Eq. 4) is acting as a standardizing constant. In contrast to [7] no weighting is carried out between the particle ﬁlters. Prediction: P (Xt |Zt−1 ) =

M

p(m) (Xt |Xt−1 )p(m) (Xt−1 |Zt−1 )dxt−1 (3)

m=1

M

Updating:

(m) (Zt |Xt )p(m) (Xt |Zt−1 ) m=1 p (m) (Z |X )p(m) (X |Z t t t t−1 )dxt−1 m=1 p

P (Xt |Zt ) = M

(4)

In addition, adaptive motion model is applied for every particle ﬁlter that are thereby individually manipulate and adaptive to the respective conditions. Adaptive motion model for the prediction step. An extended adaptive motion model is used for the prediction, which contains experience motion model (M ME ), standing motion model (M MS ) and standard motion model (M MN ) where the standard motion model is the conventional motion model. • The standing motion model (M MS ) can be used if there is no measurement for an object like with stopping objects by congestion. At this point, the standing model ensures a stop in a particle evolution at a time . Instead, the last particle will maintain in every time step until availability of a new measurement value. M MS : Xt = f (Xt ∗) + r

(5)

• The experience motion model (M ME ) is used in the case of ambiguities in the data association. In contrast to standard model the average movements of several time steps are considered d¯ = (dx,t¯, dy,t¯) in and the generated particles are not weighted by the observation model. A noise ratio r = (rx , ry ) is added, which is generated after a normal distribution N. ¯ +r M ME : Xt = f (Xt−1 , d)

(6)

The three motion models are combined to adaptive motion model (MMA) which are linked about XOR operator ⊕ with each other (Eq. 7). The controlling of M MA takes place in a data association. Thus only one motion model is active. M MA = M MN ⊕ M ME ⊕ M MS

(7)

MOT Based on Particle Filter and Data Association

137

Similarity for the updating step. The predicted particles are weighted by the observation model p(Zt |Xt ) by their likelihood on the basis of the corresponding measurement. Therefore, the measurements have to be assigned to the objects unambiguously. The identiﬁcation and allocation of several objects are reached in two steps whereas in the ﬁrst step the allocation of object prediction and current measurements takes place via similarity measure SMob (Eq. 9) which consists of two parts. The ﬁrst part is the similarity SMpar which analyzes the image parameters of an object against the acquired measurement. This similarity measure SMpar consists of three weighted similarity criteria for the position SMpos , speed SMspe and motion direction SMdir . The similarity measure is used for updating (Eq. 4). Thereby, particles which lie away from the measurement or show other motion properties should be removed. In the second part, a comparison of the color information is carried out to remove ambiguities with position and movement. This part will be used in the data association step. The c color clusters of the object predictions and measurements are analyzed about the similarities SMpos , SMext and SMdir . SMob = wF .SMpar + wC .

C

(w1 .SMpos,i + w2 .SMext,i + w3 .SMdir,i )

(8)

i=1

SMpar = w4 .SMpos + w5 .SMspe + w6 .SMdir

(9)

where wF , wC and wi = 1..6 work as standardization constant (wi = 1..6 = 0.3 and wF , wC = 0.5). The single similarity measures are given in the interval [0, 1] where the value 1 signiﬁes the maximum match. For the correlation of the respected position (px,t , py,t ) and for the reference measurement (px,ref , py,ref ) the Euclidean Distance is used. With the distmax the distances are standardized: SMpos = 1 − |(px,t , py,t ) − (px,ref , py,ref )|.(distmax + 1)−1

(10)

The similarity measures of the extent of the color clusters SMext as well as the speeds of particles and objects SMspe are determined. The diﬀerence of the speeds vt of prediction and vref of measurement is suﬃcient for the calculation. For SMext the extent parameters expt and expref are used accordingly. SMspe = [1 + abs(vt − vref )]−1

(11)

The motion direction similarity measure (SMdir ) of the particles and objects (dx,t , dy,t )T as well as the alignment of the color clusters with (dx,ref , dy,ref )T can be compared with the angle between the motion vectors. This angle is determined using the scalar product and the amounts of the vectors. Because all similarities lie in the interval [0, 1] and have the same evaluation direction, they can be used in the similarity criteria SMob and SMpar . 2.2

Data Association

For recognizing conﬂict situations the conventional two-layer association with data exclusion and data allocation are modiﬁed (Fig. 2) which is the third layer

138

A. Al-Hamadi, S.S. Pathan, and B. Michaelis

Fig. 2. The concept of the data association

of data administration. In the data exclusion step, a pre-selection allocation pairs is carried out. The measurements which lie in the prediction (X ) area are considered as an allocation candidate into this association step (Fig.2), otherwise they are excluded. All predictions and measurements establish an allocation pair to which this relation is applied. For formulation of the search area the Euclidean Distance is used. If a measurement Z lies in the search area (S ) of an object, it is a candidate for the respective O objects. For the subsequent treatment the results are entered on the candidates matrix K = (kij ) of the size (OxM) (Eq.12). 1, if Z(j) ∈ S{X(i)} kij = (12) 0, othewise, i = 1, ..., O & j = 1, ..., M where, O and M are the number of objects and measurements. The entries kij = 1 mean that measurement Z(j) is located near object X(i). So, coherence between objects and measurements are maintained and the matrix represents a basis for analyzing the scene abstractly in the data allocation step. In the data allocation, the candidate matrix K resulting from the data exclusion step is divided into candidate sets (Fig. 2). Through the subdivision in candidate sets with only coherent pairs a clear separation is reached between independent constellations. Single objects can be illustrated in a 1-element set Al (Eq. 13), single measurements according to Bl (Eq. 14) and unequivocal allocations in Cl (Eq. 15). The allocation pairs of an ambiguous situation are summarized into a set Dl (Eq. 16) where l indexes after the number of the suitable sets. M Al = {(θio )| kij = 0} (13) j=1

Bl = {(θoj )|

O

kij = 0}

(14)

i=1

Cl = {(θij )|

O i=1

kij = 1 ∧

M j=1

kij = 1}

(15)

MOT Based on Particle Filter and Data Association

Dl = {θij |

O i=1

kij > 1 ∨

M

kij > 1}

139

(16)

j=1

Besides, all allocation pairs θij and θmn belongs to a set Dl , if it is valid to its entries kij and kmn in the candidate’s matrix: kij and kmn are direct neighbors (i = m ∧ j = n) ∨ (i = m ∧ j = n) or kij and kmn are connected by a chain of direct neighbors. Now every candidate set contains one single sub-scene of the image. The sets Dl still represent ambiguous sub-scene and will be analyzed individually. By using the joint probabilities as association measure, ﬁrstly all hypotheses Hn from all allocation pairs must be put up. For this, zero allocations are inserted for all objects, so that allocation rejection is possible and thus the hypotheses generation is guaranteed. With the zero allocations and the restrictions by the search areas several combinations will arise. Thus N hypotheses Hn arise according to the rule (Eq. 17). Because the allocation pairs of the hypothesis display two independent events, the similarity values (SMob ) are used here as single probabilities. The hypotheses represent the basis for the joint probability p(Hn ). The joint probabilities p(Hn ) are determined at ﬁrst for all hypotheses Hn . These consist of the single probabilities SMob (θij ) of the allocations θij : Hn = {(θio )| ∀ θio ∧ θmn with i = m∧j = n} ,

p(Hn ) =

SMob (θio ) (17)

θij∈Hn

Hypotheses with probable allocation combinations reach a strong joint probability, while in case of unlikely allocation combinations small values will be received. The joint probabilities will be exploited for the association probability. The association probabilities p(θij ) of the allocations θij arise from the sum of the joint probabilities of all hypotheses which support this allocation (Eq. 18). p(θij ) = p(Hn ) (18) ∀Hn |θij ∈Hn

The details about the derivations can be taken from [13]. The association probabilities display attenuation or reinforcements of the single probabilities. Thus, a competing allocation possibility becomes more likely which showed before a similar single probability. After this, the association probability of an unlikely allocation is considerably reduced in the hypothesis sum. After the preceding steps, the candidate sets (Al , Bl , Cl , Dl ) and the association probabilities p(θij ) of all allocation pairs from the sets Dl are available. The all allocation pairs of the objects and measurement, also the pairs of the unambiguous sets are ﬁlled out as follows on the matrix A=(aij ) with the dimension OxM: ⎧ if θij ∈ {Al , Bl , Cl } ⎨ 1, aij = p(θij ), if θij ∈ Dl (19) ⎩ 0, othewise The entries in Eq. 19 represent the object constellations with their coherence as they were in the candidate matrix. Here wider allocation possibilities can be

140

A. Al-Hamadi, S.S. Pathan, and B. Michaelis

disintegrated between O objects and M measurements. Nevertheless, the ambiguities are not dealt in this step entirely for example the ambiguity among diﬀerent number of objects and measurements. These constellations solved in ”data administration step” (Fig. 2) where the remaining allocation possibilities will be interpreted. Based on the evaluation of the association matrix the allocation pairs will be associated to one of the following four groups which represent the basis for the interpretation step are given here. • Unequivocal allocations. Entries of the association matrix lie in non-conﬂict situation and are unambiguously associated. These allocations correspond to the sets of Cl (Eq. 15). Because a measurement could be associated and the development of the respective particle ﬁlters runs in the next steps normally, these objects receive the state Active. For these pairs the motion estimate is done by the standard motion model (M MN ) in the next time step. • Ambiguous allocations.The entries of the association matrix are analyzed which have one or several neighbors. These constellations are to ﬁnd with different measurement and object number after the data exclusion and the data allocation which are caused while merging of the objects. These allocation possibilities correspond to the sets of Dl . Here, the resultant objects are characterized by its instability. Because there is no unequivocal measurement corresponding to this object, it is interpreted as a disturbance and receives the state Dead. This induces the ﬁlter system to reject this particle ﬁlter. If it concerns established particle ﬁlters, it is assumed that several objects can divide a MB and thus an equivocal allocation arises. The particle ﬁlters get state Multi and will be propagated by the experience motion model (M ME ) in the next step. • Single objects. Here, the objects are considered to which no measurement could be associated in the data exclusion. These objects have an empty column in the association matrix and correspond to the candidates sets Al (Eq. 13) which can have several causes like congestion (stopped objects) or leaving of the scene. The particle ﬁlters which correspond to none of the cases have reason by occlusion. This leads to no measurement. They are interpreted as Searching. For the particle ﬁlter the experience motion model (M ME ) is switched on. • Single measurements. The remaining allocation pairs have empty rows (candidates sets Bl ) in the association matrix. For them no corresponding object has been found in the data exclusion. If in the data exclusion no corresponding measurements were associated to an object because of a too small search area, it has been analyzed as a single object and has been labeled with the state Searching.

3

Experimental Results

The real scenes are inﬂuenced with occlusion (Fig. 3C), deformation (Fig. 3E), brightness modiﬁcation, shadow and congestion (Fig. 3D). A framework is developed and tested on several image sequences containing multiple moving objects (Fig. 3A). Despite of disturbances (perspective deformation), the determination of trajectory are reliable. The insensitiveness to this type of interference due to

MOT Based on Particle Filter and Data Association

141

the use of the suggested approach and the co-action of its components. Figure 3B shows another sequence that contain stopped object. The stopped object due to traﬃc light cannot be detected by motion detection-based procedures. To solve this conﬂict, the suggested approach is used where when the ”stop” occurs at least one particle ﬁlter of one object remains. The particle ﬁlter of the car is estimated as Standing and is preserved therefore. The adaptive motion model is switched on a standing model for this particle ﬁlter by state standing. When the car moves on, in the next time step the particle ﬁlter receives the state ”Active” and is estimated accordingly with the standard model. In ﬁgure 3B another scene is shown. Where the person moves behind a pillar and covered at short notice completely. The person cannot be recognized with motion detection, therefore no measurement is observed. The particle ﬁlter handles this by switching now in the state Searching mode according to the analysis. Therefore the experience model is enabled for next time step. The experience model estimates and varies with the help of the average movements of the last iterations of the object. The evaluation of particle occurs again with the help of a associated measurement. Therefore the particles concentrate again around the position of the object. Figure 3D and E show other results for diﬀerent sequences, i.e. motion trajectories are illustrated. It is clear from the ﬁgure that the motion trajectories are detectable with motion parameters of moving objects despite occlusion or stop.

Fig. 3. Sequence of analysis. A) Despite of perspective deformation, the determination of motion trajectories is reliable. B and C): Sequences of analysis and a associated trajectories of rigid and non-rigid moving objects under the inﬂuence of shadow, stopped objects (car in B), and occlusion (C). D and E): Analysis results of several non-rigid objects as motion trajectories. The motion trajectories describe the motion parameters of objects despite the fact that some objects are merged and split due to occlusion.

142

4

A. Al-Hamadi, S.S. Pathan, and B. Michaelis

Conclusion and Future Work

The objective in MOT is to track object under the drastic real-time situations. In this work a processing chain using diﬀerence-based approaches is used. Both the image and color parameters are consider in the formulation and state description of the objects. An adaptive model is proposed which reacts dynamically to the motion updates of the respective object as well as adapts the particle evolution. A particle ﬁlter system with an adaptive motion model has been presented for MOT that consists of independent particle ﬁlters. This system is linked with an internal data association technique which the particle ﬁlters accordingly. The results have proved the robustness and eﬀectiveness of the proposed technique. In the future work we will extend this framework for more complex scenes.

References 1. Isard, M., Blake, A.: ICondensation: Unifying low-level and high-level tracking in a stochastic framework. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 893–908. Springer, Heidelberg (1998) 2. Sidenbladh, H.: Multi-target particle ﬁltering for the probability hypothesis density. In: 6th International Conf. on Information Fusion, Cairns, vol. 2, pp. 800–806 (2003) 3. Chang, C., Ansari, R.: Kernel Particle Filter for Visual Tracking. IEEE Signal Processing Letters 12(3), 242–245 (2006) 4. Doucet, A., de Freitas, N., Murphy, K., Russell, S.: Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In: Conference on Uncertainty in Artiﬁcial Intelligence, pp. 176–183 (2000) 5. Murphy, K., Russel, S.: Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks: Sequential Monte Carlo Methods in Practice, ch. 24, pp. 499–515. Springer, Heidelberg (2001) 6. Casella, G., Robert, C.: Rao-Blackwellisation of Sampling Schemes. Biometrika, Heft 83(1), 81–94 (1996) 7. Vermaak, J., Doucet, A., P´erez, P.: Maintaining Multi-Modality through Mixture Tracking. In: IEEE Intern. Conf. on Computer Vision, vol. 2, pp. 1110–1116 (2003) 8. Vermaak, J., Gidsill, S., P´erez, P.: Monte Carlo ﬁltering for multi-target tracking and data association. IEEE Trans. on AES, Heft 41(1), 309–332 (2005) 9. Cox, I., Hingorani, S.: An Eﬃcient Implementation and Evaluation of Reid’s Multiple Hypothesis Tracking Algorithm for Visual Tracking. In: Proc. of the 12th ICPR, vol. 1, pp. 437–442 (1994) 10. Kreucher, C., Hero, A., Kastella, K.: Multiple Model Particle Filtering for Multitarget Tracking. In: Proc. of the 12th ASAP (2004) 11. Karlsson, R., Gustafsson, F.: Entropy, Monte Carlo data association for multiple target tracking. In: IEE Internat. Seminar on Target Tracking: Algorithms and Applications, vol. 1, pp. 13/1–13/5 (2001) 12. Al-Hamadi, A., Michaelis, B.: An intelligent paradigm for multi-objects tracking in crowded environment. Journal -JDIM 4(3), 183–190 (2006) 13. Helmick, R.: IMM Estimator with Nearest-Neighbor Joint Probabilistic Data Association. In: Bar-Shalom, Y. (ed.) Multitarget-Multisensor Tracking: Applications and Advances, ch. 3, pp. 161–198. AH Publishers, Boston (2000)

Residual of Resonant SVD as Salient Feature Dmitry Chetverikov Computer and Automation Research Institute, Budapest, Hungary [email protected]

Abstract. Computer vision approaches to saliency are based, among others, on uniqueness [1], local complexity [2], distinctiveness [3,4], spectral variation [5], and irregularity [6]. Saliency can also be viewed as the information in the data relative to a representation or model [7]. When a representation is built, a residual error is often minimised. The residual can be used to obtain saliency maps for solving challenging tasks of image and video processing. We introduce the notion of the resonant SVD and demonstrate that the SVD residual at the resonant spacing is selective to defects in spatially periodic surface textures and events in time-periodic videos. Examples with real-world images and videos are shown and discussed. Keywords: saliency, SVD, periodicity, texture, defects, video processing.

1

Introduction

In human vision research, the term ‘visual saliency’ refers to the capability of certain visual primitives to ‘pop-out’ in a scene in a pre-attentive, immediate way [8]. A number of computational models for saliency-based visual attention have been developed [9,10] simulating this capability of human vision. In computer vision, saliency means eﬃcient selection of informative, distinctive and stable local regions on which further processing and decision making are based. Diﬀerent salient image and video features have been proposed and successfully used in image and video retrieval, stereo vision, attention control, motion tracking, object and event detection and categorisation, and other tasks. A vast body of literature on saliency is available. In this paper, we conﬁne ourselves to the methods that were applied to various real-world data. Much of research on saliency has been devoted to aﬃne invariant feature points and regions [11,12,13,14]. These features are frequently used in practice and wellstudied [15]. Most of them are based on a measure of distinctiveness in space and/or scale. Stability over scales [16] and insensitivity to geometric and photometric changes are important requirements. The notion of distinctiveness can be extended to the spatiotemporal domain, where invariance to linear camera motion has also been addressed [4,17]. Aﬃne invariant salient regions [14] are an extension of the previous work [2] using local complexity. Other approaches to saliency are based on uniqueness [1], spectral residual [5], and irregularity [6]. The two recent studies [5,6] are of particular interest for us L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 143–153, 2009. c Springer-Verlag Berlin Heidelberg 2009

144

D. Chetverikov

as they are related to our residual-based approach. Hou and Zhang [5] make use of the scale invariance of natural image statistics, known as the 1/f law. They attribute local variations in the log spectrum of an image to the salient objects (innovation), with the smooth part being responsible for the background (prior knowledge). In practice, the method amounts to the calculation of the spectral residual as the diﬀerence between the original log spectrum and its mean-ﬁltered version. This is essentially a simple approximation of the Laplace ﬁlter in the log spectral domain. Applying the inverse Fourier transform to the spectral residual, Hou and Zhang obtain saliency maps indicating locations of perceptually relevant objects in diﬀerent kinds of natural images. The method [5] is simple and fast, and it does not rely on any speciﬁc kind of image features. Its applicability and scalability need further testing. The basic assumption also needs more justiﬁcation. Boiman and Irani [6] deﬁne irregularity as the degree to which a local pattern can be explained, in a statistical sense, by the rest of the data. A frequently observed image pattern or video volume, supported by the data, is a regular event. Such piece of data can be composed using the rest of the data, the ‘database’. A rare pattern or volume not supported by the rest of the data is an irregular event, such as a defect or suspicious behaviour. The problem of ﬁnding salient irregularities is posed as an inference process in a probabilistic graphical model. The method [6] is quite general and ﬂexible as it can cope with limited aﬃne distortion of the data. However, the approach is time-consuming since very many pieces of data are to be matched against the database. Winkeler et al. [7] deﬁne saliency as the information in the data relative to a representation. They consider eigenspace representation using the Singular Value Decomposition (SVD) for appearance-based object recognition and pose determination. Saliency is viewed as the amount of energy not captured by the basis set, that is, the residual error. During object recognition, the saliency serves as a measure of distance from the basis set indicating that the object does not belong to a given category. A similar measure was used in face detection and identiﬁcation systems as the ‘distance from face space’ [18,19] to reject non-facial images. In this study, we show that the residual can be eﬃciently used to obtain saliency maps for challenging tasks of image and video processing. The contribution of the paper is as follows. First, we consider periodic data and introduce the notion of the resonant SVD representation. Then we propose a novel saliency map for defect detection in periodic surface textures using the resonant SVD. The method needs a good estimate of the spatial period (resonant spacing) which is obtained by an algorithm based on the principles proposed in [20] for signal period estimation. Then we extend the method to the spatiotemporal domain to obtain a spatiotemporal saliency map selective to events in a periodic video, such as a time-periodic dynamic texture. The method includes a procedure for the estimation of temporal periodicity which can be useful in video mosaicking and dynamic texture recognition [21].

Residual of Resonant SVD as Salient Feature

2

145

The Resonant SVD Representation

The SVD is a robust and powerful tool having numerous applications in signal processing, computer vision, pattern recognition and other areas. In particular, it is suitable for separation of dominant, ‘typical’ data from ‘untypical’ data. Recently, it has been used for adaptive modelling of non-stationary background and detection of novelty in image sequences [22]. The detection is based on the residual error: a region is considered novel if its residual exceeds a threshold. When the dominant data is periodic, care should be taken to learn the typical periodic structure and ﬁnd untypical, salient data as signiﬁcant local deviations from this structure. As discussed by Kanjilal et al. [20], understanding a periodic signal means extracting the pattern which is repeated, the period length, and the scaling factor of each period. The signal periodicity analysis approach proposed in [20] involves signal pre-processing and normalisation steps followed by the singular value decomposition of the signal data matrix for a range of potential period lengths. For completeness, the method is brieﬂy presented below. Consider a digital signal x(k). Assume that the period is n and place the successive n-intervals of x(k) into the rows of the m × n data matrix ⎡ ⎤ x(1) x(2) . . . x(n) ⎢ x(n + 1) x(n + 2) . . . x(2n) ⎥ ⎥ An = ⎢ (1) ⎣ ... ... ... ... ⎦ x((m − 1)n + 1) ... . . . x(mn) If n equals the period length, the rows of An are linearly dependent, even when the scaling factors of the rows are diﬀerent. The method uses the robust Singular Value Decomposition (SVD) to test the ‘rank oneness’ of the matrix An and determine the repeating pattern and the scaling factors. Applying SVD, we decompose the data matrix as An = U SV T , where U U T = U T U = I, V V T = V T V = I. The diagonal matrix S contains the sorted singular values: S = diagonal(s1 , s2 , . . . , sr : 0), s1 ≥ s2 ≥ . . . sr ≥ 0, with the rank r = min(m, n). The left (right) singular vectors ui (v i ) are the columns of U (V ), respectively. The meaning of the decomposition is the following. For strictly periodic x(k) with period N , x(k) = x(k + N ) and rank(An ) = 1 when n = N . In this case, s1 > 0, s2 = s3 = . . . = sr = 0; s1 /s2 = ∞. The vector v 1 is the (normalised) periodic pattern, the elements of u1 s1 are the scaling factors; here, they are all equal. For nearly periodic x(k) with period N , but x(k) = x(k + N ), two cases are possible: (1) same repeating pattern, but with diﬀerent scaling; (2) nearly repeating patterns, with diﬀerent scaling. In case (1), rank(An ) = 1 for n = N , s1 > 0, s2 = s3 = . . . = sr = 0, and s1 /s2 = ∞. The vector v 1 is still the periodic pattern. In case (2), the matrix An can be full-rank, and s1 s2 . Summarising, s1 s2 indicates a strong primary periodic component of the length n, given by rows of the matrix u1 s1 v T1 . To obtain the further component, one can iterate the above procedure for the residual matrix An − u1 s1 v T1 .

146

D. Chetverikov

pattern

SVD model

residual

Fig. 1. Left: Periodic test pattern with an artiﬁcial defect. Middle: The resonant SVD model for the row spacing equal to the period which is 36 pixels. Right: The residual as the saliency map indicating the defect.

We calculate the periodicity spectrum (P-spectrum) as P (n) = 1 −

s2 (n) s1 (n)

Considering the dominant maxima of the P-spectrum at n = N, 2N, 3N , etc., we obtain the periodicity index (P-index) and select N with the highest maximum count as the most probable value of the period length. In [21], this method was applied to estimation of the temporal periodicity of dynamic textures. For more technical details, including pre-processing and normalisation, the reader is referred to the papers [20,21]. In our study, we use the periodicity estimation method to ﬁnd the dominant periodic structure as the data representation u1 s1 v T1 for the largest singular value s1 at the resonant value n = N . The vector v 1 is the (normalised) periodic pattern, the elements of u1 s1 are the scaling factors. The residual of this representation, AN −u1 s1 v T1 , provides the saliency maps selective to defects in spatially periodic textures and events in temporally periodic sequences. In the rest of the paper, we will call the period length the ‘resonant spacing’ and refer to the corresponding SVD representation for the largest singular value as the ‘resonant SVD representation’.

3

Detecting Defects in Spatially Periodic Textures

Defects in periodic spatial structures are salient regions that trigger human attention. The task of texture defect detection is important for a number of applications, such as fabric inspection [23] and study of imperfections in physical microstructures [24]. When the underlying structure is highly periodic or/and the defects appear as blobs of distinct colour, their perception by either human or computer is easy. Detection of ﬁne structural imperfections like the one shown in ﬁgure 1 is not trivial. In this section, we process greyscale texture patterns with orthogonal axes of periodicity and assume for simplicity that one of the axes is vertical. (A more general case will be discussed below in section 5.) The period length in

1.0

1.0

0.8

0.8

0.6

0.6

P−index

P−spectrum

Residual of Resonant SVD as Salient Feature

0.4

0.2

0.0

147

0.4

0.2

0

20

40 spacing

60

80

0.0

0

20

40 spacing

60

80

Fig. 2. The P-spectrum and P-index of the test pattern shown in ﬁgure 1

the vertical direction is estimated as described above, by putting two, three or more image rows into each row of the data matrix An to test the period length n = 2, 3, etc. For faster operation, at this stage the image resolution can be reduced. For relatively small images such as 256 × 256, no reduction in needed. Figure 2 shows the P-spectrum and P-index of the test pattern. In the P-index, the true period of 36 rows is clearly visible. Once the period N has been obtained, the matrix u1 s1 v T1 is transformed into the resonant SVD model, while the residual matrix AN − u1 s1 v T1 is converted into the residual saliency map taking the absolute values of the matrix. Figure 1 shows the resonant SVD representation of the artiﬁcial test pattern and the corresponding residual based saliency map highlighting the defect. Figure 3 demonstrates saliency maps computed for various textile images from the textile defect database TILDA [25]. The database contains many images of real defects in diﬀerent fabrics. One can see that the resonant SVD model gives an adequate representation of each texture and the residual saliency maps indicate the defects in an appropriate way. One can also see that the models for more periodic patterns are better. Later in this section we will discuss this observation and the limits of the method in more detail.

4

Detecting Events in Time-Periodic Dynamic Textures

Detecting objects against a dynamic background is a challenging problem which is of importance for traﬃc and surveillance applications. Standard target detection algorithms using adaptive background models [26] as well as more recent approaches [27] have a limited capability to cope with varying backgrounds such as trees in the wind. When the background is time-periodic, like an escalator or a conveyor, target detection may become diﬃcult, especially if the contrast is low. In this section, we attempt modelling of periodic video, viewed as a dynamic texture or background, using the same SVD based approach as we used for static texture. The only diﬀerence is that the rows of the data matrix are now composed of multiple consecutive images rather than consecutive rows, and the

148

D. Chetverikov

pattern

SVD model

residual

Fig. 3. Saliency maps of diﬀerent textile patterns containing defects. Each row shows a textile pattern (left), its resonant SVD model (middle), and the residual as the defect saliency map (right).

periodicity is temporal rather than spatial. Note that a static camera is assumed to ensure correspondence between pixels. Figure 4 shows selected frames of the ‘Escalator Steps’ video sequence whose total length is about 300 frames. In this video an irregular white step enters, passes the viewﬁeld, then exits. The size of the step grows as it approaches the camera. To estimate the temporal period in reasonable time, the original spatial resolution of 200 × 200 was reduced. Testing a wide range of potential period values took a few seconds using the fast SVD implementation provided by the GNU Scientiﬁc Library [28]. Then the algorithm was run again, at the full resolution for the resonant spacing, to obtain the full-resolution resonant SVD representation. This took a few more seconds. The second row of ﬁgure 4 displays the corresponding frames of the resonant SVD model, in which the irregular step has disappeared as untypical. The bottom row shows the residual as the saliency map selective to the irregular step. Figure 5 shows selected frames of the ‘Man and Escalator’ video sequence whose total length is about 200 frames. In this video a man leaves the escalator and exits. The rest of the video shows the empty escalator in periodic motion. The bottom row of the ﬁgure shows the residual-based saliency map indicating the man against the time-periodic background. Note that the escalator has almost disappeared in the residual. On the other hand, the contrast of the saliency map is low where the intensity diﬀerence between the man and the escalator is

Residual of Resonant SVD as Salient Feature

f25

f50

f75

f100

149

f125

Fig. 4. Selected frames of the ‘Escalator Steps’ video sequence. Top row: Initial escalator video with an irregular white step passing the viewﬁeld. The rest of the video shows regular steps in periodic motion. Middle row: The temporal resonant SVD model for the time spacing equal to the motion period which is 21 frames. Bottom row: The residual as the saliency map highlighting the irregular step.

small. The original colour video has been converted to greyscale before applying the SVD. Using colour will hopefully improve the contrast while increasing the processing time of the second pass of the algorithm, as the SVD will be applied to each channel separately. Determining the period does not need colour information.

5

Discussion and Conclusion

The SVD based method assumes correspondence between identical data points. For video, the correspondence was provided by using a static camera and assuming temporal periodicity. For texture, in our tests it was provided by appropriate ﬁxed orientation and the assumption of orthogonal axes of periodicity. In practice, textures may have varying orientations, and the assumption of orthogonality may not be valid. In general case, an aﬃne distortion should be accounted for, at least locally. The proposed texture defect detection method can be generalised by explicitly introducing such distortion and searching for the highest periodicity under a range of distortions. If the method is applied in a sliding window, this search should only be done during the initialisation. For subsequent windows, the aﬃne parameters should be slightly adjusted to account for the natural variation within the texture.

150

D. Chetverikov

f7

f10

f13

f16

f19

f22

Fig. 5. Selected frames of the ‘Man and Escalator’ video sequence. Top row: Initial video showing a man leaving the escalator. The rest of the video shows the empty escalator in periodic motion. Bottom row: The residual as the saliency map indicating the man against the time-periodic background.

SVD model

residual

Fig. 6. Importance of resonance in static data: The SVD model and the residual of the pattern shown in ﬁgure 1, computed for row spacing set to 42. The period is 36. Compare to ﬁgure 1 where the spacing matches the period.

As we consider periodic data, signiﬁcant data reduction can be achieved by the resonant singular value decomposition. For example, the resonant SVD representation of the ‘Escalator Steps’ needs 21 frames compared to the original 300. Generally, the data reduction factor equals the ratio of the total original data size to the size of the normalised periodic pattern v 1 , that is, to the number of periods in the data. (The size of the scaling factor array u1 is negligible.) The resonant models represent the periodic data in a quite authentic way, as one can see in ﬁgures 1 and 3 periodic texture and ﬁgure 4 for periodic video. At the same time, a relatively high degree of data periodicity and a good estimate of the period length are needed for the method to work properly. In fact, the former determines the latter, since precise period estimation in low-periodicity data is diﬃcult and not robust. Figures 6 and 7 illustrate the importance of obtaining a good estimate of the period length to build a good resonant representation. In ﬁgure 6, the period is set to 42 pixels while the true period is 36. The structure of the pattern and the

Residual of Resonant SVD as Salient Feature

f25

f50

f75

f100

151

f125

Fig. 7. Importance of resonance in dynamic data: The temporal SVD model (top row) and the residual (bottom row) of the sequence shown in ﬁgure 4, computed for the time spacing set to 19. The period is 21. Compare to ﬁgure 4 where the spacing matches the period.

defect are almost completely lost for the poorly selected period. Similarly, for the ‘Escalator Steps’ sequence, the result is much worse when the period is set to 19 frames instead of the correct 21, as illustrated in ﬁgure 7. For periodic data, the period is a sensitive parameter to be selected carefully. In adaptive modelling of non-stationary background with the SVD [22], the optimal number of the singular values to use is currently a subject of research, with some proposals already made. In the case of periodic data, this parameter is essentially overruled by the period length: adding more singular values to the resonant representation would bring no signiﬁcant improvement. It may even spoil the result by moving defects from the residual to the model. Summarising our discussion, the proposed approach is limited by the degree of data periodicity. This is illustrated by ﬁgure 8 that shows saliency maps of low-regularity textile patterns containing defects, taken from the TILDA database [25]. As the periodicity decreases, the resonant SVD model tends to the average intensity. When the defect has a diﬀerent colour or intensity, it can still be detected; when the defect is rather in structure than in colour, the method will fail for low-periodicity data. We have demonstrated that the residual of the resonant SVD can be used to build saliency maps for defect detection in periodic texture and event detection in periodic video To obtain a good resonant SVD representation, one needs a precise estimate of the resonant spacing, e.g., the period length. We have proposed an eﬃcient procedure to estimate the resonant spacing based on the principles introduced earlier for signal period estimation. As a by-product, the resonant SVD model can be used for compact representation of periodic image and video data, with controllable loss of information. Estimating the temporal period length and the degree of periodicity can be useful in video mosaicking and dynamic texture recognition.

152

D. Chetverikov

Fig. 8. Saliency maps of low-regularity textile patches containing defects. Each column shows a pattern and its SVD residual based saliency map.

Future work will include extending the texture defect detection algorithm to patterns with arbitrary periodicity vectors and orientation. To test our event detection algorithm, we will need much more data with characteristic time-periodic background. The most challenging task will be applying the SVD to continuous surveillance video using the sliding data window, for adaptive background modelling and object detection. Current approaches such as [22] use an approximate solution to update the SVD, which may be a drawback in the case of periodic data. Recent developments in the SVD research will hopefully make a precise solution possible.

References 1. Schiele, B.: Object recognition using multidimensional receptive ﬁled histograms. PhD thesis, INP de Grenoble (1997) 2. Kadir, T., Brady, M.: Saliency, Scale and Image Description. International Journal of Computer Vision 45, 83–105 (2001) 3. Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91–110 (2004) 4. Lindeberg, T., Akbarzadeh, A., Laptev, I.: Galilean-diagonalized spatio-temporal interest operators. In: Proc. International Conf. on Pattern Recognition, vol. 1, pp. 57–62 (2004) 5. Hou, X., Zhang, L.: Saliency Detection: A Spectral Residual Approach. In: Conf. on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 6. Boiman, O., Irani, M.: Detecting Irregularities in Images and in Video. International Journal of Computer Vision 74, 17–31 (2007) 7. Winkeler, J., Manjunath, B., Chandrasekaran, S.: Subset Selection for Active Object Recognition. In: Conf. on Computer Vision and Pattern Recognition., vol. 2, pp. 2511–2516 (1999)

Residual of Resonant SVD as Salient Feature

153

8. Treisman, A.: Preattentive processing in vision. Computer Vision, Graphics and Image Processing 31, 156–177 (1985) 9. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Analysis and Machine Intelligence 20, 1254– 1259 (1998) 10. Sun, Y., Fisher, R.: Object-based visual attention for computer vision. Artiﬁcial Intelligence 146, 77–123 (2003) 11. Matas, J., et al.: Robust wide baseline stereo from maximally stable extremal regions. In: Proc. British Machine Vision Conf., pp. 384–393 (2002) 12. Mikolajczyk, K., Schmid, C.: Scale and aﬃne invariant interest point detectors. International Journal of Computer Vision 60, 63–86 (2004) 13. Tuytelaars, T., Van Gool, L.: Matching widely separated views based on aﬃne invariant regions. International Journal of Computer Vision 59, 61–85 (2004) 14. Kadir, T., Zisserman, A., Brady, M.: An aﬃne invariant salient region detector. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 228–241. Springer, Heidelberg (2004) 15. Mikolajczyk, K., et al.: A comparison of aﬃne region detectors. International Journal of Computer Vision 65, 43–72 (2005) 16. Lindeberg, T.: Scale-Space Theory in Computer Vision. Kluwer, Dordrecht (1994) 17. Laptev, I.: On space-time interest points. International Journal of Computer Vision 64, 107–123 (2005) 18. Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3, 71–86 (1991) 19. Hjelmas, E.: Face detection: A survey. Computer Vision and Image Understanding 83, 236–274 (2001) 20. Kanjilal, P., Bhattacharya, J., Saha, G.: Robust method for periodicity detection and characterization of irregular cyclical series in terms of embedded periodic components. Physical Review E 59, 4013–4025 (1999) 21. Chetverikov, D., Fazekas, S.: On motion periodicity of dynamic textures. In: Proc. British Machine Vision Conf., vol. 1, pp. 167–176 (2006) 22. Kahl, F., Hartley, R., Hilsenstein, V.: Novelty Detection in Image Sequences with Dynamic Background. In: Comaniciu, D., Mester, R., Kanatani, K., Suter, D. (eds.) SMVP 2004. LNCS, vol. 3247, pp. 117–128. Springer, Heidelberg (2004) 23. Kumar, A.: Computer-vision-based fabric defect detection: A survey. IEEE Transactions on Industrial Electronics 55, 348–363 (2008) 24. Harting, J., Chin, J., Venturoli, M., Coveney, P.: Large-scale lattice Boltzmann simulations of complex ﬂuids: advances through the advent of computational Grids. Phil. Trans. R. Soc. A 363, 1895–1915 (2005) 25. Workgroup on Texture Analysis of DFG: TILDA Textile Texture Database (1996), lmb.informatik.uni-freiburg.de/research/dfg-texture/tilda 26. Hull, D., Nascimento, J., Ribeiro, P., et al.: Comparison of target detection algorithms using adaptive background models. In: Joint IEEE Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS), pp. 117–128 (2005) 27. Proceedings: Int. IEEE Workshop on Visual Surveillance (VS). In conjunction with CVPR 2007 (2007) 28. Free Software Foundation, Inc.: GNU Scientiﬁc Library (GSL), version gsl-1.10 (2007), http://www.gnu.org/software/gsl/

Building Pedestrian Contour Hierarchies for Improving Detection in Traﬃc Scenes Ion Giosan and Sergiu Nedevschi Tehnical University of Cluj-Napoca, Computer Science Department G. Baritiu 26-28, 400027 Cluj-Napoca, Romania {Ion.Giosan,Sergiu.Nedevschi}@cs.utcluj.ro

Abstract. This paper presents a new method for extracting pedestrian contours from images using 2D and 3D information obtained from a stereo-vision acquisition system. Two pedestrian contour types are extracted. First is obtained from static pedestrian conﬁdence images using ﬁxed background scenes and second from general traﬃc scenes having variable background. A robust approach for building contour hierarchies of these contours is then presented. First hierarchy is built of ”perfect” contours extracted from ﬁxed background scenes and the second one is built of ”imperfect” contours extracted from images with variable background. The objective is to evaluate the two hierarchies in order to identify the best one for real time pedestrian detection. Keywords: Pedestrian contours, Stereo-vision, Fixed background, Dynamic background, Contour hierarchies.

1

Introduction

In the last 20 years, the high rate of traﬃc accidents from all world has determined the building of intelligent vehicles in a seriously way. Let’s consider the number of traﬃc accidents only in Europe for illustrating the importance of this problem. In these traﬃc accidents a number of about 150.000 pedestrians are seriously injured and about 6.000 pedestrians are losing their lives [1]. A lot of technologies such as ultrasound sensors, piezo-electrical sensors, laser scanners [2], microwave radars and video cameras [3] are frequently used for pedestrian detection [4]. Although the pedestrian detection is a simple problem for humans, it is a complex one for artiﬁcial vision systems due to the large variations of human body poses, the clothes, the accessories which they are carrying and of course due to the variations of background scenes and the environment conditions. The most important thing is that the pedestrian detection should be done in real time and this converges to consider that the detection is a complex process which involves multiple eﬃcient detection methods. The usage of contour as a feature for pedestrian detection [5] is motivated because it eliminates most of the problems presented previously which could cause weak detection. The contour describes the shape of the pedestrians and it is invariant to the pedestrians’ clothes which determine diﬀerent texture aspects L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 154–163, 2009. c Springer-Verlag Berlin Heidelberg 2009

Building Pedestrian Contour Hierarchies

155

on their image. A classical approach for object detection using a hierarchy and pattern matching with distance transform is described in [6]. In [7] is presented a system of pedestrian detection based on edges and shapes but using only monocular vision. The main idea in the pattern matching is to have a collection with a lot of pedestrian contours with diﬀerent attitudes and poses. The collection contains a set of perfect pedestrian contours obtained from ideal pedestrian images and another set of contours extracted from traﬃc stereo-reconstructed images which usually present imperfections. The hierarchies of pedestrian contours will decrease the matching time between the ROI and the collection because of reducing the number of matches and oﬀering the performance to the system of performing in real-time mode. The hierarchies of pedestrian contours consist in building a contour-templates tree. Similar systems of pedestrian detection using template trees are described in [8], [9] and [10]. This paper brings some novel methods and improvements to the existing methods from literature. We have the contribution of developing the contour extraction algorithm based on depth histogram, edge tracing and active contour reﬁnement for obtaining the pedestrian contours from scenes with variable background. The contribution in hierarchy building algorithm consists in the heuristic of initial groups’ generation which oﬀers the possibility to automatically choose the initial groups. Another contribution is that we don’t ﬁx a number of templates that should contain all the groups formed in the hierarchy tree oﬀering a ﬂexibility and possibility of grouping any number of contours if they have almost the same attitudes and orientations. This new approach permits an automatic run of the hierarchy builder without setting any initial parameters and provides better results.

2

Pedestrian Contour Extraction

In this section it is presented the main idea of the pedestrian contour extraction, the contour extraction algorithm with its input and output and the types of contour extraction obtained by using diﬀerent types of input images. A solution for real-time pixel matching is to consider a smaller number of pixels involved in the process [6]. Here it is the idea of using a pedestrian contour in pattern matching process rather than the entire intensity pedestrian image. Contour extraction is useful in building a pedestrian contours collection which contains entire body contours of pedestrians. 2.1

Types of Contours

The collection contains a set of perfect pedestrian contours obtained from ideal pedestrian images and another set of contours extracted from traﬃc stereoreconstructed images which usually present imperfections. This is useful for having a large contour collection to get a realistic valid score when matching the ROI contour in scene with the models from the collection.

156

I. Giosan and S. Nedevschi

In ﬁrst case sequences of images having ﬁxed background are used to obtain ideal conﬁdence images. Then is applied the contour extraction and stored the obtained contour in the collection. In second case sequences of motion images when background is changing from one frame to another are used to obtain the approximated zones of the pedestrian from their intensity and range image. Then is applied the contour extraction on these zones, combined and formed a single contour and stored the contour in the collection. 2.2

Contour Extraction Algorithm

An object image which can contain several parts (connected components) represents the input of the algorithm. A continuous contour formed of a sequence of contour pixels which approximates the object is deﬁning the output of the algorithm. First step is to determine all the contours of the connected components. This image is a binary one containing white pixels for pedestrian and black pixels for the background. Objective is to draw only the contours of these pedestrian connected components. A border tracing is used to determine the contour of each connected component. Next step is to calculate the minimum distance between all valid contours obtained from previous steps. This distance is calculated as a Euclidian distance between all pairs of contours’ points. The minimum distance between two contours is set as being the smallest value of the distances between their contour points. A merging process between all connected components is then applied. The order of merging components is given by the ascending order of distance values between components. Merging is done between contour points of all pairs of diﬀerent connected components that are satisfying a maximum distance constraint. The result of this step is one connected component which approximates the image of the input object. Final step is to trace the contour of this connected component which will represent the output of the algorithm. The contour trace method and the considered contour points which approximate the object are the same as we previously described. 2.3

Contour Extraction from Sequences of Fixed Background Images

In this case contour is extracted from conﬁdence images (Fig. 1). Conﬁdence images are formed of connected components of a pedestrian image. After conﬁdence image is obtained then the previously described algorithm is applied to obtain the pedestrian’s contour. 2.4

Contour Extraction from Sequences of Dynamic Background Images

First step is to mark in the intensity image the region that is representing a pedestrian. This is done with mouse dragging and the selection rectangle should

Building Pedestrian Contour Hierarchies

157

Fig. 1. Pedestrian contour extracted from images with ﬁx background: a) Conﬁdence image. b) The extracted contour

a)

b)

c)

d)

Fig. 2. Selection of pedestrian area and depth histogram computation: a) Scene image from a sequence with pedestrian selected area (red). b) Pedestrian selected area. c) Depth image of the pedestrian selected area (red). d) The computed depth histogram.

be as narrow as possible to select a small amount of background in comparison with the amount of the pedestrian pixels (Fig. 2). Next step is to compute a depth histogram in range image only in the selected pedestrian area. This is used to separate the pedestrian from the background in the selected area. In Fig. 2 is presented a depth histogram of the pedestrian selected area from the scene image. Then in the previously calculated histogram is searching for the maximum value. A binary image containing white pixels for pedestrian and black pixels for background is then constructed considering the previous peak. Next step is to extract the contour (Fig. 3a) ) using the algorithm described in previous sections. Final step in contour extraction from sequences of changing background images is to reﬁne previously extracted contour. In smoothing process of this contour is used an active contour (snake). Active contour models [11] may be used in image segmentation and understanding and are also suitable for analysis of dynamic image data or 3D image data. The snake is used to best ﬁt the pedestrian contour as near as possible to the real edges.

158

I. Giosan and S. Nedevschi

a)

b)

Fig. 3. a) Contour extraction (right) on binary image of pedestrian connected components (left). b) Edge image of pedestrian (left) and smoothed contour on edge image using snake (right).

Before starting the snake it is calculated an edge image of the pedestrian selected area on intensity image using Canny edge detector and then the result is masked by depth range (Fig. 3b) ). In the edge image are also removed the small edges as being considered noise. Snake is starting from contour points previously calculated on the edge image and tries to ﬁt better on the pedestrian shape. Smoothing with active contour removes small deviations introduced by depth masking (Fig. 4).

Fig. 4. Final pedestrian contour viewed over the intensity image

3

Building the Contour Templates Hierarchies

The contour templates hierarchy is very important in the process of pedestrian detection based on shape matching because it reduces the number of matches between the region of interest (ROI) from the scene image which represents an unknown object and the entire collection of pedestrian contour templates. Contour templates hierarchy is built as a multipath tree structure, each node representing a pedestrian contour template obtained using the methods already presented in previous chapter.

Building Pedestrian Contour Hierarchies

159

An unsupervised clustering algorithm based on a modiﬁed version of classical ”K-means” algorithm is used in building of the multipath tree. This algorithm is applied to build each level in the tree hierarchy starting from leafs level to root level. At each level the input of the algorithm is a set of pedestrian contour templates, leafs level containing the entire set of templates. The result of applying the algorithm at every level is a number of template groups, each group containing similar templates and a prototype template for each group. The algorithm is following a number of well deﬁned steps presented in next paragraphs (Fig. 5).

Fig. 5. Template tree hierarchy building algorithm at each level of the tree

Before applying the clustering algorithm it is necessary to calculate the dissimilarity matrix between all templates. Considering N contour templates of pedestrians, the elements of the matrix are calculated as evaluating the minimum Euclidian distance between that pair of templates. First step of the algorithm is the initial phase of groups’ generation. In this step are generated ﬁrst groups of templates starting from a heuristic. First of all it is considered that neither of all templates belongs to a group. Then all distances from distance matrix are ordered ascending in an array. Next procedure is an iterative one: starting from the ﬁrst distance to the last one, in the ordered array, are connected as belonging to the same group or to a new group (if neither of them belongs to a group) both templates which form that distance only if at least one of them is not a member of an existing group. The result of this step is a number of group templates. Second step is calculating the prototypes for each group previously generated. The prototype is that model from the group that has the minimum of the maximal distances between it and all other templates from group.

160

I. Giosan and S. Nedevschi

Next two steps consist in assigning each template to a prototype previously calculated. A template is assigned to that prototype that has the minimum distance from it. After this step, new groups are formed, each one with a prototype. The prototype is recalculated for each newly formed group. These two steps are repeated until there are no diﬀerences between last two classiﬁcations. The result of applying this algorithm at current level is a set of groups of templates, each one with a calculated prototype. The prototypes from current level are considered input templates for the upper level. Then the algorithm is applied at upper level and so on until a single group is reached. The prototype of this single group represents the root of the multi-path tree. The number of templates from each group isn’t a constant. There can be groups with diﬀerent number of templates but at least two models in a group. In Fig. 6 it is represented a small example of a tree hierarchy of contour templates.

Fig. 6. Example of a pedestrian contour template hierarchy

4

Experimental Results

In this section we present the results obtained on contour extraction methods and then the results of constructing two template trees hierarchies of contour (perfect and imperfect) pedestrian models. All the results have been obtained by testing the proposed methods and algorithms on thousands of images in diﬀerent environments. We subjectively evaluated the good quality of the results (the shape of the extracted contours and the building of the hierarchies). 4.1

Contour Extraction Results from Sequences of Fixed Background Images

In Fig. 7 are shown the pedestrian contours extracted from conﬁdence images. The important aspect is that the contour extraction algorithm succeeded in extracting the perfect contours of the pedestrians eliminating the holes and merging the distances between the unconnected components in the pedestrian conﬁdence image.

Building Pedestrian Contour Hierarchies

a)

b)

c)

161

d)

Fig. 7. Pedestrian contour extracted from images with ﬁx background: a), c) Conﬁdence image. b), d) The extracted contours of the pedestrians in the corresponding conﬁdence images.

Fig. 8. Pedestrian contour extracted from scene image with dynamic background: a), c), e), g) Scene image and pedestrian selected area. b), d), f), h) The extracted contour of the corresponding pedestrian.

4.2

Contour Extraction Results from Sequences of Dynamic Background Images

In Fig. 8 are shown the pedestrian contours extracted from traﬃc images where the background is changing from frame to frame. The contour extraction and reﬁnement algorithm obtained a good result which approximates the contour of the pedestrian in conditions that we have available only grayscale images of the scenes. 4.3

Hierarchy of Pedestrian’s Contours Results

Here we present the results of building a pedestrian’s contours hierarchy containing ”perfect” templates of contours. These templates were obtained by

162

I. Giosan and S. Nedevschi

Fig. 9. Example of a group from the hierarchy having 6 contour templates and one prototype

extracting contours with previously presented algorithm using scenes with ﬁxed background. The hierarchy is built of 420 templates of contours previously extracted, obtaining a four level-height multi-path tree. In Fig. 9 is presented an example of a group from the hierarchy having 6 members (contour templates) and one prototype (chosen from those members). All the members from the group have the same attitude and orientation and we can say that the algorithm worked correctly in building the entire hierarchy. We have applied the same tree generation algorithm for obtaining the pedestrian’s contours hierarchy containing imperfect templates of contours (from images with variable background).

5

Conclusions

We have presented ﬁrst a pedestrian contour extraction method applied on both ﬁxed and changing background sequences of frames. 2D images scenes were acquired by a stereo camera system. We also exploited the 3D information in the contour extraction algorithm to compute the distance points from the camera to the pedestrian area and to ﬁlter out the edges that are outside the area of interest when tracing the pedestrian contour. The pedestrian contour extraction algorithm is very fast. It could perform in real-time mode (≈20 fps) even when dealing with complex scenes having a lot of objects. After extracting the pedestrian contours we have built two pedestrian hierarchies of contour templates extracted from ﬁxed and variable background scenes.

Building Pedestrian Contour Hierarchies

163

The constructed hierarchies are very useful in improving the detection of pedestrians using pattern matching technique. The process of building the hierarchies is a time consuming one. This is performed only once and only in oﬄine mode. After the hierarchies are built they could be used in real-time pedestrian classiﬁcation. A quantitative evaluation of the extracted contours and the building of the hierarchies could be accomplished in future when they are eﬀectively used in the pattern matching classiﬁcation of the pedestrians. Future work will consist ﬁrst in reﬁnement of the pedestrians contours extracted from sequences with dynamic background by using at least one color camera. The reﬁnement will consist in extracting a set of textural features which could be exploited for separating the points belonging to the pedestrian surface from those belonging to the background. As future work we also propose the evaluation of the hierarchies in order to identify the best one for obtaining higher performances in real-time pedestrian detection.

References 1. Zhenjiang, L., Kunfeng, W., Li, L., Fei-Yue, W.: A review of vision-based pedestrian detection for intelligent vehicles. In: Proceedings of the IEEE Intelligent Vehicles Symposium, December 13-15, 2006, pp. 57–62 (2006) 2. Fardi, B., Schuenert, U., Wanielik, G.: Shape and motion-based pedestrian detection in infrared images: a multi sensor approach. In: Proceedings of the IEEE Intelligent Vehicles Symposium, June 6-8, 2005, pp. 18–23 (2005) 3. Gavrila, D., Giebel, J., Munder, S.: Vision-based pedestrian detection: the protector system. In: Proceedings of the IEEE Intelligent Vehicles Symposium, June 14-17, 2004, pp. 13–18 (2004) 4. Gandhi, T., Trivedi, M.: Pedestrian collision avoidance systems: A survey of computer vision based recent studies. In: Proceedings of the IEEE Intelligent Transportation Systems Conference, pp. 976–981 (2006) 5. Hilario, C., Collado, J., Armingol, J., de la Escalera, A.: Pedestrian detection for intelligent vehicles based on active contour models and stereo vision. In: Proceedings of the IEEE Intelligent Vehicular Sytems, October 19, 2005, pp. 537–542 (2005) 6. Gavrila, D., Philomin, V.: Real-time object detection for “smart” vehicles. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 87–93 (1999) 7. Broggi, A., Bertozzi, M., Fascioli, A., Sechi, M.: Shape-based pedestrian detection. In: Proceedings of the IEEE Intelligent Vehicles Symposium, October 3-5, 2000, pp. 215–220 (2000) 8. Gavrila, D.: Pedestrian detection from a moving vehicle. In: Proceedings of the European Conference on Computer Vision, January 01, 2000, pp. 37–49 (2000) 9. Gavrila, D., Munder, S.: Multi-cue pedestrian detection and tracking from a moving vehicle. International Journal of Computer Vision, 41–59 (2007) 10. Nedevschi, S., Tomiuc, C., Bota, S.: Stereo based pedestrian detection for collision avoidance applications. In: Proceedings of ICRA Workshop: Planning, Perception and Navigation for Intelligent Vehicles, pp. 39–44 (2007) 11. Kass, D.T.M., Witkin, A.: Snakes: Active contour models. International Journal of Computer Vision, 321–331 (1988)

Potential Field Based Camera Collisions Detection within Translating 3D Objects Adam Wojciechowski Institute of Computer Science, Technical University of Lodz Wolczanska 215, 93-005 Lodz, Poland [email protected]

Abstract. Existing collision detection methods usually need long precalculation stage or diﬃcult, time-consuming real-time computation. Moreover its eﬀectiveness considerably decreases while the complexity of the scene or objects increases. Especially dynamic scenes with moving objects invoke necessity of each frame collisions recalculation due to changeable objects position. So far seemingly promising solutions supported by potential ﬁelds do not introduce satisfactory functionality as they are mainly devoted to static scenes with one predeﬁned aim. This paper introduces method providing a new potential ﬁeld construction which lets the camera reach both static and translating objects without constraints and protects user from getting into their structure while approaching volatile goals. Additionally proposed method comprises easy to bring through precalculation stage and becomes a scene-complexity independent solution.

1

Introduction

Collision detection is a fundamental issue underlying objects simulation process. It is important especially in 3D games or VR applications devoted to various phenomena simulation when movement of the camera should be limited or behavior of mutually moving objects is considered. The process of collision detection is theoretically very easy however the cost of calculations increases steeply with a scene complexity. The phenomena can be complicated not only by complexity of an individual object but by amount of independently moving objects to consider as well. Nevertheless the most common scenario in both games or VR application is one or a few moving objects in a static environment. Collisions are usually resolved in two steps [10]. First step bases on a broad phase approach when algorithms cull away pairs of objects that can not possibly collide. This level usually makes use of various pre-calculated bounding volume hierarchies. Second step is called a narrow phase approach and applies accurate collision detection by means of real-time face-face arithmetic intersection calculations. As complexity of the objects may vary (regular box vs. very irregular tree), arithmetic calculations costs make programmers simplify considered objects. Many types of bounding boxes are used instead of originals while calculations. Considerable reduction in number of faces is rather unavoidable in contemporary real-time applications. Moreover users had already accustomed to simpliﬁcations, i.e.: nobody expects L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 164–173, 2009. c Springer-Verlag Berlin Heidelberg 2009

Potential Field Based Camera Collisions Detection

165

that thin branch of the tree can be an obstacle on his way whereas the tree trunk is always expected to stop the user. Methods based on potential ﬁelds belong to the group of approximated collisions. They require pre-calculation stage however they considerably relieve calculation process. Potential ﬁelds assigned to the scenes objects become a source of potential ﬁeld forces. These forces retrieved adequately to an actual camera position inﬂuence their consequent behavior. Nevertheless methods may vary in accuracy and memory occupation. Presented method claims to be more precise than others presented in literature.

2

Related Work

Construction of a potential ﬁled already met in literature was considered mainly in robotics [3], [6], [9]. In such environments there is usually one goal and a few static obstacles which should be passed by on the way to the predeﬁned aim. Potential distribution is in conformity with a mathematical potential ﬁeld theory. Goal object is placed in the minimum of a local potential ﬁeld and the obstacles are situated in local ﬁelds maximums. Finally algebraic superposition of the local ﬁelds is carried out. As a result global potential ﬁeld distribution is formed and robot can follow the way from its initial position to a global potential ﬁeld minimum (goal). Potential ﬁeld construction methods are used analogically in 3D environments where robot is replaced with a virtual camera. Nevertheless potential ﬁeld characteristic, prevailing currently in literature, has many destructive drawbacks [1], [2], [7], [8], [13]. Local potential ﬁelds coming out of obstacles form unpredictable extremums which are situated in accidental places and they are blocking the camera on the way to the aim [4], [5]. Forces coming out of potential ﬁeld distribution do net reﬂect objects shape, do not let the camera approach obstacles and moreover do not protect camera from getting into the objects. Such potential ﬁeld traps must be solved by additional correction algorithms [7], [8], [13]. On contrary to robotics assumptions, most of the applications require unconstrained camera movement without one predeﬁned goal. Camera should neither get into the objects nor be blocked in accidental places. Collision detection should be precise enough to let the user reach each place in the scene not occupied by the objects. Such conditions can be fulﬁlled by forces coming out of newly constructed potential ﬁeld distribution both in static and dynamic environments. The most relevant to this paper is Beckhaus CubicalPath method [1], [2] however, in case of dynamic objects, author suﬀers from time-consuming potential ﬁeld recalculation process. Presented paper is a development of a previous work on collisions with static objects [12]. As a continuation superposition of potential ﬁeld distributions, assigned to static objects, is considered. Additionally, retrievement of potential ﬁeld forces, assigned to the dynamic objects translating throughout the scene, is discussed. Consequent superposition of potential ﬁeld forces, inﬂuencing ﬁnal camera movement, is presented.

166

3

A. Wojciechowski

Potential Field Construction

Movement of the camera, representing users point of view, within a three dimensional environment can be constrained to predeﬁned limits that do not reduce methods utility. The altitude of the camera (the shortest distance between camera position and the ground) achieved while the movement is usually ﬁxed or constrained to certain levels: an ordinary height (hc ), minimum height (hmin ) and maximum height (hmax ). These are adequately: level achieved while walking mode (hc ), level achieved while squatting (hmin ) and level achieved while jumping (hmax ). Such assumption cause that only objects which parts are placed lower then actual camera height (hc , hmax or hmin) can collide with the camera. Limitations generated for hc , hmin and hmax can diﬀer and depend on objects structure. Such environments are usually called a 2,5 dimensional as 3D environment is represented as a set of 2 dimensional slices. As a result, for static objects, adequately three independent potential ﬁeld distributions are generated for each discrete camera height and interpolated if necessary [12]. In previous paper thorough study about potential ﬁeld distribution and resulting forces accompanying static objects is presented [11], [12]. As a ﬁrst stage collision contour is made by an orthogonal XZ projection of the objects faces whose at least one vertical (Y ) vertex co-ordinate is lower then hc (ﬁg. 1).

Fig. 1. Construction of an objects contour and deﬁnition of the objects vicinity (R1 )

Analogically for static objects two additional contours for hmin and hmax are introduced. In case of dynamic objects number of distributions and their levels depends on object mobility and is discussed in the 5-th chapter. Contour obtained on the ground surface reﬂects the barrier that can not be exceeded by the camera. Contours would be also used as a reference shape for a potential ﬁeld calculation. Potential values discretely deﬁned for each node of a square mesh depend on a distance between the node and the contour. Density and size of the mesh is predeﬁned and reﬂects size and complexity of the scene. In a ﬁgure 1 a simple convex object is demonstrated. However proposed method can ﬂexibly

Potential Field Based Camera Collisions Detection

167

Fig. 2. An exemplary contour and its potential ﬁeld distribution

be adjusted to more complex objects by increasing the density of a mesh the contour is projected on Δd determines the minimum thickness of the part of the object that should be considered in collision detection. Potential values (Ucol ) for each node (qi,j ) of the mesh is calculated according to equation 1. Ucol (qi,j ) =

1

1 − max 1, antie (1 −

r R)

·

1.5 R antie( Δd )

if r > R1 if r ≤ R1

(1)

where r is a distance between contour and node qij , R is a radius of objects bounding sphere, R1 is a predeﬁned vicinity of the contour that provides descent distance between the camera and the object. It compensates divergences between real space occupied by the object and its orthogonal reproduction, antie(x) is an integer part of x and Δd is a minimum acceptable objects thickness. Exemplary potential ﬁeld distribution calculated with equation 1 is presented in ﬁgure 2. Potential ﬁeld is a source of forces calculated for each node qi,j according to equation 2. F(qi,j ) = < Fx (qi,j ), Fy (qi,j ), Fz (qi,j ) > Fx (qi,j ) = [U (qi−1,j+1 ) − U (qi−1,j−1 ) + U (qi,j+1 ) − U (qi,j−1 ) + +U (qi+1,j+1 ) − U (qi+1,j−1 )] Fy (qi,j ) = U (qi,j )

(2)

Fz (qi,j ) = [U (qi+1,j−1 ) − U (qi−1,j−1 ) + U (qi+1,j ) − U (qi−1,j ) + +U (qi+1,j+1 ) − U (qi−1,j+1 )] where i, j are mesh nodes indexes. Corresponding forces distribution for an exemplary part of the object is presented in ﬁgure 3. Forces calculated for each node of the mesh can be considered individually while camera movement. However the mean force (Fmed ) for an actual camera position (pos) should be calculated. Interpolated mean of 16 neighboring, discretely deﬁned, potential ﬁeld forces (eq. 3), assures ﬂuency of the camera translation.

168

A. Wojciechowski

Fig. 3. Potential ﬁeld forces distribution for an exemplary objects contour

Fmed (pos) =

s=i+2,t=j+2

FXZ (qst ) · exp−distance (pos; qst )

(3)

s=i−1,t=j−1

where distance(pos, qst ) is a distance between actual camera position (pos) and a node of the mesh (qst ).

4

Compound Dynamic Environment

Dynamic compound environment comprises usually many static and a few dynamic objects. Introduced potential ﬁeld distributions (eq. 1) can be then superposed depending on the objects character. All distributions associated with static objects can be uniﬁed according to equation 4. ⎧ ⎪ ⎨ min{U1 (qi,j ), U2 (qi,j ), . . . , Us (qi,j )} if ∃s ∈ {1, 2, . . . , k − 1}Us (qi,j ) =1 Uk (qi,j ) = (4) ⎪ ⎩1 if ∀s ∈ {1, 2, . . . , k − 1}Us (qi,j ) = 1 where Uk (qi,j ) is an uniﬁed potential value for node qi,j calculated basing on k − 1 corresponding elementary values and Us (qi,j ) is a source potential value of a static object s. Uniﬁed potential ﬁeld, calculated according to equation 4, reﬂects universal ﬁeld distribution for the whole static part of the scene. For a whole uniﬁed static distribution corresponding ﬁeld forces are calculated with equation 2. In case of dynamic objects, potential ﬁeld distributions must be preserved individually as movement of the object may result in change of the potential ﬁeld distribution. Yet forces superposition must be performed in each frame of the simulation the process is very cost-eﬀective. The forces are pre-calculated and should be only retrieved from the memory and eventually interpolated. Only one resulting force is calculated for each dynamic object with respect to an actual camera position. Only forces derived from uniﬁed static potential ﬁeld and dynamic objects (being within a vicinity of the observer) may repulse the camera simulating collisions with them.

Potential Field Based Camera Collisions Detection

5

169

Camera Navigation in a Dynamic Environment

By default camera is steered by input devices. Application of input devices generates vector K concurrent with a direction of camera movement (K = α1 · dir). As a result position of the camera (pos) is modiﬁed by vector K in each frame of the simulation (ﬁg. 4).

a)

b)

Fig. 4. Modiﬁcation of the camera position by input devices. a) position of the camera (pos) modiﬁed by moving FRONT or BACK, i.e. pressing keyboard UP or DOWN arrows; b) direction of the camera (dir) modiﬁed by turning LEFT or RIGHT, i.e. pressing keyboard LEFT or RIGHT arrows.

Camera steered by input devices can freely navigate throughout the scene until it reaches potential ﬁeld gradient in a static or dynamic objects neighborhood. In this area potential ﬁeld forces would contradict camera tendencies to approach objects contour. Final modiﬁcation of the camera position pos), inﬂuenced by all static and dynamic objects forces, can be calculated in accordance with equation 5. pos = pos ⊕ (dir · α1 ) ⊕ (Fstat (pos) · α2 ) ⊕

F

(Fdyni (pos) · αi+2 )

(5)

i=1

where F stat is an uniﬁed force coming from static objects, Fdyni is a force coming out of i-th dynamic object and αi are scalar coeﬃcients modifying inﬂuence of input devices and collision forces. Studies on α1 and α2 correlation is tested and presented in [12]. Fstat is calculated basing on uniﬁed potential ﬁeld distribution and Fdyni are calculated individually for each dynamic object depending on their movement characteristics. All are calculated in accordance with equation 3. In general each objects transformation can be decomposed into global translation and rotation around local pivot. As this paper concentrates only on objects translations so dynamic forces coming out of translating objects would be considered (Fdyni = Fdyntrani ). At the same time any physical object translation, calculated for a single frame, can be interpolated by a linear vector and decomposed into horizontal (hor) and vertical movement (ver). That is why in the

170

A. Wojciechowski

paper translation is considered separately for horizontal (plane XZ) and vertical movement (direction Y ) and ﬁnally combined to provide solution for any translation. If object moves horizontally by vector hor the potential ﬁeld assigned to the object does not change its distribution but moves together with the object (ﬁg.5).

Fig. 5. Horizontal translation of the object results in adequate distribution movement

As a consequence, a new force inﬂuencing the camera after object translation (Fdyntrani ) is equal to the mean force (F medi ) for the modiﬁed camera position (pos − hor) (ﬁg.6) (eq.6). Fdyntrani (pos) = Fmedi (pos − hor)

(6)

where Fmedi is a force from equation 3, calculated for a modiﬁed position within i-th object distribution.

Fig. 6. Horizontal translation of the object (hor) can be used for an actual force calculation (Fdyntrani )

If object has vertical movement (ver) part of the object that can collide with a camera can change in time. Exemplary set of 4 potential ﬁeld distributions generated for vertically moving cone is presented in a ﬁgure 7. In such a case possible objects height range [h1 ; h4 ] must be discreted (hj for j = 1, 2, 3, 4) according to object complexity. The considerable change in a size of

Potential Field Based Camera Collisions Detection

171

Fig. 7. Set of 4 potential ﬁeld distributions generated for vertical object movement; a) h1 objects minimum height with adequate distribution; d) h4 objects maximum height with its distribution; b) c) h2 and h3 intermediate heights with their key distributions

objects projection contour (ﬁg. 1) or its shape should be the key aspects aﬀecting the number and position of intermediate distributions. For each discrete objects height (hj ) individual potential ﬁeld distribution must be generated according to equation 1. Respecting the fact that, simultaneously with object, camera can also change its height from hc to hmin or hmax the key distributions should cover extended height range [h1 − abs(hmax − hc ); h4 + abs(hc − hmin )]. So calculated minimum height value can be additionally modiﬁed due to possible object ﬂoor collision. Summing up, for any translating object, adequate potential ﬁeld force can be retrieved basing on initial key distributions. An actual object height (hob ·[h1 ; h4 ]) and relative to original position horizontal translation (hor) can be used for a valid force (Fdyntrani ) calculation (eq. 7). Fdyntrani (pos) =

hob − hj · Fmedih (pos − hor) + j hj+1 + 1−hj hj+1 − hob + · Fmedih (pos − hor) j+1 hj+1 + 1−hj

where Fdyntranih and Fdyntranih j

j+1

(7)

are forces coming out of the closest to an

actual object height (hj ≤ hob ≤ hj+1 ) distributions.

172

6

A. Wojciechowski

Tests and Conclusions

To proof eﬀectiveness of the method adequate tests were performed. Special scene was constructed comprising both simple convex and complex non-convex objects translating and rotating throughout the scene (ﬁg. 8).

Fig. 8. A dynamic scene comprising convex and non-convex objects moving throughout the scene. Upper part describe type of movement and point to visit for the objects.

Yet this paper considers only collisions for translating objects, potential ﬁeld method was introduced for rotating objects as well, but it would be presented in another paper. 21 people took part in the test, 2 women and 19 men. They were between 21 and 28, and most of them were 4th and 5th year students of the computer science faculty. 12 participants declared themselves as experienced users and 9 as medium experienced users. In the beginning of the test users were situated next to the entrance. The task was to visit the places marked with red dots and leave the scene through exit (ﬁg.8). Order and time were not taken into consideration. Only users experience was collected by means of questionnaire. In the anquette people were asked to write some personal details like: gender, age, occupation and 3D navigation experience. They were also asked to answer the question whether they had any problems while approaching the objects during scene exploration and whether they experienced entering the structure of the objects. The most spectacular fact is that all of the participants could approach the objects without any problems, and nobody had experienced entering objects

Potential Field Based Camera Collisions Detection

173

structure. The results of the test proof that the proposed method behaves in a proper manner, detecting camera collisions with both static and translating objects structure and let users navigate within a scene without disturbances.

References 1. Beckhaus, S., Ritter, F., Strothotte, T.: CubicalPath -Dynamic Potential Fields for Guided Exploration in Virtual Environments. In: Eighth Paciﬁc Conference on Computer Graphics and Applications IEEE (2000) 2. Beckhaus, S.: Dynamic Potential Fields for Guided Exploration in Virtual Environments. PhD thesis, Otto-von-Guericke-Universit¨ at Magdeburg (2002) 3. Dulba, I.: Metody i algorytmy planowania ruchu robotw mobilnych i manipulacyjnych. Akademicka Oﬁcyna Wydawnicza Exit (2004) 4. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. International Journal of Mobile Research 5(1), 90–99 (1986) 5. Khatib, M., Chatila, R.: An Extended Potential Field Approach for Mobile Robot Sensor-based Motions. In: Intl. Conf. on Intelligent Autonomous Systems IAS 4 (1995) 6. Latombe, J.C.: Robot Motion Planning. Kluwer Academic Publishers, Dordrecht (1991) 7. Li, T.Y., Chou, H.C.: Improving Navigation Eﬃciency with Artiﬁcial Force Fields. In: Proceedings of 14th IPPR Conference on Computer Vision, Graphics and Image Processing, Taiwan (2001) 8. Li, T.Y., Hsu, S.W.: An Intelligent 3D User Interface Adapting to User Control Behaviours. In: Proceedings of the 9th International Conference on Intelligent User Interface, Madeira, Funchal, Portugal, pp. 184–190 (2004) 9. Murphy, R.R.: Introduction to AI Robotics. MIT Press, Cambridge (2000) 10. Watt, A., Policarpo, F.: 3D Games. Real-time Rendering and Software Technology. Addison-Wesley, Reading (2002) 11. Wojciechowski, A.: Dynamic navigation support in a virtual environment. PhD Thesis, Technical University of Lodz (2005) 12. Wojciechowski, A.: Potential ﬁeld based camera collisions detection in a static 3D environment. Machine Graphics and Vision 15(3), 665–672 (2006) 13. Xiao, D., Hubbold, R.: Navigation Guided by Artiﬁcial Force Fields. In: Proceedings of CHI 1998, Los Angeles, USA (1998)

A Simple Quantitative Model of AVC/H.264 Video Coders Tomasz Grajek and Marek Doma´ nski Pozna´ n University of Technology Chair of Multimedia Telecommunication and Microelectronics ul. Polanka 3, 60-965 Pozna´ n, Poland {tgrajek,domanski}@et.put.poznan.pl

Abstract. The paper describes a simple quantitative model of AVC/ H.264 coders. The model deﬁnes the relationship between the bitstream and the quantization step (Qstep) for I- and P-frames. The whole allowed range of Qstep values has been divided into 3 intervals. In 1st and 3rd interval, the proposed model has only one parameter that depends on sequence content, whereas in 2nd interval the proposed model has three parameters that depend on sequence content. The experiments have been conducted on 4CIF sequences and showed that proposed model ﬁts experimental data very well in all intervals. Keywords: compression, video coding, AVC, MPEG-4, H.264, video coder modeling.

1

Introduction

Despite of hundreds of millions of video coders working worldwide, designing eﬃcient control algorithms is still an open problem that permanently gains a lot of attention. The problem was already quite diﬃcult for classic video encoders like MPEG-2 [2], but it has become even more severe with emerging of the new generation of advanced video encoders. Among them, there are the stateof-the-art video encoders that are compliant with ITU-T H.264 and MPEG-4 AVC standards [1], called also brieﬂy AVC/H.264 encoders. In particular, bitrate control is quite diﬃcult for such video coders. Recently, the bitrate control problem became even more crucial because of wide proliferation of video streaming in communication networks with rapidly varying throughput. It is related to numerous applications of wireless video transmission. In such applications, we need bitrate control techniques that are capable to cope with rapid variations of the available channel throughput. The techniques have to inﬂuence the Video Compression Layer in such a way that the number of produced bits will match the currently available channel throughput. Similarly to prior video compression standards, AVC/H.264 [1] does not standardize any rate control algorithm (some suggestions can be found in informative part of the standard). Therefore, encoders’ designers can freely optimize their L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 174–183, 2009. c Springer-Verlag Berlin Heidelberg 2009

A Simple Quantitative Model of AVC/H.264 Video Coders

175

rate control algorithms that have essential inﬂuence on encoding eﬃciency and performance. The basic parameter that can be used to control an encoder is the quantization parameter Qstep that deﬁnes quantization step for transform coeﬃcients. A typical goal of adjusting the parameter Qstep is to match the available channel bitrate. The quantization parameter Qstep may be adjusted on frame level, slice level as well as on macroblock level. The results of this paper are relevant to global bitrate control, i.e. bitrate control on the frame and slice levels only. Unfortunately, there is no a universal quantitative mathematical model that allows for exact calculations of coder parameters from given bitrate and video quality. In particular, we are searching for a model that will deﬁne a bitrate B as a function of the quantization parameter Qstep. Of course, such a relation depends strongly on video content, so the model would need to take it into account. Similar problems have been already considered by several authors [3-11]. A very brief review of these solutions will be given in the next section. Nevertheless, none of those references proposes a simple statistical model of AVC/H.264 coder similar to that proposed in this paper. Moreover, we assume that the relation B(Qstep ) is stable in time, so the model of the currently encoded frame may be deduced from the previous frames. Such an assumption has been justiﬁed by many empirical observations [3, 6, 10].

2

Advanced Video Coder Modeling

In informative part of AVC/H.264 standard, mode selection algorithm is based on Rate-Distortion Optimization (RDO). The algorithm needs to know a value of quantization index QP (directly related to Qstep ), but it can be determined based on the mean absolute diﬀerence (MAD) only after optimization. To cope with this problem a linear model for MAD prediction has been proposed [3]. Then quadratic rate-quantization model is used to calculate QP value [4]. Although, RDO results in a “chicken and egg” dilemma and makes rate control algorithm more complicated, many existing rate control schemes are based on these two models. Diﬀerent approach to rate control presented He and Mitra in [5] by proposing a linear rho-domain source model, where ρ denotes percentage of zeros in quantized transform coeﬃcients. Because it turned out to be very accurate in source content estimation, several new rate control models have been developed based on their observations e.g. [6, 7]. Other proposals how to avoid a “chicken and egg” dilemma are described in [8] and [9]. In [8], authors have proposed a rate-complexity-quantization model based on observations that coded bits have linear relationship with proposed frame complexity measure and exponential relationship with QP index. In [9], authors use three linear mathematical models to describe relationships between QP parameter, quality (PSNR used as a measurement metric) and bitrate. Our proposal is to treat a coder as a “black box” with one input (sequence) and one output (bitstream), controlled by only one parameter - Qstep and ﬁnd

176

T. Grajek and M. Doma´ nski

Fig. 1. The experimental curves for an I frame for 3 test sequences. For the sake of clarity, range of Qstep has been clipped to 120.

appropriate mathematical description of its behavior, which allowed us to estimate a given bitrate with high accuracy.

3

Proposed Model

Our objective is to ﬁnd the relationship between the number of bits B and quantization parameter Qstep for a given frame type in a given sequence. The relation will be established by analysis of experimental data. To collect these data, we encoded many test sequences with various, but constant Qstep values and calculated frame size as an average number of bits needed to encode frame over frames of the same type. Of course, each value of Qstep was set indirectly by properly setting the value of quantization index QP . In [10, 11], a simple global model of MPEG-2 bitstream has been proposed. It was created by function ﬁtting to experimental data [12, 13]. On the assumption that similar model may be obtained for AVC/H.264 standard, the authors applied the function ﬁtting method to experimental data obtained from AVC/H.264 encoder. Because ﬁnding a good approximation of experimental data for the whole allowed range of Qstep values turned out to be much more diﬃcult that for the MPEG-2 data, the authors decided to divide the experimental curves into 3 intervals (Fig. 1) and perform function ﬁtting in each one separately. These intervals are: – 1st − Qstep ∈< 0.625, 2.25 >; – 2nd − Qstep ∈< 2.5, 104 >; – 3rd − Qstep ∈< 112, 224 >; All research has been made on various 4CIF (704 × 576 pixels) sequences with diﬀerent motion characteristics. All sequences have been encoded with

A Simple Quantitative Model of AVC/H.264 Video Coders

177

AVC/H.264 reference software version JM 13.2 [14] (main proﬁle, CABAC and RDO enabled). Sequences bluesky, pedestrian, riverbed, rushhour, station2, sunflower and tractor have been cropped to 4CIF resolution from their original size - 720p (1080 × 720 pixels). 3.1

Model for the 1st Interval

Function ﬁtting applied to the data from the 1st interval resulted in quadratic model as follows: B(Qstep ) = a∗ Qstep 2 + b∗ Qstep + c, (1) where a, b and c are real constants that depend on sequence content and B(Qstep ) is the number of bits per frame for a given Qstep value. The parameters’ values have been estimated by minimization of maximum approximation error over the interval of the allowed values of Qstep: ε(Qstep , a, b, c) =

|Bx (Qstep ) − B(Qstep , a, b, c)| ∗ 100%, Bx (Qstep )

min max ε(Qstep , a, b, c), a,b,c Qstep

(2)

(3)

where BX (Qstep ) denotes measured value and B(Qstep , a, b, c) denotes the approximated one. Further, detailed analyses showed that there is linear relationship between those parameters. Therefore, model with only one free parameter has been evaluated separately for I- (Eq. 4) and P-frames (Eq. 5). B(Qstep ) = [(0.1169∗c) − 26042.8]∗Qstep 2 +[(−0.549∗c)+ 8228.2]∗Qstep + c (4) B(Qstep ) = [(0.1417∗c) − 41521.7]∗Qstep 2 + [(−0.58∗ c) − 58830.1]∗Qstep + c (5) Fig. 2 shows experimental and approximated curves for an I-frame for 3 exemplary sequences. Parameter’s values, maximum and average errors for all analyzed sequences have been shown in Table 1. Average relative error for most sequences is below 2% and 3% for I- and P-frames, respectively. 3.2

Model for the 2nd Interval

Function ﬁtting applied to the data from the 2nd interval resulted in hyperbolic model as follows: a B(Qstep ) = , (6) Qstep b + c where a, b and c are real constants that depend on sequence content and B(Qstep ) is the number of bits per frame for a given Qstep value. The parameters’ values can be estimated by minimization of maximum approximation error similarly as

178

T. Grajek and M. Doma´ nski

Fig. 2. Experimental and approximated curves for an I-frame for 3 exemplary test sequences

Table 1. Estimated parameters, maximum and average error for I- and P-frames (1st interval)

Sequence basket bluesky bus cheer city crew ﬂow football harbour ice icon pedestrian riverbed rushhour soccer station2 stefan sunﬂower tractor universal warner

5 4 3 4 4 3 6 3 4 2 1 2 4 2 3 3 3 3 4 1 1

c 116 477 889 774 483 682 214 102 599 507 858 699 127 481 794 542 876 399 143 660 900

391 530 417 414 201 216 351 568 594 134 923 532 412 953 015 499 281 590 423 066 413

I frames max err avg. err 2.23 0.80 2.23 0.94 1.78 0.80 2.50 1.37 1.99 0.92 3.15 2.17 2.91 1.62 4.28 2.13 2.02 1.06 4.20 1.86 3.70 2.42 3.25 1.32 2.37 1.15 6.50 3.97 1.90 0.88 2.79 1.40 1.97 0.85 3.63 2.08 2.73 1.35 3.38 1.69 2.52 0.93

4 2 3 3 3 3 3 3 4 2 2 2 4 2 3 2 3 2 3 1 2

c 007 633 044 927 675 975 587 209 091 316 040 437 485 313 583 203 144 150 357 927 473

522 699 718 236 306 767 493 110 791 404 221 282 542 825 624 927 886 666 022 193 887

P-frames max err 3.94 4.29 4.31 3.82 3.81 3.43 3.17 3.92 3.10 5.14 5.63 5.29 3.89 9.35 3.36 12.07 4.14 11.35 3.86 8.61 4.28

avg. err 2.01 2.06 2.83 2.48 2.03 1.95 1.85 2.67 2.27 3.13 4.16 2.84 1.88 5.06 2.04 6.71 2.56 5.76 1.43 6.35 2.89

A Simple Quantitative Model of AVC/H.264 Video Coders

179

Fig. 3. Experimental and approximated curves for an I-frame for 3 exemplary test sequences

Table 2. Estimated parameters, maximum and average error for I-frames (2nd interval) Sequence basket bluesky bus cheer city crew ﬂow football harbour ice icon pedestrian riverbed rushhour soccer station2 stefan sunﬂower tractor universal warner

7 6 6 5 11 3 9 2 6 3 2 8 1 10 8 3 3 6 1 1

a 684 782 073 108 301 951 196 169 955 914 365 256 236 126 422 604 423 481 731 082 187

685 429 825 101 283 910 262 293 071 719 224 976 417 003 397 979 532 803 642 796 652

b 0.9733 0.9528 1.0313 0.8916 1.1974 1.0928 0.9348 0.9924 1.0265 0.8350 1.2537 1.0105 1.1560 0.8252 1.2884 1.3068 0.8663 0.9322 1.0652 0.9454 0.9509

c max err avg. err 2.5707 3.41 1.53 2.8494 3.39 1.64 2.4884 4.41 2.22 1.1280 2.81 1.34 5.0215 8.04 4.88 0.7114 4.63 1.94 2.5798 4.10 1.75 -0.1577 3.68 1.83 2.3675 3.92 2.01 -0.8556 5.70 3.09 3.6570 5.32 2.68 0.5553 3.19 1.52 4.2279 6.11 3.27 -0.4699 5.00 2.34 6.2085 7.01 3.64 5.3327 6.94 3.79 0.6926 2.94 1.29 1.1968 3.85 1.84 2.9397 3.77 1.96 0.0177 3.69 1.87 0.0656 10.23 5.55

180

T. Grajek and M. Doma´ nski

Table 3. Estimated parameters, maximum and average error for P-frames (2nd interval) Sequence basket bluesky bus cheer city crew ﬂow football harbour ice icon pedestrian riverbed rushhour soccer station2 stefan sunﬂower tractor universal warner

7 2 3 4 20 4 11 2 18 1 9 1 3 4 1 3 1 1

a 772 776 538 158 118 714 678 238 238 618 051 854 229 143 746 420 158 147 796 143 582

281 440 115 390 704 604 402 445 524 532 675 916 901 222 844 046 669 158 236 946 011

b 1.2572 1.2466 1.2107 1.0059 1.8128 1.1975 1.5041 1.0383 1.5378 0.9022 1.0130 0.8926 1.1669 1.0798 1.1473 1.0569 1.3048 1.1594 1.1578 1.0087 0.9732

c max err avg. err 3.6677 4.17 1.89 1.0412 5.39 2.62 1.5042 8.10 3.94 1.0481 3.11 1.33 25.6179 36.67 18.60 0.9320 5.50 2.62 9.4722 14.29 7.02 -0.0666 3.76 2.05 12.7368 9.88 4.82 -1.0627 6.33 3.15 -0.0798 13.12 6.85 -0.7951 6.86 2.76 4.3571 5.01 2.28 -0.3479 4.59 1.79 0.6321 7.79 3.87 -1.6825 19.91 11.39 1.8152 9.10 5.37 -0.3354 10.47 5.51 1.2874 5.11 2.18 0.2185 5.96 3.28 0.0250 3.84 1.89

in section 3.1. Fig. 3 shows experimental and approximated curves for an I-frame for 3 exemplary sequences. The values of all three parameters, maximum and average errors for all analyzed sequences have been shown in Table 2 and Table 3. For I-frames average relative error for most sequences is lower than 4% and for P-frames is below 7%. However, sequences city and station2 have bigger average relative errors for P-frames mainly due to very complicated content and motion characteristic. 3.3

Model for the 3rd Interval

Similarly to results achieved in the section 3.1, function ﬁtting applied to data from the 3rd interval resulted in quadratic model (Eq. 1). Detailed analyses showed that the model can also be simpliﬁed. Therefore, model with only one free parameter has been evaluated separately for I-(Eq. 7) and P-frames (Eq. 8). B(Qstep ) = [(0.000014∗c)−0.00199]∗Qstep 2+[(−0.00677∗c)+11.173]∗Qstep +c (7) B(Qstep ) = [(0.000013∗c)−0.01656]∗Qstep 2 +[(−0.00671∗c)+10.5]∗ Qstep +c (8) Fig. 4 shows experimental and approximated curves for an I-frame for 3 exemplary sequences. Parameter’s values and obtained maximum and average errors for all analyzed sequences have been shown in Table 4.

A Simple Quantitative Model of AVC/H.264 Video Coders

181

Fig. 4. Experimental and approximated curves for an I-frame for 3 exemplary test sequences

Table 4. Estimated parameters, maximum and average error for I-and P-frames (3rd interval) Sequence

c

basket bluesky bus cheer city crew ﬂow football harbour ice icon pedestrian riverbed rushhour soccer station2 stefan sunﬂower tractor universal warner

164 627 151 564 97 923 183 918 62 512 54 052 239 994 47 987 105 654 44 285 19 213 39 074 64 046 49 008 46 043 27 029 121 209 101 826 94 891 29 158 31 949

I frames max err avg. err 2.64 6.63 3.00 9.21 18.44 6.22 3.28 6.53 7.37 5.46 2.47 1.64 11.11 2.44 5.30 20.10 2.80 9.65 3.88 9.79 5.04

1.26 3.85 1.77 5.39 9.79 4.02 1.61 4.11 3.95 2.97 1.22 1.11 5.45 1.56 2.70 11.27 2.14 5.68 2.29 5.59 2.56

c 48 465 18 255 28 755 85 862 12 569 31 859 24 500 40 081 20 642 20 936 23 482 29 271 76 236 11 068 42 461 438 20 814 9 701 30 638 22 693 33 976

P-frames max err avg. err 5.38 7.31 4.22 7.82 5.80 7.04 4.12 7.30 20.98 7.42 6.31 7.35 5.58 15.74 8.14 65.67 4.17 5.71 17.88 4.26 3.92

3.36 5.22 2.49 3.84 3.76 4.30 3.00 4.42 11.24 3.50 4.16 4.34 3.33 9.12 3.95 43.85 2.01 3.39 11.81 2.76 2.50

182

T. Grajek and M. Doma´ nski

For I- and P-frames average relative error for most sequences is lower than 5%. However, several sequences have bigger relative errors, what can be caused by reduced number of parameters.

4

Conclusions

A simple quantitative model of AVC/H.264 coders has been described. This model can be used to set a value of the quantization parameter Qstep for a given number of bits for an I- or P-frame. Tests showed that it ﬁts experimental data very well in all intervals. For most sequences relative approximation error is lower than 5% for I-frames and below 7% for P-frames. However sequences with speciﬁc motion characteristics like in station2 and city sequences exhibit larger approximation errors. Some experiments show that these errors can be reduced by using additional parameters in the model for 1st and 3rd intervals. The model may be used in global procedures for bitrate control. Previously encoded pictures may be used to identify the model parameters. Usually, the model parameters are nearly constant in time, mostly even in longer temporal intervals. In that way, the model may be used in order to explicitly calculate quantization parameter Qstep for the assumed bitrate. Such a technique may be used in bitrate control techniques that are appropriate for video streaming in the communication channels with rapid variations of the channel throughput, i.e. wireless channels.

Acknowledgements This work was supported by the public funds as a research project in years 2007-2009.

References 1. ISO/IEC FDIS 14496-10: Information Technology - Coding of Audio-Visual Objects - Part 10: Advanced Video Coding 2. ISO/IEC 13818-2: Information technology - Generic Coding of Moving Pictures and Associated Audio Information – Part 2: Video 3. Li, Z., Pan, F., et al.: Adaptive Basic Unit Layer Rate Control for JVT, JVT-G012, 7th meeting, Thailand (2003) 4. Chiang, T., Zhang, Y.-Q.: A New Rate Control Scheme Using Quadratic Rate Distortion Model. IEEE Transactions on Circuits and Systems for Video Technology 7, 246–250 (1997) 5. He, Z., Mitra, S.K.: Optimum Bit Allocation and Accurate Rate Control for Video Coding via ρ-Domain Source Modeling. IEEE Transactions on Circuits and Systems for Video Technology 12(10) (2002) 6. Lin, G., Zheng, S., Hu, J.: A Two-Stage ρ-Domain Rate Control Scheme for H.264 Encoder. In: IEEE International Conference on Multimedia and Expo ICME 2008, Hannover, Germany, June 23-26, 2008, pp. 713–716 (2008)

A Simple Quantitative Model of AVC/H.264 Video Coders

183

7. Lim, S.C., Na, H.R., Lee, Y.L.: Rate Control Based on Linear Regression for H.264/MPEG-4 AVC. Signal Processing: Image Communication 22(1), 39–58 (2007) 8. Zhou, Y., Sun, Y., Feng, Z., Sun, S.: New Rate-Complexity-Quantization Modeling and Eﬃcient Rate Control for H.264/AVC. In: IEEE International Conference on Multimedia and Expo ICME 2008, Hannover, Germany, June 23-26, 2008, pp. 717–720 (2008) 9. Zhuo, L., Gao, X., Wang, Z., Feng, D.D., Shen, L.: A Novel Rate-Quality Model based H.264/AVC Frame Layer Rate Control Method. In: 6th International Conference on Information, Communications & Signal Processing, Singapore (2007) 10. L uczak, A.: Digital Video Bitrate Control Using Coder Models, Doctoral Dissertation, Pozna´ n (2003) 11. Doma´ nski, M., L uczak, A.: Simple Global Model of an MPEG-2 Bitstream. In: Proceedings of IEEE International Conference on Multimedia and Expo ICME 2002, Lausanne, Switzerland, August 26-29 (2002) 12. Bronshtein, I.N., Semendyayev, K.A., Musiol, G., Muehling, H.: Handbook of Mathematics, 4th edn. Springer, New York (2004); Matematyka, poradnik encyklopedyczny, Wydawnictwo Naukowe PWN, wydanie 20, Warszawa (2004) (in polish) 13. Brandt, S.: Data Analysis: Statistical and Computational Methods for Scientists and Engineers, 3rd edn. Springer, New York (1999); Analiza danych, metody statystyczne i obliczeniowe, Wydawnictwo Naukowe PWN, wydanie 2, Warszawa (1999) (in Polish) 14. Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut Image Processing Department, http://iphome.hhi.de/suehring/tml/index.htm

Approximation of Signals by Predict Wavelet Transform Marcin Jaromin Institute of Computer Science, Silesian University of Technology, Poland [email protected]

Abstract. The article presents a general outline of the signal theory using predict wavelet transform. The predict wavelet transform (1) is a new attitude to a multiresolution signal analysis by discrete wavelet transform [8]. It is implemented in accordance with a lifting scheme, which allows realizing a signal ﬁltration by biorthogonal wavelets [10,9]. Thanks to it, there exists a possibility to receive an optimal, biorthogonal ﬁlter for analysis of chosen signal characteristic. The article describes the signal approximation method by lifting scheme (9). This method is a generalization of the predict interpolation proposed by Wim Sweldens [6,10]. It allows to predict odd signal samples using a polynomial degree much lower than an interpolation polynomial degree. This solution not only enables approximating by algebraic polynomial but also by some base functions. Thus, this method is more ﬂexible and optimal. Using an orthogonal Gram’s polynomial (23) or a trigonometric polynomial for approximation eliminates a problem of ill conditioning of matrix coefﬁcients (19). Such conditioning can cause large rounding errors during performing computer operation, so it leads to incorrect results. The article enables to get acquainted with the relation between classical and predict wavelet transform (6). The method of obtaining a biorthogonal highpass ﬁlter and lowpass ﬁlter (analysis and synthesis signal) is shown on the base of a second rank predictor example. The corresponding ﬁlters are received by summation of ﬂow-ways between Predict and Update coeﬃcients. The calculated ﬁlter coeﬃcients are described by wavelet and scaling functions in a graphical form (7, 8). The summary chapter presents the use of lifting scheme during multiresolution image analysis (9) and irregular meshes analysis 3D (10).

1

Introduction

Multiresolution analysis and digital processing by a wavelet method are nowadays one of the most dynamically developing parts of digital signal processing. Discrete wavelet transform (DWT) similarly to analysis by discrete Fourier transform (DFT), can be used to analysis of the signal, to noise reduction and image compression. Fourier analysis, being the most prevalent method of multiresolution analysis, has however a fundamental drawback: we lack the information when the event in frequency domain exactly happened. Wavelet transform has L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 184–197, 2009. c Springer-Verlag Berlin Heidelberg 2009

Approximation of Signals by Predict Wavelet Transform

185

not this inconvenience and therefore it allows to move the signal from a timevalue conﬁguration to a time-scale (time-frequency) one. It performs a split of the original signal into some elementary signals called wavelets. So, the approximated signal is a linear combination of wavelet functions. A choice of wavelet depends on signal characteristics. The better is a wavelet estimation of the function examined, the better approximation of signal is achieved on a deﬁned level. Signal decomposition by wavelet transform separates the signal into a basic process (rough representation) and processes of detailed representation, storing signal details on a required level. Signal composition, from the other hand, allows to merge a rough signal with a signal of its details, obtaining thus a higher level of signal characteristics resolution.

2

Approximation of Signals

Approximation of signal is a process which allows to obtain a rough solutions from the known signal. Thanks to it we can get a solution quite near to the exact one – with some assumed regularity. It is mostly used, when we cannot ﬁnd an analytical function deﬁning precisely the signal characteristics. The approximation function can smooth the signal, so it can become diﬀerentiable. The great advantage of an approximation over interpolation is that an approximation function doesn’t have to be a high degree polynomial with a large number of approximated points. Actually it does not have to be a polynomial at all. We read an approximation of the signal as the minimization of some error function f – so called target function [4]. N M (e) fj = w(ni )[ sj (ni ) − aM−k ΨM−k (ni ) ]2 → min. i=1

(1)

k=1

where fj is target function at j − th multiresolution level, w(·)- weight function (e) (w(ni ) ≥ 0), sj (·) - odd samples of approximation signal at j − th level and k aM−k ΨM−k (·) - linear combination of base functions Ψ (·). We calculate the minimization error as the solution of a set of equations, that they are derivatives of the expression under square root (1) compared to zero. N M δf (e) = −2 w(ni )[ sj (ni ) − aM−k ΨM−k (ni ) ]ΨM−1 (ni ) δaM−1 i=1

=0

k=1

N M δf (e) = −2 w(ni )[ sj (ni ) − aM−k ΨM−k (ni )(ni ) ]ΨM−2 (ni ) = 0 (2) δaM−2 i=1 k=1 .. .. .. . . . N M δf (e) = −2 w(ni )[ sj (ni ) − aM−k ΨM−k (ni )(ni ) ]Ψ0 (ni ) =0 δa0 i=1 k=1

186

M. Jaromin

or in the consecutive form: M

aM−k

aM−k

N

aM−k

N

w(ni )ΨM−k (ni )ΨM−2 (ni ) =

(e)

w(ni )sj (ni )ΨM−1 (ni )

N

(e)

w(ni )sj (ni )ΨM−2 (ni )

i=1

w(ni )ΨM−k (ni )Ψ0 (ni )

=

i=1

k=1

N i=1

i=1

k=1

.. . M

w(ni )ΨM−k (ni )ΨM−1 (ni ) =

i=1

k=1 M

N

.. . N

.

(3)

(e)

w(ni )sj (ni )Ψ0 (ni )

i=1

We can present the expression above in a matrix form: ⎡

N

⎤ (e) w(ni )sj (ni )ΨM−1 (ni ) ⎥

⎢ ⎢ i=1 ⎥ ⎢ ⎥ ⎢ N ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ (e) w(ni )sj (ni )ΨM−2 (ni ) ⎥ ⎢ aM−2 ⎥ ⎢ BM×M · ⎢ ⎥=⎢ ⎥ ⎢ .. ⎥ ⎢ i=1 ⎥ ⎢ ⎥ ⎣ . ⎦ .. ⎢ ⎥ . ⎢ ⎥ a0 ⎢ ⎥ N ⎣ ⎦ (e) w(ni )sj (ni )Ψ0 (ni ) ⎡

aM−1

⎤

(4)

i=1

where ⎡N ⎤ N w(ni )ΨM−1 (ni )ΨM−1 (ni ) · · · w(ni )Ψ0 (ni )ΨM−1 (ni )⎥ ⎢ ⎢ i=1 ⎥ i=1 ⎢ ⎥ ⎢N ⎥ N ⎢ ⎥ ⎢ ⎥ w(ni )ΨM−1 (ni )ΨM−2 (ni ) · · · w(ni )Ψ0 (ni )ΨM−2 (ni )⎥ ⎢ BM×M = ⎢ ⎥. (5) i=1 ⎢ i=1 ⎥ ⎢ ⎥ . . . .. .. .. ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ N N ⎣ ⎦ w(ni )ΨM−1 (ni )Ψ0 (ni ) · · · w(ni )Ψ0 (ni )Ψ0 (ni ) i=1

i=1

Transforming right-sided sums of the expressions from 4, we get the expression: ⎤ ⎤ ⎡ ⎤ ⎡ (e) sj (n1 ) aM−1 w(n1 ) 0 ··· 0 ⎢ (e) ⎥ ⎢ aM−2 ⎥ ⎢ 0 w(n2 ) 0 ⎥ sj (n2 ) ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎢ ⎥ BM×M · ⎢ . ⎥ = ΨM×N · ⎢ . .. .. ⎥ ⎢ .. .. ⎥ ⎣ .. ⎦ ⎣ .. . . . ⎦⎣ ⎦ . (e) a0 0 0 · · · w(nN )

sj (nN ) ⎡

WN×N

(6)

Approximation of Signals by Predict Wavelet Transform

⎡

where

ΨM×N

ΨM−1 (n1 ) ΨM−1 (n2 ) · · · ΨM−1 (nN )

187

⎤

⎢ ⎥ ⎢ ⎥ ⎢ ΨM−2 (n1 ) ΨM−2 (n2 ) ⎥ Ψ (n ) M−2 N ⎢ ⎥ ⎢ ⎥ =⎢ ⎥. ⎢ ⎥ .. .. .. .. ⎢ ⎥ . . . . ⎢ ⎥ ⎣ ⎦ Ψ0 (n1 ) Ψ0 (n2 ) · · · Ψ0 (nN )

(7)

So the polynomial coeﬃcients can be presented in a matrix form as follows: ⎡ ⎤ (e) ⎡ ⎤ sj (n1 ) aM−1 ⎢ ⎥ ⎢ (e) ⎥ ⎢ aM−2 ⎥ s (n ) ⎢ ⎥ 2 j ⎢ ⎥ −1 ⎢ ⎥. (8) ⎢ .. ⎥ = BM×M · ΨM×N · WN×N · ⎢ ⎥ ⎣ . ⎦ .. ⎢ ⎥ . ⎣ ⎦ a0 (e) sj (nN ) As we calculate the signal value on a base of formula sj (n) = aM−1 ΨM−1 (n) + aM−2 ΨM−2 (n) + . . . + a0 Ψ0 (n)

(9)

or in a matrix form ⎡

⎤ aM−1 ⎥ ⎢ ⎢ aM−2 ⎥ sj (n) = ΨM−1 (n) ΨM−2 (n) · · · Ψ0 (n) · ⎢ . ⎥ ⎣ .. ⎦

(10)

a0 (e)

So, value of approximated signal sj at any n point between samples n1 , n2 , . . .,nN can be presented can be presented in a predict form of signal as follows: ⎡ ⎤ (e) sj (n1 ) ⎢ ⎥ ⎢ (e) ⎥ s (n ) ⎢ ⎥ 2 j (e) ⎥ sj (n) = p · Sj = p · ⎢ (11) ⎢ ⎥ .. ⎢ ⎥ . ⎣ ⎦ (e) sj (nN ) where p = ΨM−1 (n) ΨM−2 (n) · · · Ψ0 (n) · B−1 M×M · ΨM×N · WN×N .

(12)

If the number of base functions M is equal to the number of even samples N , w(ni ) = 1 and ΨM−1 = nM−1 , we have polynomial interpolation which can be shown as p = nN −1 nN −2 · · · 1 ·B−1 (13) N×N · ΨN×N

188

M. Jaromin

where

⎡ B−1 N×N

3

· ΨN×N

−1 −2 nN nN ··· 1 1 1

⎤−1

⎢ N −1 N −2 ⎥ ⎢n n2 ··· 1⎥ ⎢ 2 ⎥ =⎢ ⎥ ⎢ .. .. . . .. ⎥ ⎣ . . .⎦ . −1 N −2 nN n · · · 1 N N

.

(14)

Update Step of Lifting Scheme

Approximation of signal using lifting scheme boils down to three steps: – Split step - splitting of the samples into even and odd ones, – Predict step - deﬁning the detailed representation characteristics by choosing a predictor, – Update step - averaging the signal of rough representation against original signal.

Fig. 1. Signal decomposition by lifting scheme

During assignment of update coeﬃcients we use the signal property 15. The average value of original signal equals to the average value of rough approximation after transformation by lifting scheme [6,10] si = si−1 .

(15)

It means that the signal average at a higher level equals to the signal average at a lower level. The coeﬃcients of approximation (rough) space and detail space at j − 1-th multiresolution level are calculated as follows (o)

(e)

di−1 (n − 1) = si (n − 1) − P {si (n − k0 + k)} (e) si−1 (n) = si (n) + U {di−1 (n − l0 + l)}

(16)

where i is a multiresolution analysis level, k and l are indices of subsequent samples used during Predict step and Update step. Constant value k0 deﬁnes index of shift coeﬃcients between even value and predict odd value during Predict operation. Similarly, the constant value l0 deﬁnes index of shift coeﬃcients between predict odd value and even value during Update operation.

Approximation of Signals by Predict Wavelet Transform

189

If Predict and Update operations have linear characteristic and the signal is periodic, we have relations ⎧ → − dim P −1 ⎪ ⎪ ⎪ ⎪ pk = 1 ⎪ ⎪ ⎨ k=0 . (17) → − ⎪ dim U −1 ⎪ ⎪ ⎪ ⎪ ul = 12 ⎪ ⎩ l=0

When predict vector length equals to update vector length, we can adopt that u = p2 .

4

Ill Conditioned of Matrix Coeﬃcients

The diﬀerence between polynomial approximation and interpolation is that, approximation function does not have to have the same values of approximated signal for analyzed nodes. The degree of approximation polynomial can be much lower than the degree of interpolation polynomial, but high enough to obtain optimal approximation of the signal. Analyzing values on discrete collections, we often use empirical methods, so the polynomial degree should be low enough to smooth random measurement errors. In practice, we use approximation as analysis of target function values fj at j − th multiresolution level. We execute iteration until the function stop diminish. This method has a great weakness, which leads to ill conditioning of matrix coeﬃcients BM×M (5). To illustrate this problem, we assume that the coeﬃcients of matrix structure BM×M have arguments within range < 0, 1 > distributed evenly, w(ni ) = 1 and Ψk (ni ) = nki . We can calculate coeﬃcients for high values N by integrate calculus [4], i.e.: N

nki ≈ N

i=1

where k ∈< 0, 2M − 2 >. Thus

BM×M

⎡

1

nk dn = N 0

nk+1 1 N |0 = k+1 k+1

1 1 ⎢ 2M − 1 2M − 2 ⎢ ⎢ ⎢ 1 1 ⎢ = N ⎢ 2M − 2 2M − 3 ⎢ .. .. ⎢ ⎢ . . ⎣ 1 1 M M −1

···

1 M

(18)

⎤

⎥ ⎥ ⎥ 1 ⎥ ⎥ ··· M −1⎥ ⎥ .. .. ⎥ . ⎥ . ⎦ ··· 1

(19)

Calculating the inverse matrix B−1 M×M we observe that for M ≥ 12, the matrix elements have values greater than 4 · 1022 . Using a computer to perform

190

M. Jaromin

such calculations is completely pointless because of rounding errors. Reassuming, approximation with baseline functions ni has meaning only for small values of M . In order to optimize arithmetic of approximation by higher polynomial degrees, we can use one of normal methods e.g. Householder transform, Golumb or other methods of matrix decomposition into a product QR. An alternative is using orthogonal polynomials base which allows to obtain solution without using inverse matrix transform.

5

Orthogonal Gram’s Polynomials

Having a structure n + 1 equidistant points x = x0 , x1 , . . . , xn where xi = −x0 x0 + iΔx and i = 0, 1, . . . , n, then we can use a linear transform i = xiΔx to analysis signal points i = 0, 1, . . . , n . If we create a conﬁguration of orthogonal polynomials P0n , P1n , P2n , . . . , Pmn

(20)

where polynomials degrees are within range < 0, m >and m ≤ n, n

Pjn (i)Pkn (i) = 0 j =k

(21)

Pkn (0) = 1 k = 0, 1, . . . , m

(22)

i=0

and when we assume we can derive the formula [s] k k+s i Pkn (i) = (−1) · · · [s] , k = 0, 1, 2, . . . , m s s n s=0

(23)

i[s] = i(i − 1)(i − 2) . . . (i − s + 1), n[s] = n(n − 1)(n − 2) . . . (n − s + 1).

(24) (25)

k

s

where [4]

Matrix B−1 M×M has following values ⎡ 1 0 ... ⎢ n P0n (i)P0n (i) ⎢ i=0 ⎢ ⎢ 1 ⎢ n 0 ... −1 ⎢ P BM×M = ⎢ i=0 1n (i)P1n (i) ⎢ .. .. .. ⎢ . . . ⎢ ⎣ n 0 0

⎤ 0 0

⎥ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ ⎥ ⎦

.. . 1 i=0 Pmn (i)Pmn (i) (26)

Approximation of Signals by Predict Wavelet Transform

191

Fig. 2. Gram’s polynomials - the plot of even samples () and odd samples () obtained by lifting scheme

(e)

Fig. 3. Gram’s polynomial - an example of matrix product B−1 M×M · ΨM×N · Sj (WN×N = IN×N ) to calculate polynomial coeﬃcients

6

Trigonometric Approximation

While examining real time phenomena, we can come across the situation that function being analyzed is periodic. Therefore, instead of using approximation with algebraic polynomials, we should use trigonometric polynomials, which method is optimal and more convenient. For further consideration we can accept that measuring points are equidistant within range < t0 , t2L−1 >, it is xi =

πk ti − t0 , where k = , i = 0, 1, 2, . . . , 2L − 1 L h

(27)

and the orthogonal base is 1, sin(xi ), cos(xi ), sin(2xi ), cos(2xi ), . . . , sin(jxi ), cos(jxi ).

(28)

Since the structure (28) is orthogonal not only within continuous range < 0, 2π >, but as well on a discrete range.

192

M. Jaromin

Thus, we obtain expressions [4]: 2L−1 i=0

⎧ =l ⎨0 k 0 , cos(kxi )cos(lxi ) = L k = l = ⎩ 2L k = l = 0

2L−1 i=0

⎧ =l ⎨0 k =0 , sin(kxi )sin(lxi ) = L k = l ⎩ 0 k=l=0 2L−1

cos(kxi )sin(lxi ) = 0

(29)

(30)

(31)

i=0

where k, l ∈ Z∩ < 0, L >. Similarly to orthogonal Gram’s polynomials, we can show that B−1 M×M is a diagonal matrix having following values: ⎡ ⎢ ⎢ B−1 = ⎢ M×M ⎣

1 2L

0 .. . 0

0 ... 0

⎤

0⎥ ⎥ . .. . . .. ⎥ . .⎦ . 1 0 L

1 L

(32)

Fig. 4. Trigonometric approximation- the plot of even samples () and odd samples () obtained by lifting scheme

Usage of a predictor with a diagonal matrix BM×M causes an automatic decrease of the calculation costs. Thanks to it, the calculation precision increases. A signal characteristics doesn’t always allow to apply the above method directly. The signal energy can be distributed inexactly when frequencies are integer multiples of the basic frequency of the examined signal fNs [3]. As an example we can take a periodical signal moved in time by a constant τ . A direct analysis of this signal causes a great approximation error. To minimize it, we can shift the signal by a time constant τ when it is possible.

Approximation of Signals by Predict Wavelet Transform

7

193

Relation between Classical and Predict Wavelet Transformation

In classical wavelet theory to deﬁne the approximation coeﬃcients during signal ˜ and decimation ↓ 2 of the signal decomposition we use a lowpass ﬁlter bank h S. Similary, we receive detail coeﬃcients after a highpass ﬁltration by g˜ and decimation ↓ 2 of the same signal, according to the scheme 5.

Fig. 5. Two-channels scheme of signal decomposition and signal composition

A relation between a classical wavelet theory and its predict equivalent can be shown by example of a interpolation predict transform with second rank predictor p = [p0 , p1 ] and update vector u = [u0 , u1 ].

Fig. 6. A relation between classical and predict wavelet transformation

The scheme 6 allows to receive a biorthogonal wavelet system in a simple way. Filter weights can be calculated by summation of ﬂow-ways for each signal sample [9]. Thus, highpass ﬁlter coeﬃcients are received evidently by the ﬂow-ways method from sj to dj−1 . ⎧ ⎪ ⎨ g˜0 = −p0 g˜1 = 1 (33) ⎪ ⎩ g˜ = −p 2 1

194

M. Jaromin

While, lowpass ﬁlter coeﬃcients taking into consideration to predict and update operations, are calculated by summation of ﬂow-ways from sj to sj−1 ⎧ ˜0 ⎪ h ⎪ ⎪ ⎪ ⎪ ˜ ⎪ ⎨ h1 ˜2 h ⎪ ⎪ ⎪ ˜3 h ⎪ ⎪ ⎪ ⎩˜ h4

= −p0 · u0 = 1 · u0 = −p1 · u0 + 1 − p0 · u1 . = 1 · u1 = −p1 · u1

(34)

An example can be an approximation by polynomial f (x) = a0 + a1 cos(x). Since we use two samples for approximation, the coordinates of the vectors are p = [0.509054; 0.490946] and u = [0.254527; 0.245473]. ˜ of decomposition are The coeﬃcients of the lowpass ﬁlter h ⎧ ˜ 0 = −0.509054 · 0.254527 = −0.129568 ⎪ h ⎪ ⎪ ⎪ ⎪ ˜ ⎪ ⎨ h1 = 1 · 0.254527 = 0.254527 ˜ 2 = (−0.490946) · 0.254527 + 1 + (−0.509054) · 0.245473 = 0.750082 (35) h ⎪ ⎪ ⎪h ˜ 3 = 1 · 0.245473 = 0.245473 ⎪ ⎪ ⎪ ⎩˜ h4 = (−0.490946) · 0.245473 = −0.120514 and the highpass ﬁlter g˜ has values ⎧ ⎪ ⎨ g˜0 = −0.509054 g˜1 = 1 . ⎪ ⎩ g˜ = −0.490946 2

(36)

A lifting scheme determines the composition ﬁlters h and g at the base of predict and update coeﬃcients. To assign the ﬁlter values we need to change the signs of predict and update coeﬃcients and to use the same scheme to calculate the coeﬃcients. At the next step we use a ﬂow-ways procedure compatible with scheme 6. The ﬁnal stage is the replacement of ﬁlter designations from ˜h to g and from g˜ to h with sequence conversion of their vector values. Thus ⎧ ˜ 4 = −0.120514 ⎪ g0 = −p1 · u1 = h ⎪ ⎪ ⎪ ⎪ ˜ ⎪ ⎨ g1 = 1 · u1 = −h3 = −0.245473 (37) g2 = −p1 · u0 + 1 − p0 · u1 = ˜h2 = 0.750082 ⎪ ⎪ ⎪ ˜ ⎪ g3 = 1 · u0 = −h1 = −0.254527 ⎪ ⎪ ⎩ ˜ 0 = −0.129568 g4 = −p0 · u0 = h and

⎧ ⎪ g2 = 0.490946 ⎨ h0 = p1 = −˜ h1 = g˜1 = 1 . ⎪ ⎩ h = p = −˜ g0 = 0.509054 2 0

(38)

Approximation of Signals by Predict Wavelet Transform

195

The process of calculating ﬁlter coeﬃcients by predict and update coeﬃcients is not always symmetric unlike this in the scheme 6. The scheme unambiguously allows to determine the biorthogonal ﬁlters for the well-known predict and update vectors.

Fig. 7. The signal decomposition - wavelet and scaling function (p = [0.509; 0.491], u = [0.255; 0.245])

Fig. 8. The signal composition - wavelet and scaling function (p = [0.509; 0.491], u = [0.255; 0.245])

An inverse situation does not give such guarantee. When we want to assign Predict and Update vectors, we must transform a two-channel decomposition scheme to a polyphase form. Next, we create a product matrix form from a polyphase matrix [1] m 1 si (z) 1 0 K 0 P(z) = . 1 0 1 ti (z) 1 0 K i=1

(39)

We can receive this product by using Euclidean algorithm for Laurent’s polynomials [8]. Since the division Laurent’s polynomial operation is ambiguous, factorization of wavelet ﬁlters may have more than one solution. Moreover, it can require more than one pair of Predict and Update operations.

196

8

M. Jaromin

Summary

The lifting scheme allows to project a inverse integer transforms.Their properties are approximated to these ﬂoat-point operations and the reconstruction error practically has zero value. We can use the transforms in the loss-less data compression algorithm (JPEG2000). A wavelet analysis of image by two-dimensional signal, uses the mostly a one-dimensional wavelet transform. At the ﬁrst step we decompose the image rows, at the second step we decompose the columns from the image received during the decomposition of rows. According to a multiresolution wavelet analysis, the original image is represented by four parts: a rough image and three detail images: horizontal, vertical and diagonal [2].

Fig. 9. Multiresolution analysis of image

A wavelet transform is used in Federal Bureau of Investigation in United States of America to compress of ﬁnger stamps. This compression uses the method of advance progressive coeﬃcient quantization which uses dependencies between wavelet coeﬃcients. We can use the multiresolution analysis e.g. in Internet, where the whole image is approximated and successively complemented by some additional details, until user cancels downloading the ﬁle. Nowadays, the predict wavelet transform is also used for diﬀerential compression of sequential video image frames [2]. A predict wavelet transform is better than DWT, because it is faster and memory sparing.

Approximation of Signals by Predict Wavelet Transform

197

All calculations are executed by substitution, without introduction of additional variables. A lifting scheme introduces a new wavelet transform method, it is the multiresolution operation for irregular grids, which are created by three-dimensional objects [5,7]. Biorthogonality does not require preliminary grid transformation into a regular or half-regular form.

Fig. 10. Comparison of irregular mesh 3D

A wavelet analysis is a young domain, it is about 10 years old. The analysis allows simultaneous presentation of time and frequency properties of a nonstationary signal. It results in signal approximation by isolating its characteristic structural elements. Wavelet analysis is also used in geophysics, physics, hydrodynamics, astronomy, biology and economy. We use it in analysis of speech and steering of industrial processes. Its popularity permanently grows, particularly within analysis of dynamic images. Therefore, we can expect its fast development soon e.g. in the direction of optimal coders which will adjust quality of displayed image to the speed of computer or to network data transfer.

References 1. Uytterhoeven, G., Van Wulpen, F., Jansen, M., Roose, D., Bultheel, A.: Waili: Wavelets with integer lifting. Technical report, Belgium (1997) 2. Vetterli, M., Kovaˇcevi´c, J.: Wavelets and Subband Coding. Signal Processing Series. Prentice Hall, Englewood Cliﬀs (1995) 3. Lyons, R.G.: Understanding digital signal processing. Addison Wesley Longman, Inc., Amsterdam (1997) 4. Ralson, A.: A ﬁrst course in numerical analysis. McGraw-Hill, Inc., New York (1965) 5. Guskov, I., Sweldens, W., Schr¨ oder, P.: Multiresolution signal processing for meshes. In: Computer Graphics Proceedings (SIGGRAPH 1999), pp. 325–334 (1999) 6. Sweldens, W., Schr¨ oder, P.: Building your own wavelets at home (1996) 7. Daubechies, I., Guskov, I., Schr¨ oder, P., Sweldens, W.: Wavelets on irregular point sets. Phil. Trans. R. Soc. Lond. A 357, 2397–2413 (1999) 8. Daubechies, I., Sweldens, W.: Factoring wavelet transforms into lifting steps. J. Fourier Anal. Appl. 4, 247–269 (1998) 9. Sweldens, W.: The lifting scheme: A new philosophy in biorthogonal wavelet constructions. In: Wavelet Applications in Signal and Image Processing III, pp. 68–79 (1995) 10. Sweldens, W.: The lifting scheme: A construction of second generation wavelets. SIAM J. Math. Anal. 29, 511–546 (1997)

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames Jaroslaw Marek Poznan University of Technology Chair of Multimedia Telecommunications and Microelectronics ul. Polanka 3, 60-965 Pozna´ n, Poland [email protected]

Abstract. The main goal of transcoding is to change bit rate of video sequence. This can be done by cascaded connection of decoder and encoder, known as Cascaded Pixel Domain Transcoder (CPDT). Decoding and re-encoding video bit stream always gives lower image quality than encoding original sequence. This paper presents a new technique of video transcoding that is able to deliver image quality superior to CPDT and has lower computational complexity. The technique is restricted to homogeneous (within the same bit stream format) transcoding of bit streams encoded according to H.264/AVC(MPEG 4) standard speciﬁcation. The standard deﬁnes diﬀerent types of encoded frames but proposed technique is designed for I(ntra) type frames only. Keywords: AVC, MPEG 4, H.264, video transcoding, video encoding, requantization, bit rate reduction.

1

Introduction

Encoded video sequences can be transmitted with various bit stream formats and bit rates. For transmission over heterogeneous networks, changing of sequence’s bit rate is sometimes required to accommodate to channel’s throughput. This can be simply done when video bit stream is scalable. When it is not, input bit stream should be transcoded to lower bit rate. Transcoding operation can be performed in various ways [1-3]. One of them is to connect decoder which decodes input bit stream with encoder which forms a new bit stream. Such a scheme is depicted in Fig. 1 and it is usually referred as Cascaded Pixel Domain Transcoder (CPDT) [2, 3]. The greatest advantage of this architecture is that it is simple to implement. There are good decoders and encoders available which can be used. Unfortunately, computational complexity of this architecture is very high. Moreover, it does not ensure the best image quality that can be achieved. Another approach is to integrate decoding and encoding processes into a single transcoding process. This can reduce computational complexity of the solution, as not all encoding/decoding operations are required for transcoding. Moreover, knowledge of an input bit stream in the encoding part, can be used L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 198–207, 2009. c Springer-Verlag Berlin Heidelberg 2009

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames

199

Fig. 1. Cascaded Pixel Domain Transcoder (CPDT)

to achieve higher image quality. Solution of this type is presented in this paper. Proposed transcoding technique is able to transform input video bit stream into another video bit stream without image reconstruction and encoding processes. This is known as transcoding in frequency domain. The technique allows for homogeneous transcoding [1], that is transcoding of bit streams within the same format. Generally, all types of frames, encoded in a given format, can be transcoded. However, this technique is designed for transcoding of (I)ntra type frames, encoded according to H.264/AVC(MPEG 4) standard speciﬁcation [4]. In the following sections one can ﬁnd grounds for a design of a new transcoder architecture, presentation of used techniques and achieved results.

2

Quality Loss Caused by CPDT

For a given bit-rate, quality of transcoded (and decoded) sequence is always lower than quality of sequence decoded from a bit stream achieved by encoding original sequence. The reason for this is that in CPDT there is no information about original image and encoding base on distorted image (result of ﬁrst time encoding). One can ﬁnd in a literature references to CPDT coding eﬃciency [5, 6] but there are no comprehensive results of research. This is why quality loss tests were performed for this kind of transcoder. Results of these tests allow to identify weak points of this solution as well as allow to compare them with a newly proposed techniques. In Fig. 2 one can ﬁnd an interpretation of quality loss value used in this paper. The higher curve refers to ﬁrst-time encoding. It is the highest image quality that can be obtained for given bit rates and for a given bit stream format. The lower curve refers to transcoding, which introduce some quality degradation. Point B shows quality and bit rate of re-encoded sequence. Point A indicates quality of ﬁrst time coded sequence with the same bit rate as for point B. Loss in quality can be calculated as a distance between these two points. All tests were conducted with H.264/AVC(MPEG 4) reference software (JM version 13.2). There were 11 high quality sequences used, namely: bluesky, city, crew, harbour, ice, pedestrian, riverbed, rushour, soccer, station, sunﬂower, station. Each sequence had 100 frames and 704 × 576 resolution. Some of these sequences originally have 1280 × 720 resolution. Smaller versions were achieved by cropping original frames to desired resolution (only images’ centres were used).

200

J. Marek

Fig. 2. Quality loss caused by transcoding

At ﬁrst, original sequences were encoded with a set of ten diﬀerent QPF values. QP is a parameter deﬁned by H.264/AVC(MPEG 4) standard determining the quantizer that should be used. The higher value QP parameter has, the wider quantization step size is. QPF refers to a value of QP used for ﬁrsttime coding. After encoding process all achieved bit streams were decoded. Each decoded sequence has been re-encoded with consecutive QPT values, starting from QPT = QPF . QPT refers to the value of QP used for transcoding. In a ﬁnal step quality losses were calculated as presented in Fig. 2. This gives a set of curves for every sequence. In Fig. 3 there is an example of a single curve that

Fig. 3. Example of quality loss (ΔPSNR ) caused by transcoding. Points on a curve indicate reencoding with diﬀerent QPT values from a single sequence encoded with QPF value. The absolute diﬀerence in QPT between neighbouring points laying on the same curve is 1. First point (from right) on the curve indicates transcoding with QPT equal to QPF .

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames

201

Fig. 4. Quality loss caused by transcoding of ’soccer’ sequence. Points on each curve indicate re-encoding with diﬀerent QPT values from a single sequence encoded with QPF value. The absolute diﬀerence in QPT between neighbouring points laying on the same curve is 1. First point (from right) on each curve indicate transcoding with QPT equal to QPF .

can be obtained for a sequence ﬁrstly encoded with QPF parameter and then transcoded with eleven QPT values, as described above. In this paper, results for only one sequence are presented. They can be found in Fig. 4. It has to be stated that these results are representative for all other sequences. For all the data gathered, when QPT equals QPF , quality loss can be as high as 0.7dB (sequence: sunﬂower, QPT = QPF = 28). This is a huge image quality degradation, especially while the encoding parameters are not being changed from ﬁrst encoding to transcoding. There are two reasons for this situation. First of all, coding of DC coeﬃcients in Intra16x16 prediction mode (deﬁned by H.264/AVC(MPEG 4) standard) is not always reversible. There are combinations of DC coeﬃcients for which decoding and re-encoding them with the same QP will result in changed coeﬃcients. Second reason is that encoder uses Lagrangian optimization (which balances between quality degradation and bits generated) for block and macroblock mode decisions. There are situations when encoder has to choose whether to use block/macroblock mode with low quality degradation and few bits needed or mode with no quality degradation and more bits needed for encoding. Choosing ﬁrst option in case of transcoding with the same QP results in additional image degradation.

202

J. Marek

The highest loss in decoded sequence quality when transcoded is when QPT − QPF diﬀerence is from 1 to 3. This loss can be as high as 2dB and generally decreases with the increase of QPF . It can be seen that when diﬀerence between QPT and QPF values raises, quality loss quickly decreases. The above suggests that there is a need for technique that could deliver lower quality loss for small bit rate reduction.

3

Proposed Requantization Technique

H.264/AVC(MPEG 4) is diﬀerent from previous video coding standards because it engages prediction of image samples for intra coded macroblocks instead of DC transform coeﬃcients prediction. Modiﬁcation of image samples in a single macroblock or a block within a macroblock can change prediction of neighbouring macroblocks. This prediction is used for neighboring macroblocks reconstruction. If its change is not compensated by transform coeﬃcients modiﬁcation, there will be a decoding error. If there will be the decoding error, prediction for the next macroblocks will be also wrong and they will be wrongly decoded. And so on. This leads to the conclusion that when one block or macroblock is being changed, all next blocks/macroblocks have to be also changed to prevent error propagation. Requantization of a macroblock results in a modiﬁcation of decoded image. Basing on information from previous paragraph, it may be stated that requantization of single macroblock forces requantization of all following macroblocks. It is because prediction for neighbouring macroblocks changes and these changes have to be compensated. Similar situation happens when during transcoding macroblock mode is being chosen diﬀerent than macroblock mode used for encoding original sequence. As a result, decoded image samples are being changed and all the above applies here. Prediction for following macroblocks is being changed and ﬁnally, requantization for all of them is required. Macroblock mode selection has a great impact on coding eﬃciency. Moreover, macroblock mode decision depends on target bit rate [7]. For these reasons, when re-encoding video, new macroblocks’ modes should be chosen. However, it can be assumed that for transcoding with small diﬀerences in bit rate, quality loss caused by wrongly chosen block/macroblock modes is also small. The main idea for minimizing quality loss caused by transcoding is to avoid requantization error. This can be done when: 1. Prediction image for following blocks after requantization remains unchanged, 2. Transcoded sequence uses the same macroblock modes as sequence coded for the ﬁrst time. When transcoder does not change macroblock modes, all of them are known for entire image prior to re-encoding. This gives an information which of coded pixels will be used to form prediction for neighbouring blocks. Furthermore, knowledge about neighbouring blocks modes allows to determine whether given

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames

203

pixel will be used directly to form prediction image or its value will be used to ﬁnd average of a few pixels (this average value will be used for prediction image preparation). When pixels will be used directly, their values cannot change after requantization. When latter applies, pixel values can be modiﬁed after requantization, but sum of pixels used for prediction have to stay unchanged. The same applies for luma as well as for chroma components. Finally, there are blocks not used for prediction. They can be modiﬁed freely as their modiﬁcation has no impact on the rest of the image. The most obvious examples are blocks of Intra16x16 mode coded macroblocks, excluding the lowest row and the most right column. These are never used for prediction as it is a rule deﬁned by H.264/AVC(MPEG 4) standard. Two schemes for coeﬃcient modiﬁcation are proposed for requantization. In both of them, consecutively, in a reverse zig-zag order (deﬁned by H.264/AVC (MPEG 4) standard), non zero DCT coeﬃcients are checked, if they can be modiﬁed according to rules presented above. There are two methods of modiﬁcation: 1. The coeﬃcient’s value is set to 0, if its amplitude is not greater than a given threshold; 2. Amplitude of the coeﬃcient is reduced by no more than a threshold value. These methods are supplementary, can be used together or separately, and allow for a bit rate control.

4

Experiments

Experiments were conducted with the software and sequences described in a Section 2. All tests were performed according to following procedure: 1. Take original sequence and encode it with a set of ten diﬀerent QPF values. 2. Decode all bit streams resulting from step 1. 3. Re-encode each sequence achieved in step 2 with ten consecutive QPT values starting from QPF . 4. Take each bit stream achieved in step 1 and transcode it using proposed technique with two proposed schemes and thresholds ranging from 1 to 4. This gives 8 transcoded bit streams for each input bit stream. Values of bit rate and PSNR obtained during realization of steps 3 and 4 were used to form rate-quality curves for CPDT and proposed technique respectively. Next, diﬀerences in PSNR (distance from rate-quality curves as illustrated in Fig. 2) were calculated for bit rates equal to these achieved in step 4. As there are no samples for CPDT with exactly the same bit rates, values of PSNR measure were achieved by using spline interpolation. Exemplary results for ’soccer’ and ’sunﬂower’ sequences can be found in Fig. 5 and Fig. 6. Proposed technique is able to deliver better results than CPDT for all bit rates. However, the results depend on video sequence and bit rate. Generally, for high values of QPF , proposed technique is better than CPDT for both schemes and all thresholds, and with the increase of QPF some scheme and threshold combinations can give worse results.

204

J. Marek

Fig. 5. Gain in PSNR for luma component achieved by using proposed technique instead of CPDT. Each curve represent transcoding of a single sequence encoded with a QPF given in a legend.

Fig. 6. Gain in PSNR for luma component achieved by using proposed technique instead of CPDT. Each curve represent transcoding of a single sequence encoded with a QPF given in a legend.

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames

205

Fig. 7. Maximum gain in PSNR of luma component achieved by using proposed technique instead of CPDT

Fig. 8. Minimum gain in PSNR of luma component achieved by using proposed technique instead of CPDT

Fig. 7 depicts the highest gain in PSNR for a proposed technique comparing to CPDT, for all analyzed sequences. Supplementary to the above is Fig. 9 which depicts bit rate reduction in relation to bit rate of ﬁrstly coded sequence (not transcoded). Additionally, Fig. 8 depicts the lowest gain (which can be at

206

J. Marek

Fig. 9. Bit rate reduction for proposed transcoding technique, achieved for the highest gain in PSNR luma

Fig. 10. Maximum bit rate reduction for proposed transcoding technique achieved in tests

some points considered as the highest loss) in PSNR for a proposed technique comparing to CPDT. Experiments showed that the above is connected with the highest bit rate reductions achieved. Values of these reductions, comparing to ﬁrstly coded sequences, can be found in Fig. 10. Proposed technique is designed for small bit

Homogeneous Video Transcoding of H.264/AVC Intra Coded Frames

207

rate reduction and as can be seen in Fig. 10 it is able to make bit stream over 18% smaller. However, this reduction is not guaranteed and can be equal to 4% for some sequences.

5

Conclusions

In this paper, new technique of H.264/AVC(MPEG 4) bit streams transcoding is proposed. It operates on intra coded frames exclusively and it is performed in the transform domain. It requires neither image decoding prior to new bit stream formatting nor very computationally intensive task of macroblock mode decision. As a result proposed technique requires much less operations than CPDT. The technique is suitable for small bit rate reduction. For 11 test sequences, the highest bit rate reduction was 18.1%. In this case image quality was not as good as the one achieved by CPDT. Putting a criterion of supporting better quality than CPDT, 16.43% bit rate reduction can be achieved. By using proposed technique, one can achieve as much as 1.4dB better results in PSNR, than employing CPDT scheme. However, it should be mentioned, that for high bit rates (low QPF values) there are situations when using proposed technique can result in worse image quality. Acknowledgments. This work was supported by the public funds as a research project in years 2007-2008.

References 1. Ahmad, I., Wei, X., Sun, Y., Zhang, Y.-Q.: Video Transcoding: an Overview of Various Techniques and Research Issues. IEEE Transactions on Multimedia 7, 793– 804 (2005) 2. Vetro, A., Christopoulos, C., Sun, H.: Video Transcoding Architectures and Techniques: an Overview. Signal Processing Magazine 20, 18–29 (2003) 3. Xin, J., Lin, C.-W., Sun, M.-T.: Digital Video Transcoding. Proceedings of the IEEE 93, 84–97 (2005) 4. Advanced Video Coding for Generic Audiovisual Services. Version 8 (including SVC extension). ITU-T Rec. H.264 and ISO/IEC 14496-10 (MPEG-4 AVC), ITU-T and ISO/IEC JTC 1 (2007) 5. Zhang, P., Lu, Y., Huang, Q., Gao, W.: Mode Mapping Method for H.264/AVC Spatial Downscaling Transcoding. In: Proc. IEEE Int. Conference on Image Processing (ICIP), Singapore, vol. 4, pp. 2781–2784 (2004) 6. Notebaert, S., De Cock, J., Van de Walle, R.: Improved H.264/AVC Requantization Transcoding Using Low-Complexity Interpolation Filters for 1/4-Pixel Motion Compensation. In: IEEE Symposium on Computational Intelligence in Image and Signal Processing (CIISP), Honolulu, pp. 307–312 (2007) 7. Lefol, D., Bull, D.R.: Mode Reﬁnement Algorithm for H.264 Inter Frame Requantization. In: Proc. IEEE Int. Conference on Image Processing (ICIP), Atlanta, USA, pp. 845–848 (2006)

Lossless and Near-Lossless Image Compression Scheme Utilizing Blending-Prediction-Based Approach Grzegorz Ulacha and Piotr Dziurzanski Szczecin University of Technology ul. Zolnierska 49, 71-210 Szczecin, Poland {gulacha,pdziurzanski}@wi.ps.pl

Abstract. An approach for lossless and near-lossless compression of still images together with its system-level multi-core hardware model utilizing blending-prediction-based technique is presented in this paper. We provide a mathematical background of the proposed approach and utilize a Network on Chip type of connection in the hardware model which beneﬁts from a new multi-path routing algorithm and heuristic algorithms for core mapping realizing subsequent stages of the compression algorithm. The experimental results conﬁrming the advantages of the proposed approach are provided. Keywords: Lossless compression, Near-lossless compression, blending predictors, Network on Chip.

1

Introduction

The main goal of the research is to develop an algorithm suitable for hardware multi-core realizations of a lossless video compression system, working in realtime. This requirement is caused by the main application of the lossless video sequences compression, namely the possibility of post-production of source materials used in television and video content production [1]. In this task, it is not recommended to use lossy video compression algorithms, as MPEG2, MPEG4, etc. Other important applications of lossless video sequence compression is archiving of: medical 2D and 3D images (in general, multi-planar 3D objects) [25], astronomic data, and compression of satellite photos [4, 29]. Since the application range of the developed system is relatively broad and may include various embedded systems, there is a strict demand in terms of the implementation eﬃciency, area and power consumption. To cope with these requirements, the Multi Processor Systems on Chips (MPSoCs) are often proposed [3]. In contrast to the contemporary chip architecture, future MPSoCs would probably include no bus-based nor point to point communication scheme due to their insuﬃcient scalability in terms of the core number in a single chip [3], but would beneﬁt from Network-on-Chip-based (NoC-based) connections, which guarantee high scalability, low contention level and high performance [12]. In this L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 208–217, 2009. c Springer-Verlag Berlin Heidelberg 2009

Lossless and Near-Lossless Image Compression

209

paper, we consider an MPSoC realizing the proposed compression system that exploits the 2D mesh architecture. Each tile is comprised of the core realizing one stage of the proposed algorithm and a router connected to its direct vertical and horizontal neighbors.

2

Related Works

The continuous attempts to obtain better eﬃciency of the lossless image compression lead to developing methods of increased implementation complexity. Among popular lossless image compression methods is, considered as one of the most eﬃcient, CALIC [24], presented in 1996. The less computational intensive method, LOCO-I, has been approved as standard JPEG-LS [23]. The second group is formed by the methods utilizing algorithms of large implementation complexity, such as TMW (1997) [16], WAVE-WLS (2002) [27] and MRP 0.5 (2005) [15]. Apart from the metioned above methods utilizing predictive modeling, there are lossless versions of wavelet codecs used for encoding in the intraframe (e.g. JPEG-2000 [13]) and interframe [17] modes. However, the obtained results are inferior in comparison with the best predictive methods. An analysis of lossless video sequences encoding has been presented in [1], where the mehods: LOPT-3D, GLICBAWLS-3D, JPEG-LS and JPEG-2000 have been compared. Taking into consideration the implementation complexity for real-time systems, in [1] it was proposed an video sequences encoding as a set of independent frames using a reduced version of the JPEG-2000 standard. It makes it possible to compress in the visually lossless mode, i.e., with the relatively low quality deterioration, invisible for humans. The proposed techniques: GLICBAWLS-3D, LOPT-3D (and its extension LPOSTC [2]) belong to the solutions characterized with high eﬃciency, but their computational complexity is too high to apply them in real-time systems. Having looked for a solution being a compromise between eﬃciency and complexity, we decided to apply the blending predictors method, which is characterized with the highest ﬂexibility. This technique has been applied, among others, in the already mentioned TMW and WAVE-WLS methods. The idea of blending predictors (forming a set of subpredictors) has been described in [20], extended by, e.g., G. Deng and H. Ye and presented in [6], [27].

19

11

8

6

9

12

22

15

7

3

2

4

10

18

13

5

1

xn

28

Fig. 1. Numbering of the neighborhood pixels

210

3

G. Ulacha and P. Dziurzanski

Proposed Algorithm for Image Modeling and Compression

In our work, we have selected a set of subpredictors based on the results of an average entropy of 45 benchmark images for a set of more than 30 constant predictors, taken from [5, 7, 10, 14, 19, 20]. An analysis of non-linear predictors based on the context-sensitive switching between constant predictors has been also performed. The simplest version is three-context median ﬁlter MED [23], two subsequent versions: GAP [24] and GAP+ [22] utilize 7 contexts. Given the vector X = [xj ] with values of 13 subpredictors: x1 = GAP+ [22], x2 = GradWest = 2P (1) − P (5), x3 = GradNorth = 2P (2) − P (6), x4 = Plane = P (1) + P (2) − P (3), x5 = Plane2 = P (1) − P (2) + P (4) [20], x6 = P (1), x7 = P (2), x8 = P (3), x9 = P (4), x10 = P (5), x11 = P (10), x12 = P (18), x13 = P (28), where P (i) denotes the pixel with index i from the closest neighborhood (see Fig. 1). With each subpredictor xj , its prediction error ej (0) = P (0)xj is associated. Then based on the vector of subpredictor weight errors S = [2, 2, 1, 1, 1, 1, 1, 1, 1, 1] and 13 vectors Ej = [ej (1), ej (2), . . . , ej (10)], where each of them is comprised of 10 prediction errors from the closest neighborhood (see Fig. 1) associated with subpredictor j, temporal significance vector is determined Y = [yj ], for j = 1, . . . , 13, where values of yj are computed according to the formula: 1 yj = (1) 1 + diag(EjT · Ej )) · S T Then, using the subpredictor power vector H = [1, 2, 2, 1, 3/2, 1, 1, 1, 1, 1, 1, 1, 1], we determine the subpredictor weight vector: W = diag(H T · Y )

(2)

and vector A with prediction coeﬃcients, which is obtained after the W vector normalization: 1 A= ·W (3) W · diag(I) where I is the identity matrix of dimensions 13×13 elements. Finally, the estimated value is computed in the following way: x ˆ = A · XT

(4)

The next stage is to apply the adaptive method of bias cancelation C[i], which is calculated by correction of the prediction error associated with the appropriate context with index i. This technique is usually referred to as contextbased error correction and is utilized in both CALIC and JPEG-LS methods. In our algorithm, we used 1024 contexts and the average value of these two methods [21]: CCALIC [i] + CJPEG-LS [i] x˙ = x ˆ+ (5) 2

Lossless and Near-Lossless Image Compression

211

The compression consists in encoding the prediction errors only, i.e., the diiﬀerence e between the real and the estimated values (rounded to the nearest integer), whose values are usually close to zero: e = P (0) − x˙

(6)

We have applied seven universal and two 16-context context-sensitive adaptive arithmetic encoders. The ﬁrst of them encodes the absolute of the error |e| quantized according to the rule T(k) ≤ |e| ≤ T(k+1) , where T(k) = {0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 20, 24, 32, 64, 128}. The value of k is sent to an arithmetic encoder. The quantization error, eq = (|e| − T(k) ), is treated as an q(k) -bit number, where q(k) is the k-th value from the vector q(k) = {0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 5, 6, 7}. If q(k) > 0, the eq value is encoded using one from the 7 universal adaptive arithmetic encoders (with the q(k) number). The role of the second contextual encoder is to encode the sign bit of the e value. Both the encoders make usage of the context switching based on the error level in the closest neighborhood, what helps to determine the individual probability distribution type for the pixel being encoded. More details on the selection of the context number can be found in [21].

4

Lossless Satellite and Space Image Compression

Photographs of the Earth, Mars and other celestial bodies made by satellites and spaceships should be sent and stored in a lossless form. It is particularly important to obtain here high compression eﬃciency with a low level of its hardware realization complexity. The transmission cost of these images can be relatively high, especially when the transmission is performed over a long distance. Taking advantage from the technological progress in the ﬁeld of minimization and power consumption decreasing, it is possible to propose an encoding algorithm of a slightly higher implementation complexity that the solution known from literature [4, 28], but remaining suitable for hardware realizations. The proposal is capable of lossless encoding of about 3 frames per second (512×512 pixels) in case of the still images and, in an extended version, up to 30 frames per second for video sequences. The sequences are then split into independent groups of 10 frames, which are encoded in a pipeline mode that takes into account temporal dependences between neighboring frames. The system tests have been performed for fourteen 8-bit grayscale images from the CCSDS benchmark set (The Consultative Committee for Space Data Systems). The results of the proposed technique are presented in Table 1 together with their comparison with the other solutions characterized with a low implementation complexity, taken from [29]. Among them, there are wavelet methods: CCSDS [29], JPEG2000 [13], SPIHT [18], ICER [9], and predictive techniques: PRDC [28], JPEG-LS [23], FPGA (utterly hardware realization dedicated to reconﬁgurable digital circuits) [4].

212

5

G. Ulacha and P. Dziurzanski

Near-Lossless Mode

To obtain higher compression ratio, it is possible to apply a non-reversible encoding. There exist a large number of lossy compression methods based on discrete cosines transform, DCT (e.g. JPEG), or wavelet transform (e.g. JPEG-2000). These methods make it possible to choose compression ratio, but their basic implementations do not allow their users to select a maximal acceptable error. This possibility is oﬀered by the near-lossless mode, where one is capable of deﬁning the maximal absolute value of d, being the diﬀerence between the pixels values of the original and the decoded images. This mode is often based on predictive encoding, where the set of prediction errors, e, is quantized in the following way [26]: e + d · sgn(e) e= (7) 2d + 1 The previous value of the prediction error can be restored with accuracy ±d, yielding: e˜ = e · (2d + 1) (8) When d = 0, we obtain the lossless mode. This rule has been applied to the near-lossless mode in our system, for both color and grayscale images. In [11] a method of lossy compression of space images, which introduces the noise similar to the one included in the source image was introduced and named as virtually lossless compression. It consists of using the quantization whose noise level is no higher than imposed by the devices for grabbing images. The algorithm decides on the maximal error, d, for each image individually. The nearly lossless method can be used for encoding medical images, where experts should establish the error tolerance, d, which does not inﬂuence on the diagnosis (such compression is sometimes referred to as visually lossless [1]). In [25] a system for hardware compression of medical images was proposed and tested on color endoscope photos with three RGB 8-bit color components and maximal resolution 640×480 pixels (capable of encoding 2 frames per second). This hardware realization is one of the examples when the algorithm simplicity and the near-lossless mode are necessary due to the minimization of the capsule, the wireless system included a camera with a lighting system, an encoding unit, a sender and battery. The size of the capsule, swallowed by a patient, is smaller than 9×20 mm. The main problem is connected with the amount of energy required for powering the system, including the lighting and the sender. The energy eﬃciency was obtained thanks to the reduction of the transmitted data due to the application of the near-lossless image compression. In this system, the near-lossless compression with the maximal error d = 2 was used, which was proposed as a trade-of between the quality of image and the compression eﬃciency. The developers decided to quantize image coeﬃcients directly, which were subjected to the reversible ﬁltering process earlier, instead of quantizing the prediction error. There was also implemented the possibility of lossless compression usage in an Region of Interest, ROI.

Lossless and Near-Lossless Image Compression

213

Table 1. Comparison of lossless compression methods on benchmark images CCSDS Images CCSDS coastal b1 3.36 coastal b2 3.22 coastal b3 3.48 coastal b4 2.81 coastal b5 3.16 coastal b6h 3.02 coastal b6l 2.35 coastal b7 3.45 coastal b8 3.66 europa3 6.61 marstest 4.78 lunar 4.58 spot-la b3 4.80 spot-la panchr 4.27 Average 3.82

PRDC JPEG-LS JPEG 2000 3.56 3.09 3.13 3.32 2.90 2.97 3.68 3.22 3.23 2.91 2.41 2.53 3.30 2.81 2.94 2.75 2.50 2.60 2.03 1.76 1.96 3.66 3.17 3.22 3.93 3.42 3.40 7.48 6.64 6.52 5.39 4.69 4.74 5.23 4.35 4.49 5.20 4.53 4.69 4.87 4.00 4.13 4.09 3.54 3.61

SPIHT 3.09 2.94 3.21 2.57 2.91 2.71 2.02 3.17 3.35 6.46 4.64 4.43 4.70 4.11 3.59

ICER 3.07 2.92 3.20 2.55 2.89 2.54 1.87 3.15 3.31 6.30 4.63 4.40 4.56 4.03 3.53

FPGA Blend-A13+ 3.00 2.934 2.84 2.781 3.14 3.052 2.37 2.372 2.79 2.709 2.52 2.432 1.84 1.842 3.10 3.009 3.28 3.241 6.42 6.100 4.60 4.345 4.20 4.114 4.43 4.365 3.90 3.829 3.46 3.366

Table 2. Average bitrates for the Blend-A13+ method applied to near-lossless image compression benchmarks CCSDS Images d=1 d=2 d=3 d=4 d=5 d=6 d=7 Average 2.007 1.476 1.180 0.991 0.852 0.750 0.668

Table 3. Average PSNR value for the Blend-A13+ encoding method applied to nearlossless image compression benchmarks CCSDS Error value: d=1 d=2 d=3 d=4 d=5 d=6 d=7 PSNR [dB] 49.956 45.331 42.417 40.153 38.429 36.893 35.590 Table 4. Comparison of PSNR for the near-lossless Blend-A13+ method with the JPEG-2000 standard for image marstest Blend-A13+ d=1 d=2 d=3 d=4 d=5 d=6 d=7 PSNR 49.882 45.154 42.185 40.028 38.327 36.882 35.703 Bitrate 2.893 2.294 1.932 1.682 1.494 1.347 1.231 JPEG-2000 d=7 d=10 d=14 d=16 d=19 d=20 d=23 PSNR 45.525 43.139 41.390 40.031 38.683 37.661 36.856 Bitrate 2.893 2.297 1.936 1.682 1.494 1.349 1.231

Using the above described example of energy saving, our further experiments are conducted for the near-lossless mode. Tab. 2 presents average bitrates, and Tab. 3 PSNR demonstrates values for benchmark images with maximal error value d ≤ 7. In Tab. 4, it is shown the PSNR comparison between the near-lossless mode and standard JPEG2000 (encoder IrfanView 4.10) for image marstest. From the analysis it follows that for the compression ratio obtained for

214

G. Ulacha and P. Dziurzanski

50 49 48 47 46

PSNR [dB]

45 44 43 42 41 40 39 38 37 36 35 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 Bitrate

Fig. 2. Comparison of PSNR values depending on the average bitrate for the BlendA13+ method in the near-lossless mode for d ≤ 7 (solid line) and JPEG-2000 (dotted line)

errors d ≤ 3 the proposed method, Blend-A13+ , yields better results than the eﬀective wavelet method, JPEG-2000 (see Fig. 2). It is worth stressing that for the wavelet method it is impossible to deﬁne the maximal error value d a priori, and comparing both these methods with the same compression ratio, the d error obtained with JPEG-2000 is distinctly higher than for Blend-A13+ , which is shown in Tab. 4.

6

System-Level Hardware Model

In our system-level hardware model, we use an MPSoC whose cores form a regular 2D mesh. Each core is connected to its horizontal and vertical neighbors forming a Network on Chip (NoC). The MPSoC follows the GALS (Globally Asynchronous, Locally Synchronous) type of synchronization; the transmission between cores is performed according to the request-acknowledge scheme. The data buses between the neighboring cores are 8-bit wide in both directions. To transfer the data, we decided to use the wormhole routing scheme [3], where a packet is split into smaller units, called ﬂits, which are small enough (8 bit) to be transmitted in parallel by the data bus. In our realization, the ﬁrst ﬂit contains the 8-bit destination address (its Cartesian coordinates), whereas the second one includes the information about the payload data length. Having obtained the ﬁrst ﬂit, each router selects the next-hop router using a particular routing algorithm and the remaining ﬂits follow the same path. In the most popular XY routing algorithm, ﬂits are ﬁrstly routed according to the X axis as long as the X coordinate is equal to the X coordinate of the

Lossless and Near-Lossless Image Compression

215

destination core, and then the ﬂit is routed vertically. In this scheme, even two data-ﬂows to be transmitted by the same link result in contention. To perform the mapping of IP cores into NoC tiles decreasing the level of contention, we developed the following method. We begin with a construction of a ﬂow network for a given stream-based algorithm, in our case, the proposed blending-predictor-based encoder. Eight processing cores (M0-M7) have been identiﬁed and the transfers between them (in bits per pixel) have been determined, as presented in Fig. 3. In the graph, the M0 node denotes a preparatory module, which sends data to parallel module M1-M4 that compute the subpredictors. M5 and M6 are two parts of the blending module, and M7 is the arithmetic encoder. Having constructed the ﬂow graph, we have to ﬁnd an appropriate mapping of the cores into the mesh structure so that the highest transfer is minimized. This value is treated as the necessary link capacity of each link in the model so as to avoid deadlocks. In such homogenous architecture, a majority of the remaining links are utilized in small percentage of this maximal value when the XY algorithm is applied. However, this situation can be improved by applying the proposed multi-path routing scheme, where each router is equipped with a routing table enlisting all the routes to a destination core sorted with respect to the length of the route and the equal length routes are sorted according to the XY rule. When a router receives a package, it transfers its ﬂits using a number of routes in such the way that the link utilization is balanced and the overall transfer is minimized. The classical Ford-Fulkerson algorithm is used to determine the paths to be transferred. For more details about this routing scheme, the reader is referred to [8].

96 72

M0

M1

M2 120

64 64

120

152

M3

M5

24

M6

48

M7

152

M4

Fig. 3. Flow network of the proposed encoder (transfers in bits per pixel)

We split the functionality of the proposed Blend-A13+ encoder according to ﬂow network shown in Fig. 3 and implemented each core in the SystemC language. Then, we embed these cores in a 3×3 mesh NoC architecture nodes. After analyzing the whole search space (9! possible mapping permutations), we found an optimal permutation where the transfer using the XY algorithm is minimal and equal to 1232 bits per pixel. On the other hand, the maximal obtained transfer is equal to 2608 bits per pixel, whereas the average for all the permutations equals 1824 bits per pixel. When the proposed routing algorithm is applied to the permutation optimal for the XY algorithm, the obtained transfer is equal to only 158 bits per pixel, i.e., it is decreased by 85 percent.

216

7

G. Ulacha and P. Dziurzanski

Conclusion

In this paper, we proposed an algorithm for a lossless and near-lossless compression of still images. According to the presented experimental results, the compression ratio obtained with the proposed method yields better results than the eﬀective wavelet method, JPEG-2000 when only low level of errors is available. In order to operate in real-time system, we implemented this algorithm in a Network-on-Chip-based model and used our routing strategy to decrease the link bandwidth by more than 80 percent in comparison with the state-of-the-art XY wormhole routing scheme.

References 1. Andriani, S., Calvagno, G., Erseghe, T., Mian, G.A., Durigon, M., Rinaldo, R., Knee, M., Walland, P., Koppetz, M.: Comparison of lossy to lossless compression techniques for digital cinema. In: International Conference on Image Processing, vol. 1, pp. 513–516 (2004) 2. Andriani, S., Calvagno, G., Mian, G.A.: Lossless Video Compression using a SpatioTemporal Optimal Predictor. In: 13th European Signal Processing Conference EUSIPCO-05 CD (2005) 3. Bjerregaard, T., Mahadevan, S.: A Survey of Research and Practices of Networkon-Chip. ACM Computing Surveys 38, Article 1 (2006) 4. Chen, X., et al.: Lossless Compression for Space Imagery in a Dynamically Reconﬁgurable Architecture. In: International Workshop on Applied Reconﬁgurable Computing, pp. 336–341 (2008) 5. Daaboul, A.: Local prediction for lossless image compression. In: Prague Stringology Club Workshop, pp. 44–50 (1998) 6. Deng, G., Ye, H.: A general framework for the second-level adaptive prediction. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3, pp. III 237–III 240 (2003) 7. Deng, G., Ye, H.: Lossless image compression using adaptive predictor combination, symbol mapping and context ﬁltering. In: IEEE International Conference on Image Processing, vol. 4, pp. 63–67 (1999) 8. Dziurzanski, P., Maka, T.: Stream Transfer Balancing Scheme Utilizing Multi-Path Routing in Networks on Chip. In: Woods, R., Compton, K., Bouganis, C., Diniz, P.C. (eds.) ARC 2008. LNCS, vol. 4943, pp. 294–299. Springer, Heidelberg (2008) 9. Kiely, A., Klimesh, M.: The ICER Progressive Wavelet Image Compressor. The Interplanetary Network Progress Report 42(155) (2003) 10. Kuroki, Y., Ueshige, Y., Ohta, T.: An estimation of the predictors implemented by shift operation, addition, and/or substraction. In: International Conference on Image Processing, pp. 474–477 (2001) 11. Lastri, C., Aiazzi, B., Alparone, L., Baronti, S.: Virtually Lossless Compression of Astrophysical Images. EURASIP Journal on Applied Signal Processing 2005(15), 2521–2535 (2005) 12. Lee, H.G., et al.: On-chip communication architecture exploration: A quantitative evaluation of point-to-point, bus, and network-on-chip approaches. ACM Transactions on Design Automation of Electronic Systems 12(3) (2007)

Lossless and Near-Lossless Image Compression

217

13. Marcellin, M., Gormish, M., Bilgin, A., Boliek, M.: An Overview of JPEG-2000. In: Data Compression Conference, pp. 523–541 (2000) 14. Marusic, S., Deng, G.: New prediction schemes for lossless coding of fullband and subband images. Signal Processing: Image Communication 14, 869–878 (1999) 15. Matsuda, I., Ozaki, N., Umezu, Y., Itoh, S.: Lossless coding using Variable BlockSize adaptive prediction optimized for each image. In: 13th European Signal Processing Conference EUSIPCO-05 CD (2005) 16. Meyer, B., Tischer, P.: TMW - a new method for lossless image compression. In: International Picture Coding Symposium, pp. 533–538 (1997) 17. Park, S.G., Delp, E.J.: Adaptive lossless video compression using an integer wavelet transform. In: International Conference on Image Processing, vol. 4, pp. 2251–2254 (2004) 18. Said, A., Pearlman, W.A.: A New, Fast, and Eﬃcient Image Codec Based on Set Partitioning in Hierarchical Trees. IEEE Transactions on Circuits and Systems for Video Technology 6(3), 243–250 (1996) 19. Sayood, K.: Introduction to Data Compression, 2nd edn. Morgan Kaufmann, San Francisko (2002) 20. Seemann, T., Tisher, P.: Generalized locally adaptive DPCM. Department of Computer Science Technical Report CS97/301, Monash University, Australia, pp. 1–15 (1997) 21. Ulacha, G., Dziurza´ nski, P.: Blending-prediction-based approach for lossless image compression. In: 1st International Conference on Information Technology, pp. 471– 474 (2008) 22. Wang, H., Zhang, D.: A linear model and its application in lossless image coding. Signal Processing: Image Communication 19, 955–958 (2004) 23. Weinberger, M.J., Seroussi, G., Sapiro, G.: LOCO-I: Lossless Image Compression Algorithm: Principles and Standardization into JPEG-LS. IEEE Trans. on Image Processing 9(8), 1309–1324 (2000) 24. Wu, X., Memon, N.D.: CALIC - A Context Based Adaptive Lossless Image Coding Scheme. IEEE Trans. on Communications 45, 437–444 (1996) 25. Xie, X., Li, G., Wang, Z.: A Near-Lossless Image Compression Algorithm Suitable for Hardware Design in Wireless Endoscopy System. EURASIP Journal on Advances in Signal Processing, 48–61 (2007) 26. Xue, X.: Prediction Based On Backward Adaptive Recognition Of Local Texture Orientation And Poisson Statistical Model For Lossless/near-Lossless Image Compression. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 6, pp. 3137–3140 (1999) 27. Ye, H.: A study on lossless compression of grayscale images. PhD thesis. Department of Electronic Engineering, La Trobe University (2002) 28. Lossless Data Compression. Recommendation for Space Data System Standards, CCSDS 121.0-B-1. Blue Book. Issue 1. CCSDS, Washington, D.C. (1997) 29. Lossless Data Compression. Recommendation for Space Data System Standards, CCSDS 120.1-G-1. Green Book. Issue 1. CCSDS, Washington, D.C. (2007)

Marker Less Vision-Based Tracking of Partially Known 3D Scenes for Outdoor Augmented Reality Applications Fakhreddine Ababsa, Jean-Yves Didier, Imane Zendjebil, and Malik Mallem IBISC Laboratory – CNRS FRE 3190 40, rue du Pelvoux 91020 Evry France {ababsa,didier,zendjebil,mallem}@iup.univ-evry.fr

Abstract. This paper presents a new robust and reliable marker less camera tracking system for outdoor augmented reality using only a mobile handheld camera. The proposed method is particularly eﬃcient for partially known 3D scenes where only an incomplete 3D model of the outdoor environment is available. Indeed, the system combines an edgebased tracker with a sparse 3D reconstruction of the real-world environment to continually perform the camera tracking even if the model-based tracker fails. Experiments on real data were carried out and demonstrate the robustness of our approach to occlusions and scene changes.

1

Introduction

Augmented Reality Systems (ARS) attempt to enhance humans’ perception of their indoors and outdoors working and living environments by complementing their senses with virtual input. Tracking computation refers to the problem of estimating the position and orientation of the ARS user’s viewpoint, assuming the user to carry a wearable camera. Tracking computation is crucial in order to display the composed images properly and maintain correct registration of real and virtual worlds. Generally, outdoor augmented reality systems use GPS and inertial sensors to measure, respectively, the position and the orientation of the camera’s view point. However, in reality several events can degrade the quality of the GPS signal like: shadowing and occlusions from buildings, self-occlusions (hand, body) and multiple signal reﬂec- tions. In the same way, inertial sensors can drift and are disturbed by local magnetic ﬁelds being in the working area. A vision-based localization provide an interesting alternative because of its accuracy and robustness against these environmental inﬂuences. Several visionbased marker less tracking approaches have been developed last years. Among them, the model-based tracking methods are most interesting for ARS and give the best results [1],[2],[3]. The mean idea of the model-based techniques is to identify, in the images, features from the object model (points, lines, etc.). One can note two main types of approaches for model-based tracking algorithms, namely the edge-based and the textured-based trackers. Both have complementary advantages and drawbacks. An interesting idea is then to combine both L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 218–227, 2009. c Springer-Verlag Berlin Heidelberg 2009

Marker Less Vision-Based Tracking of Partially Known 3D Scenes

219

approaches in the same process [4],[5]. For example, in Vachetti et al. [6] the proposed method combines edge feature matching and the use of keyframes to handle any kind of camera displacement. The model information is used to track every aspect of the target object. A set of reference keyframes is created oﬀ-line and, if there are too few of them, new frames can be automatically added online. In the same way, the framework presented in Pressigout et al. [7] fuses a classical model-based approach based on edge extraction and a temporal matching relying on texture analysis into a single nonlinear objective function that has then to be minimized. Tracking is formulated in terms of a full scale non-linear optimization. All theses approaches require that the natural points to be tracked belong on the textured area of the 3D model. However, in outdoor applications, the user needs to move freely around a large-scale environment for which it is impossible to have a complete 3D model, so classical hybrid approaches don’t work in this case. Moreover, large variations in lighting, scaling and occlusions often occur in such environments. So, developing a marker less tracking system robust against these problems is a very challenging task. In this paper, we propose an original solution combining two complementary tracking methods: An edge-based tracker with a dynamic sparse 3D key points tracking. In order to improve the accuracy and the robustness of the camera pose estimation we have integrated a M-estimator into the optimization process. The resulting pose computation algorithm is thus able to deal eﬃciently with incorrectly tracked features. The sparse key points are constructed dynamically during the motion and tracked in the current frame in order to maintain an estimate of the camera pose even if the 3D model of the scene becomes non visible in the image. We propose to use the SIFT algorithm [8] in order to extract a stable set of natural key points which are invariant to image scaling and rotation, and partially invariant to changes in illumination and view point. The remainder of this paper is organized as follows. In section 2, gives the details of our real-time 3D lines and 3D sparse key points trackers. Section 3 presents the robust formulation of the camera pose estimation problem when using point and line features. System performances and results on real data are discussed in section 4, followed by ﬁnal conclusion in section 5.

2

Description of Our Tracking Framework

Our marker less tracking framework is composed of two natural features trackers: a 3D line tracking system and a sparse 3D key points tracking one. 2.1

3D Line Tracker

Our line tracking approach is based on the moving edges algorithm (ME) [9]. The M E method is very interesting for our application because edge extraction is not required. Only point coordinates and image intensities are manipulated. This leads to real-time computation. As described in ﬁgure (1) the line model is projected with the initial camera pose into the current image plane. Let Pi be

220

F. Ababsa et al.

the sampled points along the 3D line model. Pi are projected into the 2D image points pi . Then, for each point pi, a local search is performed on a search line in the direction of the normal to the reprojected contour. We attribute to pi one correspondent qi which is the local extremum of the gradient along the search line.

Fig. 1. Line tracking approach

In our approach we only need to ﬁnd two image points (q1 and q2 ) belonging to the object line. This property make our line tracker algorithm robust to severe occlusion. So, when partial occlusion occurs, it is suﬃcient to detect small parts of the model edges to estimate the camera pose. 2.2

3D Sparse Key Points Tracker

The goal of this method is to incrementally construct a sparse set of 3D points of the viewed scene. These points are then tracked in the current frame and used in camera pose estimation process. The algorithm begins by determining the camera poses of the ﬁrst two images. Local point features are extracted and matched from these two calibrated frames and 3D key points are constructed by triangulation using the computed camera parameters. These 3D points are used to initialize the sparse key points tracker. In addition, as the system operates in incremental way, the new matched points at the current frame are automatically triangulated and their corresponding 3D points are added for the tracking. The eﬀectiveness of this method depends on the robustness of feature points extraction and matching algorithms. So, we propose to use stable natural features generated by the SIFT algorithm [8]. The strong ness of these features is their invariance to image scaling and rotation, and their partially invariance to changes in illumination and view point which make them suitable for outdoor environments. Moreover, SIFT features are present in abundance (see ﬁgure 2) over a large range of image scales, this allows an eﬃcient features tracking in the presence of occlusions. The SIFT algorithm assigns to the extracted key points an image location, a scale, an orientation (in the range of [−π, +π]) and a feature descriptor vector. Feature matching is performed using the Euclidean distance criterion. Thus, two features extracted from two successive images are matched if the Euclidean distance between their descriptors vectors is minimal.

Marker Less Vision-Based Tracking of Partially Known 3D Scenes

221

At this point, we establish several 2D-3D points correspondences which are used to estimate the camera pose at the current frame.

Fig. 2. SIFT key points extracted from the Saumur Castle image

3

Robust Camera Pose Problem Formulation

Throughout this paper, we assume a calibrated camera and a perspective projection model. If a point has coordinates (x, y, z)t in the coordinate frame of the camera, its projection onto the image plane is (x/z, y/z, l)t. In this section, we present the constraints for camera pose determination when using point and line features. 3.1

Point Constraint

Let pi = (xi , yi , zi )t , i = 1, . . . , n, n ≥ 3 a set of 3D non-collinear reference points deﬁned in the world reference frame, the corresponding camera-space coordinates qi = (xi , yi , zi ) are given by: qi = Rpi + T

(1)

where R = (rt1 , rt2 , rt3 )t and T = (tx , ty , tz )t are a rotation matrix and a translation vector, respectively. R and T describe the rigid body transformation from the world coordinate system to the camera coordinate system and are precisely the parameters associated with the camera pose problem. Let the image point gi = (ui , vi , l)t be the projection of pi on the normalized image plane. Using the camera pinhole model, the relationship between gi and pi is given by: 1 qi = t (Rpi + T ) (2) r3 pi + tz which is known as the co linearity equation.

222

F. Ababsa et al.

Fig. 3. Perspective projection of 3D line

The point constraint corresponds to the image space error, it gives a relationship between 3D reference points, their corresponding 2D extracted image points and the camera pose parameters as follows: 2 2 rt pi + tx rt pi + ty ˆ i − 1t ˆ i − 2t Eip = u + v (3) r3 pi + tz r3 pi + tz ˆ i = (ˆ where m ui , vˆi , l)t are the observed image points. 3.2

Line Constraint

Given correspondences between 3D lines and 2D lines found in the image, the goal is to ﬁnd the rotation and the translation matrices which map the world coordinate system to the camera coordinate system. Let L be an object line. Several representation for a 3D line have been proposed. In our approach, we represent the 3D line L by its two end-points p1 and p2 (see ﬁgure 3). The point pi in world coordinates can be expressed in the camera frame by equation (1). Let (x − 1, y1 , z1 ) and (x2 , y2 , z2 ) (be the camera coordinates of the end-points p1 and p2 which project onto the image plane at m1 and m2 respectively. The projection plane formed by the image line (m1 m2 ) is given by the plane (Om1 m2 ) and the 3D line L1 must lie in this plane. The normal N to the projection plane is given by: N = n1 × n2 (4) Where |vecn1 and n2 are the optical rays of the image points m1 and m2 . Thus the 3D line constraint can be formulated as: Eit = N · (Rpi + T )

(5)

The 3D line constraint represents the fact that any point on the 3D line in camera coordinates ideally must lie in the projection plane. This constraint relate both rotation and translation parameters pose to the 3D model and 2D image lines.

Marker Less Vision-Based Tracking of Partially Known 3D Scenes

3.3

223

Robust Camera Pose Estimation

Since the number of camera pose parameters is 6 (3 for rotation and 3 for translation) the pose problem can be solved if at least 3 features (points or lines) correspondences are available. In the general case, if N and M are the numbers of points and lines correspondences, respectively, the camera pose problem corresponds to the problem of minimizing the following error function: f1 (R, T ) +

N i=1

(Eip )2 +

M

(Ei1 )2

(6)

i=1

Where Eip and Eil correspond to the point and line constraints deﬁned in equation (3) and (5 ) respectively. Traditionally, this minimization is performed using a least square approach [10], [11], [12]. However, when outliers are present in the measures, a robust estimation is required. In the pose problem, outliers occur either due to incorrect 2D-3D correspondences or if parts of the 3D model is incorrect. Historically, several approaches to robust estimation were proposed, including R-estimators and L-estimators. However, M-estimators now appear to dominate the ﬁeld as a result of their generality, high breakdown point, and their eﬃciency [13]. Mestimators are a generalization of maximum likelihood estimators, they attempt to deﬁne objective functions whose global minimum 2 would not be signiﬁcantly aﬀected by outliers. So, Instead of minimizing i ri where ri are residual errors, M-estimators minimize its robust version i ρ(ri ) The function ρ(u) is designed to be a continues, symmetric function with minimum value at u = 0. Also ρ(u) must be monotonically non decreasing with increasing |u|. Many such functions have been proposed. Huber has developed the Iterative Reweighted Least Square (IRLS) method to minimize this robust error. It converts the M-estimation problem into an equivalent weighted least-squares. We have incorporated the IRLS algorithm in our minimization procedure for the camera pose parameters. This is achieved by multiplying the error functions in equation (6) by weights. The objective function to be minimized is then deﬁned by: f2 (R, T ) +

N i=1

wip (Eip )2 +

M

wi1 (Ei1 )2

(7)

i=1

Where wip and wi1 the weights and reﬂect the conﬁdence in each feature and their computation is described in [14]. Rotations can be represented by several diﬀerent mathematical entities (matrices, axe and angle, Euler angles, quaternion). However, quaternion have proven very useful in representing rotations due to several advantages above the other representations: more compact, less susceptible to round-oﬀ errors, avoid discontinuous jumps. A quaternion representation of rotation is written as a normalized four dimensional vector q = q0 qx qy qz where (q02 + qx2 + qy2 + qz2 = 1). Thus the vector β to estimate is constituted of six parameters, the three components of the translation vector and the three ﬁrst components of the unit quaternion q: β = (qx qy qz tx ty tz )

(8)

224

F. Ababsa et al.

We have used the Levenberg-Marquardt (LM) algorithm to resolve this optimization problem. The parameters updating is given by: βi+1 = βi + (J · D · J + λ · I)−1 · J · D · Ei

(9)

where D is a diagonal weighting matrix given by D = dialog(w1 , w2 , . . . , wn ). The weights wi are deﬁned by the IRLS algorithm. J is the Jacobian matrix of the objective function f2 (R, T ) and Ei the feature errors.

4

Results

To evaluate the performances of our tracking system, we considered a complex real scene (ﬁgure 4-a) representing the castle of Saumur. We recorded several image sequences from a moving camera pointing towards the castle (image size is 480 × 640 pixels). The whole 3D model of the castle is not available, we have only an incomplete 3D model of the northern tower of the castle (ﬁgure 4-b). Figure 4-c shows the wire frame model of this tower extracted from its 3D CAD model.

(a) (b) (c) Fig. 4. Outdoor scene and 3D models used for evaluation

In order to initialize the 3D sparse points tracker we used the poses of the two ﬁrst images to construct the initial set of the 3D keypoints to be tracked. Figure (5-a) shows the SIFT features matched in theses initial images. 100% of inliers have been obtained. In ﬁgure (5-b) the corresponding 3D points computed by triangulation are illustrated. These 3D points are then combined with the 3D model edges and used by the robust estimator to compute the camera pose. The 3D sparse points are dynamically re computed. Thus, for each new frame, SIFT features are detected and matched with points detected in previous frame. Obtained inliers are then triangulated in order to generate new 3D sparse points. In our approach, we don’t need neither to memorize constructed 3D sparse points nor to create oﬀ-line a set reference key points.

Marker Less Vision-Based Tracking of Partially Known 3D Scenes

225

Fig. 5. 3D keypoints tracker initialization

The ﬁrst experiment points out the eﬀectiveness of our hybrid features tracker to estimate the camera pose in several conditions. Figure 6 shows the castle tracked from diﬀerent distances and view points. In ﬁgures (6-a) and (6-b) both 3d spares points and visible 3d edges contribute in camera pose estimation. We can see that the 3D model is correctly reprojected in the images which demonstrates the tracking success. We have also evaluated the image registration error (in pixels) and have noted that the hybrid approach is more accurate than points or lines tracking approaches. In ﬁgure (6-c) and (6-d) the whole 3D model is not visible but it is still possible to estimate the camera pose using only 3D sparse points tracking. This demonstrate the robustness of our approach to the non-visibility of the 3D model in the scene and to the partial occlusion of the model.

Fig. 6.

We also evaluated the accuracy of our algorithm in presence of points and lines outliers. Due to the robustness towards outliers of the IRLS method, the

226

F. Ababsa et al.

estimated camera pose still correct even if there is certain quantity of outliers. For example, in presence of 4 outliers (2 points and 2 lines) we obtained a mean error about 3.11 pixels with a standard deviation of 0.64 pixels. Otherwise, Real-time performance of our tracking approach has been achieved by carefully evaluating the processing time to compute the camera pose. We have implemented our algorithm on an Intel Pentium IV 2.99 GHz PC. The computational time more depends on the number of tracked features (points or lines). An example of current computation times is given in Table 1. Table 1. Computation times of our marker less tracker SIFT Feature detection and description Feature matching (< 40) Camera pose computation Frame per seconds

85 ms 20 ms 15 ms 8

This table shows that he run time depends largely on the SIFT feature extraction and matching. So, for real-time (20-25 frames per seconds), each one of the three steps (detection, description, matching) should be faster still. In order to speed up the matching step, we ﬁxed the distance ratio at 0.4. This allows to only keep matches in which the ratio of vector angles from the nearest to second nearest neighbour is less than 0.4.

5

Conclusion

In this paper we presented an eﬃcient marker less method for camera pose tracking in outdoor environments. The system combines 3D edge and 3D sparse keypoints. A robust M-estimator is introduced directly in the optimization process by weighting the conﬁdence on each feature. The resulting pose computation algorithm is thus able to deal eﬃciently with incorrectly tracked features that usually contribute to a compound eﬀect which degrades the system until failure. Experimental results are very encouraging in term of accuracy. They also prove the eﬃciency of our approach to handle real scenes with model occlusion, illumination and view point changes. Future research eﬀorts will include the keypoints tracker optimization in order to improve real-time performance.

Acknowledgements This research is supported by the ANR grant TL06 (project Raxenv).

References 1. Wuest, H., Vial, F., Stricker, D.: Adaptive Line Tracking with Multiple Hypotheses for Augmented Reality. In: Proceedings of ISMAR 2005, Vienna, Austria, pp. 62–69 (2005)

Marker Less Vision-Based Tracking of Partially Known 3D Scenes

227

2. Drummond, T., Cipolla, R.: Real-Time Visual Tracking of Complex Structure. IEEE Trans. Pattern Analysis and Machine Intelligence 24(7), 932–946 (2002) 3. Yoon, Y., Kosaka, A., Park, J.B., Kak, A.C.: A New Approach to the Use of Edge Extremities for Model-based Object Tracking. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2005), Barcelona, Spain, pp. 1883–1889 (2005) 4. Vacchetti, L., Lepetit, V., Fua, F.: Combining Edge and Texture Information for Real-Time Accurate 3D Camera Tracking. In: Proceedings of ACM/IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, pp. 48–57 (2004) 5. Pressigout, M., Marchand, E.: Real-time 3D Model-Based Tracking: Combining Edge and Texture Information. In: Proceedings of the 2006 IEEE Int. Conf on Robotics and Automation (ICRA 2006), Orlando, Florida, pp. 2726–2731 (2006) 6. Vacchetti, L., Lepetit, V., Fua, F.: Stable Real-Time 3D Tracking Using Online and Oﬄine Information. IEEE Trans. Pattern Anal. Mach. Intell. 26(10), 1385– 1391 (2004) 7. Pressigout, M., Marchand, E.: Real-Time Hybrid Tracking using Edge and Texture Information. Int. Journal of Robotics Research, IJRR 26(7), 689–713 (2007) 8. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the 7th International Conference on Computer Vision, pp. 1150–1157 (1999) 9. Bouthemy, P.: A Maximum Likelihood Framework for Determining Moving Edges. IEEE Trans. Pattern Analysis and Machine Intelligence 11(5), 499–511 (1989) 10. Lowe, D.G.: Fitting Parameterized Three-Dimensional Models to Images. IEEE Trans. Pattern Analysis and Machine Intelligence 13, 441–450 (1991) 11. Haralick, R.M.: Pose Estimation From Corresponding Point Data. IEEE Trans. Systems, Man, and Cybernetics 19(6), 1426–1446 (1989) 12. Lu, C.P., Hager, G., Mjolsness, E.: Fast and Globally Convergent Pose Estimation from Video Images. IEEE Trans. on Pattern Analysis and Machine Intelligence 22(6), 610–622 (2000) 13. Huber, P.J.: Robust Statistics. Wiley, New York (1981) 14. Kumar, R., Hanson, A.R.: Robust Methods for Estimating Pose and a Sensitivity Analysis. CVGIP: Image Understanding 60(3), 313–342 (1994)

Geometric and Optical Flow Based Method for Facial Expression Recognition in Color Image Sequences Ayoub Al-Hamadi, Robert Niese, Saira S. Pathan, and Bernd Michaelis Institute for Electronics, Signal Processing and Communications (IESK) Otto-von-Guericke-University Magdeburg, Germany {Ayoub.Al-Hamadi,Robert.Niese}@ovgu.de

Abstract. This work proposes new static and dynamic based methods for facial expression recognition in stereo image sequences. Computer vision 3-d techniques are applied to determine real world geometric measures and to build a static geometric feature vector. Optical ﬂow based motion detection is also carried out which delivers the dynamic ﬂow feature vector. Support vector machine classiﬁcation is used to recognize the expression using geometric feature vector while k-nearest neighbor classiﬁcation is used for ﬂow feature vector. The proposed method achieves robust feature detection and expression classiﬁcation besides covering the in/out of plane head rotations and back and forth movements. Further, a wide range of human skin color is exploited in the training and the test samples. Keywords: Facial Expression Analysis, Human Computer Interaction.

1

Introduction

In this work, we exploit an important domain of Human Computer Interaction (HCI), which is the eﬀective visual facial expression recognition [1]. The facial expression analysis facilitates information about the person’s emotion and perception which enables an insight to interpersonal behavior. Moreover, it indexes physiological as well as psychological functioning which exploits the usability in many application areas such as patient state analysis or for evaluating preverbal infants. Another application domain is to use these facial expressions to communicate with the computers along with other devices for instance. This motivates researchers to use generalized systems which are easy to adopt in any environment. To make valid, more accurate and quantitative measurements of facial expression in diverse applications, it is needed to develop automated methods for expression recognition. Generally, these systems include automatic feature extractors and change trackers for facial features in static images and video sequences. Many approaches have been described for facial expression analysis using still images and video sequences, which can provide more information regarding expressive behavior. Black and Yacoob did one of the earliest pioneer works in this area [2]. They used local parametric motion models to regions of the face and feeding the L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 228–238, 2009. c Springer-Verlag Berlin Heidelberg 2009

Geometric and Optical Flow Based Method

229

resultant parameters to a classiﬁer. Valstar and Pantic analyzed the facial changes occurring due to expressions [3]. For this purpose, they recognized the facial muscle action units and analyzed their correspondence in the temporal domain. Kumano and colleagues proposed a pose invariant facial expression recognition method using variable intensity templates over the whole face [4]. They described that the changes in these intensity templates varied for diﬀerent facial expressions and can be used for classiﬁcation. Bartlett et al. presented a user independent facial action’s recognition approach with frontal faces in video streams using an Adaboost classiﬁer and support vector machines (SVM) [5]. They analyzed the diﬀerence between spontaneous and the posed expression. Chang and colleagues exploited the low dimensional manifold technique for modeling, tracking, and recognizing facial expressions [6]. They used the Leipschitz embedding to embed aligned facial features to build the manifold. Zeng and colleagues separated the emotional facial expressions from non-emotional ones and treated it as one class classiﬁcation problem [7]. They described that the non-emotional facial expressions could not be described distinctly and were expensive to be modelled. Torre and colleagues used spectral graph techniques for shape and appearance facial features and analyzed them temporally for expressions [8]. They used active appearance models (AAM) for the tracking of the facial features. State of the art techniques for facial expression classiﬁcation are often limited to more or less frontal poses and small skin color variations. Further, often they are sensitive to global motion with resulting perspective distortions. In order to deal with these issues, we propose an automatic real time capable method for the facial expression recognition in which strong independence from the current head pose is achieved. Besides, varying face sizes due to head movements and perspective foreshortening are tackled, like in the case of out of plane rotations.

2

Suggested Method

In this work, geometrical and optical ﬂow based expression recognition is described. Camera and face surface model as in [9,10] is used to estimate the current pose which is used in the feature extraction. Geometric features are measured between facial feature points while optical ﬂow features are computed by analyzing the changes occurring in the predeﬁned face regions. Geometric measurements are fed into support vector machines (SVM) while ﬂow features are classiﬁed by k-nearest neighbor (k-NN) technique (Fig. 1). The method consists of three main tasks, i.e. preprocessing, computation of expression features and recognition of the expression. Following paragraphs cover all these three tasks in a summarized way. 2.1

Preprocessing

In this approach, a well trained Adaboost cascade classiﬁer provided by the OpenCV, is used which detects the candidate faces in the given image [11]. The candidate faces are described by the bounding rectangles around the face. In case of multiple candidates, the candidate with the biggest size is selected for

230

A. Al-Hamadi et al.

Fig. 1. Simpliﬁed structure for the suggested facial expression method

the face. This removes the chances for wrong candidate selection. The preprocessing step contains facial points localization and face pose estimation which are described in the following subsection brieﬂy. Facial Points from Image Processing: Face detection is followed by the extraction of the eye center points ile /ire and mouth corner points ilm/irm as in Fig. 2a. HSV color model [12] is used to extract the facial feature points, which is appropriate in order to exploit the behavior of facial feature areas under diﬀerent color channels. – Eye regions are approximated using the symmetry of the face rectangle resulted from the above face detection step. Eye Blobs are extracted using the saturation channel (S) of the HSV color model where less saturation results the eye pixel which are gathered by k-means clustering algorithm [17]. Area of the eye blobs is calculated and hence the approximate centers of the eyes are computed. – Based on facial symmetry, mouth points are localized in the lower part of the face. The hue channel (H) is exploited to separate the lip pixels from the facial skin pixels. Mouth corner points, i.e. ilm /irm are obtained by traversing the mouth blob for outer most points on left and right sides. Upper and lower lip points are extracted using the information of the outer corner points. Orientation of the mouth corner points is also included in the localization of upper and lower lip points, i.e. iul /ilm respectively (Fig. 2a). – In the proposed approach, area above the eye centers is analyzed to locate the eyebrows. Contrast enhancement and sobel operator is applied in vertical direction in order to get the edges. The eyebrow contour is selected with the largest size and on the highest position. Centroid of the contour is calculated to represent the eyebrow, i.e. ileb /ireb (Fig. 2a) The complete set of image feature points is summarized in deﬁnition 1. If = {ile , ire, ileb , ireb , ilm , irm , iul , ill }

(1)

Face Pose Estimation: It is necessary to determine the current face pose in order to infer real world facial measures. The current face position and orientation is deﬁned through model pose parameter set which contains three translation and three rotation components tv = {x, y, z, ω, φ, κ}. From the current parameter set tv , corresponding pose matrix Mp is derived, simply by multiplying the basis matrices for the current translations and rotations. Anchor points

Geometric and Optical Flow Based Method

231

{are , ale , arm , alm , an } are deﬁned on the face model which help to estimate the current pose as shown in Fig. 2b. According to a deﬁned camera model Cm (in our case a pinhole model) and the current face model pose transformation matrix Mp , the image projection of each model anchor point is determined. Analogously to the pose estimation technique [9,10], the goal is to reduce error measure e (s. Eq. 2) while the image projected anchor points are approximated to the ﬁducial points ij in an iterative least squares approach. Until convergence, all pose parameters are optimized with every iteration of the diﬀerential approach.

Fig. 2. Corresponding image and surface model points for pose estimation. Facial feature points, a) In image coordinates, b) Model anchor points, c) Feature points in world projection, d) Geometric measures, whereby di shows the world distance between facial fature points while Ai denotes the angle information between mouth points.

Using camera model Cm and pose matrix Mp , the image projection of the anchor points ai is carried out. Iterative least squares technique is applied to optimize the error. N e= (ij − t(Mp · aj ))2 → min (2) j=1

where t(.) is the world to image transformation, ij and aj are the corresponding image and anchor points while N is the number of anchor points. After pose determination the image feature points are projected to the surface model, resulting in a set of 3-d points (Eq.3, Fig. 2c) Pf = {ple , pre , pleb , preb , plm , prm , pul , pll } 2.2

(3)

Feature Extraction

In the proposed method, two diﬀerent types of feature vectors are computed. One feature vector contains the direct information of the motion occurred in the diﬀerent face regions, i.e. optical ﬂow feature vector, while the other feature vector represents the change by the combination of geometric measurements, i.e. geometric feature vector. In the following subsections, both feature vectors are described brieﬂy. Geometric feature vector: The geometric feature vector consists of angles and distances between diﬀerent facial feature points in 3-d. As compared to the

232

A. Al-Hamadi et al.

neutral face, facial geometry shows some changes during expression. This information is exploited with the computation of the geometric changes between facial feature points. The combination of geometric changes varies from expression to expression and is hence helpful to recognize the expression. A set of facial feature points is deﬁned, which contains eye centers, eyebrow points, lip corner points, upper and lower lip points. Based on point set Pf (Eq. 3), the ten dimensional feature vector F is determined which represents the measurements made in 3-d. These features comprise six Euclidean 3-d distances between facial feature points while four angles are also computed between four mouth points which are used to describe the current mouth shape (Fig. 2d). – – – – – –

Distance between eye centers and eyebrow points for left and right (d1 , d2 ) Distance between eye and lip corner points for left and right portion (d3 , d4 ) Distance between upper and lower lip point (d5 ) Distance between lip corner points (d6 ) Angle of left corner point with upper and lower lip point (A1 , A2 ) Angle of right corner point with upper and lower lip point (A2 , A4 )

The raising/lowering of both of the eyebrows is gained from distances d1 and d2 . The distances between mouth corners and the eye centers (d3 and d4 ) reveal the mouth movement. The widening and opening of the mouth are represented by d5 and d6 . Further the four angles between mouth points are represented by Aj . Similar to equation 4 we the rest criteria can be deﬁned. (plm − prm ) · (pul − prm ) d1 = preb − pre , · · · A1 = arccos , etc. (4) plm − prm · pul − prm Each feature vector is normalized using the minimum and the maximum value and the input vector is afterward classiﬁed with SVM. Computation of Optical Flow Features: Flow regions are the areas in the face where motion detection is carried out. There are fourteen ﬂow regions which are deﬁned all over the face depending upon the skeleton basis points, i.e. eye points (left, right) and mouth corner points (left, right) (Fig. 3a). The division and positioning of the ﬂow regions are highly motivated by the facial physiogonomy and analysis of underlying muscular active regions during expressions [10]. 3-d boundary points and 3-d internal points where the motion is determined are projected to the 2-d image coordinates system. Lucas-Kanade method is applied to compute the optical ﬂow ﬁeld and thus the changes in the face due to an expression [13,10]. In the proposed method, local neighborhood is used with window size of ﬁve pixels while four levels are selected for the pyramidal hierarchy. This vector gives the direction in which movement is made and also the magnitude or the strength of the vector (Fig. 3d). However, problem may arise due to head motion which may supersede the local changes in the face.To measure the accurate motion Mexp caused by facial expressions, the global head motion Mhead must be suppressed (Fig. 3b). It is expressed by the following relation (Mexp = Mtotal - Mhead ). For this purpose, regional boundary points

Geometric and Optical Flow Based Method

233

Fig. 3. a) Flow region deﬁnition. p0, p1, p2, p3 represents facial ﬁducial points. Flow region rectangles are deﬁned by the ratios between ﬁducial points. b) Global motion negation. Flow vectors (in cyan color) at nose tip represents left ward head motion while empty ﬂow regions depicts eliminated global motion eﬀect c and d) Flow regions in the image. Motion sample points in image are represented by white dots.

are used to approximate the global head motion eﬀect. Boundary points of each ﬂow region in consecutive image frames are compared what provides the local position change in each ﬂow region. The nose tip point movement is analyzed to trigger the boundary points comparison (Fig. 3b). It is the combination of nose tip point motion and the position diﬀerence of ﬂow region boundaries which approximates the negation of the global head motion. The remaining motion information is ready to be used as features. To get a representative ﬂow vector per each ﬂow region, motion vectors are summed up for every ﬂow region and an average is computed. n 1 vavg = vi (5) n i=1 Where vavg is the average optical ﬂow vector and n is the number of vectors in a ﬂow region. This motion vector provides the magnitude and the angle information of the motion for a particular ﬂow region. In order to smooth the motion estimation results, it is necessary to keep the previous history of the motion. The simplest way is to ﬁlter the motion by averaging the motion vectors of n previous frames. The combination of only direction information from all ﬂow vectors makes a speciﬁc signature for an expression. For this purpose, ﬂow feature vectors use only the direction information of the representative ﬂow vectors. vf low = (v1 , v2 , . . . , v14 )T

(6)

Where is the representative ﬂow vector per ﬂow region. The ﬂow vector (Vf low ) represents the input for the afterward classifaction step using k-NN. 2.3

Classification

Classiﬁcation is the last step in the proposed facial expression recognition method, where the current expression is assigned to one of the predeﬁned classes. In the

234

A. Al-Hamadi et al.

proposed method, ﬁve expressions, i.e. happy, surprise, angry and disgust and the neutral state are dealt. SVM and k-NN technique are used for the classiﬁcation, which are described in the following subsections brieﬂy. Support Vector Machines: SVM is used for the evaluation of the geometric feature vector due to robustness and adequacy. Generally, SVM learner is based on an underlying two-class or binary classiﬁcation. The Pairwise Coupling extension is used to adapt SVM for the multi-class problem [14]. In this work, Radial-Basis-Function Gaussian kernel is used which has performed robustly with the given number of features and provided optimum results as compared to other kernels. For the optimization, kernel width and the penalty parameter C are dealt with σ = 3 and C = 5. For more details the reader may refer to [14,16]. To overcome the scaling problem of the SVM classiﬁcation, a feature normalization step is used, which is based on a statistical data analysis. K-Nearest Neighbor: Flow feature vector is classiﬁed using k-NN approach [15, 10]. Generally, k-NN achieves good classiﬁcation results when the training data is well representative and consistent. This technique requires only an accumulation of labelled template samples for training, which are further used during decision. Distance between a test and the training sample can be computed in several ways. Commonly used distance metric is the Euclidean distance. In this work, cosine distance is used for the similarity measure between training and test data. It is expressed as following vtr · vtest θ = arccos (7) |vtr | · |vtest | where vtr is the training feature vector while vtest is the test feature vector.

3

Experimental Results and Discussion

Here, a database has been built for training and test purpose comprising 3500 training samples for the expressions from ten subjects. Training is carried out for the ﬁve expressions including the neutral face. For testing, entirely diﬀerent data sets containing 1000 samples have been used. Test scenarios contain pose variations including out of plane rotation. The database also contains samples of subjects with diﬀerent skin color. Back and forth head movements are also included in training and test samples. Here, the expression can be divided into three states, i.e. onset (when expression starts), apex (expression activation) and oﬀset (relaxation phase). From these three phases, activation of the expression is the phase where complete expression is found, but the dynamics of this phase is shorter than the other phases. So it is needed to keep the state of the expression from its activation till its relaxation. The importance of the geometric feature set is that it contains the state of the expression, which helps to decide the expression when activation is over and expression has not been relaxed yet. Following is an example of the test

Geometric and Optical Flow Based Method

235

Fig. 4. Neutral expression is classiﬁed in beginning followed by the surprise. a) Classiﬁcation using geometric feature with SVM. b) Flow feature classiﬁcation using k-NN.

sequence with expression using ﬂow and geometric features. In beginning, neutral expression is classiﬁed followed by a surprise expression. Figure 4 shows the classiﬁcation using SVM and the resultant class using k-NN classiﬁcation. There is a neutral expression in the earlier part which is followed by another expression class, i.e. disgust. Almost throughout the sequence the resulting probabilities for the recognized expressions are quite high and there is a signiﬁcant diﬀerence between the probabilities of the classiﬁed expression and the rest of the classes. Again diﬀerence between the probabilities of resultant class and rest of the expressions is large. This shows the discriminating power of SVM. The expression classiﬁcation probability that is achieved for the test data using geometrical features in connection with SVM can be analyzed by the confusion matrix in Table 1. These results also include the samples with rotation and back and forth movement scenarios. The confusion matrix shows the high percentage probability of each class, which is presented by diagonal elements. Besides, there is a minimal mixing between the classiﬁcations of diﬀerent classes, which is shown by the non-diagonal elements. The confusion matrix for geometric features also gives an insight that how diﬀerent expressions are similar to each other in terms of the feature vector. Classiﬁcation of facial changes is also carried out with the help of optical ﬂow vector information, i.e. Vf low . Similar to classiﬁcation with geometrical facial based approach, ﬁve expressions have been classiﬁed using the same data set. K-NN classiﬁer is used with majority voting (Table 2). It describes the direction of the motion in a particular facial area. Only the angle information of the representative ﬂow vectors for each ﬂow region is used.

236

A. Al-Hamadi et al.

Table 1. Confusion matrix of classiﬁcation with SVM. Diagonal elements describe the accuracy of the classiﬁcation while non-diagonal elements the misclassiﬁcations. Expression Neutral Happy Suprise Angry Disgust

Neutral 99.3 0.0 0.0 2.8 0.4

Happy 0.0 98.9 0.2 0.3 3.1

Suprise 0.0 0.2 99.5 0.0 0.2

Angry 0.5 0.0 0.0 94.6 2.5

Disgust 0.2 0.9 0.3 2.3 93.7

Table 2. Confusion matrix of k-NN classiﬁcation. Misclassiﬁcation is illustrated by non-diagonal elements while classiﬁcation results are shown by diagonal elements. Expression Neutral Happy Suprise Angry Disgust

Neutral 84.2 3.9 1.2 5.6 1.6

Happy 2.8 75.5 1.8 3.1 14.9

Suprise 0.0 1.2 95.1 1.3 1.1

Angry 10.6 3.2 0.0 72.4 8.8

Disgust 2.4 16.2 1.9 17.6 73.6

On the basis of the diﬀerent facial features, facial expression analysis is carried out by two diﬀerent approaches. The following discussion compares and gives insight to both feature vectors. – The idea behind ﬂow vector computation is to localize the changes in diﬀerent parts of facial skin. The analysis shows that only the angle information of the ﬂow vector is suﬃcient for the expression recognition. The combination of the direction information in each ﬂow vector distinguishes the feature space of the expression classes and hence it is useful to diﬀerentiate between them. Flow vector features are advantageous as they also provide the length of the ﬂow vectors which can be used for the expression quantization. – An important use of the geometric features is that they can keep the state of the expression automatically unlike ﬂow vector features. Hence, they are more useful than ﬂow vector features in a sense that expression can still be classiﬁed if the subject holds the expression and does not change the face appearance. – The performance of our method is high; the processing time is 65ms per frame on a 2 GHz PC and at a camera resolution of 1024x768. We have examined our test data with a publicly available real-time system for expression classiﬁcation provided by Fraunhofer Institute for Integrated Circuits [18]. 3.1

Conclusion and Outlook

An eﬃcient framework for facial expression recognition in HCI systems has been presented. The proposed framework achieves robust feature detection and expression classiﬁcation. Test subjects cover a great variety of skin colors with variable head poses. Using 3-d computer vision techniques, real world measures have been computed for facial features in terms of distances and angles.

Geometric and Optical Flow Based Method

237

Furthermore, SVM classiﬁcation and k-NN classiﬁcation is applied for geometric and feature vector generated by ﬂow analysis respectively. The future work focuses the analysis of diﬀerent facial regions with respect to texture. Context information can be enhanced keeping the history or the current state of the subject in account. Acknowledgments. This work was supported by BMBF (Bernstein-Group) and LSA grants (C4-NIMITEK 2, FKZ: XN3621E/1005M), DFG-Schmerzerkennung (BR3705/1-1) and Sonderforschungsbereich SFB TRR/62.

References 1. Li, S.Z., Jain, A.K.: Handbook of Face Recognition (2005) ISBN: 0-387-40595-X 2. Black, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. Journal of CV 25(1), 23–48 (1997) 3. Valstar, M.F., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Proceedings of IEEE Int. Conf. Computer Vision and Pattern Recognition (2006) 4. Kumano, S., Otsuka, K., Yamato, J., Eisaku, S., Sata, Y.: Pose-Invariant facial expression recognition using variable intensity templates. In: Asian Conf. on Computer Vision (2007) 5. Bartlett, M.S., Littlewort, G., Frank, M.G., Lainscsek, C., Fasel, I., Movellan, J.: Fully automatic facial action recognition in spontaneous behavior. In: Proc. Conf. Automatic Face&Gesture Recognition, pp. 223–230 (2006) 6. Hu, C., Chang, Y., Feris, R., Turk, M.: Manifold based analysis of facial expression. Image and Vision Computing 24, 605–614 (2006) 7. Zeng, Z., Fu, Y., Roisman, G.I., Zhen, W.: Spontaneous emotional facial expression detection. Journal of Multimedia (2006) 8. Torre, F., Campoy, J., Ambadar, Z., Cohn, J.F.: Temporal Segmentation of Facial Behavior. In: International Conference on Computer Vision (October 2007) 9. Niese, R., Al-Hamadi, A., Michaelis, B.: Nearest Neighbor Classiﬁcation for Emotion Recognition in Stereo Image Sequences. ISAST Transactions on Electronics and Signal Processing, No 1(1), 88–94 (2007) 10. Niese, R., Al-Hamadi, A., Panning, A., Michaelis, B.: Real-time Capable Method for Facial Expression Recognition in Color and Stereo Vision. In: Gervasi, O., Gavrilova, M.L. (eds.) ICCSA 2007, Part I. LNCS, vol. 4705, pp. 397–408. Springer, Heidelberg (2007) 11. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (2001) 12. Gonzalez, C.R., Woods, E.R.: Digital Image Processing, 3rd edn. Prentice Hall, Upper Saddle River (2008) 13. Lucas, B., Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision. In: 7th Inter. Joint Conference on Artiﬁcial Intelligence, pp. 674–679 (1981) 14. Cristianini, N., Taylor, J.S.: An Introduction to Support Vector Machines and other kernel based learning methods (2001) ISBN: 0-521-78019-X

238

A. Al-Hamadi et al.

15. Shakhnarovich, G., Darrell, T., Indyk, P.: Nearest-Neighbor Methods in Learning and Vision: Theory and Practice (2006) ISBN:978-0-262-19547-8 16. Lin, C.-J., Weng, C.R.: Simple Probabilistic Predictions for Support Vector Regression. Technical report, Department of Computer Science, National Taiwan University (2004) 17. Byrd, R., Balaji, B.: Real time 2D face detection using color rations and k-mean clustering. In: Proc. of the 44th Southeast regional conference, Florida, pp. 644–648 (2006) 18. Fraunhofer-Institut fuer Integrierte Schaltungen IIS, Schaltungen IIS, Erlangen, Germany, Biometrics Demo, http://www.iis.fraunhofer.de/EN/bf/bv/kognitiv/biom/dd.jsp

Local Rank Patterns – Novel Features for Rapid Object Detection Michal Hradis, Adam Herout, and Pavel Zemcik Graph@FIT, Brno University of Technology, Bozetechova 2, Brno, CZ {ihradis,herout,zemcik}@fit.vutbr.cz

Abstract. This paper presents Local Rank Patterns (LRP) - novel features for rapid object detection in images which are based on existing features Local Rank Diﬀerences (LRD). The performance of the novel features is thoroughly tested on frontal face detection task and it is compared to the performance of the LRD and the traditionally used Haar-like features. The results show that the LRP surpass the LRD and the Haarlike features in the precision of detection and also in the average number of features needed for classiﬁcation. Considering recent successful and eﬃcient implementations of LRD on CPU, GPU and FPGA, the results suggest that LRP are good choice for object detection and that they could replace the Haar-like features in some applications in the future. Keywords: AdaBoost, WaldBoost, object detection, face detection, LRD.

1

Introduction

Detection of objects in images has many real-world applications ranging from camera orientation to computer-human interaction, surveillance and robot vision. One of the approaches to real-time object detection is scanning images for speciﬁc 2D patterns using fast detection classiﬁers. Such classiﬁers are in many cases modiﬁcations of the classiﬁer originally proposed by Viola and Jones [1], who designed the ﬁrst real-time frontal face detector using a cascade of boosted classiﬁers and Haar like features. The cascade structure of the classiﬁer together with rapid computation and relatively high discriminative power of the Haar like features resulted in very low average decision time which is essential in the detection task. Aside of the frontal faces, the detection classiﬁers have been used to detect various objects such as occluded faces [2], pedestrians [3], general interest points [4], automobiles, stop signs, tea cups [5] etc. Recently, Sochman and Matas [6] have proposed a uniﬁed approach how to emulate behavior of any existing application-speciﬁc detector by sequential classiﬁer which is optimal in terms of computational complexity for desired detection precision. In their approach, the main eﬀort is put into ﬁnding a suitable set of features which are then automatically combined into a WaldBoost [9] ensemble. The performance of the detection classiﬁers largely depends on the type of features they use. The ideal features should be computationally inexpensive, should L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 239–248, 2009. c Springer-Verlag Berlin Heidelberg 2009

240

M. Hradis, A. Herout, and P. Zemcik

be, to some degree, invariant to geometry and illumination changes and should provide high discriminative power - all at the same time. High discriminative power is needed to achieve high precision of detection and it also implies more compact and faster classiﬁers as lower number of features is needed to be computed for the classiﬁer to make a decision. In general, the ideal type of features can diﬀer for diﬀerent types of objects [6]. However, simple image ﬁlters have been proven to generalize well across various types of objects [4] [5]. We propose a novel feature set which is suitable for real-time detection of 2D patterns in images. These features, the Local Rank Patterns (LRP), are invariant to monotonous lighting changes, reﬂect the amplitude of local changes and also provide information about local patterns. The design of the LRP is based on the recently proposed Local Rank Diﬀerences (LRD) [10]. As shown below, the discriminative power of the LRD is unnecessarily constrained and the properties of the resulting classiﬁers can be signiﬁcantly improved by removing these constraints. The LRP present one of the possibilities how to do this. The LRP share most of the computational steps necessary to evaluate the LRD; that suggests that the recent real-time implementations of engines evaluating classiﬁers with LRD on CPU, GPU [8] and FPGA [7] can be easily modiﬁed to also accommodate classiﬁers with the LRP. The paper is structured as follows. In Section 2, general deﬁnition of the LRP is given and some of the properties of LRP are discussed. Section 3 presents the particular form of LRP which was used in experiments on frontal face detection task which are described in Section 4. The paper is concluded and hints for future work are given in Section 5.

2

Local Rank Patterns

Let us consider a scalar image f : Z2 → R. On such image, a sampling function can be deﬁned (x, u ∈ Z2 , g : Z2 → R) Sxg (u) = (f ∗ g) (x + u) .

(1)

This sampling function is parameterized by the convolution kernel g, which is applied before the actual sampling, and by the vector x which is the origin of the sampling. Next, let us introduce a vector of relative coordinates (n ∈ N) U = [u1 u2 . . . un ] , ui ∈ Z2 .

(2)

This vector of two-dimensional coordinates can deﬁne an arbitrarily shaped neighborhood and it will be used together with the sampling function to obtain a vector of values describing a neighborhood of this shape on position x in the image M = [Sxg (u1 ) Sxg (u2 ) . . . Sxg (un )] . (3) This set of values will be referred to as the mask in the following text. The term mask is reasonable as the vector was created by ”masking” global information

Local Rank Patterns – Novel Features for Rapid Object Detection

241

from the image and leaving only speciﬁc local information. For each element k in the mask, its rank can be deﬁned as n 1, if Mk < Mi Rk = , (4) 0, otherwise i=1

id est, the rank is the order of the given member of the mask in the sorted progression of all the mask members. Note that this value is independent on the local energy in the image, which is an important property useful for the behavior of the Local Rank Patterns image feature, which is deﬁned as (a, b ∈ {1, . . . , n}) LRP (a, b) = Ra · n + Rb .

(5)

Note that n is the number of samples taken in the neighborhood and therefore the result of LRP is unique for each combination of values of the two ranks Ra and Rb . This fact suggests an alternative deﬁnition of the LRP when we allow the results of LRP to be pairs of values instead of single value LRP (a, b) = [Ra Rb ] .

(6)

This implies that the LRP have higher descriptive power than the LRD which compute the diﬀerence of the two ranks [8]. The LRP have some interesting properties which make them promising for image pattern recognition. Mainly, the LRP are invariant to monotonous grayscale changes such as changes of illumination intensity. This invariance results from using ranks instead of absolute values to compute the value of the feature. In fact, using the ranks has the same eﬀect as locally equalizing histogram of the convolved image f ∗ g. Further, the LRP are strictly local - their results are not inﬂuenced by image values outside the neighborhood deﬁned by U. This is a clear advantage over wavelet features (e.g. Haar-like features) which, in the way they are commonly used, need global information to normalize their results. This locality makes the LRP highly independent for example on changes of background and on changes of intensity of directional light. The meaning of the values produced by the LRP can be understood in two ways. First and most naturally, the results give information about the image at the locations of the two ranks x + ua and x + ub and information about their mutual relation. On the other hand, the results also carry information about the rest of the neighborhood especially if the neighborhood is small. In such cases the results of LRP carry good information about local pattern in the image. In the previous text, the LRP have been deﬁned for two-dimensional images. However, the notation allows very simple generalization for higher-dimensional images by changing dimensions of x, u and of the relative coordinates in U to Z3 for 3D or Zk for general dimensionality. Further, it is possible to use more than two ranks to compute the results of the LRP. For example: LRP (a, b, c) = Ra · n2 + Rb · n + Rc .

(7)

242

3

M. Hradis, A. Herout, and P. Zemcik

Set of LRP Suitable for Object Detection

The deﬁnition of the LRP which has been given in the previous text is very general. It allows arbitrary sizes and shapes of the neighborhoods and arbitrary convolution kernels. In this section, we will deﬁne a set of LRP which is suitable for creating classiﬁers for detecting objects in images and which is used in the reported experiments. First, let us deﬁne a suitable set of neighborhoods. The number of samples taken in the neighborhood should be high enough for the features to have good discriminative power; however, the number of samples must not be unreasonably high, because computational complexity of the LRP grows linearly with the number of samples. A good ﬁrst choice is local rectangular subsampling which takes nine samples arranged in a regular 3 × 3 grid. Such neighborhoods can be deﬁned by a base neighborhood Ubase (see Figure 1) which is than scaled to generate all the required sizes.

Ubase = [0 0] [1 0] [2 0] [0 1] [1 1] [2 1] [0 2] [1 2] [2 2] Fig. 1. The base 3 × 3 neighborhood which is used to generate all other neighborhoods used in this work. The ﬁgure shows the relative coordinates of samples together with the locations in the image and also the vector Ubase which deﬁnes the neighborhood.

In the further text, Umn will refer to a neighborhood which is created from U by scaling the x-coordinates by m and scaling the y-coordinates by n: m0 mn base Ui = Ui · . (8) 0 n base

The type and the purpose of the convolution kernel g in the LRP can diﬀer. It could be a derivation ﬁlter or a wavelet, but the most basic purpose of g is to avoid aliasing when scaling the neighborhood. Aliasing could be avoided by using any low-pass ﬁlter such as the Gaussian ﬁlter. For eﬃciency reasons, we use rectangular averaging convolution kernels (see Figure 2). In the following text, g kl will stand for a rectangular averaging ﬁlter of dimensions k by l: kl

g (x) =

1, if 0 ≤ x1 < k and 0 ≤ x2 < l 0, otherwise

(9)

When scaling the neighborhood, it is reasonable to keep the size of the averaging ﬁlter the same as the distance between the samples. In such case m in Umn

Local Rank Patterns – Novel Features for Rapid Object Detection

g

22

11 = , 11

g

42

1111 = , 1111

243

⎡

g

23

⎤ 11 = ⎣1 1⎦, 11

Fig. 2. Examples of some of the convolution kernels used in the experiments

equals to k in g kl and n equals to l. In [8], the authors use a set LRD with only four neighborhoods U11 , U12 , U21 and U22 for samples of dimension 24 × 24 pixels. They report that the precision of detection of a classiﬁer using this set of features rivals the performance of a classiﬁer with Haar-like features. However, this small subset of all possible LRD is used only due to the constraints of the presented implementation on GPU and it is probable that classiﬁers using larger set of LRD would achieve better results in most cases. In this work, the set of LRP (and also LRD) is not restricted in this way and all neighborhoods which completely ﬁt into the samples are used.

4

Experiments

To evaluate the discriminative power of the LRP and the suitability of the LRP for object detection, a series of experiments was concluded. In these experiments, the LRP were compared to LRD and the traditional Haar-like features. Frontal face detection was selected as the model task, as it is the task most frequently used to report performance of novel detection algorithms and also standardized test sets are available. 4.1

Training Algorithm

The WaldBoost [9] algorithm was used to train all classiﬁers in the experiments. This algorithm is similar to the algorithm used by Viola and Jones [1] in the way it uses features to create weak hypotheses and in the way it reduces the false positive rate. However, instead of cascade structure of classiﬁers, it creates a single sequential classiﬁer with early termination thresholds. The early termination thresholds allow early classiﬁcation decision about the ”easy” samples. For detection classiﬁers, the algorithm sets the thresholds to create the fastest possible classiﬁer which still satisﬁes the desired false negative rate. This desired false negative rate is a parameter of the algorithm and will be referred to as α in the following text. In the experiments, weak classiﬁers were selected on 1500 unique positive and 1500 unique negative samples which were selected in each iteration of the WaldBoost algorithm by importance sampling from a training set which consisted of 100k positive and 100k negative samples. The selected weak hypotheses were then trained on the whole training set. The early termination thresholds were estimated on a validation set of the same size as the training set.

244

4.2

M. Hradis, A. Herout, and P. Zemcik

Datasets

The training set and the validation set were randomly bootstrapped from the same data, while guaranteeing that the two sets remain disjunct. The set of faces was generated by applying small random aﬃne transformations to 5000 handannotated faces. The non-face samples were randomly sampled from a pool of 250 million sub-windows from more than 3000 non-face images downloaded from the Internet. All samples were normalized to 24 × 24 pixels. The created classiﬁers were tested by scanning images from three separate test sets. The ﬁrst test set is the MIT-CMU frontal face database1 which is a standard benchmark for frontal face detection. The other two test sets contain photos of groups of people. The ﬁrst of them, the GROUP1 test set was gathered by a person who is experienced in face detection. The second test set GROUP2 was gathered by a person who has no knowledge about this ﬁeld. For more information about the test sets, see Table 1. The images were scanned with position displacement of 2/24 of the scanning window size and with scale factor 1.2. Non-maxima suppression algorithm proposed in [6] was used to post-process the raw detections. Table 1. The test sets used in the experiments. The table shows the number of images, the number of faces and the number of scanned position in each of the sets. DataSet # Images # Faces # Positions MIT+CMU 134 511 19M GROUP1 89 1618 142M GROUP2 109 2055 96M

4.3

The Classifiers

The tests included 5 diﬀerent classiﬁers whose results are reported in the following section. They diﬀer in the type of features they use and in the type of weak learners. See Table 2 for the complete list of the classiﬁers. The targeted false positive rate which is speciﬁed by the α parameter of the WaldBoost algorithm is 5 % for all of the classiﬁers. The ﬁnal length of the classiﬁers is 1000 weak hypotheses. The classiﬁer CHaar uses the same set of Haar-like features as in [1]. The responses of the Haar-like features are normalized by the standard deviation of pixel intensities in the sample. The weak hypotheses have 10 equidistant bins such that the 8 middle bins contain 90 % of the training samples. The classiﬁer CLRP uses the set of features deﬁned in Section 3 and the weak hypotheses have 80 bins which correspond to the individual results of the LRP. The classiﬁer CLRD uses a set of LRD features which is deﬁned by the same convolution kernels and the same neighborhoods as are deﬁned in Section 3. The weak hypotheses of CLRD have 17 bins which correspond to the individual results of the LRD features. 1

http://vasc.ri.cmu.edu/idb/html/face/frontal images/

Local Rank Patterns – Novel Features for Rapid Object Detection

245

As the number of the bins of the three classiﬁers diﬀers and it could aﬀect the results of the tests, two more classiﬁers are introduced for better comparison of the individual types of features. The classiﬁer CLRP24 uses the same features as CLRP, except that the ranks Rk (see Eq. 4) are divided by 2. The eﬀect of this change is that the weak hypotheses now have only 24 individual bins. On the other hand, the number of bins for classiﬁer CHaar80 was increased to 80 for better comparison with CLRP. Table 2. The classiﬁers which were used in the experiments Classiﬁer Features # of Bins CHaar Haar 10 CLRD LRD 17 CLRP LRP 80 CLRP24 LRP 24 CHaar80 Haar 80

4.4

Results

In each of the experiments, an average result of twelve classiﬁers is reported. The twelve classiﬁers were created with the same setting of the training algorithm, but with diﬀerent training set and validation set. The Receiver Operating Characteristics (ROC) which characterize precision of detection of the classiﬁers can be seen in Figure 3 and in Figure 4. It can be seen that the CLRP classiﬁer surpasses all the other classiﬁers. CLRP is deﬁnitely an improvement over the CLRD classiﬁer and it also reaches better results than the CHaar classiﬁer especially in the left part of the ROC. Figure 4 suggests that the better results of the CLRP classiﬁer are not caused by the higher number of bins of the weak hypotheses. For the CHaar80 classiﬁer, the higher number of bins did not improve performance of the Haar-like features. Also the results of CLRP24 do not show any signiﬁcant performance drop of the LRP when the number of bins is reduced. The average speeds of the classiﬁers can be seen in Table 3. The fastest is the CLRP classiﬁer. Table 3. The average speeds of the tested classiﬁers. The reported numbers are average number of weak hypotheses needed for decision. Classiﬁer # of weak hypotheses per window CLRP 4.8 CHaar 5.4 CLRD 6.0 CLRP24 5.0 CHaar80 6.5

246

M. Hradis, A. Herout, and P. Zemcik

Fig. 3. The Receiver Operating Characteristics of the classiﬁers CLRP, CLRD and CHaar on the three test sets. top - MIT+CMU; middle - GROUP1; bottom - GROUP2

Local Rank Patterns – Novel Features for Rapid Object Detection

247

Fig. 4. The Receiver Operating Characteristics of the classiﬁers CLRP, CLRP24 and CHaar80 on the three test sets. top - MIT+CMU; middle - GROUP1; bottom GROUP2

248

5

M. Hradis, A. Herout, and P. Zemcik

Conclusions and Future Work

Novel features the Local Rank Patterns were introduced and the performance of these features was tested on the frontal face detection task. The results show that classiﬁers with the LRP achieve better precision of detection than classiﬁers with the Haar-like features and classiﬁers with the recently proposed LRD. The speed of the classiﬁers is also improved with the LRP. These results are promising and clear path to fast and precise detection classiﬁers on platform such as GPU and FPGA. However, question still remains how well do the results generalize to other types of objects and other learning algorithms.

Acknowledgements This work has been supported by the Ministry of Education, Youth and Sports of the Czech Republic under the research program LC-06008 (Center for Computer Graphics), by the research project ”Security-Oriented Research in Informational Technology” CEZMSMT, MSM0021630528 and by the project “BIOMARKER”, 2B06052.

References 1. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001) 2. Chen, J., et al.: Modiﬁcation of the AdaBoost-based Detector for Partially Occluded Faces. In: ICPR 2006: Proceedings of the 18th International Conference on Pattern Recognition (2006) 3. Cui, X., et al.: 3D Haar-Like Features for Pedestrian Detection. In: Proceedings of IEEE International Conference on Multimedia and Expo (2007) 4. Matas, J., Sochman, J.: Wald’s Sequential Analysis for Time-constrained Vision Problems. Lecture Notes Electrical Engineering. Springer, US (2008) 5. Schneiderman, H.: Feature-Centric Evaluation for Eﬃcient Cascaded Object Detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Los Alamitos (2004) 6. Sochman, J., Matas, J.: Learning A Fast Emulator of a Binary Decision Process. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007, Part II. LNCS, vol. 4844, pp. 236–245. Springer, Heidelberg (2007) 7. Zemcik, P., Zadnik, M.: AdaBoost Engine. In: Proceedings of FPL 2007 (2007) 8. Polok, L., Herout, A., Zemcik, P., Hradis, M., Juranek, R., Josth, R.: ”Local Rank Diﬀerences” Image Feature Implemented on GPU. In: Blanc-Talon, J., Bourennane, S., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2008. LNCS, vol. 5259, Springer, Heidelberg (2008) (accepted for publication) 9. Sochman, J., Matas, J.: WaldBoost — Learning for Time Constrained Sequential Detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2 (2005) 10. Zemcik, P., Hradis, M., Herout, A.: Local Rank Diﬀerences - Novel Features for Image Processing. In: Proceedings of SCCG 2007, Budmerice, SK, pp. 1–12 (2007)

Detection of Dogs in Video Using Statistical Classifiers Roman Jur´anek Graph@FIT Brno University of Technology, Faculty of Information Technology Boˇzetˇechova 2, 612 66 Brno, Czech Republic [email protected]

Abstract. A common approach to pattern recognition and object detection is to use a statistical classiﬁer. Widely used method is AdaBoost or its modiﬁcations which yields outstanding results in certain tasks like face detection. The aim of this work was to build real-time system for detection of dogs for surveillance purposes. The author of this paper thus explored the possibility that the AdaBoost based classiﬁers could be used for this task. Keywords: Dog detection, WaldBoost, haar-like features.

1

Introduction

Statistical classiﬁers can very well be used for object detection or pattern recognition in raster images. Current algorithms even exhibit real-time performance in detecting complex patterns, such as human faces [1], while achieving precision of ˇ detection which is suﬃcient for practical applications. Recent work of Sochman and Matas [2] even suggests that any existing detector can be eﬃciently emulated by a sequential classiﬁer which is optimal in terms of computational complexity for desired detection precision. Tn their approach the eﬀort is invested into selecting a set of suitable features which are then automatically combined into an ensemble by the WaldBoost [3] algorithm. In this paper, dog detection for surveillance purposes is discussed. The aim of the work was to build simple system that will cope with low resolution and possibly low quality (noise, compression artifacts, etc.) input video. The general object detection approach with WaldBoost based classiﬁers with Haar-like features was used. The main problem to deal with in this work was the training data. Since there is no freely available database of suitable dog samples, the data had to be collected form the internet. The following section summarizes algorithms commonly used for general object detection. Section 3 discusses speciﬁc issues related to detection of dogs. Experiments carried out with the detection of dogs on surveillance recordings and their results are described in section 4. The section also describes tools that were used for training and evaluation of the classiﬁers. The last section summarizes the paper and discuses possibilities of the future work. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 249–259, 2009. c Springer-Verlag Berlin Heidelberg 2009

250

2

R. Jur´ anek

Background

Object detection with classiﬁers was ﬁrst used by Viola and Jones [1] for rapid face detection. In their work they used AdaBoost algorithm to select a set of critical features out of large set of Haar-like wavelets. The basic algorithm of AdaBoost was described by Freund and Schapire in [4]. The response of weak hypothesis is restricted to binary value and thus the algorithm is referred to as discrete AdaBoost. Schapire and Singer [5] introduced real AdaBoost which allows conﬁdence rated predictions and is most commonly used in combination with domain partitioning weak hypotheses (e.g. decision trees). Viola and Jones [1] used the AdaBoost classiﬁers to detect faces in images. In their work they used simple weak hypotheses consisting from a single Haar-like feature, and classiﬁer cascade. AdaBoost in its basic form, greedily selects weak hypotheses that are only moderately accurate to create very accurate classiﬁer. The result of such classiﬁer is based on linear combination of the selected weak hypotheses (Equation 1). fT (X) =

T

αt ht (X)

(1)

t=0

The weak hypotheses selected by AdaBoost are not optimal as the process is greedy. There have been works addressing this fact, e.g. FloatBoost [6] or Total Corrective Step [7]. 2.1

WaldBoost

The main drawback of AdaBoost classiﬁers is that each weak hypothesis within the classiﬁer must be evaluated to obtain response. To reduce the number of evaluated hypotheses a classiﬁer cascade [1] can be used. Another approach was ˇ introduced by Matas and Sochman [3] – in their WaldBoost they keep the linear structure of the classiﬁer, and for each weak hypothesis selected by AdaBoost they calculate two early-termination thresholds θA and θB using Wald’s Sequential Probability Ratio Test [8]. During classiﬁer evaluation, when a strong classiﬁer sum ft (X) exceeds a threshold evaluation ends, next stage is evaluated otherwise. The Equation 2 shows evaluation of t − th stage of a classiﬁer. ⎧ (t) ⎪ +1, ft (X) ≥ θA ⎪ ⎨ (t) Ht (X) = (2) −1, ft (X) ≤ θB ⎪ ⎪ ⎩ (t) (t) continue Ht+1 , θA < ft (X) < θB During the training, the thresholds for each stage are estimated according to parameters α and β, which represent false negative rate and false positive rate of the ﬁnal classiﬁer. In detection tasks, β is usually set to 0. As the result, positive threshold θA will be set to ∞ for each selected stage and thus a sample cannot be accepted by early termination mechanism (strong classiﬁer sum cannot possibly

Detection of Dogs in Video Using Statistical Classiﬁers

251

reach ∞). Early termination can thus only reject samples as negative. Positive detection can be reached only by evaluating the entire ensemble. The speed of the WaldBoost classiﬁers is indicated by the average number of evaluated hypotheses per sample. The speed largely depends on the classiﬁer application, training settings (namely alpha and beta parameters) and also on the number of weak hypotheses in the classiﬁer. 2.2

Features

The Properties of a classiﬁer largely depend on the low level weak hypotheses and features. In many computer vision problems, like the face detection, are the Haar-like features, commonly used since in combination with integral image they exhibit extreme performance and provide good amount of information. Other features commonly used are Gabor wavelets [9], Local Binary Patterns [10] or Local Rank Diﬀerences [11].

Fig. 1. Typical shapes of Haar-like features

Haar-like features (on the Figure 1) are based on the diﬀerence of adjacent rectangular regions of an image. Due the integral image representation, the response of a Haar-like feature can be obtained in constant time regardless of its size. The main drawback of the features is that they are dependent on light conditions and their response must be normalized

3

Detection of Dogs in Video

Compared to other tasks like face detection, the dog detection is more diﬃcult since the silhouette of dogs changes over time as the dog moves. Variety of dog shapes is very large (diﬀerent postures, orientations, etc). The texture and brightness of dogs also varies in wide range. The detection of dogs simply lacks a visually well deﬁned class. Single WaldBoost classiﬁer is, therefore, not able to detect dogs viewed from arbitrary angle. For the above reason the task in this paper is limited to the detection of dogs viewed from the proﬁle only. While this limitation may seem relatively severe, it does not introduce any serious limitation from the application point of view, as the objects in the video sequence can be tracked and whenever the classiﬁer detects a dog seen from the proﬁle, the whole track is known to represent the dog. Another issue connected with dog detection is that the silhouette is not horizontally symmetric. Unlike the face detection, the input image must be searched at least two times – ﬁrst time original image and second time image horizontally ﬂipped (or with horizontally ﬂipped classiﬁer). This ensures that all dogs could be detected regardless of their orientation.

252

R. Jur´ anek

Fig. 2. Detection of dog in a simple outdoor scene

3.1

Training Data

Since there is no freely available database with suitable dog samples, training data for our experiments – images of dogs viewed from proﬁle, are collected from the Internet and also from our own video recordings with dogs.

Fig. 3. Excerpts from training datasets

A drawback of the data is that conditions in images vary in wide range (different background, lighting, compression artifacts, etc.) which would probably result in worse classiﬁer performance. To bring best performance, the data should correspond to the conditions in the target application which in our case was dog detection in underground stations. It is clear that the data, which were taken mostly from outdoor scenes, do not correspond to the conditions in underground stations very well. From the annotated data, samples for the training framework (see section 4.1) were generated. The size of the samples was set similarly to size commonly used in face detection to 24 × 16px. In smaller samples, there would be hard to catch the silhouette of a dog. Larger samples, on the other hand, would increase number of weak hypotheses in the training which would slow down the training process rapidly. Number of dog samples in the current dataset is 618 and they are divided to training set (236 samples), validation set (190 samples) and testing set (192 samples). In the Figure 3 you can see samples form dog class (left) and background class (right) from the dataset. As the background class holds potentially inﬁnite number of images, the samples can be randomly extracted from diﬀerent images

Detection of Dogs in Video Using Statistical Classiﬁers

253

during training. The number of background samples during the training can thus, in combination with bootstrapping in WaldBoost algorithm, reach millions.

4

Experiments and Results

The experiments included training of WaldBoost classiﬁers using data described in section 3.1. The experiments also involved testing of diﬀerent false negative rate (training parameter α) and observe how this settings inﬂuence the properties of the resulting classiﬁer. The following two sections describe the classiﬁer training framework and the real-time object detection engine. In the section 4.3 are actual results of performed experiments. 4.1

WaldBoost Training

The Boosting Framework [12] was used for classiﬁer training. Overview of the training process is shown on the Figure 4. The input for WaldBoost training are conﬁguration (algorithm settings and used image features and weak hypotheses) and training data. The classiﬁer (selected hypotheses) and its evaluation are generated as an output of the process.

Fig. 4. Structure of the training framework

The framework supports many diﬀerent types of image features and weak hypotheses as well as diﬀerent training algorithms. For this paper the important is support of Haar-like features and WaldBoost training algorithm. Beside other parameters for the WaldBoost training, the conﬁguration speciﬁes the parameter α (see section 2.1) which has the greatest inﬂuence on the properties of classiﬁers. The output of an experiment is a classiﬁer and its evaluation on the training and testing sample set for each selected stage. Beside others, Receiver Operating Characteristic (ROC), Precision-Recall (PRC) and Negative Rejection Rate curves are generated. Example of stage in the XML The resulting classiﬁer is represented by a XML structure which contains a sequence of stages (Figure 5). Each stage has its early-termination thresholds and a single selected weak hypothesis. (posT corresponds to the θA and negT to the θB ). (t) ht (X) = Wk (3)

254

R. Jur´ anek

<Stage posT="1E+10" negT="-1.3"> Fig. 5. Example of a stage representation in XML

The stage holds a domain partitioning weak hypothesis with an Haar-like image feature. Evaluation of the stage is explained by the Equation 3, where N (g(X) − min) k= max − min is used as index to a table weights W (t) (responses in the XML code). The g(X) is a response of the feature on sample X. The values N , min and max correspond to parameters of Discretize element in the XML code.

Fig. 6. Example of weak hypothesis with min = −2, max = 2 and N = 8 (corresponding to the XML code in the Figure 5)

In the Figure 6 is illustrated how the weak hypothesis works. The quantized response of the feature is used as an index to the table of weights and thus to each interval of the feature response a weight value participating on the strong classiﬁer sum is assigned. 4.2

Real-Time Object Detection

One of goals of this project was to build a real-time system for detection of dogs. For this purpose, detection engine that uses classiﬁers generated by the Boosting

Detection of Dogs in Video Using Statistical Classiﬁers

255

Framework was developed. The main properties of the engine is eﬃcient Haarlike feature evaluation and multiscale object detection. The actual object detection is achieved by scanning the image. When the detection is executed, ﬁrst, input image is preprocessed and then scanned with a sliding window (ﬁgure 7), evaluating classiﬁer on each position. Encountered detections are passed to a non-maxima suppression algorithm [2] which removes possible multiple detections of same object. The image scanning technique can inherently detect multiple objects in an image. The preprocessing stage involves calculation of standard integral image and integral image of squared values. The only reason to generate the other integral image is rapid calculation of standard deviation of classiﬁed samples, which is needed for Haar-like feature response normalization.

Fig. 7. Image processing in the detection engine

Detection of diﬀerently sized objects is achieved simply by resizing the classiﬁer – position and size of each feature within the classiﬁer is adjusted for particular scan window size. The image is scanned with the classiﬁer afterwards. 4.3

Results

For the experiments, classiﬁers were trained using the framework described in the section 4.1. The training parameter α was set in range from 0.01 (few false negatives) to 0.2 (more false negatives). The plot on the ﬁgure 8 shows Receiver Operating Characteristics (ROC) of the classiﬁers and on the ﬁgure 9 there are speed characteristics of the classiﬁers. Another interesting property of WaldBoost classiﬁers is a Negative Rejection Rate (Figure 10). The curves show the ability of classiﬁers to reject negative samples by early-termination mechanism. When the classiﬁer can reject vast majority of samples before ﬁnal stage, only very few samples are needed to be classiﬁed by thresholding the ﬁnal classiﬁer response. The α = 0.1 classiﬁer can reject 99 % of negative samples before 10-th stage, whereas the α = 0.01 classiﬁer needs 100 stages to do same job. The experiments shows that classiﬁers with the α set to higher value are faster and as ROC curves shows, faster classiﬁers are less accurate because of rapid rejection of samples.

256

R. Jur´ anek

Fig. 8. ROC curves of the classiﬁers

Fig. 9. Speed of the classiﬁers – average number of evaluated weak hypotheses on test set. The classiﬁers with higher α are faster due to more rapid rejection of negative samples. Note that the speed increases logarithmically with number of stages.

The classiﬁers were tested in the real-time detection engine with a recordings of an underground station. The detection was executed on 352×288px image and the engine achieved performance of 50f ps (Intel Core2 Duo, 2.66 GHz) which was more than suﬃcient (typical frame rate in a surveillance system is 5f ps). The detection speed of course depends on many parameters, mainly on accuracy of used classiﬁer (more accurate classiﬁers takes longer time to evaluate). On the Figure 11, there are examples of detections in an underground station. The images (d) and (e) on the ﬁgure shows false detections. The false positive rate was higher due the fact that the detector is static and does not use any motion information.

Detection of Dogs in Video Using Statistical Classiﬁers

257

Fig. 10. Negative Rejection Rate of the classiﬁers. Ratio of negative (background) samples not rejected by the early-termination. More accurate classiﬁers (lower α) rejects samples slowly.

(a)

(b)

(d)

(c)

(e)

Fig. 11. Detections in the Roma underground station (classiﬁer with α = 0.05 was used). The images (d) and (e) shows false detection.

5

Conclusion and Future Work

In this paper, real-time detection of dogs in low resolution videos was studied. The used method included statistical classiﬁer training by the WaldBoost algorithm (section 2) and building of real-time detection engine – eﬃcient evaluation of WladBoost classiﬁers on image (section 4.2). In the section 3, speciﬁc

258

R. Jur´ anek

issues connected with detection of dogs were summarized and also data that was used in our experiments was described. The method of general object detection without any extra knowledge related to dogs seems promising. The experiments (see section 4.3) shows that the method is viable despite the fact that the shape of detected objects rapidly changes over time. Even on existing dataset of images that were mostly collected from the Internet, the classiﬁers exhibit good characteristics. In scenes with simple background the classiﬁers were able to detect dogs with high accuracy. In complex scenes (lots of moving people in the station) there were more false positive detections. This was caused by the static character of the detector and also by rather small positive training set. For the future, promising appears utilization of image features extracted from more than one frame (e.g. approach similar to [13]) instead of simple Haar-like features. Detector could be thus trained to that take advantage of both appearance and motion information to detect a dog. This approach would of course bring problems with training data because current dataset does not contain any motion information.

Acknowledgements ˇ The author would like to thank to Ivo Rezn´ ıˇcek for his help with collecting the data, and also to Pavel Zemˇc´ık and Adam Herout for their guidance. This work was supported by by European project CareTaker (FP6-027231).

References 1. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features, pp. 511–518 (2001) ˇ 2. Sochman, J., Matas, J.: Learning a fast emulator of a binary decision process. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007, Part II. LNCS, vol. 4844, pp. 236–245. Springer, Heidelberg (2007) 3. Sochman, J., Matas, J.: Waldboost learning for time constrained sequential detection. In: CVPR 2005: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), Washington, DC, USA, 2005, vol. 2, pp. 150–156. IEEE Computer Society Press, Los Alamitos (2005) 4. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Vit´ anyi, P.M.B. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995) 5. Schapire, R.E., Singer, Y.: Machine learning. Improved boosting algorithms using conﬁdence-rated predictions 37(3), 297–336 (1999) 6. Li, S., Zhang, Z., Shum, H., Zhang, H.: Floatboost learning for classiﬁcation (2002) 7. Sochman, J., Matas, J.: Adaboost with totally corrective updates for fast face detection. In: FGR, pp. 445–450 (2004) 8. Wald, A.: Sequential Analysis. John Wiley and Sons, Inc., Chichester (1947)

Detection of Dogs in Video Using Statistical Classiﬁers

259

9. Lee, T.S.: Image representation using 2d gabor wavelets. IEEE Trans. Pattern Anal. Mach. Intell. 18(10), 959–971 (1996) 10. Ojala, T., Pietik¨ ainen, M., M¨ aenp¨ aa ¨, T.: Multiresolution gray-scale and rotation invariant texture classiﬁcation with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002) 11. Zemˇck, P., Hradiˇs, M., Herout, A.: Local rank diﬀerences - novel features for image. In: Proceedings of SCCG 2007, pp. 1–12 (2007). 12. Hradiˇs, M.: Framework for research on detection classiﬁers. In: Proceedings of Spring Conference on Computer Graphics, pp. 171–177 (2008) 13. Jones, M., Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. In: ICCV, pp. 734–741 (2003)

Automatic Video Editing for Multimodal Meetings Radek Kubicek, Pavel Zak, Pavel Zemcik, and Adam Herout Graph@FIT, Brno University of Technology, Bozetechova 2, Brno, CZ {ikubicek,izakpa,zemcik,herout}@fit.vutbr.cz

Abstract. Meeting recording is being performed through microphones and video cameras in order to keep a permanent record of the events that are happening during the meetings. The technology to perform such recording is already mature and recording is being performed already for some time. However, eﬃcient retrieval of the information from meeting data remains a hot topic of contemporary research. Several approaches to information retrieval exist, such as indexing the data, event semantics analysis of the data, etc. This contribution focuses on automatic video editing of the data in order to prepare audiovisual material, based on several audio and video sources, that is suitable for human users to see. The video editing takes the original audio and video data as its input as well as the results of analysis of audio and video streams and user instructions. The output of the method is a simple audiovisual stream.

1

Introduction

The technology for recording audio-visual data of meetings (such as small informal technical meetings, meetings of administration boards, conference meetings, etc.) is available and is being used in practice. However, the task of information retrieval from such collected data is an open ﬁeld for research. A large European project Multi-Modal Meeting Manager [6] (MMMM, M4) has been targeting collection and processing of meeting data, both the audio and video components. The project’s goals relevant for this paper included collection of well deﬁned meeting data, annotation of the data, and research of methods to extract various events, occurrence of people and objects, and description of the processes included in the audio and video data. The records of the meetings were obtained through several synchronized video cameras and a set of microphones. Building upon the M4 project, the Augmented Multi-party Interaction project [4] (AMI) targeted interaction of the participants of meetings during the meetings. The newest, currently running Augmented Multi-party Interaction project with Distant Access [5] (AMIDA) project addes the aspect of ”remote meetings” where the participants of the meeting interact with the others remotely, In all the three consequent projects, automatic video editing based on processing of the data extracted from the audio/video streams is important. The main application areas for the video editing within the above mentioned projects include: L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 260–269, 2009. c Springer-Verlag Berlin Heidelberg 2009

Automatic Video Editing for Multimodal Meetings

261

1. ”Oﬀ-line” preparation of meeting summaries in the form of single video sequence. Such sequences may include only single audio/video stream based on ”intelligent” processing of several synchronous video streams and if necessary, may be shortened (in time) based on the semantic analysis of the input data and/or user requirements. 2. ”On-line” (real-time) video editing in the form of continuous production of a single audio/video stream based on the input. Such audio/video stream then can be shown e.g. on a large screen display to the meeting participants to enhance their perception of the meeting. 3. ”On-line” (real-time) video editing in the streaming form through Internet can also be used to transfer the audio/video information from one meeting site to another one in order to ”keep in touch” the remote meeting participants. This paper deals with the algorithms of automatic video editing that have been developed within the mentioned research projects. Section 2 summarizes the image processing and vision techniques used by the video editing engine. The automatic video editing algorithms and the proposed ﬁnal solution are described in Section 3. Section 4 deﬁnes the data types and record structuring used by the central database storage used for collecting the data from diﬀerent sources both the original recording equipment and diﬀerent processing and data-retrieval functional units. Section 5 summarizes the paper and suggests issues for future research.

2

Image Processing for Automatic Video Editing

The image processing methods used in automatic video editing include mostly the state of the art methods adjusted for the exploitation in meeting environments [1] [2] [7] [8]. The output of such methods needed for automatic audio/video editing includes mainly: – – – – – –

presence and positioning of humans and their body parts in the video, presence of objects in video, identiﬁcation of a speaker or localization of the speaker in image space, presence of events in video (related and non-related to human activities), facial expression and gesture recognition, tracking methods.

Methods used in the detection of presence of humans in the image and to identify the position and orientation of their body parts includes several known techniques, such as pre-processing of the image to ﬁnd the skin color spots, identiﬁcation of the spots (face and hand detection methods are implemented), recognition of faces through Ada-Boost based machine learning methods, and measurement of the face gaze through Gabor Wavelet Networks. The methods of detection of the presence of objects in the video are mostly based on machine learning approach but for several speciﬁc objects, such as whiteboard and/or projection wall, speciﬁc methods are proposed.

262

R. Kubicek et al.

Speaker localization is mostly done through audio signal processing (this might be done either through precise localization through microphone arrays or through processing of few microphone inputs with less precision – depending on the meeting recording equipment setup). Experiments also have been done with combination of audio and video processing through lips image detection and processing [9]. The presence of events in video are mostly recognized through detectors based on statistical processing of the scene and also on processing of lower level features through artiﬁcial intelligence methods. In fact, event detection is not only task of video processing but also of the meeting understanding research that is within the scope of the above mentioned projects but well out of the scope of this paper. The facial expression and gesture recognition methods used in the video editing are based on machine learning approaches, such as Ada-Boost based methods for image pbject recognition and Hidden Markov Models (HMMs) for scene dynamics analysis. Tracking methods used within the project are mostly based on particle tracking approaches, where the particles are mostly seen as human face, human hand, or human torso outline. The tracking methods are needed especially to cover the gaps that occur in the video processing if single frame processing does not lead in successful recognition and understanding the scene.

3

Automatic Video Editing

The approach to automatic video editing we used is based on the following facts: 1. The video presented to the humans should be well perceived by the humans from the point of view of dynamics of the shots. A large set of rules exists for this purpose [njak stihask kniha a Staova DP]. 2. The video should contain the relevant actions and objects that occur in the video. This aspect is largely dependent on the application and for the meeting data the relevant objects are humans and certain objects and actions known in advance. 3. In some cases, the video editing could be aﬀected by the user requirements, such as ”Show the activities of Mr. X in the meeting”. As the requirements can be changing over the time and can be also based on the user modiﬁcations, traditional procedural languages would not behave too well. At the same time, machine learning is not too suitable in this case, as the human perception of the video sequences are well known and well described (from the traditional ﬁlm making approaches). For the above reasons, non-procedural language Prolog was used as the implementation base of the editing decisions. The input of the editing process is a set of facts - results of audio/video stream processing, set of rules - the pre-deﬁned general video editing rules and a dynamically changeable set of rules based on the details of the audio/video processing, user requirements, etc. The output of the video editing decision process is a text description of the editing decisions

Automatic Video Editing for Multimodal Meetings

263

that is afterwards interpreted in a video editing engine that actually processes the audio/video data. The basic scheme of the process is shown in Figure 1 below.

Fig. 1. The basic video editing scheme

The main idea of building the rules for selection of views into the output video stream is that each camera should be given its ”weight” that represents the importance of the activity shown by the camera. Each person, if known, should also be given the ”weight” with similar meaning, and ﬁnally, a decision of what is going to the output is done using these weights and the human perception of video rules (that prevent e.g. too long and too shorts shots or too rapid changing of the shots). Additionally, virtual cameras can be deﬁned as rectangles within the existing camera images. See Figure 2 and Figure 3 for the editing rules.

4

Remote Access to the Audio/Video Data HUB

The input of the audio-video editing system is constituted by the multimedia audio-visual data themselves with metadata describing their properties and structure. As an input serve also events appearing in the data. These events – that are potentially somehow important for the video editing – are either manually annotated by a tool or are extracted automatically. In case of manual annotation, the data are collected before the editing process starts, the automatic

264

R. Kubicek et al.

Fig. 2. The weight deﬁnition video rules for cameras

Fig. 3. The weight deﬁnition video rules for meeting participants (persons)

event extraction can happen during the video processing, whose output is the edited video sequence. All the manually annotated and automatically extracted events need to be stored for later use and for the editing process. The example of the output of video editing is shown in Figure 4 below (along with the three input video streams).

Automatic Video Editing for Multimodal Meetings

265

Fig. 4. Example of the output of the video editing algorithm

To allow this, a component serving as the data storage of the annotations has been deﬁned, that allows a uniﬁed way of storing and retrieving the data. This complete infrastructure is referred to as the Hub. The Hub is intended to provide all of the storage that a group or a company needs for annotations about their archived meetings in one place. Each such Hub contains a database, which keeps the stored annotations and serves requests to the data. Along with the annotation/extracted data it contains all related documents, presentation slides and notes shown during the meeting. The Hub is based on the producer-consumer model, see Figure 5, where the producer sends information to the Hub, which transforms them to its internal data format and stores into the database. The client consuming the data sends a request to the Hub, which processes it and if data matching the request are found, they are sent to the client. Such requests can be speciﬁed either to match an exact data entry or regular expressions, time intervals etc. can be used. At the same moment when one event-extracting process inserts data to the Hub, the consuming application can retrieve it. This allows the Hub to be used both in real-time on a pending meeting to provide a newly connected client with all available information about the meeting as well as to post-processing the meeting data and production of summaries and similar material. The data in the Hub are stored as timed-triples. The triple contains entries: object, attribute and value, and is accompanied by a timestamp. But the application for automatic video editing (VideoEditor) uses a XML ﬁle as the source, which contains relevant information about the input data, annotated and detected events. A communication protocol was deﬁned to interface the Hub with this application. It deﬁnes the format of separate tags stored in the triples in

266

R. Kubicek et al.

Fig. 5. Producer-consumer scheme of Hub

the Hub and it handles the transformation of data read from the XML ﬁle into the correspoding timed triples and the following communication with the Hub by the means of requests. These requests are processed by a XML parser, which can construct the request and parse the correspoding response to the relevant data entries. This parser can also handle a invalid XML, which for example does not contain the termination tags or does not meet the deﬁned DTD. A comparison of the XML structure with the modiﬁed communication protocol for the Hub is in the following listing. [MID] denotes a unique identiﬁer of the meeting, [N] is represented by all indices from 0 to Count of the corresponding section and [...] denotes all diﬀerent parameters of the respective section. The output data marking the places for the shot boundary are stored with identical names, only instead of [MID] they use [MID-out]. 4.1

Data Path from the Event Producer to the Editing Process

Either during the recording or after it has been ﬁnished, the video data are sent to the annotation software and to the automatic event extractors. These processes extract relevant events and send them to the Hub. Such data are requested by the VideoEditor as a bulk. The block of data replied from the Hub is parsed from the XML into the timed-triples in the internal format of the application. Complete data path of the video editing process is shown in the Figure 7. 1 2 3 4 5 6 7 8 9 10 11

get all appropriate data from hub if received Data parse Data into TimedTriples else end application for each InternalStructure find RelevantData if RelevantData in TimedTriples fill InternalStructure with RelevantData else process next InternalStructure run video editing process

Automatic Video Editing for Multimodal Meetings

267

<EventGroups> 0individual_positions <Meaning>State<Enabled>1 1Speaking <Meaning>State<Enabled>1 <EventTypes> 0off_camera0 <Parameters individual="PM"/> <Secondary> 0<Parameters individual="ID"/> ... ... 8Start speaking0 10 <Source Camera="3">video\ES2003a.Corner_orig.avi ... <Source>audio\ES2003a.Mix-Headset.wav <TimeFormat>Milliseconds ... <Event> 8<Time>1110415 Yeah? Okay. <Parameters Person="Closeup4"/> ... Fig. 6. Example structure of XML input ﬁle for video editing application

After the editing process is ﬁnished, the resulting set of events – marking the shot boundaries – is sent back to the Hub to be used for later editing or for comparison/evaluation with other means of editing.

268

R. Kubicek et al. Table 1. Rules for saving XML events into Hub timed-triples object [MID].EventGroups.Group [MID].EventGroups.Group.[N] [MID].EventTypes.Type [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].EventTypes.Type.[N] [MID].Sources.Source [MID].Sources [MID].Sources.Source.[N] [MID].Events.Event [MID].Events.Event.[N] [MID].Events.Event.[N] [MID].Events.Event.[N]

attribute Count [...] Count [...] Parameters.Count Parameters.[N].[...] SecondaryKeys.Count SecondaryKeys.[N].[...] SecondaryKeys.[N].Parameters.Count SecondaryKeys.[N].Parameter.[...] Count TimeFormat [...] Count [...] Parameters.Count Parameter.[N].[...]

Fig. 7. Data path of video editing process

A streaming media server is being developed that will allow automatic editing on a running meeting. At present state, the data needs to be stored in a media ﬁle in the ﬁle system with the editing tool. When such streamed media is available, the Hub’s potential will be fully utilized.

Automatic Video Editing for Multimodal Meetings

5

269

Conclusions and Future Work

This paper presented an approach to automatic video editing, which builds upon a central data-collection system which collects data from diﬀerent sources - both from the original audio/video streams from the meetings and from diﬀerent metadata and extracted data as retrieved by a number of data processing units. The described system proved to be functional in a demonstration sample. However, interconnection with more data retrieval units will be sought in the future, to further extend the possibilities of the automatic video editing algorithms. The system would also beneﬁnt from an extended set of video-editing rules, which will be deﬁned based on real usage on actual data acquired from model and real meetings. The results of the AMI and AMIDA projects can be found on the web ([5]) and are available for further use.

Acknowledgements This work has been supported by the Ministry of Education, Youth and Sports of the Czech Republic under the research program LC-06008 (Center for Com puter Graphics), and by the research project ”Security-Oriented Research in Informational Technology CEZMSMT, MSM0021630528.

References 1. Ashby, S., et al.: The AMI Meeting Corpus. In: Measuring Behavior, Proceedings Book, Wageningen, NL, p. 4 (2005) 2. Kadlec, J., Potucek, I., Sumec, S., Zemcik, P.: Evaluation of Tracking and Recognition Methods, In: Proceedings of the 11th conference EEICT, Brno, CZ, pp. 617–622 (2005) ISBN 80-214-2890-2 3. Porter, S., Mirmehdi, M., Thomas, B.: Temporal video segmentation and classiﬁcation of edit eﬀects. Image and Vision Computing 21(13-14), 1097–1106 (2003) 4. Augmented Multi-party Interaction project (available as of 2008-07), http://www.amiproject.org/ 5. Augmented Multi-party Interaction with Distant Access (available as of 2008-07), http://www.amidaproject.org/ 6. Multi-Modal Meeting Manager project (available as of 2008-07), http://www.m4project.org/ 7. Chambela, T., et al.: Creating video art with evolutionary algorithms. Computers & Graphics 31(6), 837–847 (2007) 8. Bocconi, S., Nack, F., Hardman, L.: Automatic generation of matter-of-opinion video documentaries. Web Semantics: Science, Services and Agents on the World Wide Web 6(2), 139–150 (2008) 9. Sumec, S., Kadlec, J.: Event Editor -The Multi-Modal Annotation Tool, In: Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI), Edinburgh, GB (2005)

Foreground Segmentation via Segments Tracking Bogdan Kwolek Rzesz´ ow University of Technology, 35-959 Rzesz´ ow, Poland [email protected]

Abstract. In this paper we propose a video segmentation algorithm that in the ﬁnal delineation of the object employs the graph-cut. A partitioning of the image based on pairwise region comparison is done at the beginning of each frame. A set of keypoints is tracked over time via optical ﬂow to extract regions, which are likely to be parts of the object of interest. The tracked keypoints contribute towards better temporal coherence of the object segmentation. A probabilistic occupancy map of the object is extracted using such initial object segmentation and a probabilistic shape model. The map is utilized in a classiﬁer that operates both on pixels and regions. The aim of the classiﬁer is to extract a trimap consisting of foreground, background and unknown areas. The trimap is employed by graph-cut. The outcome of the graph-cut is used in on-line learning of the shape model. The performance of the algorithm is demonstrated on freely available test sequences. Keywords: Object tracking, video segmentation.

1

Introduction

Automatic segmentation of monocular video is a very challenging task. The graph-cut technique of Boykov and Jolly [1] has stimulated an explosion of interest toward automatic segmentation, rising quickly to become one of the most inﬂuential and leading algorithms for interactive image segmentation [2]. Segmentation of object in video is diﬀerent from a segmentation of object in a single image. Video segmentation should preserve temporal consistency among segmentations and in consequence should indicate that extracted segments belong to the same objects. Recent research in this area led to compelling, real-time algorithms, either based on motion [3] or stereovision [4]. Extraction of foreground objects in monocular video is less constrained and often requires additional assumptions. Research termed as layer motion segmentation [5][6] received considerable interest of the computer vision community in the past decades. The general objective of methods falling into this category is to automatically extract motion coherent regions. However, such methods are computationally expensive and often require oﬀ-line learning as well as batch processing of the whole sequences. Segmentation of objects in video under assumption that the background is static [7][8] may yield confused results due to moving background objects, shadows, etc. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 270–281, 2009. c Springer-Verlag Berlin Heidelberg 2009

Foreground Segmentation via Segments Tracking

271

Most of the current approaches to image/video segmentation rely primarily on image-based criteria, such as color, gray level, or texture regularity of image regions in context of smoothness and continuity of bounding outlines. In bottomup approaches an image is ﬁrst segmented into regions and then identiﬁcation which regions correspond to single object takes place with regard to continuity. In complementary approach referred to as top-down segmentation, prior knowledge about an object, such as its possible shape, color, or texture is utilized to guide the segmentation. The diﬃculties inherent in pure low-level segmentation stimulated elaboration of top-down, class-speciﬁc segmentation algorithms [9]. In the algorithm [10] the training is done on a set of learned pictorial structures (LPS). The learned models are then used in a detection cascade. The obj-cut algorithm provides excellent segmentations by the use in min-cut a good initialization, which was obtained by LPS. In [1] the video segmentation is achieved by passing a complete set of frames to the algorithm and treating the entire sequence as 3D grid of pixels. Motion information has been used for video segmentation for a long time [11][7][12]. The motion segmentation algorithms assume that the foreground and background objects have unlike motion patterns. The objective of methods termed as layerbased motion segmentation is to automatically extract motion coherent regions [7][5]. However, certain methods falling into this category require oﬀ-line learning and batch processing of the whole video [7][5]. Occlusions make the motion based segmentation algorithms prone to errors, in particular at object boundaries. Therefore some work has been done to elaborate methods dealing with such errors [5]. The segmentation algorithm [11] is based on Markov Random Fields (MRF) that are built on three energy terms, namely motion, intensity and boundary. The assumption behind the MRF based model is that pixels spatially close to one another have a tendency to be in an identical layer. With the help of MRF the holes in the segments are suppressed and regular boundary shapes are preferred. However, the calculation of MRF-like constraints is computationally expensive. In [13] a simple body model and MRF are utilized in video segmentation at the blob level. Temporal consistency over segmented blobs is maintained by the usage of weighted bipartite matching. However, while segmentation of videos with real-world content, some blobs can be merged into single one, whereas a single blob can be split into multiple blobs and in the last resort it can disappear. The mentioned phenomena make blob tracking a challenging task. In [3] a spatio-temporal derivatives based model, which has been learned oﬀ-line from labeled data is employed in a classiﬁer operating jointly on intensity change and contrast to distinguish between motion and non-motion. The priors for segmentation are represented by a second order, temporal Hidden Markov Model and spatial MRF. Finally, the layer segmentation is done by graph-cut. However, this algorithm needs to be trained by the use of groundtruth and is relatively slow. In the algorithm presented in this work, the ﬁnal delineation of the object is done via graph-cut. Using the object mask extracted via graph-cut a probabilistic shape model is learned on-line. A partitioning of the image based on

272

B. Kwolek

pairwise region comparison takes place at the beginning of each frame. Using a collection of the tracked point features we extract adjacent segments, which are likely to be parts of the object of interest. Next, using a component consisting of such segments a distance function is constructed. Given the motion of the object, which is determined on the basis of the tracked features an adjustment of the probabilistic shape model to likely location of the object takes place. Afterwards, we extract a probabilistic occupancy mask of the object. It is constructed on the basis of the adjusted probabilistic model of the object shape and the object distance function. A classiﬁer operating on (i) probabilistic occupancy map, (ii) mean colors of the segments and (iii) the quantity of the tracked features in each segment constructs a trimap consisting of foreground, background and unknown areas. The graph-cut responsible for ﬁnal delineation of the object is built on such a trimap. The smoothness term in the energy model is adapted gradually according to the occupancy map. The segments only partially covered by probabilistic occupancy map or holding too few features are classiﬁed as unknown. In particular, some segments arising during changes of object appearance have typically no successfully tracked features or the number of features is too small to assign such segment to the object. But with support of the classiﬁer such feature-less object fragments obtain keypoints for tracking. At last such features contribute signiﬁcantly to better object segmentation as well as temporal coherence in segmentation. The proposed method is diﬀerent from recent algorithms making use of graphcut in video segmentation. Most of the relevant work is concerned with improving the eﬃciency of graph-cut [14], the use of various image attributes [15] and models [3][8], which are used in a single-step segmentation via graph-cut only. In our approach, in several segmentation steps we employ tracking of object fragments, a classiﬁcation at intermediate level, on-line learning of the object shape, and in particular, we operate not only at pixel level but also use regions. The rest of the paper is organized as follows. In the next Section we brieﬂy describe video segmentation via pairwise region comparison. Section 3. is devoted to on-line learning of a probabilistic model of the object shape. The video segmentation via graph-cut built on the tracked object fragments and the learned shape model is explained in Section 4. In Section 5. we report results, which were obtained in experiments. Finally, some conclusions follow in the last Section.

2

Spatio-Temporal Figure-Ground Segmentation via Pairwise Region Comparison and Point Matching

Figure-ground segmentation refers to a delineation of a region in an image such that it contains the object of interest. Spatio-temporal segmentation consists in delineating one or multiple objects from images rather than partitioning video into disjoint regions. Video segmentation should keep temporal consistency among segments in consecutive frames. This means that segments of a given image should relate to the segments of the previous one such that they do not belong to diﬀerent objects. Exact estimation of optical ﬂow is computationally costly due to extensive search in the neighborhood of every image

Foreground Segmentation via Segments Tracking

273

pixel. Therefore, in order to achieve segmentation of video in real-time as well as to preserve temporal consistency of the segmentation, our algorithm relies on propagating the object seed features to the next frame. A set of adjacent blobs selected manually or automatically in the initialization step constitutes an object to be tracked. In each frame the algorithm ﬁrst partitions the image into regions, then using the tracked features it identiﬁes the image regions that correspond likely to the object of interest. The input image is partitioned into segments using method from work [16]. It selects edges from an undirected graph G = (V, E), in which every pixel corresponds to a node vi ∈ V and neighboring pixels are connected by unidirected edges (vi , vj ) ∈ E. The segmentation consists in partitioning of vertices V into components such that each component corresponds to a connected component in the graph G = (V, E ), where E ⊆ E. The evidence for a boundary between components is based on (i) intensity diﬀerences along the boundary and (ii) intensity diﬀerences between neighboring pixels within regions. The internal diﬀerence Int(C) of a component C ⊆ V is deﬁned in the following manner: Int(C) =

max

e∈MST (C,E)

w(e),

(1)

and it is the largest weight in the minimum spanning tree M ST (C, E) of the component. A component remains connected if it consists of edges with weights at least Int(C). The diﬀerence between two components C1 , C2 ⊆ V is deﬁned as the minimum weight edge in the following manner: Dif (C1 , C2 ) =

min

vi ∈C1 ,vj ∈C2 ,(vi ,vj )∈E

w(vi , vj ).

(2)

In case when there is no edge connecting C1 and C2 the diﬀerence takes ∞. The minimum internal diﬀerence M Int, which is deﬁned as follows: M Int(C1 , C2 ) = min(Int(C1 ) + τ (C1 ), Int(C2 ) + τ (C2 ))

(3)

is employed to verify if there is a boundary between a pair of components, where τ (C) = κ/| C |, and κ is a threshold. The running time of the algorithm is nearly linear and the algorithm is fast in practice. In a preprocessing stage a Gaussian ﬁlter has been applied in order to smooth the images to be segmented. During postprocessing a merging of very small components to their neighbors has been realized. In our implementation the edge set E has been constructed from pairs of neighboring 8-connected pixels. The edge weights assume the values of the Euclidean distance (L2 norm) between color components in RGB space. Figure 1 depicts some segmentation results of the Foreman image for two diﬀerent thresholds parameters κ, which were set to κ = 50 and κ = 160, respectively. We can observe that despite similar colors of the background and the helmet, for κ = 50 the algorithm quite accurately delineated the foreground boundary. Owing to graph based representation of the segmented image a single click on the mouse can activate or deactivate the whole segments. Such active segments

274

B. Kwolek

Fig. 1. Image segmentation via pairwise region comparison using diﬀerent values κ

are overlaid transparently on the input image. They constitute a very simple and intuitive way to adjust the initial object segmentation to the user liking. Given the user delineation of the object in the ﬁrst frame, the algorithm extracts object keypoints using method [17], which focuses on selecting good features for tracking. Next, some additional point features are spread evenly to cover the whole object of interest. During generation of such features a predeﬁned minimum distance to the closest feature is taken into account. At this stage the algorithm veriﬁes if each segment contains suﬃcient number of point features with respect to its area. The complete feature set consists of both such keypoints and good features. Tracking of the feature set across images allows the algorithm to ﬁnd the object from one image to another. On the other side, the segmentation information supports following the entire object instead of just a few features on it. Instead of calculating an optical ﬂow for the whole image the algorithm calculates the local displacement vector for each feature. The mean location of all valid features is used in calculating the object motion. A validity checking is performed to estimate wrong inter-frame correspondences among point features. If the color diﬀerence between keypoints at the current and previous location is above a threshold the feature is rejected. The median direction of remaining features is calculated. All features that diﬀer more than a predeﬁned value from such direction are rejected. Next, using the valid features the motion of the object is calculated. The tracking of point features is done on the basis of pyramidal implementation of the optical ﬂow [18]. The optical ﬂow is computed with sub-pixel accuracy on the basis of bilinear interpolation. Owing to pyramidal implementation of the optical ﬂow the object segmentation can cope with large motions.

3

On-Line Learning of Probabilistic Model of the Shape

The object segmentation obtained by graph-cut may drift and thus evolve into arbitrary shapes due to clutter and ambiguous backgrounds. Severe segmentation ambiguities may occur when the images are noisy or the illumination changes. Recent approaches also employ spatial Gaussian mixture models (GMM) in video segmentation [7]. However, most existing algorithms do not learn on-line or relies on small motion assumption. Hence, they assume that the mixture models from the current frame can be applied to segmentation of the next frame. In our

Foreground Segmentation via Segments Tracking

275

approach a dynamic Gaussian prior imposes preferences for shapes similar to examples that have been seen just before, as it propagates hypotheses about the object shape. In particular, the proposed dynamic Gaussian prior can also encode the knowledge about the initial shape of the object. In the ﬁnal delineation of the object the GMM model can give an estimate of the desired segmentation. At this stage it prevents the object shape from evolving into the arbitrary shape in the course of segmentation, particularly when there is a signiﬁcant overlap between foreground and background pixels. In our approach the compact object shape consists of N points {ξi }N i=1 of dimensionality d = 2. The shape is modeled with a mixture of K components. The likelihood that a pixel j belongs to the object can be expressed in the following manner: p(ξj ) =

K

p(k)p(ξj |k)

(4)

k=1

where p(k) is the prior of the k-th component and p(ξj |k) is the conditional density function. In the sequel, for probability density function consisting of Gaussians, the parameters in (4) are denoted as p(k) = πk , p(ξj |k) = N (ξj ; μk , Σk ), where μk and Σk are parameters of the model. The conditional density function p(ξj |k) takes the following form: p(ξj |k) =

1 1

d

(2π) 2 |Σk | 2

e−

(ξj − μk )T Σk−1 (ξj − μk ) 2

(5)

where d is dimensionality, {πk , μk , Σk }K k=1 are parameters of the model and denote prior, mean vector and covariance matrix, respectively. Given the initial object shape the full-covariance Gaussian mixture can be learned oﬀ-line using iterative Expectation Maximization (EM) algorithm. The k-means clustering algorithm can be used to provide a rough estimate of the model parameters. In the EM algorithm the parameters are estimated iteratively as follows: (i)

(i+1)

E-step:

pk,j

M-step:

πk

N (i+1)

=

(7)

N

j=1

= N N =

(6)

pk,j

(i+1)

pk,j ξj

j=1

(8)

(i+1)

j=1 (i+1) Σk

(i)

(i+1)

j=1

N (i+1) μk

(i)

π N (ξj ; μk , Σk ) = K k (i) (i) (i) l=1 πl N (ξj ; μl , Σl )

pk,j

(i+1)

(i+1)

(i+1) T

pk,j (ξj − μk )(ξj − μk N (i+1) j=1 pk,j

)

(9)

In each iteration i the algorithm computes the average log-likelihood as follows: L(i) =

N 1 (i) (i) (i) log πk N (ξj ; μk , Σk ) N j=1

(10)

276

B. Kwolek

The algorithm stops if L(i+1) /L(i) < th, where th is a predeﬁned value. Let us assume that in time t the learning algorithm outlined above converged in iteration (ii) (ii) (ii) ii. Given the parameters {Ek , μk , Σk }K k=1 , which were obtained in the it (ii) (ii) eration ii and stand for the cumulated posterior probabilities Ek = N j=1 pk,j , mean vector and covariance matrix, respectively, we deﬁne the following param(ii) (ii) N (ii) (ii) (ii) N (ii) eters: Eμk = μk j=1 pk,j , EΣk = Σk j=1 pk,j . Assuming that posterior (ii)

probabilities remain the same, i.e. the parameters Ek do not change their values, when in time t + 1 new data {ξj }M j=1 are considered by the model, the model can be updated according to the following steps: (i)

pk,j

M-step :

(i+1) πk

(ii)

(ii)

Eμk +

(i+1) μk

=

(ii,i+1)

= (μk

Σk

(ii)

Ek

(ii)

(ii)

(i+1) Σk

=

(i)

(i)

π N (ξj ; μk , Σk ) = K k (i) (i) (i) l=1 πl N (ξj ; μl , Σl )

(i+1)

E-step:

=

Ek

+

M j=1

(11)

(i+1)

pk,j

(12)

N +M

M

+

(i+1) j=1 pk,j ξj M (i+1) j=1 pk,j (i+1)

− μk

(ii)

(ii)

(13) (i+1) T

)(μk

− μk

(ii,i+1)

+

EΣk +Ek Σk

M

)

(i+1) j=1 pk,j (ξj − (ii) M (i+1) Ek + j=1 pk,j

(14) (i+1)

μk

(i+1) T

)(ξj − μk

)

(15)

The update of the model stops when the ratio of the average log-likelihoods from iteration i and i + 1 is below a predeﬁned threshold. Figure 2 depicts the ability of GMMs to model the compact object shapes. The tests have been carried out using the standard Foreman test sequence. In the input images #1 and #2 (left column) the ellipses resulting from the shape model are overlaid transparently. The shape model has been trained via the EM, initialized by k-means, using pixel locations taken from an object map (see image in top row and middle column). Then the shape model has been updated via the use of pixel locations from an object map reﬂecting a segmented foreground (bottom row). Looking at the ellipses at bottom left part of the object map, which cover the missing part of the torso we can observe how the model adapts to the changes of the shape. This capability can also be observed in the object probability map that is depicted in the third column. It can be observed that due to the missing part of the torso the likelihood is something smaller in this part of the image. The object likelihood has been constructed as a product of two mixtures of Gaussians. It is constructed under assumption that object color is not dependent on its location. The ﬁrst mixture consists of 20 spatial Gaussians. The second one approximates the color distribution in RGB color space. The ﬁxed color model

Foreground Segmentation via Segments Tracking

277

Fig. 2. Modeling the object shape with color-spatial GMMs

has been built using 10 Gaussians. By the change of signiﬁcance of spatial part in the color-spatial probabilistic model we can accommodate the model to the object speed, shape variability, etc. Examples shown in Fig. 3 illustrate the eﬀect of balancing between color and spatial part in the model.

Fig. 3. Balancing between color and the spatial part in the object model

Using the current object shape, the object likelihood map can be constructed in several ways. As discussed above, it can be composed on the basis of the current frame and former one. Given the motion of the object the spatial model from the previous frame can be projected forward in time trough adjustment of μ(ii) parameter. Given the pose of the object undergoing segmentation the shape model from the ﬁrst frame can be employed in the update of the model. In most cases the ﬁrst and simplest approach has proven to be eﬀective and suﬃcient.

4

Video Segmentation via Graph-Cut Using Tracked Object Fragments and Learned Shape Model

The image segmentation X can be uniquely determined by minimum cut C on the graph G with two terminal nodes such that the terminals are separated on the induced graph G = (V, E\C) [1]. The minimization is done using Gibbs energy

278

B. Kwolek

function that is formulated over the unknown labels X as the sum of data term and boundary term: E(X ) = Edata (X ) + λEsmooth (X ) = E1 (xi ) + λ E2 (xi , xj ), i∈V

(16)

{i,j}∈N

where V is the set of all vertices in graph G, whereas N is the neighborhood formed from the vertex connectivity. The regularization factor λ balances the data term E1 and the smooth cost E2 . The data term E1 is a likelihood energy, which expresses cost of assigning label xi to node i. The prior term E2 encodes the cost when the adjacent nodes i and j have the labels xi and xj , respectively. The min-cut of the weighted graph determines the segmentation that best separates the object from its background. In our work we use the recent max-ﬂow/min-cut implementation from work [19]. The weight of a t-link connecting a terminal with the pixel corresponds to cost of assigning the label to this pixel. There are two t-links for each pixel. The weights of these links depend on the state of the object trimap. If a pixel belongs to object, the value of background t-link is set to 0, whereas the foreground t-link is set to K. If a pixel has been classiﬁed as background the value of background t-link is set to K and the value of foreground t-link is set to 0. The value of K is set as follows: K = 1 + maxi∈V j:{i,j}∈N E2 (xi , xj ). To set the weights of t-links for pixels of unknown state we use the probabilities obtained from two separate GMMs. The ﬁrst GMM is used to model the distribution of background color. The second one is the model used in determining the object likelihood, discussed in Section 3. The data cost is the negative log likelihood of the color given the label. The smoothness term is a standard Potts model: E2 (xi , xj ) = dist(i, j)−1 exp(−βC(i) −C(j)2 )+λ2 , where dist(i, j) stands for the distance between pixels i and j, C(i), C(j) are RGB colors of two neighboring pixels i and j, λ2 is set to 10. β is a robust parameter that is set to (C(i)−C(j)2 )−1 [2], where the expectation stands for an average over the image. The distance term dist(·) is used to reduce a tendency of the segmentation algorithm towards diagonal cuts, whereas the aim of λ2 is to discard small and high-contrasted regions. In order to generate the trimap with very likely object pixels we employ a classiﬁer that operates both on pixels as well as on segments. The outcome of the classiﬁer is a binary mask of the object, background pixels and pixels of unknown state. We utilize a decision tree classiﬁer, which input consists of: the graph constructed during segmentation via pairwise region comparison in order to calculate the quantities given by (2) and (3), the number of valid features on each segment, the probabilistic occupancy ˜B map of the object, and the quantities d˜F k , dk that are explained below. As we already mentioned, at the initialization stage we manually extract foreground and background segments. For each segment we compute the mean colors and the clustering takes place separately for foreground and background. B M As a result we obtain cluster centroid colors {PnF }N n=1 and {Pn }m=1 , where N and M stand for the number of foreground and background clusters, respectively. At this stage we use the well-known k-means algorithm because it

Foreground Segmentation via Segments Tracking

279

acknowledged its usefulness in image segmentation [20]. At run-time, just before the classiﬁcation, for each candidate segment k we compute the minimum distance from its mean color Ck to foreground and backgrounds clusters as follows: F B B dF k = arg minn Ck − Pn , dk = arg minm Ck − Pm . Then we compute the F F F B B B B ˜ ˜ mentioned quantities: dk = dk /(dk + dk ), dk = dk /(dF k + dk ). The graph-cut extracts a binary map representing the object of interest. In the course of segmentation it adapts the value of β with reference to the probabilistic occupancy map of the object. The larger the distance of the considered background pixel to the object, the smaller the probability of the cut is. Using pixels belonging to the object map the probabilistic model of the object shape is learned on-line as discussed in Section 3. With the support of the object mask the keypoints with tentative matches are replaced by newly generated keypoints.

5

Experiments

To test the eﬀectiveness of the proposed algorithm we performed various experiments on several test sequences. We tested feature tracking via Kalman ﬁlter and a linear motion model. The conﬁdence measure that the candidate keypoint corresponds to the tracked one was built on: (i) scaled diﬀerences between gradient vectors at the tracked and candidate locations, scaled by the standard deviation of the gradient, (ii) diﬀerence between the predicted keypoint location and the candidate keypoint location, scaled by the standard deviation in position, (iii) cross-correlation between color patches. The standard deviation was used to express the reliability of the measure and was extracted from the covariance matrix of the Kalman ﬁlter. The method permits reliable keypoint matching. However, computation burden of this method is signiﬁcant because for each tracked keypoint it adds every candidate keypoint to the list of possible matches. Our research ﬁndings show that for our algorithm the optical-ﬂow based feature tracking is eﬀective and suﬃcient. At image partitioning stage we tested the well-known mean-shift segmentation algorithm [21]. This algorithm determines spatio-temporal segments with better coherence. But such stable segments were important in an earlier version of the algorithm, which has only been based on feature tracking and graph-cut in the ﬁnal segmentation. The segments, which are obtained by the current method usually correspond to large homogenous fragments of object parts. Given an initial set of image pixels corresponding to the visible area of an object, we can track it over subsequent frames, even if the viewpoint changes and diﬀerent parts of the object come into and out of view. It is worth to note that the computation time of the mean-shift based segmentation is several times larger in comparison to the method currently used. Figure 4 depicts some segmentation results that were obtained using the Foreman test sequence. The person is segmented as complete entity, despite non-stationary background, ambiguous and cluttered background as well as variability of the object shape. In order to conduct comparison tests we implemented the algorithm [3]. This method requires training with ground-truth of diﬀerent environments. Moreover,

280

B. Kwolek

#2

#15

#30

#45

#60

#70

#75

#90

#105

#120

Fig. 4. Video segmentation using Foreman test sequence

the weights of the four energy terms need to be tuned for the segmented sequence. Experiments done at AT sequence1 showed that the discussed method does not work well if the foreground has similar colors as the background, in particular when there is little motion in the foreground. Figure 5 shows some segmentation results, which were obtained by our method on the mentioned above test sequence.

#20

#30

#40

#50

Fig. 5. Video segmentation using AT sequence

The number of background pixels with similar color to the foreground color, which were labeled as the object is far smaller. Our experiments demonstrated that the objects are segmented as complete entities, despite considerable variability of the object shape as well as cluttered background. The method can even handle situations in which object boundary shares similar colors with the background.

6

Conclusions

The strength of our algorithm lies in multi-stage segmentation. In several segmentation steps we employ tracking of object fragments, classiﬁcation at intermediate level, on-line learning of the object shape, and in particular, we operate not only at pixel level but also utilize regions. The algorithm is robust to image noise and 1

Available on: http://research.microsoft.com/vision/cambridge/i2i

Foreground Segmentation via Segments Tracking

281

can achieve accurate region boundaries. One of the future research directions of the presented approach is to explore the object delineation via a graph-cut built on pre-segmented images as a means to reduce the computational cost.

References 1. Boykov, Y., Jolly, M.: Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In: Proc. of ICCV, pp. 105–112 (2001) 2. Rother, C., Blake, A., Kolmogorov, V.: Grabcut - interactive foreground extraction using iterated graph cuts. In: Proc. of ACM SIGGRAPH, pp. 332–343 (2004) 3. Criminisi, A., Gross, G., Blake, A., Kolmogorov, V.: Bilayer segmentation of live video. In: Proc. of CVPR, pp. 53–60 (2006) 4. Kolmogorov, V., Criminisi, A., Blake, A., Ross, G., Rother, C.: Bi-layer segmentation of binocular stereo video. In: Proc. of CVPR, pp. 1186–1193 (2005) 5. Xiao, J.J., Shah, M.: Motion layer extraction in the presence of occlusion using graph cut. In: Proc. of CVPR, Washington, D.C., pp. II:972–II:979 (2004) 6. Kumar, P., Torr, P., Zisserman, A.: Learning layered motion segmentations of video. In: Proc. of ICCV, pp. 33–40 (2005) 7. Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pﬁnder: real-time tracking of the human body. PAMI 19, 780–785 (1997) 8. Sun, J., Zhang, W., Tang, X., Shum, H.Y.: Background cut. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 628–641. Springer, Heidelberg (2006) 9. Leibe, B., Leonardis, A., Schiele, B.: Combined object categorization and segmentation with implicit shape model. In: Ponce, J., Hebert, M., Schmid, C., Zisserman, A. (eds.) Toward Category-Level Object Recognition. LNCS, vol. 4170, pp. 508– 524. Springer, Heidelberg (2006) 10. Kumar, M., Torr, P., Zisserman, A.: Obj cut. In: Proc. of CVPR, pp. 18–25 (2004) 11. Black, M.: Combining intensity and motion for incremental segmentation and tracking. In: Sandini, G. (ed.) ECCV 1992. LNCS, vol. 588, pp. 485–493. Springer, Heidelberg (1992) 12. Khan, S., Shah, M.: Object based segmentation of video using color, motion and spatial information. In: Proc. of CVPR (2001) 13. Park, S., Aggarwal, J.K.: Segmentation and tracking of interacting human body parts under occlusion and shadowing. In: Proc. of Workshop on Motion and Video Computing, pp. 105–111 (2002) 14. Juan, O., Boykov, Y.: Active graph cuts. In: Proc. of CVPR, pp. 1023–1029 (2006) 15. Xu, N., Bansal, R., Ahuja, N.: Object segmentation using graph cuts based active contours. In: Proc. of CVPR, pp. 46–53 (2003) 16. Felzenszwalb, P.F., Huttenlocher, D.P.: Eﬃcient graph-based image segmentation. Int. J. of Computer Vision 59, 167–181 (2004) 17. Shi, J., Tomasi, C.: Good features to track. In: Proc. of CVPR, pp. 593–600 (1994) 18. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. Int. Joint Conf. on Artiﬁcial Intell., pp. 674–679 (1981) 19. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-ﬂow algorithms for energy minimization in vision. PAMI 26, 1124–1137 (2004) 20. Plataniotis, K., Venetsanopoulos, A.: Color Image Processing and Applications. Springer, Heidelberg (2000) 21. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. PAMI 24, 603–619 (2002)

Multi-layer Background Change Detection Based on Spatiotemporal Texture Projections Roland Miezianko1 and Dragoljub Pokrajac2 1 2

Honeywell Labs, Minneapolis, MN 55418, USA [email protected] Delaware State University, Dover, DE 19901, USA [email protected]

Abstract. In this paper we explore a multi-layer background change detection method based on projections of spatiotemporal 3D texture maps. The aim of this method is to provide a background change detection of a region viewed by multiple cameras. Camera views are projected onto a common ground plane, thus creating a spatially aligned multi-layer background. The aligned multi-layer background is subdivided into nonoverlapping texture blocks, and block data is dimensionally reduced by principal component analysis. Motion detection is performed on each block, and non-moving sections of the block are clustered into multiple hyperspheres. An analysis of the clusters from spatially aligned multilayer blocks reveal regions of changed background. This method is evaluated on surveillance videos available from PETS2006 and PETS2007 datasets. Keywords: Motion, texture.

1

Introduction

Most recent video datasets of multiple overlapping camera networks [1,2] challenge researchers to address the abandoned luggage event detection. This scenario is based on real need of the security and surveillance communities. The main challenges include the ﬂexibility of a system to handle large ﬂows of people, to discriminate between slow moving or temporary changes and deliberate luggage abandonment, to handle occlusions in one or more camera views, and to minimize false alarm rates. Current approaches address the background change detection, due to an abandoned luggage event, as either a tracking problem [3,4], an object classiﬁcation problem [5,6], or as a motion detection problem [7]. The work of [8] to learn a mixture of Gaussians with set of models at each pixel location using the expectation maximization based adaptation method is the foundation of many motion detection methods. Some use the background subtraction methods [9,10,11] to detect background changes at every pixel [12] and to build regions of interest. Such methods have been supplemented to handle illumination changes, shadows, and camera-based noise and jitters. Methods based on analyzing people and object trajectories [3,4] require tracking initialization and updates [13]. This is very L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 282–291, 2009. c Springer-Verlag Berlin Heidelberg 2009

Multi-layer Background Change Detection

283

Fig. 1. Scheme of the proposed method

challenging in crowded environments where full and partial occlusions are more the norm than exception. In this paper we propose a multi-layer block based approach to detect background changes. The motivation to use this approach stems from the need to process overlapping camera views and utilize the fact that common motion regions and common background changes are present. Multiple overlapping camera views are projected onto a common background mask using homographic matrices. This projection creates a multi-layered image with co-located spatial regions. A common overlapping region limits the processing boundary where spatiotemporal texture blocks are deﬁned. Each spatiotemporal texture block occupies a small patch of the multi-layered image, and the patches are non-overlapping. Principal component analysis is used to reduce the dimensionality of the spatiotemporal texture blocks. Blocks are then analyzed for motion, by ﬁnding the largest eigenvalue of a sliding window covariance matrix at any given temporal location [14]. This leads to segregating sections of a spatiotemporal block as either moving or stationary. Stationary sections of each block are clustered into similar hyperspheres. Final analysis generates most probable changed background regions of interest. The main components of the proposed method are shown in Fig. 1.

2

Multi-camera Homographic Transformations

All images of the same scene are registered to a common ground plane. Homographic transformation is applied to align the camera views and to project common points onto the background plane (synthetic orthoimage). We compute the homographic transformations from a pair of points corresponding to the camera view plane and ground plane. The homographic transformation between a pair of corresponding points (u, v) and (x, y) is deﬁned as ⎛ ⎞ ⎛ ⎞⎛ ⎞ u h11 h12 h13 x ⎝ v ⎠ = ⎝ h21 h22 h23 ⎠ ⎝ y ⎠ (1) 1 h31 h32 h33 1 with a 3 × 3 homographic matrix. An example of constructed orthoimages is shown in Fig. 2(e), and example of background image mask used in computing the homographic matrix is shown in Fig. 2(f).

284

R. Miezianko and D. Pokrajac

Further motion detection processing and background change detection is performed on each camera’s orthoimage (bird-eye view image). All camera orthoimages are spatially aligned, and an overlapping spatial region, Ao , between all camera projections is detected. Only spatiotemporal blocks within the overlapping spatial region Ao are processed. Therefore, we process only areas commonly visible in all camera views.

(a)

(b)

(e)

(c)

(d)

(f)

Fig. 2. Camera projections and ground plane. (a-d) Individual camera views, (e) Superimposed projected view, (f) Ground plane image.

3

Motion Detection

Motion detection based on local variation of spatiotemporal texture proposed by [15] outputs a block motion vector vx,y = [m1 , m2 , ..., mM ], where mi = {0, 1}, (1 for motion), for each spatial block location (x, y) of the projected video frames. Each video frame is divided into non-overlapping blocks of size N × N pixels (we use N = 8 pix in our experiments). If a frame size of PETS 2006 video [1] is 720x576 pixels and N = 8, then there are 90 × 72 distinct spatial block locations. However, we process motion on projected frames and not on original frames as described in previous sections. The number of elements in each block motion vector vx,y is equal to M , the number of frames in the video sequence. An element of block motion vector vx,y may be viewed as a triplet (x, y, i), indicating whether motion was detected at spatial location (x, y) and time i or not. Standard parameters were used when computing block motion

Multi-layer Background Change Detection

285

vectors: C1 = 9, C2 = 3, and u = 0.99. See [14] for the implementation of spatiotemporal motion detection and description of computing the largest eigenvalue. Instead of concentrating on the moving foreground, the emphasis is placed on the detection of changes in the non-moving background block segment in the presence of previously detected motion. Therefore, only block locations with detected motion are considered, and block locations without any detected motion are not taken into account when computing background changes. This eliminates processing of block locations where no motion whatsoever was detected. Any gradual changes in the background will not be considered as eventful when no motion was detected.

4

Spatiotemporal Texture Blocks

Spatiotemporal texture blocks represent a small patch of an image at a speciﬁc spatial location. An image is subdivided into equal size blocks and there is no overlap between blocks. The spatiotemporal texture map Bx,y is created for each image location (x, y) if it lies within the overlapping spatial region Ao as deﬁned in section 2. Each spatiotemporal texture map element bx,y,i is created from 3D block of pixels taken at times i − 1, i, and i + 1, therefore, the size of each texture map element is 3 × N × N pixels. The overall dimension of a spatiotemporal texture map Bx,y is 3 × N × N × K pixels, where K is the number of block motion vector elements vx,y,i indicating no motion, and K ≤ (M − 1). Therefore, a single block of N × N pixels is present in three consecutive texture map elements. This scheme provides more invariance to shadows and to detect rapid motion changes [14]. Spatiotemporal texture maps are created in the same manner for each block location within each camera’s Ao region. Complete spatiotemporal texture maps of a single block location from all cameras is shown in Fig. 3. This ﬁgure depicts a single block (504) (a common spatial location on a ground plane) located where an abandoned luggage event has occurred. A change of texture is noticeable in the middle of the texture map. As this block represents a common spatial location in all cameras, there is a noticeable diﬀerence between them. The cameras are not color calibrated (as evident in Fig. 2), and motion is present in some cameras and not others (Fig. 3(d)). The data dimensionality of the spatiotemporal texture block is reduced with principal component analysis, as was done to detect motion [16]. Therefore, each block element is reduced from L = 3 × N × N pixels to P principal components, with P L. In our experiments L = 192 and P = 10. The ﬁrst 3 principal components of block (504) from Camera 4 are shown in Fig. 4(a), where block elements belonging to non-moving frames are shown in blue, and block elements marked as moving are shown in red. The non-moving block elements do congregate in clusters and moving block elements follow orbital paths between non-moving clusters.

286

R. Miezianko and D. Pokrajac

(a)

(b)

(c)

(d) Fig. 3. Texture maps of a single block (504) of entire video sequence S08-ABANDONED-BAG-2 from (a) Camera 1, (b) Camera 2, (c) Camera 3, and (d) Camera 4. Each column of texture map represents one texture map element b(x,y)=504,i , i = 1 . . . number of frames.

5

Temporal Background Clusters

Once motion is detected within spatiotemporal texture block, the non-moving texture elements are clustered. Initial clusters are build from consecutive texture elements in between moving block elements. In a recursive step, clusters are further merged to account for any temporary occlusions (motion). Within each spatiotemporal texture blocks, a set of background clusters comprising of nonmoving cluster elements is created. If the block has all motion elements, then there will not be a single background cluster; and no detected motion implies one background cluster. Most blocks within overlapping spatial region Ao have one or more background clusters. A cluster is deﬁned as a P − dimensional hypersphere Q. The general equation of a hypersphere Q is (x − c)T (x − c) = r2

(2)

where c is the center of the hypersphere and r its radius. The distance from a point x to Q is deﬁned as d(x, Q) = min d(x, z) z∈Q

(3)

Our distance measure d(x, Q) is a simple Euclidian distance. Speciﬁc frames of a block are marked as changed background, if the nonmoving clusters are diﬀerent from the initial background cluster. If the same block’s frames are marked as changed in all cameras, then this block is marked as changed within the speciﬁed temporal segment. First, a temporal changed background is sought within each block, next a spatial changed background is sought from all neighboring blocks. Temporal background clusters of a single block (504) from Camera 4 are shown in Fig. 4(b), where six distinct background clusters are detected. The

Multi-layer Background Change Detection

(a)

(b)

(c)

(d)

287

Fig. 4. Video sequence S08-ABANDONED-BAG-2, Camera 4, single block (504) analysis. (a) First 3 principal components, blue dots correspond to non-moving block sections, red dots correspond to detected motion block sections (orbits); (b) Detected hyperspherical clusters of non-moving block sections; (c) Extracted hyperspherical clusters; and (d) Extracted hyperspherical clusters with representative texture map section as shown in Fig. 3(d).

same clusters without the principal components are shown in Fig. 4(c) and with corresponding background textures in Fig. 4(d).

6

Experimental Results

The proposed method has been evaluated on publicly available datasets, PETS 2006 [1] and PETS 2007 [2]. These datasets consist of numerous scenarios showing abandoned luggage (or nearly abandoned luggage) and moreover, they are recorded with multiple cameras and overlapping ﬁeld of view. Each dataset contains video from four cameras and description of the background image mask. We used all cameras from PETS 2007 dataset and only three cameras from PETS 2006 dataset (Cameras 1, 3, and 4). A homographic matrix was established for each camera, and a set of orthoimages was constructed. An overlapping region between all camera projection limited the number of spatiotemporal blocks to be created and processed. Motion was detected in each spatiotemporal blocks and background clusters were found. Clustering of neighboring blocks within

288

R. Miezianko and D. Pokrajac

(a)

(b)

(c)

Fig. 5. Video sequence S08-ABANDONED-BAG-2, frame 1300, showing detected background change in orthoimage, original frame, and extracted region of interest in (a) Camera 4, (b) Camera 2, and (c) Camera 3. For additional details see Fig. 6.

Fig. 6. Video sequence S08-ABANDONED-BAG-2, composite frame 1300, showing four projected camera views (left and center columns), and superimposed projected views and background mask (right column). Background change detection blocks are shown in red and motion detection blocks are shown in green.

Multi-layer Background Change Detection

(a)

(b)

289

(c)

Fig. 7. Video sequence S1-T1-C, frame 2720, showing detected background change in orthoimage, original frame, and extracted region of interest in (a) Camera 1, (b) Camera 3, and (c) Camera 4

each frame deﬁned the changed background region of interest. An inverse transformation of the ROI from orthoimage to original image was used to extract a sub-image showing an abandoned luggage, for example. A single frame 1300 from video sequence S08-ABANDONED-BAG-2 [2] shows the orthoimages, corresponding original frames, and extracted changed background ROI in Fig. 5 (only three of four cameras are shown to allow for greater detail). The orthoimages show the common spatial location where background change was detected in all cameras. A composite video frame 1300 from video sequence S08ABANDONED-BAG-2 (Fig. 6) additionally shows motion blocks (green) and changed background blocks (red) over the background mask. The motion block is shown if the same block from all cameras was marked as moving. Another example of changed background detection is shown in Fig. 7 for single frame 2720 from video sequence S1-T1-C [1].

7

Conclusion

In this paper we present an eﬀective method for extracting changed backgrounds for regions observed by multiple overlapping cameras. A multi-layer background is constructed from orthoimages, and spatiotemporal texture blocks are used to detect motion. Detected motion delineated temporally cohesive non-moving regions to extract hyperspherical clusters and detect changed backgrounds. The proposed method allows for detecting background changes in crowded

290

R. Miezianko and D. Pokrajac

environments exhibiting large motion ﬂows and signiﬁcant occlusions. Several improvements are being considered, including tracking and object classiﬁcation, to evaluate extracted ROI as being of interest or not, and to improve forensic analysis.

Acknowledgments D. Pokrajac has been partially supported by NIH (grant #2 P20 RR01647204), DoD/DoA (award 45395-MA-ISP) and NSF (awards # 0320991, #HRD0630388). R. Miezianko initially performed this work at Temple University, Philadelphia.

References 1. PETS2006: Performance Evaluation of Tracking and Surveillance 2006 Benchmark Data (2006), http://www.cvg.rdg.ac.uk/PETS2006/ 2. PETS2007: Performance Evaluation of Tracking and Surveillance 2007 Benchmark Data (2007), http://www.cvg.rdg.ac.uk/PETS2007/ 3. Auvinet, E., Grossmann, E., Rougier, C., Dahmane, M., Meunier, J.: Left-luggage detection using homographies and simple heuristics. In: PETS 2006, pp. 51–58 (2006) 4. del Rinc´ on, J.M., Herrero-Jaraba, J.E., G´ omez, J.R., Orrite-Uru˜ uela, C.: Automatic left luggage detection and tracking using multi-camera ukf. In: PETS 2006, pp. 59–66 (2006) 5. Grabner, H., Bischof, H.: On-line boosting and vision. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 260–267. IEEE Computer Society, Los Alamitos (2006) 6. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR, vol. 1, pp. 511–518 (2001) 7. Velastin, S., Davies, A.: Intelligent cctv surveillance: advances and limitations. In: Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research (2005) 8. Stauﬀer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition., vol. 2, pp. 246–252 (1999) 9. Piccardi, M.: Background subtraction techniques: a review. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 3099– 3104 (2004) 10. Cheung, S.C.S., Kamath, C.: Robust background subtraction with foreground validation for urban traﬃc video. In: EURASIP J. Appl. Signal Process, vol. 2005, pp. 2330–2340 (2005) 11. Yao, J., Odobez, J.: Multi-layer background subtraction based on color and texture. In: VS 2007, pp. 1–8 (2007) 12. Grabner, H., Roth, P.M., Grabner, M., Bischof, H.: Autonomous learning of a robust background model for change detection. In: Proceedings 9th IEEE International Workshop on PETS (2006)

Multi-layer Background Change Detection

291

13. Hall, D., Nascimento, J., Ribeiro, P., Andrade, E., Moreno, P., Pesnel, S., List, T., Emonet, R., Fisher, R.B., Victor, J.S., Crowley, J.L.: Comparison of target detection algorithms using adaptive background models. In: ICCCN 2005: Proceedings of the 14th International Conference on Computer Communications and Networks, Washington, DC, USA, pp. 113–120. IEEE Computer Society Press, Los Alamitos (2005) 14. Latecki, L.J., Miezianko, R., Pokrajac, D.: Activity and motion detection based on measuring texture change. In: Perner, P., Imiya, A. (eds.) MLDM 2005. LNCS, vol. 3587, pp. 476–486. Springer, Heidelberg (2005) 15. Latecki, L.J., Miezianko, R., Pokrajac, D.: Motion detection based on local variation of spatiotemporal texture. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop, vol. 8, pp. 135–141 (2004) 16. Latecki, L.J., Miezianko, R., Pokrajac, D., Gao, J.: Selective hypothesis tracking in surveillance videos. In: Advances in Applied and Computational Mathematics, vol. 1, pp. 261–274 (2006)

The Development and Validation of a Method for 4D Motion Reconstruction of a Left Ventricle Michal Chlebiej1 , Marcin Denkowski2 , and Krzysztof Nowi´ nski3 1

Faculty of Mathematics and Computer Science, N. Copernicus University Chopina 12/18, 87-100 Toru´ n, Poland [email protected] 2 Faculty of Computer Science, Maria Curie-Sklodowska University pl. Marii Curie-Sklodowskiej 5, 20-031 Lublin, Poland 3 Interdisciplinary Centre for Mathematical and Computational Modeling Warsaw University, Pawi´ nskiego 5a, 02-106 Warsaw, Poland

Abstract. Echocardiographic technology has currently reached a stage where it can provide 4D visual data revealing details of the real heart motion. Possibility of spatial reconstruction and quantitative description of such motion became very important task in today’s cardiology. Unfortunately, because of the low quality such image data does not allow precise measurements. To overcome this problem images need to be processed further and moving structures have to be extracted. In this work we present a method for estimating heart motion from 3D echocardiographic image sequence. We also introduce a novel method for quantitative and qualitative validation of motion reconstruction. Keywords: heart motion reconstruction, non-linear registration.

1

Introduction

Echocardiographic examination of the heart, together with techniques based on electrocardiography, is one of the most frequently used methods of the heart examination. Using this modality, vital information about morphology and hemodynamics of the heart could be collected by simple, bedside assessment. Modern 4D ultrasound examinations became relatively inexpensive (when compared to CT or MRI) data acquisition technique. In clinical practice an analysis of data mainly relies on a visual inspection of acquired views and on a physicians’ experience. Such methods lead to a qualitative and subjective assessment without taking into account individual quantitative information included in images. Another problem of the echocardiographic analysis are artifacts from the thorax (e.g. emphysema), which severely limit diagnostic value of echocardiography for 5-10% of patients. To reveal all these vital information and decrease information noise, automated computer-based analysis is highly desirable. Several methods were proposed for the reconstruction of heart motion from 4D ultrasound images. For the left ventricle segmentation surface based methods (using shape and motion constraints) have been proposed to deal with speckle noise in the L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 292–301, 2009. c Springer-Verlag Berlin Heidelberg 2009

The Development and Validation of a Method for 4D Motion Reconstruction

293

echocardiograms [1]. Biomedical models have been investigated for the modeling of cardiac cycle in [2]. In this work we present our algorithm for left ventricle motion reconstruction [3] together with our validation technique based on detection of simulated deformations.

2

Heart Motion Reconstruction

Our reconstruction algorithm consists of several main stages. In the ﬁrst step images are ﬁltered using 4D anisotropic diﬀusion [4]. After that non-rigid registration of a 3D time sequence is performed to obtain a description of a deformation ﬁeld. Non-rigidly registered images are used to compute an average 3D dataset. The next phase consists of a shape and texture based segmentation followed by a triangulation step resulting in a 3D surface model. In the ﬁnal step deformation operator is applied to the surface model in order to recover the time motion of the left ventricle. In the ﬁrst ﬁltering stage (see Fig. 1a) we deal with the time sequence of 3D ultrasound data. We have decided to take also temporal consistency of the acquired data into account in the diﬀusion process. The diﬀusion algorithm has been extended to deal with four dimensional block of data with a time taken as the fourth dimension. As it was presented in [3] such ﬁltering drastically reduces a speckle noise and enhances structure boundaries. The speckle noise may lead to partial disappearing of the image boundaries. The time diﬀusion may help to recover some of the missing boundary parts. In the second stage of our algorithm we describe a motion of the beating heart. It is important to model the motion taking into consideration individual patient speciﬁc anatomical features. In order to achieve realistic motion we have to extract heart kinetics by studying 3D movement of a corresponding anatomy between a reference frame (at time T0 ) and following frames (T1 − TN −1 ). We recover the transformation that aligns the reference frame with all the other frames using intensity based 3D volume registration (see Fig. 1b). Such approach relies on an nonlinear transformation which allows to model local deformation of spatial objects. It is diﬃcult to describe local deformation via parameterized transformations. The method of choice is usually FFD (free-form-deformation) method [5] which is commonly used as a powerful modeling tool for 3D deformable objects. The basic idea of FFD is to deform an object by manipulating an underlying mesh of control points. The manipulated lattice determines the deformation function that speciﬁes a new position for each point of the deformed surface. The number of parameters to be optimized is equal to 3 times number of control points in the lattice. Because of a good localization of B-spline functions optimization procedures can be applied locally. This allows for acceptable running times even for very dense lattices. In this work we use MSD (mean square diﬀerence) similarity function: 1 EMSD (F I, RI, T ) = (IRI (p) − IF I (T (p)))2 (1) N where IRI represents reference image intensities, IF I represents corresponding transformed intensities of the ﬂoating image using transformation T , N is the

294

M. Chlebiej, M. Denkowski, and K. Nowi´ nski

Fig. 1. The ﬂow diagram of our heart motion reconstruction algorithm

The Development and Validation of a Method for 4D Motion Reconstruction

295

total number of overlapping voxels p. In order to deal with large displacements in the registration process, we use a classical incremental multi-resolution procedure. After obtaining the 3D frames of the deformation ﬁeld we are able to describe motion of the whole matter in the volume object. At this point we are able to apply the second step of the denoising procedure using time averaging technique. The deformation ﬁelds are used to generate new datasets elastically aligned with the reference frame T0 . After this step an average dataset from the reference frame and all the deformed datasets are created (see Fig. 1c). The noise located in the datasets is smoothed, while the boundaries of the image structures are preserved. When there is no noise correlation between time frames the process of averaging N frames results in decreasing of noise level √ N times. After denoising procedures it is possible to perform the segmentation step (see Fig. 1d) using the averaged dataset. In our work we decided to use an iterative deformable boundary approach for the segmentation of the ventricular inner surface. The selected method uses energy function consisting of texture based and shape based terms. It is a 3D extension of an algorithm proposed in [6]. Texture based energy term is calculated on the basis of texture intensity energy map which represents the probabilities of the intensity values being consistent with the current segmentation model (updated in every iteration). This term has been formulated as the Shannon’s entropy [7]. The shape energy term takes into account gradient information (revealed using Canny-Deriche’s 3D boundary detection ﬁlter [8]) available in the source image. The main idea of this term is to deform (shrink or expand) segmentation model towards image boundaries. In this segmentation algorithm, starting from an initial estimate, a deformable model evolves under the inﬂuence of the deﬁned energy to converge to the desired boundary of an image structure object. The model deformations are eﬃciently parameterized using the B-spline based Free Form Deformation. After the segmentation procedure we create a triangulated surface representing our object of interest. At this point we are able to reconstruct the cardiac motion by applying the deformation ﬁeld operator. In most heart visualization approaches presented in literature the motion of the left ventricle is described as a set of separated segmented objects. Our method gives a single geometric object deforming in time. Such approach has an important point-to-point correspondence feature which allows interpolation between deformation ﬁeld frames in order to obtain smooth motion. At this point we may use various visualization techniques (see Fig. 1e) in order to enable precise qualitative analysis of the heart’s motion. In the proposed method we obtain only a representation of the inner ventricular surface so we can not calculate all global parameters (i.e. ventricular mass, wall thickening) characterizing cardiac cycle. Using such information we are able to calculate stroke volume, ejection fraction and cardiac output parameters. The motion of the heart can be also characterized in terms of its local variations. It is possible to calculate displacement vectors: total displacement (relative to the reference frame T0 ) and displacement between consequent time frames

296

M. Chlebiej, M. Denkowski, and K. Nowi´ nski

which can be seen as an instantaneous velocity. To be able to visualize the motion occurring on the surface (twisting) we can decompose the instantaneous velocity vectors into tangential and normal components. All of these local variations can be visualized in various ways. In our work we use two kinds of techniques - color and vector based. We can colorize the moving surface according to the length values of displacement vectors. It is strongly useful when dealing with small surface deformations. When the motion is signiﬁcant it is better to visualize vector values using arrows representing length and spatial orientation of moving matter. The last method of motion visualization used in our system is called the line-paths. In this method we select a small set of surface points and visualize the path of their motion during the cardiac cycle using colorized polygons. Colors of line segments represent consequent time frames. Such method enables to estimate the viability of the heart using a single image. In the addition to the line-paths method we may also generate so called activity surface. In this method we can visualize total path length values (in a single cardiac cycle) for every surface point. Using this method on a single static image we may estimate spatial extents of pathological regions.

3

Validation Methods

As for now, there is no possibility to compare the accuracy of our reconstruction method with any other method because there is no test data available that would be commonly used by other researchers. The perfect solution for this task would involve physical phantom deforming in time with its variability precisely deﬁned. Such object could be examined using 4D ultrasound device and allow for precise and unambiguous validation of proposed motion reconstruction method. Unfortunately such solution was out of reach so we had to develop computer based solution. Left ventricle motion simulators based on 13-parameters kinematic model [9] specially dedicated to tagged MRI modality were presented in the literature [10,11]. The transformation used in this model was a modiﬁed version of an aﬃne transformation with an additional twist motion term. In this work we propose similar solution with a torsion transformation deﬁned in a simpliﬁed way. The twist is applied in relation to the center of rotation r along Z axis and its amount is deﬁned by α parameter. For every point p being transformed we add the value of twisting term to the rotation angle RotZ using following formula: pz − rz twist(p, r, α) = α (2) nz where nz is a scaling factor with its value equal to Z component of the source dataset size. Introduced modiﬁcation is responsible for twisting along Z axis of every XY plane in relation to each other. The quality tests of motion detection method were performed on artiﬁcially generated data. Source volumetric datasets were deﬁned using two coaxial ellipsoids. Next, three volumetric datasets were generated with a space between ellipsoids ﬁlled with three different kinds of texture. The ﬁrst one consists of regularly positioned spheres

The Development and Validation of a Method for 4D Motion Reconstruction

297

Fig. 2. 2D crossections of simulated datasets with noise added: regular spheres (left), random spheres (middle), random lines (right)

with equal radiuses and random intensities. The second one used spheres with random both radiuses and intensities. The last one was textured with lines of random position, orientation, length and intensities. In the next step two kinds of noise were added. The ﬁrst one was a zero mean gaussian additive nosie with σ = 8.0. The role of the second kind of noise was to make the image more similar to the one acquired by using 4D echocardiographic device. During acquisition process only 14 of the whole volume can be possessed in a single heart beat. Because of that it is necessary to stitch sub-volumes from 4 cardiac cycles to obtain full volumetric data. To make simulated image more similar to the stitched volume we add nosie diﬀering randomly in σ values (5.0 − 11.0) to 4 sub-volumes deﬁned in spherical coordinates (see Fig. 2). In the next step we deﬁned a set of 5 transformations T1 − T5 (see Table 1) representing motion frames. These transformations were used to generate deformed datasets which were next noised as original images. From that point our goal was to denoise original and deformed image pairs, register them and ﬁnally compare obtained results with the deﬁned ones. We used anisotropic diﬀusion algorithm for noise removal with parameters set as deﬁned in [3]. The RM SE (Root Mean Square) error calculated for all cases showed that anisotropic diﬀusion decreased noise signiﬁcantly and was much better in comparison to the gold standard median ﬁltering. When analyzed object includes straight lines or acute angles median Table 1. Transformation parameters used for deformation of the reference datasets Param RotX ShearX ShearY ScaleX ScaleY ScaleZ T wistZ

T1 0,0 0,0 0,0 1,1 1,0 1,0 0,0

T2 3,0 0,1 0,0 1,2 1,1 1,1 0,1

T3 4,0 0,1 0,1 1,3 1,4 1,15 0,2

T4 6,0 0,1 0,1 1,2 1,2 1,2 0,3

T5 8,0 0,2 0,2 1,1 1,1 1,25 0,4

298

M. Chlebiej, M. Denkowski, and K. Nowi´ nski

Table 2. RM SE errors for all cases dla between original noise free images and images: noise - noised, median - median ﬁltered, anisoDif f - anisotropic diﬀusion ﬁltered

t Ref T1 T2 T3 T4 T5

case1 case2 case3 noise median anisoDif f t noise median anisoDif f t noise median anisoDif f 8,6 9,2 1,3 8,2 4,1 2,9 8,5 15,6 3,8 10,2 8,6 3,5 10,3 5,0 3,5 10,1 12,7 4,8 9,9 7,1 4,5 10,1 4,6 4,1 9,9 10,1 5,6 10,1 7,5 5,2 10,2 4,8 4,7 10,1 10,7 6,6 9,9 7,4 4,8 10,1 4,8 4,4 10,0 10,6 6,3 9,8 7,2 4,6 10,0 4,7 4,2 9,9 10,4 5,9

ﬁltering can lead to further image degradation what is clearly seen in case 3 (see Table 2). In the next step the registration of all volume pairs (each 128×128×128 voxels) was applied. Relying on the resulting deformation ﬁelds it is possible to estimate reliable quantitative assessment of motion detection errors. For every matter point p and given transformation T we calculate translation vector usim (p, T ) = δp(T, p) and using deformation ﬁeld D we calculate the amount of deformation udef (p, D) = δp(D, p). Using such information we can perform detailed error analysis. Because in our work we reconstruct only a motion of inner surface of the left ventricle, error analysis was performed only for inner surface of generated models (using discretized 16000 points). For every volume pairs we calculated displacement errors uerr (pi ) = |usim (pi ) − udef (pi )|. Because in cardiac motion detection normal and tangential deformation components are also very important they should be treated separately in error analysis. For every single case we calculated maximum error - M axError, average error - AvgError, RM SError, and relative error - RM SRel deﬁned as quotient of RM SError and root mean square simulated translation: N 1 2 RM SSim = |usim (pi )| N i=0 N 1 2 RM SError = |usim (pi ) − udef (pi )| N i=0 RM SRel =

RM SError × 100% RM SSim

(3)

(4)

(5)

The results for the most interesting tangential case are presented in Table 3. Additionally we decided to analyze an error in a visual way. For every case we visualized spatial error distribution using colorized surfaces with a shape determined by translation vectors or its components. We also visualized errors using 256-bin histograms, where ﬁrst bin represents the occurrence of zero value error and the maximum error M axError value occurrence is represented by the last

The Development and Validation of a Method for 4D Motion Reconstruction

299

bin. We introduced one more visual method where every image presents two superimposed deformed ellipsoids resulting from simulated and estimated transformations. Using this method it is easy to ﬁnd areas where motion detection algorithm worked well and where it did not. Fig. 3 presents examples of error visual inspection for the most crucial cases using all of the described methods. Table 3. Registration errors of tangential components for all 3 cases T M axError[mm] AvgError[mm] RM SError[mm] RelError[%] case1 T1 0.19 0.06 0.07 13.14 T2 0.90 0.37 0.41 27.62 T3 2.15 0.94 1.06 30.02 T4 3.67 0.93 1.13 19.28 T5 1.88 0.69 0.76 13.94 case2 T1 0.33 0.11 0.12 23.18 T2 2.09 0.64 0.74 49.45 T3 4.71 2.55 2.75 78.19 T4 7.54 3.24 3.72 63.30 T5 5.77 2.41 2.74 50.21 case3 T1 0.18 0.06 0.07 13.56 T2 1.22 0.40 0.46 31.10 T3 7.88 3.55 3.99 113.27 T4 12.57 5.04 5.66 96.35 T5 8.82 4.38 4.75 87.11

After applying described qualitative and quantitative error analysis tools we can summarize the accuracy of our motion detection method. In the ﬁrst case with the highest texture regularity and gradient values the detection was very good, while in the second was slightly worse and the worst in the last case. In case of a simplest transformation T1 involving only scaling operation for all kinds of texture results were almost perfect. For more sophisticated transformations especially the ones with twist motion blended with global rotation the quality of detection depended on texture complexity. For the ﬁrst texture the quality of all matchings (absolute displacements) error values was always under 20% (relative error) what is fully satisfying result (tables not included in this article). Higher errors were obtained using irregular textures. Errors above 50% can be treated as failure of detection. For every case (besides transformations T4 and T5 at the bottom of ellipsoids) the shape was reconstructed with high accuracy. The examination revealed that according to expectations texture irregularity was the most problematic, resulting in signiﬁcant diﬃculties in detection of tangential components of motion vectors (see Table 3). Cases presented in this work were very special because the shape of the left ventricle was highly idealized. When

300

M. Chlebiej, M. Denkowski, and K. Nowi´ nski

Fig. 3. Error visualization for the most complex cases (T 4,T 5). Three top rows present error spatial distributions and histograms (color lookup tables mapped for 0 − max values). The bottom row presents reference ellipsoid and its distortions for all textures created using deﬁned transformation T 5 and calculated deformation ﬁelds.

The Development and Validation of a Method for 4D Motion Reconstruction

301

dealing with real data where signiﬁcant shape irregularities are common, such features usually can guide registration algorithm to proper motion detection.

4

Conclusions

In this paper we presented method for generation of the left ventricle model from the 4D echocardiographic image. We also presented our validation method for motion detection algorithm consisting of error calculations and visual inspections of error distributions using three diﬀerent methods. In our opinion such an approach can be used as reliable tool for detailed analysis. In the near future we plan to work on the segmentation of the ventricular outer surface to enable more accurate description of the cardiac motion.

References 1. Montagnat, J., Delingette, H.: Space and time shape constrained deformable surfaces for 4D medical image segmentation. In: Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 196–205. Springer, Heidelberg (2000) 2. Papademetris, X., Sinuas, A., Dione, D., Duncan, J.: Estimation on 3D left ventricular deformation from echocardiography. Medical Image Anal. 5(1), 17–28 (2001) 3. Chlebiej, M., Mikolajczak, P., Nowinski, K., Scislo, P., Bala, P.: Generation of dynamic heart model based on 4D echocardiographic images. In: Gavrilova, M.L., Gervasi, O., Kumar, V., Tan, C.J.K., Taniar, D., Lagan´ a, A., Mun, Y., Choo, H. (eds.) ICCSA 2006. LNCS, vol. 3984, pp. 394–403. Springer, Heidelberg (2006) 4. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diﬀusion. IEEE Trans. On Pattern Anal. and Mach. Intell. 12(7), 629–639 (1990) 5. Sederberg, T., Parry, S.: Free form deformation of solid geometric models. Computer Graphics 20(4), 151–160 (1986) 6. Huang, X., Metaxas, D., Chen, T.: MetaMorphs: Deformable shape and texture models. In: IEEE Comp. Vision and Patt. Recog., Washington, D.C. (June 2004) 7. Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal 27, 379–423, 623-656 (1948) 8. Monga, O., Deriche, R., Malandain, G., Cocquerez, J.P.: Recursive ﬁltering and edge tracking: two primary tools for 3D edge detection. Image and Vision Computing 9(4), 203–214 (1991) 9. Arts, T., Hunter, W.C., Douglas, A., Muijtjens, A.M.M., Reneman, R.S.: Description of the deformation of the left ventricle by a kinematic model. Biomechanics 25(10), 1119–1127 (1992) 10. Waks, E., Prince, J.L., Douglas, A.S.: Cardiac motion simulator for tagged MRI. In: Proceedings of the IEEE Workshop on Mathematical Methods in Biomedical Image Analysis, June 1996, pp. 182–191 (1996) 11. Chandrashekara, R.: Analysis of Cardiac Motion Using MRI and Nonrigid Image Registration, PhD thesis, University of London (2004)

Estimation of Eye Blinking Using Biopotentials Measurements for Computer Animation Applications Robert Krupi´ nski and Przemyslaw Mazurek Szczecin University of Technology Chair of Signal Processing and Multimedia Engineering 26-Kwietnia 10, 71-126 Szczecin, Poland {robert.krupinski,przemyslaw.mazurek}@ps.pl http://www.media.ps.pl

Abstract. In the paper, the estimation algorithm of eye blinking for a computer animation system using biopotentials occurring on an actor’s face is considered. The measuring system incorporates minimal 3– electrodes conﬁguration, which is relatively cheap solution. The proposed algorithm allows both the measurement of eyes orientation and detection of eyes blinking. Keywords: Computer Animation, Motion Capture, Estimation, Signal Processing, Electrooculography.

1

Introduction

Modern computer animation methods are mainly based on two techniques: animation with keyframes and motion capture systems. The former one is time consuming and requires a skilled animator or the group of animators. The latter one is partially an automatic process, where the actor’s movement is captured and then mapped to an animated avatar. Both solutions are applied and in some cases two methods are combined to overcome the constrains of each. The special case of computer animation is the realistic animation of a human face, which is the most complex, because of the wide range and nuances of possible mimical facial movements. Therefore, it demands the particular care in the animated details. The classical method with animators takes time to receive the desired eﬀect. The automatic process of the motion capture has the limited application. Particularly, the motion capture methods are restricted to the recording of actor’s movement with stationary head or the rough recording of mimical facial movements with priority to actor’s movements on the stage. These limitations mainly come from the resolution of cameras, the scene geometry, the number of cameras, the dimension of markers, the noise of converters, etc. Additionally, it should be mentioned the safety issues, which restrict the application of some devices, ex., the helmet with cameras mounted on the actor’s head to monitor mimical facial movements during action shots. The recording L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 302–310, 2009. c Springer-Verlag Berlin Heidelberg 2009

Estimation of Eye Blinking Using Biopotentials Measurements

303

of mimical facial movements during for the actor’s unconstrained movements on the stage is very limited. Contemporary, the installation from a few to several dozen markers on the face for more advanced motion capture systems is possible. It is not applicable from the safety point of view to ﬁx markers on the eyes, around the eyes and on the eyelids, because of the risk of eye damage. Consequently, the necessity for alternative technical solution overcoming these restrictions appears. The recording of biopotentials on the human face through the biomeasurement system could be one of the solutions, especially, a technique for measuring the resting potential of the retina (electrooculography; EOG) and a technique for evaluating and recording the activation signal of muscles (electromyography; EMG). In simpliﬁcation, the measured values of EOG depend on the orientation of the eyeballs and the EMG values depend on the tension of the face muscles. These signals are collected simultaneously, thus, their separation is required. This idea was presented in the patent [2] and the practical realization in the movie Beowulf [1]. The results of EOG system cooperating with the motion capture one are outlined on the Beowulf DVD 2’nd disc [1]. The application of all possible capabilities of biopotential measurements on a human face was not achieved until now, even more, the partial solutions do not fully use the certain types of signals. Figure 1 depicts the applied EOG system in recording, transferring and processing biosignals in the computer graphics of a face animation.

Biosignals Acquistion System

Biosignals Processing and 3D Visualization Actor

Avatar

Fig. 1. EOG system in recording, transferring and processing biosignals in the computer graphics of a face animation

Mimical facial movements of a human face are highly complex [13], where the estimation of all parameters is desired, at the same time, very diﬃcult. In the paper, the algorithm for the 3–electrodes system (and one additional reference electrode), which is minimal approach to the estimation of eyeballs orientation and eyelids blinking, is considered. Other electrodes placements as

304

R. Krupi´ nski and P. Mazurek

well as greater number of electrodes are possible, but the presented solution could be characterized by the implementation and application simplicity. In Section 2, the applied 3–electrodes system is described. In Section 3 the detection algorithm is discussed.

2

EOG Acquisition System

The acquisition of biosignals is a complicated process, because of the existence of interference signals, such as: power supply, electromagnetic waves, electrostatic discharge and what’s more, the movements of an actor can cause the change of electrodes’ impedance, if they are not correctly ﬁxed. The feature of biopotentials is that the recording of biosignals contains not only one type of signal (for instance, EOG), but also other ones. Therefore, the subject of signals separation is developed from many years and, specially, the removal of EOG from EEG is emphasized [3,4,5,6,7,8,9,10,11,12]. There is applied diﬀerent electrodes placement for these systems. The measured biosignals are transferred to PC for further processing and the estimation of mimical facial movements. The presented system, in the paper, and the acquisition board were designed and prepared by the authors. USB 2.0 transfer from the biosignal acquisition system to PC is applied, which assures appropriate real–time data rate for transferring data without a loss.

3

Algorithm for Separation of Eye Blinking and EOG

Nowadays, the estimation of mimical facial movements for animated avatar can be achieved in real time or as oﬀ–line processing. The collected biosignals have to be converted to the appropriate values of morphing goals of an animated character. The algorithm consists of two fundamental parts. In the ﬁrst one, the estimation of eyes blinking signal is performed. Subsequently, in the second part, the separated eyes blinking signal undergoes further noise reduction and the detection of each eye blinking. The task of estimator is to determine the eyes blinking signal, because this signal is intermixed with the EOG signal. Both of them are incorporated in the computer animation. The EOG signal is measured diﬀerentially between two electrodes with a common one. The placement of electrodes on an actor’s face is depicted in ﬁgure 2. In this conﬁguration, the collected biosignals (Fig.3) can be almost directly applied as the input values for morphing goals with minimal processing (under no blinking constraint). In a real case, the correction of signals is required because of the personal characteristics and the precision of electrodes placement. The most common technique is a calibration with the predeﬁned session of eye movements (e.g. constellation). The constellation pattern is depicted in ﬁgure 4. The recorded signal for both electrodes is shown in ﬁgure 3 and corresponding the reconstructed pattern in ﬁgure 5. Figure 5 was constructed by displaying

Estimation of Eye Blinking Using Biopotentials Measurements

305

Fig. 2. Electrodes placements on an actor’s face

18

20

22 24 time [s]

26

28

18

20

22 24 time [s]

26

28

Fig. 3. Recorded biosignals in a calibration process (two channels)

EOG signals from two channels: OX axis represents one channel, where OY axis another one. Now, it can be easily mapped from the unknown measured signals to the appropriate eyeball orientation. This device was designed to measure the signals in the order of EEG values (uV). The EOG signals are even at the level of mV, therefore, the device assures an appropriate quality for measuring EOG. In the case of eyes blinking, EOG signal is disturbed by the impulses (Fig. 6). Every impulse corresponds to a single eye blink. The task of estimator is to separate the eyes blinking signal and the signal representing the orientation of the eyeballs. The presented algorithm for separation consists of few blocks (Fig. 7) and can be deﬁned by the following equation. w = lpf (x) − med2 (med1 (lpf (x)))

(1)

where: x – an input biosignal, w – the output signal from the ﬁrst stage, lpf – a lowpass ﬁlter, med1 – the median ﬁlter no.1, med2 – the median ﬁlter no.2. The high frequency noise, especially coming from the power supply and ﬂuorescent lights, is suppressed by the input lowpass ﬁlter. The size of the median

306

R. Krupi´ nski and P. Mazurek

a

b

c

d

e

f

g

h

i

Fig. 4. Constellation pattern of eye movements for a calibration

right−down down right left−down

left up−right

up

up−left

Fig. 5. Calibration constellation

window was determined experimentally and was chosen a bit smaller than the minimal interval of eyes blinking produced by the actor. Therefore, med1 removes the impulses. Since there is no possible to eliminate all interference, the second ﬁlter med2 is applied to reduce distortion missed by med1. Fig. 8 presents the estimated EOG signal overlaid on the input signal disturbed by the impulses of eyes blinking.

Estimation of Eye Blinking Using Biopotentials Measurements

20

22

24

26

28

30

32

307

34

time [s]

Fig. 6. EOG signal disturbed by the impulses of eyes blinking (single channel)

Fig. 7. Presented algorithm for the reparation of the eyes blinking signal and the signal representing the orientation of the eyeballs

The output signal from med2 is subtracted from the lowpassed signal, thus, the estimated eyes blinking signal is depicted in ﬁgure. 9. Although, these impulses could have diﬀerent shapes, the proposed algorithm extracts them eﬀectively. This signal is an additive one for such electrodes placement, albeit, it can diﬀer depending on the position. The detection stage is depicted in ﬁgure 10. The clipping part eliminates the interference below zero. In the following step, there is required to detect impulses with high energy. Firstly, the signal is ﬁltered with a match ﬁlter. The impulse response is derived in the calibration process from several eyes blinking signals. Fig. 11 shows the estimated response model of eyes blinking signal. The peak detector returns the sum of ﬁve neighboring samples n − 2 . . . n + 2 if a center sample n is detected as a peak one. It can be deﬁned by the following equation ⎧ n+2 ⎫ ⎪ ⎪ ⎪ ⎪ ⎪ zi : zn zn−1 ∨ zn zn+1 ⎪ ⎨ ⎬ vn = i=n−2 (2) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ 0 : otherwise

308

R. Krupi´ nski and P. Mazurek

input signal estimated EOG

20

22

24

26

28

30

32

34

time [s]

Fig. 8. Estimated EOG signal overlaid on the EOG signal disturbed by the impulses of eyes blinking

20

22

24

26

28

30

time [s]

Fig. 9. Estimated eyes blinking signal

Fig. 10. Detection stage

32

34

Estimation of Eye Blinking Using Biopotentials Measurements

309

0.035 0.03 0.025 0.02 0.015 0.01 0.005 0 −0.005 0

10

20

30

40

50

60

70

n

Fig. 11. The response model of eyes blinking signal

1 0.5 0 20

22

24

26

28

30

32

34

time [s]

Fig. 12. Detected eyes blinking impulses

where zn is a sample value from the output of the match ﬁlter. The size of this window depends on the sampling frequency and the width of impulse (the window is narrower than the width of impulse). Peaks with higher energy are separated from the noise level by the application of threshold. The output of the last stage for the presented algorithm is depicted in Fig. 12. These impulses can be further applied to animate the eyelids of an avatar.

4

Conclusions

The presented algorithm of eyes blinking detection for computer animation system is based on a median ﬁltering for the separation of two combined signals: EOG and eyes blinking signals. The input signals are biopotenetials occurring on an actor’s face, which were recorded by the acquisition system designed and prepared by the authors. The system is based on the 3–electrodes conﬁguration, which is relatively cheap solution. The output of the algorithm produces the time points at which the actor’s eyes blinking appeared. These values can be applied to animate the eyelids of an avatar. The intermediate stage of the

310

R. Krupi´ nski and P. Mazurek

algorithm separates the estimated EOG signal, which can drive the orientation of the eyeballs of a virtual character.

Acknowledgments This work is supported by the MNiSW grant N514 004 32/0434 (Poland). This work is supported by the UE EFRR ZPORR project Z/2.32/I/1.3.1/267/05 ”Szczecin University of Technology – Research and Education Center of Modern Multimedia Technologies” (Poland).

References 1. E.O.G Beowulf DVD 2’nd disc, Warner Brothers (2008) 2. Patent US., International Publication Number WO 2006/039497 A2 (13.04.2006) 3. Schlogl, A., Keinrath, C., Zimmermann, D., Scherer, R., Leeb, R., Pfurtscheller, G.: A fully automated correction method of EOG artifacts in EEG recordings. Clinical Neurophysiology 118, 98–104 (2007) 4. Croft, R.J., Barry, R.J.: EOG correction: a new aligned-artifact average solution. Electroencephalography and clinical Neurophysiology 107, 395–401 (1998) 5. Liu, T., Yao, D.: Removal of the ocular artifacts from EEG data using a cascaded spatio-temporal processing. Computer methods and programs in biomedicine 8(3), 95–103 (2006) 6. Behrens, F., Weiss, L.-R.: An automated and modiﬁed technique for testing the retinal function (Arden test) by use of the electro-oculogram (EOG) for clinical and research use. In: Behrens, F., Weiss, L.-R. (eds.) Documenta Ophthalmologica, vol. 96, pp. 283–292. Kluwer Academic Publishers, Dordrecht (1999) 7. Croft, R.J., Barry, R.J.: Issues relating to the subtraction phase in EOG artefact correction of the EEG. International Journal of Psychophysiology 44, 187–195 (2002) 8. Puthusserypady, S., Ratnarajah, T.: Robust adaptive techniques for minimization of EOG artefacts from EEG signals. Signal Processing 86, 2351–2363 (2006) 9. Croft, R.J., Barry, R.J.: EOG correction: comparing diﬀerent calibration methods, and determining the number of epochs required in a calibration average. Clinical Neurophysiology 111, 440–443 (2000) 10. Hironaga, N., Haruhana, K., Liu, L.C., Fenwick, P.B.C., Ioannides, A.A.: Monitoring of eye movement and its use for artifact elimination. International Congress Series 1270, 134–137 (2004) 11. Croft, R.J., Barry, R.J.: Removal of ocular artifact from EEG:a review. Neurophysiol. Clin. 30, 5–19 (2000) 12. Croft, R.J., Barry, R.J.: EOG correction: comparing diﬀerent calibration methods, and determining the number of epochs required in a calibration average. Clinical Neurophysiology 111, 440–443 (2000) 13. Fleming, B., Dobbs, D.: Animating Facial Features & Expressions. Charles River Media (1999)

Approximation of Subtle Pathology Signs in Multiscale Domain for Computer-Aided Ischemic Stroke Diagnosis Artur Przelaskowski1, Rafal J´ o´zwiak1, Grzegorz Ostrek1 , and Katarzyna Sklinda2 1

Institute of Radioelectronics, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland [email protected] 2 Department of Radiology CMKP, CSK MSWiA Woloska 137, 02-507 Warsaw, Poland

Abstract. Computed understanding of CT images used for aided stroke diagnosis was the subject of reported research. Subtle hypodense changes of brain tissue as direct ischemia signs was estimated and extracted to improve diagnosis. Fundamental value of semantic content representation approximated from source images was studied. Nonlinear approximation of subtle pathology signatures in multiscale domain was veriﬁed for several local bases including wavelets, curvelets, contourlets and wedgelets. Diﬀerent rationales for best bases selection were considered. Target pathology estimation procedures were optimized with a criterion of maximally clear extraction of diagnostic information. Visual expression of emphasized hypodenstity was veriﬁed for a test set of 25 acute stroke examinations. Suggested methods of stroke nonlinear approximation in many scales may facilitate the early CT-based diagnosis. Keywords: Image nonlinear approximation, multiscale image representation, computer aided diagnosis, ischemic stroke.

1

Introduction

The clinical diagnosis should allow identiﬁcation of patients with hyperacute infarctions and selection of treatment, assessment of penumbra (tissues at risk of infarction), determination of etiology and follow-up of therapy and its possible complications. As therapeutic capabilities improve, brain imaging strategies are playing an increasingly important role in the initial evaluation of patients with acute stroke. Brain imaging ﬁndings, including the size, shape, form and location of ischemic changes, and vascular distribution of the infarction, as well as the presence of bleeding, aﬀect treatment decisions. In addition, information about the possible degree of reversibility of ischemic injury, diﬀerentiating between infarct and penumbra regions, intracranial vessel status, and cerebral hemodynamic status may be obtained by modern imaging studies [1]. The most commonly obtained brain imaging modality is noncontrast CT, but individual L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 311–321, 2009. c Springer-Verlag Berlin Heidelberg 2009

312

A. Przelaskowski et al.

centers are able to obtain MRI with eﬃciency equal to that of CT for patients without MR contraindications. However, additional research is required because of imaging technologies limitations - none medical imaging modality is totally eﬀective for acute stroke diagnosis. CT is relatively insensitive in detecting acute and small cortical or subcortical infarctions, especially in the posterior fossa [2]. Further studies are needed to determine the signiﬁcance of early infarct signs and their role in treatment decision making. A CT image of the brain in acute stroke patients is not self-evident. Reading of CT needs training and additional knowledge about the physical conditions of image contrast distribution with noise and artifacts-caused limitations [3]. Physicians’ ability to reliably and reproducibly recognize the early CT changes is variable. Computer-aided acute stroke diagnosis was mostly based on unenhanced CT examinations [4,5] according to concept of sensing technologies with semantic image understanding. Sensing technologies are aimed at extracting information from events that are beyond the reach of human sensory ability [6]. Because of limited range of human senses and limitations of medical imaging abilities the images are intensively processed to reveal the information content. 1.1

Disease Imaging

Stroke is a clinical manifestation of diverse pathologies, deﬁned as a syndrome characterized by rapidly progressing clinical signs and/or symptoms of focal loss of cerebral function lasting more than 24 hours or leading to death, with cause of vascular origin. Nowadays it is the third major reason of death after cardiac and oncologic diseases. Ischemic stroke is caused by some disruption of cerebral blood ﬂow (CBF) and subsequent tissue damage. A progressive 2-4% increase in brain tissue water within 4 h of middle cerebral artery (MCA) occlusion causes the lowering of brain attenuation coeﬃcients in acute ischemia, which leads to a discrepant decrease of about 1.3-2.6 HU for 1% change in water content [3]. Thus early changes with ischemia may vary within the limited range up to 10 HU within 4 h after the onset depending on cerebral infarct case, discrepant patient characteristics, bone artifacts, non-optimum scanning and acquisition conditioning. Therefore, the hypodense changes are slight, ill deﬁned, not well outlined. The CT ﬁnding of parenchymal hypodensity is highly speciﬁc for irreversible tissue damage. The hypodense area is a direct, convincing infarct sign deﬁned as any area in the brain with density lower than normal surrounding brain tissues. Moreover, early indirect ﬁndings, like obscuration of gray/white matter diﬀerentiation and eﬀacement of sulci, or ”insular ribbon sign”, may be noticed instead. Although CT examination plays a crucial role in the evaluation of stroke patients, it is not suﬃcient enough for extraction of ischemic tissue signs (see ﬁg. 1). Many infarcts do not emerge on CT until hours after the onset of stroke and hypodense changes may not be seen on the initial CT-scans during the hyperacute phase of stroke (0-6h). Additionally, the subtle hypodense changes are often masked due to artifacts, noise and other tissue abnormalities. The attenuation coeﬃcients of brain

Approximation of Subtle Pathology Signs in Multiscale Domain

313

Fig. 1. CT successive scans in a patient with ischemic stroke: 11h after stroke onset, without any visible hypodense changes (up); 4 days after the onset with clearly visible - indicated with arrows - hypodense area (down)

parenchyma vary, mainly due to the diﬀering thickness of the cranial vault. M. Bendszus et al. [4] found inter-individual diﬀerences, i.e. bone artifacts, of up to 14 HU in brain parenchyma at comparable scan levels. The CT number in noisy areas may vary in the range of 4HU. Thus assistance of computer-aided diagnosis (CAD) support to extract subtle ischemic signs was optimized. The methods used in our research were based on a concept of nonlinear approximation of subtle pathology signatures in multiscale domain to estimate hypodense changes. Applied criterion of assistance optimization was maximally clear semantic content extraction. 1.2

Disease Computing

Nowadays CAD tools were developed to improve interpretation process by generating complementary information described as a “second opinion” to be veriﬁed by experts. The expert knowledge is completed by computationally extracted diagnostic content which was inaccessible for human senses [7]. Conversion of source image that is deﬁned in a form of arrays of digits to its semantic representation, i.e. extracted in readable form diagnostic information with speciﬁc meaning of medical terms, plays a crucial role for eﬀective CAD design. A process of discovering the meaning of distinguishable picture objects, exploiting context, domain and common sense knowledge, and receiver operation characteristic is necessary. Hence detection of subtle pathology signs and determining the semantics of speciﬁc image component attributes or features, i.e. disease signatures, accurately gives additional diagnostically important information. Image processing in source data domain often fails due to complex nature of medical images. In a case of masked, subtle pathology signs, the key problem is the selection of new data representation which allows for eﬃcient semantic

314

A. Przelaskowski et al.

content analysis. Image data are transformed to new domain in order to reduce redundant objects and inter-object description, diversify important signal features from useless background (i.e. noise, artifacts, structures unimportant for diagnosis), emphasize natural proprieties of particular structures, simplify pathology model construction and consequently facilitate extraction of diagnostically useful local content. Subtle pathology signs modeling in suitable domain allows for usable signal estimation and identiﬁcation even in case of high image data masking and adverse image acquisition conditions. Due to approximate description of most important and typical features, subtle pathology models indicate particular image regions where hidden information should be uncovered. For the sake of approximate nature of semantic pathology models much attention should be paid to eﬃcient approximation methods based on diﬀerent forms of multiresolution image representation.

2

Methods of Disease Signature Extraction

Usually the searched subtle pathology information expresses as a local distortion in speciﬁc signal distribution. It can be modeled as the presence of signal function singularities, i.e. subsets of domain at which a given function behaves abnormally (i.e. it is discontinuous, non-diﬀerentiable or even undeﬁned, etc.). Singularities and irregular structures often carry essential information in a signal. In case of 1D signal we talk about point singularities (with dimensionality equals to 0), for images (2D signals) an important family of linear or curve singularities corresponds to local edges and structure boundaries (with dimensionality equals to 1). The estimation of useful signal (target function) in the presence of noise and unusual structures is possible by ﬁnding a representation that discriminates the target function features from the noise, approximation of diagnostic information in adaptive strategies depending on the signal properties and attenuation of the noise while preserving or even emphasize the disease signatures. For eﬀective target signal extraction, especially a non-linear approximation in an orthonormal basis plays an important role. Useful signal features are approximated with just a few vectors selected adaptively within a basis [8]. The key issue is optimal basis selection, which should be able to accurately describe all diagnostically important signal features within as small numbers of basis elements as possible. A wavelet multiscale representation focuses on localized signal structures with a zooming procedure that progressively reduces the scale parameter. Local signal regularity is characterized by the decay of the wavelet transform amplitude across scales. Singularities are detected and interpreted by following the wavelet domain local maxima at ﬁne scales. Adaptive thresholding of wavelet coeﬃcients make such representation extremely useful for medical imaging applications [9].

Approximation of Subtle Pathology Signs in Multiscale Domain

2.1

315

Nonlinear Approximation

The fundamental problem of approximation theory is to resolve a possibly complicated target function by simpler, easier to compute basis functions called the approximants. Formally, an approximation process can be simply deﬁned in a Hilbert space with inner product < ., . > and norm .H . Let {ϕi }i=1,2,... be an orthonormal basis for complete H . Each signal f ∈ H can be decomposed in this +∞ basis f = i=1 ai ϕi with the coeﬃcients of orthogonal expansion ai =< f, ϕi >. For linear approximation of f ∈ H, we use orthogonal projection of f over the ∼ N linear space HN := span{ϕi : 1 ≤ i ≤ N } and get f (N ) = i=1 ai ϕi . In case of nonlinear approximation we replace HN by the nonlinear space ≈ ≈ AN for expression of f ∈ H as f (N ) = i∈Λ ai ϕi where Λ ⊂ N is a ﬁnite set of indexes with the cardinality #Λ = M ≤ N In linear approximation the N basis vectors are selected a priori while for nonlinear approximation expansion basis is improved by choosing M terms depending on the meaningful features of approximated f . ≈ 2 Nonlinear approximation error ε 2N (f ) = i∈Λ / |ai | is minimal and decays as M increases if Λ corresponds to the M vectors that best correlate to f , i.e. having the largest coeﬃcients of the expansion |ai |. For the set of indexes Λr = {ik }k=1,2,..,M sorted according to decreasing order of the corresponding coeﬃcients - |aik =< f, φik > | ≥ |aik+1 |, 0 < k < M we have f approximation ≈ M +∞ ≈ f (M) = k=1 aik ϕik with the error e 2N (f ) = k=M+1 |aik |2 . The decay rate of ≈2 ε N (f )

as M increases is a measure of approximation eﬃciency. Consequently, the basis functions of approximation space should represent the most important, precisely characterized and distinguishable features of the target function. Wavelet-based decomposition. localizes signal energy in a joint space-scale domain. The source data are represented by their elementary constituents across scale using compactly supported small waves. A wavelet family basis {ψm,n }(m,n)∈Z2 is obtained by shifts and dilations of the mother function ψ, in form: ψm,n (t) = 2−m/2 ψ(2−m t − n), and characterized by the notion of local function regularity, degrees of smoothness, symmetry, limited size of the support and the orthogonality conditions. The waveforms are well concentrated in time and in frequency constituting time-frequency atoms ﬂexibly adjusted to target function non-stationary features. They are able to eﬀectively represent function discontinuities, determine whether a quick signal exists, and if so, can localize it [10]. Wavelets are tailor-made for nonlinear approximation because of fast and simple computation, simpliﬁed characterization of approximation spaces based on almost unconditional function classes with controlled regularity and transparent strategies of basis selection and target function estimates. A nonlinear approximation in a wavelet orthonormal basis deﬁnes an adaptive grid that reﬁnes the approximation scale in the neighborhood of the signal singularities. If the target function is smooth on a region of interests a coarse scale of dyadic decomposition is used. For regions where the target function is less smooth the wavelet functions of higher resolution scales are used for the approximation. More accurate

316

A. Przelaskowski et al.

Fig. 2. The indicated contour of CT hypodense area (a) expanded in wavelet basis (b), contourlet basis (c) and curvelet basis (d). Wavelet contour representation consists of large number of diﬀerent scale coeﬃcients with square-shape along the contour, affected by smooth edge. 2D wavelet-based representation with only a few coeﬃcients with diﬀerent elongated shapes and in a variety of directions following the contour is much more eﬀective.

allocating terms in a nonlinear strategy depend on signal energy distribution across scales and subbands. If f is piecewise regular then few wavelet coeﬃcients are aﬀected by isolated discontinuities and the error decay depends on the uniform regularity between these discontinuities. For smooth wavelet basis with p vanishing moments we ≈ have ε 2N = O(M −2α ) for uniformly Lipschitz function f with α < p . The more regular target function means faster error decay what corresponds to regular nature of the subtle hypodense changes and suggests denoising of CT signal before eﬀective estimation. Generalization for two and more dimensions is possible and fruitful but simple tensor product extension of wavelets to two dimensional (2D) basis is not optimal for representing images consisting of structures with smoothly varying greyvalues, separated by smooth boundaries (in medical imaging, detecting and enhancing such boundaries is of prime importance). Separable wavelet basis in L2 (R2 ) is not ﬂexible enough to represent speciﬁc image singularities. The tensor product construction of 2D wavelet basis uses three basic wavelet families: ψ (1) (x, y) = φ(x)ψ(y), ψ (2) (x, y) = ψ(x)φ(y), ψ (3) (x, y) = ψ(x)ψ(y), and scaling functions φ2 (x, y) = φ(x)φ(y). It gives wavelet image decomposition that detects isolated edge points rather than edges and the smoothness of edges is not reﬂected adequately, i.e. the edges are approximated ineﬃciently with a crucially large number of the expansion coeﬃcients. More eﬃcient nonlinear image approximation may be constructed with scaled basis functions whose support shape can be adapted to the orientation and regularity of the object contours. It refers to non-separable wavelet kernels called 2D wavelets with anisotropic dilations, rotations and translations of mother function ϕm,n,θ (·, ·) oriented by θ . The basic concepts of 2D wavelets use adaptive geometry-based approaches such as wedgelets (beamlets, platelets etc.), or directional frames such as ridgelets, curvelets, contourlets [11]. The beneﬁts of 2D wavelet representation of hypodense area contour were presented in ﬁg. 2.

Approximation of Subtle Pathology Signs in Multiscale Domain

317

Wedgelets were constructed as a base of characteristic functions of wedgeshaped sets obtained by splitting dyadic squares of entire image partition Q along straight lines [12]. Each element of dyadic partition q ∈ Q is split into at most two wedges, q = w1 ∪ w2 where wi ∈ W along linearly approximated ≈

local edge. The wedgelet approximation of the image f by f is minimizing of the ≈ ≈2 functional Hλ,f (f , W ) = f − f 2 + λ|W | where error decay for f is O(M −2 ) with requirement that the number of angles increases for ﬁner scales. Curvelets and contourlets are conceptually closer to wavelets. Both provide an essentially optimal representation of typical target function f which is C2 (twice continuously diﬀerentiable) except for discontinuities along C2 curves. The ≈ nonlinear approximation error obeys ε 2N (f ) = O(M −2 (log M )3 ) and is optimal in the sense that no other representation can yield a smaller asymptotic error with the same number of terms. The curvelet transform is a multiscale pyramid corresponding to family of functions with many directions and positions at each length scale, and needleshaped elements at ﬁne scales. This pyramid contains elements with a very high degree of directional speciﬁcity. In addition, the curvelet transform is based on a certain anisotropic scaling principle which is quite diﬀerent from the isotropic scaling of wavelets. Image approximation uses ﬁxed system of building blocks and is performed by expanding the input in the curvelet frame with coeﬃcients selection. Approximation eﬃciency critically depends on type of scaling, and the sampling of the decomposition parameters. First generation curvelets were based on ridgelets, i.e. continuous functions in the form of ρm,n,θ (x, y) = m−1/2 ψ((cos(θ)x + sin(θ)y − n)/m). Ridgelet decomposition is a form of wavelet image analysis in the Radon domain [13]. It solves the problem of sparse approximation of smooth objects with straight edges. But for ﬁner approximation of curved edges one can use a suﬃcient ﬁne scale to capture curves as almost straight edges. Thus curvelet transform was based on multiscale ridgelets combined with a spatial bandpass ﬁltering operations and subbands splitting into blocks. Second generation curvelets are deﬁned directly in via frequency partitioning without ridgelets. Digital curvelet image decomposition is based on unequally-spaced fast Fourier transforms or the wrapping of specially selected Fourier samples [14]. Contourlet image transform was initially described in the discrete domain as multiresolution and multidirectional expansion with contour segments derived from non-separable, pyramidal directional ﬁlter banks [15]. Contourlets-based sparse representation for two-dimensional piecewise smooth signals that resemble images satisfy the anisotropy scaling relation for curves. 2.2

Diagnostic Content Estimation

The best nonlinear approximation may be calculated by applying thresholding function with zeroing the expansion coeﬃcients of magnitude less than the threshold value τ . For the source signal s = f + η (with masking background (s) η) we can estimate target function f by selection of the coeﬃcients ai with

318

A. Przelaskowski et al.

N (s) thresholding function d(·) as follows: fˆ = i=1 d(ai )ϕi . Extremely useful for subtle hypodenstity estimation is semisoft thresholding function because estimated function is as regular as target function. In this case we have: ⎧ (s) (s) for |ai | > τ2 ⎪ ⎨ ai (s) t) (s) τ2 (|ai |−τ1 ) (s) d(semisof (ai ) = sign(a(s) for τ1 ≤ |ai | ≤ τ2 τ1 ,τ2 i ) τ2 −τ1 ⎪ ⎩ (s) 0 for |ai | < τ1 The thresholds are matched adaptively considering coeﬃcients distribution across scales, subbands and orientations keeping only wavelet domain transients coming from the target function. A wavelet thresholding is equivalent to estimating the signal by averaging it with a kernel that is locally adapted to the signal regularity. Good estimation results for curvelets basis were achieved with waveshrink (s) (s) ai (wavesh) (s) estimator deﬁned by dτ (ai ) = · (ai − τ )+ for complex coeﬃa(s) i (s) cients with magnitudes ai > τ . Proposed CAD method was based on nonlinear approximation of subtle hypodense signs in multiscale domain. In order to reveal unrecognized pathology from acute CT examinations and conﬁrm the presence of ischemia brain damage, the extraction and meaningful visual expression of hidden (i.e. invisible in typical assessment procedure) presence of tissue ischemia in cerebrum structures and features was realized according to the following algorithm: 1. Initial stage of image conditioning with segmentation of stroke-susceptible regions of brain tissues (see ﬁg. 3a): adaptive region growing and thresholding methods and smooth complement of segmented diagnostic areas were used. 2. Essential stage of subtle hypodensity signs extraction: a) target signal estimation with wavelet (tensor or 2D) basis for nonlinear approximation with thresholding to estimate diagnostic information by denoising; b) signal emphasizing with especially adjusted wavelet (2D or tensor) basis and additional nonlinear approximation controlled by semantic content models; 3. Final stage of hypodensity visual expression: display arrangement with greylevels quantization and contrast enhancement with diﬀerent forms of visualization (see ﬁg. 4), according to observer suggestion were used.

3

Experimental Conclusions

The ability of subtle hypodense signs estimation and expression was veriﬁed by monitoring ischemic stroke detection ability. Its eﬀectiveness mostly depends on target image features approximation (ﬁg. 3) and visual expression of lower density tissue. The experiments were concentrated on the eﬀects of as clear as possible target hypodensity revealing in the reconstructed forms (ﬁg. 4). Test set consists of 25 ischemic stroke cases of the patients aged 28-89, mean 69 years. The test examination were selected precisely from database set of over

Approximation of Subtle Pathology Signs in Multiscale Domain

319

Fig. 3. Wavelet-based approximation of hypodense changes: a) the approximation of the segmented diagnostic area (left) in successive scales (from coarse to ﬁne); b) ‘silent’ case (left), two coarse approximation with extracted hypodensity in coarse wavelet and curvelet scale, respectively, follow-up CT (right) with visible ischemic changes (see middle example in ﬁg. 4)

Fig. 4. Three examples of revealed hypodense areas; for each case top-let: acute CT examination, top-right: tensor wavelet-based extraction of hidden signatures, downleft: tensor wavelet and curvelets-based extraction, down-right: follow-up CT stroke conﬁrmation

180 cases assessed and described in details by experienced radiologists, including neurological anamnesis, acute (CT) and follow-up (CT and/or DWI) examinations analysis. Only diﬃcult ‘silent’ acute examinations (on average 3 hours after stroke onset), were selected for subtle hypodense sign extraction. For each case, we processed only the layers with subtle pathology presence possible because of follow-up conﬁrmation. It gives a total number of 108 CT single scans (slices) independently computed for subtle hypodensity estimation. Extracted and visually expressed information (ﬁg. 4) was compared to follow-up patterns and

320

A. Przelaskowski et al.

assigned to one of ﬁve categories: I – stroke symptoms are invisible; II – small indications are visible but are not interpretable due to redundant data masking; III – stroke symptoms are visible but are not unequivocal (neighbor slices should be analyzed for decision reliability); IV – stroke symptoms are clearly visible and interpretable; V – stroke symptoms are extremely and undoubtedly visible and obviously interpretable (model example). The categories I and II represent pathology extraction results without useable diagnostic information while categories III, IV and V represent images where ischemic stroke presence was more less conﬁrmed. Assessment results were the consensus of 3 experts engaged in computer-aided ischemic stroke diagnosis for more than 3 years. Against the background of all 108 CT slices, 26.85% were classiﬁed to categories I and II (23.15% and 3.7%, respectively) and 73.15% to categories III, IV and V (23.15%, 28.7% and 21.3%, respectively). Moreover, we looked on 25 ischemic stroke cases cumulatively. We assumed that if single case consists of at least one slice which was assigned independently to category IV or V, the stroke presence is acknowledged. This way we achieved 76% (19 cases) stroke detection eﬃciency while 24% (6 cases) of test cases were misclassiﬁed. Presented pathology extraction methods seem to be useful for computer assistance of acute stroke diagnosis. Wavelet-based framework of approximation is ﬂexible enough to optimize it for more formally deﬁned diagnostic problems of other pathologies, imaging modalities and protocols. However, it requires interdisciplinary cooperation of integrated research groups consisting of medical and technical stuﬀ to make pathology model more accurate and the expansion bases more semantic.

References 1. Adams, H.P., del Zoppo, G., Alberts, M.J.: Guidelines for the early management of adults with ischemic stroke. Stroke 38, 1655–1711 (2007) 2. Mullins, M.E., Schaefer, P.W., Sorensen, A.G., Halpern, E.F.: CT and conventional and diﬀusionweighted MR imaging in acute stroke: study in 691 patients at presentation to the emergency department. Radiology 224, 353–360 (2002) 3. von Kummer, R.: The impact of CT on acute stroke treatment. In: Lyden, P. (ed.) Thrombolytic Therapy for Stroke, pp. 249–278. Humana Press, New Jersey (2005) 4. Bendszus, M., Urbach, H., Meyer, B., Schultheiss, R., Solymosi, L.: Improved CT diagnosis of acute middle cerebral artery territory infarcts with density-diﬀerence analysis. Neuroradiology 39(2), 127–131 (1997) 5. Grimm, C., Hochmuth, A., Huppertz, H.J.: Voxel-based CT analysis for improved detection of early CT signs in cerebral infarction. Eur. Radiol., B315 (2005) 6. Chan, A.K., Peng, C.: Wavelets for sensing technologies. Artech House, Inc., Norwood (2003) 7. Doi, K.: Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Comp. Med. Imag. Graph 31, 198–211 (2007) 8. DeVore, R.A.: Nonlinear approximation. Acta Numerica 7, 51–150 (1998) 9. Capobianco Guido, R., Pereira, J.C. (guest eds.): Wavelet-based algorithms for medical problems. Special issue of Computers in Biology and Medicine, vol. 37(4) (2007)

Approximation of Subtle Pathology Signs in Multiscale Domain

321

10. Daubechies, I.: Ten lectures on wavelets. SIAM, Philadelphia (1995) 11. Welland, G.V. (ed.): Beyond Wavelets. Studies in Computational Mathematics, vol. 10. Academic Press, London (2003) 12. Donoho, D.L.: Wedgelets: nearly-minimax estimation of edges. Tech Retort, Statist. Depart., Stanford University (1997) 13. Starck, J.-L., Candes, E.J., Donoho, D.L.: The curvelet transform for image denoising. IEEE Tran. Image Proc. 11(6), 670–684 (2002) 14. Candes, E.J., Demanet, L., Donoho, D.L., Ying, L.: Fast discrete curvelet transforms. Technical Report, Cal Tech. (2005) 15. Do, M.N., Vetterli, M.: Contourlets. In: Wavelets, B., Welland, G.V. (eds.). Academic Press, New York (2003)

Iris Identification Using Geometrical Wavelets Mariusz Rachubi´ nski Institute of Informatics, Silesian University of Technology in Gliwice, ul. Akademicka 16, 44-100 Gliwice, Poland [email protected]

Abstract. This paper present personal identiﬁcation and veriﬁcation based on iris pattern and propose a new algorithm for iris feature extraction. The algorithm is based on texture analysis using wavelet transform. Iris code is generated using representation of the wavelet coeﬃcients based on a wedgelet dictionary. Preliminary results on CASIA iris image database conﬁrm the eﬀectiveness of this method and encourage further research. Keywords: iris representation, geometrical wavelets.

1

Introduction

User identiﬁcation and authentication are essential parts of information security. Users must authenticate as they access their computer systems, corporate and public security systems, consumer electronics and point of sale (POS) applications. There are three authentication methods that can be distinguished: simple, strong and biometric. Simple authentication consists in giving a user’s name and password. Strong authentication is based on something, that the user owns, e.g. electronic cards, tokens. This kind of authentication is often accompanied by the simple one. The biometric methods for identiﬁcation use individual physical features (ﬁngerprints, iris code, DNA, retina pattern) and behavioural ones (voice, signature, typing rhythm). Biometric methods are considered more secure than the ones based on access cards, PIN numbers, because they identify a person and not a card or password, which can be intercepted by unauthorised persons. Also hybrid systems are built, which can analyse a few biometric features. The eye is an approximate sphere of a 24mm diameter, ﬁlled mostly with an amorphous substance (vitreous humour), which is under pressure allowing it to retain its shape. The iris is a muscled part of a vascular membrane surrounding the opening called the pupil. The iris has a shape of a disc. It is the most frontal part of the central membrane of the eyeball. Due to the pigment it contains it is colourful. The muscles of the iris allow for increasing and decreasing the inﬂow of light through regulation of the pupil size [5]. The structure of the iris is highly varied and unique among the individuals in a given population. In the structure alone we can ﬁnd ridges, furrows and freckles (ﬁg. 1 [4]). Muscle ﬁbres of the iris sphincter have a circular geometry while the ﬁbres of the muscle opening the pupil are arranged radiantly inside the iris. The structure of the iris is stable with age also the left and right iris of a single person seems to be distinctive[4]. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 322–332, 2009. c Springer-Verlag Berlin Heidelberg 2009

Iris Identiﬁcation Using Geometrical Wavelets

323

Fig. 1. Anatomy of the human iris. The structure of the iris seen in a transverse section.

1.1

Outline

A new approach to iris feature extraction using geometrical wavelets is presented. Iris code is generated using representation of the wavelet coeﬃcients based on a wedgelet dictionary. Descriptions of iris image preprocessing, feature extraction and encoding are given. A brief description of the theory of geometrical wavelets used in the proposed approach to iris feature extraction is provided. A series of experiments to evaluate the performance of the proposed approach was performed. Comparisons with the other existing methods are carried out in order to exhibit the eﬃciency of proposed method.

2

Brief Overview of Existing Systems

The idea of using the iris pattern for identiﬁcation of persons was described for the ﬁrst time in an ophthalmology textbook by James Doggarts in 1949 also proposed earlier by an ophthalmologist Frank Burch in 1936. In 1987 two ophthalmologists Aran Szaﬁr and Leonard Flom [1] obtained a patent for the idea of using the iris pattern for identiﬁcation of persons. In 1989 they turned to John Daugman from Cambridge University in Great Britain with a proposition to create an algorithm of iris pattern recognition. The algorithm proposed by Daugman was granted a patent in 1994 [3] and is currently used in the existing commercial systems [16]. In the Daugman system for locating the iris he used the integro-diﬀerential operator, which searches for the biggest change intensity of the image for the round shape of the iris. In the ﬁrst stage of the search the general outline of the outer iris boundary is found. Then the exact boundaries are deﬁned. When searching for the outer iris boundary the range of the arch of the outline is limited to two opposite 90◦ cones, because the upper and the lower part of the iris is sometimes occluded by the eyelids. In order to ensure a ﬁxed size of the iris patterns stored in the database they are normalised through changing the iris co-ordinates to a polar co-ordinate

324

M. Rachubi´ nski

system of a constant size. It consists in extending the iris into a rectangle and stretching it to the demanded ﬁxed size[8]. The iris is coded using the Gabor transform. The number of Gabor wavelet coeﬃcients depends on the number and the width of the wavelets covering the iris image. Signs of the real and imaginary part of each of the obtained coeﬃcients are coded using 1 bit each. In Daugman algorithm a 2048-bit length vector is obtained[2]. Codes of two vectors of A and B iris features are compared bit by bit; their normalised Hamming distance (the number, which value is determined by the number of bits diﬀerentiating the two vectors) is deﬁned in the following way: 2048 1 HD = Aj (XOR)Bj 2048 j=1

(1)

In order to make decision about classifying the iris as authentic or imposter there is a need to choose the criterion - the point of separation for the Hamming distance between the irises. Value smaller than this point will determine authentic data and a value bigger will determine imposter. In the ﬁrst experiment the database consisted of 592 diﬀerent irises from 323 persons. For each iris there were on average three images taken. The system exhibited no False Accepts and False Rejects [4]. For ﬁnding the iris Wildes uses in his work the Hough transform. Before comparing two irises the compared image is adjusted (scaled and rotated) to the image in the database. For extracting iris features he uses the ﬁlter LoG (Laplacian of Gaussian) and creates Laplace’s pyramid. For comparing two irises he calculates the normalised mutual relation between two images. The decision of accepting the iris is made after calculating the Fisher’s linear dis-criminant. The tests were done using 60 diﬀerent irises coming from 40 persons. Each iris had 10 images taken in two sessions: 5 in the ﬁrst one and 5 in the second one, a month later. There were no observed False Accepts and False Rejects [4].

3

Iris Localization

Figure 2 is a block diagram illustrating an iris identiﬁcation algorithm.

Fig. 2. Iris identiﬁcation diagram

Iris Identiﬁcation Using Geometrical Wavelets

325

Iris localization stage has been divided into 5 substages. The ﬁrst substage is estimating the location of the pupil centre, where the preliminary co-ordinates of the centre SZs (x0Ze , y0Ze ) and the radius RZo . It consists in ﬁnding in the image I(x, y) the square, which contains the biggest number of black pixels (Fig. 3).

Fig. 3. Estimating the iris location. Search area is marked blue, wanted square is marked red.

The second substage consists in ﬁnding the exact pupil location using an integro-diﬀerential operator (2) proposed by Daugman and its ellipse-like shape. ∂ I(x, y) max(r,x0 ,y0 ) Gσ (r) ∗ ds (2) ∂r r,x0 ,y0 2πr In practice it consists in smoothening the image with a Gauss ﬁlter and then ﬁnding the biggest quotient of the pixel intensity sums located in the ellipse for the given centre and radius and the pixel intensity sum of the ellipse with the same centre and a radius smaller about 2: θ I[(a cos(θ) + x0Ze ), (b sin(θ) + y0Ze )] (3) max(a,b,x0Ze ,y0Ze ) θ I[((a − 2) cos(θ) + x0Ze ), ((b − 2) sin(θ) + y0Ze )] for a, b ∈ RZo − 5; RZo + 5 and θ ∈ 0◦ ; 360◦ and x0Ze ∈ x0Zo − 5; x0Zo + 5 and y0Ze ∈ y0Zo − 5; y0Zo + 5. In the next substage the outer boundary between the iris and the sclera is located using the same integro-diﬀerential operator. The fourth substage is ﬁnding the eyelids. It is carried out through modifying the integro-diﬀerential operator, which locates the circular edges to the operator, which ﬁnds in the image the edges describable with a parabola equation (Fig. 4). After carrying out these four substages there are four established equations: of the ellipse for the pupil, of the circle for the outer boundary of the iris and two equations of parabolas for the upper and lower eyelid. These four equations limit the iris area in the image of a ring-like shape. In the last substage the iris area is normalised by changing the from Cartesian co-ordinate system to a polar co-ordinate system simultaneously scaling the iris to a certain (ﬁxed) size (4). The internal part of the ring is taken into consideration and it consists in unfolding the ring into a rectangle (Fig. 5).

326

M. Rachubi´ nski

Fig. 4. Finding the eyelids. Parabolas describing the shape and location of eyelids are marked red. Search area of the parabolas’ vertexes is marked blue.

Fig. 5. Iris extension

I(x(rθ, ), y(r, θ)) → I(r, θ)

(4)

for r ∈ 0; 1 and θ ∈ 0; 2π where x(r, θ) and y(r, θ) are deﬁned as a linear combination of two sets of pupil boundary points (xz (θ), yz (θ)) and the points of external boundary between the iris and the sclera (xt (θ), yt (θ)): x(r, θ) = (1 − r)xz (θ) + rxt (θ) y(r, θ) = (1 − r)yz (θ) + ryt (θ)

4

(5)

Features Extraction Using Geometrical Vavelets

For extracting iris features the theory of geometrical wavelets was used. The theory of geometrical wavelets was ﬁrst described by D. Donoho in 1999 [7]. Geometrical wavelets allow for more accurate image analysis than in the case of classical wavelet methods. They allow to catch changes in the image not only in time, but also direction. It allows to catch geometrical features of the image, like edges [11]. It is especially important in the case of iris, because it’s the edges (formed here by ridges and furrows) that carry most information. It can be assumed that geometrical wavelets are a certain kind of generalisation of the classical wavelet theory. The most frequently used are two groups: wedgelets and beamlets and also curvelet and rigdelet. They all lead to diﬀerent image coding and are characterised by diﬀerent use: they allow for more eﬀective image coding than standard methods considering compression. For iris features extraction wavelets from the ﬁrst group were chosen: wedgelets and beamlets.

Iris Identiﬁcation Using Geometrical Wavelets

327

Fig. 6. Quadtree partition of the image

For any binary image with dimensions N × N , where N = 2k ,k ∈ N a quadtree partition of image is considered (Fig. 6). This division is reﬂected in the quadtree partition, which nodes may represent particular elements of quadtree partition of an image. Elements of such a division are squares Sij with dimensions 2N −1 × 2N −1 , where i = 0, ..., lg N (indexing the tree level) j = 0, ..., 4i − 1 (indexing the squares within the level). Let’s establish the square Sij from the quadtree partition. A beamlet is a nondegenerate (i.e. not laying entirely on the edge of the square) segment b connecting any two points on the edge of this square deﬁned by coordinates (v1 , v2 ). A wedgelet is a characteristic function of the area determined by beamlet b represented as: 1, for y ≤ b(x) w(x, y) = x, y ∈ S (6) 0, for y > b(x) Additionally for the wedgelet parameterisation values h1 and h2 are used to enable coding the image in grey scale[11]. In practice parameterisation based on a polar co-ordinate system is used (Fig. 7 [11]). Then the beamlet and the wedgelet are unanimously determined by the θ angle between the beamlet and the horizontal line passing by the square centre and the distance r between the centre of the square and the beamlet[11]. Wavelets parametered in such a way allow for deﬁning appropriate dictionaries which are used in image coding. By a beamlet dictionary we mean the following set: B = {bn = bij (r, θ) : i = 0, ..., lg N, j = 0, ..., 4i − 1}; (7) by a wedgelet dictionary we mean: W = {wn = bij (r, θ, h1 , h2 ) : i = 0, ..., lg N, j = 0, ..., 4i − 1}; 1

where r ∈ [0, ..., 2lg N −i− 2 ) ], θ ∈ [0, π), h1 , h2 ∈ {0, ..., 255}.

(8)

328

M. Rachubi´ nski

Fig. 7. Parameterisation of beamlets and wedgelets

Beamlet and wedgelet dictionaries deﬁned above include segments of diﬀerent location, scale and orientation. This variety allows for exact image approximation.These dictionaries can be used for image analysis and synthesis. Wedgelet analysis f is represented with the following formula: αn = f, wn

(9)

for all wn ∈ W , 0 < n ≤ |W | . Wedgelet synthesis is represented with a formula: f (x1 , x2 ) =

|W |

αn wn (x1 , x2 )

(10)

n=1

For beamlets analogous deﬁnitions of analysis and synthesis are used [11]. 4.1

Iris Code

For establishing the ’iris code’ the wedgelet theory is used. Before extracting features and coding the rectangle with the extended iris is divided into smaller, independent, so-called local regions of 8x8 pixels. This size was chosen empirically as the one most eﬀectively extracting iris features. The rectangular iris image is divided into 15 rings 8 pixels high and 512 pixels long. The length of the shorter side of the rectangle (created by the unfolded ring) is 64, so the neighbouring rings overlap and have a common part 4 pixels wide. Then the rings are divided into 127 parts - local regions, each 8 pixels long. The length of the ring is 512 pixels, which is why the neighbouring local regions have a common part 4 pixels wide. The total number of local regions is 15*127=1905. The length of the local region is 5◦ 40 . Respective local regions are set every 4 pixels, which is the length of 2◦ 50 . For an 8x8 local region a wedgelet dictionary for level 0 is determined. For a straight line passing through the two points established in the wedgelet analysis there is calculated the angle α of inclination of the line to the OX axis. This angle is coded using 2 bits: 1, for tan α ∈ −1, 1 w1 = (11) 0, for tan α ∈ (−∞, −1) ∪ (1, +∞)

Iris Identiﬁcation Using Geometrical Wavelets

w2 =

1, for tan α ∈ 0, +∞) 0, for tan α ∈ (−∞, 0)

329

(12)

The h1 and h2 values are also used - the third bit of data describing the local region is obtained. The value is determined through comparing these values: 1, for h1 ≥ h2 w3 = (13) 0, for h1 < h2

5

Comparing Samples

Comparing two codes comes down to calculating the Hamming distance of two features vectors - iris codes. HD =

1905 1 Aj (XOR)Bj N j=1

(14)

An assumption has been made: to consider two local regions Aj , Bj to be identical all three bits of the codes obtained from coding these regions have to be equal. The N symbol is the number of correct comparisons. A correct comparison is one, in which part of the iris occluded by the eyelid is not taken into consideration during the comparison. When comparing two irises it has to be noticed, that the person, who is being identiﬁed can lean his or her head during image acquisition.This problem is solved by moving the iris code about three bits, which reﬂects iris rotation about 2◦ 50 .

6

Results

Experiments on CASIA v1.0 iris image databases are performed to evaluate the accuracy of the proposed method. The experiments are completed in two modes: veriﬁcation (one-to-one) and identiﬁcation (one-to-many). In veriﬁcation mode two error rates are used to report the performance of the proposed method : False Acceptance Rate (FAR - the probability of identifying an imposter as an enrolled user), False Rejection Rate (FRR - the probability of rejecting an enrolled user, as if he is an imposter) and Equal Error Rate (ERR - the value where the FAR and FRR are equal). Two tests, studying four diﬀerent criterion values, were carried out in order to establish the criterion value. The database of iris images [17], which was tested, consists of 756 images of 108 persons. Each person had 7 images taken in two sessions at diﬀerent times. In the ﬁrst session there were 3 images taken and in the second one 4 images. In the ﬁrst test the database of iris patterns consisted of the images taken in the ﬁrst session (108*3=324 images) and were compared with the images taken in the second session (108*4=432 images). In the second test the database of iris patterns consisted of images taken in the second session and were compared with the images taken in the ﬁrst session.

330

M. Rachubi´ nski

None of the tests detected False Accept, which is classifying an imposter as an enrolled user. There has only been an error of False Reject, which is classifying an enrolled user, as if he is an imposter. The table 1 presents test results. Table 1. Test results Test 1 Test 2 False Accept 0,0% 0% False Reject 1,39% 0,31%

In identiﬁcation mode is measured Correct Recognition Rate (CRR). The Correct Recognition Rate (CRR) is deﬁned in Equation below CRR =

the number of samples being correctly classiﬁed ∗ 100 the total number of test samples

(15)

In order to establish CRR the same tests were made. The proposed method achieves CRR of 99.3% and 100% percent respectively. The table 2 presents comparision with existing methods using CASIA database. Table 2. Test results - comparision with existing methods CRR Daugman [2] 100%* Wildes [4] Boles and Boashash [6] 92,64%* Ma et al. [13] 99,60% Ma et al. [12] 100% Tan [9] 99,19% Kaushik and Bhattacharya [15] 99,81% Liu et al. [14] 97,08% proposed (test 1) 99,30% proposed (test 2) 100%

EER 0,08%* 1,76%* 8,13%* 0,29% 0,07% 0,57% 0,13% 1,79% 0,31% 0,15%

*These results were published in [10] by using the bootstrap, authors approximately predict the computational complexities.

Presented results of the research shows that the number of patterns (irises images) stored in the database of iris patterns for each person inﬂuences correct recognition of persons. Greater number of patterns gives a bigger number of comparisons of the identiﬁed person’s iris pattern with the patterns in the database, which leads to a more accurate result. However, results in table 2 may not be interpreted as a real comparison because experimental results reported in this table have been implemented on the diﬀerent datasets. This table demonstrates a comparison of the CRR and ERR. Results shows that the proposed method provides a very good recognition rate and low EER among the diﬀerent existing methods.

Iris Identiﬁcation Using Geometrical Wavelets

7

331

Summary and Perspectives

The aim of this paper is to present personal identiﬁcation and veriﬁcation based on iris pattern and propose a new algorithm for iris feature extraction using geometrical wavelets. Described solution use an already existing database of images CASIA and is concentrating just on the recognition stage, without focusing on the stage of images acquisition connected with building an appropriate system and checking the iris vitality. The input data for the algorithm is image containing a single human eye. The algorithm is performed in three main steps: iris detection and normalization, iris feature extraction and code generation, matching of the iris code and making a decision. Iris is detected in four steps: approximate pupil center is found, accurate pupil center localization and shape (ellipse) using the Integro-diﬀerential operator proposed by Daugman, iris outer bound detection, upper and lower eyelid detection using the same operator. Then the annular iris is normalized into a rectangular block. The algorithm for iris feature extraction is based on texture analysis using wavelet transform. Iris code is generated using representation of the wavelet coeﬃcients based on the wedgelet dictionary. A Hamming distance is used to quantify the diﬀerence between iris patterns. Presented in the article results of the research of coding the iris pattern using geometrical wavelets conﬁrm the eﬀectiveness of this method and they are treated as a starting point for further research.

Fig. 8. Examples of an iris obscured by eyelids and eyelashes

Current research is focused around iris area analysis in order to establish the areas not occluded by eyelids or eyelashes containing the most information for obtaining the iris code. For a big number of irises in the database of patterns part of the images used for establishing the feature vector is occluded by eyelids. An extreme case is when both eyelids occlude the iris and reach the pupil (Fig. 8 [17]). A big number of eyelashes occluding the iris also results in incorrect establishing of the iris feature vector (Fig. 8 [17]). In the proposed solution for establishing the feature vector only the inner part of the iris is used.

332

M. Rachubi´ nski

Acknowledgements Portions of the research in this paper use the CASIA V1.0 collected by the Chinese Academy of Sciences’ Institute of Automation (CASIA).

References 1. Flom, L., Saﬁr, A.: Iris recognition system, United States Patent No. 4.641.349, February 3 (1987) 2. Daugman, J.: High conﬁdence visual recognition of persons by a test of statistical independence. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(11), 1148–1161 (1993) 3. Daugman, J.: Biometric Personal Identiﬁcation System Based on Iris Analysis U.S. Patent No. 5, 291, 560, March 1 (1994) 4. Wildes, R.: Iris recognition: an emerging biometric technology. Proceedings of the IEEE 85, 1348–1363 (1997) 5. Snell, R.S., Lemp, M.A.: Clinical Anatomy of the Eye. Blackwell Publishing, Malden (1998) 6. Boles, W.W., Boashash, B.: A human identiﬁcation technique using images of the iris and wavelet transform. IEEE Transactions on Signal Processing 46(4), 1185– 1188 (1998) 7. Donoho, D.L.: Wedgelets: Nearlyminimax estimation of edges. Annals of Stat. 27, 859–897 (1999) 8. Barrett, A.: Daugman’s Iris Scanning Algorithm Biometrics Test Center, San Jose State University 2/4/00 DaugmanAlg1.doc, vs. 2.0 9. Ma, L., Tan, T., Wang, Y.: Iris recognition using circular symmetric ﬁlters. In: Proc. 16th Int. Conf. Pattern Recognition, vol. II, pp. 414–417 (2002) 10. Ma, L., Tan, T., Wang, Y., Zhang, D.: Personal Identiﬁcation Based on Iris Texture Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(12), 1519–1533 (2003) 11. Lisowska, A.: Extended Wedgelets - Geometrical Wavelets in Eﬃcient Image Coding. Machine Graphics and Vision 13(3), 261–274 12. Ma, L., Tan, T., Wang, Y., Zhang, D.: Eﬃcient iris recognition by characterizing key local variations. IEEE Transactions on Image Processing 13(6), 739–750 (2004) 13. Ma, L., Tan, T., Zhang, D., Wang, Y.: Local Intensity Variation Analysis for Iris Recognition. Pattern Recognition 37(6), 1287–1298 (2004) 14. Liu, X., Bowyer, K.W., Flynn, P.J.: Experiments with an improved iris segmentation algorithm. In: Proceedings of the 4th IEEE Workshop on Automatic Identiﬁcation Advanced Technologies (AUTO ID 2005), Buﬀalo, NY, USA, October 2005, pp. 118–123 (2005) 15. Kaushik, R., Prabir, B.: Optimal Features Subset Selection and Classiﬁcation for Iris Recognition. EURASIP Journal on Image and Video Processing 2008 (2008); Article ID 743103, 20 pages doi:10.1155/2008/743103 16. http://www.cl.cam.ac.uk/users/jgd1000/history.html 17. http://www.cbsr.ia.ac.cn/IrisDatabase.htm

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis Andrzej Zacher The Silesian University of Technology, Institute of Informatics Akademicka 2, 44-100 Gliwice, Poland [email protected]

Abstract. This paper introduces a photon subsurface scattering method to simulate light transport in human colon tissue. First the theoretical model and parameters of human tissue including autoﬂuorescence phenomenon was presented. Then it was described the Monte-Carlo model of steady-state light transport in multi-layered colon. The goal of this investigation is to simulate the light propagation in tissue and to collect the data containing the eﬀect of ﬂuorescence. This information will be used to generate images. Pictures taken for diﬀerent adjustment of light parameters should deﬁne a conﬁguration for which cancerous structures are visible quickly and precisely. Real medical devices can adjust their parameters to the simulated ones and help with eﬃcient diagnosis and recognition of diseased structures. Keywords: photodynamic diagnosis, autoﬂuorescence simulation, subsurface scattering, photon mapping.

1

Introduction

Photodynamic diagnosis and tumor therapy was invented at the beginning of 80’. Investigations of ﬂuorescence from endogenous and exogenous molecules in cells and tissues are very common, especially in vivo and ex vivo studies [1]. This is a harmless method of cancer diagnosis and removal of infected structures. Speciﬁc substances with ﬂuorescent characteristic injected into bloodstream are collected by the cancerous cells. They can be recognized by increased red color intensity, if the light of speciﬁed wavelength is applied. Photofrin is the only one ﬂuorophore that was allowed to be used by the Food and Drug Administration. Fortunately, there are also natural ﬂuorophores in human tissues that ﬂuoresces when the laser light is applied. The contrast between neoplastic and non-neoplastic tissues is characterized by changes related to several endogenous ﬂuorophores. Their distribution in the tissue is depth-dependent. The main ﬂuorophores in the epithelial layer are tryptophan, reduced nicotinamide adenine dinucleotide (NADH) and ﬂavin adenine dinucleotide (FAD), while collagen is the primary ﬂuorophore in the stroma. Recent high resolution ﬂuorescence imaging studies showed that the

Supported by Ministry of Science and Higher Education grant R13 046 02.

L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 333–344, 2009. c Springer-Verlag Berlin Heidelberg 2009

334

A. Zacher

intensity of ﬂuorescence and distribution of ﬂuorophores changes with carcinogenesis. For example, the ﬂuorescence of collagen decreases within the stroma of neoplastic tissues relatively to that of normal tissues. Within the epithelium, the NADH ﬂuorescence of dysplastic regions is increased relatively to that of normal epithelial cells, while the opposite result is observed for FAD [2]. Changes in their ﬂuorescent properties can be helpful to locate and recognize cancerous cells much faster e.g. by image processing methods [3]. That is why more and more attention is attracted to understand the microscopic source and mechanism of light-induced autoﬂuorescence (LIF) in diseased and normal human tissues. The growing interest in characterization of UV and VIS wavelength excitation (520-250nm) and ﬂuorescence emission spectra (260-750nm) of known and suspected endogenous biological ﬂuorophores and exogenous photosensitizer agents contribute to increase the number of new methods simulating the behavior of light in contact with ﬂuorophores [4]. But before any simulation can be done it is necessary to know more details about the tissue. Generally speaking, it is a multilayered structure that consists of absorbing, scattering and ﬂuorescent elements. Each component has diﬀerent properties e.g. blood is highly absorbing, but because of hemoglobin it proves ﬂuorescence eﬀect. Determination of components and their physical characteristic has the key inﬂuence on the ﬁnal results. The appropriate distribution of each substance in a tissue, the percentage amount in the simulated layer and the overall quantity describing how much ﬂuorescent matter contains each component are important questions that will be addressed in this paper.

2

Previous Work

Monte Carlo model of steady-state light transport in multi-layered tissue was presented in [5]. The method describes local rules of photon propagation, but since it is statistical in nature and requires a large amount of photons to be traced, it is very time consuming. Moreover, photons are not treated as wave phenomenon and such properties as phase and polarization were ignored. The radiant energy distribution within cells was also neglected. It is also worth to stress it, that simple model of tissue with parameters was proposed, but without ﬂuorescent properties. Despite of all those limitations, general rules can be still utilized and applied to simulate light transport correctly. In [6] a model for subsurface light transport in translucent materials was introduced. It is the extension of bidirectional reﬂectance distribution function (BRDF) that assumes that light entering a material leaves the material at the same position. This approximation is valid for metals, but fails for translucent materials which exhibit signiﬁcant transport below the surface. That is why bidirectional surface scattering distribution function (BSSRDF) was proposed to describe the light propagation between two rays that hit the surface. However, adding to this model a ﬂuorescent behavior is not straightforward and cannot be used easily to simulate such properties of tissues. In [7] a new algorithm for rendering translucent materials that combines photon tracing with diﬀusion was presented. This idea characterizes the fact that it

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis

335

allows to render eﬃciently highly scattering translucent materials while taking into account internal obstackles, complex geometry, translucent inter-scattering etc. Instead of sampling lighting at the surface, photons are traced into the material and stored volumetrically at their ﬁrst interaction with the material. The proposed approach is as eﬃcient as previous Monte Carlo sampling approaches based on the dipole or multipole diﬀusion approximations and demonstrates that it is more accurate and captures several illumination eﬀects that was previously ignored. The Monte Carlo model was extended and veriﬁed in [8]. Now, the basic idea was extended to account for the ﬂuorescent behavior in tissues. The model was validated with real experiments as well. [2] concentrated on developing ﬁber optic probe geometries to selectively measure ﬂuorescence spectra from diﬀerent sublayers within human epithelial tissues. They goal was to improve the endogenous ﬂuorescence contrast between neoplastic and nonneoplastic tissues. They proposed two basic ﬁber optic probe geometries, which are called the variable aperture (VA) and multidistance (MD) approaches and compared with ﬂuorescence measurements from human cervical epithelial tissues. They presented a simple tissue model and showed some numerical results without rendering them graphically. Tissue properties and parameters like absorption, scattering and ﬂuorescence were measured and described in many in vivo investigations like [9] or [10]. Having that information we will try to build a model of tissue that will preserve and follow all of those characteristics. Despite of the fact that a lot of work has been done to understand the light propagation in human tissue, this paper is going to be one of the ﬁrst trials to visualize the computational results. 3 Tissue model Translucent objects are very popular in our daily lives. All materials such as skin, marble, wood, wax etc. exhibit this property. They are perceived as soft and gentle due to the fact that light enters the object and is absorbed and scattered inside. This process is known as subsurface scattering [11]. It diﬀers from the other models, where the light after interaction with the object is scattered back according to BSDF. Here, the light leaves the object at diﬀerent position than it was initially captured. That is why the unique appearance of translucent materials cannot be achieved with simple surface reﬂection models. Human tissue is a good example of material where the eﬀect of translucence is noticeable and dominant. Moreover, because of the fact that the ﬂuorescence phenomenon is very important, the light is not only absorbed and scattered under the surface of the object, but also its spectrum changes relatively to material the tissue consists of. The signiﬁcant step in constructing a model of human tissue was to identify the morphological structures which ﬂuoresce. The mucosa is the top layer of tissue consisting of tubular crypts which are surrounded by a connective tissue structure mostly composed of collagen. Under the mucosa, approximately 450μm below the tissue surface there is the submucosa, a layer of connective tissue also mainly composed of collagen. However in dysplastic tissue, the crypts are irregular in shape and size and become

336

A. Zacher

Fig. 1. Layered structure and principal histologic components of normal, dysplastic, and adenomatous colon tissue [12]

disorganized. Dysplastic colon tissue proves similar morphology to that of normal colon (ﬂat dysplasia), or it can be characterized by additional gross morphological changes (adenomatous polyp). It was noticed that normal and pathologic structure of the tissue has diﬀerent ﬂuorescence properties. That observation can be utilized in the process of early cancer detection [12]. The simplest possible model of human tissue was investigated. It consists of inﬁnitely long and wide single-layered structure characterized by the following parameters: the thickness, refractive index n, absorption coeﬃcient ua , scattering coeﬃcient us and the anisotropy factor g. It is assumed that the top ambient medium is air and its refractive index is given as well. The absorption coeﬃcient ua is the probability of photon being absorbed per unit inﬁnitesimal pathlength. The scattering coeﬃcient us is the probability of photon being scattered per unit inﬁnitesimal pathlength. Finally, the anisotropy factor g deﬁnes the average of cosine value of deﬂection angle [5]. This model needs to include also ﬂuorescence phenomenon. To accomplish this goal it was necessary to decide what substances the tissue is made of that shows ﬂuorescent properties. Collagen absorbs light in ultraviolet and ﬂuoresces in the blue color and its inﬂuence is dominant among other chromophores. The consumed energy is mainly converted to heat, but can be also re-emitted as ﬂuorescence or be used for a photochemical reaction [9]. The probability of interaction a photon sample with a ﬂuorophores was deﬁned as d. Moreover, the concentration c of ﬂuorescent molecules in ﬂuorophore is diﬀerent for each substance and has to be given as well. The absorption of light in tissue is strongly wavelength dependent. A systematic characterization of UV and VIS wavelength excitation (520-250nm) and ﬂuorescence emission spectra (260-750nm) is described in so called EEM – Excitation-emission matrices [4]. It is a two-dimensional contour plot that displays the ﬂuorescence intensities as a function of a range of excitation and emission wavelengths. Each contour represents points of equal ﬂuorescence intensity [13]. The database containing the ﬂuorescence EEMs of 35 possible endogenous ﬂuorophores, was used. All data was measured using standard ﬂuorescence spectrophotometer.

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis

337

Fig. 2. Example of EEM of a molecule. The highest values of intensity are drawn in red color (inner part), the lowest in violet (outer part) [4].

3

Light Propagation in Tissue

A Monte Carlo model of light transport was chosen to simulate the photon distribution and interaction with ﬂuorescent media. This method is widely used by the family of particle-tracking algorithms including Photon Mapping. It assumes constructing paths from the light and store at each vertex the amount of incident energy represented by the photon. If required number of samples were tracked and collected, this data is used at rendering time to estimate ﬁnal values at the queried position. Photon tracing step is independent from the computing measurement part, recorded samples can be reused to generate images from different points of view. Photon Mapping algorithm applies very well to simulate light transport in a turbid media like tissue. Subsurface scattering model can be implemented easily including the ﬂuorescence properties and internal structure of the investigated object. First of all samples used to generate ray directions must be well-distributed, because it is not known in the beginning of simulation, how many paths need to be generated to get the desired number of photons that interacted with ﬂuorescent molecules. For this reason Halton sequence was used, that has good low-discrepancy properties and gives more uniform distribution of photons in the scene than random points. Each photon packet has assigned a spectrum with energy proportional to the spectrum of the light source. That radiance of a photon, described by its color spectrum, expresses its weight W . Such a packet is launched in the scene from the random light source and propagates in the air without changing direction and its weight till it intersects investigated tissue. Then the Snell’s law indicates the relationship between the angle of incidence ai and the angle of transmission at . Now, it can be now determined whether the packet is reﬂected back or enters the tissue by generating a random number ζ1 and comparing it with incident reﬂectance R(ai ): R(ai ) = 1/2[sin2 (ai − at )/ sin2 (ai + at ) + tg2 (ai − at )/tg2 (ai + at )].

(1)

338

A. Zacher

which is the average of the reﬂectances for the two orthogonal polarization directions derived from Fresnel’s formula. If ζ1 ⇐ R(ai ), then the photon is reﬂected and is not tracked any more, otherwise the photon enters tissue. Since the photon undergoes the interaction with tissue, its weight needs to be changed: W = W − W R(ai ).

(2)

Packets are then moved in the initial direction by the step size deﬁned as: s = – ln(ζ2 )/(ua + us ).

(3)

Once the photon changes its position, some attenuation of the photon weight must be computed, due to absorption by the interaction site. W = W − W ua /(ua + us ).

(4)

Now is the time to determine if the interaction was with collagen or with other substance the human tissue consists of. Another random number ζ3 is used and compared with the probability of interaction a photon with chromophore.

Fig. 3. Photon propagation in human tissue. Photon interacts many times with molecules (dots) and only some fraction of them escapes the tissue, others are completely absorbed. Fluorescence phenomenon occurs very rarely, if at all (double circle).

If ζ3 < d then random variable is used again to determine whether the interaction takes place with ﬂuorescent molecule or not. For ζ4 < c the spectrum of the photon packet changes utilizing data from EEM. To calculate the emission spectrum, the superposition principle will be used. It means that the partial emission spectrum for each excitation wavelength will be computed and summed up. The algorithm guarantees that total weight of emission spectrum is not grater than the initial excitation weight. Once the weight and spectrum of the photon was updated, the packet is ready to be scattered. There are two angles to be sampled statistically: deﬂection angle

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis

339

θ ⊂ [0, π) and an azimuthal angle Ψ ⊂ [0, 2π). The cosine of deﬂection angle can be calculated from Heney-Greenstein phase function as a scattering function of random variable ζ. For g diﬀerent than 0 it is deﬁned as: cos θ = 1/(4πg){1 + g 2 –[(1–g 2 )/(1–g + 2gζ)]2 }.

(5)

Next, the azimuthal angle Π is sampled according to equation: Ψ = 2πζ.

(6)

Fig. 4. Flow diagram of proposed Monte Carlo model

Once both angles are calculated it is easy to deﬁne a new direction a photon will follow in the next step. This process of moving, absorbing and scattering a packet is repeated until either the weight of the photon is too small or the photon hits the tissue interface again and escapes. If the photon left the structure of a tissue and its spectrum changed during the propagation due to a contact with ﬂuorescent substance, then it is stored in the photon map. Moreover, in every loop a Russian Roulette is performed to check whether photon sample should be terminated or should survive. This formula ensures that the principle of energy conservation is ensured and distribution of photon deposition is not biased. Early termination of photon is necessary, because following a path of photon with

340

A. Zacher

weight below some ﬁxed threshold does not give any useful information and only little contributes to the ﬁnal result. Proposed Monte Carlo model, excluding ﬂuorescence phenomenon, was compared and veriﬁed with other algorithms giving good approximation of light propagation and distribution in a tissue [5].

4

Simulation and Results

Endoscopy is a medical procedure used to investigate the internal surfaces of an organ by inserting a special device into the body. The endoscope consists of a thin tube with ﬁber optics to transmit light. Illuminated structures are then viewed through an eyepiece or video camera ﬁxed to the tube [15]. The example of the tip of the endoscope is presented below. Generally two light sources are available and small camera. The light sources can be treated as area lights with cone angle α = 450 . Moreover the head of endoscope may be rotated. Angle β speciﬁes the deviation from the completely perpendicular position with respect to tissue surface. In our case β = 150 . The tool that was used in this work for generating images according to photon mapping algorithm was PBRT. This is not commercial computer program written in C++. PBRT is available with the source code and the book [14] that gives highly detailed description of most of the functions. Eﬃciency was not the goal of this software. It takes often minutes or even hours to render highly detailed and complex images. Unfortunately PBRT does not support subsurface scattering model, that is why Photon mapping algorithm was changed and revised to implement full Monte Carlo model of Light Transport in turbid media. The simulation was performed for inﬁnitely thick single-layered tissue. Moreover cancerous polyp was added into the tissue model in which, besides collagen (blue color), also photofrin was concentrated (red color).

Fig. 5. Schema presenting the head of endoscope – bottom-view. Parameters are similar to the real device and were used during simulations.

Fig. 6. The head of endoscope during operation – side-view. Only ﬁrst light source is visible, the second one is behind.

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis

341

Fig. 7. Simulated human tissue for spec- Fig. 8. Simulated human tissue for spectrum light in range 360nm-750nm. Spec- trum light in range 375nm-395nm. Spectral irradiance equals to 20 Wm−2 nm−1 . tral irradiance equals to 2000 Wm−2 nm−1 . Table 1. The parameters of experiment nair 1

nlayer 1.1

ua [cm−1 ] 20

us [cm−1 ] 200

g 0.9

dcol 0.2

ccol 0.4

dpho 0.1

cpho 0.33

All parameters describing probability of photon interaction (dcol , dpho ) and concentration of ﬂuorophore molecules (ccol , cpho ) were only a good guess, they are not proven by any laboratory investigations. For this tissue model six simulations were performed with diﬀerent color spectrum of the spot light. In each case 500000 ﬂuorescent photons were stored.

5

Discussion

Results presented in Fig. 7 and Fig. 8 look similar, although energy carried by the light source is almost 3 times greater for ultraviolet laser beam than for the white light. This is because of the fact, that for white color spectrum the energy is not lost so much and more wavelengths are used in EEM transformation. To get similar color intensity for UV light beam, more energy is needed. However the

342

A. Zacher

Fig. 9. Simulated human tissue for spec- Fig. 10. Simulated human tissue for spetrum light in range 390nm-410nm. Spec- ctrum light in range 405nm-425nm. Spectral irradiance equals to 2000 Wm−2nm−1 . tral irradiance equals to 2000 Wm−2nm−1 .

spectrum of white light does not have to be so wide. For excitation wavelengths in range 600nm-750nm the emission spectrum is nearly zero, so it does not inﬂuence on the ﬁnal intensity. Collagen ﬂuoresces mostly for UV color spectrum, it means that probably the best would be to apply light beam with wavelengths in range 360nm-420nm. Looking again at Fig. 7 and Fig. 8 it is noticeable that the color of healthy part of tissue is diﬀerent at these two pictures. In both situations it is bluish. One can say that for the case where the white light was used the color of photons that contacted with collagen and photofrin are to some degree similar – photofrin photons are red and collagen photons are pink. However for ultraviolet laser beam, the contrast between photofrin and collagen is signiﬁcant, the cancerous polyp is better visible and this is exactly what we want. This two images show visually the advantage of UV light source over white light source. Also interesting eﬀect was observed for the series of images Fig. 8, Fig. 9 and Fig. 10. The narrow spectrum of UV color light was moved from left to right. It is visible very well that the collagen photons fade and their intensity is going down. The reason also comes from EEM for collagen. For higher excitation wavelengths the emission spectrum energy is going down and it is almost zero in the Fig. 10. Similar eﬀect is noticeable for photofrin photons, which intensity also decreases

Numerical Simulation of Endoscopic Images in Photodynamic Diagnosis

Fig. 11. Simulated human tissue when mercury lamp was used (light cyan color) [16]

343

Fig. 12. Simulated human tissue when xenon lamp was used (white color) [17]

with higher excitation spectrum range, but not so fast. This is because of EEM of photofrin. The initial wave spectrum after contact with ﬂuorescent molecule still possesses some amount of energy, even for higher wavelengths. When the collagen photons are almost invisible the photofrin intensity is still big enough. It means that it is worth to ﬁnd such a light source spectrum that reduces the impact of collagen ﬂuorescence and increases the intensity of photofrin. Interesting result was noticed for the mercury lamp – Fig 11. Photofrin ﬂuoresces in red color as it was expected, but the same color is visible also for collagen. This is because the spectrum of mercury lamp has very small values for ultraviolet wavelengths. It means that inﬂuence of the expected blue color is reduced at the cost of very high red color peak in mercury light spectrum. That energy multiplied by medium values retrieved from EEM gives red color as a dominant part of the emission spectrum. Finally for xenon light collagen photons seem to be violet instead of blue. The reason of that eﬀect is related with the shape of the spectrum. For UV wavelengths the function is going up, after that it stays at almost the same level to the end of the range. The contribution of those constant values is very small due to the speciﬁc structure of collagen EEM. The most important is ultraviolet part, which despite of low values has signiﬁcant role in the ﬁnal appearance.

344

A. Zacher

References 1. Latos, W., Kawczyk-Krupka, A., Ledwon, A., Kosciarz-Grzesiok, A., Misiak, A., Sieron-Stoltny, K., Sieron, A.: The role of autoﬂuorescence colonoscopy in diagnosis and management of Solitary Rectal Ulcer Syndrome. In: SPIE Photonics West Conferences and Courses. Biomedical Optics. Imaging, Manipulation and analysis of Biomolecules, Cells and Tissues VI. Cell and tissue functional imaging, San Jose (2008) 2. Zhu, C., Liu, Q., Ramanujam, N.: Eﬀect of ﬁber optic probe geometry on depthresolved ﬂuorescence measurements from epithelial tissues: a Monte Carlo simulation. Journal of Biomedical Optics 8(2), 237–247 (2003) 3. Ledwon, A., Bieda, R., Kawczyk-Krupka, A., Polanski, A., Wojciechowski, K., Sieron- Stoltny, K., Sieron, A.: The possibilities of improvement the sensitivity of cancer ﬂuorescence diagnostics by computer image processing. In: SPIE Photonics West- Conferences and Courses. Biomedical Optics. Biomedical Optics. Imaging, Manipulation and analysis of Biomolecules, Cells and Tissues VI. Advances in Bioimaging: Computation and Image Analysis, San Jose (2008) 4. Dacosta, R.S., Andersson, H., Wilson, B.C.: Molecular Fluorescence Excitation – Emission Matrices Relevant to Tissue Spectroscopy. Photochem. Photobiol. 78(4), 384–392 (2003) 5. Wang, L., Jacques, S.L.: Monte Carlo Modeling of Light Transport in Multi-layered Tissues in Standard C. University of Texas M. D. Anderson Cancer Center (1992) 6. Jensen, H.W., Marschner, S.R., Levoy, M., Hanrahan, P.: A Practical Model for Subsurface Light Transport. In: Proceedings to Siggraph, pp. 511–518 (2001) 7. Donner, C., Jensen, H.W.: Rendering Translucent Materials Using Photon Diﬀusion. In: Eurographics Symposium on Rendering, pp. 234–251 (2007) 8. Quan, L., Nimunkar, A., Hagl, D.M.: Veriﬁcation of Monte Carlo modeling of ﬂuorescence in turbid medium. Optical Society of America (2001) 9. Eker, C.: Optical Characterization of tissue for medical diagnostics. Doctoral Thesis. Department of Physics. Lund Institute of Technology (October 1999) 10. Marin, N.M.: Autoﬂuorescence and diﬀuse reﬂectance patterns in cervical spectroscopy. Partial Fulﬁllment of the Requirements for the Degree of Doctor of Philosophy. The University of Texas at Austin (December 2005) 11. Mertens, T., Kautz, J., Bekaert, P., Van Reeth, F., Seidel, H.P.: Eﬃcient Rendering of Local Subsurface Scattering. Computer Graphics Forum 24(1), 41–49 (2005) 12. Zonios, G.I., Cothren, R.M., Arendt, J.T., Wu, J., Van Dam, J., Crawford, J.M., Manoharan, R., Feld, M.S.: Morphological Model of Human Colon Tissue Fluorescence. IEEE Transactions on Biomedical Engineering 43(2) (1996) 13. Ramanujam, N.: Fluorescence Spectroscopy In Vivo. In: Meyers, R.A. (ed.) Encyclopedia of Analytical Chemistry, pp. 20–56. John Wiley & Sons Ltd., Chichester (2000) 14. Pharr, M., Humphereys, G.: Physically based rendering. From theory to implementation. Morgan Kaufmann, San Francisco (2004) 15. Swenson, E.A., Rosenberger, A.E., Howell, P.J.: Validation of Endoscopy for Determination of Maturity in Small Salmonids and Sex of Mature Individuals. Transactions of the American Fisheries Society 136, 994–998 (2007) 16. Called de Calipso, N.: Optical Short Course International. In the box Optics of Digital Projectors Bi-monthly eNewsletter. 6679, Tuscon, AZ, USA, www.oscintl.com 17. Day, D.C.: Technical Report. Spectral Sensitivities of the Sinarback 54 Camera, http://www.art-si.org

Mixtures of Classifiers for Recognizing Standing and Running Pedestrians Raluca Borca-Mure¸san, Sergiu Nedevschi, and Florin M˘ aguran Technical University of Cluj-Napoca Barit¸iu 26-28, Cluj-Napoca, Romania http://www.ac.utcluj.ro

Abstract. Recognizing pedestrians in traﬃc scenarios is an important task for any smart vehicle application. Within the context of a realtime stereo based driving assistance system, this paper presents a novel method for recognizing pedestrians. We have designed a metaclassiﬁcation scheme composed of a mixture of Bayesian and boosted classiﬁers that learn the discriminant features of a pedestrian space partitioned into attitudes like pedestrian standing and pedestrian running. Our experiments show that the mixture of classiﬁers proposed outperforms a single classiﬁer trained on the whole un-partitioned object space. For classiﬁcation we have used a probabilistic approach based on Bayesian Networks and Adaptive Boosting. Two types of features were extracted from the image: anisotropic gaussians and histograms of gradient orientations (HOG). Keywords: Pedestrian recognition, classiﬁcation, anisotropic gaussians, histogram of gradient orientations.

1

Introduction

In the context of a driving assistance system the term “pedestrian” is very complex and encompasses a large variety of forms. It basically means a human that is situated in a traﬃc scene. Pedestrians may have diﬀerent sizes, shapes, colors, poses, wore many accessories. Very often, pedestrians are found in heavy cluttered urban environments and the problem of separating the pedestrians from the background becomes a non-trivial task. The expression “pedestrian recognition” refers to separating pedestrians from all the other objects that may appear in a traﬃc scene, i.e. recognizing which objects are pedestrians and which are not. The aim of our research is to develop a reliable pedestrian recognition module able to distinguish pedestrians from other objects that appear in a highly-cluttered urban environment. The pedestrian recognition module is integrated in a driving assistance system [1] that, based on 3D and 2D information, generates the pedestrian object hypotheses. The 3D pedestrian hypothesis is projected onto the image plane generating a 2D pedestrian assumption represented by a 2D image window. This 2D image window is scaled to a ﬁxed size and send L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 345–355, 2009. c Springer-Verlag Berlin Heidelberg 2009

346

R. Borca-Mure¸san, S. Nedevschi, and F. M˘ aguran

as input to our meta-classiﬁcation module whose purpose is to evaluate each pedestrian assumption and to provide a conﬁdence measure. The proposed pedestrian recognition module divides the space of pedestrian hypotheses into two classes: pedestrians running and pedestrians standing. The division of attitudes reduced the complexity of the humanoid object set. For each class we have trained a classiﬁer and we have used two diﬀerent feature sets. Choosing the features implied in the design of a classiﬁer represents a very important step. A robust set of attributes must be used in order to recognize the humanoid shape in a cluttered backround and under diﬃcult illumination conditions. Our approach exploits descriptors based on histogram of gradient orientations and descriptors based on anisotropic Gaussians. The contribution of this paper resides in the development of a mixed classiﬁcation scheme for pedestrian recognition based on a partitioned pedestrian space. We have trained diﬀerent classiﬁers for two categories of pedestrian attitudes: standing and running. We show that the obtained meta-classiﬁer outperforms previous approaches that use the whole un-partitioned pedestrian space. The rest of the paper is structured as follows: section 2 presents previous work on the pedestrian detection problem. Details about the construction of the metaclassiﬁcation scheme are given in section 3. The way of dividing the pedestrian object space, the number of images used for training and tested are underlined in section 4. Experiments and results are discussed in section 5 and section 6 ends the paper with conclusions and ﬁnal remarks.

2

Related Work

Research in the ﬁeld of pedestrian detection has examined a range of technologies, including contact/impact sensors [2], laser, radar and computer vision (both visible and infrared spectrum). Vision sensors are the most promising, because they provide a high spatial resolution, enabling better discrimination between pedestrians and non-pedestrians. Imaging sensors are used in various conﬁgurations and they utilize visible light and infrared radiation. Video sensors are a natural choice for intelligent driver support systems. The processing of video data to extract useful information is a complex task. Many existing approaches in the literature use shape techniques for recognizing spatial human features. For example Gavrila in [3] used a shape ﬁnding method called the Chamfer system to ﬁnd pedestrian shapes with a set of hand crafted image templates of pedestrians in diﬀerent poses. Haar wavelet transform is used by [4] to extract a pedestrian shape representation. They train a Support Vector Machine to learn a model of pedestrians in a front/ rear pose, producing strong results. In [5] the classiﬁcation is performed based on the vertical symmetry that the human ﬁgure exhibits. An eﬃcient cascade classiﬁer used to recognize image patterns is developed by [6]. The cascade classiﬁer is trained by an exhaustive selection of the best weak classiﬁers, and the combination of these weak learners results in a strong classiﬁer with impressive results. The classiﬁer was originally

Mixtures of Classiﬁers for Recognizing Standing

347

created for face detection and then it was applied to pedestrian detection for surveillance cameras [6]. Motion techniques are used to detect human walking patterns in image sequences. A model of the human gait in various poses is used by [7] to detect temporal patterns which resemble walking pedestrians. The drawback of this method is that it will not detect stationary pedestrians nor any pedestrian not moving across the camera’s ﬁeld of view. Other methods are based on stereo data and make use of depth information. Most systems rely on the the disparity map and make some kind of segmentation of this map to detect objects [8], [9] or use a v-disparity approach [10].

3

Overview of the Method

The aim of this research is to propose a 2D image based pedestrian recognition expert. This expert is a component of a probabilistic pedestrian detection framework exploiting 2D, 3D and motion information. The connection between the framework and the expert is bidirectional, the ﬁrst providing pedestrian hypothesis with the associated 2D image window, and the second one returning the conﬁdence score. The probabilistic pedestrian detection framework is described in [1] and [11]

Fig. 1. Pedestrian recognizer within the 3D pedestrian detection framework

348

R. Borca-Mure¸san, S. Nedevschi, and F. M˘ aguran

Figure 1 presents how the pedestrian classiﬁer is applied within the context of the probabilistic pedestrian detection framework. The framework detects the obstacles, using 3D information and it generates 2D image windows on which the pedestrian recognition algorithm we propose is applied. The steps we performed for constructing the mixed classiﬁcation scheme are: (1) Divide the space of pedestrians into two classes of attitudes: running and standing pedestrians; (2) For each of the classes train a classiﬁer. Our experiments have shown that best recognition results were obtained using a Bayesian network trained on HoG descriptors for the pedestrians running class and a boosted classiﬁer build using anisotropic gaussians for the pedestrians standing class; (3) Perform classiﬁcation and evaluate the accuracy of the detection. The ﬂow of our pedestrian recognition algorithm is presented in Figure 2.

Fig. 2. Flow of the pedestrian recognition algorithm for two categories: pedestrians running and pedestrians standing

3.1

Feature Description

In our work we use two sets of features: anisotropic gaussians and histogram of gradient orientations. Anisotropic Gaussian features were introduced by [12]. They are constructed from base functions of an over complete basis. The expansion of any image in the base is not unique. The generative function φ(x, y) : R2 → R is 2 described by the equation φ(x, y) = xe−(|x|+y ) . It is made of a combination of a Gaussian and its ﬁrst derivative. This presents the ability of approximating eﬃciently contour singularities with a smooth low resolution function in the direction of the contour and it approximates the edge transition in the orthogonal direction. Figure 3 shows some anisotropic gaussian kernels with diﬀerent scaling, bending, rotating and translating parameters. Figure 4 displays some features computed on a pedestrian image.

Fig. 3. Some examples of anisotropic gaussian kernels

Mixtures of Classiﬁers for Recognizing Standing

349

Fig. 4. Original image and two anisotropic gaussians computed on it

Diﬀerent transformations can be applied to this generative function: – Translation by (x0 , y0 ): Tx0 ,y0 φ(x, y) = φ(x − y0 , y − y0 ) – Rotation by θ: Rθ φ(x, y) = φ(xcosθ − ysinθ, xsinθ + ysinθ). – Bending by r: y φ(r − (x − r)2 + y 2 , r arctan( r−x )) , if x < r Br φ(x, y) = π φ(r − |y|, x − r + r × 2 ) , if x ≥ r – Anisotropic scaling by (sx , sy ): Ssx ,sy φ(x, y) = φ( sxx , syy ) By combining these four basic transformations, we obtain a large collection D of ψsx ,sy ,θ,r,x0 ,y0 as deﬁned by equations bellow: ψi (x, y) = ψsx ,sy ,θ,r,x0 ,y0 (x, y) = Tx0 ,y0 Rθ Br Ssx ,sy φ(x, y) The obtained anisotropic features were normalized and as their number was very large we have selected randomly 2640 features. Histogram of Oriented Gradients turned out to have good results for pedestrian classiﬁcation [13]. For each point of an image I in the dataset we have computed the gradient magnitude M and orientation θ as follows: Dy Dx = (I ∗ B) ∗ Gx , Dy = (I ∗ B) ∗ Gy , M = (Dx )2 + (Dy )2 , θ = arctan Dx where B is a Gaussian smoothing kernel, Gx = [−1, 0, 1]T , Gy = [−1, 0, 1]. Next, the image is divided into non-overlapping rectangular cells of equal dimension as depicted in Figure 5. For each cell we compute a weighted histogram of gradient orientations. In each pixel location the orientation gives to the histogram a vote weighted by the gradient magnitude at the position of the respective pixel. The orientation bins are evenly spaced over 0◦ - 180◦ . The last step for HOG descriptors extraction is represented by normalization. The cells are grouped into overlapping blocks of various dimensions. Block overlapping ensures the fact that each scalar cell response contributes to several components of the ﬁnal feature vector. The ﬁnal vector of descriptors will contain all the components of the normalized cell responses from all of the blocks in the detection window.

350

R. Borca-Mure¸san, S. Nedevschi, and F. M˘ aguran

Fig. 5. a) Original Image; b) Gradient magnitude and cell division; c) Histogram of Oriented Gradients computed on each cell; d) Cell grouping into blocks within which normalization is made

Normalization scheme. For both feature types we use L2-norm normalization algorithm. Suppose that we have a vector of k features denoted by fd . The value of a feature fd (i) can either be the response to an anisotropic gaussian kernel or a histogram value (for HOG attributes). The normalization equation is: fd (i) = √ k fd (i) 2 where is a small constant. i=1 (fd (i))

3.2

+

Classification

Classiﬁcation has a decisive role in any object detection system. We have implemented a an adaptive boosting classiﬁcation scheme and a Bayesian network learning scheme. Adaptive Boosting is a learning technique that has been used for pedestrian detection providing good results [6]. The classiﬁer is a set of weak classiﬁers arranged in cascade. Each classiﬁer in the cascade is a thresholded sum of weighted features and each feature is a thresholded sum of gaussian features. Let F be a gaussian feature with the corresponding gaussian ﬁlter f and threshold θ. The 1 if response(f ) × p ≥ θ × p output of the weak learner is: h(F ) = , where 0 otherwise response(f ) is the response of the gaussian ﬁlter f, p ∈ {±1}. The output of a classiﬁer in the cascade is then: n 1 if i=1 αi hi (Fi ) ≥ θclassif ier outputclassif ier = 0 otherwise where θclassif ier is the classiﬁer threshold and {F1 , F2 , . . . , Fn } are the features in the classiﬁer with the associated weak learners hi (Fi ) and weights αi Bayesian networks (or belief networks) have been used successfully for pedestrian classiﬁcation in [14], [15].

Mixtures of Classiﬁers for Recognizing Standing

351

The Bayesian network U is a pair B = G, Θ, where G is a directed acyclic graph whose vertexes correspond to attributes in the training set, and whose edges represent direct dependencies between attributes. We model the set of attributes by the random variables A1 , . . . , An . The graph G encodes independence assumptions: each variable Ai is independent of its nondescendants given its parents in G. Θ represents the set of parameters that quantiﬁes the network. It contains a parameter θai |Πai = PB (ai |Πai ) for each possible value ai of Ai , and Πai , of ΠAi , where ΠAi denotes the set of parents of Ai in G. The Bayesian network B deﬁnes distribution over U given n a unique joint probability n by: PB (A1 , . . . , An ) = i=1 PB (Ai |ΠAi ) = i=1 θAi |ΠAi So, being given a set of HOG attributes a1 , . . . , an and an attribute describing the class, c, the classiﬁer based on B returns the label c that maximizes the posterior probability PB (c|a1 , . . . , an ). In our experiments we have used WEKA[16] for training a Bayesian Network.

4

Dataset and Methodology

Dataset. We have considered two classes of attitudes for pedestrians: standing and running. For each class we have used a training set and a testing set. We have collected our samples in the category standing from the MIT pedestrian database [17] and from the INRIA pedestrian database [18] that contain images of pedestrians in city scenes. We have collected pictures for the category ’running pedestrians’ from several images taken from the Internet or from our personal photos and some from the INRIA [18] database. We build a set of 300 images of diﬀerent running pedestrians. In all the pictures the pedestrians occupy the central position. We have applied a 4 small in-plane rotations with 5◦ , 10◦ , 5◦ and -10◦ to each image from both classes of attitudes, hence we have obtained a dataset of 1500 pictures of running pedestrians and 2500 pictures of standing pedestrians. Methodology. For each category we have considered images of dimension 18x36 pixels. We have divided the datasets for the standing pedestrian attitude into 2100 training samples and 400 testing samples and the dataset for running pedestrians was split into 1100 train pictures and the 400 test pictures. The initial negative training set had 12000 images (also of dimension 18x36 pixels) sampled randomly from person-free training photos. An initial training is made and the obtained detector is tested on a larger set of negative images that were not in the initial negative training set. All the false positives are added to the training set and the process is repeated until we reach a good accuracy of the detection. Hence, we obtain a classiﬁer for each category of the considered pedestrian attitudes i.e. running and standing. For each category we have trained two separated classiﬁers, one that uses HOG features and another that exploits anisotropic gaussians. We have compared the results obtained using these two feature sets, HOG providing the best results for pedestrians running and anisotropic gaussians gave best results for pedestrians standing.

352

R. Borca-Mure¸san, S. Nedevschi, and F. M˘ aguran

We have grouped the classiﬁers trained for each category in a mixture of classiﬁers. The next step was the comparison of our method based on the partitioning of the pedestrian space into several classes, with a method that trains a classiﬁer using the whole pedestrian set. In order to perform this comparison we build a third classiﬁer that used as positive training set 1100 images from the category running and the 2100 train images from the category standing and the same set of negative samples as the classiﬁers build separately for each category. Considering the two test sets (for pedestrians running and for pedestrians standing) we evaluated the performance of a Bayesian detector trained using the whole object space, an adaptive boosting classiﬁer trained on the whole object space and the performance of the mixture of classiﬁers (that uses belief networks for pedestrians running and adaboost for pedestrians standing). As detailed in section 5 the best results of pedestrian recognition are obtained when applying the mixed scheme. The detection window is of 18x36 pixels. We have not described the way in which the mixture of classiﬁers is applied for larger images because the recognition module is designed to work within an existing probabilistic pedestrian detection framework[11], [1] which provides the pedestrian hypotheses.

5

Experimental Results

In this part we will present some results of our system. For both categories, pedestrians running and pedestrians standing, we have computed the two feature sets: histograms of gradient orientations and anisotropic gaussians. For our detection window of 18x36 pixels we have retained for HoG features: a cell size of 3x6 pixels with a histogram having 8 bins, and a block size of 3x3 cells resulted in a feature set of 128 attributes. We have evaluated our method with 400 positive samples of running pedestrians, 400 positive images of standing pedestrians and 10000 non-pedestrians.

Fig. 6. Results for pedestrians running

Figure 6 shows the detection rate of the classiﬁer for the category pedestrians running(PedRun). The negative training set is shortly referenced with NonPed. We have depicted the true positive rate (TP), precision (Prec.), recall (Rec) and

Mixtures of Classiﬁers for Recognizing Standing

353

Table 1. Recognition rate of the classiﬁer trained on the running set using HOG TP Prec. Rec. ROC-A PedRun 0.901 0.02 0.901 0.964 Table 2. Recognition rate of the classiﬁer trained on the standing set using Anisotropic Gaussian features TP Prec. Rec. ROC-A PedStand 0.87 0.57 0.87 0.674

Fig. 7. Results for pedestrians standing

area under ROC (ROC-A). Figure 6 also includes the ROC for the classiﬁer trained on pedestrian running set. Figure 7 shows the detection rate of the classiﬁer trained for the category pedestrians standing (denoted in the table with PedStand) using anisotropic gaussian features. Figure 7 plots the ROC for the classiﬁer for the category pedestrians standing. Table 3 makes a comparison of the recognition rates for the Bayesian Network trained using HOG features and the meta-classiﬁer proposed in our research. We depict the values of the true positive rate (TP) and of the true negative rate (TN). The results show that our approach which trained a classiﬁer for each class of objects and formed a meta-classiﬁcation scheme gives better results than the Bayesian Network trained on the mixed set of pedestrians (running and standing). Table 3 makes a comparison of the recognition rates for the Adaptive Boosting trained using anisotropic gaussian features and the metaclassiﬁer proposed. We present the values of the true positive rate (TP) and of the true negative rate (TN). The results show that our approach which trained a classiﬁer for each class of objects and formed a meta-classiﬁcation scheme gives better results than the boosted classiﬁers trained on the mixed set of pedestrians (running and standing). For both sets of features, the resulting meta-classiﬁer outperforms the detection rate of a classical learner (either Bayesian Network or Adaptive Boosting classiﬁer) trained on the whole pedestrian feature space.

354

R. Borca-Mure¸san, S. Nedevschi, and F. M˘ aguran

Table 3. (a) detection rates obtained using a Bayesian Network trained on the unpartitioned object space using HOG features and the proposed metaclassiﬁcation scheme. (b) detection rates obtained using boosted classiﬁers trained on the unpartitioned object space using anisotropic gaussians and the proposed metaclassiﬁcation scheme. Classiﬁer trained on Run test Stand test

6

ped. ped. running & running standing standing TP TN TP TN TP TN 0.90 0.95 0.55 0.91 0.76 0.91 0.68 0.90 0.89 0.91 0.86 0.91

Classiﬁer trained on Run test Stand test

ped. ped. running & running standing standing TP TN TP TN TP TN 0.75 0.70 0.52 0.78 0.70 0.82 0.57 0.63 0.87 0.85 0.79 0.82

Conclusion

The presented approach shows the development of a pedestrian recognition module based on the mixture of classiﬁers obtained using belief networks and adaptive boosting. The features used for classiﬁcation are histograms of gradient orientations and anisotropic gaussians. The result of our research is a pedestrian recognition module that is integrated in a stereo-based driving assistance system. As pedestrians represent complex objects with numerous attitudes we will perform a deep analysis of the pedestrian attitude space and investigate the factors that lead to a good partitioning scheme of the pedestrian space into classes of non-overlapping attitudes ( for examples pedestrians running or walking in a certain direction, pedestrians standing in front of the camera etc.) . The metaclassiﬁer we have build runs its classiﬁers in parallel and the one that gives the best detection result labels an instance under test. Diﬀerent ideas are currently under evaluation for improving the results. We want to build an automatic method for pedestrian attitude recognition, that will allow a single classiﬁer from the meta-classiﬁcation scheme to be active at one moment (ensuring by this a gain in speed, as not all the classiﬁers will run in parallel). More advanced classiﬁcation algorithms can increase the accuracy of our results.

References 1. Nedevschi, S., Danescu, R., Marita, T., Oniga, F., Pocol, C., Sobol, S., Tomiuc, C., Vancea, C., Meinecke, M.M., Graf, T., To, T.B., Obojski, M.A.: A sensor for urban driving assistance systems based on dense stereovision. In: Proceedings of Intelligent Vehicles, pp. 278–286 (June 2007) 2. Fredriksson, R., H¨ aland, Y., Yang, J.: Evaluation of a new pedestrian head injury protection system with a sensor in the bumper and lifting of the bonnet’s rear edge. In: Proceedings of the 17th International Conference on the Enhanced Safety of Vehicles (ESV), Paper no 131 (2001) 3. Gavrila, D., Munder, S.: Multi-cue pedestrian detection and tracking from a moving vehicle. In: International Journal of Computer Vision, vol. 73, pp. 41–59. Springer, Heidelberg (2007)

Mixtures of Classiﬁers for Recognizing Standing

355

4. Papageorgiou, C., Poggio, T.: Trainable pedestrian detection. In: International Conference on III Pattern (1999) 5. Fascioli, A., Broggi, A., Bertozzi, M., Sechi, M.: Shape-based pedestrian detection. In: IEEE Intelligent Vehicle Symposium (2004) 6. Snow, D., Viola, P., Jones, M.: Detecting pedestrians using patterns of motion and appearance. International Journal of Comupter Vision 63, 153–161 (2005) 7. Curio, C., Edelbrunner, J., Kalinke, T., Tzomakas, C., von Seelen, W.: Walking pedestrian recognition. IEEE Transactions on Intelligent Transportation Systems 1(3), 155–163 (2000) 8. Kelly, P., O’Connor, N.E., Smeaton, A.F.: Pedestrian detection in uncontrolled environments using stereo and biometric information. In: VSSN 2006: Proceedings of the 4th ACM international workshop on Video surveillance and sensor networks, pp. 161–170. ACM Press, New York (2006) 9. Nedevschi, S., Tomiuc, C., Bota, S.: Stereo based pedestrian detection for collision avoidance applications. In: Proceedings of ICRA 2007 Workshop: Planning, Perception and Navigation for Intelligent Vehicles, pp. 39–44 (2007) 10. Bertozzi, M., Binelli, E., Broggi, A., Rose, M.D.: Stereo vision-based approaches for pedestrian detection. In: Conference on Computer Vision and Pattern Recognition, p. 16. IEEE Computer Society Press, Los Alamitos (2005) 11. Nedevschi, S., Danescu, R., Frentiu, D., Marita, T., Oniga, F., Pocol, C., Schmidt, R., Graf, T.: Stereovision approach for obstacle detection on non-planar roads. In: IEEE and IFAC International Conference on Informatics in Control, Automation and Robotics, August 2004, pp. 11–18 (2004) 12. Peotta, L., Granai, L., Vandergheynst, P.: Very low bit rate image coding using redundant dictionaries. In: Proceedings of the SPIE, Wavelets: Applications in Signal and Image Processing X. SPIE 2003, vol. 5207, pp. 228–239 (2003) 13. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, June 2005, pp. 886–893 (2005) 14. Dimitrijevic, M., Lepetit, V., Fua, P.: Human body pose detection using bayesian spatio-temporal templates. Comput. Vis. Image Underst. 104(2), 127–139 (2006) 15. Wu, B., Nevatia, R.: Detection and tracking of multiple, partially occluded humans by bayesian combination of edgelet based part. International Journal of Computer Vision, 247–266 (November 2007) 16. WEKA: Data mining software in java, http://www.cs.waikato.ac.nz/ml/weka/ 17. Center for biological and computational learning at MIT: Pedestrian database, http://cbcl.mit.edu/software-datasets/PedestrianData.html 18. INRIA pedestrian dataset, http://pascal.inrialpes.fr/data/human/

Automatic Classification of Wood Defects Using Support Vector Machines Irene Y.H. Gu1 , Henrik Andersson1 , and Ra´ ul Vicen2 1

Dept. of Signals and Systems, Chalmers Univ. of Technology, 41296, Sweden [email protected], [email protected] 2 Dept. of Signal Theory and Communications, Univ. of Alcal´ a, Spain [email protected]

Abstract. This paper addresses the issue of automatic wood defect classiﬁcation. We propose a tree-structure support vector machine (SVM) to classify four types of wood knots by using images captured from lumber boards. Simple and eﬀective features are proposed and extracted by ﬁrst partitioning the knot images into 3 distinct areas, followed by applying an order statistic ﬁlter to yield an average pseudo color feature in each area. Excellent results have been obtained for the proposed SVM classiﬁer that is trained by 800 wood knot images. Performance evaluation has shown that the proposed SVM classiﬁer has resulted in an average classiﬁcation rate of 96.5% and false alarm rate of 2.25% over 400 test knot images. Our future work includes more extensive tests on large data set and the extension of knot types. Keywords: wood knot classiﬁcation, wood defect detection, wood defect inspection, support vector machine, feature extraction, machine vision.

1

Introduction

This paper addresses the issue of automatic classiﬁcation of wood defects. There has been a growing interest in research and development of automatic wood defect detection and classiﬁcation methods and systems, especially in wood-rich countries such as Scandinavian. Currently, most lumber board inspections (such as detecting defects including freckle, bark, pitch pockets, wane, split, stain, hole and various types of knots) and wood grading are still performed by trained human experts. Long hours exposing to wood inspection would cause eye fatigue hence low eﬃciency and accuracy in the inspection. It is found [1,2] that human visual inspections rarely achieved better than 70% performance in grading lumbers largely due by eye fatigue. The growing number of new imaging techniques, available databases of wood images, and sophisticated techniques in computer vision have made online automatic wood inspections an ever approachable reality, further more, with a performance that well exceeds those of human inspectors. Many promising techniques have been studied and developed on automatic detecting and classiﬁcation of wood defects in the last decade. For example, [3] proposed a genetic algorithm for the classiﬁcation of 10 types of defects (birds L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 356–367, 2009. c Springer-Verlag Berlin Heidelberg 2009

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

357

eye and freckle, bark and pitch pockets, wane, split, blue stain, stain, pitch, dead knot, live knot, and hole). Using a database containing color images from 2958 board faces where each defect is characterized by a feature vector of size 182, the online system has resulted 80% classiﬁcation rate. [5,4] proposed a recognition system of defects on lumber boards using simple RGB color percentile features calculated from arbitrary rectangular regions. Several defect recognition stages, starting from sound wood elimination, defect identiﬁcation, to classiﬁcation of knots, color defects and splits, are used. Each stage reduces the amount of image data needed to be processed by the subsequent stages. The ﬁnal stage is quality grading, which collects the information on recognized defects on each board surface, and determines the quality grade according to the grading rules. The system uses a non-supervised clustering-based approach for detecting and recognizing defects in lumber boards. A self-organizing map (SOM, a special type of artiﬁcial neural network) is employed for discriminating between sound wood and various types of defects. Human involvement needed for training is minimal. The method has achieved low false alarm rate. Further improvement has been made in [7] for the classiﬁcation of wood defects. Four diﬀerent types of knots (encased, leaf, edge and sound knots) were included in their study. The classiﬁer employs supervised and non-supervised artiﬁcial neural networks (self organized neural network - SONN and fuzzy SONN) as well as feedforward perceptron where the knot features are obtained from Gabor ﬁlters. The system using fuzzy SONN has resulted in a classiﬁcation rate of 91.17%, and of 85% for the perceptron on the test knot set, however, the false alarm rate is not mentioned. Despite their promising in classifying wood defects, many challenging and open issues remain to be solved. On the other hand, support vector machines (SVMs) [8,9] have drawn increasing interest lately for pattern classiﬁcation. SVM classiﬁers have the sound mathematic background One of the attractions of SVM classiﬁer is that it has a solid mathematical foundation, the statistical learning theory. Unlike artiﬁcial neural network classiﬁers that minimize the error for the training set, a SVM classiﬁer is designed to minimize the generalization error, i.e., the error on the test set under the structural risk minimization (SRM) principle. There is a guarantee of the upper error bound on the test set based on statistical learning theory. Further more, the complexity of a SVM classiﬁer can be controlled and is associated with the so-called VC dimension. Despite many advantages of statistical learning and SVMs, they have not been used for wood defect or wood knot classiﬁcation. Motivated by the above, we propose an automatic wood defect classiﬁcation method based on support vector machines (SVMs). A tree structure SVM classiﬁer is employed for classifying four diﬀerent types of wood knots, including sound knots, dry knots, pin knots, black and rotten knots. To the best of our knowledge, this is the first reported work on using SVMs for wood knot classification. Simple features are examined for characterizing each type of wood knots. The SVM classiﬁer has been trained by using 800 wood knot images from Sweden, as well as by using some mixture of wood knot images from Sweden and Finland.

358

I.Y.H. Gu, H. Andersson, and R. Vicen

Despite using simple features in our system, the proposed SVM classiﬁer has yielded an excellent performance [10]. The remaining paper consists of the description of wood types, knot images and their related properties (in Section 2), a brief description of theory of SVMs (in Section 3), feature selection and extraction methods (in Section 4), the structure of the SVM classiﬁer (in Section 5), experimental results and performance evaluations (in Section 6), and ﬁnally conclusions (in Section 7).

2

Wood Knot Types

In this section we brieﬂy describe ﬁve wood knot types that will be studied in the paper. Although there are many more possible types of wood knots, readers are referred to [11] for more details and their speciﬁcation. Fig.1 shows typical image examples for these ﬁve types of wood knots, namely, sound knots, dead knots, black knots, rotten knots and pin knots. As can be seen from the ﬁgure, their images diﬀer in color, shape, texture and size. Sound knots are usually considered as harmless, formed from tree branches. Images of typical sound knots may have some circular shaped curves reﬂecting the age of the branches, also have slightly dark colors and diﬀerent textures as compared with the background wood, as shown in Fig.1(a). Dead knots are formed when a tree branch ceased to function. An image of a dead knot usually shows a darker circular shaped curve where the interior area of knot has a slightly darker color than the normal wood color, see Fig.1(b). Black knots are dead knots where the interior of a knot is also dark colored, see Fig.1(d). Rotten knots are considered as harmful. Images of rotten knots usually show a dark interior area with a darker circular shape boundary, see Fig.1(d). Pin knots are small dark knots, usually with a diameter no more than 7mm (see Fig.1(c)). Since black knots and rotten knots in a lumber board are usually considered as unsuitable to use, and would usually be cut oﬀ when processing the wood board, we shall treat them as one class in our classiﬁcation system.

3

Statistical Learning and SVM Classifiers

A special characteristic of a SVM is that, instead of dimension reduction commonly employed in pattern classiﬁcation, the input space is nonlinearly mapped by Φ(·) onto a high dimensional feature-space, where Φ(·) is a kernel satisfying Mercer’s condition. As classes are more likely to be linearly separable in the high dimensional space rather than in a low dimensional space. Let N input training samples and their class labels be (xi , di ), xi ∈ Rm0 (i.e. m0 dimensional input-space) and di ∈ Y (i.e. the decision-space). A nonlinear mapping function Φ(·) is ﬁrst applied which maps the input-space Rm0 into a high→ Φ(xi ). Then, another function dimensional feature-space F : Φ : Rm0 → F , xi

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

359

Fig. 1. Image from diﬀerent types of wood knots. From top to bottom: Rows (a)-(d): sound knots, dead knots, black knots and rotten knots, and pin knots.

f (·) ∈ F is applied to map the high-dimensional feature-space F onto a decision→ f (Φ(xi )). The best function f (·) ∈ F that may corspace, f : F → Y, Φ(xi ) rectly classify a unseen example (x, d) from a test set is the one minimizing the generalization error, R(f ) = l [f (Φ(x)), d] dP (Φ(x), d), where l(·) is the loss function, and P (Φ(x), d) is the probability. An error is occurred if f (Φ(x)) = d. Since P (x, d) is usually unknown, we cannot directly minimize R(f ). Although there exist many f (·) ∈ F that give nearly perfect classiﬁcation on the training set, f (·) will be chosen among them, such that it results in the minimum error on the test set, based on the structural risk minimization (SRM) principle. For all δ > 0 and f ∈ F, it is shown that the bound of the generalization error for a SVM classiﬁer is

δ h(ln( 2N h + 1) − ln( 4 ) (1) N N where the empirical error is Remp (f ) = N1 i=1 l (f (Φ(xi )), di ), and h is the VC dimension for the function class F. (1) holds with a probability of at least (1 − δ) for N > h. Choosing a nonlinear mapping function Φ in a SVM is associated with selecting a kernel function. Kernel functions in a SVM must satisfy Mercer’s condition [8].

R(f ) ≤ Remp (f ) +

360

I.Y.H. Gu, H. Andersson, and R. Vicen

Soft-Margin C-SVMs A soft-margin C-SVM is used for the wood knot classiﬁcation as it can handle linearly nonseparable classes. A C-SVM allows some training error Remp (f ) in order to achieve less error on the test set. Roughly speaking, optimal solution to a SVM classiﬁer is associated with ﬁnding the maximum margin (∝ 1/||w||2 ), where a margin is the shortest distance between the separating boundary and a correctly classiﬁed training sample that is nearest to the separating boundary. A soft-margin SVM can be described by the quadratic optimization problem, N 1 minw,b,ξ w2 + C ξi (2) 2 i=1 subject to the condition that all training samples being classiﬁed satisfy, di ( w, Φ(xi ) + b) ≥ 1 − ξi ,

ξi ≥ 0, i = 1, 2, · · · , N

(3)

where C ≥ 0 is a regularization parameter determined empirically, ξi are the slack variables. This leads to the Lagrangian optimization problem (primal form), L(w, b, α, ξ, μ) =

N N 1 w2 + C ξi − μi ξi − 2 i=1 i=1

−

N

αi [di ( w, Φ(xi ) + b) − 1 + ξi ]

(4)

i=1

where μi is the lagrange multiplier to enforce the positivity of ξi . Using the kernel k(·, ·), an important concept in SVMs, a dual form equivalent to (4) is ⎧ ⎫ N N ⎨ ⎬ 1 maxα αi − αi αj di dj k(xi , xj ) ⎩ ⎭ 2 i,j=1 i=1 (5) N subject to : αi di = 0, C ≥ αi ≥ 0, i = 1, 2, · · · , N i=1

The optimal solution satisﬁes the KKT (Karush-Kuhn-Tucker) conditions [8] and can be obtained by solving the (convex) quadratic programming (QP) problem. Finally, the decision function for a soft-margin SVM classiﬁer is, N s f (x) = sgn αi di k(x, xi ) + b i=1

where Ns is the number of support vectors xi , where they are either on the margin (correct classiﬁed training samples) or inside the margin (incorrectly classiﬁed training samples).

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

361

Fig. 2. Block diagram of a SVM classiﬁer for a two-class case

Fig.2 shows the block diagram of a SVM classiﬁer where x is the input vector drawn from the test set, and xi , i = 1, · · · , Ns , are the support vectors drawn from the training set. Despite the block diagram structure of the SVM looks similar to a RBF neural network, it is worth mentioning their fundamental diﬀerences: First, the weights αi di to the SVM outputs contain lagrange multipliers αi (associated with the constrained optimization in SVM learning that minimizes the generalization error), while the weight in the RBF neural network are selected for minimizing the empirical error. Second, for a general multilayer neural network, activation functions are used in the hidden layer to achieve the nonlinearity, while a SVM employs a kernel function k(·, ·) for nonlinearity, and the kernel is only dependent on the dot product of x from the test set and the support vectors.

4

Feature Selection and Extraction for Wood Knots

Each wood knot sample is described by a rectangular image area containing the knot (referred to as the ”knot image” in the subsequent text), and is then characterized by a feature vector. Selecting features for knots is one of the most essential steps towards a successful classiﬁcation. Observing images from diﬀerent types of knots indicates: – color and texture of knots are essential for human inspections. Since colors of wood knots may vary signiﬁcantly in respect to diﬀerent wood type, wood origins (countries or regions), camera type and settings and lighting conditions, it is therefore essential that the relative color diﬀerences (as reference to their background wood) rather than the absolute colors are exploited. – knot sizes are in most cases not essential features, with an exception for pin knots (featured by its small size), as knot sizes within each knot type can vary signiﬁcantly.

362

4.1

I.Y.H. Gu, H. Andersson, and R. Vicen

Partition Each Knot Image into 3 Areas Based on the Outer Boundary

The following basic idea is exploited for deﬁning features: Noting that each knot image is chosen to be rectangular, while the shape of a wood knot is usually ellipse, round, or with a smooth boundary. Therefore, each knot image contains a small exterior area outside the wood knot that belongs to the background, and can be used as the reference of the ”normal wood” color (or, reference texture) related to the given lumber board. We divide each knot (rectangular) image into 3 areas: the interior area of knot (i.e., the knot area), the exterior area (i.e., a small background area) and the boundary area (set to be the thick outer boundaries of a knot). To detect the outer boundary of a knot, a simple edge detection algorithm is ﬁrst applied to a knot image, followed by edge chaining where nearby edge points with similar edge directions are connected. Outer edge curves that are long (exceed certain lengths) are picked out as the outer boundary curves. Since the outer boundary of wood knot is assumed to be closed, unconnected outer boundaries are then closed by using e.g. B-spline, assuming a smooth wood boundary. It is worth mentioning that: (a) for weak edges, double edge curves may present; (b) edge positions from a simple edge detection algorithm may not exactly correspond to the peak position of an edge proﬁle, if it is not carefully tuned. Observing that for some knot types (e.g., black knots and rotten knots), the color diﬀerence between the boundary points and the interior area of a knot is one of the important characteristics. Therefore, edge curves that are selected as the outer boundary curves shall be widened. Ideally, for double edge curves, the width of a corresponding widened boundary curve would be set as the distance between the double edges so that the area between the double edges can be included (see Fig.3). Otherwise, an outer boundary curve would be extended in both sides of the edge peak location to prevent the possible misplacement of the exact peak position. To simplify the processing for these two occasions, a lowpass ﬁlter is applied to yield widened edge proﬁles. In such a way, a widened outer boundary curve would always include the true boundary proﬁle from the wood knot.

Fig. 3. Double edges are detected for a weak edge. Region 2 is the area of interest for characterizing the wood boundary. A widened boundary for double edges should therefore include regions 1, 2 and 3.

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

4.2

363

Feature Deﬁnition, Average Pseudo Colors and Order Statistic Filters

Once the closed outer boundary of a knot is obtained, a set of image pixels that belong to the interior, exterior or a boundary area can then be determined. For each area in the knot image, a simple feature vector is then deﬁned, x = [x1 , x2 , x3 , x4 ]T

(6)

where x1 is the knot size, x2 , x3 and x4 are the average normalized pseudo colors for the interior, exterior and outer boundary area of knot, respectively. Pseudo colors in each area is normalized by the maximum value in the corresponding area (i.e. interior, exterior, or boundary) selected from the entire training set. 4.3

Order Statistic Filter for Computing Average ”Pseudo Color”

First, let us deﬁne the (normalized) ”pseudo color”. Let the color of an image pixel si be I(si ). A pseudo color is a ﬁltered image color G(si ) = h ∗ (I(si )), where I is the original image, h is a ﬁlter kernel, and ∗ is the linear convolution. To compute the average pseudo color of an area, an order statistic ﬁlter (OSF) is used. Let the area (either interior, exterior or widened boundary area) contains N image pixels. Let the pseudo color of pixels si be G(si ), i = 1, · · · , N . The pseudo color values are ﬁrst sorted in an ascending order, resulting in a sequence G(s(1) ) ≤ G(s(2) ) ≤ · · · ≤ G(s(N ) ). A nonlinear order statistic ﬁlter (or, L-estimator) is the applied, yielding an average (normalized) pseudo color as follows: F (G(s(1) ), G(s(2) ), · · · , G(s(N ) ) = αj G(s(j) ) (7)

j

1/L, if i = (N − L + 1)/2 , and L is empirically determined. (7) 0, otherwise is computed for each area, and then assigned to the corresponding xi in (6). The principle of choosing the ﬁlter for generating the pseudo color is as follows: For each knot color image contains 3 component images (e.g., R,G,B), a lowpass ﬁlter shall be selected so that the ﬁltered color image contains suﬃcient averaging texture and knot boundary information, while most ﬁne details of wood structure lines are smoothed out. Further, the image becomes a onecomponent image. Ideally, the ﬁlter could also be desired to correct the wood color due to the origins of wood and the type of wood, and due to lighting conditions. To simplify the processing, and to limit the scope of this paper, a simple ﬁlter is used. Fig. 4 shows an example of a wood image consisting of 3 component images. Observing the 3 component images indicates that (a) Red color images are somewhat blurred (equivalent to apply a lowpass ﬁlter) where the ﬁne details of the knot are smoothed out, however, the average texture and contrast of the knot against its background are maintained. (b) Green color images contain some more detail structure of knots, although the images are also where αi =

364

I.Y.H. Gu, H. Andersson, and R. Vicen

Fig. 4. Three component images from a sound knot

blurred (or, lowpass ﬁltered). (c) Blue images have a rather low contrast. Since the important knot features used in our method are based on the averaging area features in the interior/exterior/boundary of knots and their relative diﬀerences, we decide to simply use the red component image for our experiments, as this would satisfy our requirement. 4.4

Knot Size

For knot size feature in (6), it is estimated by counting the number of pixels on the outer boundary curve. Since a pin knot signiﬁcantly diﬀer from other types of knot in size, knot size feature is used for distinguishing pin knots from the remaining types of knot without exploring other features.

5

Topology Selection for the SVM Classifier

As shown in Fig.5, a tree structure SVM classiﬁer is proposed for classifying diﬀerent types of wood knots.

Fig. 5. Using a tree-structure SVM for wood knot classiﬁcation

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

365

The basic idea of using a tree-structure SVM classiﬁer is to divide a large classiﬁer into a few smaller and simpler ones. Since pin knots signiﬁcantly diﬀer in size from the remaining types of knots, the classiﬁer would only require to use the size feature x1 to distinguish between pin knots and the remaining knots. After that, the feature vector in the remaining levels of classiﬁer would reduce to [x2 , x3 , x4 ]T . Based on the observation that dead knots clearly diﬀer from the remaining 3 types by having a darker boundary curve against the interior and exterior area, the 2nd level of classiﬁer is to distinguish black and non-black knots. In the last level, ”not recognized knot” type is also deﬁned so that knots do not belong to the 4 designed classes would not be classiﬁed.

6

Experimental Results and Performance Evaluations

First, wood knot images required were manually cut from color images captured from lumber boards. A total of 1200 knot images were used, of 800 for training and 400 for testing. For each knot image, a feature vector is then extracted. For each ”average pseudo color” feature, it is then normalized by using the maximum value of the corresponding feature in the training samples. The same normalization factor is then saved, and subsequently applied to the corresponding feature in the test set. Matlab SVM toolbox from OSU [14] was used. A C-SVM, with a Gaussian 2 RBF (radial-basis function) kernel k(x, y) = exp( −x−y ), was chosen. The γ two parameters (C, γ) were determined prior to the training process. A simple grid-search method in [12] and leave one out cross-validation were used to ﬁnd the best parameters (C, γ). The optimal parameters obtained from such a way are (C, γ)=(3, 6),(6, 6.5) and (5, 6.5) corresponding respectively to 2nd, 3rd and 4th level in the tree-structure SVM. Once (C, γ) were determined, the SVM is then re-trained with the ﬁxed (C, γ) using the entire training data (see Fig.6).

Fig. 6. Block diagram: RBF kernel SVM learning, then classifying x from a test set

6.1

Training SVM Classiﬁers

Two classiﬁers were trained both using 800 knot images (200 for each type): classifier-1 was trained using features from knot images in Sweden, while for

366

I.Y.H. Gu, H. Andersson, and R. Vicen

classifier-2 training, 200 sound knot images consisted of 184 from Sweden and 14 from Finland [13]. 6.2

Testing SVM Classiﬁers

Tests were then performed by 400 knot images (100 for each type). For classifier1, knot images in the test set were from Sweden. However, for classifier-2, 100 sound knot images were from Finland [13], and the remaining 300 were from Sweden. Tables 1 and 2 summarize the test results from these two classiﬁers. To further evaluate the performance, false alarm rate was also tested and included. Table 1. Test results from SVM classiﬁer-1, using all Swedish wood images. Pc : classiﬁcation rate; PF A : false alarm rate. knot type pin dead sound black/rotten total number 100 100 100 100 classiﬁcation 97 96 95 98 false alarm 0 4 2 3

average (%) Pc =96.5 PFA =2.25

Table 2. Test results from SVM classiﬁer-2, using mixture of Swedish (S) and Finnish (F) wood images. Pc : classiﬁcation rate; PF A : false alarm rate. knot type

pin dead sound black (S) average (%) (S) (S) (F) /rotten(S) total number 100 100 100 100 classiﬁcation 97 96 96 98 Pc =96.75 false alarm 0 5 2 2 PFA =2.25

Observation the performance in Tables 1 and 2, our tests showed that using the simple feature vector and the tree-structure SVM classiﬁers have yielded good performance.

7

Conclusion and Future Work

The proposed tree-structure SVM classiﬁers has been tested for classifying four diﬀerent types of wood knots by using simple and eﬃcient features. Evaluation of our test results over 400 images yielded an average classiﬁcation rate of 96.5% and a false alarm rate of 2.25%. Future work includes reﬁning the methods, more extensive tests using larger wood data sets, and also on extending the classiﬁer to include more wood knot types and other wood defects.

Acknowledgement The Swedish wood images were kindly provided by Ros´ens Maskin AB, Sweden.

Automatic Classiﬁcation of Wood Defects Using Support Vector Machines

367

References 1. Gr¨ onlund, U.: Quality improvements in forest products industry. Dissertation, Lule˚ a University of Technology, Sweden (1995) 2. Huber, H.A., McMillin, C.W., McKinney, J.P.: Lumber defect detection abilities of furniture rough mill employees. Forest Products Journal 35, 79–82 (1985) 3. Estevez, P.A., Perez, C.A., Goles, E.: Genetic input selection to a neural classiﬁer for defect classiﬁcation of radiata pine boards. Forest products journal 53(7), 87–94 (2003) 4. Silv´en, O., Niskanen, M., Kauppinen, H.: Wood inspection with non-supervised clustering. Machine Vision and Applications 13(5-6), 275–285 (2003) 5. Silv´en, O., Kauppinen, H.: Recent Development in Wood Inspection. Internation Journal of Pattern Recognition and Artiﬁcial Intelligence 19(1), 83–95 (1996) 6. Kauppinen, H., Silv´en, O.: The Eﬀect of Illumination Variations on Color-Based Wood Defect Classiﬁcation. In: Proc. IEEE Int’l conf. ICPR, pp. 828–832 (1996) 7. Chacon, I.M., Alonso, G.R.: Wood defects classiﬁcation using a SOM/FFP ap˙ proach with minimum dimension feature vector. In: Wang, J., Yi, Z., Zurada, J.M., Lu, B.-L., Yin, H. (eds.) ISNN 2006. LNCS, vol. 3973, pp. 1105–1110. Springer, Heidelberg (2006) 8. Shawe-Taylor, J., Cristianini, N.: An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge University Press, Cambridge (2000) 9. Sch¨ olkopf, B., Smola, A.J.: Learning With Kernels - Support Vector Machines, Regularization, Optimization and Beyond (2001) 10. Andersson, H.: Automatic classiﬁcation of wood defects using support vector machines, MSc thesis, Chalmers Univ. of Technology, Sweden (2008) 11. F¨ ooreningen Svenska S˚ agverksm¨ an FSS, Nordiskt tr¨ a (Nordic trees), Markaryds Graﬁska (1999) ISBN: 91-7322-251-8 12. Hsu, C.-W., Chang, C.-C., Lin, C.-J.: A practical guide to support vector classifcation (2007), http://www.csie.ntu.edu.tw/~ cjlin/papers/guide/guide.pdf 13. Finnish sound wood knots, http://www.ee.oulu.fi/research/imag/knots/KNOTS/ 14. Matlab SVM toolbox from OSU, http://svm.sourceforge.net

Automatic Surveillance and Analysis of Snow and Ice Coverage on Electrical Insulators of Power Transmission Lines Irene Y.H. Gu1 , Unai Sistiaga1 , Sonja M. Berlijn2 , and Anders Fahlstr¨ om3 1

Dept. of Signals and Systems, Chalmers Univ. of Technology, Sweden [email protected] 2 Statnett, Box 5192, Majorstuen, 0302, Oslo, Norway [email protected] 3 STRI AB, Box 707, 77180, Ludvika, Sweden [email protected]

Abstract. One of the large problems for electrical power delivery through power lines in the Northern countries is when snow or ice accumulates on electrical insulators. This could lead to snow or ice-induced outages and voltage collapse, causing huge economic loss. This paper proposes a novel real-time automatic surveillance and image analysis system for detecting and estimating the snow and ice coverage on electric insulators using images captured from an outdoor 420 kV power transmission line. In addition, the swing angle of insulators is estimated, as large swing angles due to wind cause short circuits. Hybrid techniques by combining histogram, edges, boundaries and cross-correlations are employed for handling a broad range of scenarios caused by changing weather and lighting conditions. Experiments have been conducted on the captured images over several month periods. Results have shown that the proposed system has provided valuable estimation results. For image pixels related to snows on the insulator, the current system has yielded an average detection rate of 93% for good quality images, and 67.6% for images containing large amount of poor quality ones, and the corresponding average false alarm ranges from 9% to 18.1%. Further improvement may be achieved by using video-based analysis and improved camera settings. Keywords: electric insulator, insulator snow detection, insulator ice detection, insulator swing angle, insulator image analysis, histogram, boundary detection, cross-correlation.

1

Introduction

Northern countries, e.g. Scandinavian, north Canada, Russia and China often encounter snow and ice during cold winter or in high areas. One of the problems for electrical power delivery through power lines is when snow or ice accumulates on electrical insulators. When the accumulated snow melts and freezes or in case of freezing rain, long ice bars hanging down along the edge of insulators could be L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 368–379, 2009. c Springer-Verlag Berlin Heidelberg 2009

Automatic Surveillance and Analysis of Snow and Ice Coverage

369

formed. Also, the coverage of snow on insulators could be thick. When the ice or snow melts, a conducting layer is formed on the insulator or on the outside of the ice, and short circuit or ﬂashover may occur. This may lead to ice-induced outages and voltage collapse, causing huge economic loss for the power company and the related users. For example, Norwegian power companies have observed ice-induced outages especially during 1987 and 1993. In Sweden, it has recently caused a number of large blackouts. In Canada it has led to large problems and sever blackout in the end of 1990’s. Because snow and ice related outages happen during severe weather conditions, very little information and knowledge are available about the process of ice and snow accretion ﬁnally leading to ﬂash over. Further, when upgrading power lines, ice performance is one of the important aspects of the insulation selection process. A better understanding of insulator’s sensitivities to snow and ice would be useful to help improving future design. In Norwegian environment, classical freezing rain rarely occurs. Hence, it is unlikely that rain would lead to outages in their networks. However, it is assumed that accretions from wet snow (possibly in combination with heavy rime icing) that occur regularly could result in ice accretions with electrical properties similar to those of freezing rain. Eﬀorts have made for arranging surveillance cameras along these power lines as they are often located in very remote locations. The captured videos from the cameras are then transferred through the Internet to the corresponding utilities or power companies where the situations are monitored by humans. It is desirable that one automatically monitor and detect possible snow/ice accretions on electrical insulators. Further, once the snow/ice on the insulator are detected, automatic analysis shall be followed to estimate the percentage of snow/ice coverage related to the distance between two neighboring shells of an insulator. The analysis results can be fed to network operators if the snow/ice coverage reaches a risk level, and necessary intervention can then be taken before short circuits occur. Motivated by the above issues, this paper proposes a full automatic image analysis system for the surveillance, detection and analysis of snow/ice coverage in electrical insulators of power lines through using images captured by a visual camera. To be able to capture images in dark and night scenarios, the insulator is lighted up by a lamp. The system also automatically estimate swing angles, since large swing angles of insulators may lead to short circuits. To be able to determine, monitor and verify the snow or ice coverage on diﬀerent types of insulators, outdoor laboratory tests and a on-site, on-line web-based surveillance system is developed [1,2] To the best of our knowledge, this is the first successful insulator snow and ice surveillance system based on automatic image analysis techniques. As a by-product, these tests may also provide the experts with a better understanding of ice bridging phenomena hence possible improvement in future insulator design [3]. In the remaining part of the paper, we shall describe the insulator and surveillance system settings and the block diagram of image analysis system (Section 2), complex image scenarios impacting the surveillance (Section 3), image registration for ROI extraction (Section 4), snow analysis (Section 5), swing angle estimation (Section 6), experimental results and evaluation (Section 7), and ﬁnally conclusions (Section 8).

370

2

I.Y.H. Gu et al.

Settings for the Web-Based Surveillance System

The measurements are performed for insulators on a 420 kV power transmission line that is set for outdoor tests. Basic components that are installed in the proposed surveillance system includes: (a) visual cameras and lamps; (b) a weather station; (c) a communication system between the remote test site and the network operator; (d) a web-based database; (e) a real-time automatic image analysis system. Measurements are done in the remote site (where 230 V supply is not available) in a ﬁx time interval and during severe conditions such as low winter temperatures and darkness, where the power is suﬃcient for heating the lamps and the cameras. Images are captured in each 10 minute interval. Weather and wind conditions are recorded. The communication system is established between the remote site and the power network operator, where measurement data are transferred from the test site to the database. The data are then stored in a database, being accessible via a web interface immediately after the measurement. Meanwhile, real-time image analysis is performed for each newly captured image and the results are displayed and stored in the system. Other data, such as voltage, power, current, reports and results of analysis, can be added to the database. Further detail of the system can be found in [1,2,4].

Fig. 1. Insulators used for our tests. From the left to right: type (a), (b) and (c).

As shown in Fig.1, three diﬀerent types of insulators are monitored in our outdoor tests. type (a): composite insulator (Φ=168 and 129 mm): manufactured by Lapp insulators gmbh & co, consisting of 11 large and 33 small shells. The length (include corona rings placed above and below) of the insulator is 2.12 meters. type (b): desert type insulator (Φ=420 and 280 mm): manufactured by Sediver, consisting of 6 large diameter shells and 7 small diameter shells. The length of the insulator is 2 meters. type (c): coastal type insulator (Φ= 330 mm): manufactured by Sediver, consisting of 13 shells. With a corona ring placed below, the insulator is 2 meters long. Automatic image analysis, which is the main focus of this paper, is applied to detect snow/ice events and wind events. As shown in Fig.2, this image analysis system is designed to detect the snow (or ice) on an insulator. Once snow is detected, the percentage of snow coverage on the insulator is then estimated. The swing angle of insulator is also estimated in the system as large swing

Automatic Surveillance and Analysis of Snow and Ice Coverage

371

Reference angle

Enhancement

Estimate swing angle

Image Extract ROI

Analyze insulator context

Estimate outer boundaries A priori knowledge

Detect / estimate snow / ice coverage

Fig. 2. Block diagram of the proposed surveillance system (automatic part only)

can cause short circuit. In addition to this, a semi-automatic analysis part is also provided to the network operator, which generates the desired snow areas (referred to as the ’ground truth’) for each given ROI image. The ground truth is used when evaluating the performance (such as detection rate and false alarm rate) of automatic analysis results.

3

Images from Complex Outdoor Scenarios

Despite insulators are rigid objects and cameras are stationary, automatic detection and analysis of snow or ice coverage is non-trivial. Since images were captured from natural scenes, not only light conditions and background can change abruptly (e.g. sunshine, cloudy, foggy, drizzle, raining and snow; moving clouds, unexpected moving objects such as airplanes or birds within camera views), but also a camera is often slightly moved due to strong wind (causing insulator positions in images drifting with time). Among them, events that signiﬁcantly impact the image analysis are: – strong wind: this may cause camera movement hence the insulator position in the image can drift; – dark weather: this may lead to low visibility or low contrast in images;

Fig. 3. Scenarios of insulators captured by a outdoor camera (only the areas containing the insulator is shown). From the left to right: (a) ideal image with sunny clear sky; (b) insulator with reflections from sunlight; (c),(d),(e) cloudy where clouds forms a nonuniform background and can change fast; (f),(g) blurred images from foggy conditions; (h),(i),(j) dark and night images; (k) snow on shells; (l): ice on shells.

372

I.Y.H. Gu et al.

– cloudy weather: this may lead to non-uniform fast changing background; – foggy weather: this may lead to low visibility and severely blurred images; – dark night: images may vary signiﬁcantly, depending on the snow, reﬂection of lighting and camera incident angle. – strong sun: depending on the incident angle of camera, images may contain bright regions due to the reﬂection from the insulator. Fig.3 shows some examples scenarios. To be able to handle a broad range of varieties, robust image processing methods shall be employed.

4

Extract the ROI through Image Registration

For real-time processing, an essential step is that image analysis is only applied to a small region that contains the insulator (or, the region of interest - ROI). It is observed that insulator’s positions in image frames drift with time. This is caused mostly by the minor camera movement but also from the swing of insulator. Extracting ROI therefore requires image registration. Since the size of the insulator under a given camera setting remains a constant, it can be used as a priori information. Since daytime and nighttime images are signiﬁcantly diﬀerent, diﬀerent methods are employed to extract the ROI from these two types of images. 4.1

For Nighttime Images

From the illumination setting in the current system, only the central axis of insulator in an image is most visible (see Fig.4(a)). Since the histogram of night image contains a narrow sharp peak, a binary image B is generated by thresholding the histogram. To determine the ROI, vertical accumulations and horizontal accumulations are performed, yielding: av (i) =

N

B(i, j),

ah (j) =

M

j=1

B(i, j)

i=1

The vertical accumulated curve av (i) usually has one narrow peak (see Fig.4(c)) and is related to the central axis of the insulator. The peak position is thus Hor acc

Vert acc

250

600

500

200

400

150 300

100 200

50 100

0

0

100

200

300

400

500

600

700

800

0

0

100

200

300

400

500

600

700

Fig. 4. Extract a ROI from a large image. Left to right: (a) night image; (b) binary image after histogram thresholding; (c) vertical accumulations av ; (d) horizontal accumulations ah ; (e) extracted ROI.

Automatic Surveillance and Analysis of Snow and Ice Coverage

373

assigned to the x-coordinate center of the ROI. The horizontal accumulated curve ah (j) usually has two large peaks (see Fig.4(d)), corresponding to the top and bottom frame where the insulator is ﬁxed. The valley region between the two peaks in ah (j) is related to the insulator, the center position in this valley is thus set as the y-coordinate center of the ROI. The width and the height for the ROI are then assigned by the pre-determined values (they are ﬁxed constants for each type of insulator in respect to its camera setting). This results in the ROI (see Fig.4(e)). 4.2

For Daytime Images

Extracting the ROI from a daytime image is more diﬃcult. Since a broad range of possible complex scenarios, most image processing techniques (e.g. segmentation [5], corner detection, histogram thresholding) are only eﬀective for a subset of images. The basic idea is to exploit the prior information and pre-store a template that contains the broadened outer boundaries of the insulator in concern, and apply cross-correlation to the given image. First, a simple edge detector is applied to the given image, resulting a binary edge image E. Then, normalized crosscorrelations are computed between E and the template ET containing broadened outer boundaries (width w = 7 for our tests) obtained from the ROI of an ideal insulator image, x,y E(x − u, y − v)ET (x, y) ρ(u, v) = (1) , u, v ∈ RE 2 2 E(x, y) E (x, y) x,y x,y T where u, v are the lags for cross-correlation, the range of u, v is within the size of image E. The reason to use broadened boundaries in the template is to avoid sensitivity in the correlation when the edges and boundaries from two images are slightly shifted. The best matching position is found (u∗ , v ∗ ) = argmaxu,v ρ(u, v). The ROI extracted in such a way can be further reﬁned by applying horizontal and vertical accumulations in the similar way that is described for extracting ROI from nighttime images. Tests of these methods over 3 months of images have resulted in about 88.5% of success in the extracted ROIs.

5

A Hybrid Method for Snow Detection and Analysis

Once a ROI image is given, detection of snow and subsequently analysis of snow (or, ice) coverage will be performed. Noting that snow (or, ice) scenarios can vary signiﬁcantly (see Fig.5(a)-(e)), the following prior information is exploited: – An insulator is a rigid object, its size and shell edges and outer boundaries are ﬁxed and known. These may be changed if an insulator is covered by something, e.g. snow or ice. – There exist some intensity diﬀerence between the snow/ ice and the insulator/background.

374

I.Y.H. Gu et al.

– The intensity diﬀerences between snow, shells (or, background) cause extra edge curves on the top half of insulator shells. Further, it is reasonable to assume that snow is only accumulated on the top or along the side of insulator’s shells based on observations.

Fig. 5. Variety of scenarios of: snow on insulator and swing of insulator. From the left to right: (a)(b) snow; (c) melting snow; (d)(e) Rim frost; (f): night image of insulator with almost no wind. (g) night image of insulator with a relatively large swing (measured wind speed 10.4m/sec).

5.1

Detect and Analyze Snow Regions

To determine the snow (or ice) regions for images with a broad range of complex scenarios, a joint edge and context analysis method is applied. Detect extra regions Since snow may generate extra image edges and regions, an edge detector is ﬁrst applied to the median ﬁltered ROI image, followed by a simple edge closing algorithm. The reason of applying a median ﬁlter is to obtain a ROI image with a smoother background, hence less edge noise. Each enclosed area surrounded by edge curves forms a region. Find regions above the shells Since snow on insulator shells, and other changing conditions (e.g.local clouds, illuminations and reﬂections) could generate new extra regions, extra regions need to be analyzed. By using the prior information of standard shell positions and the ’ellipse’ shaped regions of shells as the reference, these extra regions (including split regions) can be found. Further, since snow or ice is more likely to accumulate on the top and/or side part of insulator shells, only extra regions related to these locations are considered, while extra region next to the bottom half shells are removed. Tighten the width of ROI To further limit the areas to be analyzed, a ROI is narrowed down by setting a tight width determined by the most left and most right outer points of shells. This is done by setting two parallel lines on the outer sides of shells: a left line is

Automatic Surveillance and Analysis of Snow and Ice Coverage

375

set to only touches the extremal left points of shells, and a right line only touches the extremal right points of shells. These extremal points can be detected either from the outer boundaries, or the silhouette of insulator shells (see Section 6). To make the boundary or silhouette estimate more robust, cross-correlations with a pre-stored template (containing broadened outer boundaries or silhouette) from the ideal insulator can be applied. Compare the intensities Region analysis is then performed by comparing the range of intensity values in each extra region with those of the shell and of the background. Decision on snow area is then made by combining the comparison results and the prior knowledge of snow intensity. 5.2

Compute the Snow Coverage

Once the snow (or ice) regions are determined, a narrow-width vertical bar (see red bar in Fig.7) parallel to the vertical center axis of insulator, is then is placed on the insulator, and is swept from the left to the right side of the insulator. For each area under the sweeping bar, the heights of detected snow regions are accumulated. This accumulated height value is then compared with the total length of insulator shells under analysis, resulting the percentage of snow coverage. One can also examine the change in the gap between two neighboring shells, and the percentage of average (or maximum) change in the gap can be used.

6

Estimate Swing Angles

It is worth noting that due to the camera view angle, the angle of insulator in an image does not have to be 0◦ with respect to the vertical axis of the Cartesian coordinate system (see example scenarios in Fig.5(f)-(g)). The swing angle for an insulator should be estimated as the relative angle, which is the diﬀerence between the absolute insulate angle in the given image and the reference insulator angle in an image captured when no wind is present. We shall focus on estimating the absolute angle, since the remaining step for computing the relative angle is straightforward. Two methods are described below, one generating silhouette of insulator shells, another generating outer boundaries of insulator shells. (a) Histogram-based method for generating insulator silhouette This method will generate a silhouette image for insulator shells. Ideally, the histogram of a ROI image is bimodal, with one peak related to the insulator and another to the background. The insulator can thus be extracted by applying two thresholds in the left and right side of the histogram peak related to the insulator. The distance between the two thresholds is determined by the insulator image size (a priori information), so that the thresholded peak area is equal to the insulator image size. A binary image is the generated by assigning pixels in the thresholded peak area as value one and the remaining as zero. The

376

I.Y.H. Gu et al.

Fig. 6. Thresholding the histogram and the corresponding image region. 3 sub-figures in the left side: insulator image with a clear background, histogram, thresholded image; 3 sub-figures in the right side: insulator image with a cloudy background, histogram, thresholded image.

method is simple however, may fail in some non-ideal scenarios. Fig.6 shows two examples. (b) Cross-correlation for estimating shell outer boundaries This method will generate outer boundaries of insulator shells. The basic idea is very similar to the cross-correlation used in Section 4, where the edges from the ROI is correlated with a template containing the broadened insulator outer boundaries. However, instead of translating the template to ﬁnd maximum crosscorrelation location in Section 4, the template is now rotated in order to ﬁnd the maximum correlation with the insulator in the ROI. This corresponding crosscorrelation is obtained by (1), by introducing a rotation angle and using only zero-lag (i.e., (u, v) = (0, 0)), described below:

E(i, j)ET (i, j, θk ) , θk = kΔθk , θk ∈ [0, θ1 ] 2 (i, j, 0) 2 (i, j) E E i,j i,j T

ρ(0, 0, θk ) =

i,j

where θk is the rotation angle of the template, (Δθk = 0.25◦ , θk ∈ [0◦ , 1.5◦ ] used in our tests). The best matched angle is found when the cross-correlated is maximized, θ∗ = argmaxθk ρ(0, 0, θk ). The original thin outer boundaries from the template at matched positions are then assigned as the outer boundaries for the given ROI image. The method is more robust and works for a relatively broad range of images. Estimate the central axis and compute the angle Once the silhouette, or the outer boundaries of shells are found, two parallel vertical lines can be determined by either shifting lines to touching the outer most (extremal) points on the shell silhouette, or detecting and linking the extremal boundary points. The central axis is then obtained by adding a parallel line in the middle of these two outer lines. The absolute angle between the central axis and y-axis in the Cartesian coordinate system is then computed. For dark nighttime images, the central axis is directly estimated through vertical accumulations av of the ROI image.

Automatic Surveillance and Analysis of Snow and Ice Coverage

7 7.1

377

Experimental Results and Evaluation Test Results

As shown in Fig.7, the proposed system contains a graphical user interface (GUI), where the bottoms in the 1st column of GUI is related to the steps in automatic snow and swing angle analysis, and the results (image and estimated parameters) are then display.

Fig. 7. GUI for the insulator surveillance system

Fig.8 (left) shows 5 examples of several good and not so good results from the snow analysis. It is observed that detection results are signiﬁcantly aﬀected by the variety of background scenarios, also, they are aﬀected by the setting of camera view angle. For example, the camera setting for insulator type (b) generates better view angle for insulator as compared with that for the insulator type (c) . Further, the image resolution for insulator type (a) is insuﬃcient for the analysis of snow regions. 7.2

Estimate Ground Truth by Semi-automatic Processing

To obtain the ’ground truth’ of insulator snow regions, a semi-automatic assisted analysis is provided to the network operator through using the 2nd column functions in the GUI (see the right part of Fig.8). In the semi-automatic analysis, individual shell from the ROI is extracted and analyzed in each time.

378

I.Y.H. Gu et al.

Fig. 8. Left part: Automatic analysis results. This part contains 5 results where the detected snow regions are visually enhanced: the first 3 insulators: good results from image with one clear and 2 dark backgrounds; the next 2 insulators: not so good results where only partial snow areas are detected from images with cloudy and dark background. Right part: Semi-automatic analysis to find snow ground truth for shell 2 and 3. From the left to right, top to bottom in each column: Selecting a shell (inside green box) from ROI, the selected shell, closed edge curves after modification, resulted snow region ground truth with visual enhancement.

The user is allowed to manually select 2 thresholds based on the histogram of the selected region. The detected edges of the shell can be manually closed or modiﬁed, until the generated snow regions are satisﬁed. This process is repeated for all shells, as shown in Fig.8. The resulted snow regions from diﬀerent shells are then combined to form an insulator snow ground truth image, which will be used as the reference for performance evaluation of the automatic analysis results. 7.3

Performance Evaluation

The performance of automatic analysis results are evaluated by using the corresponding insulator snow ’ground truth’ as the reference. The evaluations have only been performed on insulator type (b) due to the following reasons: Images of insulator type-(a) are too low due to the small shell size and the distance between the camera and the insulator. Consequently, snow regions are too small for analyzing with suﬃcient accuracy. For insulator type (c), the current setting of camera view angle has led to signiﬁcantly reduced in-between-shell regions and large overlap areas in neighboring shells. Our preliminary evaluations for insulator type (b) images show that: For good quality images, the average detection rate is about 93% with false alarm rate about 9%, (deﬁned by the correctly and falsely detected pixels related to the snow, respectively). However, it is noticed that the average performance is signiﬁcantly dragged down by poor quality images with low visibility, very weak edges and dark snow. The average detection rate is dropped to 61.03% for bright background images and 74.22% for dark background images. Meanwhile, the average false alarm also increased to 21.5% for bright background, and 18.13% for dark background. The highest snow coverage during that period was 14.73%.

Automatic Surveillance and Analysis of Snow and Ice Coverage

8

379

Conclusion and Future Work

The proposed system, designed for automatically detecting and estimating insulator snow (or, ice) coverage and insulator swing angles, has been tested for real-time processing of outdoor measured images captured from insulators on 420 kV transmission lines in a remote test site. Our results showed that the proposed hybrid method is relatively robust for a broad range of complex images, with an average detection rate ranging from 92% for good quality images to about 67.6% with large percentage of poor quality images, and the corresponding range of false alarm from 9% to 18.1%. Further improvement will be made by exploiting temporal information in videos and by improving the settings of image capture system. The system has also increased the interest for a long term research, as our test results have demonstrated that automatically monitoring ice and snow phenomena, previously considered as not feasible, is now possible.

References 1. WAP project website, http://wap.stri.se 2. Berlijn, S.M., Gutman, I., Halsan, K.A., Gu, I.Y.H.: Laboratory Tests and a Web Based Surveillance to determine the Ice- and Snow Performance of Insulators. IEEE Trans. on Dielectrics and Electrical Insulation, Special Issue on Flashover of Ice- or Snow-Covered Insulators 14(6), 1373–1380 (2007) 3. Gutman, I., Halsan, K., H¨ ubinette, D., Ohnstad, T.: Ice progressive stress method: repeatability during full-scale testing of 400 kV line and apparatus insulators and application of the test results. In: Proc. of IEEE Int’l Symp. Electrical Insulation, Indianapolis, USA, September 19-22, 2004, pp. 560–563 (2004) 4. Sistiaga, U.: Automatic image analysis methods for estimating snow coverage and swing angle of insulators in power transmission lines, M.Sc. thesis, Dept. of Signals and Systems, Chalmers Univ. of Technology, Sweden (2007) 5. Gu, I.Y.H., Gui, V.: Joint space-time-range mean shift-based image and video segmentation. In: Zhang, Y.-J. (ed.) Advances in Image and Video Segmentation, pp. 113–139. Idea Group Inc. Pub. (2006)

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature Adam Herout, Radovan Josth, Pavel Zemcik, and Michal Hradis Graph@FIT, Brno University of Technology, Bozetechova 2, Brno, CZ {fherout,ijosth,zemcik,ihradisg}@fit.vutbr.cz Abstract. A currently popular trend in object detection and pattern recognition is usage of statistical classiﬁers, namely AdaBoost and its modiﬁcations. The speed performance of these classiﬁers largely depends on the low level image features they are using: both on the amount of information the feature provides and the processor time of its evaluation. Local Rank Diﬀerences is an image feature that is alternative to commonly used haar wavelets. It is suitable for implementation in programmable (FPGA) or specialized (ASIC) hardware, but -as this paper shows -it performs very well on graphics hardware (GPU) used in general purpose manner (GPGPU, namely CUDA in this case) as well. The paper discusses the LRD features and their properties, describes an experimental implementation of the LRD in graphics hardware using CUDA, presents its empirical performance measures compared to alter native approaches, suggests several notes on practical usage of LRD and proposes directions for future work.

1

Introduction

Current algorithms of statistical classiﬁcation for object detection or pattern recognition exhibit real-time performance in detecting complex patterns, such as human faces [1], while achieving precision of detection which is suﬃcient for practical applications. Recent work of Sochman and Matas [2] even suggests that any existing detector can be eﬃciently emulated by a sequential classiﬁer which is optimal in terms of computational complexity for desired detection precision. In their approach, human eﬀort is invested into designing a set of suitable features which are then automatically combined by the WaldBoost [3] algorithm into an ensemble. In practical applications, the speed of the object detector or other image classiﬁer is crucial. Recent advances in development of graphics processors attract many researchers and engineers to the idea of using GPU’s not for their primary purpose – rendering 3D graphics scenes. Diﬀerent approaches to so-called GPGPU (General-Purpose computation on GPUs) [4] exist and also the ﬁeld of image processing and computer vision has seen several successful uses of these techniques (e.g. [5], [6]). Statistical classiﬁers are built by using low level weak classiﬁers or image features and the properties of the classiﬁer largely depend on the quality and performance of the low level features. In face detectors and similar classiﬁers, Haar-like L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 380–390, 2009. c Springer-Verlag Berlin Heidelberg 2009

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature

381

wavelets [1], [2], [3], [7] are frequently used, since they provide good amount of discriminative information and they provide excellent performance. Other features are used in diﬀerent contexts, such as the Local Binary Pat terns [8]. Recently, designed especially for being implemented directly in programmable or hard-wired hardware, Local Rank Diﬀerences [9] have been presented. These features are described in more detail in Section 3 of this paper. The main strengths of this image feature are inherent gray-scale transformation invariance, the ability to capture local patterns and the ability to reject quantitative changes in lightness of image areas. Prior to this GP-GPU [4] in CUDA [10] implementation and related research, the authors of this paper implemented the LRD features in the GPU as shaders [11]. The Cg implementation is fairly eﬃcient, the main disadvantage was the need of complicated control of the rendering pipeline from the CPU (by issuing commands to render quads, lines or other primitives in a complex pattern that covered the searched area of the image). This disadvantage is minimized by the properties of the GP-GPU philosophy. The CUDA implementation presented here, compared to the Cg one [11] beneﬁts also from some memory arrangement improvements, from improved training process and other minor advances. The following Section 2 of this paper brieﬂy presents the Local Rank Diﬀerences (see [9] for more detail) image feature. In Section 3, the notes on implementation of LRD using CUDA are given. Section 4 presents the experimental results of the implementation carried out and its comparison to other approaches. Conclusions and suggestions for future research in the area are given in Section 5.

2

Local Rank Diﬀerences

Let us consider a scalar image I(x, y) → R. On such image, a sampling function can be deﬁned (x, y, m, n, u, v, i, j ∈ Z) mn Sxy (u, v) =

m−1 n−1 1 I(x + m(u − 1)) + i, y + n(v − 1) + j). mn i=0 j=0

(1)

This sampling function is parameterized by the sampling block dimensions m, n, and by the origin of the sampling (x, y), which is a pixel in the image. Note that this function ,,subsamples” the image by a multiple of pixels in each direction. Note please also that this function can be deﬁned in other manners, namely not by summing rectangular blocks of the image but by convolving them with a suitable wavelet ﬁlter kernel, etc. Based on this sampling function a rectangular mask can be deﬁned: ⎡ mn ⎤ mn mn Sxy (1, 1) Sxy (2, 1) . . . Sxy (w, 1) ⎢ mn ⎥ mn mn ⎢ Sxy (1, 2) Sxy (2, 2) . . . Sxy (w, 2) ⎥ mnwh ⎥. Mxy =⎢ (2) .. .. .. ⎢ ⎥ .. ⎣ ⎦ . . . . mn mn mn Sxy (1, h) Sxy (2, h) . . . Sxy (w, h)

382

A. Herout et al.

The mask is parameterized by sampling block dimensions m, n and sampling origin (x, y), just as the used sampling function S. Along with these parameters, the mask has its dimensions w, h. Experiments (see [11]) show that in the context of AdaBoost and WaldBoost object detection, the masks of dimensions 3 × 3(w = 3, h = 3) are suﬃcient. For diﬀerent classiﬁers and applications, different sampling block sizes are necessary. For face detectors operating on image windows with resolution of 24 × 24 pixels, sampling sizes of 1 × 1(m = 1, n = 1 etc.), 2 × 2, 2 × 4, 4 × 2 are suﬃcient, and also the set 1 × 1, 1 × 2, 2 × 1 and 2 × 2 provides good performance, enabling eﬃcient implementations. For each position in the mask, its rank can be deﬁned: w h mn mn 1, if Sxy (i, j) < Sxy (u, v) mnwh Rxy (u, v) = , (3) 0, otherwise i=1 j=1

id est, the rank is the order of the given member of the mask in the sorted progression of all the mask members. Note that this value is independent on the local energy in the image, which is an important property useful for the behaviour of the Local Rank Diﬀerences image feature, which is deﬁned as: mnwh mnwh mnwh LRDxy (u, v, k, l) = Rxy (u, v) − Rxy (k, l).

(4)

The notation can be slightly facilitated by vectorizing the matrix M by stacking its rows (it is just a convention that row rather than column stacking is used):

mn mnwh mn mn Vxy (u, v) = Sxy (1, 1)Sxy (1, 1) . . . Sxy (1, 1) . (5) mnwh The rank of a member of the vector then is (note that for clarity, Vxy (i) th denotes the i member of the vector): w×h 1, if V mnwh (i) < V mnwh (a) mnwh xy xy Rxy (a) = (6) 0, otherwise i=1

The Local Rank Diﬀerence of two positions a, b within the vector obviously is:

mnwh mnwh mnwh LRDxy (a, b) = Rxy (a) − Rxy (b).

(7)

Empirical experiments carried out so far show that one w × h dimension used in a classiﬁer is suﬃcient (currently we are using 3 × 3 mask dimension only), i.e. for the purpose of constructing a classiﬁer, there is no need to mix several combinations of mask dimensions, which simpliﬁes the training and evaluation process. Weak LRD classiﬁers available to the statistical classiﬁer therefore oﬀer varying position x, y within the window of interest and varying size m, n of the sampling block used. 2.1

Input Image Pre-processing

mn For increasing the performance of the LRD evaluation, the function Sxy deﬁned on the input image can be pre-calculated. As stated above, low number of combinations of m × n is suﬃcient for learning an object classiﬁer -experiments show

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature

383

that 1 × 1, 2 × 2, 2 × 4 and 4 × 2 combinations are enough. The input image I can be convolved with ⎡ 1 ⎤ 1 . . . wh

wh 1 1 1 1 1 1 ⎢ . . . ⎥ ⎥ h2×2 = 41 41 , h4×2 = 81 18 18 81 , hw×h = ⎢ (8) ⎣ .. . . .. ⎦ 4 4 8 8 8 8 1 1 wh . . . wh and the resulting images at given location (x, y) can contain the values of the sampling function. Such pre-processing of the input images can be done eﬃciently and the LRD evaluation then only consists of 9 look-ups to the memory (for the case of 3 × 3 LRD mask) into appropriate pre-processed image and then evaluation of ranks for two members of the mask. The evaluation then can be done in parallel on platforms supporting vector operations; both GPU and FPGA are strong in such kind of parallelism. 2.2

Local Rank Diﬀerences Compared to Haar Wavelets

Comparing LRD with Haar wavelets is only natural as both of these types of features were ﬁrst intended to be used in detection classiﬁers. There are two fundamental aspects in respect to the detection classiﬁer which must be addressed: the computational complexity of evaluating the features and the amount of dis criminative information the features provide. Haar wavelets can be computed very rapidly on general purpose CPUs by using the integral image representation [1] which can be created in a single pass through the original image. The simple Haar wavelets of any size can be computed using only six accesses into the integral image, six additions and two bit-shifts. When scanning the image in multiple scales, this gives the possibility to scale the classiﬁer instead of down-sampling the image. The Haar wavelets are usually normalized by the size of the feature and the standard deviation of pixel values in the classiﬁed sub-window. Computation of the standard deviation requires additional integral image of squared pixel values and uses square root. While the Haar wavelets can be computed relatively eﬃciently on general purpose CPUs, it may not be the same on other platforms. On FPGAs, the six random accesses into memory would signiﬁcantly limit the performance (only single feature evaluated per every six clock cycles) and the high bit-precision needed for representing the integral images would make the design highly demanding. On the other hand, the nine values needed to compute LRD with grid size 3 × 3 can be obtained on FPGAs with only single memory accesses [9] (when preprocessed as mentioned in Section 2.2) and on GPUs with three or six accesses [11]. Some detection classiﬁers evaluate on average very low number of features (even less than 2). In such cases, computing the normalizing standard deviation poses signiﬁcant computational overhead. Further, the square root which is needed can not be easily computed on FPGAs. The LRD inherently provide normalized results, whose normalization is in fact equivalent to local histogram equalization.

384

A. Herout et al.

The detection performance of classiﬁers with the LRD has been evaluated on the frontal face detection task and it has been compared to the performance of classiﬁers with the standard Haar features. The results suggest that the two types of features provide similar classiﬁcation precision. One of the two classiﬁers compared in Figure 1 uses the same Haar wavelets as in [1] and the other uses the LRD with block sizes of the sampling function (see equation 2) restricted to 1 × 1, 1 × 2, 2 × 1 and 2 × 2. The classiﬁers were trained using 5000 hand annotated faces normalized to 24 × 24 pixels and the nonface samples were randomly sampled from a pool of 250 million sub-windows from more than 3000 non-face images. The results were measured on a set of 89 group photos which contain 1618 faces and total 142 million scanned positions (scale factor 1.2, displacement 2/24). Although the set of LRD features is very limited in this experiment, the detection performance it provides is similar to the full set of Haar wavelets. This is probably due to the localized normalization of the results of the LRD which provides information about local image patterns that goes beyond simple diﬀerence of intensity of image patches.

Fig. 1. ROC of two WaldBoost classiﬁers on a frontal face detection task. Length of the classiﬁers is 500 and they diﬀer only in type of features which they use (Haar features, LRD).

3

LRD Implementation Using CUDA

The implementation of the LRD using CUDA corresponds with the theoretical description of the LRD in a straightforward way. It appears that a wise choice is relying on the combinations 1 × 1, 1 × 2, 2 × 1 and 2 × 2 of the LRD sampling function. Such sampling limits the descriptive power of the features slightly, but allows nice performance improvements. Thanks to the built-in texture sampling with bilinear interpolation on the usable graphics cards, sums of 2 neighboring pixels in vertical or horizontal direction or sum of four neighboring pixels consume the same amount of time as sampling just one source pixel. The scanned image can be used in such way without any pre-processing stage. The following Figure 2 contains the central part of the CUDA code implementing the LRD evaluation.

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature

385

Compared to the previously published Cg implementation of the LRD [11], the CUDA oﬀers some advantages. The biggest problem of the shader version was the need for rather complicated drawing of geometric primitives on the ,,screen” to control the object detection process. The whole of the input image needs to be covered by the primitives, but for eﬃciency reasons, simple drawing of one rectangle of the same size as the input image was not possible. In the GP GPU version, all the coding and control is simpler and more straightforward. As shown later in the performance evaluation section 4, the price to pay for such feasibility of programming is the performance, or rather performance distribution depending on the input size and on the count of the weak classiﬁers. __device__ int GetRankDiff(unsigned int posX, unsigned int posY, unsigned int BlockSizeId, unsigned int BlockABId) { unsigned int mempos = threadIdx.x*9; // address to the temp mem float uiBlockWidth = _uiBlockSizeId >> 3; // mask size float uiBlockHeight = _uiBlockSizeId & 7; // mask size float px = posX + AbsX1 + float(BlockSizeId >> 3)/2.0f; // curr pixel X float py = posY + AbsY1 + float(BlockSizeId & 7)/2.0f; // curr pixel Y // get sums of each matrix block (1x1, 1x2, 2x1, 2x2) s_fBlockSum[mempos+0] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+1] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+2] = tex2D(tImage1, px, py).x; px -= 2.0f*uiBlockWidth; py+=uiBlockHeight; // shift to next line s_fBlockSum[mempos+3] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+4] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+5] = tex2D(tImage1, px, py).x; px -= 2.0f*uiBlockWidth; py+=uiBlockHeight; // shift to next line s_fBlockSum[mempos+6] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+7] = tex2D(tImage1, px, py).x; px+=uiBlockWidth; s_fBlockSum[mempos+8] = tex2D(tImage1, px, py).x; // compute the rank difference between blockA and blockB int iRank = 0; unsigned int uiBlockA = _blockABId >> 4; unsigned int uiBlockB = _blockABId & 15; for (unsigned int bi = 0; bi < 9; bi++) { if (s_fBlockSum[mempos+bi] < s_fBlockSum[mempos+uiBlockA]) iRank--; if (s_fBlockSum[mempos+bi] < s_fBlockSum[mempos+uiBlockB]) iRank++; } return iRank; } __device__ unsigned char LRD() { float ret = 0.0f; for (unsigned cid=0; cid < WCCount-1; cid++) { // loop over weak class. uint4 w0 = tex1D(tWeakParam, cid); // get WeakClassifier parameters // Compute WeakClassifier rank and convert it to predictor value ret += tex2D(PredValues, GetRankDiff(w0.x, w0.y, w0.z, w0.w)+8, cid); } return (unsigned char)ret; }

Fig. 2. The central part of the CUDA implementation code. The LRD() function loops over all the weak classiﬁers in the boosted cascade (stored in a 1D texture), gets the rank diﬀerence (by calling GetRankDi(. . .) and uses the diﬀerence as an index to the table of alpha values obtained by training the classiﬁer.

386

4

A. Herout et al.

Performance Evaluation and Analysis

To evaluate the eﬃciency of the presented GP-GPU implementation of the LRD, these implementations were compared: LRD on GPU using CUDA (sect. 3 above), LRD on GPU using Cg , Haar on GPU using Cg , and LRD on CPU/MMX. Evaluation was performed for diﬀerent resolution of the image, for diﬀerent sizes of the classiﬁed window and for diﬀerent amount of the weak hypotheses calculated for each classiﬁed window. Note that this evaluation is to determine the evaluation speed of the weak classiﬁers only, not the overall performance of the boosted classiﬁer. Table 1. Performance table for LRDonGPU, HAARonGPU and LRDonMMX; the table contains the times of sole evaluation of the classiﬁer, since the pre-processing for the Haar wavelets (integral image calculation), cannot be easily implemented in the GPU frame-time [milli sec] time-per-wc [nano sec] num lrd lrd haar lrd lrd lrd haar lrd resol wc CUDA GPU GPU MMX CUDA GPU GPU MMX 320 × 200 5 13.90 0.244 0.370 17.7 43.44 0.872 1.325 55.29 320 × 200 10 13.63 0.527 0.469 25.0 21.29 0.942 0.839 46.71 320 × 200 50 13.50 2.524 3.010 82.0 2.20 0.902 1.076 40.04 640 × 480 5 56.87 1.173 1.642 101.8 37.03 0.810 1.134 58.55 640 × 480 10 53.82 2.232 2.159 149.0 17.52 0.771 0.745 51.82 640 × 480 50 32.95 11.066 15.731 493.0 2.14 0.746 1.086 44.05 Table 2. Evaluation of the pre-processing stage (convolutions for the LRD, integral im age for Haar wavelets); the pre-processing needs to be performed on every frame. Times are given in milliseconds. Note that pre-processing for the LRD is notably cheeper, even on CPU and performs excellently on GPU. Note also, that the presented CUDA implementation requires no pre-processing stage. resol LRDonCUDA LRDonGPU HAARonCPU LRDonCPU 320 × 200 — 0.72 1.22 2.52 640 × 480 — 1.22 10.29 9.13 800 × 600 — 3.51 16.41 13.80 1024 × 768 — 3.75 27.94 24.80 1280 × 1024 — 4.53 45.16 37.45

In Table 2, a coarse comparison of the performance of the pre-processing stage is given. It is diﬃcult to compare the pre-processing for the Haar wavelets with the LRD convolutions, because the integral image calculation is diﬃcult to implement on the GPU. Note that this is an important advantage of the LRD over the Haar wavelets, especially when in GPU implementation. The actual CUDA implementation works without the pre-processing, because it relies on the 1 × 1, 1 × 2, 2 × 1 and 2 × 2 set of mask dimensions. As indicated by the graph in Figure 1, such limited set of sampling function dimensions is still suﬃcient and well comparable with the commonly used Haar features.

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature

387

Table 3. Behavior of the CUDA implementation for a range of image sizes and number of weak classiﬁers per scanned window. Two parts of the table show the time consumed per frame and this measure divided per the number of weak classiﬁers in the frame. The times are structured into Load time of the program, Execution time and the sum of these both.

resol 128 × 128 256 × 256 512 × 512 1024 × 1024 1600 × 1600 128 × 128 256 × 256 512 × 512 1024 × 1024 1600 × 1600 128 × 128 256 × 256 512 × 512 1024 × 1024 1600 × 1600 128 × 128 256 × 256 512 × 512 1024 × 1024 1600 × 1600

num of WC 5 5 5 5 5 40 40 40 40 40 160 160 160 160 160 640 640 640 640 640

Frame-time Load Exec Total [ms] [ms] [ms] 2.18 4.26 6.44 2.29 11.94 14.23 2.35 46.17 48.53 3.37 169.67 173.04 4.86 403.88 408.74 2.18 4.85 7.03 2.26 11.38 13.64 2.30 42.62 44.93 2.79 165.58 168.38 3.47 399.88 403.35 2.01 4.37 6.38 2.34 11.34 13.69 2.36 42.36 44.73 2.65 165.29 167.95 3.60 411.19 414.80 1.98 12.31 14.30 2.24 25.54 27.79 2.36 98.16 100.52 2.70 385.54 388.25 4.25 1028.21 1032.46

Future time Load Exec Total [ns] [ns] [ns] 26.69 52.00 78.69 7.00 36.43 43.44 1.79 35.23 37.02 0.64 32.36 33.00 0.38 31.55 31.93 3.32 7.40 10.73 0.86 4.34 5.20 0.22 4.06 4.28 0.06 3.94 4.01 0.03 3.90 3.93 0.76 1.66 2.43 0.22 1.08 1.30 0.05 1.01 1.06 0.01 0.98 1.00 0.01 1.00 1.01 0.19 1.17 1.36 0.05 0.60 0.66 0.01 0.58 0.59 0.00 0.57 0.57 0.00 0.62 0.63

Table 1 includes such regimes of evaluation, that were designed to correspond to real-time operation even on slower platforms, as is the C code for the CPU (it is considered slow compared to the parallel architectures as FPGA or GPU). In that table, the CUDA code does not perform excellently, but a tremendous increase of performance is observed when the number of weak classiﬁers is increased (towards 50 in the table). Further exploration showed that the CUDA platform (at its current version – 2.0beta) exhibits relatively slow and constant load-time of the code to be executed. Also the current implementation of the boosted classiﬁer, as indicated in Table 3, consumes constant run time for wide range of increasing number of weak classiﬁers – though the computational load should be linearly proportial to it. This anomaly must be further explored and may be related to some characteristic of the GPU architecture or a ﬂaw in the compiler. However, if the boosted classiﬁer would be a standard AdaBoost [1] or similar, the number of weak classiﬁers would be constantly high (hundreds). In such case the CUDA implementation outperforms tremendously any other solution available to our knowledge.

388

A. Herout et al.

Several notes about the diﬀerent alternative implementations used in the comparison follow. GPU Implementation using Cg Shading Language. An eﬃcient memory layout is used (utilizing 3D textures and other techniques) to allow the shader to access all the nine values of the LRD mask in 3 or 6 texture look-ups. The pixel data are stored as components of the .rgba vector, and vector operations can be used in the calculation. For the pre-processing task, which is constituted by several passes of subsampling by an integer fraction (see section 2.1), built-in hardware means of texture sampling are used on the GPU – see Table 2 for results. Haar Features. Only the simplest (two-fold) Haar wavelet features were used in this testing implementation (though also three-fold features are used in the object detectors, whose evaluation is slightly slower). The Haar wavelets require normalization by the energy in the classiﬁed win dow – both to evaluate the energy and to evaluate the features themselves, integral images are used, which is the fastest method available to our knowledge. The calculation of the integral images constitutes the preparatory phase evaluated in the comparison. Please note that (to our knowledge) there is no eﬀective way of calculating the integral image in the shading language, and the implementation in CUDA is also not straightforward and eﬃcient, so the preparatory phase is implemented in the CPU. The shader code evaluating the classiﬁers can be found in [11]. LRD on the CPU using MMX instruction set. The performance of the GPU implementation was compared to an implementation on standard Intel CPU using MMX instructions. To simplify the feature evaluation as much as possible, the convolutions of the image with the sampling function kernel are pre-computed and stored in the memory in such manner that all the results of the LRD grid can be fethed into the CPU registers through two 64-bit loads. This positively aﬀects the evaluation that is performed in MMX CPU instructions (introduced by Intel). The authors are currently ﬁnishing experiments with the SSE instruction set, which improves the results of the MMX set slightly and whose source code is shorter and clearer. The SSE implementation’s performance seems promising on the CPU but still does not compare to the GPU/shader or GP-GPU performance.

5

Conclusions and Future Work

This paper presents an experimental implementation of the Local Rank Diﬀerences image feature using the CUDA GP-GPU environment and its comparison to other approaches, speciﬁcally to a CPU implementation and to the (commonly used) Haar features on the GPU. The LRD features seem very well suitable for pattern recognition by image classiﬁers. They exhibit inherent gray-scale transformation invariance, ability to

GP-GPU Implementation of the “Local Rank Diﬀerences” Image Feature

389

capture local patterns, and the ability to reﬂect quantitative changes in lightness of image areas. The implementation on the GPU is reasonably eﬃcient, and a great advantage of the LRD over the common Haar wavelets in the GPU environment is the feasibility of the pre-processing stage, or even ability of excellent performance without any input image pre-processing. The authors of this paper are currently working on an eﬃcient implementation of the whole WaldBoost engine utilizing the LRD features on the GPU, both written as shaders and using a GP-GPU environment. At the moment, the partial implementation is reasonably fast (1.6 ms looking for face in a 256 × 256 image). However, the authors have several clues how to improve the current implementation and increase its speed possibly several times. Another direction of future research is improving the training process of WaldBoost detectors using diﬀerent low-level features or even combinations of several kinds of features in a single detector. Although the implementation of the LRD on CPU, which is used in the comparison (section 4) is eﬃcient (by using recent multimedia instructions of the processor), better implementations and variation of the LRD will also be looked for on the Intel CPU platform. In any case, the results of the presented work deﬁnitely lead in a conclusion that that the Local Rank Diﬀerences features present is a vital low level image feature set, which outperforms the commonly used Haar wavelets in several important measures. Fast implementations of object detectors and other image classiﬁers should consider the LRD as an important alternative.

Acknowledgements This work has been supported by the Ministry of Education, Youth and Sports of the Czech Republic under the research program LC-06008 (Center for Computer Graphics), by the research project ,,Security-Oriented Research in Informational Technology,, CEZMSMT, MSM0021630528, and by Czech Grant Agency, project GA201/06/1821 ,,Image Recognition Algorithms”.

References 1. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001) 2. Sochman, J., Matas, J.: Learning A Fast Emulator of a Binary Decision Process. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007, Part II. LNCS, vol. 4844, pp. 236–245. Springer, Heidelberg (2007) 3. Sochman, J., Matas, J.: WaldBoost -Learning for Time Constrained Sequential Detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2 (2005) 4. General-Purpose Computation on GPUs (2008-07), http://www.gpgpu.org 5. Sinha, S.N., Frahm, J.M., Pollefeys, M., Genc, Y.: GPU-based Video Feature Tracking And Matching, Technical Report TR 06-012, Department of Computer Science, UNC Chapel Hill (May 2006)

390

A. Herout et al.

6. Michel, P., et al.: GPU-accelerated Real-Time 3D Tracking for Humanoid Locomotion and Stair Climbing. In: Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (2007) 7. Lienhart, R., Maydt, J.: An extended set of Haar-like features for rapid object detection. In: ICIP 2002, pp. I:900–I:903 (2002) 8. Ojala, T., Pietik¨ ainen, M., M¨ aenp¨ aa ¨, T.: Gray scale and rotation invariant texture classiﬁcation with local binary patterns. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 404–420. Springer, Heidelberg (2000) 9. Zemcik, P., Hradis, M., Herout, A.: Local Rank Diﬀerences -Novel Features for Image Processing. In: Proceedings of SCCG, Budmerice, SK, pp. 1–12 (2007) 10. CUDA (2008-07), http://www.nvidia.com/cuda 11. Polok, L., Herout, A., Zemcik, P., Hradis, M., Juranek, R., Josth, R.: Local Rank Diﬀerences Image Feature Implemented on GPU. In: Blanc-Talon, J., Bourennane, S., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2008. LNCS, vol. 5259. Springer, Heidelberg (2008) 12. Schapire, R., Singer, Y.: Improved boosting algorithms using conﬁdence-rated predictions. Machine Learning 37(3), 297–336 (1999)

Image Recognition Technique for Unmanned Aerial Vehicles Karol J¸edrasiak1 and Aleksander Nawrat2 1

2

Polish-Japanese Institute of Information Technology Aleja Legionow 2, 41-902 Bytom, Poland [email protected] Silesian University of Technology, Department of Automatic Control and Robotics Akademicka 16, 44-100 Gliwice [email protected]

Abstract. Developing fast and accurate 2D image processing algorithms is an important task for the practical use of cybernetics. This paper presents an algorithm for fast and accurate blob detection and extraction based on the usage of two parameters ζ and χ. The algorithm is aimed to work in the color domain to prevent any loss of information but can also be implemented on gray-scale images. Achieved regions of interest can be further processed to achieve high level description. The algorithm is implemented in Java environment in order to adduce results on diﬀerent video devices and system platforms. Keywords: robotics, image recognition, color segmentation.

1

Introduction

The most important part of computer vision is detection of the required features for further processing. No exact deﬁnition was developed for a feature, therefore it is important to specify whether edges, corners or regions are required. Numerous approaches of feature detection were developed, such as edge and corner detectors or blob detectors. In cybernetics it is important to detect speciﬁc objects by their color, shape or both, in order to maximize the chance for further successful recognition. Detection is a low-level image processing operation which often appears computationally expensive. Each of the detectors vary in terms of speed and accuracy. Classic blob detection aims at detecting regions in the image that are described by the intensity value diﬀerent than that of the surroundings. The most popular approaches are the Laplacian of Gaussian (LoG) and the Diﬀerence of Gaussian (DoG). It is required to convolve the image by a Gaussian kernel and compute the LoG or DoG operator [5]. It is an eﬃcient technique but the convolving phase of the process with kernels 5 by 5 or bigger is computationally expensive. For many years the research in the feature detection ﬁeld concentrated on gray-scale image processing. The human vision model [3],[2] can be used to improve the algorithms accuracy. Detection problem is often solved using color L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 391–399, 2009. c Springer-Verlag Berlin Heidelberg 2009

392

K. J¸edrasiak and A. Nawrat

comparator. Thanks to its simplicity and eﬃciency, the Euclidean Distance Vector is often applied to solve the color detection problem. The result depends on the deﬁnition of distance and the color model. Diﬀerent results will be achieved using Minkowski distance or Chebyshev distance. The color model used in the most of the image acquisition devices and displays is RGB color model, therefore it is the standard model for color domain images. This model stores individual values for red, green and blue color component. It uses additive color mixing [1], [4] to achieve complex colors. Real-life objects colors depend on illumination. It is easier to detect intensity changes in HSV (Hue, Saturation, Value) color model, but additional conversion from RGB color space to HSV color space is required. In HSV color model colors are separated by angles and have a diﬀerent saturation level and intensity value. Instead of using the HSV model Vector Angle metric was introduced. However both solutions or even their joint usage have certain limitations. The techniques mentioned above are used for edge detection, but can also be used for seeking for regions of interest. As an example of the robotic platform the Qwerk Robot Controller will be used. Qwerk board is capable of controlling up to four motors and sixteen servos. There is a Logitech QuickCam Communicate Deluxe webcam communicating via USB port through wireless 802.11 network adapter. The Qwerk platform used to test proposed solution was equipped with two motors and 200 MHz ARM9 RISC processor with MMU and hardware ﬂoating point unit (ﬁg. 1.a). Measurements of the used robot are 5.1 x 5.8 x 1.3 [in.].

a)

b)

Fig. 1. Used robot platforms: (a) The Qwerk robot platform, (b) The modiﬁed Hirobo Eagle GS Long

The Qwerkbot do not process data on-board therefore communication delay exists. Direct-connect mode allows external Pc to connect to Qwerk through wireless Ethernet network and process acquiring data. In such conditions the delay is a major factor requiring fast processing data. When the time constraint is very strict it is impossible to use more accurate and complicated color algorithms such as [6] or CIEDE2000. Processing computational complexity has to be as low as possible. Therefore it is not allowed to use global image data or complex ﬁlters. Presented algorithm using described robot was capable of autonomically

Image Recognition Technique for Unmanned Aerial Vehicles

393

tracking a speciﬁed object. Algorithm was also tested using the modiﬁed Hirobo Eagle GS Long UAV (Unmanned Aerial Vehicle) (ﬁg 1. b). Tests of both robots proved suﬃcient speed of presenting solution for real-time color object recognition. Though fast and accurate blob detection algorithms, which could be used in cybernetics, are still yet to ﬁnd.

Fig. 2. Action spectrum of human cones and rods

2

Algorithm Overview

The algorithm aims at fast processing and accuracy of detecting speciﬁc color and extracting blobs for further analysis, usually edge detection. The speed requirement is fulﬁlled by iterating over the image only twice and using as simple methods as possible. Accuracy is controlled by two parameters, ζ and χ. At the outset, the algorithm iterates over pixels vertically. For each pixel luminous intensity is calculated using the formula based on discrete values taken from Fig.2: 587rp + 479gp + 216bp Iv = , (1) 1000 where: rp red color intensity, gp green color intensity, bp blue color intensity, Iv - pixel luminous intensity. At the same step the distance vector between current pixel and reference color is computed. To perform this task, the previously extracted color intensities, are used in the following formula: δ = (rp − rr )2 + (gp − gr )2 + (bp − br )2 , (2) in this algorithm the following estimation of (2) is used: δ = |rp − rr | + |gp − gr | + |bp − br |,

(3)

where: rr reference pixel red color intensity, gr reference pixel green color intensity, br reference pixel blue color intensity, δ - length of distance vector between current pixel and reference pixel.

394

K. J¸edrasiak and A. Nawrat

The parameter δ is used to control the tolerance of detected pixels. If δ < ζ then current pixel is marked with a special color for the next stage of process. Real-life images consist of objects which reﬂect light and in natural way produce diﬀerent level shades and scales. To detect such smooth changes in pixels intensity a method similar to adaptive threshold is used (Fig. 3.). Previous intensity value is stored in ψ variable and intensity of the following pixel is stored in κ variable. Therefore the estimation of standard deviation is calculated by the following three values. σ = (Iv − ψ)2 + (Iv − κ)2 . (4) The usage of χ parameter enhances sensitivity of the algorithm to processing shades.

Fig. 3. A result of color detection with usage of vector distance (red line) and implemented improvement(dashed red line). Red dot marks reference pixel.

If δ < χ then current pixel is also marked for next stage. To avoid detecting every pixel with slight change in intensity (4) is calculated only for pixels in close neighborhood of the pixel marked previously as correct. The next stage of algorithm performs similar approach but horizontally. It is also checked if pixel is within the tolerance range of parameters ζ and χ. The neighborhood is calculated as the previous and the following pixel in the same row. The blob extraction is also performed at this stage. Pixels marked during the prior horizontal phase and vertical phase are the subject of extracting. Storing the whole region of interest in the created blob object is ineﬃcient. Therefore only four peripheral pixels are stored. To avoid computable expensive creating and iterating over all blob objects, it is important to check whether currently examining pixels neighborhood contains previously marked pixels. The presented approach performs a preselect stage to reduce unnecessary iterations over possibly high amount of detected blob objects. The ﬁnal stage of the algorithm ﬁnds and destroys blobs that do not match requirements such as minimum width or area. Before presenting extracted data for each blob the estimation of the center of mass is calculated using the simplest formula and previously found four peripheral pixels: Rx − Lx Bx − Tx R = Rx = Lx + , Ry = T x + , (5) 2 2 where: Rx - x coordinate of the center of the mass, Lx - x coordinate of the left peripheral pixel, Rx - x coordinate of the right peripheral pixel, Ry - y coordi-

Image Recognition Technique for Unmanned Aerial Vehicles

395

nate of the center of mass, Ty - y coordinate of the top peripheral pixel, By - y coordinate of the bottom peripheral pixel.

3

Experimental Results of the Color Object Detection

The outcome images with marked detected blobs are presented for two diﬀerent webcams. The ﬁrst is Logitech Quickcam Express [7] and the second is Trust 150 SpaceC@m Portable [8]. Each camera will be tested: – without detection, with distance vector detection, with implemented changes, – speed comparison. Example tests are shown an output image and algorithm speed measured in FPS. 3.1

Color Detection Examples

Testing scene was built of objects in interesting colors with variation of illumination. The outcome images are screens from a test application using video stream of 320x240 pixels resolution. Fig. 4. Illustrates the scene and Fig. 5. eﬀects of detection without any enhancements. Fig. 6. and 7. come from proposed solution. Finally Fig. 8. display an example of further processing.

a)

b)

Fig. 4. Screen without detection: (a) Trust 150 SpaceC@m Portable, (b) Logitech QuickCam Express

Fig. 4. presents artiﬁcially made scene for color detection testing. In the screen there are two red objects on both sides of the screen. First can be seen on the left, second is much smaller and located in the bottom-right part of the image. Bottom of the screen is ﬁlled with unsaturated dark and light colors. The background of the screen is made of blue object surrounded by a white object. Fig. 5. is illustrating a usage of Euclidean Distance Vector color detector (χ = 0) to detect green color. Green color is detected only in close distance to the reference green point. Therefore it was not possible to correctly detect smoothly illuminated object.

396

K. J¸edrasiak and A. Nawrat

a)

b)

Fig. 5. Green color detected with distance vector, tolerance 118 (ζ = 118, χ = 0). (a) Trust 150 SpaceC@m Portable screen (b) Logitech QuickCam Express.

a)

b)

Fig. 6. Green color detected (ζ = 118, χ = 9) (a) Trust 150 SpaceC@m Portable screen (b) Logitech QuickCam Express

Fig. 6. is an example of green color detection with enhancements (χ = 9). Smooth changes in illumination were successfully recognized and detected. The estimation of the center of the mass points to the middle of the detected object. Yellow arrow connects the center of the mass with the center of the screen and suggests the direction and the distance of possible future servo movements. Parameter (χ) with value 9 was set manually to eliminate any acquisition noise. Fig. 7. is an example of blue color detection with enhancements (χ = 8). Smooth changes in illumination were successfully recognized and detected. Problems occurred while processing sharp changes in illumination due to diﬀerences greater than set threshold value. It can be seen that un-sharp image contain less sharp changes and therefore was possible to detect by using (χ) parameter with value of 8. Fig. 8. displays one of many usages of proposed solution. Due to calculation simplicity proposed solution fast and easy to implement. Therefore further processing sub-images for extracting edges from colored objects and shape recognition or pattern matching can be performed. Presented solution architecture makes easy storing any additional data in created blob objects. Processing only sub-images is always much faster and allow future parallel calculations. Each of detected blob objects can be processed individually for more complex and speciﬁc problem adjusted solutions. One of possible applications may be detection of previously beaconed targets or colored object tracking.

Image Recognition Technique for Unmanned Aerial Vehicles

a)

397

b)

Fig. 7. Blue color detected (ζ = 50, χ = 8). (a) Trust 150 SpaceC@m Portable screen. (b) Logitech QuickCam Express.

Fig. 8. An example of further processing only within speciﬁed sub-images created during blob detection (The Canny Edge detector )

3.2

Speed Comparison for Diﬀerent Blob Sizes Detected

The most important requirement for real-time applications is suﬃcient processing speed. Typical size of the detected object is about 20% and the minimal speed requirement is 5 frames per second. Proposed solution was tested against blobs with diﬀerent size varying between 0% and 70%. Results were acquired using two diﬀerent video cameras to allow comparing the resulting speed values. The obtained results are presented in the Tab. 1. It is shown that the speed of the algorithm varies depending on the size of the detected blob. Typical detected object’s size is 20%, therefore empirical results prove that 12 FPS is suﬃcient speed of processing for real time decisions.

4

Conclusions

The presented algorithm aims at practical use in cybernetics therefore it was designed to detect and extract blobs eﬃciently. It was shown that the popular Euclidean distance vector approach can fail in real-life cases, and the above solution successfully corrected the inaccuracy. The χ parameter is sensitive to noise. Overall algorithm detection eﬃciency can be further improved by convolving the image with Gaussian blur mask at the beginning of the detection

398

K. J¸edrasiak and A. Nawrat

Table 1. Comparison of algorithm speed for diﬀerent blob sizes and video cameras. All values are the outcome from counting FPS. Blob size is measured in the percentages of the video stream size. TRUST 150 SpaceC@m Portable LOGITECH Quickcam Express Blob Size 320x240 320x240 0% 30 30 10% 15 14 20% 12 13 30% 10 12 40% 9 10 50% 7 9 60% 5 7 70% 4 6

Fig. 9. Graphical representation of the data from Tab. 1. No signiﬁcant diﬀerence is seen between the results of two diﬀerent video cameras.

process. Another possible improvement is performing erosion and dilatation before extracting blobs. The simplicity of the proposed approach and the stability between diﬀerent video cameras (Fig. 9.) should make it an attractive tool for cybernetic implementations.

Acknowledgement This research has been supported by State Committee for Scientiﬁc Research (MNiSW) under grant O R00 0021 06 and O R00 0022 06 in years 20082010.

Image Recognition Technique for Unmanned Aerial Vehicles

399

References 1. Luong, Q.-T.: Color in computer vision. In: Chen, C.H., Pau, L.F., Wang, P.S.P. (eds.) Handbook of Pattern Recognition and Computer Vision, pp. 311–368. World Scientiﬁc Publishing Company, Singapore (1993) 2. Amini, A., Weymouth, T., Jain, R.: Using dynamic programming for solving variational problems in Vision. PAMI 12(9) (1990) 3. Healey, G.: Segmenting images using normalized color. IEEE T. on Sys., Man, and Cyb. 22 (1992) 4. Swain, M., Ballard, D.: Color indexing. IJCV 7(1) (1991) 5. Terzopoulos, D.: On matching deformable models to images: Direct and iterative solutions. OSA Technical Digest Series, vol. 2 (1987) 6. Hedley, M., Yan, H.: Segmentation of color images using spatial and color space information. Journal of Electronic Imaging 1, 374–380 (1992) 7. Logitech, http://www.dealtime.co.uk/ xPO-Logitech-Quickcam-Express-640x480-USB-W98-Logitech 8. Trust International B.V., http://www.dealtime.co.uk/xPF-Trust-150-SpaceCam

The Performance of Two Deformable Shape Models in the Context of the Face Recognition Adam Schmidt and Andrzej Kasinski Poznan University of Technology, Institute of Control and Information Engineering ul. Piotrowo 3a, 60-965 Poznan, Poland {Adam.Schmidt,Andrzej.Kasinski}@put.poznan.pl http://www.cie.put.poznan.pl

Abstract. In this paper we compare the performance of face recognition systems based on two deformable shape models and on three classiﬁcation approaches. Face contours have been extracted by using two methods: the Active Shapes and the Bayesian Tangent Shapes. The Normal Bayes Classiﬁers and the Minimum Distance Classiﬁers (based on the Euclidean and Mahalanobis metrics) have been designed and then compared w.r.t. the face recognition eﬃciency. The inﬂuence of the parameters of the shape extraction algorithms on the eﬃciency of classiﬁers has been investigated. The proposed classiﬁers have been tested both in the controlled conditions and as a part of the automatic face recognition system. Keywords: face recognition, active shapes, normal bayes classiﬁers.

1

Introduction

The face recognition is one of the most actively researched biometric identiﬁcation tasks. Its popularity is caused mainly by two reasons. Firstly the face appearance is a prominent clue used by humans in the visual identiﬁcation. Secondly the automatic face recognition is a non-invasive method, and as such can be used even without a person’s knowledge or permission. A human face is a complex 3D body which appearance depends not only on its structure but is also strongly inﬂuenced by many external factors (such as lighting, make-up, facial hair etc.). The main diﬃculty in building an eﬃcient automatic face recognition system is to deﬁne the proper features which capture the person’s characteristic description and can be extracted under the varying conditions of the image registration. Our main goal was to test if the shape of the face and its components (eyes, nose, mouth and eyebrows) contains suﬃcient information for reliable human recognition. Therefore, two diﬀerent face shape extraction methods have been selected and compared: the Active Shape Model and the Bayesian Tangent Shape Model. To investigate the role of classiﬁcation scheme on the overall system eﬃciency three types of classiﬁers have been tested: the Normal Bayes Classiﬃer and the Minimum Distance Classiﬁer based on both Euclidean and Mahalanobis L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 400–409, 2009. c Springer-Verlag Berlin Heidelberg 2009

The Performance of Two Deformable Shape Models

401

metrics. Moreover, the inﬂuence of the reduced subspace dimension and of the length of the gradient proﬁles used on the recognition rates has been asessed. In this paper we present results of our tests. We start with a review of the currently used deformable shape models and a short presentation of the implemented shape models and classiﬁcation methods. Than we describe the data used and the environment of the conducted experiments. Finally, we give the results of applying classiﬁers to manually marked and automatically detected faces.

2

Deformable Shape Models

Kass et al. [1] were the ﬁrst to introduce a deformable shape model. The Active Contours represented each curve as an ordered set of points. The state of contours was deﬁned by two energy functionals: the internal energy responsible for the smoothness and continuity of the contour and the external energy attracting the contour to characteristic image features. The contour ﬁtting task was obtained by the minimalization of the energy sum. The main drawback of the Active Contours lies in the fact that they do not represent any knowledge of the shape of the extracted objects. The Active Shape Model (ASM) introduced by Cootes [2][3][4] was free of this fault. Statistical modelling of both the shape distribution and the appearance of the contour points resulted in a ﬂexible and powerful shape extraction method. Zhao et al.[5] proposed the Weighted Active Shape Model, which utilized the information on the contour points stability. After ﬁtting to the image contour points were projected to the shape space in a way that minimized the reconstruction error of points with the smallest movement. Additionally the authors introduced an image match measure of the whole shape rather than its particular points. This approach facilitated choosing the ﬁnal contour. Zuo and de Witt[6] noticed that the Active Shapes needed precise initialization. To provide good initial conditions for their algorithm they matched a face template to the gradient image. They also resigned from using the gradient proﬁles to ﬁt contours to the image. Instead they used a N × N pixels neighborhood decomposed with the Haar wavelets. The system developed by Ge et al. [7] strived to improve the robustness of the Active Shapes to the face pose variability. As the ﬁrst processing stage two face detectors were used: one detecting the frontal faces and the other detecting face proﬁles. Their responses were used to estimate the head rotation. The rotation angle was used to decide which of the ten previously trained Point Distribution Models should be used. Wan et al. [8] observed that diﬀerent parts of face contour model were diﬀerently disturbed by face pose changes and decided to split the face model into two submodels: one representing only the face outline and the second modelling the eyes, eyebrows, nose and mouth. To further increase the robustness of the system three independent models were created for the frontal and proﬁle face views. The genetic algorithm with chromosomes describing both submodels parameters and the similarity transformation was used to ﬁt contours to the image.

402

A. Schmidt and A. Kasinski

The ﬁtness function was based on the both submodels match measures and the third component describing the correctness of the submodels relative position. The whole procedure was computationaly expensive and authors admit that the slight improvement in contours quality was paid with a few times longer processing time. Zhou et al. reformulated the shape extraction task in the Bayesian framework. In their Bayesian Tangent Shape Model (BTSM) the shape vector was treated as a hidden state and was estimated using the Expectation-Maximization algorithm. They also used an adaptive weighting depending on the contour points stability which improved the model robustness to the noise and partial occlusions.

3

Shape Extraction

Two shape models have been used in the experiment: the basic ASM [2][3][4] and the BTSM [9]. They both utilize two submodels: the Point Distribution Model (PDM) and the Local Structure Model (LSM). The PDM is used to represent the statistical distribution of shapes in the training set. It is created by aligning set of exemplary shapes to a common coordinate frame (by using the Generalized Procrustes Analysis) and then applying the PCA to reduce the dimensionality of the model and to suppress the noise. The extracted shape is represented as a t-dimensional vector in the subspace deﬁned by the selected Principal Components. The purpose of the LSM is to model the typical neighborhood of contour points. It is achieved by sampling the gradient along the proﬁles perpendicular to the contour and passing through the model points. For each point the mean proﬁle and the covariance matrix are estimated according to the training set. The complexity of the model can be set by changing the length of sampled proﬁles. This length is deﬁned by the parameter k, which corresponds to the number of pixels sampled on each side of the contour. While ﬁtting the shape to the image particular contour points are moved to the positions which minimize the Mahalanobis distance between the sampled proﬁles and those in the LSM. Both deformable shape models diﬀer in the ﬁtting procedure (Figure 1) and the regularization function used to validate shapes. In case of the ASM the shape is extracted by iterative ﬁtting the model to the image using the LSM and ﬁnding shape parameters with PDM. In order to obtain a plausible shape parameters they are regularized by truncation to a speciﬁed range. The whole procedure is repeated until convergence. The extraction procedure of the BTSM is slightly more complicated. At each iteration the shape is calculated as a weighted mean of two shapes: one aquired by ﬁtting points positions to the image, and the second one obtained by applying continuous regularization function to the previous estimate of shape. The weight is adaptively changed to reﬂect the relation between stability of contour points and the speciﬁcity of PDM.

The Performance of Two Deformable Shape Models

403

(a) reconstructing shape x from a t-dimensional subspace

fitting shape to the image

regularization through projection and truncation (b) continuous regularization

fitting shape to the image

updating shape from both shape parameters and observed shape

Fig. 1. Fitting procedure scheme: (a) ASM (b) BTSM

The more detailed speciﬁcation of both algorithms can be found in the cited literature. Figure 2 presents examplary shapes extracted with the ASM and the BTSM.

Fig. 2. Examples of the extracted face shapes: upper row the ASM, lower row the BTSM

4

Classification

Three classiﬁcation methods have been used in the experiments: the Normal Bayes Classiﬁer (NBC), the Minimum Distance Classiﬁer using the Mahalanobis metric (MDC-M) and Minimum Distance Classiﬁers using the Euclidean metric (MDC-E) [10][11]. All of them select the class with the highest a posteriori probability according to the Bayes rule:

404

A. Schmidt and A. Kasinski

P (ωm |b) =

P (b|ωm )P (ωM ) P (b)

(1)

where ωm is the m-th class and b is the shapes parameter vector. As all classes are considered to be equally probable (P (ωM ) = const) and P (b) is equal for all classes the Eqn. 1 simpliﬁes to: P (ωm |b) = const · P (b|ωm )

(2)

The classiﬁcation methods used diﬀer in the degree of simplyﬁng assumptions used. The main supposition of NBC is that P (b|ωm ) can be modeled as a multivariate gaussian: 1 1 −1 P (b|ωm ) = exp(− (b − bm )T Cm (b − bm )) 2 2π|Cm |

(3)

where bm is a mean vector and Cm is the covariance matrix of the class m. With the assumption that all classes have the same covariance matrix (Cm = C) maximizing the a posteriori probability is achieved by minimizing the term (Eqn. 3): −1 DM = (b − bm )T Cm (b − bm ) (4) which in fact is the Mahalanobis distance. Classiﬁcation in MDC-M is achieved by ﬁnding the class m with the lowest DM . If the covariance matrix C is an identity matrix Eqn. 4 simpliﬁes even further to the Euclidan distance: DE = (b − bm )T (b − bm )

(5)

Decision in MDC-E is based on the Euclidan distance of the new vector b under consideration to the mean vectors bm .

5 5.1

Experiments Data

A new base of almost 10000 images of 100 people was gathered to provide high quality, high resolution color face images. The images were acquired in partially controlled illumination conditions over an uniform background and stored as 2048x1536 pixels JPEG ﬁles. Each picture was provided with a manually marked face and eyes ROIs which were stored in the OpenCVStorage YAML ﬁles. The main purpose of creating such an extensive database was to ensure the statistical signiﬁcance of obtained results. Our research was conducted on the selected subset of 2193 images corresponding to the near-frontal views of faces (Figure 3). Each of the images used was

The Performance of Two Deformable Shape Models

405

Fig. 3. Examples of pictures treated as near-frontal face images

manually marked with 194 point forming a contour model. The contour model is composed of the following subcontours (Figure 4): – – – – –

face outline - 41 points nose outline - 17 points eyes outlines - 20 points each eyebrows outlines - 20 points each lips outlines (inner and outer) - 28 points each

The proposed system has been developed using the Visual C++ 6.0 and Open Computer Vision Library[12].

Fig. 4. Contour model - mean shape

5.2

Experiments Setup

To test the inﬂuence of the dimensionality of the PDFM (t) on the recognition rate 7 diﬀerent PDMs with t = {10, 15, 20, 25, 30, 35, 40} were created. The fraction of the training set variance modeled by those components is shown in

406

A. Schmidt and A. Kasinski

Table 1. In order to investigate the inﬂuence of the length of the gradient proﬁle (k) on the system performance 10 LSM were created with k = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}. The PDMs and LSMs were created using face contours of 50 people (half of the base). Shapes were extracted using every possible combination of the LSM, the PDM and the shape extraction method used, which gives a total of 140 variants. The shape models have been initialized with both manually marked and automatically detected eyes positions. The eyes positions used for the automatic initialization have been detected by using the system proposed by Kasinski and Schmidt in [13][14]. For each variant three distinct sets have been created: – training set - half of shapes extracted with the manual initialization – test set I - the other half of shapes extracted with the manual initialization – test set II - shapes extracted with the automatic initialization The training sets have been used to create MDC-E, MDC-M and NBC classiﬁers, which performance has been assessed using the corresponding test sets. Table 1. The percentage of training set variance modeled by t-dimensional subspace t 10 15 20 25 30 35 40 % 95.8 97.5 98.5 99.0 99.3 99.5 99.6

5.3

Results

The main goal of this research was to determine the inﬂuence of face shape extraction method and its parameters as well as the classiﬁer type on the classiﬁcation eﬃciency. As all classes are almost equally numerous we used the True Positive Ratio (TPR) as a measure of classiﬁcation eﬃciency (Fig. 5 and 6). The MDC-E has proven ineﬀective in classifying face shapes regardless of the extraction algorithm used. It has achieved its peak eﬃciency of TPR = 58.7% in the test set I for the ASM with k = 3 and t = 40. This is the result of using PCA for dimensionality reduction. The scale of particular shape vectors is directly connected with the size of corresponding eigenvalues (due to regularization function) which practically renders the Euclidean metric useless. Using the MDC-M has given signiﬁcantly better results. Its highest TPR = 96% has been achieved in test set I for the ASM with k = 6 and t = 40. The highest TPR in the test set II was equal to 92.1% for the ASM with k = 6 and t = 35. The high reliability of those classiﬁers conﬁrms that all classes have similar covariance matrices and the Mahalanobis metric can be succesfully used to discriminate between them. The MDC-M classiﬁes shapes extracted with the ASM better than those extracted with the BTSM, although this diﬀerence is smaller in the test set II. The best classiﬁcation results have been obtained with the NBC. It has achieved the highest TPR = 98% in the test set I for the ASM with k = 6 and t = 40. In the test set II the NBC has had the peak TPR = 94.3% for the

The Performance of Two Deformable Shape Models

(a)

(b)

(c)

100

100

100

50

50

50

0 40

30

10 20

0 40

5

10

30

10 20

(d)

0 40

5

10

100

50

50

50

10 20

0 40

5

10

10 20

30

10 20

0 40

5

10

5

10 (f)

100

30

30

(e)

100

0 40

407

30

10 20

5

10

Fig. 5. The TPR in test set I as a function of k and t: (a) ASM + MDC-E (b) ASM + MDC-M (c) ASM + NBC (d) BTSM + MDC-E (e) BTSM + MDC-M (f) BTSM + NBC

(a)

(b)

(c)

100

100

100

50

50

50

0 40

30

10 20

0 40

5

10

30

10 20

5

10

(d)

0 40

100

50

50

50

10 20

10

5

0 40

30

10 20

10

5

10 (f)

100

30

10 20

(e)

100

0 40

30

5

0 40

30

10 20

10

5

Fig. 6. The TPR in test set II as a function of k and t: (a) ASM + MDC-E (b) ASM + MDC-M (c) ASM + NBC (d) BTSM + MDC-E (e) BTSM + MDC-M (f) BTSM + NBC

ASM with k = 6 and t = 35. This shows that the preassumption of modelling the classes probabilities as multivariate gaussians is admissible. The NBC also gives better results with the ASM than with the BTSM, although the diﬀerence is smaller then in case of the MDC-M. The shapes extracted with the ASM have been classiﬁed more reliably. It is probably caused by using truncation as a regularization function. This ensures that all shape parameters fall into a certain range, thus the of covariance matrices of diﬀerent classes are similar and their elements are smaller.

408

A. Schmidt and A. Kasinski

The diﬀerence between the ASM and the BTSM is less visible while using the NBC. This may lead to the conclusion that classes of shapes extracted with the BTSM have diﬀerent covariance matrices. As the BTSM has proved to be less reliable than the ASM w.r.t. the classiﬁcation eﬃciency while being computationally more complex it is not recommended to use it in a face recognition system. The length of the gradient proﬁle used strongly inﬂuenced the recognition rates. Increasing it to the value of k = 6 improved the classiﬁers reliability but further increase resulted in eﬃciency drop (especially visible for the BTSM). This can be explained by gradual addition of the information on the contour background. Initially it helps to ﬁnd the correct contour position, but after a certain point the abundance of information hinders the decision. This impacts the BTSM stronger, as it causes the instability of contour points and reduces the inﬂuence of ﬁtting the shape to the image (due to the weighted average). Increasing the PDM dimensionality initially signiﬁcantly boosted the classiﬁers performance (up to t = 20). For t greater than 30 the gain is hardly noticeable. This can be explained by the fact, that 30 ﬁrst principal components already model 99% of the training set variance. The remaining 1% is likely to contain mostly the noise introduced by manual annotation of images. The obtained results also show, that the k and t parameters inﬂuence the recognition rates independently and can be adjusted individually. Initializing deformable shape models with automatically detected eyes positions led to slight performance drop. The main reason of this is the True Negative rate of the eyes detector, which fails to detect almost 3% of eyes.

6

Conclusions

Our computations proved that the MDC-M and NBC successfully classify face shapes extracted with the ASM. The complete processing time of a single image (including face and eyes detection, shape extraction and classiﬁcation) on a PC with Intel Celeron 2800MHz and 512MB RAM was about 600ms. This clearly shows that the proposed method can be used to build fast and reliable face recognition systems. The choice of the classiﬁer used should be based on the number of available training pictures per class. The MDC-M gives lower TPR but it needs fewer training images, as the covariance matrix is assumed to be equal for all classes. Training the NBC requires more images as each class covariance matrix has to be estimated independently, but this type of classiﬁer yields much better results. Another important property of the NBC is that it facilitates easy modiﬁcations of the base. Adding new persons does not require retraining whole classiﬁer but only estimating the covariance and mean vector of the new class. This is especially useful in managing large, dynamically changing databases such as those used in population traﬃc analyses.

The Performance of Two Deformable Shape Models

409

References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1987) 2. Cootes, T., Taylor, C.: Statistical models of appearance for computer vision. Technical report, University of Manchester, Wolfson Image Analysis Unit, Imaging Science and Biomedical Engineering (2004) 3. Cootes, T., Cooper, D., Taylor, C., Graham, J.: Active shape models - their training and application. Computer Vision and Image Understanding 61(1), 38–59 (1995) 4. Cootes, T.: Image Processing and Analysis, Chapter Model-Based Methods in Analysis of Biomedical Images. Oxford University Press, Oxford (2000) 5. Zhao, M., Li, S.Z., Chen, C., Bu, J.: Shape Evaluation for Weighted Active Shape Models. In: Proc. of the Asian Conference on Computer Vision, pp. 1074–1079 (2004) 6. Zuo, F., de With, P.H.N.: Fast facial feature extraction using a deformable shape model with haar-wavelet based local texture attributes. In: Proc. of ICIP 2004, pp. 1425–1428 (2004) 7. Ge, X., Yang, J., Zheng, Z., Li, F.: Multi-view based face chin contour extraction. Engineering Applications of Artiﬁcial Intelligence 19, 545–555 (2006) 8. Wan, K.-W., Lam, K.-M., Ng, K.-C.: An accurate active shape model for facial feature extraction. Pattern Recognition Letters 26(15), 2409–2423 (2005) 9. Zhou, Y., Gu, L., Zhang, H.: Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2003) 10. Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 2nd edn. Elsevier Academic Press, Amsterdam (2003) 11. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classiﬁcation, 2nd edn. Wiley, Chichester (2001) 12. Open Computer Vision Library, http://www.intel.com/technology/computing/opencv/ 13. Kasinski, A., Schmidt, A.: The Architecture of the Face and Eyes Detection System Based on Cascade Classiﬁers. In: Computer Recognition Systems 2. Advances in Soft Computing, vol. 45. Springer, Heidelberg (2007) 14. Schmidt, A., Kasinski, A.: The Performance of the Haar Cascade Classiﬁers Applied to the Face and Eyes Detection. In: Computer Recognition Systems 2. Advances in Soft Computing, vol. 45. Springer, Heidelberg (2007)

A Hierarchical Model for the Recognition of Deformable Objects Martin Stommel and Klaus-Dieter Kuhnert Institute of Real-Time Learning Systems, University of Siegen Hoelderlinstrasse 3, 57076 Siegen, Germany {Stommel,Kuhnert}@fb12.uni-siegen.de

Abstract. This paper proposes a hierarchical model for the recognition of deformable objects. Object categories are modelled by multiple views, views in turn consist of several parts, and parts consist of several features. The main advantage of the proposed model is that its nodes can be tuned with regard to the spatial selectivity. Every node in a category, views or part can thus take on the shape of a simple bag of features or a geometrically selective constellation model including all forms in between. Together with the explicit modelling of multiple views this allows for the modelling of categories with high intra-class variance. Experimental results show a high precision for the recognition of a character from a cartoon data base.

1

Introduction

Visual object recognition has advanced from modelling rigid objects by structural 3d-primitives [9] to appearance based methods [11]. These methods avoid the problem of often unstable image segmentations by implicitly taking into account variations in lighting conditions. Appearance based methods like principle component analysis [20,11] however, treat images often as a whole and are thus sensitive to occlusion and clutter. Also, training and recognition depend on the accurate centering of the object in an image. To overcome these problems research has recently focussed on part-based approaches [21,5,6,10,12,15]. However, the recognition of deformable objects still is a diﬃcult problem. Deformations together with changes in viewing perspective lead to a high intra-class variation. This causes a large variety of possible views, where often parts of the model cannot be found reliably. Here we propose a novel part-based model that is especially designed with respect to the recognition of objects with high intra-class variation. The model stores multiple views of objects which in turn consist of several parts. Parts again are modelled as compositions of feature points. The model represents geometric relationships between parts and sub-parts as they are needed and is robust to missing parts and geometric distortions of the stored patterns. Because of a hierarchy of 4 abstract levels of image and object descriptions and the possibility to share sub-parts between multiple more abstract parts, the model beneﬁts more from the principle of compositionality than models with just one abstraction L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 410–419, 2009. c Springer-Verlag Berlin Heidelberg 2009

A Hierarchical Model for the Recognition of Deformable Objects

411

level. The model is proved by the training and recognition of a character from a cartoon data base.

2

Related Work

Currently, there is a discussion on the extend to which geometric relationships between parts should be modelled. On the one hand there is the highly restrictive constellation model proposed by Weber, Perona et al. [21,6]. In this model objects are described as a set of parts together with geometric constraints on the relative scales and positions between all pairs of parts. The geometric part relationships are inferred from co-occurrence statistics, so detection, training and recognition depend on the maximization of the image probability given a certain model. In practice the constellation model is limited to only small numbers of parts, because of the exponential complexity of possible assignments of part positions to model constellations. Crandall et al. [3] and Fergus et al. [5] ﬁnd out that models with small clique-sizes, i.e. fewer part dependencies, perform better because they can comprise more parts and are less prone to overﬁtting. On the other hand, detection fails if the reference parts can not be found, e.g. due to occlusion. Besides, the expectation maximisation algorithm used to train these models optimises only one single view. This is disadvantageous for the recognition of objects with high intra-class variation. The appearance of parts is often handled independently of the spatial relationships. Usually edge and region detectors are used to choose the position of a part in an image and represent it by a region descriptor [8,10]. To overcome the problem that no single descriptor suits all types of images, feature combinations are proposed [2,8]. Often the space of possible part representations is subsampled and stored in a codebook. This provides generic part descriptions comparable to the visual alphabet found in the inferior temporal cortex [18,13]. In contrast to the constellation model Serre et al. [15] propose a model in the form of a bag of features without geometric relationships at all. Motivated by the high performance of biological vision systems, the authors propose a part based recognition system that is modelled on the mammalian visual cortex. It consists of a hierarchy of wavelet ﬁlters and radial basis functions which are trained to mimic the behaviour of real nerve cells. Since the model is also successfully used for object recognition, this raises the question of the meaning of geometric information. To clarify this aspect, Crandall and Huttenlocher [3] compare systematically models with diﬀerent spatial constraints and show a signiﬁcant improvement of a star-shaped model over the bag of features, but no improvement when introducing stronger dependencies. However, the constellation model has only one abstract part layer where geometric relationships are modelled, whereas the biologically inspired model has up to 11 layers. Keeping in mind that in the mammalian visual cortex the receptive ﬁelds of neurons increase signiﬁcantly from one area to a subsequent one [7,14,19], the importance of geometry could also depend on the level of abstraction.

412

M. Stommel and K.-D. Kuhnert

Aside from the geometry, the higher number of compositional levels can itself be advantageous for object recognition as stated by Ommer and Buhmann [12]. They introduce a multi-level part hierarchy, where parts are composed to more abstract parts if the resulting composition leads to better class probabilities on the training samples. This corresponds to a divide-and-conquer strategy, where complex classes are split into sub-classes which are easier to learn. Finally, the geometric relationships of the best parts from the hierarchy are modelled by a Bayesian network.

3

Hierarchical Model

We use a feature detector previously used for the recognition of cars in highway scenes [16,17] which provides simple edge and area features. This feature detector is extended by additional feature points from a skeletonization/Medial Axis Transfrom. Although often more elaborate feature detectors are used [8], we chose this one because it worked for natural images and provides a combination of diﬀerent features as proposed by Burge or Leibe [2,8] to enhance generalisability. To account for the possibly diﬀerent importance of geometry on diﬀerent levels of abstraction we introduce a part hierarchy, where parts are modelled as constellations of sub-parts and sub-parts must be found within a certain radius around their expected position (Fig. 1). The positions of the sub-parts are modelled with regard to the part center, so a single missing sub-part cannot break the whole geometry like in the star-shaped variation of the constellation model [5]. Using a high value for the radius, a part can be described as a bag of features. With a low value on the other hand, a part can only be recognised if the geometry matches exactly. Together with a threshold for the number of sub-parts to be recognised object recognition can be done very quickly using a Hough-like voting strategy. With this model, we do not need do decide for or against geometry in advance but can exploit it as it is useful. The geometric dependencies of the model are suﬃciently simple to be learned eﬃciently [3]. Taken together, the model can be described more precisely as a tree, where each node p ∈ {p1 , p2 , ...}, (1) corresponds to a certain feature vector, part, view or category. A node can be written as a tuple p = {m, C, L, θ, σ, k} (2) which stores the node indices C = {c1 , c2 , ..., cm }

(3)

of the m child nodes as well as the corresponding child positions L = {x1 , y1 , x2 , y2 , ..., xm , ym }

(4)

A Hierarchical Model for the Recognition of Deformable Objects

413

Fig. 1. Hierarchical model: Features are edges (Sobel) and regions (skeletonisation). Sets of 50 to 200 features are grouped into single parts based on spatial proximity. Sets of 100 to 20.000 parts are then combined in single views. Diﬀerent views are created in order to fully cover the training samples despite high intra-class variation. Category nodes are then created to integrate ten to 100 views into single classiﬁers.

relative to the node position. The parameters θ and σ denote the threshold for the number of child nodes necessary for part recognition and the spatial tolerance, respectively. A vector k stores feature attributes and the feature type of leaf nodes. For inner nodes k stores the index of the node, so the position of a node within the hierarchy can be determined easily.

4

Object Recognition

Although the graph based model structure is convenient for training and illustration, during object recognition a simpler form is used. To achieve a bottom-up directed information ﬂow from the leaf nodes of the model to the category nodes at the top, all nodes are stored in a look-up-table LU T : kj → {i|j ∈ Ci },

(5)

where the feature vector kj of node pj serves as a key to look up the node indices i of more abstract nodes pi . This LUT corresponds to the r-table in the generalised Hough-transform [1].

414

M. Stommel and K.-D. Kuhnert

After the initial extraction of features of a test image, object recognition is done by repeatedly looking up features and instances of nodes in the LUT and propagating the detections to the top nodes. This is done in the following way: 1. First the parent nodes i1 , i2 , ... of the instance j are determined using the LUT. 2. For every parent node i the corresponding geometric relationship {xu , yu } ∈ Li is determined, where u indicates the child node j according to cu = j. 3. The position of the detected instance j is shifted by −xu , −yu towards the position of the parent node. The recognition of the sub-part j is indicated by setting bit u in an accumulator corresponding to node i. The accumulator is a map with a depth of mi bits at each position indicating which child nodes are recognised over the image coordinates x and y. 4. After all child nodes are inserted into a certain accumulator the spatial tolerance is introduced by dilating all set bits to squares with a width of σ. 5. Then the number of set bits is counted for every image position and compared to threshold θi . For positions with higher numbers of recognised child nodes new instances are created corresponding to the detection of node i. 6. This is repeated until there are only instances of category nodes. The advantage of this method is that only look-up-tables and bitwise dilation/counting is necessary for recognition. The method is accelerated further by subsampling the accumulators depending on the value of σ.

5

Training

Training comprises the grouping of parts, the selection of views and the combination of views to categories. The selection of parts is based only on positive samples. The aim is to ﬁnd parts that generalise well over the training samples. By extensive experiments concerning the cooccurrence of feature points (Fig. 2) we came to the conclusion that the spatial distance between feature points is crucial for the aggregation of features to parts. This is in accordance to other approaches [12] although explanatory statements are rarely given. We evaluate geometric relationships of features by ﬁrst clustering features based on spatial proximity. As a result for every sample image a proximity graph in the form of a dendrogram is obtained. Every node in the dendrograms is a candidate for a part. The next step in training is to transform the huge set of candidates into a compact generic part set comparable to Tanaka’s visual alphabet [18]. Therefore, we must eliminate repeating candidates and determine useful radii σ and thresholds θ. The compact part set is obtained by clustering the candidates. Since most similarity measures for tree-structures are more mathematically than visually inspired we compare candidates by the similarity of patterns that they recognise. We exploit here that a candidate can be either regarded as a pattern or a classiﬁer recognising the pattern. The similarity between two candidates is thus computed by comparing which candidate patterns can be recognised by

A Hierarchical Model for the Recognition of Deformable Objects

415

Fig. 2. Cooccurrance of feature points: First in one image a reference point and a neighboring feature point are selected. The neighboring feature point is then searched for over all sample images. In every sample image the nearest feature point with corresponding attributes is selected. The graph shows the mean distance of correspondences from the expected position (i.e. the position of the neighbor) computed for about 40.000 features.

the candidate classiﬁers in question. For every resulting cluster the candidate recognising the most other candidates in the same cluster is selected as a part. In our experiments clustering reduces a subsampled set of 30.000 candidates to only 5.000 parts. Since it turned out experimentally that high thresholds result in more precise matches, the threshold θ of all candidate parts was set to 90%. To achieve a good trade-oﬀ between geometrical correspondence and generalisability clustering is evaluated for diﬀerent radii σ depending on the size of a candidate. The best results are ﬁtted by a 2nd order polynomial, where σ = (−3.7 ∗ 10−3 ∗ size + 0.3775) ∗ size.

(6)

This leads to radii between 2 and 10 pixels for parts smaller than 60 pixels. Next, the view level of the model is trained. The function of the view level is to model enough distinct object poses to cover all training samples. The training samples are therefore ﬁrst divided into overlapping groups of preferably similar samples. For every group a single view node is created by combining all parts that recognise the samples within the group and by choosing appropriate parameters σ and θ. To ﬁnd homogeneous groups of training samples a third clustering step is executed. By recursively composing the two most similar training samples or groups of already composed samples to a new group a binary similarity tree is created (see Fig. 3). The similarity of two samples is determined by comparing the output of part classiﬁers. A complete coverage of the training samples can be achieved by creating views for all nodes along a horizontal cut through the similarity tree. Fortunately, it is not necessary to exhaustively test all possible cuts and classiﬁcation parameters in order to train the model. By estimating the discriminative power of diﬀerent models it turned out that the height and depth of a node in the similarity tree are most important for the creation of

416

M. Stommel and K.-D. Kuhnert

Fig. 3. Division of the training samples into homogeneous groups: Sample images are ﬁrst grouped with regard to their similarity. View nodes are then created for homogeneous groups.

views compared to other possible criteria like the sample sizes in a subtree or the number of features. Good views could be achieved for 1.125d − h − 2 ≥ 0,

(7)

where d and h are the depth and the height of a node in the similarity graph. A similar dependency holds for the spatial tolerance, so the radius could be linearly approximated by the equation σ = 4.41h − 1.88d + 21.89.

(8)

This results in radii between 15 and 75 pixels, which is considerably more than for nodes on the lower part level. The threshold is optimised with regard to the accuracy of recognising positive samples from a group and rejecting negative samples. At the highest level of the hiearchy category nodes are created to combine views from overlapping sample groups. Since for recognition 3d-relationships between views are not necessary and view nodes already exploit all geometric dependencies between parts, category nodes are modelled as bag of features. To reduce the number of false positives, for every training sample a category node is created which combines the outputs of all recognising view nodes. The threshold of every category node is optimised for a low number of false positives. Category nodes are thus more general than view nodes. On the other hand, single category nodes do not cover all positive samples. The presented method of category training optimises the precision of object recognition at the expense of recall. We also tested other methods but the results are only reported brieﬂy.

6

Experiments

Training was conducted on a cartoon database (Fig. 4) with 820 positive samples and 820 negative samples scanned from printed comics [4]. The results reported here are preliminary in the sense that samples were not randomised and there

A Hierarchical Model for the Recognition of Deformable Objects

417

Fig. 4. Some positive and negative sample images from the cartoon data base Table 1. Confusion matrix of the test results

Recognised as positive Recognised as negative

715 Positive samples 715 Negative samples 380 19 335 696

were some repetitions in the positive training samples. Since this may degrade generalisability we are presently conducting experiments on a correct data base. However, the test results reported here, are computed on 715 randomised positive and 715 randomised negative samples without repetitions. Table 1 shows the confusion matrix of the recognition results. Only 335 out of 715 positive samples are recognised, which corresponds to a recall (recognised positives/all positives) of 53%. This may partly result from the preliminary character of the training data base. On the other hand, only 19 out of 715 background images are recognised as positives, which is a precision (true positives/all positively recognised) of 95%. This is certainly a result of the restrictive category training which aims at a low false positive rate. As a result, an accuracy of 75% is achieved over all classiﬁcations. Two other training methods for category nodes balanced the trade-oﬀ towards a higher accuracy of 80% (precision = 87%, recall = 71%) or a higher recall of 96% (precision = 70%, accuracy = 78%), respectively.

7

Conclusion

A hierarchical part-based object model is proposed. Each node of the model is tuned to the geometric accuracy needed at a particular level of abstraction. During training, radii of 2–10 pixels are selected for parts, radii of 15–75 pixels for views and bag-of-feature classiﬁers for categories. This is in parallel to the increasing receptive ﬁeld sizes reported for the visual cortex [7,14,19]. In this regard, the results may also contribute to understand the meaning of geometric part relationships as raised by the contrary approaches of Fergus et al. [6] and Serre et al. [15]. However, due to limitations of the training samples the results

418

M. Stommel and K.-D. Kuhnert

reported here are only preliminary. Though, the high accuracy of 75% is an encouraging result for the strongly varying samples of the cartoon data base. Especially the precision of 95% indicates a high robustness.

References 1. Ballard, D.H.: Generalizing the Hough transform to detect arbitrary shapes. Pattern Recognition 13(2), 111–122 (1981) 2. Burge, M., Burger, W.: Learning Visual Ideals. In: Proc. of the 9th ICIAP, Florence, Italy, pp. 316–323 (1997) 3. Crandall, D.J., Felzenszwalb, P.F., Huttenlocher, D.P.: Spatial Priors for PartBased Recognition Using Statistical Models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10–17 (2005) 4. Disney, W.: Lustiges Taschenbuch. vol. 204, 320, 323, 327, 328, 336, 357, 367, Spezial 13, Enten Edition 7, 20, Sonderband 12, Egmont Ehapa, Berlin, Germany 5. Fergus, R., Perona, P., Zisserman, A.: A Sparse Object Category Model for Eﬃcient Learning and Complete Recognition. In: Ponce, J., Hebert, M., Schmid, C., Zisserman, A. (eds.) Toward Category-Level Object Recognition. LNCS, vol. 4170, pp. 443–461. Springer, Heidelberg (2006) 6. Fergus, R., Perona, P., Zisserman, A.: Weakly Supervised Scale-Invariant Learning of Models for Visual Recognition. International Journal of Computer Vision 71(3), 273–303 (2007) 7. Hubel, D.H.: Eye, Brain and Vision. Scientiﬁc American Library (1988) 8. Leibe, B., Schiele, B.: Analyzing Contour and Appearance Based Methods for Object Categorization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2003) 9. Marr, D.: Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman and Company, New York (1982) 10. Mikolajczyk, K., Leibe, B., Schiele, B.: Multiple Object Class Detection with a Generative Model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2006) (June 2006) 11. Murase, H., Nayar, S.K.: Visual learning and recognition of 3-d objects from appearance. Int. Journal of Computer Vision 14(1), 5–24 (1995) 12. Ommer, B., Buhmann, J.M.: Object Categorization by Compositional Graphical Models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 103–113. Springer, Heidelberg (2005) 13. Rolls, E.T., Treves, A., Tovee, M.J.: The representational capacity of the distributed encoding of information provided by populations of neurons in primate temporal visual cortex. Experimental Brain Research 114(1), 149–162 (1997) 14. Schein, S.J., Desimone, R.: Spectral Properties of V4 Neurons in the Macaque. Journal of Neuroscience 10(10), 3369–3389 (1990) 15. Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual cortex. In: Proc. of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 994–1000. IEEE Computer Society, San Diego (2005) 16. Stommel, M., Kuhnert, K.-D.: Appearance based recognition of complex objects by genetic prototype-learning. In: Proc. 13th Int’l Conf. In: Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG), Plzen, Czech Republic, January 31–February 4 (2005)

A Hierarchical Model for the Recognition of Deformable Objects

419

17. Stommel, M., Kuhnert, K.-D.: A Learning Algorithm for the Appearance-Based Recognition of Complex Objects. World Congress in Computer Science, Computer Engineering, and Applied Computing (WORLDCOMP 2006). In: Proc. The 2006 International Conference on Machine Learning; Models, Technologies & Application (MLMTA 2006), Las Vegas, Nevada, USA (2006) 18. Tanaka, K.: Inferotemporal cortex and object vision. Annual Reviews of Neuroscience 19, 109–139 (1996) 19. Trappenberg, T.P., Rolls, E.T., Stringer, S.M.: Eﬀective Size of Receptive Fields of Inferior Temporal Visual Cortex Neurons in Natural Scenes. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems, vol. 14(1), pp. 293–300. MIT Press, Cambridge (2001) 20. Turk, M., Pentland, A.: Face recognition using eigenfaces. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591 (1991) 21. Weber, M., Welling, M., Perona, P.: Unsupervised Learning of Models for Recognition. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 18–32. Springer, Heidelberg (2000)

Exploiting Quaternion PCA in Virtual Character Motion Analysis Tomasz Grudzinski The Silesian University of Technology Faculty of Automatic Control, Electronics and Computer Science Institute of Informatics 16 Akademicka Street, 44-100 Gliwice, Poland [email protected] http://www.polsl.pl

Abstract. Animating virtual human-like characters has been a challenge in computer graphics for many years. Until now no single technique exists which addresses all of the arising problems. This paper presents a short overview of the methods and directions that were taken around the world to develop a solution to the subject. Following that, the paper concentrates on example-based algorithms and statistical analysis of the virtual character motion recorded using motion capture class devices or hand-made animations. The application of PCA technique on joint orientation correlations is introduced and two approaches to the crucial issue of quaternion linearization are compared with experimental results given for each of them. Then the summary of current state of research is provided and conclusions and future work possibilities are discussed. Keywords: Animation, PCA, Motion editing, Motion capture, Motion generation.

1

Background and Purpose

The present advances in computer graphics and games industry have greatly increased the demand for the eﬀective methods of virtual character animation. The most commonly used techniques are based on the Motion Capture motion acquisition [2][12] in which the real actor movement is digitalized and used to drive the virtual character animation. Such approaches, though of very high quality, lack the level of interactivity required for most real-time applications, like computer games for example. On the other hand, highly interactive methods of generating animations in real-time basing on mathematical data, physics simulation or forward and inverse kinematics solvers [9][3], even if mathematically correct, result in animation which hardly resembles real human motion. Until now no single technique exists which addresses all of the arising problems. Many solutions were presented worldwide, but none is found satisfying both the interactivity and expressivity requirements. There is a strong need for further investigation into the subject and looking to ﬁnd a universal solution that could be used in real-time applications like games. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 420–429, 2009. c Springer-Verlag Berlin Heidelberg 2009

Exploiting Quaternion PCA in Virtual Character Motion Analysis

2

421

State of the Art

Although the inverse kinematics algorithms are able to mathematically solve most problems, they are not enough if used solely. In addition to eﬃciency issues which are fairly well addressed through CCD based techniques [9] these solutions give results that look often far from realistic. Therefore researchers are trying to work out more precise algorithms. Methods presented worldwide may be divided into three groups - procedural models, dynamic models and examplebased models. Recently some hybrid solutions combining strong points of these techniques have been presented either. In our work we have concentrated on example-based models, where the statistical analysis plays a crucial role. While procedural and dynamic models may still oﬀer some valuable ideas and solutions, recent works point to a conclusion that basing on a database of known motions, possibly taken from motion capture class devices, can result i more universal and applicable solutions and procedural algorithms like IK and FK should be rather used as supporting techniques than a basis for the whole animation system. 2.1

Example-Based Models

Both procedural and dynamic animation models are based on creating a new motion without knowledge or analysis of existing one. Although such an approach seems animator-independent and therefore attractive, the results showed by researchers investigating these methods are not really encouraging. It is very diﬃcult to achieve a high level of expressivity and a truly realistic motion without any support from the professional graphics artists or data gathered from the Motion Capture class devices. Nowadays more and more articles and research reports relates to approaches that concentrate on deriving a new motion from the database of existing, artist- or mocap-made animations. Brand and Hertzmann [4] describe a very interesting method of generating animations according to the behaviour style determined on the base of example animations. They have chosen a pure statistical approach based on Hidden Markov Models [6]. The authors describe the method of creating the basic motion model using the pure Motion Capture data, using it as a reference and create more motion models varying in motion style so that unique animations can be generated for diﬀerent virtual characters. Such animations can diﬀer in details that are insigniﬁcant from the generator’s point of view, but strongly aﬀect the ﬁnal impression. The authors state that stylistic variations in the input data are detected using the PCA algorithm but do not cover the PCA application in details. Moreover, the described method is not capable of working in realtime but rather designed to support motion editing in motion building software applications. A similar approach is presented in [17]. Again the authors propose using PCA technique on a wide set of example animations to detect the stylistic degrees of freedom and then by modifying the eigenvector weights create new animations that diﬀer in style from the input motions. The work concentrates on locomotion so it can hardly be called universal, nevertheless it is yet another

422

T. Grudzinski

example of using statistical analysis on examplary data to generate new, unique animations. Some basic application of PCA method is also described in PhD thesis of M. P. Johnson [9]. The whole thesis is a wide summary of the animation issues along with the introduction of several new, interesting ideas. It starts with the comparison of rotation representation methods and following the conclusion, that the quaternion is the most robust and universal one it concentrates on the joint statistical analysis and using the joint statistical model in inverse kinematics algorithms like Cyclic Coordinate Descent. Through the logarithmic mapping Johnson converts the non-linear quaternion values in S3 to the linear tangent space in R3 so the values may be processed through the linear algorithms like PCA that assume euclidean data. Having these the author introduces the Eigenpostures algorithm that uses PCA to compute the principal postures of the virtual character skeleton using the animation database as a set of samples. However, the work does not evolve into more sophisticated use of PCA but rather ends with the conclusion, that more work should be done in the subject. Another attractive technique has been described in [14]. Taking a dancing character as an example the authors present a solution which they call a motion texture deﬁned as a set of textons and their time and space distribution that describes the nature of the captured motion. The texton is given as a linear dynamic system (LSD) and texton distribution is a transition matrix indicating how likely each texton is switched to another. The texton themselves and relations between them are created on the base of the original motion. Such a two-level architecture provides wide possibilites of generating new animations - ﬁrst by modifying single textons through the standard forward and inverse kinematics algorithms and second by making use of the transition matrix generate a kind of new choreography while still preserving the high probability of achieving very realistic outcome of the ﬁnal animation. The combination of these two methods of changing the animations delivers a strong tool of creating new unique animations. The weakness of this method is that is is hardly possible to generate the motion that is not represented by any similar one in the input animation database and therefore the generation algorithm is restricted to the animation family given by the input database. The research results presented in [13] point to a similar direction. The proposed solution is once again based on the analysis of the database of example human motion taken by Motion Capture devices and once again divides the motion data into two levels of details. There are animation transition graphs created on each level and once again Hidden Markov Models are used as a mathematical tool to implement the algorithm. Similar to the previous method, modiﬁcations on any of these levels allow changing the animation both in small stylistic variations and the whole motion choreography. Additionally, three interfaces for driving the model are presented. Unfortunately, the solution suﬀers from the same problems as the previous one - the generated animations must be at least similar to the basis motions.

Exploiting Quaternion PCA in Virtual Character Motion Analysis

423

A little diﬀerent though interesting either is the approach presented in [7][16]. The authors present an original system of controlling the virtual character animation that is a bit more like a procedural model, but is worth mentioning as we took some of the ideas from their solution. The animation engine called FreeWill+ implemented by the authors introduces a very interesting idea of an intelligent recurrent decomposition of complicated motion commands (tasks) passed to the virtual character into several simpler subtasks and then again and again until the single subtask comes down to a simple forward or inverse kinematics problem. The system assumes the possibility to write additional motion solvers that can operate on every level of motion decomposition therefore extending the engine capabilities. Such architecture assures that the system is ﬂexible and easily extendable and - what is very signiﬁcant - eﬃcient enough to be working in real-time applications. In the following sections we introduce shortly the base concept of the original animation system and then present its statistical analysis subsystem concentrating on the issue of quaternion PCA which is crucial of the whole system.

3

Animation Engine Concept

Deriving from the research results in the most popular example-based methods a new, hybrid method is proposed. The solution assumes an open program architecture which automatically divides the highly complicated motion tasks to simpler ones, similar to the method described in [7]. However, unlike in [7], every time it is possible the example motion database is used to generate the resulting motion, and kinematics solvers used only to tweak the result where required. There is a possibility to implement automatic plug-ins handling ’special cases’ which oﬀers a chance to deal with the most important task in a speciﬁc way without interfering with the system architecture. 3.1

Data Representation

The motion data is represented by a series of keyframed bone tracks containing translational and rotational data deﬁned relative to the parent bone along with keyframe occurrence time. An animation is a set of one or more bone tracks describing the skeleton motion. For the purpose of some advanced statistical techniques used in our engine it is essential to gather as many basis animations as possible. For the same purpose we need the animations to be uniﬁed in terms of sampling frequency, i.e. the frame time intervals should be constant among the entire animation database. The representation form does not guarantee that and therefore we need to preprocess the database before getting to statistical calculations to achieve its uniform state. Therefore we create quaternion curves using the S3 into R4 map introduced in [10] which will be covered later in more details and Catmull-Rom standard splines in linear R4 space, then resample the curves with the constant time interval. Eﬀectively, we get the uniform animation database with every animation consisting of one or more bone tracks, and the tracks deﬁned by the series of keyframes with regular time intervals.

424

3.2

T. Grudzinski

Animation Blending

Displaying a single animation seems trivial and so it is. If more than one animation is played simultanously, animation blending algorithms are used. We use standard LERP and SLERP [shoemake85] algorithms for translation (vector) and rotation (quaternion) blending respectively. This technique gives perfect results when it comes to blend between two motions and if fairly eﬀective with a greater number of motions as well. 3.3

Joint Limits Estimation

From all the approaches towards incorporating the joint limits into the animation system the most appealing seems to be the one introduced in [9]. The author uses the term QuTEM to describe the statistical attributes of the joint: – – – –

M - the mean rotation (quaternion) R - local principal axes of the joint motion (quaternion) V - variances on the principal axes of motion (vector in R3 ) d - maximum allowable distance from the Mean

The attributes are estimated on the basis of a database of existing animations. We use similar structure to QuTEM, but instead of using a single distance metric between joint positions we check the joint constraints separately on all of the principal axes of the joint motion. The tests we conducted gave better results when using three single-axis checks separately.

4

Statistical Analysis of Motion Structure

After getting the base of the engine to work we can concentrate on the most important part which is the statistical analysis of the animation database. We have chosen the PCA algorithm as our primary analysis tool and decided to use it on our joint orientation data. The most basic version of Eigenpostures PCA algorithm was described in [9], but the author performed only initial testing and found that there was little correlation between posture data when the whole skeleton was to be considered. The problem with applying Principal Component Analysis technique to joint motion data is that the rotational data (typically in quaternion form) does not form linear space and PCA is a strictly linear algorithm. As Johnson stated, to apply PCA to quaternion data one needs to convert non-linear quaternion values to a linear space where PCA can be performed. Such conversion must be invertible to allow converting back the result to the quaternion form and preserve as many quaternion properties as possible. We found two mappings that meet these requirements in the literature. The ﬁrst one evolves just from the Euler’s theorem and is described in [9] - it is logarithmic mapping from S3 to R3 (1) and exponentiating back from R3 to S3 (2).

Exploiting Quaternion PCA in Virtual Character Motion Analysis

ln Q = n

θ 2

425

(1)

θ θ θ en 2 = (n sin , cos ) (2) 2 2 The second one is the transformation mentioned above in section concerning quaternion curves and described in [10]. The transformation takes the quaternion value from S3 to a vector value in R4 (3) and is unambigous and invertible except the identity quaternion, which does not map to a proper vector in R4 (4). Though it might seem problematic, in fact one can treat the identity quaternion as a special case and map it into zero length vector in R4 and back. We have found this solution acceptable.

Q.x V .x = 2(1 − Q.w) Q.y V .y = 2(1 − Q.w) Q.z V .z = 2(1 − Q.w) 1 − Q.w V .w = 2(1 − Q.w) 2V .xV .w V + V .y 2 + V .z 2 + V .w2 2V .yV .w Q.y = V .x2 + V .y 2 + V .z 2 + V .w2 2V .zV .w Q.z = V .x2 + V .y 2 + V .z 2 + V .w2 V .x2 + V .y 2 + V .z 2 − V .w2 Q.w = V .x2 + V .y 2 + V .z 2 + V .w2 Q.x =

(3)

.x2

(4)

In the PCA algorithm we found in [9] we have included several enhancements that we found helpful in verifying the linearization methods and PCA analysis as a whole. The two of them that are crucial for the result presented here are: – Specifying a subset of joints to analyze – Specifying a method of quaternion linearization Below we present the comparison of two linearization methods as RMS (Root Mean Square) error measure as a function of eigenpostures used in posture reconstruction. The error measure relies on the quaternion distance metric which deﬁnes the distance between the quaternions R1 and R2 as the length of the logarithmed quaternion R1∗ R2 . The results were measured for the 21-joint character

426

T. Grudzinski

Fig. 1. Reconstruction error as a function of Eigenpostures used in two methods of Quaternion linearization for 21-joint character motion analysis

Fig. 2. Reconstruction error as a function of Eigenpostures used in two methods of Quaternion linearization for 4-joint leg motion analysis

and 4-joint leg to show the diﬀerences between the whole body and a single lamb cases. As can be seen, the results obtained while linearization method was logarithmic mapping are better than the second method. In following images we present the posture images for eigenposture reconstruction using logarithmic linearization method with diﬀerent numbers of eigenpostures used so the reconstruction error may be assessed visually.

Exploiting Quaternion PCA in Virtual Character Motion Analysis

427

Fig. 3. Lola projected postures, from left to right: A - all 63 Eigenpostures, B - 35 Eigenpostures, C - 30 Eigenpostures, D - 25 Eigenpostures, E - 15 Eigenpostures

Fig. 4. Lola projected postures (left leg analysis), from left to right: A - all 12 Eigenpostures, B - 7 Eigenpostures, C - 5 Eigenpostures, D - 3 Eigenpostures, E - 2 Eigenpostures

5

Conclusions and Future Work

The animation engine and its PCA subsystem introduced in this paper are far from complete. The paper presents the initial results from early motion analysis algorithms, but the work in the subject must be continued. The initial tests were run on a very small example database and there is no doubt that more research must be done, but some conclusions can already be made: – Logarithmic mapping gives far better results than map described in [10] that is used in creating quaternion curves. Additionally, the logarithimc mapping uses R3 rather than R4 to represent the quaternions in linear space and has therefore 3 DOFs per joint, while the second transformation has 4 of them; – There might be a need to ﬁnd a better linearization method, as the results of the PCA projection and reconstruction error tests are not as good as expected;

428

T. Grudzinski

– If no better quaternion linearization method can be found, there may be a need for another quaternion mapping methods, some sort of kernel tricks could provide a more reasonable results; – PCA analysis seems to be more eﬀective for the small sets of joints, for example a single hand, as in wider sets of joints the correlations seems to be quite irrelevant; – It might be an interesting idea to measure not only joint orientations, but their angular velocities either. Angular velocities may be computed through calculating tangents to joint orientation quaternion curves at speciﬁc times. Beside the example motion analysis, forward and inverse kinematics solvers should be enhanced to make use of the knowledge of the motion structure. Basing on the constraint values (eﬀector position, motionless joints) and eigenposture values an initial guess for most of the interactive movements could be proposed and that would greatly increase the expressivity of the eﬀecting motion.

References 1. Baraﬀ, D., Witkin, A.: Large Steps in Cloth Simulation. Computer Graphics, 43–54 (July 1998) 2. Barbic, J., Safonova, A., Pan, J.-Y., Faloutsos, C., Hodgins, J.K., Pollard, N.S.: Segmenting Motion Capture Data Into Distinct Behaviours. In: Proceedings of Graphics Interface 2004 (GI 2004), London, Ontario, Canada, May 17-19 (2004) 3. Blow, J.: Inverse kinematics with joint limits. Game Developer Magazine (2002) 4. Brand, M., Hertzmann, A.: Style machines. In: Computer Graphics (Proceedings of SIGGRAPH 2000) (2000) 5. Cohen, M.F.: Interactive Spacetime Control for Animation. Computer Graphics 26, 293–302 (1992) 6. Eddy, S.R.: What is a Hidden Markov Model? Nature Biotechnology 22, 1315–1316 (2004) 7. Francik, J., Szarowicz, A.: Character Animation with Decoupled Behaviour and Smart Objects. In: 6th International Conference on Computer Games CGAIMS, Louisville, Kentucky, USA, July 27-30 (2005); extended Paper 8. Herda, L., Urtasun, R., Fua, P., Hanson, A.: Automatic Determination of Shoulder Joint Limits using Quaternion Field Boundaries. International Journal of Robotics Research 22(6) (June 2003) 9. Johnson, M.P.: Exploiting Quaternions to Support Expressive Interactive Character Motion. PhD thesis, Massachusetts Institute of Technology (2003) 10. Johnstone, J.K., Williams, J.P.: A Rational Quaternion Spline of Arbitrary Continuity (August. 1999) 11. Kalisiak, M., van de Panne, M.: A Grasp-based Motion Planning Algorithm for Character Animation. Department of Computer Science, University of Toronto (2001) 12. Kovar, L., Gleicher, M.: Flexible Automatic Motion Blending with Registration Curves. In: ACM SIGGRAPH (2003) 13. Lee, J., Chai, J., Reitsma, P.: Interactive Control of Avatars Animated with Human Motion Data. In: Proceedings of the 29th annual conference on Computer graphics and interactive techniques (2002)

Exploiting Quaternion PCA in Virtual Character Motion Analysis

429

14. Li, Y., Wang, T., Shum, H.-Y.: Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis. In: Proceedings of ACM SIGGRAPH (2002) 15. Shoemake, K.: Animating Rotations with Quaternion Curves. In: SIGGRAPH 1985, San Francisco, vol. 19(3), pp. 245–254 (1985) 16. Szarowicz, A., Francik, J.: Integrate and Conquer - the Next Generation of Intelligent Avatars. In: ACM SIGCHI International Conference on Advances in Computer Entertainment Technology ACE 2005, Special Session on Computer Games Technology, Valencia, Spain, 15-17 June (2005) 17. Urtasun, R., Fua, P., Glardon, P., Thalmann, D.: Mahalanobis Motion Generation (2004)

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics Jakub Grudzi´ nski The Silesian University of Technology, Institute of Informatics ul. Akademicka 16, 44-100 Gliwice, Poland [email protected]

Abstract. The paper presents a novel approach to particle systems dedicated to the use in real-time 3D graphics. A new particle system architecture is presented, with hierarchical structure of objects, implementing a scene graph- speciﬁc interfaces which automatically incorporate them into the main graph of the whole interactive scene. With the uniform parameter passing scheme, the objects can be easily edited in a way similar to any other objects (like models, lights, sounds etc.), even during the simulation, with an instant visual eﬀect. A complex hierarchy of objects grants a unique ﬂexibility and extensibility; either by parameterization of existing objects in an editor or by implementing additional task-speciﬁc objects, which are automatically incorporated into the whole hierarchy, any speciﬁc behaviour of particles may be achieved, which greatly increases the level of visual attractiveness of generated eﬀects. Keywords: Particle systems, particle eﬀects, scene graphs, real-time graphics.

1 1.1

Background and Purpose Fuzzy objects

Most of visual objects in interactive 3D graphics applications are represented with the use of a mesh consisting of a set of polygons (most commonly triangles) and deﬁning the surface of the object. However, fuzzy objects like smoke, dust, ﬁre, explosions etc. cannot be represented this way; they do not have a deﬁned surface and their shape is strongly variable in time. Since 1980s these objects have been modelled with the use of particle systems [1]. In such a system, a single particle represents an elementary volume of a physical body and the whole body is a set of particles. Since then many new technologies have been introduced and knowledge in the ﬁeld of particle systems has been greatly widened. Many natural phenomena are simulated with the use of dedicated particle systems [2, 3] and nowadays even artiﬁcial neural networks are incorporated in fuzzy objects modelling [4]. In case of interactive 3D graphics, particle systems are most commonly used for the so called particle eﬀects instances of particle systems, where the simulation, although present, is not as important as an attractive look. Such eﬀects (explosions, L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 430–440, 2009. c Springer-Verlag Berlin Heidelberg 2009

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics

431

ﬁre, smoke, dust, ﬁreworks, sprinkling water, fountains, rain, snow, visual phenomena like sun ﬂares, magical spells in fantasy games etc.) are either programmed or edited in dedicated editors [5, 6]. The main idea of this paper is to incorporate the particle system deeper into the application core, which would greatly increase the ﬂexibility of the whole system and grant a unique ability to edit particle eﬀects simultaneously with any other object directly in a ‘living’. scene. 1.2

3D Graphics Engines and Scene Graphs

Modern 3D graphics applications are implemented with the use of extensive programmable platforms, called graphics engines [7, 8]. Such engines implement the mathematical and physical basis of a virtual world (dynamic terrain generation, collision detection, physics etc., in some cases even weather simulation) and an audiovisual basis for realistic graphics rendering (skeletal animation, particles rendering, postprocessing etc.) and sounds and music support [5, 9], leaving only the logic of the ﬁnal application for developers to implement. The core of an interactive 3D graphics engine is a scene graph, a hierarchical structure of objects, representing their spatial distribution and logical relations [10]. Any object in a virtual scene is a node in the scene graph which represents the whole scene. Such objects may both represent physical bodies (e.g. animated or non-animated 3D models) and abstract objects, like a trigger placed in a selected area and calling a speciﬁed action should any object of another class come into the area during the simulation. The root of the scene graph tree is an arbitrary deﬁned global coordinate frame; other objects in the scene deﬁne their own local coordinate frames, forming a hierarchy of transformations; each object has its transformation in the coordinate frame of its parent object, and the transformation automatically deﬁnes the local coordinate frame of the object. The exemplary room with a table and a plate on it would be represented as a scene graph consisting of a room being the root object with arbitrary deﬁned global coordinate frame, a table with its transformation in the global frame and a plate with its transformation in the local coordinate frame of the table. With the graphics engines being the core of any 3D graphics application and the scene graph being the core of any engine, the incorporation of particle eﬀects into the core of any 3D graphics application means the incorporation of the eﬀects into the scene graph layer of the engine core. In such an architecture, an eﬀect would be an object similar to any other (e.g. a model) and could be placed in a scene graph tree. Following the example of a room with a table and a plate, a steam from hot soup in a plate would be a particle eﬀect located in the local coordinate frame of the plate. Treating a particle eﬀect as an object in a scene graph is not a novel idea, though. In most graphics engines particle eﬀects may be placed on a map (i.e. in a scene graph) during the edition of the map in the main level design editor1 1

Each graphics engine has a level design editor, i.e. the editor in which a designer creates a map the whole scene for further simulation (e.g. a game). For example, in case of Unreal Engine 3 the editor is named UnrealEd [5].

432

J. Grudzi´ nski

of the engine. However, the eﬀects alone are edited in external tools. The novel approach presented in this paper introduces a hierarchy of parameterizable objects, every of which is a scene graph object and is a component of a particle eﬀect.2 Such architecture makes it possible to edit particle eﬀects directly on a map (i.e. in a scene) in the level design editor with immediate visual eﬀect, even during the simulation and without any external editing tool.

2

System Architecture

2.1

Scene Graph Objects and Executives

The current implementation of the presented particle system corresponds to the scene graph layer of Flexible Reality Simulation engine [12]. The engine core in this case is made of two groups of components: scene graph objects (sgo) and scene graph executives (sge). The ﬁrst of the two are the objects which form the graph (i.e. the scene) following the example of a room with a table and plate, all those objects would be scene graph objects. Such objects are ‘program resources’ they are passive and do not perform any action on their own. However, they must implement basic functionality, depending on the classes of certain objects: for example, an animated model would have a PlayAnimation method. They also must implement the basic interfaces of the core scene graph layer of the engine, so that the engine treats them as scene graph objects. Scene graph executives are those components which perform the simulation: they operate on scene graph objects, call their methods and use the data stored in them. In most cases each of the scene graph object libraries has its own scene graph executive (or even a set of executives); each executive operates only on a certain class of objects. 2.2

Parameter Schema

Apart from the basic scene graph-oriented interfaces, every scene graph object implements a speciﬁc parameter schema interface a uniform parameter passing scheme, which makes it possible to edit parameters of the selected object in the level design editor. An object deﬁnes its editable ﬁelds, their types and their names, which are then displayed in a standard object inspector in the editor. In case of an exemplary model, such ﬁelds would be as follows: ModelFileName, ModelScale, IsVisible etc. In case of editable particle eﬀects, even an implementation of such uniform parameter passing interface is not enough, should an eﬀect be represented by a single scene graph object. Much more complex hierarchy is therefore introduced.

2

Basic idea of scene graph object hierarchy of particle eﬀects, greatly extended in this paper, has been presented in [11].

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics

2.3

433

Scene Graph Objects Hierarchy

In the presented system, particle eﬀects consist of four classes of objects: eﬀect, emitter, action and region. An eﬀect is an abstract object, which represents the whole eﬀect and provides a basic ‘play-pause-stop’ functionality and very basic set of parameters. An effect does not perform any particle simulation, but is rather a container which combines various emitters into a single whole. An emitter is a key object, responsible for the generation of particles and initialization of their parameters. Emitter does not provide any functionality through its outer interface, but provides an extensive set of parameters, including those, which deﬁne the spatial and time distribution of generated particles and those which deﬁne the initial values of parameters of particles. Any number of emitters may be assigned to a single eﬀect. An action is an object which performs the simulation (e.g. integration, colour modiﬁcations etc.). It operates on every living particle in every simulation frame. Any number of actions may be assigned to a single emitter. In most cases actions perform only a limited set of tasks, and far more than one action is assigned to a single emitter. The presented system provides a wide variety of predeﬁned actions (although any other may be added), which include: – MotionAction performs the integration, given the resultant force aﬀecting each particle (the force is stored in the particle.s data). – ColourAction performs colour modiﬁcations deﬁned by the user (i.e. a person, who creates an eﬀect with the use of the level design editor). The action provides a wide variety of parameters and operates both on RGB and HLS colour spaces. – SizeAction modiﬁes size of each particle in a way speciﬁed by the user. – DeathAction allows the user to deﬁne any condition which, if met, causes a particle to cease to exist. – A set of ForceActions add user-deﬁned forces of various classes (constant force, gravity3 etc.) which interact with the particles generated by the parent emitter. – A set of special actions deﬁning a speciﬁc, ‘eﬀect-oriented’ behaviour of particles; e.g. a WindAction based on Perlin Noise [13] causes the particles to ﬂoat as if they were moved by a realistically variable wind. A region deﬁnes a region in a local coordinate frame of its parent emitter, where the emitter generates particles. A region may deﬁne a simple 1D, 2D or 3D shape (box, sphere, ball, segment etc.) and any number of regions may be assigned to a single emitter. While an emitter generates a particle, it randomly chooses one of its regions, with probability distribution deﬁned by the volume 3

The gravity action allows to deﬁne a point mass in space which attracts particles according to an adequate equation.

434

J. Grudzi´ nski

of each region and an additional weight parameter, set by the user. The selected region returns a random position within its volume and the position is assigned to the generated particle by the parent emitter. In such a hierarchy, an emitter is responsible for the initialization of parameters of particles. However, with many types of actions, particle data structure (i.e. a set of parameters deﬁning the current state of a particle) is extensive most of the parameters are action-speciﬁc. Initialization of every parameter in most cases is therefore not necessary, because an exemplary emitter may not control actions of every class. Consequently, in the presented system, the emitter does not initialize every parameter; each action initializes its own parameters itself. The parameter schema of actions include not only parameters deﬁning the simulation (e.g. in case of ColourAction a model of colour modiﬁcation in time), but also the initial parameters of particles (initial colour4 in case of ColourAction). Because the actions are assigned to an emitter, each emitter generates only particles of an abstract class deﬁned by a certain set of actions. Moreover, each emitter generates particles which are rendered in a single speciﬁed way, deﬁned by a shader5 [14] and texture assigned to it (each particle is rendered as a billboard with a texture assigned to an emitter). Consequently, an emitter cannot be the very top object in the system hierarchy, because the user may want to combine various classes of particles in a single visual eﬀect. An eﬀect is therefore the top object; with any number of emitters assigned to it, each eﬀect may generate various types of particles with greatly diversiﬁed behaviour. The particles generated by a certain emitter may also be of various spatial relations to other objects in a tree, which adds to the uniqueness of the class of particles generated by a single emitter. Each emitter may be set as absolute or relative. Both types of emitters generate particles in positions deﬁned by a hierarchy of transformations in the scene graph tree (i.e. in their local coordinate frame) with additional local translation (the position returned by a region), but the particles emitted by an absolute emitter are then not aﬀected by the motion of their parent emitter6 . Such extension is crucial for ‘trace’ eﬀects, like smoke left by a rocket (vide Fig. 3) or smoke generator in a visualisation of planes performing air acrobatics. 4

5

6

In very most cases the initial values of particles. parameters are set with the use of normal distribution: the user deﬁnes the mean and standard deviation of each parameter. An action, which initializes parameters of each particle may therefore assign diﬀerent values to various particles. As a result, instances of the same particle eﬀect may slightly diﬀer from each other, which greatly increases the visual attractiveness of eﬀects. There are many ways of rendering particles, many of which have been introduced in [14]. An emitter is an abstract object in space, which can move in a way similar to particles. The motion of each emitter is deﬁned by a user; the emitter has an additional set of parameters for this purpose. An extended study of the motion of emitters may be found in [15].

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics

2.4

435

Scene Graph Executives

As was stated in section 2.1, any hierarchy of scene graph objects is controlled by a set of managers, the executives, each of which operates only on objects of a certain class. In case of the Flexible Reality Simulation engine, there are two particle eﬀects executives in the scene graph layer: the low-level executive and an eﬀect controller. The low-level executive is responsible for rendering and particle data memory management and in its basic functionality is similar to a classical non-scene graph- oriented particle manager [16]. In each simulation frame the low-level executive calls simulation methods provided by emitters and actions and collects rendering-oriented particle data from them. Given the sizes, colours, positions and textures of each particle and knowing the shaders set by emitters, the system renders all particle eﬀects in a virtual scene. Any other particle data is not analyzed by the executive, save only for memory allocation and deallocation for new and dead particles. The low-level executive therefore communicates only with emitters and actions and does not analyze the rest of the logical data structure of the scene graph tree. However, there must be a high-level controller, which keeps the whole scene graph coherent in terms of particle eﬀects. The eﬀect controller is responsible for registering/unregistering objects of certain classes in parent objects of superior classes, while the user deletes, copies, pastes, moves and creates new objects. Even during the simulation, an emitter moved from one parent eﬀect to another, or a region moved from one emitter to another, keeps working, providing an instant visual eﬀect. The eﬀect controller therefore operates on all objects of particle eﬀects and is always given the corresponding data (i.e. handles of the moved/copied/deleted/added objects and their previous and new parents) by the engine core, should any modiﬁcation of the scene graph tree be performed. The whole scene graph object hierarchy, scene graph executives and their relations are presented in Fig. 1.

Fig. 1. Scene graph-oriented particle system architecture: objects on the left and executives on the right. The low-level executive operates only on data provided by actions and emitters, while the controller operates on any of the objects.

436

3

J. Grudzi´ nski

Editable and Programmable Objects

Even with the much extended parameter set of objects of particle eﬀects and many predeﬁned actions, always there is a class of eﬀects which cannot be made with the use of the presented editable objects, because the behaviour of their particles is exceptional. However, with the presented architecture, the developer may easily add his own objects (speciﬁcally actions), provided that it implements a uniform scene graph-oriented interface. The user may specify his own rendering method by providing his own shader and setting it in a corresponding ﬁeld in a parameter schema of a selected emitter, and any behaviour of particles may be achieved by providing a set of user-deﬁned actions. Both the user-deﬁned and the predeﬁned objects are ‘visible’ for the editor and can be edited during the simulation. In the Ainu editor, the main Flexible Reality Simulation level design editor, the scene is rendered in real time, while the user operates on the scene graph and edits parameters of its objects. Any object may be edited, moved, copied or deleted during the simulation with instant visual eﬀect. An exemplary map view in the editor is presented in Fig. 2.

Fig. 2. Ainu level design editor. Both ﬂames and comic-like eﬀect of a ringing telephone are particle eﬀects.

In Ainu, a tree view (bottom left corner in the screen) displays the scene graph, and an object inspector (top right corner in the screen) displays the parameters of the selected object. There are three particle eﬀects in the depicted scene: two ﬂames and one comic-like eﬀect of a ringing telephone. Each of the ﬂames has only one emitter, but the telephone eﬀect has two emitters: one for notes and one for ‘rings’, each with diﬀerent set of actions. The notes’ emitter is selected in the scene graph tree view and its parameters are displayed in an object inspector.

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics

4

437

Results and Future Work

Exemplary eﬀects created with the use of current implementation prove that the presented system is capable of generating any type of particle eﬀects, from simple smoke through falling snow, comic-like eﬀects and even complex explosions. As was shown in the previous sections, the main advantage of the system is an extraordinarily high level of ﬂexibility and extensibility; the eﬀects may be edited during the simulation together with the whole scene with instant visual eﬀect and without a dedicated editor and developers may deﬁne their own additional programmable objects and modify the rendering method without interfering with the simulation. Exemplary results are presented in Fig. 3 - Fig. 9.

Fig. 3. Rocket engine ﬂame during edition: the icon and axes indicate the position of the eﬀect. The curved shape of the eﬀects is caused by the motion of emitters.

Fig. 4. Fountain eﬀect (consecutive stages)

However successful in terms of usability and visual attractiveness, the use of the presented system in four commercial games proved that it has one disadvantage: complex eﬀects consist of many scene graph objects which results in scene graphs with many eﬀects being enormously large. This noticeably slows down the loading and saving of the whole data tree of a scene. In the future, the high-level executive, the eﬀect controller, will be replaced with a more complex eﬀect manager. The actions and regions will be registered in multiple emitters and emitters will be registered in multiple eﬀects in a way which eliminates redundancy in the scene graph tree. However challenging, should the task be successfully accomplished, it may eliminate the problem of enormous size of scene graphs.

438

J. Grudzi´ nski

Fig. 5. Three types of ﬂames: torch-like (left), candle-like (middle)and match-like (right)

Fig. 6. Various stages of a complex explosion. This eﬀect consists of several emitters.

A Scene Graph-Oriented Particle System for Real-Time 3D Graphics

439

Fig. 7. A complex eﬀect: ﬁreworks

Fig. 8. Various stages of a shooting tank. The eﬀect (axes indicate its position) consists of two emitters: one for ﬂame and one for smoke.

Fig. 9. A comic-like eﬀect of a ringing telephone and two magic spells

440

J. Grudzi´ nski

References 1. Reeves, W.: Particle Systems: A Technique for Modeling a Class of Fuzzy Objects. ACM Transactions on Computer Graphics 17(3), 359–376 (1983) 2. Schpok, J., Simons, J., Ebert, D.S., Hansen, C.: A Real-Time Cloud Modeling, Rendering and Animation System. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, San Diego, California, pp. 160–166 (2003) 3. Wei, X., Li, W., Mueller, K., Kaufman, A.E.: The Lattice-Boltzmann Method for Simulating Gaseous Phenomena. IEEE Transactions on Visualization and Computer Graphics 10(2), 164–176 (2004) 4. Hastings, E., Guha, R., Stanley, K.O.: NEAT Particles: Design, Representation, and Animation of Particle System Eﬀects. In: Proceedings of the IEEE 2007 Symposium on Computational Intelligence and Games, pp. 154–160 (2007) 5. Unreal Technology, http://www.unrealtechnology.com 6. Particle Illusion 3.0, http://www.wondertouch.com/pIllusion3.asp 7. Eberly, D.: 3D Game Engine Design. Morgan Kaufmann, San Francisco (2000) 8. Eberly, D.: 3D Game Engine Architecture. Morgan Kaufmann, San Francisco (2005) 9. CryENGINE 2 Speciﬁcations, http://www.crytek.com/technology/cryengine-2/specifications/ 10. Woolford, D.: Understanding and Using Scene Graphs. COMP4201 Lectures (2003) 11. Grudzi˜ nski, J.: From Programming to Visual Content Design: A Scene Graph- Oriented Particle System for Computer Games. In: Proceedings of SAGSET 2008 38th Anniversary Conference Teaching and Learning through Gaming and Simulation, Nottingham, pp. 203–209 (2008) 12. Grudzi˜ nski, T., Mikuszewski, R.: Splitting It into Components: Designing and Interface-oriented 3D Game Engine. In: Learning with Games 2007 Conference Proceedings, Sophia Antipolis, pp. 245–252 (2007) 13. Elias, H.: Perlin Noise, http://freespace.virgin.net/hugo.elias/models/m perlin.htm 14. Engel, W.: ShaderX5: Advanced Rendering Techniques. Charles River Media (2006) 15. Ilmonen, T., Kontkanen, J.: The Second Order Particle System. The Journal of WSCG 11(2), 240–247 (2003) 16. Van der Burg, J.: Building an Advanced Particle System. Game Developer Magazine, 44–50 (2000)

Debugging, Object and State Management with OpenGL 1.x and 2.x Jaroslaw Konrad Lipowski Institute of Computer Science, Computer Graphics Laboratory Warsaw University of Technology Nowowiejska 15/19, 00-665 Warsaw, Poland [email protected]

Abstract. In this paper we present solutions to most important deﬁciencies of the current version of OpenGL library which are connected to the lack of proper debugging facilities, state management ineﬃciency and bind-to-change object management scheme. Our solutions diﬀer from the ones that have surfaced earlier in that, they are relatively lightweight, low-level and portable. Keywords: OpenGL, state, batching, management, debugging.

1

Introduction

The most notable problems associated with low-level, procedural graphics libraries such as OpenGL [1] can be associated with the lack of: – direct support for verbose debugging; – eﬃcient mechanism for saving and restoring graphics pipeline state; – eﬃcient object management scheme. First category comes from the fact that the OpenGL library was initially [2] designed as a high-performance, low-level interface to the underlying graphics hardware. At that time the hardware acceleration was mostly restricted to efﬁcient rasterization of basic geometrical primitives. The operations performed by a typical application were rather simple, with no complicated dependencies between results of their invocation. But as the hardware evolved, so did the state machine abstractly represented by the library. The dependency chains became very hard to resolve based on a simple last-error-query mechanism provided by the original library implementation. In this paper we describe our solution to this issue that is lightweight and portable, but at the same time suﬃciently verbose. Relative simplicity of the initial OpenGL revision led its authors to designing suitably simple mechanism of context state management. As a result the state is managed by ﬁne-grained state-changing commands and saved and restored using stack mechanism which incorporates operations on multiple state variables into a single function call. This mechanism worked perfectly until the number of state variables introduced by subsequent revisions of the library had grown so much, L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 441–450, 2009. c Springer-Verlag Berlin Heidelberg 2009

442

J.K. Lipowski

that saving and restoring variables belonging to diﬀerent groups proved to be a very expensive operation. In some cases it can be even more expensive, then simply querying the values of some state variables and restoring them manually. We will describe our solution to this problem in one of the following chapters. As the ﬁrst version of OpenGL focused mainly on procedural tasks of the graphics hardware, it did not concerned itself with the data storage and management problems. With the evolution of underlying architectures it became apparent that there is a need for hardware-side storage of at least geometrical and image data. This in turn led to the introduction of object-oriented concepts into clearly procedural library. First sign of this change were so called displaylists and other objects followed shortly [3]-[9]. It is common for all OpenGL data objects to be represented by handles in a form of integer values, but the methods of creation and usage of such handles can diﬀer greatly among diﬀerent object types. This somehow reassembles the attempts at development of management scheme that would be eﬃcient and at the same time compatible with existing OpenGL speciﬁcation. The most widely adopted scheme is the one represented by texture objects [2]-[3]. Despite its obvious ineﬃciency coming from the fact that each operation on an object requires it to be bound to the current context, it has became de-facto standard of OpenGL data object management. One of the most notable exceptions are query [7] and shader [8] objects, which for most operations ignore the object binding and require the user to supply a handle. We will describe our attempt at adoption of a similar scheme for all data objects, which allows us to bind objects only when they are actually used for rendering.

2

Debugging Context

OpenGL debugging capability is mostly restricted to simple get-last-error mechanism complemented by per data object queries used to obtain information speciﬁc to given object type. Such information may be presented as a string [8] or an enumeration token [9]. As the state machine has grown in complexity, this mechanism in most cases proves to be insuﬃcient to localize true cause of an error. Ideally we would like to be able to track all the library commands and report the oﬀending ones in a clear, human-readable form. Existing attempts at solving this problem, such as GLIntercept [10] or gDEBugger [11] take a form of external dynamic-linking libraries sitting between the application and the actual OpenGL implementation. Such solutions force the user to accept all the issues connected to the existence of additional invocation layer and tie her to platforms supported by the tool in question. To free ourselves of this issues, we designed a mechanism that is implemented directly in the application code. Our solution can be divided into two main parts: – generating C++ header ﬁle representing OpenGL interface as a series of preprocessor macro-deﬁnitions and – designing a class of objects storing OpenGL context related data. Detailed descriptions of both these parts follow.

Debugging, Object and State Management with OpenGL 1.x and 2.x

2.1

443

Interface Header File

The standard header ﬁle supplied with most OpenGL implementations can be insuﬃcient to perform some more complicated tasks. In our case we lacked the ability to use the code generation properties of C++ preprocessor and additional data that is present in the interface speciﬁcation ﬁles [19] but is stripped from the standard header. We process the interface speciﬁcation ﬁles with a parser prepared using the Spirit library [20] to obtain a set of macro deﬁnitions that can be directly used by C++ preprocessor for code generation. The results are stored in a single header ﬁle which contains macro deﬁnitions: – – – – – – –

mapping OpenGL types into C types; mapping .spec ﬁles meta-types into OpenGL types; resolving function return expressions for .spec ﬁles meta-types; declaring all functions at once; declaring one function at a time; deﬁning all enumeration types at once and deﬁning one enumeration type at a time.

Each macro deﬁnition for function declaration consists of: – – – –

function name; function return type; display list capability ﬂag; arguments list with meta-types and argument names.

These ﬁelds are then enclosed in expressions that can be passed as macro invocation arguments. The one-at-time versions are prepared for cases, when we want to bypass automatic code generation for a speciﬁc function or enumeration type, by the means of manual macro redeﬁnition. 2.2

Context Object

To best reﬂect the state of current OpenGL speciﬁcation [17], we chose to encapsulate all interface related data in an object representing OpenGL context. This data consists of a set of implementation dependent function pointers, which are obtained with operating system dependent routines. Each function pointer can then be invoked using member function of an interface object. There are two types of interface objects per context object: – standard interface and – debugging interface. The ﬁrst one simply resolves a call to a function pointer with given arguments, therefore imposing as small overhead as possible. The second is more heavyweight and involves optional function call logging and error checking. Each call in the debugging interface is processed as follows: – we perform the actual library function call;

444

J.K. Lipowski

– we construct human-readable string representation of a call according to function name, call arguments and optionally its result; – while in display list creation mode, we add it into string list representation; – we check for library errors, if needed; – we output the string representation of an error, if needed; – while the function call logging is active, we output the string representation of a call along with a display list string for display list execution commands. The string representation is constructed with human readability in mind. Therefore enumeration arguments are resolved into their string form and other types are presented in an expected way (see Listing 1). This is realized through appropriate specializations of type traits class templates. Debugging interface additionally stores information about display list objects. When a list creation command is called, the appropriate string representation of a list is cleared and the interface is switched to a mode appending subsequent command strings to a display list string. We omit commands that are not listable. They can be identiﬁed according to special ﬂag in the interface header ﬁle. As clearly stated in the speciﬁcation [17], we should refrain from error checking while drawing in an immediate mode. This means in our case that the debugging interface is put into no-error-checking state, whenever we begin immediate mode and put back into error-checking state, when ending this mode. To reduce debugging overhead, all the interface and mode switching is handled using appropriate interface and function pointers.

3

State Management

When it comes to managing the OpenGL state, the library oﬀers the user ability to change and query the state in a very ﬁne-grained fashion but at the same time saving and restoring is handled in large, predeﬁned blocks using stack mechanism. Such a solution was acceptable in the early days of the library development. But as the number of state variables increased over time, the problems coming from the grain diﬀerences of both types of operations became more apparent. Nowadays the stack mechanism is even considered suboptimal and cumbersome to implement to such degree, that it is dropped from API speciﬁcations that can break with backwards compatibility [12]. One of the solutions to the state management problem is to use higher level library, that can handle the optimization through appropriate hierarchical representation of the rendering problem. It is most common to use a scene-graph approach, which is implemented by such libraries as OpenSceneGraph [13], OpenGL Performer [14], Open Inventor [15] or NVIDIA Scene Graph [16]. But in some cases use of such rather large libraries can prove to be prohibitive. 3.1

State Lists

Our solution in based on a concept that we like to call state lists. It assumes that the largest, in the sense of number of commands involved, state changes

Debugging, Object and State Management with OpenGL 1.x and 2.x

445

can be grouped into user deﬁned blocks. This state changes can be for example associated with drawing diﬀerent visual eﬀects in rendering pipeline. It is also beneﬁcial from the usage point of view for the lists to represent state changes with respect to some default state. Fortunately such an assumption perfectly ﬁts the deﬁnition of OpenGL state machine, which initial state is clearly described by the speciﬁcation [17]. If the above is true, then we can simply restrict ourselves to setting the state variables that diﬀer from the default state and restoring them to their default values after the rendering of given eﬀect is done. In fact the above default state does not need to reassemble the default state as found in the speciﬁcation [17]. Nevertheless it should be assumed by all eﬀects rendering routines upon their entrance and exit. We choose to represent all state changing commands as objects that can set and restore values of given state variables. Each such object should therefore be able to perform three basic tasks: – retrieve values of state variables that are associated with given command; – set values of state variables that are associated with given command and – restore the variables to their values retrieved earlier. To implement such a scheme we need at least two copies of each state variable corresponding to the state to be set and restored later. The functionality of state retrieval can be implemented with the use of appropriate state queries. The state setting can be handled with appropriate state changing command using values supplied at the object creation time. The same applies to state restoration with the exception that we should use the values obtained during state retrieval. The simplicity and elegance of presented scheme breaks down, when it comes to implementation. As there is no clear and standard way of mapping the state changing commands to their query or default state counterparts, we have to do most of the work manually. Fortunately most of the commands can be associated with appropriate get-query supplied with suitable enumeration token. For such commands we can use preprocessor to do most of the work for us. We can also handle the mapping of arguments of state changing and query commands using argument lists described later in this paper. State changing objects are grouped into blocks using state lists. State lists implement similar functionality to this oﬀered by state changing objects: – set values of state variables that are associated with given list and – restore the variables to their earlier retrieved values. One notable diﬀerence is the lack of explicit state retrieval functionality. It comes from the fact that this step is performed automatically during ﬁrst invocation of the list. Changing state is performed by invoking each command object in the order that it was originally stored in the list. During ﬁrst list invocation each of the state changing commands is preceded with appropriate state query. State restoration is handled by invoking the state changing commands in reversed order and arguments coming from state retrieval. To further optimize the setting and restoration routines, the commands associated with them are encapsulated into display list objects. This approach works well, even when we consider the state

446

J.K. Lipowski

queries present during ﬁrst invocation of the list, as the state query commands are not compiled into display lists according to the speciﬁcation [17]. This brings up the problem of state changing commands that can not be put into display lists. Upon encountering such a command during state setting or restoration, the current display list is ended and a new one is started, immediately after such command or group of commands (see Listing 1). This ensures that the dependencies between subsequent commands are met. The not listable commands can be easily identiﬁed thanks to the header ﬁle that we described earlier. To minimize the number of display lists created, one should therefore group all the not listable commands (preferably at the start or end of a state list). Each state list therefore comprises of possibly multiple display lists interleaved with not listable state changing commands. Considering list implementation details, we have two options to work with: – implementation using dynamic list object or – static implementation using template meta-programming [18]. The second is possible with most of the modern C++ compilers, as long as the lists are short enough to ﬁt in-lined recursive invocation limits. They can give us small boost in performance during ﬁrst list invocation. Table 1. Average time in ns as measured with timer queries [25] for OpenGL stack and our lists in case of diﬀerent number of state commands and GL stack groups Commands GL State Groups GL Stack 2 1 7636.80 4 2 7612.90 6 3 8206.18

3.2

State List 1846.24 1840.32 2217.28

Command Binders

To ease the implementation of objects with complicated relationships between arguments of state changing and query commands, we additionally designed new invocation scheme. We call it command binders expressing its similarities to the standard C++ library functional class templates. Each binder object represents a single OpenGL command and additionally stores its arguments. To implement such a scheme we could use one of the following approaches: – one class data member per argument of given command or – single class data member for all arguments. As we planned to use automatic generation of such classes of objects based on interface header ﬁle prepared earlier and we could not predict the maximal number of arguments needed by commands introduced by future versions of the library, we decided to use the second method of implementation. Therefore all arguments associated with a given invocation are represented by a single list object. This list is in turn implemented as a list of virtual nodes, each potentially storing value of diﬀerent type. During the actual command invocation this list

Debugging, Object and State Management with OpenGL 1.x and 2.x

447

is traversed and each of the arguments is resolved from appropriate virtual node through pointer casting. If the command in question returns a result, it is stored in the ﬁrst element of the argument list, which is expected to be of an appropriate reference type. Obviously if we want to use object pointers or references to external data and avoid copy operation, than we should additionally design some kind of reference counting mechanism. The easiest solution is to use automatic pointers from the standard C++ library. For convenient notation of argument list creation we can use properly overloaded comma operator, that matches expected behavior when it comes to associativity and precedence rules. But we should remember to supply it with appropriate ﬁrst operand (list or node object), so that the expression can be resolved correctly. As in the case of state lists we can choose to implement the argument lists using dynamic or static lists based on template meta-programming [18].

4

Object Management

Most of the OpenGL data object types suﬀer from the negative aspects of bindto-change strategy adopted ﬁrst in texture objects. This strategy requires an object to be bound to the active context before being modiﬁed. Binding an object without a clear intent to use it for actual drawing may have some performance implications. For some object types it may be hard for an underlying driver to decide beforehand whether object will be used for drawing or just requires a state change. Taking the former decision may require moving the data object between diﬀerent regions of memory or other expensive operation. From the user’s perspective it can be hard to resolve and track bindings for objects with complicated binding strategies. The best example can be aforementioned texture objects, which require the user to specify not only object handle but also texture unit and target on that unit to which an object should be bound. To deal with this problem we decided to use the command binder objects to deﬀer some of the object state changing commands until the object is bound for actual drawing. We decided that each object can be fully described by its handle and optional target. For most of the types the target property, if present, is immutable and object can be correctly bound only to the target speciﬁed at creation time. But there are some exceptions from this rule, as for example buﬀers [6] can be bound to multiple targets. For such objects the target can be perceived as a context or binding point property. As for the diﬀerent texture binding units, they are clearly a context property, because a single object can be bound to multiple units and there can only be one active unit at a time. Therefore we represent each object as a handle, optional target and a container of command binders. The container is ﬁlled by the user with commands that should be executed on an event of next binding. The commands are executed in an order that they were stored and for that purpose we can use single linked list of binders. Executed commands are removed from the container and the operation ends when there are no more commands to execute.

448

J.K. Lipowski

For some commands the user may be required to specify the target for which the command should take eﬀect [17]. This target may be speciﬁc to an object, binding operation or the command in question. The latter can be encountered in the case of cube-map textures, where a command specifying two-dimensional image for a cube face requires a target corresponding to that face. In such cases we should store the target value just as any other command argument. We can also choose to store the target speciﬁed at object creation time, which corresponds to the object-speciﬁc case. When the target is mutable and depends on binding operation, we should provide each stored command binder with a reference to variable holding the value of actual target. This in turn can be seen as an argument for storing target value as an object property even for objects like buﬀers. As for units, they do not occur in any object changing command according to current speciﬁcation [17] and therefore there is no need to store them as a per object property.

5

Conclusions and Future Work

In this paper we presented solutions to most important deﬁciencies of the current version [17] of OpenGL library: – the lack of proper debugging facilities; – state management ineﬃciency and – object management ineﬃciency. Our solutions diﬀer signiﬁcantly from the ones that have surfaced earlier in that, they are lightweight, low-level and portable. To solve the debugging problem we have chosen direct, in-application debugging as opposed to external library [10][11], which gives us undeniable beneﬁts of ﬁne-grain control and portability. We have chosen similar goals, while designing the context state management. It is relatively simple, API independent and, what is most important, does not involve large data structures and complicated algorithms associated with the use of a scene-graph approach [13]-[16]. We can also use it to complement the functionality missing from embedded devices version of the library [12]. Finally the proposed object management scheme can be used to resolve most of the bind-to-change issues. We have chosen to implement it with smallest amount of data possible restricted to handle, binding target and command binders storage. The interface used for objects management within our framework is also minimal, restricted to binding-event signaling and binders container traversal. This gives us the advantage of independence of the API changes. We designed our solutions with future in mind, omitting API dependence whenever possible. It will be interesting to see in what degree we managed to fulﬁll our goals when new major revision of the OpenGL library [21]-[24] arrives, which is said to be solving some of the issues presented in this paper. We are looking forward to using this new version as a foundation of our solutions.

Debugging, Object and State Management with OpenGL 1.x and 2.x

Listing 1: Debugging Context Log for List with One Not Listable Command. // First Invocation of List Setting with State Queries. (Boolean) FALSE IsEnabled((EnableCap)DEPTH_TEST); (List) 1 GenLists((SizeI)1); (void) NewList((List)1, (ListMode)COMPILE); (void) Enable((EnableCap)DEPTH_TEST); (void) GetIntegerv((GetPName)ARRAY_BUFFER_BINDING, (Int32 *)06A70340); (void) BindBufferARB((BufferTargetARB)ARRAY_BUFFER, (UInt32)0); (Boolean) FALSE IsEnabled((EnableCap)VERTEX_ARRAY); (void) EndList((void)); (void) EnableClientState((EnableCap)VERTEX_ARRAY); (void) GetIntegerv((GetPName)ELEMENT_ARRAY_BUFFER_BINDING, (Int32 *)0... (List) 2 GenLists((SizeI)1); (void) NewList((List)2, (ListMode)COMPILE); (void) BindBufferARB((BufferTargetARB)ELEMENT_ARRAY_BUFFER, (UInt32)0); (void) EndList((void)); // First Invocation of List Restoration. (List) 3 GenLists((SizeI)1); (void) NewList((List)3, (ListMode)COMPILE); (void) BindBufferARB((BufferTargetARB)ELEMENT_ARRAY_BUFFER, (UInt32)3); (void) EndList((void)); (void) DisableClientState((EnableCap)VERTEX_ARRAY); (List) 4 GenLists((SizeI)1); (void) NewList((List)4, (ListMode)COMPILE); (void) BindBufferARB((BufferTargetARB)ARRAY_BUFFER, (UInt32)1); (void) Disable((EnableCap)DEPTH_TEST); (void) EndList((void)); // Second Invocation of List Setting. (void) CallList((List)1); (void) Enable((EnableCap)DEPTH_TEST); (void) BindBufferARB((BufferTargetARB)ARRAY_BUFFER, (UInt32)0); (void) EnableClientState((EnableCap)VERTEX_ARRAY); (void) CallList((List)2); (void) BindBufferARB((BufferTargetARB)ELEMENT_ARRAY_BUFFER, (Uin... // Second Invocation of List Restoration. (void) CallList((List)3); (void) BindBufferARB((BufferTargetARB)ELEMENT_ARRAY_BUFFER, (Uin... (void) DisableClientState((EnableCap)VERTEX_ARRAY); (void) CallList((List)4); (void) BindBufferARB((BufferTargetARB)ARRAY_BUFFER, (UInt32)1); (void) Disable((EnableCap)DEPTH_TEST);

References 1. OpenGL Overview, http://www.opengl.org/about/overview/ 2. The OpenGL Graphics System: A Speciﬁcation (Version 1.1), http://www.opengl.org/documentation/specs/version1.1/GLspec1.1.ps 3. GL EXT texture OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/EXT/texture.txt 4. GL ARB vertex program OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/ARB/vertex_program.txt

449

450

J.K. Lipowski

5. GL ARB fragment program OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/ARB/fragment_program.txt 6. GL ARB vertex buﬀer object OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/ARB/vertex_buffer_object.txt 7. GL ARB occlusion query OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/ARB/occlusion_query.txt 8. GL ARB shader objects OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/ARB/shader_objects.txt 9. GL EXT framebuﬀer object OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/EXT/framebuffer_object.txt 10. GLIntercept, http://glintercept.nutty.org/ 11. gDEBugger, http://www.gremedy.com/products.php 12. OpenGL ES Common Proﬁle Speciﬁcation Version 2.0.22 (Full Speciﬁcation) (April 30, 2008), http://www.khronos.org/registry/gles/specs/2.0/es full spec 2.0.22.pdf 13. OpenSceneGraph, http://www.openscenegraph.org/projects/osg/wiki/About/Introduction 14. OpenGL Performer, http://www.sgi.com/products/software/performer/overview.html 15. Open Inventor, http://oss.sgi.com/projects/inventor/ 16. NVSG SDK Details, http://developer.nvidia.com/object/nvsg_details.html 17. The OpenGL Graphics System: A Speciﬁcation (Version 2.1) (December 1, 2006), http://www.opengl.org/registry/doc/glspec21.20061201.pdf 18. Template Metaprograms, http://ubiety.uwaterloo.ca/~tveldhui/papers/ Template-Metaprograms/meta-art.html 19. OpenGL Extension Registry, http://www.opengl.org/registry/ 20. Spirit User’s Guide, http://spirit.sourceforge.net/distrib/spirit_1_8_5/ libs/spirit/index.html 21. Superbuﬀers Working Group Update, http://www.opengl.org/pipeline/article/vol001_4/ 22. The New Object Model, http://www.opengl.org/pipeline/article/vol002_3/ 23. Using the Longs Peak Object Model, http://www.opengl.org/pipeline/article/vol003_4/ 24. Another Object Lesson, http://www.opengl.org/pipeline/article/vol004_4/ 25. GL EXT timer query OpenGL Extension Speciﬁcation, http://www.opengl.org/registry/specs/EXT/timer_query.txt

Estimation of State–Space Spatial Component for Cuboid Track–Before–Detect Motion Capture Systems Przemyslaw Mazurek Szczecin University of Technology Chair of Signal Processing and Multimedia Engineering 26. Kwietnia 10, 71126 Szczecin, Poland [email protected] http://www.media.ps.pl

Abstract. In the paper spatial component estimation for Track–Before– Detect (TBD) based motion capture systems is presented. Using TBD algorithms is possible to track markers at low Signal–to–Noise Ratio level that is a typical case in motion capture system. In the paper is considered special and the most popular rectangular capture area (cuboid volumen) for three configurations of image processing schemes: full frame processing, single camera optimized and multiple cameras optimized. In the article separate TBD processing for every camera is assumed. Keywords: Estimation, Motion Capture, Track–Before–Detect, Image Processing.

1

Introduction

Motion capture systems are widely used in computer graphic animation and many others applications like biomechanic, ergonomy. The most popular and usefully are marker based optical tracking systems due to superior time and spatial resolution performance. Infrared reﬂective balls (markers) are used typically. Using a set of calibrated cameras, markers are detected, tracked, assigned to tracks and after 3D triangulation motion of markers can be obtained. The most serious problem is size of markers – due to reliability of tracked motion hundreds of markers should be user for every actor. Large markers (like table tennis balls) are obsolete contemporary due to lack of physical possibilities of placements of them on actor’s face. Small reﬂective markers are also problematic because they reﬂect small amount of energy from IR lighting system. Increasing amount of IR light on set is not recommended due to safety of actor’s eye. Using advanced tracking algorithms (Track–Before–Detect) based on tracking at ﬁrst and then the detection it is possible to track markers even if Signal–to– Noise Ratio (SNR) is near or smaller than one, so markers can be small and actors’ eye safety can be obtained. There are number of TBD algorithms: Spatio– Temporal Filters [1], Likelihood Ratio TBD [2] and Particle Filters [4,3]. Paper L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 451–460, 2009. c Springer-Verlag Berlin Heidelberg 2009

452

P. Mazurek

[5] shows possibility of tracking and method of computation cost reduction using Likelihood Ratio TBD for motion capture system. In [6] is considered special case of motion capture systems that uses cylindrical volumen (measurement space of this system) and circularly placed cameras around volumen. That paper shows implementation possibilities of TBD motion capture systems and signiﬁcant computation cost reduction. This paper considered the second one popular case of motion capture volumen that is cuboid based.

2

Geometry of Cuboid Motion Capture System

Considered cuboid motion capture system uses set of camera placed around rectangular basis (room ﬂoor) at constant height.

Fig. 1. Geometry of scene (top view)

VFOV

Volumen area

Fig. 2. Geometry of scene (left side view)

There is N x cameras marked as a Cxi where i is camera number for direction X and there is N y camera marked as a Cyi for direction Y . There are always two cameras placed oppositely, e.g. Cx5 and Cx6 for both direction sets. For all cameras there are two constant internal parameters: HF OV (Horizontal Field of View) and V F OV (Vertical Field of View).

Estimation of Spatial Component for Cuboid TBD Motion Capture Systems

3

453

Track–Before–Detect Algorithms for Motion Capture Systems

In TBD systems algorithm tries to track markers ﬁrst in the state space without knowledge if a marker is located in this state space cell or not. It allows tracking low SN R (Signal to Noise Ratio) signals (very dim markers) but this process is computationally demanding. The Likelihood Ratio TBD algorithm, detailed described in the book [2], uses recurrent method for reduction of computations and every step has a constant cost. Pseudoalgorithm can be written in the following form: Start // Initial Likelihood Ratio: Λ(t0 , s) =

p(k = 0, s) p(k = 0, φ)

f or s ∈ S

(1)

For k ≥ 1 and s ∈ S // Motion Update: Λ− (k, s) = qk (s|φ) +

qk (s|sk−1 )Λ(k − 1, sk−1 )dsk−1

(2)

S

// Information Update: Λ(k, s) = L (yk |s)Λ− (k, s)

(3)

EndFor Stop The most signiﬁcant cost of algorithm is the integral of motion update. This is a serious limitation for contemporary available computers because measurement likelihood is a camera frame size. There are possibilities [5] using additional knowledge about actor placement that allows reduce cost signiﬁcantly. Application to multiple camera motion capture systems is analyzed in [6].

4

Estimation of Computational Costs of State Space Component (Multiple Cameras Optimized)

Because both directions are orthogonal single direction set can be considered for estimation. Distance between camera and boundary of volumen can be calculated using geometrical property. b=

2 tan

Δ HF OV 2

(4)

454

P. Mazurek

b

HFOV

Volumen area

Fig. 3. Camera to volume distance geometry

Intersections of HF OV boundaries between cameras for single direction set creates lattice and number of lattice cells depends on number of cameras. Boundary placed cameras are less eﬀectively used because they utilize only partially own Field–of–View (FOV), but for serious motion capture systems hundreds of them are used and they are placed dense so Δ is small value and this assumption is acceptable.

Fig. 4. Lattice for single direction

It can be shown that for large number of cameras every cell is observed by constant number of cameras. There are two kinds of cells: tetragonal and triangular. Triangular cells outside volumen area are not considered. For case shown

Estimation of Spatial Component for Cuboid TBD Motion Capture Systems

455

in ﬁgure 4 every cell is observed by 6 cameras if there are additional left Cxi<1 and right Cxi>10 cameras. Volumen area size By is described by following formula: By = 4b = Δ

Ny 2

(5)

or

(N y + 1) (6) 2 depending on camera placement. The ﬁrst one formula assumes conﬁguration show in ﬁgure 1 and the second one assumes more optimal camera placement that assumes additional boundary oﬀset placement equal to Δ/2. In this paper is assumed ﬁrst variant. The number of cameras that observe every cell CN can be described by following formula. By = 4b = Δ

CN = By + 2

(7)

Depending on distance between camera C and point P computation cost changes also. Estimation of this distance is important for derivation of total computation cost for system, because if distance is larger angular size of actor is smaller and computation cost for particular camera is smaller. Replacing point P by 3D model of actor (like cylinder what is described in paper [6]) two variables can be introduced: AV F OV (Actor Vertical Field of View) and AHF OV (Actor Vertical Field of View). If actor has height marked as H and diameter d, for camera number i following formula can be derived. H AV F OVi = 2 arctan (8) 2|Ci P | d AHF OVi = 2 arctan (9) 2|Ci P | Only a few cameras can acquire image of actor and this set is deﬁned as a AC 2 so i ∈ AC and total computation cost related to actor image only is AF OVmax . 2 AF OVmax = AV F OVi AHF OVi (10) i∈AC

Both AV F OVi and AHF OVi are arctan(k/x) functions. Complete set of AV F OVi and AHF OVi functions is shown in ﬁgure and depending on position of P they moves left and right, changes amplitude due distance changes between camera and point P and distance changes between camera and volumen area (Ci P line outside volumen). Additionally depending on lattice call the number of odd and even cameras changes but there are always two important local maxima for P –point nearest to odd and even numbered camera. They give largest values and one of them is global maxima. Additionally P point can be moved in any direction (but inside volumen) and if this point is located on volumen boundary global maxima (corresponding to worsted computational requirements) can be obtained. Largest values located

456

P. Mazurek AHFOV AVFOV

C

volumen area

x

Fig. 5. AVFOV and AVHFOV functions shape odd camera row local maxima

even camera row local maxima

P

Fig. 6. Set of AVFOV and AHFOV intersections

on volumen boundary are for points located between cameras what is shown in ﬁgures 7 and 8. Exactly due to geometrical repetition of cameras conﬁguration there are multiple global maxima between two neighboorhoods cameras. Single case of global maxima is suﬃcient for formula derivation.

Fig. 7. Example of cost function depeded on P position (Enhanced contrast image. White is largest value of AV F OV . Cameras positions are marked as a dots. Six cameras fo single lattice cell. AV F OV = 900 ).

Angular size of actor models for camera number i is descript by following formulas where l is distance between centre of model P and camera. H d AV F OVi = 2 arctan ; AHF OVi = 2 arctan (11) 2li 2li

Estimation of Spatial Component for Cuboid TBD Motion Capture Systems

1 Cx

3 Cx

5 Cx

7 Cx

457

9 Cx

b P b

b Volumen area b

b

b

2 Cx

4 Cx

6 Cx

8 Cx

10 Cx

Fig. 8. Position of global maxima

Binding together all angular sizes following formulas are obtained where By/b + 2 is number of used cameras at one time for one orthogonal direction. b=

2 tan

⎛

Δ HF OV

(12)

2

⎞ H

⎠ AV F OVmax = 4 arctan ⎝

2 + 4 2 b2 + Δ 2

By b −3

arctan (fH (i))

H fH (i) = 2 2 2 Δ2 12 + i + (By + b) ⎛

⎞ d

⎠ AHF OVmax = 4 arctan ⎝

Δ 2 + 4 2 2 b + 2

(13)

i=0

(14)

By b −3

arctan (fd (i))

(15)

i=0

d fd (i) = 2 2 2 Δ2 12 + i + (By + b)

(16)

For overall systems (including orthogonally placed cameras) computation cost is described by following formula. 2 r r c c AF OVmax = AHF OVmax AV F OVmax + AHF OVmax AV F OVmax

(17)

458

5

P. Mazurek

Estimation of Computational Costs of State Space Component (Single Camera Optimized)

This estimation is much simpler and motion capture system uses simpler algorithm. Only information about maximal angular size is used and multiplied by number of active cameras By/b + 2. For overall systems (including orthogonally placed cameras) computation cost is described by following formula. H AV F OV1 = 2 arctan (18) 2b d AHF OV1 = 2 arctan (19) 2b By 2 AF OVmax = 2 + 2 AHF OV1 · AV F OV1 (20) b

6

Estimation of Computational Costs of State Space Component (Full Frame Processing)

Full frame processing is reference to above estimations and algorithms. Images from all cameras N are processed without additional knowledge so cost is described by following formula. 2 AF OVmax = N · HF OV · V F OV

7

(21)

Example of TBD Cuboid Motion Capture System

Markov matrix in TBD algorithms describes motion vectors and this matrix is large if assumed dynamic of marker is complex. For one megapixel camera and 50 fps rate and only 100 motion vectors there are 5G basic Multiply and Accumulate operations (this value is similar to possibilities of contemporary available PC computers). It will be assumed that single full frame is processed by single computer. Spatial component is proportional to products of angular size (actor or camera). Assuming 4 : 3 angular ratio camera and HF OV = 900 , V F OV = 67.50 and rectangular volumen By = 4b, Bx = 4b there are 24 but not optimally placed cameras in conﬁguration like in ﬁgure 1. The height of volumen is assumed 3m. 1.5m 0 = 2.445m tan 67.5 2 0 90 Δ = 2b tan = 4.89m 2 b=

Actor has height H = 2.3m and diameter d = 0.4m assumed.

(22) (23)

Estimation of Spatial Component for Cuboid TBD Motion Capture Systems

459

Volumen

Fig. 9. Example motion capture system

For full frame processing 24 cameras are always used so cost is maximal: 24 computers – 100%. For single camera optimization 12 cameras are always used but only part of image (due to availability of information about actor placement) is used. AV F OV1 = 50.40 ; AHF OV1 = 9.40 ;

(24)

Single camera optimization is very powerful – it needs only 3.8% of full frame processing system (0.9 of PC computer). For multiple camera optimization method 12 cameras are always used but this method has important advantage. If actor is quite near to the particular camera he/she has large angular size but for other cameras is smaller so computation cost is additionally lower. For analyzed example it is 3.2% of full frame processing system (0.77 of PC computer). Shown results are very signiﬁcant, but is should be considered that single actor is in the volumen area. For single camera optimization and multiple camera optimization methods number or computers (computation cost) should be multiplied by number of actors.

8

Conclusions

Proposed is method for estimation of ﬁxed computation cost for multiple camera cuboid motion capture system with TBD algorithm. Three methods are compared: full processing, single camera optimization and multiple camera optimizations. Example (that use derived formulas) shows signiﬁcant reduction of computation cost that is very important due to considered TBD algorithm requirements. Obtained results extends knowledge about TBD based motion capture systems – previous work [6] is related to cylindrical volumen.

460

P. Mazurek

Acknowledgments. This work is supported by the MNiSW grant N514 004 32/0434 (Poland). This work is supported by the UE EFRR ZPORR project Z/2.32/I/1.3.1/267/05 ”Szczecin University of Technology - Research and Education Center of Modern Multimedia Technologies” (Poland).

References 1. Barniv, Y.: Dynamic Programming Algorithm for Detecting Dim Moving Targets. In: Bar–Shalom, Y. (ed.) Multitarget–Multisensor Tracking. Artech House, Norwood (1990) 2. Stone, L.D., Barlow, C.A., Corwin, T.L.: Bayesian Multiple Target Tracking. Artech House, Norwood (1999) 3. Ristic, B., Arulampalam, S., Gordon, N.: Bayesian Beyound the Kalman Filter: Particle Filters for Tracking Applications. Artech House, Norwood (2004) 4. Doucet, A., De Freitas, N., Gordon, N. (eds.): Sequential Monte Carlo Methods in Practice. Springer, Heidelberg (2005) 5. Mazurek, P.: Likelihood Ratio Track–Before–Detect Algorithm for Motion Capture Systems. Polish Journal of Environmental Studies 16(5B), 280–285 (2007) 6. Mazurek, P.: Estimation Track–Before–Detect Motion Capture Systems State Space Spatial Component. In: Kropatsch, W.G., Kampel, M., Hanbury, A. (eds.) CAIP 2007. LNCS, vol. 4673, pp. 149–156. Springer, Heidelberg (2007)

Automatic Extraction of Graph-Like Structures from Binary Images Marcin Iwanowski Institute of Control and Industrial Electronics, Warsaw University of Technology ul.Koszykowa 75, 00-662 Warszawa, Poland [email protected]

Abstract. The paper describes a method for the analysis of the content of a binary image in order to ﬁnd its structure. The class of images it deals with consists of images showing a groups of objects connected one to another forming a graph-like structure. Proposed method extracts automatically this structure from image bitmap and produces graph adjacency matrix describing it. The method is based on morphological image processing, skeletonization and labelling.

1

Introduction

The paper describes a method for the analysis of the content of a binary image in order to ﬁnd its structure. The class of images it deals with consists of images showing at its foreground groups of objects connected one to another forming a graph-like structure. Described method extract automatically this structure from image bitmap and produces a matrix containing connections between all the objects shown on the input image - graph adjacency matrix. An example is shown in Fig. 1 - binary image shown on the left-hand side is characterized by graph-like structure of the foreground. This image consists of thicker and more compact parts which will be referred to as objects and thinner, elongated - further referred to connections. Such structure of the foreground resembles the graph where objects are its nodes, while connections are the edges of this graph. The graph describing the content of above mentioned image is shown on the right-hand side of the same picture. The proposed method is based on morphological image processing [3,9] and skeletonization [6,8,10]. In the ﬁrst step binary objects presented on the image are classiﬁed into one of two groups: objects and connections. Classiﬁcation of the image content based on spatial characteristics of image fragments follows an idea of morphological classiﬁcation described in [2,1]. In the classiﬁcation step, the morphological approach is used, which allows pixels’ classifying based on theirs positions within the image. Next, the joints of connections to objects are detected. Finally the description of each joint is extracted based on labelings. This description consists of label assigned to joint, label of object and label of connection. The set of all above descriptions is stored in a temporary vectors which describes the graph structure present on the input image. Based on these vectors, the ﬁnal product is computed - the L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 461–468, 2009. c Springer-Verlag Berlin Heidelberg 2009

462

M. Iwanowski

adjacency matrix which is of size nO × nO where nO stands for the number of graph nodes (vertices, objects). In this matrix, both rows and columns refer to graph nodes, each element of this matrix has value equal 1 iﬀ nodes related to both indices of this element are connected, and 0 otherwise. An example of such matrix describing the image from Fig. 1 is shown in Table 1. In the paper, an extended adjacency matrix is computed. Thanks to this extension, each element of this matrix contains the number of connections between every two nodes.

Fig. 1. Graph-like image content and related graph Table 1. Adjacency matrix of graph shown in Fig. 1

A B C D

A 0 1 1 1

B 1 0 0 1

C 1 0 0 0

D 1 1 0 0

The paper is organized as follows. Section 2 introduces all the necessary tools used in the proposed method. Section 3 describes the method itself. Finally, section 4 concludes the paper.

2 2.1

Tools Used Base Morphological Operators

Two base morphological operators are erosion and dilation [9,3], which - in the binary case - are deﬁned as, respectively: X B = ∩b∈B Xb ; X ⊕ B = ∪b∈B Xb ,

(1)

where X is a input image, B - is a structuring element and Xb = {x + b, x ∈ X}. Structuring element B describes pixel’s neighborhood. The simplest neighborhood in 2D case it is often 4- or 8-connected elementary structuring element and consists of a central pixel plus 4 (horizontal and vertical, B = N4 ) or 8 (horizontal, vertical and diagonal, B = N8 ) closest neighbors. Structuring element may also consist of neighboring pixels covering wider pixel neighborhood, of a given radius.

Automatic Extraction of Graph-Like Structures from Binary Images

463

By combining erosion and dilation two base morphological ﬁlters [3,9,5] are deﬁned: opening (erosion followed by dilation) and closing (dilation followed by erosion). In the current study, opening operator is used to detect objects referring to graph nodes. 2.2

Anchored Homotopic Skeletonization

The second class of image content which is necessary to detect graph structure are connections between objects (nodes) which refer to graph edges. In order to perform this task, the homotopic thinning approach is used. This way of performing skeletonization is based on the notion of simple (or deletable) pixel [10,8] i.e. such a pixel which can be removed without changing the homotopy of the binary image. A pixel p belonging to the image X is simple if and only if it satisﬁes the following three conditions: 1. NG (p) ∩ X = ∅, 2. NG (p) ∩ X C = ∅, 3. ∃S ∈ CC G (N8 (p) ∩ X) such that NG (p) ∩ X ⊆ S, where X C stands for the complement of image X. NG and NG represents the closest neighborhood of foreground and background pixels respectively. Due to connectivity paradox, diﬀerent connectivity should be used for the foreground and for the background. So either G = 8 and G = 4, or inversely. Function CCG returns the set of G-connected components of its argument. By the successive removal of simple pixels the image is thinned. Image obtained by this thinning performed till idempotence is a skeleton which - due to its homotopy preservation property - is also referred to as homotopic marking. Usually, thinning is performed in two stage iterative process. In the ﬁrst phase the simple pixels are detected within the whole image (but not yet removed), while the proper removal of these pixels is performed in the second phase. These two phases are performed iteratively until thinned image stops to change. In order to have the possibility to control the thinning process, the notion of anchor pixels has been introduced [6]. These pixels are deﬁned separately and by deﬁnition - cannot be removed during the thinning process even if they are simple. The anchored skeletonization requires two input images - the image to be thinned and the image containing anchor pixels referred to as anchor image.

3 3.1

Graph Extraction Method Detection of Objects and Connections - Graph Nodes and Edges

Images containing rasterized graph-like structures consist of two types of regions: regions referring to graph nodes (objects) and regions referring to graph edges

464

M. Iwanowski

(connections). In order to extract the graph structure from this image, at ﬁrst, one should extract from input image pixels belonging to one of two classes: objects and connections. At the beginning, objects are localized within the image. It is performed using the morphological opening operator. Usually, in majority of image processing applications, this operator is used to remove ’salt’ noise. Side-eﬀect which appears along with this removal is the modiﬁcation of shape of image foreground. This feature, considered as disadvantage from ﬁltering point of view, is a great advantage as far as the discussed task is taken into account. It allows detecting relatively thick object connected with relatively thin and elongated connections. The result of binary opening is often described as a region covered by structuring element in all possible positions totally included in the foreground of the input image. Consequently, thin fragments of of the foreground, thinner than the structuring element are not included in the result of opening. Among all such fragments of the image, also connections between objects are included. On the other hand, a majority of the surface of objects are present in the result of the opening. In case of image foreground structure resembling a graph, opening transforms in this way the single connected component of the foreground of input image into several connected components, each of which refer to single object - node of the graph. This is illustrated in Fig. 2, the result of an opening of the image on the left is shown on the right-hand side.

(a)

(b)

Fig. 2. Input test image (a) and the result of of morphological opening (b)

The result of opening contains thus simpliﬁed shape of all objects. Formally, it can be described as: IO = (I B) ⊕ B T , (2) where I stands for the input image, IO - image with simpliﬁed objects, B structuring element and B T = {p : −p ∈ B} - for its transposition.

Automatic Extraction of Graph-Like Structures from Binary Images

465

There exists direct relation between the structuring element used and the number of detected objects. The choice of structuring element is thus important since it determines which fragments of the foreground are considered as object. Consequently, it allows to ﬁnd out which of them are graph nodes, and which are not. Generally, the bigger structuring element is, the lower number of object is extracted. This is due to the fact that opening with bigger structuring element( i.e. covering wider neighborhood) would cause the removal not only of connections but also of smaller potential objects. In order to ﬁnd connections between object, the residue of opening will be analyzed1 . This residue contains all the fragments of foreground which was too thin to be able to include moving structuring element. Within these fragments, among others, also connections between objects are present. In order to ﬁnd them, the anchored homotopic thinning of the input image is applied. Since the resulting skeleton - by deﬁnition - is homotopic to the original shape, the single connected component of a graph-like structure will be transformed into a single connected component of the skeleton. Moreover, if the result of opening will be considered as the set of anchor pixels, the skeleton will contain all the skeleton lines connecting them. Result of such skeletonizing is shown in Fig. 3(a). This result is used to compute simpliﬁed connections using the following equation: IC = (IO − askel(I, IO )) ⊕ N8 ,

(3)

where IC stands for simpliﬁed connections, askel(I, IO ) is the anchored skeletonization of I with anchor image IO . An additional dilation in the Eq. 3 makes the skeleton lines thicker. Thanks to that the simpliﬁed connections image has a non-empty intersection with simpliﬁed objects one. This fact is used to detect the joints - places where connection is attached to the object. It is computed as simple intersection: IJ = IO ∩ IC .

(4)

Detected joints on the test image are shown in Fig. 3(b). Joints are marked there in black, the simpliﬁed objects and connections as solid gray, the original foreground shape as gray outline. Finally, each of three binary images: simpliﬁed objects, simpliﬁed connections and joints are labeled and three label image are created called: LO , LC and LJ , respectively. The number of connected components of IO , IC and IJ (or in other words, number of labels in LO , LC and LJ ) will be denoted as nO , nC and nJ , respectively. Labeled images will be used to match objects, connections and joints and to produce to ﬁnal adjacency matrix. 3.2

Adjacency Matrix Computation

Three labeled images obtained in the previous step are now processed in order to produce the adjacency matrix. The appropriate algorithm consists of two stages and is presented below. 1

Residue of opening i.e. diﬀerence between original image and the result of opening is called white top-hat.

466

M. Iwanowski

(a)

(b)

Fig. 3. Result of anchored skeletonization of a test image (a) and result of joints detection (b)

Algorithm Adjacency matrix computation I LO ,nO LC ,nC LJ ,nJ vi ei a

image deﬁnition domain image with labeled objects (nodes) and the number of them image with labeled connections (edges) and the number of them image with labeled joints and the number of them label of object (node) attached to i-th joint label of connection (edge) attached to i-th joint graph adjacency matrix of size nO × nO

1. for i ← 1, ..., nJ do 2. for all p ∈ I do 3. if LJ (p) = i then 4. vi ← LO (p) ; ei ← LC (p) 5. break the loop 2-5 and goto 1 6. for i ← 1, ..., nO do 7. for j ← 1, ..., nO do 8. a(i, j) ← 0 9. for i ← 1, ..., nJ − 1 do 10. for j ← i + 1, ..., nJ do 11. if ei = ej then 12. a(vi , vj ) ← a(vi , vj ) + 1 13. a(vj , vi ) ← a(vj , vi ) + 1

In the ﬁrst stage (lines 1-5) three images are analyzed and two temporary vectors v and e are produced. They contain information on the objects and connections attached to each joint: vi is the label of object attached to i-th joint, while ei stands for the label of connection attached to that joint. In the second stage (lines 6-13) the graph adjacency matrix a is computed. First, matrix elements are set to 0 (lines 6-8). Next, all the joints are analyzed.

Automatic Extraction of Graph-Like Structures from Binary Images

467

The analysis of each joint result in the increment of an appropriate element of matrix a. Since the graph - by deﬁnition - is undirected, the adjacency matrix is symmetric, so the increment is performed twice (lines 12 and 13). Since the increment is performed following appropriate joints, ﬁnally an element of the matrix contains the total number of connections (edges) between two objects (nodes). Table 2. Graph adjacency matrix of graph shown in Fig. 2

A B C D E F G H I J K L M N O

A 2 0 0 0 0 0 1 0 0 0 0 0 0 0 0

B 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

C 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0

D 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0

E 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0

F 0 0 0 1 0 0 2 1 0 0 0 0 0 0 0

G 1 0 0 0 0 2 0 1 0 1 0 0 0 0 0

H 0 0 1 0 0 1 1 0 1 0 0 0 1 0 0

I 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1

J 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0

K 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

L 0 0 0 0 0 0 0 0 0 1 0 0 2 0 0

M 0 0 0 0 0 0 0 1 0 0 1 2 0 0 0

N 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0

O 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0

Example of graph adjacency matrix of an image shown in Fig. 2(a) is presented in Table 2.

4

Conclusions

In the paper a method for extraction from binary images, structures of the foreground, that resemble graph structures, was presented. Such foreground conﬁgurations consists of thick objects connected with thin and elongated connections. The objects and connections are detected using morphological image processing, in particular morphological opening (objects) and anchored homotopic skeletonization (connections). Depending on the structuring element used, various image regions can be classiﬁed as objects. Apart from object and connections, also their joints are detected. In the next step, based on the labeling of all these three images, the graph adjacency matrix is computed, which describes the structure of resulting graph. The method can be applied in various areas of image processing where the graph-like structures are present on images under consideration. It may be used to describe these structures by providing with the graph adjacency matrix which can be used independently as ﬁnal image description or as a feature for further recognition.

468

M. Iwanowski

References 1. Iwanowski, M.: Binary Shape Characterization using Morphological Boundary Class Distribution Functions. In: Kurzynski, M., Puchala, E., Wozniak, M., Zolnierek, A. (eds.) Proc. of CORES 2007 Conference, October 22-25, 2007. Computer Recognition Systems 2 – Advances in Soft Computing, vol. 45. Springer, Heidelberg (2007) 2. Iwanowski, M.: Morphological Boundary Pixel Classiﬁcation. In: Proc. of IEEE EUROCON Conference, Warsaw, pp. 146–150 (2007) 3. Soille, P.: Morphological image analysis. Springer, Heidelberg (1999) (2004) 4. Vincent, L.: Morphological grayscale reconstruction in image analysis: applications and eﬃcient algorithms. IEEE Trans. on Image Processing 2(2) (April 1993) 5. Serra, J., Vincent, L.: An overview of morphological ﬁltering. Circuit systems Signal Processing 11(1) (1992) 6. Vincent, L.: Eﬃcient computation of various types of skeletons. In: Loew, M. (ed.) Medical Imaging V: Image Processing. SPIE, vol. 1445, pp. 297–311 (1991) 7. Vincent, L.: Exact Euclidean distance function by chain propagations. In: Proc. IEEE Computer Vision and Pattern Recognition, pp. 520–525 (1991) 8. Kong, T., Rosenfeld, A.: Digital topology: Introduction and survey. Computer Vision, Graphics, and Image Processing 48, 357–393 (1989) 9. Serra, J.: Image analysis and mathematical morphology, vol. 1. Academic Press, London (1983) 10. Rosenfeld, A.: Connectivity in digital pictures. Journal of the ACM 17(1), 146–160 (1970)

‘SNN3DViewer’ - 3D Visualization Tool for Spiking Neural Network Analysis Andrzej Kasi´ nski1 , Juliusz Pawlowski1, and Filip Ponulak1,2 1

2

Institute of Control and Information Engineering Poznan University of Technology, Poland Bernstein Center for Computational Neuroscience Albert-Ludwigs University Freiburg, Germany

{Andrzej.Kasinski,Filip.Ponulak}@put.poznan.pl, [email protected]

Abstract. We present a specialized visualization system dedicated to support analysis of dynamical processes in large spiking neural networks. The key features of the considered system are: clarity of visual representation, easy accessibility of multiple views and parameters related to the network analysis, advanced and ﬂexible GUI and the system interoperability. In this paper we focus speciﬁcally on the implementation issues related to the network 3D representation, design of the graphical objects, visualisation functions and system performance. Keywords: data visualization, 3D graphics, spiking neural networks.

1

Introduction

Biomimetics searches for biological inspirations to create improved engineering solutions. Prominent examples of biomimetic systems are artiﬁcial neural networks which attempt to mimic such properties of biological neural systems, as their learning ability and high information processing capability. Recently, spiking neural networks (SNN), i.e. networks of neuron-like units communicating by sequences of impulses [4], are the subject of growing interest. This interest is motivated by the evidence that SNN can provide eﬃcient models to explain mechanisms of neural computation or are able to eﬃciently perform real-world computational tasks [6]. However, the progress in the ﬁeld of neural computation with SNN depends strictly on the available tools for the simulation and analysis of neural ensembles. Yet, the complexity of spiking neural networks is often a constraint for the traditional analysis methods. On the other hand it is known that humans can eﬃciently process complex information if represented graphically - visual information is supporting their intuitive reasoning and hypotheses rising. Following this observation we introduce a new 3D-visualization tool designed to support analysis of dynamical processes in spiking neural networks. The presented application, called SNN3DViewer, refers exclusively to the postsimulation L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 469–476, 2009. c Springer-Verlag Berlin Heidelberg 2009

470

A. Kasi´ nski, J. Pawlowski, and F. Ponulak

visualization. The idea is to separate visualization from the simulation part, so the user can select any of the available applications for the design and simulation of spiking neural networks (such as e.g. CSIM, GENESIS, MVASPIKE, NCS, NEST, NEURON, SPLIT, XPPAUT [1]) and to visualize the results using SNN3DViewer.

2

Implementation Tools

Assuring appropriate work-speed of the visualizer (i.e. maintaining the maximum animation frame rate) was one of the main design criteria which determined selection of C++ as a programming language. C++ provides a good usage of hardware resources of a central processing unit in any computer with any operating system. A very important feature of this language is its object-orientation, which is particularly helpful in the implementation of programs designed to represent hierarchical structures of real/virtual objects. For the development of a main part of the program, that is a 3D visualization, we considered two graphics libraries: Direct3D (a part of Microsoft DirectX) and OpenGL. Eventually, we decided to use Direct3D, since it is evolving much faster than OpenGL and since the initially assumed target platform was Microsoft Windows. Implementation of the SNN3DViewer with OpenGL is planned in the near future, as to make it easier to assembly a version running under other operating systems (e.g. Linux). SNN3DViewer oﬀers an advanced graphical user interface, implemented with a CEGUI library [2], and designed according to the recommendations about the visual presentation clarity and easy graphical objects manipulation.

3

Graphical Network Representation

It is essential to elaborate proper graphical models of the visualized objects (neurons, interneuronal connections, synapses, and spikes) to ensure realistic and informative 3D neural network representation on one hand, while ensuring minimum requirements for the memory and processor resources on the other hand. Neurons are represented schematically as spheres. Each neuron is described by its 3D location, radius, ﬁring times and optionally by the internal state trace (usually corresponding to the membrane potential of the biological neurons). The internal state Vm (t) is visualized by a neuron colour. By default, black and white colours are used and brighter tones correspond to the higher values of the membrane potential, so actually coding is done by saturation. A user can select any other colourmap for representing the Vm (t) values. However, the default colours are recommended, since coding by saturation has an advantage over coding by palette as the smaller number of colours provides better readability of visualization. In such a case there is no spurious contrast between the particular objects [7], and the proportion of the colour change to the value change is

‘SNN3DViewer’ - 3D Visualization Tool for Spiking Neural Network Analysis

471

Fig. 1. Typical results of network visualization with SNN3DViewer

noticeable in semantically correct way, unlike e.g. when changing a colour from red to blue (using a discrete palette). Due to the high density of connections observed in many SNN architectures, a graphical representation of the connections is the most demanding problem in terms of clarity of visualization and computational power requirements. There are contradictory goals there: a high frame rate on one hand and a suﬃcient readability on the other hand. In order to avoid a heavy load on a processor, we therefore excluded from our considerations Bezier curves, Hermit curves or B-Splines [3,5] as models for the parametrization of the 3D representation of the interneuronal connections. We also dismissed a straight line representation, since it increases a likelihood of connection crossing or overlay, and hence reduces clarity of visualization. As a solution, a simple spline-based algorithm is used: curves representing the particular connections are divided into several segments, each one described by an n-th order polynomial (usually n = 1, .., 3). Points in space, where the adjacent segments are supposed to link to each other, are sampled within some range, which results in a randomization of a distribution of connections in the 3D space. This algorithm augments the image readability by essentially reducing the likelihood of connection overlays. The particular segments of the interneuronal connections are modelled as cylinders. We use cylinders instead of lines, because in case of a high density neuronal connections shaded cylinders can be much easier distinguished from one another than simple lines.

472

A. Kasi´ nski, J. Pawlowski, and F. Ponulak

(a)

(b)

Fig. 2. (a) Visual representation of interneuronal connections (spline curves) and spikes (small, white cylinders). (b) Strength of a connection is coded by the transparency of its graphical model. With this approach a user can easily determine inputs that have the highest impact on a postsynaptic neuron. In the illustrated example all neurons in the bottom right part of the ﬁgure send converging inputs to the neuron shown in the top left corner. Only several connections, corresponding to the strongest inputs, are clearly visible.

Each connection is characterized by its strength (synaptic weight) and the delay. By default excitatory connections are displayed in bright brown, whereas inhibitory connections are in magenta. Connection strength is visualized by the transparency level of its graphical model - weaker connections are less visible (Fig.2). This approach is useful to visualize the dynamics of synaptic plasticity (learning) in the neural networks. With the strength-to-transparency coding the user can easily determine the impact that the particular synaptic inputs have on the postsynaptic neuron (Fig.2b). It is also easier to observe how this impact evolves over time. In case of a network with high density connections, it can be a futile attempt to display all connections in a clear way. For this reason, the user can optionally turn on/oﬀ the visibility of the particular connections or groups of connections (Fig. 3). Such an option supports analysis of the whole networks, their speciﬁc regions or single neurons. Signal transmitted between spiking neurons is not continuous, but represented rather by short impulses (spikes or action potentials) generated in neurons. Neuronal ﬁring is illustrated by a short increase of a neuron radius. In some applications this (optional) eﬀect improves marking of the ﬁring times in the particular neurons. Immediately after each ﬁring time, a graphical object representing a spike is generated and propagated along the interneuronal connections to the successor neurons. SNN3DViewer enables the users to emulate and visualize this process. Given the network topology, the ﬁring times of the particular neurons as well as a connection length and delay, our application estimates spike velocity and accordingly calculates a spike position along the connection at every time step (Fig.2a). The number of impulses to be presented on a screen at a certain

‘SNN3DViewer’ - 3D Visualization Tool for Spiking Neural Network Analysis

473

time step can be very large, so there is a need to reduce a graphical representation of spikes to the very simple 3D objects. For this reason spikes are visualized as simple white cylinders.

4

Application Performance

As mentioned before, SNN3DViewer has been designed to meet the requirements of the large network visualization, while maintaining appropriate work-speed. Typically our system is able to render up to 100 000 interneuronal connection in the real-time. More connections can be displayed at the expense of the visualization speed. For the large networks it is important to display only these connections that are in the observer camera visibility frustum. In order to check what connections are actually visible a three-step test is performed. First, the network space is divided into cuboids (oct-trees [3,5]) which themselves are tested for the visibility. If a particular cuboid is not visible all contained connections are marked invisible. At a second step, every connection-segment is ”surrounded” by a sphere (not visible, like cuboids) with diameter equal to the segment length. The visibility test on a sphere is very fast [3,5]. If a particular connection ever passes this test, the program performs the third test on a surrounding cylinder (equal to the connection-segment cylinder). The ultimate test is the most accurate one. This three-stage test assures that the graphics card power is used eﬃciently. Visibility tests are performed only whenever there are some modiﬁcations of the orientation or position of the observer camera. Such an event causes the frame-rate to drop. This happens if the user is navigating through the network space. Removing the third stage of the test would signiﬁcantly increase application work-speed in such circumstances. However, we observed that most of the time, which a user spends working with the described application, is used for the observation of a certain network region, while not moving a virtual camera. To achieve the highest possible frame rate, we used a so called ”geometry instancing” technique. Shortly speaking, this method consists in sending the largest possible data portions to the graphical card. Instead of sending data for every connection piece one at a time, these data are sent in batches, prepared (modiﬁed) only whenever a camera position/orientation changes. Because every connection segment is represented by an identical parametrized graphical model, one needs only the transformation matrix [3,5] to geometrically transform the model in a proper way. This matrix is computed and saved in a memory during the generation of a graphical model of a given connection (at the network data loading stage). The advantage of this method arises from a simple remark, that by sending data to the graphics card a rendering initialization procedure is called (”draw call”). This procedure takes (signiﬁcant) time, so sending dozens of instances of a single model at a time saves card’s power for the proper rendering process. This method not only results in the increase of a frame-rate, but also reduces a RAM memory usage. For example, each connection segment is represented as a

474

A. Kasi´ nski, J. Pawlowski, and F. Ponulak

(a)

(b)

Fig. 3. Example of a network visualization with all- or selected connections. (a) Examining the network while all connections are visible could be diﬃcult. (b) Analysing the network part by part is sometimes a better choice.

cylinder composed of 18 vertices. Every vertex contains data about a position (x, y, z - three 32 bit variables) and a normal vector (next 3x32 bits), which is used for lighting and shading calculations by the Direct3D engine. Thus a description of a whole model requires 432 bytes all together. Size of a transformation matrix is 4x4, thus it takes 64 bytes. So storing only transformation matrix instead of all transformed data saves 368 bytes for a single instance. It amounts to 140 MB for 100 000 connections (every connection being split into four pieces). ”Geometry instancing” has some drawback - it requires the graphics card to support interface Vertex Shader 3.0. If a card does not support this interface, the application uses a ”geometry batching”. In this case the program stores transformed vertex data, but not the transformation matrix for connection segments, thus saving on memory is lost, but still data can be sent in batches. In practice this method can be actually faster than the ”geometry instancing” due to the resource constraints of graphics cards which might occur in the latter case (because of the limited number of card registers, which are involved in the ”geometry instancing” process). Due to the strength-to-transparency coding scheme used in our application, the particular connection segments need to be sorted with respect to their distance to the virtual camera (for proper rendering, transparent objects need to be rendered from back to front of the scene). Application speed criterion requires sorting to be fast, even at the expense of its accuracy. We have implemented a quick algorithm, which splits the whole list of records to be sorted into several sublists. The actual sorting is performed only for a single sublist per frame. This assures that whenever the user moves camera, the program starts a new sorting without suspending the whole application for a certain amount of time. The application is designed in such a way that the user does not lose any information related to the dynamics of the visualized network even if the program

‘SNN3DViewer’ - 3D Visualization Tool for Spiking Neural Network Analysis

(a)

475

(b)

Fig. 4. Graphical user interface: (a) All parameters related to the program conﬁguration are easily accessible through the pop-up and list menus. (b) With a navigation window the user can easily identify speciﬁc regions of a whole network to be analysed.

work-speed drops (e.g. due to the high processor load). This is because the visualization speed is tightly related to the frame-rate, so when the latter drops, the number of executed visualization steps per second is reduced accordingly. The application is equipped with a set of tools which additionally support the network analysis. These are e.g. graph windows which display the ﬁring times or membrane potential of the selected neurons. Several graph windows can be presented simultaneously on the screen. The user can easily modify their parameters (position, size, etc). They are half-transparent to maintain the clarity of the visualization. Another important tool implemented in SNN3DViewer is a camera trajectory planner, which supports a design of predeﬁned paths for the camera movement in 3D. This option frees up the user from the burden of manual navigation and allows e.g. to design a camera trajectory matched with particular events that take place in a visualized network. This tool, along with the implemented ’screen movie recorder’, allows to prepare a presentation of the network under study, and to replay it oﬀ-line. In addition SNN3DViewer oﬀers a whole bunch of conﬁguration tools to modify the application settings, to improve the program performance or to adjust the parameters of the visualization individually for a given project (Fig.4).

5

Discussion

An important design criterion for SNN3DViewer was to provide a tool for the neural network analysis by the means of enhanced 3D network representation and a careful visualization control structure (supporting 3D navigation, aspect and scale change). The main purpose of this particular graphical representation is not to maximize the aesthetical impressions, but to help the user to obtain maximum information from the analysis. Therefore, we purposely did not use

476

A. Kasi´ nski, J. Pawlowski, and F. Ponulak

many of the available functions implemented in modern graphics cards (such as anisotropic ﬁltering, etc.), giving the priority to the possible size of the visualized network. Our application assembles several graphical representation and visualization techniques into a consistent, highly integrated, interactive system facilitating analysis of the dynamical processes observed in spiking neural networks. The application not only provides support for the scientiﬁc research, but can also serve demonstration and education purposes – it vividly illustrates the properties of spiking neurons and networks, such as input integration and generation of spikes in the individual neurons, formation and propagation of patterns of activity in neural circuits, modiﬁcation of a network structure and behaviour by the means of synaptic plasticity, etc. Another advantage of our application is that it provides a common graphical representation of networks generated with various SNN-simulators – thus it can form a common platform for the benchmark tests or comparisons for the diﬀerent neurosimulation tools. Future development steps will focus on the multi-platform implementation of SNN3DViewer and on the further enhancement of system performance for large neural networks.

References 1. Brette, R., et al.: Simulation of networks of spiking neurons: a review of tools and strategies. J. Comput. Neurosci. 23, 349–398 (2007) c 2. Crazy Eddie’s GUI System Mk-2, Copyright 2004 - 2005 CEGUI Team & Contributing Authors, http://www.cegui.org.uk 3. Dunn, F., Parberry, I.: 3D Math Primer for Graphics and Game Development. Wordware Publishing (2002) 4. Gerstner, W., Kistler, W.: Spiking Neuron Models. In: Single Neurons, Populations, Plasticity. Cambridge University Press, Cambridge (2002) 5. Lengyel, E.: Mathematics for 3D Game Programming and Computer Graphics. Charles River Media (2004) 6. Maass, W., Bishop, C.: Pulsed Neural Networks. The MIT Press, Cambridge (1999) 7. Ware, C.: Information Visualization. Perception for Design. Morgan Kaufmann Publishers, San Francisco (2004)

Using Graph Transformations in Distributed Adaptive Design System Leszek Kotulski1 and Barbara Strug2 1

Department of Automatics, AGH University of Science and Technology Al. Mickiewicza 30, 30 059 Krakow, Poland [email protected] 2 Department of Physics, Astronomy and Applied Computer Science Jagiellonian University, Reymonta 4, Krakow, Poland [email protected]

Abstract. In this paper a graph transformation using the parallel derivation approach is used to model the process of distribution and adaptation for computer aided design. It is based on earlier research in formal language theory, especially graph grammars, and distributed models. The motivation for the ideas presented here is given and some possible ways of application are described. The application of this idea by the graph distribution toolkit proposed as a multi-agent framework is also considered. The approach is illustrated by an example from the domain of flat layout design. Keywords: computer aided design, adaptive design, graph transformations, distributed grammar systems.

1

Introduction

Distributed model of computing is more and more popular, especially with a rapid development of the Internet and availability of distributed development platforms. Such a model seems to be very useful in the domain of computeraided design. Many design problems can be divided into a number of tasks, each of them working nearly independently and either only occasionally exchanging pieces of information or contributing its capabilities to a common object being designed. Yet there is not much research into proposing a formal model of such a cooperative distributed design so far. This paper deals with a linguistic approach [2] to distributed parallel design. The formal model of computer-aided design is based on graph structures for which a lot of research has been done in context of design [3]. Semantic knowledge about the design is expressed by attributes. Graphs representing design object structures are generated by grammars which consist of graph rules. Usually a complex design problem is divided into a number of subproblems. Therefore we propose a distributed design system consisting of several cooperating graph grammars. Searching for each subproblem solution is supported by one grammar. Solving the whole problem requires the ability of grammars assigned to subproblems to communicate [4,5,6,7,8,9]. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 477–486, 2009. c Springer-Verlag Berlin Heidelberg 2009

478

L. Kotulski and B. Strug

Our ﬁrst attempt to solve the problem of cooperation strategy, where the sequence (in which component grammars are activated) is determined by a control diagram, is presented in [10,11,12,20]. In this paper the system communication is realized by a specially introduced agents and the notion of complementary graphs [13] and conjugated graph grammars [14]. These concepts, supported by the multi-agent GRADIS framework, will be brieﬂy reviewed in section 3 .The presented system of graph grammars generates a set of graphs representing design task solutions compatible with the given design criteria. The proposed approach is illustrated on the example of designing house layout and internal rooms arrangements (in section 4). Final remarks and future investigation directions are presented in conclusions.

2

Design Process

The model of distributed design presented in this paper constitutes a part of a design system which contains also other modules. These modules can be considered to be a context in which the grammar system works. A general framework of a design system is depicted in Fig 1. The process is started by the designer who formulates a set of design requirements on the basis of a design task (for instance after discussion with a customer). These requirements together with legal norms and standards constitute the design criteria for a given project. These criteria are then matched against the existing designs from the design database (the ”experience” part of design process).

DESIGN SOLUTIONS DESIGN CRITERIA

ENVIRONMENT

DATABASE

STARTING PATTERNS

AXIOMS

DESIGNER

DESIGN TASK

GRAMMAR SYSTEM

DESIGN SPACE

GRAMMARS

Fig. 1. A diagram of the design process

If one or more designs, satisfying the designer requirements, are found they are presented to the designer. It is rather rare that the user wants one of the ”oﬀ-the-shelf” designs. But frequently the designer wants something similar to one of the presented samples or a combination of several of them. If a modiﬁcation is desired a description of a selected design is copied from the database.

Using Graph Transformations in Distributed Adaptive Design System

479

This description should allow for easy modiﬁcation. In a model based on formal grammars the description consists of a set of numbers of productions that were originally used to generate the design. Then the process can be ”rewound” as much as needed and then diﬀerent productions or grammars can be used to satisfy user requirements. If there is no design matching user requirements (or database is empty) the process is started from scratch. The starting pattern is selected and the design is generated by a grammar system working in parallel on diﬀerent parts of design. The resulting graph structure is passed to the interpretation module and then visualized. Its description is added to the design database.

3

GRADIS Agents Model

The GRADIS framework (an acronym for GRAph DIStribution toolkit) makes possible a distribution of a centralized graph and controlling of its behavior with the help of concurrent processes. The proposed solution is based on a multiagent approach; an agent is responsible for both a modiﬁcation of the maintained (local) graph, in a way described by the graph transformation rules that are associated with it and a cooperation with other agents in order to maintain the consistency of the graph system. The GRADIS agent model assumes the existence of two types of agents, called: maintainers and workers. The maintainer agent is responsible for maintaining a local graph. The whole set of maintainers deals with the keeping of the system cohesion which is understood here as a behavior equivalent to the graph transformations carried out over the centralized graph. We assume, that at the start of the process one maintainer controls the full, centralized graph, but at the same time it is able to split itself into a set of maintainer agents controlling parts of the previous graph, that is also split accordingly. The cooperation of the maintainers is based on the exchange of information among the elements of the agent’s local graph structure. The graph transformation rules are inherited from the centralized structure. The worker agents are created: temporarily, in order to carry out a given action (eg. to ﬁnd a subpattern), or permanently - to achieve a more complex result (eg. for detail design of an element represented at the lower level of a graph hierarchy). The worker’s graph structure is generated during its creation (by a maintainer agent or another worker agent) and is associated with a part of the parent’s graph structure. This association in not one between some nodes of a graph structure maintained by these agents. The parent-worker association is done at the graph transformation level i.e.some worker’s transformations enforce the application of some graph transformations over the parent’s graph structure Complementary graphs. The data structure which is maintained and transformed by agents, has a form of labeled (attributed) graphs. Let Sv and Se be sets; the elements of Sv are used as node labels and the elements of Se are used as edge labels. The graph structure is deﬁned as follows:

480

L. Kotulski and B. Strug

Definition 1. A (Sv , Se )− graph is a triple (V, D, v − lab) where V is a nonempty set, D is a subset of V × Se × V , and v − lab is a function from V into Sv . For any (Sv , Se ) − graph G, V is set of nodes, D is set of edges and v − lab is a node labeling function. One can extend this graph deﬁnition eg.by introduction attributing functions both for nodes and edges, but these extensions will not inﬂuence the rules of the centralized graph distribution and their transformation, so they will not be considered here. Our aim is to split a given graph G into several parts and to distribute them to diﬀerent locations. Transformation of each subgraph Gi will be controlled by a maintainer agent. To maintain the compatibility of centralized graph with the set of split subgraphs some nodes (called border nodes) should be replicated and placed in the proper subgraph. Subgraphs containing copy of a given node v are complementary to a graph that contains an ”original” of v. A node v is then a border node. Graphically, we will mark a border node by a double circle representation. During the splitting of the graph we are interested in checking if the connection between two nodes crosses a border between subgraphs The algorithms for splitting a common graph and joining partial graphs into the common graph are presented in [13]. For any border node v in the graph Gi we can move boundary in such a way that, all nodes that are connected with v (inside other complementary graphs) are incorporated to Gi as a border nodes and the node v copies are removed from another graphs (i.e. v becomes a normal node). Such an operation is called an incorporation operation. The GRADIS framework associates with each distributed complementary graph the maintainer agent, that not only makes the local derivations possible, but also assures the coordination and synchronization of parallel derivations on diﬀerent complementary graphs. The main idea is to either apply transformation locally or to create the appropriate local environment (with the use of incorporate operation) in which to properly apply transformations locally. The application of a transformation is strongly dependent on the graph transformation mechanism; for the ETPL(k) graph grammars the formal description is presented in [13]. The presented approach does not depend on the speciﬁc properties of the graph transformation mechanism like NLC embedding transformation (in case of the algorithmic approach) [15] or single- and double-pushout (in the case of the algebraic approach) [16]. In [13,17] the cooperation among the agents in the case of these types of graph transformations is considered and a detail algorithms based on these transformation properties are presented. Cohesion graph grammar. The maintainer cooperation is based on complementary graphs. Thus it makes possible distributing a centralized graph, being a solution of some design task, into the distributed agent’s environment. Each agent can maintain a ”reasonable” part of the centralized graph, what makes the parallel computation eﬀective. As it was mentioned, GRADIS framework

Using Graph Transformations in Distributed Adaptive Design System

481

oﬀers one more type of agents called worker agents. The worker agent is created by the maintainer agent temporarily or permanently to support the maintainer creativity. The formal background for the above problem is based on a conjugated graph grammars theory [14,18]. In a conjugated graph a diﬀerent type of nodes is used - it is called a remote node.The fundamental diﬀerence between complementary and conjugate approaches consists in the fact that while in the ﬁrst one the agents create a local environment for correct application of a given production (by incorporating border nodes so there is no synchronization at the production level), in the concjugated approach the local graphs that consist of partially replicated nodes are synchronized with the help of application of the conjugated transformation rules. More precisely remote nodes represent the nodes appearing in other graph structures. This approach requires some provisions in the graph grammar itself to make it possible. In the conjugated graphs grammars we assume that, the graph transformations P, applied to thee ﬁrst agent’s graph structure in which there is a remote node w, exists and this transformation is associated with the another graph transformation Q (applied to the other agent’s graph structure), such that it modiﬁes the neighborhood of the node represented by w. The pair P and Q are called conjugated transformations in context of the remote node w. In other words if v is a remote node and some transformation is applied that inﬂuences v an associated transformation must be applied to the other copies of v. Such a awy of applying transformation may result in a need for some sort of synchronization. So in order to synchronize the set of conjugated graph transformations we assume that GRADIS assures that both P and Q graph transformations will be successfully performed. In [14] it is proved that the fulﬁllment of eight conditions guarantees strictly conjugated model for double-pushout graph grammars.

4

Distributed Adaptive Designs

The model described in previous section seems to be very appropriate for the design process. The example of such design process is presented below. For the clarity of presentation the design patterns are restricted to minimal elements that allows us to illustrate the introduced concept. Thus in our example we will use a ﬂat layout and arrangement design problem. This process is composed of several phases: ﬁrstly we create basic plans (eg. of a building design), then in the next step we can plan rooms arrangement and ﬁnally details of the furniture. Graph transformations seem to be very promising in this application domain [10]. In ﬁg 2a a layout of a ﬂat is presented. A graph representing this layout at the highest level is presented in ﬁg. 2b. This graph contains only nodes representing spaces (labelled P), walls (labelled S) and doors (labelled D). It can be easily noticed that even for such small apartment and a very high level of representation (i.e. without any details of internal elements of a room like chairs,

482

L. Kotulski and B. Strug

D

S S

S

S

D

P

P

S

P

S

S

S P

S

S

D

S

D P S

P

S

S

P S S

D

P

S

S

S

D

S

Fig. 2. a. A layout of a flat , b. a graph representing the layout from

D

D

S

S

S

S P

S

D

S

S

P

S

P

S

S P D

B

D

S

S

T

Fig. 3. Three of 7 complementary graphs generated after splitting a full graph from Fig. 2

tables, wardrobes etc.) this graph is relatively large. Adding to it nodes representing elements of furniture, and then parts of every single piece of furniture would make it rather overwhelming. So the use of approach from the GRADIS framework seems to be a good way to reduce amount of data that has to be taken into account at a given time. We assume that, the building plan (in a graph form) is maintained by a set of maintainer agents. Each of them takes care of a subgraph responsible for a parameterization (that is the allocation of walls, doors, windows, etc.) of one or more rooms. In general case the problem of splitting a large graph into a number of reasonably-sized graphs is not trivial as it is not obvious which nodes should become border ones [19] In case of design problems, however, the splitting is performed across ”natural” borders.That may also not be a simple process but is less time consuming then the generalized one. In house design problem nodes representing walls and doors seem to be good candidates for such natural border nodes. In ﬁg. 3 three subgraphs resulting

Using Graph Transformations in Distributed Adaptive Design System

maintainers

X

flat

T

X

B

L

room

L L

483

L

chair

chair

room room

table

workers

Fig. 4. A graph representing internal structure of a chair and a diagram representing a hierarchy of maintainer and worker agents

from splitting the full graph are depicted. The nodes depicted as double circles represent border nodes. It means that any operation including these nodes can not be carried out locally but must involve cooperation with other agents maintaining subgraphs containing the same border nodes. Let’s imagine that we want to move doors (change attributes associated with a node labeled by D). As D is a border node for any rule to be applied to it we must ﬁrst carry out a sequence of incorporate operations. Each incorporate operation collects the neighbourhood nodes of D in other subgraph and then ”moves” it into the graph in which the rule is to be applied. This operation thus moves the border and, at the same time, makes a node ”D” internal one. Then the required transformation may be applied. Let’s note that this incorporation disturbs the ”natural borders” among the complementary graphs. Fortunately, we can delegate to the agents community the global behaviour of the restoration of ”natural borders”. So after the local transformation has been ﬁnished, the system automatically performs one or more reverse incorporate actions. Moreover with each of the rooms a permanent worker agent is associated. This agent is responsible for the given room arrangement. Let’s note that, in this case, the maintainer-worker relationship is more complex. Moving walls or doors (as described above) has a very strong inﬂuence on the room arrangement process. Putting the chair or a table again the wall with door is possible only when these door will be transferred to another place. Thus the cooperation between graph transformation systems supported by maintainer and worker agents should be very close. In ﬁg.3a the subgraph representing one of rooms is depicted with a bit more details as additional nodes represent a chair. From the point of view of the maintainer agent for this room only two nodes labeled B and T (representing, respectively, the seat and the top of a chair) are important as this agent is responsible for positioning the chair. So for this agent the internal structure of a chair (i.e. number of legs, type of back etc) has no importance. This structure is thus maintained by another agent which is responsible for designing a chair. Such an agent uses its own grammar designed especially for chair design. It may

484

L. Kotulski and B. Strug

introduce diﬀerent types or styles of chairs. But it may also change sizes of chairs by setting or changing appropriate node attributes. This agent can not on the other hand, change the location of the chair. An example of the structure for such an agent is depicted in ﬁg 4. Any change in this structure that does not inﬂuence the remote nodes as it is local and can be performed by any type of grammar designed with this aim in mind. It is important to notice here that,, as mentioned above, a change must not necessarily mean structural one but it may involve a change in attributes value. For example lets an attribute changing transformation makes all legs of a chair wider. Such a change propagates to the nodes representing seat and back as they must accommodate it. This change in turn requires a cooperation between the chair agent and a room agent as a wider chair may, for example, be to wide to ﬁt into its current position. In ﬁg. 4 a general hierarchy of workers and maintainers in a considered example is presented. It should be noticed here that the full graph (representing a ﬂat) actually does not exist (hence it is marked with dashed line) but it can be regenerated by joining all complementary graphs. The level of a furniture design (supported by the worker created for the support of the room arrangement) points out yet another problem. Let’s assume that, we would like to furnish the house in the style Louise XVI and all chairs should be identical. The ﬁrst part of the problem can be solved by preparing the proper graph grammar. The second at the ﬁrst sight demands the nonhierarchical worker’s cooperation. It can be solved by the introduction of worker’s equivalence, i.e. during a worker creation its parent can decide whether to create a new agent’s instance or to use already existing agent (what is marked by twoway arrows in worker-maintainer hierarchy depicted in ﬁg 4). The common agent reacts to the demand of any of its parents, but its reaction is broadcasted to all of them. This solves the problem of some subsystem uniﬁcation. In our example when the one room arranging agent suggests, for example, to make chairs wider, then this modiﬁcation (if accepted by other arranging agents) will be carried over to all chairs.

5

Conclusions

In this paper a general framework for distributing large graph structures into smaller and thus easier to maintain and transform was presented It is worth noticing that this approach does not make any assumptions about the type of graph transformations that are used as rules at diﬀerent levels (by diﬀerent agents). This grammars may nor only be of diﬀerent classes (context-free, regular etc.) but even diﬀerent approaches can be used (algebraic or algorithmical).If only there appropriate rules available for each agent to work their type is not important. The proposed formalism was applied to the domain of distributed design that is in particular need of a good formalism that would allow for both distribution and adaptation within one general framework. The dynamic character of design in which external as well as internal constraints may require fast adaptation of a design can be well described by such a formalism. Moreover the

Using Graph Transformations in Distributed Adaptive Design System

485

problem of the subsystems modiﬁcation is solved. This paper lays a ground for further research in both general methodology and its application to the domain of dynamic design. The use of this method in design context both simpliﬁes it and adds complexity. On one side by, for example, introducing natural borders for splitting and natural hierarchy of tasks for maintainers and workers it makes the problem simpler then in a more general case. On the other hand the domain of design adds its own problems, especially a potential for deep propagation of cooperation (a change of a width of a chair leg requires a change in a chair size, which in turn may cause a chair not to ﬁt in its current position and thus require moving it, which in turn may require the movement of other pieces of furniture etc.) and lead to a cascade of requests. In future we plan to test more intensively the diﬀerent types of cooperation (conditional, transactional) and their inﬂuence on the working of this methodology in real situations.

References 1. Rozenberg, G.: Handbook of Graph Grammars and Computing By Graph Transformation, vol. I, Foundations Ed. World Scientific Publishing Co., NJ (1997) 2. Ehrig, H., Engels, G., Kreowski, H.-J., Rozenberg, G.: Handbook of Graph Grammars and Computing By Graph Transformation: Application, Languages and Tools, Ed., vol. II. World Scientific Publishing Co., NJ (1999) 3. Grabska, E., Palacz, W.: Hierarchical graphs in creative design. MG & V 9(1/2), 115–123 (2000) 4. Csuhaj-Varju, E., Vaszil, G.: On context-free parallel communicating grammar systems: Synchronization, communication, and normal forms. Theoretical Computer Science 255(1-2), 511–538 (2001) 5. Csuhaj-Varju, E., Dassow, J., Kelemen, J., Paun, G.: Grammar systems. A grammatical approach to distribution and cooperation. Topics in Computer Mathematics, vol. 8. Gordon and Breach Science Publishers, Yverdon (1994) 6. Csuhaj-Varju, E.: Grammar systems: A short survey. In: Proceedings of Grammar Systems Week 2004, Budapest, Hungary, July 5-9, 2004, pp. 141–157 (2004) 7. Kelemen, J.: Syntactical models of cooperating/distributed problem solving. Journal of Experimental and Theoretical AI 3(1), 1–10 (1991) 8. Martin-Vide, C., Mitrana, V.: Cooperation in contextual grammars. In: Kelemenov, A. (ed.) MFCS 1998, pp. 289–302 (1998) 9. Simeoni, M., Staniszkis, M.: Cooperating graph grammar systems. In: Paun, G., Salomaa, A. (eds.) Grammatical models of multi-agent systems, pp. 193–217. Gordon and Breach, Amsterdam (1999) 10. Grabska, E., Strug, B.: Applying cooperating distributed graph grammars in computer aided design. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 567–574. Springer, Heidelberg (2006) 11. Grabska, E., Strug, B., Slusarczyk, G.: A Graph Grammar Based Model for Distributed Design. In: International Conference on Artificial Intelligence and Soft Computing, ICAISC (2006) 12. Kotulski, L., Strug, B.: Distributed Adaptive Design with Hierarchical Autonomous Graph Transformation Systems. In: Shi, Y., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2007. LNCS, vol. 4488, pp. 880–887. Springer, Heidelberg (2007)

486

L. Kotulski and B. Strug

13. Kotulski, L.: GRADIS - Multiagent Environment Supporting Distributed Graph Transformations. In: Bubak, M., et al. (eds.) ICCS 2008, Part I. LNCS, vol. 5101, pp. 386–395. Springer, Heidelberg (2008) 14. Kotulski, L., Fryz, L.: Conjugated Graph Grammars as a Mean to Assure Consistency of the System of Conjugated Graphs. Accepted at the Third International Conference on Dependability of Computer Systems DepCoS - RELCOMEX (2008) 15. Engelfriet, J., Rozenberg, G.: Node Replacement Graph Grammars. In: 3-94 in [1] 16. Ehrig, H., Heckel, R., L¨ owe, M., Ribeiro, L., Wagner, A.: Algebraic Approaches to Graph Transformation - Part II: Single Pushout and Comparison with Double Pushout Approach. In: [1], pp. 247–312 17. Kotulski, L.: Distributed Graphs Transformed by Multiagent System. In: Rutkowski, L., et al. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 1234–1242. Springer, Heidelberg (2008) 18. Kotulski, L., Fryz, L.: Assurance of system cohesion during independent creation of UML Diagrams. In: Proceedings at the Second International Conference on Dependability of Computer Systems DepCoS - RELCOMEX 2007, IEEE Computer Society Order Number P2850, Poland, June 14-16 2007, pp. 51–58 (2007) 19. Kotulski, L., Sedziwy, A.: Agent framework for decomposing a graph into the subgraphs of the same size. In: WORLDCOMP 2008 (FCS 2008), Las Vegas (accepted, 2008) 20. Kotulski, L., Strug, B.: Parallel Graph Transformations in Distributed Adaptive Design. In: Workshop on Graph Computation Methods, International Conference on Graph Transformations (GCM-ICGT, Leicester (accepted, 2008)

The Lifting Scheme for Multiresolution Wavelet-Based Transformation of Surface Meshes with Additional Attributes Agnieszka Szcz¸esna The Silesian University of Technology, Institute of Computer Science Akademicka 16, 44-100 Gliwice, Poland [email protected]

Abstract. There are a variety of applications areas that take advantage of the availability of three-dimensional data sets. These objects are represented as complex polygonal surfaces formed by hundreds of thousands of polygons, which causes a significant increase in the cost of storage, transmission and visualisation. Such models are usually not only geometrically complex, but they may also have various surface properties such as colour, textures and temperature, etc. This paper presents a extension of lifting scheme for the multiresolution decomposition and reconstruction of irregular triangle surface meshes with additional attributes. Keywords: lifting scheme, multiresolution, irregular surface mesh.

1

Introduction

Many applications in computer graphics and visualisation take advantage of 3D models. Better and more popular 3D geometry acquisition systems result in million-polygons models. Such models are usually not only geometrically complex, but they may also have various surface properties especially appearance attributes such as color values and textures coordinates. Multiresolution analysis and wavelets have been very popular lately, after processing signals, sounds, images and video, wavelets have been applied for digital geometry processing. This new class of application needs a processing toolbox of fundamental algorithms such as denoising, compression, transmission, enhancement, detection, analysis and editing with suitable mathematical and computational representations. The basic framework for these tools is the multiresolution analysis that decomposes an initial data set into a sequence of approximations and details. This paper presents a natural extension of lifting scheme framework presented in my previous papers [22], [23], [21] to new wavelet-based multiresolution framework for decomposition and reconstruction of irregular triangle surface meshes with additional attributes. This solution is based on complete lifting scheme as a second generation wavelet construction tool. The generalisation from the ﬁrst generation of wavelets to the second one gives an opportunity to construct wavelets on irregular meshes. L. Bolc, J.L. Kulikowski, and K. Wojciechowski (Eds.): ICCVG 2008, LNCS 5337, pp. 487–495, 2009. c Springer-Verlag Berlin Heidelberg 2009

488

2

A. Szcz¸esna

Definition of Surface Mesh

The triangle mesh is a pair M = (P, K) where P is a set of n points pi = [xi , yi , zi ] with 1 ≤ i ≤ 1n and K is a simplicial complex which contains information about topology of a mesh. The complex is a set of three types of subsets called simplices: vertices vi = {i} ∈ V , edges ei,j = {i, j} ∈ E, faces fi,j,k = {i, j, k} ∈ F . Two vertices {a} and {b} are neighbours if {a, b} ∈ E. The 1-ring neighbourhood of vertex {i} is the set N (vi ) = {j|{i, j} ∈ E}. The valence of a vertex vi is the number of edges meeting at this vertex #N (vi ). There are three types of triangular meshes: regular (every vertex has valence equals 6), semi-regular (the most vertices have valence 6, few isolated vertices have any valence), irregular (vertices can have any valence). Many models have surface properties beyond simple geometry. In computer graphics, the most common is RGB color values. Vertices of surface mesh, in addition to its position in space, have additional values which describe other properties. Each vertex will be treated as a extended vector v ∈ Rn . The ﬁrst 3 components of v will be spatial coordinates, and the remaining components will be attributes values.

3

Related Work

This paper concerns multiresolution analysis based on wavelet theory so classic simpliﬁcation algorithms [19], [18], [6], [17] are not described. The ﬁrst connection between wavelets and subdivision (semi-uniform Loop subdivision) allows to deﬁne multiresolution surface representation for semiregular mesh with subdivision connectivity, was presented in [9], [10]. In this case input, irregular mesh must ﬁrst be rebuild with remeshing algorithms [10]. The BLaC-wavelets [11], [12] deﬁne local decomposition which is generalisation of Haar wavelets for non-nested spaces. The connection of hierarchical Delaunay triangulation allows to build multiresolution model of planar or spherical, irregular surface mesh. The next proposition in literature is wavelet-based multiresolution analysis of irregular meshes using a new irregular subdivision scheme [13], [14]. This is expansion of the solution presented in [9], [10]. Another direction in research is the use of non-uniform subdivision schemes to build wavelet analysis. In [8], [7] the central ingredient of multiresolution analysis is a non-uniform relaxation operator which minimises divided diﬀerences. This operator together with the mesh simpliﬁcation method (progressive mesh [17]) and pyramid algorithm (Burt-Adelson pyramid) allows to deﬁne signal processing tools for irregular connected triangle meshes. In [15] progressive meshes and a semi-uniform discrete Laplacian smoothing operator were used to perform multiresolution editing tool for irregular meshes. Lifting scheme is used for multiresolution analysis in [16] where a prediction block is non-uniform relaxation operator based on surface curvature minimisation. This transformation is not a complete lifting scheme because it does not have update block.

The Lifting Scheme for Multiresolution Wavelet-Based Transformation

489

There has been comparatively less work done on processing meshes with additional attributes. There is few extensions of simpliﬁcation algorithms, for example Quadric Error Metrics [20], Progressive Meshes [17], Texture Deviation [24], DePreSS [27] and algorithm with new iso-perceptual error metric [25]. The wavelet transformation of meshes with additional properties are discussed in [26] which is extension of [10]. The multirsolution framework based on not complete lifting scheme presented in [16] allows processing irregular meshes with attributes.

4

General Lifting Scheme for Decomposition and Reconstruction of Irregular Surface Mesh

The most important advantage of the lifting scheme is generality which allows to construct the second generation wavelets on data with non-standard structures [1], [3], [4] while keeping all powerful properties of the ﬁrst generation wavelets (classic wavelets [5]) such as speed and good ability of approximation. This section summarises, developed in [22], [21], [23], proposition of transformation based on the lifting scheme for multiresolution analysis of irregular, triangle closed mesh (2-manifold). Main operations of lifting scheme (Fig. 1) can be singled out:

Fig. 1. The lifting scheme for decomposition of surface mesh

– Select from input mesh M j+1 (where j is the resolution level) the even and odd vertices (split block, S). The odd vertex (vn ) will be removed from mesh by half-edge collapse. The even vertices are 1-ring neighbourhood of odd vertex (N (vn )). The vertex is selected based on the value of criterion which is vertex distance to the regression plane deﬁned by 1-ring neighbourhood of the odd vertex (Sd (vn )). This criterion reﬂected the curvature of surface very well. So the vertices from plain area of mesh are removed ﬁrst. S(M j+1 ) = (vnj+1 , N (vnj+1 )) – Update even vertices (update block, U ). This block smoothes mesh in lower resolution, which inﬂuences on better approximation by this mesh and in consequence smaller wavelet coeﬃcients. The update value is added to every vertex in the neighbourhood of processed vertex in this lifting scheme step.

490

A. Szcz¸esna

The suggested update block is: equal average value of single coordinates (AVV). Odd and even vertices are used in computing so as to avoid allocation memory for additional coeﬃcients in multiresolution model, update block is before prediction block (”update ﬁrst lifting scheme”, [28]). vkj = vkj+1 + U (vnj+1 , N (vnj+1 )) j+1 vk ∈N (vn )

Prediction odd vertex based on even ones (predict block, P ). The proposed prediction blocks take advantage of deterministic spatial interpolation methods such as natural neighbours (NNI) and thin plate splines interpolation based on ﬁrst (TPS1) and second ring neighbourhoods (TPS2). To calculate prediction value the local parameterization of odd vertex and its neighbours should be determined, so the proposed prediction block consists of two parts: parameterization and interpolation. The searched parametric plane is the regression plane (total least squares plane) used also in split block. This is the best-ﬁtting plane to a given set of vertices in 1-ring neighbourhood by minimizing the sum of the squares of the perpendicular distances of the points from the plane. – Compute detail vector (wavelet coeﬃcient, d) as diﬀerence between predicted and original vertex. Detail is stored in the multiresolution model [21]. dj = vnj+1 − P (N (vnj+1 )) – Remove the odd vertex from the mesh by an half-edge collapse. The output is the coarser mesh M j that is the approximation of initial mesh. The inverse lifting scheme consists of following operations (Fig. 2):

Fig. 2. The reverse lifting scheme for reconstruction of surface mesh

– Prediction odd vertex based on even vertices from mesh M j . – Compute original value of odd vertex as sum of detail vector remembered in multiresolution model and prediction vertex. – Update even vertices. – Insertion computed vertex to mesh (merge block, M ). Reconstruction of topological connections between odd vertex and its neighbors (even vertices). The output is the mesh M j+1 .

The Lifting Scheme for Multiresolution Wavelet-Based Transformation

491

We can write the equation of one step of wavelet transform: M j+1 → M j + dj The original mesh (M ) is decomposed into base mash in lowest resolution (M 0 ) and sequence of detail vectors: M = M0 +

m

dj

j=1

This is the resulting multiresolution representation. Application of second generation wavelets allows to build multiresolution processing directly on data from acquisitions systems without preprocessing step. In this step the irregular mesh is rebuilt to regular or semi-regular one.

5

Processing Meshes with Attributes

Vertices with additional attributes are treated as extended vector v ∈ Rn . For example vertex with RGB color values is vector [x, y, z, r, g, b]. The ﬁrst 3 elements are coordinates values and next are color values. Ak (vi ) is the k-th value of vertex vector vi. 5.1

Extension of Split Block

The vertex to remove criterion Sd is extended by additional criterion which minimises changes among next attributes of vertices in lower resolution mesh. From vertices with the smallest value of Sd criterion to remove is chosen this with the smallest value of criterion Sa . This is the minimal diﬀerence between average attribute value of neighbours vertices and attribute value of this vertex: Sa (vi ) =

d

αk |Ak (vi ) − Ak (N (vi ))|

k=4

Where Ak (N (vi )) is average attribute value of neighbours vertices of vertex vi : vj ∈N (vi ) Ak (vj ) Ak (N (vi )) = #N (vi ) The coeﬃcient αi is weigh of attribute i in multiresolution processing of mesh. 1 When all attributes are treated equally we can take αi = |Ai,max −A where i,min | values Ai,max and Ai,min are the minimal and maximal value of attribute i for all vertices of processing mesh. 5.2

Update Block

Additional attributes are treated in the same way as coordinates so this block assures equal average value of single coordinates and separate attributes.

492

5.3

A. Szcz¸esna

Prediction Block

In prediction block three coordinates of vi are treated as value of function in two dimension parameter space (ti , ui ). This is computed by the local parameterization of odd vertex and its neighbours on the regression plane. The prediction vertex is value of spatial interpolation in (t0 , u0 ) which is the central point of its neighbours. In this same way can be computed prediction values of attributes. When the result of interpolation function is outside of range of property value (for example range of RGB color is form 0 to 1) it is clamped to nearest proper value.

6

Results

Presented results concern only meshes with RGB color attributes. The criterion Sd + Sa in split block classiﬁes vertices to remove ﬁrst based on distance to the local regression plane next based on diﬀerences between attributes. Vertices which removing cause the smallest change in curvature of local surface (vertices from plain area of mesh) and in attributes (vertices with this same value of attributes) will be processed at the beginning. On Fig. 3 and Fig. 4 we can see that this criterion will retain vertices on the borderline between colors.

(a) criterion Sd

(c) criterion Sd + Sa

(b) criterion Sd

(d) criterion Sd + Sa

Fig. 3. Comparison meshes after removing of 60% of vertices by the lifting scheme with criterion Sd and extended criterion Sd + Sa

The Lifting Scheme for Multiresolution Wavelet-Based Transformation

(a)

493

(b)

Fig. 4. Mesh after removing 80% of vertices by the lifting scheme with criterion Sd +Sa .

(a) smoothing mesh

(b) enhancement mesh

Fig. 5. Applying of smoothing and enhancement filter to multiresolution representation

7

Summary

This paper presents a extension of lifting scheme for the multiresolution decomposition and reconstruction of irregular triangle surface meshes with additional attributes. These attributes can be color values, texture coordinates, temperature and pressure values, etc. The result of processing is multiresolution representation which supports LOD and ROI questions [21]. The lifting scheme is second generation wavelets construction tool so this representation can be a base of digital geometry processing algorithms, such as smoothing or enhancement details of surface mesh (Fig. 5).

References 1. Sweldens, W.: The lifting scheme: A construction of second generation wavelets. SIAM J. Math. Anal. 29(2), 511–546 (1997) 2. Sweldens, W.: The lifting scheme: A new philosophy in biorthogonal wavelet constructions. Wavelet Applications in Signal and Image Processing III, 68–79 (1995) 3. Jansen, M., Oonincx, P.: Second Generation Wavelets and Applications. Springer, Heidelberg (2005)

494

A. Szcz¸esna

4. Stollnitz, E.J., DeRose, T., Salesin, D.H.: Wavelets for Computer Graphics: Theory and Applications. Morgan Kaufmann, San Francisco (1996) 5. Stang, G., Nguyen, T.: Wavelets and Filter Bank. Wellesley-Cambridge Press (1996) 6. Puppo, E., Scopigno, R.: Simplification, LOD and Multiresolution - Principles and Applications. Eurographics Tutorial Notes (1997) 7. Daubechies, I., Guskov, I., Schr¨ oder, P., Sweldens, W.: Wavelets on Irregular Point Sets. Phil. Trans.: Math., Physical, Engrg. Sci. 357(1760), 2397–2413 (1999) 8. Guskov, I., Sweldens, W., Schr¨ oder, P.: Multiresolution Signal Processing for Meshes. Computer Graphics Proceedings, 325–334 (1999) 9. Lounsbery, J.M.: Multiresolution analysis for surfaces of arbitrary topological type. Ph.D. thesis. Department of Mathematics, University of Washington (1994) 10. Eck, M., DeRose, T., Duchamp, T., Hoppe, H., Lounsbery, M., Stuetzle, W.: Multiresolution Analysis of Arbitrary Meshes. ACM Transactions on Graphics 16(1), 34–73 (1995) 11. Bonneau, G.-P.: Multiresolution analysis on irregular surface meshes. IEEE Transactions on Visualization and Computer Graphics 4(4), 139–149 (1998) 12. Bonneau, G.-P., Gerussi, A.: Hierarchical decomposition of datasets on irregular surface meshes. In: Proceedings CGI 1998, pp. 59–65 (1998) 13. Valette, S., Kim, Y.S., Jung, H.J., Magnin, I., Prost, R.: A multiresolution wavelet scheme for irregularly subdivided 3D triangular mesh. In: IEEE Int. Conf. on Image Processing ICIP, Japan, vol. 1, pp. 171–174 (1999) 14. Valette, S., Prost, R.: Wavelet Based Multiresolution Analysis Of Irregular Surface Meshes. IEEE Transactions on Visualization and Computer Graphics 10(2), 113– 122 (2004) 15. Kobbelt, L., Campagna, S., Vorsatz, J., Seidel, H.-P.: Interactive Multi-Resolution Modeling on Arbitrary Meshes. In: Computer Graphics, Annual Conference Series, vol. 32, pp. 105–114 (1998) 16. Roy, M., Foufou, S., Truchetet, F.: Multiresolution analysis for irregular meshes with appearance attributes. In: Proceedings of International Conference on Computer Vision and Graphics, pp. 80–86 (2004) 17. Hoppe, H.: Progressive meshes. In: Proceedings of ACM SIGGRAPH, pp. 99–108 (1996) 18. Heckbert, P., Garland, M.: Survey of Polygonal Surface Simplification Algorithms. Siggraph Course Notes: Multiresolution Surface Modeling (1997) 19. Luebke, D.: A Developer’s Survey of Polygonal Simplification Algorithms. IEEE Computer Graphics and Applications 21(3), 24–35 (2001) 20. Garland, M., Heckbert, P.: Simplifying Surfaces with Color and Texture using Quadric Error Metrics. IEEE Visualization, 263–270 (1998) 21. Szcz¸esna, A.: The Second Generation Wavelet-based Multiresolution Representation of Irregular Triangle Mesh. In: IX International PhD Workshop OWD, Conference Archives PTETiS, vol. 23, pp. 77–80 (2007) 22. Szcz¸esna, A.: The multiresolution analysis of triangle surface meshes with lifting scheme. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 274–282. Springer, Heidelberg (2007) 23. Szcz¸esna, A.: Designing Lifting Scheme for Second Generation Wavelet-Based Multiresolution Processing of Irregular Surface Meshes. In: Proceedings of IADIS International Conference on Computer Graphics and Visualization (2008) 24. Cohen, J., Olano, M., Manocha, D.: Appearance-Preserving Simplification. In: Proceedings of SIGGRAPH, Computer Graphics, Annual Conference Series, ACM SIGGRAPH, pp. 115–122 (1998)

The Lifting Scheme for Multiresolution Wavelet-Based Transformation

495

25. Rigiroli, P., Campadelli, P., Pedotti, A., Borghese, N.A.: Mesh Refinement with Color Attributes. Computers and Graphics 25(3), 449–461 (2001) 26. Certain, A., Popovic, J., DeRose, T., Duchamp, T., Salesin, D., Stuetzle, W.: Interactive Multiresolution Surface Viewing. In: Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 91–98 (1996) 27. Cignoni, P., Montani, C., Scopigno, R., Rocchini, C.: A general method for preserving attribute values on simplified meshes. In: Proceedings of the Conference on Visualization. IEEE Visualization, pp. 59–66. IEEE Computer Society Press, Los Alamitos (1998) 28. Nunes, M., Knight, M., Nason, G.: Adaptive lifting for non-parametric regression. Statistics and Computing 16(2), 143–159 (2006)

Author Index

Ababsa, Fakhreddine 218 Al-Hamadi, Ayoub 133, 228 Andersson, Henrik 356

Lech, Piotr 43 Lipowski, Jaroslaw Konrad Luchowski, Leszek 100

Berlijn, Sonja M. 368 Borca-Mure¸san, Raluca

Mallem, Malik 218 Marek, Jaroslaw 198 Mazurek, Przemyslaw 24, 302, 451 Michaelis, Bernd 133, 228 Miezianko, Roland 282 Mikolajczak, Pawel 90 M˘ aguran, Florin 345 M¨ unzenmayer, Christian 14

345

Chetverikov, Dmitry 143 Chlebiej, Michal 90, 292 Cyganek, Boguslaw 1 Denkowski, Marcin 90, 292 Didier, Jean-Yves 218 Doma´ nski, Marek 174 Dziurzanski, Piotr 208 Fahlstr¨ om, Anders

368

Giosan, Ion 154 Grajek, Tomasz 174 Gruca, Aleksandra 80 Grudzi´ nski, Jakub 430 Grudzinski, Tomasz 420 Gu, Irene Y.H. 356, 368 Herout, Adam Hradis, Michal

239, 260, 380 239, 380

Iwanowski, Marcin

441

Nawrat, Aleksander 391 Nedevschi, Sergiu 154, 345 Niese, Robert 228 Nowi´ nski, Krzysztof 292 Nurzy` nska, Karolina 111 Okarma, Krzysztof 43, 55 Ostrek, Grzegorz 311 Pathan, Saira Saleem 133, 228 Pawlowski, Juliusz 469 Piotrowski, Jacek 66 Pokrajac, Dragoljub 282 Pola´ nczyk, Maciej 33 Ponulak, Filip 469 Przelaskowski, Artur 311

461 Rachubi´ nski, Mariusz

Jaromin, Marcin 184 J¸edrasiak, Karol 391 Josth, Radovan 380 J´ o´zwiak, Rafal 311 Jur´ anek, Roman 249 Kasi´ nski, Andrzej 400, 469 Kotulski, Leszek 477 Krawczy´ nska, Agnieszka 33 Krupi´ nski, Robert 302 Kubicek, Radek 260 Kuhnert, Klaus-Dieter 410 Kuleschow, Andreas 14 Kwolek, Bogdan 270

322

Schmidt, Adam 400 Sistiaga, Unai 368 Skabek, Krzysztof 123 Sklinda, Katarzyna 311 Skublewska-Paszkowska, Maria Skurowski, Przemyslaw 80 ´ Slot, Krzysztof 33 Smolka, Jakub 70 Spinnler, Klaus 14 Stommel, Martin 410 Strug, Barbara 477 Strzelecki, Michal 33 Szcz¸esna, Agnieszka 487

70

498

Author Index

Ulacha, Grzegorz Vicen, Ra´ ul

208

356

Wojciechowski, Adam

164

Z¸abik, L ukasz 123 Zacher, Andrzej 333 Zak, Pavel 260 Zemcik, Pavel 239, 260, 380 Zendjebil, Imane 218

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part II (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part I (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Mathematical optimization in computer graphics and vision

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part II (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part I (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part III (Lecture ... Vision, Pattern Recognition, and Graphics)

Image processing for computer graphics and vision

Handbook of Pattern Recognition and Computer Vision

Combinatorial Image Analysis: 12th International Workshop, IWCIA 2008, Buffalo, NY, USA, April 7-9, 2008, Proceedings (Lecture Notes in Computer Science ... Vision, Pattern Recognition, and Graphics)

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 14th Iberoamerican Conference on Pattern Recognition, CIARP 2009, ... Vision, Pattern Recognition, and Graphics)

Kernel Methods in Computer Vision (Foundations and Trends in Computer Graphics and Vision)

Object Categorization (Foundations and Trends in Computer Graphics and Vision)

Energy Minimazation Methods in Computer Vision and Pattern Recognition: 8th International Conference, EMMCVPR 2011, St. Petersburg, Russia, July ... Vision, Pattern Recognition, and Graphics)

Energy Minimization Methods in Computer Vision and Pattern Recognition: 7th International Conference, EMMCVPR 2009, Bonn, Germany, August 24-27, 2009, ... Vision, Pattern Recognition, and Graphics)

Motion in Games: Third International Conference, MIG 2010, Utrecht, The Netherlands, November 14-16, 2010, Proceedings (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Pattern Recognition: 31st DAGM Symposium, Jena, Germany, September 9-11, 2009, Proceedings (Lecture Notes in Computer Science Image Processing, Computer Vision, Pattern Recognition, and Graphics)

Pattern Recognition and Image Analysis: Third Iberian Conference, IbPRIA 2007, Girona, Spain, June 6-8, 2007, Proceedings, Part I (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Applied Graph Theory in Computer Vision and Pattern Recognition

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 15th Iberoamerican Congress on Pattern Recognition, CIARP 2010, ... Vision, Pattern Recognition, and Graphics)

Lecture notes on Computer Graphics for Engineers

Computer Vision -- ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: International Conference, ICCVG 2008, Warsaw, Poland, November 10-12, 2008 Revised Papers (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part II (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part I (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Mathematical optimization in computer graphics and vision

Mathematical Optimization in Computer Graphics and Vision

Mathematical Optimization in Computer Graphics and Vision

Mathematical optimization in computer graphics and vision

Mathematical optimization in computer graphics and vision

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part II (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part I (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part III (Lecture ... Vision, Pattern Recognition, and Graphics)

Image processing for computer graphics and vision

Handbook of Pattern Recognition and Computer Vision

Combinatorial Image Analysis: 12th International Workshop, IWCIA 2008, Buffalo, NY, USA, April 7-9, 2008, Proceedings (Lecture Notes in Computer Science ... Vision, Pattern Recognition, and Graphics)

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 14th Iberoamerican Conference on Pattern Recognition, CIARP 2009, ... Vision, Pattern Recognition, and Graphics)

Kernel Methods in Computer Vision (Foundations and Trends in Computer Graphics and Vision)

Object Categorization (Foundations and Trends in Computer Graphics and Vision)

Energy Minimazation Methods in Computer Vision and Pattern Recognition: 8th International Conference, EMMCVPR 2011, St. Petersburg, Russia, July ... Vision, Pattern Recognition, and Graphics)

Energy Minimization Methods in Computer Vision and Pattern Recognition: 7th International Conference, EMMCVPR 2009, Bonn, Germany, August 24-27, 2009, ... Vision, Pattern Recognition, and Graphics)

Motion in Games: Third International Conference, MIG 2010, Utrecht, The Netherlands, November 14-16, 2010, Proceedings (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Pattern Recognition: 31st DAGM Symposium, Jena, Germany, September 9-11, 2009, Proceedings (Lecture Notes in Computer Science Image Processing, Computer Vision, Pattern Recognition, and Graphics)

Pattern Recognition and Image Analysis: Third Iberian Conference, IbPRIA 2007, Girona, Spain, June 6-8, 2007, Proceedings, Part I (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Applied Graph Theory in Computer Vision and Pattern Recognition

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 15th Iberoamerican Congress on Pattern Recognition, CIARP 2010, ... Vision, Pattern Recognition, and Graphics)

Lecture notes on Computer Graphics for Engineers

Computer Vision-Computer Graphics Collaboration Techniques, 3 conf., MIRAGE 2007

Information Theory in Computer Vision and Pattern Recognition

Computer Vision-Computer Graphics Collaboration Techniques - Mirage 2011

Computer Graphics

Computer Vision -- ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II ... Vision, Pattern Recognition, and Graphics)

Computer Graphics

Computer Vision and Graphics: International Conference, ICCVG 2008, Warsaw, Poland, November 10-12, 2008 Revised Papers (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part II (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Computer Vision and Graphics: Second International Conference, ICCVG 2010, Warsaw, Poland, September 20-22, 2010, Proceedings, Part I (Lecture Notes ... Vision, Pattern Recognition, and Graphics)

Mathematical optimization in computer graphics and vision

Mathematical Optimization in Computer Graphics and Vision

Mathematical Optimization in Computer Graphics and Vision

Mathematical optimization in computer graphics and vision

Mathematical optimization in computer graphics and vision

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part II (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part I (Lecture ... Vision, Pattern Recognition, and Graphics)

Computer Vision -- ACCV 2009: 9th Asian Conference on Computer Vision, Xi'an, China, September 23-27, 2009, Revised Selected Papers, Part III (Lecture ... Vision, Pattern Recognition, and Graphics)

Image processing for computer graphics and vision

Handbook of Pattern Recognition and Computer Vision

Combinatorial Image Analysis: 12th International Workshop, IWCIA 2008, Buffalo, NY, USA, April 7-9, 2008, Proceedings (Lecture Notes in Computer Science ... Vision, Pattern Recognition, and Graphics)

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 14th Iberoamerican Conference on Pattern Recognition, CIARP 2009, ... Vision, Pattern Recognition, and Graphics)

Kernel Methods in Computer Vision (Foundations and Trends in Computer Graphics and Vision)

Object Categorization (Foundations and Trends in Computer Graphics and Vision)

Energy Minimazation Methods in Computer Vision and Pattern Recognition: 8th International Conference, EMMCVPR 2011, St. Petersburg, Russia, July ... Vision, Pattern Recognition, and Graphics)

Energy Minimization Methods in Computer Vision and Pattern Recognition: 7th International Conference, EMMCVPR 2009, Bonn, Germany, August 24-27, 2009, ... Vision, Pattern Recognition, and Graphics)

Motion in Games: Third International Conference, MIG 2010, Utrecht, The Netherlands, November 14-16, 2010, Proceedings (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Pattern Recognition: 31st DAGM Symposium, Jena, Germany, September 9-11, 2009, Proceedings (Lecture Notes in Computer Science Image Processing, Computer Vision, Pattern Recognition, and Graphics)

Pattern Recognition and Image Analysis: Third Iberian Conference, IbPRIA 2007, Girona, Spain, June 6-8, 2007, Proceedings, Part I (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics)

Applied Graph Theory in Computer Vision and Pattern Recognition

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 15th Iberoamerican Congress on Pattern Recognition, CIARP 2010, ... Vision, Pattern Recognition, and Graphics)

Lecture notes on Computer Graphics for Engineers

Computer Vision-Computer Graphics Collaboration Techniques, 3 conf., MIRAGE 2007

Information Theory in Computer Vision and Pattern Recognition

Computer Vision-Computer Graphics Collaboration Techniques - Mirage 2011

Computer Graphics

Computer Vision -- ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II ... Vision, Pattern Recognition, and Graphics)

Computer Graphics

Recommend Documents