TIME-VARYING IMAGE PROCESSING AND MOVING OBJECT RECOGNITION, 4
This Page Intentionally Left Blank
TIME-VARYING IMAGE PROCESSING AND MOVING OBJECT RECOGNITION, 4 Proceedings of the 5th InternationalWorkshop Florence, Italy, September 5-6, 1 996 Edited by V. CAPPELLINI Department of Electronic Engineering University of Florence Florence, Italy
1997 ELSEVIER Amsterdam
- Lausanne - New York- Oxford - Shannon - Tokyo
ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25 P.O. Box 211,1000 AE Amsterdam, The Netherlands
ISBN: 0 444 82307 7 91997 Elsevier Science B.V. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box, 521,1000 AM Amsterdam, The Netherlands. Special regulations for readers in the U.S.A. - This publication has been registered with the Copyright Clearance Center Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923. Information can be obtained from the CCC about conditions under which photocopies of parts of this publication may be made in the U.S.A. All other copyright questions, including photocopying outside of the U.S.A., should be referred to the copyright owner, Elsevier Science B.V., unless otherwise specified. No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. pp. 69-76, 184-189, 190-196: Copyright not transferred. This book is printed on acid-free paper. Printed in The Netherlands.
PREFACE The area of Digital Image Processing is of high actual importance in terms of research and applications. Through the interaction and cooperation with the near areas of Pattern Recognition and Artificial Intelligence, the specific area of "Time-Varying Image Processing and Moving Object Recognition" has become of increasing interest. This new area is indeed contributing to impressive advances in several fields, such as communications, radar-sonar systems, remote sensing, biomedicine, moving vehicle tracking-recognition, traffic monitoring and control, automatic inspection and robotics. This book represents the Proceedings of the Fifth International Workshop on Time-Varying Image Processing and Moving Object Recognition, held in Florence, September 5-6, 1996. Extended papers reported here provide an authorative and permanent record of the scientific and technical lectures, presented by selected speakers from 10 nations. Some papers are more theoretical or of review nature, while others contain new implementations and applications. They are conveniently grouped into the following fields: m. B.
C. D. E. F. G. H. I.
Digital Processing Methods and Techniques Pattern Recognition Computer Vision Image Coding and Transmission Remote Sensing Data and Image Processing Digital Processing of Biomedical Images Motion Estimation Tracking and Recognition of Moving Objects Application to Cultural Heritage.
New digital image processing and recognition methods, implementation techniques and advanced applications (television, remote sensing, biomedicine, traffic, inspection, robotics, etc.) are presented. New approaches (i.e. digital filters, source coding, neural networks .... ) for solving 2-D and 3-D problems are described. Many papers are concentrated on the motion estimation and tracking-recognition of moving objects. The increasingly important field of Cultural Heritage is also covered. Overall the book presents - for the above outlined area - the state of the art (theory, implementation, applications) with the next-future trends. This work will be of interest not only to researchers, professors and students in university departments of engineering, communications, computers and automatic control, but also to engineers and managers of industries concerned with computer vision, manufacturing, automation, robotics and quality control. V. Cappellini
vi
WORKSHOP CHAIRMAN
V. CAPPELLINI, University of Florence, Florence, Italy
STEERING COMMITTEE
J.K. A GGARWAL, University of Texas, Austin, U.S.A. M. BELLANGER, Conservatoire National des Arts et Mgtiers, Paris, France J. BIEMOND, University of Delft, The Netherlands M. BRA CALE, University of Naples, Italy A.G. CONSTANTINIDES, Imperial College, London, England T.S. DURRANI, University of Strathclyde, Glasgow, Scotland G. GALATI, H University of Rome, Italy G.H. GRANLUND, University of Link6ping, Sweden T.S. HUANG, University of Illinois at Urbana-Champaign, U.S.A. G. IMMOVILLI, University of Modena, Italy M. KUNT, Ecole Polytechnique Fe'd~rale de Lausanne, Switzerland A.R. MEO, Polytechnic of Turin, Italy S.K. MITRA, University of California, Santa Barbara, U.S.A. F. ROCCA, Polytechnic of Milan, Italy A. ROVERI, University of Rome "La Sapienza", Italy G. L. SICURANZA, University of Trieste, Italy A.N. VENETSANOPOULOS, University of Toronto, Canada G. VERNAZZA, University of Cagliari, Italy
o~
VII
Sponsored by: European Association for Signal Processing (EURASIP) IEEE Central & South Italy Section European Association of Remote Sensing Laboratories (EARSeL) International Center for Signal & Image Processing (ICESP), Florence Centro d'Eccellenza Optronica (CEO) Dipartimento di Ingegneria Elettronica, University of Florence Istituto di Ricerca sulle Onde Elettromagnetiche (IROE) "Nello Carrara" - C.N.R., Florence Fondazione Ugo Bordoni Fondazione IBM ITALIA Fondazione per la Meteorologia Applicata Associazione Italiana di Telerilevamento (AIT) Gruppo Nazionale Telecomunicazioni e Teoria dell 'lnformazione (T. T.I.) - C.N.R. Sezione di Firenze dell'A.E.I. Associazione Italiana di Ingegneria Medica e Biologica (A.I.LM.B.) CESVIT- Agenzia per l' Alta Tecnologia Regione Toscana - Giunta Regionale
Co sponsored by: Alenia Spazio Alinari AXIS Esaote Biomedica Nuova Telespazio OTE SAGO S.M.A - S istemi per la Meteorologia e l'Ambiente Syremont Telecom Italia Telesoft Ente Cassa di Rispaxmio di Pistoia e Pescia
This Page Intentionally Left Blank
ix
CONTENTS
AO
DIGITAL PROCESSING METHODS AND TECHNIQUES
A.1
"On 3-D Space-time Interpolation and Data Compression of Digital Image Sequences Using Low-order 3-D IIR Filters" H.-L.M. CHENG and L.T. BRUTON
A.2
"Flicker Reduction in Old Film Sequences" (Invited) P.M.B. VAN ROOSMALEN, R.L. LAGENDIJK and J. BIEMOND
A.3
"Multichannel Filters in Television Image Processing"
19
K.N. PLATANIOTIS, S. VINAYAGAMOORTHY, D. ANDROUTSOS and A.N. VENETSANOPOULOS
BO
PATTERN RECOGNITION
25
B.1
"Blotch and Scratch Detection in Image Sequences based on Rank Ordered Differences" (Invited)
27
M.J. NADENAU and S.K. MITRA B.2
"Feature Matching by Optimization using Environmental Constraints"
36
A. BRANCA, E. STELLA, G. ATTOLICO and A. DISTANTE B.3
"@stem Identification for Fuzzy Controllers"
42
G. CASTELLANO, G. ATTOLICO, T. D'ORAZIO, E. STELLA and A. DISTANTE
Co
COMPUTER VISION
49
C.1
"Computer Vision for Autonomous Navigation: from Research to Applications"
51
G. GARIBOTTO, P. BASSINO, M. ILIC and S. MASCIANGELO C.2
"An Optimal Estimator of Camera Motion by a Non-Stationary Image Model" G. GIUNTA and U. MASCIA
57
C.3
"A Simple Cue-Based Camera Calibration Method for Digital Production of Moving Images"
63
Y. NAKAZAWA, T. KOMATSU and T. SAITO C.4
"Exploration of the Environment with Optical Sensors Mounted on a Mobile Robot"
69
P. WECKESSER, A. VON ESSEN, G. APPENZELLER and R. DILLMANN
DO
IMAGE CODING AND TRANSMISSION
77
D.1
"Time-Varying Image Processing for 3D Model-Based Video Coding" (Invited)
79
T.S. HUANG, R. LOPEZ and A. COLMENAREZ D.2
"A New Arbitrary Shape DCT for Object-Based Image Coding"
87
M. TANIMOTO and M. SATO D.3
"Picture Coding Using Splines"
93
M. BUSCEMI, R. FENU, D.D. GIUSTO and G. LIGGI D.4
"A 10 kb/s Video Coding Technique Based on Spatial Transformation"
99
S. BONIFACIO, S. MARSI and G.L. SICURANZA D.5
"Image Communications Projects in ACTS" (Invited)
105
F. BIGI D.6
"Conveying Multimedia Services within the MPEG-2 Transport Stream"
115
L. AZTORI, M. DI GREGORIO and D.D. GIUSTO D.7
"A Subband Video Transmission Coding System for ATM Network"
121
M. EYVAZKHANI D.8
"A High Efficiency Coding Method"
127
K. KAMIKURA, H. JOZAWA, H. WATANABE, H. KOTERA and K. SHIMAMURA D.9
"A Sequence Analysis System for Video Databases"
133
M. CECCARELLI, A. HANJALIC and R.L. LAGENDIJK D.10
"Subjective Image Quality Estimation in Subband Coding: Methodology and Human Visual System Application" Z. BOJKOVIC, A. SAMCOVIC and B. RELJIN
139
xi
EO
REMOTE SENSING DATA AND IMAGE PROCESSING
145
E.1
"Neural Networks for Multi-Temporal and Multi-Sensor Data Fusion in Land Cover Classification"
147
A. CHIUDERI E.2
"Influence of Quantization Errors on SST Computation Based on A VHRR Images"
153
P.F. PELLEGRINI, F. LEONCINO, E. PIAZZA and M. DI VAIA E.3
"Study of Ecological Condition Based upon the Remote Sensing Data and GIS"
159
M. ZHANG, J. BOGAERT and I. IMPENS E.4
"PEICRE PROJECT: a Practical Application of Remote Sensing Techniques for Environmental Recover and Preservation"
165
M. BENVENUTI, C. CONESE, C. DI CHIARA and A. DI VECCHIA E.5
"A Wavelet Classification Chain for Rain Pattern Tracking from Meteorological Radar Data"
171
P. GAMBA, A. MARAZZI and A. MECOCCI E.6
"Frequency Locked Loop System for Doppler Centroid Tracking and Automatized Raw Data Correction in Spotlight Real-Time SAR Processors"
176
F. IMPAGNATIELLO and A. TORRE E.7
"Use of Clutter Maps in the High Resolution Radar Surveillance of Airport Surface Movements"
184
G. GALATI, M. FERRI and M. NALDI E.8
"Simulation of Sequences of Radar Images for Airport Surveillance Applications"
190
F. MARTI, M. NALDI and E. PIAZZA E.9
"Data Fusion and Non Linear Processing of E.L.F. Signal for the Detection of Tethered Satellite System" 197 S. MONTEVERDE, R. RUGGERONE, D. TRAVERSO, S. DELLEPIANE and G. TACCONI
Fo F.1
DIGITAL PROCESSING OF BIOMEDICAL IMAGES
203
"A Simple Algorithm for Automatic Alignment of Ocular Fundus Images"
205
L. B ALLERINI, G. COPPINI, G. GIACOMELLI and G. VALLI
xii F.2
"Automatic Vertebrae Recognition throughout a Videofluoroscopic Sequence for Intervertebral Kinematics Study"
213
P. BIFULCO, M. CESARELLI, R. ALLEN, J. MUGGLETON and M. BRACALE F.3
"An Evaluation of the Auditory Cortex Response to Simple Non-Speech Stimuli through Functional MRI"
219
A. PEPINO, E. FORMISANO, F. DI SALLE, C. SAULINO and M. BRACALE
GO
MOTION ESTIMATION
225
G.1
"Temporal Prediction of Video Sequences Using a Region-Based Image Warping Technique" (Invited)
227
N. HERODOTOU and A.N. VENETSANOPOULOS G.2
"High Performance Gesture Recognition Using Probabilistic Neural Networks and Hidden Markov Models"
233
G. RIGOLL, A. KOSMALA and M. SCHUSTER G.3
"Image Segmentation Using Motion Estimation"
238
K. ILLGNER and F. MOLLER G.4
"A Phase Correlation Technique for Estimating Planar Rotations"
244
L. LUCCHESE, G.M. CORTELAZZO and M. RIZZATO G.5
"Tracking by Cooccurrence Matrix"
250
L. FAVALLI, P. GAMBA, A. MARAZZI and A. MECOCCI G.6
"Robust Pose Estimation by Marker Identification in Image Sequences"
256
L. ALPARONE, S. BARONTI, A. BARZANTI, A. CASINI, A. DEL BIMBO and F. LOTTI G.7
"Markov Random Field Image Motion Estimation Using Mean Field Theory"
262
A. CHIMIENTI, R. PICCO and M. VIVALDA G.8
"Moving Object Detection in Image Sequences Using Texture Features"
268
F. MOLLER, M. HOTTER and R. MESTER G.9
"Determining Velocity Vector Fields from Sequential Images Representing a Salt-Water Oscillator" A. NOMURA and H. MIIKE
274
xiii
HO
TRACKING AND RECOGNITION OF MOVING OBJECTS
281
H.1
"'Long-Memory' Matching of Interacting Complex Objects from Real Image Sequences"
283
A. TESEI, A. TESCHIONI, C.S. REGAZZONI and G. VERNAZZA H.2
"Spatial and Temporal Grouping for Obstacle Detection in a Sequence of Road Images"
289
S. DENASI and G. QUAGLIA H.3
"Attitude of a Vehicle Moving on a Structured Road"
295
A. GUIDUCCI and G. QUAGLIA H.4
"An Algorithm for Tracking Pedestrians at Road Crossing"
301
M. LORIA and A. MACHI
1.1
APPLICATION TO CULTURAL HERITAGE
307
"Cultural Heritage: The Example of the Consortium Alinari 2000-SOM"
309
A. DE POLO, E. SESTI and R. FERRARI 1.2
"Color Certification"
313
A. ABRARDO, V. CAPPELLINI, A. MECOCCI and A. PROSPERI 1.3
"Image Retrieval by Contents with Deformable User-Drawn Templates"
319
A. DEL BIMBO and P. PALA 1.4
"Synthesis of Virtual Views of Non-Lambertian Surface through Shading-Driven Interpolation and Stereo-Matched Contours"
325
F. PEDERSINI, A. SARTI and S. TUBARO
AUTHOR INDEX
331
This Page Intentionally Left Blank
A DIGITAL PROCESSING METHODS AND TECHNIQUES
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.)
9 1997 Elsevier Science B.V. All fights reserved.
On 3-D Space-time Interpolation and Data Compression of Digital Image Sequences Using Low-order 3-D IIR Filters H.-L. Margaret Cheng and Leonard T. Bruton Department of Electrical and Computer Engineering, The University of Calgary, Calgary, Alberta, CANADA A bstract--A method is proposed for the data compression and spatio-temporal interpolation of temporally sub-sampled digital image sequences using a first-order 3-D Linear Trajectory (LT) IIR filter.
1. I N T R O D U C T I O N Data compression of image sequences can be achieved by spatio-temporal sub-sampling. In this contribution, we propose a method for recovering a sequence of digital images from the temporally sub-sampled version using a low-order spatio-temporal 3-D IIR (infinite impulse response) filter to perform the required spatio-temporal interpolation. A firstorder 3-D Linear Trajectory (LT) IIR filter [1] is employed for this purpose, followed by a smoothing operation performed in the direction of the motion vector. Experimental results suggest that high compression ratios may be possible. We assume for simplicity that, in each spatio-temporal sub-image sequence, the 3-D spatio-temporal signal contains only one object moving with a constant velocity. This assumption is valid for many practical situations and is the underlying assumption of MPEG-2 and other compression methods.
2. R E V I E W
O F S P A T I O - T E M P O R A L S U B - S A M P L I N G OF I M A G E
SEQUENCES A 3-D LT signal pc(x, y, t), (x, y, t) E ~3 is defined as a continuous-domain space-time signal having a value that is everywhere constant in the direction of the motion vector ~ = (v~'x + vyey-4-vte.t), where ~x, C-u,C~tare the unit basis vectors in the spatial and temporal directions, respectively. The region of support (ROS) of the 3-D Fourier transform of a LT signal is the plane passing through the origin and perpendicular to ~, i.e. w~v~ + wuv u + cotvt 0. The 2-D spectrum on this plane represents the spatial frequency components of the intersection of the 3-D signal with the plane perpendicular to ~ [1]. We assume that this continuous-domain LT signal pc(z, y, t) is 3-D rectangularly sampled at a sufficiently high 3-D sampling frequency that aliasing is negligible. However, temporal sub-sampling of pc(x, y, t) by M introduces aliased replicated 3-D frequency planes (referred to as replica hereafter), at locations w~v~ + wyvy + wtvt = -t-2rrvtj/M, j C [1,..., M-l]. These replica must be completely eliminated by an ideal interpolator. To achieve close-to-ideal interpolation, we employ motion-compensated (MC) interpolation (lower part of Figure 1), where the orientation of the interpolator's passband =
,~, ,~/~ [:?i!~i~!~
(a)
co x
-
(b)
':::~....
:.........
li :: :': :: : :l~i i: :: :IV
(c)
Figure 1: Spectral representation of temporal (upper) and motion-compensated (lower) interpolation, shown for the 2-D case. Dashed lines show aliased replicated signal planes under temporal sub-sampling by M=2. Shaded regions represent passbands of interpolators. The problem of aliasing is shown in (a), and its solution (i.e. pre-filtering) is shown in (b). Interpolation of properly pre-filtered signals is shown in (c). Adapted from [2]. is adapted to that of the spectrum of the sub-sampled signal. In Figure 1 we review the advantage of using this method by comparing it with temporal (upper part of Figure 1) interpolation [2]. For ease of illustration, a 2-D signal that has been temporally sub-sampled by M = 2 is used. Its spectrum is shown in Figure l(a), where the solid line represents the original spectrum of the signal prior to sub-sampling, the dashed lines represent replica introduced by sub-sampling, and the shaded regions represent the passbands of the interpolators. Clearly, the temporal interpolator transmits the undesirable replica and, therefore, fails. To avoid such aliasing in the case of the temporal interpolator, the high-frequency components may be eliminated by separably pre-filtering the signal prior to sub-sampling (Figure l(b)). This seriously attenuates the 3-D planar spectrum of the signal, causing spatio-temporal blurring. However, MC interpolation ideally eliminates the replica and, therefore, does not require pre-filtering. In Figure l(c) we show the two interpolators operating on appropriately pre-filtered and sub-sampled sequences. Aliasing is avoided in both cases. However, because M C interpolation is performed in the direction of the motion vector ~, it does not attenuate the 3-D planar spectrum resolution of the signal and is, therefore, much more effective than simple temporal (or spatial) interpolation. A D E S I G N T E C H N I Q U E TO O B T A I N T H E 3-D LT I I R DISCRETE-DOMAIN FILTER FOR MC INTERPOLATION
3.
To achieve motion-compensated interpolation, we wish to design a stable 3-D IIR discretedomain LT filter having a 3-D passband that is approximately planar, where this passband closely surrounds the planar ROS of the 3-D LT signal. The design process commences with a suitable continuous-domain 3-D frequencyplanar filter [1] having a 3-D Laplace transform transfer function of the form [1]
T(s:,sy, st) = R/[R + s:L: + syLy + stL,l
(1)
The passband of T(s:, sy, st) closely surrounds a 3-D plane [1] passing through the origin and having a normal fi = +(n:~: + Ly~y + Lt~t)/l[LII2. The parameters R, L:, Ly, Lt
Figure 2: Resonant plane of the first-order 3-D LT IIR filter determine the orientation of the passband, and the "thickness" of the passband is determined by its 3-D bandwidth B3 = 2R/llLll~ (Figure 2) [1]. The proposed 3-D discrete-domain interpolating filter is obtained from the above continuous-time prototype by applying the triple s-to-z domain transform [3],
si =
1+
2
ai z i - 1 --,
O
zi nu ai
(2)
The case where az=ay=at=l corresponds to the triple bilinear transform (BLT). This discrete filter has the 3-D Z-transform transfer function [1] 1
1
1
1 --i
1
1
"
H(z~, zy, zt) = ~-~ E E aijkz~ zy' ztk/ E E ~_. bijkz~-izy-j zt k i=0 j = 0 k=0
(3)
i = 0 j = 0 k=0
where the coefficients aijk and bijk are real. The corresponding first-order 3-D recursive equation is [1]
qzyt -
b~176~~~-~akjipxy-'j=o itkj-~~~bkjiqzy-'itkjk=o ]=iO
i=o j=o k=O
(4)
i+j+kr where p~yt and q~yt are the respective discrete input and output sequences. Three advantages of this filter, relative to a non-recursive (i.e. FIR) 3-D filter, are evident. First, computational requirements are low: only 16 multiplies and 14 adds are needed to compute each output pixel. Second, memory requirements are small since as few as one frame store is needed for input or output. Third, the low order allows rapid adaptation to velocity changes. 3.1. C o n s e q u e n c e of the W a r p i n g Effect of the 3-D BLT Application of the B LT causes 3-D warping of the planar passband of the continuousdomain LT filter, such that the passband of the discrete-domain filter is given by Lx tan (-~) + Ly tan (_a~)+ Lt tan (-~) = 0, Jail <_ 7r, i = x,y,t. This 3-D warping can be shown to cause high frequency "speckles" to appear inside and outside of the interpolated object in the corresponding space-time dimensions. However, the transformation (2), with ai < 1, reduces the passband gain in the high frequency regions, where warping is most severe, and in the dimensions in which it is applied. This has two effects. First, signal components from replicated planes that pass
M-1 zero frames
p (x,y,t) temporally up-sampled input sequence
I temporal . I , . II intensity scaling q (x,y,t)l post-processor for temporal intensity scaling
r (x,y,t) spatio-temporally interpolated output sequence
Figure 3: 3-D space-time interpolation scheme through warped regions of the passband are further attenuated. Second, and more importantly, high frequency components of the baseband signal spectrum are reduced, thereby eliminating much of the texture and artifacts appearing both within and without the object. 4. T H E P R O P O S E D 3-D S P A T I O - T E M P O R A L MOTION-COMPENSATED INTERPOLATION
SCHEME
Assuming a highly temporally sub-sampled digital video sequence has been obtained by means of an appropriate frame sub-sampling strategy, we focus here on the problem of reconstructing an approximation to the original video sequence by means of 3-D spatiotemporal interpolation. That is, the interpolator operates on an image sequence that has been sub-sampled temporally by a factor M . A priori knowledge of the corresponding motion vector 0 is assumed (~ may be found using motion estimation techniques [4]). The proposed interpolation scheme consists of conventional temporal up-sampling of the image frames followed by 3-D spatio-temporal filtering to obtain the interpolated values in space-time. The proposed 3-D filtering is performed in two steps (Figure 3). 4.1. O b t a i n i n g the first-level a p p r o x i m a t i o n of the original image sequence We apply the 3-D discrete-domain LT filter to the temporally up-sampled 3-D signal in order to obtain a first-level approximation to the original signal by recovering the missing frames. By orienting the passband of the filter such that 6 = ~, we achieve lowpass filtering in the space-time direction corresponding to ~ . So, the main signal plane in the baseband is retained while the replica introduced by sub-sampling are attenuated. However, due to the low order of the filter, the replica are not sufficiently attenuated. As a result, the intensity of the interpolated output sequence sustains a ripple whose period equals M and whose rate of decay depends on both B3 and 6 (see Figure 4(a)). 4.2. E l i m i n a t i n g intensity variations due to t e m p o r a l ripple A second stage is employed to smooth out the temporal ripple. Either one of two proposed methods may be utilized. The first method involves using an oriented 1-D FIR filter that performs a movingaverage operation in the direction of 0. The order of the filter must equal M to ensure that intensity fluctuations are almost completely eliminated. Here, the difference equation of the 1-D FIR filter is
1M~lq(x -- i(~),y -- i(~t),t -- i) r(x,
,--o
(5)
When the pixel locations x - i(v~:/vt) and y - i ( v y / v t ) are non-integers, the nearest pixel is used. Hence, non-linearities arise. If, however, the quantities v~/vt and vy/vt are integers, we may take the Fourier transform to obtain the 3-D frequency response ___
S (e j(a'~+au+at)) --
M-,~o
~
sin( M--(~O.-~+ ~-flY + fit) 2 \ vt "'~ vt + ~ f l y + f~t)
e-J( 2 ,,, ..x+~ay+a,) x sin( 89
(6)
This is a good approximation even in the case of non-integer pixel locations. In the 3-D Fourier domain, S (e j(a'+a~'+a')) is a 3-0 sinc-like function in the direction of 0 and has equi-gain planes perpendicular to 0. For integer pixel locations, the 3-D planes where the gain is zero correspond exactly to the locations of the temporally replicated signal planes. This is a very effective 3-D lowpass filter for removing the replica introduced by temporal sub-sampling. The second method is to scale each frame of the output of the LT filter by a predetermined corrective intensity-scaling factor. Since intensity variations are only a function of the sub-sampling factor M, the bandwidth B3 of the LT filter, and its orientation fi, we can pre-determine the temporal intensity ripple fluctuations and thereby obtain the normalizing scale factor required for each frame. Although this method does not further attenuate the replicated planes, it is computationally efficient when compared with the first method. k
F
5. E X A M P L E
We present an example to demonstrate the capability of the proposed system to interpolate an up-sampled sequence involving an object that moves at constant velocity and also compare the results obtained using the two methods for removing temporal intensity variations. We use a 3-D digital image sequence in which a 40x40 pixels square, of value 100, moves at a velocity of (0.75, 0.5) pixels per frame. The frame has dimensions 256x256 pixels. A temporal sub-sampling factor M = 20 is used, implying a data reduction of 20. The other parameters are/33 = 0.04 and a factor of ai = 0.9 for i = x, y, t. In Figure 4(a) we show the average intensity of the interpolated square object as a function of image frames. Evidently, both methods remove the temporal intensity fluctuation present at the output of the LT filter, shown by the dotted line. However, differences occur in the spatial characteristics of the output, as seen in Figure 4(c) (d), where we show frame 50 of the interpolated output. Comparison of the FIR and scaling methods shows that the former removes aliased textural artifacts (ripples) due to sub-sampling that the scaling method does not eliminate. However, the difference between the two results is small, and for computational efficiency, scaling is preferred to FIR-filtering. For comparison, Figure 4(b) shows output frame 50 when the B LT is used in conjunction with intensity scaling. Artifacts are evident both within and without the square. However, these artifacts can be mostly removed by using ai < 1. 6. C O N C L U S I O N A 3-D space-time interpolation system is proposed that uses information about the motion of an object to recover the missing frames in a temporally sub-sampled digital image sequence. The first-order 3-D LT IIR filter [1] is proposed for performing MC interpolation.
Readjustment of the output intensity is performed by further filtering or by scaling. It is shown that the system works well for interpolating objects moving at constant velocity and, though not shown here, those undergoing sudden velocity changes. This method is effective for data reductions up to 20, implying the potential for compression ratios much larger than is achieved by the MPEG-2 method.
Figure 4: (a) Comparison of average intensities of output image. (b) Frame 50 of output of LT filter obtained by using BLT. Frame 50 of post-processed output obtained by (c) FIR filtering and (d) intensity scaling for M=20, ai=0.9, i = x, y, t. REFERENCES
[1] L. T. Bruton and N. R. Bartley. The Enhancement and Tracking of Moving Objects in Digital Images Using Adaptive Three-Dimensional Recursive Filters. I E E E Transactions on Circuits and Systems, CAS-33(6), June 1986. [2] M. Ibrahim Sezan and Reginald L. Lagendijk. Motion Analysis and Image Sequence Processing. Kluwer Academic Publishers, Norwell, Massachusetts, USA, 1993. [3] P. Agathoklis, L. T. Bruton, and N. R. Bartley. The Elimination of Spikes in the Magnitude Frequency Response of 2-D Discrete Filters by Increasing the Stability Margin. I E E E Transactions on Circuits and Systems, CAS-32(5):451-458, May 1985. [4] Didier Le Gall. MPEG: a Video Compression Standard for Multimedia Applications. Communications of the ACM, 34(4), April 1991.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
Flicker Reduction in Old F i l m Sequences* P.M.B. van Roosmalen, R.L. Lagendijk and J. Biemond Delft University of Technology, Department of Electrical Engineering, Information Theory Group,, P.O. Box 5031, NL-2600 GA Delft, The Netherlands.
Image flicker, undesirable fluctuations in image intensity not originating from the original scene, is a common artifact in old film sequences. After describing possible causes for image flicker this paper models the effects of flicker as local phenomena. Unfortunately estimation of the model parameters from the degraded sequence is hampered due to presence of noise, dirt and motion. In the latter cases the model parameters can not be estimated directly from local data and are interpolated using the found model parameters of regions nearby. Once the model parameters have been estimated the film sequence can be corrected, taking care that no blocking artifacts occur. The application of this technique in combination with other restoration techniques is discussed.
1. INTRODUCTION Unique records of historic, artistic and cultural developments of every aspect of the 2 0 th century are stored in huge stocks of moving picture archive material. Many of these historically significant items are in a fragile state and are in desperate need of restoration. However, the high cost and lengthy processing time required to restore archive material limit the preservation of these records on a large scale. The aim of the AURORA project (AUtomated Restoration of ORiginal film and video Archives) is the development of technology that significantly reduces the cost and processing time of the restoration processes. Areas of interest within AURORA include Noise Reduction [1], Blotch Detection and Removal [2], Scratch Removal [3], Film Unsteadiness Correction [4], Flicker Reduction, Line Registration Correction [5] and Color Correction. There are several reasons why the artifacts covered by these areas are to be addressed.
* This work is funded by the European Community under the ACTS contract AC072 (AURORA).
10 The first being the explosive growth in number of broadcasters for television, in the near future the home viewer will be able to choose from a hundred or more channels and all of them require programming. The costs for creating new, high quality programs are tremendous. Recycling old programs form a good alternative, if the image (and audio) quality expectations of the modem viewer are met. The second reason for image restoration is that preservation implies storage. The presence of artifacts, and noise in particular, causes compression algorithms to dedicate many bits to irrelevant information. After processing, image sequences of higher quality can be stored using less bits. In this paper we concentrate on the reduction of flicker artifacts. Image flicker is a common artifact in old film sequences. It is defined as unnatural temporal fluctuations in perceived image intensity (globally or locally) not originating from the original scene. Image flicker can have a great number of causes, e.g. aging of film, dust, chemical processing, copying and aliasing (e.g. when transferring film to VCR using a twin lens telecine). To our knowledge very little research has been done on this topic. Neither equalizing the intensity histograms nor equalizing the mean frame values of consecutive frames, as suggested in [6], form general solutions to the problem. These methods do not take changes of scene contents into account and they do not appreciate the fact that flicker can be a spatially localized effect.
2. A MODEL FOR IMAGE FLICKER Due to the lack of detailed knowledge on how the various mechanisms mentioned above cause image flicker, it is difficult to come to models for image flicker based on these mechanisms. Even if these models are known there still is the problem of selecting one of those models for correcting the film sequence. Often only the degraded sequence is available, it is not known what mechanism caused the image flicker. What can be said about flicker is that in any case it causes unnatural changes in image intensity (locally and/or globally) in time. Our approach models image flicker as a local effect independent of the scene contents. We want to limit fluctuations in image intensity in time by locally preserving the intensity mean and the intensity variance. The following model is assumed:
r(x, y, t) = ot(t)(I(x, y, t)+ ~[(x, y, t))+ [J(t) + rl(x, y, t)
~ct(t) = constant for [ [J(t) = constant
x, y e f ~
(I)
where Y(x,y,t) and l(x,y,t) indicate the observed and real image intensities respectively, ix(t) and 13(t) are flicker gain and offset parameters and f~ indicates a small image region and makes that the flicker is modeled as a local effect. In the ideal case (no fading, no flicker) ct(t) = 1 and 13(0 =0. Both flicker dependent noise ~(x,y,t) and flicker independent noise rl(x,y,t) add to the overall amount of noise, which can be estimated, for example, as in [7]. An example of flicker dependent noise is granular noise already on the film before flicker is introduced. Flicker independent noise can be thermal noise due to electronic processing.
11
3. E S T I M A T I O N OF F L I C K E R P A R A M E T E R S Flicker correction requires estimation of the flicker parameters o~(0 and [3(t). The estimates resulting from the initial approach (section 3.1) are optimal for stationary scenes. The estimation of image statistics in non-stationary scenes are usually influenced by the presence of motion. To avoid this one would like to apply some form of motion compensation. Unfortunately the presence of flicker hampers motion estimation as motion estimators usually have a constant luminance constraint, i.e. pel-recursive methods and all motion estimators that make use of block matching in one stage or another. For this reason we choose to merely detect the presence of motion (section 3.2). For regions in which motion was detected the flicker parameters are then interpolated using the flicker parameters of nearby regions not containing motion (section 3.3).
3.1. Flicker parameter estimation in the motion free case For the moment a stationary scene is assumed, let I(x,y,t) = l(x,y). It is also assumed that the distribution of y(x,y,t) does not change in time. This is acceptable under the assumption that the physical quality of the fdrn is constant and, as mentioned before, the scene is stationary. Taking the expected value and the variance of Y(x,y,t) in (I), in a spatial sense, gives for
x,y~
:
E(Y(x, y,t)) = o~(t)E(I(x, y) + y (x, y,t))+ ~(t) + E(rl(x, y,t))
(II)
var(Y(x, y,t)) = var(o~(t)(I(x, y) + y (x, y,t))+ [3(t) + rl(x, y,t)) = Ot2 (t)var(I(x, y)+ y (x, y,t)+ rl(x, y,t))+ (1-Or 2 (t))var(rl(x, y,t))
(III)
When assuming zero mean noise, rewriting these equations give o~(t) and I](t) for x, y e ~ : (IV)
[3(t) = E(Y(x, y,t))-o~(t)E(l(x, y))
Jvar(Y(x, O~(t) = ~
Y,'))
var(l(x,y)+~[(x,y,t)+rl(x,y,t))
(v)
Following [8] it can be shown that these estimates for o~(0 and 13(0 are optimal in the sense that they result in the linear minimal mean squared error between real image intensity and the estimated image intensity. If the variance of the flicker-independent noise is small compared to variance of the observed signal and/or o~(t)= 1, (V) can be approximated by:
I
var(Y(x,y,t))
o~(t) --- var(I(x,y)+y(x,y,t)+rl(x,y,t))
(VI)
12 In order to solve (IV) and (VI) in a practical situation estimates in a temporal sense of expected means and variances at frame t can be used:
E(I(x'Y))t= ET(E(I(x'Y)))= ET(E(Y(x'y't))-ct(t) ~(t)) 1
(VII)
t-1 ~,
E(Y(x,y,n))-fS(n) N - 1n=t-N o~(n) var(I(x, y) +'t (x, y,t) +rl(x, y,t)) = ET (var(I(x, y) +'t (x, y,t) +rl(x, y, t))) = ET
(
var(Y(x,y,t))~ 1 )=N-1 ~2(t')
t-1
var(Y(x, y, n))
~" n=t-N
O~2 (n)
To reduce memory requirements and computational load, first order of (VII) and (VIII) in a practical situation:
(VIII)
IIR filters are used instead
E(l(x, Y))t = rd~(I(x,Y))t-2 + ( l - K ) E(Y(x, y,t-
1))- ~ ( t - 1) a ( t - 1)
(ix)
var(l(x, y)+T(x, y,t)+Tl(X, y,t))=K var(l(x, y)+T(x, y,t-2)+rl(x, y,t-2))+ (1 -K) var(Y(x, y, t - 1)) a2(t-1)
(X)
where K signifies the importance of the previous estimate. Depending on the value for K this method allows the estimates of the original image mean and variance to be adapted to changes in scene lighting (e.g. during a fade or when a light is switched on). Low frequency image flicker is not removed in that case.
3.2. Motion detection in image sequences containing flicker A number of motion detection mechanisms that can be applied to image sequences containing image flicker are described in this section. As these mechanisms rely on detecting changes in image statistics not only motion but also dirt, drop outs and scene changes trigger the motion detectors. Where motion is detected the recursive filters for estimating the mean and variance have to be reset. 3.2.1. Motion detection using the flicker parameters Motion causes local changes in temporal statistics: significant changes in intensity variance and/or mean result in a large deviations from 1.0 for o~(t) and/or from 0 for fS(t)/o~(t), respectively. Regions containing motion can be detected by comparing all ct(t) and [5(t)/ct(Oto threshold values l__.Ta and _+T~. Motion is flagged when either flicker parameter surpasses its threshold value (typical values for Ta and T~ are 0.3 and 20 respectively).
13
3.2.2. Motion detection using frame differences A different method for detecting the presence of motion is the following. For each block in the current frame ~(t) and 13(t) are estimated using (IV) and (VI). The corrected frame is generated using (XI) (see section 4). In the absence of motion the variance of local frame differences between the corrected frame and the previous corrected frame should be twice the total noise variance. Where this is not the case motion is detected.
3.2.3. A hybrid motion detection system The method in section 3.2.2. has the disadvantage that it is very sensitive to film unsteadiness. Slight movements of textured areas may lead to large frame differences and thus to "false" detection of motion. The method in section 3.2.1 is robust against film unsteadiness. The drawback in comparing the flicker parameters ~(t) and ~(t)/et(t) to threshold values is that it is difficult to find good threshold values: false alarms and misses will always occur. Combining the two methods leads to a robust algorithm. First, the motion detection algorithm from section 3.2.1. is applied where T~ and TI~are chosen relatively small leading to relatively many false alarms and few misses. Second, the algorithm from section 3.2.2 is applied to those regions for which motion was detected: the correctness of the found flicker parameters is verified.
3.3. Interpolation of unreliable flicker parameters Where motion is detected the flicker parameters o~(t) and [5(t) computed according to (IV) and (VI) are unreliable. They are to be interpolated using the flicker parameters found in nearby regions. This approach leans on the assumption that the flicker parameters vary slowly (are correlated) in a spatial sense, and, as stated before, are independent of image contents. One pitfall is to be avoided. For uniform regions corrupted by image flicker it is difficult to tell what part of the image flicker is due to variances in gain and what part is due to variances in offset. These regions should not be included in the interpolation process. Moreover, from section 4 it will become clear that that the estimated flicker parameters for these regions should be marked unreliable. In the case of the restoration of old film sequences no problems are to be expected as granular noise is always present (we implicitly assume that granular noise is affected by flicker in a similar manner as the original scene intensities). The iterative interpolation process is as follows. Consider the matrix containing the values of all ~(t) for a certain image. Figure l a shows an example of such a matrix. The gray area indicates the image blocks for which o~(t) are known, the white area indicates the image blocks in which motion was detected. For blocks in the latter region the values ~(0 can be estimated at the boundary of the two regions, by taking the average value of the ~(t) in adjacent blocks in the still region (fig. lb). By repeating this dilation process an estimate for ~(t) can be assigned to each image block in regions where motion was detected (fig. l c,d). The procedure for estimating the unknown 13(t) is similar.
14
Figure 1. (a) gray indicates known parameter values, white indicates the unknown values. (b), (c) and (d) indicate what parameters have been estimated after 1, 2 and 3 steps of the dilation operation.
This method is not optimal in the sense that jumps might occur between the values for or(t) and I](t) in adjacent image blocks near the center of the dilated region (e.g. when the values in the top-left hand side of the still region are very different from the values in the bottom right hand side). This can be resolved by smoothing the found results using, for instance, a Laplacian kernel (see section 4). As the region containing motion becomes larger, more steps are required for the dilation process. This implies more uncertainty about the correctness of the interpolated values. Applying biases towards unity for o~(t) and to zero for 13(t) that grow with each step reduces the probability that flicker is enhanced due to incorrect estimation of the flicker parameters.
4. C O R R E C T I N G
IMAGE FLICKER
Once the flicker parameters have been estimated the sequence can be corrected. But first an extra step is required. As the flicker parameters are computed on a block by block basis, blocking artifacts will be introduced if the found flicker parameters are applied for correction without preprocessing. This preprocessing consists of upsampling the matrices containing the flicker parameters to full image resolution followed by smoothing using a low-pass filter. As mentioned before, when sources other than film are used the contribution to changes in gain and offset to the flicker can not be determined for uniform regions using (IV) and (VI). It is necessary that the flicker parameters in the uniform regions are estimated using the interpolation scheme in section 3.3. If not, smoothing would have the unreliable flicker parameters of these regions influence the reliable flicker parameters of neighboring regions. Now the new flicker free image can be estimated according to: i(x, y, t) = Y(x, y, t ) - ~(x, y, t) ~(x,y,t)
(XI)
15 5. E X P E R I M E N T S
AND RESULTS
Figure 2. Clips of original and corrected frames. In our experiments we used a test sequence of 50 frames containing image flicker and motion (introduced by a man entering the scene through a tunnel). When viewing this sequence it can clearly be seen that the amount of flicker varies locally. Also the presence of granular noise is
16
MEAN: 68.6 STD. DEV.: 2.5
80.0
1600.0
75.0
tu
o z <
uJ
MEAN: 1397.9 STD. DEV: 77.5
1500.0
rr
<
70.0
~
65.0
Z uJ k_Z 1300.0
_z
60.0 0.0
10.0
20.0
FRAME
30.0
40.0
1400.0
1200.0 0.0
50.0
10.0
20.0
(a)
1600.0
75.0
tu
z <
tu
or)
40.0
50.0
(b)
MEAN: 67.2 STD. DEV.: 1.0
80.0
30.0 FRAME
MEAN: 1353.7 STD. DEV.: 52.0
1500.0
E 70.0
~
14oo.o
65.0
_z
130o.o
z ILl I-Z
60.0
0.0
10.0
20.0
30.0 FRAME
(c)
40.0
50.0
1200.0 0.0
10.0
20.0
30.0
40.0
50.0
FRAME
(d)
Figure 3. (a), (b) Mean frame intensities and variances of original sequence. (c), (d) Mean frame intensities and variances of corrected sequence. clearly visible. The signal to noise ratio was estimated to be 21 dB. Equalizing the mean field intensities did not lead to a reduction in image flicker. Figure 2 shows clips of frames 13 and 15, which contain excessive amounts of flicker, before and after correction. Figure 3 shows the field means and variances of the original and the processed sequence. The smoother curves resulting from the processed sequence in figure 3 imply that the amount of image flicker has been reduced. Subjective evaluation confirms this. A (very) small amount of low frequency flicker remained, which can be explained by keeping the last paragraph of section 3.1 in mind. No blocking artifacts are visible and no blurring occurred. No new artifacts were visible.
17
6. DISCUSSION IN
IMAGE STABILIZATION
2D NOISE REDUCTION
v
COMPUTE
FLICKER CORRECTION
OUT Figure 4. Flicker correction as part of an automatic image restoration system
In practical situations the proposed scheme for flicker correction will be applied in combination with other restoration techniques as in many old films combinations of various artifacts are present simultaneously. Two common types of artifacts are noise and image unsteadiness. An example of the place of flicker correction in an automatic restoration system is shown in figure 4. Here the flicker parameters a(t) and 13(t) are estimated from a noise reduced, stabilized sequence. The simultaneous image flicker correction and image stabilization is applied to the original sequence. The output of this system forms the input for subsequent stages of the restoration system where noise, dirt and dropouts are removed making use of motion estimation and motion compensation. The flicker correction scheme can easily be extended to include camera panning, as the panning vectors can be estimated from the image stabilization vectors. Including camera zoom is more troublesome. A major problem is that the characteristics of observed texture changes depending on distance to the camera and on camera parameters such as aperture and focal point. It is difficult to adjust for these. Including scene rotation (perpendicular to the camera) is possible. The first frame of a sequence is chosen as a reference, later frames are compensated for their rotation with respect to the reference frame. Flicker can then be corrected for and the result is rotated back again. Note that aliasing caused by correction for rotation may well influence the results. As the rotation angle becomes larger less of the frames corrected for rotation overlaps with the reference frame. It is then necessary to pick a new reference frame. This can be the current frame, with the disadvantage that the overall brightness of this frame may be noticeably different from the overall brightness of the corrected preceding frame. Another possibility is to choose the corrected preceding frame as a reference (in doing so the loop is closed and the system might become unstable). Fortunately only old film sequences seldom contain zoom and rotation.
18
REFERENCES P.M.B. van Roosmalen, S.J.P. Westen, R.L. Lagendijk and J. Biemond, "Noise Reduction for Image Sequences using an Oriented Pyramid Thresholding Technique", Proceedings of ICIP-96, Vol I, pp. 375-378, Lausanne Switzerland, IEEE 1996. [21 A.C. Kokaram, R.D. Morris, W.J. Fitzgerald and P.J.W. Rayner, "Interpolation of Missing Data in Image Image Sequences", IEEE Transactions on Image Processing, Vol.4 no. 11, pp. 1496-1508, 1995. R.D. Morris, W.J. Fitzgerald and A.C. Kokaram, "A Sampling Based Approach to Line [31 Scratch Removal from Motion Picture Frames", Proceedings of ICIP-96, vol. I, pp. 801-804, Lausanne Switzerland, IEEE 1996. [4] T. Vlachos and G. Thomas, "Motion Estimation for the Correction of Twin-Lens Telecine Flicker", Proceedings of ICIP-96, vol. I, pp. 109-111, Lausanne Switzerland, IEEE 1996. [51 A.C. Kokaram, P.M.B. van Roosmalen, P.J.W. Rayner and J. Biemond, "Line Registration of Jittered Video", submitted to ICASSP 97, Munich 1997. [6] P. Richardson and D. Suter, "Restoration of Historic Film for Digital Compression: A Case Study", Proceedings of ICIP-95, Vol II, pp. 49-52, Washington D.C. USA, IEEE 1995. [7] J.B. Martens, "Adaptive Contrast Enhancement through Residue-Image Processing", Signal Processing 44, pp. 1-18, 1995. [8] K.S. Shanmugan and A.M. Breipohl, "Random Signals", pp. 529-534, J. Wiley & Sons, New York, 1988. [1]
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
19
M u l t i c h a n n e l F i l t e r s in T e l e v i s i o n I m a g e P r o c e s s i n g K.N. Plataniotis, S. Vinayagamoorthy, D. Androutsos, A.N. Venetsanopoulos a aDigital Signal & Image Processing Laboratory, Department of Electrical and Computer Engineering, University of Toronto, M5S 3G4, Toronto, CANADA, E-mail: [email protected] URL: http://www.comm.toronto.edu/dsp/dsp.html A novel multichannel filtering approach is introduced in this paper. The new filter, which is perfectly suitable for real time implementation, can be used to remove impulsive noise and other impairment from color TV signals. The principles behind the new filter are explained in detail. Simulation results indicate that the new filter offers some flexibility and has excellent performance. Due to its inherent parallel structure and high regularity, the new filter can be implemented using array processors on VLSI hardware. With the advent of the all-digital TV system, such filters can lead to systems which would retain accurate image reproduction fidelity despite possible transmission problems. 1. I N T R O D U C T I O N Image filtering refers to the process of noise reduction in an image. As such, it utilizes the spatial properties of the image and is characterized by memory. Filtering is an important part of any image processing system whether the final image is utilized for visual interpretation or for automatic analysis [1]. Filtering of multichannel images has recently received increased attention due to its importance in the processing of color images. It is widely accepted that color conveys information about the objects in a scene and that this information can be used to further refine the performance of an imaging system. Thus, the generation of high quality color images is of great interest. Noise in an image may result from sensor malfunction, electronic interference, or flaws in the data transmission procedure. In considering the signal-to-noise ratio over practical mediums, such as microwave or satellite links, there would be a degradation in quality due to the weak received signal. Degradation of the broadcasting quality can be also a result of processing techniques, such as aperture correction which amplifies both high frequency signals and noise. The appearance of the noise and its effect is related to its characteristics. Noise signals introduced during the transmission process are random in nature resulting in abrupt local changes in the image data. These noise signals cannot be adequately described in terms of the commonly used Gaussian noise models [1]. Rather, they can be characterized as 'impulsive' sequences which occur in the form of short time duration, high energy spikes attaining large am-
20 plitudes with probability higher than the probability predicted by a Gaussian density model. There are various sources that can generate impulsive noise, such as, man made phenomena, such as car ignition systems, industrial machines in the vicinity of the receiver, switching transients in power lines and various unprotected electric switches. In addition, natural phenomena, such as lightning in the atmosphere and ice cracking in the antarctic region, also generate impulsive noise. Impulsive noise is frequently encountered during the transmission of TV signals through UHF, VHF, terestial microwave links and FM satellite links. It is therefore important to develop a digital signal processing technique that can remove such image impairment in real-time and thus, guarantee the quality of service delivered to the consumers. Such a system is proposed here. A new two-stage multidimensional color filter is developed. The color filter is applied on-line on the digitized image frames in order to remove image noise. A number of digital techniques have been applied to the problem aiming to smooth out impulsive noise and restore TV images. In [2], [3] a multi-shell median filter has been introduced. The approach introduced in [3] is applicable only to gray-scale images. Since the TV signal is a color signal, such an approach can be applied only to the luminance component of the transmitted signal without any reference or association to the corresponding chrominance signals. However, there is some indication that noise correlation among the different image channels exists in real color images. Particularly, in the case of NTSC television broadcast signal, if there is any degradation of the chrominance signal that is broadcast, both the I and Q components would be affected simultaneously [4]. Therefore, noise removal operations on only one channel are not adequate and a multichannel filter is necessary to remove the noise and restore the originally transmitted signal. 2. A M U L T I C H A N N E L F I L T E R F O R I M P U L S I V E N O I S E R E D U C T I O N Impulsive noise can be classified as a short duration high energy spike, which results in the alteration of the digital value of the image pixel. After the effect of the noise, the altered value of the image pixel usually differs from the corresponding values of the neighboring pixels. However, in TV signals, any kinds of scenes, pictures or images are transmitted. Thus, it is important for the filter to differentiate between impulsive noise and other image features, such as intended dots or thin lines in the image, which may resemble this kind of noise. For the removal of impulsive noise the class of median filters is considered the most appropriate[l]. However, repeated applications of a median filter in a filtering window centered around a pixel of the image will probably remove the noise but will also reduce the resolution of the image by filtering out thin lines and details. Similarly, using a larger size of filtering window (e.g., 5 x 5 instead of 3 x 3) might result in better noise removal, but will blur the fine details of the image. Thus, to filter out noise and preserve image details a different approach is necessary. A two stage adaptive median filter is introduced. As with any other nonlinear filter, a working area (window or template)is centered around an image pixel [1], [5]. To prevent thin lines and intended spots in the image from being altered through the nonlinear filter-
21 ing process, we applied directional median filters inside the processing window. In other words, instead of a combined median filter applied to the whole window, four different median filters are applied across the four main directions at O ~ , 45 ~ , 90 ~ , 135 ~ (see Fig. 1). Be aware that the pixel at the window center (pixel under consideration) belongs to all four sets. If the pixel under consideration has considerably larger or smaller values than those of the other pixels along a specific direction it will be treated as an outlier and it will be replaced by the median value across this specific direction. Otherwise the value remains unchanged during this operation. Thus, by employing filtering across the main directions, lines and other fine details will be preserved. In a second stage, another median operates on the four filtered results to generate the final output. This directional vector processing median can be considered as an extension of the different multistage medians [6]-[8] to vector processing. The mathematical description of the filter can be summarized as follows: Let y(x): Z l --~ Z TM, represent a multichannel signal and let W E Z l be a window of finite size n • n (square window with filter length n2), where n is generally an odd number. The pixel under consideration xi,j is at the window center. The noisy vectors (n 2 in total) inside the window W are noted as: xi+k,j+t
k
l = O +1 +2 ... '
'
~
'
+(n-l) '
(1)
2
The median filter applied along the 0 ~ direction operates on the horizontal pixels, across and including the center pixel xi,j, noted as (see Fig. 1): xi,j+ l
1 - O, •
+ 2 , - - . , + (n - 1)
2
(2)
For simplification and clarity, let these vectors be h i . . . hn (h stands for horizontal direction). Now, according to standard vector median operation, a scalar distance dp can be defined for vector hp, p = 1 , . . . , n, as: dp = F_,qn~ Ilhp - hqllL ~ where Ilhp - hq[IL ~ is the L1 n o r m or the city block d i s t a n c e between the vectors hp and hq. An ordering of the dp's as d(1) < d(2) _< ... ___ d(n) implies the same ordering to the corresponding hp's: h(1) _< h(2) _< .-- _< h(n), where, h(p) is the pth order statistics [1]. The vector median Yl along the 0 ~ direction is defined as: Yl = h(~). Similarly, the process is repeated for the other three directions. The vectors fp, p = 1 , . . . , n ( f stands for 45 ~ direction) representing those pixels along the 45 ~ direction are (see Fig. 1): xi-k,j+k
k - 0, +1, + 2 , . . . , + (n - 1) 2
(3)
The vector median y2 along the 45 ~ direction is then defined as: Y2 - f(1). For the 90 ~ direction, the corresponding vectors vp, p - 1 , . . . , n (v stands for vertical, i.e. 90 ~ direction) are (see Fig. 1): xi-k,j
k - O, +1, •
• (n - 1) 2
(4)
22 The vector median Y3 along the 90 ~ direction is given as: Y3 - v(1). Finally, the vectors rp, p - 1 , . . . , n (r stands for reverse 45 ~ i.e. 135 ~ direction) representing those pixels along the 135 ~ direction are (see Fig. 1): k = 0, +1 • '
+ (n - 1) '
(5)
2
The vector median Y4 along the 135 ~ direction is thus defined as: y4 = r(1). In the second stage, a vector median filter is applied to the four vector median outputs Yl, y2, y3 and y4 obtained in the directional filtering of the previous stage. Hence, the final output X D V M F of this Directional Vector Median Filter (DVMF) is derived as: XDVMF
=
Y(1)
(6)
where, y(1) is the first order statistic of the ordered sequence of vectors yp, p = 1 , . . . , 4. This new Directional Vector Median Filter (DVMF) is applied to different color images, namely Lenna, Pepper and Lake to assess qualitatively the performance. First, the original images are corrupted with 4% impulsive noise and 50% noise correlation between the Red, Green, and Blue channels using an appropriate noise generator [1]. Then, the DVMF, with a window size of 3 • 3 (small window), is applied to the corrupted images and the filtered output images are displayed and compared visually with the original images. In all three images, no impulsive noise is visible. In addition, all the edge information, thin lines and fine details, are well preserved. 3. R E M O V A L O F M I S S I N G L I N E S An additional motivation to introduce directional filtering, is the problem of missing lines in TV signals in addition to impulsive noise which has been observed to be a common problem in TV signals transmitted over satellite or microwave links. Usually, the signal along a horizontal line of one pixel width will be lost and appears as either a white or black line along the image. In other words, these lines appear like continuous impulsive noise along the horizontal direction. Normally, on a single frame one or more such lines can appear. Since, such lines are horizontal, and most of the time have a width of one pixel, the horizontal direction filtering within the filtering window W is not considered in such cases. Therefore, the DVMF for images having missing line is re-defined as: XDVMF
~-- Y(1),
(7)
where, f(1), v(1) and r(1) are the first order statistics of the ordered vector sequences fp, vv and rp respectively as before, and Y(1) is the first order statistics of the ordered sequence of vectors yp, p = 1 , . . . , 3. Two types of simulations are made to assess the performance of this modified DVMF for removing missing lines. When some missing lines are inserted at random positions in the original image the proposed directional multichannel filter was able to perfectly remove the missing lines and nothing atypical could be visually detected on those locations where those lines were introduced. In order to examine the robustness of the proposed filter, an extreme case was investigated in another simulation experiment by adding 4% impulsive noise to the random missing line. Again, the filter performed well in removing
23 the impulsive noise as well as the missing lines. However, at some positions where missing lines existed, the filter failed to remove the noise completely and few pixels with impulsive noise are visible. This could be attributed to the fact that both missing lines and noisy pixels are contained within the filter window at those locations, and thus more than 50% of the pixel values are outliers. Since the break-down point c* of the median filter is 0.5 [1], the directional median filter failed to remove the noise when the filter window contained more than 50% outliers. Nevertheless, the results seem to be fairly acceptable for viewing (Figs. 3-4). If the impulsive noise percentage is approximately to 2%, an actual figure for most real systems, then the filter performance improved and almost no noise could be detected visually. The proposed methodology can be applied on-line for any of the existing TV systems. Since it is a digital image processing technique, analog-to-digital (A/D) converters are necessary to transform the in-coming analog TV signal to its digital form. After that, a real-time digital signal processor board can be designed to implement the method. Due to its inherent parallel structure and high regularity, the filter has regular computational structure, and can be implemented using array processors on VLSI hardware. Alternatively, a network of dedicated multiple microprocessors can be devised for its implementation. 4. C O N C L U S I O N S A new adaptive filter was introduced in this paper. The new filter perfectly suitable for real time implementation was used to remove impulsive noise and other impairment from color TV signals. Experimental results have been used to illustrate our discussion and to demonstrate the effectiveness of our method. In addition, we have outlined its hardware implementation which makes the proposed solution particularly attractive. With the advent of the all-digital TV system, such filters can lead to systems with accurate image reproduction fidelity despite any unforeseen transmission developments. REFERENCES 1. I. Pitas, A.N. Venetsanopoulos, Nonlinear Digital Filters: Principles and Applications, Kluwer Academic, Norwell Ma., 1990. 2. J. Siu, J. Li, S. Luthi, 'A real-time 2-D median based filter for video signals', IEEE Trans. on Consumer Electronics, vol. 39, no. 2, pp. 115-121, 1993. 3. C.J. Juan, 'Modified 2D median filter for impulse noise suppression in a real-time system', IEEE Trans. on Consumer Electronics, vol. 41, pp. 73-80, 1995. 4. B. Grob, Basic Television and Video Systems McGraw-Hill, N.Y., 1984. 5. A.N. Venetsanopoulos, K.N. Plataniotis, 'Colour Image Software', Proceedings of the European Conference on Circuits and Systems Design, pp. 247-251 , 1995. 6. G.R. Arce, 'Multistage order statistic filters for image sequence' IEEE Trans. on Signal Processing, vol. 39, pp. 1146-1163, 1991. 7. X. Wang, 'Adaptive multistage median filter', IEEE Trans. on Signal Processing, vol. SP-40, pp. 1015-1017, 1992. 8. X. Yang, P.S. Toh, 'Adaptive fuzzy multilevel median filter', IEEE Trans. on Image Processing, vol. 4, pp. 680-683, 1995.
24
135
9(i'
o
/
h
X
X
X
i-l.j-I
-I.j
i.j-I
,.j
i+l.j-I
t+l.j
X
X
X
45
o
i-l.j+l
.,...-
i.j+l
i+l.j+l
/ Figure 1. The New Filter
i X
i,j.! ' V
I ___J
I
HI: I
X i.j
L
Figure 3. Corrupted image (4% impulsive noise and missing line)
I
x ik~'~ 1 VMF
VMF
[,
J
VM[:
L
J x'.j ( = x
=y,h)
Figure 2. Parallel Implementation
Figure 4. Filtered result of (4)
B PATTERN RECOGNITION
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
27
Blotch and Scratch Detection in Image Sequences based on Rank Ordered Differences M. J. Nadenau and S. K. Mitra Department of Electrical and Computer Engineering University of California, Santa Barbara, CA 93106 The film material of old movies is very often degraded by randomly located blotches and scratches. These artifacts are caused by damages to the film surface or by dust, covering small areas of the surface. Digitizing this kind of film material results in images having patches with gray level values that are uncorrelated to the pixel neighborhood. To avoid a distortion of the unaffected parts of the image, first the locations of the blotches and scratches have to be detected before the restoration algorithm can be applied. These locations are especially characterized by a discontinuity in the sequence. In this paper we present an algorithm that works much more effectively and with less computational load than other known algorithms [ 1]. The proposed algorithm is based on rank ordered differences (ROD), which are calculated from the data of the current image flame, and the preceding and the succeeding motion compensated frame. The new algorithm is compared to existing detection algorithms in the form of probability plots and images indicating the correct, false and missing detections. 1. INTRODUCTION Old movies are often valuable historical records, but most of them progressively deteriorate in visual quality during the years, decreasing their usefulness. The main visual defects are dust which gets attached to the film or abrasions of the thin emulsion layer and scratches caused by foreign bodies, which have been in the camera or the projector. Usually two types of distortions can occur. The first one, called a scratch, appears as a thin line of pixels of arbitrary shape with nearly equal gray level values. The second one, called a blotch, is a block or a small coherent area of pixels with very similar gray level values. Individual pixels in a scratch or a blotch are a kind of impulsive noise distortion, but they are specifically characterized by two properties. First, these distortions build a discontinuity, because they appear randomly in the image sequence and the probability that scratches or blotches in two succeeding image frames are located in the same place is very low. Second, these distortions are coherent areas which have almost the same gray level. As a result traditional impulse noise removing algorithms, like median filters, do not work very well in removing these distortions. We proposed a new algorithm that takes advantage of the distortion characteristics mentioned above. To reduce the reconstruction errors at uncorrupted sites we have to separate the process of restoration from the process of detection. After the corrupted locations have been detected in a first stage, in a second stage the missing data is restored by an interpolation process. In this paper we only present an algorithm for efficient scratch or blotch detection. Using both blotch and scratch properties for the detection, we make use of data from the current frame, which allows us to detect the coherence of the blotches, but we also use data from the preceding and succeeding frame to detect a discontinuity of the gray levels in the temporal sequence of the
28 frames. The latter one is only possible in combination with a motion estimation. Otherwise, every motion in the movie would also be detected as a temporal discontinuity confusing the detector. Then the usage of temporal filtering would not represent any improvement in relation to a filter using only the data of the current frame. The motion estimation we used is based on a hierarchical block matching algorithm [2] which is very stable and sufficient for our purposes. It is likely that the application of another, but also more complex, motion estimation algorithm would give slightly better results, but the additional computational load is not justified by the associated improvement. In this paper we present a detection algorithm which is attractive on the one hand by its low computational load and on the other hand by its efficiency, which is better than that of any other comparable algorithm. The detector uses image data of the spatial and temporal pixel neighborhood and calculates some rank ordered differences (ROD). In a second step, three threshold tests are used to indicate wether the examined pixel is corrupted or not. Three different detectors are compared and applied to a sequence of 64 black and white images that have been distorted by artificial blotches. These blotches represent real blotches quiet well and the knowledge about the blotch positions makes it possible to calculate detection error rates. The generation of these blotches is described in [ 1]. The results of the three detectors, that have been applied to a single image frame and to the whole image sequence, are presented here along with error rate plots. Images indicating the correct, false and missing detections, provide a visual verification of the effectiveness of the algorithms. 2. MOTION ESTIMATION In our approach we use a hierarchical block matching motion estimation that is similar to the motion estimator described in [2]. In a movie sometimes an object is covered or uncovered from one frame to the next, but this kind of "natural discontinuity" usually appears only from the preceding to the current frame or from the current to the succeeding frame. To avoid, false detections of these discontinuities as corrupted areas, we make use of all three frames, the current, preceding and succeeding image frames, for detection. First, the algorithm builds image pyramids of each of the three frames. Starting with the original image, the next level is generated by filtering with a Gaussian circularly symmetric filter of a kernel size 7 x 7 and down sampling by factor 2 in both horizontal and vertical directions. This procedure, repeated recursively, results in representations of the original image at different resolutions. In our case we use a four level image pyramid. The original frame is level 0 with a size of 256 x 256, level 1 represents the same image by 128 x 128, and so forth.
Figure 1: Image pyramid
29 Next we begin the estimation of the motion at the highest level of the pyramid. These estimated motion vectors are used as initial vectors on the next lower level. This kind of hierarchical process ensures that all the different magnitudes of motion in a movie can be detected.
Figure 2: Seeking process At every level the same seeking procedure is applied. The image data of the particular level is segmented in rectangular macroblocks. In our approach we use a size of 4 x 4 pixels. Each of these macroblocks is shifted around in a certain seeking area to figure out which motion vector for this macroblock delivers the best match. This is done by thresholding the summed absolute difference (SAD) between the pixels in the current block and those in the block at the shifted position in the previous or succeeding frame. This seeking procedure is started only, when the SAD for the non-shifted macroblock is greater than a certain threshold (here 120). The search space we used is defined bysetting the maximum expected displacement to + 4 pixels. The displacement with the lowest SAD (Emin) is stored as a possible motion vector on this level. This value Emi n is compared to the SAD of "no motion" E 0. If the ratio r=E0/Emi n is greater than a threshold T O(here T0=2.3), the motion vector is accepted at the next level as an initial vector. Otherwise it is assumed as a spurious match [3]. In the latter case the motion vector of this macroblock is generated by an interpolation. It is set to the average of the motion estimations of the three neighbours with the lowest SAD. After estimation at the lowest (256 x 256 ) level, the vector estimations are compared with the difference picture which represents the pixel differences between the motion uncompensated previous image frame and the current one. At those locations, where the difference is less than a certain threshold, the motion vector is set to zero. This reduces the errors that have been accumulated from one level to the next level. Finally, the vector estimation for every pixel of the frame is done by an interpolation of the macroblock motion vectors. 3. DETECTORS 3.1. ROD
- Detector
The rank-ordered difference (ROD) detector presented in this paper is a modified form of the signal-dependent rank ordered mean filter (SD-ROM) used for restoration of an impulse noise corrupted image [4]. While the SD-ROM filter works exclusively in the spatial area of one image frame and is only able to remove one or two pixel wide distortions, the ROD filter is designed to work on image sequences. The ROD filter processes in both spatial and temporal
30 domains; that is, it uses data from the current frame, and from the preceding and succeeding ones. It is able to detect both thin scratches and blotches. The aim of the algorithm is to determine whether a pixel x(k_) in the current frame is corrupted or not. The vector k_ = (x0,Y0,n) describes the position of the pixel being tested. In the first step we define a six element vector p(k_) 9 p (k_) = [p 1 (k_) ,P2 (k_), P3 (k_), P4 (k_),p5 (k_), P6 (k_) ]
= [X(Xo, Y o - l , n - 1
) ,X(Xo, Yo, n - 1 ) , X ( X o , Yo+ 1, n - l ) ,
(1)
X(Xo, Yo - 1, n + 1),X(Xo, Yo, n+ 1),X(Xo, Yo+ 1, n + 1) ] p(k_) contains three pixels of the preceding frame n - 1 and three pixels of the succeeding frame n + 1. The center pixel x(k_) and the six elements of p(k_) build the input data for the filter. Figure 3 clarifies the arrangement of the input pixels. If k_ is pointing to a blotch, usually the value x(k_) is different from the gray values of the elements of p(k_). Depending on the motion, gray value change caused by light conditions in the scene and the gray value of the blotch itself, the difference between x(k_) and the elements of p(k_) varies between very small and large values.
n
P4 P5
C Y
".',"."".. "
succeedingframe
current fra n
'p6 Pl
x
i
preceding frame.. n-1
P2 P3 Figure 3: Pixel arrangement The values above and below k_, in the n-th frame are excluded because in the case of blotches, their values are very similar to the value of x(k_). Therefore including both of these values would only reduce the possibility of detecting an unusual difference of x(k_) compared to the neighborhood pixels. However, similar values for the pixels above and below k_ do not necessarily imply the presence of a blotch, because scratches can be one pixel wide. If k_ points to the border of a blotch, only the value above or below k_ would be similar to x(k_). Taking all these different cases into account, the complexity of the algorithm would be unnecessarily increased. The six pixel values of p(k_) are ordered by rank, which gives the vector r (k_) 9 r(k_) = [r 1 (k_),r 2 (k),r 3 (k),r 4 (k_),r5 (k) ,r 6 (k) ]
(2)
31 containing the values P l ' " P 6 mean m (k_) as: m (k_) =
in a rank ordered sequence. Next we define the rank-ordered
( r 3 (k) + r 4 ( k ) ) / 2
(3)
The r a n k - o r d e r e d d i f f e r e n c e s are defined as d (k_) = [d 1 (k_) ,d 2 (/!:_),d 3 (k) ] , where: r i (k_) - x (k)
x (k_) < m (k_)
X (k_) - r7_ i (k_)
x (k_) > m (k_)
d i (k_) =
V i = 1... 3
(4)
The rank-ordered differences provide the information about the likelihood of corruption of a pixel at location k_. Finally the comparison with preselected thresholds determines if a pixel is corrupted or not. The location k_ is detected as corrupted, if at least one of the following inequalities is true: d i (k_) > T i
i = 1...3
(5)
T 1 , T2 and T3 are preselected threshold values with T 1 < T2 < T3 . The most important threshold value for detection is T 1 9the thresholds T2 and T 3 are necessary, but of secondary importance and can be used to optimize the rate of correct detection by varying both of these values. On the other hand the rate of correct detection depends greatly on the appropriate selection of T 1 . In case of a restoration process for a commercial application, it could be useful to extend the algorithm by an adaptive threshold setting of T 1 , to automatically obtain an optimal detection for every frame. 3.2. SDIa - Detector
The SDIa detector is similar to the spike detection index (SDI) algorithm, presented in [5]. It is the simplest and earliest detector of the three detectors discussed in this paper. Like the ROD, the SDIa detector is also based on an heuristic approach to detect temporal discontinuities in a image sequence. The main idea of the detector is given by the following equation: e b = ( I n (~r) - I n _ 1 (~r + ~ n , n - 1 ( ~ ) ) ) 2
ef = ( I n (~r) - I n + 1 (~r + ~n,n + 1 ( ~ ) ) ) 2
/.
1 DSDIa (]') = I \ 0
(e b > et) /x ( e f > et)
(6)
otherwise
where I n (r~) is the pixel intensity at the location r~ in the n-th image frame and the motion c o m p e n s a t i o n vectors in forward and backward direction are Vn, n - 1 (r), Vn. n + 1 (r) . A n examined location is marked as corrupted, if the forward and backward squared frame difference (ef, eb ) are greater than a certain threshold el. 3.3. MRF
- Detector
The M R F detector is based on a Markov random field (MRF) model. This model is not applied to the image itself, but it is used to model the blotches of an image by creating a blotch detection frame D. This can be considered as an additional virtual frame between two real image frames of the sequence, containing only the blotch and not any real image information. For a possible configuration (D = d) of the complete detection frame D the presence of a
32 blotch at position ~r is indicated by d (r~) = 1 , while d (r~) = 0 represents an uncorrupted location. In [ 1] the following equation for an a posteriori joint distribution for the detection frame D is given P (D = d l I = i) = ~exp -
[a(1-d(r~))
(i(~r)-i(Nc)) 2
_
[31f(d(~r)) + J 3 2 6 ( 1 - d ( r ~ ) ) ]
where 0~, J3~ an J32 are certain parameters, i (~r) is the pixel intensity at the location r~ in the current frame, t(Nc) is the pixel intensity of the motion compensated other real image, the function f ( d (r~) ) gives the number of the four neighbors of d (r~) with the same value as d (~-), 6 ( ) is the delta function and S describes the possible area for r~ , namely the whole image frame. With Eq. (7) the probability for a certain configuration of D can be evaluated. The Gibbs sampler with annealing is used to find the maximum a posteriori (MAP) configuration of the detection frame D, given the data and the model for blotches. First, this technique is aimed to the current frame and preceding frame; next, it is applied to the current frame and succeeding frame. Only at those sites, where both times a discontinuity is estimated, the location is classified as corrupted. The search for the MAP is carried out in an iterative manner. After approximately 5 iterations the algorithm is assumed to have converged. 4. SIMULATIONS To compare the efficiency of the three detectors we use the same black and white image sequence, WESTERN, which has been used in [ 1]. The images have a size of 256 x 256 pixels and contain gray values in the range between 0 and 255. The sequence is artificially corrupted with blotches of random gray values, quite realistic in size and shape. This makes it possible to compute the false alarm and correct detection rate. First, we discuss the detector efficiencies by applying all three methods to the whole sequence of 64 image frames. We then show a typical frame to provide a visual demonstration of the detection algorithm. The motion in the image sequence is estimated from the degraded frames using a four-level estimation process, described above in Section 2. It is a different motion estimation process than used in [ 1], therefore the results for the MRF and SDIa detector might be slightly different. Figure 4 shows a plot of the correct detection rate versus false alarm rate for the ROD, MRF and SDIa detector, applied to the whole sequence. The probability of false alarm and correct detection are defined as:
nfa = Number of false detections
nfa Pfa = "~
nco P co =
nco + nmi
nmi
-
"
Number of missing detections
nco = Number of correct detections
(8)
N = Number of pixels per frame We used 1 < T 1 < 38, T2 = 39, T 3 = 55 as parameters for the ROD detector. For the MRF detector the best results out of the parameter range of 6 < e 1 < 34, 14 < e 2 < 54 have been chosen and the SDIa curve has been generated by measurements for 50 < e I < 2000 in steps of 25.
33
i::~
0.9
M.F .
~
0.
0.6 10 -4
.
.
.
.
.
1 0 -3 Probability
.
.
.
.
: - : .
.
.
.
1 0 -2
of false alarm
Figure 4: Performance of detectors applied to the whole sequence WESTERN
.
1 0 -1
.!
~0.7~ .
!i!i,! :~-r::
!
i
.
0.6l
i
.......
! . i~i i
.
10 -3
~ ! i!ii!i
./
: .... :
.
.
:
.
.
:
::::
..........
.
.
i
i ! i i!!
" ........................
ROD
.
:-
[ :
.
10 -2
Probability of false alarm
10
Figure 5: Performance of detectors applied to frame 49
Obviously the performance of the MRF detector and the SDIa detector are very similar. A slightly better result is obtained with the MRF detector, but the difference is really marginal. The ROD curve shows the fundamental improvement of the new detector versus the other approaches. For a correct detection rate of 80% the ROD detector has about 2.5 times less false detections, which provides a much more feasible basis for the restoration process to be carried out in the second step. The MRF detector, although a very complex algorithm, does not show a better performance versus the SDIa or the ROD approaches. Figure 5 provides a comparison of the detector performances as applied to frame 49 of the sequence W E S T E R N shown in Figure 6. This frame contains more marginally contrasted blotches on the average than all others. In this special case the MRF detector is able to take advantage of its better capability to detect poorly contrasted blotches by its spatial connectivity. In fact, the MRF detector performs perceptually better than the SDIa detector, but the new ROD approach provides a still superior performance. For a correct detection rate of 93%, the false alarm rates are - ROD 0.096% - MRF 0.81% - SDIa 0.95%. That means the performance of the ROD detector is 10 times better than the performance of the SDIa detector. Also the difference between the ROD and MRF detector is almost of one order. To visualize the difference of detection Figure 7 - 9 show the detector results. Green colored pixels indicate correct detections, red colored pixels mark missing detections and brown colored pixels represent false detections. The chosen bias point of the algorithms provide the same correct detection rate, that is, the number of green marked areas should be almost equal. To compare the detector performances, attention should be focused on the number of brown pixels. All three detectors produce false alarms in the area of the white coat lining, because it appears only in frame 49. In the preceding frame and succeeding frame it is covered. For a three frame based approach, this demonstrates the limit of an automatic detection process. In the same way all detectors miss the blotch on the right shoulder of the main person. This blotch is too marginally contrasted for all algorithms. While the ROD detector provides nearly perfect detection with the fewest number of false detections, the other two detectors result sometimes in small coherent areas of false detections.
34 A restoration process applied to these locations definitely causes a degradation of the fine details of the picture. From the implementation point of view, the computational complexity of these algorithms is of greater interest. We define the cost for an addition, subtraction, multiplication or division as 1, the cost for an exp-function as 20, according to the numbers used in [ 1]. The SDIa detector uses only 6 operations, while the MRF approach needs about 60 operations in forward and 60 operations in backward direction. For 5 iterations, this results in about 600 operations per pixel. That is a difference of two orders to the SDIa approach. The ROD detector requires only 24 operations and is thus a very easily implementable algorithm. 5. CONCLUSIONS In this paper we introduced an very efficient blotch and scratch detector of quiet low computational complexity. The proposed detector delivers a very solid basis for the next steps in image restoration. If it will be necessary to increase the efficiency of the ROD detector, even when this causes a higher computational load, the spatial information of the current frame has to be used in a sophisticated way, combined with the algorithm presented in this paper. ACKNOWLEDGEMENT This research is part of an ongoing Alexandria digital library project being carried out at the University of California, Santa Barbara under NSF Grant Number IR194-11330. REFERENCES [ 1] A. C. Kokaram, R. D. Morris, W. J. Fitzgerald and P.J.W. Rayner, "Detection of Missing Data in Image Sequences", IEEE Trans. Image Processing, Vol. 4, No. 11, pp. 1496-1508, Nov 1995 [2] W. Enkelmann, "Investigations of Multigrid Algorithms for the Estimation of Optical Flow Fields in Image Sequences", Computer Vision Graphics and Image Processing, Vol. 43, pp. 150-177, 1988 [3] J. Boyce, "Noise reduction of image sequences using adaptive motion compensated frame averaging", IEEE ICASSP, vol. 3, 1992, pp. 461-464 [4] E. Abreu and S. K. Mitra, "A Signal-Dependent Rank Ordered Mean (SD-ROM) Filter- A New Approach for Removal of Impulse from Highly Corrupted Images", IEEE ICASSP, Detroit, MI, USA, 9-12 May 1995, pp. 2371-4 vol.4 [5] A. C. Kokaram and P. J. Rayner, "A system for the removal of impulsive noise in image sequences", SPIE Visual Communication Image Processing, 1990, pp. 122-133 [6] R. D. Morris, "Image sequence restoration using Gibbs distributions", Ph.D. thesis at University of Cambridge, UK, May 1995
35
Top left illustration - Figure 6: Corrupted frame 49 of sequence WESTERN Top fight illustration - Figure 7: SDIa detector applied to frame 49 Bottom left illustration - Figure 8: MRF detector applied to frame 49 Bottom right illustration - Figure 9: ROD detector applied to frame 49
Time-Varying Image Processing and Moving Object Recognition, 4 - V. Cappellini (Ed.) 36
9 1997 Elsevier Science B.V. All rights reserved.
Feature Matching by Optimization using Environmental Constraints A. Branca, E.Stella, G.Attolico, A. Distante a aIstituto Elaborazione Segnali ed Immagini- C.N.R. Via Amendola, 166/5- 70126 Bari- ITALY Phone (39) 80-5481969 Fax (39) 80-5484311 bran ca ~ie si. ha. cnr.i t Matching is the capability to find correct correspondences among features extracted in two images of the scene acquired from different points of view or after TV camera motion. 3D stereo reconstruction and optical flow estimation are contexts of image understanding where matching has a fundamental rule. We describe a feature-based approach to solve the correspondence problem. Our goal is to correct initial matches, obtained by correlation, minimizing an appropriate energy function using as constraint the invariance of the cross ratio evaluated among coplanar points. 1. I n t r o d u c t i o n
Time-varying images of real-world scenes can provide kinematical, dynamical, and structural information of the world. To estimate the 3D motion and the structure of objects from image sequences, it is often necessary to establish correspondences between images, i.e., to identify in the images the projections corresponding to the same physical part of the sensed scene. The existing techniques for general two-view matching roughly fall into two categories: continuous and discrete. In this work a general method to perform discrete feature matching between images acquired in different times or from two different views is proposed. Generally, discrete matching techniques proposed in literature are implemented by direct methods, using local constraints on features [8], [1], or through optimization methods, using 'global constraints on features [5], [2],[10] to formulate an energy function to be minimized. While the direct methods are fast but more sensitive to noise, the optimization based techniques are more reliable though the drawback to require a burdensome processing. The energy minimization based approaches have been extensively used in literature [6][9] and most of them formulate the energy functional using constraints, determined considering feature characteristics, such as: uniqueness, ordering, disparity continuity. An unexplored field is that to consider projective geometrical invariance constraints in the optimization process computing features correspondences. In this paper an optimization method including cross-ratio invariant constraint of five coplanar points is proposed to solve the correspondence problem. The geometric invariance of cross-ratio of five coplanar
37 points has been used in literature as constraint for optimal matches selection in tracking algorithms, planar region detection or object recognition using probabilistic analysis [3],[11],[4],[12]. The performance of probabilistic approaches depends on the choice of rule for deciding whether five image points have a given cross-ratio [7]. In our method projective invariant constraints are included directly in the optimization process. We propose a feature-based approach to solve the correspondence problem by minimizing an appropriate energy function where constraints on radiometric similarity and projective geometric invariance of coplanar points are defined. The method can be seen as a correlation based approach which take into account the projective invariance of coplanar points in computing the optimal matches. In the following sections the algorithm used for optimal match selection (section 2) and the minimization technique implemented (section 3) to correct all mismatches are described. The experimental results (section 3) show as the approach provides good estimates of visual correspondences.
2. Raw Match C o m p u t a t i o n and Mismatch Selection Our aim is to define a new optimization algorithm for solving the correspondence problem using perspective invariance of cross ratio. Displacement vectors should be estimated only for features of "high" interest (using the algorithm proposed in [8]) which are salient points that can be more easily matched than other points. Initially, raw matches are computed maximizing the radiometric similarity among windows in the first image cenetered on high variance features and second image candidate features. Such matches represent the initial guess will be improved through an optimization process. Our idea is to use the geometric invariance of cross ratio C R ( P ) of five coplanar points P = (Pl, P2, Ps, P4, ps)
CR(P) - sin(a13)* sin(a24)
(1)
(where sin(ao) is the sin of the angle p(p-ff'pj) to verify the goodness of matches estimated through radiometric similarity and at same time to correct all mismatches. This require to satisfy the constraint that for all matches among neighboring points, given five points Pi~kl~ = {Pi,Pj,Pk,Pt,P~} in the first image and the corresponding points in the second image Qokl,.,, = {qi, qj, qk, qz, q,.,,}, the cross ratio computed for each subset must be the same. Previous works proposed in literature use the value of cross ratio computed on a group of five points to verify their coplanarity or match correctness. Evaluation performed on a single group must involve the use of thresholds, actually a small error in locating points can determine large variations in cross ratio values. These problems can be overcome if many combinations of five image points are considered. A mismatch or a point not coplanar with its neighborhood will be easily identified by considering the cross ratio similarity computed on all groups containing it.
38 3. Mismatch Correction through Optimization Cross ratio similarity computed on a large number of groups can be used as constraints to correct all mismatches generated using radiometric similarity. We propose to solve the correspondence problem by imposing that the sum of all differences between the cross ratio computed for each considered subset of five features of the first image and the cross ratio computed for the matched points in the second image, must be minimized. The energy function to be minimized to solve the correspondence problem is: E =
x"lvsaa ~,=1 IICR(P,)
-
IVFeattl C R ( Q , ) I I ~ + V, z_.,=~ ,~ _ R4)
(2)
where P,~ and Q,, are the n t h subsets of five points from the first and second image respectively, and the term P~ imposes that corresponding features in the first and the second image must have a radiometric similarity. The norm E will be minimized only when its partial derivatives with respect to all points qi in the second image equal zero. Satisfying this condition for each of the qi then a system on N F e a t simultaneous equation in N F e a t unknowns is generated. Since the problem is nonlinear, we use an iterative approach to compute the optimal solution. Each match is update iteratively by an amount given by the partial derivative of energy function with respect the same point scaled by a parameter fl determined at each iteration using the method of conjugate gradient:
Vi, q,+=Z ~ -~qi k--l,qh~qi
(3)
Since the partial derivatives of cross ratio estimated for a subset Qn with respect to a point qi G Qn depends to all points qk, k = 1...5 of Qn, the update should depend to radiometric similarity of other points {qk C Qn, qk ~ qi; k = 1...4}. Correct matches (with high radiometric similarity) will influence positively the update, on the other hand, mismatches (with a low radiometric similarity) will avoid any update depending of them. Starting from some approximate matches, the algorithm improve the solution until a predetermined convergence criterion is satisfied. Due to the nonlinearity of the system more than one solution can exist. The success to reach a global minimum, and not be trapped in local minimum, depends on having a good first-guess for the solution. The goal is to reject the noise introduced from correlation measurements. The approach we propose converges through iteration upon the desired correspondence points {qi} by implementing gradient descent along the E(q~) surface, which expresses the quadratic cost function's dependency on all of the {qi} points. The correct matches are not changed because the computed adapting signal control is zero due to the satisfaction of the geometrical constraint. On the other hand, mismatches are influenced by correct matches determining the noise rejection. When a stable state is reached, the energy function value in each subset will provide useful information to identify coplanar features.
39 4. Experimental Results The experimental results reported in this paper have been obtained from tests performed on time varying image sequences. The tests have been performed by considering pairs of image relative to a same static scene and acquired in different times while the TV camera is moving forward along its optical axis (the resulting optical flow must have a radial topology). Once highest interest features are extracted in the first image of a sequence, the raw matches, computed imposing the radiometric similarity, have been considered to define a large number of five point groups between neighboring features, in order to select all mismatches or to correct all mismatches through the optimization process. In the reported results we can observe as the performances of the approach are independent of the planarity of the scene. Actually, to satisfy the cross-ratio constraint it is sufficient that near features are coplanar, but it is not necessary that all extracted features to be coplanar. We can compare the results obtained from a sequence of coplanar image features in fig.(1) and those obtained from the image sequence in fig.(2) where features are extracted on different planes: the algorithm recovers the correct matches in both sequences. Finally, the ability of the system to select all correct matches from the raw measurements obtained through correlation, without apply the optimization process is shown in fig.(3). 5. Conclusions In this paper, we have proposed a new approach to solve the correspondence problem between sparse features of images acquired at different times or from two different point of view. The approach is based on cross-ratio similarity between five coplanar points. Cross ratio invariance constraint computed on a large number of combinations of five image points can provide an useful mean to identify mismatches generated by radiometric measurements and at same time to correct all mismatches through an optimization process. REFERENCES 1. N.Ayache, "Artificial Vision for Mobile Robots",MIT Press, 1991 2. N.Ayache and B.Faverjon, "Efficient Registration of Stereo Images by Matching Graph Descriptions of Edges Segments" The Int. Journal of Comp. Vision 1(2):107-131, April 1987. 3. H.Chabbi, M.O.Berger "Using Projective Geometry to Recover Planar Surfaces in Stereovision" Pattern Recognition Vol.29, No.4, pp.533-548, 1996. 4. S.Carlsson "Projectively Invariant Decomposition and recognition of Planar Shapes" International Journal of Computer Vision Vol 17 No 2 pp 193-209 1996. 5. Y.Ohta and T.Kanade, "Stereo by Intr- and Inter-Scanline Search. IEEE Trans. on Pat. Anal, and Mach. Intell.,7,No.2:139-154,1985. 6. J.J.Lee, J.C.Shim,Y.H.Ha "Stereo correspondence using the Hopfield Neural Network of a new energy function", Pattern Recognition, Vol.27,No.ll,1994.
40
Figure 1. (a)Start image of the sequence with extracted feature over-imposed. (b)Optical flow estimated after matching with correlation. (c)Optical flow estimated after matching with our optimization technique.
Figure 2. (a)Start image of the sequence with extracted feature over-imposed. (b)Optical flow estimated after matching with correlation. (c)Optical flow estimated after matching with our optimization technique. 7. S.J.Maybank "Probabilistic Analysis of the Application of Cross Ratio to Model Based Vision" International Journal of Computer Vision Vol 16 pp 5-33 (1995). 8. H.P.Moravec "The Stanford Cart and the CMU Rover",Proc. IEEE,1983. 9. J.P.Pascual Starink, E. Backer "Finding point correspondences using simulated annealing", Pattern Recognition, Vol.28,No.2,1995. 10. L.Robert and O.D.Faugeras "Curve-based Stereo: Figural Continuity and Curvature" In CVPRgl, 57-62. 11. D.Sinclair, A.Blake "Qualitative Planar Region Detection" International Journal of Computer Vision Vol 18 No 1 pp 77-91 (1996). 12. C.A.RothweU,A.Zisserman,D.A.Forsyth,J.L.Mundy"Planar Object Recognition using Projective Shape Representation" International Journal of Computer Vision Vol 16 pp 57-99 1995.
41
Figure 3. (a)Start image of the sequence with extracted feature over-imposed. (b)Second image of the sequence with features computed through correlation over-imposed. (c)Second image of the sequence with features corrected through optimization overimposed. (d)Optical flow estimated after matching with correlation. (e)Matches selected from the flow in (d). technique.
42
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
System identification for fuzzy controllers G. Castellano, G. Attolico, T. D'Orazio, E. Stella, A. Distante " Istituto Elaborazione Segnali ed Immagini- C.N.R. Via Amendola, 166/5 - 70126 Bari- ITALY at t olico @iesi.ba. cnr.it Several robotic applications can be accomplished through a direct mapping between perceptual situations and control commands. Fuzzy logic is a useful tool for realizing such a mapping. It allows either explicit programming or automatic learning of control rules from suitable training data. A fuzzy control system for wall-following has been developed studying the problem of automatic extraction of rules from training data. Using a machine learning technique we build a compact rule base by estimating the relevance of each input signal in the control decisions. The derived fuzzy rules have successfully driven a TRC Labmate inside an indoor environment. 1. I N T R O D U C T I O N Some tasks involved in autonomous mobile vehicle navigation can be solved efficiently without using plan-based methods, that need some internal representations to be built and updated. Obstacle detection and avoidance, wall-following, door-crossing are examples of low-level strategies that can be realized by a direct mapping between the sensory input and the output control spaces, thus avoiding the delay introduced by the update of an environment model and enabling the use of simple and cheap sensors, as the ultrasonic ring used in our experiments. To describe mathematically the mapping between perceptual situations and control commands may be not easy or desirable. Therefore the use of techniques for designing good approximation of the desired behaviors is preferred. Both neural networks ([1], [2]) and fuzzy controllers ([3], [4]) have proved to give good results as function approximators in robot navigation applications. The learning process for a neural network is generally slow and the final knowledge is represented in a way difficult to evaluate, integrate and refine. The linguistic representation of fuzzy rules is easily understandable instead, allowing validation and correction at any time using information provided by human experts. The learning process for a fuzzy controller is quicker and scales gracefully with the size of the problem. The design of fuzzy systems requires the choice of input and output data, the definition of linguistic values (with the associated membership functions) for each fuzzy variable and finally the derivation of rules from the available knowledge (human experts and real data). To make automatically these choices means to improve the autonomy of the system in learning the initial strategy and in its tuning during on-the-job runs in a dynamic environment.
43 In [5] and [6] the rules are obtained by iteratively dividing the input and the output space into regions to which numerical input-output data are assigned Also machine learning approaches [7], neural networks [8] and genetic-based learning algorithms [9] have been used to derive a feasible set of rules. In [10] we developed a rule construction algorithm to automatically build a fuzzy wall-follower. The resulting rule base is efficient but contains a large number of fuzzy rules depending on the number and the granularity of input variables. This paper addresses the problem of automatically building a compact rule base by estimating the relevance of each input signal in the control decisions. Using a well known machine learning technique the number of produced fuzzy rules is drastically reduced with respect to [10] without weakening the controller. Experimental results will be shown using a fuzzy controller for the wall-following task. 2. T H E F U Z Z Y W A L L - F O L L O W E R 2.1. N o t a t i o n s and definitions Let us consider a fuzzy system with n inputs xl,...,xn and a single output y. Each input linguistic variable Ak E Ux, k = 1, ..,n, is characterized by Nk linguistic terms Akl, Ak~., ..., AkN~.. The linguistic variable B E Uo is characterized by M linguistic terms B1,B2,...,BM. A fuzzy set is associated with a crisp representative value, that we define as the modal point for triangular or gaussian shaped sets. Trapezoidal sets instead will be represented by the midpoint of the range of points having membership value 1.0. We will denote by aki and bj the representative value for sets Aki and Bj respectively. Let the rule that maps the i th multivariate fuzzy input variable A i to the jth univariate output set be labelled by rij, i.e.: rij: IF (xl is A~) A N D . . . AND (x~ is A~)THEN (y is B j) where A~ (respectively B j) is the linguistic value of the input fuzzy variable Ak (respectively the output fuzzy variable B) in rule rij. 2.2. Fuzzification, Rule Inference and Defuzzification We have adopted a nonsingleton fuzzifier, which is more adequate than singleton fuzzification when dealing with noisy data [11]. Our fuzzifier maps an input crisp value x into the fuzzy set A* for which pa*(x) > 0.5. A product-max inference has been applied. Rule evaluation by product operator retains more input information than using the min operator and generally gives a smoother output surface [11], a desiderable attribute for a controller. Given an input x = (zl,z2,..,zn), the firing strength of a rule is n
#,'o (x) - #A' (X) -- II /Za~ (Xk) k=l
(1)
while the final maximum membership value for the output set Bj, j = 1...M after the inference of all rules is given by: #-Bj (Y) - minCPej(y), im.a.~(p,,j (x))) where Y represents the support values of the output membership functions.
(2)
44 The center of area defuzzification method has been adopted, since it yelds a better performance of the controller than the mean of maxima method [11]. However, to reduce the computational cost of the method, we have considered the crisp representative value bj of the set Bj instead of its centroid. Thus the crisp control value is obtained as:
M y*-- Ej=I - ~ B j ( bj )bj Eg=I~Bj (bj)
(3)
-
2.3. R u l e C o n s t r u c t i o n Following the machine learning approach of [7], we have developed a rule construction method, which applies the ID3 algorithm to construct a compact rule base by selecting only the most relevant input variables. The rule construction algorithm involves building a decision tree by recursively taking as root of each subtree the input variable with greater information content and with less homogeneous branches (in terms of values of that variable in the training set). Rules are obtained by crossing all possible branches of the tree from the main root the leaf nodes, representing the output fuzzy values. At each level of the decision tree we define:
nt the total number of training samples at that point of the tree nsj the number of the nt samples with y in Bj nAhi is the number of the nt samples with xk in Aki nAkiBj is the number of the nt samples with x~ in A~ and y in Bj In order to evaluate the importance of each input variable, the Quinlan's Gain Ratio has been adopted as information measure. For an input variable ~ this is defined as: GR(xk) -
INF(y)-
Mk
INF(x~)
where Mk -- ~ nAh______A (_i i nt j
nAh,Bj 1og2nAkiBj ) nBj nBj
is the information content if xk is selected as root of the current subtree, and N~
INF(=k) - - ~
z9
nab, 1og2nAh, nt
,
M
I N F ( y ) - - ~ , :~J log2 n ' i
nt
39
nt
are the total information content for the input variable xk and the output variable y respectively. In addition, in order to avoid useless details in the decision tree (normally producing about 100 rules) at each step we create a sub-tree only if it produces a relevant reduction of the error rate, that is if
N
E eAk~ ~ et--s i=l
where
45
~q - - 4 V nt nAhi eAki -
et
-
-
-
nt
-
TI,AhiBrna~ TI,Bma,. ~
-
0.5 0.5 - - maxj=l,...M n A k i B i maxj-1,...M TI,B j -
-
7~AkiBrna~ -~
nB,~~
+
3. E x p e r i m e n t a l R e s u l t s The Sistema AUtonomo RObotizzato (AUtonomous RObotic System) SAURO, an autonomous vehicle oriented to transportation tasks in indoor environments [fig.l), has been used for collecting the training data and for testing the fuzzy wall-follower. SAURO is a LABMATE mobile base provided with a VME bus processing system and a ring of 18 ultrasonic sensors.
Figure 1. The mobile robot SAURO.
Figure 2. Arrangement of ultrasonic sensors and sensor suits.
Input to the fuzzy controller are the ultrasonic sensors measures grouped into suits, according to the required spatial resolution, each one providing a single value to the control system (fig.2). Also the number of linguistic labels associated to each fuzzy variable depends on the position (and relevance for the wall-following task) of the corresponding suit (fig.3). The motion control of the mobile robot is realized by setting its steering velocity w (fig.4). For simplicity a constant forward speed has been assumed. Training data (sensory input and corresponding control output) have been collected during navigation sessions of the vehicle driven by a human operator along a wall on its right-hand side. Fig.5 shows the training environments and the corresponding trajectories of SAURO. The rule construction method used in [10] has derived about 170 fuzzy rules, with a visible degree of redundancy. With the application of the ID3 method, only 15 rules have been extracted, without decreasing the performance of the controller. Each rule does not
46
N
F
V
~. ( m m )
150
300
10000
500 (a)
..
150 200 300
500
600
NB
NM NS t PS PM
PB
-15
30
,. (mm) 10000
(b)
Figure 3. Membership functions of the input variables (a) LeftSack, LeftFront, Front and (b) RightFront and RightBack.
-60
-30
-5 0 5 15
60
Figure 4. Membership functions of the output variable.
use necessarily all the 5 input data, therefore exploiting the real relevance of each input on control commands. The final number of rules is comparable with the size of hand-written fuzzy rule-base for similar tasks. The compact controller has successfully driven SAURO to follow unknown configurations of walls in both simple (fig. 6) and complex (fig. 7) environmental situations. It can be noted that the robot is also able to avoid unespected obstacles by correctly changing its trajectory and still following the wall.
STTS
ST~__&T
3
START
~\
Figure 5. Situations used for collecting the training data.
4. C O N C L U S I O N S Fuzzy navigation controllers can be an effective solution for the implementation of navigation behaviors that do not require internal representations of the environment, so hard to acquire and to update, necessary for conventional plan-based techniques. Automatic learning and continuous adaptation of the control strategy by representative real data can produce fuzzy rules, that experts can then evaluate and tune with their skills. A first automatic derivation of the fuzzy rule base, has produced redundant rule bases. By estimating the relationship between input and output data, we have build a fuzzy wall-f011ower with a reduced number of rules. Simplifying the fuzzy controller is especially important in prospect of the extension of the control system to the complete set of behaviors required by a safe navigation in indoor environments.
47
~,~
START/
Figure 6. SAURO's trajectory in a simple environment.
STAR
Figure 7. SAURO's trajectory in a complex environment cluttered with obstacles.
REFERENCES
1. D.A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3, 1991. 2. H. Meng and P.D. Pincton. Neural network for local guidance of mobile robots. In Proc. of Inter. Conference on Automation, Robotics and Computer Vision, pages 1238-1242, Singapore, November 1994. 3. K.T. Song and J.C. Tai. Fuzzy navigation of a mobile robot. In Proc. of I E E E / R S J Inter. Conference on Intelligent Robots and Systems, volume 1, pages 621-627, Raleigh, NC, July 1992. 4. W. Li. Fuzzy logic based robot navigation in uncertain environments by multisensor integration. In Proc. of the 1994 IEEE International Conference on Multidensor Fusion and Integration for Intelligent Systems (MFI '94), pages 259-265, Las Vegas, NV, October 1994. 5. M. Lan S. Abe. Fuzzy rules extraction directly from numerical data for function approximation. IEEE Transactions on Systems, Man and Cybernetics, 25(1):119129, January 1995. 6. L.X. Wang and J.M. Mendel. Generating fuzzy rules by learning from examples. IEEE Transactions on Systems, Man and Cybernetics, 22(6):1414-1427, November 1992. 7. J.Y. Hsu S.C. Hsu and I.J. Chiang. Automatic generation of fuzzy control rules by machine learning methods. IEEE Proc. of lnt. Conference on Robotics and Automation, pages 287-292, 1992. 8. Y. Lin and G. A. Cunningham III. A new approach to fuzzy-neural system modeling. IEEE Transactions on Fuzzy Systems, 3(2):190-197, May 1995. 9. A. Homaifar and Ed McCormick. Simultaneous design of membership functions and rule sets for fuzzy controllers using genetic algorithms. IEEE Transactions on Fuzzy Systems, 3(2):129-138, May 1995. 10. G. Castellano, G. Attolico, E. SteUa, and A. Distante. Learning the rule base for a fuzzy controller. In 4th IEEE Mediterranean Symposium on Control and Automation (MSCA '96), Crete, Greece, June 1996. 11. M. Brown and C. Harris. Neurofuzzy Adaptive Modelling and Control. Prentice Hall, 1994.
This Page Intentionally Left Blank
C COMPUTER VISION
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
Computer Vision Applications
for
Autonomous
51
Navigation" from
Research
to
G.Garibotto, P.Bassino, M.Ilic, S.Masciangelo Elsag Bailey -TELEROBOT, via Hermada 6, Genova, Italy The results described in this paper represent the conclusion of a cycle of researches carried out in the last few years in the field of Artificial Vision applied to mobile robotics. A first prototype system was experimented in the field of service robotics, including light material transportation (mail, documents, medicines, clinical data), and museum guide services. This technology has been recently applied to the driving control of standard transportation vehicles for palletised goods and materials. A prototype system named ROBOLIFT has been developed and is presently in the stage of industrial exploitation. The referred results are a concrete demonstration of technology transfer from basic research studies to the application domain and the level of maturity of Computer Vision technology for industrial use in Service Robotics. 1. INTRODUCTION TO ROBOTIC VISION In this section we briefly refer the main steps of our research efforts, from the beginning of the '80s with a strong industrial investment in robotics applications. At that time the driving force of the research effort was the development and integration of prototypes of Vision systems for a manipulating robot to perform flexible assembly operations [1 ]. Beside modelbased image recognition and object positioning and orientation, the main contribution was the effective integration of the system in a fully autonomous robotic assembly cell. The achieved performance were satisfactory in terms of flexibility and processing speed, but the high costs of the parallel implementation on the proprietary EMMA2 architecture [1] and the lack of a standardisation prevented a wider industrial exploitation of these results. In the mid of the '80s an international project has been established to better investigate 3D stereovision and motion analysis and realise a special hardware machine to perform such image processing functions almost at video rate. The main result of this European project (ESPRIT P940)[2] has been a multi-DSP parallel processing architecture based on M E - b u s which allowed 5 Hz 3D stereo reconstruction, using trinocular (three cameras) stereo arrangement and tracking of linear segment features at a rate of 10 Hz. The limited market size, as well as the not yet proved reliability and robustness of the on-going research did not allow a consolidation of the system as an industrial product. On the other hand the P940 machine has represented for many years (since the end of the project in 1992) a strong competitive advantage for the European industry in Computer Vision and a very powerful advanced research environment for real time experiments in the field of Computer Vision [3]. We have successfully applied this technology in different contexts (quality inspection and control), and
52 in robotic metrology, using camera calibration [4] for the 3D reconstruction of surface patches of object models. Anyway, one of the most challenging problem for Computer Vision was clearly identified, since the end of the '80s, in the development of intelligent sensors for autonomous navigation control. This is the context where almost all features of Vision research can be fully exploited, in terms of adaptability, dynamic response, visual servoing, learning and understanding of the environment, perception of global features (self-orientation) and local features (obstacle detection) [5]. From 1987 to 1992 our team has participated to an international project, ESPRIT P2502 [6], aimed to develop vision technologies for mobile robotics, together with the most qualified European Research centres in the field. The final demonstration in Genova was a combination of interactive teleguidance, and stereo-based obstacle detection, using off-board processing with a special hardware workstation. Moreover, a strong experience was gained in monocular vision with the development of perspective inversion tools and geometric reasoning by 3D model based techniques. In 1993, to demonstrate the maturity of vision-based navigation, using on-board low-cost PC-based processing hardware, a fully integrated system, SAM (Autonomous Mobile System), was realised, to address a wide class of autonomous navigation and transport tasks, in an indoor environment, in presence of people. In the following section 2 a brief description of this first prototype system is referred. More recently, at the beginning of 1994, an industrial oriented project has been started, to put into practice Vision technology to achieve competitive results both in terms of performance and costs. The goal of this project was the automation of an existing, conventional fork-lift carrier using an intelligent on-board control system, driven by Computer Vision. The reference application was the transportation of self-supporting palletised goods in a warehouse. Section 3 is devoted to briefly recall the Vision techniques which has been developed and used in this project as well as the current experimental results of the engineered version ofRoboLift. A more detailed description of the system can be found in [7]. 2. DESCRIPTION OF THE MOBILE ROBOT SAM The logic architecture of SAM was implemented as a series of almost independent layers of competencies, each one in charge of a single, well defined task, such as obstacle avoidance and global position maintenance. The obstacle avoidance strategy is reflexive, that is the trajectory is heuristically determined on the basis of sensor readings rather than accurately planned starting from a reconstructed local environmental map. The suboptimality of the obtained trajectory is largely compensated by the fast response time which allows to navigate safely at an acceptable speed also in presence of cluttered environments. The hardware solution is based on a PC platform as the main computational infrastructure to reduce costs, minimise the development time and take advantage of the wide choice among a great variety of add-on boards which can be integrated to improve the system functionality. The navigation system needs a periodic position and orientation estimation coming from an external sensor in order to reset drifts of odometry. This is provided through a Vision system able to detect and recognise navigation landmarks placed in known position along the robot routes [8], and to recover the robot position and orientation with respect to them.
53
Figure 1: The mobile robot SAM
Figure 2: The artificial landmark
The selected artificial landmark consists of a black annulus on a white background, as depicted in Fig.2. The 3D position and attitude of the camera with respect to the landmark reference system is obtained from a single image through model based perspective inversion.
2.1. Summary of experimental results Beside extensive laboratory experiments, the system SAM has been tested for two years in our office environment, for document transport and guest accompany service, in normal operating conditions with a lot of people wandering around. Later, the robot SAM has been equipped with a Soundblaster board for sound generation and a radio modem providing a link to an host computer at a remote control station. The remote computer is able to select the most appropriate sound or voice file according to the robot position or the navigation status (presence of obstacles, landmark search, and so on) as communicated by the robot navigation system. The robot in such a configuration was installed in an historical building during the Christmas '93 exhibitions, as reported in [9]. The results has been very encouraging in terms of performance and good acceptance by the people who visited the museum during the exhibition days. 3. ROBOLIFT: AUTONOMOUS FORK-LIFT IN LOGISTIC SERVICES The problem of autonomous transport of material in workshops and warehouses has been traditionally approached through the use of automated guided vehicles (AGV). They have been introduced into the market in the early '70s and have provided significant improvements in terms of efficiency, precision and co-ordination of material flow in manufacturing, as compared to conveyor belts, single track railways, etc. The main drawbacks of the consolidated AGV technology [10] comes from the heavy installation requirements (inductive guides buried into the floor), the need of continuous central control, the rigidity of fixed navigation pathways, the requirement to design specialised machines for each different application. Moreover there is a severe limitation in flexibility, and
54 the position of all the palletised loads in the warehouse is supposed to be known in advance with high precision (within the range of 10 mm.). Our answer is RoboLift the first Autonomous Fork Lift developed jointly by Elsag Bailey Telerobot and Fiat OM Carrelli Elevatori SpA (Patent pending). This system is based on Vision technology both for autonomous navigation and for the recognition of the pallets to be transported. 3.1 Main Characteristics of the vehicle The selected basic vehicle is the classical frontal fork lift carrier (from the well known EU family of Fiat OM Carrelli Elevatori SpA, operating in the range of 1.2, 1.5 ton), being one of the most commonly used in the market. The kinematics of the vehicle is made of three wheels (two driving and one steering). A schematic drawing of the vehicle and the list of sensors which have been introduced for autonomous control is referred in figure 3.
Figure 3 Sensor arrangement in ROBOLIFT
Fig.3 Model based vision and 3D recognition and positioning of the pallet
3.2: Computer Vision for Autonomous Navigation Vision processing is performed primarily to support autonomous navigation. Through the recognition and 3D location of some artificial landmarks (H shaped) placed on the floor along the planned navigation path, the Vision system is able to self-localise the vehicle in the scene, by integrating this information with the odometric values coming from sensors on the wheels (both drive and steering).
55 3D model based vision is used to identify and recognise the H-shaped landmarks placed onto the floor, by exploiting all a priori information available to simplify image processing analysis. To avoid errors and ambiguities caused by other features in the scene or noise effects, geometric reasoning is performed directly in 3D, by reprojecting all features onto the 3D floor. Using extended geometric features, this process is proved to be extremely robust also when the landmarks is partially occluded or damaged by stains. Computer Vision is performed on-line, during the motion of the vehicle passing through these landmarks, along the navigation path. The success of Artificial Vision is strongly related to the accuracy of camera calibration, computed with respect to the odometry of the vehicle, to obtain a homogeneous data representation suitable for the navigation commands (steering and driving). 3.3" Computer Vision for the recognition of the pallet pose A second fundamental vision function which is implemented in ROBOLIFT is pallet detection and recognition and is performed by a camera placed within the forks and rigidly connected to them, so that it can move up and down, searching for the different position of the palletised load. A model-based Vision algorithm has been implemented, to search for the central cavities of the pallets and compute the size and shape of these holes. A prediction-verification paradigm has been implemented. It consists in projecting onto the image the geometry of the pallet model, from the expected position in the 3D world. An adaptive estimation of the contrast is performed into the expected hole position in the image, followed by a controlled region growing process aimed to propagate this grey level up to the border of the hole, with a constraint on the expected size and shape. A schematic example is referred in figure 3. Once the holes are correctly identified and localised, the current 3D position of the pallet is computed. If this new position is within the tolerance bounds to be carried by the fork-lift, the forks will be properly shifted to left or right of the appropriate amount to take the load in a centred position. The project has been developed by taking as a reference the standard Europallett of size 1200 x 800 mm., with loading side 1200. The Computer Vision system has proved to be able to recognise the presence of the pallet, also in a wide range of operating conditions, from intense sun-light, artificial light and shadows, and compute the current distance and orientation of the pallet with respect to the vehicle.
4. RESULTS AND STATUS OF THE PROJECT The ROBOLIFT project started in 1994, with the definition of the basic architecture and the selection of the necessary modifications implemented on a first prototype. During 1995 a first laboratory prototype has been integrated and a first release of control software has been made available in July '95, followed by an extensive experimentation phase in a suitably equipped warehouse environment. A second engineered prototype has been integrated at the beginning of'96, and it has been officially presented at the Hannover Fair in April '96. Further engineering work is in progress to improve the system performance, its robustness and reliability as well as the level of integration with the application field.
56 5. CONCLUSIONS One of the most important objective of advanced research institutions, including EEC programmes, is the promotion of industrial exploitation of research results with particular attention to A.I. technologies which received significant research funds in the last few years. The paper describes our recent experience in exploiting Computer Vision technology in transport service automation, following the necessary fundamental steps of basic research development, laboratory prototype implementation, and the final acquisition of a strong integration knowledge and expertise. The achieved results consists in the integration of an autonomous fork-lilt carrier, which can be also driven in the conventional way by the human operator. The system makes use of model based passive vision techniques, without any external active lighting support. Computer Vision represents the main sensory component for both autonomous navigation and pallet recognition. The possibility to use a standard PC-based multiprocessing architecture allows the implementation of a competitive industrial system. The extensive experimental results collected during many hours of test demonstrate a high maturity of Vision technology in advanced mobile robotics, to come out from the research labs an be used as an established and accepted technology in Industrial Automation. REFERENCES 1. L.Borghesi, et al. "A Modular Architecture for a flexible real-time Robot Vision System", Proc. of the Int. Conference on Digital Signal Processing, Firenze, Sept. 1987. 2. G.Garibotto, S.Masciangelo, "Depth and Motion Analysis P940: development of a realtime Computer Vision System", ESPRIT Workshop at ECCV'92, S.Margherita, May 1992. 3. Faugeras, "Three-Dimensional Computer Vision; a geometric viewpoint", The MIT press, 1993. 4. E.Bruzzone, F.Mangili, "Calibration of a CCD Camera on a Hybrid Coordinate Measuring Machine for Industrial Metrology", International Symposium on Industrial Vision Metrology, Winnipeg, Manitoba, Canada, July 1991. 5. G.Garibotto, S.Masciangelo, "3D Computer Vision for Navigation/Control of Mobile Robots, in Machine Perception, AGARD Lecture Series, 185, 1992. 6. B.Buxton, et al. "The Transfer of Vision Research to Vehicle Systems and Demonstrations", Proc. of the ESPRIT Conference, 1991, Brussels. 7. G.Garibotto, "ROBOLIFT: Vision-guided Autonomous fork-lilt", Service Robot, An International Journal, Vol.2, n.3, 1996, pp.31-36. 8. Garibotto, M. Ilic, S. Masciangelo, "An Autonomous Mobile Robot Prototype for Navigation in Indoor Environments", Proc. of the Int. Symposium on Intelligent Robotic Systems '94, Grenoble (France), July 1994. 9. Garibotto, S. Masciangelo, M Ilic, "Vision Based Navigation in Service Robotics", pp. 313-318, Image Analysis and Processing, Lecture Notes in Computer Science, Springer, 1995 10. Warnecke, C.Schaeffer, J.Luz, "A driverless and free-ranging fork-lift carrier, Proc. of the 24th ISIR, session El, Nov. 1993.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
57
An Optimal Estimator of Camera Motion by a Non-Stationary Image Model G. Giunta
U. Mascia
INFO-COM Department, University of Rome "La Sapienza", Rome, Italy e-mails: [email protected] [email protected] Camera zooming can be regarded as a 2D Doppler effect. Techniques for Doppler estimation from 1D signals, based on data partition and linear regression on a set of time-delay measurements, were presented in the literature. Such basic idea is here extended to fine motion estimation. The devised algorithms, estimating four global motion parameters (viz: horizontal and vertical translation, zooming, and rotation), are based on a non-stationary model. They have been validated by both synthetic and experimental tests. 1. I N T R O D U C T I O N The analysis and estimation of motion are very important in time-varying image processing. Many algoritms have been developed to estimate 2D motion for different applications [ 1]-[ 12], such as object tracking, image segmentation, environment sensing for autonomous vehicle navigation, image sequence coding, object-oriented analysis and synthesis coding, TV standard conversion, frame rate conversion, bandwidth compression for HDTV, very low bit-rate video coding for audiovisual services, 2D motion parameter acquisition from image sequences, camera motion estimation and compensation, etc.. Fast discrete-time techniques for time-delay estimation with sub-sample accuracy, based on a parabolic interpolation of estimated cross-correlation samples, were devised and analysed for random 1D signals [ 13]. Camera zooming (or radial motion) causes an isotropic change of the scale in the whole image, that can be regarded as a 2D Doppler effect on the magnitude of spatial polar coordinates. Moreover, any rotation can be modeled as a 2D Doppler effect on the phase of spatial polar coordinates. Techniques for Doppler estimation of 1D signals were proposed in the literature [14]-[16]. Among them, an indirect estimation method based on data partition and linear regression on a set of time-delay measurements (linearly related to the actual Doppler coefficient) was devised [ 17]. Such basic idea is here extended to a 2D fast estimator of spatial position. Motion compensation brings about a saving in bit-rate, due to the smaller prediction error as well as to the reduction in motion-vector coding. A fine estimation with sub-pixel accuracy can take account of more information, giving better results in the prediction of the picture. Stationarity is widely assumed in image modeling for the sake of simplicity, while it is well known that this assumption is far from reality. A motion estimation procedure with a subpixel accuracy is here presented. It is based on a non-stationary model depending on local properties. This approach can be usefully employed to reach our aim, by extending the method devised in [ 17], for estimating the four global motion parameters (viz: horizontal and vertical translation, zoom, and rotation). In particular, a 2D paraboloid is used for interpolating the inter-pixel cross-correlation estimates. This method can be applied to 2D Doppler estimation, after a block partition of the whole image, by a linear regression in the complex domain of the spatial dispacements. Such measurements can be also weighted according to a proper error function, derived from a confidence measure, accounting for the local statistics of each block. Such a non-stationary method is then based on the minimization of the weighted mean square error.
58 2. I M A G E MODEL AND MOTION ESTIMATOR
2.1 Time-varying image model Let z = x + j y be the Cartesian coordinates, expressed in the complex domain. Let us consider a pair of sequential image frames, say R and P. Let us assume the following model of instantaneous motion: R(x,y) = R(Z) = S(Z) (1) (referencepicture) P(x,y) = P~) = S[~-~i) / ~ + E~) (2) (moved picture) where E(/,) is the model error image, that also accounts for the two noises. In particular, Re{fi} and Im{fi} represent the horizontal and vertical displacements, while the term o~=pexp [j 0] accounts for both the zoom factor (p) and the rotation angle (0). It is interesting to point out that any rotation can be directly included in the MSE estimation in the complex domain: in fact, a complex change of scale takes zoom as well as rotation into account (i.e. modulus represents zoom, and phase represents angle of rotation).
2.2 Displacement estimation Our method performs a fine (sub-pixel) estimation by means of a fast digital algorithm. The whole picture is divided into small blocks and the relative position displacements are extracted by a conventional matching algorithm based on cross-correlation measurements. The estimated displacements are linearly related in the complex domain and the four parameters can be derived by performing a complex linear regression. The relationships so obtained can be weighted according to their accuracy, which depends on the contents of each block of data. In paticular, we divide the whole reference image into small blocks. For each block, we search for the best matching block in the moved picture (like accomplished in several widely used standard), by evaluating the magnitude of displaced frame difference, i.e.: MDFD(.-I) = { ~k I R(_zk) - P(_zk+~) I } (3) where the sum extends over all the pixels of the considered block. We then estimate the linear motion between the considered pair of blocks as the displacement ld, that minimizes the magnitude of displacedframe difference, i.e.: R = "c 9arg MDFD(Id,) = arg mini: MDFD~) (4) by performing a sub-pixel parabolic interpolation of the square of the estimated MDFD(.TJ: MDFD2(.~) = MDFD2(~,~) _=a ~2 + b ~2 + c ~ + d ~ + e ~ + f (5) with :t = ~ + j ~ . The displacement ld, is determined as the minimum argument of the parabolic function, fitted by six samples of the squared MDFD(jO chosen around its coarse minimum.
2.3 MSE global motion estimation Let /5 be the center of the i-th reference block. We may write a (usually overdetermined) set of linear equations in the complex domain:
o~zi + ~ = R i
(6)
In fact, there are 2 complex unknowns (8 and ~ accounting for 4 real parameters (viz: horizontal and vertical displacement, zoom factor and rotation), while the number of equations is equal to the number of considered blocks in the whole picture.
59 If we chose the origin of the coordinates at the exact center of the reference frame and we take a symmetic arrangement of N blocks into account, a standard pseudo-inverse solution, based on the mean square error (MSE), can be employed: N N
~5= 1 ~
i=l
~l,i (7/
zi Ri
o~=~
(8)
N
i=l
~ z i zi i=l
2.4 WMSE global motion estimation
If we have no particular symmetries or if we wish to take the available equations into account with different weights, a weighted mean square error (WMSE) can be defined: /5=~-1 I [ ~ w i2 z i*zi][~ w i2 i]~l,- [ ~ w i2 *i l [ ~z w i2 z i ~l,i] 1 i=l
i=l
i=l
ct= 1 I[i__~1 w2l [i__~1w2z~Ril-[i__~1 w2z~l [i__~1w2Ril I with
A= ~ i=l
~w2zi z
~wiz i
i=l
i=l
(9)
i=l
(10,
wiz i=l
In fact, while the MSE criterion minimizes (6), the solutions (9)-(10) minimize the set of equations: Wi [ (t ~ + ~5 ] = Wikl, i (11) We have employed a parabolic approximation of the squared MDFD (5) near the minimum in the displacement estimation. It is well known the dependence of time-delay estimation performance on such a parameter (its second derivative provides the asymptotic error variance). The curvature of the same squared MDFD function is the simplest local confidence measure that we may use. In particular, we have employed the curvature of the estimated squared MDFD (depending on the cross-correlation function) along the direction of the local motion gi estimated for each block, i.e.: --
2
wi
M D F D 2(.~i)
~ ~i 2
I
(12)
,~i= Ri
MDFD (depending on the cross-correlation coefficient), divided by the variances 0..Kz- and o.. 2 of the reference and the moved blocks, i.e." t" and the normalized
1 . wi2 OROp
i
~,i2
MDFD 2r
(13)
~i=ai with Y'i= ~'i exp[Jei]' evaluated along the same direction (i.e. ei=angle{Ri}).
60
3. SYNTHETIC AND EXPERIMENTAL RESULTS 3.1 Synthetic tests with known global motion parameters A number of standard still images (viz: "Airplane", "Barbie", "Baboon", "Boats", and "Gold") have been considered. The test images were reduced by the factor 4 from the original size of 512.512 pixels, becoming 128.128 pixels sized. Each image was compared to a copy of itself, deformed by a known set of global motion parameters. The parameter values were randomly chosen, each one with a uniform probability density function, in the range [-0.5,0.5] pixels for the horizontal and vertical displacements, in [0.95, 1.05] for the zoom factor, and in [-3,3] degrees for the rotation angle. Each image was tested 500 times, and a total of 2500 test images was then obtained. The motion vectors were estimated from 8.8 pixels sized blocks. Three different weights were used: NOW = no weight; DCCF = directional convexity of the cross-correlation function; DCCC = directional convexity of the cross-correlation coefficient. The numerical results of accuracy of the four estimates are reported in tabs. 1-4 for the cases of few (16) and many (195) blocks used in the performed tests. H.Dis.(pels) NOW DCCF DCCC
few (16) blocks Bias.10 Var.ol02 MSE.102 0.13 1.46 1.48 -0.09 0.27 0.27 -0.12 0.29 0.30
many (195) blocks Bias.10 Var..102 MSE.102 0.04 0.17 0.17 0.01 0.33 0.33 0.02 0.30 0.30
V.Dis.(pels) NOW DCCF
Bias.10 0.63 0.14
Var.~ 1.78 0.65
MSE.102 2.17 0.67
Bias.10 0.08 0.04 0.04
Var..102 0.15 0.14 0.12
MSE.102 0.16 0.14 0.12
Zoom factor NOW DCCF DCCC
Bias.103 4.88 1.37 1.31
Var.ol06 68.1 21.0 21.1
MSE.106 91.9 22.9 22.8
Bias.103 0.11 0.17 0.11
Var..106 1.02 1.54 1.78
MSE.106 1.03 1.57 1.79
Rot.angle (o) NOW DCCF DCCC
Bias.10 0.97 0.36 0.37
Var..102 43.5 9.8 9.7
MSE.102 44.4 9.9 9.9
Bias.10 -0.05 0.01 -0.05
Var..102 0.31 0.22 0.23
MSE.102 0.31 0.22 0.23
Tables 1-4. Estimation accuracy of the synthetic tests for a small and a large number of blocks.
3.2 Experimental tests with unknown global motion parameters Pairs of frames from two standard test image sequences (viz: "Foreman" and "Tabletennis") were extracted, because of their global motion characteristics. The subsequences were alternately cut forward and back. The motion vectors were estimated from 8-8 pixels sized blocks and collected to simultaneously estimate the four motion parameters. Some significant results of the three estimates (NOW, DCCF, and DCCC) of the rotation angle from the "Foreman" sequence (320 blocks per frame, 141 frames) and of the zoom factor from the "Table-tennis" sequence (1176 blocks per frame, 93 frames) are shown in the figs. 1-2.
61
Frames
w
from
"Foreman"
3
i!
L,
N0W [XX~
~ t 9
R
ii
il~.
l i i i~ t~
-
l/,J
//!
l
ol c t~
c o
in
-1
I I , .... ,.h,
m o
-2
9
90
i
9
100
l
9
110 frames
i
120
130
Fig. 1. Estimates of the rotation angle from an experimental sub-sequence. Frames
from
.
#
"Table-tennis"
1,03 L_
o r
1,02
N,-
E o o N
, t -"
,, x...~,....,,f .~
~\
1,01 DCCC 1,00
50
.
,
60
.
,
70
frames
9
,
80
. 90
Fig. 2. Estimates of the zoom factor from an experimental sub-sequence. 4. C O N C L U D I N G
DISCUSSION
The results of the synthetic tests, performed on actual standard images to validate the devised algorithm, have shown that the WMSE-based method is suited in the presence of a small number of blocks, while its accuracy is comparable to a simpler MSE-based method for a larger number of available data. No significant difference appears between the results obtained with weights derived from the local cross-correlation function or from the local crosscorrelation coefficient. The method has been also applied to standard test image sequences. It visually appears that the WMSE criterion enhances the dynamic properties of the algorithm (this can be useful for fast tracking the camera motion), while MSE-based estimates are usually more smoothed.
62 The specific criterion (namely: uniform MSE, CCF-based WMSE, CCC-based WMSE) should be chosen in practice according the particular application. As a general remark, a simple MSE-based technique is suited for estimating the camera motion of high resolution image sequences, while both the examined WMSE-based methods should be preferred on small images, such as a region of interest extracted by a segmentation algorithm and containing a moving object. Future research investigations will include the case of multiple objects moving on a background. The mathematical problem then becomes a multi-linear regression, that can be resolved after a proper clustering of the available measurements. REFERENCES
[ 1] J.K. Aggarwal and N. Nandhakumar, "On the computation of motion from sequences of images - A review", Proc. IEEE, Vol. 76, No. 8, 1988, pp. 917-935. [2] G.J. Keesman, "Motion estimation based on a motion model incorporating translation, rotation and zoom", in Signal Processing IV: Theory and Applications, 1988, pp. 31-34. [3] S.F. Wu and J. Kittler, "A differential method for simultaneous estimation of rotation, change of scale and translation", Signal Processing: Image Communication, Vol. 2, 1990, pp. 69-80. [4] J.H. Moon and J.K. Kim, "On the accuracy and convergence of 2D motion models using minimum MSE motion estimation", Signal Processing: Image Communication, Vol. 6, 1994, pp. 319-333. [5] Z. Eisips and D. Malah, "Global motion estimation for image sequence coding applications", Proc. 17th Conv. of Elec. and Electronics Eng., Israel, 1991, pp. 186-189. [6] Yi Tong Tse and R.L. Baker, "Global zoom/pan estimation and compensation for video compression", Int. Conf. Acoust. Speech Signal Proc., 1991, Vol. 4, pp. 2725-2728. [7] G. Giunta, T.R. Reed and M. Kunt, "Image sequence coding using oriented edges", Signal Processing: Image Communication, Vol. 2, No. 4, 1990, pp. 429-440. [8] M. Bierling and R. Thoma, "Motion compensating field interpolation using a hierarchically structured displacement estimator", Signal Processing, Vol. 11, 1986, pp. 387-404. [9] Y. Ninomiya and Y. Ohtsuka, "A motion compensated interframe coding scheme for television pictures", IEEE Trans. Commun., Vol. COM-30, 1982, pp. 201-211. [ 10] A. Amitay and D. Malah, "Global-motion estimation in image sequences of 3-D scenes for coding applications", Signal Processing: Image Communication, Vol. 6, 1995, pp. 507-520. [11] M. Hoetter, "Differential estimation of the global motion parameters zoom and pan", Signal Processing, Vol. 16, 1989, pp. 249-265. [ 12] P. Migliorati and S. Tubaro, "Multistage motion estimation for image interpolation", Signal Processing: Image Communication, Vol. 7, 1995, pp. 187-199. [ 13] G. Jacovitti and G. Scarano, "Discrete time techniques for time delay estimation", IEEE Trans. on Signal Proc., Vol. 41, No. 2, 1993, pp. 525-533. [14] C.H. Knapp and G.C. Carter, "Estimation of time delay in the presence of source or receiver motion", J. Acoust. Soc. Am., Vol. 61, No. 6, 1977, pp. 1545-1549. [15] J.W. Betz, "Comparsion of the deskewed short-time correlator and the maximum likelihood correlator", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-32, No. 2, 1984, pp. 285-294. [ 16] J.W. Betz, "Effects of uncompensated relative time companding on a broad-band cross correlator", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-33, No. 3, 1985, pp. 505-510. [17] E. Weinstein and D. Kletter, "Delay and Doppler estimation by time-space partition of the array data", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-31, No. 6, 1983, pp. 1523-1535.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
63
A simple c u e - b a s e d c a m e r a calibration m e t h o d for digital production of m o v i n g images Y. Nakazawa, T. Komatsu
and
T. Saito
Department of Electrical Engineering, Kanagawa University 3-27-1 Rokkakubashi, Kanagawa-ku, Yokohama, 221, Japan
One of the keys to new-generation digital image production applicable even to domestic uses is to construct simple methods for estimating the camera's motion, position and orientation from a moving image sequence observed with a single domestic video camera. For that purpose, we present a method for camera calibration and estimation of focal length. The method utilizes four definite coplanar points, e.g. four corner points of an A4 size paper, as a cue. Moreover, we apply the cuebased method to the digital image production task of mixing real and CG moving image sequences. The cue-based method works well for the digital image mixing task.
1. INTRODUCTION
- BACKGROUND AND MOTIVATION -
Recently some research institutes have started studying digital production of a panoramic image sequence from an observed moving image sequence, construction of a virtual studio with the 3-D CG technology and so on, with intent to establish the concept and the schema of the new-generation digital image production technology. Such an image production technology, utilizing information about the camera's motion, position, orientation and so on, integrates consecutive image frames to produce such an enhanced image as a high-resolution panorama or mixes a synthetic 3-D CG image sequence and a real moving image sequence taken with a video camera. The key to the newgeneration digital image production applicable even to domestic uses is to develop simple methods for estimating the camera's motion, position and orientation from a real moving image sequence observed with a single video camera [1 ]-[4]. In this paper, to render it feasible to pertbrm such 3-D estimation of the camera's motion, position and orientation when we use a single do~nestic handy video camera whose camera parameters are not given in advance, we present a method for performing camera calibration along with estimation of focal length of the camera accurately by using four definite coplanar points, which usually correspond to four vertices of a certain quadrilateral plane object such as an A4 size paper, as a cue. The practical computational algorithms for the cue-based method of camera calibration are composed of simple linear algebraic operations and arithmetic operations, and hence they work so well as to provide accurate estimates of the camera's motion, position and orientation stably. Furthermore, in this paper, we apply the cue-based camera calibration method to the image production task of mixing a synthetic 3-D CG image sequence and a real moving image sequence taken with a video camera according to the recovered estimates of the camera's motion, position and orientation.
64 2. CUE-BASED C A M E R A C A L I B R A T I O N In this paper, we assume the following situation. The situation is as follows: while moving the single video camera arbitrarily by hand, we image the scene which includes not only the objects of interest but also four definite coplanar points P~ -- P4 whose relative positions are known in advance and which usually correspond to four vertices of a certain quadrilateral plane object with known shape and are used as a cue for camera calibration. We perform camera calibration, that is to say, determination of the camera's position and orientation at each image frame, from 2-D spatial image coordinates of the four definite coplanar cue points, which are detected with our recently presented active line-contour model [5] and tracked temporally over consecutive image frames. Under such conditions, we perform camera calibration and estimate the focal length f o f the camera at the same time.
2.1. Image Coordinate System Here for each image frame we define the 3-D viewing coordinate system o'-x'y'z' which is associated with the 2-D image coordinate system O-XY as shown in figure 1. In figure 1, we represent the 3-D viewing coordinates and the 2-D image coordinates with ( x' y' z' ) and ( X Y ) respectively. We represent the 3-D viewing coordinates of the four coplanar cue points Pl " P4 with { p~' = ( x / y / z i' )t ; i = 1,2, 3, 4 }, and we represent the 2-D image coordinates of the imaged coplanar cue points, perspectively projected onto the image plane, with { Pi - ( X Y )" i = 1, 2, 3, 4 }. 2.2. Camera Calibration The problem of camera calibration is to recover geometrical transformation of 3-D world coordinates of an arbitrary point in the imaged scene into its corresponding 2-D image coordinates, from given multiple pairs of the 3-D world coordinates and their corresponding 2-D image coordinates. The camera calibration problem is concisely formulated with the homogeneous coordinate systems. Given both 4-D homogeneous world coordinates a - ( x y z 1 )' of an arbitrary point in the imaged scene and their corresponding 3-D homogeneous image coordinates b = h. ( X Y 1 )', then the foregoing transformation will be represented as the linear transformation, which is
m3 3-D world coordinate ,
Y
~
ml
X
Figure I. Coordinate systems.
system
65 defined as follows:
(x'
y
,
z
,)t
-M'(x
v
=(m I
m2
)t 1
z
m3
m4).(x
v z. 1)t
=x.m 1 +y.m 2+z.m 3+m 4
(1)
Xt
X=--7.f
,
Y= y ' . /
Z
(2)
.7
where the focal length )"is explicitly handled. Here the camera calibration problem is defined as the problem to recover the 3 x 4 matrix M and the focal length f o f equation 1 from given multiple pairs of the homogeneous world coordinates and their corresponding homogeneous image coordinates. Equation 1 means that the 3-D viewing coordinates ( x' y' z' ) are expressed as a linear combination of the three vectors { in 1 rnz in 3 }, and hence we may regard the three vectors { m I in 2 m 3 } as the basis vectors of the 3-D world coordinate system o-xyz. On the other hand, the vector m 4 means the displacement vector shifting from the origin of the 3-D viewing coordinate system to that of the 3-D world coordinate system. Here we imagine a plane quadrilateral whose four vertices are given by the four definite coplanar cue points, and we refer to the plane quadrilateral as the cue quadrilateral. As a common coordinate system to all image frames, we define the 3-D world coordinate system o-xyz whose x-y cross section contains the cue quadrilateral, that is to say, whose z-axis is normal to the cue quadrilateral. Moreover, without loss of generality, we put the origin of the 3-D world coordinate system o-xyz at one of the coplanar cue points, e.g. P l In this case, we can represent the 3-D world coordinates of the four coplanar cue points P~ - P4 with
I,-(xi
}
y, :.,),-(-,, y, o)'-o Pi-(xi
Yi
zi
Yi
0) t
"i-2,3,4
(3)
Assuming that the focal length f o f the camera is accurately estimated some way or other, which will be described in the next section, we can easily recover the 3 x 4 transformation matrix M of equation 1 from the four pairs of the 3-D world coordinates Pi - ( x; Yi 0 )' of each cue point P; and its corresponding image coordinates P ~ - ( X; Y; )'. Substituting the four coordinate pairs into equation 1, then we will reach the simultaneous equations:
, (x'i
Yi
xi/z'i
,)t zi
=N.(xi
Yi
1) t =
in,1n,2 n21
n22
n24
n31
n32
n34
"
i
/
t
Xi -
,
Yi - Yilz'i
" i - l, 2, 3, 4
(4)
where the focal length f i s implicitly included in the expression of the 3 x 3 matrix N and the matrix
66 N is related with the matrix M as follows" / mll m21
m12 m22
m24
m31
m32
m34
ml411 =
nillf n21/f
nl2/f
nl41f]
n22/f
n24/f
tt31
n32
(5)
t/34 J
The simultaneous equations of given by equation 4 are linear with respect to the nine unknown matrix components n's, and we can easily solve them. However, their solution should be expressed with one scale factor, and hence here we set the value of the matrix component n~ to one. Moreover, given the focal length f o f the camera, then we can recover the column vectors { m 1 m z m 4 }of the matrix M by applying the relation of equation 5. With regard to the column vector m 3 o f the matrix M, we should employ a vector which is normal to both the two column vectors { m 1 m z }, e.g. m3 =
{Im~l/Iml x m2]}.
(m I x m2)
(6)
Thus we can recover the 3 x 4 transformation matrix M of equation 1. 2.3. E s t i m a t i o n
of Focal Length
Once we recover the foregoing transformation matrix N of equation 4, we can estimate the relative depth z i' of each coplanar cue point Pi as follows: t
zi = m31 "xi + m32 ' Yi + m34 = n31 "xi + n32 "Yi + n34 Thus we get an estimate of the 3-D viewing coordinates Pi' - ( P. as follows"
,( x'i Yi) tZi( = Xi "z.i l l
I
Pi =
Yi "z i l l
Xi' Yi' Zi' )' o f
;
(7) each coplanar cue point
(8)
zi
The lengths of the four sides of the cue quadrilateral are assumed to be known in advance, and furthermore taking account of the fact that the ratio of lengths of two sides arbitrarily chosen out of the four sides is invariant irrespective of the definition of the 3-D coordinate system, we get the relation:
I'
'1
P2 - P l I / I P 4 - P l
= IP2
-
Pl
12/Ip4-p,I
2
=
r
(9)
Substituting equation 8 along with equation 7 into equation 9, then we will obtain the quadratic equation with respect to the focal length f. The solution is given by (10)
f = ~/(r.C-A)/(B-r.D) -
- x,. z;)
+
-
z;)
C = (X 4'2:.'4 - X l'z' 1)2+ (Y4"z4-YI'z'I)2,
,)2
67
3. DIGITAL MOVING IMAGE MIXING We have imaged the scene in our laboratory while moving a 8-mm domestic handy video camera arbitrarily by hand, and then we have applied the foregoing cue-based camera calibration method to the moving image sequence, each image frame of which is composed of 720 • 486 pixels. In the imaged scene we have put an A4 size paper on the table in the scene, and we have used the four comer points of the A4 size paper as cue four coplanar points. Moreover, we have put a book on the table as a still obstacle. We have detected the four edges of the A4 size paper with our recently presented active line-contour model [5], and identified its four comer points as their intersections, and thus we obtain estimates of the image coordinates of the four comer points.
Figure 2. Image frames chosen from the resultant mixed moving image sequence.
68 We have performed the digital image production task of mixing a synthetic 3-D CG image sequence of moving two toy cars and a moving toy robot and the real moving image sequence of our laboratory, according to the recovered estimates of the camera's motion, position and orientation. Figure 2 shows some image frames chosen fiom the resultant mixed moving image sequence. As shown in figure 2, we can hardly identify any artificial distortions in the mixed image sequence, which demonstrates that the cue-based camera calibration method works well for the foregoing digital moving image mixing task. 4. CONCLUSIONS In this paper, we present a method for performing camera calibration along with estimation of focal length of the camera accurately by using four definite coplanar points as a cue. The practical computational algorithms for the cue-based method of camera calibration are composed of simple linear algebraic operations and arithmetic operations, and hence they work so well as to provide accurate estimates of the camera's motion, position and orientation stably. Moreover, in this paper, we apply the cue-based camera calibration method to the digital moving image production task of mixing a synthetic 3-D CG image sequence and a real moving image sequence taken with a domestic video camera according to the recovered estimates of the camera's motion, position and orientation. Experimental simulations de~nonstrate that the cue-based camera calibration method works well for the digital moving image mixing task. The key to the accurate cue-based camera calibration is how to detect the feature points used as a cue in an input image highly accurately. Sub-pixel accuracy will be possibly required for the detection task. To detect the feature points with sub-pixel accuracy, in advance we should enhance spatial resolution of the image region containing the feature points. It seems that we can apply our recently presented temporal-integration resolution-enhancement method [6] to this purpose. Moreover, to complete a practical image processing algorithm for the digital moving image mixing task, in addition to the camera calibration, we should take account of many other points, that is to say, occlusion between real objects and synthetic CG objects, 3-D shape of real objects, and so on. Further studies on these points will be required. REFERENCES
1. K. Deguchi, "Image of 3-D Space : Mathematical Geometry of Computer Vision", Shoukodo Press, Tokyo, Japan, 1991. 2. C. Longuet-Higgins , "A Computer Algorithm for Reconstructing a Scene from Two Projections", Nature, .293 ( 1981) 133. 3. R. Horaud, et al., "An Analytic Solution for the Perspective 4-Point Problem", Computer Vision, Graphics, and hnage Processing, 47 (1989) 33. 4. C. J. Poelman and T. Kanade, "A Paraperspective Factorization Method for Shape and Motion Recovery", Lecture Notes in Computer Science, 801 (1994) 97. 5. Y. Nakazawa, T. Komatsu and T. Saito, "A Robust Object-Specified Active Contour Model for Tracking Smoothly Deformable Line-Features and Its Practical Application to Outdoor Moving Image Processing", IEEE 1996 International Conference on Image Processing, 17P8.13, 1996. 6. Y. Nakazawa, T. Komatsu and T. Saito, "Temporal Integration Method for Image-ProcessingBased Super-High-Resolution Image Acquisition", lEE Proc. Vis. Image Signal Process., 143 (1996) in press.
Time-Varying Image Processing and Moving Object Recognition, 4 - V. Cappellini (Ed.) 69
1997 Elsevier Science B.V. All fights reserved.
E x l ) h ) r a t i o n of t h e e n v i r o n m e n t w i t h o p t i c a l s e n s o r s m o u n t e d on a m o b i l e robot P. Weckesser, A. von Essen, G. Al)l)enzeller, R. Dillmann Institute for Real-tiine Computer Systems & Robotics Prof. Dr. U. Reinhold, Prof. Dr. R. Dillmann University of Karlsruhe, Department for Comt)uter Science I(arlsruhe, Germany The exI)loration of unknown enviroimmnt is an iInportant task for the new generation ()f mol)ile service robots. These rol)ots are supposed to operate in dynalnic and changing envir()nlnents together with human beings an(1 other static or moving objecl, s. Sensors tha,t are capable of providing tile quality of information that is require(t for the described scenario are optical sensors like digital cameras and laserscanners. In this I)al)er sensor integration and fusion for sli(:h sens()rs is described. Complementary sensor information is transformed into a coInmOn representation in order to achieve a coot)crating sensor system. 1. I n t r o d u c t i o n
In this Imper an apt)roach to fuse sensor infl)rnmtion fr(ml (:{mlplementary sensors is I)resented. 'File mobile rol)()t PlqlAMOS (figure 1) was used as an experimental testlmd. A multisensor systmn supports the vehicle witll o(lometric, sonar, visual and laserscanner information. This work is part of a large project with tile goal of lnaking r()bot navigation safer, faster, more rdiable and l~lore stalJle under changing environtnental conditions. An architecture for actiw; and ta.sk-driw'~l processing of sens()r data is presented in [10]. With this architecture it is p()ssit)le to control the sensor system acc()r(ling to envir(mmental conditions, per(:eiw;d sensor infornlation, a1)riori knowledge and the task of the robot. The system's 1)erfl)rmance is demonstrate(l for tile task of exploring an llnkn()wn environment and incrementally lmihling up a geometrical model of it.
Figure 1. PRIAMOS
70 Sensor fusion is perh)rmed by matching the local perception of a laserscanner and a camera system with a global model that is being built up incrementally. The Mahalanobis distance is used as matching criterion and a Kalman filter is used to filse matching features. A common representation inclu(ling the uncertainty and the confidence is used for all scene features. 1.1. Mobile robot navigation Navigation tasks of a mobile robot can be subdivided into three subproblems.
1. collision avoidance: this is the basic requirement for safe navigation. The problem of collision avoidance is solve(1 for dynamic environments with different kinds of sensors like sonars or laserscanners.
2. mobile robot positioning: if geometrical a priori information about the environment is available to the robot the folh)wing questions can be asked. 'Where am I?', Where am I going?' and 'How do I get there' [7]. With today's sensors these questions can be answered for static environments [6]. Though the problem is not solved in general for dynamic and changing environments.
3. exploration and environmental modelling: The problem of exploring an unknown environment was apI)roached by various groups [4,1,9] but is by far not solved. Most approaches aim at building up a 2-dimensional map of a static environment. In this paper a 3-dimensional map of the environment is built up with an integrated use of a laserscanner and a trinocular vision system. The laserscanner only provides 2dimensional information. The vision system is capable of perceiving the environment 3-dimensionally. The goal of this paper is to develop and to apply sensor fusion techniques in order to improve the system's performance for 'mobile robot positioning' and 'ext)loration of unknown environments'. The approach is able to deal with static as well as dynamic environments. On different levels of t)rocessing geometrical, topological an(l semantical mo(lels are generated (exploration) or can be used as a priori information (positioning). The system's l)erformance is demonstrated for the task of building a geometrical model of an unknown environment. 2. Obtaining 3D descriptions of the scene Ill this section the reconstruction of the 3-dimensional scene with the trinocular vision system and the laserscanner is descril)e(|. As scene featurcs linear edge segments are used. These edge segments are reI)resented by nlidI)oint, direction-vector and half-length. The uncertainty of the segments is rel)resented by a covariance matrix [3]. The xz-I)lane is the ground-plan of the coordinate system and the y-axis represents the h(:ight. As the laserscanner only provides 2-(limensional data the y-coordinate is always equal to the height in which the sensor is mounted on the robot.
71
2.1. 3 D r e c o n s t r u c t i o n f r o m t r i n o c u l a r s t e r e o
The process to reconstruct scene features from camera images is relatively complex but it is possible to derive a 3-(limensional (lescription of the scene. The first step of stercoima,ging is the calibration of the, cameras. In [12] a photogrammetric aI)t)roach to highly accurate camera calibration of zo()m lenses is developed. The result of the calil)ration is the matrix MDLT which (lescril)es the transformation from scene- to image-coordinates for a camera. In homogeneous coor(linates this tra, nsformation is given by ( 'W"'U?' ) "W~'U~
i
ix/ y
"
-- MDLT
wi
Z
(1) "
1
In the presented system linear edge segments which are extracted fi'om the camera images in real-time are used as image and scene features. It is possible to reconstruct scene features if corresponding image features in at least two camera images are known. This means that the stereo correspondence 1)rol)lenl has to be solved. In [8] a trinocular stereo-matching algorithm using the eI)ipolar c()nstraint combined with a local n()rmalized cross-correlation technique has been developed. The stereo matching algorithm provides c()rresl)onding image I)oints (u i, v i) in the camera images. For the presented system it was ext)erimentally proved that the uncertainty of the matches can generally be estimate(t to be below one pixel. F()r the stereo-reconstruction of a scene point the following overconstrained linear system has to be solved: A
y
- b
~
(,)
p-
y
z
-(ATA)-'ATb.
(2)
z
In [5] it is shown that the uncertainty for the reconstruction of a scene point can be written in a first order approximation by a covariance matrix E p -- J E,,,,,
jT
with
J
0((ATA)-'ATb)
0(',,,~, v')
(3)
Are, asonable estimation for E~,,~ is given by Eu,,, = 1. In order to reconstruct a line segment the endpoints of the lille Pl and P2 are reconstructed. The equations 4 to 8 (lescril)e the representation of a line segment by midpoint, normalized dire, ctiou-vector and halflength and tile correspon(ling covariance matrices for the representation of the m~certainty. For a minimal reI)resentation there is no uncertainty ret)resente(1 for the halflength. m -
midpoint
(4)
r -
m-v2 [[PI --O2[[
normalized direction
(5)
l =
[[Pl-P2112
ha lflength
(6)
E m ---
Epl +Ep:~.4
(:()variance of midI)oint
(7)
Er -
r..~ +r.p22 [[Pl -p2ll
covariance of direction
(8)
-
-
p~+p22
Tlle state vector for a line segment in rol)ot coordinates is given by k r - (m, r, l) T
72 3D descriptions obtained by a laser scanner Figure 2 shows a laserscan that is ac(luired in a corridor environment. The sensor (lata 1)r()vi(led by the laserscanner are 2-(limensional (ground plan) so y-coordinate is always 0. In order to show the quality of the laserscan the cad-model of this environment is overlayed in grey lines. 2.2.
_:__=_.-_ .....................................~a~t.a1~,m.=~r
_-::-
:-=_-
,-.1
_ _ ..............I
Figur(; 2. raw data from laserscan
Figure 3.
.....
example for edge extraction by
iterative end point fit combined with leastsquare approximation
An experimental evaluation for tile accuracy of the laserscanner measurements was carried out with the result that within a distance of 10 meters the variance of the distance measurement and tile variance, 1)erp(',ndicular to the measuring direction can 1)e estimated to be
a~ - (2cm)2 - 4cm 2
'
a~ - d2 tan2 (0"25~ 2
"
(9)
From the scanner's polar ret)resentation of the measurements a cartesian rcpresentation is computed which results in the following covariance matrix for the uncertainty of a single scan point of the laserscanner.
((~) + alcos2((~,) (a'Z_L- a~)cos(t~)sin(c~)
)
The next step of processing is the extraction of linear edge segments fronl the laserscan. This is done by using the iterative end-point ,fit algorithm for the determination of points belonging to a line segment. A least square solution is applied to compute a symbolic representation as defined 1)y the equations 4 to 8 (see also [11]). This is displayed in figure 3.
73
2.3. T r a n s f o r m a t i o n t o w o r l d c o o r d i n a t e s The line segments are so far rel)reseute(l in robot coordinates. The state vector of the r()b()t is given by x~ - (x,., z,., qS) 't' For a c o m m o n robot indeI)endent ret)resenta.ti()u the t.ransformation to world coor(linates is necessary. This transformation is given t)y the t()llowing rotation and translatiou:
R -
0 - sin(C)
1 0 0 cos(qS)
,
T -
() zr
.
(1())
The transforination of the state vector in rol)ot coordinates k r = (m, r, l) t() the state v(,ct()r in world coordinates k w 1)e('.()lu(;s k w = ( R m + T, R r, 1). The 1)r()t)agatioIl of the uncertainty of a r a n d o m vector x with covariance matrix Ex lln(l(;r the transformation y = f(x) is in a first order aI)proximation given by ]~-] y
z
Of(x) Ox
~x X--~
Of(x)"' Ox
(11) X-=X
\Vitll this equation the ulw(;rtainty ()f the state vector iu worl(t coor(tinates [5] 1)eCOllW.s
y]~,,w __ I t Y],rn R T -~-
-0~-I11
~-]R
-0-~ Ill
-4- Y]T
(12)
and r. w - R ~ r R " ' +
r
~
(oR,), ,, -bT~
(13)
with E R a Il(1 ET beeing the llu(:('rl,aiuties in R and T. 3. E x p l o r a t i o n
II~ ()r(ler to explore an ellvir()llillent the rol)()t is 1)rovi(le(t with a topologicaJ m()(lel and a, (:('rtaiu)uission ((lirectiou and (listau(~(, t() travel) is sI)ecified. The geomet, ri('al worl(t m()(lel is 1)lfilt uI) incremeutally. The lo(,al 1)('.rcet)tion is matched with the gl()l)al m()(lel a u(1, if 1)()ssit)le, fused acc()r(liug t() ttl(', f()ll()willg section. 3.1. F u s i o n o f s y m b o l i c e d g e s e g m e n t s Iu this work linear edge segnlents a r(' rei)reseuted t)y midI)oint, nornlalized (lire('ti()n vcct.()l a u(l llalltength 1)e(:a.us('. this reI)r(,seutati()xl is advantageous for the fusiou of segments. 'l'his lIl(~/,IIS ~tll edge seguw.ut is (l(;Iine(1 1)y a state vector k = (71~.:~,mu, 7/~.z,r:,;, r.~, r~, l) '1' aTl(l l,h(' uucertainty is givelt 1)y the (:()variau(:r lua.trix Ek. I~l ()r(ler to lind ('.orresi)()u(ling s(:eue f(,atllres (nearest neighbor matching) in tll(; h)ca] 1)cr(:Cl)ti()u ;tnd the gl()t)al mo(lel the Matlalan()t)is distance is apt)lied. The Mahalall()l)is (lisl:al~(:(' is a distance crit,eri()u f()r tw() stat, e ve(:tors normalized by the sllm ()f their (:()Val'iance lnatIiC('s. The s(tlmre(l Nlahalall()l)is (lista, n(:e is given by ~(ko, kl) - ( k o
- k l ) T (Y]ko --[- ~]kl) --I ( k o -
kl).
(1,[)
74 The squared Mahalanobis distance defines a x2-distribution [2] for the means ko = k l . For a segment in the local perception the nearest neighbor in the global model is defined by the minimal Mahalanobis distance. The two segments can be fused using a Kalman filter. K :
Eo (Eo + ~ 1 ) -1
Kahnan-Gain
(15)
k :
ko - K (ko - kl)
fusc(l segment
(16)
E =
(I - K) Eo
covariancc matrix
(17)
Every time an edge segment is fllsed with a segment from the local t)erception its confidence is incremented by one [3]. For multiple observations it is so possible to reject segments from the world model whicll have only been observed once and could be dynamic objects or artefacts.
3.2. Ground plan exploration In the following example (figure 4) the robot has driven about 50 meters through a hallway environment and incrementaly generated a geometrical map using the lascrsca, nncr and the above described techniques.
h~
!ii
......................
Figure 4. sensor fllsion with laserscanner
The accuracy of tile generated real) is basically defined by the o(lometry of the robot. This means that this map could be used as a priori information for later navigation tasks in order to enable position correction.
3.3. 3-dimensional map b u i l d i n g In this second example an al)l)roach to fusing 3-dimensional stereo-reconstructions is presented. In this case the rol~ot is h)cate(l in a hallway corner. In figure 6 the rcconstruc-
75 tions from a sequence of stereo triples (tigure 5) are displayed, again overlaye(1 with tile exact ca(t-model of the environment. Here the 3-D reconstruction is fully automatically (:omtmted with an accuracy of • f()r the vertical edges.
Figure 5. camera image from sequence of stereo-tril)les
Figure 6. reconstruction from a sequence of stereo-triples
The stereo vision system is tile most I)owerflfl sensor that is currently used on PR,IAM()S. It requires more effort to reconstruct scenes by stereo than with a laserscanner though.
4. Conclusion In this paper it was shown that stereo-reconstructions and planar laserscans can be ret)r(:sented in a coInmon format. This enables the fusion of complementary sensor information. Experimental results w(',re 1)resente(l for 3-(limensional mat) building ()f an lmknown environment. More mathematical details can be found in [11]. Th(' accuracy of th(, generated model enal)les using this model for position correction during later navigation tasks. It will also be the goal to derive topological and semantic knowlege from the geometrical reconstructions using generic ol)ject models. The planar laserscanner will s()(m be replaced by a 3-D scanner that provides (leI)th images with a resolution of 128 • 64 1)ixels at a rate of 5Hz. This step will again improve the perf()rmance of the t)resente(l system, especially for 3-D modelling.
Acknowledgement The authors would like to thank F.Al)egg and C.Wetzel for their contributions to the iml)l('.mentation of this work. This work was 1)erformed at tile Institute for Ii,eal-Tinle Conlplll,er Control Systems & Rob()tics, Prof. Dr.-Ing. U. Rembold and Prof. Dr.-Ing. 11. Dillmann, Department of Computer Science, Uniw'xsity of Karlsruhe, 76128 I(a.rlsruhe, Germany.
76 REFERENCES 1. F. Arman and J.K. Aggarwal. Model-based object-recognition in dense range images - a review. A CM Computin, g Surveys, 25(1):5- 43, 1993. 2. J.L. Crowley. Principles and Techniques for Sensor Data Fusion, volume 99 of NATO AS I, Multisensor Fusion for computer Vision, chapter 1, pages 15-36. J.K. Aggarwal, 1989. 3. J.L. Crowley, P. Stehnaszyk, and P. Skordas, T. Puget. Measurement and integration of 3-D structures by tracking edge lines. International Journal of Computer Vision, 8(1):29- 52, 1992. 4. T. Endlinger and G. WeiB. ExI~loration, navigation and self-localization in an autonomous mobile robot. In Autonome mobile Systeme, pages 142 151. SpringerVerlag, 1995. 5. O.D. Faugeras. Three-Dimensional Computer Vision. MIT-Press, 1995. 6. L. Feng, J. Borenstein, and H.R. Everett. Where am I? Sensors and Methods for Autonomous Mobile Positioning. University of Michigan, 1994. 7. J.J Leonard and tt.F. Dllrrant-Whyte. Directed sonar sensing for mobile robot navigation. MIT-Press, 1992. 8. P. Steinhaus. Bin- ~md trinokulare Matchingalgorithmen t)asierend auf Korrelationsverfahren und epipolarer Geometrie, 1996. Studienarbeit, Institut fiir ProzeJ]rechentechnik und Rol)otik, Universit/it Karlsruhe, Deutschland. 9. E. Triendl and D. J. Kriegman. Stereo vision and navigation within tmildings. In bdcrnational CoT@fence on Robotics and Automation. IEEE, 1987. 10. P. Weckesser, G. Al)penzeller, A. von Essen, and R. Dilhnann. Exl)loration of the environment with an actiw; and intelligent optical sensor system. In b~.ternational Conference on Intelligent Robots and Systems; Ituman Robot Interaction and Cooperative Robots. IR,OS, November 1996. 11. P. Weckesser a.nd I{. Dillmann. Sensor-fusion of intensity- and laserrange-images. In MFI, 1996. 12. P. Weckesser and G. Hetzel. Ph()togrammetric calibration methods fl)r an active stereo vision system. In Prof. A. Borkowsky and Prof. J. Crowley, editors, bt.tclligent Robotic Systems (IRS), pages 326333, 1994.
D
IMAGE CODING AND TRANSMISSION
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
79
Time-Varying Image Processing for 3D Model-Based Video Coding 1 T. S. Huang, R. Lopez, A. Colmenarez University of Illinois at Urbana-Champaign ~05 N. Mathews Ave., Urbana, IL 61801
Abstract This paper presents a framework for tracking moving features using a 3D model and an analysis-by-synthesis approach. First, the global pose is estimated with 2Dto-3D position correspondences of rigid features. Next, non-rigid features are tracked after compensating for the global pose. Our approach is tested in the context of facial feature and head pose tracking. Results confirm that the system is robust, accurate, and can be implemented in real-time. This framework finds applications ranging from model-based video coding to facial expression analysis to face recognition using video sequences.
1
Introduction
Time-varying imagery has been increasingly prevalent in computer vision and image processing. For computers to be able to interact with man, it is vital that they be able to analyze such data and extract higher level knowledge of the world they perceive. One of the most basic tasks, which has produced a wealth of research, is the "simple" act of recognizing and tracking objects in a time-varying scene. While this is intuitive to humans, it has been shown many times to be a formidable task for a computer in all but very limiting scenarios. Indeed, much of the work in automatic object tracking has sought to relax the constraints under which such systems will produce accurate results. Image understanding and motion analysis of video sequences are two aspects of computer vision that connect and relate to a number of applications ranging from video annotation to video coding to human computer interfaces. In most 1 This work was supported in part by Joint Services Electronics Program Grant N00014-96-1-0129 and an AT&T Bell Labs Fellowship.
80
scenarios, objects are not rigid because their motion, with respect to the scene, is combined with deformations and changes in light conditions. A framework for automatic detection and tracking of moving objects in such complex conditions is very important in time-varying imagery, and would provide a tool for further analysis and understanding of the higher level aspects of the observed scene. In early work, detection and tracking of moving objects was carried out in limited scenarios and only superficial understanding of the scene was achieved [1-4]. Low level modeling such as snakes, active contours, and deformable templates introduced some improvements [5-7]. However, to obtain a robust and complete analysis of the objects and their motion, high level modeling is required. Note that in approaches such as analysis-by-synthesis, the scene understanding is limited by the model's capability to represent the scene. A particular example of major importance in time-varying imagery is the detection, tracking, and motion analysis of human faces. Two distinctive applications are: (i) very-low-rate video coding, and (ii) human-computer interfaces. In the former, motion analysis is used to improve the motion compensation used in compression schemes that code the displaced-frame-difference [9-11], and to drive the synthesis of the video sequences in pure model-based schemes [8]. In human-computer interfaces, the motion analysis of the face is useful for gesture recognition [12], lip reading [13], and other high level tasks. In this paper we present a scheme for model-based tracking of global and local motion that is robust, accurate, and can be implemented in real-time. The system consists of three modules acting in a feedback loop: (i) 3D Object Modeling, (ii) 2D-3D Pose Estimation, and (iii) Synthesis-based Template Matching. Because the scheme performs analysis-by-synthesis using a 3D model, the system overcomes many of the common problems such as error accumulation over long sequences, light conditions, and occlusions. We demonstrate its use in the context of facial feature tracking with both synthetic and real image sequences. The synthetic sequences are helpful in examining the accuracy of the system with respect to ground truth. The real images sequences help test the robustness of the algorithm is practical applications.
2
S y s t e m Overview
As indicated in the block diagram in Fig. 1, the system consists of three modules acting in a feedback loop: (i) 3D head modeling, (ii) feature detection, and (iii) pose estimation. Given a predicted head pose, the 3D head model provides a synthetic view from which templates are made. Features are then detected via template matching using the newly generated templates. Finally,
81
the 2D feature positions and their corresponding 3D positions in the head model are used to estimate the new head pose. Kalman filters are used to predict the head pose and the 2D feature location using previous measurements. Note that if this procedure is applied repeatedly on the same frame, it can be considered an iterative approach to refining the current head pose estimation. C~borscan Data 3D
"---
Features
KalmanPredictor
1~1
[
Render
) ~ [Synthetic View, and Feature Positions
/~
Predicted Pose
[
Estimated Pose I ~--- I PoseEstimator 1
I KalmanPredictorI ~
[ Feature Position l
( TemplateMaker ]
[Predicted Feature Position Video Sequence
I
/
Fig. 1. Feature-Based Global Head Pose Tracker: The Block Diagram. The system is initialized assuming a front view in the first frame, and the rough location of the facial features; the visual pattern recognition technique in [14] provides these initial facial feature locations. A feature-based pose estimation algorithm relies on the precise detection of the feature positions. Small errors in the feature locations can produce large errors in the pose estimation. Good spatial localization is achieved with weightedtemplate matching; the match error is weighted with a Gaussian function centered at the position of the features. Since the 3D location of the facial features are known from the head model, no error in the feature locations is accumulated over the sequence.
3
Head Modeling
For the purpose of this research, the object modeling, or more specifically, head modeling, was performed with a 3D Cyberware range scanner, Fig. 2. While these scans provide detailed geometric data for a specific subject, it is somewhat intrusive and requires the subject to be available for the scan. Preliminary experiments, however, indicate that crude, low-resolutional models may be sufficient and that it may be possible to use an "average" head scan or a generic 3D head model. With the use of texture mapping techniques, the high resolution input image can be mapped onto a low resolution geometric
82
Fig. 2. The Cyberware head scan data. (a) Color data. (b) High-resolution range data. (c) Low-resolution wireframe used in tracking. surface and still provide accurate renderings of the necessary templates for a wide range of 3D poses. As part of the initialization, a texture for the generic model is obtained from the first frame by aligning the facial features with their correspondences in the 3D model.
4
Pose Estimation
One of the main steps in the proposed system is the update of the head pose from the corresponding 2D-3D facial features. A large set of 3D features are extracted manually from the initial head scan and a smaller subset of 2D features are obtained from the feature tracking module. Using these correspondences, we can compute an alignment transform that maps the 3D features to their 2D locations [15-17]. One difficulty is that, since only 3 feature points are used, the pose estimation results in 2 mathematically equivalent transforms, only one of which is correct for our purposes. We limit ourselves to 3 features (2 eye corners and nose tip) because of rigidity constraints and also to reduce complexity. However, a 4th feature point (mouth center) can be roughly approximated and used to resolve the ambiguity in the transforms. Since we know the 3D location of this 4th feature, the two resulting transforms can be applied to this point and compared to the estimated 2D position. The correct transform will result in a significantly smaller mapping error. Our pose computation module is based on the work in [17] where 3 points in each the model and image are used for object alignment. The main underlying assumption made in this method is that 2D images of an object obey a weak perspective imaging model. Using this assumption we can apply an affine approximation to this model and obtain a closed form solution to the pose estimation problem.
83
5
Experimental Results
Several sequences were tested with the proposed system, two of which are presented here. The sequences were captured at 30 fps and 320x240 resolution. The head models for each subject were created with the range scanner and the necessary 3D features were extracted manually. In the sequence of Figure 3, three rigid points were used for the pose estimation: the two outer eye corners and the middle of the nostrils. Several non-rigid points, the tips of the eyebrows and mouth corners, were also tracked successfully. The images in Figure 4 show the results of wireframe tracking. The 3D wireframe at the computed pose is overlayed over the original image. We also tested our system with synthetic video sequences to determine its accuracy. Figures 5-7 show a comparison between the measured pose parameters and the ground truth for the angles about the X, Y, and Z axes. Each figure shows plots for the optimal, Kalman filtered, and ground truth values. Many more results can be obtained from the authors as MPEG video clips, or check ftp://ftp.ifp.uiuc.edu/rlopez/00INDEX.
6
C o n c l u s i o n s and F u t u r e W o r k
In this paper we have presented a robust and novel approach to real-time feature tracking using a 3D model-based framework. A small set of facial features were tracked successfully over a large range of head motion. Global rigid motion was recovered in the form of three angles and a scale factor. Nonrigid points were also tracked successfully after compensating for the computed global motion. The combination of the 3D model, head pose estimation, and texture mapping avoided the error accumulation problem and allowed better localization of the features. Future work includes more detailed modeling of the head to aid in the analysis of local facial motion such as expressions, gestures, and deformations (wrinkles, etc.). Also, constraints on the movement of nonrigid model points need to be developed to allow for the estimation of actual 3D motion vectors from 2D trajectories.
References
[1] I. K. Sethi and R. Jain, Finding Trajectories of Feature Points in a Monocular Image Sequences, PAMI, Jan 1987.
84
Fig. 3. Feature Tracking Results: 10 points tracked: 3 rigid points used for the pose estimation + 7 non-rigid points. Selected frames are shown. [2] D. Huttenlocher, J. Noh, W. Rucklidge, Tracking Non-rigid Objects in Complex Scenes, ICCV 1993. [3] A Framework for Real-Time Window-Based Tracking Using Off-The-Shelf Hardware, [4] Y. Yao and R. Chellappa, Dynamic Feature Point Tracking in an image sequence, IEEE Int. Conf. Pattern Recognition Oct 1994 [5] F. Leymarie and M. D. Levine, Tracking Deformable Objects in the Plane Using an Active Contour Model, PAMI Jun 1993 [6] C. Kervrann and F. Heitz, Robust Tracking of Stochastic Deformable Models in Long Image Sequences, IEEE Int. Conf. Mach. Intel. Jun 1993
85
Fig. 4. Wireframe Tracking Results
Fig. 5. Synthetic Sequence: Optimal, Filtered and Actual Angles (X axis)
Fig. 6. Synthetic Sequence: Optimal, Filtered and Actual Angles (Y axis) [7] F. G. Meyer and P. Bouthemy, Region-Based Tracking Using Affine Motion Models in Long Image Sequences, CVGIP:Image Understanding Sep 1994. [8] K. Aizawa and T. S. Huang, Model-Based Image Coding: Advanced Video Coding Techniques for Very Low Bit-Rate Applications, Proceedings of the IEEE Vol. 83, Feb. 1995
86
15
, x
Optimal
.....
Filtered
10 ~
,
Real
Frame
#
Fig. 7. Synthetic Sequence: Optimal, Filtered and Actual Angles (Z axis) [9] Y. Altunbasak, A. m. Tekalp, and G. Bozdagi, Two-Dimensional Object-Based Coding Using a Content-Based Mesh And Affine Motion Parameterization, [10] C. Toklu, A. Erdem, M. Sezan, and A. Tekalp, 2-D Mesh Tracking For Synthetic Transfiguration, [11] Y. Wang and O Lee, Active Mesh- A Feature Seeking and Tracking Image Sequence Representation Scheme, IEEE Tran. Image Processing Sep 1994 [12] I. A. Essa and A. Pentland, A Vision System for Observing and Extracting Facial Action Parameters, CVPR 1994 [13] D. Stork and M. Hennecke, Speechreading: An Overview of Image Processing, Feature Extraction, Sensory Integration and Pattern Recognition Techniques. Int. Conf. Automatic Face and Gesture Recognition 1996 [14] A. Colmenarez and T. S. Huang, Maximum Likelihood Face Detection, Int. Con/. Automatic Face and Gesture Recognition 1996 [15] Ricardo Lopez and Thomas Huang. Head pose computation for very low bitrate video coding. In Vaclav Hlavac and Radim Sara, editors, Computer Analysis o/Images and Patterns, pages 440-447, Prague, Czech Republic, September 1995. Springer. [16] Ricardo Lopez and Thomas Huang. 3d head pose computation from 2d images: Templates versus features. In IEEE International Conference in Image Processing, pages 220-224, Washington DC, USA, October 1995. IEEE, IEEE Press. [17] Shimon Ullman and D. P. Huttenlocher. Recognizing solid objects by alignment with an image. International Journal of Computer Vision, 5(2):195-212, 1990.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
87
A New Arbitrary Shape DCT for Object-Based Image Coding Masayuki Tanimoto and Mario Sato Department of Information Electronics, Faculty of Engineering, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-01, Japan A new DCT-based scheme is proposed for object-based image coding. It expresses pixels in arbitrary shape regions as linear combination of a part of DCT basis functions. The number of basis functions used here is the same as that of the pixels to be encoded. Computer simulation shows that the proposed scheme is superior to Region Filling DCT and conventional block-based DCT. 1
INTRODUCTION
Transform coding for arbitrary shape regions is one of the key issues of object-based image coding. Such schemes are divided into two categories. The first category uses basis functions of orthogonal transform depending on the shape of the region. Arbitrary Shape KLT (AS-KLT) [1] and Arbitrary Shape DCT (AS-DCT) [2] [3] come into this category. The second category uses basis functions of conventional block-based orthogonal transform such as DCT. Region Filling DCT (RF-DCT) [4], Shape Adaptive DCT (SA-DCT) [5] and Region Support DCT (RS-DCT) [6] come into this category. The schemes of the second category are discussed here. RF-DCT makes rectangular blocks by filling the outside of the region, and applies the conventional two dimensional DCT to the blocks. So, the number of samples to be coded is increased. In SA-DCT, one dimensional DCT is applied to each horizontal line, shifting the coefficients to low frequency side and one dimensional DCT is applied to vertical lines. In this scheme, the shift of the coefficients decreases vertical correlation. In RS-DCT, pixels in the region is expressed approximately by using a limited number of basis functions of DCT. This paper proposes a new arbitrary shape transform coding scheme of the second category. Pixels in the arbitrary shape region are expressed exactly by using the same number of basis functions as that of the pixels. The coding performance of the proposed scheme is examined. 2
2.1
PROPOSED
SCHEME
Outline
Figure 1 shows the outline of the proposed scheme, bl(i,j) denotes the values of pixels in the region to be coded. First, the outside of the region is filled with 0 to make a rectangular block. Then, block-based orthogonal transform is applied to the block. Components Xl(k,1) whose number is the same as that of the pixels in the region are selected. The selected components Xl (k, l) are converted to Yl (k,/), and components x2(k,1) which are not selected are substituted by 0. Thus, a new rectangular block is made in the transformed domain. The conversion of Xl(k, 1) to yl(k, 1) is determined so that the inverse orthogonal transform of this new rectangular block reconstructs the
88
OT
< Orthogonal~ Transform/
x I (k,.l) I
tc:mp~
A
be~kl,
i)
redundant I components
conversion
I bl (i, j) I I
decoded pixels
-_ IOT ( Inverse ~
y 1 (k,l)
components ]
~Orth~176 I \Transform/
Figure 1. Outline of proposed scheme.
values of pixels bl (i, j). b2(i, j) which are the values of pixels in the outside of the region are not used. In the proposed scheme, the number of samples is not increased. The pixels bl(i, j) in the region are reconstructed perfectly if the coefficients yl (k.1) are not quantized.
2.2
Derivation of the conversion equation The equation for the conversion of Xl(k, l) to yl(k, l) is derived. As shown in Figure
2, $1 denotes the region to be coded in the pixel domain and S~ denotes the region of components to be coded in the transformed domain. Sl (i, j) is a sampling function taking the region S1 and expressed as Sl (i
' J)
_ [ 1 (i,j) 9 [0 else.
(1)
t
$1
region of pixels to be coded (a) pixel domain
region of / components// to be co~//
(b) transformed domain
Figure 2. Sampled region in each domain.
89
Defining Ckl(i, j) as basis fllncti()ns of the ()rthog(mal transfl)rln,
(2)
= Xl(k,/)
(sl(i,j)r
holds for (k, l) C S'1. From the inverse transfrom, S l(i, j)b~(i, j)
=
sl(i, j)
Z
(k,,t,)~s',
y, (k', l')r (i, j) (~1,/1)~S/I Yl (]~" /t) {Sl(i, j)r (i, j)}
(3)
holds. Substituting (3) to (2), a set of equations
Z
(Sl(i,j)~kl(i,j),81(i,j)~k,l,(i,j))" yl(k',l')
= Xl(k,l)
(4)
(k',l')~S', are obtained. Solving (4), 'gl(k,l) is obtained from Xl(k, l) in case that the determinant given by
det = I(sl(i,j)r162
(5)
is not equal to O. 3 3.1
CODING
SIMULATIONS
Conditions
Computer simulation of the proposed coding scheme is performed. Figure 3 shows encoding and decoding process used here. Before orthogonal transform is applied, DC component in the region $1 is removed to decrease the discontinuity of pixel values at the boundary. Blocks including the boundary are chosen, and both sides of the boundary are coded in each block. 8 x 8 DCT is used as the orthogonal transform. Bit allocation for the quantization of each coefficient in the transformed domain is determined by considering the power of the coefficients. Figure 4 shows the algorithm of selecting basis flmctions. Spatially localized basis are not orthogonal though orthogonal basis functions are desirfunctions sl(i, j)r able for high coding performance. This algorithm selects basis functions which are as nearly orthogonal as possible. 3.2
Results
Examples of selected basis functions are shown in Figure 5. Gray boxes in the upper blocks denote pixels to be coded. Gray boxes in the lower blocks denote selected basis functions. The selected basis functions are distributed though the pixels to be coded are localized. Figure 6 shows the dependence of SN ratio of the reconstructed picture on entropy. The entropy of the proposed scheme and RF-DCT includes the information of selected basis functions, but does not include the shape information. As seen from this figure, SN ratio of the proposed scheme is about 3dB higher than that of RF-DCT at the same entropy.
9O 0
0 P removal
DC
orthogonal
of
conversion
transform
component
and
selection
of
components
pixel
domain
~
transformed
domain
quantization
[encoder] [decoder]
0
0 9
4 addition
DC
of
component
q
selection
inverse
orthogonal
of r e g i o n
pixel
transform
domain
~
transformed domain
Figure 3. Encoding and decoding process.
To achieve the same SN ratio, the proposed scheme needs entropy about l bit/pixel less than the conventional DCT. The shape information by chain coding is estimated to be about 0.3bits/pixel. Therefore, the proposed scheme is superior to the conventional DCT even if the shape information is included. 4
CONCLUSION
We proposed an image coding scheme which expresses an arbitrary shape region by using block-based orthogonal basis functions. High coding performance is obtained by proper selection of the basis functions since it uses the same number of basis functions as that of pixels to be coded and the selected basis functions are nearly orthogonal. ACKNOWLEDGEMENT The work reported in this paper was supported in part by the Grant-in-aid for General Scientific Research (B), No.07455159, 1995-1996, the Ministry of Education, Science and Culture.
9!
START
Select [unction
a basis having
the largest coefficient as a candidate
No
Subt the component in the pixel domain
Yes Adopt the basis function
L
~ A b a n d o n the ~ basis function
J
No
Figure 4. Algorithm of selecting basis functions.
Figure 5. Examples of pixels to be coded and selected basis functions.
92
34
32
30 propo ,_..., m
2
n-" Z cO
/,"
28
,d"
26
24 ,," ~,,"
22
....... conventional DCT
,,"'"
0
.'
..
,
i
|
i
|
,
|
|
,
0.2
0.4
0.6
0.8
]
1.2
1.4
1.6
1.8
entropy [bits/pixel]
2
Figure 6. Dependence of SN ratio on entropy in various schemes.
REFERENCES
[1] I. Matsuda, S. Itoh, T. Utsunomiya, "Adaptive KL-Transform Image Coding Based on Variable Block Shapes", THE TRANSACTIONS OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, B-I Vol. J76B-I No. 5, pp.399-408, May 1993. [2] M. Gilge, T. Engelhardt, and R. Mehlan, "Coding of Arbitrary Shaped Image Segments Based on a Generalized Orthogonal Transform", Signal Processing Image Communication, 1, pp.153-180, Oct.1989. [3] Y. Kato, "A Study of Orthogonal Transform for Arbitrary Shape Image Segment", 1990 AUTUMN NATIONAL CONVENTION RECORD, THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, D-326, p.328, Oct.1990. [4] N. Ito, H. Katata, H. Kusao, "A Method of DCT for Arbitrary Shape", The Proceedings of the 10th Picture Coding Symposium of Japan, 5-2, pp.77-78, Oct. 1995. [5] T. Sikora and B. Makai, "Shape-Adaptive DCT for Generic Coding of Video", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 5, NO. 1, pp.59-62, Feb 1995. [6] Y. Shishikui, "A Study on Arbitrary Shape Coding using Region Support DCT", TECHNICAL REPORT OF IEICE, CS95-157, pp.61-66, Dec.1995.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
93
Picture c o d i n g u s i n g splines M.Buscemi, R.Fenu, D.D. Giusto, and G.Liggi Dept. of Electrical and Electronic Engineering, University of Cagliari Piazza d'Armi, Cagliari 09123 Italy email : [email protected]
A new interpolation algorithm for 2D data is presented that is based on least-squares minimization and splines. This interpolation technique is then integrated into a two-source decomposition scheme for image data compression. The major advantages of this new method over traditional block-coding techniques are the absence of the tiling effect and a more effective exploitation of interblock correlation.
1. INTRODUCTION Although 1-D spline functions have important applications in many areas of scientific and engineering research, they are frequently insufficient to meet the need of the encoming recent technological developments. For instance, to represent a mathematical model described by several parameters and to interpret higher dimensional data, functions of two or more variables are often required. Among these, the family of piecewise polynomial functions in more than one variable, which are usually called multivariate splines, is the most useful in many applications. Moreover, the subject of multivariate splines [1] has become a rapidly growing field of mathematical research. Using spline-methods it is possible to carry out interpolations of discrete points: the major advantage of these algorithms for image data compression lies in the possibility of reducing the tiling effect typical of simple block coders and in the gradualness of changes between adjacent pixels. Before expounding the content of this paper, it is important to underline that very often, for natural images, the hypothesis of sufficiently high correlation and 1st order Markov process is not worth; so an higher approximation fidelity can be achieved by optimising the interpolation algorithm using special parameters as variance, covariance, standard deflection, and so forth.
2. THE CODING STRATEGY In order to interpolate images, a new strategy has been adopted, based on two-source decomposition [2]. The first source is related to the low spacial frequencies, while the second one allows to extract the residual information, which refers to the high spacial frequencies. Two different techniques have been improved for each component: they are the spline
94 interpolation and the entropic analysis, respectively. The sources are elaborated in an independent way, but in the phase of decoding the reconstructed image is obtained by their recombination through a specific algorithm.
2.1. Image data interpolation The interpolation of a source requires the definition of a model, which can be either deterministic or stocastic, depending on whether it uses casual variables or not. It is reasonable to interpret several neighbouring points of an image, as belonging to a polynomial function. Using 3rd degree functions with two variables the generic pixel value z=f(x,y) can be interpolated by z = c 1 + c2Y + c3x + c4xY + c5Y 2 + c6 x2 + c7y2x + c8Yx 2 + c9Y 3 + Cl0x3
(1)
When the number of points is greater than the number of polynomial coefficients, one strategy of minimization of error is unavoidable: if the MSE is chosen as indicator-parameter, the resulting function could not pass exactly in the interpolation points, but minimizes the MSE [3]. This objective is achieved by solving a linear system expressible in the matrix form, where [A]is a 10x 10 symmetrical matrix; c is the vector of the polynomial coefficients (the unknows); t is the vector of the known terms and depends on the image pixels. This tecnique can be successfully adopted until pixel values are not abruptly varying and pixel number is not exaggeratedly major than the degree-freedom of the polynomial functions. These problems define the application limits.
2.2. Least square 2D spline interpolation Spline function and least-square block interpolation present, both, a contrastanting behavour: the first one links excellently adjacent blocks but induces an evident effect of smoothing with loss of settlement; the last one approaches in a correct way but only locally, giving rise to tiling effects. Appreciable results can be attained by combining these two methods for the whole image or, better, for the low frequency component. The initial strategy (in a second moment it has been abandoned) was based on total links about adjacent blocks for continuity, 1st and 2nd derivatives for each directions (x,y). Calling c 1 ' c0.,2 c 3. the coefficients of three adjacent blocks and x f and y f their extreme position
geometrical values, a set of 18 equations can be written:
' f3 + c6x ' f + 4xi clOX
+cl .
0;.
.
4x} +clxl +cl 4 o .
...
1 f3 +c5y 1 f2+ c l y f + c ~ _ c 3 1 =0 ; c9Y
(2)
c7Y 1 f2+ c l y f + c { - c 3 = 0
Applying totally the MSE technique, a linear system results but there are superabundant equations, so a QR factorization (q orthogonal, R triangular) must be obtained for getting the solution (among the infinite possible solutions) which minimizes
II[A]c-tl12,where [A] is a
In x n] matrix, and c and t are [n x 1] vectors. Although this method could be highly correct, the results are not excessively good because the innumerable constrins on the edges produce insufficient approximation in the central part of each block. So a second strategy has been
95 developed inverting the order of application of spline-approximation and SE minimization. A local SE minimization is applied for the first block of the image and, only after this operation, the new pixel values (derived by the polynomial function) come into play for the following block. The process is iterative and the links are warranted by the method of minimization of the SE which examines both the pixels of the new block and (this is the real innovation) the last adjacent pixels previously calculated. This is equal to write a linear system [4] for each block [A]c = t , where [A] is a symmetric matrix in the form (n+l) 2
EZy
ZZx
... ...
ZZ y
EZ y 2
ZZxy
ZZx
ZZxy
ZZx
,
.
2
...
.
and [t] a vector (I is the pixel) in the form Z
I /x
but now both [A] and [t] have new limits for their summations. They are extended for containing a convenient number of external pixels: N+ot N+~ Z... Z... ~, [3 estimated according to the position held in the image 0 0 The sequence of analysis of the blocks of the whole image is illustrated in Figure 1, which points out the overlap between the marginal parts of adjacent blocks.
Figure 1. The sequence of analysis of the blocks in the whole image; the overlap between the marginal parts of adjacent blocks is shown.
96 This overlap allows to extract a partial reconstructed image with a very low presence of the tiling effect although the details are not perfect. The objective to achieve trough the low frequency image is the elaboration of a prototype which will be improved by the residual image, but with good connections.
2.3. Residual coding Through the analysis of the pixel entropy, two similar methods have been created for the memorization of the residual component. The difference can be summarized considering that the first one presents a specific quantization matrix, the second one is based on a tree code (arithmetic coding [5]). The procedures based on the entropy are then applied to the residual information. Since the correlation is high, the final image can possess a bit-rate compression higher than 40:1 (8bpp for the original image).
3. RESULTS The analysis of reconstructed images shows that the Least-Square Spline Interpolation method achieves PSNR values over 30dB if bitrate is almost 0.20 bpp. This confirms the high fidelity between the original image and the reconstructed one; in fact, the residual is sufficiently restricted so a good visual quality is observed. The most problematical zones as textures and edges do not give rise to unpleasant effects for each value of bpp. In Figure 2, images are respectively: the original one, the low pass, the correct residual, the codified residual and the reconstructed one with bpp .20. In order to point out the performances with different compression-rates and observe the gradual improvements, a particular of the whole image is provided in Figure 3 with the following values of bitrate: 8 (the original), 0 . 2 0 , 0 . 2 5 , 0 . 3 0 , 0 . 4 0 , 0 . 5 0 bpp. After this graphic presentation of the results, it is very interesting to make a comparison between the techniques which we have developed and other two important methods: the JPEG standard [7] and the interpolation with B-splines [8]. These last two techniques have a range of bitrates less wide because they are not able to achieve considerable compression rates owing to their operation limits. Our methods present the best results until a compression rate equal to .25 bpp. In Figures 4 and 5 PSNRs for the various methods are reported relevant to different values of bitrate.
REFERENCES 1. C. K. Chui. Multivariate Splines. CBMS 54, Siam, 1988. 2. J. K. Jan and D. J. Sakrison. Encoding of images based on a two-component source model. IEEE Trans. Comm. COM-25(11), 1977, pp. 1315-1322. 3. S. Van Huffel and J. Vandewalle. The Total Least Squares Problem, Computational Aspects and Analysis. Siam, Philadelphia, 1991. 4. D. Bini, M. Capovani, G. Lotti, e F. Romani. Complessita' numerica. Boringhieri, Torino, 1981. 5. G. C. Langdon. An Introduction to Arithmetic Coding. IBM J. Res. Dev., vol. 28, March 1994, pp.135-149.
97 6. A. M. Mood, F. Graybill, and D. C. Boes. Introdizione alla statistica. McGraw-Hill 1988, pp.481-500. 7. R. J. Clarke. Transform Coding of Images. Academic Press, London 1985. 8. Carl de Boor. A Pratical Guide to Splines. Applied Mathematical Sciences vol. 27, Springer-Verlag, 1978.
Figure 2. Results obtained by the proposed approach.
Figure 3. Details at different bitrates.
98
1"1~ JPEG
mllmlll~~~
"161"181201"251"3~176176176 ~ ~
Etllm I~
~
~J'~
1.s.spline entr. 1.s.spline arithm. 13-spline Figure 4.
Figure 5.
nh,mlllmL~~l
: ~ rata ~
~W~~]~:o~
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
99
A 10 k b / s V i d e o C o d i n g T e c h n i q u e B a s e d o n S p a t i a l T r a n s f o r m a t i o n
*
Stefano Bonifacio, Stefano Marsi and Giovanni L. Sicuranza DEEI, University of Trieste via A. Valerio, 1 0 - 34127 Trieste- Italy This paper describes a method for coding QCIF image sequences at 10 kb/s and 10 frame/s. The proposed algorithm performs motion compensation between consecutive frames using a spatial transformation. It is divided in two steps: the first one consists of the motion estimation and compensation, the second one includes the prediction error processing and transmission. This technique is of help in overcoming the limits presented at very low bit-rate by traditional hybrid block-based techniques. 1. I N T R O D U C T I O N Recent developments in video coding, aiming at video transmission on wireless communication and telephone channels, point out the inadequacy of coding techniques based on block-matching and bidimensional discrete cosine transform at very low bit rate (< 16 kb/s). In fact, the coarse quantisation of the DCT coefficients, and the limits on the number of blocks which can be correctly updated, cause serious degradations on decoded images introducing the so-called mosquito and blocking artifacts. One solution, to overcome these problems, can be obtained using a spatial transformation to realize a motion compensation in the sequence. Considering the high temporal correlation between two consecutive frames in the sequence, the actual frame can be predicted from the previous one using a spatial transformation and the knowledge of the motion in the scene. Of course using this technique not all the possible movements in the sequence can be reconstructed so this methodology must be, of course, integrated by a second step in which the reconstructed error is evaluated, compressed and transmitted to the decoder. 2. T H E P R O P O S E D
ALGORITHM
The main goal of this project has been the development of an algorithm for video sequence coding working at 10 Kb/s for very low bit-rate applications like video-phone tele-conference or tele-surveillance. Focusing to these applications some hypotheses must be assumed; in particular the sequence to code should have only slow motions inside, no change scene and no panning or zooming of the camera. The proposed algorithm is composed of two fundamental steps: in the first step specific motion estimation and compensation technique is used to predict the actual frame from the previous one while in the second one the predicted error is processed and transmitted *This work was partially supported by "MURST" and "ESPRIT-LTR 20229 Noblesse" projects.
100 2.1. M o t i o n e v a l u a t i o n a n d c o m p e n s a t i o n The spatial transformation-based approach [1-3] achieves a compensation of the movements in the scene warping the previous frame, i.e. the nth predicted frame I n ( X , y ) is N synthesised from the decoded previous one I n - l ( X , y). The process can be written as
]n(X,y)- In_l(Xt, y I) X'
--
y~
--
ailx + ai2y -4- ai3 + aisy + ai6
(1)
(2)
ai4x
where (x', y') is the estimated position in the previous frame for the current (x, y) pixel. This relationship is modelled by means of the geometric atone transformation, as described in (2) with six motion parameters a i l . . . a i 6 . The image is divided in non-overlapping triangular equilateral patches, so we have to estimate the motion parameters for each of them and send this information to the decoder, where the warping process is performed. Since the transformation offers six degrees of freedom, if we estimate the motion vectors (MVs) of the three vertices, and use (2), the six parameters can be determined. In (2), the i subscript for the parameters specifies that they are referred to the ith patch. When (x', y') is not a sampling point of the image (e.g. if x' or y' are not integer), we use the intensity value of the nearest one. The prediction for the whole frame needs the estimation of the motion vectors of all the vertexes of the patches, that therefore make up a grid of regularly placed points, called "grid-points"(GPs). In the next step all the MVs must be evaluated. This can be done through two different methodologies: the MVs can be evaluated using an optimisation process which allows to minimise the error between the predicted image and the original one; even if this technique is able to reach the highest quality in the predicted image, the evaluation of the MVs uses an iterative algorithm that, cycling for all the possible MVs configurations, reaches the better one. Of course the computation complexity in such a case is so high that a real time application for the system could be very expensive. A cheaper solution can be reached if we can consider the real movement for all the grid points. In such a case we accept the hypothesis that the movement of the grid points is sufficient for the reconstruction of the motion in the sequence. In this paper we adopt this solution, which requires a low computation level, even if the result is a sub optimal predicted frame. Describing in detail the process, for each grid point, a MV is estimated with a classical block matching (BM) technique [4]: it is searched the best matching block of pixels in the previous frame, which minimises the mean absolute difference in a search region centred on each GP; the corresponding displacement is the estimated vector for the GPs. For simplicity, according to the hypothesis of no motion of the camera, we do not perform BM on the GPs disposed on the border of the frame: the points on the four corners are assumed to be motionless, while the other GPs on the borders are allowed to move only along the border of the picture, and their MVs are deduced from the MVs of their corresponding closest GPs inside the frame. No information is then needed to be sent with regard to the peripheral GPs. The estimated motion vector field (MVF), is the highest priority information to be transmitted. The field is zig-zag scanned, and the difference between the components of
101
Figure 1. Evaluation of the no-motion areas: absolute difference between two consecutive frames (LEFT), threshold application (CENTER), median filter and evaluation of the no-motion vectors (RIGHT)
successive MVs is coded using a variable-length code, while sequences of null vectors are run-length coded. Since an irregular MVF involves high information rate, a particular method is studied to prune badly estimated MVs, mainly in still background zones of the image, where MVs should be set to zero. A binary image is composed by thresholding the absolute difference between two consecutive original frames, then a 3 x 3 median filtering eliminates the isolated points corresponding to local noise difference. A vector is set to zero if there are no associated filtered pixels in a region around the associated GPs. An example of this process is illustrated in fig 1. The resulting MVF is further smoothed by means of a vector median filter [5], which operates on each vector and the six adjacent ones. This filter works jointly on the vector components, so that the filter output is one of the original input vector (the quality of the decoded images is improved by the fact that we do not introduce new vectors). With this approach,the MVF becomes more uniform according to the natural motion of the sequence, and also the information rate, related to the transmission of the MVF, can be significantly reduced. The average amount of bits, required to code MVF after this processing, is about 400 bits per frame for the sequence "Miss America". An example of this process is shown in fig.2. 2.2. E r r o r p r o c e s s i n g a n d t r a n s m i s s i o n Considering that the main goal of this project is the development of a system able to code a video sequence at 10 kb/s and 10 frames/s, it is quite evident that the average number of bits that we can spend to code every frame is 1000. Looking at the conclusions of the previous paragraph we can consider that while about 400 bits must be used to transmit the MVF, a bit margin of about 600 bits can be used to transmit additional informations. The last important step of the method involves prediction error processing and transmission. Because of the inexact estimation of some vectors, and the difficulty to model particular movements and effects of the scene, like rotations of objects and uncovering of new areas in the picture, warping techniques does not always provide an adequate ira-
102
Figure 2. Motion Vector Field Processing: the MVF obtained after the Block-Matching (LEFT), after the evaluation of the no-motion areas (CENTER) and after the median vector filter (RIGHT)
age quality. An experimental comparison between several method for error elaboration (pixel-based, region based and block-based) points out that a block-based approach gives better results. Whatever method we choice, due to the low amount of bits remaining after MVF transmission, the task of coding prediction error is really hard. Using a block-based approach, the prediction error is evaluated inside 8 x 8 pixel blocks, placed in all the possible positions in the frame. The error related to the blocks with the highest mean absolute difference is DCT-transformed and quantised. The obtained coefficients, the quantisation step and the position of the blocks are transmitted. For each frame the number of coded blocks and transmitted coefficients depends on the remaining amount of bits after MVF transmission. On the receiver a post-processing is performed for all the blocks which have been improved by the transmission of the error. Typically the most important DCT coefficients of the error patches are located at low frequencies; in such a case, in the reconstruction phase while the low frequencies errors are recovered, the noise is concentrated in the high frequencies. Using a low-pass filter as post processing the advantage is twofold: the high frequencies noise is reduced and even if the reconstructed image becomes little smoother, a subjective evaluation of the image quality rewards this effort. 3. S I M U L A T I O N
RESULTS
The simulation was performed on three video-conference sequences in the QCIF format (picture size 176 x 144 pixels, luminance only), with a frame rate of 10 Hz: "Miss America", "Salesman" and "Akiyo". The grid is made up of 332 GPs, but only the 281 MVs of the internal GPs need to be coded and transmitted. In BM, the MVs are allowed to have integer values in the range +7 pixels vertically and horizontally, and 7 • 7 pixel blocks were used in the matching search. In the error coding phase, due to the energy compaction property of DCT only low-frequency coefficients are scalar quantised and coded with a variable length code. About 80 bits are required for error recovering in a 8 x 8 block. Moreover the first frame is J-PEG coded and transmitted.
103
sequence name "Miss America" "Akiyo" "Salesman"
min PSNR 30.97 dB 29.50 dB 27.01 dB
mean PSNR 32.47 dB 30.44 dB 30.04 dB
Table 1 Minimum and mean PSNS obtained for three test sequences
Figure 3. Last frame of the sequence: original (LEFT), reconstructed (RIGHT)
In spite of decreasing the temporal correlation between the first and the current frame in the sequence, warping and error recovering techniques achieves good results, particularly when the scene content does not produce sharp changes. In fact, PSNR drops significantly only when the movements in the scene give rise to a complex or irregular MVF, so that motion information increases and only few error blocks can be transmitted. On the other hand, when the movements become less decided, the MVF can be decoded using only few bits and a larger bit margin can be applied on the transmission of the error. In such a way frame after frame the quality in the decoded sequence significantly increases. In fig. 3 the last frame of the decoded sequence is represented together with the original one, fig. 4 depicts the plot of the PSNR for the sequence "Miss America" of 50 frames (5 seconds). Finally, Table 1 summarises the results obtained for the three test sequences. 4. C O N C L U S I O N S In this paper an innovative method to code video sequences at very low bit-rates has been presented. The proposed algorithm uses a spatial transformation to predict the actual frame from the previous one and the transmission of some error patches for the
104
Sequence "Miss America" 42 ............................................................. 44t i i 40
..............................................................................................
38
.....................................................................
::..................... : .................... ! i 'i i ~:--.
..........................
36 Z
34 32 28 26
' 5
10
15
20 25 30 frame number
35
40
45
t
50
Figure 4. Diagram of the PSNR vs. frame number for the reconstructed sequence "Miss America" reconstruction of critical motion areas which are not well reconstructed through spatial transformations. In typical video-conference sequences simulation results demonstrate that in a 10kb/s channel a good quality can be reached for the reconstructed sequence. R E F E R E N C E S
1. Y. Nakaya and H. Harashima, "Motion compensation based on spatial transform", IEEE Transactions on Circuits and Systems for Video Technology, vol. 4, no. 3, pp. 339-356, June 1994. 2. G . J . Sullivan and R. L. Baker, "Motion compensation for video compression using control grid interpolation", Proc. 1991 IEEE ICASSP, M9.1, pp. 2713-2716, Toronto, May 1991. 3. J. Nieweglowski, T. G. Campbell and P. Haavisto, "A novel video coding scheme based on temporal prediction using digital image warping", IEEE Transactions on Consumer Electronics, vol. 39, no. 3, pp. 141-150, Aug. 1993. 4. J. Jain and A. Jain, "Displacement measurement and its application in interframe image coding", IEEE Transaction on Communication, vol. COM-29, no. 12, pp. 1779-1808, Dec. 1981. 5. J. Astola, P. Haavisto and Y. Neuvo, "Vector median filters", Proceedings of the IEEE, vol. 78, no. 4, pp. 678-689, Apr. 1990.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
105
I m a g e C o m m u n i c a t i o n s Projects in A C T S Franco BIGI Responsible for Service Engineering in the Directorate B "Advanced Communications Technologies and Services in the Directorate General XIII "Telecommunications Information Market and Exploitation of the Research of the European Commission.
1 9I N T R O D U C T I O N
On 27 July 1994, the European Council adopted, as part of the fourth framework programme of the European Union (1994-1998), a specific programme for research and technological development, including demonstrations in the field of Advanced Communication Technology and Services (ACTS). The ACTS programme is to be considered as the third phase of the implementation of the IBC (Integrated Broadband Communications). Considerable progress has been made since telecommunications operators and industry defined IBC as the common vision of the future communications infrastructure for Europe and as objective for RTD collaboration in the RACE programme. But, right from the conceptual stage in 1984, three phases were distinguished on the route towards implementing IBC: a Phase I concentrating on the system engineering, specifications and key technologies, a Phase II concentrating on integration and the prototyping of the new services and applications and a Phase III beyond the RACE Programme consisting of user-oriented experimentation. With the ACTS programme, with a considerable paradigm shift, the objectives, as stated in the Council Resolution are ...to develop advanced communication systems and services for economic development and social cohesion in Europe, taking into account of the rapid evolution in technologies, the changing regulatory situation and opportunities for development of advanced transeuropean networks and services.
The aims are to support European policies for early deployment and effective use of advanced communications in consolidation of the internal market, and to enable European industry to compete effectively in global markets. The work will enable the re-balancing of public and private investments in communications, transport, energy use and environment protection, as well as experimentation in advanced service provision. In conjunction with the work in the specific programme on information technologies, it will provide a common technological basis for applications research and development in the specific programme on telematic systems and will prepare the ground for the development of a European market for information services..." The programme Acts has been implemented with two calls for proposals: the first call resulted in 115 projects started in the second half of 1995 or at the beginning of 1996 for a total
106 contribution of 440 million ECU's; the second call resulted in the approval of around 45 new projects and in the reinforcement of some of the existing projects for a total contribution of 110 million of ECU's. The new projects started or are due to start in the second half of 1966.
2 . AREAS IN TECHNOLOGY DEVELOPMENT IN ACTS The workplan of ACTS has been developed with an intensive preliminary consultation of all the sector actors, coming from all the main European universities, the manufacturing industry, the network operators and their research centres, the broadcasters, the service and content providers. The projects have responded to the tasks, as described in the workplan, classified into six specific domains, as follows: Interactive digital multimedia services Photonic technologies High-speed networking Mobility and personal communications Intelligence in networks and service engineering Quality, security and safety of communications systems Horizontal actions. The image communication is dominant in the first area, but is present also in all the remaining, because it is very difficult to imagin a high speed network or an advanced mobile system not open to the video signals. In the same way a complex interactive digital multimedia service can not operate in an unsafe and unsecure system, without the appropriate quality, with a good identification and an accurate description of the service components in close cooperation with equally complex network functions.
3 . INTERACTIVE DIGITAL MULTIMEDIA SERVICES In the next ten to fifteen years emerging interactive multimedia services will have a strong impact on social and cultural life. European standards and multilingual services are essential for social cohesion in Europe, and a strong multimedia sector will create new employment opportunities. By the year 2010 there will be many more TV programmes available to the customer and many of these will be interactive in nature. The boundaries between computing, communications and broadcasting will have largely been eliminated and user friendly terminals with fiat panel displays will provide access to a wide range of entertainment, communication and education services. Broadband connections to both workplace and home will eliminate the local loop bottleneck. Service designers will be able to trade bandwidth and coding complexity to achieve the desired service features and usability. If residential and small business information highways are to flourish, an open competitive market is vital. Vendor-independent standards are needed to enable the commercial separation
107 of service provision from network provision. European and international standardisation bodies have shown the technical feasibility and time-scales for the introduction of the requested standards. The MPEG standard defines a range of state of the art image and sound coding and multiplexing techniques. The standard has been accepted by international standardisation bodies Like ETSI as the basis for new television broadcasting services and by DAVIC as the basis for multimedia services via telecom, CATV and satellite networks. The provision of multimedia services over advanced communication networks involves the development and integration of a number of key technologies. The projects addressing these technologies cover the following service elements and background knowledge: 1. 2. 3. 4. 5. 6. 7.
Multimedia content manipulation and management Presentation, interaction and storage Transmission media Interworking across different networks Enabling commercial services Trials Transition scenarios
This structure has been chosen in order to present the general intention to reach the needed convergence, more than reflecting the subdomain structure of the 58 projects. For the majority of these projects (44) the work has started in the second half of 1995; for 7 the commencing date is July 1996 and for the remaining 7 the starting date is planned in the second half of this year, for some of them is subject to the completion of the negotiation. The following list is established in purely alphabetical order of the chosen acronym: AMPA AMUSE ATLANTIC ATMAN AURORA BIDS BURBON CABSINET CATVDC CICC CINENET CODIS COVEN CRABS DAM DIANE DIGISAT DVBIRD DVP
Advanced Multimedia Parallel Accelerator Advanced MUltimedia SErvices for residential users Advanced Television at Low bit rates And Networked Transmission over Integrated Communication system Digital Audio Visual Work Trading by ATM Automatic Restoration of ORiginal film and video Archives Broadband Infrastructure for Digital television and multimedia Services Broadband Urban Rural Based Open Networking Cellular Access to Broadband Services and Interactive Television Concerted Actions in Support of the CATV Delivery Chain Collaborative Integrated Communications for Construction CINEma, films and live events via satellite, cable and Atm NETworks Clip On Demand Interactive Services Collaborative Virtual ENvironments Cellular Radio Access for Broadband Services Davic Accompanying Measures Design Implementation and operation of a distributed Annotation Environment Advanced Digital Satellite Broadcasting and Interactive Services Digital Video Broadcasting Integrated Receiver Decoder Distributed Video Production
108 European MM Services for Medical Imaging Architectures Software and Hardware for MPEG4 Systems EUROpean Research and consensus on Interactive Multimedia Generic Architecture for Information Availability Integrated Broadband Communications on Broadcast Networks Integrated MultiMedia Project INTERACTive Television and multimedia return channel service trials Interactive Satellite Information Systems Kiosk based Integrated Multimedia Service Access for Citizens Learn from Video Extensive Real ATM Gigabit Experiments Multimodal Verification for Teleservices and Security Applications MAintEnance System based on Telepresence for Remote Operators Multimedia Environment for MObiles Multimedia Interactive DemonStrator TElePresence Manipulation of Images in Real time for the creation of Artificially Generated Environments MOMUSYS Mobile Multimedia SYStems MUSICIAN MUltimedia Services Integration Chain In Advanced Networks Multimedia User Interfaces for Interactive Services and TV MUSIST Offer of Contents through Trusted Access LinkS OCTALIS Open Kernel for Access to Protected Interoperable interactive services OKAPI OPARISOD OPen ARchitecture for Interactive Services On Demand PANORAMA PAckage for New OpeRational Autostereoscopic Multiview systems and Applications QUO VADIS Quality Of Video and Audio for Digital television Services REconstruction using Scanned Laser and Video RESOLV Scalable Architectures with Hardware Extensions for Low Bitrate Variable SCALAR Bandwidth Real-time Videocommunication Set Top Box for Interactive Services on Demand SETBIS Scalable Interactive Continuous Media Servers Design and Applications SICMA Storage for Multimedia Application Systems in the Home SMASH Software Open MultiMedia Interactive Terminal SOMMIT Service Provisioning Environment for Consumers Interactive Applications SPECIAL TALISMAN Tracing Authors rights by Labelling Image Services and Monitoring Access Network TAPESTRIES The Application of Psychological Evaluation to Systems and Technologies in Remote Imaging and Entertainment Services Team based European Automotive Manufacture TEAM TELEBORG Communications and Hardware Requirements for Telepresence Supporting Human Like Physical Presence in Real Remote Environments TELESHOPPETelescoping services using virtual reality and interactive multimedia VALIDATE Verification And Launch of Integrated Digital Advanced Television in Europe VANGUARD Visualisation Across Networks based on Graphics and the Uncalibrated Acquisition of Real Data VIDAS Video assisted with audio coding and representation VISEUM Virtual Museum International
EMERALD EMPHASIS EURORIM GAIA IBCoBN IMMP INTERACT ISIS KIMSAC LEVERAGE M2VTS MAESTRO MEMO MIDSTEP MIRAGE
109 4.1. MULTIMEDIA CONTENT MANIPULATION AND M A N A G E M E N T The scope of this area will be to develop audio-visual signal representation, manipulation and management tools able to support the creation of advanced digital multimedia services. The activities will concern software tools and hardware components for generic applications in 3D telepresemce, mobile multimedia communications, program production and distributed virtual environments. New image related standardisation activities in MPEG4 herald a revolution in terms of representation and processing of audio visual signals. The scope goes beyond compression to include new functionality such as object identification and manipulation, hybrid scenes (computer graphic and real images) or 3D images representation. ACTS projects aim at different services and applications but are likely to share a large number of tools and algorithms. State-of-the-art audio visual coding (e.g. H324 for very low bit rate coding over the telephone network) will be enhanced thanks to the use of audio information combined with advanced image synthesis techniques (VIDAS). Technology providing new audio visual functionality (tools, algorithms and syntactic description language) for mobile multimedia systems (MOMUSYS) and videotelephony (SCALAR) will be developed and contributed to MPEG4 and the implementation aspects of this new standard will be pursued (EMPHASIS). Telepresence extends the video conferencing concept so that participants can use non verbal aspects of communication (eye contact, spatial perception, body movement, gesture, facial expressions) in the same way as they would in a face to face meeting. Within ACTS, telepresence services and the underlying technology will be demonstrated in field trials by a number of projects in a variety of application scenarios. The technology involved is often a combination of relatively stable networking techniques and leading edge applied research results such as 3D image acquisition, modelling of virtual reality, and high performance displays (RESOLV, VANGUARD, PANORAMA). A computational service for virtual presence in support of future co-operative teleworking systems will be developed and trialled (COVEN). The main focus will be on the added value of networked Virtual Reality involving varying degrees of multi sensory presence. The benefits of these advanced technologies in the construction of buildings will also be shown (CICC). Following the ACTS second call, the MCM group will also integrate new projects on telepresence including telemanipulation and augmented reality (TELEBORG, MAESTRO, MIDSTEP). Compared to existing multimedia services projects, these initiatives will go one step further towards telepresence. Input and output devices will not only be audio visual based but will also cover touch and physical feedback. The multidisciplinary nature of the work is in line with current global high potential activities (DAVIC, MPEG4, ..). New telemanipulation projects are likely to impact on virtual reality, augmented reality and real time registration techniques (CAD model on video..), video based telepresence (higher flexibility, frame rate view points..), hypermedia data management, broadband networking (QoS, traffic analysis for telepresence applications), teleoperation techniques (robotic arms, data glove..), advanced displays techniques (3D, VR helmets, forcereflecting glove / arm display..), sensors, actuators, human/computer interaction tools and artificial intelligence. Advanced image manipulations, telepresence and distributed operations have also the potential to transform the way TV programs are produced and to cut costs. A testbed for
110 distributed production, post-production, rehearsal, archiving, indexing and retrieval forming a distributed virtual studio over ATM will be developed by DVP. Television virtual production involving various aspects of creation and display of Virtual Reality and manipulation of virtual environments will be the focus of MIRAGE. Parallel architecture for computation intensive multimedia operations such as those needed in advance programme production will be developed in AMPA and content will be deployed over broadband transatlantic links (VISEUM). Advances in signal processing will also be applied for automatic artefact removal aimed at the restoration of old films (AURORA) or the analysis of videosequences for access control (M2VTS). The next step in MPEG2 coding will be the implementation of a full MPEG2 chain starting in the studio and involving switching of compressed streams (ATLANTIC). Finally, understanding the key psychological factors contributing to customer acceptance of new entertainment and informational services will be the aim of TAPESTRIES and will include subjective evaluation of virtual reality, MPEG4 and 3D images (this will also be offered as a service to other projects). MPEG4 is and will continue to be the glue between most of the MCM projects. It is the forum bringing together major actors and technologies from the computer, television and telecom areas. ACTS projects have already achieved an outstanding degree of participation, contribution and leadership within this group. This is specially the case in the Video-VM, MSDL, Implementation and Future workgroups, all of which are led by ACTS project members. This is true also for the MPEG4 audio group in which ACTS is equally well represented. As part of the Integration group, the relatively new SNHC ad hoc group intends to standardise tools for representation and coding of real and synthetic audio visual scenes in support of the creation, manipulation and efficient management of increasingly hybrid content. ACTS projects are stepping up their involvement in this group as well. VIDAS is already an active member and contributions will be reinforced through new project extensions such as COVEN bringing in computer graphic expertise and MOMUSYS extending its activities toward higher bitrates in a hybrid framework. A deeper involvement in SNHC is also expected from projects DVP, MIRAGE (distributed virtual studio, VR), VANGUARD, RESOLV (3D analysis and modelling), CICC, MAESTRO (augmented reality) and PANORAMA (3D handling and display), ensuring a significant impact for European technology in this key area. 4.2. PRESENTATION, INTERACTION AND STORAGE The advent of interactive multimedia services points to the need for intelligent service interfaces capable of assisting the user to understand the extent of many services available, to locate relevant material, access services, contact other parties and retrieve, store and manipulate information. This entails the development of high capacity storage units and terminals equipped with advanced navigation tools. R&D activities in this area will concern set top units, servers and storage units. Innovative navigation and dialogue tools for interactive multimedia services integrating agent technology and exploiting new standards such as MHEG or MPEG4 will be demonstrated. Brokerage services will be developed. The aim of the SICMA project is to design a cost effective, scaleable server for the delivery of images, data and continuous multimedia information. Its efficiency will be demonstrated by a
111 "virtual museum" application, including links to an art museum in Russia. The server will fully realise the DAVIC standard. Set top units are the focus of both the SETBIS and SOMMIT projects. SETBIS is constructing a set top prototype with special emphasis on low cost hardware, for acceptance testing of interactive services. The set top box will be evaluated in a mini trial with real users, using the infrastructure of the CaTV German National Host in Baden Wttrtenberg. The overall goal of SOMMIT is to define an open interactive multimedia terminal architecture, application and delivery media independent, and to achieve a full software implementation. SOMMIT is building on work carried out in the RACE project MARS, and will develop a toolkit of modules which can be reused in different designs of terminals depending on the actual time, manufacturer and product. The project aims at compliance with DAVIC and ATM Forum standards. DVBIRD is developing an optimised and integrated chip set for digital terrestrial TV receivers, starting from the DVB-T (Digital Video Broadcasting Terrestrial) specification, and from work already carried out within the RACE project dTTb. DVBIRD will deliver a demonstrator to other ACTS projects to carry out field trials for validation purposes. It will also take into account the exploitation of commonalties with satellite and cable receiving chains, leading to the definition, design and fabrication of an optimised chip set to be used in a common receiver for satellite, cable and terrestrial broadcasting, compliant with DVB. SMASH is developing a storage system to satisfy both the demand for large storage, and the need to access the stored information in a fast and interactive way. Studies will be done on how to interconnect magnetic tape and disk systems in optimal way, so that they behave as one integrated system to the user. An adverse consequence of the information highway is the possibility of getting lost and being unable to locate the desired information. Electronic programme guides, brokerage services, mobile intelligent agents and user interfaces are the subject of study for the projects MUSIST, KIMSAC and DIANE. 4.3. TRANSMISSION MEDIA
Instead of being integrated on one single medium, the information society will utilise all available transport media in order to create user friendly and economically viable services. This is reflected in the diversity of the projects. Fibre, coax, twisted pair, satellite and terrestrial transmission are all covered. A major effort is to creme a return channel for interactive services on the traditional media for distribution services or to creme broadband capability on the traditionally narrowband switched services. Many direct-to-home satellite and terrestrial broadcasters, CaTV operators and telecoms companies are looking to migrate their residential networks towards bi-directional broadband capability, in a race to pioneer the new interactive multimedia markets. VALIDATE will co-ordinate the tests needed to achieve sufficient confidence in the DVB-T specification to be adopted by ETSI. The laboratory tests and field trials carded out by VALIDATE will also produce information needed for the planning of Digital Terrestrial
112 Television Broadcasting (DTTB) services. INTERACT is studying and implementing return channels for use in terrestrial and cable DVB systems, and will contribute to the debate within Europe (via the DVB and EBU) and in world wide fora (via the ITU and DAVIC) regarding the return channel specification. MEMO will trial the data transfer capability of DAB (up to 1,7 Mbit/s, with a GSM return channel). The DTH (Direct To Home) satellite project ISIS aims to provide tele-education, on line catalogues, newspaper distribution, access to Internet and to LANs in major offices for those working from home. It aims to incorporate a low speed return channel at an increase in cost of around 20% compared to the DTH receive-only terminal. Another satellite project (DIGISAT) will develop the technology for interactive services via SMATV (Satellite Master Antenna TeleVision) networks, based on the studies started in the RACE project DIGISMATV. A key target of the project is to develop a return channel via satellite using VSAT technology. In the downstream-only direction, CINENET aims to distribute cinema quality digital movies, sports, live events and business presentations to conference halls, cultural centres, hotels, etc. over satellite, cable and ATM networks. It will build on the experience gained in the RACE project HDSAT. IBCoBN will concentrate on upgrading CaTV networks to support symmetrical broadband applications such as videotelephony and videoconferencing, and on publicising the potential of these services. IBCoBN also plans to lay the groundwork for establishing a common research facility for independent cable operators in Europe. CaTV operators can carry two way broadband communications over existing coaxial cable networks, using Hybrid Fibre Coax HFC topologies upgraded with a return channel. In HFC, optical fibre runs between head end and a street cabinet, with the existing coaxial cable covering the last km or two to the home. The return channel uses bus techniques to share bandwidth on the coaxial cable which is laid in a tree topology. Many ACTS projects see a switched ATM overlay on an HFC network as necessary, for efficient use of bandwidth, and so that the subscriber can establish connections on demand to servers in different locations. Telecom operators can also carry two way broadband communications over the existing telephony twisted pair local loop, by deploying ADSL (Asymmetrical Digital Subscriber Loop) electronics in the home and at the local exchange. Alternatively they can use a FTTC (Fibre To The Curb) network with a passive fibre extending from local exchange in tree topology to termination points in street cabinets, from which individual twisted pair or coax drops are provided in star topology to the home. Typical bandwidth are 6 Mbit/s downstream and several hundred kbit/s upstream. The trend is to bring nearer and nearer to the home, but FTTH (Fibre To The Home) is generally too expensive today, because of the cost of the optical to electrical conversion in the home. AMUSE will demonstrate the viability of using ATM end-to-end from server to TV set top units, and also PCs over ADSL,FTTC and FTTH networks MPEG2 will be transported over ATM variable bit rate and possibly available bit rate services, for maximum efficiency. The IMMP project is also using end-to-end ATM. IMMP foresees that in the longer term business services to the home and very small enterprises will be of major importance, it is therefore
113 developing a platform that will support business applications as well as entertainment. The focus of OPARISOD is the development of an interactive services on demand Media switching Centre which receives, via ATM-SDH interfaces, a number of information streams from the Media Servers. These streams are switched then mapped into DAVIC transport frames for transmission through the access network to the users. A successful introduction of advanced digital services relies upon the management of the quality of service; the project QUOVADIS is looking to mechanisms for controlling and supervising the performance of digital TV distribution across gathering, transport and broadcast networks. 4.4. INTERWORKING ACROSS DIFFERENT NETWORKS Access to servers in many different locations, and peer-to-peer broadband communication, requires that the various types of access networks be interconnected via core telecom networks. The existing core network infrastrucure is mainly optical fibre, and over this, many operators are now deploying a layer of ATM switches. This ATM switching layer provides instant switched bandwidth on demand, carrying any mix of voice, data and video traffic, with the economy of sharing physical links through statistical multiplexing.
Interconnection between participating countries, and between projects is a key obiective of ACTS. It is planned that this interconnection will be via a pan-European research network> Access in a given country will be provided by the National Host(s) in that country. NH offer access protocols such as ATM and SMDS (Switched Multimegabit Data Services, and were set up to support the national communication needs of ACTS and other EU R&D programmes (A country may have more than one NH, based on different types of network). Several ACTS projects aim to demonstrate open DAVIC and/or DVB compliant interfaces. The ACTS project DAM will test DAVIC profilesover vired access networks, linking servers with access networks between Finland, France, Germany and Italy via an ATM network. DAM is also sisseminating information about the DAVIC specifications to other ACTS projects and to European companies, and will co-ordinate input towards DAVIC. 4.5 ENABLING COMMERCIAL SERVICES The development of technologies for the digital transmission of multimedia information may be impressive, but these technologies will not be implemented unless they can be exploited in commercially viable services. However, besides the higher bandwidth efficiency, the fact that digital information streams can readily be encrypted is an important drive towards digitalisation of existing services. The rapid multiplication of the number of the television programmes and the limited financial resources make it necessary to provide the content only to customers who have paid for the access right and to protect the multimedia material against illegal copying. ACTS projects that study these systems are respectively OKAPI and TALISMAN. Customer care, tariffing and billing concepts for interactive multimedia services will be evaluated by SPECIAL. The introduction of a multimedia broadband service network will give the opportunity to provide a much more efficient and cost effective mechanism to connect consumers with providers, with the introduction of a suitable set of standard brokerage interfaces. Electronic
114 brokering services range from meta-catalogues compiling information from a large number of different providers to intelligent agents that visit the provider on behalf of the user. The brokerage for broadband services is part of the work within DAVIC and the appropriate references to standards will be part of future releases of the DAVIC specifications. GAIA is addressing the task on the development of open architectures for brokerage systems. It will develop a generic scaleable architecture for information availability having as infrastructure platform heterogeneous already established media, networks and protocols. ATMAN, CODIS and MUSICIAN will develop, integrate and test the full supply handling delivery chain for electronic brokerage systems trading digital audio video material of different kind: data from Internet (WWW), AV material through the ATM network, AV material through SNG (Satellite News Gathering), video clips and AV material through broadcasting. 4.6. TRIALS Several projects aim to carry out trials of the whole multimedia service chain, from server to television set and PC. AMUSE will carry out trials on ADSL, FTTC and FTTH networks in Italy, and HFC on CaTV networks in Belgium and Germany. IMMP will use the UK, Swedish and Finnish National Hosts to provide access to the Pan-European ATM Pilot. OPARISOD will carry out a mini trial, in which the users will be chosen from the 4000 subscribers connected to the Baden WOrtemberg pilot system. SPECIAL plans to develop and implement a customer care concept for consumer services in the InfoCity trial in Nord Rhine Westfalen (the InfoCity trial is a major initiative to provide an alternative infrastructure to that of the dominant operator, involving 10,000 users). BURBON will run a trial on cost effective access to ATM based advanced services for SMEs in 6 member states. TELESHOPPE will apply advanced virtual reality technologies in the creation and trialling of multi lingual teleshopping services. TEAM will provide a framework for virtual integration of the entire automotive supply chain. LEVERAGE is focusing on the support of language learning using broadband video services; EMERALD will do the same in the medical environment. 4.7. TRANSITION SCENARIOS The project BIDS will address the main options for progressive and economically sound introduction of digital TV, HDTV, and interactive TV services across Europe, including user requirements and regulatory considerations.
5. THE FUTURE The ACTS programme is due to run until the end of 1998. Already some thought is being given to the direction that future programme should take. Advanced telecommunications will continue to remain of importance both in the direct impact that they have as a major industry in themselves, but more importantly owing the services which they can offer to the business community and to the individual. Multimedia, with special emphasis on image, will remain the major building block of the European R&D activity in telecommunications.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
115
C o n v e y i n g m u l t i m e d i a services within the M P E G - 2 T r a n s p o r t S t r e a m Luigi Atzori, Massimiliano Di Gregorio and Daniele Giusto Department of Electrical and Electronic Engineering, University of Cagliari, piazza D'Armi, 09123 Cagliari Italy In this paper, we present a new system for the transmission of multimedia information within the digital TV channel. The scheme proposed is based on some DSM-CC functions, while the transport structure used is the MPEG-2 Transport Stream, that is the most widespread platform for new digital television systems. The multimedia information conveyed in such application is structured as in the WWW system, where the text is integrated with images, sounds and animation by the means of the html language. 1. INTRODUCTION It is a matter of fact that the standard MPEG-2, defined by ISO/IEC 13818, is the most widespread platform for video and audio coding. This standard provides also the transport protocol in order to multiplex different bit streams and to convey the audio-video synchronization signals ~. The transport structures defined in the standard for the digital television broadcasting is the Transport stream. Such a structure allows to multiplex in one bit stream, different mpeg audio and mpeg video signals as well as every other type of digital stream any way defined, refered to as private data. In point of fact, new elementary streams can be handled at the transport layer without hardware modifications, giving the opportunity to implement additional services inside the TV digital channel. A digital teletext version has been already standardized by the ETSI, giving the syntax to insert the data inside the Transport Stream 2. Furthermore, the part six of the ISO/IEC 138183 describes a global server-network-client system and the related bit stream syntax for the implementation of additional services related to the broadcast and interactive multimedia applications. This is the Digital Storage Media Command and Control (DSM-CC) specification. Our idea has been to carry multimedia information inside the Transport Stream MPEG-2, making use of the DSM-CC features and introducing a service that is somehow similar to the WWW information available in the Internet Network. In the last section of this paper, a simulation of coding-transmission-decoding of this kind of data is described, based on two UNIX processes connected by a socket structure.
2. MULTIMEDIA INFORMATION SYSTEM BROADCASTING An hypertextual system consists of a group of different documents linked each other, as a database where the user can "jump" from a region to an other inside the database. Most of the hypertextual systems allow the integration of documents with other types of data, as graphs, images, animation, making up multimedia information 4.
116 For the purpose of our work, we refer to the WWW database structure, in which every web is made up of html files, images of different type and audio-video sequences. The main characteristic of the WWW database is that it branches out in several local hypertexts, that are separate open systems. This feature cannot be realised in a broadcast application where it is required to send only "closed web", that is every link must be inside. The implementation of such a service requires to send the whole web file system, being care to rebuild, in the client side, the same path for every file as to hold the same file names to preserve the links. The final service offered is a periodic transmission of the file systems (data carousel), where the user can download the webs he is interested in. In order to provide the user with sufficient information to make this selection, data carousel has to include, inside the multimedia data, a directory mechanism that describes the information of the carousel and provides a procedure to group the information. 3. C O D I N G LAYER STRUCTURE The coding and transmission processes of the hypertextual system are made up by three layers. The first one has been introduced to convey specific information about every file such as the name and the path. This information is necessary for the correct building of the file system, so that every link is preserved. The second layer provides the functions necessary to set up the transmission and to carry the files to the client. For this purpose the DSM-CC download messages have been adopted, in order to get a stream that can be parsed by a generic DSM-CC decoder. This choice has introduced some unnecessary fields in the syntax but makes this application more portable. The third layer is the transport layer, that is the well-known MPEG-2 Transport Stream.
3.1. File section The file_section is a structure in which is encapsulated the file, that has been designed to associate basic file information with the file data. As table 1 shows, a name_length field is conveyed to provide the number of bytes used to code the file name in the following field. So a char string is carried, that codes the name in a ASCII fashion. To give the right path to the file, it has been designed to associate a byte code to every path inside the file system. This is carried inside the path_code field that shall be decoded through a table previously sent to the client in a DSM-CC message. m
Syntax
N. of Bytes
file_section() { name_length file_name path_code for(i=O; i
1
name_length 1 1
Table 1. File_section syntax.
3.2. The download layer The Digital Storage Media Command and Control specification provides a set of protocol to realise different applications such as movie on demand, tele-shopping, news on demand,
117 remote database access, etc. The main characteristic of this syntax is that it doesn't specify the underlying physical, data link, transport layers of the overall protocol stack. It is intended to provide a unified signalling layer over a wide variety of underlying network topologies. A particular feature of the DSM-CC is the download function, that is intended to be a very lightweight and fast data or software download from the server to the client, or from the network to the client. Flow-controlled operation, as well broadcast download option, are supported and are based on the same message set. A complete download operation transfers an "image" data, that is made up of logically separate section called modules. In order to meet the transmission constraints, such as bit error rates and maximum transport packet size, each module is then divided in blocks, all of the same size. During a download session the whole web file system is sent to the client, managing each file as a module. In order to realize a web download it has been