This Page Intentionally Left Blank
Selections of the International Conference on Intelligent Robots and Systems I994, IROS 94, Munich, Germany, 12-i6 September i 9 9 4
E D I T E D BY
V o T.K~.R G RAE F E
Institut ffir Messtechnik Universitgt der B W Mi~nchen Neubiberg, Germany
1995 ELSEVIER A m s t e r d a m - Lausanne - New Y o r k - O x f o r d - Shannon - Tokyo
ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25 P.O. Box 2II, IOOO AE Amsterdam, The Netherlands
Library oF C o n g r e s s
Cataloging-In-Publication
Data
IEEE/RSJ/OI Internationa] ConFerence on I n t e l l i g e n t R o b o t s and Systems (1994 : M u n i c h , Oermanv) Inte]ligent r o b o t s and s y s t e m s : s e l e c t i o n s oF t h e I n t e r n a t i o n a ] C o n F e r e n c e on I n t e l l i g e n t R o b o t s and Systems 1994, IROS 9 4 , M u n i c h , Oerman#, 12-16 S e p t e m b e r 1994 / e d i t e d bv V o l k e r O r a e ~ e .
p. cm. Includes bibliographical references. ISBN 0-444-82250-X I. Robotics--Congresses. 2. Inte111gent control systems-Congresses. I. OraeFe, Volker, 1938. II. Title. TJ210.3.I447 629.8'92--dc20
1994a 95-36630 CIP
ISBN: o 444 82250 x 9 i995 Elsevier Science B.V. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box 52i, iooo AM Amsterdam, The Netherlands. Special regulations for readers in the U.S.A. - This publication has been registered with the Copyright Clearance Center Inc. (CCC), 222 Rosewood Drive, Danvers, MA, oi923. Information can be obtained from the CCC about conditions under which photocopies of parts of this publication may be made in the U.S.A. All other copyright questions, including photocopying outside of the U.S.A., should be referred to the copyright owner, Elsevier Science BV, unless otherwise specified. No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. This book is printed on acid-free paper. Printed in The Netherlands.
PREFACE In September 1994 the seventh International Conference on Intelligent Robots and Systems- IROS '94-- was held in Neubiberg, a suburb of Munich, Germany, on the premises of the German Armed Forces University Munich. Of the 300 papers presented during the conference I have selected 48 ones which, in my subjective opinion, are particularly significant and characteristic for the present state of the technology of intelligent robots and systems. This book contains the selected papers in a revised and expanded form. Robotics and intelligent systems constitute a very wide and truly interdisciplinary field. The papers in this book have been grouped into the following categories: Sensing and Perception Learning and Planning Manipulation Telerobotics and Space Robotics Multiple Robots - Legged Locomotion - Mobile Robot Systems Robotics in Medicine
-
-
-
-
-
-
They cover, however, many additional fields, e.g. control, navigation, and simulation, to mention only a few. When preparing the conference, and this book, I found that most researchers in robotics are now apparently interested in some combination of learning, mobile robots, and robot vision. Consequently, most of the articles in this book relate to at least one of these fields. A book like this is always the result of many persons' efforts. Special thanks are due to Mr Michael LiJtzeler who helped with the compilation of the book by creating and maintaining various data bases, and to my secretary, Mrs Sigrun Haussmann, who among other things, helped some of the authors to improve the style and grammatical correctness of their texts. Thanks also to Ms Drs. Eefke Smit &Elsevier Science BV for competently and patiently managing all matters related to the publishing of the book.
Neubiberg, March 1995 Volker Graefe
This Page Intentionally Left Blank
vii
TABLE OF CONTENTS
Preface
SENSING AND PERCEPTION
A new High-performance Multisonar System for Fast Mobile Robot Applications U.D. Hanebeck and G. Schmidt Proper Selection of Sonar and Visual Sensors in Vehicle Detection and Avoidance N. Moghaddam Charkari, K. Ishii and H. Mori
15
Selective Refinement of 3-D Scene Description by Attentive Observation for Mobile Robot H. Takizawa, Y. Shirai and J. Miura
29
Active Vision Inspired by Mammalian Fixation Mechanism T. Yagi, N. Asano, S. Makita and Y. Uchikawa
39
Realtime Range Imaging Using An Adjustment-free Photo VLSI A. Yokoyama, K. Sato, T. Yoshigahara and S. Inokuchi
49
A Pedestrian Discrimination Method Based on Rhythm S. Yasutomi, H. Mori and S. Kotani
61
Visual Recognition of Obstacles on Roads U. Regensburger and V. Graefe
73
Visual Collision Avoidance by Segmentation I. Horswill
87
LEARNING AND PLANNING
Using Perceptions to Plan Incremental Adaptions A.J. Hendriks and D.M. Lyons Robot Task Programming by Human Demonstration: Mapping Human Grasps to Manipulator Grasps Sing Bing Kang and Katsushi Ikeuchi Robot Learning By Nonparametric Regression S. Schaal and C.G. Atkeson
101
119
137
~ 1 7 6
VIII
Behavioral Control in Mobile-Robot Navigation Using a Fuzzy Decision Making Approach H.R. Beom, K.C. Koh and H.S. Cho Learning Emergent Tasks for an Autonomous Mobile Robot D. Gachet, M.A. Salichs and L. Moreno Efficient Reinforcement Learning of Navigation Strategies in an Autonomous Robot J. del R. Milldn and C. Torras
155
169
185
A Lifelong Learning Perspective for Mobile Robot Control S. Thrun
201
A Multilevel Learning Approach to Mobile Robot Path Planning F. Wallner, M. Kaiser, H. Friedrich and R. Dillmann
215
A Self-Organizing Representation of Sensor Space fi~r Mobile Robot Navigation B.J.A. KrOse and M. Eecen
229
Path Planning and Guidance Techniques for an Autonomous Mobile Cleaning Robot C. H ~ w r and G. Schmidt
241
MANIPULATION
Grasp Strategies for a Dextrous Robotic Hand K. Woelfl and F. Pfeiffer
259
Motion Scheme for Dextrous Manipulation in the Intelligent Cooperative Manipulation System - ICMS M. Buss and H. Hashimoto
279
Designing a Behavior of a Mobile Robot Equipped with a Manipulator to Open and Pass through a Door K. Nagatani and S. Yuta
295
The Development of Re-usable Software Systems for Intelligent Autonomous Robots in Industrial and Space Applications E. Freund and J. Roflmann
311
TELEROBOTICS AND SPACE ROBOTICS
Toward Integrated Operator Interface for Advanced Teleoperation under Time-Delay A.K. B~jczy, P. Fiorini, W.S. Kim and P.S. Schenker
327
Active Understanding of Human Intention by a Robot through Monitoring of Human Behavior T. Sato, K Nishida, J. Ichikawa, K Hatamura and H. Mizoguchi
349
Human/machine Sharing Control for Telerobotic Systems T.-J. Tarn, N. Xi, C. Guo and A.K. Bejczy
373
Task-Directed Programming of Sensor-Based Robots B. Brunner, K. Arbter and G. Hirzinger
387
Telerobotics with Large Time Delays - the ROTEX Experience G. Hirzinger, K. Landzettel and Ch. Fagerer
401
Feature-Based Visual Servoing and its Application to Telerobotics G.D. Hager, G. Grunwald and K. Toyama
415
Practical Coordination Control Between Satellite Attitude and Manipulator Reaction Dynamics Based on Computed Momentum Concept K. Yoshida
431
A 5-Axis Mini Direct Drive Robot for Teleoperation and Biotechnology M. Moreyra, P.-H. Marbot, S. Venema and B. Hannaford
445
A Laboratory Test Bed for Space Robotics: The VES II S. Dubowsl,y, W. DurJ~)e, T. Corrigan, A. Kuklinski and U. MOiler
463
MULTIPLE ROBOTS
Hierarchical Prediction Model for Intelligent Communication in Multiple Robotic Systems K. Sekiyama and T. Fukuda
477
Proposal for Cooperative Robot "Gunryu" Composed of Autonomous Segments S. Hirose, T. Shirasu and E.F. Fukushima
491
Unified Control for Dynamic Cooperative Manipulation K Kosuge, H. Yoshida, T. Fukuda, M. Sakai, K. Kanitani and K. Hariki
501
LEGGED LOCOMOTION Comparative Study of Adaptive Controllers for a Pneumatic Driven Leg M. Guihard
513
An Efficient Forward Gait Control for a Quadruped Walking Robot D.J. Pack and A.C. Kak
533
The Six-Legged TUM Walking Robot H.-J. Weidemann and F. Pfeiffer Vision-Based Adaptive and Interactive Behaviors in Mechanical Animals Using the Remote-Brained Approach M. Inaba, S. Kagami, T. Ishikawa, F. Kanehiro, K. Takeda and H. Inoue
549
559
MOBILE ROBOT SYSTEMS
Autonomous Sonar Navigation in Indoor, Unknown and Unstructured Environments W.D. Rencken
577
Control and Localisation of an Autonomous Mobile Vehicle M. Adams, N. Tschichold-Garman, S. Vestli and D. yon FlOe
589
The Robotic Travel Aid "HITOMI" H. Mori, S. Kotani and N. Kiyohiro
607
Vision-based Naviation for an Office Messenger Robot T. Gomi, K. Ide and P. Maheral
619
Trajectory Tracking Control for Navigation of the inverse Pendulum Type Self-Contained Mobile Robot Y.-S. Ha and S. Yuta
637
An Outdoor Robots System for Autonomous Mobile All-purpose Plattbrm T. Okui and Y. Shinoda
657
Mechanism and Control of Propeller Type Wall-Climbing Robot A. Nishi and H. Miyagi
669
Automated Guided Vehicles in Japan T. Tsumura
679
ROBOTICS IN MEDICINE Robotics in Medicine P. Dario, E. Guglielmelli and B. Allotta Planning, Calibration and Collision-Avoidance for Image-Guided Radiosurgery A. Schweikard, M. Bodduluri, R. Tombropoulos and J.R. Adler
691
721
Intelligent Robots and Systems V. Graefe (Editor) 1995 Elsevier Science B.V.
A new High Performance Multisonar System for Fast Mobile Robot Applications Uwe D. Hanebeck and Giinther Schmidt Department for Automatic Control Engineering (LSR) Technische Universit~t Miinchen 80290 Miinchen, Germany A new type of actively sensing multisonar system has been developed for obstacle detection, localization and map updating in indoor environments. It comprises 24 individual intelligent transmitter and receiver elements arranged in a horizontal plane around a high-speed motion platform. The physical sensors are electronically configured to form various types of virtual sensor arrays for specific tasks and situations. A method for high-frequency firing compensates for the low speed of sound in air. Various schemes for intelligent evaluation of the array outputs provide highly accurate localization estimates. Experiments with the developed multisonar system in structured and cluttered environments demonstrate improved perception capabilities compared to state-of-the-art approaches. 1. I n t r o d u c t i o n
Ultrasonic sensor systems in mobile robot applications serve three major objectives: 9 detecting collisions with unexpected, possibly mobile obstacles, 9 localizing known environmental structures, 9 updating an internal map with previously unknown objects. Different approaches have been reported. One class of systems uses one rotating time-offlight sonar sensing device to obtain a 360 degree scan of the environment [4]. The width of the sonar beam causes a blurred image. Attempts to narrow the beam by phased arrays [2], [6] or horn constructions [3] have also been undertaken. Mechanical positioning makes these systems slow and inadequate for collecting data during fast motion. In another class of systems several (20-30) individual time-of-flight sensors surround a robot. Fixed firing schemes independent of mission and situation are employed [7], [13]. Undesired mutual influences of the sensors result in slow firing rates and make error correction algorithms necessary [1]. Crosstalk effects are eliminated by identifying and rejecting pulses received from other sensors. The fastest system known to the authors achieves a sampling interval of 60 ms for each sensor with a maximum range of 2.5 m [1]. The separate evaluation of sensor outputs results in an unsatisfactory angular resolution. This feature complicates many sensory tasks, like determining the vehicle's own position with respect to a landmark or the position of a moving obstacle. While the second class of systems tries to avoid crosstalk effects, another class takes advantage of these phenomena. Sparse sensor arrays are used and the characteristics of sound propagation are taken into account. This approach permits determination of the normal directions of walls [10], I12] or tracking
of moving obstacles as discussed in [8]. The need for mechanical aiming renders these systems too slow for the intended application. The sensor system discussed in this paper is used with an omnidirectional locomotion platform as shown in Figure 6. For this application, omnidirectional measurement capabilities are required. In addition, the system is expected to concurrently perform various sensory tasks around the mobile robot. For example, localizing of a wall is performed while reliably scanning the current travelling direction. Data has to be provided at a sufficiently high rate to guide a fast (2 m/s) mobile robot with steering commands generated every 10 ms. The proposed system which adopts useful aspects of today's sensor systems is discussed in detail in Section 2. A method to alleviate complicated cross-talk error correction schemes is presented. High-frequency firing schemes compensate for the low speed of sound in air. Furthermore, it is shown how the system is dynamically configured during operation. Section 3 presents algorithms for extracting several relevant environmental features with the proposed system. Section 4 introduces a current prototypical implementation. Experiments with the multisonar system mounted on a fast motion platform are described in Section 5. Compared to state-of-the-art systems the new approach gives improved perception capabilities and simplifies subsequent algorithms like obstacle detection and localization. The techniques presented in this paper apply to what is called conformal arrays in radar technology, i.e. sensor arrays which conform with a given vehicle geometry. 2. S y s t e m O v e r v i e w The proposed multisonar system is designed for operation in office or laboratory type indoor environments. Unknown objects like humans, chairs or trash cans must be detected in a wide surveillance area around the mobile robot even during fast motion. The estimate of the obstacle positions should, nevertheless, be accurate enough to avoid blocking of narrow passages. Section 2.1 details on a method that allows combined firing of many sensors while avoiding cross-talk errors. We assume that static structures such as walls, edges and doorframes are stored in a 2D map as natural landmarks. These a-priori known features are used for localization estimation. A fast-firing method that takes advantage of this prior knowledge is introduced in Section 2.2. The sampling rate is increased far beyond the limits usually imposed by the low speed of sound in air. To allow for several tasks to be performed concurrently, only the most appropriate subsets of sensors are used. This form of dynamic resource management is discussed in Section 2.3. 2.1. M e t h o d of V i r t u a l Point Source Moving with high speed in a cluttered hallway requires early detection of the closest unknown obstacles and the determination of their positions. Usually several sensors are used separately for this purpose. Disturbing crosstalk effects require elaborate error correction schemes. A more appropriate configuration for this application uses one single transmitter with the desired beam form and a receiving array configured of several receivers in order to evaluate incoming echos as reported in [8]. A typical surveillance area is shown in Figure 1 a). Since the objective is to use only transmitters located on the
Figure 1. a) Typical surveillance area. b) Combining physical transmitters to form a virtual transmitter. vehicle surface, this beam shape cannot be achieved with a single element. A virtual transmitter with the desired characteristics is obtained by appropriately phasing several adjacent transmitter elements with the main axis intersecting at the origin, Figure 1 b). For developing an adequate firing scheme, we assume a virtual circular wave which is starting from the origin. When the virtual wave front passes a physical transmitter, an ultrasonic pulse is fired. For M transmitters involved, an index vector I can be defined such that CIi < CIi+l, i = 1, 2, . . . , M -
1,
(1)
with Ci the distance from transmitter i to the origin. The time differences ti between firing transmitters Ii and/i+1 are given by CIi+~ - CIi ti =
V~
, i-
1, 2, . . . , M -
1,
(2)
with V~ the velocity of sound in air. This idea result,, in the following firing scheme: FORi=I,2
..... M-1
~
Fire S e n s o r I i Wait t~ Fire S e n s o r I M
! i
Let Tj denote the time difference between firing of the first sensor 11 and detection of an echo at receiver j, j = 1, 2, . . . , N, where N is the number of receivers. The distances Rj from virtual transmitter to the object and back to receiver j are then given by n j : T j . Ys -Jr-CI1, j • 1, 2, . . . , N . (3)
Figure 2. Fast firing
combining windowing and interlacing.
The number M of neighbouring transmitters employed depends on the desired beam width. Since the physical sensors' main axis intersect at the origin, superposition of the pulses r o u g h l y a p p r o x i m a t e s one pulse of a single centrally located virtual transmitter. Since we assume only one single pulse stemming from the origin, no decision on which physical transmitter produced the pulse detected by an individual receiver is required. Rotation of the beam to another focus of attention is achieved by electronically shifting the sensor arrangement. 2.2. Fast F i r i n g It is common engineering practice to focus on a certain interval around a nominal or predicted measurement value. For example, prior knowledge from a map could be used in conjunction with an estimated position to a-posteriori discard inconsistent measurements. However, we propose to go one step further and use prior knowledge at an earlier stage, i.e. during the data gathering process. Echos are received only in certain precalculated time-slots. This scheme is also applicable to reflectors not represented in the map. In this case, for windowing the distance predicted from previous data is used. Missing an obstacle occluding the expected reflector is impossible because the firing scheme is changed when the predicted time-slot is empty for several times. Due to the low speed of sound in air, the elapsed time between transmitting a pulse and receiving the echo is approximately 3 ms per meter travelled. With longer distances, interlaced transmitting and receiving becomes more and more appropriate. Thus, windowing and interlacing are combined in a highly effective manner. After initializing with a standard firing method, i.e. determining the expected echo return time T~, we receive the echo from the previous pulse after the next pulse has been transmitted. A window of length Tv + TN, with Tv = TN, is centered around the expected time of arrival. The most straightforward scheme for one sensor is shown in Figure 2. The pause time Tpkus~ is given by
Tpkus~ - T~ - 2. Tdead- 2 . Tv -- T N .
(4)
The expected echo return time Tk is updated with the knowledge from every new sample. Td~d corresponds to the dead zone of the transmitter/receiver-combination. Tv = TN = 0.5 ms and Td~ad = 1.5 ms results in the firing rate 1/T~ with
T~ = T E - T d ~ d - Tv
~
d - - " 6 m s - 2 ms. m
(5)
N "1-
)
'~--~ 4 5 5 .i....,
250 C U_
loo i
45!
. . . .
t + _ _ •
~
L
i ..........
!....
0.7
i
1
*
2
>
4 Distance d / m
Figure 3. Firing rate for fast firing of first order as a function of distance.
The dependency of the firing rate on distance d is shown in Figure 3. The order of the fast firing scheme is defined according to the number of pulses interlaced in the pause times. With simple first order fast firing, sampling frequencies above 100 Hz are achieved for distances of up to 2 m. Thanks to windowing, the scheme is invulnerable to noise and secondary reflections. It is applicable for d > (2. Tdead -4- 2" Tv + TN)" V~/2, in this case approximately 70 cm. If the tracking process breaks down because the object is moving too fast, the system restarts with initialization measurements. Fast firing of higher order is achieved by interlacing more pulses during the pause times.
2.3. Dynamic Reconfiguration The 24 transmitter and receiver elements arranged on a horizontal plane around the fast mobile robot constitute a homogeneous multisonar system. They are, however, combined to a variety of virtual sensing arrays depending on tasks and situations. This type of operation is similar to a concept described in [5]. Since virtual sensors are aimed electronically, there is no need for moving parts. All virtual sensor arrays consist of N arbitrarily distributed receiving elements. W h e n the angular region of interest is wide, a virtual point source combining several physical transmitters is used to illuminate the scene. On the other hand, when prior knowledge on reflector positions is available, the angular region of interest is narrow and one physical transmitter may suffice. 3. O b j e c t L o c a l i z a t i o n Based on the concept of dynamic reconfiguration, virtual sensors are formed from several physical transmitting and receiving elements. In general, we assume a virtual point source located at the vehicle's origin and an array of N arbitrarily distributed receivers. For the sake of simplicity, only two types of reflecting surface primitives are considered here: i) line reflector ~ walls, objects with small curvature, ii) point reflector --~ edges, doorframes. Both reflectors represent classes of real targets within the 2D sensor plane.
Figure 4. Measurement model for a) line type reflector b) point type reflector.
Given a reflector hypothesis and N measured distances Ri, i -- 1, 2 , . . . , N, an advanced object localization scheme should 9 allow for arbitrary receiver distribution, 9 provide position estimates with high angular accuracy, 9 discard inconsistent measurements, 9 quantify the estimates' uncertainty, 9 use redundancy: N receivers -~ N equations, 9 be valid for near/far-field reflectors, 9 be simple, linear, and non-iterative. The derivation of schemes that fulfill the above set of requirements is given next. 3.1. Line R e f l e c t o r Assuming specular reflection and introducing the mirror image of the transmitter, the following relationship can be obtained from Figure 4 a). R 2 - (C[ - 2x) z 4 - ( C ~ - 2y) 2 , i - 1, 2 , . . . , i
(6)
with c; = c,
= e,
(7)
With r 2 = x 2 + y2 the set of Eqs. (6) is rewritten in vector/matrix form
z+4r21-H'(X)+e_ y
(8)
with
c~ - n~
c~ ci'
c~-n~ z
,H
-
4
1
C~ C~
-
.
c ~ - n~
.
,
1
-
c~ c~
.
(o)
,
i
where e denotes an error vector. In a first step, x, y and r 2 are assumed to be independent variables. Then a weighted least-squares estimate of 3, 9 in terms of 2 2 is given by 9
- ~+4
~
(10)
with a
-
a.z,
G
-
(HTE-IH)
(11) (12)
~-G.1 -1. H T. E -1.
E denotes the weighting matrix. With the relationship ~
,
(13)
the following quadratic equation for 2 2 is obtained 16flTz~ 4 + ( 8 a T f l - 1)~z + a T a -
0.
(14)
Substitution of 2 2 back into Eq. (10) gives the desired solution. The optimal choice for E is derived next by adopting a set-theoretic point of view. The errors Ai in the measurement distances R{, i - 1, 2 , . . . , N, are assumed to be located inside an ellipsoid defined by a symmetric, positive definite matrix SA, i.e. (A,, A2, . . . , AN) S~' (A1, A2, . . . , AN) T
<__1.
(15)
According to the appendix, the error e will then be constraint by an ellipsoid defined by (16)
E - D. SA. D T
with D ~ {I-1
8(x,0) G}. - 8(&,9)/~ - 1
RD,
RD --
--2 diag(/~,,/~2, .-
"'
/~N).
(17)
The noise free value of 9 is denoted as ~ and I is the identity matrix. The distance of the actual error from the surface of the error ellipsoid in Eq. (16) is given by d- 1-
[ z + 4r2_1- H ( ~ ~) ] f
E -1 [ z + 4r21_- H ( 3 ~) 1
.
(18)
If d is negative, the solution is not valid within the constraints given in Eq. (15) and may be discarded. Otherwise (0 <_ d < 1) the solution is valid and contained in an ellipsoid given by - d. (HTE-1H)
-1 ,
(19)
which is centered at 5, ~. Since the matrix D contains the true distances/~i and the true position (~,~), some approximations are required to arrive at a feasible solution for the weighting matrix E. The measured distances R i are used to approximate the unknown true distances /~i. In a first step, R D 9S A 9R D is used in place of E to solve for a, and then ~2. Next, the resulting ~2 is used as an approximation of the true ~2 in Eq. (17) to calculate D. Although the described process could be repeated to obtain even better solutions, iteration does not lead to significant improvements in practice. Special Case: Linear Array For a linear arrangement of receivers, a simpler solution may be obtained and may replace the above formulas. When the transmitter is also lined up with the receivers, this solution must be used to avoid singularity of G in Eq. (12). Without loss of generality the linear array is assumed to be arranged along the x-axis. Then Eq. (6) simplifies to C~ - R ~ -
4 C[ x + 4 y,
i -
l, 2,...,
N,
y -
C V y - r 2.
(20)
In matrix form, this reads as C~2 -
( )x
z-H
+e
z-
,
R~
C~- R~ ,
H-4
C~v- R~v
C~
1
~
o
C~1 C}
(21)
9
1
With ~2 _ 52 + ~2 an estimate ~ is found from ~2 _ CV?~+ ~ + 52 _ 0.
(22)
The error e is then contained in an ellipsoid given by E -
RD.
S A " RTD .
(23)
A straightforward first-order error propagation analysis provides A~
~ F.
A,)
, F-
(,
2~ cy-2~
0)
1 cy-26
9
(24)
and it follows that -
d. F.
( H T E -~
H) -1.
(25)
F T .
In the special case, the solution is obtained in one step. 3.2. P o i n t R e f l e c t o r From the measurement model of Figure 4 b) follows that
( n ~ - ~)~
__
(c~x - 9)2 + ( c ~ -
y)~, i - ~, 2 , . . . ,
g.
(26)
This can be written as
z+2rR-H
(x)
n,
+e
R-
n~ .
c i ~ c~'
,H-2
c~c~ . .
.
(27)
Again, x, y and r are first assumed to be independent variables. The estimate ~, ~ in terms of ~ is then given by
with ~
=
G -
G . z , ~ = G . R
(29)
(H TE -1H)-1. HT " E_I.
(30)
With Eq. (13), a solution for ~ is obtained from (4/3T/3 -- 1)~ 2 § 4aTl3i" + aTa = 0.
(31)
Using similar arguments as for the wall reflector, the error e will be located within an ellipsoid defined by
E = D. SA. D T
(32)
with D
~
(I -
h2(~,~)~_ ~
a]
+
(33)
Special Case: L i n e a r A r r a y When the receivers a n d the transmitter are lined up, G in Eq. (30) becomes singular. Without loss of generality, the array is again assumed to stretch along the x-axis. A solution for ~, ~ is then obtained from
c~ z-H(X] +e'H-2v
Cf.
-R1 -R2 (34)
c~ --RN and
(35) in a very similar fashion as for the line reflector. 3.3. D i s c r i m i n a t i n g R e f l e c t o r T y p e s Besides localizing objects for which the reflector type is a-priori known, it is often useful to identify reflector types based on actual measurement data. The additional surface information simplifies the determination of line endpoints and outlier elimination for subsequent segmentation algorithms. The hypothesis "line reflector" is accepted, if the line condition Eq. (18) is true. In analogy, the hypothesis "point reflector" is accepted, if the respective condition is true. No decision is possible, if both conditions are true. Both hypothesis are rejected, if no condition is true. For the prototype implementation considered, the measurement error is in the order of magnitude to the differences between the distances for the two reflector hypothesis. Numerous misinterpretations are the result. The amplitude of the returned signals is very well suited to support decisions in conflicting cases. Unfortunately, amplitude information is not yet available in the current prototype.
10
Figure 5. Overview of the ultrasonic sensing hardware.
4. I m p l e m e n t a t i o n Issues The prototype implementation comprises 24 piezo-ceramic emitter and receiver elements operating at 40 kHz. They are fixed in a horizontal plane around the mobile platform. The maximum range is about 4 m for most reflector types. The minimum range the dead zone is approximately 0.25 m. Each sensor is connected to an analog signal processing unit which generates pulses and detects the returned echos. A tandemarranged digital signal processing unit determines the elapsed time between transmitting a pulse and the returning echos. The evaluation units are fully decoupled. The digital electronic is mapped to the memory of a local transputer network consisting of four T805 transputers, Figure 5. This network takes care of configuration, the sensor firing, data gathering, and prefiltering. For experimental evaluation of the proposed multisonar system, the omnidirectional service robot ROMAN (ROving MANipulator) has been used. It is a full scale mobile robot, Figure 6, adapted to indoor requirements: width 0.63 m x depth 0.63 m x height 1.6 m. Three independently steerable wheel systems provide excellent maneuverability. Wheel diameters of 0.2 m allow travel across rough surfaces like carpeted floor. The maximum speed of 2 m/s has been chosen to correspond with human walking speeds and wheel-chair velocities. Steering commands are emitted every 10 ms based on actual position information. A lightweight 7 DOF manipulator with a maximum range of 80 cm is mounted in one corner of the platform. Besides the ultrasonic sensors, ROMAN employs a laser navigation system, a wheel based dead-reckoning system and a low-cost gyroscope. The local sonar transputer network is part of a larger distributed network which takes care of the sensor system. 5.
Applications
So far, a number of tools for enhancing the perception capabilities of sonar array sensors have been introduced. Their application depends on how structured the environment is, whether the objects are static or moving, and the degree of prior knowledge. Table 1 lists the rules that allow the system to automatically select sensing strategies depending on the current task and situation.
ll
Table 1 Rules for automatic selection of sensing methods. # of transmitters per array
# of concurrent arrays
# of receivers i per array i
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
data rate per array
fast firing . . . . .
_
virtual point source __+
.
.
.
.
.
.
.
.
I
j
Localization for navigation
1
typ. 2-3
cf. Fig. 3
5
X
i
i
i
I
Obstacle detection
!
6-10
1 .
updating
2-3
.
.
.
.
.
.
.
.
t
1
.
.
.
.
5
.
.
X
25 ms
6-10 .
!
25 ms i
5.1. E x p e r i m e n t s in Cluttered Environment In a cluttered environment, the multisonar system is used to detect and localize obstacles. Absolute localization of the vehicle is performed by the laser navigation system with respect to known artificial landmarks. To demonstrate the performance of the proposed system, the following experiment has been conducted: While following a given course, the robot encounters two obstacles of different size. A virtual point source consisting of 8 physical transmitters points to the travelling direction of the vehicle. 6 receiver elements are employed to detect the incoming echoes. For simplicity, it is assumed, that the reflecting surfaces may be approximated by lines within their 2D projection. Every 25 ms, 6 distances Ri corresponding to the first echoes are measured. The procedure from Section 3.1 is used in an order-recursive manner to combine these measurements. Inconsistent combinations are discarded. Valid combinations are kept and provide the reflector position in the vehicle frame. Via the known transformation vehicle ~ world, the object positions are converted to the world frame and plotted in Figure 7 without further processing. Only a few outliers are generated. For the small obstacle, however, the line reflector assumption leads to errors in the localization estimate.
5.2. E x p e r i m e n t s in Structured Environment In a structured environment, a few a-priori known natural landmarks such as walls and edges are tracked and used for updating the vehicle's position and orientation. The number of employed virtual sensors is equal to the number of simultaneously visible landmarks, typically 2 to 3. Every sensor array employs one physical transmitter and up to 5 receiving elements. At any instant of time, the transmitter pointing closest to the predicted landmark is chosen. Fast firing is used to exploit a-priori knowledge on the reflectors' positions. This leads to very high update rates. For demonstration purposes, the vehicle moved along a wall. Based on the measured distances, the wall position is calculated with respect to the vehicle frame with the scheme discussed in Section 3.1. Since the wall position is known, the vehicle position in world coordinates may be obtained. The resulting data is compared with localization data from the laser navigation system, Figure 8. A satisfactory correspondence of data from both measurement systems is achieved. The raw data is used subsequently to calculate the actual position estimate. The previous estimate is propagated by means of dead-reckoning
12
Figure 6. Mobile manipulator ROMAN.
and gyroscope and fused with localization data from every visible landmark. The respective uncertainties are again modeled based on a set-theoretic approach. Statistical modeling would not capture the systematic errors caused by coordinating the three wheels. According to this point of view, the fusing process results in uncertainty set intersection [11]. Going into more detail would take us too far afield.
6. C o n c l u s i o n s A new high performance multisonar system for application with fast (2 m/s) omnidirectional mobile robots has been developed. 24 transmitter and receiver elements are arranged in a horizontal plane around a mobile robot and constitute a homogeneous construction set for forming of transmitting and receiving arrays. Depending on mission and situation the individual elements are combined to form several heterogeneous virtual sensing arrays, each designed for carrying out a given task. Virtual sensors are aimed electronically, i.e. there is no need for moving parts. High frequency firing schemes exploit a-priori information about the current environment to purposefully collect data in a surveillance area of 8.6 m in diameter. This makes the proposed system superior to those with fixed data gathering schemes. The results of experiments underline the system's improved perception capabilities in detecting obstacles during motion. Other tests demonstrate sonar localization results that are in good accordance with data from a laser navigation system.
13
Figure 7. Obstacle detection during motion.
Figure 8. Sonar localization of the vehicle compared with a laser navigation system.
REFERENCES
1. Borenstein, J. and Koren, Y., "Noise rejection for Ultrasonic sensors in Mobile Robot Applications". Proceedings of the 1992 IEEE International Conference on Robotics and Automation, Nice, France, S. 1727-1732, May 1992. 2. Burks et al., "Autonomous Navigation, Exploration, and Recognition Using the HERMIES IIB Robot". IEEE Ezpert, Vol. 2, No. 4, S. 18-27, 1987. 3. Crowley, J. L., "Dynamic world modelling for an intelligent mobile robot using a rotating ultrasonic ranging device". Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, MO, 1985. 4. Drumheller, M., "Mobile Robot Localization Using Sonar". IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, No. 2, S. 325-332, 1987. 5. Fukuda, T. and Nakagawa, S., "Dynamically reconfigurable robotic systems". Proceedings of the 1988 IEEE International Conference on Robotics and Automation, Philadelphia, PA, S. 1581-1586, 1988. 6. Kampmann, P.; Freyberger, F.; Karl, G.; and Schmidt, G., "Real-Time Knowledge Acquisition and Control of an Experimental Autonomous Vehicle". Intelligent Autonomous Systems, S. 294-307, 1986.
14
.
10.
11.
12.
13.
Kortenkamp et al., "Integrated Mobile-Robot Design, Winning the AAAI '92 Robot Competition". IEEE Expert, Vol. 8, No. 4, S. 61-73, 1993. Kuc, R. and Barshan, B., "Bat-Like Sonar for Guiding Mobile Robots". IEEE Control Systems Magazine, Vol. 12, No. 4, S. 4-12, 1992. Kuc, R. and Siegel, M. W., "Physically Based Simulation Model for Acoustic Sensor Robot Navigation". IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, No. 6, S. 766-778, 1987. Nagashima, Y. and Yuta, S., "Ultrasonic sensing for a mobile robot to recognize an environment - Measuring the normal direction of walls -". Proceedings of the 1992 I E E E / R S J International Conference on Intelligent Robots and Systems, Raleigh, NC, S. 805-812, July 7-10, 1992. Sabater, A. and Thomas, F., "Set Membership Approach to the Propagation of Uncertain Geometric Information". Proceedings of the 1991 IEEE International Conference on Robotics and Automation, Sacramento, CA, S. 2718-2723, 1991. Sabatini, A. M., "Active Hearing for External Imaging Based on a Ultrasonic Transducer Array". Proceedings of the 1992 IEEE//RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, S. 829-836, July 7-10, 1992. Walter, S. A., "The Sonar Ring: Obstacle Detection for a Mobile Robot". Proc. IEEE International Conference on Robotics and Automation, Rayleigh, NC, March 31. - April 3., S. 1574-1579, 1987.
Appendix The error e in Eq. (8) may be written as ~ - s + ZXz + 4 ( ~ ~ + z x ~ ) l -
H
~
(36)
- ZXz + 4 z X ~ , ! .
From Eq. (14) follows that 16Z~Z(~ ~ + ~x~) ~ + ( s ( a + ~x.)~Z -
~)(~ +
~ x ~ ) + (a + ~x.)~(a + A . ) = 0.
(37)
If squared error terms are neglected, error At2 is given by 2(4~3 ~ +
a~)
Ar~ ~ - 8(4~2/~T~ + 5 T 3 ) _ 1
a.
= _ AO~
"
(3s)
From Eq. (11) it is concluded that An = G" Az and we get from Eq. (9)
Az ,-~ RD "
A1 A2 .
9
(39)
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
15
P r o p e r S e l e c t i o n of S o n a r a n d V i s u a l S e n s o r s in V e h i c l e D e t e c t i o n a n d A v o i d a n c e N. Moghaddam Charkari, Kazmmri Ishii and Hideo Xlori * Department of Electrical Engineering and Computer Science "~unanashi University
A general 1)aradigm for vehicle detection and avoidance by proper selection of visual and sonar sensors is 1)resented. The most appropriate sensor for detection and navigation of the robot is selected in accordance with the sensor fimctions and limitations. This strateg T is activated when one sensor can't obtain enough information for navigation of the robot because of sensor limitations. Five types of actions based on stereotyped motion are used for th(, navigation system: Moving Along Wall, Moving Along, Moving Toward, Tracking Motion and Danger Estimation. The first is sonar based and the rest are vision based a('tions. Visual detection of the vehicle is 1)ase(l on the locomotion strateg3' "Sign PattcrT~ based Stereotyped MotioTi'. Th(" sign pattern of the vehicle is the shadow underneath the vehMe, which is dark('r than any other parts of the paved roa(l. The rol)ot detects the sign pattern of a vehMe fi'om a (listance of 30 m and extracts its front shape 1)y video rate pro('essing.
1. Introduction One of the most important tasks for the navigation of mol)ile robots in outdoor envirom nents is the ability to detect and avoid moving or I)ark('(l vehicles. In this paper, sonar sensors are used in combination with vision to compensate for some of the difficulties and limitations encountered in the machine processing of visual information. The proper use of multiple sensors is becoming an in(,reasingly important area of research and appli('ation to improve the capabilities of intelligent machines, because of the advantages and the limitations of each type of sensor, most mobile robots use a combination of different sensors to perform some tasks. Each of the sensors in a defined task may be operated synchronously or asynchronously with the other sensors, depending on the Selection Process. Selecting the most appropriate sensor, or group of sensors, during the navigation of the mobile robot is also perform('d by the Selection Process. The strategy of selection dep('nds on the limitations of the s('nsors. For instance, a visual sensor observes an entire scene and can be used for dete('tion an(l re('ognition of objects that are distant, while tactile and sonar sensors are used to ol)tain inibrmation about a few objects that are nearl)v. *This work is parti~dly supported l)y Grmlt-in-Aid for Sci(,ntifi(" Research from the .Ministry of Education, No. 03650343 and fr(nn tile Okawa Institute of Information and Tel(wommunication No. 91-32.
15 Two (liffcr,'nt al)l)roachcs to the s('nsor Sclcctiou Process are Preselectiug during initialization and On.line selection in resl)OnSC to changing (,nvironment or syst(un conditions. Gaskell and Prol)crt [1] discussed the use of De,.cisiou Theoretic PTi'aciplcs for sensor management in multiscnsor systems. Their fi'amcwork i)r(~vidcs a way of rcl)rcsc:lting sensory data and choosing actions under uncertainty. Luo and Kay [2] surveyed a variety of recent al)l)roa('hes to the pro])lcm of multiscnsory information. G('neral multiscnsor fitsion m('tho(ls, sensor selection strategy, an(l some other apl)roachcs and applications are discussc(l in their report. In the NAVLAB paper [3] sonar, a lapser rangcfindcr and a color vision system are used for object detection as well as navigation in outdoor environment. The design of the architecture can support parallel processing and th(, development of multiscnsor integration has been the major goal of this research. h:imoto and "~]lta [4] dcvclol)cd a method based on the fusioll of internal sensors and ultrasonic sensors, ms external sensors, using an extended Kahnan filter for the navigation of the YAMABICO rol)ot. Information on the 1)lamlcd path and environmental data are the prior knowledge of the robot. Sonar, a laser rangcfin(lcr and two color vi(h'o camera.s have been Cml)loycd for roadfollowing and ol)jcct detection appli('ations on road environments in ALV [5]. Color video information is us('(l to locate the road, as the las('r rangcfindcr may siml)ly be confllscd if there is litth" diffcrcncc between the road au(l th(, surroundings. On the other hand, la.,~cr range information is used to ol)tain an accurate (lcscril)tion of the geometrical features of obstacles on the road. Visual ol)jcct detection is one of the main subj(,(,ts in robotic studies. Among such objects, vehicles are the :::()st troublesome ones in out(loor cnviromncnts. Several methods for vision 1)ascd vehicle detection have 1)con prol)oscd l)y researchers. A dynamic 2-D model has been iml)lcmcntc(l on VaMotLs [6]. This model is l)ascd on the notion that the world is structured, rather than monolithic, and consists of individual objects. It allows the determination of the relative Sl)ced and distance in real time and may guide appropriate maneuvers if necessary. In this method, the right and left. sides and the bottom of a car arc detected horizonta.lly and vertically by a local edge operator. Symmetry is another method which provides a useful feature since vehicles' rears are generally contour-symmetric and spatially gray-level-symmetric about a vertical centerline [7,8]. It means that an appropriately shifted mirror image of a vehicle overlaps itself. However, the error rate may increase when the mctho(1 is performed in a highly structured scene.
In our earlier article [9] we discussed a method 1)ascd on "Sign Pattern based Stereotyped Motiou" for vehicle detection. There the robot (lots not need to fully understand the obstacle as it reconstructs the 3-D shal)e of the obstacle in its memory. The robot only needs to obtain signs to such a degree that they inform it of the existence of an object in its way. The sign wa~s referred to as "sign 1)attcrn" in the earlier work. We found shadow underneath a vehicle to 1)c a rclial)lc sign 1)att('rn of the vehicle [10,11]. \Tchicle detection ba.,~ed on sign 1)attern is one of the lowcr-lcv('l COml)ctcn('ics of the rol)ot's information set of the cnviromncnt. Therefore, it does not nc('d highly computational image processing. Moreover, it cnal)les the s.x'stcm to overcome the tra.cking process, i.e. the ability to keep a vehicle in track l)y continuously sensing its sign 1)attcrn in real time.
17
Figure 1. The structure of HARUNOBU-6
In section 2, the structure of the HARUNOBU-6 robot and its explained. A simple reflection model of the road surface is discussed developments of the visual vehicle detection module can 1)e found in paradigm of proper sensor selection is discussed in section 5. Finally, the experimental results and conclusion respectively.
2.
sensory system are in section 3. Recent se(-tion 4. A general sections 6 and 7 are
S t r u c t u r e of the R o b o t
The work on developing the compact rol)ot HARUNOBU-6 (54 cm x 84 cm x 84 cm) has been undertaken in our laboratory since 1992. The structure of the robot is shown in Figure 1. The undercarriage is a motor-tricycle developed by Suzuki Co. Ltd.
2.1. Sensor S y s t e m Two types of sensors have been installed on the rol)ot: internal and external sensors. The internal sensors include encoders, a magnetic directional sensor and a gyroscope. The external sensors are a video camera and six sonar sensors. Sonar sensors are attached to the right, left and front of the robot. Two sonar sensors which are attached on the front of the robot are used for obstacle detection. The remaining sensors are used for moving along an object. Data from sonar sensors is sampled every 50 ms while the frequency of each sensor is about 40 kHz. The video camera is set at a height of 84 cm with a tilt angle of 12 ~ A VME image memory system (CPU 68040, 25 MHz) is used for ilnage processing and navigation. The communication between the image processing system and the vehicle motion control systern is performed through an RS232C connection. Data from internal and sonar sensors are fed to a UP P (universal pulse processor) for flu'ther processing. The UP P contains a 10-bit A / D converter, 1 kB RAM and a pulse modulator with counter. The control of the UPP is done by a single board control 1)ro('cssor (CPU V25, 8 MHz).
18 !
urce I Y
L _
(o
o•'4000
I
o Io
/'
//7
~ i~L ~' r /-/ J -- ~ ' ~ ' - :
vJow
|
$
E
C
un
.
oo E
D i f f u s e Light
_ . . I: ~
...
,~4,
: , 'IX
.:
9
3000
I """"'=,=
I
5
'
I~o
'
3~o angle
Figure 2. Schematic of specular and diffilsed light. Incident angle is Oi degrees in the x-y 1)lane. View position is defined by
(0~, 0,.).
~ r
Figure 3. Image intensity curve of the reflected light on the a.sphalt road when view position 0,. changes from 0 to 360 ~ and Oi =70 ~ and ~ = 60 ~
3. N o t e s o n t h e T e r m i n o l o g y
Tile terminologT employed in this stu(ly to describe the reflectance geometry and tile reflectance model is illustrated inFigure 2. A surfa~'e of an object is illuminated by the sun light I~,, and the sky light I~ky. The light s o u r c e is assumed to lie in the x-y plane with the zenith angle 0i. The light is reflected in the ((r , 4),.) direction, where 8~ denotes the angle between the normal vector of the surface and the reflected.light which is seen from the viewpoint of the video camera. ~ is the reflected light angle between the plane of I~,, and the normal vector. When the view position lies in the reflected direction (0i = 8,.), then the luminance of a surface is mostly evaluated by the specular component of the light source. Furthermore, if I,,,, becomes ahnost parMlel to the surface, the surface behaves like a mirror and all the light is reflected in the ~)i = 0~ direction. The reflected light is called "specular light" and represented by R,~. The specular component ha~s a spectrMenergy distribution close to that of the incident light. The directional characteristic of R~pr is called the "specular lobe" and is expressed by R~
= L,,,, " s i ,
e,. . cos4(t~i - 0 , ) . cos40,..
(1)
When I.,,,, is nearly perpendicular to the surface the light penetrates into the substance of the object and is partially scattered in many layers. The scattered light is called "diffused light" and is represented by RdL,. The (liffused light of a surface is given by Rd/s = P" h~,, " cos(Oi - 0r)(1 -- kCO.S" 20,.)/(1 -t-/,:),
(2)
where p is the reflectance factor of the sllrfa(,e and k denotes the fraction of diffuse light. Equations (1) and (2) show that surface reflectance not only depends on the material properties, but also on the viewing geometry. Figure 3 shows the image intensity curve of the reflected light on the road surface in one of our experiments. 0i and 0r are assumed to be almost equal (0i = 70 ~ , 0,. = 60 ~ while viewing position 0~ varies from 0 to 360 ~
19 Tal) le 1 Luminance of road surface on a clear day Clear weather Road surface type In specular light In diffused light Shadow of vehicle Underneath vehicle
Luminance (cd/m 2) Dry surface Wet surface 3600 to 4800 2020 to 3000 2600 to 3300 850 to 2200 400 to 900 275 to 500 100 to 200 60 to 150
Table 2 Luminance of road surface on a cloudy day Cloudy weather Road surface tyl)e Open parts Underneath vehMe
Lmninance (cd/m2) Dr3' surface Wet sm'fa,ce 2610 to 3500 2000 to 3000 105 to 140 80 to 100
On tile other hand, tile reflected light of the sun R.,,,,, is generally expressed by
R:,,, = R~p~ + Rd/.~.
(3)
The. total reflected light R, is given 1)3: R -
a l 9R , , , , + 0'.2. R , k y + c~a. R~,,v,
(4)
where at, (~'2and a3 are transparency factors equal to or less than 1.0. From the viewpoint of optical science, the sunny part of the asphalt road is illuminated by tile sun and the sky light (al - a2 = aa = 1) while the sha.d('d part is only illuminated by the blue sky light (a'l ~ 0, a2 = 1, aa - 1). The area underneath a vehicle is illuminated through the gap between the vehicle and the ground l)y the faint environment light R,,,,, (at - ~2 = 0, oa - 1). Equation 4 shows that although the luminance of the shadow underneath and the other shaded parts on the road apl)ear similar to the human visual system, this is an illusion of brightness constancy. To determine 1)recisely the above differences, the luminance of sunny regions, shadows, and the areas underneath a vehicle on dr3" and wet road surfa.ces were measured. Tables 1 and 2 show the result of the experiments on a clear day and during cloudy weather conditions respectively. It was found that the area ml(lerneath a vehicle on a ('lear day is markedly dm'ker than the shaded parts and wet patches in luminance. Similarly, tile area. underneath is nmch darker than the other open parts of the asl)halt road on a cloudy day when there is no shadow. Moreover, the luminance of the shadow underneath shows little change while those of sunny and shaded parts have no consistency because of the different sources of illunfinating light. Finally, the shadow underneath is not responsive to change in the viewing position and weather conditions.
20
Figure 4. Flow diagram of vehicle detection and tracking process
4. Vehicle D e t e c t i o n Algorithm Visual vehicle detection and tracking is (lecoml)osed into three layers. In the lowest layer the candidate vehicles are detected l)y the Selection sul)-module. In the middle layer, the pseudo-objects which arc wrongly selectte(l as vehicles are identified. In this l~'er, some additional features of vehicles are investigated by the Confirmation sul)-module. In the third layer, the type of motion is investigated and the candidate vehicles are tracked. Moreover, the trajectory of vehicles in motion and the motion direction, which are important in danger estimation, are found during the tracking process. The entire system structure is composed of the following sub-modules: Window Set-up, Searching Underneath, Confirmation, Motion Estimator, Danger Estimation, and Tracking and is shown in Figure 4. In the following the operation of ea~'h sub-module is discussed in brief. (1) W i n d o w Set-up: A fi'ame is digitized by its intensity and the right and left lane marks are detected by the line-fitting method. Then a thin window Wt, with its length equal to the road width, is set up at a position corresponding to a distance of 30m ahea~l of the robot on the image plane. (3) Searching U n d e r n e a t h : The vertical projection is performed on the window rWt. The candidate pixels are extracted and linke(l into the line segments by comparing the projection with the threshold value Trl. Then, the short line segments are eliminated and the collinear line segments are merged. The result is a new line segment L, = (~,, ui, ~, ui)T representing the a.rea underneath the vehicle.
21
Figure 5. Ilesult of vertical projection on a road partly covered by shadows. The area underneath a vehicle is clearly distinguished from other parts of the road surface when it enters the search window.
Figure 5 shows ml example of the vertical projection when the area underneath a vehicle enters the search window on a. road partly covered by shadows. As can be seen, the area underneath is clearly distinguished from other parts of the asphalt road by the intensity vahle. (4) Onc0ming/Parking Detection: Physiological studies suggest that the human visual system may use instantaneous motion information COmlmted fl'om two successive flames. In our work, once two feature points of t lw area underneath the vehicle ((~,, I~,)T and ((~,ui) T with length of Li are extracted in the search window 11,'1, then the next window II.) is set up at (ui + 7~e) as the next image. (ui + ~ite) indicates the shift of the search window on the 2-D image. Then, the corresl)onden('e of these points in successive images is determined. If the correspondence is estalJlished within an interval of time tl th('n the detected vehicle in lI'l is identified as an oncoming vehicle; otherwise it is a.s a stationary w'hicle. Since moving vehicles have a smooth and continuous and not a. transverse motion, we have emplyed a time interval tl which indicates the teml)oral 1)roperty of the area underneath when it is in motion. ( 5 ) C o n f i r m a t i o n : Moving vehicle detection based on two successive fl'anws may include some errors caused by distortion, random noise, 1)lurring, etc. The uncertainty can be considerably reduced by the following assumptions: (5.1) Number of Line Segments: The presence of at most two line segments in the vertical projection. When these are two line segments, one line segment corresponds to the shadow underneath all oncoming vehicle a~ld the other to that of a departing vehicle. (5.2) Length of the Line Segment: The length of the line segment should be greater
22 than the thresholcl value/Ji defined as tile metal value of vehicle width on the 2-D image. The threshold value increases with decrease in the depth of view. (5.3) Vertical Edge O p e r a t i o n : Vertical edge operation is performed on the t)eginning and the points of the line segment to detect the left and right sides of the vehicle. Confidence is defined to find the accuracy of measurement. It is represented 1)y the square root of the inverse of the vaa'iance of the candidate edge points. The smaller the variance, the higher the confidence of the measurement. (5.4) Vehicle Side S y m m e t r y O p e r a t i o n : The cletected eclges on tile left. and right windows belong to tile vehicle sides if they are symmetric about the vertical centerline defined by the line segment L,, = (g~,,,~'i,~,:~,,,(vi - Li/2)) T. ~,~,, is the mean of {, and ~ and Li denotes the length of the line segment expressed by ({~ - ~,). (5.5) G r a y Level S y m m e t r y : Window ITQ,,d of size (Li x K ) is set up at tile position of tile candidate. Two columns of pixels in Wu,,d are gray level symmetric al)out tile vertical eenterline Lv if
k
~-~[(a(eent-i)j-
I < A,.
(5)
j=l
where Ag indicates the threshold value during tile symmetry operation. This process is done for i = 1,2 ..... , ({+ - {,)/2. The extent of symmetry is found to be proportional to the number of pixels which are gray level symmetric. As the vehicle's underneath is only illuminated by faint light from the environment, the difference between two column pixels gradually increases from the vehicle's centerline to tile sides of tile vehicle, in tile small range of Ag. This feature is effectively use(l to clistinguish the area underneath the vehicle from pseudo-objects in tile mid-range distance of tile vehicle on tile road in tile vehicle detection module. (5.6) H o m o g e n e i t y : Homogeneity is one of the main properties of the area underneath a vehicle. As this area is not directly illuminated by tile sun or tile sky light, the intensity dispersion of pixels has little change. Therefore, the average intensity and tile standard deviation of this area can be efficiently used to distinguish the shadow underneath a vehicle from pseudo-objects, such as, low-contrast shaded parts of an asphalt road. If the confirmation is successfully completed then tile detected segment is assumed to be the sign pattern of a vehicle, otherwise it is noise. (6) Tracking Vehicles: Tile tracking process gives tile robot the ability to estimate the safe distance between the robot and the vehicle. A vehicle is detected and tracked in successive images by shifting the dynamic windows Wi (i - 3,4,...) to tile lower side of the image. Tile size and the position of tracking window Wi is determined by the length, center position, and ordinate of tile line segment in tile tracking window Wi-l. The length of the tracking window is given by
Wi = W~-I + AL
(6)
where A L is expressed by A L --
L i - l [ R i - ~ - Di • 2tan(aJ/2)] Di • 2 tan(a/2)
(7)
23 and a a n d / ? i - I are the field of view (F()\,') and the n,lative width of tile area subtended by the camera at distance Di-t respectively. L;-t denotes the length of line segment in window IV/_ I. Finally, depth, velocity, and width are determined by analyzing the time series of the ordinate of the area underneath the segment.
5. Proper Sensor Selection Two types of external sensors, visual and sonar, are currently used in vehMe detection and avoidance in HARUNOBU-6. The proper selection of sensors is the main purpose of the control s3"sWm of this robot. In the whoh, system, the most al)propriate group of sensors, or only one type of sensor, is selected to use as a response to changing conditions during the robot's motion. This means that all the sensors do not seem to l~e combined but rather, depending on the environmental conditions and the sensors' fimctionalities, the data fi'om one sensor tends to dominate. This helps to simplify the control program, while in the case of combining or filsion the complexity of the control sysWm grows exponentially [2]. Five types of actions are currently undertaken in HARUN()BU-6; Moving Along Wall (MAW), Tracking Motion (TM), Moving Toward (MT), Danger Estimation (DE), and Moving Along (MA). The first action is a sonar based and the remaining actions are vision based. The sign pattern of MA action is white lane marks or long edges along the ground, while a line segment found from the shadow underneath the vehicle is the sign pattern in TM, M T and DE actions. The principles of MA W and MA actions have been discussed in [9] and [12]. Each of these actions consists of simple processing whicll enables the robot to respond to sudden changes in its environment quickly. Selection of an action is determined by knowledge of tile limitations and the advantages of each type of sensor as well as by the current action of the rol)ot. The relevant action is reliable if its response to the enviromnent is positive. Two new actions M T and TM applicable to vehMe detection and avoidance are presented in this paper. Tile selection of TM and M T is done in the Motion Estimator module as shown in Figure 4. TM is used fox" tracking an oncoming vehMe. Distance 1)etween tile robot and tile vehicle as well as tile collision time are continuously estimated in this action. It is a useful action which ensures the robot maintains a salt' distance and finds a free route for its motion. On the other hand, M T is used fox" tracking a parked vehicle. Continuous detection and tracking of a parked vehicle helps the robot to evaluate its distance from that vehicle as it moves. Moreover, the width of the vehicle and the free path are estimated and fed to tile control system for the avoidance process. A free path in the road is found from knowledge of the road width and tile estimated wMth of the vehicle. It indicates whether tile path can be followed or not. Finally, DE is activated during the tracking process. The trajectories of vehicles as well as their direction of motion are found during this action. The output of this action indicates whether the route is safe or not. The result may be used to decide the next action of the rol)ot: for instance, the decision to ('ross an intersection, make an elnergency stop, or continue in motion, and so on. Tile detail of this action has been discussed in [13].
24
Figure 6. Stereotyped motion of the robot in avoiding a vehMe parked at the side of the roa(l
Fig~lre 7. The state transition diagram
5.1. O r d e r of P r o c e s s Figure 6 shows the Stereotyped Motion of the rol)ot in tile vehMe avoidance process. A general 1)aradignl for vehicle detection an(l avoidance l)y the l)roper use of visual and sonar sensors is shown in Figure 7. In the first phase, one of two actions, M A W or MA, is activated. In both cases the global information concerning the environmellt is ol)tained ])y the visual system. This information is ('omI)osed of the position an(l the angular directions of the left and right lane marks. If a vehMe is detected during the rol)ot's motion then one of two types of action, TM or MT. is started as a result ()f the motion estimator. At the same time, DE is activated to estimate the danger coefficient of vehM('s. The relevant action is continued until the visual sensor fails to provide sutfi('ient information about the vehicle. In other words, the uncertainty of visual sensory data progressively increases a,s the detection
25
fre(lllency decreases. Thus. the rehwan('e and the r('lial)ilitx" of this sensory data to the ('ulrent action is re(luce(l an(l the resl)()nse fl'()m ()ther t.X'l)eS of sensory data ten(ls to dolninate. Ill 1)ractice, this hal)l)ens at ('h)se l"ang(', less than 61n, when the visual sensor can't see the entire vehicle an(l fails to dete('t the sign 1)attern ()f the vehicl('. In this case, the reliaMlitv of the visual sensor reaches z('r() while that of the sonar sensor increases and 1nay reach unity. The estimat(,d width of the v(,hi('le, the al)l)roximate distance 1)etween the robot and the vehicle, and tile free path. are sent to the control system at this 1)oint. In the secon(l phase, the fl'ont sonar sensor is activated to prepare inforlnation on the position of the rear of the vehicle. The robot continues its motion until it is f meters from the vehicle, where distance f is provided by the front sonm" sensors. In the third phase, avoidall('e occurs. The avoidance lnaneuver is set up to sidestep to the right of tile 1)arked vehMe. The inforlnation ()l)tained fi'om the visual sensor, i.e. width and fl'ee path, as well as that fi'Oln the front sonar sensors, i.e. distance f, is used for ax'oiding the vehicle. The robot avoids the vehi('le 1)y following the sylnmetrical arc R. The interior angle 0 and the radius r of the arc ar(, given 1)y .)
II 2 -1- I',
-
~
(8)
,
4u II
0 - 2arctalt-,
(9)
U
where u and v are define(l as follows: u-
a
IV
+ s
~,-f+~,
+ -~-,
L
(10)
(11)
where L and W are the length and width of the rol)()t, a is equal to half of the vehicle width, and s is define(l as the distance between the rol)ot and the vehicle's side while it is moving along the vehicle. In the fourth phase, the M A W action is activated. MA can't be selected, as the left, lane mark is not in the visual field of the rol)()t. Xloving along the vehicle's side continues mgil the vehicle's side is not detected by the sonar sensors attached on the left side of the robot. In this case, the robot uses its locati()n 1)efor(, the avoidance process and returns to its position during the M T action. Finally, phw~e 1 will 1)e activated again.
6. E x p e r i m e n t a l R e s u l t s The vehicle detection and avoidance systenl was tested in the university carol)US. A vehicle was parked 30m in front of the rol)ot. Figure 8 shows the locus of the robot during motion and avoidance processes. The order of 1)ro('ess('s was as exl)lained in the previous section. The avoidance process was activated when the rol)ot reached 2m in front of the vehicle. The robot continued its motion, keeping l m fl'om the right side of the vehicle, using the ilffOrlnation received fl'om the s(mar sensors attached on the left side of the robot. Figure 9 shows the result of six frames of visual vehicle detection flom 30m to
26
Figure 8. The locus of the robot during motion
Figure 9. The result of vehicle detection in six images
5m in distance. Figure 10 shows tile result of estimating the vehicle width during M T action. Tile estimated width is 165(,m where tile real width of the vehicle is 160cm. The accuracy of more than 90% in many exl)eriments as sures tile robot of safe avoidance. Finally, Figure 11 shows the time series results of vehi('le detection by: vision-based and sonar based actions fl'om 30m to 2m in distan('e. The number of d('te('tions has been found in a defined time interval T. Because of the limitation of visual sensors in detecting tile sign pattern of the vehicle at a near distance, tile number of visual vehicle detections, shown by x, has reached 0 at a distance of 6m. Similarly tile number of sonar-1)ased vehicle detections, shown b v . , has increased at a near distance. 7. C o n c l u s i o n
A general paradignl for vehicle detection and avoidance by proper use of visual and sonar sensors has 1)een 1)resented. The 1)erfornmn(,e of the system has 1)een evaluated in
27
.-.. K
|
i
300
|
Vehicle Width Estimated Width
r
,,,
=
,,|
O (J (D
160 cm " 165 cm 9
60
Visual Sensor
Cl
"o E
O
200
d z Z
40
"-
r
uJ
1O0
20
5 2'o
lh
'
Ib
'
,
g"
30
20
Distance(m)
Figure 10. The estimated width of the vehicle. The result has been found from 20111 to 5m in distan('e during MT a ctioll.
10
a1
0
Distance(m)
Figur(, 11. The llUllll)el of v(,hMe dete('tions l)y visual and sonar s('nsors (hu'ing th(" l"(~l)ot's nlotion frOlll 3()m to 3111 in (list alwe. x indicates the nUlll])er of det(,('ti()ns 1)v visual sensor. ,, indicates the llllllll)('l" of (h't('('tions by sonar sensor.
Table 3 Results of vehMe detection
Road surface Clear weather (1)artly shadow) Clear weather (entirely shadow) Clear weather( llO shadow ) Cloudy weather
Detection success 99.8~! 100% 100% 100(~
()vet d(,t('('tion 0.4(/ ().8(/(~ 0.2(/: 0.6(/
Under detection 0.2(~ 0.0% 0.0% 0.0%
Total 11o. of vehMes 850 1050 880 860
tile university camlmS. The order of using s('ns()rs is sele('tcd according to the sensors' ('al)al)ilities. Tile lnethod of selection uses knowh'(lg(' of th(, functionalities and linfitations of each type of sensor a.s well as the outdoor cnvil-Olnn('nt situation. Five types of actions, originated froln Stereotyped Motion. have 1)ten al)l)li('(l for detection and a voidallce problems: they are lklovillg Along Wall, Moving Along, Tra('king NIotion, Nioving Toward, and Danger Estinmtion. Visual vehicle detection is discussed in 1)fief. Th(' method is 1)ased on Sign Pattcrn Stereotyped Motion. The Sign Pattern of the vchich' is the shadow underneath the vehicle. It. is unique to the time, shadow, weather, and light conditions. Tal)h ~ 3 shows the robustness of the shadow underneath for vchi('le d('t.ection in different conditions. This result has been obtained Kern one hour of videotal)e consisting of more than 3000 scenes of vehicles in different road and weather conditions. ()ver-det.ectioll occurs when pseudo-objects are detected as vehicles by mistake. Similarly, under-detection indicates the system is failure to detect, vehicles.
28
:s research on 1)rol)er selection of sensors is still ('ontimfing. hnl)roving the navigational system and multi-ol)ject detection are flltm'e work. REFERENCES
1. Alex Gask('ll and Penelope Probcrt, "Sensor Models and a Framework for Sensor Managenwnt", SPIE 2059, United States, Boston, pp. 2 1 3 , 1993. 2. R. luo and M. Kay, "Multisensor Integration and Fusion in Intellig('nt Systems", IEEE Trans. on Systems, Man, and Cyl)ernetics 19, No. 5, pp. 901- 931, 1989. 3. S.A. Shafer, A. Stentz and E. Thrope, "An Architecture for Sensory Fusion in a Mobile Rol)ot', Proc. of IEEE Int. Conf. Robotics and Automation, Raleigh NC, pp. 4 0 2 - 4 0 8 , 1987. 4. K. Kimoto and S. Yuta. "Sonar Ba~se(l Outdoor Navigation - Nax'igation using Natural Landnmrks ", IC'AR, Tokyo, Japan. Pl). 239 244. 1993. 5. M.A. Turk. D.G. ~Iorgenthaler, h:.D. Ch,'ml~an and M. Marra, "\'ideo Road Following for the Autonomous Land \,~,hich,", in Proc. of IEEE Int. Conf. Rol)oti('s and Automation, pp. 727-733, 1988. 6. U. Solder and V. Graefe, "O1)ject Detection in Real Time", in Pro(-. of the SPIE Symposiunl on Advances Intelligent System. Boston, 1)P- 104111, 1990. 7. A. Kuehnel. "Symmetry based Recognition of Vehicle Rears", Pattern Recognition Letters 12, North Holland, pp. 249--258, 1991. 8. T. Zielke, M. Braukman and W.Seelen. "Intensity and Edge-Based Symmetry Detection aI)I)lied to Car Following", Coml)ut('r Vision- ECCV '92, 1)1). 865-874, 1992. 9. H. Mori, M. Nakai, H. Chen, S. Nakayanm and T. '~51guchi, "A Mol~ile Robot Strategy - Stereo typed Motion by Sign Pattern - " , ~IIT press, pp. 161-172, 1990. 10. N. Moghadam and H. Xlori, "A New A1)proach for Real time VehMe Detection", IEEE Int. Conf. on Intelligent Robots and Systems IROS, Yokohanm, Japan, pl ). 273--278, 1993.
11. Hideo Mori. N. Moghadam Charkari and Takeshi ~Iat.sushita, "On-Line VehMe and Pedestrian Detections Based on Sign Pattern", IEEE Transaction on Industrial Electronics 41, No. 4, pp. 384-391, 1994. 12. H. Xlori, "Guide Dog Robot HAI:/UNOBU-5 - Stereotyped Motion and Navigation -", Springcr-\:erlag Press, Germany, Pl). 135 149, 1991. 13. Hideo Mori, N. Moghaddam Charkari and Shinji Kotani, "Danger Estimation of Vehicles at Intersection", IEEE Int. Conf. on Rol)oti('s and Automation, Nagoya, Japan, 1995.
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
Selective Refinement of 3-D Scene Description for Mobile Robot
29
by Attentive
Observation
Hotaka Takizawa ~, Yoshiaki Shirai and Jun Miura ~Osaka University, 2-1, Yamadaoka, Suita, Osaka 565, JAPAN This paper describes a method of constructing a 3-D scene description by attentive observation to find a path to a given destination. From low-resolution stereo images, features are extracted and a scene description is constructed with the constrained Delaunay triangulation. Then, a path is searched for in the description. If no reliable paths are found, an unknown image region on the most promising path is examined in zoomed-in stereo images and new features are extracted from the zoomed-in images. The new features are integrated to the original ones. A new description is similarly constructed of the integrated features, and a path is again searched for in the description. This procedure is repeated until a path is found or no more ambiguous regions are left. The experimental results for a real indoor scene including a chair, desks and computers are shown. 1. I n t r o d u c t i o n Vision is useful for an autonomous mobile robot to reach a given destination safely in an environment [1]. The cost of vision may be high if the robot has to observe the entire environment to make a detailed description [2] [31. There have been attempts to reduce the cost of making a scene description by selecting observation regions. Xie [4] proposed a method of viewpoint selection to determine whether or not the environment is different from a given map. Jasna et al. [5] proposed to observe an unknow environment (without maps) from several different viewpoints. The viewpoints are selected so that the robot observes the entire environment and that the number of the observations is minimized. Subhodev et al. [6] proposed to acquire range information of an environment with low-resolution stereo images to observe selected regions with high-resolution images. The observation regions are selected so that the fine description of the entire scene is constructed with minimum camera movement. These methods, however, are not effective for the task of finding a path to a given destination. In Blake et al. [7], the environment is first observed to find a destination. If it is not found, a freespace to pass through is searched for. The robot moves to the freespace and observes the environment again. The search of freespace may be inefficient because the robot has no global information of the environment. In our method, the robot selects the region on the most promising path out of the candidates obtained by low-resolution images. Then, the robot observes the region attentively. Fig. 1 shows a typical situation where the method is useful. If only the features Ti(i = 1 ,,~ 4) are detected from the low- resolution image, the movable area for the robot
30 is the movable area-1 because the feature points are regarded as lying on a surface. If the destination of the robot is behind the plane containing T2 and T3, the region corresponding to the area is observed attentively. If any textures on the back wall are observed, the movable area expands to the movable area-2 and the path to the destination might be found.
Figure 1. Movable areas for the robot.
Figure 2. Panther-VZ on a mobile robot.
In this paper, a 3-D scene description is constructed by attentive observation to find a path to a given destination. From low-resolution stereo images, features are extracted and a scene description is constructed. Then, a path is searched for in the description. If a safe path is found, the robot goes to the destination. If no reliable paths are found, impassable image regions and unknown regions are determined and an unknown region on the most promising path is examined in a stereo pair of zoomed-in images. From the zoomed-in images, new features are extracted. The new features are integrated to the original ones. A new description is similarly constructed of the integrated features and a path is again searched for. This procedure is repeated until a path is found or no more ambiguous regions are left. We use an autonomous mobile robot equipped with an Active Vision System (See Fig. 2), named Panther-VZ [8]. The camera system has four degrees of freedom: pan, tilt and vergence for the two cameras. A motorized lens is attached to each camera. Zoom, focus and aperture of the lens are controllable. 2. O v e r v i e w of t h e m e t h o d The procedure of the method to find a path to a given destination is as follows:
31 1. Take a pair of low-resolution stereo images and compute the range data. 2. Constructed a scene description from the data(described in Section 3.1). 3. If a safe path is found from the description, the robot goes to the destination and the procedure ends successfully. Otherwise, extract impassable regions and undefined regions (described in Section 3.2). 4. If the undefined regions are found, select a region to observe attentively. Zoom in the region and take a pair of high-resolution stereo images of the region(described in Section 4.1). Integrate the new range data with the original data (described in Section 4.2). Go to Step (2). 5. Otherwise. If there are no next viewpoints, the procedure fails. Otherwise, the robot goes to the nearest viewpoint. Go to Step (1).
3. Scene Interpretation 3.1. 3-D Scene Description with Constrained Delaunay Triangulation The 3-D positions of feature points(edges) are obtained by our feature-based stereo method [9]. The straight line segments are extracted by merging connected edges. The scene description is constructed by interpolating edges and segments with triangles. Delaunay triangulation is made use of to generate the triangles. It generates triangles such that nearer pairs of edges are preferably linked. In this description, a generated triangle may not correspond to a real surface. We will verify whether or not the triangles are real surfaces later.
3.1.1. Delaunay Triangulation The Delaunay triangulation is calculated from a Voronoi diagram [10]. The Voronoi diagram and the Delaunay triangulation on a 2-D plane are defined as follows(See Fig. 3). For a given set of points (called generators), Mi, on a plane, the corresponding Voronoi diagram consists of polygons, each of which includes a generator such that the polygon region is closer to the generator than to the other generators. The Delaunay triangulation draws line segments between adjacent generators in the Voronoi diagram. The Voronoi diagram and the Delaunay triangulation in a 3-D space is represented by polyhedra.
""/
"
M3 - ~ ~
M 4 ~ ~ . ~ i
Voronoi diagram
M2
....
it"............
~ M5 Delaunay triangulation
Figure 3. Voronoi diagram and Delaunay triangulation.
32
3.1.2. 3-D Scene Description with Constrained Delaunay Triangulation In indoor scenes, straight line segments are useful for 3-D scene description. Although the ordinary Delaunay triangulation is computed only with the generators, it is desirable to preserve straight line segments in the triangulation. The constrained Delaunay triangulation keeps the line segments as a part of edges of the final triangulation [11]. Floriani et al. [12] proposed to apply the 2-D constrained Delaunay triangulation to the 3-D description. First, the 2-D constrained Delaunay triangulation is generated from segments in an image. Then, the 2-D triangulation is backprojected to the 3-D space using the 3-D positions of the segments. We apply the method to segments and points. Given edges obtained by the stereo method, the straight line segments are extracted. The 2-D constrained Delaunay triangulation is constructed from both the segments and the remaining isolated points. Finally, the 2-D triangulation is backprojected to 3-D space. Fig. 4(a) shows an example of scene and the extracted segments (the white arrow indicates the destination). Fig. 4(b) shows the 3-D scene description with the constrained Delaunay triangulation.
Figure 4. Scene description.
3.2. Interpretation of Delaunay Triangulation The safe path to a given destination is calculated using the scene description. If the destination is in front of the triangles, the path is the straight line from the current position to the destination. Otherwise, the path is undefined because nothing is known about the space behind the triangles. For example, in Fig. 4(b), the robot can get no reliable paths, because the destination is behind the triangles. Some triangles of Delaunay triangulation, however, are generated on the plane which is not object surface. The robot can go into the space through those triangles. We search such regions. Triangles of the Delaunay triangulation are classified into the following two categories: 9 impassable: The robot cannot pass the triangle region.
33
9 undefined: The passability cannot be decided. The procedure is as follows. 1. Triangles which are higher than the height of the robot are eliminated. Triangles which are farther than the destination are also eliminated. 2. Triangles whose width is narrower than the robot are classified as impassable because the robot cannot pass the region. The other triangles are classified as undefined. Fig. 5 shows examples of the undefined triangles. 3. The undefined triangles are merged into a polygon if they lie on a plane. For every pair of neighboring triangles(or polygons), a plane is fitted by the least squares method. The pair is selected which has the smallest squared error. If the value is less than a threshold, the pair is merged. The merged pair is added to the undefined triangles(or polygons) set and the procedure is repeated until no more pair is merged. Fig. 6 shows the result of this process.
4. Undefined polygons are examined using the light intensity image to see if they are the surfaces of an object. If the angle between the normal vectors of two adjacent polygons is large, there must be a visible boundary between them. Thus, if there exist edges along a boundary between two polygons, at least one side of the boundary can be regarded as an object surface. (Broken lines in Fig. 6 indicate such boundaries.) If a robot cannot pass through either of the polygons without interfering the boundary, the both polygons are labeled impassable. The remaining polygons are labeled undefined. The image regions( See Fig. 7) which consist of the neighboring undefined polygons are the candidates for attentive observation to verify the passability. 4. A t t e n t i v e O b s e r v a t i o n 4.1. D e t e r m i n a t i o n of R e g i o n for A t t e n t i v e O b s e r v a t i o n The image region for attentive observation is selected from the candidates as the one which is closest to the promising path. The camera direction is set to the center of the region. The zoom value(magnification) is determined so that the zoomed-in image includes the region. It is also necessary that the zoomed-in image contains segments in various directions, because they are used to register the position of the zoomed-in image and the original image. The positional difference of two segments can be defined only in the perpendicular direction to the that of segments. For example, in Fig. 4(a), the destination is the left most computer on the desk. The right region shown in Fig. 7 is chosen for attentive observation, because the region is on the path which is straight line from the current position to the destination. Fig. 8 shows the zoomed-in image including the region for attentive observation and the extracted segments.
34
Figure 5. Undefined triangles.
Figure 6. Result of merging. The broken lines indicate object boundaries.
4.2. Integration of Observed Data This section describes how to integrate the base data and the zoom data. The base data are defined as the 3-D positions of the segments and the points which are extracted from the initial images. The zoom data are defined as the 3-D positions which are extracted from the zoomed-in images and are transformed into the coordinate system of the base data. The zoom data include errors from the pan and tilt angles of the camera platform. We correct the errors using the positional differences in the image plane between the corresponding segments in both data. The following procedure is repeated until the sum of the differences is less than a threshold. First, the matching pairs of segments are determined. Second, the difference vector (described in Section 4.2.2) in the image plane between both data is computed. Finally, the zoom data are modified with the difference vector. After the final difference vector is obtained, the segments and the points are integrated to the base data. We assume that the zoom value is always correctly controllable.
4.2.1. D e t e r m i n a t i o n of M a t c h i n g of S e g m e n t s We define the following evaluation values to be used for matching: 9 The average contrast is the average of the contrast value of the pixels which compose a segment. 9 The average direction is similarly defined. 9 The non-overlap-ratio is the ratio of the length of the shorter segment to the nonoverlapping length between the both segments. It is given as follows: N o v e r = (18 - lo)/l,,
(1)
where 18 is the length of the shorter segment and the lo is the overlap length as shown in Fig. 9.
35
Figure 7. The neighboring undefined polygons are the candidates for attentive observation.
Figure 8. Zoomed-in image.
9 The mean-squared-error is defined as follows:
Msqua-
1 ~)~ {e(t) }2 dt,
t 2 - t,
(2)
where e(t) is the distance from the point at t, on the shorter segment, to the longer segment, tl and t2 are the positions of the endpoints of the overlapping part along the longer segment as shown in Fig. 10. For a segment in the base data, the segment in the zoom data is selected which minimizes the following expression:
wlCnij + w2Drij + waNoverij + w4Msquaij,
(3)
where Cnij is the difference of the average contrast, Drij is the difference of the average direction between the i-th segment of the base data and the j-th segment of the zoom data. If the minimum value is smaller than a threshold, the selected segments in the zoom data correspond to the segment in the base data.
4.2.2. Computation of Difference Vector With a set of matched pairs of segments, we determine how much the zoom data should be translated on the image plane. The optimal amount of translation is called the difference vector Z. The difference vector is determined so that the sum of the mean-squarederror weighted by the overlapping length may be minimized, that is, Z is given by,
{ where errk is mean-squared-error of the k-th matched pair, lo,k is the overlapping length of the pair.
36
Figure 9. Non-overlap-ratio.
Figure 10. Mean-squared-error.
Let us consider Fig. 11 where the solid lines are in the base data, the broken lines are in the zoom data. The difference vector is determined to minimize the sum of the meansquared-error in the shaded area. The difference vector is computed by the contragradient method.
Figure 11. Computation of the difference vector.
Figure 12. Scene description with the integrated data.
5. Experimental Results The initial input image of a scene, shown in Fig. 4(a) is taken from the start point and the scene description is constructed with the constrained Delaunay triangulation. Fig. 4(b) shows the result of the triangulation. The triangles are merged into polygons and the polygons are classified into two categories: impassable and undefined. Fig. 6 shows the undefined polygons. The regions which include the undefined polygons are the candidates for attentive observation. Fig. 7 shows
37 the attentive region candidates. If the left most computer in Fig. 4(a) is the destination, the path is set to the straight line course from the current position to the destination. Since the right region in Fig. 7 is on the path to the destination, the region is selected for attentive observation. The zoomed-in image including the region is shown as Fig. 8. The constrained Delaunay triangulation with the integrated data is shown as Fig. 12. Fig. 13 shows the top views of the descriptions. With some of the newly obtained features, the surface composed of the edges of the chair and the edges of the desk is determined to be passable(See Fig. 13(b)). Finally, the path from the current position to the destination is found.
Figure 13. Top view of the description.
6. C o n c l u s i o n This paper describes a method of constructing a 3-D scene description to find a path to a given destination by attentive observation. The 3-D scene description is constructed with the constrained Delaunay triangulation and a path to a given destination is determined from the description. If the path is not found, the unknown image region which is on the most promising path is observed attentively. New data is obtained with higher-resolution. The new data, however, includes the position displacement by the errors from the pan and tilt angles of the camera platform. The both images are registrated by matching segments in both images and by computing the displacement. The new data is integrated with the original description, and a path is searched for in the new description.
38 Experiments have been performed for an indoor scene which includes a chair, desks, and computers and so on. The results have proved that the proposed method is valid for cluttered scenes.
Acknowledgment This research is partially supported by Mr.Fredric Solomon and by the Science and Technology Agency of Japan as one of the fundamental technical research project in their Sensor Fusion Program and by Grant-In-Aid for Scientific Research under Grant No.06555071 from the Ministry of Education, Science, and Culture, Japanese Government. The authors would like to thank those concerned for their cooperation and support.
REFERENCES 1. 2.
3.
4. 5.
6.
7. 8.
9.
10. 11. 12.
"Vision-Motion Planning with Uncertainty", J.Miura and Y.Shirai, Proc. International Conference on Robotics and Automation, pp.1772-1777, 1992. "Omni-Directional Stereo For Making Global Map", Hiroshi Ishiguro, Masashi Yamamoto and Saburo Tsuji, International Conference on Computer Vision, pp.540-547, 1990. "Map Building For a Mobile Robot Equipped With a 2D Laser Rangefinder", Javier Gonzalez, Anibal Ollero and Autonio Reina, Proc. IEEE International Conference on Robotics and Automation, vo13, pp.1904-1909, 1994. "View Planning for Mobile Robots", S.Xie, Proc. IEEE Int.Conf. on Robotics and Automation, pp.748-754, 1990. "Occlusions as a Guide for Planning the Next View", Jasna Maver and Ruzena Bajcsy, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.15, No.5, May 1993. "Multiresolution Image Acquisition and Surface Reconstruction", Subhodev Das and Narendra Ahuja, IEEE International Conference on Computer Vision, pp485-488, 1990. "Visual Exploration of Free Space", Andrew Blake, Andrew Zisserman and Roberto Cipolla, ACTIVE VISION(The MIT Press), pp.175-188. "Tracking a Moving Object by an Active Vision System: PANTHER-VZ", J.Miura, H.Kawarabayashi, M.Watanabe, T.Tanaka, M.Asada, Y.Shirai, Proc. International Symposium on Robotics, Mechatronics and Manufacturing System'92, pp.957-962, 1992. "A Multistage Stereo Method Gibing Priority to Reliable Matching", O.Nakayama, A.Yamaguchi, Y.Shirai, M.Asada, Proc. International Conference on Robotics and Automation, pp.1753-1758, 1992. "Voronoi Diagrams - A Survey of a Fundamental Geometric Data Structure", Franz Aurenhammer, ACM Computing Surveys, Vol.23, No.3, pp.345-405, 1991. "Representing Stereo Data with the Delaunay Triangulation", O.D.Faugeras, E.Le Bras-Mehlman, J.D.Boissonnat, Artificial Intelligence, vo144, pp.41-87, 1990. "Constrained Delaunay Triangulation for Multiresolution Surface Description", De Floriani L., Puppo E., Proc. International Conference on Pattern Recognition'88, 1988.
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
39
A C T I V E VISION INSPIRED BY M A M M A L I A N F I X A T I O N M E C H A N I S M
T. Yagi, N. Asano, S. Makita, and Y. Uchikawa Bio-Electronics Engineering Laboratory, Dept. of Information Electronics Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-01, Japan*
Focusing on the mammalian fixation mechanism, we propose a localization technique for object-detection in a scene, and implement it into an active vision system. In our technique, large and small camera movements are coordinated to detect an object precisely at the optical axis of the camera. Each type of movement is performed via two separate localization methods: approximate localization and adjustment localization. In the former localization, a preliminary fixation point is detected using the multi-resolution retina. After the optical axis of the camera is pointed in the direction of this point, the latter localization method determines a precise fixation point by means of visual feedback control, which minimizes the angle between the object's center and the optical axis of the camera. Experimental results show that the active vision system using our proposed technique succeeded in fixating objects one after another in a scene, even if several objects are placed within it. According to these results, we have confirmed that bottom-up processing using primitive visual features is an effective technique for localizing an object.
1. INTRODUCTION Active vision is expected to solve ill-posed problems in the understanding of 3-D scenes from 2-D images, and to be useful for more sophisticated visual information processing in industries [ 1][2][3]. One of the important components of active vision is "localization," which is the detection of an object and determination of its position. Conventional studies have used various techniques to localize an object in a scene; among them, segmentation by means of clustering several extracted features (such as intensity, color, texture, etc.) in a feature space, so called "feature-clustering" [4][9], is widely used. The problem inherent in this technique, however, is the long processing time required, which increases in proportion to the number of extracted features. Furthermore, it is difficult to apply this technique in scenes where each object has features that vary from the others only slightly. The other segmentation techniques, "region-growing" [15] or "split and merge procedure" [5], perform better in some cases; however, they are greatly affected by noise and sharp edges, as well as by lines which form disconnected boundaries. As a result, it is difficult to formulate simple criteria to decide whether or not they form true region boundaries. Of course, the "spatial-filtering" procedure [6] can be applied to find region boundaries; however, it also is sensitive to noise, and requires several other refining processes like noise-reduction or smooth-filtering in order to obtain clear and continuous edges. Consequently, conventional localization techniques have some problems in terms of processing time and noise. * E-mail:
[email protected]
40 In order to effect better and easier localization in active vision, we have for years focused on biological visual systems, and have studied the mammalian fixation mechanism from the computational point of view [12][13][14]. Fixation is the process of looking at a single point using eye movements [8][11]. Because spatial resolution in vision is the highest near the optical axis and decreases towards the periphery, only an object placed at the center of the retina is identifiable. Therefore, fixation is one of the inevitable functions of mammalian visual systems. Fixation is usually completed with a fast eye movement called saccadic eye movement; however, sometimes overshooting or undershooting of the object occurs. Such detection errors are compensated with a small saccade (catch-up saccade) and/or a slow eye movement known as smooth pursuit eye movement. In addition, fixation is generally considered to be driven by at least two different visual systems: the primary visual system (i.e. r e t i n a - lateral geniculate body - striate area), and the secondary visual system (i.e. r e t i n a - superior colliculus - pulvinar). Because identification of an object is carried out in the occipital lobe, fixation through the primary visual system seems to be related to top-down (or knowledgebased) visual information processing. On the other hand, some experimental results have shown that localization of an object can be achieved using only the secondary visual system, such as in lesion-induced malfunction of the primary visual system [7][10]. Therefore, fixation through the secondary visual system seems to be related to bottom-up visual processing. Moreover, it has been theorized that the secondary visual system in fact specializes in finding where objects are. Consequently, the understanding of this system is a key to accurate emulation of localization in active vision. In our previous psycho-physical and computational studies, we have hypothesized the importance of retinal signal processing in the secondary visual system, and revealed that certain fixation is correctly executed only if the mathematical model of the retina has linearly increased dendritic field size in the retinal ganglion cells [13][14]. Based on this hypothesis, in this paper we propose a localization technique which is similar to the mammalian fixation mechanism. Implementing this technique in an active vision system, we demonstrate that the active vision system is able to fixate objects one after another in a scene where several objects exist. In the last part of this paper, we discuss the advantages of our proposed technique and our future research work using this system.
2. B I O L O G I C A L L Y - I N S P I R E D L O C A L I Z A T I O N In order to fixate an object at the optical axis of a camera, the object must first be detected in a scene, then its location determined, prior to camera movements. Following two
Figure 1. The proposed localization method.
41 different types of localization, approximate localization and adjustment localization, work interactively for this purpose (Fig. 1). Whenever a camera is moved, a new image is sent to a host computer, then either type of localization is executed. The former localization directs large camera movements and determines a preliminary fixation point, which is approximately identical to the object's position. When overshooting or undershooting occurs, small camera movements are effected for compensation by the latter localization method until the object is foveated at the optical axis of the camera. Such coordination of large and small camera movements, which correspond to saccadic eye movements and smooth pursuit eye movements, respectively, is expected to perform precise localization.
2.1. Approximate localization Our previous studies have revealed that retinal signal processing is considered to play a great role in the secondary visual system [12]. Moreover, fixation points seem to be computed primarily at the retina [13][14]. Therefore, we use a multi-resolution retina "Eye Field" (Fig.2) in approximate localization. This Eye Field increases resolution near the optical axis and lowers it in the periphery. It is a simplified version of the mathematical retina model, and has non-overlapping rectangular-shaped receptive fields. (Although the mathematical model could be applied, we have modified the model in order to reduce computational load.)
Figure 2. Various types of Eye Fields. Fig.3 shows how a camera-taken image is converted into the image processed by the Eye Field. Every camera-taken image is split into smaller sizes near the optical axis than in the periphery (Fig.3a). In each split area, the intensity values of each pixel are added. The total intensity is divided by the split area to obtain the mean intensity (Fig.3b). Fig.3c indicates the mean intensity in each split area. The Eye Field is expected to intensify each object differently, corresponding to its location. In addition, it is supposed to reduce the effect of noise, especially at the peripheral region. From among several peripheral split areas, the one which has the maximum intensity value is selected, and the center of that split area is defined as a preliminary fixation point. The camera is then pointed in the direction of this
Figure 3. Approximate localization.
42 selected position (Fig.3d). These processes are repeated until the image part of the object is located at the optical axis of the camera. As shown in Fig.2, several types of Eye Fields could be proposed. Because the dendritic field size in the retinal ganglion cells greatly affects fixation [13][14], the localization ability of the Eye Field was hypothesized to depend on its splitting format. Therefore, we evaluated each type of Eye Field on a computer in terms of the following: suppose several objects are placed at random in a scene. When they are the same in size, intensity, shape, and color, it is known that human beings tend to fixate the object which is more closely located to the fovea. From the facts, the nearness of the object, otherwise termed the proximity of the visual stimulus to the fovea, is one of the factors which induce fixation. Focusing on this tendency, we simulated the localization ability of each Eye Field in various scenes. For quantitative analysis, we employed the "Proximity Index," which was defined in our previous studies [12]. Proximity Index refers to the normalized traveling distance of the optical axis of the camera in fixation. Assume the following distances: L from an initial fixation point to a detected fixation point, Lmax t o the farthest object, and Lmin t o the closest. Then, Proximity Index p is given as follows; p =
tmax m t
(1)
Lma x - Lmi n
As shown in this equation, this index gives a value of 1.0 if the detected object is located close to a fixation point, and a value of 0.0 if it is farther away; consequently, we can determine how far the detected object is located from an initial fixation point. In addition to Proximity Index analysis, we also checked if each localization was executed without overshooting or undershooting. In this analysis, we simply observed whether or not an object was detected at the central region of the Eye Field. Fig.4 is the histogram where the frequencies of Proximity Index in steps of 0.1 were graphed. It shows very high frequency in the category 0.9-1.0 in every type of Eye Field; therefore, it can be said that the object which was more closely located to the center of the Eye Filed tended to be detected. From this analysis, we confirmed that Type-I Eye Field resulted in slightly better localization than the others in terms of the proximity of the visual stimulus to the fovea. Also shown below, Table 1 indicates the percentage of localization without overshooting and undershooting. Note that 67% of localizations were performed without overshooting and undershooting in the case of Type-l, the highest score among the four Eye Fields. As a result of its superior performance, we have used Type-1 Eye Field in following experiments. Table 1 The percentage of the localization without overshooting or undershooting.
Figure 4. Proximity Index analysis.
Eye Field
Percentage
Type- 1 Type-2 Type-3 Type-4
67% 62% 35% 40%
43
2.2. Adjustment localization After approximate localization is carried our, adjustment localization takes place. Because approximate localization may result in overshooting or undershooting, it is necessary to compensate for such errors. In adjustment localization, only the central region of the Eye Field (the smallest split areas) is used for processing (Fig.5a). After approximate localization has occurred, the smallest central region is divided into horizontal and vertical halves (Fig.5b). The mean intensity values of each split area within regions A and B, or C and D are then added. In order to determine the traveling direction of the camera, we compare the total value in each region. If the total value in region A is greater than in region B, the camera is pointed in an upward direction. If not, the camera is moved downward. Likewise, comparison of the total values in regions C and D determines the horizontal traveling direction, left or right, respectively. The traveling distance of the camera is computed using the sum of the values of intensity in each pair of regions. Fig.5c shows an example of this procedure. The maximum traveling distance is defined as the distance from the center of the fovea to its edge. (It corresponds to 4 blocks in the figure.) As shown, region A is 14.50, and B is 4.25, hence the total is 18.75, and the difference 10.25. The ratio, therefore, between the total and the difference is 0.547. As for the horizontal case, the sum, difference and ratio are 18.75, 14.75, and 0.787 respectively. The traveling distance is defined as the multiplication of the sumdifference ratio by the maximum traveling distance. Therefore, in the case of Fig.5, the vertical traveling distance is computed as 0.787 x 4 (blocks), and the horizontal as 0.547 x 4 (blocks). These processes are repeated until the total values in regions A and B, as well as C and D, reach equality. When they become equal, an object is expected to be localized at the center of a camera-taken image.
Figure 5. Adjustment localization.
44
2.3. Subsequent object detection Based on the detected position of the object via the above, the camera is then aimed in such a way that the object is placed at the center of the Eye Field after the completion of localization. In the next phase, a new target is selected from the periphery again using approximate localization; however, the next target might be one which has been previously localized. In order to avoid such repetitive localization, we set the following rule: whenever an object is detected, its position is memorized. During approximate localization in the next phase, the position of the newly-detected split area's center is compared with stored positions. If it coincides to a previously memorized position within a c e r t a i n range, it is excluded as a candidate for the next fixation point. Accordingly, the area which has the second highest intensity is then selected.
3. EXPERIMENT 3.1. Experimental settings Our proposed localization technique was evaluated in the experimental settings described below. We applied the technique to an active vision system, called "Eye Mover" (Fig.6). In Eye Mover, a small CCD camera (EM-102II, produced by ELMO CO.) was incorporated into a universal joint linked to a motion driver via a sliding rod. Because the camera is able to rotate around the rotational center of the joint, it is possible to point the optical axis of the camera in any direction. An X-Y plotter (GRAPHTEC WX1000, produced by GRAPHTEC CO.) was used as the motion driver of the camera. Camera-taken images were transferred to a host computer (EPSON PC-386V, produced by EPSON, with an Intel 80386 20 MHz CPU, and 80387 co-processor) through an image digitizer (PHOTRON FDM 4-256, produce by PHOTRON, with 256 x 256 pixel resolution and 1/60 second sampling time). In Eye Mover, the rotational centers of the camera and the focusing point are identical and fixed at one position. In addition, the angle between the optical axis of the camera and the vertical axis of Eye Mover, is easily obtained from the camera-positioning data in the cameramotion driver. Therefore, the object's position in the 3D space could be geometrically computed using the angle and the height of the rotational center from the floor. The maximum traveling speed of our system was 180 (degrees/second), and its acuity was within 0.4 degrees. Unlike hand-eye systems, the size of our Eye Mover system was relatively small. In addition, the simple mechanism enabled the camera to execute movements within a wide rage of motion and to fixate an object quickly.
Figure 6. Active Vision System, "Eye Mover."
45
3.2. Results
The experiment was executed in a scene which contained several randomly placed objects on a plane background. Fig.7 shows a typical example of approximate localization (Fig.7a and 7b) and adjustment localization (Fig.7c and 7d). In the figure, camera-taken images are indicated on the left, and processed images on the fight. The camera-taken image at the initial fixation point is depicted in Fig.7a-left. Fig.7a-right shows the processed image using the Eye Field. Using the approximate localization process, the most intensified split area, the middle upper-fight in Fig.7a-fight, was selected. The center of the area was then defined as a preliminary fixation point, and the optical axis of the camera pointed in that direction. Fig.7b-left shows the camera-taken image after the camera movement. In this case, the object was correctly foveated using only approximate localization, and adjustment localization was skipped in order to select the next object for localization. Overshooting or undershooting, however, happened sometimes, as can be seen in Fig.7c. Because the large camera movement resulted in pinpointing only the edge of the ball (Fig.7c-left), adjustment localization was performed. Using the central region of the processed image (Fig.7c-right), a more accurate position was then computed. After the second small camera movement, the object was fixated precisely at the optical axis of the camera (Fig.7d-left).
Figure 7. Experimental results. (a)(b) Approximate localization. (c)(d) Adjustment localization.
Fig.8 indicates a locus of camera movements. Open triangles and open circles denote fixation points in approximate localization and adjustment localization, respectively. Starting from the center of the scene, Eye Mover pointed towards each object and foveated them one after another within a few small camera movements. In addition, it can be observed from this figure that repetitive detection was avoided. At the second to last fixation point (near the bottom of a can), the system detected the can because it was closely located to the camera's optical axis. The detected position, however, coincided within a pre-described range to the previously detected position of the can, so the can was consequently excluded as the next fixation target. Finally, the last object (a wooden piece) was localized.
46
Figure 8. Locus of camera movements.
4. DISCUSSION AND CONCLUSION The experimental results reveal that an active vision system implementing our proposed localization technique is able to achieve precise localization when simple scenes are displayed. The optical axis of the camera is directed close to each object on the first camera movement during approximate localization; each object is then correctly localized within a few small additional movements in the adjustment localization phase. In approximate localization, camera images are split into various sizes, and the intensity values of each pixel are successively added in order to compute the mean intensity in each split area. On this point, approximate localization is similar to the conventional "split and merge" procedure. One of differences between the two methods, however, is the splitting format. In the Eye Field, split areas are inhomogenenous. This makes resolution with our technique high at the center and lower at the periphery, while resolution is uniform in the "split and merge" procedure. Therefore, the proposed technique enables every object in a scene to be weighted differently according to the distance from the center of the scene to its location. Moreover, approximate localization is more applicable to a noisy image, because the effect of noise is reduced by means of blurring the peripheral region of the image. The second difference is the time required in the splitting procedure. Because the splitting format of the Eye Field is already determined before processing, the time in the approximate localization phase is always constant, while in the "split and merge" procedure it is variable. This constant time element allows us to estimate the processing time of approximate localization in advance. Consequently, the simplified computation in approximate localization overcomes the disadvantages experienced by conventional techniques in terms of noise and the processing time. Our proposed technique can be seen as bottom-up processing. Based on a low-level visual feature such as intensity, an object is localized without the assistance of top-down processing. Moreover, this technique does not require any noise-reduction or segmentation processing, thus consequently reducing processing time. Of course, some criticism can be leveled against such simple processing. Because low-level visual features mainly affect the determination of a fixation point, the localizing ability of the system is still at a primitive
47 level. The experimental results, however, show that our technique achieves adequate localization in a scene where several objects were placed. As we mentioned at the beginning of this paper, localization was realized without object identification. Hence it is suggested that such primitive processing is a key element of localization, and a key to understand "attention," which involves top-down processing. Based on this assumption, we have undertaken psycho-physical experiments on human eye movements, and are formulating a mathematical model of fixation and attention in order to reveal the relationship between bottom-up and top-down processing. Results will be published soon. In addition, we are attempting to apply our proposed active vision system to robot-manipulation tasks. In this study, a robot hand is manipulated by means of the positioning-data computed via our technique. After the position of an object is determined, the hand is moved to grasp the object. Preliminary analysis shows that the acuity of the detected position is precise enough to manipulate the hand. Gradually incorporating top-down processing into our active vision system, we will seek to further refine our technique in future studies.
REFERENCES
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
[ 13]
[14]
[15]
J. Aloimonos, and I. Weiss, and A. Bandyopadhyay, "Active vision," Proc. 1st Int. Conf. Computer Vision, pp. 35-54, 1989. R. Bajsy, "Active perception vs. passive perception," Proc. IEEE Workshop on Computer Vision, 1985. D . H . Ballard, "Reference Frame for Animate Vision," Proc. 11th Int. Joint Conf. Artificial Intelligence, pp. 1635-1641, 1989. G.B. Coleman and H. C. Andrews, "Image segmentation by clustering," Proc. IEEE, 67, 5, pp. 773-785, 1979. S.L. Horowitz and T. Pavlidis, "Picture segmentation by a directed split-and-merge procedure," Proc. 2nd Int. Joint Conf. on Pattern Recognition, pp. 424-433, 1974. D. Marr, Vision, W. H. Freeman and Company, New York, 1982. M. Mishkin, L. G. Ungerleider, and L. A. Macko, "Object vision and spatial vision: Two cortical pathways," Trends in Neurosciences, 6, pp. 414-417, 1983. D. Noton and L. W. Stark, "Scanpaths in eye movements during pattern perception," Science, 171, pp. 308-311, 1971. R. Ohlander, et. al., "Picture segmentation using a recursive region splitting method," CGIP, 8, 3, pp. 313-333, 1978. G.E. Schneider, "Two visual systems," Science, Vol. 163, pp. 895-902, 1969. A.L. Yabus, Eye movement and vision, Plenum Press, 1967. T. Yagi, K. Gouhara, and Y. Uchikawa, "An algorithm of eye movement in selective fixation," Proc. of IEEE International Conference on Neural Networks, Vol. 2, pp. 761765, 1993. T. Yagi, K. Gouhara, and Y. Uchikawa, "The influence of the inhomogeneous dendritic field size of the retinal ganglion cells on the fixation," Proc. of The European C o ~ on Artificial Neural Networks, Vol. 1, pp. 26-29, 1994. T. Yagi, K. Gouhara, and Y. Uchikawa, "A computational study on fixation and the effect of an inhomogeneous morphological feature in the retina," Proc. of IEEE International Conference on Systems, Man and Cybernetics, Vol. 1, pp. 777-781, 1994. S.W. Zucker, "Region growing: childhood and adolescence," Comput. Graph. Image Process., 5, pp. 382-399, 1976.
This Page Intentionally Left Blank
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
Realtime Range Imaging Using An Adjustment-free
49
PhotoVLSI
Atsushi Yokoyama ~ , Kosuke Sato b , Takayuki Yoshigahara ~ and Seiji Inokuchi ~ ~Production Technology Group, Sony Corporation, 6-7-35 Kitashinagawa Shinigawa-ku Tokyo, 141 Japan bNara Institute of Science and Technology, 8916-5, Takayama, Ikoma, 630-01 Japan CDepartment of Systems Engineering, Osaka University, Toyonaka, Osal~, 560 Japan We report a VLSI sensor which gives a range image of a scene with high robustness and high accuracy at a video frame rate. No adjustment is required. Photodiodes and operational circuits are integrated into each pixel of the VLSI sensor so they function as analog/digital micro-rangefinders. The 2D array of the micro-rangefinders acquires frames of range data in parallel manner as a lightstripe is swept across the scene. Complementary pairs of photodiodes in each cell reliably acquire range data without being disturbed by temperature drift, ambient light, manufacturing irregularities or other noise. A prototype system has been built using a 24 x 24 VLSI array of sensing elements. It gives us 24 x 24 range images at a video-frame rate. 1. I n t r o d u c t i o n Obtaining 3D information of a scene at a high speed and with high accuracy is important in factory automation tasks such as inspection and assembly. Numerous studies have been made of rangefinding techniques[ill2]. The lightstripe-rangefinding method is one of the most common and reliable methods of rangefinding. In this paper we report an adjustment-free range-imaging VLSI sensor which receives a complete range image consisting of lightstripe range data with high robustness at a video frame rate. The speed of a conventional lightstripe rangefinder using a CCD camera does not exceed one image/s because a lightstripe rangefinder needs hundreds of intensity images in order to construct a 2D range image. The conventional CCD camera captures intensity images of the scene on which the lightstripe is projected. The position of the projected lightstripe is extracted from each image and used to calculate the range data. For each position of the lightstripe, only a one-dimensional range image is acquired by triangulation. In order to acquire a complete set of the range data in the camera's field of view, these operations should be iterated as the lightstripe moves across the scene in steps, which requires more than one second. Cell-parallel lightstripe rangefinders have been reported by Araki et al. [3], Kida et al. [4] and Kanade et al. [5][6]. In these rangefinders, the lightstripe continuously sweeps
50 across the scene. During one sweep of the lightstripe, the time at which the image of the lightstripe crosses a cell indicates the range between the rangefinder and the scene along the line-of-sight of each cell. The time is detected by each cell and a complete set of range data in the camera's field of view is acquired during each sweep of the lightstripe. Araki et al. [3] have built a realtime rangefinder with 47 • 47 resolution with discrete phototransistors connected to separate signal processors. This rangefinder holds the timestamp by digital latches. Kida et al. [4] have built a rangefinder with 4 • 4 resolution with discrete photodiodes. This rangefinder holds the time-stamp in the shape of an analog sawtooth voltage signal by a sample-hold circuit. For large arrays, a high bandwidth link between each photosensor and signal processor is required and is provided by a 2D array of micro-rangefinders - each cell includes a photosensor and a signal processor. Kanade et al. [5] [6] constructed a VLSI sensor which consists of an array of cells, each of which contains a photodiode and signal processor. This sensor holds the time-stamp in the shape of a charge in a MOS capacitor with an analog sawtooth signal. One disadvantage of this sensor is that the time at which the image of the lightstripe crosses each cell is determined by comparing the photocurrent of the photodiode in each cell with the external threshold. Precise adjustment of an external threshold is therefore necessary, taking into account the ambient light, the reflectance of the object's surface and the operating temperature. Another disadvantage is that leakage of the holding capacitance makes it difficult to obtain accurate range data. In this paper we report a realtime, adjustment-free range-imaging VLSI sensor which gives a complete frame of robust range-image data at a video-frame rate. We call this new sensor a SRF(Silicon Range Finder) chip. In order to determine the time at which the image of the lightstripe crosses each cell with high robustness, a photo-comparator which consists of a complementary pair of photodiodes and analog/digital circuitry is integrated in each cell of the SRF chip. This has the advantage of allowing reliable 3D data to be obtained without being disturbed by noise. A prototype system using this SRF chip and a CCD chip has been developed. This system provides both a range image (from the SRF chip) and an intensity image (from the CCD chip). 2. C e l l - p a r a l l e l L i g h t s t r i p e R a n g e f i n d e r In this technique, each cell of the sensor array acquires range data while the lightstripe continuously sweeps across the scene. As shown in Figure 1, a lightstripe sweeps across the scene from right to left at a constant angular velocity of the scanning mirror. The image of the lightstripe on the sensor array moves left to right across each cell. Each cell detects the time at which the image of the lightstripe crosses it during one sweep of the stripe. The time-stamp signal is sampled and held, and converted to the orientation of the lightstripe, then the range data is calculated based on the triangulation principle. Figure 2 illustrates the cell-parallel lightstripe rangefinder geometry. The range data Zi,j of the cell Pi,j is given using baseline Bi,j and A, the direction of the line-of-sight/3i,j and the orientation of the lightstripe ai,j,
tan(~i,jtan/~i,j A Zi j = ( Bi,j + ), ' tan(~i,j + tan/3i,j tanc~i,j where/3i,i,
Bi,j, A
are constant.
(1)
51 The range Zc,r of each cell Pc,r on the sensor array is acquired during one sweep of the lightstripe by the detection of the orientation ac,r of the lightstripe. It is obvious that robust detection of the orientation of the lightstripe is essential for accurate range finding.
Figure 1. Cell-parallel Lightstripe Rangefinder
Figure 2. Cell-parallel Lightstripe Rangefinding Geometry
3. Silicon Range Finder The reliability and robustness of the range measurement of each cell depend on accurate determination of the timing. The VLSI sensor designed by Kanade et al. [5][6] determines the time by comparing the photocurrent of a photodiode with an external fixed threshold. For highly reliable and robust rangefinding, however, the time at which the image of the lightstripe crosses each cell must be determined without being affected by the ambient light, the reflectance of the object's surface or the operating temperature.
3.1. A N e w Timing D e t e c t i o n M e t h o d As shown in Figure 3, each cell of the SRF chip consists of an optical comparator capable of distinguishing minor differences in light intensity across the cell. The optical comparator has a complementary pair of adjacent photodiodes which generate photocurrents proportional to their incident light intensity. The photocurrents are fed into a current comparator which determines which photodiode is receiving the brighter light. As the lightstripe sweeps from right to left, the dominant photocurrent moves from right to left. The time at which the image of the lightstripe crosses each cell is determined by the transition of the dominant photocurrent. Even though the lightstripe is blurred due to defocus, the transition of the dominant photocurrent occurs at the exact center of the pair of photodiodes. In addition to this, the transition of the dominant photocurrent is not affected by ambient light, thermal drift or other noise.
52
Figure 3. Optical Comparator
Figure 4. Sensing element circuitry and Clock
3.2. Sensing Element Circuitry As shown in Figure 4, the circuit consists of five main components: photodiodes, current mirrors with gain, inverting voltage buffers, a clocked comparator, and comparator read circuitry. The photodiodes are made by N + doping in a P - substrate. They are surrounded by guard rings to ensure a solid ground connection and isolation fl'om adjacent cells. A complementary pair of photodiodes forms part of the optical comparator which determines which photodiode is receiving the brighter light. The size of the photodiode pair is slightly different to prevent disturbance from weak noises- the right photodiode is 3% bigger than the left one. The current mirror acts as a current amplifier and a buffer between the photodiodes and the inverting voltage buffers. It serves three purposes: first, it amplifies the photocurrent (a ratio of 2:1 is used); second, it isolates the photodiode cathode from the buffer input, which is susceptible to the influence of clock breakthrough during buffer clamping and unclamping; and third, it allows the back bias voltage on the photodiode to be independent of the large capacitance of the photodiodes. The buffer is a simple CMOS inverter with a CMOS switch between the input and output. Closing the CMOS switch clamps the buffer to its highest gain state.
53 A clocked comparator detects very weak differential voltage of the pair of buffers. The comparator is constructed using two back-to-back low drive CMOS inverters with a CMOS switch in parallel to allow initialization. The comparator is initialized during each sensing cycle before the buffer outputs are connected to the comparator inputs. The result of the comparison is converted to a digital value and sent to the output (Yn) via the inverting tri-state buffer selected by the output control signal Xn. 3.3.
Circuit
Operation
The sensing element is operated under three phase signals - r Figure 4. These signals iterate the following four phases:
r
r
-, as shown in
9P h a s e - A The input and output of the buffer are clamped together to place the buffer
in its highest gain state. The comparator holds the previous state. 9P h a s e - B The buffer clamping is released and the buffers are fl'ee to work. The stray capacitance at the buffer input nodes will be charged, via the current mirror, at a rate proportional to the photocurrents of the two photodiodes, and therefore proportional to the incident light intensity. As the charge of the photocurrent slightly changes the input voltage, the output voltage will be changed by the gain of the inverter. If the incident light intensity is different on the photodiodes, the resultant voltage change on each buffer output will be different. This differential voltage is compared by the following latched comparator. Sufficient time is allowed for PhaseB to ensure that enough voltage movement is achieved at the buffer outputs to switch the comparator. 9P h a s e - C The comparator is clamped into a cross-coupled neutral state in preparation for comparison. During this phase, the buffer continues to output. 9P h a s e - D The comparator is unclamped and the outputs fl'om the two buffers are switched to the comparator inputs. At the end of this phase, when the buffer drive is released, the comparator flips to a definite logic state, depending upon the polarity of the differential input voltage. This state will be read out near the end of Phase-B in the next cycle. A peripheral circuit addresses each column (X~) and reads the output (7~;,) in turn at a clock of 30kHz to 100kHz. When a pixel (X~,Yj) transits low to high, the corresponding digital latch in a frame buffer acquires the time-stamp count in 16 bits. Thus, the resolution of the rangefinding depends on both scan rates - the clock rate of addressing each column and the rate of the lightstripe scanning. For example, when the lightstripe sweeps at 30Hz and when the peripheral circuit scans the output (}2) at this clock rate, the resolution of rangefinding ranges from l@00 to ~ of the measurement range. 3.4. P e r i p h e r a l C i r c u i t r y The SRF chip has n-parallel column select lines (X~) and n-parallel row read lines (?~;,). Peripheral circuitry encodes the column address lines and decodes the row read lines, as shown in Figure 5. A time-stamp counter holds the current time. A scanning mirror controller resets the time counter, so that the time stamp is synchronous to the mirror scan. The frame buffer
54 is a 2D array of digital latches, with the same pixel resolution as the SRF chip. According to the trigger generated by the decoder module, the corresponding digital latch holds the time stamp. After the lightstripe was scanned a scene, all digital latches in the frame buffer hold the time stamp. An interface module transfers the time-stamp data to a computer. The computer then calculates the range data based on the triangulation principle.
.n~
C,o~ ~
7 \ q-
Eq i
i
,..,rror Con,,o,,.r
Line Xn
.
i "
---
Reset
"1o
" 0
II'
*
--
II -I1
SRF Chip
Scanning Mirror
rim e-S tam p
II
I
~' 'IC~ inter I
Row Read Line Yn
Trigger
~IL IlL IILI ~ ~ I L IlL I I Interface ]
~~IL
IlL
I
1
Computer Frame Buffer
Figure 5. Peripheral Circuitry
3.5. S e n s o r L a y o u t a n d F a b r i c a t i o n A prototype SRF chip has been fabricated in a 1.5# CMOS using a double metal, single polysilicon process. Figures 6 and 7 show the layout of a sensing element and a 24• element range sensor array with read-out circuitry, respectively. Each cell is 250pro • 250#m and the two photodiodes in each cell are 66#m • 100/tm and 64pm • 100pro. Figure 8 shows the SRF chip within the 84 pin ceramic PGA package, which includes 24 column-address pins, 24 row-read pins, and 3 comparator-control (phase inputs) pins.
4. P r o t o t y p e R a n g e f i n d e r A prototype rangefinder has been built using the new SRF chip. Figure 9 shows the prototype system. Figure 10 shows the configuration of the prototype rangefinder system, which includes a camera module, a range-image memory unit, a lightstripe generation module, and a host computer. The camera module consists of the SRF chip, a CCD chip, a lens, and a dichroic mirror which splits the incident light into visible ray and perpendicularly reflected infrared (Figure 11). Infrared focuses on the SRF chip and visible ray focuses on the CCD. The SRF chip and the CCD are mounted at the same optical distance from the lens. A range image and an intensity image in the camera's field of view are obtained at the same time.
55
Figure 6. Layout of Sensing Element
Figure 7. 24 • 24 SRF chip
Figure 8. SRF chip within the PGA package
The range-image memory unit consists of the frame buffer and a time-stamp counter. As mentioned before, this unit outputs the control signals to each cell on the SRF chip through 24 column address lines, and reads out the comparator state through 24 row read lines from the SRF chip in parallel. The counter value - the time-stamp data - at which the comparator changes its state from low to high is stored in the frame buffer. During one scan of the lightstripe, the counter values corresponding to each pixel are stored in the frame buffer. The lightstripe generation module consists of a 30roW infrared (780nm) laser diode, cylindrical lenses through which the laser beam goes to the lightstripe, a scanning mirror mounted on a galvanometer and a controller. This module outputs the reset signal to the time-stamp counter in the range-image memory unit. A host computer converts the counter values of each cell in the frame buffer into range data through a look-up table. In the next section, we see that the look-up table which maps the counter values into range data along all sensing element line-of-sight rays is generated by a calibration process.
56
Figure 9. The Prototype Rangefinder
Figure 10. System Configuration
5. C a l i b r a t i o n For accurate measurement, the system must be calibrated to reflect the exact relationship between the time-stamp count and the range data. The conversion function f ( C ) is derived by measuring four known parallel planes, as shown in Figure 12. In this figure, the lightstripe and the SRF chip are perpendicular to the plane of the figure, a reference plane SO is located at a known distance and the time-stamp counter Co of each cell is recorded. After measuring the distance of the reference plane SO, this process is repeated for planes S1 and $2, which are located at d and 2d from SO, and time-stamp count C1 and C2 are recorded. Using these data, the conversion function f ( C z ) which maps time-stamp count C z into range data Z from the reference plane along the line of sight is obtained as 2dT~(T~ - Tz) Z = f ( C z ) = TzT~ - 2TzT~ + T~T~'
(2)
where T~ =
tan(kC1 - kC2)
T~ = T~ =
tan(kCz - kC2) tan(kC0 - kC2), k = const.
The constant k in (2) is derived from measuring another plane, $3, which is parallel to the reference plane at the known position of Zx(r 0, d, 2d). In practice, a look-up table for each cell is generated using conversion function (2) for realtime conversion. p f -
57
Figure 11. The Camera Module
Figure 12. Calibration
6. E x p e r i m e n t s 6.1. R a n g e a c c u r a c y a n d r o b u s t n e s s In order to test range accuracy and the robustness of the prototype system, a white plate was positioned on a translation stage in the measurable area and range data were obtained by moving the white plate toward the sensor. Initially, the white plate on the stage was positioned 30cm away from the sensor and stepped 2500 times at 4pro pitch, with range data measured at each position. This experiment was done both in a dark room and under ambient light (50cm below a 30W fluorescent lamp) without thermal control. An experimental system using the SRF chip was built to the following specifications. B a s e line
920cm,
Focal l e n g t h of lens : 55mm, M e a s u r a b l e r a n g e : 28cm to 32cm, Theoretical maximum resolution S c a n r a t e of l i g h t s t r i p e
930Hz,
C l o c k r a t e of r o w s c a n : 50kHz, P o w e r of l a s e r : 30roW, W a v e l e n g t h of l a s e r : 780nm.
924#m,
s
.
o
.
.
.
.
.
.
.
.
]: ]!}
o -0.02 i l -0.04 0
,
,
5
,
'
10 15 Column
20
"
(a) under ambient light
o 8e-05
I
0
.
-o.o: .i
.
.
.
.
.
.
,
m*" -0.04
0
5
.
,
,
10 15 Column
,
.
20
(b) without ambient light
I under ambient I I light -I 1 without ambient ligh.t.:-..---
"= 4e-05
>
.
e
""
0
""-I
......
5
I''"
I0 Column
I
I
15
20
(c) variance of errors
Figure 13. Range Measurement Errors
Measured range data were compared to the actual range data. Averaged errors of 2500 measured range data are plotted in Figure 13(a) and (b). Figure 13(a) shows the averaged range measurement error of the each cell on the 14th row of the SRF chip under ambient light. Figure 13(b) shows the averaged measurement error in a dark room. The diamondoid points in the plot represent the mean errors and the vertical lines represent the ranges of the 2500 measurement errors. Figure 13(c) shows the variance of range measurement errors under each condition. The maximum range error under ambient light is - 2 4 # m to +10#m and that without ambient light is - 2 4 g i n to +14#m. This shows that the experimental error is less than the quantization error (+24#m). Because the accuracy and reliability of sensing data are the same in both cases, we conclude that a complementary pair of photodiodes - an optical comparator - is robust for the ambient light, and adjustments are not necessary. 6.2. R a n g e I m a g e of a Scene w i t h C o m p l e x V a r i a t i o n s in R a n g e We acquired a range image of a PCB mounting two QFP ICs, eight chip capacitors and two chip resistances. The heights varied between 0.5ram to 1.6mm. Figure 14(a) shows the PCB intensity image captured by a CCD. The PCB was positioned 180mm ahead of the SRF camera under the ambient light without thermal control. Figure 14(b) shows an elevation map of the 24 × 24 range image acquired by the SRF chip. In order to avoid occlusion, light generation modules were set at both the right and left side of the camera module. A lightstripe was projected from the right and left in turns and each range image was acquired at a video frame rate. The complete range image was generated by
59
Figure 14. ICs and Chip Parts on PCB
integrated two range images. The range image was a few pixels wider than the intensity image because the SRF chip is wider than the CCD chip. Although there are several different reflectances of the object's surface, such as mold surface and solder surface, the range image shows that the acquired range data are quite accurate. We also acquired a range image of a human face (Figure 15(a)). The face was positioned 50cm ahead of the SRF camera with a 20ram lens under ambient light without thermal control. Figure 15(b) shows an elevation map of 24 x 24 range image acquired by the SRF chip. The curved surface of the face is clearly visible and the lips, the nose and the cheeks are readily distinguishable.
7. C o n c l u s i o n \Ve have reported a new photo-sensing structure for a cell-parallel lightstripe rangefinder capable of reliably acquiring range data without disturbance from ambient light, the temperature drift of the environment, or the reflectance of the target object. We fabricated a smart vision VLSI sensor (SRF chip) based on this new structure. An experimental prototype system using the SRF chip demonstrates that it is able to measure a dynamic 24 x 24 range image with 99.9% resolution at a video fl'ame rate. Unlike other rangefinder chips, it is adjustment free. The prototype system also acquires intensity images by a CCD chip in the camera's field of view at a video flame rate. The combination of these range and intensity images of the scene should be useflll for various factory automation tasks, such as visual inspection and assembly.
50
Figure 15. Face
REFERENCES
1. Y. Nishikawa and S. Inokuchi, "Range Data Entry Techniques Using a Laser Spot Scanner and Two Linear Solid-State Image Sensors", IEEE Trans. on Instrumentation and Measurement, VOL.IM-30, No.4, pp.270-273, 1981. 2. K. Sato and S. Inokuchi, "Range-Image System Utilizing Nematic Liquid Crystal Mask ", 1st Int. Conf. of Computer Vision, pp.657-661, 1987. 3. K. Araki, Y. Sato and S. Parthasarathy, "High Speed Rangefinder", Proc. of SPIE, Vol.850, pp.184-188, 1987. 4. T. Kida, K. Sato and S. Inokuchi, "A Realtime Range Imaging Sensor", Proc. of 5th Sensing Forum, pp.91-96, (1988) in Japanese. 5. T. Kanade, A. Gruss and L. R. Carley, "A Very Fast Rangefinder", Proc. of IEEE 1991 Robotics & Automation, pp.1322-1329, April 1991. 6. A. Gruss, T. Kanade and L. R. Carley, "Integrated sensor and range-finding analog signal processor", IEEE Journal of Solid-State Circuit, vol.26, pp.184-191, March 1991. 7. K. Sato, A. Yokoyama and S. Inokuchi, "Silicon Range Finder- A Realtime Range Finding VLSr', Proc. of Custom Integrated Circuit Conference '94, pp.339-342, 1994.
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
51
A Pedestrian Discrimination Method Based on Rhythm S. Yasutolni , H. ~h)ri and S. Kotalfi Department of Electrical Engineering and Computer Science, Yamana.shi University 4-3-11, Takeda, Kofll, Yamanashi 400, Japan A new metho(l of fin(ling pedestrians in a time sequence of visual images has beell proposed, and its great 1)ra('ti(:al al)l)li('al)ility has ])een demonstrated. Walkers exhil)it 1)eriodic motion in (luite a different way from other moving ()l)jects. This 1)eriodicity, or in other words rhythm of walking, is the result of the walking action and can ])e ol)served in two kinds of perio(lic intensity change of the 1)e(lestrian's image. So, pe(lestrians can 1)e ibun(l if we l()()k for these p('rio(licities whi('h are (llfite unique to them. To realize this idea, we estimate the teml)oral-fre(llmn(.y and the spatial-period ()f the moving ()l)je(.ts. The temporal-fie(luen('y is the os(,illati()n of intensity against time. The spatial-1)erio(l represents the fluctuation of intensity along the traje('tory of the object, an(l is referred to as the stride of walking. A pe(lestrian detection meth()(l using these two oscillations has the following advantages. First, this metho(l is in(lel)en(lent of the al)solute intensity or contrast of the ol)je('ts. Secondly, clothes, hairstyle, shape of 1)ody, illumination and the distance fi'om the camera do not affect the relial)ility of detection. Finally, the algorithm is simple enollgh for real time operation. The (letaile(l algorithm is presented here and its vali(lity and 1)racti('al appli('ability are examined 1)3" an experiment on asphalt paved road. 1. I N T R O D U C T I O N
We focus on how to discriminate pedestrians flom other moving objects such as bicy(-les, animals and swaying plants. This technique is inq)ortant ill lnany fields: for example, detecting intruders for a. security,,' gual(ling systeln, a.n(l finding an avoidance-1)ath lnechanism for mobile rol)ots. However, variety,, of clothes, hairstyle and bodyshal)e make it. difficult to realize this end. So, we propose a new (l('te('tioll method which makes fldl use of periodicity in walking [1]. The recognitioll of an ol)je(-t in motion is (me of the tol)i('s of 1)sychol)hysics. ~\}' use the word "motiolF' here to refer to the challge of shape of the ol)ject and not to the change of its position. The moving light disI)lay (5ILD) nlctho(l proposed l)x" Johansson [2] is capable of perceiving the l)ody and the lnotioll of a walker. Kozlowski and Cutting [3] have demonstrated that the MLD can tell the sex of a walker using the patterll of the motion. These facts suggest that the motion of the ol)ject ("(mid 1)e the key to its recognition. R ashid [4] has reconstructed the structure of an ol)ject k o m the MLD kom a.n ellgineering point of view. So, given the structure of the lmlnan body, pedestrians can be selected from other moving ol)jects. However, this idea will not work because it. will be difficult
62 to know the in(livi(h:al ln(~tion of tit(' lmlts (if" th(, l~(,(l(,st:ian, i.('. arlns, f'(,('t, h('a(l an(l so ()lI. \\'(' can't ('Xl)('('t ('v(u'y 1)('(h'strial~ t(~ w('ar Sl)('cial marks at ('v(,ry j()int. P(>lana's a l)l)r()a('h [5] is s(mwwhat (lose to ours" It(' has tal:('n th(' m()ti(m int() a(-(-()lmt 1)y al)l)lying F F T and its 1)(uio(li('ity against tim('. A new ln('thod w(' 1)lOl)OSC in this 1)al)(U t(,lls 1)('d('st:ians h o m oth(u ln()ving ()l)j(,('ts 1)y utilizing two tyl)CS of 1)('rio(li('ity uni(lll{' t{) th(,m. Th(, first ()1:(' is the pitch (): 1)a('(, of walking. \V(' shall ('all this 1)(':i()(li('it)' against tim(' "t(unl)()ral-fr('q~:('ncy". Th(' oth('r tyl)(' is st ri(l(' of walking whi('h wc ('all "Sl)atial-1)('rio(l"" it is a n('w ('on('('l)t a(l(l('(l t() th(' apl)roach by Polana. The pace of walking at natural Sl)r has its o.wn Ul)l)(U' and low(u limits. So has the walking stride. Thus, t lwsc two featur(,s can represent all the motion of 1)c(h'strial~s. In order to ol)sc:vc t h('sc 1)('ri()di('iti('s in mono('ular images, w(' look at th(' intcmsity diffi'r('m'(' l)etwc(,l~ suc('essivc flam(,s. The l~e:'iodi(" ('hal~g(' of intel:sity l~rought al)out 1)y t h(' motion of t h(, f(,et can 1)(, ()l)s(uv(,(l and is ~s(,(l t() ('stimat(' t h(" t('ml~oralh'('qucnt'y. Tlw aml)lit~td(' of tlw os('illati(m will not aff'('('t th(' tcml)()ral-fr('(l~:(u~('y. ()m'(' the 1)a('(' aim dire('tiol: and distan('c of a m()ving ()l)j(,(-t a:(' knr th(, sl)atial-1)('ri(~(l ('an 1)e 1):'('(li('t('(l. Again, thcs(' 1):'o('c(hu('s a:'(' n()t aff'('('t('(l 1)y vis~al (listurl)am'(' as m('nti(m('(l a])ov('. Pedestrian (h,t(,('tio:~ 1)as(,d on t h(, a l)ove i(h'a is :('aliz('(l l)y tim,(' lm~('('ss('s: 1) m()vingol)j('ct (h't('ction, 2) the t ra('kil~g 1)l'O('('ss, 3) s('l('(ti(m l~y rhythln. In section 2, w(" (lcs('ril)(' t h(' m e t h o d of (l(,t('('ting a m()ving ol)j(,('t and its 1)()siti(m in successive h'alnc data. Then section 3 will l~(, (h,v(,t(,d to a tracking m(,th()(l l)asr on th(' kincmati(" model and ('onsi(h~latioll of its :vlial)ility. Cal('~:lation (,f t(,ml)oral-h('(l,:Cn(y and sl)atial-p('riod as wcll as a mod('l mat('hing t('('hniq~:c ::,ill 1)(, Dilly lnVs('nt('d in s(,('ti(m 4. The CXl)(Uiln('ntal results ol)tain(,d (m an asphalt lmV('d roa(l at(' (h,s('ril)(,(l in s(,('ti(m 5. Th(, final section gives some conch:(ling r(una:ks.
2
MOVING
OBJECT
DETECTION
A moving ol).jcct ('an 1)e detect(,d in m a n y ways such as sul)tra('ti(m of su(('(,ssiv(' frames [6], sul)tra('tion from background imag(', ol,ti('al flows [7] and so on. Sin('(" th(, sul)tractioll of su('c'r frames can 1)e dolw in the sh(n't(,st 1)ro('cssing t inw, w(, al)l)li('d this at the l~cginning of the data pro(cssing. The detail('d dctecti(m ln'O('cdm(' is as folh)ws. 1. Th(' illt('nsity ('hang(' 1)('tween su('('('ssiv(' flam('s is ('ah'ulatc(l and its al)solut(' vah:(,s arc t ransferr('d to the following lnOc('ss as sh()wll ill figlll(' 1. 2. To d(:'fin(" the moving-ol~jcct region a('('urat('ly, a vertical ('(lg('s arc sear('h('d fin and d a t a l~oints :cmoved whi('h hay(, h'ss int(,nsity than a t hrcsh(~M vahw. Th~,se lnOCeSs('s a.rc us(,ful bccaus(' imag('s ~)f"p(,d(,strians at(' mo:'~' likc,ly t.o hay(, vcrti~'al ('(lg('s than arc ()tlwr obj('cts. Th(' t hr(,sh(fl(l int(,nsiti(,s I/'t//el and Iv'nl~2 arc fix('(l so to cut off th(' 99 % of the intensity ('hange Inought h o m the 1)a('k-ground of t lw images. 3. To get th(' l~ott(ml position of t h(, ol)j(,(t :vgion, th(, v(,rti('al and h(n'iz(mtal lnoJ('('tions of the intensity a.rc ('al('ulatcd as shown in figure 2. The ol)je(,t-rcgion width
63
F i g u r e 1. D(,t(,ctit)n ()f a m o v i n g ol)ject
F i g l m ' 2. P o s i t i o n an(l wi(lth on t h e imag('
u'i(lt:xl(;(~.) is d(~t('rmined as t h a t of t h( ~ vcrti('al 1)roj(,('ti()n, a n d tit(, h o r i z o n t a l 1)osition u(A.) p o i n t s th(, (cnt(~r of th(' wi(lth. Th(' v('rti('al 1)()sition ~,(~.) is (h't('rmin(~(l as the 1)()ttt)m of the 1)rojti~(:tioll so t() fi)(.us 111)t)n th(, m()ti(m of t ht ~ t'eet. St'e figurt 92 t'()r (l(,tails of tht' r(qations 1)(~twt~en u'i(//.xtc,.(z.), u(~.) a n d ~'(~.). Her( ~ th(" in(l('x L" s t a n d s for tht, tim(, at whi('h th(' flamt, is ol)s(,rv(,tl. Th( ~ 1)ro('t'ss('s list t'd al)ov(, art, 1)el'fOrln('(| ()vt, r t h(' wh()l(' t)f th(' imagt~s ol)tain(~(l at t h(" v('ry 1)(~ginning of th(, d a t a 1)roct'ssillg. H()w(,v(~r, if tht, 1)t)siti()n of th(, ()l)jt,('t ca.n 1)(" 1)redi('t(~d in tit(' t't)urs(, of t r a c k i n g , only th( , (lata in a s m a l l t l a ( k i n g w i n d o w ('Clltt,lt,(l Olt t h(, ol)j(,('t art, 1)r()('t'ss(,(l. T h i s grt'atly r(~thl('('s 1)r()('('ssing cost a n(l nois(' t'a.llst,(| ])y ()th('r m o v i n g ol)j(~('ts.
3. T R A C K I N G Thtu(, art, tw() k i n d s ()fmt, t ht)(l tt) realiz(' t r a c k i n g . Tit(' first is tt) s('t'k th(" mfitllu 9shal)(" of the t)l)j(-'(-t. Gill)(,rt t't al. [8] h a v e (l(,mt)nstrat(,(l this mt, th()d in t r a c k i n g an airl)lan( ~ using its twt)()rth()g()nal 1)rojt'ctit)llS. Howt, vt'r. this mt,thotl rt,tluircs t ht~ ol)jt,('t to 1)t, rigid an(l is nt)t suital)h' for ()1lr l m r p o s ( '. T h t ~ s('('t)n(l att('tho(l is l)ased ()n t ht' ldnt'mati('s of the ()l).j(,(,t. Lt, gtt,rs anti Y ( m n g [9] hart, a i)l)lit'(l a m('tht)tl whi('h us(,s kintunati(" mo(h,ls an(l t.ra('killg filters to the sinmlatt~(| scent'. Alth()ugh thi.s ln('tho(| is us(~ful a n(l suital)h' f()r ()ur goal, it is n()t 1)ow('rflfl (m()ugh. \V(' h()l)(' th(, t r a c k i n g will (-ontinu(, (,Vt,ll if t h(, m('asur(un('nt ('n('()untt'rs ()utli('rs. Th(' ilnl)r()vt'(l r(~('()v('ry svst(un for th( ~ t ra('king will 1)(, stat('d in sul)s(,('tion 3.2.
3.1. K i n e m a t i c s a n d M e a s u r e m e n t M o d e l B('t'or(' s t a t i n g th(' m('asllr(un(mt or (lata h a n d l i n g t('('hni(llt(', th(' s('tting ()f t h(, vi(h,() ('amtua sht)lfl(l l)(' ('xl)laint'(l. \\-(' assure(' it tt)1)(, a 1)in-h()h' c a m t u a an(l alignt'(l 1)aralh'l tt) th(' r()ad. T h i s makt's it 1)()ssil)h ~ to st't, a m a x i m l m ~ mmfl)('r t)f 1)edt'strians in a vi(,w. Tit(' horiz()ntal axis ()f" vi(,w is also 1)aralh'l tt) th(' ,gr()lln(| s() as tt) ltlak(' 1)(,d('strians hay(, m()rt ' vtuti('al ('(lg('s as (letaih~d in the 1)rt'vi()lls s(,('ti()n. L('t .~'(a.)and !I(L.) giv(' th( ' 1)()siti()n
64
Z 9
,,0
/'-'-'-'-'-'-'-'O ~- / D l " " ",,
(~,,v(~)
.I
~ ~ 1 ',, ...--" )- ' 4 ~ ~ . / " , ,
ax,
~(X(k) ,y(k) )-Y
X"
Figure 3. ~Ieasurelnent model and ('OOl'(linat(' s3"stelns of a 1)edestrian at tim(' k. The state v(,('tor X(t.) is conll)osed of the position .r(~.) and y(k.) an(l their (lerivatives with respect to tilne .i'(/,,) and /1(i,.), i.e. velocity. Now we suppose a pedestrian to have an ahnost constant velo('ity. Then the estimated 1)osition of the pedestrian at time (k + 1)-Ys will 1)e
(i)
X(k+t) = FX(,.) + w(t.) where
X(t.)=
,i'(t.) y(k) !)(,.)
1 00
,F-
0 0 0
1 0 0
0 0 1 Ts' 0 1
,w(t.)-
i01 ,'t(t.) 0 ,"2(~.)
"
Here Ys is the nornlalized time interval and w(i,.) is calh'd system noise, w(t.) represents the fact that no pedestrian goes exactly straight, and it. is supposed to be a sequence with zero lnean, white noise and Gaussian with ('ovaiiallce Q. The Q is set to be 0.27Tm/s 2, as determined by experiment. The relation between the position in real-world coordinates with that in the linage plane Y(k) can l)e expressed using translation operator hit. ) as
(2)
Y(~'i = hIk'l(X(~'l) + z(k)
where Y(k)
=
~'(k)
, Zlk)
--
-'2(k)
9
The operator h(t.) defines the relative position of tile video camera and the ground (see figure 3) mid is read as
" fc :r(t.) uc + fc--Y(k) sin(O-Tr/2)+h cos(O-Tr/2) h(k.j(X(k.i) =
(3)
b fc (Y(t.) cos(O-Tc/2)+h s i n ( O - r / 2 ) ) fc-Y(k) sin(O-Tr/2)+h, cos(O-Tr/2)
65 where .f(:, h, and 0 correspond to focal length, height an(l depression angle of the ('amera. The optical axis is set at tile center of the image plane and its position is (u(,, vc). a and b are the aspect, ratios needed to correct the rati() 1)(,tween wi(lth and height of the image plane. Z(k) represents the measurement error and again is set to be zero mean, white noise and Gaussian with cova.rimlce R(k.). R(k) del)ends on the distance 1)etween pedestrian and tile video camera: the measurement noise becomes smaller when a 1)edestrian goes farther from the camera. Extended h:ahnan filter (EKF) is used to get the estimated state vector X(~.]~.) and predicted measurement Y(klk-1). X(klk) is the result of the incremental procedure of EKF. The predicted ~r(klk._l) points the center of the window at the next time step as is mentioned in section 2.
3.2. Checking the Validity of M e a s u r e m e n t The pedestrian may be hidden 1)3;other moving ol)jects and noise may lessen the validity of measurement. These facts reduce the reliability of measur('ment and will lead tracking to end in failure. For these reasons, we should make s~tr(, that observation is reliable. If the observation does not satisfy the following ('()nditions, th(' ol)serve(l l)osition Y(~.) is judged to 1)e wrong and should 1)e replaced with th(" 1)redi('ted 1)ositi()n ~r(~.l~._l). 1. Occlusion When the image of a pedestrian ix covered by another moving object, occlusion may arise. In this case, the 'mcas'urcd width of the object may not lie with in the range of the pedestrian width. So. th,c width of the object is a good measure of occlusion. The width of the object in real spacc wid(~.) is determined by applying an inverse perspective transformation to tt,i(11,~l(;(~.) at (u(~.), t'(~.)), and ch(:cki'ng by thc following condition. I'VI > u'id(k) > H2
(4)
where l't'l and H,~ dcnote the maximum and minimum widths of pedestrian respectively. .
Outlier When the noise cause an outlier in th,c measurement of position, the estimation may not be valid. We check the following condition for validity.
[X(k',/~'-1)-s
p(kl/,._l)-I[X(k',~,-l)-- s 2_
(5)
where ~( is a tentative estimation by the mcas'uTvmcnt "Y at I,. P(~I~-J) is the prediction error covariance determined by EKF. This condition means that wh,en the Mahalanobis distance between ~((k'l~'-l) and ~( is greater than 3.0 .) in the chisquare distribution of 4 degrees of freedom, then the measurement ~Y ix not reliable.
4. D I S C R I M I N A T I O N B A S E D ON R H Y T H M Tile rllythm of walking is caused 1)y a two-stage 1)il)edal action: first, a pedestrian stauds still for a relatively long time on both feet" second, oue of the feet steps forward rather swiftly. This is clearly seen in figure 4 as the 1)eriodi(' intensity change of the sul)tracted image data around the feet. In this section we will r the ways of placing a small window which is especially useful for detecting rhythln and ascribing it to walking.
66
Figure 4. Sul)tra('ted images at the fi,(,t of a 1)edestrian. Th(' ('(mtrast was enhan('ed fin. clarity
Fig~lr(, 5. \Vin(h)w fin d(,te(-tion of int(,nsitv ('hauge
4.1. D e t e c t i o n of R h y t h m A small win(low which in useful fin" (h,t(,('ting th(, rh.xthm ()f 1)('(h'strians in 1)1lt within the tra('king window as shown in figure 5. Th(' small win(low in l l l in wi(lth an(l 11~ in height. The actual values of 11, and ll~ will l)(, stated ill s('('ti(m 5. Th(' int('nsitv in this window will oscillate during walking lint th(' (listan('(' 1)('tw('en th(' camera and t h(, pedestrian aftb('ts its a nll)litude. Since the (listan('(' is inv(u'se 1)rol)ortional to the win(l()w size, the total int(msity within tit(" window in divide(l 1)3" the x,~:in(h)w size ill th(' imag(' plane. The norlnalized tilne se(tuen('c of total int(ulsity (~f"the window (l('ll()t,'(l by I(~.) in pa.rti('ularl.x,, suit al)le fl)r detecting the rh.x'thm. To calculate th(, 1)ower sl)e('trum P(.f) flOln I(~.) (1,: - 0 , . . . , / ~ ) for a d m a t i o n ()f :X' 9T.s. seconds, F F T is not slfital)le be('ause res(flllti()n is lilnit('d 1)y t.h(' numl)er of d a t a 1)r in a sampling interval. So, we use the n mxim, ml cntrol)y m(,thod (~IEM) [10] for its 1)etter fl'eqllency resolution. The final 1)l'('dicti(m Crl'()r in used to estimate the (n'(h'r 3I tbr the aut(~regressive lno(h'l for ~IE~I. The f'~uMaln('ntal t elnl)oral-fl(,(l~ten('y F of I(~.) ol)tain('(l in this way is related to the sl)atial-1)(ui()d T 1)y r -
D F - N . T,s,
(6)
where D iel)resents the distan('e of the ol)je('t at _Y. J-:.s'.
4.2. M o d e l M a t c h i n g Before 1)roceeding to the matchillg 1)roc('ss. a whit(, 1lois(, ('(mtl)(ment is r~unove(l flOln the power sl)e('truln ol)tained. Peaks slnalh,r than PrH/~ are set to 1hake no ~'ontril)uti(m. Walking a.t natm'al speed has its own limitatiollS fl)r th(' flmdamental t('ml)()ral-fl('q,Wll('y F and sl)atial-1)eriod T. Let F:~l.:l.\', F~ll:v and T.~I i.\', T~t/,x l)e the Ul)l)er and lower limits of these quantities. The dimensions of tiles(' 1)aramet('rs at(' presented in sul~section 5.1. The (letaih~d process in as follows: 1. To nornmliz(, the power Sl)ectrunl shal)e: a l l / } larg('r than P/'t11,' are smnme(l:
67 ,\:p
(7) /--1
H('r(' ~ is th(' p()w('r at th(, fl'('qll('ll('y f I (/ - 1 , . . . , A ~ , ) . If th('r(' is n() 1)()W('l ~/ gr('at('r than P~H~r no 1)('(lcstrian is in th(' win(low. S(), tit(, ('vahlati()n valu(' 5" is set to () and th(' 1)ro('('ss is lmltcd. 2. F()r ('a('h fr('(lu(ul('y _F!, th(' ('()rr('sl)()n(ling Sl)atial-1)('ri()(l T/ is ('ah'ulat('(l llsing th(' r('latioll 6. 3. The fr('qu(uwy F i is found at which Pi is a m a x i n m m and slwh that
F,~ttx < F/ < F.~t.~x, T,~llx < T / < T.~I ~.x-.
(s)
4. The evaluation valu(' E is t lw normaliz('(l 1)ow('r Pi, i.(,. E - P j / S . 10() 5. If E is grcat('r than t h(" giv(,n yah1(, E/'H t~, w(, (()n(ht(h ~ that th(u(' is a 1)(,(l(,strian in th(' window. ()n th(, oth(,r hand, if E is smalh'r than ErH/r t h(u'(" is n() 1)('(l('strian. ErH I~ is ('Xl)('rim(mtally s(,t so as t() nfininfiz(' th(, dis('riminati()n ('rr()r and th(, vahw is givcll in slll)s('ction 5.2. In th(, r(,al (mvir()nment, irwgular nois(:, will fr('(lll('ntly al)l)car whi('h has almost th(' sam(' Sl)c('trunl as that a ssigne(l t o 1)edestrians. This t('n(huwy is strong if' th(' ()l)s('rvati()n is ('arri(,d out fl)r only. a sh()lt. tiln(,. . To less(u1 th(, ('()ntril)lltion ('alls(,(l 1)v nois(', th(' . 11111111)o1 of o])s(uvations _V should l)e largc. In oth(,r words, the Salnl)lillg 1)(uio(l sh()uhl 1)(' long. Howcv('r, this will 1hake t lw d a t a 1)r()('('ssing tim(' l()ng. As a r(,sult ()i'a vari('tv of ('Xl)erim(ults. we t'oluld 64 d a t a 1)oints to 1)(, ('n()l@l t() lnak(, t h(' disturl)an(-(' smalhu than E/'Hfr 1)rovid('d only tw()('y('h:'s of 1)(u'i()(li(" ('hang(' wh()s(, fr(,(llWn(-y is th(, sam(, as that of walking, at(' a('cid(,ntally m(,rg('(l in th(, (lata.
5. E X P E R I M E N T S 5.1. E x p e r i m e n t a l S y s t e m The algorithm fi)r 1)('(l('strian dete(tion dcs('ril)(,(l in th(, 1)rcvious s('('ti()ns was t('stc(l using imag(' s('(lu(uwcs of an asl)halt paved road ()n th(' university ca nll),ls. Th(' vi(h'() c a m e r a S o ~ g C C D - TRI()00 was 1)la('('d at 1.()m h('ight, with 15.0 ~ (l('l)r('ssi()n angh'. In t lw RGB dcco(l(,(l vi(h'o signal, only tlw G (,Oml)()n('nt was us('d, whi('h has al)out 60(X: of th(" total hmfinosity. The A N D R O X ICS-400X~I9 imag(' 1)ro('('ssillg system ( C P U 68030, OS-9) was (Unl)loy('d with i b m digital signal 1)r()('('ss()rs. The saml)ling of th(' images and t la('king of the ('Xl)('rim('ntal s('t-ul) arc 1)r('s(ultc(l in tal)h,s 1 an(l 2. 5.2. E x p e r i m e n t a l R e s u l t s Two exanll)los of imag(, sequences arc shown in tigmc 6. Th(u(' are six 1)('(h'strians (Nil-El6), two 1)i('.x'('l('s (NIt, :~i~) and a dog (NI~)). The video ('anwra can look at ol)j(,('ts at distan('cs of 3.0 to 20.0m. Figure 7 plots t h(' cstimatvd traj('('tori('s and v(,lo('iti('s of t h(, ol).ic('ts. Th(' time illtcnsity an(l its power Sl)(,('trlml fl)l ('a(-h ol)j(,(t arc shown in figur(' 8. The int('nsity (l(,1)('n(ls on the a ctioll of walking (n ln()titm such as riding a 1)i('y('h,
68
Table 1 Parameters for the exl)eriments Threshold for moving object detection Number of intensity change data Order of AR model Threshold of power Pace model Stride model Pedestrian width model Ankle height model
Ir..i N.ll -
-
11 64
I r H R2 =
1?
15 P r H , =10.0 F.~/i,v = 1.50
F,~,.ux,
TMI,v =0.45
Tal..~x =0.99
ll~l= li,~--
14:2=
0.66 0.13
-2.60
Hz 111
0.13
Ill 111
Table 2 Examples of the standard deviation of measurement noise Distance frolll the video camera 3.0 5.0 10.0
20.0
m
/R(t:)ll
10.4
7.5
3.2
0.5
1)ixel
R(t,)22
4.9
3.9
2.2
0.7
1)ixel
o1" the running of a dog, as well as on the intensity diff('ren('(, l)etw(,en t lw ol)ject and the background. The pedestrians MI-.NI~i show 1)(,rio(lie change of intensity, but nonpedestrians lklr-M9 exhibit no periodicity. The estimated 1)ace, stride, evahmtion value and real pace for each object are talmlated in tal)h, 3. Th(' real pace for the pedestrialls is that of an average over 10 steps of walking. \Vith regard to non-pedestrian, pace and stride are estimated kom the frequency which has the maximum power. The estimated pace for non-pedestrian Mr is very (,lose to the lower ])omld of F, i.e. FM~A'. However, the stride is estimated to be about 2m which is not within the range of pedestrians (0.45-0.99m). So, we can tell Mr is not a pedestrian. Indeed, the evaluation value for 1~I7 is equal to zero.
The polmlation of the evaluation value E for 533 1)edestrians is shown in figure 9(a) and for 109 non-1)edestrians in figure 9(b). Of tht' 1)('d(,strians 82cZ~.wear trousers, 11% are in knee-length skirts or short 1)ants and 7(X, art' in long skirts. Of tht, non-pedestrians 807c, are bicycles, 18% (logs and 2c/(:. swaying 1)lants. Figure 9(a) shows that evaluation values E for 1)edestrians have a peak at the right side of the figure. On the other hand, for non-pedestrian there is a peak on the left. If the threshold value E'rHI~ is set to be 10, 94.0% of pedestrians and 95.40/(,, of non-pedestrians are corre('tly recognized. This m('ans that 94.2~: of all moving objects are correctly identified. The reasons for error will be summarized as follows. For pedestrians, 1) jittering of the observed images disturb the detection of the rhythnl of walking, 2) with swaying 1)lants often al)l)ear in the tra('king window, 3) walkers light-coh)red shoes diminish the contrast between the images of feet and the paved road. Reasons (1) and (2) are also true for n~ Jitters between successive fram('s is the main cause of error.
69
Figure 6. hnage sequences for.the eXl)eriments
6. C O N C L U D I N G
REMARKS
\Ve have I)roposed a new method for discrimination of pedestrians fl'om other moving objects based on the rhythm of walking. This method is achieved by the following three processes: moving-object detection in successive frames, tracking of moving objects using kinematic models and measurement applying extended 1,2ahnan filter, and searching for the walking rhythm in the power spectra of intensity of the images. These processes arc realized and executed in a realtime operation. The experiment has shown 94.2% of moving ol)jects to 1)e correctly
70
20
,
10
'
'
' .,
' ' ., ' ' '~ " '\ \~2" ,,. ,,,,~, M1 'x " 5.09 9 \\\' \, km/h ,, 4.71 \ \ "\, km/h \\ "". M3 \ ", ",, '", 6.33 \,\ "\ ", km/h ".,, \,, ""
\
-,4'
'
-'2
'
20'
'
',
';\ ' '\
M5 3.92 km/h
'
'
'
'
"\'x,\\,,X M6 \'", "'x"~ 4.04 ~ '\ \~ km/h "'\, Ix\
10
~r -4 i
20r
\,, \ M4 ',,X~,),,~: X 4.27 \ ~,, \ \ _ ,_ '~ \ . \ Km/n ""'\."*'\ X
~ i,
6 x(~) m
'
i
',
-'2
!
6 'x(~) m
I
, ,
\
',
\
M7 '" '" 10.54 ",, 101"-
km/h
\
\
i
'
9 "",
6.86 km/h .
\\
xx ~
M8 7.00
',
M9
~ -4.
I
\',,,~ ~kk\\
~.. !
i
. -'2.
\\
\\k m / h
"% N "',
"\, \
6
x(;0 m
Figure 7. Traj(,('tori(,s of l nOVillg ol)j('cts (1)ir(l's-(,y(, vi('w)
Tal)l(, 3 Estimat('d 1)a('('. st.ri(l(,, ('valuatioll Ol)j('('t lmznl)(u" ~Ii NI.2 Estilnat(,d 1)a('(' 1.92 1.91 Estimat('(l stri(l(' 0.74 0.68 Evaluation vahw E 48 92 R('al 1)a('(' 1.91 1.93
yah1(, Nla 1.94 0.91 73 2.00
and r('al 1)a('c of ea('h ()l)j(,ct Nil NI.~ Nlci NIt his 1.86 1.85 1.89 1.39 0.42 0.64 ().59 0.61 2.11 4.63 85 100 100 0 0 1.88 1.86 1.84
M!) 0.71 2.69 0
Hz m Hz
and 1)e(l('strians t() 1)(, re('ogniz('(l w('arillg various kin(Is of cl()th('s, and with diff(uing hairstyl('s and 1)()dyshal)('s. The exp(uim(,ntal results v('riI~" the vali(lity and r()l)ustness of th(, met h()(l. Th(" a(lwmtag('s of ()ur ln(,thod al'(': 1) the ('l( )t h('s, shal)(' of 1)()(ly, hairstyh' and s() on (l() not aff('('t th(, :(,suits, 2) th(' 1)a('( 9(t('lnl)()ral-h('(lll('ll('y) and th(' stride (Sl)at.ial-1)(ui()(l) at(' well ('()ml)incd tog('th(,r t.() make th(un 1)()w('rful (,n()ugh t() r(,m()v(, mils(, su('h as swaying plants, 3) the m('th()(l is simple and can 1~(, 1)(,rf'():m(,(l in r(,al time. Pedestrians in sha(low may not 1)(, (l(,t(,(,t(,(l 1)y ()lu 1):('s('nt m('tho(l. This 1):ol)l('n: should 1)(' ()v(u'('(mw in th(' :war fltt,u('. ACKNOWLEDGMENTS
\V(' at(' thankfld to Associat(' Prof. Dr. I(. Fujima fin valual)l(, sl:gg(,sti()n to iml)r()v(' this 1)at)(u". This work has 1)(,on 1)artia.lly SUl)l)()lt('(l 1)y Grant-Ai(l for S('i(,ntifi(" R('s('al('h flom ~Iinistrv of Edu('atioll No. 03650343. ()kawa Institut(' ()f hfformation and T('l('('ommuni('ati(m No. 91-32 and OMI1ON Co. Lt(l. REFERENCES
1.
H. 1[():'i, N.1I. Charkari and T. XIats,:shita. "()n-line \),hi('l(' an(l Pedestrian D('t(,cti(ms Based on Sign Pattern", IEEE Trans. In(h:s. El(,ctroni('s. \2)1. 41, No. 4,
71
>, 6 0 t l , , , , , , ,
,
60
oo 40
,
,
,
,
,
,
,
,
,
60
i
,
0 0
'
40
40
20
20
,
,
,
,
' 2
'
,
,
,
,
,
c-
(D
__E 20 0"()
'
601_,
'
' :2 '
, 1, ,
,
'
' 4s
,
, ,
0 0
o~ 4020 i-i, 0
2
(~[~)
4
601~ , , , , ,
,a'
Hz
,
0
~ 60
i'
2
,
__E 20
20
;
O"
' I-I
' '
~-o ~ 1000I
' 2
'
'
'
'
'
'
'
'
;,s'
",'
,
6
Hz
'
,
2
i
(~[,~)
0
(3 Hz
I'
'
'
'
'
'
'
0";
'
'
' 2 '
'
' ~,s I
::4~
'
, :
4
'
4
60
.'.',,., 2
'
'
,
,
,
4
(x[:~)
,
,
,
,
6
Hz
,
,
4s '
'
'
0
'
2
'
'
'
0 0
4~f,
0
2
4s
'
,-
4
'
I
I
t
O
6 Hz
(~[~-,)
'
'
'
I
I
I
2 ~-I
o
1000
140,
'
4s
100
20~
60
2
20
10I~,
Oh--'b-'~
0
~
2011
A t
0
40
0 0
'
,
~ ,
4
(x[._,)
40
(D
, i
' 4s
1 24~
0
6
--(oo-9 40
o~
60i,
'~176176I
0
60
4s
2 0 0 0 ~ ~ ~ r ~
B_
>,
2
60~,
2
,
,
4~
,
v 0
I
.
~
0
(~[.) ,
4
6 Hz
,
,
,
I
I
I
2 I
.I
I
I
,
4
1
4s I
J
11
&.
0
0
0
2
(~[:)
4
6 Hz
I- I
0
2
4
6 Hz
F i g u r e 8. I n t ( ' n s i t y ('haltgc a n d p o w e r Sl~(,('tllun ~ff' c a ( h ol~j(,(t
0
2
4
6 Hz
72
100 104 o
2oo7 1 94 OO/o
I [
100-
e-
t_ U.
0
10
50 Evaluation Value (E)
90 100
(a) Pedestrians
0 10
50 90 100 EvaluationValue (E) (b) Non-1)edestrians
Figure 9. Histograms of evaluation value
pp. 384-391, 1994. 2. G. Johansson, "S1)atio-Teml)oral Differentiation and Integration in Visual Motion Perception", Psych. tt,'s., Vol. 38, Pl). 379-383, 1976. 3. L.T. I(ozlowski and J.E. Cutting, "Recognizing the Sex of a Walker fi'om a Dynanfic Point-Light Disl)lay", Perception & Psychophysics, Vol. 21, No. 6, pp. 575-580, 1977. 4. R.F. Rashid, "Towards a. System for the hlterl)retation of Moving Light Disl)lays", IEEE Trans. Pattern Anal. Math. Intell., Vol. PAMI-2, No. 6, pp. 574581, 1980. 5. R. Polana, "Detecting Activities", Pro('. of IEEE Computer Soc. Conf. on Computer Vision and Pattern Recognition, pp. 2 7, New York, 1993. 6. M. Yachida, M. Asada and S. Tsuji, "Automatic Analysis of Moving hnages", IEEE Trans. Pattern Anal. ~[ach. Intell., Vol. PAMI-3, No. 1, pp. 12-20, 1981. 7. A. Shio and .I. Sklansky, "Segmentation of Peol)le in Motion", Proc. of IEEE Workshop on Visual Motion, pp. 325-333, 1991. 8. A.L. Gill)ert, M.G. Giles, G.M. Flachs, R.B. Rogers and Y.H. U, "A Real-Time Video Tracking System", IEEE Trans. Pattern Anal. Math. Intell., Vol. PAMI-2, No. 1, pp. 49-56, 1980. 9. G.R. Legters, Jr. and T.Y. Yomlg, "A Mathematical Model for Computer hnage Tracking", IEEE Trans. Pattern Anal. Math. Intell., Vol. PAMI-4, No. 6, pp. 583594, 1982. 10. D.F. Elliott (ed.), Handbook of Digital Signal Processing Engineering Applications, pp. 701-740, A('ademic Press, 1987.
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
73
Visual Recognition of Obstacles on Roads Uwe Regensburger and Volker Graefe Institut ftir MeBtechnik Universit~it der Bundeswehr Mtinchen Neubiberg, Germany Fax (+49 89) 6004-3074
As part of a driver support system for motor vehicles on freeways an obstacle recognition system was developed within the EUREKA project PROMETHEUS over the last four years. The obstacle recognition system uses a single monochrome TV camera and a multi-processor robot vision system. The recognition system includes a road tracker and an obstacle detector that were described in previous publications, and an object classification and tracking module presented here. The module is based on generic 2-D object models. It recognizes obstacles in real time and from distances of 200 to 300 m, sufficient for high-speed driving. The module was extensively tested in real-world scenes on the German Autobahn and on various other roads, including city streets with heavy traffic.
1. INTRODUCTION A driver support system for road vehicles is a great challenge for the technology of machine vision. The goal is to relieve the driver from monotonous tasks, to warn him of imminent danger (for instance, if he is about to fall asleep), and - possibly- to assume control of the vehicle in precarious situations. The necessary technology is being developed in numerous laboratories all over the world; there is hope that it will eventually lead to a significantly increased traffic safety [Masaki 1992]. Among the characteristics demanded of such a system are high reliability, low costs, and independence of any special infrastructure. Within the European research project PROMETHEUS the Institute of Measurement Science has developed real-time vision technology that may be used for a driver support system [Graefe 1993 ]. Freeways were chosen as the principal domain for testing and demonstrating the visual recognition of objects that are relevant for the understanding of traffic situations. The reason for choosing freeways, in spite of the typically high driving speeds, is that the complexity of the traffic situations and the variety of objects are much lower on freeways than on other roads. Obstacle recognition is an essential subtask of a driver support system. (In the sequel "obstacle" refers to any visible object on the road in front of the ego vehicle that is, or could possibly become, a danger for the ego vehicle.) According to a concept for obstacle recognition proposed by [Graefe 1989] the task should be decomposed into several largely independent subtasks, or modules, including the initial detection of possible obstacle candidates, the classification of obstacle candidates into false alarms (caused e.g. by shadows), and real physical objects, the tracking of obstacles, etc.. A module for detecting obstacles in distances adequate for high-speed
74 driving was introduced by [Solder 1992] and [Solder, Graefe 1993], and road modules by [Tsinas, Graefe 1992] and [Wershofen 1992]. A multitude of methods for recognizing obstacles in monocular image sequences has been described in the literature. Essentially three different classes of methods have been used. Gestalt approaches, e.g. [Davies 1990], aim at a direct reconstruction of the 3-D world as a basis for object recognition. Other approaches, e.g. [Enkelmann 1990], [Carlsson, Eklundh 1990] and [Young, G. et al. 1992] evaluate displacement vector fields and the differences in image motion between points within the road surface and points above the road surface. Due to the large amounts of computation required by these methods, they have not yet been demonstrated in real time and, thus, do not appear suitable for vehicle applications. The third class of methods is based on object models. According to this approach objects are recognized by certain combinations of visible features. This may be accomplished in a number of different ways. As one example, [Moil et al. 1989] and [Raboisson, Even 1990] first segment the image. Irregularities within the road segment are then taken as indications for objects. Additional criteria (edges, color, etc.) are used to minimize the frequency of false detections, while additional sensors, such as laser range finders [Alizon et al. 1990] or radars [Young, E. et al. 1992], may be used to measure the distance from an object. [Korn 1989] bases the tracking of vehicles on histograms of edge orientations. [Seitz et al. 1991] use a hierarchically structured multi-layer representation of edge directions for building object models and recognizing objects in images. [Hartmann, Mertsching 1992] describe object images by a "hierarchical structure code." [Ktihnle 1991] and [Zielke et al. 1992] track vehicles on the basis of edges and symmetry. [Graefe, Jacobs 1991], [Efenberger et al. 1992], [Solder 1992], and [Solder, Graefe 1993] detect and track vehicles and obstacles on roads by comparing generic 2-D object models with edges detected in image sequences. [Thomanek, Dickmanns 1992] and [Schmid, Thomanek 1993] use spatio-temporal models for tracking objects.
2. THE CONCEPT OF OBSTACLE RECOGNITION An object-oriented concept for dynamic robot vision has been proposed by [Graefe 1989; 1992]. It is based on the notion that the world is structured, rather than monolithic, and consists of individual objects. Only a few physical objects in a robot's environment affect the control of the robot in any instant. Following this concept, the task of object recognition should, therefore, be structured according to the relevant objects. This is done by associating one individual object process within the vision system with each relevant external object. The following object processes are needed for recognizing obstacles on roads (Figure 1): 9 Recognition and tracking of the road (because only objects that are actually sizuated on the road are relevant as obstacles) 9 Detection of obstacle candidates (the corresponding object is either a hypothetical object that has not yet been found in the image, or an object that has just been detected) 9 Tracking of an obstacle; multiple copies of this process may exist in the system to allow multiple obstacles to be tracked simultaneously.
75
Video section
V Road recognition
V
0otection II tr. ,o0
V Obstacle
tracking
L _ _ '_--'-'-- . . . . . _; Situation assessment
Vehicle control
Figure 1 Structure of an object-oriented obstacle recognition system Each object process receives image data (pixels) from the video section of the vision system and continuously generates a description of its assigned object. All object descriptions are evaluated by the situation assessment module and combined into a situation description. Depending on the perceived situation, vehicle control commands or messages for the human operator may be issued. Step by step the information contained in the pixels is transformed into more and more abstract forms. Starting with a large number of pixels a chain of data fusion processes eventually leads to a description of an object or a situation. At each step specific knowledge in the form of appropriate models is introduced into the process (Table 1). At the first step, the feature extraction, groups of pixels are combined into features, such as edges or corners. Knowledge about characteristics of the camera and of the features being searched is used in this process. Feature extraction may be supported by the next higher level in the processing hierarchy, the 2-D object level. Here, knowledge about the visual appearance of specific physical objects is used to generate hypotheses regarding the nature of the object being seen and expectations as to which of its features should be visible, and where in the image they should show up. Feature extraction is thus not a form of schematic image processing, but part of a model- and knowledge-based hypothesis verification or falsification process; its task is to provide answers to very specific questions as to the presence and location of certain expected features in the image. A very robust image interpretation may be realized by such an interaction between the feature extraction and the 2-D object levels. It is possible to add additional 3-D or 4-D (spatio-temporal) levels if a description of the 3-D shape of physical objects, or of their motions, is desired.
76
Table 1 Levels of Modeling and Data Fusion Model Level
Data Fusion Performed at that Level
groups of pixels feature
-, elementary features complex features
2-D object
groups of features 2-D objects
4-D object
2-D objects, time, sensor data 4-D objects
environment and situation
robot state, external objects -- situation
3. THE FALSE-ALARM PROBLEM
For reasons of safety an obstacle detection system must be perfectly reliable in the sense that it never fails to detect any obstacle that is actually present. Moreover, in order to allow a timely reaction, obstacles must be detected while they are still far away. These requirements make it practically unavoidable to generate false alarms occasionally, caused, for instance, by shadows, dirt, markings or patches on the road surface. Although false alarms are not directly dangerous they must be processed and, thus, constitute an unnecessary load for the vision system. To minimize this load, false alarms should be identified and discarded as quickly as possible. Distinguishing between actual obstacles and shadows is easy in many cases, but sometimes it can be extremely difficult. We use a set of heuristics to reject most shadows within the detector module as quickly as possible. Therefore, all alarms reported by the detection module are classified by matching the visible features that have been detected in the image against generic 2-D models of obstacles. If a fairly good match is possible, the obstacle candidate is accepted as an actual obstacle. This decision is final. If, on the other hand, the detected features do not match the expectations derived from the model, the obstacle candidate is considered a possible false alarm. A final rejection, however, does not occur at this point. Rather, the detector is requested to report the same obstacle candidate again if it can be detected in one of the subsequent images. Due to the motion of the vehicle the subsequent images will differ from the previous ones in respect of perspective, specular reflections, occlusions, etc., and, accordingly provide an independent chance to recognize an actual obstacle if there is one. A final rejection thus occurs only if an obstacle candidate has been classified many times and has not been accepted even once.
77 If the time required for one cycle of detection and rejection is short, many independent images may be analyzed while approaching an obstacle candidate. The probability of rejecting an actual obstacle is then very small. On the other hand, it is possible that once in a while false alarms are accepted as an obstacle. They must be rejected during tracking.
4. GENERIC 2-D OBJECT M O D E L S 4.1 Effects of Distance on the Visual Appearance
As Figure 2 shows, the visual appearance of an object depends strongly on the distance from which it is seen. Moreover, for a given camera and object the area of the object's image is inversely proportional to the square of the distance. Images of nearby objects are clear with large contrast and many visible details. Many features may, therefore, be detected in such an image. A lateral motion of the object causes a significant motion of its image. Images of distant objects have a low contrast and do not display much internal structure. Their contours tend to be somewhat blurred. Because of their small size and low contrast hardly any internal features can be detected. Rotation of the camera is the dominating cause for motions of objects in the image. In an intermediate range of distances the contours of objects tend to be clearly visible, but the object images do not have much internal structure. Motions of the camera and of the objects have similar effects on the motions of the image. Because of these differences specific models are used to model the appearance of physical objects, depending on their distance. The models include a number of free parameters in order to let them match objects of different shapes. Therefore, they are called generic object models.
4.2 The 2-D Model for the Nearby-range
Most of the obstacles that exist on freeways display a nearly rectangular contour with vertical and horizontal sides when seen from an approaching vehicle. Therefore, parts of rectangular contours are searched in the image.
Figure 2 Different 2-D object models (top) take into account the significant differences in the appearance of nearby (left) and far away (right) objects.
Characteristic points on the contour that may be used for an object description are the corners or the centers of the sides of the rectangle. The upper part of an obstacle is often seen before a structured background, such as trees, mountains, or other vehicles; this makes it difficult to locate the upper part of the contour in the image. Usually numer-
78 ous horizontal edges are visible, and it is not easy to determine which one is the upper edge of an obstacle. The lower part of the contour is usually seen before the surface of the road and may, therefore, be detected and tracked more easily. Recognizing only the left, right, and lower edge is sufficient for tracking the object in the image and for measuring its distance. Therefore, only these edges are looked for in the image. Nevertheless, the large number of features that are often visible in the images of nearby objects makes it difficult to single out those belonging to the contour of the object. A statistical approach was chosen to solve this problem. The image of a typical object contains a number of homogeneous areas, delimited by approximately vertical edges. Many of these homogeneous areas begin, or end, near the le~ and right contours of the object. In an area of interest that encloses the image of the object candidate, the le~ and right ends of homogeneous areas, and their centers, are counted in each image row. Local maxima of the distribution of these features indicate a possible location of the object contour and its center line. Combining these yields strong expectations where the contours of the object should be found in the image. Vertical edge detectors based on controlled correlation [Kuhnert 1986] are then applied in the vicinity of the expected contours to find the exact locations of the contours. Generally, the lower edge of an object is distinctly visible. In case of vehicles the lower edge of the shadow underneath the vehicle is taken as the lower edge of the object. It may usually be detected reliably and robustly because the shadow appears much darker than the pavement. 4.3 The 2-D Model for Distant objects
Images of distant objects are typically small and of low contrast (Figure 2). Only few features may be extracted; therefore, a statistical approach is not feasible for recognizing such objects. The left, right, and lower edges are detected directly by applying suitable edge detectors. Due to the large image motions induced by unavoidable camera motions, the tracking of distant objects would not be robust if it were based only upon a few local edge elements. Using additional features, such as corners and symmetry, improves the robustness. In contrast to the near range where, due to the high spatial resolution of the image, object contours rarely contain any sharp corners, corners may be detected robustly in the far range. Specific corner masks are used for detecting the lower left and right corners of objects by the method of controlled correlation. The images of many distant objects, including the rear-views of vehicles, have a vertical symmetry axis. Symmetry is, therefore, included as an additional feature in the object model. In the near range symmetry would be difficult to detect because of the many visible details, and finding a symmetry axis would be time-consuming due to the large areas in the image that must be processed. In the far range, where object images are small and do not contain many features, symmetry detection is fast and robust.
5. DETECTING VERTICAL SYMMETRY AXES Numerous methods for detecting and exploiting symmetry have been proposed. If object contours have been determined and converted into B-splines, symmetries may be extracted according to [Saint-Marc, Medioni 1990]. A method for detecting symmetries of polyhedrical objects has been described by [Jiang, Bunke 1990]. Shape recognition based on symmetry was introduced by [Yuen 1990]. [Friedberg 1986] determined symmetry by computing a moment matrix. A known
79 symmetrical object may be detected by convolving its image with a mirror image of a model; atter detection it may be localized by convolving it with its own mirror image.
Figure 3 Pairs of pixels, centered around a pixel, x, are evaluated. For each pair the difference of the grey-levels is computed. The absolute values of the differences are added up. Their sum is the symmetry function at the center pixel x.
All these methods presuppose a good segmentation of the image (e.g. into object and background) or have been tested only on synthetic images or line drawings. Moreover, they require computing times of tens of seconds.
Methods that are suitable for real-time and real-world applications have also been proposed. [Zielke et al. 1992] determine for each pixel a symmetry measure, based on the even and the odd part of the symmetry function. A symmetry-based statistical approach has been used by [Ki~hnle 1991] to determine the center of the image of a car as seen from the rear. Histograms of symmetrical contour pixels, of symmetrical grey-level pixels, and of the lengths of horizontal edges are computed; local maxima in all three histograms indicate candidates for a vertical symmetry axis [Marola 1989]. In the sequel a method is described that detects individual symmetry points and then determines image columns containing a maximum number of symmetry points. For finding a symmetry point the even part of the symmetry function is determined. To accomplish this the similarity of greylevels of pairs of pixels, located symmetrically to the left and to the right of a point in question, is studied (Figure 3). The absolute values of the differences of the grey-levels of each pair of pixels are added up and assigned to the pixel in question. A symmetry function is obtained by repeating this procedure for all pixels within an area of interest. Local minima within a row of the symmetry function indicate symmetry points; in Figure 4 the symmetry points have been marked. Figure 5 shows the corresponding symmetry function, computed over a 40 pixels by 40 pixels area of interest around the vehicle. When symmetry points are searched, the width of the environment upon which the symmetry function is based should be somewhat larger than the width of the object. However, if the
Figure 5 Figure 4 Symmetry points determined from 12-pixel wide environments
The symmetry function in an area of interest covering the car in Figure 4; each graph corresponds to one image row.
80 environment is chosen too large, many background pixels are included in the computation. A strongly asymmetrical background may then severely distort the symmetry function.
6. EXPERIMENTAL RESULTS 6.1 Test Environment
A real-time vision system BVV 3 [Graefe 1990] was used for testing the obstacle recognition system. Each of the 3 sub-processes, road recognition, obstacle detection, and obstacle tracking, was implemented on a separate parallel processor within the B VV 3. The obstacle recognition system was designed to cooperate closely with other modules of a driver support system, for instance modules for modeling the dynamics of the recognized obstacles. Such a cooperation has been tested earlier and described by [Dickmanns, Christians 1989] and [Regensburger, Graefe 1990]. The tests reported in the sequel were, however, performed without any additional modules. The results may, therefore, be directly attributed to the modules described here. The situation assessment module (cf. Figure 1) was replaced by a message passing system that allowed communication between the recognition modules, but did not perform any additional processing. 6.2 Recognition Distance
The ability to recognize obstacles from a great distance is of critical importance for driving at high speed. Experiments have shown that cars (moving and resting) can be tracked in distances of about 200 m; under cooperative conditions, e.g. high contrast and little camera motion, cars could be tracked in distances of over 300 m. As Figure 6 shows, a car in a distance of 300 m appears rather small in the camera image. The obstacle tracking module does not include any distance measurement. The distance from the obstacle was, therefore, measured by external means. Visible markers were placed on the pavement in known intervals. As this was impossible on a public road, the runway of a closed airfield was used for the experiments to measure the maximum recognition and trackingrange,
Figure 6 Passenger car seen from a distance of 300 m through the camera of the vision system, with overlaid markings showing the detected contour and symmetry axis
6.3 Robustness
Although the obstacle recognition system was designed for freeways, due to the model-based tracking module it proved to be so robust that it was also tested on ordinary highways and furthermore in heavy city traffic. There, other cars, as well as the own car, changed lanes frequently causing time-varying occlusions, and together with structured backgrounds and markings on the pavement generated a high degree of visual complexity. Even in such difficult scenes (Figure 7) vehicles could be tracked reliably over extended periods of time.
81 The 2-D object models employed by the recognition system assume a nearly rectangular object contour and they have been optimized especially for cars and trucks. Nevertheless, it could be shown that various other objects that may occasionally be seen on roads, such as boxes, barrels and, in some cases, persons, may also be classified and tracked. However, tracking such objects that do not match the generic models well is less reliable, especially under conditions of poor lighting or when they are far away and their images are small. Increasing the repertoire of different object models in the system should further improve the system performance in this respect.
6.3 Speed Figure 7 The computing time required by a module of Tracking a vehicle in heavy city traffic a real-time vision system is always of prime importance. The cycle time of the object classifier and tracker was, therefore, measured in a number of experiments. One parallel processor of the robot vision system BVV 3 using an Intel 386/20-MHz CPU, augmented by a special co-processor for fast feature extraction [Graefe, Fleder 1991], was used as a hardware platform in the experiments. The basic cycle time of the vision system is the image field period of the camera's video signal, i.e. 20 ms. Thus, all cycle times are integer multiples of 20 ms. Tracking a distant object requires always less than 20 ms of computing time; the cycle time is in this case 20 ms. The time required for tracking a nearby object depends on the size of its image (see Table 2). On freeways the cycle time is usually between 40 and 60 ms. Only when the distance to the preceding vehicle is exceptionally short, e.g. in stop-and-go traffic, a cycle time of 80 ms may be observed. In such situations it would be preferable anyway to evaluate the image of a wideangle camera instead of the telescopic image used, because the lower edge of the vehicle may not even be visible if a vehicle in such a short distance is imaged by the telescopic camera.
Table 2 Cycle times for tracking nearby-objects Object width
Cycle time
< 45 pixels
40 ms
45 - 90 pixels
60 ms
90 - 120 pixels
80 ms
82 6.4 Reliability of Classification
Parameters such as scene complexity or image quality that are essential for judging the performance of the tracking module are difficult to define quantitatively. Therefore, long image sequences covering a wide variety of situations were evaluated. A 20-min run on a freeway may be used as an example. It included 60,000 images processed in real time without any preselection. Goal of the classification of obstacle candidates is the discrimination between false alarms and actual obstacles. It should be remembered that primarily it must be avoided to ever mistakenly reject any actual obstacle as a false alarm. (The other type of misclassification, accepting a shadow as an obstacle, is much less dangerous.) Table 3 shows the results obtained during the test. All objects that were reported by the detector were correctly classified, and only a very small fraction of the system resources were wasted tracking false alarms. Table 3 Results of object classification during a 20-min run on a freeway
Number of actual objects misclassified as false alarms Percentage of false alarms that were correctly rejected
0
immediately
67%
within a few seconds of tracking
33%
Number of final misclassifications
0
Fraction of time spent tracking false alarms
1.3%
6.5 Quality of Tracking
Because the true motions of the objects tracked in natural scenes were not measured, it is difficult to determine the quality of the tracking, or the accuracy of the localization of object features, quantitatively. Figures 8 and 9 are intended to give at least a qualitative impression of the quality of tracking. The ranges of applicability of the two models, for the near range and for the far range, overlap to a large extent. This allows a direct comparison between the two models by using an object in the overlap range. In each of the Figures, 8 and 9, the left graphs show the image coordinates of the two sides of a car driving in front of the vehicle with the camera. The right graphs show the width of the car's image in pixels. The scene was recorded on video tape while driving in a car, and then evaluated in real time in the laboratory, using the methods for the near range (Figure 8) and for the far range (Figure 9) on the same analog data. Both methods yield nearly identical results. The method for the far range yields slightly smoother curves (in both cases raw, unfiltered data are shown). This is probably due to the robust localization of the lower corners. Altogether both curves indicate that the unfiltered measured data are consistent and free from significant outliers, in spite of the image deterioration caused by the video tape recorder. This may be taken as evidence for a robust tracking.
83
image column 240
/
width of object (pixels)
"'-~ ,,,~_
"~--- I " ~
60
200
-- _ ---"---
\ 40
160 , 120
20 0
100
200
300 400 program cycles
0
100
200 300 400 program cycles
Figure 8
Results of tracking a car in an intermediate distance range using the model for the far range (1 program cycle is equivalent to 20 ms)
image column
width of object (pixels)
240
200
F'-
6(1 ~ ' ~
9, - , - ~
n r -
-
4()
160 /
"~1.. " ' , v ~ ~
120 0
100
200 program cycles
2C 0
100
200 program cycles
Figure 9
Results of tracking a car in an intermediate distance range using the model for the near range (1 program cycle is, on average, approximately equivalent to 40 ms)
7. SUMMARY As an essential part of a driver support system for road vehicles a module for the robust classification and tracking of objects on roads has been realized. Obstacles are recognized in real time and from great distances (200 to 300 m) as required when driving at high speed on freeways. The obstacle recognition module was originally designed for the relatively simple scenes on freeways, but it has proven sufficiently robust to allow the tracking of moving vehicles even in heavy city traffic. The module is based on generic 2-D models optimized for the rear views of cars and trucks. However, the genetic models are sufficiently general to also allow the tracking and classification of various other objects that may occasionally be encountered on roads. For a truly reliable recognition of arbitrary objects the repertoire of internal object models should be expanded.
84
ACKNOWLEDGEMENT Part of the work reported has been performed with support from the Ministry of Research and Technology (BMFT) and from the Daimler-Benz AG within the EUREKA project PROMETHEUS.
REFERENCES Alizon, J.; Gallice, J.; Trassoudaine, L.; Treuillet, S. (1990): Multi-sensory data fusion for obstacle detection and tracking on motorway. In Monique Thonnat (Ed.): Proceedings of the Workshop on Vision (Prometheus, Pro-Art). Sophia Antipolis, pp 179-187.
Carlsson, S.; Eklundh, J.-O. (1990): Object detection using model based prediction and motion parallax. In Monique Thonnat (Ed.): Proc. of the Workshop on Vision (Prometheus, Pro-Art). Sophia Antipolis, pp 139-145. Davies, E.R. (1990): Machine Vision: Theory, Algorithms, Practicalities. London: Academic Press, 1990. Dickmanns, E.D.; Christians, Th. (1989): Relative 3-D State Estimation for Autonomous Visual Guidance of Road Vehicles. Proceedings, Intelligent Autonomous Systems. Amsterdam. Efenberger, W.; Ta, Q.-H.; Tsinas L.; Graefe V. (1992): Automatic Recognition of Vehicles Approaching from Behind. Proceedings of the IEEE Intelligent Vehicles '92 Symposium. Detroit, pp 57-62 Enkelmann, W. (1990): Obstacle detection by evaluation of optical flow fields. In Monique Thonnat (Ed.): Proceedings of the Workshop on Vision (Prometheus, Pro-Art), Sophia Antipolis, pp 146-166. Friedberg, S.A. (1986): Finding Axes of Skewed Symmetry. Computer Vision, Graphics, and Image Processing, 34, Nr. 2. San Diego: Academic Press, pp 138-155. Graefe, V. (1989): Dynamic Vision Systems for Autonomous Mobile Robots. Proceedings, IEEE/RSJ International Workshop on Intelligent Robots and Systems, IROS '89. Tsukuba, pp 12-23. Graefe, V. (1990): The BW-Family of Robot Vision Systems. In O. Kaynak (ed.): Proceedings of the IEEE Workshop on Intelligent Motion Control, Istanbul, pp IP55-IP65. Graefe, V. (1992): Vision for Autonomous Mobile Robots. Proceedings, IEEE Workshop on Advanced Motion Control. Nagoya, pp 57-64. Graefe, V. (1993): Vision for Intelligent Road Vehicles. Proceedings, IEEE Symposium on Intelligent Vehicles. Tokyo, pp 135-140. Graefe, V.; Fleder, K. (1991): A Powerful and Flexible Co-processor for Feature Extraction in a Robot Vision System. International Conference on Industrial Electronics, Control, Instrumentation and Automation (IECON '91). Kobe, pp 2019-2024. Graefe, V.; Jacobs, U. (1991): Detection of Passing Vehicles by a Robot Car Driver. IEEE/RSJ International Workshop on Intelligent Robots and Systems, IROS '91. Osaka, pp 391-396. Hartmann, G.; Mertsching, B. (1992): A Hierarchical Vision System. Proceedings, IEEE Intelligent Vehicles '92 Symposium. Detroit, pp 18-23.
85 Jiang, X.Y.; Bunke, H. (1990): Detektion von Symmetrien polyedrischer Objekte. In GroBkopf (Hrsg.): Mustererkennung 1990, 12. DAGM-Symposium, Oberkochen-Aalen, InformatikFachberichte 254. Berlin: Springer, pp 225-231.
Korn, A. (1989): Zur Erkennung yon Bildstmkturen durch Analyse der Richtungen des Grauwertgradienten. In Burkhardt et al. (Hrsg.): Mustererkennung 1989, 11. DAGM-Symposium, Hamburg, Informatik-Fachberichte 219. Berlin: Springer, pp 507-511. Kiihnle, A. (1991): Symmetry-based recognition of vehicle rears. Pattern Recognition Letters, Volume 12, Number 4. North-Holland, pp 249-258. Kuhnert, K.-D. (1986): A Model-driven Image Analysis System for Vehicle Guidance in Real Time. Proceedings of the Second International Electronic Image Week. CESTA, Nice, pp 216-221. Marola, G. (1989)" Using Symmetry for Detecting and Locating Objects in a Picture. Computer Vision, Graphics, and Image Processing, Vol. 46, Nr. 2, San Diego: Academic Press, pp 179-195. Masaki, I. (1992): Vision-based Vehicle Guidance. I. Masaki, editor. New York: Springer 1992. Mori, It.; Nakai, M.; Chen, H. (1989): A Mobile Robot Strategy: Stereotyped Motion by Sign Pattern. Preprint of the 5th International Symposium of Robotics Research. Tokyo, pp 225-236. Raboisson, S.; Even, G. (1990): Road Extraction and Obstacle Detection in Highway Environment. 2nd Prometheus Workshop on Collision Avoidance (CED 3). Coventry, pp 74-83. Regensburger, U.; Graefe, V. (1990): Object Classification for Obstacle Avoidance. Proceedings, SPIE Symposium on Advances in Intelligent Systems. Boston, pp 112-119. Saint-Marc, Ptt.; Medioni, G. (1990): B-Spline Contour Representation and Symmetry Detection. In G. Goos, J. Hartmanis (Ed.): Computer Vision- ECCV '90, Lecture Notes in Computer Science, Vol. 427. Berlin: Springer, pp 604-606. Seitz, P.; Lang, G.K.; Giiliard, B.; Pandazis, J.C. (1991): The robust recognition of traffic signs from a moving car. In Radig (Ed.): Mustererkennung 1991, 13. DAGM-Symposium, Informatik-Fachberichte 290, Mtinchen, Springer, pp 287-294. Schmid, M.; Thomanek, F. (1992): Real-time Detection and Recognition of Vehicles for an Autonomous Guidance and Control System. Pattern Recognition and Image Analysis, Vol. 3, pp 337-38O. Solder, U. (1992): Echtzeitfa,hige Entdeckung von Objekten in der weiten Vorausschau eines StraBenfahrzeugs. Dissertation, Fakultfit fur Luft- und Raumfahrttechnik der Universit~it der Bundeswehr Munchen. Solder, U.; Graefe, V. (1993): Visual Detection of Distant Objects. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS '93. Yokohama, pp 1042-1049. Thomanek, F.; Dickmanns, D. (1992): Obstacle Detection, Tracking and State Estimation for Autonomous Road Vehicle Guidance. Proc. 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS '92. Raleigh, pp 1399-1406. Tsinas, L., Graefe, V. (1992): Automatic Recognition of Lanes for Highway Driving. IFAC Conference on Motion Control for Intelligent Automation. Perugia, pp 295-300.
86 Wershofen, K. P. (1992): Real-time Road Scene Classification Based on a Multiple-lane Tracker. Proceedings, International Conference on Industrial Electronics, Control, Instrumentation and Automation (IECON '93). San Diego, pp 746-751. Young, E.; Tribe, R.; Conlong, R. (1992): Improved Obstacle Detection by Sensor Fusion. IEE Colloquium on "Prometheus and Drive", Savoy Place.
Young, G.-S.; Hong T.-H.; Herman, M.; Yang, J.C.S. (1992): Obstacle Detection for a Vehicle Using Optical Flow. Proc. IEEE Intelligent Vehicles '92 Symposium. Detroit, pp 24-29. Yuen, Sh.-Y.K. (1990): Shape from Contour Using Symmetries. In G. Goos und J. Hartmanis (Ed.): Computer Vision- ECCV '90, Lecture Notes in Computer Science, Vol. 427. Berlin: Springer. pp 437-453. Zieike, T.; Brauckmann, M.; v. Seelen, W. (1992): Intensity and Edge-Based Symmetry Detection Applied to Car-Following In G. Sandini (Ed.): Computer Vision- ECCV '92, Lecture Notes in Computer Science, Vol. 558. Berlin: Springer, pp 865-873.
Intelligent Robots and Systems V. Graefe (Editor) 1995 Elsevier Science B.V.
87
Visual Collision A v o i d a n c e b y S e g m e n t a t i o n Ian Horswill* *MIT Artificial Intelligence Laboratory, Cambridge, MA 02139 USA Visual collision avoidance involves two difficult subproblems: obstacle recognition and depth measurement. We present a class of algorithms that use particularly simple methods for each subproblem and derive a set of sufficient conditions for their proper functioning based on a set of idealizations. We then discuss and compare different implementations of the approach and discuss their performance. Finally, we experimentally validate the idealizations.
1
Introduction
One of the major difficulties of visual obstacle avoidance is that obstacles can seemingly have arbitrary appearance. They can be tall or fat, plain or textured, rigid or flexible. Different algorithms can be seen as embodying different assumptions about the appearance of obstacles. Moravec's Stanford Cart [11] can be seen as assuming that obstacles (1)project significantly from the ground plane and (2) are densely covered with visual corners. Kriegman, Triendl and Binford's Mobi system [9] can be seen as assuming that obstacles (1) have at least a certain height and (2) have vertical edges around their perimeters. The stereo algorithm of Storjohann et al. [13] is similar, but does not require the presence of vertical edges, only some sort of texture. The motion-based systems of Coombs and Roberts [2] and of Sandini et al. [ 12] also make these assumptions. Each system is compatible with a particular class of obstacles and incompatible with another. For example, none of these systems would know to avoid an oil slick, a patch of ice, or a delicate cable lying on the floor, because none of these objects would project sufficiently from the floor to be geometrically distinguishable from the floor. The other major difficulty with visual collision avoidance is depth recovery. Both the Stanford Cart and Mobi use stereo triangulation to determine position. Triangulation involves computationally expensive correspondence calculations and generally requires the use of well-calibrated cameras. In this paper, we will discuss a class of algorithms based on assumptions about the appearance of the background. Simply put, the method is as follows: take an image, use some criterion to discard pixels that look like the floor, and avoid driving toward the remaining pixels. Our current implementations are designed to operate in environments with textureless carpeted floors. In such environments, the presence of an edge is a cue to the presence of an obstacle, so the robot need only steer to avoid image edges. The algorithm uses 2D image position to determine obstacle position: higher edges are assumed to be farther away. The algorithm has several advantages: 9 All computations can be performed on 2D rasters. 9 The algorithm requires very little computation. 9 The algorithm is very conservative, preferring false positives to false negatives. We have implemented the algorithm on five different robots here at MIT. The algorithm has also been implemented and tested in other labs [5][ 14]. The algorithm is quite robust and has seen hundreds of hours of testing in many different environments.
88
Figure 1: Polly (left), the first generation vision-based robot, Frankie (center), the second generation, and Elvis (right), the third generation. The next section describes the edge-based algorithm. Section 3 shows that the algortihm's behavior is equivalent to the behavior of a system with a perfect depth map, provided that a set of assumptions about the environment and the robot dynamics are valid. Section 4 gives the details of several implementations done here at MIT. Section 5 discusses their performance and failure modes and concludes that the assumptions about environment and vehicle dynamics are accurate enough for the purposes of the analysis. Finally, section 6 provides conclusions.
2
Algorithm
We assume a forward-looking camera. Let I(t) be the image at time t and let E(.) be some edge detection operator. We define the bottom projection ("bprojection") of a binary image J to be a vector b ( J ) , indexed by the x (horizontal) coordinate, whose x'th element is the height of the lowest marked pixel in the x'th column of J:
b~(J) = min{y : J ( x , y ) =
1}
It will be shown below that under the right conditions, b ( E ( I ) ) is a radial depth map: a mapping from direction to the amount of freespace in that direction. Given the radial depth map b(E(I(t)))), we define the left, right, and center freespaces to be the distances to the closest objects to the left, right and center:
l(t)
=
r(t)
-
c(t)
=
min
b~(E(I(t)))
min
b~(E(I(t))) b~(E(l(t)))
~<xe
x~>~c
min
(1)
where xc is the x coordinate of the center column of the image, and w is a width parameter which in our implementations has been 1/4 the width of the image. The exact choices of the ranges over which the minima are taken is not important. Finally, we define a simple reactive control system for a holonomic base:
dO dt (t)
v(t)
-
co ( l ( t )
r(t))
(2)
=
c. (c(t) - dm,~)
(3)
-
89 where (dO/dt)(t) is the rotational velocity of the robot, v(t) is its translational velocity, d,~n is the closest that the robot should ever come to an obstacle, and co, c~ are user-defined gains. The algorithm turns in the direction of greater freespace at a rate proportional to the difference in freespace on the two sides. It drives forward if the distance to the closest object ahead is greater than d~,~, otherwise it drives backward. Of course, real systems will implement velocity caps (maximum velocities), which introduce min and max terms to the definitions above. We have left these out for simplicity.
3
Analysis
The discussion above leaves open a number of questions: in what environments will this algorithm work? What types of edge detectors can be used for N? Do the gains matter? In this section, we will examine these questions by comparing the algorithm to one which uses the same velocity equations (eq. 2), but which computes the distances l(t), T(t) and c(t) using a perfect oracle. 1 Space precludes a formal presentation in full detail. The interested reader is referred to the proofs of lemma 3 and theorem 2 in [7]. We can represent the perfect-information system schematically as follows: image
oracle
l, r, c
velocity eqs
-d-~' v
An image is fed to an oracle that somehow determines for each direction the distance to the closest object in that direction. The l, r and c values are computed from the resulting depth map and these values are passed to the velocity equations (eq. 2), which control the motors. Suppose that we were to warp the oracle's distance estimates by some strictly increasing function f: q
oracle ~ - ~
/, v, c H velocity eqs b
Because f is strictly increasing, this transformation will preserve the signs of both dO/dr and v. If we model the rotation and translation subsystems as first-order, zero delay systems, and if we can treat them as approximately separable (see section 5), then each subsystem will always converge to the same angle/distance, regardless of our choice of f. Only the rate of convergence will be affected. To say it more abstractly, if we treat the robot-environment combination as a dynamical system (whose state space is the configuration space), then the dynamical system will have the same attractors, repellors, and fixed points, regardless of the choice of f. Thus, the qualitative behavior of the system is invariant with respect to the introduction of a strictly increasing function. The practical import of this is that that we do not need to use a calibrated depth m e a s u r e ~ a n y monotonic measure will do, so long as we adjust d ~ n accordingly. Furthermore, the system should be robust in the presence of variations in sensor parameters. An oracle is a theoretical device generally used to prove that some function is not Turing computable. It is ,assumed to correctly compute the value of some function whether or not that function is formally computable. Oracles ,arenot intended to be physically realizable. We use the term here simply to mean a system that somehow correctly computes the right answer.
90 Now suppose we have an oracle that can identify obstacle pixels in the image: its input is an image, and its output is a binary image whose pixels are 1 iff the corresponding pixels in the input were projected from an obstacle. Call this the "figure/ground" image. It has been known at least since Euclid (see Cutting [3]) that for points projected from a ground plane, image plane height is a strictly increasing function of distance. This result can be extended to show ([7], theorem 2) that in the right environments, the bottom projection of the figure/ground image is equal to the radial depth map, modulo a strictly increasing function. Therefore we can substitute the figure/ground oracle for the distance oracle:
q F/Goracle H bproj H l,r,c H vel. eqs ground-planeconstraint
without altering the behavior of the system, provided that the (GPC) is true of the robot's environment. The GPC requires that all obstacles rest on the ground plane and that their projections be contained within the interiors of their regions of contact with the ground plane (see [7]). To complete the transformation of the distance oracle algorithm into the section 2 algorithm, we need only replace the figure/ground oracle with an edge detector. We would like to justify this substitution and to know what constraints there are on the choice of edge detector. Suppose the surface markings of the ground plane (e.g. the carpet) consist of a high frequency component and a constant (DC) component. Let ta be the lowest spatial frequency in the high frequency component. If the image plane is parallel to the ground plane, looking down at the ground plane from a distance d, then the to component will appear on the image plane with frequency tat
__
tad
Y where f is the lens focal length. Increasing d or changing the camera orientation can only increase this frequency. If we apply a low-pass filter with cutoff w t to the image, only the DC component will be preserved. Since an edge detector is effectively a band-pass filter, it should not respond to regions of the image projected from the ground plane, provided that it's high cutoff is less than w t. Thus we can substitute the edge detector for the F/G oracle:
q edgedet. H bprojH l, r,c~ vel.eqs b backgroundtextureconstraint
provided that the (BTC) holds: that the environment is uniformly illuminated, that the ground plane's surface markings have no components less than w other than DC, and that the scene is viewed from a distance of at least d. 2 The derivation of our algorithm from the oracle algorithm tells us that we can expect them to behave equivalently provided that the GPC and BTC hold of the environment, and that the robot base can be approximated as two non-interacting first-order systems. We will return to these
2This substitution step is not strictly valid, since it assumes that the edge detector fires for all the object pixels. In real life, the edge detector will only fire for the outline of the object and part of its interior. To be strictly valid, the substitution step should replace the F/G oracle with an edge detector combined with a fill algorithm (the fill would mark the object pixels missed by the edge detector). However, it is easy to show that the bprojection is invariant w.r.t, the fill operation, so it can be safely omitted.
91
Robot Polly Gopher Frankie Wilt Elvis
Speed lrn/s 0.2m/s lm/s lm/s 2m/s
Processor TMS320C30 MC68332 MC68332 MC68332 TMS320C31
Frames/s 15 5-10 5-10 5-10 15
Cost $11,000 $700 $7OO $7OO $1,000
Power <40W <3W <3W <3W 12W
Weight <15kg 2kg 2kg 2kg 6kg
Table 1: MIT implementations of the algorithm. Cost, weight and power refer only to the vision system, not to the base or power supply. The frame rates of the 68332-based systems are variable because of varying CCD integration times. assumptions shortly. The derivation also formalizes the computational value of the individual assumptions: the ground plane constraint allows a particular formal simplification of the depth estimation problem, and the BTC allows a particular formal simplification of the figure/ground problem.
4 Implementations To date, we are aware of several implementations of the algorithm that are presently in use. Most have been done at MIT (see table 1), although we know of two others that have been implemented by outside labs [5][ 14].
Polly The first implementation was part of a vision-based mobile robot called Polly (Horswill [6][7]). Polly used vision to seek out visitors and give them primitive tours of our laboratory. Polly processed images at 15Hz using an on-board, off-the-shelf DSP card and frame grabber mounted on an RWI B 12 holonomic base. The complete robot was built for less than $20K US. Polly uses several different algorithms in parallel. The motor outputs of the collision collision avoidance algorithm discussed here are summed with the outputs of a simplified version of Bellutta's vanishing-point algorithm [ 1] for corridor following. All the MIT implementations, except for Elvis and Gopher, run at roughly lm/s. Higher-level navigation was performed by separate processes that attempt to recognize landmarks and choose paths. The details of these other algorithms are outside the scope of this paper. See [7] for further information.
Frankie and Wilt Frankie and Wilt are newer implementations based on new-generation hardware. Polly's major reliability problem was that its off-the-shelf hardware was not intended to run on batteries, nor was it intended to be knocked around and shaken on board a mobile robot. Frankie and Wilt, by contrast, use only a small 68020-class microcontroller board designed to run on batteries. The board is much more robust, both mechanically and electrically, than Polly's processor. Image acquisition is performed by directly interfacing a CCD image sensor to the processor bus by way of an A/D converter (see Gavin and Yamamoto [4], Horswill and Yamamoto [8]). The CCD is mounted on a pan/tilt head built from Futaba model airplane servo motors. The complete vision system is less than $700 US (parts cost). The collision avoidance algorithm was modified to take advantage of the pan/tilt head; Instead of using the center distance to control
92 the rate of translation, the center distance is used to tilt the head up or down so as to keep the nearest obstacle centered in the image. The translate velocity is then based on the tilt angle of the camera. This increases the effective field of view of the system and helps compensate for various mechanical deficiencies (see below). The Polly, Frankie and Wilt implementations work by subsampling the image to 64 x 48 pixels, and then smoothing the result with a 3 x 3 filter. Edge detection is done by computing the sum of the absolute differences of each pixel with its upper and right neighbors. If the sum is greater than a threshold (15 grey levels out of a possible 510 on Polly, 22 grey levels for Frankie and Wilt which have more camera noise). The Polly system is described in detail in [7]. The important parts of the code for the Frankie system are given in section 6. Elvis
The Frankie and Wilt implementations were more robust and were dramatically cheaper, however, their 1-2MIP processors severely limited the processing they could perform. Elvis, our third-generation robot is based on much higher-performance (50MFLOP) hardware that is still relatively inexpensive ($1,000 parts cost). Because Elvis can use a higher frame rate and has a lower center of gravity, it can run dramatically faster than the other implementations. We have reliably achieved speeds of up to 2m/s. The Elvis vision hardware, designed by Chris Barnhart, consists of a TMS320C31 floatingpoint DSP with 128KW of ram, a memory-mapped NTSC color frame grabber, and a 24 bit memory-mapped video display for debugging. We hope to use the color capabilities of the machine to simplify place recognition.
5
Evaluation
The algorithm has been thoroughly tested, having seen many hundreds of hours of service. Polly has given over 100 tours and has patrolled the lab for periods of up to two hours before exhausting its motor batteries. Wilt, while more limited in functionality, is light enough and robust enough to permit much greater testing~ It is reliable (and safe) enough that researchers regularly bring their children to play with it. The researchers often demo the system themselves, even though they did not design it and do not know how it works. Frankie and Wilt have also been demo'ed off site by researchers unfamiliar with their workings. The algorithm is dramatically more reliable than the sonar-based systems running in the lab, which are very sensitive to obstacle geometry. In all, the various implementations of the algorithm have been tested in a few dozen different environments: 9 Various floors of our lab (see below). 9 Various class rooms and auditoriums of MIT. 9 The AAAI-93 convention center, exhibition halls, and hotel. 9 The Brown University Computer Science Department. 9 West Point Military Academy 9 The New Hampshire offices of DEC 9 A local elementary school.
93
5.1
Environmental approximations
The analysis of section 3 showed that the behavior of our algorithm is equivalent to the behavior of a system with a perfect distance oracle, provided that the environment satisfied certain properties: that all obstacles rest on a ground plane and have a certain type of shape (the GPC), and that the ground plane was textureless and uniformly lit (the BTC). These are idealizations; real environments contain shadows, real carpets have stains. The usefulness of the algorithm is determined by the range of environments which satisfy these properties well enough to allow the system to work. The ground-plane constraint has turned out to be mostly non-problematic, even though many objects, such as chairs and tables, violate its shape restrictions. It is particularly unproblematic for Frankie because it tilts its head downward as it approaches an obstacle. In the limit, when the head is pointed straight down, the equivalence of the bprojection to a depth map can be shown to be independent of object shape. Thus the active head actually makes the system somewhat more robust. Frankie's biggest problem is that its head happens to be exactly at desktop level. As Frankie approaches a desk and looks downward to keep the desk legs in view, the desktop goes out of view and Frankie hits itself in the head. The background-texture constraint is more complicated. All floor surfaces in our building satisfy its surface marking requirements, but the basement and top floor both have shiny tile floors that reflect images of the overhead lights like mirrors. The robot treats these reflections as obstacles and tries to avoid them. Depending on room geometry, the robot may successfully navigate or may back away from the lights until it collides with something in the rear (Frankie can't see backward). Another problem occurs when a room has two different carpets. Even though each carpet satisfies the BTC, the boundary between them looks like an obstacle to the robot. Polly uses a separate algorithm to identify specific carpet boundaries and override the edge detector, but we hope to handle the situation more intelligently on Frankie. Shadows have not been a problem in our environment, although scenes which are illuminated both by sunlight and by room light often have illumination gradients strong enough to trigger the edge detector. Such scenes also cause the automatic gain control of the camera to fail. Although such experiments are purely anecdotal, we have also run the robots in a number of off-site environments. Frankie has been run successfully at a number of sites on campus: classrooms, the main auditorium, and the so-called "infinite corridor." One problem in auditorium environments is that the overhead lights are often point-sources which generate strong shadows. These can sometimes confuse the robot. Polly has been run successfully in those parts of the Brown University computer science building that were well enough lit for the video camera to operate. Frankie has also been taken off-site by other researchers to give demos. A smaller robot using an early version of the Frankie vision hardware was run on-site at last year's AAAI. The robot suffered many hardware problems and did not perform at all well in the office environment that had been carefully engineered for the sonar-based robots of the AAAI robot contest. However, it was the only robot that could function effectively in the real lounge area, the sessions, or the conference hotel.
94
5.2 Dynamicapproximations Our analysis assumed that the translation and rotation systems of the robot are independent and that they are both first order dynamic systems. In real life, the two systems are coupled: rotations in response to the l and d distances change the c distance, while translations in response to c can change the I and r. This coupling raises the possibility of oscillation. Fortunately, we have not experienced this problem in practice because the robot is almost always either blocked or in rotational equilibriummit tends either to translate or rotate, but rarely both. Thus the coupling between rotation and translation rarely shows itself. While most modern robot bases have good velocity control mechanisms, they can only generate finite accelerations, so the first-order approximation is also only partly accurate. Another limitation of our robots is that their centers of gravity are high enough for them to tip over when the robots stop suddenly. Deceleration limits cause an interesting problem: if the robot fails to detect a faint obstacle until it is nearby, the robot may not have time to stop fully before the obstacle goes out of view. Since the algorithm is purely reactive, it forgets the out of view obstacle and accelerates into it. Any reactive control system will have this problem, whether it is based on a first-order model or not, so the problem does not really lie with our analysis. On Frankie, the problem is greatly reduced because it can compensate with its pan/tilt head. The low mass head is not subject to the same acceleration restrictions as the body, so Frankie can always keep the object in view. Frankie is also drastically lighter and has a lower center of gravity, so its dynamic problems are less to begin with. Elvis, being still lighter and lower to the ground, can safely run at up to 2m/s and yet stop without falling.
6
Conclusions
We have presented a simple collision avoidance algorithm and an analysis of its behavior based on a set of assumptions regarding the environment and the robot's dynamics. We also presented various implementations of the algorithm and evaluated their performance and the empirical validity of the assumptions used in the analysis. Experiments show that a surprisingly wide range of environments satisfy the constraints closely enough to allow the algorithm to work effectively and that the real robot dynamics generally match the approximations used in the analysis. Thus the environment and dynamic models, while imperfect, are sufficient for the analysis. The algorithm is simple, conservative, extremely efficient, and quite reliable within our local environment. In our environment, it is far more reliable than the sonar-based methods. The algorithm belongs to a general class of algorithms that exploit the fact that it is often easier to recognize the background than to recognize an obstacle. Depending on the context, a robot might use any number of cues to distinguish the background from the obstacles. The algorithm presented here uses an edge detector, and therefore relies on an assumption about the surface markings of the background. One can easily imagine other algorithms using other cues (e.g. color or scale) in other contexts. To change the cue, one need only change the final substitution in the analysis given in section 3. For example, one could substitute a color-based identification system (e.g. Swain [ 15]) for the edge detector:
q
H
H ',',oH ve' b
95 if the BTC didn't hold of the environment but the floor could be relied on to have a predictable color. In theory, one could even use an adaptive system that dynamically switched between an array of cues, depending on local context, but this is more speculative. The reader may object that this algorithm is just a particular hack for a particular environment. In a sense, this is true. However, it is a hack which has been demonstrated to operate in a larger number of environments than any other vision-based robot of which we are aware. In addition, nearly any vision system will make some set of assumptions about its environment (see Marr [ 10] for the canonical discussion of this point). A stereo or motion system, for example, requires the presence of texture not only on the obstacles, but on the floors and walls also, at least if it is to make accurate depth readings there. This property simply isn't true of our laboratory. We suggest that the issue is not so much whether we should specialize our systems to tasks and environmentsmsince nearly all systems are specialized--but what specializations are most useful.
Acknowledgments Support for this research was provided in part by the Advanced Research Projects Agency under Office of Naval Research contract N00014-91-J-4038 and in part by the NASA Jet Propulsion Laboratory under contract 959333. Additional support was provided by Matsushita Electric Industrial Co., Ltd.
Appendix: code excerpts from Frankie Below is most of the important code from the Frankie system. Some variable names have been changed to be more consistent with the text. Device drivers and obvious functions, such as a b s have been omitted. The code is written in C and makes extensive use of pointer arithmetic to speed up the inner loops of the vision code. Note that only 1D iterations are performed over the images, rather than 2D iterations. This makes the code somewhat faster and simpler, but does produce erroneous results at the edges of the image. The rest of the code is written to ignore the values it finds there.
#define image_width 64 #define image_height 48 #define image size (image_width*image height)
* ipointer is the type of a pointer into an image. * pointer() casts an image into a pointer. */ typedef unsigned char image [image_height] [image_width ] ; typedef unsigned char *ipointer; #define pointer(a) ((ipointer) (a))
96
/* I t e r a t i o n m a c r o s */ # d e f i n e c o u n t d o w n ( v a t , count) \ for (var : (count)-l; v a r > = 0 ; var--) # d e f i n e d o t i m e s ( v a r , count) \ for (var = 0; v a r < ( c o u n t ) ; var++) /* P o i n t e r a r i t h m e t i c for i m a g e p o i n t e r s */ #define shift_left(i,pix) ((i)-(pix)) #define shift right(i,pix) ((i)+(pix)) #define shift_down(i,pix) ((i)+image_width*(pix)) #define shift_up(i,pix) ((i)-image_width*(pix)) /* T h i s is a 3x3 s e p a r a b l e l o w - p a s s void smooth(image in, i m a g e out)
filter
*/
{ image scratch; i p o i n t e r left, int c o u n t e r ;
middle,
right,
up,
down,
i,
i = pointer(in) ; s = p o i n t e r (scratch) ; o = s h i f t _ r i g h t (s, i) ; left = i ; middle = shift right(i,l) ; right = shift right(i,2) ; countdown(counter, image_size-2) *o++ = ( * l e f t + + + * r i g h t + + + ( * m i d d l e + + >> 2;
o,
<
o = s h i f t _ d o w n ( p o i n t e r (out) , i) ; up = s; middle = shift down(s,l) ; down - shift down(s,2) ; countdown(counter, image size-(image_width*2)) *o++ : ( ( i n t ) * u p + + + ( i n t ) * d o w n + + + ( ( i n t ) * m i d d l e + + << I)) >> 2;
void
edge_detect
(image in, i m a g e out, int e d g e t h r e s h o l d )
{ i p o i n t e r up, int c o u n t e r ,
left, center, c, dx, dy;
i ,o;
s;
97
center : i : pointer(in) ; o : p o i n t e r (out) ; up : s h i f t _ u p (i, i) ; l e f t = s h i f t l e f t ( i , i) ; countdown(counter, i m a g e size) c = *center++; d x = (int) ( * l e f t + + ) -c; dy = (int) (*up++) -c; *o++((abs(dx) + abs(dy))> ?255-0;
#define
thresh
edge_threshold)
22
/* Do all t h e v i s i o n to and center distances v o i d do v i s i o n ( )
compute */
left,
right,
{ grab frame(framebuf) ; s m o o t h ( f r a m e b u f , bufl) 9 e d g e _ d e t e c t (bufl, b u f 2 , t h r e s h ) ; b p r o j e c t i o n (buf2, d e p t h _ m a p ) ; /* c o m p u t e l, r, a n d c d i s t a n c e s */ left=region_min ( d e p t h _ m a p , 4,25) ; r i g h t = r e g i o n _ r a i n ( d e p t h _ m a p , 35,25) ; center=region_min ( d e p t h _ m a p , 20,24) ;
int
pan,
/*
tilt-
T i l t t h e c a m e r a so as obstacle */ void cameracontroller()
to
center
the
nearest
{ int int
vertical_error = center - desired_height; control_signal = min(max(-(vertical_error>>l),-l),5);
/* t i l t t h e c a m e r a u p / d o w n to c a n c e l o u t control_signal *,I tilt camera relative( control signal) ;
98
/* R o t a t e the b a s e to b a l a n c e the free s p a c e on the left a n d right. D r i v e f o r w a r d / b a c k to k e e p the n e a r e s t o b s t a c l e at a c o n s t a n t d i s t a n c e */ void
base
controller()
{ int int int if
rotational_error = right-left; translational_error = center - desired_height; tilt_error = 20-current camera_tilt();
(current c a m e r a _ t i l t ( ) = = b o t t o m _ l i m i t ) /* s p e e d up turns w h e n v e r y c l o s e to o b s t a c l e this m a k e s it r e s p o n d f a s t e r w h e n it r e a c h e s a d e a d end */ rotational error = (rotational_error>0) ?
4000- (-4000); set_rotate_velocity
(rotational_error*200)
/* If too c l o s e to object, m o v e b a c k full this p r o b a b l y isn't n e e d e d n o w that we m o v e the h e a d */ if (center< d e s i r e d h e i g h t -i0) set_translate_velocity(-1000); else set_translate velocity(tilt error*100);
/* D r i v e r main()
loop
*/
{ w h i l e (I) { dovi s ion ( ) ; c a m e r a _ c o n t r o l l e r () ; base controller() ;
}
; speed can
99
References [ 1] R Bellutta, G. Collini, A. Verri, and V. Torre. Navigation by tracking vanishing points. In AAM Spring Symposium on Robot Navigation, pages 6-10, Stanford University, March 1989. AAAI. [2] David Coombs and Karen Roberts. "bee-bot": using peripheral optical flow to avoid obstacles. In Proc. of the SPIE Conf. on Intelligent Robots and Computer Vision XI: Algorithms, Techniques, and Active Vision, (Boston, MA, November 15-20, 1992), 1992. [3] James E. Cutting. Perception with an Eye for Motion. MIT Press, 1986. [4] Andrew S. Gavin and Masaki Yamamoto. A fast, cheap, and easy system for outside vision on mars. In Intelligent Robots and Computer Vision XI, Cambridge, MA, September 1993. SPIE. [5] Takashi Gomi and Koh ichi Ide. Vision based navigation for an office messenger robot. In this volume, 1995. [6] Ian Horswill. Polly: A vision-based artificial agent. In Proceedings of the Eleventh National Conference on Artificial Intelligence, pages 824-829. AAAI, MIT Press, 1993. [7] Ian Horswill. Specialization of perceptual processes. PhD thesis, Massachusetts Institute of Technology, Cambridge, May 1993.
[8] Ian Horswill and Masaki Yamamoto. A $1000 active stereo vision system. In Submitted to CVPR-94, 1994. [9] David J. Kriegman, Ernst Triendl, and Tomas O. Binford. A mobile robot: Sensing, planning and locomotion. In 1987 IEEE lnternation Conference on Robotics and Automation, pages 402--408. IEEE, March 87. [ 10] David Marr. Vision. W. H. Freeman and Co., 1982. [ 11] Hans R Moravec. The stanford cart and cmu rover. Technical report, Robotics Institute, Carnegie-Mellon University, February 1983. [ 12] G. Sandini, E Gandolfo, E. Grosso, and M. Tistarelli. Vision during action. In Yiannis Aloimonos, editor, Active Perception, chapter 4, pages 151-190. Lawrence Erlbaum Associates, Inc., 1993. [ 13] K. Storjohann, T. Zeilke, H. A. Mallot, and W. von Seelen. Visual obstacle detection for automatically guided vehicles. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 761-766, May 1990. [ 14] Michael Swain. Personal communication. [ 15] Michael J. Swain. Color indexing. Technical Report 390, University of Rochester Computer Science Department, November 1990.
This Page Intentionally Left Blank
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
101
Using Perceptions to Plan Incremental Adaptations A.J. Hendriks and D.M. Lyons Philips Laboratories, Philips Electronics North America Corporation, 345 Scarborough Road, BriarcliffManor, NY 10510, USA. [email protected], [email protected], com
Abstract
A robot system operating in an environment in which there is uncertainty and change needs to combine the ability to react with the ability to plan ahead. In a previous paper we proposed a solution to the problems of integrating planning and reaction: cast planning as adaptation of a reactive system. The planner, asynchronously tuning the reactor, decides on the appropriate parts of the reactor to be modified based on perceptions: information gathered in the reactor for the express purpose to inform the planner. In this paper, we show the benefits of using perception to adapt the reactor where it needs to be updated most and present our first experimental results from the planner-reactor architecture. 1.
INTRODUCTION
The importance of integrating deliberative ("planning") capabilities and reactive capabilities when building robust, 'real-world' robot systems is becoming widely accepted [1, 6, 13]. Purely deliberative approaches (e.g., [4]) are good at determining globally good courses of action for a robot, selecting an optimal assembly sequence for example. They are not very robust, however, in the face of changing or uncertain conditions in the robot's environment. Typically, a deliberative system expends a lot of effort deriving a good course of action for the robot, only to find on starting that the environment has changed and rendered the actions unsuitable [13]. Purely reactive approaches (e.g., [2, 7]) operate very robustly in some uncertain and dynamic e n v i r o n m e n t s - namely those for which the reactions have been manually p r e - p r o g r a m m e d - but lack any ability to plan ahead. However, many application domains such as mobile robots [6], autonomous explorers [1], emergency response planning [13] as well as our domain, robotic kitting, demand the integration of deliberative and reactive capabilities. In [11] we suggested a straightforward approach to this integration: allow the deliberative component to incrementally adapt the reactive component to suit
102 current goals and operating conditions. In subsequent papers, we looked at the theoretical issues of incremental adaptation [9] and of safely making changes to an operating reactive system [10]. Here we present our first performance results, using our approach to construct a robot kitting work cell. Some alternative approaches to this integration have been suggested. Schoppers[14] amongst others has proposed an 'off-line' generator for reactive systems; Xiaodong and Bekey [16] also suggest something similar for an assembly cell equipped with a reactive scheduler. This approach works well only when the deliberative component can be separated 'off-line'. We address the problems that arise when both reactive and deliberative components need to be 'on-line'. Connell's SSS [3] addresses this integration for mobile robots. In SSS, though, the deliberative component is restricted to enabling/disabling behaviors, while we need to be able also to generate new behaviors. Bresina & Drummond's ERE [1] comes closest to our approach. A key difference is that our domain, kitting, is a repetitive activity. Our concept of the improvement of a reactive system exploits this repetition, whereas ERE was designed for once-off activities. We begin with a summary of the planner-reactor architecture we developed to implement of our approach. Section 3 describes the kitting robot domain. Sections 4 and 5 provide more details of our implementation as a prelude to our experimental results in Section 6.
Adaptations
PLANNER
Actions
REACTOR
WORLD
J Perceptions
Sensing
Figure 1: Planner-Reactor-World System
2.
THE P L A N N E R - R E A C T O R A R C H I T E C T U R E
Our definition of planner is a system that continually modifies a concurrent and separate reactive system (the reactor) so that its behavior becomes more goaldirected. Figure 1 illustrates this architecture. The reactor contains a network of reactions: hardwired compositions of sensor and motor processes. In our framework, these are sensory processes composed with effector (or action) processes in such a fashion that should the sensory process detect its triggering condition, then it will in turn trigger the effector process. The reactor has two key properties:
103 1.
2.
It can produce action at any time. Unlike a plan executor, a reactor can act independently of the planner; it is always actively inspecting the world, and will act should one of its reactions be triggered. A reactor should produce useful behavior even without a planner~ It is a real-time system. The central objective of the reactor is to produce the appropriate action at the appropriate time. To make this possible, all the 'knowledge' in a reactor is encoded into reactions and cross-connections between them, and annotated with time constraints.
The reactor encapsulates what would typically be called the plan in more traditional approaches. By making it a completely separate system, we remove the delays involved when a general deliberation subsystem is always interposed between sensing and action. The reactor is also responsible for the default behavior of the system in the following sense: It is initialized with a number of generally useful behaviors already in place, for example, distinguishing objects from background, locally avoiding obstacles, and scanning its sensors in the environment. Unlike a plan executor or a hierarchical system of planner and reactive system, our reactor acts asynchronously and independently of the planner. It is always actively inspecting the world, and will act should one of its reactions be triggered. A reactor should produce timely, useful behavior even without a planner. Rather than viewing the planner as a higher-level system that loads plans into an executor, we see the planner as an equal-level system that continually tunes the reactor to produce appropriate behavior. Interaction between the planner and reactor is entirely asynchronous. The planner continually determines if the reactor's responses to the current environment would indeed conform to the given objectives. If not, then the planner makes an incremental change to the reactor configuration to suit the reactor's behavior better to the objectives. As indicated in figure 1, there are two routes of interaction between planner and reactor. 1. 2.
Adaptations These are structural alteration commands issued by the planner causing changes in the reactor structure. Perceptions These are sensory data collected by the reactor to be sent to the planner.
An adaptation is essentially an instruction to delete part of the reactor structure or to add in some extra structure. Each individual adaptation should be small in effect and scope; large changes in reactor behavior only come about as the result of iterated adaptation. Large adaptations would be equivalent to downloading 'plans' to the reactor; something we need to avoid. A planner can only adapt a reactor in a well-defined way, as is explained shortly. This allows a formal analysis of the adaptation mechanism. To distinguish between reactor and planner sensory data, we refer to any data collected by the reactor as sensory data, but sensory data collected to be sent to the planner will be referred to as perceptual data. In the remainder of this section, we briefly reiterate the details of the planner-reactor approach as presented in our previous work[9, 10]. We begin by introducing the formal language ~U we use to describe the reactor and its incremental adaptation.
104
2.1. The T5 Model The ~5 model [8] was developed to r e p r e s e n t and analyze the kind of prog r a m s involved in sensory-based robotics. This language allows a user to construct programs by 'gluing' together processes in various ways. This style of language is called a process-algebra language. These languages have the additional a d v a n t a g e t h a t they facilitate the m a t h e m a t i c a l analysis of programs. A description of a process, or n e t w o r k of processes, is called a schema. For example, L o c a t e m (X) denotes a process t h a t is an instance of the schema L o c a t e with one i n p u t p a r a m e t e r m and one result x. Networks are built by composing processes together using several kinds of process composition operators. This allows processes to be ordered in various ways, including concurrent, conditional and iterative orderings. At the bottom of this hierarchy, every n e t w o r k m u s t be composed from a set of atomic, pre-defined processes. The composition operations are the following:
1. 2. 3. 4. 5.
sequential concurrent conditional negation disabling
A; B,
do A t h e n B.
(A I B), A: B,
do A and B concurrently.
~A,
do A but fail if A succeeds and vice-versa.
A#B,
do B only if A succeeds. do A and B concurrently until one terminates.
We use two r e c u r r e n t operators:
6. 7.
synchronous recurrent A: ; B =A: (B; {A: ; B) ), while A succeeds do B. asynchronous recurrent A: : B =A: (B I (A: 9B) ), while A succeeds s p a w n off B.
It is i m p o r t a n t to represent how process networks evolve over time, as processes dynamically t e r m i n a t e or are created. This is captured by the evolves operator: We say t h a t process A evolves into process B u n d e r condition ~ if A can possibly become equal to B w h e n condition a occurs; we write this as A -~ B. A process must evolve to one of the set of processes to which it can possibly evolve.
2.2. I n c r e m e n t a l R e a c t o r A d a p t a t i o n We define the reactor R as a set of concurrent reactions, and a well-defined interface t h r o u g h which the p l a n n e r can modify the reactor structure:
REACTOR
=
(EI#pI I ......I En#pn
I (ADD(k): ~ (Ek#p k) )
(1 )
w h e r e there are three m a i n items: 1.
pi is a reaction or reactor segment.
2.
E i is a g u a r d process by which the p l a n n e r can remove pi from the reactor.
3.
ADD is the interface t h r o u g h which the p l a n n e r (by the definition of : :) can add new guarded reactions.
A reactor does not contain/execute plans in the same sense t h a t a classical plan executor does, i.e., as a d a t a structure. The reactions are represented procedurally, so the reactor is more like a compiled controller. Nonetheless, certain
105
p a r t s of the reactor s t r u c t u r e can be identified as corresponding to the instructions necessary for a specific t a s k or subtask; we call these parts reactor segments. It is possible to write down how the reactor is affected by the changes described in items 2 and 3 of eq. 1 above using the T5 evolves operator: ( R
Dk
[ Ek#p k )
--> Ak -->
R
where Dk is where A k is
R ( a
(2)
I Ek#p k )
STOP, and
gk --+ ADD(k> --->
STOP.
Saying t h a t a process P successfully t e r m i n a t e s (under some condition) is the s a m e as saying t h a t P evolves to STOP (under t h a t condition). So in eq. 2 above, Dk can be used to delete reactions, and A k can be used to add reactions. Using the evolves operation, we can express the process of reactor a d a p t a t i o n as follows: R
~
(3)
R'
w h e r e a' is the a d a p t e d reactor and a is some combination of A k and Dk conditions. In [10] we describe how to m a k e these a d a p t a t i o n s in a safe m a n n e r despite the fact t h a t the p l a n n e r does not know w h e n or which reaction m i g h t be firing. Ideal React.or. The u l t i m a t e objective of the p l a n n e r is the construction of a reactor t h a t achieves its goals not only in the current environment, but because change is possible, in all e n v i r o n m e n t s t h a t m i g h t occur, as indicated by the e n v i r o n m e n t model. A reactor t h a t fulfills this criterion is called an ideal reactor, and is similar in behavior to Schoppers' universal plans[14]. It is unrealistic to suppose t h a t a p l a n n e r will always be able to generate the ideal reactor in one step, and in [9] we introduced the more restricted m-ideal reactor am. In p l a n n e r terms, m is a set of assumptions u n d e r which the reactor R~ has been constructed. We propose the following i n c r e m e n t a l reactor construction strategy: m is initially chosen to allow the p l a n n e r to quickly produce a goal achieving reactor; t h e n co is g r a d u a l l y relaxed over time. The evolution in eq. 3 can be used to r e p r e s e n t this s t r a t e g y as follows: R m1
for
~
e(col) >
Rm2
e(co2)
a2 --+ >
a--+ n
... ....
>
R*
(4)
e(O)
where the ai is an a d a p t a t i o n c o m m a n d g e n e r a t e d by the p l a n n e r and e0 is an evaluation function. The a s s u m p t i o n relaxation priorities are governed by the ordering on o~ dictated by e0. The conditions t h a t trigger the p l a n n e r to produce an improved reactor are described in [9] and the constraints on convergence to a* overviewed.
106 3.
THE KITTING ROBOT PROBLEM
Assembly components are fed from stores to a kitting station on a conveyor belt. The kitting station consists of one or more kitting robots. Each kitting robot must take parts off the belt, place them in the appropriate slots of a kitting tray, and then place the tray onto the belt. The trays are stored in a stack in the robot's work space. The product we are kitting is a small DC servomotor manufactured by Philips. It has three parts: a cap, a motor and a body, all of which have a number of variants. The planner-reactor approach provides a way to incrementally construct reactive systems with improving performance. At the heart of this iterative mechanism is the concept of characterizing the environment by a set of assumptions and methods to detect whether they hold or not. Kitting Assumptions. The set of assumptions we have developed for the kitting environment is described below in order of likelihood of not holding: the ordering we have chosen for the planner to relax them (i.e., the e0 function in eq. 4). The planner uses these assumptions to help it to quickly build a reasonable reactor. Later, as it has time, it relaxes assumptions and fashions improved versions of the reactor. When the environment goes through regimes where different subsets of assumptions hold, the planner can capitalize on this, especially when new factory goals have been received and resultant changes in behavior are necessary. Some of the assumptions, such as that of the quality and substitutability of parts, must eventually be relaxed to get to a robust work cell. Some other assumptions, such as that of cooperation in kit assembly and tray disturbance, describe contingencies that make a more versatile work cell, but are not necessary for a robust system. Still other assumptions pertain to unusual operating conditions, such as the assumptions of no downstream disturbances. The kitting robot problem uses nine assumptions in total. In our experiments, however, only the following five were relaxed: 1. 2. 3. 4. 5.
Assumption of Parts Quality (AQ): All the parts coming into the kitting work cell are of good quality and do not need to be tested. Assumption of non-substitutability of parts (AS): Each part has only one type. Assumption of no parts motion (AM): The kit parts do not move around once delivered on the belt to the work cell. Assumption of no downstream disturbance (ADD): Downstream automation is always ready to receive finished kits. Assumption of parts ordering (AO): The robot cannot modify the preplanned parts ordering.
In addition to the iterative relaxation of assumptions based on likelihood, the planner can also be forced to relax assumptions. Whenever the planner builds a reactor that depends upon a particular assumption holding, it needs to embed a monitor process that can detect whether the assumption actually holds in practice or not. The monitor process triggers a 'clean up' action if the assumption ever fails, and notifies the planner that it needs to relax this assumption next.
107 4.
SITUATIONS
Reactor situations are a m e c h a n i s m to group related reactions together in a hierarchical fashion. This m o d u l a r i t y is i m p o r t a n t because it allows (human) designers to more easily u n d e r s t a n d the reactor and diagnose any problems t h a t m a y occur; it provides the p l a n n e r with a unit around which to define safe changes to the reactor s t r u c t u r e [10] and it provides a way to give computational resources to those reactions t h a t currently need it, while 'suspending' others. Intuitively: a situation being active expresses the a p p r o p r i a t e n e s s for the reactor to enable the set of reactions associated with t h a t situation. Situations can be nested hierarchically, and m a n y situations m a y be active at one time so t h a t the execution of their reactions will be interleaved. We introduce a small set of basic processes with which to build reactor situations in T5 (table 1). For example, an instance of a situation is asserted by the execution of the A S S E R T s p l , ~ 2 .... process, w h e r e p l, p 2 .... are the values for the p a r a m e t e r s associated wit~ the situation. The SIT s ( k , p l , . . . ) process, w h e n executed, suspends itself until an instance of situation s is asserted. It t h e n t e r m i n a t e s and passes on the details of the situation instance as its results.
* ASSERTS,p Assert situation s and initialize the situation p a r a m e t e r s p,. . . . . . This t e r m i n a t e s w h e n the situation is t e r m i n a t e d and stops or aborts depending on w h e t h e r FAIL or SUCCEED was used. * FAILS, k T e r m i n a t e s instance k of situation s with fail status. * SUCCEEDs, k T e r m i n a t e s instance k of situation s with success status. * SIT s ( k , p l , p 2 , . . . ) T e r m i n a t e s if an instance of situation s is currently asserted. k is the instance n u m b e r and p l , p2, ... are the current values of the situation parameters. Table 1: Situation basic processes We use the S I T s process to ensure t h a t a reaction associated with a situation is only 'enabled' w h e n an instance of t h a t situation is asserted. For example, let p1, . . . . , p n be the reactions for situation s, t h e n we would r e p r e s e n t this in the reactor as follows:
PS
=
SITs:;pI I SITs:;P2 I ""I SITs:;Pn
(5)
The ':;' operation ensures t h a t as long as the situation is active, t h e n the reactions are continually re-enabled. Note t h a t once enabled, a reaction cannot be disabled until it has t e r m i n a t e d . The asserting and t e r m i n a t i o n of instances of situation s h a p p e n as the side effects of ASSERTs and FAIL s or SUCCEEDs processes in reactions in other situations.
108
5.
THE PLANNER
The planner needs to adapt the reactor to suit new goals and/or changing operating conditions. New goals may be issued by factory management. Through feedback from perceptions t h a t the planner has inserted in the reactor the planner is made aware of the current operating conditions, e.g. the current set of assumptions t h a t hold in the environment. Based on these two sources of information, the planner reasons to arrive at an incremental set of adaptations, such t h a t the reactor's behavior becomes ever more goalworthy. This section outlines the principles of operation of the planner with emphasis on its interactions with the reactor, a detailed description of the planner is given in [9].
5.1. The P l a n n e r Algorithm At every planner iteration t, based on the environment model TM~ and current goals Gt, the planner generates what we call an expectation T_t:an abstract description of the changes it expects to make to the reactor to achieve Gt. At every iteration, the combination of the reactor model and the expectation is exactly the co-ideal reactor (where co is the current set of assumptions the planner is working with).The process of incremental improvement of the reactor can now be seen as incremental reduction of the planner's expectation by some ATt and corresponding increment of the reactor ARt, until when Tt is null, the current reactor is exactly the ideal reactor. Additionally, when assumptions are used to construct the reactor increment, the planner will insert assumption monitors AIt into the reactor. Similarly, if the planner needs specific information for its own planning purposes, it can insert additional perception processes APt, processes t h a t collect information in the reactor and report back to the planner. Goals G
T,
r I I I I I Z~
t
I
i i i
current reactor
Rt
AIr ~
perception monitors
SI
t
I I I I
Pt
=1
Wt
PS i
PLANNER
',
Fit
REACTOR
.i
Figure 2: Planner-Reactor Interactions
WORLD
12
109 The input/output interactions between planner and reactor are as shown in Figure 2. The planner incrementally produces changes in the reactor structure ARt plus the associated assumption monitors AI t and additional perception processes APt. It receives back information from existing perception process vP t and any signals from assumption monitors that have failed FI t .
Figure 3: The Planner Architecture
5.2. The p l a n n e r architecture. The block diagram of the planner's internals is shown in figure 3. Based on the environment model TM and the goals G,, the planner accumulates a current Problem-Solving Context (PSC). Within that PSC, the planner generates an abstract plan, the expectation Tt. The current set of assumptions ~ s ~ is used to filter the environment model to generate a simpler model in the PSC. The current expectation Tt is analyzed to determine how much of it can be reduced A~. This incremental expectation is decomposed (in the planning sense) in the context of this filtered world, to generate an adaptation of the reactor ARt and the set of assumption monitoring processes to go along with it AIt. A set of perception requests can also be generated at this point APt; these requests will return data that clarify uncertainty in T M t and allow further decomposition of ~. The assumption monitor processes may signal that the assumptions they protect have been violated; this results in ~ 5 ~ being decremented by that assumption, and it may thus result in being revised and probably increased again.
110
5.3. Expectation reduction. To reduce this expectation, the planner reasons within a Problem Solving Context consisting of the relevant parts of the environment model TM, the action repertoire A and the assumptions o~ The outcome will be a reactor adaptation ARt, the reactions necessary to implement the expectation reduction ATt. The manageable size of AT_t is constrained by the following factors: 1.
The response time of the planner. This can come from the design specification of the planner, or from the temporal constraints on goals and actions in the plan. 2. The effects of uncertain knowledge on the plan. The planner may need to issue perception requests before more of the expectation can be reduced. 3. Resource use and mappings. Initially, the planner reduces an expectation by looking for an abstract plan that achieves the given goals. This abstract plan, representing the structure of a more detailed plan, is maintained through insertion of situations into the reactor. The more detailed actions corresponding to the expansion of an abstract action are encoded as reactions in the situation, representing the abstract action. Any reactor segment sent to the reactor must be directly executable. However it need not necessarily be concrete, i.e., a reaction of a given situation may only contain an STUB process, a process that acts as a 'placeholder' and, when executed, only sends a perception back to the planner stating that this particular situation has become active. We call this perception signal a stub-trigger. In this way, the planner can adapt the reactor without needing to fully refine its plan first. Before inserting any adaptation into the reactor however, the planner proceeds to detail the first step in the abstract plan, such that for this first step a concrete plan is in place (i.e., no STUB) when the adaptation is sent. This avoids an immediate stub-trigger signal from the reactor for this segment. Instead it allows the reactor to start operating immediately, while the planner can proceed in parallel to flesh out other abstract parts of the plan. For the kitting robot domain, an initial abstract plan would be a sequential composition of the situations AcquireTray, FillTray, and RemoveTray. Only AcquireTray needs to be concrete in order for the planner to adapt the reactor. The other situations can have S'I~'B reflexes initially. The planner can refine these while the reactor acquires a tray. The s ~ m processes in the reactor provide the planner with much needed focus-of-attention information, i.e. the stub trigger signal tell the planner which part of the reactor configuration is in immediate need of refinement. Hence the planner can concentrate on refining the relevant parts of the reactor first, as indicated by the environment by a stub-trigger. The following process descriptions show a reactor segment generated by the planner for this initial adaptation. Each reaction in the segment below is in an ifthen-else form ( ( - C o n d : I f T r u e ) : E l s e ) . The REASSERT process continually reasserts failing situations. P0 is necessary to assert the topmost situation repeatedly. The process MOVeobj,loc repositions a grasped object obj to a location loc.
111 S T O P : ; ASSERTKitting
PO
=
P1
=
SITKitting(k >: ; (- ( (REASSERTAcquireTray (xl> : REASSERTFilITray, xl : REASSERTFinishTray ,xl ) " SUCCEEDKitting, k) : FAILKitting, k)
P2
=
SITAcquireTray(k >: ; ( - ( ( REASSERTFindPart (xl> : MOVExl ' trayarea : (UPDATEAcquireTray, k, xl; SUCCEEDAcquireTray, k ) ) : FAILAcquireTray, k )
P3
=
SITFindpart(k> : ; ( " ( ( Findtray(xl> : UPDATEFindPart, k, xl ; SUCCEEDFindPart, k ) ) : FAILFindPart, k )
P4
=
SITFillTrayt(k, x l } : ; (STUBFillTray: SUSPEND2000; FAILFiUTray, k)
P5
=
SITFinishTrayt(k , xl>. ; (STUBFinishTray : SUSPEND2000; FAILFinishTray, k)
Barring any perceptions received from the reactor, the planner continues this process, incrementally fleshing out abstract segments. At every iteration, the expectation is reduced by fleshing out a situation. The situation hierarchy is restricted to be an acyclic digraph, i.e., abstract actions used higher in the plan hierarchy cannot be used to construct a concrete reaction. If all the reactor segments are made concrete and all the STUB processes removed. Then, the planner has achieved an co-ideal reactor, and can proceed to relax the next assumption.
5.4. Perceptions Perceptions may change the course of operation of the planner. They provide key information for the planner to decide what part of the reactor to elaborate upon next. The planner specifies which perceptions it wants to receive, i.e. which sensory data the reactor needs to collect for the planner's purpose. Reactions are ideally suited for situations where the data yielded by the embedded sensing is of no interest to the planner, because such data has only local effect within the reflex. Unknown or uncertain data that can have an effect on future reactor segments, however, should be collected differently. For example, in the case of a constant, but unknown, fact, it is inefficient to manufacture a reaction with a conditional on this fact, since we know a priori that all but one branch of that conditional is wasted. Factoring in resource use constraints means that some problems would become impossible when implemented with conditionals. There simply won't be enough resources available to cover all the a priori options. In addition, posting perception processes is useful for longer term information that is important to the planner, e.g. in the kitting domain, the fact of whether downstream automation is always ready to receive finished kits. In these cases, the planner produces perception requests: these are actions that report information back to the planner. The planner uses this information to refine its knowledge about the world, and thus remove the troublesome uncertainty. In contrast, insertion of communicating networks in the reactor is clearly most appropriate for transitory information, e.g., the position of a cap on the belt, or for determining which of the situations, sit(t), of a task, t, is currently present.
112 This fosters a fast response on the part of the reactor, it allows the reactor to deal with contingencies, and to take advantage of opportunities such as fortuitously occurring goal conditions. Apart from informational perceptions, the planner relies on four specific perceptions to set its priorities for the planning process. These perceptions are the following: 1.
2.
3.
4.
Guard-kills. The planner is informed when an ~. process is killed. ~. processes ('guards') form the basis by which an adaptation can remove old structure and install new structure in the reactor. This perception indicates to the planner the completion of an adaptation. Stub trigger. The execution of a sTtm process in the reactor signals the planner that the reactor has started to execute a segment for which no concrete plan yet exists. Situation failure. This signals the planner that a reactor segment failed (usually through an invalidated assumption), and needs to be modified as soon as possible. Assumption failure. The planner installs dedicated processes (A]:t) in the reactor to monitor assumptions. If one of the assumptions is invalidated, its monitor will signal the planner.
Guard-kills govern the adaptation cycle and are necessary for the planner to preserve the safety of the adaptation process. If a situation is still being adapted, the planner cannot send out a next adaptation for that same situation since the safe adaptation mechanism is only safe for one adaptation per situation at the time. Thus the planner holds back the next adaptation instructions for that situation until the guard-kill perception is received. The other three types influence the planner's priority in planning adaptations. The planner uses these to decide which part of the expectation to reduce first. In effect, the environment (through the reactor) tells the planner what parts of the reactor should be considered for adaptation. The priory ordering for planning is the following: Failed situations > Forced assumption relaxation > Active stubs > Quiescent stubs.
In absence of any perceptions received i.e in a fully cooperating environment, the normal procedure is to build an initial reactor first using all assumptions. This initial reactor is built in increments with stub refinement proceeding along temporal ordering of the reactor segments. Subsequently, assumptions are relaxed to widen the scope of the reactor.
113 5.5. F o r c e d R e l a x a t i o n .
Any of the assumption monitors or stub triggers i n the initial reactor segments can potentially signal the planner and divert its attention from the a priori established ordering of assumptions relaxation. Failure of a situation to successfully complete its reactions is also cause for a perception. These three sources of perceptual input have different effects on the planner. Stub trigger. On receiving a stub trigger signal, the planner redirects its focus of attention for refining the abstract plan to the portion of the plan containing the stub trigger. The planner inserts the appropriate refinement into the reactor. The reactor then can start executing that segment immediately, since the situation is still active. Assumption failure. An assumption failure perception has a larger effect than simply refocusing attention. The assumption failure signal causes the planner to negate the assumption in its Problem Solving Context and begin to rebuild the effected portions of its plan. Situation failure. If a situation relying on that assumption failed, that particular situation is given highest priority for adaptation, even though other assumptions may have priority in the assumption ranking. This means the planner may have to turn its focus back to parts of the plan it had previously considered finished. Apart from its cause, a forced relaxation or refinement results in the same reactor segment and adaptation sequence as a normal relaxation or refinement. 6.
EXPERIMENTAL RESULTS
The kitting example described section 3 was implemented and several experimental runs conducted on a PUMA-560 based kitting work cell. Trace statistics were gathered on each run on, e.g., the times when assumption failures were detected, when adaptations were made, the number of assumptions in use in the reactor, and so on. In each run, the planner utilized all the assumptions described in section 3. However, only five of the assumptions were actually relaxed in these trials. The purpose of these experimental runs was to begin to explore the behavior of a planner-reactor system in practice. Results from the runs are presented in the following subsections. Initial Reactor Construction. Figure 4 contains trace statistics for the start-up phase on a typical experimental run of the kitting work cell. Graph (a) shows the number of assumptions in use at any time in the work cell by the planner and by the reactor. Graph (b) shows the number of adaptations issued (by the planner) and the number of discrete changes to the reactor structure (adaptations applied). Initially there is an empty reactor in operation. After 26 seconds (in this example) the planner is in a position to start to update the reactor. (This time include planning plus communication time.) The planner begins by sending adaptations to the reactor (adaptations issued). Each adaptation involves one or more changes to the reactor structure. Thus, eventually, the adaptations applied will outnumber the adaptations issued.
114 The constraints involved in safe adaptation can cause a time lag in implementing changes. Active situations cannot be i n t e r r u p t e d - that would leave the reactor in an undefined s t a t e - instead the adaptation waits for the situation to end before applying the changes; the theory behind this is presented in [10]. Thus the adaptations applied trace always lags the adaptations issued trace. The lag time varies, depending on the activity of the reactor. This is what causes the reactor assumptions trace in Figure 4(a) to lag the planner assumption trace.
Figure 4: Start-up phase statistics for one run of the Planner-Reactor based Kitting Robot The first reactor was in place roughly 28 s after start-up. Even though this reactor was not complete, the kitting robot could start kitting; its kitting actions were overlapped in time with the further refinement of its kitting 'program'. This reactor was elaborated over the following 20 s until by 40 s after start-up the first complete reactor was in place. This reactor employed all nine assumptions. Up to this point, there was little delay between the planner reducing an assumption and the reactor following suit. Assumption Relaxation. Then the planner began to relax assumptions to incrementally improve the reactor. It took roughly 5 s to relax the first assumption (the length of the 'plateau' in the planner trace in Figure 4(a)). It then began to issue the adaptations to the reactor. Typically one or more adaptations is necessary to relax any one assumption. Each adaptation will give rise to one or more structural changes in the reactor, again subject to the delays of the safe adaptation constraint. In the case of this first assumption relaxation, its took the reactor roughly 60 s before it was able to implement the last structural change to relax the assumption (the 'plateau' on the reactor trace in Figure 4(a)). It took the planner roughly 30 s to relax the second assumption; but the change was implemented in the reactor almost immediately after the first relaxation. These lags and overlaps in operation underscore the highly asynchronous and independent operation of planner and reactor.
115 In these experiments, we restricted to planner to only relax two assumptions voluntarily. Having done this, the planner will only relax other assumptions when the environment forces their failure. This could occur within the start-up phase, of course, but for convenience of presentation we have separated the two here.
Figure 5: Forced relaxation statistics Environment Forced Relaxation. Figure 5 shows trace statistics for the ongoing operational phase in a typical experimental run of the kitting work cell. Again, graph (a) displays the assumptions in use by the planner and reactor, and graph (b) displays the trace of adaptations issued and applied. Additionally, Figure 5 (b) now also shows the trace of assumption failure perceptions. These are the perceptions that force the planner to relax an assumption. The two forced assumption relaxations in Figure 5 are the no-motion (AM) and downstream disturbance (ADD) assumptions. The failure perception for AM is generated when the robot fails to acquire a part it had identified in an initial visual image of the work space. The failure perception for ADD is manually invoked, and would normally be an input signal from downstream automation. At 152 s the AM assumption failure was signaled (Fig. 5(b)). The planner responded very quickly and began to issue adaptations by 155 s, continuing for about 10 s. There was little delay in the safe adaptation procedure and the structural changes to the reactor lag the adaptations issued by only about ls. This is also evident from the assumptions trace in (b). Part of the changes in this adaptation was to remove the assumption failure monitor for this assumption; thus, partway into the adaptation (161 s) the signals from the assumption monitor cease. The total forced relaxation response time ~ the time from failure to final change in the reactor m is roughly 11 s. The ADD failure occurred at 215 s and proceeded in a similar fashion. Reinstating Assumptions. In our theoretical analysis [9, 10] we did n o t address the problem of reasserting previously failed assumptions. Nonetheless, this is convenient in practice to model operating regimes, and therefore is part of our objectives. In the experimental run, at time 357 s the ADD assumption was reinstated
116 (Figure 5b)). On the relaxation of such reinstateable assumptions, the planner adds a reactor monitor that will signal when the assumption holds again. (It doesn't show up as an assumption failure, since it isn't). As the trace statistics show, our implementation does indeed handle the reintroduction of assumptions. However, further work is necessary to extend our convergence results to include this case. Ordering of Assumption Relaxation. The graphs of assumptions versus time do not indicate which assumptions are relaxed at each step. Figure 6 shows the interval for which each assumptions is in use in both planner (solid line) and reactor (broken line). The acronyms on the Y-axis refer to assumptions. Four assumptions m ADT (tray disturbance), ACP (cooperation), ACT (competition), and AF (filled trays) m are not relaxed in this experiment and hence hold throughout. The assumptions of quality (AQ) and substitutability (AS) are introduced first. Note that the reactor does not drop these assumptions for some time after the planner has dropped them. This is also shown in Figure 4 but there it wasn't evident which assumptions were involved. The downstream disturbance assumption (ADD) is introduced, then relaxed and then reintroduced, as shown in Figure 5.
Figure 6: Reactor assumption usage over time
Figure 7: The response to immediate failure of assumptions
Immediate Assumption Failure. The assumption relaxation in the experiment shown in graphs 4 and 5 is artificial: The planner-directed relaxation of assumptions is carefully separated in time from the environment-directed relaxation.
117 This is unlikely in practice, both planner and environment will be simultaneously directing relaxation. The assumptions graph in Figure 7 was generated in simulation (the previous graphs have been from real robot experiments) by assuming that whenever the reactor tests an assumption, the assumption failso The first difference with the previous experiment, (Figure 4) is that the planner never gets to use all the assumptions. Failure signals begin as soon as the first reactor update is made, even before the entire first ideal reactor is constructed. The planner quickly relaxes the AQ, AS and AO assumptions. It then finishes the reactor, introducing the ADD assumption which also immediately fails (the 'blip' at 44 s) and is replaced. 7.
DISCUSSION
In summary, we have reviewed our motivation and approach to casting planning as adaptation of a reactive system and have presented the first performance results of the approach, a first for integrated systems of this kind. These results show the feasibility of our approach. The incremental reactor construction and assumption relaxation strategy minimizes planning delays. The reactor can start kitting while the planner still is refining later parts of the task. The asynchronous communication between planner and reactor allows the latter to be real-time and not bogged down by the planner. The planner stays informed through its implantation of perceptions in the reactor. Information from these perceptions gets shuttled back to the planner, which employs it to decide on the appropriate parts of the reactor to modify next. Thus, the planner is able to support the reactor's direct needs, despite the asynchronous link between the two. REFERENCES
1. J. Bresina & M. Drummond, Integrating Planning and Reaction, in: AAAI Spring Workshop on Planning in Uncertain, Unpredictable or Changing Environments, Stanford, CA, J. Hendler (ed.), Systems Research Center, Univ. of Maryland (1990). 2. R. Brooks, A Robust Layered Control System for a Mobile Robot, IEEE Journal Robotics & Automation, 2 (1986). 3. J. Connell, SSS: A hybrid architecture applied to robot navigation. IEEE Int'l Conf. on Robotics & Automation (1992). 4. R.E. Fikes and N.J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence 2, (1971), 189-208 5. B.R. Fox and K.G. Kempf, Opportunistic scheduling for robotic assembly. IEEE Int'l Conf. Robotics & Automation, St.Louis MO (1985), 880-889. 6. Th. Fraichard & C. Laugier, On-line Reactive Planning for Non-Holonomic Mobile Robot in a Dynamic Environment, IEEE Int'l. Conf. on Robotics & Automation, CA (1991). 7. E. Gat, Alfa: A language for programming reactive robot control systems, IEEE Int'l Conf. on Robotics & Automation, Sacramento, CA (1991), 1116-1121. 8. D.M. Lyons, Representing and analyzing action plans as networks of concurrent processes. IEEE Trans. Rob. & Aut}, 9 (3), (1993).
118 9. D.M. Lyons & A.J. Hendriks, Planning for Reactive Robot Behavior, IEEE Int'l Conf. on Robotics & Automation, Nice, France (1992). 10.D.M. Lyons & A.J. Hendriks, Safely Adapting a Hierarchical Reactive System, SPIE Intelligent Robotics & Computer Vision Boston, MA, (1993). 11. D.M. Lyons, A.J. Hendriks & S. Mehta, Achieving Robustness by Casting Planning as Adaptation of a Reactive System, IEEE Int'l Conf. on Robotics & Automation, Sacramento, CA (1991), 198-203. 12. A.J. Hendriks and D.M. Lyons, A Methodology for Creating and Adapting Reactive Systems, IEEE Int'l conference on Tools for AI, San Jose, CA (1991), 220227. 13.D. McDermott, Robot Planning, Tech. Report YALEU/CSD/RR #861, Yale University, (1991). 14.M. Schoppers, Representation and Automatic Synthesis of Reaction Plans, Ph.D. Thesis, Tech. Rep. uiucdcs-r-89-1546, CS dept., University of Illinois at Urbana-Champaign (1989). 15. C.J. Sellers & S.Y. Nof, Performance Analysis of robotic kitting systems, Rob. & Comp. Integ. Manuf. 6 (1989). 16.X. Xiadong and G.A. Bekey, Sroma: An adaptive scheduler for robotic assembly systems, IEEE Int'l Conf. on Robotics & Automation, Philadelphia, PA (1988).
Intelligent Robots and Systems V. Graefe (Editor) 9 1995 Elsevier Science B.V. All rights reserved.
119
Robot task programming by human demonstration: Mapping human grasps to manipulator grasps Sing Bing Kang and Katsushi Ikeuchi
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
Abstract To alleviate the problem of overwhelming complexity in grasp synthesis and path planning associated with robot task planning, we adopt the approach of teaching the robot by demonstrating in front of it. A system with this programming technique is able to temporally segment a task into separate and meaningful parts for further individual analysis and recognize the human grasp employed in the task. With such derived information, this system would then map the human grasp to that of the given manipulator, plan its trajectory, and proceed to execute the task. This paper describes how grasp mapping can be accomplished in our system. The mapping process essentially comprises three steps. The first step is local functional mapping, in which grasps of functionally equivalent fingers are established. This is followed by gross physical mapping which produces a kinematically feasible manipulator grasp. Finally, by carrying out local grasp adjustment using some task-related criterion, we arrive at a locally optimal manipulator grasp. We describe these steps in detail in this paper and show results of example grasp mappings.
1. Introduction There is always a need for a portable, safe, easy, fast, and robust means of programming robots to perform tasks. Current conventional methods of programming robots such as programming using a robot language and direct manipulation of the robot through the desired trajectory (i.e., manual lead-through), while useful, fall short on some of these characteristics. Completely automatic task planning (with the inputs being just the high-level specifications of the task goals) is difficult to achieve in practice. This is attributed to the high combinatorial complexity of grasp synthesis and path planning in the presence of obstacles. We seek to mitigate this problem by using cues extracted directly from human performance of the task of interest. The
120 approach of programming a robot by allowing it to observe the human demonstrating the task is also known as Assembly Plan from Observation (APO) [9]. A major component in task planning is planning the manipulator grasp. In the following section, we briefly discuss some of the issues involved in grasp planning and reiterate the justification for our approach.
2. Grasp Planning Grasp planning involves determining the hand placement relative to the object of interest as well as the posture of the hand assumed in grasping the object, both in a manner consistent to the requirements of the task. It is computationally intensive due to both the large search space and the incorporation of geometric and task-oriented constraints. Approaches to grasp planning can be categorized as geometric and mechanical [ 19]. The geometric approach considers only the spatial aspects of the objects and arm/hand system, while the mechanical approach incorporates the arm/hand kinematics, system dynamics, and their task-related constraints. The large search space for an optimal grasp can be reduced in size or dimensionality by using certain heuristics or structural constraints. For a 2-finger gripper, for example, the surface element pair selection can be guided by a series of constraints, such as face parallelism, permissible face separation, and non-zero overlap between projected faces [24]. By imposing fixed contact locations or fixed types of grasp, Pollard computes a 6D projection of the 15 DOF configuration space for the Salisbury hand [20]. This reduced space represents the space of wrist poses for geometrically feasible grasps, and is used to search for a globally optimal grasp. In addition, a popular strategy for reducing the complexity of planning the dextrous hand grasp is by using a small set of predetermined grasps or grasping behaviours (e.g., [2], [7] and [21 ]). We adopt the approach to make grasp planning more tractable by using cues extracted from human execution of the grasp. The approach first involves the temporal segmentation of the recorded task executed by the human which locates the instant when the grasp occurs [ 11]. The subsequent steps are recognizing the type of grasp [12][13] and recovering the approximate grasp contact locations from observed data. This is followed by determining the initial approximate grasp which satisfies kinematic and geometric constraints, and subsequently performing local optimization of the grasp using the approximate grasp as the starting condition and a taskoriented objective function. Our grasp mapping approach is described in detail in the following sections.
3. Mapping grasps 3.1. General approach At the highest level of description, planning the manipulator grasp based upon the observed human grasp comprises the following steps:
121
1. Local functional mapping This is done by observing the type of grasp used by the human, and the functions and grouping of the palm and individual fingers. The latter is determined using the real-finger-to-virtual finger mapping described in [ 13]. The mapping of the manipulator end-effector to the virtual fingers is dependent on the effective cohesive index and the cohesive indices of the individual virtual fingers.
2. Adjustable or gross physical mapping This operation is carried out using the outcome of the local functional mapping and the initial location of the manipulator (near the pose of the human hand while carrying out the task). It results in an approximate form of the manipulator grasp of the object which is geometrically feasible.
3. Fine-tuning or local adjustment of grasp The grasp determined from steps 1 and 2 may not exhibit static stability, or that it may not be locally optimal according to a chosen task-oriented criterion. By analyzing the criterion and iteratively perturbing the initial grasp configuration, we arrive at a locally optimal grasp. In the case of the power grasp which involve high kinematic constraint, this step is skipped. 3.2. A n a l y t i c m e a s u r e s of a g r a s p Cutkosky and Howe [5] summarizes a number of analytic measures that have been used by various researchers in their analyses of grasps, several of which are for the purpose of grasp synthesis. We can group these grasp analytic measures into different categories according to both the depth of knowledge of the grasp and hand mechanism, and the type of analysis (Fig. 1). These analytic grasp measures are instrumental in determining the manipulator grasp. The overall strategy of mapping grasps is depicted in Fig. 2. The human grasp can be described in a hierarchical fashion [ 12]. With a priori knowledge of the human hand and data derived from observation, certain analytic grasp measures can be extracted (such as connectivity and mobility). Armed with these information, together with a priori kinematic, structural and geometric knowledge of the manipulator, the grasp translator would then attempt to produce a manipulator grasp with compatible analytic grasp measures. In our work reported in this paper, as a starting point, we employ the manipulability measure as the criterion used in generating the locally optimal manipulator grasp.
122
Analytic Grasp Measures Degree-of-freedom Measures connectivity mobility
Static Measures
Increasing depth of knowledge required of grasp and hand mechanism
force closure form closure internal forces resistance to slipping manipulability grasp isotropy
Structural and Servo Parameter Measure$ compliance stability Fig. 1 Categories of analytic grasp measures
f
Human grasp
~
]g l ras7 ulat~
observation
| | ofHierarchicalgrasp ~ description
a priori
knowledge of human ' hand ~.,
i/
~.~ ~ ~ ~" '~ ~ ~ [ "~ ~
.
description of grasp Analytic grasp measure________~s
I a priori
knowledge f manipulator
~
J
Fig. 2 Grasp mapping strategy
The functions of the hierarchical description of the human grasp are local functional mapping and gross physical mapping. Meanwhile, the functions of the analytic grasp measures are: 1. Verification of appropriateness of gross physical mapping; and 2. Local fine-tuning or perturbation of grasp to a locally optimal posture.
4. Local functional mapping This first step in mapping grasps is common to all types of grasps, i.e., power and precision grasps. The local functional mapping associates the manipulator fingers to those of the hand,
123 and is generally different for different grasps. To guide the local functional mapping, we first assign manipulator fingers as either a primary finger, secondary finger, or a palm.
4.1. Assigning manipulator fingers Prior to mapping the the manipulator fingers to those of the human in the observed grasp, the manipulator finger has to be assigned as one of the groups of fingers:
1. Primary finger/s A finger in this group would correspond to the human thumb, which has at least the same number of degrees of freedom as the other fingers. It serves not only to secure the hold of the object against the palm and/or other fingers, but also to manipulate the object.
2. Secondary finger/s This group would correspond to the fingers of the hand other than the thumb. This group of fingers would serve to secure the hold of the object against the palm and/or primary finger/s. A special case of a "finger" which is also assigned is the palm; it is passive with no local degree of freedom (i.e., discounting the global 6 DOF of the manipulator). Manipulators are designed with specific functions or level of dexterity in handling objects. The repertoire of grasping abilities may be as limited and simple as just clamping the object between its jaws as a means of securing a hold on it (as in the parallel-jaw gripper). The manipulator may be anthropomorphic to some degree and be able to mimic certain grasps of the human hand; examples of such hands are the Utah/MIT hand [10], Salisbury hand [22], Okada hand [18], Minnesota hand [15], and the Southampton hand [4]. It is usually simple to label a priori each of the manipulator fingers as either a primary or secondary finger. Once the primary and secondary fingers have been identified, the hand-to-manipulator finger mapping can proceed with the aid of another type of mapping. This mapping relates real fingers to groups of functionally equivalent fingers called virtual fingers [1][13]. A consequence of this real-finger-to-virtual finger mapping is the cohesive index [ 13], which is associated with the degree to which the real fingers in a virtual finger act in a similar manner (by comparing the object normals at the contact points). A cohesive index of a virtual finger is maximum at unity, and would indicate that the fingers within that virtual finger act exactly alike. 4.2. U s i n g t h e
cohesive index to guide the mapping of fingers
An indication of how the cohesive index is used to guide the virtual finger mapping between those of the human hand and the manipulator is illustrated in Fig. 3. (VF n refers to the nth virtual finger with C n being its associated cohesive index.) The palms, primary fingers, and secondary fingers are mapped appropriately as seen in Fig. 3. The cohesive index serves as a check - the virtual finger corresponding to the group of secondary fingers normally has a cohesive index less than that of the collection of primary fingers. The virtual finger composition determined from the real-finger-to-virtual finger mapping [ 13] take precedence, however.
124
VF o = {0} (palm) .~ VF 1={1} VF 2 = {2, 3, 4, 5}
C o = 1.0 C 1=1.0 C 2 < 1.0
5-fingered cylindrical power grasp VF o' = {0 } (palm) VF 1' = {1} VF 2' = {2}
VF0" = {0 } (palm) VFI" = {1} VF2" = {2, 3}
VF0'" = {0 } (palm) VFI'" = {1} VF2"' = {2, 3, 4}
(Decompose VF 1 and VF 2 this way because C 2 < C 1)
Two-fingered parallel-jaw gripper
Three-fingered Salisbury hand
Four-fingered Utah-MIT hand
Fig. 3 Mapping grasps according to cohesive indices of virtual fingers Once the functional finger assignments have been accomplished, the more detailed manipulator grasp planning can then proceed.
5. Approach in generating grasp and pregrasp trajectory The diagram representing the entire manipulator grasp and trajectory planning approach is shown in Fig. 4. Prior to the more detailed manipulator grasp planning, the manipulator assumes a fixed pose and posture relative to the human hand throughout the task execution; local planning is carried out within the vicinity of this pose and posture. Note that by virtue of the different form and function of the power and fingertip precision grasps, they are planned in a different manner elucidated in subsequent sections. Once the local planning is done, starting with the pose and posture of the manipulator during the grasping stage, the pregrasp trajectory is then planned backwards, with the goals of a smooth and collision-free path. This path would closely follow that assumed by the human hand trajectory.
125 Depending on the type of grasp, the approach in manipulator grasp planning differs. In the subsequent description of such mapping techniques, we concentrate on the most widely used grasps, namely the power grasp and the fingertip precision grasp. Identify human grasp Generate grasp hierarchy
$ Functional finger group assignment for manipulator xV
Follow approximate hand trajectory
Power grasp < t~
Fingertip precision grasp ~,,i~
Gross grasp posture followed by fine-tuning of grasp posture ~V
Regenerate the pregrasp trajectory
Fig. 4 Scheme for generating the manipulator grasp and pregrasp trajectory
6. Generating the power grasp The primary purpose of the power grasp, in which the hand generally envelops the object, is to ensure a secure hold of the held object at the expense of dexterity. Hence, we can dispense with the third step of local grasp adjustment.
6.1. General approach The approach taken to generate the power or "enveloping" grasp follows that taken in "reactive" planning (similar to that described in [2]). Note that this approach is taken in generating the grasp, and is not used in the actual execution of the grasp itself. The approach is characterized with the following phases:
126
1. Aligning the dextrous hand with the approach vector assumed by the human hand just prior to grasping (approach alignment phase). 2. Translating the hand (preserving its orientation) towards the object until the palm touches the object (approach phase). 3. Adjust the hand to "fit" the object to the palm (adjustment phase). 4. Flex the fingers to envelope the object (wrapping phase). Central to the alignment motions of the dextrous hand is the pivot point. It is predefined to be a point on the surface of the palm between the fingers. Its frame is oriented such that its z-direction points away from the palm and the x-direction points from the thumb (or a primary finger/ s) to the rest of the fingers. 6.2. A p p r o a c h a l i g n m e n t of t h e d e x t r o u s h a n d In order to minimize the problem of collision with the object during the pregrasp and post grasp phases, we first align the pivot frame of the dextrous hand to the approach vector assumed by the human hand just prior to grasping the object. This is known as the approach alignment phase. The approach vector is determined as the translation component of the differential transformation AT, where AT = Tg Tg_k-1, where Tg is the transformation of the hand at the grasp frame and Tg_k is the transformation of the hand k frames prior to the grasp frame (k is chosen to be 2). Once the rotational transformation (about the pivot point) has been determined, the dextrous hand is rotated in steps, with the intermediate collision detection with the object. If collision is detected, the dextrous hand is translated or "repulsed" away (preserving its orientation) from the surface of object that makes contact with the hand. The normals of the surface or surfaces in question determine the direction of the repulsion vector. 6.3. A d j u s t i n g t h e h a n d a n d e n v e l o p i n g t h e o b j e c t Once the desired orientation of the hand is assumed, planning proceeds with the following behaviors:
1. Pure translation toward the object (approach phase) The translational motion is dictated by attractive forces by the object on the pivot point on the dextrous hand. This behavior is active prior to any collision between the object and the manipulator palm.
127
~ Object n j--
~
Fig. 5 Determining direction of translation From Fig. 5, p is the pivot point on the manipulator palm, and qj is the jth sampled object surface point, rj is the vector from point qj to p, i.e., rj = qj - p. fij is the object normal at qj. The motion at each stage is M:
d M = 8=-=, Idl
d = -
E llj. rj>o
fD.r. J J
where 8 is the step size in 3D space. r is the weighting factor dependent on Irjl; the function used is o~j = Irj1-2.
Translation and rotation about the pivot point (adjustment phase) Once collision between the object and the manipulator palm is detected, this behavior becomes active. The motion associated with this behavior is defined by both the point or points of contact and the entire object. The points of contact determine the motion which tend to push away the manipulator (repulsive forces) while the entire object tends to pull the manipulator (via the pivot point) towards the object. This has the property of trying to align the manipulator relative to the object prior to grasping. It should be noted that while these behaviors are active, the posture of the manipulator is that of full extension of the fingers (for parallel jaw gripper, maximally distanced fingers). Note: The circumflex (^) over the symbol indicate its normalized version. The rotation about the pivot point is ~, where Q-
E Qj qjCP
and P is the palm, Qj is the quaternion associated with the rotation due to the jth contact point and is given by Qj =
2 Ecos ~-Oj, mjsin-Oj
0j = 8/Irjl and mj = -rj x nj.
128
3. Flexion of fingers around the object (wrapping phase) Once the manipulator has been adequately aligned with respect to the object, the manipulator would then proceed to wrap its fingers around it. This is done by flexing successively distal joints (starting with the most proximal joint) until contact is made between the associated link and the object (Fig. 6 (a) and (b)).
Fig. 6 Results of mapping a cylindrical power grasp for Utah/MIT hand (a) and Salisbury hand (b) This approach is a simple one, and is relatively independent of the manipulator (once the manipulator models have been built). This approach is made possible by the cues given in the human execution of the same task.
7. Generating the fingertip precision grasp 7.1. G e n e r a l a p p r o a c h The approach taken to plan the fingertip precision grasp is different than that for the power grasp, primarily because this precision grasp does not envelope the object. As with the power grasp, there are basically two stages:
1. Initial grasp pose and posture determination phase Let the (N+ 1) contact points on the object be denoted Pc0, Pc 1 . . . . PcN" The initial grasp pose is determined by calculating the pose of the contact frame F c and orienting the pivot frame Fpivot centered at the pivot point on the hand by a certain fixed transformation relative to F c (Fig. 7).
129
Pcl Pco
9
d .....~ P c N Fpivot
Fig. 7 Orienting the pivot frame with respect to the contact frame Subsequently, the pose of the hand is searched at this neighhorhood until a kinematically feasible grasp is found. The contact frame F c is centered about point c, defined to be
1
N
1
1/
N
c -- ~ Z 2 (Pco +pc/) = ~ Pco + ~ Pc/ i=l i=1
/
The orientation of F c is determined as follow (note that the caret ^ on a vector term indicates its normalized version):
N N l = ~ (Pc/-Pco) - Z P c / - N P c o i=1 i=1 If the best fit plane H to the contact points is r 90 = a, where r is any point on H, and 0 its unit normal, then
h = I 0,
l
if
-0,
(P X1) 9(PeN--Pc0) >0 otherwise
and~h = ~ x ~ .
2. Local grasp pose and posture adjustment phase This is done by locally optimizing a task-oriented criterion with respect to the pose and posture of the hand. The local adjustment is done at two levels; the two levels are associated with the optimization with respect to the pose (outer level) and with respect to the joint angles (inner level). Note that the inner level optimization require iterations only if redundancy of finger DOFs exists. For the Utah/MIT hand, each finger has a redundant DOF since there are 4 DOFs per finger with only the 3D position of each fingertip being fixed. There is no redundancy of DOF for the three fingers of the Salisbury hand.
130 Let Aj = (tj, rj) be the vector representing the hand pose, Ojk be the vector of joint angles representing the hand posture, and C0 be the task-oriented criterion to be optimized. Then the optimization can be symbolically represented as:
max fmax C (Aj, Ojk) ) = max C (Aj, Oj, opt)
A.J Lo.
)
A.j
The outer optimization level uses the Nelder-Mead simplex algorithm (see, for example, [ 17]), while the inner optimization is accomplished using Yoshikawa's scheme for optimizing manipulator postures with redundant DOFs [25]. For a given initial hand pose Aj, the locally optimal hand posture Oj, opt is determined using the following pseudocode:
Repeat until either the joint angle limitations are exceeded or the magnitude of joint angle change vector 6)8t is below a threshold {
= ( I - J+J) ~k since ic = O, the contact points being fixed at each iteration.
%k <---%k + l~jkSt
} where J is the hand Jacobian and J+ is the pseudoinverse of the Jacobian, i.e.,
j+ = jT ( j j T ) - I
8. Estimating the dextrous hand-object contact points Approach is as follows:
1. Determine the rough positions of the human hand-object contact points. Fit these contact points with a plane H: p 9h = d.
2. Cull off planes and edges as potential grasping places by imposing the constraint that the associated object normal should not be too close in direction to h. At each point on edge i (edge point) and point on face j (face point), let the associated normals be nei and hf respectively. Then, disregard points on edge i if Ihe 9h I > COS0e and points on face j if lhfj. h > cos0f. Since we favor faces over edges, Oei> Of, with 0 e = 75 ~ and Of = 45