Distributed Video Sensor Networks
Bir Bhanu Chinya V. Ravishankar Amit K. Roy-Chowdhury Hamid Aghajan Demetri Terzopoulos Editors
Distributed Video Sensor Networks
Editors Prof. Bir Bhanu Center for Research in Intelligent Systems University of California, Riverside Riverside, CA 92521 USA
[email protected] Prof. Chinya V. Ravishankar Dept. Computer Science & Engineering University of California, Riverside Riverside, CA 92521 USA
[email protected] Assoc. Prof. Amit K. Roy-Chowdhury Dept. Electrical Engineering University of California, Riverside Riverside, CA 92521 USA
[email protected]
Prof. Hamid Aghajan Dept. Electrical Engineering, Packard Bldg. Stanford University Serra Mall 350 Stanford, CA 94305-9505 USA
[email protected] Prof. Demetri Terzopoulos Dept. Computer Science University of California, Los Angeles Boelter Hall 4731 Los Angeles, CA 90095-1596 USA
[email protected]
ISBN 978-0-85729-126-4 e-ISBN 978-0-85729-127-1 DOI 10.1007/978-0-85729-127-1 Springer London Dordrecht Heidelberg New York British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library © Springer-Verlag London Limited 2011 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licenses issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Cover design: eStudio Calamar S.L. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Preface
On May 11–12, 2009, the University of California, Riverside, hosted a two-day workshop on “Distributed Video Sensor Networks—Research Challenges and Future Directions.” The workshop was attended by researchers from academia, industrial labs, and government, and was sponsored by the US National Science Foundation (NSF), the US Army Research Office (ARO), and the US Office of Naval Research (ONR). This book is a collection of articles authored by the leading researchers who attended this workshop, outlining their vision in this emerging, interdisciplinary research area. Large-scale video networks are becoming increasingly important for a wide range of critical applications such as video surveillance, the monitoring of disaster zones and traffic, elderly care, the tracking of people and vehicles in crowded environments, and the acquisition of more realistic images for consumer electronics and entertainment. In most of these applications, multiple sensors such as video cameras, infrared (IR), or range sensors gather data from various points of view, which are then sent to a central processing unit. In most existing systems, there is no intelligent processing of the data locally at each camera, and the monitoring stations, which are staffed with people, generally observe and oftentimes store multiple video streams with very limited automated processing. Many fundamental problems need to be solved before these networks can be used more effectively. It is extremely challenging to develop automated techniques for aggregating and interpreting information from multiple video streams acquired by large-scale networks of camera sensors in real-life scenarios. Research in video sensor networks is highly interdisciplinary, requiring expertise in a variety of fields, among them sensor networks, video analysis, cooperative control, communications, sensor design, real-time and embedded processing, graphics and simulation, and the development of real-world applications. However, these disciplines have their own core sets of problems, and it is not easy for researchers to address critical challenges that demand advanced knowledge from multiple areas. This volume is a collection of chapters contributed by leading researchers from these different areas, offering an in-depth understanding of the state-of-the-art, current research directions, and future challenges in distributed video sensor networks. Its objective is to address critical interdisciplinary problems at the intersection of v
vi
Preface
large-scale video camera networks and related disciplines. Starting with an overview of distributed video sensor networks and research opportunities, the subsequent parts of the book discuss issues pertinent to: (a) Video Processing and Video Understanding, (b) Simulation, Graphics, Cognition, and Video Networks, (c) Wireless Video Sensor Networks, Communications, and Control, (d) Embedded Cameras and Real-Time Video Analysis, (e) Applications, and (f) Educational Opportunities and Curriculum Development. We hope that this book will catalyze a research agenda and interdisciplinary collaborations in large-scale camera networks, an emerging field with many exciting scientific and technological opportunities, thereby setting the stage for potentially transformative breakthroughs. We appreciate the suggestions and support received from Dr. Scott Midkiff (NSF), Dr. Jie Yang (NSF), Dr. Liyi Dai (ARO), and Dr. Martin Kruger (ONR). Bhanu and Roy-Chowdhury would like to acknowledge the support from NSF (ENGR 0910614, ENGR 0622176, IIS 0551741), ARO (W911NF-09-1-0115, W911NF-07-1-0485) and ONR (N00014-07-1-0931, N00014-09-C-0388). Bhanu, Ravishankar and Roy-Chowdhury would like to acknowledge the support received from ONR (N00014-07-0311). Bhanu, Roy-Chowdhury and Terzopoulos would like to acknowledge the support from NSF (IIS 0905671). Terzopoulos acknowledges the support of Dr. Thomas Strat (formerly of DARPA), and UCLA. We would like to thank Dean Reza Abbaschian, Victor Hill, Eilene Montoya, Mitch Boretz, Jhon Gonzalez, Bill Bingham and Jim Dexter for their support with the workshop. Riverside, CA, USA August 20, 2010
Bir Bhanu Chinya V. Ravishankar Amit K. Roy-Chowdhury Hamid Aghajan Demetri Terzopoulos
Contents
Part I 1
Distributed Video Sensor Networks and Research Challenges
Report on NSF/ARO/ONR Workshop on Distributed Camera Networks: Research Challenges and Future Directions . . . . . . . . Bir Bhanu and Amit Roy Chowdhury
Part II
3
Video Processing and Understanding
2
Motion Analysis: Past, Present and Future . . . . . . . . . . . . . . . J.K. Aggarwal
27
3
Projective Joint Invariants for Matching Curves in Camera Networks 41 Raman Arora and Charles R. Dyer
4
Multiple-View Object Recognition in Smart Camera Networks . . . Allen Y. Yang, Subhransu Maji, C. Mario Christoudias, Trevor Darrell, Jitendra Malik, and S. Shankar Sastry
5
A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yiming Li and Bir Bhanu
55
69
6
Distributed Sensing and Processing for Multi-Camera Networks . . Aswin C. Sankaranarayanan, Rama Chellappa, and Richard G. Baraniuk
85
7
Tracking of Multiple Objects over Camera Networks with Overlapping and Non-overlapping Views . . . . . . . . . . . . . . . . 103 Jenq-Neng Hwang and Victor Gau
8
Toward Robust Online Visual Tracking . . . . . . . . . . . . . . . . 119 Ming-Hsuan Yang and Jeffrey Ho vii
viii
9
Contents
Modeling Patterns of Activity and Detecting Abnormal Events with Low-Level Co-occurrences . . . . . . . . . . . . . . . . . . . . . . . . 137 Yannick Benezeth, Pierre-Marc Jodoin, and Venkatesh Saligrama
10 Use of Context in Video Processing . . . . . . . . . . . . . . . . . . . 149 Chen Wu and Hamid Aghajan Part III Simulation, Graphics, Cognition and Video Networks 11 Virtual Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Demetri Terzopoulos and Faisal Z. Qureshi 12 Virtualization and Programming Support for Video Sensor Networks with Application to Wireless and Physical Security . . . . 179 Azer Bestavros and Michael J. Ocean 13 Simulating Human Activities for Synthetic Inputs to Sensor Systems 193 Jan M. Allbeck and Norman I. Badler 14 Cognitive Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . 207 Thomas C. Henderson 15 Ubiquitous Displays: A Distributed Network of Active Displays . . . 215 Aditi Majumder Part IV Wireless Video Sensor Networks, Communications and Control 16 Research Challenges for Wireless Multimedia Sensor Networks . . . 233 Tommaso Melodia and Ian F. Akyildiz 17 Camera Control and Geo-Registration for Video Sensor Networks . 247 James W. Davis 18 Persistent Observation of Dynamic Scenes in an Active Camera Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Bi Song, Chong Ding, Amit Roy-Chowdhury, and Jay Farrell 19 Proactive PTZ Camera Control . . . . . . . . . . . . . . . . . . . . . 273 Faisal Z. Qureshi and Demetri Terzopoulos 20 Distributed Consensus Algorithms for Image-Based Localization in Camera Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . 289 Roberto Tron, Andreas Terzis, and René Vidal 21 Conditional Posterior Cramér–Rao Lower Bound and its Applications in Adaptive Sensor Management . . . . . . . . . . . . . 303 Ruixin Niu, Long Zuo, Engin Ma¸sazade, and Pramod K. Varshney
Contents
Part V
ix
Distributed Embedded Cameras and Real-Time Video Analysis
22 VideoWeb: Optimizing a Wireless Camera Network for Real-time Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 Hoang Thanh Nguyen and Bir Bhanu 23 VideoWeb Dataset for Multi-camera Activities and Non-verbal Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Giovanni Denina, Bir Bhanu, Hoang Thanh Nguyen, Chong Ding, Ahmed Kamal, Chinya Ravishankar, Amit Roy-Chowdhury, Allen Ivers, and Brenda Varda 24 Wide-Area Persistent Airborne Video: Architecture and Challenges 349 Kannappan Palaniappan, Raghuveer M. Rao, and Guna Seetharaman 25 Collaborative Face Recognition Using a Network of Embedded Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Vinod Kulathumani, Srikanth Parupati, Arun Ross, and Raghavender Jillela 26 SATware: A Semantic Approach for Building Sentient Spaces . . . . 389 Daniel Massaguer, Sharad Mehrotra, Ronen Vaisenberg, and Nalini Venkatasubramanian Part VI Applications of Distributed Video Networks 27 Video Analytics for Force Protection . . . . . . . . . . . . . . . . . . 405 Peter H. Tu, Glen W. Brooksby, Gianfranco Doretto, Donald W. Hamilton, Nils Krahnstoever, J. Brandon Laflen, Xiaoming Liu, Kedar A. Patwardhan, Thomas Sebastian, Yan Tong, Jilin Tu, Frederick W. Wheeler, Christopher M. Wynnyk, Yi Yao, and Ting Yu 28 Recognizing Activity Structures in Massive Numbers of Simple Events Over Large Areas . . . . . . . . . . . . . . . . . . . . . . . . 427 Raymond D. Rimey 29 Distributed Sensor Networks for Visual Surveillance . . . . . . . . . 439 Zeeshan Rasheed, Khurram Shafique, Li Yu, Munwai Lee, Krishnan Ramnath, TeaEun Choe, Omar Javed, and Niels Haering 30 Ascertaining Human Identity in Night Environments . . . . . . . . . 451 T. Bourlai, N. Kalka, D. Cao, B. Decann, Z. Jafri, F. Nicolo, C. Whitelam, J. Zuo, D. Adjeroh, B. Cukic, J. Dawson, L. Hornak, A. Ross, and N.A. Schmid
x
Contents
Part VII Educational Opportunities and Curriculum Development 31 Educational Opportunities in Video Sensor Networks . . . . . . . . . 471 Thomas C. Henderson Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
Introduction Distributed Video Sensor Networks
The automated aggregation and interpretation of information from multiple video streams acquired by large-scale networks of camera sensors in real-life scenarios presents highly challenging scientific and technological problems. Research in video sensor networks is highly interdisciplinary, requiring expertise from a variety of fields. Examples include sensor networks, video analysis, cooperative control, communications, sensor design and embedded systems, graphics and simulation, and the development of real-world applications. However, these disciplines have their own core sets of problems, and it is not easy for researchers to address challenges that require advanced knowledge from multiple areas. This volume is a collection of chapters contributed by leading researchers from these different areas with additional overview material, offering an in-depth understanding of the state-of-the-art, future challenges, and current research directions in distributed video sensor networks. Its objective is to address critical challenges at the intersection of large-scale video camera networks and related disciplines. The book consists of seven parts, as follows: Part I—Recommendations of the NSF/ARO/ONR Workshop on “Distributed Video Sensor Networks—Research Challenges and Future Directions” Part II—Video Processing and Video Understanding Part III—Simulation, Graphics, Cognition, and Video Networks Part IV—Wireless Video Sensor Networks, Communications, and Control Part V—Embedded Cameras and Real-Time Video Analysis Part VI—Applications of Distributed Video Networks Part VII—Educational Opportunities and Curriculum Development An introduction of each of the above parts and their constituent chapters follows: PART I: Distributed Video Sensor Networks and Research Challenges Chapter 1 by Bhanu and Roy-Chowdhury summarizes the NSF/ARO/ONR Workshop on “Distributed Video Sensor Networks”. It presents the recommendations of the workshop in each of the topical areas that were discussed at the meeting— Video Processing and Video Understanding; Simulation, Graphics, Cognition, and xi
xii
Introduction
Video Networks; Wireless Video Sensor Networks, Communications and Control; Distributed Embedded Cameras and Real Time Video Analysis; Applications of Distributed Video Networks; Educational Opportunities and Curriculum Development—which are reflected in the organization of this volume. We include the workshop report with the hope that it will catalyze a research agenda and interdisciplinary collaborations in large-scale camera networks, thereby setting the stage for potentially transformative breakthroughs in the exciting scientific and technological challenges that uniquely define this emerging field. PART II: Video Processing and Understanding This part consists of nine chapters, which cover research in motion analysis, invariants, and the detection, tracking, and recognition of objects and activities in video networks. Motion is of the essence for any dynamic scene analysis and understanding in a video network. Chapter 2 by Aggarwal provides an overview of the past, present, and future of motion analysis. It presents an early history of motion, and offers highlights from philosophy, psychology, and neurobiology. Further, it focuses on motion in computer vision from its beginnings to current work on human actions and activities. Finally, it discusses diverse applications of motion analysis. Chapter 3 by Arora and Dyer on projective joint invariants presents an algorithm for matching curves across widely varying viewpoints in camera networks. The equivalence of curves under projective transformation is established by matching sections of invariant signature manifolds, which are local, compact descriptors of curves. The method first establishes the best possible correspondence between two curves using sections of the invariant signature manifold and then performs a test for equivalence. Chapter 4 by Yang et al. on multiple-view object recognition focuses on object recognition in low-power, low-bandwidth smart camera networks. To accommodate limited transmission bandwidth between the cameras and the base station computer, their method utilizes the available computational power on the smart sensors to locally extract SIFT-type image features that represent individual camera views, thus enabling the camera network to compute a sparse pattern that encodes the acquired data. In accordance with distributed compressive sensing theory, the base station then decodes and recovers the set of multi-view features. Tracking is a basic functionality in distributed video networks. This may require collaboration among the cameras and the ability to perform camera selection and hand-off. These issues are the focus of Chapter 5 by Li and Bhanu, which presents a theoretical and experimental comparison of four approaches for camera selection and hand-off—a game theoretic approach, a co-occurrence to occurrence ratiobased approach, a constraint satisfaction-based approach, and a fuzzy-logic-based approach. Chapter 6 by Sankaanarayanan, Chellappa, and Baraniuk identifies challenges in designing distributed sensing and processing for multi-camera networks. It demonstrates the use of geometric constraints in designing optimal estimators for object
Introduction
xiii
detection and tracking. It also highlights the use of the theory of compressive sensing for data reduction in sensing and communication within a camera network. Chapter 7 by Hwang and Gau focuses on the tracking of multiple objects by camera networks with overlapping and non-overlapping camera views, in which tracking within a camera’s field of view is achieved using Kalman filtering and particle sampling based on a set of landmark points. The process is extended to multi-camera setups using image registration and appearance parameters. Chapter 8 by Yang and Ho on robust online learning develops techniques that exploit a combination of a priori information acquired offline with online information using learning algorithms. The chapter discusses learning nonlinear appearance manifolds and online multiple instance boosting for tracking humans and other objects with the goal of simultaneous tracking and recognition in changing environments. In Chapter 9 by Benezeth, Jodoin, and Saligrama on modeling patterns of activity and detecting abnormal events with low-level co-occurrence of activity, the authors describe a location-based approach for behavior modeling and abnormality detection. In contrast to conventional object-based approaches for which objects are identified, classified, and tracked to detect suspicious behavior, their proposed method directly links low-level features with event characterization and behavior modeling. A statistical model for normal activities is computed based on observations, resulting in a co-occurrence matrix that is then employed to detect moving objects whose behavior differs from the object motions observed during the training phase. Chapter 10 by Wu and Aghajan explores the use of context for data processing in a video network. The authorscategorize the context available in an application as environmental context and user-centric context, which may also be classified as static or dynamic. They present examples to show how context can be acquired and used in a multi-camera video processing scenario within a smart home environment. PART III: Simulation, Graphics, Cognition, and Video Networks This part consists of five chapters dealing with virtual vision, network security, human activities, cognitive architecture, and displays. To facilitate research in multi-camera sensing and control in large, real-world video networks, in Chapter 11 Terzopoulos and Qureshi present a “virtual vision” approach that performs a virtual reality simulation of populated urban environments, camera networks, and computer vision algorithms. The authors demonstrate their approach by developing two highly automated surveillance systems consisting of passive and active pan/tilt/zoom cameras that are deployed in a simulated train station populated by lifelike, autonomous virtual pedestrians. Chapter 12 by Bestavros and Ocean discusses the integration of wireless networking security services into a sensor network workbench called snBench. The workbench provides an extensible framework that enables the rapid development and deployment of sensor network applications on a shared, embedded, and actuation infrastructure. snBench enables the rapid prototyping and testing of methods for intrusion detection and response services.
xiv
Introduction
Chapter 13 by Allbeck and Badler presents a system, called CAROSA, for simulating human activities that can be used as synthetic spatiotemporal inputs to sensor systems. The objectives are to build statistical models of pedestrian flow and density, and to correlate these data with population movement and density models. CAROSA enables the authoring of functional crowds of people performing role-, context-, and schedule-dependent activities. Chapter 14 by Henderson discusses a domain-theoretic cognitive architecture for the autonomous and robust operation of senor-actuation networks. These networks are able to perceive, learn, reason, and act by means of a distributed, sensor-actuator computation and communication system. The elements of this architecture include domain theories, world model construction by mapping observations onto theory, and controller behaviors. In Chapter 15 Majumder examines the high-level design of environments with ubiquitous active displays that interact with other components of workspaces, including users and other devices. The seamless integration of a collection of such displays into an environment enables the design of advanced collaborative workspaces capable of recognizing natural gestures as a means of supporting user interactions. PART IV: Wireless Video Sensor Networks, Communications and Control This part consists of six chapters focusing on multimedia networks, registration, the control of cameras (in simulated and real networks), as well as localization and bounds on tracking. In Chapter 16 Melodia and Akyildiz discuss the major research challenges in wireless multimedia networks. The authors argue that it may be impossible in many applications, such as those requiring rapid deployment in adverse conditions, to deliver or obtain multimedia data over wired connections. After discussing the existing solutions at the various layers of the protocol stack, the chapter discusses open research issues and outlines future trends. The management of camera networks can be challenging for human observers due to the large amount of data that need to be analyzed. In Chapter 17 on control and registration of video sensor networks, Davis addresses this problem by discussing methods for efficient camera network management and access to cover large areas. The output of the proposed system is a geo-referenced map representation that enables the coordinated control and analysis of multiple cameras. Chapter 18 by Song et al. presents a game theoretic approach to the decentralized, cooperative control of active cameras in a video network. The goal of cooperation among cameras is to acquire multiple objects at different resolutions for the persistent observation of dynamic scenes. Chapter 19 by Qureshi and Terzopoulos introduces proactive Pan/Tilt/Zoom camera control in a sensor network, treating the control of cameras as a planning problem whose solution achieves optimal camera utilization with respect to predefined observation goals. The cameras plan ahead to select optimal camera assignment and hand-off with the passive cameras providing the tracking information that is used to control the active cameras.
Introduction
xv
Traditional computer vision algorithms are centralized. However, many applications in camera networks will require decentralized solutions, just as in traditional sensor networks. In Chapter 20 on distributed consensus algorithms for image-based localization, Tron, Terzis, and Vidal present such a method for addressing the problem of estimating the pose of an object and the cameras observing it. They also analyze the convergence properties of their proposed algorithm. Chapter 21 on the conditional posterior Cramer-Rao lower bound and its application in adaptive sensor management, Niu et al. introduce a performance metric for nonlinear, non-Gaussian tracking problems, and discuss how a proposed sensor management approach leads to estimation performance similar to that of information theoretic measures. PART V: Distributed Embedded Cameras and Real Time Video Analysis This part focuses on system aspects of video networks with five chapters on providing testbed environments, data collection on activities, new integrated sensors for airborne sensors, face recognition, and building sentient spaces. In order to perform research in the real-world, one needs a testbed environment for distributed video networks. In Chapter 22 on VideoWeb, Nguyen and Bhanu present a real-time distributed camera network laboratory that has been developed at the University of California, Riverside. After a brief overview of the laboratory, the chapter focuses on the problem of optimizing the network for surveillance with respect to the configuration of the network. The parameters of the network include video resolution, video compression, average frame rate, standard deviation of frame rate, and longest lag between the acquisition of two complete frames. Currently, human activity analysis is accomplished mostly with a single camera that is used to recognize simple motions and atomic activities performed by an individual. There exists a great need to collect a dataset depicting realistic scenarios in a multi-camera network environment. Chapter 23 by Denina et al. describes the details of an activity dataset (with available ground-truth) collected using the VideoWeb. This publicly available dataset involves multiple actors performing dozens of activities with a focus on non-verbal communication. Chapter 24 by Palaniappan, Rao and Seetharaman presents an eight-camera airborne sensor for the acquisition of wide-area persistent airborne video for the purposes of urban-scale activity analysis. The system is capable of acquiring 88 megapixels per frame over a wide field of view of 160 degrees or more at low frame rates of several Hertz. The chapter describes the architecture, processing challenges, and scene interpretation complexities of the system. Chapter 25 by Kulathumani et al. presents a decentralized network of embedded cameras for collaborative real-time face recognition. They consider scenarios that simulate typical corridors and passages in airports and other indoor public spaces. Cameras are built using off-the-shelf components. They quantify the system performance and the impact of multiple views on the accuracy of face recognition. Chapter 26 introduces a semantic approach for building sensor-based sentient spaces. Messegaur et al. describe SATWARE, a middleware environment for implementing such spaces. Their techniques explicitly model the semantics of the sentient
xvi
Introduction
space, and mechanisms to map sensor data into the higher-level abstractions defined by their view of the sentient space. The components of the model echo the entityrelation approach widely used in data management that permits SQL-like queries on the virtual sensors defined in the model. The work provides new opportunities for critical ancillary tasks in sensor networks, such as scheduling and recalibration. PART VI: Applications of Distributed Video Networks This part comprises four chapters on real-world applications of distributed video networks dealing with force protection, wide-area activities, port security, and recognition in nocturnal environments. In Chapter 27 on video analytics for force protection, Tu et al. present a set of video technologies relevant to the general problem of providing contextual awareness, with specific application to the challenge of force protection. The central thrust of the work is the integration of a variety of technologies, such as ground-based monitoring and aerial analysis for object detection and tracking, person detection from mobile platforms, as well as biometrics at a distance. The integration of these technologies also enables the inference of intent. An added benefit is the ability to infer the social networks that exist among the monitored subjects. Camera networks usually span large areas and, hence, it is necessary to understand activities that are spread over such areas. In Chapter 28 on recognizing activity structures in massive numbers of simple events over large areas, Rimey provides a taxonomy of various wide-area activity structures, and examples of how to visualize and understand them. Chapter 29 by Rasheed et al. discusses the challenges of distributed heterogeneous sensor networks for surveillance, focusing on the efficient processing of highresolution video data and discusses challenges and associated solutions for fusion, robustness to changing environmental conditions, user interface, and visualization. They present a case study of a wide-area video analysis system deployed at ports, including results for geo-registration, video processing, and event detection, plus a discussion of processing time and memory usage conditions. The recognition of humans and their activities in nocturnal environments finds several applications. In Chapter 30 Bourlai et al. present techniques for the extraction, processing and matching of biometrics under adverse night conditions given the available natural or artificial illumination. The employed sensors cover a wide range of the electromagnetic spectrum, including visible, near infrared, and thermal infrared. PART VII: Educational Opportunities and Curriculum Development This final part of the volume deals with educational considerations from the Electrical Engineering and Computer Science perspectives. In Chapter 31 Henderson discusses the educational opportunities and implications of research in video sensor networks. The field of sensor networks is delineated by a rich selection of topics with substantial pedagogical value, including
Introduction
xvii
such areas as modeling the physical world, complexity and correctness issues, distributed and parallel processing, networking, and data management. There are also unique opportunities for teaching system integration issues. In addition to surveying these possibilities, the chapter provides an overview of some courses currently being taught in this and related areas. To reiterate, this book presents a collection of the latest research efforts in an emerging research domain—Distributed Video Sensor Networks—which has numerous, high-impact applications. The volume covers a broad spectrum of work from video understanding to graphics and simulation, and from communication and control to embedded systems. The coverage also extends to practical applications, and on incorporating research and applications into the educational curriculum. We hope that this timely collection will prompt new investigations of the scientific and engineering aspects of the field and its applications, leading to further contributions.
Part I
Distributed Video Sensor Networks and Research Challenges
Chapter 1
Report on NSF/ARO/ONR Workshop on Distributed Camera Networks: Research Challenges and Future Directions Bir Bhanu and Amit Roy Chowdhury
Abstract Large-scale video networks are becoming increasingly important for a wide range of critical applications. The development of automated techniques for aggregating and interpreting information from multiple video streams in large-scale networks in real-life scenarios is very challenging. Research in video sensor networks is highly interdisciplinary and requires expertise from a variety of fields. The goal of this effort was to organize a two-day nationally recognized workshop in the domain of camera networks that brings together leading researchers from academia, industry and the government. The workshop was held at the University of California at Riverside on May 11–12, 2009. The workshop was attended by 75 participants. The workshop was sponsored by the US National Science Foundation, US Army Research Office and US Office of Naval Research. The workshop addressed critical interdisciplinary challenges at the intersection of large-scale video camera networks and distributed sensing, processing, communication and control; distributed video understanding; embedded real-time systems; graphics and simulation; and education. The recommendations of the workshop are summarized in the following order of topics: A. Video Processing and Video Understanding B. Simulation, Graphics, Cognition and Video Networks C. Wireless Video Sensor Networks, Communications and Control D. Distributed Embedded Cameras and Real Time Video Analysis E. Applications F. Educational Opportunities and Curriculum Development We include the workshop report in this book with the hope that it will catalyze a research agenda and interdisciplinary collaborations in large-scale camera B. Bhanu () · A. Roy Chowdhury Center for Research in Intelligent Systems, University of California at Riverside, Riverside, CA 92521, USA e-mail:
[email protected] A. Roy Chowdhury e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_1, © Springer-Verlag London Limited 2011
3
4
B. Bhanu and A. Roy Chowdhury
networks—an emerging field with significant and exciting scientific and technological challenges, which will set the stage for potentially transformative breakthroughs. The workshop website is http://videonetworks2009.cs.ucr.edu/. Keywords Video networks · Workshop report · Sensor networks · Graphics and simulation · Embedded systems · Communication and control sensors · Applications · Education and curriculum
1 Introduction This document summarizes the activities and recommendations of NSF/ARO/ONR sponsored Workshop on Distributed Video Sensor Network: Research Challenges and Future Directions. A multitude of real-world problems can be addressed now using video networks. Some of the examples are: • Monitoring civilian conditions such as disasters, evacuations, urban patterns, and transportation. • Monitoring natural habitats, colony collapse disorder, bird habitats. • Home care with personal robots in households. • Object-ensemble tracking: annotations of scene by type, and by contents (object classes, motion patterns, events and anomalies). • Webcams for science for active research in phenology, smart offices, classrooms and education outreach. The workshop discussed critical interdisciplinary challenges at the intersection of large-scale video camera networks and distributed sensing, processing, communication and control; distributed video understanding; embedded real-time systems; graphics and simulation; and curriculum development. The agenda for the workshop, the composition of various research groups and the list of participants are given at the end of this report. In the following we present the Workshop Report and topics for major research projects.
2 Workshop Recommendations The recommendations of the workshop are summarized in the following order of topics: A. Video Processing and Video Understanding (Group 1) B. Simulation, Graphics, Cognition and Video Networks (Group 2) C. Wireless Video Sensor Networks, Communications and Control (Group 3) D. Distributed Embedded Cameras and Real-Time Video Analysis (Group 4) E. Applications (Group 5) F. Educational Opportunities and Curriculum Development (Group 6)
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
5
2.1 A. Video Processing and Video Understanding This group was led by Profs. Rama Chellappa and Gerard Medioni. The group discussed the core issues associated with camera control, algorithms and architectures for distributed processing, and real-time/forensics scenarios and applications. The group also identified high risk and payoff areas and outreach efforts to other groups. The following are identified as new research directions within this group: • New algorithms for – Camera control (e.g., game theoretic approach). – Camera placement in dynamic environments, scheduling cameras for data acquisition to optimize tracking, and choosing the most informative cameras for a task. – Architecture for distributed processing, real-time vs. forensics; semantic models for group activities, theory of crowd behavior. Better understanding of physical phenomena, local context, and low-level image features. – Theory of visual information and representation to support the inference of scene properties (topological, geometric, photometric, dynamic) from images. – Central role of occlusion with respect to network topology and mobility, control authority vs. actionable information, and theory of active vision. – Scalable representations to handle increases in object complexity, count and dynamics; and sparse representations (e.g., compressive sensing). – Statistical learning and adaptation for data driven, scene specific models. • Several core issues are related with the algorithms. They include – Calibration. – Localization using non-line of sight sensors. – Tag and track. – Integration of multiple trackers. – Integration of modeling and tracking from sample to ensemble (statistics of crowds, objects). – Dynamic compensation for imaging inconveniences (using multiple sources). – Trade-off between camera mobility/fixed cameras. – Distributed vision algorithms based on consensus approaches. – Power aware vision algorithm. – Inference (decision, analysis and classification) in high-dimensional (multiple properties, objects and dynamics) time series. – Efficient and robust integration of information through the network. – Parameter free grouping and clustering. Additional items were building models of complex objects from parts, modeling relationships among objects and phenomena, automatic identification of informative features, modeling complex motions, and advanced sensing to reduce processing, integration of multiple cues and strategies for reducing the need for low-level coordination among cameras. Risks associated with the algorithms for distributed video sensor networks include calibration, complexity of nuisance factors (illumination, deformations, etc.),
6
B. Bhanu and A. Roy Chowdhury
non-linear (or even linear) increase in complexity with problem size, fragility of low-level vision algorithms, new potential issues due to integrating visual data through a network with communication constraints, and run-time issues such as the synchronization of multiple video streams.
Calibration Risk and Payoffs The calibration problem has been studied for a long period of time but not solved for automated calibration of the network. Algorithms for self-calibration and opportunistic use of environmental data can address this need. If done right, calibration simplifies all data fusion problems and increases the accuracy of registration, recognition, tracking etc.
Nuisance Factors Risks and Payoffs Complexity of nuisance factors is caused by environmental conditions, occlusion and scene clutter. New directions are to improve current representations and decrease the dependency on extensive training sets to learn known nuisances. On the other side, a larger amount of data, obtained intelligently, can overcome many of the nuisance factors. Developing representations that are designed to be insensitive/invariant to nuisance factors simplifies learning (no extensive datasets necessary to “learn away” spurious variability). Further, the development of a theory of representation would enable provable bounds, given specific sets of assumptions. Intersection/Outreach of Video Processing and Video Understanding (Group 1) to other groups (a) Simulation, Graphics, Cognition and Video Networks (Group 2) Video understanding results do the following: • Feed into graphics simulator. • Populate the synthetic world as vision can generate descriptors of objects and their behaviors, information on gender, height, gait parameters, behavior patterns of humans, dimensions of vehicles, their traffic flows, etc. • Immersive environments where real video and synthetic environments can be integrated. • Synthetic environments can produce data for validating vision algorithms. and graphics provides the context for designing robust vision algorithms and interfaces; and builds more on computational photography. (b) Wireless Video Sensor Networks, Communications and Control (Group 3) Understanding the impact of compression on vision algorithms (quality loss, use of metadata); finding the impact of transmission error on vision system performance (track failures, maintaining identity); developing application-aware compression and transmission; and collaborative processing algorithms.
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
7
(c) Distributed Embedded Cameras and Real-Time Video Analysis (Group 4) Understanding trade-offs between local, collaborative and centralized processing (cost-benefit of implementing on embedded platforms, computation vs. bandwidth (information exchange); studying mobility issues; codesigning hardware and vision software (selecting algorithms for hardware, GPU, etc.); building robustness to camera and network failures; designing power aware vision algorithms; and designing novel sensors. At the system level, issues include the following: • Scalability (trade-off accuracy to computational time). • Architectural issues (distributed implementations, trade-offs in bandwidth, cost, accuracy (depends on kinds of questions posed in server/edge device)). • Architecture—link to Neuroscience/Psychology. • Performance models of distributed sensing, intelligence and control. • Human factors issues and new concepts when dealing with distributed sensing, intelligence and active control, novel visualizations; algorithmic issues (develop algorithms which do not need manual tuning).
2.2 B. Simulation, Graphics, Cognition and Video Networks This group was led by Profs. Demetri Terzopoulos and Chuck Dyer. The group discussed the issues of analysis in camera networks using simulations. Since there are practical challenges in installing camera networks for research purposes and getting access to the real-life data, simulations through computer graphics methods can be very useful in understanding the performance of these methods. Moreover, simulation studies can be used to create virtual scenarios through active camera control. Such simulations require models of the areas, environmental models and dynamic models of people and vehicles. The group discussed a number of advances in computer graphics that would enable this research direction. They included simulation in camera networks, realistic renderings of humans and their activities, tele-immersion systems, and ubiquitous displays. The issue of distributed processing and architectures was also discussed. Some researchers raised the issues of camera control, tracking and biometric recognition. The following were identified as future research directions. 1. The use of biometrics in simulations has not been studied, but they are obviously important in scene analysis in camera networks. Information from the sensor network can be used to obtain higher quality simulations since this information can be obtained from different views and over time. Also, realistic biometric simulations can be used in the scene analysis algorithms for tracking and control. 2. The issue of view-invariant matching across camera networks, which are spread over wide areas, was considered to be a challenging problem.
8
B. Bhanu and A. Roy Chowdhury
3. The issue of building intelligent architectures that would be adaptive to different representations, categories and constraints was discussed. This would be especially important in efficiently organizing the infrastructure prior to deployment. 4. Distributed processing and camera control is another area of future work in this domain. 5. Simulations must be application specific since they have specific requirements. For example, tele-immersion systems have hard time constrains, while surveillance systems can have weaker restrictions.
2.3 C. Wireless Video Sensor Networks, Communications and Control This group was led by Prof. Antonio Ortega and Prof. Venkatesh Saligrama. The group discussed issues related to infrastructure, synthesis, design and integration of low-level tools leading to a system-level understanding of the video network and its functionalities. The inputs to the system are the low-level data acquisition and processing methods, algorithms capable of analyzing the low-level data like image/signal processing algorithms, the design constraints and objectives, the quality metrics that need to be satisfied and evaluation datasets. The outcome of this system-level understanding will be models (either deterministic or stochastic), protocols that describe the communication between the different sensor nodes, centralized and distributed algorithms for system-level analysis, and overall performance analysis of the entire system. A number of key research problems need to be solved to achieve all these outcomes. The group identified the following areas where interactions with the other groups were necessary. The system analysis is dependent on the application domain, specifically the video analysis methods. Simulated environments could be helpful for their analysis if the system-level issues, e.g., networking constraints, could be built into the simulation methods. The proposed ideas need to be analyzed for possible realtime applications. Finally, how the proposed methods will be built into major products and standards was an issue that was discussed and it was felt that these applications can provide the ultimate metrics for quality of service and performance analysis. The group also brought up the issue of how video sensor networks can provide unique examples in teaching different core systems theory courses like signal processing, information theory, controls, networking, etc. The general challenges in the system-level understanding are the following: • Scalability, which includes the ability to deal with high data rates, latency and large number of nodes; • Computation-communication trade-off to decide on distributed vs. centralized processing; • Spatial coordination and its potential benefits compared to the coordination overhead; • Identification of generic tools that can be used across application domains;
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
9
• Quality metrics for Resources vs. Analysis quality; • Robustness of the overall system to faults, losses and security threats. Some desired characteristics of the solution methods were also identified. • For the physical layer, the group felt that new MAC layer tools can be developed for video sensor networks. • The networking protocols should be capable of handling distributed architectures, provide different levels of reliability for different data types, and lead to cooperative approaches for sensing, processing and communication. • The ultimate goal of the system will be optimization of quality vs. available resources (power, bandwidth, time). A major question will be to decide what are the desirable metrics for measuring “distortion” in the system and the group felt that it was application specific. There is very limited work on “analysis-based” metrics. • The system analysis can be lossy for the data, but lossless for the information. • Application specific compression techniques need to be studied. • Information driven adaptive sensing, followed by distributed fusion of the sensed data, will be a major focus area for this research. • Collaboration between the different entities is necessary and leads to a number of sub-problems like opportunistic communication, parallel processing, distributed algorithms, distributed control and resource management, and the interaction between control and communication.
2.4 D. Distributed Embedded Cameras and Real Time Video Analysis This group was led by Dr. Hamid Aghajan and Dr. Paul Brewer. Its focus was to analyze the issues related to developing smart cameras capable of performing the tasks necessary in a distributed video sensor network (DVSN). The group identified the issues in this regard with respect to the other groups. • For Group 1 (Video Processing and Video Understanding), the challenges identified were related to developing efficient architectures for sensor processing so that the algorithms could run in real time. It was raised that most video understanding algorithms are computationally intensive and, therefore, it is necessary to focus on how to make them more efficient. • Another issue in this regard was the methodology that would be followed in creating the metadata that could be exchanged among sensors. • With respect to Group 2 (Simulation, Graphics, Cognition and Video Networks), the issues were related to architectures that would allow such simulations to run. It was, however, felt that this might not be a critical issue since simulations were being done offline and using powerful GPUs. • It was felt that the real-time processing issues were closely related to the networking, control and communication aspects. The network protocols would directly
10
B. Bhanu and A. Roy Chowdhury
affect the smart camera architectures. Communication resources would dictate the trade-off between local and distributed processing, which would, in turn, affect the architecture. Control mechanisms have to work in real time and, thus, need significant computational power. • With the Applications group, the main issues would be deciding upon the platforms, operating system and architectures.
2.5 E. Applications The applications group was led by Prof. Glenn Healy and Dr. Bogdan Matei. This group identified a number of application areas of video sensor networks. Some of the examples mentioned are given below. • One interesting problem was recognizing activity patterns over large areas from a number of simple events. This requires advanced video analysis techniques in camera networks. • The integration of biometrics with wide area surveillance was considered another interesting problem domain. This is especially important because it is hard to track individuals over large areas without identification information. • Another application area was in the health care industry, especially assisted living facilities. • In all of these, the integration of video with other sensory signals was considered critical. • The group members expressed concerns about the real-time abilities of video analysis algorithms and felt that closer interaction was necessary between the low-level signal analysis, communication protocols and real-time implementation.
2.6 F. Educational Opportunities and Curriculum Development This group was led by Prof. Sharad Mehrotra and Prof. Jenq-Neng Hwang. The related talks were also given by Prof. Tom Henderson and Prof. Narendra Ahuja. The group identified the challenges associated with distributed video sensor networks because of its interdisciplinary nature and offered potential solutions. Several ideas for interdisciplinary curriculum were discussed. The highlights of this group are: • The field of Distributed Video Sensor Network (DVSN) is interdisciplinary. It includes many aspects of computer vision, pattern recognition, distributed algorithms, signal processing, graphics, simulation, verification and validation, sensors, robotics, operating systems and networking, hardware and embedded systems, data management, analysis of large datasets, cognitive systems, ethics, intellectual property rights, science and math and technical writing. Each of these
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
•
•
•
•
•
11
technical areas has its core set of problems, techniques, approaches, biases, languages, research methodologies, acceptable validation mechanisms, etc. One idea is to look at the intersection of disciplines involved and expand disciplinary curriculum with concepts/ideas/abstractions from different fields. The associated challenges are: What are the core disciplinary concepts? How do we create common understanding of different areas, while being flexible enough to meet the needs of educators from different perspectives? One possible solution is to create testbeds highlighting different application areas such as the UCR VideoWeb camera network testbed and Responsphere at UCI. There could be a testbed for open access similar to ORBIT project sponsored by NSF Network Research Testbed for large-scale wireless networks with remote data collection and remote algorithms porting. Building, usage, validation can all be incorporated as strategies to improve graduate education. Note that testbeds are not easy to share and require dedicated staff and researchers and economic models for sustainability and sharing of testbed are needed. Some solutions are NSF Site REU, NSF IGERT, DoD MURI, Summer school, Educational Programs. Currently the sustained funding for infrastructure and its maintenance is very limited. NSF may consider modifying CRI, MRI programs. The community can help the funding agencies by promoting the field through establishing the needs of DVSN. Another possible solution is to develop a systems approach to education. Experts design micro-modules in their areas of expertise. Modules consist of course slides, test data, etc. Modules can be combined to create courses offering with different perspectives. Modules are refined over time. This will allow the shared burden with huge payoffs at the community level. Another possible solution is the creation of “grand challenge” tasks similar to ones done in robotics, web search, etc. In this regards calibrated, time synchronized, and ground truth labeled benchmark videos (e.g., VideoWeb of UCR, Human ID of Southampton, IEEE PETS dataset) will be helpful. Another possible solution is to develop Network Simulator-2 like camera network simulators. The simulator will allow for camera placement and specification, event building and scheduling, protocol for information exchange and an integrated display. Other ideas for introducing DVSN relevant material into curriculum are: – Exploit electives to influence curriculum e.g., adding project oriented special course. – Create interdisciplinary classes with projects requiring students from different disciplines to work together and make courses interesting for students. – Revise existing classes with new relevant material. – Create opportunities to improve course quality and allow participation in “large” multi-university courses.
Teaching methodology must adapt to how students learn today. Include active learning techniques such as gaming, novel and interactive applications, virtual reality. Interested schools need to create summer schools/courses/seminars and set the prerequisites to include right set of courses.
12
B. Bhanu and A. Roy Chowdhury
3 Suggested Major Research Topics 3.1 Topic 1: Robust Scalable Video Networks for Wide Area Analysis Disciplines Involved Image Processing, Computer Vision, Pattern Recognition, Graphics, Databases, Learning, Real-time systems, Applications Research Concentration (1) Detecting, tracking and recognizing objects of interest in changing environmental conditions. (2) Extraction and manipulation of real-time sensory data for wide area object and activity recognition. (3) Simulating real environments for increased robustness and scalability of vision algorithms and systems. (4) Integrating analysis-by-synthesis and synthesis-by-analysis in a learning framework for increased robustness. (5) Strategies for synthesizing and analyzing data for many sensors and their performance characterization. (6) Recognizing objects and evolving activities over time and their interaction with databases.
3.2 Topic 2: Active, Distributed and Communication Aware Video Sensor Networks Disciplines Involved Image Processing, Computer Vision, Pattern Recognition, Distributed Computing, Communications and Control, Real-time Systems Research Concentration (1) Develop distributed algorithms for image processing, computer vision and pattern recognition to detect, track and recognize objects and actions. (2) Develop strategies for real-time control of sensors for robust and efficient feature acquisition, recognition and targeting. (3) Model communication constraints for operation of camera networks in a theater of operations. (4) Perform information theoretic analysis for dynamic optimization of networks for practical applications such as tracking, recognition and targeting. (5) Perform system theoretic analysis that includes aspects of communication, control, image processing, computer vision and pattern recognition, and develop fundamental theories and practical algorithms to achieve them.
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
13
3.3 Topic 3: Large-scale Heterogeneous Sensor Networks for Wide Area Analysis Disciplines Involved Image Processing, Computer Vision, Pattern Recognition, Distributed Computing, Fusion Architecture, Statistics, Performance Characterization, Aerial and Ground Vehicles, Applications Research Concentration (1) Physics and statistics-based multisensory fusion algorithms that include control and communication among various ground and aerial sensors. (2) Coordination of ground and aerial platforms for wide area robust performance for detection, tracking and recognition. (3) Integration of global trajectory analysis with local processing algorithms for detection, tracking and recognition. (4) Real-time sensing, monitoring and control for recognition and targeting. (5) Distributed fusion architectures for real-time processing, communication and control at different abstraction levels. (6) Analysis of complex behavior over long term. (7) Develop strategies for dynamic optimization of available resources and their effects on performance.
Appendix 1: List of Attendees in Alphabetical Order Note: All the abstracts by participants are available on the workshop website: http://videonetworks2009.cs.ucr.edu/. J.K. Aggarwal, The University of Texas at Austin Abstract: Sensor Networks and Recognition of Human Activities Hamid Aghajan, Stanford University Abstract: Vision: Ideas for Enabling Ambient Intelligence and Serving Social Networks Narendra Ahuja, UIUC Abstract: Image Representation for Image Understanding Ian Akyildiz, Georgia Tech Abstract: Research Challenges for Wireless Multimedia Sensor Networks Rick Allan, BBN Kevin Almeroth, UC-Santa Barbara
14
B. Bhanu and A. Roy Chowdhury
Norman Badler, UPenn Abstract: Simulating a Functional Populace Ruzena Bajcsy, UCBerkeley Abstract: Active Perception: Past, Present, and Future Azer Bestavros, Boston University Abstract: Virtualization and Programming Support for Video Sensor Networks Bir Bhanu, UCR Abstract: VideoWeb: Design of a Wireless Camera Network for Real-time Monitoring of Activities Paul Brewer, ObjectVideo Abstract: Maritime Persistent Surveillance Mark Campbell, Cornell University Abstract: Sensing, Cooperation and Control in Networked Systems Rama Chellappa, UMD Abstract: Exploiting Sparsity, Geometry and Statistics for Video Processing and Understanding Using a Camera Network Jie Chen, University of California Abstract: Best Achievable Tracking Performance Under Limited and Constrained Information Feedback Hui Cheng, Sarnoff Corporation Abstract: Real-Time Entity Tracking Using Wide Area Surveillance Videos Bojan Cukic, West Virginia University Abstract: Fault Tolerant Sensor Networks for Border Activity Detection Liyi Dai, US Army Research Office Raju Damarla, US Army Research Laboratory Larry Davis, UMD Abstract: Event Modeling and Recognition in Camera Networks Jim Davis, Ohio State University Abstract: Control, Registration, and Exploitation of Video Sensor Networks Sudhir Dixit, Center for Internet Excellence Abstract: Challenges of Wireless Communication in Video Sensing
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
15
Chuck Dyer, University of Wisconsin-Madison Abstract: Tracking in Camera Networks Using Joint Projective Invariants Jay Farrell, U of CA Riverside John Fisher, MIT CSAIL Abstract: Information-driven Inference Under Resource Constraints Arun Hampapur, IBM Abstract: Media Analytics Research at IBM Mary Ann Harrison, WVHTC Foundation Abstract: Tactical Analysis of Video Imagery Parag Havaldar, SONY Pictures, Imageworks Abstract: Performance Driven Character Animation on a Real Production Set Using Multiple Cameras Glenn Healey, UC Irvine Abstract: Processing Multispectral/Hyperspectral Video Tom Henderson, University of Utah Abstract: Cognitive Video Sensor Networks Jenq-Neng Hwang, University of Washington, Seattle Abstract: Intelligent Surveillance and Event Understanding in Distributed Embedded Camera Networks of a Large Scale Community Ted Isaacson, ONR/MTCSC Ioannis Kakadiaris, University of Houston Abstract: Face Recognition: Your Face is Your Password Behzad Kamgar-Parsi, ONR Aggelos Katsaggelos, Northwestern University Abstract: Content-based Scalable Video Streaming in Video Sensor Networks Iouri Kompaniets, Physical Optics Corporation Abstract: Advanced Hyperspectral Zoom Optics Sensor Rick Kremer, Logostech Martin Kruger, Office of Naval Research
16
B. Bhanu and A. Roy Chowdhury
Vinod Kulathumani, West Virginia University Abstract: Collaborative Face Recognition Using Network of Smart Cameras Mohan Kumar, The University of Texas at Arlington Abstract: Collaborative Virtual Observation (CoVO) in Dynamic Environments Aditi Majumder, UCI Abstract: Ubiquitous Displays via a Distributed Framework of ProjectorCamera Systems Bogdan Matei, Sarnoff Corporation Abstract: Multi-Camera Tracking and Data Association Shean T. McMahon, Physical Optics Corporation Abstract: Advanced Hyperspectral Zoom Optics System Gerard Medioni, University of Southern California Abstract: Distributed Vision: Algorithms, Scalability, Forensics Sharad Mehrotra, UCI Abstract: SATWARE: A Semantic Enhanced Middleware and Database System for Sentient Spaces Scott Midkiff, National Science Foundation Vassilios Morellas, UMN Abstract: Linking Video Information Across Cameras Thinh Nguyen, Oregon State University Abstract: Towards Building a Robust Video Sensor Network Antonio Ortega, University of Southern California Abstract: Challenges in Practical Distributed Video Compression Robert Pless, Washington University in St. Louis Abstract: Passive Vision: Observing the World While Sitting Still T.V. Prabhakar, Indian Institute of Science Abstract: An Energy Efficient Distributed Algorithm for Image Slicing in WSNs Ranjit Pradhan, Physical Optics Corporation
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
17
R.R. Venkatesha Prasad, Delft University of Technology Abstract: An Energy Efficient Distributed Algorithm for Image Slicing in Wireless Sensor Networks Faisal Qureshi, UOIT Abstract: Proactive Camera Control for Collaborative Sensing V. Ramesh, Siemens Abstract: Robust Video Understanding Systems—Review and Challenges Raghuveer Rao, Army Research Laboratory Chinya Ravishankar, UC Riverside Ray Rimey, Lockheed Martin Abstract: Recognizing Activity Structures in Massive Numbers of Simple Events Over Large Areas Amit Roy Chowdhury, University of California, Riverside Abstract: Video Understanding in Distributed Active Camera Networks Maya Rubeiz, Office of Naval Research Venkatesh Saligrama, Boston University Abstract: Video Analytics in Multicamera Networks Wes Snyder, NCSU Abstract: Shape Recognition Based on Accumulators Stefano Soatto, UCLA Abstract: From Shannon to Gibson: Actionable Information in Video Bi Song, UCR Abstract: Scene Analysis, Control and Communication in Distributed Camera Networks Tanveer Syeda-Mahmood, IBM Almaden Research Center Demetri Terzopoulos, University of California, Los Angeles Satish Tripathi, University at Buffalo, SUNY Mohan Trivedi, University of California at San Diego Abstract: Distributed Video Networks in Intelligent Vehicles and Transportation Systems
18
B. Bhanu and A. Roy Chowdhury
Yanghai Tsin, Siemens Corporate Research Peter Tu, GE Abstract: Video Analytics from Homeland Protection to Theft Prevention and Crowd Analysis Pramod Varshney, Syracuse University Abstract: Sensor Selection and Information Fusion in Sensor Networks Rene Vidal, Johns Hopkins University Abstract: Distributed Scene Understanding Wayne Wolf, Georgia Tech Abstract: Distributed Smart Cameras Ming-Hsuan Yang, University of California at Merced Abstract: Visual Tracking with Online Multiple Instance Learning Allen Y. Yang, UCB Abstract: Multiple-View Object Recognition in Band-Limited Distributed Camera Networks
Appendix 2: Groups and Group Leaders
Group 1: Video Processing and Video Understanding Group Co-Leaders: Rama Chellappa (UMD), Gerard Medioni (USC) Participants: J.K. Aggarwal also group 2 Narendra Ahuja also group 6 Robert Pless Vinod Kulathumani Amit Roy Chowdhury also group 3 Stefano Soatto also group 2 Rene Vidal Ming-Hsuan Yang
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
19
Bir Bhanu Jim Davis also group 2 Larry Davis also group 6 V. Ramesh also group 4 Wes Snyder also group 4 Bi Song Yanghai Tsin Allen Y. Yang also group 4 Group 2: Simulation, Graphics, Cognition and Video Networks Group Co-Leaders: Demetri Terzopoulos (UCLA), Chuck Dyer (U. Wisconsin) Participants: Ruzena Bajcsy also group 6 Jim Davis also group 1 Ioannis Kakadiaris Aditi Majumder Faisal Qureshi Avideh Zakhor also group 6 J.K. Aggarwal Norman Badler Parag Havaldar also group 5 Tom Henderson also group 6 Stefano Soatto also group 1 Group 3: Wireless Video Sensor Networks, Communications and Control Group Co-Leaders: Antonio Ortega (USC), Venkatesh Saligrama (BU) Participants: Ian Akyildiz Jie Chen Mohan Kumar Thinh Nguyen Pramod Varshney also group 6 Mark Campbell Raju Damarla Jay Farrell John Fisher also group 6 Aggelos Katsaggelos also group 6 Amit Roy Chowdhury also group 1 Satish Tripathi
20
B. Bhanu and A. Roy Chowdhury
Group 4: Distributed Embedded Cameras and Real Time Video Analysis Group Co-Leaders: Hamid Aghajan (Stanford), Paul Brewer (Object Video) Participants: Jenq-Neng Hwang also group 6 Sharad Mehrotra also group 6 Ray Rimey also group 5 Wes Snyder also group 1 Mohan Trivedi also group 6 Wayne Wolf (phone/internet) Allen Y. Yang also group 1 Rick Allan Azer Bestavros Rick Kremer V. Ramesh also group 1 Raghuveer Rao Group 5: Applications Group Co-Leaders: Glenn Healey (UCI), Bogdan Matei (Sarnoff) Participants: Bojan Cukic Mary Ann Harrison Iouri Kompaniets R. Prasad/T. Prabhakar Parag Havaldar also group 2 Ted Isaacson Shean T. McMahon Ranjit Pradhan Ray Rimey also group 4 Peter Tu also group 6 Group 6: Educational Opportunities and Curriculum Development Group Co-Leaders: Sharad Mehrotra (UCI), Jenq-Neng Hwang (UW) Participants: Narendra Ahuja also group 1 Ruzena Bajcsy also group 2 Larry Davis also group 1 John Fisher also group 3
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
21
Tom Henderson also group 2 Aggelos Katsaggelos also group 3 Mohan Trivedi also group 4 Peter Tu also groups 5 Pramod Varshney also group 3 Avideh Zakhor also group 2
Appendix 3: Talks with Titles and Presenters Plenary Presentations Past, Present and Future of Active Vision, Ruzena Bajcsy (UCB) Past, Present and Future of Motion Analysis, J.K. Aggarwal (UT Austin) Event Modeling and Recognition in Camera Networks, Larry Davis (UMD) Robust Video Understanding Systems—Review and Challenges, V. Ramesh (Siemens) Simulating a Functional Populace, Norm Badler (UPenn) Information-Driven Inference under Resource Constraints, John Fisher (MIT) Research Challenges for Wireless Multimedia Sensor Networks, Ian Akyildiz (GTech) Content-based Scalable Video Streaming in Video Sensor Networks, Aggelos Katsaggelos (Northwestern) Sensing, Cooperation and Control in Networked Systems, Mark Campbell (Cornell) snBench: Virtualization and Programming Support for Video Sensor Networks, Azer Bestavros (BU) Video Analytics from Homeland Protection to Theft Prevention and Crowd Analysis, Peter Tu (GE) VideoWeb: A Network of Wireless Video Cameras, Bir Bhanu (UCR) Education Opportunities: Computer Science Perspective, Thomas Henderson (UU) Education Opportunities: Electrical Engineering Perspective, Narendra Ahuja (UIUC) ONR Programs, Martin Kruger (ONR) NSF Opportunities, Scott Midkiff (NSF) ARO Opportunities, Raghuveer Rao (ARO)
22
B. Bhanu and A. Roy Chowdhury
Group Presentations to the Entire Workshop and Discussions Group 1: Video Processing and Video Understanding Group 2: Simulation, Graphics, Cognition and Video Networks Group 3: Wireless Video Sensor Networks, Communications and Control Group 4: Distributed Embedded Cameras and Real Time Video Analysis Group 5: Applications Group 6: Educational Opportunities and Curriculum Development Recommendation by the Groups for Research Directions and Educational Opportunities/Curriculum Development Group Presentations and Discussions Group 1: Video Processing and Video Understanding Exploiting Sparsity, Geometry and Statistics for Video Processing and Understanding Using a Camera Network, Rama Chellappa (UMD) Distributed Vision: Algorithms, Scalability, Forensics, Gerard Medioni (USC) Presentations by other group participants (see abstracts on the website) Group 2: Simulation, Graphics, Cognition and Video Networks Virtual Vision: A Simulation Framework for Camera Sensor Networks Research, Demetri Terzopoulos (UCLA) Tracking in Camera Networks Using Joint Projective Invariants, Chuck Dyer (U. Wisconsin) Presentations by other group participants (see abstracts on the website) Group 3: Wireless Video Sensor Networks, Communications and Control Video Analytics in Multicamera Networks, Venkatesh Saligrama (BU) Challenges in Practical Distributed Video Compression, Antonio Ortega (USC) Presentations by other group participants (see abstracts on the website) Group 4: Distributed Embedded Cameras and Real Time Video Analysis Vision: Ideas for Enabling Ambient Intelligence and Serving Social Networks, Hamid Aghajan (Stanford) Maritime Persistent Surveillance, Paul Brewer (Object Video)
1 Report on NSF/ARO/ONR Workshop on Distributed Camera Networks
23
Presentations by other group participants (see abstracts on the website) Group 5: Applications Processing Multispectral/Hyperspectral Video, Glenn Healey (UCI) Multi-Camera Tracking and Data Association, Bogdan Matei (Sarnoff) Presentations by other group participants (see abstracts on the website) Group 6: Educational Opportunities and Curriculum Development Educational Opportunities and Curriculum Development, Sharad Mehrotra (UCI), Jenq-Neng Hwang (UW) Presentations by other group participants (see abstracts on the website)
Part II
Video Processing and Understanding
Chapter 2
Motion Analysis: Past, Present and Future J.K. Aggarwal
Abstract The subject of motion has been the center of interdisciplinary studies since the time when Zeno posed his paradox circa 500BC. However, computer vision, the use of a camera and a computer to recognize objects, people and/or events automatically, is a relatively young field of research. Its development began in the early 1960s; however, it has matured fairly quickly. Today, it is contributing to the solutions of some of the most serious societal problems. Motion analysis of a sequence of images is an important part of computer vision. This chapter briefly presents the contributions to motion analysis from other fields followed by the computer vision-based analysis of motion from a sequence of images. Analysis and understanding of images based on both feature tracking and optical flow estimation are presented. Early works focused on the computation of structure from motion of objects from a sequence of images via point features. This was followed by the computation of optical flow to characterize motion. Applications today focus on the monitoring of traffic, providing guidance to a motorist in terms of his/her position relative to traffic lanes and traffic ahead, and inspection of complicated three-dimensional industrial parts, to mention a few. Research focus has shifted from inanimate objects to people, for example monitoring people and their activities in public places or monitoring activities from an unmanned aerial vehicle. These applications are dominating the research scene through the belief that computer vision/motion analysis can contribute to the solution of societal surveillance and biometric problems. The chapter ends with a discussion of the future directions of research in motion analysis and possible applications.
Keywords Motion · Tracking · Human activities · Past · Present · Future J.K. Aggarwal () The University of Texas at Austin, Austin, TX, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_2, © Springer-Verlag London Limited 2011
27
28
J.K. Aggarwal
1 Introduction to Motion: An Early History Timely response to moving objects is of critical importance; at times, it is a matter of life and death. It is natural that the mammalian visual system has evolved specialized neural hardware to detect and respond to an ever-changing environment full of stationary objects, moving objects, and objects that appear to move due to the movement of our bodies or eyes. Thus, it is not surprising that human interest in motion as a subject of scientific thought emerged long ago. This interest has spanned diverse disciplines including philosophy, psychophysics, neurobiology, robotics, computer graphics, and, of course, computer vision. Motion and human vision has been the subject of intense inquiry both at experimental and theoretical levels in computer vision research. Interest in computer vision itself began in the early 1960s and research in computer vision-based analysis and understanding of motion in a sequence of images began around the early 1970s. The publication of the paper by Johansson [23] provided significant impetus for the study of motion in several diverse disciplines. Author’s interest in motion analysis began around that time. The paper “Computer Analysis of Moving Polygonal Images,” in IEEE Transactions on Computers [4] is one of the earliest papers published on motion in computer vision literature. This paper was motivated by the desire to track objects; more details follow in Sect. 3. Author’s interest in tracking three-dimensional objects, and Norman Badler’s interest in movements of dance led to the 1979 Workshop on Computer Analysis of Time Varying Imagery in Philadelphia. It was one of the first meetings devoted to research on machine perception of motion [7]. A subsequent workshop of note includes the 1982 NATO Advanced Study Institute on Image Sequence Processing and Dynamic Scene Analysis directed by T.S. Huang [22]. There were several other workshops sponsored by the IEEE Computer Society and by the Association for Computing Machines (ACM). In addition, several other workshops were held outside the USA, for example those chaired by Vito Cappellini in Florence, Italy. Shimon Ullman at MIT submitted his dissertation in 1977 on the motion of objects [38]. The 1980s were a very exciting period for research in motion analysis and understanding. Berthold Horn (with Brian Schunck) published his seminal paper on Optical Flow in 1981 [21] and published his book on Robot Vision in 1986 [20]. In 1988, Worthy Martin and the author edited a collection of papers on various facets of motion entitled Motion Understanding: Robot and Human Vision [26]. The authors included Shimon Ullman, Thomas Huang, Hans Nagel, and David Fleet and others. In addition, several applications, which are the lifeblood of any research, were also presented in the book. Today, the diverse applications motivating motion research include medicine, tomography, autonomous navigation, communications, television, video-conferencing, unmanned aerial vehicle imagery, athletics, dance choreography, and meteorology. Given the constraints of space and time, it is difficult to cover all aspects of motion analysis and understanding. In this chapter, the author presents an overview of the past, present and future of research in Motion and related issues. In Sect. 2, highlights of motion research from other disciplines are briefly mentioned. Section 3 covers some of the beginnings of computer vision-based research in motion.
2 Motion Analysis: Past, Present and Future
29
Section 4 is devoted to Optical Flow. Section 5 describes studies in recognition of human activities. Section 6 presents a view of the future directions of research and applications.
2 Motion: Highlights from Philosophy, Psychology and Neurobiology Understanding motion, the movement of objects through time and space, has interested humans for at least 2500 years. In the 5th century BC, the Greek philosopher Zeno of Elea raised the first major scientific question to inquire regarding the nature of motion: Is it discrete or continuous? He presented the “paradox of arrow” to point out the logical problem encountered when motion is conceived to be the repositioning of an object at successive time instances in a given spatial coordinate system. While some modern philosophers like Bertrand Russell [33], and certain aspects of theory of special relativity as detailed on the Math Pages website [27] have “resolved” Zeno’s paradox, the point remains that the phenomenon of motion has played a long and pivotal role in the evolution of scientific thought. Aristotle, more than two millennia ago, posed questions about the nature of motion. He made one of the first references to a motion illusion: the motion after effect. The motion after effect is created when the human visual system receives a prolonged stimulation from a moving field in a particular direction like a swift river or a waterfall. When one views a stationary object after one’s vision has accommodated to the motion field, the stationary object appears to move in the opposite direction. This illusion was rediscovered by Addams in 1834 [1]. While Aristotle talks about a swiftly moving river, Addams mentions a waterfall. The human vision system and its relationship to motion have been studied intensely by psychophysicists of various persuasions. Based on the findings presented in the excellent review article by Derrington et al. [12], one may observe that there are two distinct visual subsystems that analyze retinal motion, a feature-tracking subsystem and a motion-sensing subsystem.1 Paraphrasing Derrington et al.: A feature-tracking motion subsystem tracks features or objects across multiple frames. A motion-sensing subsystem consists of two components: A “first order” motion filtering system is the substrate of our direct sense of motion. It consists of local orientation-selective motion sensors that respond to the motion of luminancedefined features in a particular direction. The location, speed and direction of motion of a moving object are encoded in the visual system by the identity of the sensor that responds to it. A “second order” motion system is based on sensors that respond to motion of patterns defined by spatial modulation of local contrast or other image properties. 1 The
interested reader may note that the first physiological model of motion detection based on the “Autocorrelation, a principle for evaluation of sensory information by nervous system” by Reichardt W. (1961) is presented at the website: http://en.wikipedia.org/wiki/Motion_perception.
30
J.K. Aggarwal
The nature of models used for motion detectors in the visual system has been the subject of considerable thought and research. The most commonly accepted models are by Adelson and Bergen [2] and Watson and Ahumada [42]. There is no clear agreement on how the above mechanisms give rise to vection and optical flow. Also, there is disagreement on the mode of operation of the motion filters and how their outputs are combined to resolve the direction of 2D motion. Moreover, it is difficult to specify the differences in performance between the feature-tracking mechanisms and the motion-sensing mechanisms. Another class of study that addresses the visual perception of motion is in the field of neurobiology. The work of Semir Zeki [45] led to the acceptance of the concept of functional specialization in the visual cortex, where color, form, and motion are processed separately. Four parallel systems handle different attributes of vision: one for motion, one for color, and two for form. It is important to note that the two channels that are computationally most distinct from one another are motion and color systems. For the human visual-motion system, the pivotal area of the brain is called V5, a lesion in this area produces akinetopsia where patients neither see nor understand the world in motion. At rest, objects appear perfectly visible, whereas motion relative to them causes the object to vanish. Motion and human understanding of motion have been studied by a number of different disciplines. Motion is a fascinating subject with a number of interesting unsolved problems. This fascination with motion and the human visual system is likely to continue far into the future!
3 Motion in Computer Vision: The Beginnings Having presented some of the highlights from philosophy, psychology and neurobiology, let us consider the beginnings of motion studies in computer vision. The problem of determining cloud motion from a sequence of satellite images motivated the paper [4]. The difficulty of approximating clouds and cloud motions led to the idealization of polygonal figures moving in planes. The assumed model of cloud motion consists of polygons at different levels moving independently and the observed view of cloud motion is the overlapping view of various polygonal figures at various layers. The polygons are assumed to be rigid in order to determine the linear and angular velocities of the polygons, and to decompose the sequence of images into component sequences of images. In reality, clouds are neither polygonal nor rigid; indeed, clouds change shape, appear and disappear. Nevertheless, the study revealed a number of interesting and fundamental relations concerning overlaying polygonal figures. The study yielded one program to generate a sequence of images from the moving polygons in different layers and another program (analysis program) to systematically analyze the generated overlay of polygonal image sequences. The analysis program correctly constructed the description of the underlying polygons. The features used in tracking polygons included vertices, angles subtended at the vertices and the length of sides of polygons.
2 Motion Analysis: Past, Present and Future
31
The above scenario is similar to a person observing a scene through a narrow slit. Let us assume that a car passes by the slit. At no time does the observer see the car in its entirety but the observer is able to construct a description of the car and determine its speed. Our motivation for studying two-dimensional motion of objects in a plane was a matter of convenience and it was a meaningful first step. Further, the use of simple polygonal shapes was also driven by a lack of digitized images, digitizing hardware for video and sufficiently powerful computing equipment. However, the resulting ability to ‘understand’ the sequence of polygonal images led to more realistic scenarios. In general, researchers are interested in three-dimensional (3D) objects in threedimensional space. However, psychologists have reported several cases of humans discerning a 3D structure from moving 2D images. Wallach and O’Connell [41] presented silhouettes of moving wire-frame objects to subjects and the subjects reported seeing a three-dimensional structure while individually the frames contained no information about the three-dimensional structure. Further, under the right circumstances the observer could describe the wire-frame structure correctly. Wallach and O’Connell called it the “kinetic depth effect”. Shimon Ullman [39] called this the “structure from motion” effect. He considered the estimation of a threedimensional structure from moving light displays, i.e. images formed by the orthographic projection of rigid configurations of localized feature points. Ullman derived the result that four non-coplanar points in three distinct views are sufficient to give the three-dimensional structure of the points. John Roach [32] considered the same problem but with central projection (or perspective projection). The problem here was significantly more complex because the determination of the structure depended upon the solution to a set of nonlinear equations. Roach2 discovered that two views of six points led to an over-determined set of equations. Roach also observed that one may use five points but the solution was not always stable. This is an example of feature tracking which corresponds to the first subsystem (of the human visual system) mentioned in Sect. 2. In this case, the points are presumably extracted from features on moving objects of interest. The underlying assumption is that the corresponding points represent the feature points on the moving object. In the above analysis, features played an important role in computing motion and regions of ‘similar’ motion. Another important contribution was made by Jon Webb [43], inspired by Johansson’s [23] experiments in which Johansson showed his subjects a movie of a person walking around in a dark room with lights attached to the person’s major joints. Even though only lights could be seen, there was a strong impression of three-dimensional motion in these movies. In these experiments, each rigid-body part was represented by two points. Webb considered the motion of two rigidly connected points and observed that any motion may be decomposed into a rotation and 2 The methods for solving nonlinear equations have progressed significantly, as well as the available
computing power, so that now there is a software package, the EOS Systems Photomodeler [13], which uses two views of six points to generate a structure. The software package first appeared on the market in 1999, indicating the long interval between an idea and a product.
32
J.K. Aggarwal
a translation, however, Webb made an additional assumption that the axis of rotation is fixed in a direction for a short period of time. Webb was able to exploit this to determine the structure of jointed objects under orthographic projections. Webb applied his algorithm to real data with excellent results. This is one of the earliest studies on the motion of jointed objects and the analysis of human motion. In the works described above, feature points are used to estimate motion as well as the structure of objects. There are two difficulties with this paradigm. First, the correspondence between points in successive images must be established, and second, one must take into account the disappearance of points due to occlusion and possibly noise. The establishing of correspondence is a serious problem even in simple environments. An alternative way of computing motion follows where correspondence is established between regions of the image plane as the first step.
4 Optical Flow-Based Motion Detection It is mentioned in Sect. 2 that the human visual system responds to motion stimulus in mainly two ways: feature-based detection of motion and through a process driven by direction-of-motion sensors, even though it is not known how the motion sensors give rise to optical flow. Section 3 above described feature-based sensing and discussed briefly the advantages and disadvantages of feature tracking and sensing motion based on features. In this section, the beginnings of the optical-flow based techniques are presented. The motion of an object in three-dimensional space when imaged by a camera system yields a vector field of the motion vectors corresponding to each point of the object. It is assumed that the image surface is a plane and the vector field is constructed by observing the position of a given point on the object at two successive instants of time. The duration of the time interval between the two instants of time may be made arbitrarily small. At any given time, the direction of the flow vector at a specific point on the object is the projection of the direction of motion of the point in space, and the magnitude of the vector is the projection of the length of the velocity vector of the object point. This flow field is called the optical-flow field. Thus, optical flow is a mapping of the “apparent” velocities of points on the object as projected on the image plane. This distribution of velocities can provide a wealth of information including motion of objects, presence of multiple objects, and rate of movement of objects. It has been the subject of intense study by computer-vision researchers. A fundamental contribution is due to Horn and Schunck [21]. The brightness of an object point, f (x, y), may be assumed to be constant in a small window for the purposes of computing optical flow. They formulated the problem using (1) as follows: df (x, y) = 0. dt
(1)
2 Motion Analysis: Past, Present and Future
33
Equation (1) may be expanded as: ∂f (x, y) dx ∂f (x, y) dy ∂f + + =0 ∂x dt ∂x dt ∂t
(2)
∂f (x, y) ∂f ∂f (x, y) u+ v+ = 0, ∂x ∂x ∂t
(2a)
or
dy where u = dx dt and v = dt . Here f is the image intensity at point (x, y) in the image plane and (u, v) is the vector describing the velocity vector and ∂ is the partial derivative operator. This is only one equation. Obviously, (1) is not adequate by itself to solve for the velocity vector (u, v). Additional constraints must be imposed. A variety of constraints may be imposed to solve this equation. Horn and Shunck suggested two constraints as follows:
(i) By minimizing the square of the magnitude of the gradient of the optical-flow velocity 2 2 2 2 ∂u ∂v ∂v ∂u + and + . (3) ∂x ∂y ∂x ∂y (ii) Or by minimizing the sum of the squares of the Laplacian of x- and ycomponents of flow ∇2u =
∂ 2u ∂ 2u + ∂x 2 ∂y 2
and ∇ 2 v =
∂ 2v ∂ 2v + . ∂x 2 ∂y 2
(4)
Today there are a number of techniques for computing optical flow. Fleet and Weiss [14] present a review of methods for the computation of optical flow. The computation of optical flow is important in many applications and the method by Ogale and Aloimonos [29] is particularly useful. Lucas and Kanade [25] presented an iterative technique for image registration with application to stereo vision.3 Later, Tomasi and Kanade [36] adapted it for detection and matching of feature points. The proposed two frame method is probably the most commonly used method for computing optical flow. Optical flow is based on image patches and an affine model for the flow field. Essentially the flow is assumed to be locally constant. At a more theoretical level, several researchers are pursuing more elaborate methods for computing optical-flow and motion boundaries. In particular, the methodology introduced by Mumford and Shah [28] for computation of homogeneous regions and smooth boundaries by minimizing a functional has become a tool of choice. Deriche et al. [11] document results on optical-flow estimation based on 3 Professor
Takeo Kanade once remarked to the author at a conference that the complexity of the Roach and Aggarwal [32] solution to the feature-tracking problem contributed to the Lucas and Kanade [25] simple solution. Further, the Lucas and Kanade solution is probably his most contributive paper among all his contributions to date.
34
J.K. Aggarwal
a variational approach and Vazquez et al. [40] present results on region and motion estimation. This is an active area of research at present. These methods will probably grow further and be incorporated in the solution of real world problems in the near future.
5 Human Actions and Activities In the above two sections a discussion of detection of motion paralleling the physiological model for detection of motion has been presented. There are several other methodologies for detection of motion in computer vision. Instead of focusing attention on just detection of motion, an application of computer vision, the recognition of human actions and activities, will be considered in this section. Recently, computer vision-based recognition of human actions and continued activities has gained significant importance in view of the intense desire for safety, security and surveillance in almost every aspect of life. The surveillance of people at public places like airports and subway stations is well known, however, there are a number of other applications that make the observation of persons in normal activities at home and in the workplace as interesting and as challenging. Several funding agencies have initiated substantial programs to study human activities in various forms including observation of human activities from unmanned aerial vehicle based cameras. Monitoring the elderly in a ‘smart’ home equipped with multiple cameras and other sensors and alarms, analysis and understanding of sports video, and the content-based video summarization and retrieval (useful for video sharing websites) are several different types of applications. Another application involves the monitoring of industrial environments where humans perform hazardous tasks wearing important protective clothing. In this application the safety of the product and the safety of persons are at stake. The movie industry is interested in synthesizing a given person’s actions, gait and other characteristics based on a model video. In military applications there is an added constraint of real-time analysis and understanding, thus, there is a plethora of applications where computer vision-based recognition of human activities may contribute to society. Before one is able to understand an activity based on a sequence of images, many low-level image processing steps in along chain of steps must be performed. The motion analysis methods presented in Sects. 3 and 4 serve as low-level components for activity-level analysis. The efficacy of these steps determines the overall success of determining high-level activities. A look at real imagery for a given application, for example imagery in subway station, requires careful algorithms for segmentation and extraction of features. In addition, surveillance and certain other applications, monitoring is a 24/7 problem including nighttime, rain and fog. Low-level issues are not discussed here but this is not an attempt to slight them. They must be dealt with and the results at higher levels depend on the efficacy of the tools at the low level. Yamato et al. [44] introduced recognizing human actions in time-sequential images using hidden Markov models (HMMs). However, significant human activity
2 Motion Analysis: Past, Present and Future
35
recognition had its origin at the MIT Media Lab in Alex Pentland’s group. Starner and Pentland [35] document early work on American Sign Language detection using HMMs. The paper by Oliver et al. [30] is another important paper delineating the segmentation of human blobs, estimating their position and velocity, and recognizing two-person interactions by applying coupled HMMs. Several other works appeared at about the same time including Haritaoglu et al. [19], Haritaoglu and Davis [18] and Cai and Aggarwal [8]. (The group at The University of Texas at Austin derived much inspiration from these works.) It is worth emphasizing that a person is just a blob in the above works. Later, Park and Aggarwal [31] introduced the motion of body parts to recognize human actions. This was a distinct departure from the earlier paradigm. Other works with similar flavor include Gavrila and Davis [17] and Campbell and Bobick [9]. Most of the above works focus on simple and short duration activities. Michael Ryoo [34] presents a context-free-grammar based representation scheme as a formal syntax for representing composite and continued activities. The system is able to recognize recursive activities such as assault and fighting. The approach is able to recognize continued and crowd/group activities. The period of late 1990s and early 2000s has had an enormous amount of research activity in the area of human actions and activities. The earlier reviews by Cedras and Shah [10], Aggarwal and Cai [3] and Gavrila [16] document the research prior to 2000. The more recent reviews by Turga et al. [37] and Aggarwal and Ryoo [6] detail the recent human activity research. An approach based taxonomy is outlined in the review by Aggarwal and Ryoo [6]. In particular, the methodologies are classified as single layered approaches and multi-layered approaches. Single layered approaches are more suitable for recognizing simple actions whereas multi-layered approaches are more relevant for complex or continued human activities. Single layered approaches may be further subdivided into space-time approaches or sequential approaches. Multi-layered or hierarchical approaches may also be further classified as statistical, syntactical or description-based approaches. Statistical approaches use Hidden Markov Models, syntactic approaches use grammar syntax such as stochastic context-free grammars, and description-based approaches describe events in terms of sub-events of actions and their temporal, spatial and logical structure. The last approach is particularly flexible and is to capture the variability of human activities. The author believes that in the not too distant future significant results on the recognition of human actions and activities will lead to products in industrial and military domains.
6 Motion: Future Obviously, we have made significant progress in developing methodologies and algorithms for analyzing and understanding sequences of images. In addition, we have made significant progress because computation is faster, memory is cheap, and cameras are relatively inexpensive. In the 1970s, it used to take a VAX 11/780 to process
36
J.K. Aggarwal
images and the cost of a respectable system was about $100,000, whereas today one can build a very good computer vision system with $2,000. Further, the ability to acquire and process images fairly effortlessly has contributed to the development of a large number of applications. However, the author would be remiss without mentioning that problems are difficult and solutions are incomplete. A presentation at a computer vision workshop entitled “Structure and Motion From Images: Fact and Fiction” [5] outlined the difficulties by using points, lines, and contours as discrete features and optical-flow and range images as additional features to analyze motion. It also presented difficulties arising from occlusion, low-level processing, and time reliability constraints. Paraphrasing the conclusion in 1985, the same holds today “. . . we have performed fairly but much remains to be resolved.” Computer vision-based research in motion is already contributing to some of the pressing societal problems. In particular several corporations have products. Siemens has installed a traffic monitoring system to monitor traffic in tunnels. It is able to estimate the level of traffic and signal to motorists. It has been installed and tested in a variety of conditions. Another system, manufactured by MobilEye, installed inside an automobile, warns the driver of an impending collision or if the car is going outside the prescribed lane. It is also available as a product. National Instruments is producing an inspection system for automobile parts and this particular system inspects a complicated part by measuring its dimensions in a three- dimensional coordinate system. The part has to satisfy very precise dimensions. There are systems being deployed at airports to detect the direction of the movement of passengers. So it is fair to say that computer vision is producing systems for the well being of society. The future of computer vision-based research in motion is bright. One factor contributing to rapid progress is the tools that are coming on line. The “computing scene” including speed and packaging is improving by the day! The price of memory is going down and the resolution (definition) of cameras has increased substantially. The research of Shree Nayar [24] at Columbia University and of Narendra Ahuja [15] at The University of Illinois, Urbana/Champaign is contributing to the developments of new camera technologies. Embedded cameras are available with a significant amount of computing power and more innovations in camera technology are on the way. These cameras are becoming relatively cheap. In addition to intensity image cameras, cameras with other modalities are finding use in daily life. Image processing and understanding using other modalities offers new avenues of investigation. Robot vision with understanding of motion and environment poses the greatest challenges and rewards. The success of the MARS Rover has brought forth a lot of interest in computer-vision driven robot exploration. With real-time constraints, the problems are truly challenging. It is a given that every cell phone has a camera and there are billions of cell phones in the world today. In addition, people are carrying them everywhere they go. It may not be too far-fetched to think that cameras will be used for navigation and monitoring. That is clearly a great application of cell phone and computer vision. In the not too distant future, the following scenario of a synthetic movie of a person appears likely. We observe a person in various poses and activities and we
2 Motion Analysis: Past, Present and Future
37
are able to construct a repertoire of motions of the person. Based on the analysis of these motions, we synthesize images of the person in any given posture and possibly a motion picture of that person performing various activities not included in the repertoire. In this way, we will be able to synthesize movies of persons in their absence. This will be a true interplay between computer vision and computer graphics. A number of researchers are pursuing the recognition of everyday human activities when observed from an unmanned aerial vehicle. The difficulties of imaging noise, stabilizing images and a host of other problems make the problem very difficult at this stage. Problems of monitoring workers in the food industry or workers performing other hazardous tasks, where protective gloves and other clothing are necessary, are beginning to receive attention. Monitoring patients in a hospital environment is another area of application. Given all the problems begging for a solution, the future is truly bright for motion research and applications. Motion is a subject of serious study in computer vision, image processing, computer graphics, and robot vision communities. The objectives are different in each of the communities in view of their applications. For example: in computer vision one is interested in human activities; in image processing video compression and transmission take the front seat; in computer graphics the production of life-like synthetic images takes precedence; and in robot vision real-time navigation applications dominate. The future of research and applications in motion in each of these endeavors is paved with low hanging fruit. Acknowledgements It is a pleasure to acknowledge the help and comments of Dr. Michael Ryoo, Professors Bill Geisler and Amar Mitiche and Mr. Birgi Tamersoy and Mr. Chia-Chih Chen. Also, my sincere thanks go to Ms. Selina Keilani for editing the manuscript. The research was supported in part by Texas Higher Education Coordinating Board award # 003658-0140-2007.
References 1. Addams, R.: An account of a peculiar optical phenomenon seen after having looked at a moving body, etc. Lond. Edinb. Philos. Mag. J. Sci., 3rd Ser 5, 373–374 (1834) 2. Adelson, E.H., Bergen, J.R.: Spatiotemporal energy model for the perception of motion. J. Opt. Soc. Am. A 2, 284–299 (1985) 3. Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Comput. Vis. Image Underst. 73(3), 428–440 (1999) 4. Aggarwal, J.K., Duda, R.O.: Computer analysis of moving polygon images. IEEE Trans. Comput. 24(10), 966–976 (1975) 5. Aggarwal, J.K., Mitiche, A.: Structure and motion from images: fact and fiction. In: Proceedings of the Third Workshop on Computer Vision: Representation and Control, Bellaire, MI (1985) 6. Aggarwal, J.K., Ryoo, M.S.: Human activity analysis: a review. ACM Comput. Surv. (2010, to appear) 7. Badler, N.I., Aggarwal, J.K.: Abstract of the Workshop on Computer Analysis of TimeVarying Imagery, Philadelphia, PA (1979) 8. Cai, Q., Aggarwal, J.K.: Tracking human motion in structured environments using a distributed camera system. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1241–1247 (1999) 9. Campbell, L.W., Bobick, A.F.: Recognition of human body motion using phase space constraints. In: IEEE Int. Conference on Computer Vision (ICCV), pp. 624–630 (1995)
38
J.K. Aggarwal
10. Cedras, C., Shah, M.: A motion-based recognition: a survey. Image Vis. Comput. 13(2), 129– 155 (1995) 11. Deriche, R., Kornprobst, P., Aubert, G.: Optical-flow estimation while preserving its discontinuities: a variational approach. In: Proceedings Second Asian Conference on Computer Vision. Lecture Notes in Computer Science, vol. 1035, pp. 71–80 (1995) 12. Derrington, A.M., Allen, H.A., Delicato, L.S.: Visual mechanisms of motion analysis and motion perception. Annu. Rev. Psychol. 55, 181–205 (2004) 13. Eos Systems, Inc. Photomodeler, http://www.photomodeler.com/ 14. Fleet, D.J., Weiss, Y.: Optical flow estimation. In: Paragios, et al. (eds.) Handbook of Mathematical Models in Computer Vision. Springer, Berlin (2006) 15. Gao, C., Hua, H., Ahuja, N.: A hemispherical imaging camera. Comput. Vis. Image Underst. 114(2), 168–178 (2010) 16. Gavrila, D.: The visual analysis of human movement: a survey. Comput. Vis. Image Underst. 73(1), 82–98 (1999) 17. Gavrila, D., Davis, L.: Towards 3-D model-based tracking and recognition of human movement. In: Bichsel, M. (ed.) International Workshop on Face and Gesture Recognition, pp. 272– 277 (1995) 18. Haritaoglu, I., Davis, L.: Hydra: multiple detection and tracking silhouettes. In: IEEE Workshop on Visual Surveillance, Fort Collins, CO, pp. 6–13 (1999) 19. Haritaoglu, I., Harwood, D., Davis, L.: W4: who, when, where, what: a real system for detecting and tracking people. In: Third Int. Conf. on Automatic Face and Gesture Recognition, Nara, Japan, pp. 222–227 (1998) 20. Horn, B.K.P.: Robot Vision. MIT Press, Cambridge (1986) 21. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981) 22. Huang, T.S.: Image Sequence Processing and Dynamic Scene Analysis. NATO ASI Series. Springer, Berlin (1983) 23. Johansson, G.: Visual motion perception. Sci. Am. 232(6), 76–88 (1975) 24. Krishnan, G., Nayar, S.K.: Cata-fisheye camera for panoramic imaging. In: IEEE Workshop on Application of Computer Vision (WACV) (2008) 25. Lucas, B.D., Kanade, T.: An iterative image registration technique with application to stereo vision. In: Proceedings of 7th International Joint Conference on Artificial Intelligence (1981) 26. Martin, W.N., Aggarwal, J.K. (eds.): Motion Understanding: Robot and Human Vision. Kluwer Academic, Dordrecht (1988) 27. Math Pages Website (http://www.mathpages.com/rr/s3-07/3-07.htm), Reflections on relativity—Zeno and the paradox of motion 28. Mumford, D.S.: Boundary detection by minimizing functionals. In: Proceedings of CVPR (1985) 29. Ogale, A.S., Aloimonos, Y.: A roadmap to the integration of early visual modules. Int. J. Comput. Vis. 72(1), 9–25 (2007). Special Issue on Early Cognitive Vision 30. Oliver, N., Rosario, B., Pentland, A.: A Bayesian computer vision system for modeling human interactions. In: Proceedings of International Conference on Vision Systems 99, Gran Canaria, Spain, pp. 255–272 (1999) 31. Park, S., Aggarwal, J.K.: A hierarchical Bayesian network for event recognition of human actions and interactions. Multimed. Syst. 10(2), 164–179 (2004) 32. Roach, J.W., Aggarwal, J.K.: Determining the movement of objects from a sequence of images. IEEE Trans. Pattern Anal. Mach. Intell. 2(6), 554–562 (1980) 33. Russell, B.: The problem of infinity considered historically. In: Salmon, W.C. (ed.) Zeno’s Paradoxes, p. 51. Bobbs-Merrill, Indianapolis (1970) 34. Ryoo, M.S., Aggarwal, J.K.: Semantic representation and recognition of continued and recursive human activities. Int. J. Comput. Vis. 82, 1–24 (2009) 35. Starner, T., Pentland, A.: Real-time American sign language recognition from video using hidden Markov models. In: International Symposium on Computer Vision, p. 265 (1995) 36. Tomasi, C., Kanade, T.: Detection and tracking of point feature shape and motion from image streams: a factorization method—part 3. Technical Report CMU-CS91-132 (1991)
2 Motion Analysis: Past, Present and Future
39
37. Turga, P., Chellappa, R., Subrahmanian, V.S., Udren, O.: Machine recognition of human activities: a survey. IEEE Trans. Circuits Syst. Video Technol. 18(11), 1473–1488 (2008) 38. Ullman, S.: The Interpretation of Visual Motion. MIT Press, Cambridge (1979) 39. Ullman, S.: The interpretation of structure from motion. Proc. R. Soc. Lond. B 203, 405–426 (1979) 40. Vazquez, C., Mitiche, A., Langaniere, R.: Joint segmentation and parametric estimation of image motion by basis function representation and level set evolution. IEEE Trans. Pattern Anal. Mach. Intell. 28(5), 782–793 (2006) 41. Wallach, H., O’Connell, D.C.: The kinetic depth effect. J. Exp. Psychol. 45(4), 205–217 (1953) 42. Watson, A.B., Ahumada, A.J.: Model of human visual-motion sensing. J. Opt. Soc. Am. A 2(2), 322–342 (1985) 43. Webb, J.A., Aggarwal, J.K.: Structure from motion of rigid and jointed objects. Artif. Intell. 19, 107–130 (1982) 44. Yamato, J., Ohya, J., Ishii, K.: Recognizing human action in time-sequential images using hidden Markov model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 379–385 (1992) 45. Zeki, S.: The visual image in mind and brain. Sci. Am. 69–76 (1992)
Chapter 3
Projective Joint Invariants for Matching Curves in Camera Networks Raman Arora and Charles R. Dyer
Abstract A novel method is presented for distributed matching of curves across widely varying viewpoints. The fundamental projective joint-invariants for curves in the real projective space are the volume cross ratios. A curve in m-dimensional projective space is represented by a signature manifold comprising n-point projective joint invariants, where n is at least m + 2. The signature manifold can be used to establish equivalence of two curves in projective space. However, without correspondence information, matching signature manifolds is a computational challenge. Our approach in this chapter is to first establish best possible correspondence between two curves using sections of the invariant signature manifold and then perform a simple test for equivalence. This allows fast computation and matching while keeping the descriptors compact. The correspondence and equivalence of curves is established independently at each camera node. Experimental results with simulated as well as real data are provided.
Keywords Camera networks · Curve matching · Projective invariants · Joint invariant signatures · Cross ratios
1 Introduction Object recognition in automated visual surveillance systems must be capable of matching features which represent distinctive parts of objects (such as people or vehicles) in complex environments in an online fashion across multiple viewpoints. R. Arora () Department of Electrical Engineering, University of Washington, Seattle, WA 98105, USA e-mail:
[email protected] C.R. Dyer Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI 53706, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_3, © Springer-Verlag London Limited 2011
41
42
R. Arora and C.R. Dyer
Commercial, law enforcement, and military applications abound, including detection of loiterers, monitoring vehicles on highways, patrolling borders, measuring traffic flow, counting endangered species, and activity monitoring in airports. As costs for cameras and computers continue to drop while the desire for security and other applications increases, research in this area has been developing rapidly over the last decade [4, 5, 9, 11, 20]. Matching curves across widely varying viewpoint requires local image features that are invariant to changes in pose, occlusion, illumination, scale, and intrinsic differences between cameras. This chapter describes a method that uses projective joint invariants to match curves across multiple views. Given a pair of images taken from unknown viewpoints, a set of curves is extracted from each image that consist of projections of unknown 3D curves in the scene. The objective is to determine if any two given curves, one from each image, match, i.e. if they correspond to the same curve in the scene. An invariant is defined to be a function on the set of points (or a subset) of an image, of a planar or 3D object, that remains constant under a collection of transformations of the object. Two images (or sub-images) with the same values of the invariant are identified as images of the same object under a transformation, thereby making the problem of multiple hypothesis detection direct. Due to the utility of transformation-invariant features in their ability to reduce the set of possible matches and speed up the search for similar classes or objects, invariant-based approaches to problems in computer vision have been well studied [16, 17]. Invariant-based methods may be classified as global or local: global invariants utilize the entire image to compute feature values whereas local invariants are computed from much smaller subsets. Local invariants are more desirable due to their robustness to occlusion and noise. However, one of the fundamental problems with the use of local invariants is that they must be computed on corresponding subsets of points in each view.
1.1 Related Work Projective invariants have been applied to various computer vision tasks such as localization [13, 21], autonomous navigation [26], 3D reconstruction [24], and surveillance [27]. A few researchers have focused on the probabilistic analysis of projective invariants. In [2], a probability distribution was derived for the four-point cross ratio, a classical planar projective invariant, under various assumptions on the distribution of the four points. The distribution of cross ratios was further examined in [10] with constraints on relative distances of the four points. The performance of cross ratios was described quantitatively in terms of probability of rejection and false alarm in [15]. Unfortunately, in all these works, the correspondence of points between images was given a priori or external makers were used to assist with the correspondence. Without correspondence information, the classification methodology breaks down since the cross ratios are not unique.
3 Projective Joint Invariants for Matching Curves in Camera Networks
43
In other related work, Scale-invariant feature transform (SIFT) [12] was used to compute a large set of local feature vectors from each image and the correspondence between two images was established using RANSAC [6]. This approach has been used for 3D model reconstruction in the Photo Tourism system [25]. However, the computational complexity of SIFT and RANSAC makes it difficult to use for real-time video surveillance applications with non-overlapping field-of-view camera networks. Rothwell et al. [22, 23] presented an approach for planar object recognition by constructing a canonical frame for determining projectively invariant indexing functions for planar curves. The idea is to identify four distinguished points on the curve and then compute the projective transformation that maps these points to the four corners of a square. The distinguished points are chosen using tangency conditions that are preserved under projective transformations. Similar ideas have been put forth by Hann and Hickman [7, 8] and by Orrite and Herrero [19]. These methods also utilize bi-tangency points on curves to learn a projective map. The key difference in the more recent work [7, 8, 19] from the algorithms presented in mid-nineties [22, 23] is that they learn the best projective transformation between two given planar curves in an iterative fashion whereas the earlier work focussed on solving for a projective transformation that best maps the bi-tangency points to four corners of the unit square. There are several shortcomings of existing methods for matching curves across multiple viewpoints that preclude their deployment to applications like video surveillance in camera networks. The methods based on learning projective transformations [7, 8, 19, 22, 23] between given curves are inherently centralized and computationally expensive. In order to deal with differences in sampling of images (resulting from different grids in imaging devices), existing methods employ an iterative scheme where the learnt projective transformation is corrected-for based on the resulting mismatch in the image domain. For specialization to matching curves in video streams, this will require complete exchange of images at each iteration and repeated estimation of projective transformations. Furthermore, the methods depend on the ability to consistently identify bi-tangents. But, due to possible occlusions, the curves extracted from the images may not admit any bi-tangents. Finally, the methods based on detecting interest points and representing images in a visual dictionary obtained by clustering SIFT descriptors [3] are inherently offline. In applications such as video surveillance (where the object to be matched may be moving), these methods require frame synchronization across video feeds from different cameras as well as dictionary computation and exchange every few frames.
1.2 Our Approach We present a method for matching curves in different views without assuming any knowledge of the relative positions and orientations of the viewpoints. Our approach is based on the computation and comparison of projective invariants that are expressed as volume cross ratios of curves extracted from images of an arbitrary 3D
44
R. Arora and C.R. Dyer
scene. Signatures based on these cross ratios are computed from each image and a clustering-based method is presented for distributed matching. This work was inspired by recent advances in joint invariants [18] and probabilistic analysis of random cross ratios. Joint-invariant signatures were recently studied by Olver [18] for various transformation groups. However, due to the sheer size and global nature of the signatures, they cannot be directly employed for curve matching. The novel ideas in this chapter include generating compact local signatures independently from each image and clustering-based matching. We systematically reduce the size and computational complexity of the matching by reformulating the problem and offering a tradeoff between the size of feature space and size of the search space for registration parameters. Our method alleviates the aforementioned shortcomings of existing methods. Unlike existing methods that match curves in the image domain, the proposed method matches curves in an invariant domain. The classification rule is based on comparing the projective invariants of a given pair of curves. The invariants are known to be complete and therefore uniquely represent the corresponding curve. The joint invariants are also robust to noise as a small perturbation of points on the curve results in a small relative error in the invariant domain [1]. The matching of two curves can be performed efficiently in the presence of bi-tangents on the given curves. It should be remarked that unlike previous work [7, 8, 19, 22, 23], the proposed method does not critically depend on the existence of bi-tangents. However, whenever bi-tangents are present, our approach utilizes them to speed up the search for registration parameters. It is important to note that a description of a curve using projective joint invariants remains invariant to Euclidean transformations as well. Therefore, in video surveillance applications, the representation is redundant across frames of the video feed when the object undergoes rigid-body motion. This saves network and computational resources and allows for robust matching of curves between two cameras without frame synchronization. The chapter is organized as follows. Section 2 describes the curve-matching problem in a multiview setting and briefly presents mathematical preliminaries in projective joint invariants. Section 3 describes the joint-invariant signature and challenges associated with distributed matching of signature manifolds. In Sect. 4, we discuss the use of sub-manifolds and local signatures to establish correspondence and equivalence of curves across viewpoints. Section 5 presents experimental results with simulated as well as real datasets. Table 1 shows the notation used in this chapter.
2 Problem Formulation and Preliminaries This section introduces the notation and presents problem formulation for pairwise curve matching across different viewpoints. Let Ci represent an unknown viewpoint or camera location. Let Sij denote the j th continuous planar curve observed
3 Projective Joint Invariants for Matching Curves in Camera Networks
45
Table 1 Notation table Symbols
Description
Ci
i th camera or i th viewpoint
Sij
j th continuous curve at viewpoint Ci
I
an interval of R
Dij
j th discrete curve j in image plane of camera Ci
G
transformation group
M
manifold
F(n)
n-point joint-invariant vector-function [F1 , . . . , Fl ]
V (zi , zj , zk )
area of the triangle with vertices zi , zj , zk
CR(z1 ; z2 , z3 , z4 , z5 )
volume cross ratio given in (5)
Sijn
n-times Cartesian product of the curve Sij
Jij
invariant signature manifold associated with curve Sij
ij
(n)
dkl
distance function used for matching; see (10)
t∗
∗ ) pivot points (t1∗ , t2∗ , . . . , tn−p
(n)
π
a permutation of n points
At ∗ ,π
slice of the signature manifold determined by t ∗ , π
Ut ∗ ,π
a section of the slice At ∗ ,π
at viewpoint Ci . The curve Sij is obtained from a space curve S under an unknown projective transformation. Formally, Sij is defined to be a parametric curve Sij : Iij → Rm t → Sij (t),
(1)
where Iij ⊂ R is a real interval and m is the dimensionality of the ambient Euclidean space for the curve. For t ∈ T , Sij (t) = [x1 (t) · · · xm (t)]T gives coordinates in Rm of the corresponding point on the curve. For planar curves, m = 2 and for space curves m = 3. The n-dimensional Cartesian product of the curve Sij is written as, Sijn : Iijn → Rm×n t n → Sijn t n ,
(2)
where Iijn ⊂ Rn is an n-dimensional interval, t n = (t1 , . . . , tn ) and Sijn (t n ) = (Sij (t1 ), . . . , Sij (tn )). For a pair of curves Sij , Skl observed at viewpoints Ci , Ck respectively, the objective is to determine if the two curves in the image space at the two unknown viewpoints represent the same space curve in the observed scene. The sampled, discretized version of Sij is denoted as Dij . Lets denote the manifold associated with the given curve as M. A given curve may undergo various transformations like rotation, translation, scaling and projection. These transformation can be described as a Lie group, denoted G, acting on the manifold M. The joint action of a group on a manifold describes how the group
46
R. Arora and C.R. Dyer
transforms any given n-tuple on the manifold. Formally, the joint action of the group G on the Cartesian product M n is a map (G × M n ) → M n given as: g · (z1 , . . . , zn ) = (g · z1 , . . . , g · zn ),
(3)
for g ∈ G and z = (z1 , . . . , zn ) ∈ With a slight abuse of notation we use zi to represent a point on the manifold M ⊆ Rm as well as its Euclidean coordinates in the ambient space Rm . An n-point joint invariant of the transformation group G on M n is defined to be a function M n.
F(n) : Rm×n → z
Rl (n) (n) → F1 (z), . . . , Fl (z) ,
(4)
which is invariant to the joint action of the group on the manifold: F(n) (g · z) = F(n) (z). The projective transformation group, G = PSL(m + 1, R), which is the subject of Az+b study in this chapter, acts on the projective space RPm as w = g · z = c·z+d , where A is an m × m matrix, b, c are m × 1 vectors, and d is a scalar. The transformation g maps the point z ∈ Rm ⊂ RPm to w ∈ Rm . For planar curves (m = 2) consider n = 5. The fundamental invariants for fivepoint joint action of the projective transformation group PSL(3, R) are given by the volume cross ratios [18]: CR(z1 ; z2 , z3 , z4 , z5 ) =
V (z1 , z2 , z3 )V (z1 , z4 , z5 ) , V (z1 , z2 , z5 )V (z1 , z3 , z4 )
(5)
CR(z2 ; z1 , z3 , z4 , z5 ) =
V (z1 , z2 , z3 )V (z2 , z4 , z5 ) , V (z1 , z2 , z5 )V (z2 , z3 , z4 )
(6)
and
where V (zi , zj , zk ) is the area of the triangle defined by zi , zj and zk . The cross ratio defined in (5) is described as the ratio of the product of the areas of the nonshaded triangles in Fig. 1 and the product of areas of the shaded triangles. Therefore, for PSL(3, R), F(n) is given as F(n) (z) = CR(z1 ; z2 , z3 , z4 , z5 ), CR(z2 ; z1 , z3 , z4 , z5 ) . (7) For m = 3, we consider six-point joint invariants. There are three fundamental volume cross ratios: CR(z1 , z6 ; z2 , z3 , z4 , z5 ), CR(z1 , z3 ; z2 , z4 , z5 , z6 ), and CR(z3 , z6 ; z1 , z2 , z4 , z5 ). Geometrically, CR(z1 , z6 ; z2 , z3 , z4 , z5 ) is the ratio of the volumes of four tetrahedrons: V (z1 , z2 , z3 , z4 )V (z2 , z3 , z5 , z6 ) CR(z1 , z6 ; z2 , z3 , z4 , z5 ) = . (8) V (z1 , z2 , z4 , z5 )V (z3 , z4 , z5 , z6 ) Figure 1 shows a bird’s eye view of the double pyramid with common base resulting from the union of the tetrahedrons [18]. The probabilistic analysis of random five-point cross ratios reveals that no single cross ratio is unique on smooth manifolds [1]. Consequently, for planar curves, the matching schemes based on comparing single cross ratios are not discriminative.
3 Projective Joint Invariants for Matching Curves in Camera Networks
47
Fig. 1 The five-point projective joint invariant is the ratio of the product of areas of the non-shaded triangles and the product of areas of the shaded triangles
The six-point joint invariants for 3D space curves lend themselves to the same analysis as the empirical distributions are found to exhibit characteristics similar to the planar case. Furthermore, it is argued using jitter-analysis that the cross ratios are robust to noise. For more details the reader is referred to [1].
3 Joint-Invariant Signatures The non-uniqueness of any single cross-ratio value implies that no single cross-ratio value can be used for matching without establishing correspondence of points [1]. However, the joint-invariant signature defined to be the manifold comprising crossratio values generated by all possible n-point sets on a curve represents the curve uniquely up to a projective transformation [18]. Let F(n) be the n-point joint-invariant map for the projective transformation group PSL(3, R) given in (7). Consider the composition maps Jij = F(n) ◦ Sijn , Jij : I n → Rl
t n → F(n) Sijn t n .
(9)
The invariant signature manifold at viewpoint Ci for j th curve is defined to be Jij (Iijn ). We now focus on restriction of the curves to a common domain. Consider I ⊆ Iij ∩ Ikl . Note that the intervals Iij , Ikl can be translated, flipped and scaled appropriately so that I = ∅. We denote the restriction of the curves to the common interval I by S˜ij and S˜kl . Now, if the curves S˜ij , S˜kl are related by a projective transformation, i.e., S˜ij = g · S˜kl for some g ∈ G = PSL(3, R) on interval I ⊂ R, then from the definition of n . This implies that the invariant signature manifold for the joint action, S˜ijn = g · S˜kl two curves coincide: For all t n ∈ I n , n (n) n n n t = F ◦ S˜kl t = Jkl t n . Jij t n = F(n) ◦ S˜ijn t n = F(n) ◦ g · S˜kl
48
R. Arora and C.R. Dyer
More importantly, S˜ij = g · S˜kl for some g ∈ G if Jij (I n ) = Jkl (I n ) [18]. Therefore, two curves are equivalent up to a projective transformation if and only if their signature manifolds coincide. Consider the following distance function that measures the degree of mismatch between two curves in terms of the mismatch in the associated invariant signature manifolds, n ij Jij t − Jkl t n 2 dt n , U ⊆ I n , dkl (U ) ≡ (10) 2 U
where · 2 represents the standard Euclidean norm. For robustness to noise in observation, we adopt the following test for matching two curves, ij
dkl (I n ) < ,
(11)
where threshold > 0 depends on the amount of noise resulting from differences in quantization and the sampling grids of two cameras. The matching criterion in (11) is straightforward if the two curves are defined on the same domain and the true correspondence between the curves is known. However, without the correspondence information between the two curves there is no direct method to compute the distance function in (10). This problem is further compounded by various practical aspects of matching in a camera network. First and foremost, the curves observed at each camera are discrete curves (Dij obtained from sampling Sij ). We assume that the curves are sampled dense enough to admit a good approximation in an interpolating basis. Secondly, due to differences in the sampling grids for different cameras, the discretization of the interval I differs for each camera. Finally, the size of signature manifold associated with each curve grows exponentially with the number of samples on the curve. Therefore, estimating correspondence between the two curves using entire signature manifolds is not a feasible solution due to computational constraints. In this chapter we first establish correspondence efficiently by restricting ourselves to certain sub-manifolds of the signature manifold and then ij test for equivalence by estimating dkl from given observations (Dij , Dkl ).
4 Toward Local Signatures The joint-invariant signature manifold is a global descriptor. Owing to the lack of robustness of global signatures to occlusion, we restrict our attention to sections of invariant signature sub-manifolds. We first define a slice of the signature manifold and then discuss local signatures computed on sections of slices.
4.1 Slices and Sections of Signature Manifold A slice of the signature manifold may be generated by freezing one or more coordinate directions of I n (at pivot points denoted as t ∗ ) while at least one of the coordinate direction spans the entire interval. Matching slices of the signature manifold
3 Projective Joint Invariants for Matching Curves in Camera Networks
49
across varying viewpoints provides an efficient method for establishing correspondence of curves. Such slices (or sub-manifolds) may not be generated arbitrarily: If the pivot points in RP2 (RP3 ) result in collinear (coplanar) points on the curve, then the resulting slice may comprise all zero cross-ratio values. A p-dimensional sub-manifold of the n-point joint-invariant signature manifold is obtained by pivoting n − p of the n points. Consider a canonical p-dimensional slice of the interval I n ⊂ Rn , ∗ At ∗ = {t1∗ } × {t2∗ } × · · · × {tn−p } × I p,
(12) (t ∗ , t p ).
where first (n − p) coordinates for ∈ At ∗ are fixed: = In order to generate all possible slices, we need to introduce the notion of permutation of coordinates. Let π denote a permutation of integers {1, . . . , n}, tn
tn
π : {1, . . . , n} → {1, . . . , n} i → π(i).
(13)
The permutation π acts on At ∗ to give another slice, At ∗ ,π = t˜ ∈ I n : t˜π(i) = ti , for i = 1, . . . , n, where (t1 , . . . , tn ) ∈ At ∗ . (14) Given a slice At ∗ , consider a local section Ut ∗ ⊆ At ∗ defined as ∗ Ut ∗ = {t1∗ } × {t2∗ } × · · · × {tn−p } × U p,
(15)
⊆ I p.
where = Un−p+1 × Un−p+2 × · · · × Un A section of At ∗ ,π is denoted as Ut ∗ ,π . Using sections of signature manifolds offers a tradeoff between the size of the invariant descriptors versus the computational load associated with searching pivot points. Consider the special case of projective transformations of planar curves (m = 2, n = 5). A 2D slice of signature manifold for the letter “W” from the license plate dataset is shown in Fig. 2(c) with permutation 13 24 35 41 52 . The pivot points are Up
zπ(i) = Sij (ti∗ ) for i = 1, 2, 3 and zπ(4) , zπ(5) span the entire interval I . A section of the slice is shown in Fig. 2(b) with free points restricted to the region marked by squares in Fig. 2(a).
4.2 Correspondence and Equivalence from Matching Sections Correspondence between curves can be established efficiently using the sections of the signature manifold. Lets denote the section of the signature manifold Jij collected on Ut ∗ ,π as Jij (Ut ∗ ,π ). Camera Ck reconstructs the curve Sij from samples of the curve received from camera Ci and the curve Skl from the samples captured locally. Then pivot points tˆ that generate the section of Jkl which is the best possible match to section Jij (Ut ∗ ,π ) can be found in an iterative manner as outlined below. For t n ∈ I n , define Jkl (t n ) − J . tn = (16) inf J ∈Jij (Ut ∗ ,π )
2
50
R. Arora and C.R. Dyer
Fig. 2 The points z3 , z4 and z5 are the pivot points and points z1 , z2 span the 2D slice of the (n) signature sub-manifold associated under F1
Consider the set V comprising all n-point sets in the domain of Jkl that result in cross ratios within an -neighborhood of the given section Jij (Ut ∗ ,π ): (17) V = t n ∈ I n : t n < ∗ . The set V is learnt as follows. Starting with a uniform distribution on I n , samplepoints t n are generated and tested for inclusion: t n ∈ V as given by (17). Alternatively, picking pivot points as points of inflections on the curve allows initializing with a prior distribution on I n that is a Gaussian mixture model with components centered at inflection points. This yields tremendous speed-up in matching. See next section for more details on picking pivot points. As the set V grows, the clusters in the set V are learnt using Density-Based Scan Algorithm with Noise (DBSCAN). For each cluster, Hausdorff distance is computed between the given section Jij (Ut ∗ ,π ) and the signature manifold Jkl evaluated on points comprising the cluster. The distribution on I n is updated in accordance with the score given by the Hausdorff distance and the sampling process is repeated. A concentrated cluster with small Hausdorff distance indicates a matching section between the two signature manifolds. Multiple matching sections provide significant statistical evidence for the equivalence of curves at two viewpoints.
3 Projective Joint Invariants for Matching Curves in Camera Networks
51
4.3 Picking Pivot Points A robust method for choosing pivot points and sections consistently across varying viewpoints is based on the identification of inflection points of curves. Inflection points are defined to be the points on the curve at which the curvature changes sign. Consider the motion of the tangent to a given planar curve at a point as the point moves along the curve. The tangent either rotates clockwise or anti-clockwise in the plane. The rate of rotation of the tangent is given by the curvature of the curve. The points at which the rotation changes direction (clockwise to anti-clockwise or vice versa) are the inflection points of the curve. It is well known that inflection points are invariant to projective transformations. Thus they can be found consistently across different perspectives and result in the same segmentation of the curve. However, inflection points are very sensitive to noise. Figure 3 shows inflection points for various contour images extracted from the license plate dataset. Due to the quantized nature of the contours and associated noise or discontinuities, a simple test for inflection points results in a host of possible candidates as seen in Fig. 3(a). Smoothing the curve using a simple low-pass filter eliminates most of the noisy candidates (Fig. 3(b)). Further elimination based on the area under the curvature plot about each candidate point reveals the significant inflection points as seen in Fig. 3(c). This pre-processing method is robust to severely noisy conditions as well as widely varying perspectives. It allows for robust segmentation of curves. It should be remarked that most interesting shapes admit inflection points in the resulting contours. However, in the case where no inflection points are observed, the pivot points can be picked arbitrarily without affecting the matching accuracy of the iterative clustering method described in the previous section. Thus, the matching methodology does not depend critically on the presence of inflection points.
Fig. 3 Inflection points marked with dots on (a) original contours, (b) smoothed contours and (c) post-elimination based on the amount of rotation of tangent about inflection points
52
R. Arora and C.R. Dyer
5 Matching Performance This section discusses performance of the clustering-based algorithm on simulated data using the Epipolar Geometry Toolbox [14] as well as on a license plate image database [28]. Case: m = 2, n = 5 Figure 2 shows contour plots from license plate dataset along with invariant signatures. Figure 2(a) shows the contour of the letter “W” (extracted from images of the license plate WMP619). The set of five points on contours that generated the invariant signature (in Fig. 2(c)), are highlighted with dots. The points z3 , z4 and z5 are the pivot points and points z1 , z2 span a 2D slice of the signature sub-manifold (n) associated with F1 in (7). The surface plot of the 2D slice is shown in Fig. 2(c) and surface plot of a section is shown in Fig. 2(b). The images from the license plate dataset captured at two different viewpoints are shown in Figs. 4(a, c). The test dataset comprising the 12 contour images was enlarged by generating random projective transformations of the given contours. The confusion matrix for this experiment involving 100 random projective transformations is shown in Fig. 4(e). It is evident from the experimental results that the method enjoys a good specificity as well as sensitivity. The number of detected inflection points for the test images ranged from zero (for the digit “8” after smoothing) to eight (for the letter “X”). Case: m = 3, n = 6 The Epipolar Geometry Toolbox [14] was used to simulate a 3D scene with moving space curves being tracked by two pinhole cameras. Since the signature submanifolds are invariant to Euclidean as well as perspective transformations, they uniquely describe these space-time curves. Therefore, the signature sub-manifold need not be recomputed at every frame of the trajectory of the curve. This allows for robust matching of curves in the image planes of the two cameras at every instant, without frame synchronization. For more details and results on matching space curves, refer to [28].
Fig. 4 Contour images from the license plate database [28]. The digits extracted from license plates as seen at (a) Camera 1, (b) at a randomly generated viewpoint of the curves observed at Camera 1, (c) at Camera 2, and (d) at a randomly generated viewpoint of the curves observed at Camera 2. (e) Confusion matrix for 100 random projective transformations of curves in the license plate database
3 Projective Joint Invariants for Matching Curves in Camera Networks
53
6 Discussion This chapter presented an efficient algorithm for matching curves across widely varying viewpoints using joint invariants. The equivalence of curves under projective transformation is established by matching sections of invariant signature manifold which are local, compact descriptors of curves. Matching results are provided using both simulated data and a license plate dataset.
References 1. Arora, R., Hu, Y.H., Dyer, C.R.: Estimating correspondence between multiple cameras using joint invariants. In: Proc. Int. Conf. Acoust., Speech and Signal Process (2009) 2. Astrom, K., Morin, L.: Random cross ratios. In: Proc. 9th Scand. Conf. Image Anal., pp. 1053– 1061 (1995) 3. Chum, O., Philbin, J., Zisserman, A.: Near duplicate image detection: min-hash and tf-idf weighting. In: Proc. British Mach. Vision Conf. (2008) 4. Clarot, P., Ermis, E.B., Jodoin, P.M., Saligrama, V.: Unsupervised camera network structure estimation based on activity. In: Proc. 3rd ACM/IEEE Int. Conf. Dist. Smart Cameras (2009) 5. Collins, R., Lipton, A., Fujiyoshi, H., Kanade, T.: Algorithms for cooperative multisensor surveillance. Proc. IEEE 89(10), 1456–1477 (2001) 6. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 381–395 (1981) 7. Hann, C.E.: Recognizing two planar objects under a projective transformation. PhD dissertation, Univ. of Canterbury (2001) 8. Hann, C.E., Hickman, M.S.: Recognising Two Planar Objects under a Projective Transformation. Kluwer Academic, Dordrecht (2004) 9. Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Trans. Syst. Man Cybern. Part C, Appl. Rev. 34(3), 334–352 (2004) 10. Huynh, D.Q.: The cross ratio: a revisit to its probability density function. In: Proc. 11th British Machine Vision Conf. (2000) 11. Lee, L., Romano, R., Stein, G.: Monitoring activities from multiple video streams: establishing a common coordinate frame. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 758–767 (2000) 12. Lowe, D.: Object recognition from local scale-invariant features. In: Proc. 7th Int. Conf. Computer Vision, pp. 1150–1157 (1999) 13. Marhic, B.M., Mouaddib, E.M., Pegard, C.: A localisation method with an omnidirectional vision sensor using projective invariant. In: Proc. Int. Conf. Intelligent Robots Systems, pp. 1078–1083 (1998) 14. Mariottini, G., Prattichizzo, D.: EGT for multiple view geometry and visual serving. IEEE Robot. Autom. Mag. 12(4), 26–39 (2005) 15. Maybank, S.J.: Probabilistic analysis of the application of the cross ratio to model based vision. Int. J. Comput. Vis. 14, 199–210 (1995) 16. Mundy, J.L., Zisserman, A. (eds.): Geometric Invariance to Computer Vision. MIT Press, Cambridge (1992) 17. Mundy, J.L., Zisserman, A., Forsyth, D. (eds.): Applications of Invariance in Computer Vision. Springer, Berlin (1993) 18. Olver, P.J.: Joint invariant signatures. Found. Comput. Math. 1, 3–67 (2001) 19. Orrite, C., Blecua, S., Herrero, J.E.: Shape matching of partially occluded curves invariant under projective transformation. Comput. Vis. Image Underst. 93, 34–64 (2004) 20. Remagnino, P., Velastin, S., Foresti, G., Trivedi, M.: Novel concepts and challenges for the next generation of video surveillance systems. Mach. Vis. Appl. 18(3), 135–137 (2007)
54
R. Arora and C.R. Dyer
21. Roh, K.S., Lee, W.H., Kweon, I.S.: Obstacle detection and self-localization without camera calibration using projective invariants. In: Proc. Int. Conf. Intelligent Robots and Systems, pp. 1030–1035 (1997) 22. Rothwell, C.A., Zisserman, A., Forsyth, D.A., Mundy, J.L.: Canonical frames for planar object recognition. In: Proc. 2nd European Conf. Computer Vision (1992) 23. Rothwell, C.A., Zisserman, A., Forsyth, D.A., Mundy, J.L.: Planar object recognition using projective shape representation. Int. J. Comput. Vis. 16, 57–99 (1995) 24. Shashua, A.: A geometric invariant for visual recognition and 3D reconstruction from two perspective/orthographic views. In: Proc. IEEE Workshop on Qualitative Vision, pp. 107–117 (1993) 25. Snavely, N., Seitz, S., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 25(3), 835–846 (2006) (Proc. SIGGRAPH) 26. Tsonis, V.S., Chandrinos, K.V., Trahanias, P.E.: Landmark-based navigation using projective invariants. In: Proc. Int. Conf. Intell. Robots Systems, pp. 342–347 (1998) 27. Velipasalar, S., Wolf, W.: Frame-level temporal calibration of video sequences from unsynchronized cameras by using projective invariants. In: Proc. Advanced Video Signal-based Surveillance, pp. 462–467 (2005) 28. Wisconsin Computer Vision Group: Repository for joint-invariant matching. http://www.cae. wisc.edu/~sethares/links/raman/JICRvid/space.html
Chapter 4
Multiple-View Object Recognition in Smart Camera Networks Allen Y. Yang, Subhransu Maji, C. Mario Christoudias, Trevor Darrell, Jitendra Malik, and S. Shankar Sastry
Abstract We study object recognition in low-power, low-bandwidth smart camera networks. The ability to perform robust object recognition is crucial for applications such as visual surveillance to track and identify objects of interest, and overcome visual nuisances such as occlusion and pose variations between multiple camera views. To accommodate limited bandwidth between the cameras and the base-station computer, the method utilizes the available computational power on the smart sensors to locally extract SIFT-type image features to represent individual camera views. We show that between a network of cameras, high-dimensional SIFT histograms exhibit a joint sparse pattern corresponding to a set of shared features in 3-D. Such joint sparse patterns can be explicitly exploited to encode the distributed signal via random projections. At the network station, multiple decoding schemes are studied to simultaneously recover the multiple-view object features based on a distributed compressive sensing theory. The system has been implemented on the Berkeley CITRIC smart camera platform. The efficacy of the algorithm is validated through extensive simulation and experiment. A.Y. Yang () · S. Maji · C.M. Christoudias · T. Darrell · J. Malik · S.S. Sastry Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, USA e-mail:
[email protected] S. Maji e-mail:
[email protected] C.M. Christoudias e-mail:
[email protected] T. Darrell e-mail:
[email protected] J. Malik e-mail:
[email protected] S.S. Sastry e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_4, © Springer-Verlag London Limited 2011
55
56
A.Y. Yang et al.
Keywords Distributed object recognition · Wireless camera networks · Compressive sensing
1 Introduction Object recognition has been extensively studied in computer vision. In the traditional formulation, a vision system captures multiple instances of an object from a set of object classes, and is asked to classify a new test image that may contain one or more known object classes. Successful methods have been demonstrated in the past, including pedestrian detection [16], general object detection [1, 27] (e.g., vehicles and animals), and scene annotation [17, 24] (e.g., buildings, highways, and social events). A large body of these works have been based on analysis of certain local image patches that are robust/invariant to image scaling, affine transformation, and visual occlusion, which are the common nuisances in image-based object recognition. The local image patches are typically extracted by a viewpoint-invariant interest point detector [20] combined with a descriptor, e.g., SIFT (Scale-Invariant Feature Transform) [3, 18]. In this paper, we consider a relatively new scenario where a network of smart cameras are set up to simultaneously acquire an ensemble of images when a common object can be viewed from multiple vantage points. Traditionally, investigators often assume that the cameras are reliably connected to a central computer with no bandwidth limitation. As a result, the multiple-view images (or their SIFT representations) would be streamlined back to the computer, and the whole recognition process would be constructed in a centralized fashion. Recent studies in distributed object recognition have been mainly focused on two directions. First, when multiple images share a set of common visual features, correspondence can be established across camera views. This indeed was the original motivation of the SIFT framework [18]. More recently, [11, 27] proposed to harness the prior spatial distribution of specific features to guide the multiple-view matching process and improve recognition. Using random projections, [31] argued that reliable feature correspondence can be estimated in a much lower-dimensional space between cameras communicating under rate constraints. Second, when the camera sensors do not have sufficient bandwidth to streamline the high-dimensional visual features and perform feature matching, distributed data compression [10] can be utilized to encode and transmit the features. In particular, several methods have been proposed to directly compress 2-D visual features, such as PCA, entropy coding, and semantic hashing [14, 19, 26, 29]. For SIFT-type visual histograms, [5] proposed a rate-efficient codec to compress scalable tree structures in describing the hierarchy of histograms. In another work, [8] studied a multipleview SIFT feature selection algorithm. The authors argued that the number of SIFT features that need to be transmitted can be reduced by considering the joint distribution of the feature histograms among multiple camera views. However, the selection of the joint features depends on learning the mutual information among different camera views, and their relative positions must be fixed.
4 Multiple-View Object Recognition in Smart Camera Networks
57
1.1 Contributions We propose a novel distributed object recognition system suitable for band-limited smart camera networks. The contributions of this paper are two-fold: First, based on compressive sensing theory, we propose an effective distributed compression scheme to encode high-dimensional visual histograms on individual camera sensors. In particular, we explicitly exploit the nonnegativity and the joint sparsity properties in multiple-view histograms to achieve state-of-the-art compression for multipleview recognition. No communication between the cameras is necessary to exchange mutual information about the scene. Random projections will be used to provide dimensionality reduction, which is particularly adept for sensor network applications. Note that the paper does not address compression of object images or SIFT-type visual features per se, which may not exhibit joint sparse pattens across multiple camera views. Second, we detail the design of the distributed recognition system. On the sensor side, a smart camera sensor platform called CITRIC [6] is utilized. The substantial computational capability on CITRIC enables a fast implementation of the SURF (Speeded-Up Robust Features) detector [3] and compression of the histograms. On the station side, we demonstrate that the multiple-view object histograms can be jointly recovered using 1 -minimization (1 -min) solvers. Finally, the object class from the multiple views is classified using standard support vector machines (SVMs). We conduct extensive simulation and a real-world experiment to validate the performance of the system, in which the Columbia COIL-100 object image database [21] is used.
2 Encoding Multiple-View Features via Sparse Representation Suppose multiple camera sensors are equipped to observe a 3-D scene from multiple vantage points. As the wireless network and its placement is not an essential part of the paper, we can assume the sensors communicate with a base station via a single-hop wireless network. Using a visual feature detector (e.g., SIFT or SURF), viewpoint-invariant features can be extracted from the images, as shown in Fig. 1. These local features are called codewords. If one is given a large set of training images that capture the appearance of multiple object classes, the codewords from all the object categories then can be clustered based on their visual similarities into a vocabulary (or codebook). The clustering
Fig. 1 Detection of SURF features (red circles) on two image views of a toy. The correspondence of the interest points is highlighted via red lines
58
A.Y. Yang et al.
Fig. 2 The histograms representing the image features in Fig. 1
normally is based on a hierarchical k-means process [22]. The size of a typical vocabulary ranges from thousands to hundreds of thousands. Given a large vocabulary that contains codewords from many object classes, the representation of the visual features in a single object image is then sparse, which is called a feature histogram. Since only a small number of features are exhibited on a specific object, their values (or votes) in the histogram are positive integers, and the majority of the histogram values should be (close to) zero, as shown in Fig. 2. We define the problem of multiple-view histogram compression: Problem 1 (Distributed Compression of Joint Sparse Signals) When L camera sensors are equipped to observe a single 3-D object, the extracted histograms x 1 , x 2 , . . . , x L ∈ RD are assumed to be nonnegative and sparse. Further, the corresponding images may share a set of common features in the multiple views. Hence, a joint sparsity (JS) model is defined as: x 1 = x˜ + z1 , .. . x L = x˜ + zL .
(1)
In (1), x˜ is called the common sparsity, and zi is called an innovation [10]. Both x˜ and zi are also sparse and nonnegative. Suppose no communication is allowed between the L cameras: 1. On each camera, employ an encoding function f : x i ∈ RD → y i ∈ Rd (d < D) that compresses the histogram. 2. At the base station, once y 1 , y 2 , . . . , y L are received, simultaneously recover the histograms x 1 , x 2 , . . . , x L and classify the object class.
2.1 Random Projections We first discuss choosing a projection function to encode the histogram vectors x. In particular, a linear projection function is defined as: f : y = Ax, where A ∈ Rd×D is in general a full-rank matrix with d < D.
(2)
4 Multiple-View Object Recognition in Smart Camera Networks
59
Recently, a special projection function called random projections has gained much publicity in applications where the prior information of the data and the computational power of the sensor modalities are limited [4, 6, 31]. In this case, each element aij of A is independently drawn from a zero-mean Gaussian distribution. One can further simplify the implementation by using a Bernoulli distribution of two values (+1, −1) with equal probability. Compared with other linear projections, the main advantages of random projections are two-fold: 1. Random projections are efficient to generate using a pseudorandom number generator, and they do not depend on any domain-specific training set. 2. In terms of robustness to wireless congestion and packet loss, if (part of) the projected coefficients are dropped from the communication, the node needs not resend the coefficients, so long as the receiver can keep track of the packet IDs to reconstruct a partial random matrix with a lower dimension d in (2). Clearly, encoding x using an overcomplete dictionary A in (2) leads to infinitely many solutions. Before we discuss how to uniquely recover x, remember the goal of Problem 1 is to classify the object class in a 3-D scene. Indeed, one can directly utilize the randomly projected features y in the d-dim feature space for recognition. An important property of random projections is that they preserve the pairwise Euclidean distance, known as the Johnson–Lindenstrauss (J–L) lemma [13]: Theorem 1 (Johnson–Lindenstrauss Lemma) Let 0 < < 1 and an integer n for the number of any point cloud X ⊂ RD . For any d ≥ 4( 2 /2 − 3 /3)−1 log n, random projections f : RD → Rd preserve the pairwise Euclidean distance with high probability: 2 (3) (1 − )x i − x j 22 ≤ f (x i ) − f (x j )2 ≤ (1 + )x i − x j 22 , where x i and x j are any two points in the point cloud X . The J–L lemma essentially provides the following guarantee: For applications where only pairwise 2 -distances are concerned, it suffices to use the randomly projected features y and “throw away” the original data. In machine learning, random projections have been applied to reducing the data complexity for k-nearest neighbor (kNN) [2, 31]. On the other hand, note that Gaussian random projections do not guarantee the bounds for other p -norms with (p < 2). Particularly in object recognition, the similarity between different histograms is often measured w.r.t. the 1 -norm using histogram intersection kernels [12], which will be discussed in more detail later. For clarity, in the paper, our discussion will be limited to Gaussian random projections. Another observation about the J–L lemma is that the lower bound of the projection dimension d depends on the number of samples n. However, the lemma does not assume any special structure of the point cloud in the high-dimensional space. If we further assume the source signal x is sufficiently sparse, e.g., in the case of feature histograms computed over a large vocabulary, each x then can be reliably recovered from its random observations y. This “inverse” process is the main subject in compressive sensing [4, 9].
60
A.Y. Yang et al.
Theorem 2 Given a sparse signal x 0 , denote k as the sparsity (i.e., x 0 0 = k). Then for large D, with high probability, there exists a constant ρ = ρ(A) in (2) such that for every x 0 with its sparsity k < ρd, x 0 is the unique solution of the 1 -min program: (P1 ):
min x1
subject to y = Ax.
(4)
Clearly, the condition ρ in Theorem 2 is a function of the matrix A. In fact, for a particular A matrix, ρ can be exactly quantified in convex polytope theory [9]. In the rest of this subsection, we will first overview this relationship. . Figure 3 illustrates a projection between a cross polytope C = C 3 ⊂ R3 and its image AC ⊂ R2 . In general, a cross polytope C D in RD is the collection of vectors {x : x1 ≤ 1}. For any k-sparse vector x, x1 = 1, one can show that x must lie on a (k − 1)-face of C D . With projection A ∈ Rd×D , AC is an induced quotient polytope in the d-dim space. It is important to note that some of the vertices and faces of C may be mapped to the interior of AC, i.e., they do not “survive” the projection. Theorem 3 1. For a projection matrix A ∈ Rd×D , the quotient polytope AC is called kneighborly if all the k-faces of C D are mapped to the boundary of AC. Any sparse signal x ∈ RD with (k + 1) or less sparse coefficients can be recovered by (P1 ) if and only if AC is k-neighborly. 2. For a specific (k +1)-sparse signal x ∈ RD , x must lie on a unique k-face F ⊂ C. Then x can be uniquely recovered by (P1 ) if and only if AF is also a k-face of AC. Theorem 3 is a powerful tool to examine if a sparse signal under a projection A can be uniquely recovered by (P1 ). For example, in Fig. 3, AC is 0-neighborly. Therefore, any 1-sparse signal can be uniquely recovered by (P1 ). However, for a specific x on a 1-face of C, x is 2-sparse and it is projected to a 1-face of AC. Hence, x also can be uniquely recovered via (P1 ). For a specific A matrix that depends on the application, one can simulate the projection by sampling vectors x on all the k-faces of C. If with high probability, the projection Ax survives (i.e., on the boundary of AC), then AC is at least
Fig. 3 Projection of a cross polytope C in R3 to a quotient polytope AC via projection A. The corresponding simplex is T at the shaded area. Both AC and AT are 0-neighborly
4 Multiple-View Object Recognition in Smart Camera Networks
61
k-neighborly. The simulation provides a practical means to verify the neighborliness of a linear projection, particularly in high-dimensional data spaces. On the other hand, a somewhat surprising property guarantees the well-behavior of random projections: In a high-dimensional space, with high probability, random projections preserve most faces of a cross polytope. A short explanation to this observation is that most randomly generated column vectors in A are linearly independent.
2.2 Enforcing Nonnegativity in 1 -Minimization Given the observations y 1 , y 2 , . . . , y L , (P1 ) provides a solution to independently recover each of the ensemble elements x 1 , x 2 , . . . , x L . However, such a solution fails to observe that in our application the sparse signals x represent image histograms and therefore are strictly nonnegative, and it also fails to enforce the possible joint sparse patterns that are shared among multiple camera views. In this subsection, we first discuss how to impose nonnegativity in 1 -min. Assuming nonnegative x is normalized to have unit 1 -norm without loss of gen. erality, we denote T = T D−1 as the standard simplex in RD , i.e., (5) T = x : x1 = 1 and x ≥ 0 . Figure 3 shows the relationship between C D and T D−1 . Hence, a k-sparse nonnegative vector x must lie on a (k − 1)-face of T , which is only a small subset of the cross polytope. The following theorem shows that the nonnegativity constraint reduces the domain of possible solutions for 1 -min [9]: Theorem 4 1. Any nonnegative sparse signal x ∈ RD with k or less sparse coefficients can be recovered by (P1 ):
min x1
subject to y = Ax and x ≥ 0
(6)
if and only if all (k − 1)-faces of T D−1 survive the projection A. 2. For a specific nonnegative k-sparse signal x, x must lie on a unique (k − 1)-face F ⊂ T . Then x can be uniquely recovered by (P1 ) if and only if AF is also a (k − 1)-face of AT . The nonnegative 1 -min (6) is a linear program, and can be solved by efficient algorithms, such as orthogonal matching pursuit (OMP) and polytope faces pursuit (PFP) [23]. These algorithms are usually preferred in sensor network applications compared to other more expensive programs (e.g., interior-point methods [7, 28]). In both simulation and experiment on real-world data, we have found that PFP is a more efficient algorithm than interior-point methods to impose the nonnegativity constraint, and produces better results than OMP. In the rest of the paper, PFP is the 1 -solver of our choice.
62
A.Y. Yang et al.
2.3 Estimation of Joint Sparse Signals We propose to adopt a joint sparsity (JS) model [10] to directly recover the common sparse signal and the sparse innovations as the following: y 1 = A1 (x˜ + z1 ) = A1 x˜ + A1 z1 , .. . y L = AL (x˜ + zL ) = AL x˜ + AL zL ,
(7)
where both x˜ and z1 , . . . , zL are assumed to be nonnegative. The JS model can be directly solved in the following linear system via PFP: ⎡ ⎤ ⎤ x˜ ⎡ ⎤ ⎡ A1 A1 0 · · · 0 ⎢ ⎥ y1 ⎥ ⎢ z1 ⎥ ⎢ .. ⎥ ⎢ .. .. .. ⎦ ⎢ .. ⎥ ⎣ . ⎦=⎣ . . . ⎣ . ⎦ AL 0 · · · 0 AL yL zL ⇔
y = A x ∈ RdL .
(8)
The global projection function (8) projects a D(L + 1)-dim nonnegative sparse signal x onto a dL-dim subspace defined by matrix A . The new linear system also improves the sparsity w.r.t. the total data space. As an example, suppose for each camera in (2), ρ = dk , and x ˜ 0 = k2 and zi 0 = k2 . Then the new sparsity ratio in (8) becomes ρ =
(L + 1)k/2 L + 1 = ρ. dL 2L
(9)
With a large L for the number of the cameras, x becomes much sparser and can be recovered more accurately via 1 -min. Example 5 Suppose the triplet D = 1000, d = 200, and k = 60. To simulate multiple-view histograms, three k-sparse histograms x 1 , x 2 , x 3 ∈ R1000 are randomly generated with nonzero coefficients between 0 and 1, and then randomly projected to a 200-dim space. Among the 60 nonzero coefficients in each histogram, different combinations of common sparsity and innovation are constructed, as shown ˜ 0 = 60 and z0 = 0 to x in Table 1, from x ˜ 0 = 30 and z0 = 30. We evaluate the performance of OMP, PFP, and JS, based on their 0 -norm distortion (i.e., sparse support error) and 2 -norm distortion between the ground truth and the estimate. First, since both OMP and PFP do not consider any joint sparsity, each x is independently recovered from its projection y. Hence, their performance should not change w.r.t. different sparsity combinations. Without enforcing the nonnegativity, OMP basically fails to recover any meaningful nonnegative sparse patterns. On the other hand, the average sparse support error for PFP that enforces the nonnegativity is much smaller.
4 Multiple-View Object Recognition in Smart Camera Networks Table 1 Average 0 -error and 2 -error of OMP, PFP, and JS over 100 trials. The two numbers in the parentheses indicate the sparsity in the common sparse signal and the innovation, respectively. The best results are indicated in bold numbers
63
Sparsity
(60, 0)
(40, 20)
(30, 30)
0OMP
56.14
56.14
56.14
2OMP
1.76
1.76
1.76
0PFP 2PFP 0JS 2JS
3.48
3.48
3.48
0.05
0.05
0.05
1.85
1.65
1.95
0.02
0.02
0.02
Overall, the JS model achieves the best performance. First, w.r.t. different combinations of common sparsity and innovation, the average support error stays consistent, which shows the method adapts well to the presence of innovation signals in the multiple-view histograms. More importantly, the method achieves a very low estimation error both in 0 and 2 . Out of 60 nonzero coefficients, only one coefficient is misidentified in the 1000-D ambient space. Finally, taking advantage of the JS model, flexible strategies can be proposed for choosing the random projection dimensions di . A necessary condition for simultaneously recovering x 1 , . . . , x L can be found in [10]. Basically, it requires each sampling rate δi = di /D guarantees that the so-called minimal sparsity signal of zi is sufficiently encoded, and the total sampling rate must also guarantee that both the common sparsity and the innovations are sufficiently encoded.1 This result suggests a flexible strategy for choosing varying sampling rates and communication bandwidth, that is, the random project dimensions di need not to be the same for the L sensors to guarantee perfect recovery of the distributed data. For example, sensor nodes in a network that have lower bandwidth or lower power reserve can choose to reduce the sampling rate in order to preserve energy.
3 System Implementation The complete recognition system has been implemented on the Berkeley CITRIC smart camera platform [6] and a computer as the base station. The design of the CITRIC platform provides considerable computational power to execute SURF feature extraction and histogram compression on the sensor board. Each CITRIC mote 1 The
strategy of choosing varying sampling rate is a direct application of the celebrated Slepian– Wolf theorem [25]. In a nutshell, the theorem shows that, given two sources X1 and X2 that generate sequences x1 and x2 , asymptotically, the sequences can be jointly recovered with vanishing error probability if and only if R1 > H (X1 |X2 ),
R2 > H (X2 |X1 ),
R1 + R2 > H (X1 , X2 ),
where R is the bit rate function, H (Xi |Xj ) is the conditional entropy for Xi given Xj , and H (Xi , Xj ) is the joint entropy.
64
A.Y. Yang et al.
Fig. 4 SURF features detected from a corridor scene (left) and a tree object (right)
consists of a camera sensor board running embedded Linux and a TelosB network board running TinyOS. The camera board integrates a 1.3 megapixel SXGA CMOS image sensor, a frequency-scalable (up to 624 MHz) microprocessor, and up to 80 MB memory. We have ported an Open SURF library to extract SURF features.2 The algorithm is based on sums of approximated 2D Haar wavelet responses, and it also makes use of integral images to speed up the keypoint detection and descriptor extraction. The quantization process yields a 64-D vector. Figure 4 illustrates two examples. The TelosB network board uses the IEEE 802.15.4 protocol to communicate between camera nodes and the base station. The typical bandwidth is 250 Kbps. To measure the speed of the system on the camera sensor, we have conducted a real-world experiment at multiple locations in an office building [30]. Overall, the CITRIC system takes about 10–20 seconds to extract SURF features from 320× 240 grayscale images and transmits the compressed histograms y to the base station, depending on the number of SURF features and the dimension of random projections. We believe the limitation of the CITRIC platform for computation-intensive applications can be mitigated in a future hardware update with a state-of-the-art floating-point mobile processor and a faster data rate between the CITRIC mote and the TelosB network mote. At the base station, upon receiving the compressed features from the L cameras, the original sparse histograms x 1 , x 2 , . . . , x L are simultaneously recovered. In order to identify M object classes w.r.t. each individual camera view, we train one-vs.-one SVM classifiers for every pair of categories. We use LibSVM3 with the histogram intersection kernel for learning classifiers. This kernel and its variants such as the pyramid match kernel [12] have been shown to work quite well for visual recognition. 2 The
Open SURF project is documented at: http://code.google.com/p/opensurf1/.
3 http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
4 Multiple-View Object Recognition in Smart Camera Networks
65
When multiple views are available, there are various ways to use them to improve recognition. The simplest being one that enforces agreement between the views by means of majority voting. One may also learn to classify in a joint representation domain directly. On the other hand, most existing methods must assume the relative camera positions are known and fixed. We leave the exploration of this direction for future research.
4 Experiment To demonstrate the performance of the algorithm on real multiple-view images, we utilize the public COIL-100 dataset. This dataset consists of 72 views of 100 objects imaged from 0 to 360 degrees in 5 degree increments. In this setting we perform instance-level recognition and demonstrate the performance of our approach with varying number of random projection dimensions. The imaging process on the CITRIC mote is simulated by directly uploading the COIL images to the camera memory for processing. A local feature representation was computed for each image using 10-D PCASURF descriptors extracted on a regular grid with a 4 pixel spacing that were combined with their image location to form a 12-D feature space. The features from a subset of the COIL-100 images were then used to compute the vocabulary of a multi-resolution histogram image representation found using hierarchical k-means with LIBPMK [15]. We used 4 levels and a branching factor of 10 to get a 991 word vocabulary at the finest level of the hierarchy. We represent each image and perform 1 recovery using the finest level of the hierarchical histogram, and similarity between images is computed using histogram intersection over the resulting 991-D histogram vectors corresponding to each image. For each object, 10 training examples are sampled uniformly from the complete 360 degree viewing circle. To classify a query image, we use the projected features from its neighboring views in the dataset to jointly recover the features, but the classification is performed on a per-view basis for a fair comparison of the quality of the distributed compression algorithm. Figure 5 shows the performance of various compression methods on this dataset. The solid line on the top shows the ground-truth recognition accuracy assuming no compression is included in the process, and the computer has direct access to all the SURF histograms. Hence, the upper-bound per-view recognition rate is about 95%. When the histograms are compressed via random projections, in the low-dimension regime, the random projection space works quite well to directly classify the object classes. For example, at 200-D, directly applying SVMs in the random projection space achieves about 88% accuracy. However, the accuracy soon flattens out and is overtaken by the 1 -min methods when the projected feature dimension becomes high enough. Since the 1 -min scheme provides a means to recover the sparse histograms in the high-dimensional space, when the dimension of random projections becomes sufficiently high, the accuracy via PFP surpasses the random projection features, and
66
A.Y. Yang et al.
Fig. 5 Per-view classification accuracy (in color) w.r.t. random projection dimensions: 1. Randomly projected features. 2. PFP in one view. 3. JS in two to three views. The top curve is the baseline recognition rate without histogram compression
Fig. 6 (Left) 1 -error and (Right) 2 -error using 1–3 camera views
approaches the baseline performance beyond 600-D. Furthermore, when more camera views are available, the joint sparsity model significantly boosts the accuracy by as much as 50%, as seen in Fig. 5. For example, at 200-D, the recognition accuracy for PFP is about 47%, but it jumps to 71% with two camera views, and 80% with three views. JS has also been shown to reduce the 1 - and 2 -recovery-error in Fig. 6.
4 Multiple-View Object Recognition in Smart Camera Networks
67
5 Conclusion and Discussion We have studied the problem of distributed object recognition in band-limited smart camera networks. The main contribution of the solution is a novel compression framework that encodes SIFT-based object histograms. We exploit three important properties of multiple-view image histograms of a 3-D object: histogram sparsity, nonnegativity, and multiple-view joint sparsity. The complete recognition system has been implemented on the Berkeley CITRIC smart camera platform. One of the limitations in the current solution is that the algorithm lacks a mechanism to classify and associate multiple objects in the scene. This is due to the fact that each histogram is being treated as a holistic representation of the 3-D scene. Another limitation is that the classification via SVMs is conducted on a per-view basis, although majority-voting can be trivially applied to incorporate the multiple views to some extent. Future solutions to these questions must carefully study the detailed structure of sparse histograms in full granularity, and answer how the association of visual features can improve the classification across multiple camera views in a band-limited sensor network. Acknowledgements This work was supported in part by ARO MURI W911NF-06-1-0076. The authors thank Kirak Hong and Posu Yan of the University of California, Berkeley, for the implementation of SURF on the CITRIC camera platform.
References 1. Agarwal, S., Roth, D.: Learning a sparse representation for object detection. In: Proceedings of the European Conference on Computer Vision (2002) 2. Ailon, N., Chazelle, B.: Approximate nearest neighbors and the fast Johnson–Lindenstrauss transform. In: Proceedings of the ACM Symposium on Theory of Computing (2006) 3. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. Comput. Vis. Image Underst. (2008) 4. Candès, E., Tao, T.: Near optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inf. Theory (2006) 5. Chen, D., et al.: Tree histogram coding for mobile image matching. In: Proceedings of the IEEE Data Compression Conference (2009) 6. Chen, P., et al.: CITRIC: A low-bandwidth wireless camera network platform. In: Proceedings of the International Conference on Distributed Smart Cameras (2008) 7. Chen, S., Donoho, D., Saunders, M.: Atomic decomposition by basis pursuit. SIAM Rev. (2001) 8. Christoudias, C., Urtasun, R., Darrell, T.: Unsupervised feature selection via distributed coding for multi-view object recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2008) 9. Donoho, D., Tanner, J.: Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. (2005) 10. Duarte, M., et al.: Distributed compressed sensing of jointly sparse signals. In: Proceedings of the Thirty-Ninth Asilomar Conference on Signals, Systems and Computers (2005) 11. Ferrari, V., Tuytelaars, T., Van Gool, L.: Integrating multiple model views for object recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2004)
68
A.Y. Yang et al.
12. Grauman, K., Darrell, T.: The pyramid match kernel: discriminative classification with sets of image features. In: Proceeding of International Conference on Computer Vision (2005) 13. Johnson, W., Lindenstrauss, J.: Extensions of Lipschitz maps into a Hilbert space. Contemp. Math. (1984) 14. Ke, Y., Sukthankar, R.: PCA-SIFT: A more distinctive representation for local image descriptors. In: Proceedings of International Conference on Computer Vision and Pattern Recognition (2004) 15. Lee, J.: A pyramid match toolkit. MIT CSAIL Tech Report: MUT-CSAIL-TR-2008-017 (2008) 16. Leibe, B., Seemann, E., Schiele, B.: Pedestrian detection in crowded scenes. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2005) 17. Liu, C., et al.: SIFT flow: dense correspondence across different scenes. In: Proceedings of European Conference on Computer Vision (2008) 18. Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of International Conference on Computer Vision (1999) 19. Makar, M., Chang, C., Chen, D., Tsai, S., Girod, B.: Compression on image patches for local feature extraction. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (2009) 20. Mikolajczyk, K., et al.: A comparison of affine region detectors. Int. J. Comput. Vis. (2005) 21. Nene, S., Nayar, S., Murase, H.: Columbia object image library (COIL-100). Columbia University Tech Report: CUCS-006-96 (1996) 22. Nistér, D., Stewénius, H.: Scalable recognition with a vocabulary tree. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2006) 23. Plumbley, M.: Recovery of sparse representations by polytope faces pursuit. In: Proceedings of International Conference on Independent Component Analysis and Blind Source Separation (2006) 24. Quattoni, A., Collins, M., Darrell, T.: Transfer learning for image classification with sparse prototype representations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2008) 25. Slepian, D., Wolf, J.: Noiseless coding of correlated information sources. IEEE Trans. Inf. Theory (1973) 26. Takacs, G., Chandrasekhar, V., Gelfand, N., Xiong, Y., Chen, W., Bismpigiannis, T., Grzeszczuk, R., Pulli, K., Girod, B.: Outdoor augmented reality on mobile phone using loxelbased visual feature organization. In: Proceedings of International Multimedia Conference (2008) 27. Thomas, A., et al.: Towards multi-view object class detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2006) 28. Tibshirani, R.: Regression shrinkage and selection via the LASSO. J. R. Stat. Soc. B (1996) 29. Weiss, Y., Torralba, A., Fergus, R.: Spectral hashing. In: Proceedings of Advances in Neural Information Processing Systems (2008) 30. Yang, A., et al.: Distributed compression and fusion of nonnegative sparse signals for multipleview object recognition. In: Proceedings of International Conference on Information Fusion (2009) 31. Yeo, C., Ahammad, P., Ramchandran, K.: Rate-efficient visual correspondences using random projections. In: Proceedings of International Conference in Image Processing (2008)
Chapter 5
A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network Yiming Li and Bir Bhanu
Abstract Video networks are becoming increasingly important for solving many real-world problems. Multiple video sensors require collaboration when performing various tasks. One of the most basic tasks is the tracking of objects, which requires mechanisms to select a camera for a certain object and hand-off this object from one camera to another so as to accomplish seamless tracking. In this chapter, we provide a comprehensive comparison of current and emerging camera selection and hand-off techniques. We consider geometry-, statistics-, and game theory-based approaches and provide both theoretical and experimental comparison using centralized and distributed computational models. We provide simulation and experimental results using real data for various scenarios of a large number of cameras and objects for in-depth understanding of strengths and weaknesses of these techniques.
Keywords Utility-based game theoretic approach · Co-occurrence to occurrence ratio · Constraint satisfaction problem · Fuzzy-based
1 Introduction The growing demand for security in airports, banks, shopping malls, homes, etc. leads to an increasing need for video surveillance, where camera networks play an important role. Significant applications of video network include object tracking, object recognition and object activities analysis from multiple cameras. The cameras in a network can perform various tasks in a collaborative manner. Multiple cameras Y. Li () · B. Bhanu University of California, Riverside, Riverside, CA, USA e-mail:
[email protected] B. Bhanu e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_5, © Springer-Verlag London Limited 2011
69
70
Y. Li and B. Bhanu
enable us to have different views of the same object at the same time, such that we can choose one or some of them to monitor a given environment. However, since multiple cameras may be involved over long physical distances, we have to deal with the hand-off problem. Camera hand-off is the process of switching from the current camera to another one to follow an object seamlessly [1]. This has been an active area of research and many approaches have been proposed. Some camera networks require switches (video matrix) to help monitor the scenes in different cameras [2]. The control can be designed to switch among cameras intelligently. Both distributed and centralized systems are proposed. Some researchers provide hardware architecture design, some of which involve embedded smart cameras, while others focus on the software design for camera assignment. This chapter first gives a comprehensive review for the existing related works and then focuses on a systematic comparison of the techniques for camera selection and hand-off. Detailed experimental comparisons are provided for four selected techniques. This chapter is organized as follows: Sect. 2 gives a comprehensive background of the current and emerging approaches for camera selection and hand-off. Comparisons tables are provided for a macroscopic view of the existing techniques. Section 3 focuses on the theoretical comparison and analysis of four key approaches. Experimental comparisons are provided in Sect. 4. Finally, the conclusions are drawn in Sect. 5.
2 Related Work and Contributions The research work in camera selection and hand-off for a video network consisting of multiple cameras can be classified according to many different aspects, such as whether it is embedded /PC-based; distributed/centralized; calibrationneeded/calibration-free; topology-based or topology-free; statistics-based/statisticsfree, etc.
2.1 Comparison for Existing Works Some researchers work on the design for embedded smart cameras, which, usually, consist of a video sensor, a DSP or an embedded chip and a communication module. In these systems, such as [3–7], since all the processing can be done locally, the design work is done in a distributed manner. There are also some PC-based approaches that consider the system in a distributed manner, such as [8–11]. Meanwhile, a lot of centralized systems are proposed as well, such as [12–16]. Some work, such as [16], requires the topology of the camera network while some are image-based and do not have requirements for any priori knowledge of the topology. As a result, calibration is needed for some systems, while some systems, such as [17–20] are calibration-free. Active cameras (pan/tilt/zoom cameras) are used in some systems, such as [15, 16, 18], to obtain a better view of objects. However, to our knowledge, only a small amount of work has been done to propose a large-scale active camera
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
71
Table 1 Merits of various characteristics encountered in distributed video sensor networks Properties
Advantages
Disadvantages
Distributed
Low bandwidth requirement; No time requirement for image decoding; Easy to increase the number of nodes
Lack of global cooperation
Centralized
Easy for cooperation among cameras; Hardware architecture is relatively simple compared with distributed systems
Require more bandwidth; High computational requirements; May cause severe problem once the central server is down
Embedded
Easy to be used in real-world distributed system; Low bandwidth
Limited resources, such as memory, computing performance and power; Only simple algorithms have been used
PC-based
Computation can be fast; No specific hardware design requirements, like for embedded chips or DSPs
A bulky solution for many cameras
Calibrated
Can help to know the topology of the camera network; A must for PTZ cameras
Pre-processing is required; Calibration process may be time consuming
Uncalibrated
No offline camera calibration is required
Exact topology of cameras difficult
Active cameras
Provide better view of objects; Can save the number of cameras by pa/tilt to cover larger monitoring range
Camera calibration may be required, especially when zooming. Complex algorithms to account camera motions
Static/Mobile cameras
Low cost, high for mobile; Easy to determine topology of the camera network; Relatively simpler algorithms as compared with those for active (and mobile) cameras
More (static) cameras are needed to have a full coverage; Have no close-up if the object is not close to any cameras
network for video surveillance. More large-scale camera networks generally consist of static cameras. Images in 3D are generated in some systems, such as [6]. However, in most approaches proposed for the camera selection and hand-off, only 2D images are deployed. There are also other considerations, such as resource allocation [21], fusion of different types of sensors [22], etc. In Table 1, we compare the advantages and disadvantages for some of the important issues discussed above. Table 2 lists sample approaches from the literature and their properties. It is to be noticed that, not all the distributed systems are realized in an embedded fashion.
72
Y. Li and B. Bhanu
Table 2 A comparison for of some properties for selected approaches. (Legends for the table: HW—Hardware-wise; SW—Software-wise; E—Embedded; A—Active camera; D—Distributed; C—Calibration needed; RT—Real-time; RD—Real data; NC —Number of cameras; NP —Number of objects; T—Tracking algorithm used; O—Overlapping FOVs; Y—Yes; N—No; Y+—Yes but not necessary) Approaches
Quaritsch et al. [4]
HW
SW
Experiment Details
E
A
D
C
RT
RD
NC
NP
T
O
Y
N
Y
N
Y
Y
2
1
Camshift
N
Flech and Straßer [6]
Y
N
Y
N
Y
Y
1
1
Particle filter
Y
Park et al. [7]
N
N
Y
N
N/A
N
20
N/A
N/A
Y
N
Morioka et al. [8]
N
Morioka et al. [10]
N
Y
N
N/A
N
6
1
N/A
Y+
Y
Y
Y
Y
3
3
Kalman filter
Y
Qureshi et al. [11]
N
Kattnaker et al. [13]
N
Y
Y
Y
N
N
16
100
N/A
Y+
N
N
N
Y
Y
4
2
Bayesian
Everts et al. [14]
N
N
Y
N
Y
Y
Y
1
1
Histogram based
N
Li and Bhanu [17]
N
N
N
N
Y
Y
3
2
Camshift
Y+
Javed et al. [18]
N
N
N
N
Y
Y
2
2
N/A
Y
Jo and Han [20]
N
N
N
N
Y
Y
2
N/A
Manual
Y
Gupta et al. [23]
N
N
N
N
Y
Y
15
5
M2Tracker
Y
Song et al. [24]
N
N
Y
N
N
Y
7
9
Particle filter
N
Song et al. [25]
N
Y
Y
N
N
N
14
N/A
N/A
Y
For instance, a distributed camera node can consisted of a camera and a PC as well, although the trend is to realize distributed systems via embedded chips. That is why we treat distributed systems and embedded systems separately in Table 1. In Table 2, some approaches are tested using real data, while some provide only the simulation results. There is no guarantee that the systems, which are experimented on using synthetic data, can still work satisfactorily and realize real-time processing when using real data. So, the real-time property is left blank for those approaches whose experiments use simulated data. Similarly, most of the experiments are done for a small-scale camera network. The performance of the same systems for a large-scale camera network still needs to be evaluated.
2.2 Our Contributions The contributions of this chapter are: • A comprehensive comparison of recent work is provided for camera selection and hand-off. Four key approaches are compared both theoretically and experimentally. • Results with real data and simulations in various scenarios are provided for an in-depth understanding of the advantages and weaknesses of the key approaches.
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
73
The focus of comparison is solely on multi-object tracking using non-active multicameras in an uncalibrated system. The comparison considers software and algorithm related issues. Resource allocation, communication errors and hardware design are not considered.
3 Theoretical Comparison We selected four approaches [9, 11, 17, 20] for comparison. They are chosen as typical approaches because these approaches cover both distributed systems [9, 11] and centralized systems [17, 20]. Although none of these approaches needs camera calibration, some of them do a geometry correspondence [20] while some do not [9, 11, 17]. Approaches such as [11, 17] provide a more systematic approach to camera selection and hand-off. This section focuses on the comparison of theoretical ideas for while experimental comparison is provided in the next section. In this section, we first describe the key ideas of these approaches. Analysis of the advantages and disadvantages are provided in Table 3.
Table 3 Relative merits and shortcomings of the selected approaches Approaches
Pros
Cons
Utility-based game theoretic approach [17]
Provides a mathematical framework; Can deal with the cooperation and competition among cameras; Can perform camera selection based on user-supplied criteria
Communication among cameras is not involved, can be extended for distributed computation; The local utility has to be designed that will align with the global utility in a potential game
Co-occurrence to occurrence ratio approach [20]
Intuitive efficient approach; Acceptable results when there are few occlusions and few cameras and objects
Time consuming; When correspondence fails or occlusion happens, there is ambiguity; Becomes complicated when # of camera nodes/objects increases; FOVs have to overlap
Constraint satisfaction problem approach [11]
Provides a distributed system design; Camera nodes can cooperate by forming coalition groups; Conflicts among cameras are solved by the CSP
The backtracking approach is time consuming for solving the constraint satisfaction problem; Only simple constraints are provided; Only simulation (no real video) results are provided
Fuzzy-based approach [8]
Distributed approach; Camera state transition and handoff rules are both intuitive
Only simulation results are provided; Tracking has to be accurate; Not robust when occlusion happens; No guarantee for convergence in a large-scale network
74
Y. Li and B. Bhanu
3.1 Descriptions of the Key Ideas of Selected Approaches The Utility-Based Game Theoretic Approach This is the most systematic approach among the selected ones. It views the camera selection and hand-off problem in a game theoretic manner. There is the trend to consider the camera assignment problem as a cooperative multi-agent problem. The merit of [17] is that the authors come up with a complete mathematical mapping of the problem to a classical vehicle-target problem in game theory by viewing the cameras that can “see” an object as the multiple players in a game. The problem formulation considers both cooperation and competition among cameras for tracking an object, which demonstrates the main advantage of applying game theory. Camera utility, person utility, and the global utility are calculated: UCj (a). (1) • Global utility: Ug (a) = Cj ∈C
•
Person utility for Pi : UPi (a) = Ug (ai , a−i ) − Ug (C0 , a−i ).
•
Camera utility for Cj :
UCj (a) =
NCrt nP
Crtil .
(2) (3)
i=1 l=1
a = (ai , a−i ) is the camera assignment result. ai stands for the camera used to track person Pi , while a−i stands for the camera assignment for all the other persons other than Pi . The person utility implies the marginal contribution of camera ai to the global utility. Crtsl are the user-supplied criteria. It is shown that the design of the utility functions as above makes it a potential game. The final assignment result is given in the form of a mixed strategy: 1
eτ
pil (k) = e where l U Pi (k
+ 1) =
⎧ ⎨ U Pl (k) + i ⎩ U l (k), Pi
1 1 τ U Pi (k)
l
U P (k) i
1
+ ··· + eτ
n
U PC (k)
(4)
,
i
l 1 (UPi (a(k))) − U Pi (k), pil (k)
ai (k) = Ali ,
(5)
otherwise
is the predicted person utility in the (k + 1)th iteration step. Due to the limited space of this chapter, for more detailed explanations, please refer to [17].
The Co-occurrence to Occurrence Ratio (COR) Approach This approach decides whether two points are in correspondence with each other by calculating the co-occurrence to occurrence ratio (COR). If the COR is higher than some predefined threshold, then the two points are decided to be in correspondence with each other. When one point is getting close to the edge of the field of
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
75
view (FOV) of one camera, the system will hand-off to another camera that has its corresponding point. The COR is defined as p(x, x ) , (6) R(x, x ) = p(x) where p(x) =
T Nt 1 K2 x − xti T
(7)
t=1 i=1
is the mean probability that a moving object appears at x, i.e. the occurrence at x. K2 is claimed to be circular Gaussian kernel. Similarly,
Nt T Nt 1 x − xti p(x, x ) = K2 x − xti T
t=1 i=1
(8)
i=1
is the co-occurrence at x in one camera and x in another camera. It is intuitive that if two points x and x are in correspondence, i.e. the same point in the views of different cameras, then the calculated COR should be 1 ideally. On the contrary, if the x and x are completely independent on each other, i.e. two distinctive points, then p(x, x ) = p(x)p(x ), which leads the COR R(x, x ) to be p(x ). These are the two extreme cases. If we chose some threshold θr such that (x ) < θr < 1, then by comparing with θr , the correspondence of two points in two camera views can be determined. Another threshold θ0 is needed to be compared with p(x) to decide whether a point is detected in a camera. Thus, camera hand-off can be taken care of by calculating the correspondence of pairs of points in the views of different cameras and performed when necessary.
The Constraint Satisfaction Problem (CSP) Approach The approach discussed in [11] focuses on the system design. Unlike the previous two centralized systems, this system is designed to be distributed by deploying the local visual routines (LVRs). Camera controllers are modeled as a finite state machine with the Idle state, the ComputingRelevance state and the PerformingTest state. The cameras cooperate with each other by forming coalition groups, which is achieved by involving leader nodes and the auction/bidding mechanism for recruiting new nodes. When multiple cameras nodes are available for joining the group, a conflict resolution mechanism is realized by solving the constraint satisfaction problem. Three elements of a CSP are a set of variables {ν1 , ν2 , . . . , νk }, the domain of each νi Dom[νi ] and a set of constraints {C1 , C2 , . . . , Cm }. The authors apply backtracking to search among all the possible solutions and rank them according to the relevance to solve the CSP. BestSolv, which is based on the quality of the partial solution, is compared with the Allsolv, which is an exhaustive manner.
76
Y. Li and B. Bhanu
The Fuzzy-based Approach This is another decentralized approach. Each candidate camera has two states for the object that is in its FOV: the non-selected state and the selected state for tracking. Then, camera handoff is done based on the camera’s previous state S i and the tracking level state SS i , which is defined by estimating the position measurement error in the monitoring area. The two states for the tracking level are: unacceptable, meaning that the object is too far away and acceptable, meaning that the object is within the FOV and the quality is acceptable. The block diagram for camera state transition and the fuzzy rule for camera handoff are given in Fig. 1 [9] and Fig. 2 [9], respectively.
Fig. 1 Diagram for camera state transition
Fig. 2 Fuzzy-based reasoning rules
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
77
4 Experimental Results In this section, we perform experiments for the above four approaches in different cases. Although some of the approaches [9, 11] do not have results with real data, in this chapter, both indoor and outdoor experiments with real data are carried out for all the approaches. For convenience of comparison among different approaches, no cameras are actively controlled.
4.1 Data The experiments are done using commercially available AXIS 215 cameras. Three experiments are carried out with an increase in complexity. Case 1: two cameras three persons, indoor. Case 2: three cameras five persons, indoor. Case 3: four cameras six persons, outdoor. The frames are dropped whenever the image information is lost during the transmission. The indoor experiments use cable-connected cameras, with a frame rate of 30 fps. However, for the outdoor experiment, the network is wireless. Due to the low quality of the images, the frame rate is only 10–15 fps on average. The images are 60% compressed for the outdoor experiment to save bandwidth. Images are 4CIF, which 704 × 480. They are overlapped randomly in our experiments, which is not required by some of the approaches but required by some others.
4.2 Tracking None of the approaches discussed here depends on any particular tracker. Basically, ideal tracking can be assumed for comparing the camera selection and hand-off mechanisms. Trackings in all the experiments are initialized manually at the very beginning and then done with a color-based particle filter. The dynamic model used is random walk. Measurement space is two-dimensional: hue and saturation values of a pixel. The sample number used for each object to be tracked is 200 for indoor experiments and 500 for outdoor experiments. Tracking can be done in real time by implementing the OpenCV structure CvConDensation and the corresponding OpenCV functions. Matches for objects are done by calculating the correlation of the hue values using cvComparehist Minor occlusion is recoverable within a very short time. Tracking may fail when severe occlusion takes place or the case that an object is not in the scene for too long and then re-enters. Theoretically, this can be solved by spreading more particles. However, having more particles is computationally expensive. Thus, we just re-initialize the tracking process.
78
Y. Li and B. Bhanu
4.3 Parameters We first define the following properties of our system: • A person Pi can be in the FOV of more than one camera. The available cameras for Pi belong to the set Ai . • A person can only be assigned to one camera. The assigned camera for Pi is named as ai . • Each camera can be used for tracking multiple persons. (1) The Utility-based Game Theoretic approach: The utility functions are kept exactly the same as they are in [17]. The criteria used for calculating the cameras are the combined criterion mentioned in [17], i.e. a weighted sum of the other three criteria. For instance, the criterion for Pi is calculated as: Crti = 0.2Crti1 + 0.1Crti2 + 0.7Crti3
(9)
(a) Crti1 : The size of the person. It is measured by r=
# of pixels inside the bounding box . # of pixels in the image plane
(10)
Assume that λ is the threshold for best observation, i.e. when r = λ this criterion reaches its peak value, then 1 when r < λ, λ r, Crti1 = (11) 1−r 1−λ , when r ≥ λ. (b) Crti2 : The position of the person in the FOV of a camera. It is measured by the Euclidean distance that a person is away from the center of the image plane (x − xc )2 + (y − yc )2 Crti2 = , (12) 1 2 2 2 xc + yc where (x, y) is the current position of the person and (xc , yc ) is the center of the image plane. (c) Crti3 : The view of the person. It is measured by R=
# of pixels on the face . # of pixels on the entire body
(13)
We assume that the threshold for best frontal view is, i.e., when R = ξ the view of the person is the best, where 1 when R < ξ, ξ r, (14) RCrti3 = 1−R , when R ≥ ξ. 1−ξ (2) The COR Approach: The COR approach in [20] has been applied to two cameras only. We generalize this approach to the cases with more cameras by comparing the accumulated COR in the FOVs of multiple cameras. We randomly select 100 points on the detected person, train the system for 10 frames to construct the
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
79
correspondence for these 100 points, calculate the cumulative CORs in the FOVs of different cameras and select the one with the highest value for hand-off. (3) The CSP Approach: According to the assumption made earlier, we allow one camera to track multiple persons but one person can only be tracked by one camera. So, for each camera Cj , we let all those persons that can be seen by this camera form a group gj . For instance, if, in our case, the camera Cj can see person P1 and P2 , then the domain of gj , noted as Dom[gj ], is {{P1 }, {P2 }, {P1 , P2 }}. The constraint is set to be di ∩ dj = {∅} for i = j , where di ∈ bi ∪ ∅ is the camera assigned to track person Pi , where bi and bj belong to Dom[gj ] and i = j . By doing so, we mean that the persons to be tracked are assigned to different cameras. (4) Fuzzy-based Approach: We apply the same fuzzy reasoning rule as the one in Fig. 2, which is given in [9]. The tracking level state is decided by the Criterion 2, i.e. Crtt2 , which is used for the utility-based game theoretic approach.
4.4 Experimental Results and Analysis Due to limited space, only those frames with camera hand-offs are shown (actually, only some typical hand-offs, since the video is long and there are too many hand-offs.). These camera hand-offs for case 1, 2, 3 are shown in Figs. 3, 4, 5 respectively. Since no topology of the camera network is given, tracking is actually performed by every camera all the time. However, for easy observation, we only draw the bounding box for an object in the image of the camera which is selected to track this object. Case 1 and Case 2 are simple in the sense that there are fewer cameras and objects, and the frame rate is high enough to make the objects trajectories continuous. So, we only show some typical frames for these cases and give more hand-off examples in Case 3, which is more complicated. We show some typical hand-offs for Case 1 and Case 3, while for Case 2, we show the same frames for the four approaches to see the differences caused by different approaches.
Fig. 3 Selective camera handoff frames for the four approaches (Case 1)
80
Y. Li and B. Bhanu
Fig. 4 Selective camera handoff frames for the four approaches (Case 2)
It is clear that the utility-based game theoretic approach considers more criteria when performing the camera selection. Camera hand-offs take place whenever a better camera is found based on the user-supplied criterion in this case. So, cameras that can see persons’ frontal views, which has the highest weight in Crti , are more preferred most of the time. The other three approaches have similar results in the sense that they all consider hand-off based on the position of the objects. Ideally, hand-offs should take place near the FOV boundaries most of the time. Different results are caused by different iterative methods to get the solutions. The design for new constraints and tracking levels are non-trivial. On the contrary, if we just want to consider camera hand-offs when a person is leaving and entering the FOV of a camera by using the utility-based game theoretic approach, we can achieve this by just apply the Criterion 2 in [17]. In [17], this is compared with the results using the combined criterion. Based on the error definition, the combined criterion produces much better results. In this sense, the game theoretic approach is more flexible to perform camera hand-offs based on different criteria. The modification of a criterion will have no influence on the decision-making mechanism. Figure 3 shows the camera hand-off results for a very simple case. All the four approaches achieve similar results, although the utility-based game theoretic approach prefers frontal view. As the scenario being more complex, the COR approach and the fuzzy-based approach have less satisfactory results. The CSP approach needs relatively long time for computing the solutions when the camera network is growing larger, as what is shown in Fig. 6. Error rates for different approaches in the each case are given in Table 4.
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
81
Fig. 5 Selective camera handoff frames for the four approaches Case 3
5 Conclusions and Future Work In this chapter, we analyzed existing and emerging techniques for the camera selection and hand-off problem. Pros and cons of distributed and centralized systems are discussed. Four selected approaches are discussed in details. Both theoretical and experimental comparisons are provided. It is shown that the utility-based game theoretic approach is more flexible and has low computational cost. However, it is
82
Y. Li and B. Bhanu
Fig. 6 Comparison for the number of iterations with a fixed number of cameras (10) and various numbers of the objects
Table 4 Error rates of the selected approaches
Utility-based
COR
CSP
Fuzzy-based
Case 1
3.86%
4.23%
3.92%
4.64%
Case 2
4.98%
10.01%
6.33%
7.11%
Case 3
7.89%
45.67%
12.96%
21.33%
centralized unlike the CSP approach and the fuzzy-based approach. The COR approach is not applicable when the scenario is complicated. There is the trend to have a hierarchical structure which hybrids the distributed and centralized control. There is a lack research on camera selection and hand-off in a large-scale network of active cameras. Current research is short on experimental results with real data processed in real time. Embedded systems are attracting increasing attention. However, the limitation of resources requires for more efficient software algorithms that can run on embedded systems reliably. Acknowledgement This work was partially supported by NSF grants 0551741 and 0622176, 0905671 and ONR grants (DoD Instrumentation and the Aware Building). The contents of the information does not reflect the position or policy of the US Government.
References 1. Li, Y., Bhanu, B.: Utility-based camera assignment in a video network: a game theoretic framework. IEEE Sens. J., Special Issue on Cognative Networks (2010) 2. Kim, H., Nam, C., Ha, K., Ayurzana, O., Kwon, J.: An algorithm of real time image tracking system using a camera with pan/tilt motors on an embedded system. In: ICMIT. Proc. of SPIE, vol. 6041, p. 60 (2005) 3. Quaritsch, M., Kreuzthaler, M., Rinner, B., Bischof, H., Strobl, B.: Autonomous multicamera tracking on embedded smart cameras. EURASIP J. Embed. Syst. (2007) 4. Rinner, B., Jovanovic, M., Quaritsch, M.: Embedded middleware on distributed smart cameras. In: ICASSP (2007) 5. Flech, S., Straßer, W.: Adaptive probabilistic tracking embedded in a smart camera. In: CVPR (2005)
5 A Comparison of Techniques for Camera Selection and Hand-Off in a Video Network
83
6. Flech, S., Busch, F., Biber, P., Straßer, W.: 3D surveillance—A distributed network of smart cameras for real-time tracking and its visualization in 3D. In: CVPR Workshop (2006) 7. Park, J., Bhat, P.C., Kak, A.C.: A look-up table based approach for solving the camera selection problem in large camera networks. In: Workshop on DSC (2006) 8. Morioka, K., Kovacs, S., Lee, J., Korondi, P., Hashimoto, H.: Fuzzy-based camera selection for object tracking in a multi-camera system. In: IEEE Conference on Human System Interactions (2008) 9. Micheloni, C., Foresti, G.L., Snidaro, L.: A network of co-operative cameras for visual surveillance. Vis. Image Signal Process. (April 2005) 10. Bashir, F., Porikli, F.: Multi-camera control through constraint satisfaction for persistent surveillance. In: AVSS (2008) 11. Lim, S., Davis, L.S., Elgannal, A.: A scalable image-based multi-camera visual surveillance system. In: AVSS (2003) 12. Kattnaker, V., Zabih, R.: Bayesian multi-camera surveillance. In: CVPR (1999) 13. Lu, Y., Payandeh, S.: Cooperative hybrid multi-camera tracking for people surveillance. Can. J. Electr. Comput. Eng. 33(3/4) (2008) 14. Everts, I., Sebe, N., Jones, G.: Cooperative object tracking with multiple PTZ cameras. In: ICIAP (2007) 15. Kang, S., Paik, J., Koschan, A., Abidi, B., Abidi, M.A.: Real-time video tracking using PTZ cameras. Proc. of SPIE, vol. 5132 16. Li, Y., Bhanu, B.: Utility-based dynamic camera assignment and handoff in a video network. In: ICDSC (2008) 17. Javed, O., Khan, S., Rasheed, Z., Shah, M.: Camera handoff: tracking in multiple uncalibrated stationary cameras. In: IEEE Workshop on Human Motion (2000) 18. Valera, M., Velastin, S.A.: Intelligent distributed surveillance systems: a review. Vis. Image Signal Process. 152(2) (2005) 19. Moller, B., Plotz, T., Fink, G.A.: Calibration-free camera hand-over for fast and reliable person tracking in multi-camera setup. In: ICPR (2008) 20. Jo, Y., Han, J.: A new approach to camera hand-off without camera calibration for the general scene with non-planar ground. In: VSSN’06 21. Chen, C., Yao, Y., Page, D., Abidi, B., Koschan, A., Abidi, M.: Camera handoff with adaptive resource management for multi-camera multi-target surveillance. In: AVSS (2008) 22. Nayak, J., Gonzalez-Argueta, L., Song, B., Roy-Chowdhury, A., Tuncel, E.: Multi-target tracking through opportunistic camera control in a resource constrained multimodal sensor Network. In: ICDSC (2008) 23. Gupta, A., Mittal, A., Davis, L.S.: COST: an approach for camera selection and multi-object inference ordering in dynamic scenes. In: ICCV (2007) 24. Song, B., Roy-Chowdhury, A.K.: Robust tracking in a camera network: a multi-objective optimization framework. IEEE J. Sel. Top. Signal Process. 2(4) (2008) 25. Song, B., Soto, C., Roy-Chowdhury, A.K., Farrell, J.A.: Decentralized camera network control using game theory. In: ICDSC Workshop (2008)
Chapter 6
Distributed Sensing and Processing for Multi-Camera Networks Aswin C. Sankaranarayanan, Rama Chellappa, and Richard G. Baraniuk
Abstract Sensor networks with large numbers of cameras are becoming increasingly prevalent in a wide range of applications, including video conferencing, motion capture, surveillance, and clinical diagnostics. In this chapter, we identify some of the fundamental challenges in designing such systems: robust statistical inference, computationally efficiency, and opportunistic and parsimonious sensing. We show that the geometric constraints induced by the imaging process are extremely useful for identifying and designing optimal estimators for object detection and tracking tasks. We also derive pipelined and parallelized implementations of popular tools used for statistical inference in non-linear systems, of which multi-camera systems are examples. Finally, we highlight the use of the emerging theory of compressive sensing in reducing the amount of data sensed and communicated by a camera network.
Keywords Multi-camera systems · Multi-view tracking · Particle filtering · Compressive sensing
1 Introduction Over the past decade, camera networks have become increasingly prevalent in a wide range of applications [1] for a number of different perspectives. These systems A.C. Sankaranarayanan () · R.G. Baraniuk Dept. of Electrical and Computer Engineering, Rice University, Houston, TX 77005, USA e-mail:
[email protected] R.G. Baraniuk e-mail:
[email protected] R. Chellappa Dept. of Electrical and Computer Engineering, University of Maryland, College Park, MD 20740, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_6, © Springer-Verlag London Limited 2011
85
86
A.C. Sankaranarayanan et al.
are deployed with the objective of capturing, analyzing, and storing high bandwidth video data. Such an infrastructure, when provided with the necessary computing power, is capable of delivering highly sophisticated services that can potentially improve quality of life. Despite recent advances in computer vision, however, many challenges remain before these systems will be truly autonomous and able to detect, track and analyze behaviors in real-time. For example, real-time scene and behavior analysis on visual sensor networks is the first step toward effective scene understanding. Large visual networks also sense a deluge of data which significantly increases the processing and communication costs in a network. In order to effectively mine real-time data from such large camera networks, there is an immediate need for computationally efficient and robust algorithms for video analytics from as little data as possible. Distributed video processing in a network of smart cameras raises research challenges that can be broadly clustered into three areas: Robust Statistical Inference How does one fuse the information extracted at the individual camera nodes in order to solve detection, tracking, and recognition tasks in a visual sensor network? Invariably, these fusion algorithms involve the interplay between geometric constraints that arise from models of the scene and the imaging process. Inference in such a setting is about designing appropriate statistical estimation techniques that can contend with varying sources of error under the geometric constraints introduced by the imaging process. Computationally Efficient and Distributed Algorithms In addition to the distributed sensing aspect in a multi-camera network, the availability of computing resources at each sensing node (such as in a smart camera) raises the possibility of distributed algorithms that minimize communication and energy costs as well as provide robustness to node failures. However, such a framework requires a complete reworking of the commonly used tools and algorithms. In particular, statistical inference techniques that are widely used across many application domains need to address not just the distributed sensing but also the distributed processing nature of the camera network. Opportunistic and Parsimonious Sensing One of the key challenges in the deployment of large camera networks is the processing and storage of vast amounts of data produced. A fundamental reason for this “data deluge” is that traditional cameras do not exploit the redundancy in the signal at sensing. For many applications, this is extremely wasteful. Toward this end, it is important to design novel sensors and sensing protocols that sense at the information rate of the scene under view. It is also important that the sensing process be adaptive and tuned to the application. There are numerous applications of distributed visual sensing algorithms. In this chapter, we concentrate on the problems of distributed detection and tracking. These form integral subsystems of any robust distributed visual sensing network. The number of cameras connected in a distributed network can vary greatly: from a few (less than ten) cameras for video conferencing to tens of cameras to monitor a building,
6 Distributed Sensing and Processing for Multi-Camera Networks
87
to hundreds of cameras connected in a traffic monitoring network, and thousands over a city (such as London or New York). The specific challenges encountered in each of these applications vary with the number of cameras that are connected in the network. Nevertheless, some of the basic principles of algorithm design and optimization remain the same in all these scenarios. This chapter is organized as follows. We introduce the distributed tracking problem in Sect. 2. In Sect. 3, we discuss implementations of statistical inference algorithms that are inherently distributed and parallelized. Finally, in Sect. 4, we discuss novel sensing strategies and associated inference algorithms that address the data deluge problem.
2 Statistical Inference for Tracking In this section, we describe the problems of detection and tracking fin multi-camera networks. Our approach relies heavily on the geometric constraints specific to multicamera networks. We introduce one such constraint called the homography, and demonstrate the use of homography for multi-view detection and tracking.
2.1 Homography Central projection is the fundamental principle behind imaging with a pinhole camera and serves as a good approximation for lens-based imaging for the applications considered here. In the pinhole camera model, rays (or photons) from the scene are projected onto a planar screen after passing through a pinhole. The screen is typically called the image plane of the camera. We introduce the notation of homogeneous coordinates. We use the tilde ( · ) notation to represent entities in homogeneous coordinates. Given a d-dimensional vector u ∈ Rd , its homogeneous representation is given as a (d + 1)-dimensional vector u [u, 1]T , where the operator denotes equality up to scale. Dealing with homogeneous quantities allows for a scale ambiguity in our representation that enables elegant representations. As an example, the imaging equation given by central projection can be written as u f x, where x = (x, y, z)T is a three-dimensional (3D) point and u = (u, y)T is its projection on a camera with focal length f . The imaging equation is non-invertible in general. However, there is one special scenario where it becomes invertible on, and that is when the world is planar. Consider two views of a planar scene labeled View A and View B. Let uA and uB be the projections of a 3D point x lying on the plane onto views A and B respectively. The homography constraint introduces the following relationship between uA and uB : uA , uB H
(1)
where H is a 3 × 3 matrix called the homography. This implies that a point uA in View A maps to the point uB in View B as defined by the relationship in (1). Note
88
A.C. Sankaranarayanan et al.
that H is a homogeneous matrix, and the transformation defined by it is unchanged when H is scaled. Further, H is invertible when the world plane does not pass through pinholes at either of the two views. The premise of the induced homography depends on the fact that the pre-image of a point on the image plane is a unique point on the world plane. If we use a local 2D coordinate system over the world plane, then the image plane to world plane transformation (from their respectively 2D coordinate systems) can be shown to be a projective transformation, which as before can be encoded as a 3 × 3 homogeneous matrix, say Hπ . This transformation is useful when we wish to estimate metric quantities, or quantities in an Euclidean setting.
2.2 Detection The first and foremost task in distributed visual sensing is to detect objects of interest as they appear in the individual camera views [2, 3]. In typical visual sensing scenarios, the objects of interest are those that are moving. Detection of moving objects is a much easier task, because object motion typically leads to changes in the observed intensity at the corresponding pixel locations, which can be used to detect moving objects. Detection of moving objects is typically performed by modeling the static background and looking for regions in the image that violate this model. The simplest model is that of a single template image representing the static background. A test image can then be subtracted from the template and pixels with large absolute difference can be marked as moving. This simple model introduces the idea of background subtraction—essentially the process of removing static background pixels from an image. There exist a host of background subtraction algorithms that work very well in many scenarios [2, 3]. We denote the image frame at time t as It and background model as Bt , and the value/model of their i-th pixel as Iti and Bti . A commonly used model represents each pixel as Gaussian density whose parameters are learnt from training data. Another background model that robustly handles periodic background disturbances is the mixture of Gaussians (MoG) model [3]. This model adaptively learns a MoG distribution at each pixel. The multi-modality of the underlying distribution gives the ability to capture repetitive actions as part of the background. An adaptive learning method can be used to keep track of global changes in the scene. Given a test image, we can compute the likelihood image (the probability of the intensity value observed at each pixel belonging to the background model) and threshold it to obtain pixels that violate the background model.
2.3 Multi-View Fusion and Tracking Once the objects of interest have been detected in each of the individual cameras, the next task is to track each object using multi-view inputs. Multi-camera tracking
6 Distributed Sensing and Processing for Multi-Camera Networks
89
in the presence of ground-plane constraint has been the focus of many recent papers [4–6]. The key concepts in many of the proposed algorithms are: • Association of data across views by exploiting the homography constraint: This can be done by projecting various features associated with the silhouette onto the ground plane via the homography. The vertical axis from each segmented human is used as the feature in [5], while points features [6, 7] or even the whole silhouette [4] form alternate choices. • Temporal continuity of object motion to track: Typically, a particle filter [5, 6] is used to temporally filter the data after association. Alternatively, [4] builds a temporal consistency graph and uses graph-cuts [8] to segment tracks. Consider the problem of location estimation of a point object moving on a plane. At each camera, background subtraction provides an estimate of where the object lies. We can now project the image plane locations to arrive at individual estimates of the world plane location of the tracked point. In the presence of noise corrupting the image plane observations, errors in calibration and inaccuracies in modeling, the world plane location estimates will no longer be identical. We now need a strategy to fuse these estimates. To do so in a systematic fashion we need to characterize the statistical properties of these estimates. The projective mapping is in general a non-linear transformation involving ratios. The statistics of random variables, when transformed under such a ratio transformation, change significantly. Given that the projective transformations linking different views of the same scene are different, one can expect that the statistics of random variables on the world plane arising from different views will necessarily be different, even when the original random variables are identically distributed. Given M cameras and the homography matrices Hi , i = 1, . . . , M between the camera views and the ground plane (see Fig. 1), one can derive an algorithm for fusing location estimates. Let Ziu be the random variable modeling the object location on the image plane of the ith camera. Let us assume that the random variables {Ziu }M i=1 are statistically independent. Now, each of these random variables can be projected to the world plane to obtain Zix , such that Zix Hi ZiU , i = 1, . . . , M. Consider the distribution of Zix under the assumption that the Ziu are jointly Gaussian. Specifically, when certain geometric properties are satisfied (for details, see [6]), we can show that the distribution of Zxi is closely approximated by a Gaussian distribution. Further, we can relate the mean and the covariance matrix of the transformed random variable to the statistics of the original random variable and the parameters characterizing the projective transformation. This result is useful for designing strategies to fuse {Zix , i = 1, . . . , M} in an optimal sense. In the case of multi-view localization, if the covariances of the estimates Zix is Σi , then the minimum variance estimate Zmv is computed as M −1 M −1 −1 Σi Σj Zix . (2) Zmv = i=1
j =1
90
A.C. Sankaranarayanan et al.
Fig. 1 (Left) A schematic showing densities in the image planes of cameras and their transformations to the ground plane. (Right) Variance ellipses are shown for the image planes. The corresponding color coded ellipse on the ground plane shows the covariance when transformed to the ground plane. The ellipse in black depicts the variance of the minimum variance estimator. Figure courtesy of [12]
The covariance of the Zmv , Σmv is given as −1 M −1 Σj . Σmv =
(3)
j =1
We refer the reader to [6, 9–11] for details of the derivation. Hence, given a true object location on the ground plane, Σmv provides an estimate of the maximum accuracy (or minimum error) with which we can localize the object on the ground plane given modeling assumptions on the image plane (see Fig. 1). Finally, we can embed the concept used in constructing the minimum variance estimators to formulate a dynamical system that can be used to track objects using multi-view inputs. As before, we efficiently fuse estimates arising from different views by appropriately determining the accuracy of the estimates characterized by their covariance matrices. Figure 2 shows tracking outputs from processing video data acquired from six cameras. Each object is tracked using a particle filter, and object to data associations are maintained using Joint Probability Data Association (JPDA) [13]. Here, tracking is reformulated as Bayesian inference for the dynamical system. While exact inference is analytically intractable, approximate Bayesian inference for this dynamical system can be performed using the particle filter [14]. Toward this end, efficient implementations of particle filters influence multiple problems. In addition to faster run-time, these implementations help in reducing the resource utilization of the camera network as well. We discuss this next.
3 Efficient Particle Filtering Particle filtering has been applied to a wide variety of problems such as tracking, navigation, detection and video-based object recognition. This generality of particle filters comes from a sample based approximation of the posterior density of the state vector. This allows the filter to handle both the non-linearity of the system as well
6 Distributed Sensing and Processing for Multi-Camera Networks
91
Fig. 2 Output from the multi-object tracking algorithm working with input from six camera views. (Top row) Four views of a scene with several humans walking. Each camera independently detects/tracks the humans using a simple background subtraction scheme. The center location of the feet of each human is indicated with color coded circles in each view. These estimates are then fused taking into account the relationship between each view and the ground plane. (Bottom row) Fused tracks overlaid on a top-down view of the ground plane. Figure courtesy of [12]
as the non-Gaussian nature of noise processes. However, the resulting algorithm is computationally intensive and as a result, the need for efficient implementations of the algorithm. We first briefly describe particle-filtering algorithms and then propose modifications to the general filtering framework that make it highly amenable to pipelined and parallel implementations.
3.1 Particle Filtering: A Brief Overview Particle filtering addresses the problem of Bayesian inference for dynamical systems. Let X ⊆ Rd and Y ⊆ Rp denote the state space and the observation space of the system respectively. Let xt ∈ X denote the state at time t, and yt ∈ Y the noisy observation at time t. We model the state sequence {xt } as a Markovian random process. Further we assume that the observations {yt } to be conditionally independent given the state sequence. Under these assumptions, the system is completely characterized by the state transition density p(xt |xt−1 ), the observation likelihood p(yt |xt ) and the prior state probability p(x0 ). Given statistical descriptions of the models and noisy observations, we are interested in making inferences I (ft ) of the form I (ft ) = Eπt ft (xt ) = ft (xt )p(xt |y1:t ) dxt , (4)
92
A.C. Sankaranarayanan et al.
where ft is some function of interest and p(xt |y1:t ) is the posterior probability is recursively estimated using the Bayes theorem, p(yt |xt ) p(xt |xt−1 )p(xt−1 |y1:t−1 ) dxt−1 . (5) πt = p(xt |y1:t ) = p(yt |y1:t−1 ) A popularly used inference is that of the conditional mean, where ft (xt ) = xt . Particle filtering approximates the posterior πt with a discrete set of particles or
N (i) (i) N (i) samples {xt }N i=1 wt = 1. The i=1 and normalized weights {wt }i=1 such that approximation for the posterior density is given by πˆ t (xt ) =
N
(i) (i) wt δxt xt ,
(6)
i=1
where δxt (·) is the Dirac delta function centered at xt . The set St = {xt(i) , wt(i) }N i=1 can be estimated recursively from St−1 . The initial particle set S0 is obtained from sampling the prior density π0 = p(x0 ). Particle filters sequentially generate St from St−1 using the following steps: (i) 1. Importance Sampling: Sample xt(i) ∼ g(xt |xt−1 yt ), i = 1, . . . , N . This step is also called the proposal step and g(·) is called the proposal density. 2. Computing Importance Weights: Compute the unnormalized importance weights (i) w˜ t , (i)
(i)
(i)
w˜ t = wt−1
p(yt |xt ) (i)
(i)
g(xt |xt−1 yt )
,
i = 1, . . . , N.
(7)
(i)
3. Normalize Weights: Obtain the normalized weights wt , (i)
w˜ (i) wt = t N
(j ) ˜t j =1 w
,
i = 1, . . . , N.
(8)
4. Resampling: To avoid sample degeneracies, the weighted sample set {xt(i) , wt(i) } is resampled to get an unweighted sample set St = {x˜t(i) }. The resampling step, at the end of each iteration is performed to avoid certain degeneracies in the finite sample representation of the posterior density. Resampling essentially replicates particles with higher weights and eliminates those with low weights. The most popular resampling scheme, called systematic resampling (SR) (i) [14], samples N particles from the set {xt } (samples generated after proposal) according to the multi-nomial distribution with parameters wt(i) to get a new set of N particles St . The next iteration uses this new set S˜t for sequential estimation. Systematic resampling is a computationally tricky as it requires the knowledge of the normalized weights. Resampling based on SR cannot start until all the particles are generated and the value of the cumulative sum is known. This is the basic limitation that we overcome by proposing alternative techniques.
6 Distributed Sensing and Processing for Multi-Camera Networks
93
3.2 Metropolis Hastings Algorithm Particle filtering is a special case of more general MCMC based density sampling techniques that are especially tuned for dynamical systems. The Metropolis Hastings Algorithm (MHA) [15, 16] is considered the most general MCMC sampler. We first present the a theory of MCMC sampling using a variant called the Independent MHA (IMHA) algorithm and then state the conditions under which the general theory fits into the particle-filtering algorithm presented before. The IMHA generates samples from the desired density (say p(x), x ∈ X ) by generating samples from an easy to sample proposal distribution, say q(x), x ∈ X . IMHA produces a sequence of states {x (n) , n ≥ 0}, which by construction is Markovian in nature, through the following iterations. 1. Initialize the chain with an arbitrary value x (0) = x0 . Here, x0 could be user specified. 2. Given x (n) , n ≥ 0, generate xˆ g(·), where g is the sampling or proposal function. 3. Accept xˆ with probability p( x ) g(x (n) ) (n) ,1 . (9) x ) = min α(x , g( x ) p(x (n) ) Both IMHA and SISR are algorithms designed to generate samples according to a probability density function, with the SISR suited specifically to the sequential nature of dynamical systems. We show, next, how the IMHA can be used to perform resampling in particle filters without the drawbacks associated with systematic resampling.
3.3 Particle Filtering with IMHA-Based Resampling The bottlenecks introduced by the SR technique can be overcome by using IMHA (i) for resampling. Given a set of unweighted samples {xt−1 , i = 1, . . .} sampled from the posterior density p(xt−1 |y1:t−1 ) at time t − 1, we can approximate the posterior by N (i) 1 δxt−1 xt−1 . p(xt−1 |y1:t−1 ) ≈ N
(10)
i=1
Using (5) and (10), we can approximate the posterior at time t as p(xt |y1:t ) ≈
N p(yt |xt ) 1 (i) p xt |xt−1 . p(yt |y1:t−1 ) N
(11)
i=1
Sampling from this density can be performed using the IMHA. The issue of choice of importance function now arises. The importance function typically reflects
94
A.C. Sankaranarayanan et al.
and exploits the knowledge of application domain or could be a clever approximation to the posterior. For this reason, we would like to reuse the importance function corresponding to the underlying model. Keeping this in mind, we propose a new importance function of the form, g (xt |yt ) =
N 1 i yt . g xt |xt−1 N
(12)
i=1
Note that g (xt |y1:t ) qualifies to be an importance function for use in IMHA given the independence of the function on current state of the chain. To sample from g (xt |yt ), we need to first sample I U [1, 2, . . . , N], and then sample from I y ). The sampling of I can be done deterministically given the ease of samg(·|xt−1 t pling from uniform densities over finite discrete spaces. Finally, although the new importance function is functionally different from the one used in the SISR algorithm, the generated particles will be identical. The overall algorithm proceeds similar to IMHA. We first propose particles using the new importance function g (xt |y1:t ). The acceptance probability now takes the form α(xt , x ) = min{w ( x )/w (xt ), 1} with
N (i) i=1 p(xt |xt−1 ) . (13) w (xt ) = p(yt |xt ) N (i) i=1 g(xt |xt−1 yt ) Further, if the choice of the importance function were the same as the state transition model, i.e., g(xt |xt−1 yt ) = p(xt |xt−1 ), then the acceptance probability be(n) (n) x ) = min{p(yt | x )/p(yt |xt ), 1}. comes a ratio of likelihoods, α(xt , We can now avoid the systematic resampling of traditional particle-filtering algorithms and use IMHA to generate the unweighted particle set/stream from the desired posterior. Figure 3 present a schematic diagram of possible implementation (see [17] for additional details). We can now compare the algorithm given above with the classical SISR. Note that the SISR algorithm involves a weight normalization step (see (8)). However, the proposed algorithm works with ratios of unnormalized weights and requires no such normalization. This obviates the need for knowledge of normalized importance weights, as we can work with unnormalized weights. This allows the IMH sampler to start parsing through the particles as they are generated, and not wait for the entire particle set to be generated and the importance weights computed.
Fig. 3 Parallel implementation of the IMHA-based particle filter with a single Metropolis–Hastings Chain
6 Distributed Sensing and Processing for Multi-Camera Networks
95
Fig. 4 (Left) Actual time (in seconds) taken to process 20 frames, with a filter of 840 particles, with varying number of Rw . (Right) Speedup obtained by replication of the weight computation node. Note the linear speedup obtained with the proposed algorithm. Figure courtesy of [17]
3.4 Experimental Results The design methodologies proposed have been verified for two applications: a synthetic example originally discussed in [14] and the problem of visual tracking. We chose to implement over a multi-processor cluster framework since the underlying theory applies both to hardware based design as well as to clusters. We implemented the particle-filter-based online tracking algorithm presented in [18] using the Red Cluster. The algorithm was used to process 20 frames of a video sequence, tracking a car. The filter was run with 840 and 1680 particles, with the number of cluster nodes for weight computation Rw varying. Rw = 1 corresponds to the sequential implementation, and Rw > 1 corresponds to the parallel implementation with a single chain. Under the same setup, we tried an implementation of SISR, replacing the IMH Chain with a systematic resampler. Figure 4 shows the actual time taken (in seconds) and the speedup to process 20 frames of video, with 840/1680 particles for the proposed algorithm and SISR. Note the linear increase in speedup with the number of processing nodes. The two plots demonstrate the pipelinability of the proposed algorithm. It can be seen that the speedup tapers-off as number of cluster nodes increases. As we use more and more processors, inter-processor communication becomes the dominant source of delay, and further parallelization does not help.
4 Compressive Sensing The emerging field of compressive sensing (CS) provides an alternate way of handling the data deluge problem. CS relies on the fact that much of real-world signals are inherently sparse in some known basis such as wavelets. Sparsity of the signal hints at the fact that the inherent information content of a signal is significantly
96
A.C. Sankaranarayanan et al.
lower than its ambient dimension. This motivates the design of a parsimonious sensing strategy that samples the signal at its information rate. In this section, we first introduce compressive sensing, and show that some of the fundamental operations in a multi-camera system such as background subtraction and multi-view tracking can be performed directly on these compressive measurements with a high level of fidelity. This leads to a sensor network that senses only at the information rate required for the task at hand.
4.1 Compressive Sensing Compressive sensing [19] relates to the concept of reconstructing sparse signals from under-sampled linear measurements. Consider a signal x ∈ RN , which is sparse in a basis B, that is, s ∈ RN defined as x = Bs is sparse. We call a vector K-sparse if it has at most K non-zero components, or equivalently, if s0 ≤ K, where · 0 is the 0 norm or the number of non-zero components. We are interested in the problem of sensing the signal x from linear measurements. With no additional knowledge about x, we would require N linear measurements of x, which would then form an invertible linear system. The theory of compressed sensing shows that it is possible to reconstruct x from M measurements even when M N by exploiting the sparsity of s = BT x. Consider a measurement vector y ∈ RM obtained using a M × N measurement matrix , such that y = x + e = Bs + e,
(14)
where e is the measurement noise. For M < N , estimating x from the linear measurements is an ill-conditioned problem. However, when x is K sparse in the basis B, then CS enables recovery of s (or alternatively, x) from M = O(K log(N/K)) measurements. In particular, when B is a fixed basis, it can be shown that using a randomly generated measurement matrix allows for the recovery of x with a high probability. Typical choices for such measurement matrix are the Bernoulli matrix and the Gaussian matrix [20, 21]. Further, x can be recovered by solving a convex problem of the form: (P1) :
min
s1
s.t. y − Bs ≤
(15)
with being a bound for the measurement noise e in (14). It can be shown that the solution to the (P1) is with a high probability the K sparse solution that we seek. There exist a wide range of algorithms that solve (P1) to various approximations or reformulations [19, 22]. Most of them note that the problem (P1) is a convex problem, and in particular, can be recast as a Second Order Cone Program (SOCP) for which there exist efficient numerical techniques.
6 Distributed Sensing and Processing for Multi-Camera Networks
97
4.2 Compressive Background Subtraction For background subtraction, our objective is to recover the location, shape and appearance of the objects given a test image over a known background. Let us denote the background, test, and difference images as x b , x t , and x d , respectively. The difference image is obtained by pixel-wise subtraction of the background image from the test image. Note that the support of x d , denoted as Sd = {n|n = 1, . . . , N; |x d (n)| = 0}, gives us the location and the silhouettes of the objects of interest, but not their appearance. Suppose that x b and x t are typical real-world images in the sense that when wavelets are used as the sparsity basis for x b , x t , and x d , these images can be well approximated with the largest K coefficients with hard thresholding [23]. The images x b and x t differ only on the support of the foreground, which has a cardinality of P = |Sd | pixels with P N . As a consequence, we can expect to require a much smaller number of samples to reconstruct the difference image than the background or foreground images. Assume that we have multiple compressive measurements y bi (M × 1, i = 1, . . . , B) of training background images x bi , where x b is their mean. Each compressive measurement is a random projection of the whole image, whose distribution we approximate as an i.i.d. Gaussian distribution with a constant variance y bi ∼ N (y b , σ 2 I ), where the mean value is y b = x b . When the scene changes to include an object which was not part of the background model and we take the compressive measurements, we obtain a test vector y t = x t , where x d = x t − x b is sparse in the space domain. In general, the sizes of the foreground objects are relatively smaller than the size of the background image; hence, we model the distribution of the elements of the literally background subtracted vector as y d = y t − y b ∼ N (μd , σ 2 I )(M × 1). Note that the appearance of the objects constructed from the samples y d would correspond to the literal subtraction of the test frame and the background; however, their silhouette is preserved. We performed background subtraction experiments with a single pixel camera [24]; in our test, the background x b consists of the standard test Mandrill image, with the foreground x t consisting of a white rectangular patch as shown in Fig. 5. Both the background and the foreground were acquired using pseudorandom compressive measurements (y b and y t , respectively). We obtain measurements for the subtraction image as y d = y t − y b . We reconstructed both the background, test, and difference images, using total variation (TV) minimization. The reconstruction is performed using several measurement rates ranging from 0.5% to 50%. In each case, we compare the subtraction image reconstruction with the difference between the reconstructed test and background images. The resulting images are shown in Fig. 5, and it can be seen that for low rates the background and test images are not recovered accurately, and therefore the subtraction performs poorly; however, the sparser foreground innovation is still recovered correctly from the difference of the measurements, with rates as low as 1% being able to recover the foreground at this low resolution.
98
A.C. Sankaranarayanan et al.
Fig. 5 Background subtraction experimental results using an SPC. Reconstruction of background image (top row) and test image (second row) from compressive measurements. Reconstruction using conventional subtraction using the above images (third row) and directly from compressive measurements (last row). The columns correspond to measurement rates of 50%, 5%, 2%, 1% and 0.5%, from left to right. Figure courtesy of [25]
Fig. 6 Tracking results on a video sequence of 300 frames. Results are obtained using only 1/5 the bandwidth required of traditional central servers. (Left) The first two rows show sample images and background subtraction results using the compressive measurements, respectively. The background subtracted blobs are used to detect target location on the ground plane. (Right) The detected points using CS (blue dots) as well as the detected points using full images (black). The distances are in meters. Figure courtesy of [25]
4.3 Multi-View Ground-Plane Tracking Background subtraction forms an important pre-processing component for many vision applications. In this regard, it is important to see if the imagery generated using compressive measurements can be used in such applications. We demonstrate a multi-view tracking application where accurate background subtraction is key in determining overall system performance. In Fig. 6, we show results on a multi-view ground-plane tracking algorithm over a sequence of 300 frames with 20% compression ratio. We first obtain the object silhouettes using the compressive samples at each view. We use wavelets as the sparsifying basis B. At each time instant, the silhouettes are mapped on to the ground planes and averaged. Objects on the ground plane (e.g., the feet) combine in synergy while those off the plane are in parallax and do not support each other. We
6 Distributed Sensing and Processing for Multi-Camera Networks
99
then threshold to obtain potential target locations as in [4]. The outputs indicate the background subtracted images are sufficient to generate detections that compare well against the detections generated using the full non-compressed images. Hence, using our method, the communication bandwidth of a multi-camera localization system can be reduced to one-fifth if the estimation is performed at a central location.
5 Conclusions and Future Directions In this chapter, we have discussed some of the main challenges in distributed video processing, namely robust and computationally efficient inference, and opportunistic and parsimonious sensing. Advances in these are crucial to the deployment of large camera networks. We conclude by highlighting several recent trends that we believe will have significant impact on various aspects of distributed video processing.
5.1 Distributed Bayesian Inference The algorithms for tracking presented here were both based on online inference using particle filtering. Therefore, in order to make these algorithms truly distributed and enable their implementation on huge camera networks containing hundreds of cameras, one needs to pay attention to methods that enable these particle-filter-based estimates to be performed in a distributed manner. This can be achieved using either synchronized particle filtering or the more general means of distributed function estimation. (1) Synchronized Particle Filtering: One way to decentralize the filter operations is to replicate it identically at each node. For particle filtering, this can be done easily if the random number generators are made identical across nodes. Such a scheme is referred to as synchronized particle filtering [26]. By initializing the random number generator with the same seed, all nodes can be made to generate the same particles, which in turn makes fusion of the associated weights simpler. The communication costs are then limited to the transmission of the associated weights across the whole network. The immense flexibility of this approach allows for it to be effective in any particle-filtering algorithm. (2) Distributed Function Estimation: We can relax the need to make our distributed inference algorithm identical to the centralized one. There are a host of methods that allow for the computation of average mean through explicit global communication or through local consensus [27]. An alternative to the concept of synchronous filtering can be by approximating the inference at each camera with a Gaussian mixture model [28] or, in general, any parametric density family. The parameters can then be transmitted to all nodes in the sensor network, each of which locally updates their densities.
100
A.C. Sankaranarayanan et al.
5.2 Manifold-Based Dimensionality Reduction (NLDR) Images and video data lead to extremely high-dimensional data sets and in order to represent, visualize, analyze, interpret, and process such high-dimensional data, one needs to encapsulate the salient characteristics of the data in a lower dimensional representation. Traditionally, this has been performed using linear dimensionality reduction techniques such as principal component analysis (PCA). Such linear dimensionality reduction methods are limited in applicability for vision applications, because the geometric constraints imposed by the imaging device and the lighting characteristics lead to non-linear constraints on image and video data. NLDR approaches such as the locally linear embedding (LLE) [29], Isomap [30], and the Laplacian eigenmap [31] construct and represent the high-dimensional data using a low-dimensional representation. The parameters of the representation are obtained by optimizing an appropriate cost function that leads to embeddings that are ‘Euclidean-like’ locally, but globally they are highly nonlinear. Moreover, most of these techniques also have an elegant graph formulation that explicitly describes the local properties that these approaches preserve. Such approaches for NLDR find several natural and compelling applications in vision tasks, as both image and video data are inherently high-dimensional and lend themselves to non-linear dimensionality reduction. Acknowledgements This research was partially supported by the Office of Naval Research under the contracts N00014-09-1-1162 and N00014-07-1-0936, the U.S. Army Research Laboratory and the U.S. Army Research Office under grant number W911NF-09-1-0383, and the AFOSR under the contracts FA9550-09-1-0432 and FA9550-07-1-0301. The authors thank Prof. Volkan Cevher, Prof. Ankur Srivastava, Dr. Ashok Veeraraghavan, Dr. Marco Duarte and Mr. Dikpal Reddy for valuable discussions and collaborations.
References 1. Aghajan, H., Cavallaro, A.: Multi-Camera Networks: Concepts and Applications. Elsevier, Amsterdam (2008) 2. Lv, T., Ozer, B., Wolf, W.: A real-time background subtraction method with camera motion compensation. In: Proceedings of IEEE International Conference on Multimedia and Expo, vol. 1, pp. 331–334 (2004) 3. Elgammal, A., Duraiswami, R., Harwood, D., Davis, L.S.: Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc. IEEE 90(7), 1151–1163 (2002) 4. Khan, S.M., Shah, M.: A multi-view approach to tracking people in crowded scenes using a planar homography constraint. In: European Conference on Computer Vision, vol. 4, 133–146 (2006) 5. Kim, K., Davis, L.S.: Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering. In: Proceedings of European Conference on Computer Vision, vol. 3, pp. 98–109 (2006) 6. Sankaranarayanan, A.C., Chellappa, R.: Optimal multi-view fusion of object locations. In: Proceedings of IEEE Workshop on Motion and Video Computing (WMVC), pp. 1–8 (2008) 7. Fleuret, F., Berclaz, J., Lengagne, R.: Multi-camera people tracking with a probabilistic occupancy map. Technical report, EPFL/CVLAB2006.07 (July 2006)
6 Distributed Sensing and Processing for Multi-Camera Networks
101
8. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000) 9. Smith, R., Self, M., Cheeseman, P.: Estimating uncertain spatial relationships in robotics. In: Autonomous Robot Vehicles, pp. 167–193 (1990) 10. Smith, R.C., Cheeseman, P.: On the representation and estimation of spatial uncertainty. Int. J. Robot. Res. 5(4), 56–68 (1986) 11. Kanatani, K.: Statistical Optimization for Geometric Computation: Theory and Practice. Elsevier, New York (1996) 12. Sankaranarayanan, A.C., Veeraraghavan, A., Chellappa, R.: Object detection, tracking and recognition for multiple smart cameras. Proc. IEEE 96(10), 1606–1624 (2008) 13. Bar-Shalom, Y., Fortmann, T.: Tracking and Data Association. Academic Press, San Diego (1987) 14. Gordon, N., Salmon, D., Smith, A.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc. Radar Signal Process. 140, 107–113 (1993) 15. Chib, S., Greenberg, E.: Understanding the metropolis hastings algorithm. Am. Stat. 49, 327– 335 (1995) 16. Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57, 97–109 (1970) 17. Sankaranarayanan, A.C., Srivastava, A., Chellappa, R.: Algorithmic and architectural optimizations for computationally efficient particle filtering. IEEE Trans. Image Process. 17(5), 737–748 (2008) 18. Zhou, S., Chellappa, R., Moghaddam, B.: Visual tracking and recognition using appearanceadaptive models in particle filters. Trans. Image Process. 11, 1434–1456 (2004) 19. Candes, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207 (2006) 20. Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3), 253–263 (2008) 21. Candˇcs, E.J.: The restricted isometry property and its implications for compressed sensing. C. R. Math. (2008) 22. Needell, D., Tropp, J.A.: CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. (2008) 23. Mallat, S.: A Wavelet Tour of Signal Processing. Academic Press, New York (1999) 24. Duarte, M., Davenport, M., Takbar, D., Laska, J., Sun, T., Kelly, K., Baraniuk, R.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008) 25. Cevher, V., Sankaranarayanan, A., Duarte, M.F., Reddy, D., Baraniuk, R.G., Chellappa, R.: Compressive sensing for background subtraction. In: Proceedings of European Conference on Computer Vision, pp. 12–18 (2008) 26. Coates, M.: Distributed particle filters for sensor networks. In: Proceedings of the Third International Symposium on Information Processing in Sensor Networks, pp. 99–107 (2004) 27. Xiao, L., Boyd, S., Kim, S.J.: Distributed average consensus with least-mean-square deviation. J. Parallel Distrib. Comput. 67(1), 33–46 (2007) 28. Sheng, X., Hu, Y.H., Ramanathan, P.: Distributed particle filter with GMM approximation for multiple targets localization and tracking in wireless sensor network. In: Proceedings of the 4th International Symposium on Information Processing in Sensor Networks (2005) 29. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000) 30. Tenenbaum, J.B., Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000) 31. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)
Chapter 7
Tracking of Multiple Objects over Camera Networks with Overlapping and Non-overlapping Views Jenq-Neng Hwang and Victor Gau
Abstract In this book chapter, we propose a fully automated approach for tracking of multiple objects across multiple cameras with overlapping and non-overlapping views in a unified framework without initial training or prior camera calibration. For tracking with a single camera, Kalman filter and adaptive particle-sampling techniques are integrated for multiple objects tracking. When extended to tracking over multiple cameras, the relations between adjacent cameras are learned systematically by using image registration techniques for consistent handoff of trackingobject labels across cameras. In addition, object appearance measurement is employed to validate the labeling results. Experimental results demonstrate the performance of our approach on real video sequences for cameras with overlapping and non-overlapping views.
Keywords Object tracking · Kalman filter · Particle filter
1 Introduction Distributed camera networks have emerged in various applications including urban surveillance for identification of terrorist and criminal events, building/campus monitoring with anomaly detection, health and safety care in nursing home or kindergarten, and battlefield or hazardous area visualization for better decision making. Surveillance tracking of multiple video objects across large area requires camera J.-N. Hwang () · V. Gau Department of Electrical Engineering, Box 352500, University of Washington, Seattle, WA, USA e-mail:
[email protected] V. Gau e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_7, © Springer-Verlag London Limited 2011
103
104
J.-N. Hwang and V. Gau
networks work cooperatively for reliable handoff of tracked objects from one camera to another. If tracking within a single camera is effective, the main task for tracking across cameras is to establish the correspondence between the tracks of the same object when it is seen in a new view [1]. Multiple object tracking within a single-camera view has been investigated intensively due to its important role in various applications [2]. Based on some restricted assumptions, the Kalman Filter (KF) [3] can achieve computational simplicity, which is a desired property, for real-time tracking systems. For video object tracking, researchers have also turned to particle filter based tracking methods [4] due to the argument that the object movements are not linear, especially tracking under occlusion or segmentation errors. The main issue is how to update the particle filters with correct measurements when occlusion or segmentation errors occur. This problem can be effectively solved by using adaptive particle sampling under a normal KF based tracking [5] for measurement selection only in case of occlusion or segmentation errors. While we use particle-sampling techniques to provide reasonable measurement candidates, the mathematical tractability and closedform solutions provided by KF do not need to be sacrificed in the overall tracking model. Once the tracking of multiple objects within a single camera is effectively handled, we take one step further to track objects across multiple cameras. Two typical scenarios for tracking across multiple cameras, the so-called consistent labeling problem, are commonly considered, one is for cameras with overlapping field of views (FOVs), and the other is with non-overlapping (disjoint) FOVs, i.e. “blind” area, as shown in Fig. 1. For two adjacent cameras, the FOV lines for both overlapping and non-overlapping cases, which will be briefly described in Sect. 4.1, can be automatically established by using effective image registration techniques without using training video sequences as in [1]. Our approach also uses the appearance measurement in single-camera tracking to validate the labeling result, instead of using FOV lines as the only feature for consistent labeling [6]. Moreover, the labeling result is further used to continuously update the pre-computed FOV lines. The rest of this chapter is organized as follows: In Sect. 2, we review some of the related work in tracking within a single camera and across multiple cameras. Next, in Sect. 3, we describe the single-camera tracking method. Then, Sect. 4 gives the details about how to establish FOV lines for both overlapping and non-overlapping
Fig. 1 The configuration of camera network system
7 Tracking of Multiple Objects over Camera Networks
105
cases, as well as shows the consistent labeling process across cameras. The experimental results are presented in Sect. 5, followed by conclusion in Sect. 6.
2 Related Work The multi-object tracking problem can be modeled as a dynamic system represented by states in discrete time domain. A Kalman filter [2] assumes a linear state transition function and a linear observation model as well as Gaussian noises to obtain optimal state prediction and update formulations. Kalman filtering is based on restricted assumptions and can therefore achieve computational simplicity, which is a desired property for real-time tracking systems. If the linear and Gaussian assumptions are relaxed, the solutions cannot be determined analytically in general and there exist only sub-optimal approximations. The extended Kalman filter (EKF) [7] relaxes the restriction of linearity on the state transition function and observation model, i.e., the system transition function and observation model can be any differentiable functions in EKF. Foresti [8, 9] utilized a modified EKF model which does not require a priori information to perform tracking. Particle filters further relax the Gaussian assumption to model non-Gaussian and multi-modal posterior distributions. For more complex models, particle filters have their merits when dealing with objects that are closer to the camera and having dramatic motion changes. However, a large number of particles are required in such cases in order to capture the large and dramatic motion changes, and the computational complexity of the particle filters is usually very high. Moreover, particle filters still need to cooperate with occlusion detection and handling techniques in order to have better performance. Maggio and Cavallaro [10] designed a combined tracker utilizing particle filters and mean-shift techniques to produce a smaller number of samples than traditional particle filters and more reliable results than traditional mean-shift tracking. However, their tracker was not extended to multi-target tracking. Furthermore, without explicit occlusion analysis, particle filters that continue relying on outlier pixels and updating the appearance model can significantly deteriorate the tracking accuracy in occlusion situations [11]. Therefore the ability of handling occlusion cases of their hybrid tracker was quite limited. In fact, although researchers are turning to use particle filters because Kalman filtering is only optimal when the linear and Gaussian assumptions are held, many recent works using particle filters [11–14] still use linear functions as the underlying system models. This is reasonable because when the video frame rate is high enough and the camera is set up with a certain distance away from the objects under surveillance, which is often the case in many surveillance applications, the object motion is close to linear as we look at them frame by frame. Therefore, even with occlusion or segmentation errors, the use of Kalman filter with linear prediction can still be justified. It also explains why Kalman filters are still adopted in many recently proposed systems [15–17]. The approaches for solving the consistent labeling problem for tracking across camera networks with overlapping FOVs can be roughly categorized into the following three schemes: (1) The Feature Matching Scheme is a common strategy for
106
J.-N. Hwang and V. Gau
consistent labeling which matches geometrical or other features of tracked objects directly. The FOV lines, which delineate the region of one camera that is visible in another camera, have been used for consistently labeling [1]. Instead of using training videos, a projective invariants based method was proposed in [18], which manually selects corresponding feature points from two camera images to establish FOV lines. Multiple features, such as epipolar geometry, homography, landmark, apparent height and color, can also be combined using a Bayesian Network for better similarity measure [19]. (2) The 3D Information Based Scheme achieves the consistent labeling by projecting the location of each object in the world coordinate system, and establishing equivalence between objects from various camera views, if camera calibration and 3D site model are created [20]. (3) The Alignment Scheme achieves consistent labeling by aligning the different camera views through geometrical transformation, e.g., the transformed motion trajectories of objects as observed in different cameras and computing plane homographies for each match [21, 22]. On the other hand, tracking targets across multiple cameras with non-overlapping FOVs requires the solving of a correspondence problem by learning the relations between non-overlapping cameras. In [23], the concept of FOV lines is extended to non-overlap views. Different camera views can be transformed to a common ground plane, and tracking in the “blind” region is achieved by motion prediction [24]. However, the feature correspondence in transforming camera views is manually selected. In [4], spatio-temporal and appearance cues are integrated to constrain correspondences. In [25], the topology of camera network is learned and the gap between non-overlapping cameras is bridged by providing the probabilistic estimation of the location and time with which a target may re-appear.
3 Tracking within a Single Camera The proposed tracking system framework within a single camera is illustrated in Fig. 2. Before applying the tracking algorithm, a background model is estimated and updated from the video to segment the video objects (VOs) from the background scene. For a new object entering at time instance k, the system initializes its system state xk = [uk vk u˙ k v˙k ak bk ]T and an appearance model ξk for it. Commonly used appearance models are color values of the fitted ellipse (color matrices), and compact summarization of color distribution such as histograms or mixture of Gaussians [11]. As shown in Fig. 3, the position (uk , vk ) is coordinate of the centroid of an object in the image plane. The velocities u˙ k and v˙k are initialized as zeros. The sizes (ak , bk ) are the length of the major axis and the minor axis of the ellipse fitted on the VO. The measurement state is defined as yk = [uk vk ak bk ]T . After the initial states for the objects are obtained, we perform KF prediction. Afterwards, we obtain the measurements using the measurement selection procedure and utilize an enhanced version of probabilistic data association (EPDA) [26] to associate the measurements with each target object for filter update. In the measurement selection procedure (see Fig. 2), the system detects if an occlusion or a segmentation error occurs. If there is no occlusion or segmentation error,
7 Tracking of Multiple Objects over Camera Networks
107
Fig. 2 The proposed tracking system framework within a single camera
Fig. 3 Object initialization
the segmented VOs are reliable and the measurements for a target object can be obtained by referring to the segmented VOs. Otherwise, we do not trust the segmentation results and perform adaptive particle sampling to find reasonable measurement candidates instead.
3.1 Detection of Occlusion and Segmentation Errors Based on the predicted state and the segmented VOs, we design a reasoning logic to detect occlusion and segmentation error. Occlusion are discovered when two or more tracked objects start to merge with one another, as illustrated in examples in Fig. 4(a). Occlusion detection can be accomplished by checking the predicted state of every pair of target objects in the tracking list. In order to discover segmentation errors, a validation gate (see [26]) is built upon the predicted system state of the
108
J.-N. Hwang and V. Gau
Fig. 4 Different cases of occlusion and segmentation errors. (a) Occlusion cases; (b) segmentation error cases
target object. Several segmentation error examples are shown in Fig. 4(b). If we cannot find any measurement within the validation gate of a target object, then we regard it as a segmentation error because either the positions of all the VOs are far away from the predicted position, or the major axes and minor axes of the ellipses fitted on the VOs are very different from the predicted ones for the target object. Under such circumstances, we do not trust the segmented VOs. A segmentation error is also discovered when a segmented VO is within the validation gate but there is a big difference between the size of the VO and the size of the tracked object.
3.2 Measurement Selection via Segmented VOs When there is no occlusion or segmentation error, the measurements for a target object are obtained by referring to the segmented VOs within the validation gate of the target object. If a measurement is within the validation gate, we put it into a measurement candidate list LMC . A weight is computed for each measurement candidate in LMC . If there is only one measurement candidate in LMC , the weight for this measurement candidate is set to one. If more than one measurement candidates are present in LMC , the weights are proportional to the similarities between the measurement candidates and the target object. The similarities depend on the appearance model and the chosen similarity metric, which are elaborated in Subsect. 3.4.
3.3 Measurement Selection via Adaptive Particle Sampling Under occlusion or segmentation error, we cannot refer to the segmented VOs when selecting measurements. Therefore, under such circumstances, we perform particle sampling to select measurement candidates for the object. First, Np random samples are generated in the four-tuple state space (uk , vk , ak , bk ) around the point (uˆ k|k−1 , vˆk|k−1 , aˆ k|k−1 , bˆk|k−1 ), where uˆ k|k−1 , vˆ k|k−1 , aˆ k|k−1 , and bˆk|k−1 are obtained from the predicted system state xˆk|k−1 of the target object at frame k. Then, each particle is associated with a weight. Finally, particles with higher weights are put into the measurement candidate list LMC of the target object.
7 Tracking of Multiple Objects over Camera Networks
109
Fig. 5 Example of overlap area and non-overlap area in (3)
The Gaussian distribution is considered for particle sampling, where the mean of the Gaussian is set as the predicted state (uˆ k|k−1 , vˆ k|k−1 , aˆ k|k−1 , bˆk|k−1 ). The innovation covariance matrix Sk−1 obtained at update phase of KF in the previous time instance can be used as the covariance matrix of the multivariate Gaussian. Since the measurement state is four-dimensional, the innovation covariance matrix Sk−1 is a four-by-four matrix. Let the four diagonal elements of Sk−1 be s11 , s22 , s33 , and s44 . The parameters of sampling range u, v, a, and b can be dynamically determined by referring to the diagonal elements of the innovation covariance matrix Sk−1 using (1) and (2). √ √ u = max 5, s11 , v = max 5, s22 , (1) √ √ ˆ b = max bk|k−1 /20, s44 . (2) a = max aˆ k|k−1 /20, s33 , Each four-tuple particle j corresponds to an ellipse region Ellipsej in the twodimensional image plane. The weight wj for particle j is computed as follows: NonOverlapAreaj OverlapAreaj wj ∝ ω1 + ω2 1 − + ω3 · Similarityj . (3) size(VO) size(Ellipsej ) In (3), the summation of the adjusting factors, ω1 , ω2 , and ω3 , equals to one. The OverlapAreaj and NonOverLapAreaj between Ellipsej and the segmented VOs are illustrated in Fig. 5. The term Similarityj is the similarity measure between the appearance of Ellipsej at the current frame and the appearance of the KF tracked target object (see Subsect. 3.4).
3.4 Adaptive Appearance When the degree of occlusion for an object exceeds a threshold under an occlusion case, adaptive appearance should be applied to ensure the robustness against occlusion. Suppose that an object O1 inter-occludes with objects O2 , O3 , . . . , the degree of occlusion for O1 is defined in (4), and the adaptive appearance ξ˜k (O1 ) is defined in (5). size(O1 ∩ O2 ) + size(O1 ∩ O3 ) + · · · , size(O1 ) ξ˜k (O1 ) = ξk O1 − (O1 ∩ O2 ) − (O1 ∩ O3 ) − · · · .
drg_occ(O1 ) =
(4) (5)
110
J.-N. Hwang and V. Gau
Notice that the size and position information required in (4) and (5) are based on the predicted state from KF. In implementation, we have to maintain a minimum region for the appearance of the object that is seriously occluded. Also, if the degree of occlusion of an object exceeds a threshold, we should stop updating the appearance of the object. We can adopt the proposed adaptive appearance concept on various kinds of appearance models such as histograms and color matrices. Commonly used metrics to calculate the distance between two discrete distributions such as histograms include L1, L2, and Bhattacharyya distances [27]. To convert distances into similarities, (6) can be used. Similarity(h1 , h2 ) = √
1 2πσ
e
D(h1 ,h2 ) 2σ 2
.
(6)
In (6), h1 and h2 are two color histograms, either obtained from adaptive or nonadaptive appearance. On the other hand, similarity metrics used to compare color matrices include distance based similarities and cross-correlation coefficients. Suppose that two color matrices are arranged in two n × 1 vectors An and Bn , respectively. Let A and B denote the mean values of An and Bn . The cross-correlation coefficient of the two vectors can be obtained using (7). Notice that particles at different scaling levels need to be resized in order to compute the cross-correlation coefficients when using color matrices. (An − A )(Bn − B ) . (7) CR (An , Bn ) = n ( n (An − A )2 )( n (Bn − B )2 )
4 Tracking Across Multiple Cameras In order to consistently label new objects entering a camera, the relations between adjacent cameras are first estimated offline by applying image registration techniques. After that, the brightness difference is compensated by matching the histograms of overlapping area. The brightness difference will be also used to modify the histogram of an object when it passes across cameras. Both the distance to FOV lines and appearance are used to measure the degree of matching between each candidate object correspondence pairs to select the best global label.
4.1 Establish Field of View (FOV) Lines The FOV of a camera Ci is a rectangular pyramid in the space. The intersection of each planar side s of this pyramid with the ground plane defines 3D FOV lines [1], which mark the viewing limit of Ci from side s. If the viewing coverage of camera Ci is overlapping with that of Cj from side s, there would be a 2D line Li,s j that marks the limit of what may be visible in the image plane of Cj , and this line is
7 Tracking of Multiple Objects over Camera Networks
111
called an FOV line. At most, there are four FOV lines of one camera as visible in the other one, which correspond to four sides of image plane. Therefore, if we stitch two images Ii and Ij from overlapping cameras Ci and Cj , the overlapping area will mark the common visible area from both cameras, and the intersection of the four sides of Ii with Ij are the approximation of FOV lines of Ci in Cj . Two images Ii and Ij can be aligned automatically through the following steps.
Extracting Landmark Points The landmark points in each image are extracted by using scale-invariant feature transformation (SIFT) [28]. First, candidate landmark points are extracted by searching for the scale-space extrema over all scales and locations using difference of Gaussian function. Once a candidate point is found, a 3D quadric function is fitted to refine the landmark locations. Then, one or more orientations are assigned to each landmark point based on local image gradient information. Finally, orientation histograms are computed to represent all those landmark points.
Findling Matching Landmark Points After obtaining landmark points in each image, the corresponding pairs are indexed using an efficient nearest neighbor search method, called Best Bin First algorithm [29]. It is a k-d tree based algorithm with a modified search order by taking into account the position of the query location, i.e., the bins in feature space are searched in order of increasing distance from the query location. A good approximation can still be found by cutting off further search after a certain number of the nearest bins have been explored.
Aligning Two Images Given a set of corresponding landmark points (x, y) ↔ (x , y ) from two images, the homography H that describes the projective transformation between those points can be estimated by the Random Sample Consensus (RANSAC) algorithm [30] in an iterative way. In each iteration k, four pairs of landmark points are randomly selected to calculate a candidate homography Hk . Then all other landmark points in image Ii are transformed with Hk , and the number of inliers in this iteration is counted as those with distance dH (Hk (x, y), (x , y )) < th , where dH is the Euclidean distance between two points and th is the tolerance threshold. After the iterative process is finished, the largest set of inliers are used to estimate the best homography H ∗ in a least-square sense. Finally, image Ii is registered to Ij by transforming it with H ∗ . The FOV lines of Ci into Cj are approximated by the intersection between the transformed four sides of image Ii and image Ij .
112
J.-N. Hwang and V. Gau
For cameras with non-overlapping views, we assume that there is an intermediate image Iij which is overlapping with both images Ii and Ij . This is a reasonable assumption for most surveillance scenarios where cameras views may not be overlapping, while taking some bridging images can be easily feasible. In this way, images Ii and Ij can also be registered sequentially in a similar way as in reconstructing panoramas. The extended FOV lines are defined as the extended virtual boundaries of FOV of one camera. One such extended FOV line example, which is created by the use of one intermediate image, is illustrated in Fig. 10. j After registration, a visibility map Vi is built for image Ii , where Vi (x, y) = 1 represents that position (x, y) is also visible in camera Cj . The subset of all cameras C in which the nth object Oin in Ci can be seen in other cameras is given by j (8) C i (n) = j | Vi (x, y) = 1, ∀i = j and j ∈ C , where (x, y) is the current feet position that denotes the lower end of the ellipse major axis fitted on the object.
4.2 Brightness Calibration of Neighboring Cameras After image registration, the grayscale cumulative histograms of overlapping areas are calculated for both images. Histogram matching method [15] is used to compensate for brightness changes.
4.3 Consistent Labeling Across Cameras When the nth object Oin enters Ci from a specific side s, the visibility map is first checked for the visibility of Oin in other cameras. If Oin is only visible in Ci , then assign a new global label to Oin . Otherwise, Oin should be also visible in other cameras defined in C i (n). Suppose all camera videos have been synchronized, the corresponding label of Oin is searched in the following steps. First, a list of corresponding candidates is generated for Oin . For each Cj ∈ C i (n), search for objects Ojm with d(Ojm , Li,s j ) < td that moves toward the visible region of Ci from Cj from side s, where d(Ojm , Li,s j ) is the minimum distance between the feet position of Ojm and FOV lines Li,s j associated with the specific side s that Oin enters Ci . An object is defined to be moving toward the visible area of Ci , if it enters Ci from invisible area of Cj and has been tracked in Cj for a certain time, e.g., one second. In particular, if two cameras Ci and Cj have nonoverlapping views, then the feet position of an object in the “blind” region is predicted by a constant velocity model in which the constant velocity is approximated by using least-square fitting of its previous visible feet positions in Cj .
7 Tracking of Multiple Objects over Camera Networks
113
According to the FOV constraints in [1], the candidate object with the minimum distance to its corresponding FOV line is selected as the most possible match for Oin . We define the likelihood of matching two objects in terms of the distance to the FOV line as 2 m i,s d (Oj , Lj ) 1 Pd Oin , Ojm = √ , (9) exp − σd2 2πσd where σd controls the width of the region considered for selecting candidate matches. To take into account of the errors in detecting feet position and estimating FOV lines, we also define the likelihood of matching two objects in terms of appearance using color dissimilarity as 2 n m d (h1 , h2 ) 1 exp − Pc Oi , Oj = √ , (10) σc2 2πσc where h1 , h2 are the color histograms of the two objects Ojm and Oin respectively. And d(h1 , h2 ) is the dissimilarity (in terms of Euclidean distance) between the histograms. Then, for each candidate, the likelihood of assigning the global label of Ojm to Oin is defined as the product of Pd and Pc as (11) P Oin , Ojm = Pd Oin , Ojm Pc Oin , Ojm . If the object with the highest likelihood as computed in (11) has been globally labeled, then we assign its global label to Oin . Otherwise, a new global label is assigned to both of them. After handoff labels, the corresponding feet position in Cj is saved. With sufficient number of feet positions being collected, a least-square line fitting method can be used to update FOV lines. There is a special case about initial occlusion. If more than one corresponding candidates have very high and close likelihoods as computed in (8), it is possible that these candidates may be in occlusion initially when they enter Ci . In this case, a list of labels is created for Oin . If Oin splits later, then only the appearance likelihoods are computed between the objects split from Oin and that from the label list to select the most likely global labels.
5 Experimental Results 5.1 Performance of Tracking within a Single Camera Since particles are generated around the predicted position of the target object and the parameters used to generate the particles are always updated by KF, only a small number of particles (empirically 40–50) are required to keep the positioning errors and scaling errors low for both sampling distributions. Some of the test cases are shown in Fig. 6. Since the average object size in the test cases is around 100 × 160
114
J.-N. Hwang and V. Gau
Fig. 6 Typical test cases for tracking
Fig. 7 Tracking results on a seriously occluded object using different appearance models
pixels, the positioning errors and scaling errors are relatively small compared to the object sizes. As shown in Fig. 7 [5], applying the proposed adaptive appearance based on color matrices (7(a)) and color histograms (7(b)) significantly improves the positioning and scaling accuracy compared to non-adaptive appearances.
5.2 Performance of Tracking Across Multiple Cameras The proposed approach was tested for two cameras with overlapping and nonoverlapping views separately. Figure 8 shows the results of two surveillance cameras monitoring of a parking lot with overlapping views. The left camera is camera 1, the right one is camera 2. Two tracked persons P1 and P2 in camera 2 were passing through the FOV lines almost at the same time (see Fig. 8(c)). The approach can successfully label them after they entered camera 1. Also, it can be seen from Figs. 8(c) and 8(d) that the FOV lines are established accurately since when P2 was passing through FOV line L12 , at the same time he appeared in camera 1. And as he was passing through L12 in camera 1, he was leaving camera 2 in the meantime. The tracking results for non-overlapping case are shown in Fig. 9, where two persons P1 and P2 were walking from right to left. P1 entered the blind region first but with a slower speed than that of P2. The established extended FOV line for nonoverlapping case is given in Fig. 10. Although the scene in the intermediate image had changed slightly since it was not taken at the same time as the testing sequence had been taken, the image registration and extended FOV line can still be effectively established. It can be seen from Fig. 9(c), based on the predicted position approximated from the constant velocity, that P2 entered the left camera from the “blind”
7 Tracking of Multiple Objects over Camera Networks
115
Fig. 8 Tracking results for cameras with overlapping views. FOV lines are marked with blue color
Fig. 9 Tracking results for cameras with non-overlapping views
Fig. 10 Motion prediction in “blind” region. The extended FOV line L12 is marked with blue color. The predicted positions for P1 and P2 when P2 appears in camera 1 are marked with red color
region earlier since it is closer to the extend FOV line. The tracking results for both cases with occlusions are also given in Figs. 8(b), 8(c) and 8(d).
116
J.-N. Hwang and V. Gau
6 Conclusion We have presented a unified framework for tracking objects within a single camera and across multiple cameras for both overlapping and non-overlapping scenarios. Taking advantage of both the closed-form equations for optimal prediction and update from Kalman filters and the versatility of particle sampling for measurement selection under occlusion or segmentation error cases, the proposed adaptive particle and KF based approach for tracking within a single camera achieves both high tracking accuracy and computational simplicity. Compared with traditional particle filter based tracking methods, the proposed method generates particles only when necessary. Also, we can avoid degeneracy problem because the sampling position and range are dynamically determined by parameters that are updated by Kalman filters. There is no need to spend time on processing particles with very small weights. After measurement selection, the enhanced probabilistic data association is able to successfully associate the measurements with each target object for filter update. For tracking across multiple cameras, the (extended) FOV lines can be established automatically without training. Combining the distance to FOV lines and color similarity to select correct matches makes our approach more applicable to initial occlusion scenarios and less sensitive to the errors in estimating FOV lines and feet position of objects being tracked. Updating FOV lines with tracking results further increases the adaptation capabilities of our approach.
References 1. Khan, S., Shah, M.: Consistent labeling of tracked objects in multiple cameras with overlapping fields of view. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1355–1360 (2003) 2. Qu, W., Schonfeld, D., Mohamed, M.: Real-time distributed multi-object tracking using multiple interactive trackers and a magnetic-inertia potential model. IEEE Trans. Multimed. 9(3), 511–519 (2007) 3. Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82, 35–45 (1960) 4. Chen, K., Lai, C., Hung, Y., Chen, C.: An adaptive learning method for target tracking across multiple cameras. In: CVPR, pp. 1–8 (2008) 5. Cheng, H.Y., Hwang, J.N.: Resolving occlusion and segmentation errors in multiple video object tracking. In: SPIE Conference on Computational Imaging, San Jose (2009) 6. Zhu, L., Hwang, J.-N., Cheng, H.-Y.: Tracking of multiple objects across multiple cameras with overlapping and non-overlapping views. In: IEEE Int’l Symposium on Circuits and Systems, Taipei, Taiwan, May (2009) 7. Uhlmann, J.K.: Algorithms for multiple target tracking. Am. Sci. 80(2), 128–141 (1992) 8. Foresti, G.L.: A real-time system for video surveillance of unattended outdoor environments. IEEE Trans. Circuits Syst. Video Technol. 8(6), 697–704 (1998) 9. Foresti, G.L.: Object recognition and tracking for remote video surveillance. IEEE Trans. Circuits Syst. Video Technol. 9(7), 1045–1062 (1999) 10. Maggio, E., Cavallaro, A.: Hybrid particle filter and mean shift tracker with adaptive transition model. In: Proc. of IEEE Signal Processing Society Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. 221–224 (2005) 11. Zhou, S.K., Chellappa, R., Moghaddam, B.: Visual tracking and recognition using appearanceadaptive models in particle filters. IEEE Trans. Image Process. 13(11), 1491–1506 (2004)
7 Tracking of Multiple Objects over Camera Networks
117
12. Wu, L., Deng, Z., Jia, P.: A post-resampling based particle filter for online Bayesian estimation and tracking. In: The Sixth World Congress on Intell. Control and Automation, vol. 1, pp. 4331–4334 (2006) 13. Gustafsson, F., et al.: Particle filters for positioning, navigation and tracking. IEEE Trans. Signal Process. 50(2), 425–437 (2002) 14. Xu, G., Ma, Y.F., Zhang, H.J., Yang, S.Q.: An HMM-based framework for video semantic analysis. IEEE Trans. Circuits Syst. Video Technol. 14(11), 1422–1433 (2005) 15. Magee, D.R.: Tracking multiple vehicles using foreground, background and motion models. Image Vis. Comput. 22(2), 143–155 (2004) 16. Veeraraghavan, H., Masoud, O., Papanikolopoulos, N.P.: Computer vision algorithms for intersection monitoring. IEEE Trans. Intell. Transp. Syst. 4(2), 78–89 (2003) 17. Haritaoglu, I., Harwood, D., Davis, L.S.: W4: Real-time surveillance of people and their activities. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 809–830 (2000) 18. Velipasalar, S., Wolf, W.: Recovering field of view lines by using projective invariants. In: ICIP, vol. 5, pp. 24–27 (2004) 19. Chang, T., Gong, S.: Tracking multiple people with a multi-camera system. In: WOMOT with ICCV (2001) 20. Cai, Q., Aggarwal, J.K.: Tracking human motion in structured environments using a distributed camera system. IEEE Trans. Pattern Anal. Mach. Intell. 21(11), 1241–1247 (1999) 21. Lee, L., Romano, R., Stein, G.: Monitoring activities from multiple video streams: establishing a common coordinate frame. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 758–767 (2000) 22. Arora, R., Hu, Y.H., Dyer, C.R.: Estimating correspondence between multiple cameras using joint invariants. In: Proc. Int. Conf. Acoustics, Speech, and Signal Processing (2009) 23. Javed, O., Rasheed, Z., Alatas, O., Shah, M.: Knight M: a real time surveillance system for multiple overlapping and non-overlapping cameras. In: ICME, vol. 1, pp. 649–652 (2003) 24. Kumar, P., Chilgunde, A., Ranganath, S., Huang, W.: Multi-camera target tracking in blind regions of cameras with non-overlapping fields of view. In: BMVC, pp. 397–406 (2004) 25. Makris, D., Ellis, T., Black, J.: Bridging the gaps between cameras. In: CVPR, vol. 2, pp. 205– 210 (2004) 26. Cheng, H.Y., Hwang, J.N.: Multiple target tracking for crossroad tracking utilizing modified probabilistic data association. In: IEEE Int’l Conf. on ASSP, Honolulu, Hawaii, April (2007) 27. Nummiaro, K., Koller-Meier, E., Gool, L.V.: An adaptive color-based particle filter. Image Vis. Comput. 21(1), 99–110 (2003) 28. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004) 29. Beis, J., Lowe, D.: Shape indexing using approximate nearest-neighbour search in highdimensional spaces. In: CVPR, pp. 1000–1006 (1997) 30. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Chapter 8
Toward Robust Online Visual Tracking Ming-Hsuan Yang and Jeffrey Ho
Abstract We pursue a research direction that will empower machines with simultaneous tracking and recognition capabilities similar to human cognition. Toward that, we develop algorithms that leverage prior knowledge/model obtained offline with information available online via novel learning algorithms. While humans can effortlessly locate moving objects in different environments, visual tracking remains one of the most important and challenging problems in computer vision. Robust cognitive visual tracking algorithms facilitate answering important questions regarding how objects move and interact in complex environments. They have broad applications including surveillance, navigation, human computer interfaces, object recognition, motion analysis and video indexing, to name a few.
Keywords Visual tracking · Object tracking · Online learning · Incremental learning
1 Introduction While we have witnessed significant progress in visual tracking over the last decade [3, 7–9, 11, 12, 23, 28, 30, 59, 63], developing visual tracking systems that match human cognitive abilities is still a very challenging research problem. Existing visual tracking systems tend to perform well over short durations, and more importantly M.-H. Yang () Electrical Engineering and Computer Science, University of California, Merced, CA 95344, USA e-mail:
[email protected] J. Ho Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32607, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_8, © Springer-Verlag London Limited 2011
119
120
M.-H. Yang and J. Ho
only when the target objects stay visible in camera view (i.e., not in and out of the scene). One main reason is that most existing algorithms employ a static representation of the target object and operate on the premise of constancy in appearance. In other words, most algorithms assume that the appearance of a target object does not change rapidly. For such algorithms to perform robustly, it is imperative to collect a large set of training images to account for all possible appearance variations caused by change of viewing angles and illumination. These models do not exploit rich and important information (e.g., most recent appearance and illumination condition) that becomes available online during tracking. More importantly, it is of great interest to develop algorithms that leverage prior knowledge and online learning to enhance the recognition and tracking capabilities. Another reason is that most existing algorithms are not able to detect and recover from drifts accumulated during tracking. Once the target position is initialized, most tracking algorithms operate as a series of predictions, and consequently accumulated drifts are inevitable unless they are able to reinitialize their positions periodically. Finally, tracking articulated objects poses additional difficulties due to high dimensionality of the state variables and partial occlusion. The above-mentioned problems entail the need for learning robust appearance models adaptively which in turn facilitate the tracking processes as well as algorithms to detect and correct deviations from the true target locations. Specifically, a robust appearance model should constantly learn a compact notion of the “thing” being tracked rather than treating the target as a set of independent pixels, i.e., “stuff” [2]. For visual tracking, an appearance model needs to be learned efficiently and effectively to reflect the most recent appearance change of any target objects. We note that it is a daunting, if not impossible, task to collect a large set of data encompassing all appearance variation of a target object caused by change of pose, illumination, shape, and occlusion. Meanwhile, it is equally important to exploit prior knowledge or model, when available, within the online learning framework. Clearly, there is a need to develop robust algorithms that can learn to update appearance models online for any objects, and use these models to address drifting problems. We emphasize that the problems of detection, tracking, recognition and appearance models can be simultaneously addressed with online and prior learning. Here we present our works in addressing these problems.
2 Appearance Modeling for Visual Tracking Visual tracking essentially deals with non-stationary data, both the target object and the background, that change over time. Most existing algorithms are able to track objects, either previously seen or not, in short durations and in well controlled environments. However, due to drastic change in the object’s appearance or large lighting variation in its surroundings, these algorithms usually do not perform well after some period of time or have significant drifts. Although such problems can be ameliorated with recourse to richer representations and effective prediction schemes, most algorithms typically operate on the premise that the model of the target object
8 Toward Robust Online Visual Tracking
121
does not change drastically over time. These algorithms usually adopt a model of the target object first and then use it for tracking, without adapting the model to account for appearance change of the object due to variation of imaging conditions (e.g., viewing angles and illumination). Furthermore, it is usually assumed that all images are acquired with a stationary camera.
2.1 Learning Nonlinear Appearance Manifold It is well known that images of an object taken under different lighting and pose can be well modeled with nonlinear manifold via a set of linear subspaces [6, 40, 43]. However, prior work has focused on learning such models in a batch mode offline fashion. Here we describe algorithms that use nonlinear appearance models learned offline and show how they facilitate tracking and recognition tasks. We propose to learn nonlinear manifold with online update and clustering, as well as their underlying constraints. The nonlinear manifold is modeled with a set of submanifolds constructed in an online manner, where each submanifold is approximated with a PCA (Principal Component Analysis) subspace. It entails the need to efficiently process the incoming images into clusters from which submanifolds are constructed and updated. Each submanifold model is expected to capture certain appearance variation of the target object due to illumination and pose change. In addition, a nonlinear manifold provides a way to retain “long-term memory” of the target rather than to rely on one single subspace which has only “short-term memory.”
Learning Nonlinear Manifold Online The complex nonlinear appearance manifold of a target object k, Mk , is partitioned by a collection of submanifolds, C k1 , C k2 , . . . , where each models the appearances of the target object under illumination and pose change. The submanifold is approximated by a low-dimensional linear subspace computed by PCA using images observed sequentially (see Fig. 1). The tight coupling between the tracking and recognition components is achieved via the shared appearance models M1 , . . . , MN . Another difficulty is related to the definition of the 2 -distance d(I, Mk ) between an image I and a manifold Mk in the image space. By definition, d(I, Mk ) = d(I, x ∗ ) with x ∗ is a point on Mk having minimal 2 -distance to I (see Fig. 1). Even if an analytic description of Mk were available, finding x ∗ is generally not an easy problem. In our case Mk is, at best, modeled by a modest number of images sampled from it; therefore, Mk is available to us only through a very coarse and sparse representation with many “gaps” in which we have inadequate or incompletely information. The main focus of our work is to provide an effective definition for d(I, Mk ) that works for a coarse representation of Mk .
122
M.-H. Yang and J. Ho
Since the appearance manifold Mk is nonlinear, it is reasonable to decompose Mk into a collection of m simpler disjoint submanifolds, Mk = C k1 ∪ · · · ∪ C km , with C ki denoting a submanifold in a decomposition of person k’s appearance manifold. Each C ki is assumed to be amenable to linear approximations by a low-dimensional linear subspace computed through Principal Component Analysis (i.e., a PCA plane). We define the conditional probability p(C ki |I ) as the probaki bility m that kiC contains a point x with minimal distance to I . With pMk (x|I ) = i=1 p(C |I )pC ki (x|I ), we have, m d(x, I )pMk (x|I ) dx = p C ki |I d(x, I )pC ki (x|I ) dx d(I, Mk ) = Mk
=
m p C ki |I d I, C ki .
i=1
C ki
(1)
i=1
The equation above shows that the expected distance d(I, Mk ) can be treated as the expected distance between I and each C ki . In addition, this equation transforms the integral to a finite summation which is feasible to compute numerically. The details of this formation can be found in [25, 31]. For visual tracking and recognition in video sequences, we can exploit temporal coherence between consecutive image frames. As shown in the right panel of Fig. 1, the 2 -distance may occasionally be misleading during tracking/recognition. But if we consider previous frames in an image sequence rather than just one, then the set of closest points x ∗ will trace a curve on a submanifold C ki . In our framework, this is embodied by the term p(C ki |I ) in (1). We apply Bayesian inference to incorporate temporal information to provide a better estimate of p(C ki |I ) and thus d(I, Mk ); this will then yield better tracking/recognition performance. We show that
Fig. 1 (Left) Appearance manifold: A complex and nonlinear manifold Mk can be approximated as the union of several simpler submanifolds; here, each submanifold C ki is represented by a PCA plane. (Right) Difficulty of frame-based tracking/recognition: The two solid curves denote two different appearance manifolds, MA and MB . It is difficult to reach a decision on the identity from frame It−3 to frame It because these frames have smaller 2 -distance to appearance manifolds MA than MB . However, by looking at the sequence of images It−6 . . . It+3 , it is apparent that the sequence has most likely originated from appearance manifold MB [25, 31]
8 Toward Robust Online Visual Tracking
123
it is a recursive formulation that depends on the generative model, p(I |C ki ), and the kj transition probability, p(Ctki |Ct−1 ) [25, 31]. The connectivity between the submanifolds is modeled as transition probabilities, between pairs of submanifolds, and these are learned directly online via frequency estimation or simple counting. The integrated task of tracking and recognition is formulated as a maximum a posteriori estimation problem. Within our framework, the tracking and recognition modules are complementary to each other, and the capability and performance of one are enhanced by the other. Our approach contrasts sharply with more rigid conventional approaches in which these two modules work independently and in sequence. Recent work on incremental clustering data streams [10, 29] has shown its promise for its applicability to numerous types of data, including web documents, routing packages, financial transactions, and telephone records. We may draw on such ideas and extend to learning image data online for learning a set of PCA subspaces. Specially, it is worthwhile exploiting the characteristics pertaining to 2D image data in developing new algorithms to handle image sequences. In the vision context, it is important to exploit the fact that the similarity measure can be better modeled with distance from an image to a subspace. For image sequences, one may compute the distance from an incoming image I to the submanifold, C ki (see Fig. 1) rather than the 2 -distance to other images. On the other hand, randomization and sampling schemes have shown much promise in fast approximation of clustering data streams. We will exploit both characteristics in developing online approximate algorithms that are able to assign each image I to one or a couple of submanifold C ki for weighted update.
Online Update of Submanifold For each submanifold modeled by a PCA subspace, we have developed an efficient online subspace update algorithm [35] for appearance model based on the R-SVD algorithm [21] and the sequential Karhunen–Loeve method [32]. The proposed method not only updates the orthonormal basis but also the subspace means, which is of great importance for certain applications. For example, it can be applied to adaptively update the between-class and within-class covariance matrices used in Fisher linear discriminant analysis [37]. Experimental results show that our subspace update algorithm is 20% more efficient than the most related work [24]. We develop robust tracking algorithms using online appearance model with subspace update [35, 47, 48]. In contrast to the eigentracking algorithm [7], our algorithm does not require a training phase but instead learns the eigenbases online during the object tracking process. Thus our appearance model can adapt to account for change in pose, view angle, and illumination which is not captured by the set of training images. Our appearance model provides a richer description than simple curves or splines as used in [28], and has a compact notion of the “thing” being tracked [2]. The learned representation can also be utilized for other tasks such as object recognition. Furthermore, our algorithm is able to simultaneously track and learn a compact representation of the target object even when the camera is moving.
124
M.-H. Yang and J. Ho
Fig. 2 A person undergoing large pose, expression, appearance, and lighting change, as well as partial occlusions. The red window shows the maximum a posteriori estimate of the particle filter, and the green windows show the other particles with large weights. The images in the second row show the current sample mean, tracked region, reconstructed image, and the reconstruction error respectively. The third row shows the top 10 principal eigenvectors [35, 48]. The MATLAB code and data sets can be found at http://faculty.ucmerced.edu/mhyang
Our experiments [35, 48] show that robust tracking results can be obtained using this representation without employing more complicated wavelet features as in [30], although this elaboration is still possible and may lead to even better results. Figure 2 shows some experimental results using our algorithm. Note also that the view-based eigenbasis representation has demonstrated its ability to model the appearance of objects in different poses [40], and under different lighting conditions [6]. Consequently, the learned eigenbasis facilitates tracking objects undergoing illumination and pose change.
2.2 Leveraging Prior Knowledge with Online Learning Cognitive psychologists have suggested computational models to explain human visual cognition in terms of long-term and short-term memories [26]. Numerous studies suggest that interplay between long-term and short-term memories explains how humans track and recognize objects. For vision problems such as object recognition, we have access to prior knowledge of the objects. One natural way is to exploit the prior knowledge obtained offline with the information obtained online, thereby simultaneously enhancing the abilities to recognize and track objects robustly. In such situations, the prior knowledge can be encoded as long-term visual memory via construction of nonlinear manifold offline while the proposed online update algorithm serves as short-term memory to account for the most recent appearance change. Our algorithms [25, 31] facilitate the integration of long-term and short-term memories via the use of submanifold construction and update. The long-term memory provides rich prior information about the object appearance that helps in assigning one or more subspaces to account for appearance change when a new image arrives. In our study with one single submanifold, the newly arrived images are always added to the retained covariance matrix. It is of great interest to develop algorithms with soft assignments for multiple submanifolds to account for the appearance of a newly arrived image. In addition, algorithms may not take an irrevocable action when a data point arrives, and may modify the current model after
8 Toward Robust Online Visual Tracking
125
a group of image data arrives. Another direction to pursue is to periodically check the reconstruction errors after a group of points have been added. If the recognition error increases over time, it suggests the data points previously added should be discarded or downweighted. As each submanifold is modeled by a PCA subspace modeled with a covariance matrix, adding or deleting particular data points can be carried out efficiently with matrix update and downdate [21].
3 Learning Detectors Online for Visual Tracking It has been shown that in many scenarios an adaptive appearance model, which evolves during the tracking process as the appearance of the object changes, is the key to good performance [30, 48]. Another choice in the design of appearance models is whether to model only the object [5, 48], or both the object and the background [3, 11, 22, 37]. Many of the latter approaches have shown that training a model to separate the object from the background via a discriminative classifier can often achieve superior results. In this case, the tracking problem becomes a detection one as the target is located by scanning through the image region as shown in the left panel of Fig. 3. In particular, the recent advances in face detection [61] have inspired some successful real-time tracking algorithms [22]. However, almost all the detectors are constructed offline for a specific object class (e.g., faces, cars, and pedestrians) which demand significant efforts in collecting data as well as training time [13, 38, 50, 51, 56, 61]. Another challenge that is often not discussed in the literature is how to choose positive and negative examples when updating the appearance model. Most commonly this is done by taking the current tracker location as one positive example, and
Fig. 3 (Left) Tracking by detection with a greedy motion model: Generally, the appearance model is a discriminative classifier that can be trained in an online manner. A greedy motion model is used to search for the most probable location of the object in a frame within some search window. An alternative is to use particle filter. (Right) Updating a discriminative appearance model: (A) Using a single positive image patch to update a traditional discriminative classifier. The positive image patch chosen does not capture the object perfectly. (B) Using several positive image patches to update a traditional discriminative classifier. This can confuse the classifier causing poor performance. (C) Using one positive bag consisting of several image patches to update a MIL classifier [4]. The C++ code and data sets can be found at http://faculty.ucmerced.edu/mhyang
126
M.-H. Yang and J. Ho
sampling the neighborhood around the tracker location for negatives. If the tracker location is not precise, however, the appearance model ends up getting updated with a suboptimal positive example. Over time this can degrade the model, and can cause drift. On the other hand, if multiple positive examples are used (taken from a small neighborhood around the current tracker location), the model can become confused and its discriminative power can suffer as illustrated in the right panel of Fig. 3. Similar problems are encountered in object detection because it is difficult for a human labeler to be consistent with respect to how the positive examples are cropped. In other words, the exact object locations are unknown. In fact, Viola et al. [62] argue that object detection has inherent ambiguities that make it more difficult to train a classifier using traditional methods. For this reason they suggest the use of a Multiple Instance Learning (MIL) [15] approach for training object detectors offline. The basic idea of this learning paradigm is that during training, examples are presented in sets (often called “bags”), and labels are provided for the bags rather than individual instances. If a bag is labeled positive it is assumed to contain at least one positive instance, otherwise the bag is negative. For example, in the context of object detection, a positive bag could contain a few possible bounding boxes around each labeled object (e.g., a human labeler clicks on the center of the object, and the algorithm crops several rectangles around that point). Therefore, the ambiguity is passed on to the learning algorithm, which now has to figure out which instance in each positive bag is the most “correct.” Although one could argue that this learning problem is more difficult in the sense that less information is provided to the learner, it is actually easier in the sense that there is less risk of correct information being lost.
3.1 Multiple Instance Learning We present an online learning algorithm that builds detectors specific to the target object online for robust visual tracking [4]. The basic flow of the tracking system is illustrated in the left panel of Fig. 3, and it contains three components: image representation, appearance model and motion model. Local feature-based or partbased representations have been demonstrated to perform well when the objects are partially occluded [38, 61]. Our image representation consists of a set of Haar-like features that are computed efficiently for each image patch [16, 61]. The appearance model is comprised of a discriminative classifier which is able to return p(y = 1|x) (we will use p(y|x) as shorthand), where x is an image patch (or the representation of an image patch in feature space) and y is a binary variable indicating the presence of the object of interest in that image patch. At every time step t, our tracker maintains the object location lt∗ . Let l(x) denote the location of image patch x. For ∗ } that each new frame we crop out a set of image patches Xs = {x|s > l(x) − lt−1 are within some search radius s of the current tracker location, and compute p(y|x) for all x ∈ X s . We update the tracker location with maximum likelihood (ML). In other words, we do not maintain a distribution of the target’s location at every frame
8 Toward Robust Online Visual Tracking
127
and use a motion model where the location of the tracker at time t is equally likely to appear within a radius s of the tracker location at time (t − 1), although it could be extended with something more sophisticated, such as a particle filter, as is done in [48]. Once the tracker location is updated, we proceed to update the appearance model. We crop out a set of patches X r = {x|r > l(x) − lt∗ }, where r < s is the positive radius, and label this bag positive (recall that in MIL we train the algorithm with labeled bags). On the other hand, if a standard learning algorithm were used, there would be two options: set r = 1 and use this as a single positive instance, or set r > 1 and label all these instances positive. For negatives we crop out patches from an annular region X r,β = {x|β > l(x) − lt∗ > r}, where r is same as before, and β is a scalar. Since this generates a potentially large set, we then take a random subset of these image patches and label them negative. We place each negative example into its own negative bag. Note that we could place all negative examples into a single negative bag. However, our intuition is that there is no ambiguity about negative examples, so placing them into separate bags makes more sense. Figure 3 (right panel) contains an illustration comparing appearance model updating using MIL and a standard learning algorithm. Traditional discriminative learning algorithms for training a binary classifier that estimates p(y|x) require a training data set of the form {(x1 , y1 ), . . . , (xn , yn )} where xi is an instance (in our case a feature vector computed for an image patch), and yi ∈ {0, 1} is a binary label. In the Multiple Instance Learning framework the training data has the form {(X1 , y1 ), . . . , (Xn , yn )} where a bag Xi = {xi1 , . . . , xim } and yi is a bag label. The bag labels are defined as: yi = max(yij ), j
(2)
where yij are the instance labels, which are assumed to exist but are not known during training. Numerous algorithms have been proposed for solving the MIL problem [15, 62]. The algorithm that is most closely related to our work is the MILBoost algorithm [62], which uses the gradient boosting framework [19] to train a boosting classifier that maximizes the log likelihood of bags: log L = (3) log p(yi |Xi ) . i
Notice that the likelihood is defined over bags and not instances, because instance labels are unknown during training, and yet the goal is to train an instance classifier that estimates p(y|x). We therefore need to express p(yi |Xi ), the probability of a bag being positive, in terms of its instances. In [62] the Noisy-OR (NOR) model is adopted for doing this: p(yi |Xi ) = 1 − j (1 − p(yi |xij )). This equation has the desired property that if one of the instances in a bag has a high probability, the bag probability will be high as well. However, the MILBoost algorithm is a batch algorithm and cannot be trained in an online manner as we need in our tracking application. We propose an online MIL boosting algorithm to learn object specific detectors for visual tracking. The goal of boosting is to combine many weak classifiers h(x)
128
M.-H. Yang and J. Ho
(usually decision stumps) into an additive strong classifier: H(x) = K k=1 αk ht (x) where αk are scalar weights. There have been many boosting algorithms proposed to learn this model in batch mode [18, 19], but generally this is done in a greedy manner where the weak classifiers are trained sequentially. After each weak classifier is trained, the training examples are re-weighted such that examples that were previously misclassified receive more weight. If each weak classifier is a decision stump, then it chooses one feature that has the most discriminative power for the entire training set. In this case boosting can be viewed as performing feature selection, choosing a total of K features, which is generally much smaller than the size of the entire feature set.
3.2 Learning Detectors with Online Multiple Instance Boosting In [42], Oza develops an online variant of the discrete AdaBoost algorithm [18], which minimizes the exponential loss function. As such, this online algorithm is limited to classification problems. We take a statistical view of boosting similar to in [19] where the algorithm minimizes a generic loss function J . In this view, the weak classifiers are chosen sequentially to optimize the following criteria: (hk , αk ) = argmin J (Hk−1 + αh), h∈H,α
(4)
where Hk−1 is the strong classifier made up of the first (k − 1) weak classifiers, and H is the set of all possible weak classifiers. In batch boosting algorithms, the loss function J is computed over the entire training data set. In our case, for the current video frame we are given a training data set {(X1 , y1 ), (X2 , y2 ), . . .}, where Xi = {xi1 , xi2 , . . .}. We would like to update our estimate of p(x|y) to minimize the negative log likelihood of these data (3). We model the instance probability as p(y|x) = σ (H(x)) where σ (x) = 1+e1 −x is the sigmoid function, and the bag probabilities p(y|X) using the NOR model described above. To simplify the problem, we absorb the scalar weights αt into the weak classifiers, by allowing them to return real values rather than binary. To perform online feature selection, our algorithm maintains a pool of M > K candidate weak classifiers h. We update all of these weak classifiers in parallel, similar to [22]. Note that although examples are passed in bags, the weak classifiers in a MIL algorithm are instance classifiers, and therefore require instance labels yij . Since these are unavailable, we pass in the bag label yi for all instances xij to the weak training procedure. We then choose K weak classifiers h from the candidate pool sequentially, using the following criteria: hk =
argmin log L(Hk−1 + h).
h∈{h1 ,...,hM }
(5)
We use classifier hk that can be updated online, and each classifier is composed of a Haar-like feature fk and modeled with univariate Gaussian distributions whose parameters are updated online. The classifiers return the log likelihood ratio based
8 Toward Robust Online Visual Tracking
129
on the estimated Gaussian distributions. When the weak classifier receives new data {(x1 , y1 ), . . . , (xn , yn )} we use the weighted running mean and variance for update. To learn detectors online for real-time visual tracking, we represent each image patch as a vector of Haar-like features [61] where they are randomly generated, similar to [16]. The feature value is then a weighted sum of the pixels in all the rectangles, which can be computed efficiently using the integral image described in [61]. The proposed online visual tracking algorithm with Multiple Instance Learning is dubbed as MILTrack algorithm. We evaluate the proposed MILTrack algorithm on several challenging video sequences, some of which are publicly available. For comparison, we implemented a tracker based on the Online-AdaBoost (OAB) algorithm described in [22]. We plug this learning algorithm into our system, and used the same features and motion model as for MILTrack. We demonstrate the merits of the proposed MILTrack algorithm with experiments where all algorithm parameters were fixed (i.e., no tuning for particular sequences). To further gauge performance we also compare our results to the recently proposed algorithms using online discrete AdaBoost [22] and local histograms [1]. For MILTrack we sample positives in each frame using a positive radius r = 5, which generates a total of 45 image patches composing one positive bag. For the OAB tracker we experiment with two variations. In the first variation we set r = 1 generating only one positive example per frame; in the second variation we set r = 5 as we do in MILTrack (although in this case each of the 45 image patches is labeled positive). The reason we experiment with these two versions was to show that the superior performance of MILTrack is not simply due to the fact that we extract multiple positive examples per frame. Some results are shown in Fig. 4 and Fig. 5. These sequences exhibit many occlusions, lighting and appearance variations, and fast motion which causes motion blur. For the “Occluded Face” sequence,
Fig. 4 Screen shots of tracking results with zoom-in images [4]. Videos and source code can be found at http://faculty.ucmerced.edu/mhyang
130
M.-H. Yang and J. Ho
Fig. 5 Error plots for three test video clips. See [4] for details
FragTrack performs poorly because it cannot handle appearance changes well (e.g., when the subject puts a hat on, or turns his face). In all cases our MILTrack algorithm outperforms both versions of the Online AdaBoost Tracker, and in most cases it outperforms the FragTrack algorithm as well. The reason for the superior performance is that the Online MILBoost algorithm is able to handle ambiguously labeled training examples, which are provided by the tracker itself. On the other hand, when the Online AdaBoost Tracker is updated with multiple positive examples it performs quite poorly because the classifier is unable to learn a good decision boundary between positives and negatives. We notice that even when MILTrack loses the target due to severe occlusions, it is able to recover quickly since the temporary distraction to the appearance model is not as significant. The proposed online MILBoost algorithm can easily exploit the prior knowledge of the target object. From a set of training images, we can extract a set of Haarlike features that best model the target object before applying online MILBoost for visual tracking. In addition, the motion model we used here is fairly simple, and could be replaced with something more sophisticated, such as a particle filter as in [48] for additional gain in performance. We also plan to investigate the use of other part-based appearance models [1] with our algorithm and evaluate these alternative representation methods. The proposed algorithms provides the basic mechanism to
8 Toward Robust Online Visual Tracking
131
detect and recognize the objects when they come in and out of the camera view. One straightforward way is to set thresholds on the detector confidence. However, more sophisticated algorithms will be investigated to empower machines to mimic human-level visual cognition.
3.3 Articulated Objects Tracking articulated objects is of great importance for motion analysis with broad impact. Once we are able to track single objects robustly with online algorithms, the next question is how we can extend these ideas to articulated objects. A major difficulty in applying Bayesian tracking methods for tracking 3D human body pose is the high dimensionality of the state vector—typically 20–60 dimensions depending on the parameterization [14, 52, 55, 60]. Although the state vector is high dimensional, the pose parameters typically can be assumed to lie on a low-dimensional manifold embedded in the high-dimensional space. We propose to approximate the low-dimensional manifold so that the dimensionality of the state vector is reduced for efficient and effective Bayesian tracking [36]. To achieve this goal, a statistical model known as the globally coordinated mixture of factor analyzers (GCMFA) is learned from motion capture data. This model provides a global parametrization of the low-dimensional manifold. Each factor analyzer in the mixture is a “locally linear dimensionality reducer” that approximates a part of the manifold. The global parametrization of the manifold is obtained via aligning these locally linear pieces in a global coordinate system. The parameters of the GCMFA model for our application are learned from motion capture sequences. Since the GCMFA is effective in preserving important information during dimensionality reduction [49, 57], it can capture the key kinematic information with the use of motion capture sequences as training data. The global coordination of the local linear factor analyzers ensures that poses have a globally consistent parameterization in the latent space. The global coordination also preserves the continuity of the manifold as similar poses are mapped to the coordinates that are close to each other on the manifold. The density of high-dimensional pose data is approximated by the piecewise linear Gaussian factor analyzers in the low-dimensional space. By encouraging the internal coordinates of the factor analyzers to agree, a single, coherent low-dimensional coordinate system can be obtained for dimensionality reduction. The mixing and coordination of the linear factors provides nonlinear bidirectional mappings between the low-dimensional (latent) space and the pose space. Because the nonlinear mapping functions are broken down into linear factors, the learning algorithm is efficient and can handle large training data sets with grace. Once the GCMFA model is learned, we demonstrate its use in a multiple hypothesis tracker with a dimensionality reduced state space for 3D human tracking. The performance of this tracker is currently being evaluated on the HumanEva benchmark data sets [54]. In experiments with real videos, the proposed system reliably tracks body motion during self-occlusions and in the presence of motion blur. Figure 6 shows some tracking results using the proposed algorithm, annealed particle
132
M.-H. Yang and J. Ho
Fig. 6 Sample tracking results from the test video sequence of S2 (Left) and S3 (Right) performing boxing using the annealed particle filter [14] (APF), the GPLVM-based method [58, 60] (GPLVMPF), and the proposed method [33]
[14] (APF) and Gaussian Process Latent Variable Model (GPLVM) [58, 60]. The proposed algorithm is able to accurately track large movements of the human limbs in adjacent time steps by propagating each cluster’s information over time in the multiple hypothesis tracker. Some quantitative evaluation [33] using the HumanEva benchmark data sets shows that our method produces more accurate 3D pose estimates than those obtained via two previously-proposed Bayesian tracking methods [14, 60]. Although the GCMFA framework has all the desirable properties of a dimensionality reduction algorithm for tracking, a main disadvantage is that one has to choose the optimal structure of the GCMFA model empirically or manually. We address this issue by proposing a variational Bayesian solution [34] for automatic selection of the optimal model structure in a way similar to [20]. In addition, we plan to learn part-based object detector online using the adaptive appearance models presented in Sect. 2.1 as well as the algorithm described in Sect. 3.2, and exploit the constraints enforced among them with dimensionality reduction techniques such as GCMFA. Several methods have shown the potential of exploiting constraints among subspaces and parts in vision applications [53]. With the current algorithm, the dimensionality reducers are used mainly to map between the input and low-dimensional spaces. One way to extend the current algorithm is to further exploit the temporal and spatial constraints of the clusters in the low-dimensional space. The appearance model in our current algorithm can be improved with online update using the algorithms discussed in Sects. 2.1 and 3 to better account for change in illumination and shape. It is of great interest to extend the online algorithms discussed in Sect. 3.2 to learn detectors for parts of an articulated object. Different from the recent work that learn body parts offline [17, 27, 38, 39, 45, 46, 53], we aim to exploit the potential of learning detectors online with their constraints aside from relying on prior knowledge. As it involves learning multiple detectors simultaneously as well as their kinematic constraints, we expect to explore top-down and bottom-up approaches for efficient visual tracking.
8 Toward Robust Online Visual Tracking
133
One single model or algorithm is not expected to succeed in all tracking scenarios, and we will explore other representations such as integral histograms [44] for tracking articulated objects at a distance. Our recent results show that articulated objects at a distance can be well tracked by integrating online appearance models, object segmentation and spatial constraints of the articulating parts [41]. We plan to pursue this line of research to account for larger shape deformation and self occlusions.
4 Conclusions The ultimate goal of our research focuses on developing efficient and effective algorithms that mimic human cognitive abilities for tracking as well as recognizing objects. Toward that, we have developed several algorithms that leverage online and offline information for robust tracking and recognition. As one single model or method is not expected to succeed in all tracking scenarios, we plan to exploit generative and discriminative algorithms for tracking objects in different scenarios. We also aim to further explore the interplay between online and offline learning for robust appearance models.
References 1. Adam, A., Rivlin, E., Shimshoni, I.: Robust fragments-based tracking using the integral histogram. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 798–805 (2006) 2. Adelson, E.H., Bergen, J.R.: The plenoptic function and the elements of early vision. In: Landy, M., Movshon, J.A. (eds.) Computational Models of Visual Processing, pp. 1–20. MIT Press, Cambridge (1991) 3. Avidan, S.: Ensemble tracking. IEEE Trans. Pattern Anal. Mach. Intell. 29(2), 261–271 (2007) 4. Babenko, B., Yang, M.-H., Belongie, S.: Visual tracking with online multiple instance learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 983–990 (2009) 5. Balan, A.O., Black, M.J.: An adaptive appearance model approach for model-based articulated object tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 758–765 (2006) 6. Belhumeur, P., Kreigman, D.: What is the set of images of an object under all possible lighting conditions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–277 (1997) 7. Black, M.J., Jepson, A.D.: Eigentracking: Robust matching and tracking of articulated objects using a view-based representation. Int. J. Comput. Vis. 26(1), 63–84 (1998) 8. Bregler, C., Malik, J.: Tracking people with twists and exponential map. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 8–15 (1998) 9. Cham, T.J., Rehg, J.M.: A multiple hypothesis approach to figure tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 239–245 (1998) 10. Charikar, M., Chekuri, C., Feder, T., Motwani, R.: Incremental clustering and dynamic information retrieval. SIAM J. Comput. 33(6), 1417–1440 (2004) 11. Collins, R.T., Liu, Y., Leordeanu, M.: Online selection of discriminative tracking features. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1631–1643 (2005).
134
M.-H. Yang and J. Ho
12. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 564–577 (2003) 13. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005) 14. Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 126–133 (2000) 15. Dietterich, T.G., Lathrop, R.H., Perez, L.T.: Solving the multiple-instance problem with axis parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997) 16. Dollár, P., Tu, Z., Tao, H., Belongie, S.: Feature mining for image classification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007 17. Forsyth, D., Arikan, O., Ikemoto, L., O’Brien, J., Ramanan, D.: Computational Studies of Human Motion: Part 1, Tracking and Motion Synthesis. Now publishers, Hanover (2006) 18. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997) 19. Friedman, J.H.: Greedy function approximation: A gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001) 20. Ghahramani, Z., Beal, M.: Variational inference for Bayesian mixtures of factor analysers. In: Advances in Neural Information Processing Systems, pp. 449–455 (2000) 21. Golub, G.H., Van Loan, C.F.: Matrix Computations. The Johns Hopkins University Press, Baltimore (1996) 22. Grabner, H., Grabner, M., Bischof, H.: Real-time tracking via on-line boosting. In: Proceedings of British Machine Vision Conference, pp. 47–56 (2006) 23. Hager, G.D., Belhumeur, P.N.: Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Anal. Mach. Intell. 20(10), 1025–1039 (1998) 24. Hall, P., Marshall, D., Martin, R.: Adding and subtracting eigenspaces with eigenvalue decomposition and singular value decomposition. Image Vis. Comput. 20(13–14), 1009–1016 (2002) 25. Ho, J., Lee, K.-C., Yang, M.-H., Kriegman, D.: Visual tracking using learned linear subspaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 782–789 (2004) 26. Humphreys, G., Bruce, V.: Visual Cognition: Computational, Experimental and Neuropsychological Perspectives. Psychology Press, London (1989) 27. Ioffe, S., Forsyth, D.: Probabilistic methods for finding people. Int. J. Comput. Vis. 43(1), 45–68 (2001) 28. Isard, M., Blake, A.: CONDENSATION—conditional density propagation for visual tracking. Int. J. Comput. Vis. 29(1), 5–28 (1998) 29. Jain, A.K., Murty, M.N., Flynn, P.J.: Data clustering: A review. ACM Comput. Surv. 31(3), 264–323 (1999) 30. Jepson, A.D., Fleet, D.J., El-Maraghi, T.F.: Robust online appearance models for visual tracking. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1296–1311 (2003) 31. Lee, K.-C., Ho, J., Yang, M.-H., Kriegman, D.: Visual tracking and recognition using probabilistic appearance manifolds. Comput. Vis. Image Underst. 99(3), 303–331 (2005) 32. Levy, A., Lindenbaum, M.: Sequential Karhunen-Loeve basis extraction and its application to images. IEEE Trans. Image Process. 9(8), 1371–1374 (2000) 33. Li, R., Yang, M.-H., Sclaroff, S., Tian, T.-P.: Monocular tracking of 3D human motion with a coordinated mixture of factor analyzers. In: Proceedings of European Conference on Computer Vision, pp. 137–150 (2006) 34. Li, R., Tian, T.-P., Sclaroff, S., Yang, M.-H.: 3D human motion tracking with a coordinated mixture of factor analyzers. Int. J. Comput. Vis. 87(1–2), 170–190 (2010) 35. Lim, J., Ross, D., Lin, R.-S., Yang, M.-H.: Incremental learning for visual tracking. In: Advances in Neural Information Processing Systems, pp. 793–800. MIT Press, Cambridge (2005)
8 Toward Robust Online Visual Tracking
135
36. Lin, R.-S., Liu, C.-B., Yang, M.-H., Ahuja, N., Levinson, S.: Learning nonlinear manifolds from time series. In: Proceedings of European Conference on Computer Vision, pp. 239–250 (2004) 37. Lin, R.-S., Ross, D., Lim, J., Yang, M.-H.: Adaptive discriminative generative model and its applications. In: Advances in Neural Information Processing Systems, pp. 801–808. MIT Press, Cambridge (2005) 38. Mohan, A., Papageorgiou, C., Poggio, T.: Example-based object detection in images by components. IEEE Trans. Pattern Anal. Mach. Intell. 23(4), 349–361 (2001) 39. Moselund, T., Granum, E.: A survey of computer vision-based human motion capture. Comput. Vis. Image Underst. 81(3), 231–268 (2001) 40. Murase, H., Nayar, S.: Visual learning and recognition of 3d objects from appearance. Int. J. Comput. Vis. 14(1), 5–24 (1995) 41. Nejhum, S.M.S., Ho, J., Yang, M.-H.: Online articulate object tracking with appearance and shape. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2008 42. Oza, N.C.: Online Ensemble Learning. Ph.D. Thesis, University of California, Berkeley (2001) 43. Pentland, A., Moghaddam, B., Starner, T., Oligide, O., Turk, M.: View-based and modular eigenspaces for face recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 84–91 (1994) 44. Porikli, F.: Integral histogram: A fast way to extract histograms in Cartesian spaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 829– 836 (2005) 45. Ramanan, D., Forsyth, D.: Finding and tracking people from the bottom up. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 467–474 (2003) 46. Ronfard, R., Schmid, C., Triggs, B.: Learning to parse pictures of people. In: Proceedings of the Seventh European Conference on Computer Vision, pp. 700–714 (2002) 47. Ross, D., Lim, J., Yang, M.-H.: Adaptive probabilistic visual tracking with incremental subspace update. In: Proceedings of European Conference on Computer Vision, pp. 470–482 (2004) 48. Ross, D., Lim, J., Lin, R.-S., Yang, M.-H.: Incremental learning for robust visual tracking. Int. J. Comput. Vis. 77(1–3), 125–141 (2008) 49. Roweis, S., Saul, L., Hinton, G.E.: Global coordination of local linear models. In: Advances in Neural Information Processing Systems, pp. 889–896 (2001) 50. Rowley, H., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20(1), 23–38 (1998) 51. Schneiderman, H., Kanade, T.: Object detection using the statistics of parts. Int. J. Comput. Vis. 56(3), 151–177 (2004) 52. Sidenbladh, H., Black, M.: Learning image statistics for Bayesian tracking. In: Proceedings of IEEE International Conference on Computer Vision, pp. 709–716 (2001) 53. Sigal, L., Bhatia, S., Roth, S., Black, M., Isard, M.: Tracking loose-limbed people. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 421–428 (2004) 54. Sigal, L., Black, M.: HumanEva: synchronized video and motion capture dataset for evaluation of articulated human motion. Technical Report CS-06-08, Brown University (2006) 55. Sminchisescu, C., Triggs, B.: Covariance scaled sampling for monocular 3D body tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–454 (2001) 56. Sung, K.-K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20(1), 39–51 (1998) 57. Teh, Y.W., Roweis, S.: Automatic alignment of local representations. In: Advances in Neural Information Processing Systems, pp. 841–848 (2002) 58. Tian, T.-P., Li, R., Sclaroff, S.: Tracking human body pose on a learned smooth space. Technical Report 2005-029, Boston University (2005) 59. Toyama, K., Blake, A.: Probabilistic tracking with exemplars in a metric space. Int. J. Comput. Vis. 48(1), 9–19 (2002)
136
M.-H. Yang and J. Ho
60. Urtasun, R., Fleet, D., Hertzmann, A., Fua, P.: Priors for people tracking from small training sets. In: Proceedings of IEEE International Conference on Computer Vision, pp. 403–410 (2005) 61. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 511– 518 (2001) 62. Viola, P., Platt, J.C., Zhang, C.: Multiple instance boosting for object detection. In: Advances in Neural Information Processing Systems, pp. 1417–1426. MIT Press, Cambridge (2005) 63. Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM Comput. Surv. 38(4), 1–45 (2006)
Chapter 9
Modeling Patterns of Activity and Detecting Abnormal Events with Low-Level Co-occurrences Yannick Benezeth, Pierre-Marc Jodoin, and Venkatesh Saligrama
Abstract We explore in this chapter a location-based approach for behavior modeling and abnormality detection. In contrast to conventional object-based approaches for which objects are identified, classified, and tracked to locate objects with suspicious behavior, we proceed directly with event characterization and behavior modeling using low-level features. Our approach consists of two-phases. In the first phase, co-occurrence of activity between temporal sequences of motion labels are used to build a statistical model for normal behavior. This model of co-occurrence statistics is embedded within a co-occurrence matrix which accounts for spatio-temporal cooccurrence of activity. In the second phase, the co-occurrence matrix is used as a potential function in a Markov-Random Field framework to describe, as the video streams in, the probability of observing new volumes of activity. The co-occurrence matrix is thus used for detecting moving objects whose behavior differs from the ones observed during the training phase. Interestingly, the Markov-Random Field distribution implicitly accounts for speed, direction, as well as the average size of the objects without any higher-level intervention. Furthermore, when the spatiotemporal volume is large enough, the co-occurrence distribution contains the average normal path followed by moving objects. Our method has been tested on various outdoor videos representing various challenges.
Y. Benezeth () Orange Labs, France Telecom R&D, 4, rue du Clos Courtel, 35512 Cesson Sévigné, France e-mail:
[email protected] P.-M. Jodoin Université de Sherbrooke, 2500 bd. de l’Université Sherbrooke, Sherbrooke, J1K 2R1, Canada e-mail:
[email protected] V. Saligrama Boston University, Department of Electrical and Computer Engineering, 8 Saint Mary’s Street, Boston, MA 02215, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_9, © Springer-Verlag London Limited 2011
137
138
Y. Benezeth et al.
Keywords Video surveillance · Abnormality detection · Motion detection · MRF
1 Introduction In this paper, we present a low-level location-based approach for activity analysis and abnormal detection. In several traditional approaches (e.g. [5]), moving objects are first detected, analyzed and then tracked. Subsequently, behavior models are built based on object tracks and non-conformant ones are deemed abnormal. The main problem with this approach is that in case of complex environments, object extraction and tracking are performed directly on cluttered raw video or motion labels. We propose performing activity analysis and abnormal behavior detection first, followed possibly by object extraction and tracking. If the abnormal activity is reliably identified, then object extraction and tracking focus on region of interest (ROI) and thus is relatively straightforward, both in terms of difficulty and computational complexity, on account of sparsity and absence of clutter. A question arises: How to reliably identify abnormalities from a raw video? Some approaches have been proposed to perform such low-level abnormality detection (see for instance [1] and [6]). Nevertheless, we point out that these methods process each pixel independently and thus ignore spatial correlation across space and time. These correlations may not only be important in improving false alarms and misses but also in detecting abnormality of event sequences, such as a person in the act of dropping a baggage, tracking the person who dropped the baggage, a car making an illegal U-turn, etc. In our method, we account for these scenarios through spatio-temporal models. Although this model is simple, it results in extremely interesting results (see Figs. 3, 6 and 7). Note that our scheme does not rely on object tagging, tracking or classification. Furthermore, the co-occurrence can be readily generalized to higher-dimensions and other interesting features can be augmented.
2 Context, Overview and Notations 2.1 Context Motion labels obtained from background subtraction are atomic information often used by surveillance applications. A comparative study of background subtraction methods can be found in [2]. In this paper, we implemented a basic background subtraction method based on a Euclidean distance metric. Although many video analytics methods use it only in early stages of processing (mainly to locate moving objects) we argue that motion labels carries fundamental information on the content of the scene and thus, can be used to perform high-level tasks. Motivated by this perspective, some authors have already shown that low-level motion labels can be used to summarize videos [8], recognize human movements [4] and detect abnormalities [6]. In general, motion-label sequences provide valuable information for characterizing “usual behavior” observed at each pixel. For instance, consider patterns asso-
9 Modeling Patterns of Activity and Detecting Abnormal Events
139
Fig. 1 Binary signature for three pixels, two being highly correlated (A and B). The abscissa axis is w.r.t. time
ciated with random activity (shaking tree), regular activity (highway traffic), bursty activity (due to traffic light), or simply inactivity. All of these scenarios are characterized by patterns of motion-label sequences at the pixel level (or in general location). Consequently, abnormal behavior can be detected using low-level features whenever the observed pattern is unlikely under the normal activity model. In these cases, object identification and tracking can be circumvented for detecting abnormal behavior. However, the pure pixel-by-pixel approach is insufficient in applications where abnormality is manifested spatially as, for instance, cars running against traffic flow, cars making illegal U-turns, etc. Consequently, we need a strategy for incorporating spatial patterns in addition to the temporal patterns of motion-label sequences. The shortcomings of characterizing purely temporal behavior is further depicted in Fig. 1, which shows two pixels with identical signatures (except for a time-shift
140
Y. Benezeth et al.
arising from cars going from right to left). Normal/Abnormal behavior arising from the pattern of activity between the two pixels cannot obviously be captured through a purely pixel-by-pixel analysis. For instance, a burst of activity occurring at pixel A before pixel B would mean that a car now runs from left to right.
2.2 Overview and Notation The reader can follow the upcoming exposition through Fig. 2. Let Ix,k be the luminance (or color) of a video sequence sampled on a 2D lattice of size W0 × H0 at discrete times k, i.e., x ∈ W0 × H0 ⊂ R 2 , k ∈ Z + . To simplify notation, we use s to denote the pixel location x at time t. Xs is the motion label estimated through simple background subtraction where Xs ∈ {0, 1}, 0 and 1 denoting the “inactive” and “active” states. We also define the motion-label sequence centered at s = ( x , t) s . In x,t = [Xx,t−η , . . . , Xx,t+η ] where 2η + 1 is the length of the vector X as being X x,t is a one-dimensional binary sequence at pixel x and time t as shown in short, X Fig. 1. A contiguous sequence of ones denotes a busy period and is associated with a passing object while a sequence of zeros corresponds to an idle period of activity. The entire spatio-temporal sequence can be alternatively defined over a 3D lattice S of size W0 × H0 × T0 with s ∈ S being a point in the spatio-temporal space, Is being the corresponding luminance (or color) and Xs the corresponding motion label. Now lets consider for each pixel x at time t, a spatio-temporal neighborhood centered at s = ( x , t). This neighborhood is a 3D lattice Ms ⊂ S with size W × H × T , W < W0 , H < H0 and T T0 , centered on s ∈ S. Let us also consider a location r = ( y , τ ) ∈ Ms in the spatio-temporal neighborhood of s = ( x , t). The spatial neighborhood of a pixel x is the set of all pixels y such that s = ( x , t) and r = ( y, τ ) are both in Ms for all t. As we mentioned previously, whenever a moving object passes in front of x at time t, it leaves a spatio-temporal trace as some sites r = ( y , τ ) ∈ Ms co-occur with s = ( x , t). Interestingly, several moving objects exhibiting regular behavior (think of cars on a highway going in the same direction) leave, after a while, similar traces
Fig. 2 3D lattice S with spatio-temporal neighborhood Ms
9 Modeling Patterns of Activity and Detecting Abnormal Events
141
in the spatial neighborhood of Ms . Interestingly, the co-occurrence of two spatiotemporal neighbors s and r is not only due to the position and orientation of the camera in the scene, but also due to the shape, velocity and direction of the moving objects passing in front of a given spatial location x . In this context, the goal of the co-occurrence matrix is to estimate how frequently a site r co-occurs with s given a training video sequence exhibiting normal activity. We next define the notion of cooccurrence. A site r ∈ Ms co-occurs with s whenever their corresponding motion s and X r exhibit a similar signature. The similarity between motion vectors vector X at s and r is expressed using the mutual information defined as: PX s X r (x, y) r) = s, X PX s X r (x, y) · log sim(X , (1) PX s (x)PX r (y) x∈{0,1} y∈{0,1}
r (i) = y, x and y ∈ {0, 1}, i = [t − η, . . . , t + η], P (x, y) s (i) = x and X where X X s Xr r and P (x) and P (y) are s and X is the joint probability of discrete variables X Xs Xr the marginal probabilities. The mutual information is a useful tool for determining whether two motionlabel sequences contain the same activity. For example, a temporal sequence of motion labels containing random values due to noise of false detections (caused, say, by an unstable background) will have a low mutual information with almost any other sequence. On the other hand, two sequences containing the trace left by the same moving object will have a large mutual information. In this way, the mutual information criteria minimizes the influence of spurious false detections and noisy environments.
3 Our Method In this section, we present how, for a given site s, a co-occurrence matrix and the associated statistical model can be estimated from a training video sequence. Our statistical model is a Markov-Random Field (MRF) model that accounts for the likelihood of the co-occurrences. We later present how abnormal events can be detected and how low-level connected graphs can be used to follow relevant moving objects.
3.1 Training Phase Nominal Model Let Os denote a motion-label volume in the spatial neighborhood of location s, i.e. Os = (Xr : r ∈ Ms ). We are interested in modeling the likelihood of the normal observations, i.e., PN (Os ). We do this using an MRF model parameterized through co-occurrences: 1 u, X v) , αuv sim(X (2) PN (Os ) = exp Z u,v∈Ms
142
Y. Benezeth et al.
u, X v ) is the mutual information between motion-label vectors X u and where sim(X Xv . αuv is the co-occurrence potential between site u and v determined in a learning phase as will shortly be described (for the remainder of the paper, αuv will be refereed to as the co-occurrence matrix). Z is the usual partition function, which is a normalization constant to ensure that the right hand side sums to one.
Learning the Co-occurrence Matrix As mentioned previously, the co-occurrence matrix αuv accounts for how many times sites u and v co-occur during the training phase. Two sites are said to co-occur v exhibit a similar profile. In this paper, u and X whenever their motion signature X we measure the similarity between two sites based on their mutual information. The co-occurrence matrix αuv of two spatio-temporal locations, u, v ∈ Ms can be empirically computed as follows: αuv =
βuv T0 − T
T0 −T /2
u, X v ), sim(X
(3)
t=T /2
where T0 is the total number of frames in the training video sequence and βuv is a constant that can depend on distance between the locations u and v (in this paper we assume βuv = 1). Note that by definition, αuv does not depend on the time index t. Therefore, αuv = α(y1 ,t+τ1 ),(y2 ,t+τ2 ) = α(y1 ,τ1 ),(y2 ,τ2 ) .
(4)
A Specific Case for Co-occurrence Benezeth et al. [3] show that the co-occurrence between two sites s an r can be determined by considering motion-label values Xs and Xr instead of the motion s and X r . In this way, two sites co-occur whenever X s = X r = 1. label sequences X In this case αuv can be easily computed. However, this formulation is sensitive to noise and spurious false positives caused by unstable background. As can be seen in Fig. 7, accounting for plane co-occurrence between motion labels (third row) generates a large number of false positives and poor detection of true moving objects. This clearly shows how mutual information allows for essential co-occurrences, i.e. co-occurrences caused by real moving objects only.
Complexity Issues & Conditional Independence The main issue is the cost of computation of all of the edge potentials, since they are combinatorially many. In our practical implementations, we typically only consider a sparse number of well-separated locations for testing abnormalities. In many of our applications, abnormalities are typically associated with patterns of abnormal
9 Modeling Patterns of Activity and Detecting Abnormal Events
143
activity as opposed to inactivity. Motivated by this perspective, we make the following simplifying assumption: for any spatio-temporal neighborhood, Ms centered around s = ( x , t), the co-occurrences are conditionally independent given Xs is active (namely Xs = 1). It will become clear why this assumption is not meaningful when Xs = 0. In other words, given Xs the values realized at the spatio-temporal locations Xv and Xu are statistically independent. Alternatively, one may think of this assumption as an instantiation of a naive Bayes perspective, namely, we assume that the pairwise co-occurrences in the spatial neighborhood of a location s are all independent. Practically, this assumption implies that we must have, αuv = 0,
u = s, v = s.
(5)
In practice we have found this assumption does not severely degrade performance in our applications. Note that from a pure implementation perspective, the cooccurrence matrix [αuv ] is a 3D array with each component accounting for the number of times site u co-occur with v while translating Ms . In this way, (5) reduces the complexity of the method from (W × H × T )2 pairwise co-occurrences down to W × H × T .
3.2 Observation Phase Abnormal Model It is generally difficult to describe an abnormality model except to say that abnormality is anything that does not look normal. However, from a classification perspective it becomes necessary to make some implicit assumptions about abnormality. Several researchers implicitly assume that abnormal observations are uniformly distributed in the feature space [7]. In this paper, we consider an action to be suspicious when its occurrence is rare (or simply nonexistent) in the training sequence. We also assume that abnormal observations are independent and identically distributed across the different pixels. This assumption amounts to a multinomial distribution. For simplicity, let N0 = |Ms | be the total number of spatio-temporal locations and N1 the total number of co-occurring pixels, i.e., u, X s) f (X (6) N1 = u∈Ms
with
u, X s) = f (X
u, X s ) > τ, 1 if sim(X 0 otherwise;
(7)
then the probability distribution of observations under the abnormal distribution is given by N 1 p (1 − p)N0 , (8) PA (Os ) = pN1 (1 − p)N0 −N1 = 1−p u, X s ) = 1. where, p is the probability that f (X
144
Y. Benezeth et al.
Abnormality Detection Consider now a test video sequence S defined on a 3D lattice of size W0 ×H0 ×Ttest , x , t) in the test video, and its correa spatio-temporal neighborhood Ms with s = ( sponding motion-label observations Os . The goal now is to detect every time instant t ∈ [0, Ttest ] for which the observations Os has a low probability under nominal distribution in comparison to likelihood of abnormality. It is well-known that the likelihood ratio test (LRT) is the optimal test for deciding between the two hypothesis: nominal vs. abnormal. The likelihood ratio (Os ) is the ratio of the probability of observations under nominal and abnormal hypothesis, from (2), (5) and (8), it follows that (Os ) =
PN (Os ) PA (Os )
(1 − p)N0 s, X r) αsr sim(X = exp Z r∈Ms p r , X s) , − log f (X 1−p
(9)
r∈Ms
where, as before, N0 is the number of spatio-temporal locations and Z is a normalization constant. The likelihood ratio test is to decide between nominal and abnormal hypothesis based on a global threshold η: αsr sim(Xs , Xr ) − τ f (Xr , Xs ) (Os ) = exp r∈Ms
r∈Ms
nominal
> <
η,
(10)
abnormal
where τ = log(p/1 − p). Here we have absorbed Z, pN0 into η. A related test obtained by choosing η = 1 above reduces to a test for positivity or negativity of the argument of the exponential function. This reduces to the following simple test: nominal r∈Ms αsr sim(Xr , Xs ) > τ. (11) < f ( X , X ) r s r∈Ms abnormal
4 Experimental Results We present in this section some results obtained on various outdoor sequences representing different challenges. For each sequence, a co-occurrence matrix of size ranging between 130 × 70 × 300 and 210 × 210 × 150 have been used. The number of frames T used to estimate PN (see (2)) varies between 2000 and 7000 (i.e. from 1 and 4 minutes of video) depending on the sequence. Note that results are presented in thumbnails of Figs. 3, 6 and 7. The green moving objects are ones classified as
9 Modeling Patterns of Activity and Detecting Abnormal Events
145
Fig. 3 Example video in which cars following the regular traffic flow are tagged in green, while the car making an illegal U-turn has been picked up by our algorithm and tagged in red
Fig. 4 (a) Co-occurrence matrix of a regular traffic flow; (b) one car moving using the regular path; (c) the trace left by a car making an illegal U-turn
being normal and the red moving objects are those classified as being abnormal, i.e., whose trace is significantly different from the co-occurrence matrix equation (11). The first example (see Fig. 3) shows normal traffic and a car making an illegal U-turns. In Fig. 4(a), a co-occurrence matrix associated with a normal traffic flow is presented. As shown in Fig. 4(c), the trace left by the U-turn significantly differs from the usual traffic flow illustrated in the Fig. 4(b). Cars following the regular path are tagged in green and cars making an illegal U-turn are tagged in red. The second example shows a person dropping a baggage and abandoning it. In this video, pedestrians usually walk from left to right and from right to left, hence the X shape of the co-occurrence matrix (see Fig. 5(a)). When the person drops the bag, the abandoned package leaves a straight elongated line which differs from the cooccurrence matrix and thus causes this situation to be suspicious (see Fig. 5(b) and Fig. 6). Note that the likelihood ratio test (in (11)) is computed only when the key pixel is active. When considering Fig. 6, that is when the person passed through the key pixel, not before. This having been said, with a connected component analysis (in both space and time) we are able to tag the overall action as being “normal” or “abnormal”. That is the reason why the person dropping the bag is tagged in red during the overall action. The third example, in Fig. 7, shows how our method deals with noisy environments. The third row presents results obtained considering co-occurrences with mo-
146
Y. Benezeth et al.
Fig. 5 (a) Co-occurrence matrix of pedestrians walking from left to right and from right to left and (b) the trace left by a person dropping a bag
Fig. 6 Example video in which people walking are tagged in green while the person dropping a bag is tagged in red
tion labels [3] while the fourth row presents results obtained considering motionlabel vectors and mutual information. Clearly, the use of mutual information reduce the sensitivity to noise as the boat is clearly detected. The spatio-temporal trace left by the boat is shown in Fig. 8.
5 Conclusion We propose in this chapter a method to perform behavior modeling and abnormality detection based on low-level characteristics. We use the spatial and temporal dependencies between motion-label vectors obtained with simple background subtraction. To do this, we built an MRF model parameterized by a co-occurrence matrix. Although simple, this matrix contains the average behavior observed in a training sequence. It also implicitly contains information about direction, speed and size of objects usually passing through one (or more) key pixel(s). Equipped with
9 Modeling Patterns of Activity and Detecting Abnormal Events
147
Fig. 7 Illustration of detection in a challenging environment. The first row present the input images sequence, the second present the result of a background subtraction detection followed by the detection of abnormal activities (the boat displacement) considering co-occurrences with motion labels in the third row [3] or considering motion-label vectors in the fourth row
Fig. 8 Trace left by the displacement of the boat detected using (7)
148
Y. Benezeth et al.
the co-occurrence matrix, we can detect abnormal events by detecting traces which significantly differ from our nominal model following a likelihood ratio test. The main advantages of our method are threefold. First, in contrast to conventional object-based approaches for which objects are identified, classified and tracked to locate those with suspicious behavior, we proceed directly with event characterization and behavior modeling using low-level characteristics and thus avoid the risk of errors propagation (e.g. due to the tracking algorithm limits in complex environments). Second, our method does not require any a priori knowledge about the abnormal event detection. We learn the usual behavior of moving objects in a scene and detect activity which significantly differ from usual ones. Third, our method is robust to noise and can detect unusual activities using very noisy background subtraction masks.
References 1. Adam, A., Rivlin, E., Shimshoni, I., Reinitz, D.: Robust real-time unusual event detection using multiple fixed-location monitors. Trans. Pattern Anal. Mach. Intell. 30(3), 555–560 (2008) 2. Benezeth, Y., Jodoin, P.-M., Emile, B., Laurent, H., Rosenberger, C.: Review and evaluation of commonly-implemented background subtraction algorithms. In: International Conference on Pattern Recognition (ICPR) (2008) 3. Benezeth, Y., Jodoin, P.-M., Saligrama, V., Rosenberger, C.: Abnormal events detection based on spatio-temporal co-occurrences. In: International Conference on Computer Vision and Pattern Recognition, pp. 2458–2465 (2009) 4. Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. Trans. Pattern Anal. Mach. Intell. 23(3), 257–267 (2001) 5. Hu, W., Tab, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. Trans. Syst. Man Cybern. Part C, Appl. Rev. 34(3), 334–352 (2004) 6. Jodoin, P.-M., Konrad, J., Saligrama, V.: Modeling background activity for behavior subtraction. In: International Conference on Distributed Smart Cameras 7. Polonik, W.: Minimum volume sets and generalized quantile processes. Stoch. Process. Appl. 69, 1–24 (1997) 8. Pritch, Y., Rav-Acha, A., Peleg, S.: Non-chronological video synopsis and indexing. Trans. Pattern Anal. Mach. Intell. 30(11), 1971–1984 (2008)
Chapter 10
Use of Context in Video Processing Chen Wu and Hamid Aghajan
Abstract Interpreting an event or a scene based on visual data often requires additional contextual information. Contextual information may be obtained from different sources. In this chapter, we discuss two broad categories of contextual sources: environmental context and user-centric context. Environmental context refers to information derived from domain knowledge or from concurrently sensed effects in the area of operation. User-centric context refers to information obtained and accumulated from the user. Both types of context can include static or dynamic contextual elements. Examples from a smart home environment are presented to illustrate how different types of contextual data can be applied to aid the decision-making process. Keywords Contextual information · User-centric context · Environmental context · Activity recognition
1 Introduction While offering access to rich information from the environment, computer vision often remains challenged in the face of requirements for accurate and meaningful content interpretation in practical systems. This is partially due to the complexities associated with the choice of visual features. These features are often subject to becoming unavailable or irrelevant as conditions vary. Another source of challenge is the presence of different possible interpretations from the observations acquired in limited temporal or spatial spans. C. Wu () · H. Aghajan Department of Electrical Engineering, Stanford University, Stanford, CA, USA e-mail:
[email protected] H. Aghajan e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_10, © Springer-Verlag London Limited 2011
149
150
C. Wu and H. Aghajan
Fig. 1 Relationship between different context types
Despite the availability of many possible sources of contextual information, especially when working with video sequences or in multi-camera networks, the use of context in video processing is an area in its infancy. In most video-based applications, contextual information may be available from a plethora of vision-based or non-vision-based sources. Contextual data can be employed across the various processing layers from early vision to event interpretation in order to achieve better overall performance in the presence of uncertainties. Generally speaking, the types of context in an interactive system can be loosely categorized as environmental context and user-centric context. Environmental context refers to information derived from domain knowledge or from concurrently sensed effects in the area of operation. User-centric context refers to the elements of information obtained and accumulated from the user. Both types of context can include static or dynamic contextual elements. Relationship between the above four context types is illustrated in Fig. 1. The figure also lists some examples for environmental and user-centric contexts. For example, environmental context can be in the form of application-specific domain knowledge which could be a static context, or in the form of dynamic context such as spatiotemporal persistence of features which enforces a logical link between observations over a period of time or between multiple camera views. Henricksen et al. [3] summarize some characterizations of context in pervasive computing, where contextual information is provided by sensor data. User-centric context can be available in the form of static context such as a set of preferences explicitly entered by the user into the system, or in the form of dynamic context such as the activity that the user is observed in during real-time operation. In a multi-camera network, different cameras can serve to provide contextual information to each other, thus helping the network to operate more efficiently and more effectively. A particular source of context in multi-camera video networks is the model for camera priors, which provides clues on the value or confidence of each camera’s observations based on the consistency and value of past observations. Geometric constraints and geographic information are other types of environmental context. Logic and high-level reasoning in turn can offer contextual information to the vision processing module through accumulation of a knowledge base from past observations, or in the form of various behavior models associated with the user, the camera network, or the observed environment.
10
Use of Context in Video Processing
151
Context awareness has been widely studied in ubiquitous computing as mobile devices and distributed sensors naturally provide rich contextual information for ubiquitous computing problems. A survey on approaches to modeling context can be found in [10], including description of key-value models, markup scheme models, graphical models, object-oriented models, logic-based models, and ontologybased models. The authors discuss strengths and weaknesses of each model in the end. RFID sensor data are used to sense interactions of the user with objects at home in [15], where a dynamic Bayesian network is used to model common sense activity descriptions in terms of object usage such that the user activity can be inferred from object usage. In [7], the authors adopt a variety of sensor data such as PC usage, phone usage, PDA location, etc., to infer the user’s availability for a spontaneous meeting. A large schema is created to model dependencies between different sensor data and user status. The user’s intention and preference has also been used as context in [16] to help with navigation. In computer vision, use of context has mainly focused on contextual information provided within the images. In recent years research on using context for object recognition in images has seen much progress. Rabinovich and Belongie [9] summarizes two approaches to using context in image-based object recognition, i.e., scene-based context and object-based context. A representative work on scenebased context can be found in [11]. The approach is based on the observation that the background of an object in a scene normally has certain structures. Contextual priors are learned from features that represent the holistic scene, which include object priming, focus of attention, and scale selection. In [1] a system of object recognition based on object co-occurrence, location and appearance is presented. The image is first segmented into multiple stable segmentations. Each segment is assigned to object labels with confidence. A context model with object location and co-occurrence counts is learned. The final assignment of object labels to segments maximizes probability of labels to the segments’ content as well as contextual agreement with other segments. In [8], objects in the office space are segmented from user activities, where the high-level activity is used as a context for low-level action recognition, and interaction signatures are used as context for object labeling. Recently, contextual information has also been applied in human activity analysis [2] by incorporating object detection information (e.g., presence/absence, counts and distance) in the bag of detectors. The authors applied a Bayesian classification method based on Gaussian processes with multiple kernel covariance functions to automatically select and assign weights to multiple features. They show that for activities that involve objects (“GetOutCar”) and multiple persons (“HandShake”) object-level information may be potentially more discriminative than standard image features.
2 Case Study: Environment Discovery In this section, we present a case study in environment discovery for smart homes to illustrate the use of a few types of contextual data in a multi-camera video processing setting. Multiple cameras are installed in a home environment (Fig. 2) and the
152
C. Wu and H. Aghajan
Fig. 2 Philips HomeLab’s floor plan and camera views
goal is to automatically discover objects regularly used in the home. Appearancebased object recognition is challenging in our problem. Many objects such as chairs may look very different in appearance. The cameras have different views to the objects, and the environment is cluttered. Our approach employs observations of the user’s interactions with the objects as contextual information to recognize the objects through their function. The user’s location and height are measured by data fusion between the cameras. Pose and activity of the user are inferred by a conditional random field (CRF) process from the detected height and aspect ratio of the user’s bounding box as features. Activity and location information are then input to a knowledge base of weighted first-order logic to infer the identity of the object that the user interacts with. More details of the system setup and activity recognition method can be found in [13].
2.1 Environmental Context Sensors can be a source of dynamic environmental context. Given our multi-camera setup, geometric constraints from camera parameters and camera priors in activity recognition are the two sources of context. Camera parameters are used to remove false foreground detection results through validating the user’s location and height based on geometric constraints of the environment. Camera priors provide a contextual basis for arbitrating between the activity declarations by different cameras. Another type of environmental context comes from domain knowledge related to the application, which is the relations between user activity and objects in this problem. In this section we describe these latter two sources of context.
10
Use of Context in Video Processing
153
Camera Priors in Activity Recognition In a multi-camera network, the camera views often overlap and hence decisions about events can be made by combining data from multiple views. For many tasks such as tracking, it is helpful to incorporate such information related to camera views as constraints to the problem. Javed et al. in [4] model camera relationships for tracking in non-overlapping views. Spatiotemporal constraints including locations of exits and entrances in an environment, direction of movement and average time taken to travel between the cameras are learned. Brightness transfer functions are also learned between cameras to match objects in different cameras. The space-time constraints and appearance transformation are example representations of camera priors. In our application, the user’s activity is determined by fusing activity recognition decisions from different cameras. Given the conditions each camera may have, this decision fusion needs to consider heterogeneity across cameras. The frequency of each activity type in a camera’s field of view and the viewing angle to the activity observed over time provide helpful prior information for assigning a confidence level to new activity reports. One way to model how “well” a camera detects a certain activity is to use P (A|E) where A is the ground truth and E is the estimate. Let Cobsv represent the set of indices of cameras that observe the user and have his activity estimated as E c (c ∈ Cobsv ) at time t. The global decision on the activity A∗ can be chosen such that A∗ = argmax P Acj |Eic P Eic , (1) Aj ,j =1,2,3 c∈C
obsv
i=1,2,3
where P (Eic ) is the probability of activity i (standing/sitting/lying) in camera c obtained from the CRF analysis. In Fig. 2, only cameras 2 and 5 observe the lying activity. Table 1 shows activity classification performance of cameras 2 and 5. Differences can be seen in the number of true positives and misclassifications for each category. Camera 2 never estimates lying correctly, because the dining table and chairs block the camera’s view to the user. Comparatively, camera 5 has a much higher precision in detecting lying. The global decision maker takes into account such priors to estimate the activity. Table 1 Activity classification results for cameras 2 and 5. Rows are ground truth and columns are estimates (a) camera 2 Standing Standing
(b) camera 5 Sitting
Lying
2582
167
0
Sitting
449
1639
Lying
13
44
Standing
Sitting
Lying 0
Standing
697
127
59
Sitting
740
1566
77
0
Lying
71
183
132
154
C. Wu and H. Aghajan
Object Recognition Through User Activities Our approach aims to detect and label objects in a given environment solely based on their interactions with users. This offers a complementary method to appearancebased object recognition. Moore et al. [6] propose an approach to use the context information of certain objects in the scene to recognize human actions. When the hands touch certain objects, this information is used to suggest a hidden Markov model to classify the action. They also show that once the action is learned, it can be used to detect unknown objects using a naive Bayesian classifier. Veloso et al. [12] propose an algorithm named FOCUS to recognize objects based on use and structure. It defines the affordance property of objects with prior knowledge. The affordance property is suggested by human activities and used to detect the object. In [8] image region patches are classified into object categories in an office room also by observing human interactions. In our approach, the contextual information to detect objects, i.e., their interactions with the user, is modeled as domain knowledge in a Markov logic network which is able to handle complex relationships. We define three types of object behaviors in terms of their relational structure with user activities or proximity between multiple objects (Fig. 3): • Direct actions: For example, in Fig. 3(a) the sitting and lying activities of the user hint about the likelihood of a sofa at the observed location. Or the appearing and disappearing places of the user are likely to be doors. • Sequential actions: This behavior refers to temporal relationship between subsequent user activities. When the user actions are related in time, so are the objects he interacts with in such actions. For example, in Fig. 3(b), if the observation is that in the kitchen the user first takes out something from a place and then puts that into a nearby place for a while, and then takes it to a sitting place to eat, a likely inference is that the first position may imply a fridge while the second may imply a microwave and the third a dining table. • Concurrent actions: This behavior refers to spatial relationship between user activities and corresponding objects. Sometimes the user interacts with certain ob-
Fig. 3 The three types of object behaviors. (a) Immediate relationship; (b) temporal relationship; (c) spatial relationship
10
Use of Context in Video Processing
155
jects from a distance, for example gazing at a screen. The activity features such as the attention region of interest can provide clues to infer the object’s existence and location. For example, if the user is observed staring at a fixed direction at different times of the day, a likelihood of a TV at that location can be inferred (Fig. 3(c)). Common location relations between objects in the environment can also add to our knowledge base, e.g., a coffee table is usually in the living room with sofas or chairs surrounding it. Details of problem formulation based on employing a Markov Logic Network (MLN) to infer object types and experimental results can be found in [13, 14]. The room area is divided into grids. For each grid, its object type is inferred based on the user activity observed at the grid. This inference is based on the knowledge base as follows: w1
walking => floor
(2)
w2
sitting => chair
(3)
w3
sitting => sofa
(4)
w4
lying => sofa
(5)
w5
watching => TV
(6)
Each rule in the knowledge base is written in a first-order logic formula to construct the MLN model. We define the predicates Hasact(t, act), Hastype(t, obj) and After(t2 , t1 ). Hasact(t, act) = 1 means at time t activity act is observed at the grid, while Hastype(t, obj) = 1 indicates the object type of the grid is obj at time t. After(t2 , t1 ) means t2 is after t1 . Therefore, (3) is formulated as w1
Hasact(t2 , walking) ∧ Hastype(t1 , NK) ∧ After(t2 , t1 ) ⇒ Hastype(t2 , floor),
(7)
where NK means not-known. The other formulas can be written in a similar way. wi are the weights associated with each formula. The bigger the wi , the more confident we are on the relationship represented by the associated formula. Some examples are given in Fig. 4, which shows how object type probability changes given activity observations. “No activity” means there is no activity observed at the grid, which happens either when the user is not present, or when the user’s activity does not affect that grid. The probability of an object P (obj) equals P (Hastype(t, obj) = 1). We use P (obj) for short. In Fig. 4(a) there are Hasact(ti , sitting) in the middle of the sequence, and Hasact(ti , NA) otherwise. P (NK) drops gradually, and P (floor) remains 0, while P (chair) and P (sofa) increase at the same pace as a result of the sitting actions. In Fig. 4(b) the sequence of observations are Hasact(ti , walking), Hasact(ti , sitting), Hasact(ti , lying). The results of MC-SAT shows that before frame 16 P (floor) rises, then P (chair) and P (sofa) rise with the same speed (since only the sitting action is observed), and finally the lying activity increases P (sofa) and decreases all others. Figure 4(c) illustrates the effect of temporal sequence of Hasact(ti , watching) versus other activities on P (TV). In (c), Hasact(ti , watching) at the beginning increases P (TV). But later
156
C. Wu and H. Aghajan
Fig. 4 Activity observations through time and the corresponding inferred object type probability. The first row of each sub-figure shows different activities in colors (the activities are also marked on the plotted bars). Probability of object types from inference is shown in the second row with different line patterns (triangle: floor; circle: chair; plus sign: sofa; cross: TV; square: NK)
10
Use of Context in Video Processing
157
Hasact(ti , walking) indicates that although the grid is within the gaze area, it actually belongs to the floor area. In (d), the grid is identified to be the floor area at the beginning. Therefore, even when it is later observed to be in the gaze area, P (TV) remains zero. With the activity observations, the object layout of the room can be created by inferring the object type of each grid. Performance of activity analysis, object recognition, and comparison with an appearance-based object detection method can be found in [14].
2.2 User-Based Context A source of user context is the preferences stated over time by the user, which are either explicitly entered by the user via an interface device, or are learned by the system through observing user activities and reactions to events or services. In a more general term, the system can obtain and accumulate a knowledge base through observing the user’s behavior or via purposeful interaction with the user. In the problem of object recognition discussed in the previous section, a form of user-based context can be interacting with the user to receive an input verifying a hypothesis about the presence of an object in a specific location in the environment. A motivation for such a query with the user may be that the domain knowledge encoded into the system in the form of rules linking activities with object types may not reflect the user’s pattern of using some objects. Interacting with the user enables the system to adapt its domain knowledge to the specific user according to the user context.
Adapting Domain Knowledge from User Feedback If the user provides feedback to the system in the form of specifying a location in the environment to be occupied by an object (such as a chair or a sofa), the system can convert the input into a probability P (obj) = 1. While the system may simply assign the user-specified object type to the location, an alternative approach would be to adjust the weights of the formulas in the knowledge base so that the inferred object probability is as close as possible to the user input given the previous observations. In this way, the knowledge base is also adapted to the user context. This problem can be seen as a discriminative learning problem: min F (x, y) = − log Pw (Y |X) + αβw − w0 1 .
(8)
The second term in (8) denotes regularization. w0 contains the current weights, and we do not want the weights to change drastically. α denotes how much we want to preserve the previous model. If α is small, the optimization tends to use the feedback more; otherwise, when α is large, w tends to remain unchanged. β denotes the extent to which we trust the user’s feedback. A smaller β means we trust the feedback more.
158
C. Wu and H. Aghajan
Fig. 5 The three scenarios for eating. The user may eat at the dining table (top), at the coffee table while watching TV (middle), or at the desk while reading (bottom). With the default knowledge base, the latter two cases are identified as dining table. After they are corrected from user feedback, the knowledge base is adapted accordingly
This is discriminative learning since we try to maximize P (Y |X). X is the set of all evidence predicates. Y is the set of query predicates which we know their true values. However, there are some query predicates as well which we do not know their true values. For example, we know the object types at the beginning and at the end (from user feedback), but we do not know about them in between. So P (Y |X) is a marginal distribution summing over all unobserved predicates Y : Pw (Y = y|X = x) = Pw (Y = y, Y = y |X = x). (9) y
With the hidden variables Y , F is not convex anymore. But we can still use the gradient and find a local minimum with gradient descent. Optimization uses preconditioned scaled conjugate gradient described in [5]. Here is an example showing how weights are adapted given the feedback and regularization. Depending on the user’s habit and preference, there can be three situations about where the person eats (Fig. 5). The user may eat at the dining table, or eat at the coffee table while watching TV, or at the desk while reading books. The following rules describe these situations: w1
eating => dining table
(10)
w2
eating => coffee table
(11)
w3
eating => desk
(12)
Weights of the rules are listed at the beginning of the lines. Here the weight wi and w probability pi can be interchanged via pi = 3e i wj . Initially, the probability of j =1 e
10
Use of Context in Video Processing
159
Fig. 6 Probability of each of the three formulas being true versus different α values. When α is small, feedback has more effect on the formula’s likelihood
the three rules are set to 0.8, 0.1, and 0.1. If eating action is observed, the location is more likely to be a dining table rather than coffee table or desk. Suppose now the user indicates that this location is actually a desk. This feedback is applied to re-estimate the weights again according to the objective function in (8). Assume the user is trusted so that β = 1. Then the learned weights and inference based on the new weights will be dependent on α. Figure 6 shows how the probabilities of the rules change with α. When α = 0.1, probability of eating => desk increases to 0.6 while probability of eating => dining table drops to about 0.3. As expected, the influence of feedback is smaller when α is bigger.
3 Conclusion In this chapter we discussed the types of contextual information that are helpful and sometimes crucial in video processing. Context has been widely and effectively used in areas such as pervasive computing, while work in vision research has mainly focused on using context within one image. We group context for a vision-based application into two types, environmental context and user-centric context. The context may be static or dynamic. It may come from images, videos, other sensors, prior domain knowledge, or user input. Through a case study in a smart home environment, several examples were presented to illustrate how to acquire and employ contextual information in a multi-camera system.
160
C. Wu and H. Aghajan
References 1. Galleguillos, C., Rabinovich, A., Belongie, S.: Object categorization using co-occurrence, location and appearance. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008) 2. Han, D., Bo, L., Sminchisescu, C.: Action recognition in context. In: IEEE International Conference on Computer Vision, 2009 3. Henricksen, K., Indulska, J., Rakotonirainy, A.: Modeling context information in pervasive computing systems. In: Proceedings of the First International Conference on Pervasive Computing, pp. 167–180 (2002) 4. Javed, O., Shafique, K., Rasheed, Z., Shah, M.: Modeling inter-camera space-time and appearance relationships for tracking across non-overlapping views. Comput. Vis. Image Underst. 109(2), 146–162 (2008) 5. Lowd, D., Domingos, P.: Efficient weight learning for Markov logic networks. In: Proceedings of the 11th European Conference on Principles and Practice of Knowledge Discovery in Databases, pp. 200–211 (2007) 6. Moore, D.J., Essa, I.A., Hayes, M.H.: Exploiting human actions and object context for recognition tasks. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 80–86 (1999) 7. Mühlenbrock, M., Brdiczka, O., Snowdon, D., Meunier, J.L.: Learning to detect user activity and availability from a variety of sensor data. In: Proceedings of the Second IEEE International Conference on Pervasive Computing and Communications, p. 13 (2004) 8. Peursum, P., West, G., Venkatesh, S.: Combining image regions and human activity for indirect object recognition in indoor wide-angle views. In: Proceedings of the Tenth IEEE International Conference on Computer Vision, vol. 1, pp. 82–89 (2005) 9. Rabinovich, A., Belongie, S.: Scenes vs. objects: A comparative study of two approaches to context based recognition. In: First International Workshop on Visual Scene Understanding at IEEE Conference on Computer Vision and Pattern Recognition, pp. 92–99 (2009) 10. Strang, T., Linnhoff-Popien, C.: A context modeling survey. In: Workshop on Advanced Context Modelling, Reasoning and Management, the Sixth International Conference on Ubiquitous Computing, 2004 11. Torralba, A.: Contextual priming for object detection. Int. J. Comput. Vis. 53(2), 169–191 (2003) 12. Veloso, M., Hundelshausen, F., Rybski, P.E.: Learning visual object definitions by observing human activities. In: The Proceedings of IEEE-RAS International Conference on Humanoid Robots, 2005 13. Wu, C., Aghajan, H.: Using context with statistical relational models—object recognition from observing user activity in home environment. In: Workshop on Use of Context in Vision Processing at ICMI-MLMI, 2009 14. Wu, C., Aghajan, H.: User-centric environment discovery with camera networks in smart homes. IEEE Trans. Syst. Man Cybern. Part A PP(99), 1–9 (2010) 15. Wu, J., Osuntogun, A., Choudhury, T., Philipose, M., Rehg, J.M.: A scalable approach to activity recognition based on object use. In: IEEE 11th International Conference on Computer Vision, pp. 1–8 (2007) 16. Ziebart, B.D., Maas, A.L., Dey, A.K., Bagnell, J.A.: Navigate like a cabbie: Probabilistic reasoning from observed context-aware behavior. In: Proceedings of the 10th International Conference on Ubiquitous Computing, pp. 322–331 (2008)
Part III
Simulation, Graphics, Cognition and Video Networks
Chapter 11
Virtual Vision Virtual Reality Subserving Computer Vision Research for Camera Sensor Networks Demetri Terzopoulos and Faisal Z. Qureshi
Abstract Computer vision and sensor networks researchers are increasingly motivated to investigate complex multi-camera sensing and control issues that arise in the automatic visual surveillance of extensive, highly populated public spaces such as airports and train stations. However, they often encounter serious impediments to deploying and experimenting with large-scale physical camera networks in such real-world environments. We propose an alternative approach called “Virtual Vision”, which facilitates this type of research through the virtual reality simulation of populated urban spaces, camera sensor networks, and computer vision on commodity computers. We demonstrate the usefulness of our approach by developing two highly automated surveillance systems comprising passive and active pan/tilt/zoom cameras that are deployed in a virtual train station environment populated by autonomous, lifelike virtual pedestrians. The easily reconfigurable virtual cameras distributed in this environment generate synthetic video feeds that emulate those acquired by real surveillance cameras monitoring public spaces. The novel multi-camera control strategies that we describe enable the cameras to collaborate in persistently observing pedestrians of interest and in acquiring close-up videos of pedestrians in designated areas.
Keywords Smart cameras · Camera networks · Sensor networks · Computer vision · Visual surveillance · Persistent human observation · Virtual reality
D. Terzopoulos () Computer Science Department, University of California, Los Angeles, USA e-mail:
[email protected] F.Z. Qureshi Faculty of Science, University of Ontario Institute of Technology, Oshawa, Canada e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_11, © Springer-Verlag London Limited 2011
163
164
D. Terzopoulos and F.Z. Qureshi
1 Introduction Future visual sensor networks will rely on smart cameras, which are self-contained vision systems, complete with increasingly sophisticated image sensors, on-board processing and storage capabilities, power, and (wireless) communication interfaces. This opens up new opportunities to develop sensor networks capable of visually surveilling extensive public spaces, disaster zones, battlefields, and even entire ecosystems. These multi-camera systems lie at the intersection of Computer Vision and Sensor Networks and they pose challenging technical problems to researchers in both fields. In particular, as the size of a camera sensor network grows, it becomes infeasible for human operators to monitor the multiple video streams and identify all events of possible interest, or even to control individual cameras directly in order to maintain persistent surveillance. Therefore, it is desirable to design intelligent camera sensor networks that are capable of performing advanced visual surveillance tasks autonomously, or at least with minimal human intervention. Unfortunately, deploying a large-scale physical surveillance system is a major undertaking whose cost can easily be prohibitive for most computer vision and sensor network researchers interested in experimenting with multi-camera systems. As a means of overcoming this barrier to entry, as well as to avoid privacy laws that restrict the monitoring of people in public spaces, we have introduced the Virtual Vision paradigm for fostering research in surveillance systems [6, 17]. Thus far, we have pursued our unique approach using a dynamic virtual environment populated by autonomously self-animating, lifelike virtual pedestrians [7, 8]. Cost and legal impediments aside, we have also found that virtual vision facilitates the scientific method by offering significant advantages during the surveillance system design and evaluation cycle. Within the virtual vision framework, we review in this chapter surveillance systems comprising smart cameras that provide perceptive coverage of a large virtual public space—in our case, a reconstruction of New York City’s original Pennsylvania Station, which was demolished in 1963, populated by autonomously selfanimating virtual pedestrians (Fig. 1). Virtual passive and active cameras situated throughout the expansive chambers of the train station generate multiple synthetic video feeds that emulate those generated by real surveillance cameras monitoring public spaces (Fig. 2). The advanced pedestrian animation system combines behavioral, perceptual, and cognitive human simulation algorithms [14]. The simulator can efficiently synthesize well over 1000 self-animating pedestrians performing a rich variety of activities in the large-scale indoor urban environment. Like real humans, the synthetic pedestrians are fully autonomous. They perceive the virtual environment around them, analyze environmental situations, make decisions, and behave naturally within the train station. They can enter the station, avoiding collisions when proceeding though portals and congested areas, queue in lines as necessary, purchase train tickets at the ticket booths in the main waiting room, sit on benches when they feel tired, purchase food/drinks from vending machines when they feel hungry/thirsty, etc., and proceed from the concourse area down the stairs to the train platforms if they wish to board a train. A graphics pipeline renders the busy urban scene with considerable geometric and photometric detail, as shown in Fig. 1.
11
Virtual Vision
165
Fig. 1 A large-scale virtual train station populated by self-animating virtual humans [14]
Espousing our virtual vision paradigm, we have developed novel camera control strategies that enable simulated camera nodes to collaborate both in tracking pedestrians of interest that move across the fields of view (FOVs) of different cameras and in capturing close-up videos of pedestrians as they travel through a designated area. These virtual camera networks demonstrate the advantages of the virtual vision paradigm in designing, experimenting with, and evaluating prototype largescale surveillance systems. Specifically, we have studied control and collaboration problems that arise in camera networks by deploying simulated networks within the virtual train station. These simulated networks have performance characteristics similar to those of physical camera networks; e.g., latency, limited bandwidth, communication errors, camera failures, compression artifacts, etc.
2 The Case for Virtual Vision In virtual vision we combine computer vision and advanced graphics technologies to facilitate our development of camera network control algorithms, through the deployment of virtual networks in simulated environments (Fig. 3). This enables us
166
D. Terzopoulos and F.Z. Qureshi
Fig. 2 Plan view of the (roofless) virtual Penn Station environment, revealing the concourses and train tracks (left), the main waiting room (center), and the shopping arcade (right). (The yellow rectangles indicate pedestrian portals.) An example camera network is illustrated, comprising 16 simulated active (pan-tilt-zoom) video surveillance cameras. Synthetic images from cameras 1, 7, and 9 (from [14])
Fig. 3 The virtual vision paradigm
to investigate high-level control problems, such as camera assignment and handoff, which frequently arise in networks comprising smart cameras under realistic conditions. Virtual vision offers several advantages:
11
Virtual Vision
167
• The virtual vision simulator runs on (high-end) commodity PCs, obviating the need to grapple with special-purpose hardware.1 • The virtual cameras are very easily relocated and reconfigured in the virtual environment. • The virtual world provides readily accessible ground-truth data for the purposes of surveillance algorithm/system validation. • Experiments are perfectly repeatable in the virtual world, so we can easily modify algorithms and their various parameters and immediately determine the effect. • Our simulated camera networks run on-line in “real time” within the virtual world, with the virtual cameras actively controlled by the vision algorithms. By suitably prolonging virtual-world time relative to real-world time, we can evaluate the competence of computationally expensive algorithms, thereby gauging the potential payoff of accelerating them through more efficient software and/or dedicated hardware implementations. An important issue in camera network research is the comparison of camera control algorithms. Simple video capture suffices for gathering benchmark data from time-shared physical networks of passive, fixed cameras, but gathering benchmark data for networks that include any smart, active PTZ cameras requires scene reenactment for every experimental run, which is almost always infeasible when many human subjects are involved. Costello et al. [4], who compared various schemes for scheduling an active camera to observe pedestrians, ran into this hurdle and resorted to Monte Carlo simulation to evaluate camera scheduling approaches. They concluded that evaluating scheduling policies on a physical testbed comprising even a single active camera is extremely problematic. By offering convenient and limitless repeatability, our virtual vision approach provides a vital alternative to physical active camera networks for experimental purposes. Nevertheless, skeptics may argue that virtual vision relies on simulated data, which can lead to inaccurate results. Fretting that virtual video lacks all the subtleties of real video, some may cling to the dogma that it is impossible to develop a working machine vision system using simulated video. However, our high-level camera control routines do not directly process any raw video. Instead, these routines are realistically driven by data supplied by low-level recognition and tracking routines that mimic the performance of a state-of-the-art pedestrian localization and tracking system, including its limitations and failure modes. This enables us to develop and evaluate camera network control algorithms under realistic simulated conditions consistent with physical camera networks. We believe that the fidelity of 1 With
regard to software, a virtual vision simulator consists of an environmental model, character models, an animation engine, and a rendering engine. Most commercial modeling/animation systems enable users to create 3D virtual scenes, including virtual buildings populated by virtual characters, and they incorporate rendering subsystems to illuminate and visualize the scenes. The animation subsystem can animate the virtual characters, but autonomous pedestrian animation is an area of active research in the computer animation community and there are as yet no adequate commercial solutions.
168
D. Terzopoulos and F.Z. Qureshi
our virtual vision emulator is such that algorithms developed through its use will readily port to the real world.2
3 Related Work Preceding virtual vision, a closely related software-based approach to facilitating active vision research was proposed, called animat vision [18], which prescribed eschewing the hardware robots that are typically used by computer vision researchers in favor of biomimetic artificial animals (animats) situated in physics-based virtual worlds. Salgian and Ballard describe another early use of virtual reality simulation, which employed synthetic video imagery as seen from the driver’s position of a simulated car cruising the streets of a virtual town [12], in order to develop a suite of visual routines running in a real-time image processor to implement an autonomous driving system. Rabie and Terzopoulos demonstrated their animat vision approach by implementing biomimetic active vision systems for artificial fishes and for virtual humans [10]. Their active vision systems comprised algorithms that integrate motion, stereo, and color analysis to support robust color object tracking, vision-guided navigation, visual perception, and obstacle recognition and avoidance abilities. Together, these algorithms enabled the artificial animal to sense, understand, and interact with its dynamic virtual environment. The animat vision approach appeared to be particularly useful for modeling and ultimately reverse-engineering the powerful vision systems found in higher-level animals. Furthermore, it obviated the need to grapple with real hardware—cameras, robots, and other paraphernalia—at least during the initial stages of research and development, thereby yielding substantial savings in terms of the cost in money and time to acquire and maintain the hardware. The algorithms developed within the animat vision approach were subsequently adapted for use in a mobile vehicle tracking and traffic control system [11], which affirmed the usefulness of the animate vision approach in designing and evaluating complex computer vision systems. The virtual vision paradigm for video surveillance systems research was proposed in [17]. Its central concept was to design and evaluate video surveillance systems using Reality Emulators, virtual environments of considerable complexity, inhabited by autonomous, lifelike agents. The work reviewed in this chapter realizes that concept within the reality emulator developed by Shao and Terzopoulos [14, 15]—a virtual train station populated with lifelike, self-animating pedestrians. In concordance with the virtual vision paradigm, Santuari et al. [1, 13] advocate the development and evaluation of pedestrian segmentation and tracking algorithms using synthetic video generated within a virtual museum simulator containing scripted characters. Synthetic video is generated via a sophisticated 3D rendering scheme, which supports global illumination, shadows, and visual artifacts like 2 We
are currently validating our virtual vision paradigm in a collaborative project with the University of California, Riverside, through the development of a virtual vision simulator that emulates a large-scale physical camera network that they have deployed.
11
Virtual Vision
169
depth of field, motion blur, and interlacing. They have used their virtual museum environment to develop static background modeling, pedestrian segmentation, and pedestrian tracking algorithms. Their work focuses on low-level computer vision. By contrast, our work has focused on high-level computer vision issues, especially multi-camera control in large-scale camera networks, which is a fundamental high-level problem that must be tackled in order to develop advanced surveillance systems [7–9].
4 Smart Camera Nodes Each virtual camera node in the sensor network is able to render the scene from its own vantage point in order to generate synthetic video suitable for visual surveillance (Fig. 4). It is an active sensor that is able to perform low-level visual processing and has a repertoire of autonomous camera behaviors. Furthermore, it is capable of communicating (wirelessly) with nearby nodes in the network. We assume the following communication model: (1) nodes can communicate with their neighbors, (2) messages from one node can be delivered to another node if there is a path between the two nodes, and (3) messages can be broadcast from one node to all the other nodes. Furthermore, we assume the following network model: (1) messages
Fig. 4 Synthetic video feeds from multiple virtual surveillance cameras situated in the (empty) Penn Station environment
170
D. Terzopoulos and F.Z. Qureshi
can be delayed, (2) messages can be lost, and (3) nodes can fail. These assumptions ensure that our virtual camera network faithfully mimics the important operational characteristics of a real sensor network. The following sections describe the capabilities of a camera node in greater detail.
4.1 Synthetic Video Virtual cameras use the OpenGL library and standard graphics pipeline to render the synthetic video feeds. Our imaging model emulates imperfect camera color response, detector and data drop-out noise, compression artifacts, and video interlacing (Fig. 5). Noise is introduced during a post-rendering phase, and the amount of noise present determines the quality of the input to the visual analysis routines, which affects the performance of the pedestrian segmentation and tracking module. We model the variation in color response across cameras by manipulating the HSV channels of the rendered image. Similarly, we can adjust the tints, tones, and shades of an image by adding the desired amounts of blacks, whites, and grays, respectively [2]. Our visual analysis routines rely on color-based appearance models to track pedestrians; hence, camera handovers are sensitive to variations in the color responses of the different cameras. Bandwidth is usually at a premium in sensor networks, especially so in camera networks. To keep bandwidth requirements within acceptable limits, camera nodes typically compress the captured video frames before transmitting them to the monitoring station or to other nodes for the purposes of camera coordination, camera handover, and multi-camera sensing operations. Compression artifacts and the limited resolution of the captured video pose a challenge to visual analysis routines and are therefore relevant to camera network research. To enhance realism, we introduce compression artifacts into the synthetic video by subjecting it to JPEG compression and decompression before providing it to the pedestrian recognition and tracking module.
4.2 Visual Processing The sensing capabilities of a camera node are determined by low-level visual routines (LVR). The LVRs, which implement basic functionalities such as pedestrian detection, tracking, and identification, are computer vision algorithms that directly operate upon the synthetic video generated by the virtual cameras and mimic the performance and limitations of a state-of-the-art surveillance video analysis module. Our virtual vision simulator affords us the benefit of fine tuning the performance of this module by taking into consideration the ground truth data readily available in the virtual world.
11
Virtual Vision
171
Fig. 5 Simulating noise in synthetic video: (a) Detector noise. (b) Data drop-out noise. Compression artifacts in synthetic video: (c) Uncompressed image. (d) Enlarged region of the rectangular box in (c). (e) JPEG-compressed image. (f) Enlarged region of the rectangular box in (e). Video interlacing effects: (g) Video frame obtained by interlacing two consecutive fields. (h) Close-up view of a pedestrian in (g)
We have employed appearance-based models to track pedestrians. Pedestrians are segmented to compute robust color-based signatures, which are then matched across subsequent frames. Color-based signatures have found widespread use in tracking applications [3], but they are sensitive to illumination changes. This shortcoming can be mitigated, however, by operating in HSV rather than RGB color
172
D. Terzopoulos and F.Z. Qureshi
Fig. 6 Tracking pedestrians 1 and 3. Pedestrian 3 is tracked successfully; however, (a) track is lost of pedestrian 1 who blends in with the background. (b) The tracking routine loses pedestrian 3 when she is occluded by pedestrian 2, but it regains track of pedestrian 3 when pedestrian 2 moves out of the way (c)
space. Furthermore, zooming can drastically alter the appearance of a pedestrian, thereby confounding conventional appearance-based schemes. We employ a modified color-indexing scheme [16] to tackle this problem. Thus, a distinctive characteristic of our pedestrian tracking routine is its ability to operate over a range of camera zoom settings. Note that we do not assume that the active cameras are calibrated. The tracking module emulates the abilities and, importantly, the limitations of a state-of-the-art tracking system. In particular, it can lose track due to occlusions, poor segmentation (the quality of segmentation depends upon the amount of noise introduced into the process), or poor illumination (Fig. 6). Tracking sometimes locks onto the wrong pedestrian, especially if the scene contains multiple pedestrians with similar visual appearance; i.e., wearing similar clothes. Tracking also fails in group settings when the pedestrian cannot be segmented properly. Each camera can fixate and zoom in on an object of interest. The fixation and zooming routines are image-driven and do not require camera calibration or any 3D information such as a global frame of reference. The fixate routine brings the region of interest—e.g., the bounding box of a pedestrian—into the center of the image by rotating the camera about its local x and y axes. The zoom routine controls the FOV of the camera such that the region of interest occupies the desired percentage of the image. The implementation details of the various LVRs are presented elsewhere [5].
4.3 Camera Node Behavioral Controller The camera controller determines the overall behavior of the camera node, taking into account the information gathered through visual analysis by the LVRs (bottomup) and the current task (top-down). We model the camera controller as an augmented hierarchical finite state machine (Fig. 7).
11
Virtual Vision
173
Fig. 7 The top-level camera controller consists of a hierarchical finite state machine (FSM). The inset (right) represents the child FSM embedded within the PerformingTask and ComputingRelevance states in the top-level FSM
In its default state, Idle, the camera node is not involved in any task. It transitions into the ComputingRelevance state upon receiving a queryrelevance message from a nearby node. Using the description of the task that is contained within the queryrelevance message, and by employing the LVRs, the camera node can compute its relevance to the task [9]. For example, it can use visual search to find a pedestrian that matches the appearance-based signature forwarded by the querying node. The relevance encodes the expectation of how successful a camera node will be at a particular sensing task. The camera node returns to the Idle state if it fails to compute its relevance because it cannot find a pedestrian matching the description. Otherwise, when the camera successfully finds the desired pedestrian, it returns its relevance value to the querying node. The querying node passes the relevance value to the supervisor node of the group, which decides whether or not to include the camera node in the group. The camera goes into the PerformingTask state upon joining a group, where the embedded child finite state machine hides the sensing details from the top-level controller and enables the node to handle transient sensing (tracking) failures. All states other than the PerformingTask state have built-in timers (not shown in Fig. 7), which allow the camera node to transition into the Idle state rather than wait indefinitely for a message from another node. The child FSM (Fig. 7 (inset)) starts in Track state, where video frames are processed to track a target without panning and zooming a camera. Wait is entered when track is lost. Here camera zoom is gradually reduced in order to reacquire track. If a target is not reacquired during Wait, the camera transitions to the Search state, where it performs search sweeps in PTZ space to reacquire the target. A camera node returns to its default state after finishing a task, using the reset routine, which is a proportional-derivative (PD) controller that attempts to minimize the difference between the current zoom/tilt settings and the default zoom/tilt settings.
174
D. Terzopoulos and F.Z. Qureshi
5 Surveillance Systems To date, we have studied the problems of active camera scheduling and collaborative, persistent observation within smart camera networks. We have been able to rapidly develop novel camera control strategies to address these problems by deploying virtual camera networks in the virtual Penn Station’s large-scale simulated indoor urban environment.
5.1 Active Camera Scheduling In 2005, we introduced a camera scheduling strategy for intelligently managing multiple, uncalibrated active PTZ cameras, supported by several static, calibrated cameras in order to satisfy the challenging task of automatically recording close-up biometric videos of pedestrians present in a scene. Our approach assumes a nonclairvoyant model of the scene, supports multiple cameras, supports preemption, and allows multiple observations of the same pedestrian [7]. To conduct camera scheduling experiments, we populated the virtual train station with up to twenty autonomous pedestrians, who enter, wander, and exit the main waiting room of their own volition. We tested our scheduling strategy in various scenarios using anywhere from 1 to 18 PTZ active cameras. For each trial, we placed a wide-FOV passive camera at each corner of the main waiting room. We also affixed a fish-eye camera to the ceiling of the waiting room. These passive cameras were used to estimate the 3D location of the pedestrians. We formulated the multi-camera control strategy as an online scheduling problem and proposed a solution that combines the information gathered by the wide-FOV cameras with weighted round-robin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. Figure 8 compares weighted and non-weighted scheduling schemes for active PTZ cameras assignment.
5.2 Collaborative Persistent Surveillance In [8], we developed a distributed coalition formation strategy for collaborative sensing tasks in camera sensor networks. The proposed model supports task-dependent node selection and aggregation through an announcement/bidding/selection strategy combined with a constraint satisfaction problem (CSP) based conflict resolution mechanism. Our technique is scalable as it lacks any central controller, and it is robust to node failures and imperfect communication. In response to a sensing task, such as, “observe pedestrian i during his stay in the region of interest,” wide-FOV passive and PTZ active cameras organize themselves into groups with the objective of fulfilling the task. These groups evolve as the pedestrian enters and exits the fields
11
Virtual Vision
175
Fig. 8 Comparisons of Weighted (W) and Non-Weighted (NW) scheduling schemes. The weighted scheduling strategy, which takes into account the suitability of a camera for recording a particular pedestrian, outperforms its non-weighted counterpart, as is evident from its (a) higher success rates and (b) shorter lead, (c) processing, and (d) wait times. The displayed results are averaged over several runs of each trial scenario. Trials 1–6 involve 5 pedestrians and 1, 2, 3, 4, 5, and 6 cameras, respectively. Trials 7–12 involve 10 pedestrians and 3, 4, 5, 6, 7, and 8 cameras, respectively. Trials 13–18 involve 15 pedestrians and 5, 6, 9, 10, 11, and 12 cameras, respectively. Trials 19–24 involve 20 pedestrians with 5, 8, 10, 13, 15, and 18 cameras, respectively
of view of different cameras, ensuring that the pedestrian remains persistently under surveillance by at least one camera. Figure 9 illustrates the 15-minute persistent observation of a pedestrian of interest as she makes her way through the train station. For this example, we placed 16 active PTZ cameras in the train station, as shown in Fig. 2.
6 Conclusions Virtual Vision is a unique synthesis of virtual reality, artificial life, computer graphics, computer vision, and sensor network technologies, with the purpose of facilitating computer vision research for camera sensor networks. Through the faithful emulation of physical vision systems, any researcher can investigate, develop, and evaluate camera sensor network algorithms and systems in virtual worlds simulated on high-end commodity personal computers. Without having to deal with specialpurpose surveillance hardware, we have demonstrated our prototype surveillance systems in a virtual train station environment populated by lifelike, autonomous pedestrians. This simulator has facilitated our ability to design visual sensor networks and experiment with them on commodity personal computers. In this chapter we described two prototype multi-camera surveillance systems capable of autonomously carrying out high-level visual surveillance tasks. Our first surveillance system comprised calibrated passive and uncalibrated active cameras, and it relied upon a scheduling strategy for managing the multiple active cameras in order to capture close-up videos of pedestrians as they move through designated areas. The second surveillance system managed multiple uncalibrated passive and active cameras intelligently in order to persistently observe pedestrians of interest that enter and exit the FOVs of different cameras as they travel across the train station. Our companion chapter in this volume reviews our most recent work on proactive planning for PTZ camera assignment and handoff [9].
176
D. Terzopoulos and F.Z. Qureshi
Fig. 9 15-minute persistent observation of a pedestrian of interest as she makes her way through the train station. (a–d) Cameras 1, 9, 7, and 8 monitoring the station. (e) The operator selects a pedestrian of interest in the video feed from Camera 7. (f) Camera 7 has zoomed in on the pedestrian, (g) Camera 6, which is recruited by Camera 7, acquires the pedestrian. (h) Camera 6 zooms in on the pedestrian. (i) Camera 2. (j) Camera 7 reverts to its default mode after losing track of the pedestrian and is now ready for another task. (k) Camera 2, which is recruited by Camera 6, acquires the pedestrian. (l) Camera 3 is recruited by Camera 6; Camera 3 has acquired the pedestrian. (m) Camera 6 has lost track of the pedestrian. (n) Camera 2 observing the pedestrian. (o) Camera 3 zooming in on the pedestrian. (p) Pedestrian is at the vending machine. (q) Pedestrian is walking towards the concourse. (r) Camera 10 is recruited by Camera 3; Camera 10 is observing the pedestrian. (s) Camera 11 is recruited by Camera 10. (t) Camera 9 is recruited by Camera 10
The future of advanced simulation-based approaches for the purposes of low-cost prototyping and facile experimentation appears promising. Imagine an entire city, including indoor and outdoor environments, subway stations, automobiles, shops and market places, homes and public spaces, all richly inhabited by autonomous virtual humans. Such large-scale virtual worlds will one day provide unprecedented opportunities for studying large-scale camera sensor networks in ways not currently possible in our train station simulator. Future work on virtual vision research will therefore benefit from long-term efforts to increase the complexity of virtual worlds.
11
Virtual Vision
177
Acknowledgements We thank Wei Shao for developing and implementing the train station simulator and Mauricio Plaza-Villegas for his valuable contributions. We thank Tom Strat, formerly of DARPA, for his generous support and encouragement.
References 1. Bertamini, F., Brunelli, R., Lanz, O., Roat, A., Santuari, A., Tobia, F., Xu, Q.: Olympus: An ambient intelligence architecture on the verge of reality. In: Proc. International Conference on Image Analysis and Processing, Mantova, Italy, pp. 139–145 (2003) 2. Birren, F.: Color Perception in Art. Van Nostrand Reinhold, New York (1976) 3. Comaniciu, D., Ramesh, V., Meer, P.: Real-time tracking of non-rigid objects using mean shift. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00), Hilton Head Island, SC, vol. 2, pp. 142–151 (2000) 4. Costello, C.J., Diehl, C.P., Banerjee, A., Fisher, H.: Scheduling an active camera to observe people. In: Proc. ACM Int. Workshop on Video Surveillance and Sensor Networks, New York, pp. 39–45 (2004) 5. Qureshi, F.Z.: Intelligent Perception in Virtual Camera Networks and Space Robotics. PhD thesis, Department of Computer Science, University of Toronto, Canada (2007) 6. Qureshi, F.Z., Terzopoulos, D.: Towards intelligent camera networks: A virtual vision approach. In: Proc. Joint IEEE Int. Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS05), Beijing, China, pp. 177–184 (2005) 7. Qureshi, F.Z., Terzopoulos, D.: Surveillance camera scheduling: A virtual vision approach. ACM Multimed. Syst. J. 12, 269–283 (2006) 8. Qureshi, F.Z., Terzopoulos, D.: Smart camera networks in virtual reality. Proc. IEEE 96(10), 1640–1656 (2008) (Special Issue on Smart Cameras) 9. Qureshi, F.Z., Terzopoulos, D.: Planning ahead for PTZ camera assignment and control. In: Proc. Third ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC 09), Como, Italy, pp. 1–8 (2009) 10. Rabie, T., Terzopoulos, D.: Active perception in virtual humans. In: Vision Interface (VI 2000), Montreal, Canada, pp. 16–22 (2000) 11. Rabie, T., Shalaby, A., Abdulhai, B., El-Rabbany, A.: Mobile vision-based vehicle tracking and traffic control. In: Proc. IEEE International Conference on Intelligent Transportation Systems (ITSC 2002), Singapore, pp. 13–18 (2002) 12. Salgian, G., Ballard, D.H.: Visual routines for autonomous driving. In: Sixth International Conference on Computer Vision, Bombay, India, pp. 876–882 (1991) 13. Santuari, A., Lanz, O., Brunelli, R.: Synthetic movies for computer vision applications. In: Proc. IASTED International Conference: Visualization, Imaging, and Image Processing, Spain, pp. 1–6 (2003) 14. Shao, W., Terzopoulos, D.: Autonomous pedestrians. In: Proc. ACM SIGGRAPH/EG Symposium on Computer Animation, Los Angeles, CA, pp. 19–28 (2005) 15. Shao, W., Terzopoulos, D.: Environmental modeling for autonomous virtual pedestrians. In: Proc. SAE Digital Human Modeling Symposium, Iowa City, IA (2005) 16. Swain, M.J., Ballard, D.H.: Color indexing. Int. J. Comput. Vis. 7(1), 11–32 (1991) 17. Terzopoulos, D.: Perceptive agents and systems in virtual reality. In: Proc. ACM Symposium on Virtual Reality Software and Technology, Osaka, Japan, pp. 1–3 (2003) 18. Terzopoulos, D., Rabie, T.: Animat vision: Active vision in artificial animals. Videre, J. Comput. Vis. Res. 1(1), 2–19 (1997)
Chapter 12
Virtualization and Programming Support for Video Sensor Networks with Application to Wireless and Physical Security Azer Bestavros and Michael J. Ocean
Abstract Network Security Systems are heavily anchored in the digital plane of “cyber space” and hence cannot be used effectively to derive the physical identity of an intruder in order to prevent further malicious wireless broadcasts (i.e., escorting an intruder off the premises based on physical evidence). Embedded Sensor Networks (SNs) can be used to bridge the gap between digital and physical security planes, and thus can provide reciprocal benefit to security tasks on both planes. Toward that end, we present our experience integrating wireless networking security services into SN B ENCH (the Sensor Network workBench). SN B ENCH provides an extensible framework that enables the rapid development and automated deployment of SN applications on a shared, embedded sensing and actuation infrastructure. SN B ENCH’s extensible architecture allows an engineer to quickly integrate new sensing and response capabilities into the SN B ENCH framework, while high-level languages, compilers and execution environments allow novice SN programmers to compose SN service logic, unaware of the lower-level components on which their services rely. Concrete examples are provided to illustrate the power and potential of Wireless Security Services that span both the physical and digital plane.
Keywords Wireless sensor networks · Video networks · Intrusion detection · Physical security · Software environments This research was supported in part by a number of NSF awards, including CISE/CSR Award #0720604, ENG/EFRI Award #0735974, CISE/CNS Awards #0524477, #0952145, CNS/NeTS Award #0520166, CNS/ITR Award #0205294, CISE/EIA RI Award #0202067, and CISE/CCF Award #0820138.
A. Bestavros () · M.J. Ocean Computer Science Department, Boston University, Boston, MA, USA e-mail:
[email protected] M.J. Ocean Computer Science Department, Endicott College, Beverly, MA, USA e-mail:
[email protected]
B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_12, © Springer-Verlag London Limited 2011
179
180
A. Bestavros and M.J. Ocean
1 Motivation A variety of Wireless Intrusion Detection Systems (WIDS) have been created to address Wireless Network Security concerns. WIDS employ wireless probes/sensors to monitor the Media Access Control (MAC) frames transmitted on the wireless medium and identify misuse by observing either suspicious characteristics of individual frames (e.g., exhibiting characteristics imprinted by standard hacking tools) or a particular pattern in a sequence of frames (e.g., sequences in violation of protocol standards). Wireless misuse includes illegitimate users attempting to gain access to the network (intrusion), man-in-the-middle attacks (e.g., luring legitimate users into communication with a rogue access point), and various Denial of Service (DoS) attacks [4] (e.g., spoofing a legitimate wireless Access Point (AP) and sending a disauthenticate beacon to legitimate users). Wireless intrusion is often dealt with using Layer-3 mechanisms (e.g., content based packet filtering, IP address isolation), essentially ignoring the option of Layer2 detection and prevention. Layer-3 IDSs are likely popular because there are far more data available at Layer-3, making it straightforward to respond to attacks, and because detection and response at Layer-3 is independent of the Layer-2 connection medium. On the other hand, Layer-3 response to Layer-2 wireless DoS attacks is limited given that attackers will likely utilize fictitious or spoofed MAC addresses and may not have an IP address to retaliate against. Ultimately the only way to respond to these types of attack is to utilize information derived from the wireless medium (e.g., received signal strength) to reconstruct physical location toward the goal of preventing further wireless transmissions from that user [7]. Wireless Intrusion Detection Systems provide mechanisms to identify, detect and locate DoS attacks, yet these systems are generally limited to logging or email alert response mechanisms. Many works ultimately recommend dispatching administration personnel to further analyze and respond to a detected attack—a costly and impractical solution in many situations. Instead, once the physical area of an attack has been derived it is possible to utilize automated responses from a variety of actuation hardware, if available; e.g., embedded pan-tilt-zoom video cameras to capture an image, wireless detectors on pan-tilt motors to pinpoint a signal, programmable robots to triangulate signal, a common message display (virtual bulletin board) in the environment informing users why their service has been interrupted and who is responsible. Additionally, there would be a clear benefit from including other, non-network centric inputs to the Wireless Network Security System (e.g., a MAC whitelist from Bluetooth/RFID tracking, analysis of security camera images or passcard logs). Generally, attaining such cross-modal interaction within the context of a Network Intrusion Detection tool would require the generation of highly customized, package and deployment specific software (modules, scripts, etc.) that are, by their very nature, cumbersome to maintain. Indeed, such an approach is wrong headed. We observe that Wireless Network Security Services are specific, narrowly focused instantiations of an Embedded Sensor Network wherein sensory data include the output of such monitoring tools. Rather than “hack” a Wireless Security System to
12
Virtualization and Programming Support for Video Sensor Networks
181
include Sensor Network functionality, we advocate the inclusion of Wireless Security within a Sensor Network. Thinking differently about Network Security, the integration of new sensory data (e.g., motion detection, face detection) and actuation responses expand Network Security beyond the digital plane and into the physical plane. This chapter details the inclusion of wireless network monitoring devices in our Sensor Network infrastructure, SN B ENCH (Sensor Network Workbench) to achieve precisely these goals. SN B ENCH provides a high-level programmatic interface to the resources of a Sensor Network (SN) and thus the inclusion of wireless network sensors enables intrusion detection and response services to be written quickly and easily. SN B ENCH has been designed with extensibility and modularity as a central tenet and therefore the changes required to include these new sensing modalities are quite modest. Moreover, the framework’s modular nature allows a user to swap in any improved emergent wireless surveillance tool or technology (be it algorithmic or a physical turn-key device) with nominal effort and such changes would be transparent to their dependent services. We submit that our programmable, adaptable SN framework is the ideal foundation on which to compose Wireless Network Security services and physical security services alike, providing reciprocal benefit to each. The example programs given provide some insight into the highly customized, cross-modal Wireless Security behaviors that are possible in this context.
2 Related Work While many Network Intrusion Detection (Security) Systems exist (both commercial and open-source), we are presently unaware of any other work that leverages a programmable Sensor Network framework toward joint physical and Wireless Network Security, and thus believe we are unique in this regard. We present works that are related in three major thrusts; We distinguish between works that provide detection on a single wireless source (probe) as Wireless Intrusion Detectors (WIDs), those works that detect events across multiple detectors simultaneously as Intrusion Detection Systems (IDSs) and finally those that determine attack location as Wireless Intrusion Detection Systems (WIDSs). Although WIDSs contain a WID component, these works are not necessarily proper subsets of each other, as IDSs may not provide wireless detection. The distinction that a WIDS must determine attack location is sensible, considering that MAC addresses are easily spoofed [4] and that Layer-2 DoS attack response generally requires physical intervention [7]. Wireless Intrusion Detection Kismet [8] is the de facto open-source Layer-2 Wireless Intrusion Detector. Kismet passively scans 802.11 channels for activity and will generate alert events when suspicious frames are detected (among other uses). A Kismet deployment may consist of three distinct components, (1) a Kismet Drone that passively captures the wireless frames from its local interface and sends them to (2) a Kismet Server that processes the frames from drones to detect either fingerprint or trend based suspicious activity and (3) an optional remote Kismet
182
A. Bestavros and M.J. Ocean
client that connects to the Server to receive notifications and render the results. By writing a custom client (using the published client protocol) Kismet may drive “external” wireless event notification. Kismet may be configured as an IDS by associating several drones with a single server process to build a single, central wireless event log. Kismet has also recently been updated to track which physical drone is responsible for an alert, enabling ad-hoc spatial intrusion tracking; thus, assuming a custom client that processes these data, such Kismet deployments would be considered Wireless Intrusion Detection Systems by our definition. Other notable WID tools have existed prior to Kismet but have been unmaintained in recent years. Intrusion Detection Systems As Kismet is to Layer-2 WID and IDS, Snort [11] is the de facto standard IDS for Layer-3 (IP traffic analysis). Snort is a mature IDS with a large user base and comprehensive set of detection rules for detecting malicious content in IP packets for a wide range of attacks. Snort also offers very basic response mechanisms (e.g., logging or email alert mechanisms) and projects (e.g., Barnyard) that claim to enable the creation and use of custom output plug-ins. As Snort is aimed at Layer-3, it offers no support for wireless-specific events; plans to integrate wireless frame capture appear to have been abandoned. In many ways, our vision is similar to that of modular (or so-called “Hybrid”) IDSs (e.g., [13, 14]). These systems are designed to allow various Intrusion Detection Software packages to be integrated as “sensors” in the IDS. This modular approach is similar in spirit to the cross tool integration that we hope to provide to the Network Security community, yet these works are narrowly focused on issues of traditional Network Security. Our work enables sense and respond programs that manipulate both network and physical sensory data (e.g., image processing on embedded video cameras) in a manner that would be impossible on these platforms without significant changes. Wireless Intrusion Detection Systems Many approaches to derive location from multiple sensors’ Signal Strength Information (SSI) of RF transmissions have been undertaken, including addressing issues of transmission reflection, diffraction and interference (e.g., [3, 15]). The WIDS architecture detailed in [1] provides detailed analysis of specific directional antennas (as opposed to the typical, omni-directional antennas) to form a sweeping perimeter around an access point and is able to accurately pinpoint wireless intruders. Not only would our work be compatible with the use of sweeping directional antenna, SN B ENCH could likely direct the servos that control antenna movement explicitly within the security logic (easing future changes). Finally, commercial offerings provide turn-key detection and response systems for corporate wireless networks (e.g., [2]). Responses to wireless attack detection in these systems are more proactive (e.g., disauthenticating malicious users from the network), yet they do not provide integration with third-party tools or offer a programming interface to adjust the sense and respond behavior. Commercial sense and respond WIDSs lack the extensibility required to enable cross-modal monitoring (e.g., utilizing video frames).
12
Virtualization and Programming Support for Video Sensor Networks
183
3 SNBench Overview To orient the reader to the platform to ease further discussion, in this section we briefly highlight the salient features of SN B ENCH. The vision, goals and high-level overview of the SN B ENCH infrastructure have been reported elsewhere [5] and implementation details may be found in [9]. SN B ENCH consists of programming support and a runtime infrastructure for Sensor Networks comprised of heterogeneous sensing and computing elements that are physically embedded into a shared environment. We refer to such a physical space with an embedded SN as a Sensorium. The SN B ENCH framework allows Sensorium users to easily program, deploy, and monitor the services that run in this space while insulating the user from the complexity of the physical resources therein. We liken the support that SN B ENCH extends to a Sensor Network to the support that higherlevel languages and operating systems provide to traditional, single machine environments (language safety, APIs, virtualization of resources, scheduling, resource management, etc.). SN B ENCH is designed such that new hardware and software capabilities may be painlessly folded into the infrastructure by its advanced users and those new capabilities easily leveraged by its novice users. SN B ENCH provides a high-level programming language with which to specify programs (services) that are submitted to the resource management component which in turn disseminates program fragments to the runtime infrastructure for execution. At the lowest level, each sensing and/or computing element hosts a Sensor eXecution Environment (SXE) that abstracts away specific details of the host and attached sensory hardware. SXEs are assigned tasks by the resource management components of SN B ENCH; the Sensorium Service Dispatcher and Sensorium Resource Manager in tandem monitor SN resources, schedule (link) and deploy (bind) tasks on to available SXEs. The Virtual Instruction Set Architecture of SN B ENCH is the Sensorium Task Execution Plan (STEP), a tasking-language used to describe complete programs and fragments alike. A STEP program is a graph of an SN program’s data-flow and computational dependency, with the nodes of a STEP graph representing the atomic computation and sensing operations and edges representing data flow. In execution, demand for evaluation is pushed down from the root of the graph to the leaves, and values percolate up from the leaves back to the root. STEP nodes describe data, control flow (e.g., repetition, branching) and computation operations that we refer to as STEP Opcodes, and the SXE maintains implementations of the Opcodes with which it may be tasked. Opcodes do not directly manipulate sensors, but rather manipulate SN B ENCH typed data. Specific details of the sensor hardware of the SXE are abstracted away by a SensorHandler module that is capable of communicating with and reformatting the data from a specific sensor to produce to SN B ENCH typed data; support for new sensor device types require the addition of new SensorHandler modules.1 In 1 SXEs
can retrieve Opcode implementations at runtime; however, support for loading new sensing devices at runtime is not currently supported. Such functionality is not difficult to support, and it is analogous to dynamically loading device drivers to support new hardware.
184
A. Bestavros and M.J. Ocean
SN B ENCH there is a distinction between a SN Service Developer who uses highlevel programming languages to compose Services by gluing together Opcodes and sensors (generally without regard for how the Opcodes are actually implemented beyond their type signature) and the SN B ENCH “engineers” who are responsible for expanding the Opcode and SensorHandler libraries to enable new functionalities.
4 Enabling Wireless Monitoring SN B ENCH is extensible by design insofar as support for new sensing devices may be added to the Sensor eXecution Environment (SXE) by providing implementations of two relatively small interfaces; a SensorHandler translates SN B ENCH requests to interact with a specific device and a SensorDetector module must provide a facility to detect new devices of this type and inspect their state. The SensorHandler is akin to a device driver, abstracting away the specific idiosyncrasies of the particular device’s interface and enabling the device to be accessed by higher-level programming constructs. As far as the SN B ENCH framework is concerned, the abstracted device becomes just another managed input device/event generator only different from a video camera or motion sensor insofar as the datatype of its output. To enable wireless network security service composition on SN B ENCH, two new sensors and a new actuator were added; the WifiAlertSensor reports wireless alert detection events, the WifiActivitySensor reports MAC addresses and Received Signal Strength Indication (RSSI) for any passively observed wireless activity, and the WifiResponder actuator sends a disauthenticate flood to a particular MAC address. Rather than implement wireless Layer-2 tools from scratch, we opted to leverage several existing open-source software packages.
WifiAlertSensor The WifiAlertSensor is a SensorHandler implementation that leverages the Kismet [8] wireless intrusion detector via a self-contained customized Kismet client. The Java based WifiAlertSensor class is hosted by a “nonlightweight” SXE and translates the proprietary Kismet client-server protocol into structured, typed SN B ENCH objects (tagged XML) that encapsulate notifications from the Kismet server. The decision to use Kismet stems from its passive scanning ability, wide range of hardware support, and modular design (described in Sect. 2). While the decision to use this package in particular may be debated, the inclusion of any another functionally-equivalent Wireless Intrusion Detector would be equally straightforward. A Kismet client may request to receive several types of Kismet messages from a Kismet server/drone pair (client traffic, AP detection, suspicious activity alerts, etc.). In the case of the Alert Sensor, the client requests notification of all wireless alerts supported by the current stable build of Kismet. Whenever the Kismet server detects an alert condition from its corresponding drone’s data feed, an alert is sent to the WifiAlertSensor client which translates and buffers the alert message. In addition to translating the Kismet protocol, the WifiAlertSensor adds additional fields to the alert message: a local timestamp to measure buffer service delay, a sensor source to
12
Virtualization and Programming Support for Video Sensor Networks
185
identify the physical sensor (drone) that produced the message, and a severity field that indicates the relative threat of the particular attack. The WifiAlertSensor’s message buffer is configurable in length (where length is measured in either size or time) and alert messages are retrieved from the buffer by Opcodes requesting data from this sensor. Implementation of the retrieval Opcode may impose a blocking or nonblocking semantic, as needed. In our experimentation we implemented a single alert-centric Opcode, sxe.core.wifi.get, that performs a nonblocking read from the Alert Sensor’s buffer to populate and return a WifiAlert. The WifiAlert datatype is a subtype of snStruct, with tagged fields corresponding to the fields populated by the WifiAlertSensor and thus, accessing the data within a WifiAlert, reuses the existing snStruct manipulation Opcodes. A Service Developer retrieves WifiAlerts via the high-level function DetectWifiAlert() that is compiled into a call to the Opcode sxe.core.wifi.get with a WifiAlertSensor (or set of sensors) as a parameter. High-level service logic examples are given in Sect. 7. WifiActivitySensor The Activity sensor provides data regarding wireless transmissions that have been detected by a passive, promiscuous-mode wireless sensor. In particular, we are interested in the MAC address of a transmission, the observed signal strength (RSSI) and the mode of the transmission (i.e., Access Points, Clients, Ad-hoc participants). While determining physical location from RSSI is imperfect (as RSSI readings themselves may not be entirely accurate depending on the driver implementation and other physical factors), the use of RSSI readings can better estimate the physical location of a MAC address beyond the simple cell-of-origin. WifiActivitySensor maintains a hash-table of the detected wireless activity (keyed by MAC address), which can be used either to report new/updated wireless activity (similar to the Alert Sensor) or to query the activity log to find information about a particular MAC address. Like the Alert Sensor, the activity sensor also communicates with a remote sensor “server” process responsible for gathering data. As the Kismet drone/server cannot retrieve the RSSI on all hardware platform, two different physical implementations for the activity sensor server are supported. For Kismet’s RSSI-supported hardware, the client, which is derived from the WifiAlertSensor implementation, requests and parses NETWORK and CLIENT messages from the Kismet server rather than ALERT messages. For the OpenWRT platform, a custom monitoring program sends ioctl’s to the wireless device to put the device in passive monitor mode, accept frames, and retrieve data from the frames and device (including the RSSI). This program is based on code from the opensource WiViz [12] package for OpenWRT, which contains the ioctl codes needed to achieve the proper device state and interaction. Like the Kismet server, this program provides notifications of activity messages which are received and hashed by the WifiActivitySensor. The high-level Opcode DetectWifiActivity() is compiled into sxe.core.wifi.get with a WifiActivitySensor as a parameter and blocks until a new activity message is available from that sensor. In addition QueryWifiActivity() (compiled into sxe.core.wifi.find) searches the WifiActiv-
186
A. Bestavros and M.J. Ocean
itySensor’s hash table for the latest reading associated with the specified MAC address. As with the WifiAlertSensor, returned data are snStruct derivatives. WifiResponder In addition to the wireless network sensing described above, the Layer-2 wireless actuator (i.e., output device) WifiResponder may be used as a retaliatory action against a detected attacker. The WifiResponder invokes a script on a trusted (whitelisted) device running Linux with a compatible 802.11 interface and the airreplay-ng [6] tool. The Opcode APDeauth() takes as arguments a WifiResponder that will send a flood of deauthenticate messages to a particular MAC address (the second argument) from a particular MAC address (the third argument).2 An actuator is nearly identical to a Sensor in its implementation within SN B ENCH . The Handler for WifiResponder invokes the remote common gateway interface (CGI) script to initiate the deauthenticate “attack” against the specified host.
5 Deployment Environment Our test-bed deployment contains several OpenWRT[10] Linux enabled Linksys WRT54GL Access Points (APs), each with the kismet-drone, airreplay-ng, and signal strength monitor packages installed. The APs are configured to use their wireless interface in client mode, and are connected to our gigabit research LAN by its 100 Mbit Ethernet port. To support the WifiAlertSensor, each of the APs run a Kismet drone process, while the Kismet server process runs on the same host as the SXE. Although the Kismet server process could also be run directly on the AP, the RAM and CPU limitations of these devices lead to a less responsive system in that scenario. As the Kismet server did not distinguish the results from different Kismet drones at the time of our experiments, one Kismet server process was required per drone, and each WifiAlertSensor connects to a unique Kismet server process thus allowing SN B ENCH to distinguish which drone generated a wireless event. Running one Kismet server per drone also carries the advantage of minimizing the impact of a Kismet server process hanging, or failing to process updates from its drones (admittedly a fairly uncommon occurrence). In our tests of the WifiAlertSensor we were able to simulate and detect all relevant attacks detected by Kismet and were unable to measure any significant induced delay on event detection in the SN B ENCH infrastructure. Analysis confirmed the expectation that the amount of time a single Kismet message spent in the Sensor buffer was directly related to the computation load on the SXE host and the alert generation rate. In general the observed buffer service delay oscillated between zero and 15 ms per alert under moderate load with unrealistically high message flooding arrival rates (in practice, Kismet can and will throttle alert notification rates, 2 Readers
may readily note that this Opcode is a loaded weapon and may gasp or recoil in horror. In fact, this is not the first Opcode that requires special user privileges to ensure correct use.
12
Virtualization and Programming Support for Video Sensor Networks
187
however this was disabled for our performance tests). Under heavy load conditions with alert message flooding, we experienced queuing delay as long as 300 ms. This gives us a good indication as to the maximum acceptable workload for an individual SXE before it is no longer a viable host for wireless sensing tasks. Ultimately any response detection under one second is reasonable as it is unlikely that the attacker would, say, flee the premises (or video frame) within that amount of time.
6 Service Programming Primer To understand the Wireless Security Services examples it is important to understand the key concepts and unique constructs of SN B ENCH programming.3 The Sensorium Task Execution Plan (STEP) language has a functional-style, high-level sibling called SNAFU (Sensor Network Applications as Functions). SNAFU serves as a readable, accessible language that is compiled into the graph-centric STEP for execution. Broadly speaking, functions in SNAFU correspond to the computational nodes of a STEP graph while terminals represent nodes that convey sensors, actuators, and constant values. SNAFU provides symbolic assignment and function definition, however it forbids explicit recursion by reference. Instead SNAFU provides iteration constructs called triggers. A trigger takes two arguments: a predicate and a response clause. The predicate is repeatedly evaluated until it evaluates to true, at which point the response clause is evaluated and returned as the result of the trigger expression. For example, consider the expression Trigger(P,Q) in which P is the detection of an AP intrusion and Q is an expression that shuts down the AP. The WhileTrigger(P,Q) is similar to the previous trigger, except that it evaluates Q every time P evaluates to true and when P eventually evaluates to false returns the last value of Q (or NIL if P was initially false). Persistent triggers extend the basic triggers in that they return a stream of values over their persistent evaluation. A LevelTrigger evaluates the predicate P indefinitely (or for some specified length of time or conditional termination) and evaluates and returns a value of Q every time P evaluates to true. In practice P may be the detection of a particular MAC address being used in the network and Q is the recording of an image at the detected locale. An EdgeTrigger continually evaluates the predicate, but will only evaluate and return the clause Q whenever the predicate P transitions to be true (i.e., on the edge of the signal P). Consider, if the expression P represents detection of two deauthenticate beacons (indicating the start of a deauthenticate flood) and Q is an SMS pager alert, we do not want to generate a separate notification for every consecutive deauthenticate beacon for the duration of the flood. SNAFU also allows a programmer to refer to an expression by symbolic reference (e.g., let X = Y in Z, wherein X stands for the complete expression Y 3 We
tion.
refer the reader to [9] for a more thorough treatment of the SNAFU language and its evalua-
188
A. Bestavros and M.J. Ocean
in the expression Z) or refer to a computational result by symbolic reference (e.g., let_const X = Y in Z, wherein X stands in for the result of the expression Y in the expression Z). Finally the trigger construct begs for the creation of a unique reference that allows the symbol to be recomputed once per iteration of the trigger. The “let_once” binding (e.g., let_once X = Y in Z) provides exactly that facility, ensuring the expression Y is evaluated once per iteration of the trigger (Z) at the first occurrence of the symbol X, while all latter instances of the symbol X in the same iteration of Z are evaluated by reference to the previous evaluation.
7 Wireless Security Services An example SNAFU program that provides simple logging is given in Program 1. A level_trigger is used to assign an event handler to the detection of a high severity wireless alert. The storage.append Opcode modifies a named storage entity (i.e., table) by inserting a data object and its corresponding unique key. The storage table is keyed by timestamp and includes entries for each detected violation containing the recorded MAC address, the sensor from which the alert was detected, and the type of alert. Unlike the logging provided by Kismet as an IDS, this service records which sensor has detected the event and is backed by an SQL server. The logged data are available programmatically via storage access Opcodes or direct SQL queries, or through a standard web browser via the SXE host’s web service that performs XSL translations to render the local data storage. This sample SNAFU Program 1 Add to a central log on detection of a wireless alert. let_once ALERT = DetectWifiAlert(sensor(WifiAlert,ALL)) in level_trigger( equals(ALERT."SEVERITY","HIGH"), storageappend("ALERTLOG", concat(ALERT."TIMESTAMP",ALERT."SOURCE"),ALERT))
SNAFU program could easily be extended to establish a log of all observed wireless activity (not just attacks) by adjusting the predicate of the trigger from DetectWifiAlert to DetectWifiActivity and removing the severity check. Another simple example is given in Program 2, which automatically emails an administrator when a specific wireless attack is detected. The previous examples are essentially the status quo for a response to the detection of a breach in a Wireless Network—an entry into a log file or an email alert. The advantage of employing the SN B ENCH in the wireless security domain is the wider range of responses possible. Nominally, the email operation in Program 2 could be replaced with any number of response mechanisms including sending an explicit deauthorization to the detected MAC address4 using the WifiResponder and 4 A MAC address is far from the best way to uniquely identify an attacker, as the attacker will likely
use a fictitious MAC address or worse, clone a legitimate user’s MAC during an attack.
12
Virtualization and Programming Support for Video Sensor Networks
189
APDeauth Opcode described in Sect. 4. Instead, we explore the unique cross section of the network plane (e.g., wireless data frames) with the physical plane (e.g., signal strength and signal loss of signal over distances). For example, an embedded, cross-modal Sensor Network such as the Sensorium can utilize both wireless network sensors (i.e., network plane) and a pan-tilt-zoom video camera network (i.e., physical plane) to catch an image of the attacker “in the act.” SNAFU Program 2 E-mail an admin when a specific wireless alert is detected. let_once ALERT = DetectWifiAlert(sensor(WifiAlert,ALL)) in level_trigger( equals(ALERT."TYPE","DEAUTHFLOOD"), email("
[email protected]", concat("$NOW$", ": Deauth flood detected from MAC ", ALERT."MAC", " at time ", ALERT."TIMESTAMP", " by sensor ", ALERT."SOURCE")))
Any user detected engaged in wireless network intrusion is clearly within a bounded distance from the detecting sensor. This coarse, cell-of-origin based physical location of wireless users is available, imprinted in all wireless data returned from the WifiSensors (determined by which sensor has detected the user). A very simple wireless cell-of-origin location example is specified in Program 3. The program’s content is very similar to the previous examples and introduces some pantilt-zoom sensor (PTZCamera) specific Opcodes, the function of which should be clear from context. This sample streams images of a region where an attack has been detected. The location estimation is explicit in the service logic, selecting an image from the camera that best covers the physical space within the signal coverage region of the relevant Wifi sensor, which (in this example) requires some knowledge of the specific physical layout of the sensor deployment. The case expression takes the same syntax as in StandardML and is used for readability as syntactic sugar (i.e., a macro) for nested conditionals. Connecting this program fragment to either of the previous examples would log or email images that correspond to the attack location.
SNAFU Program 3 Whenever a wireless alert is detected, pan a PTZ camera to that region and return its image. def BestPTZForViewOf(alert) = case APName(alert.SOURCE) of "CS Grad Lab West" => List(45,0,0,sensor(PTZCamera,"PTZ1"), | "CS Grad Lab East" => List(15,0,0,sensor(PTZCamera,"PTZ1"), | "CS Grad Lab Lounge" => List(0,0,0,sensor(PTZCamera,"PTZ3"), | "CS UGrad Lab" => List(0,0,0,sensor(PTZCamera,"PTZ4") let_each ALERTSENSORS = sensor(WifiAlert,"ALL") in let_once ALERT = DetectWifiAlert(ALERTSENSORS) in level_trigger( not(isNull(ALERT)), PTZSnapshot(BestPTZForViewOf(ALERT)))
190
A. Bestavros and M.J. Ocean
Alternatively user location reconstruction could be implemented within an Opcode, resulting in IDS logic that is agnostic to the particular location resolution mechanism used. Such an approach makes sense if the deployment environment already contains a wireless location infrastructure (e.g., network appliance, all knowing oracle) that could be accessed from an Opcode call. An example of this approach is given in Program 4. WifiLocateMac encapsulates the physical location of MAC addresses and PTZLocate determines the best PTZ Camera (and corresponding angle) to capture an image of that location. The implementation of WifiLocateMac is functionally similar to BestPTZForViewOf in the example in Program 3, yet uses a received signal strength from multiple sensors to estimate the target’s location between the sensors. SN B ENCH not only eases the composition of such alert services, it also eases deployment by automating the re-use of existing computation/deployments to improve resource utility. All the examples given thus far share the same predicate logic and could share a single instantiation of that portion of the logic.
SNAFU Program 4 Equivalent to Program 3, but uses “black-box” Opcodes. let_each ACTSENSORS = sensor(WifiActivity,ALL) in let_each PTZSENSORS = sensor(ptz_image,ALL) in let_once ALERT = DetectWifiAlert(sensor(WifiAlert,ALL)) in level_trigger( not(isNull(ALERT)), PTZSnapshot( PTZLocate(QueryWifiAlert(ALERT."MAC",ACTSENSORS)), PTZSENSORS)))
8 Future Work and Conclusions Wireless Access Lists from Physical Data With this infrastructure in place, programs may use information detected on the physical plane to (re-)configure the wireless network. For example, an embedded camera network and face detection Opcodes can be used to detect the identities of individuals entering or leaving as a trigger to enable the detected user’s wireless MAC address for service in a physical space.Put simply, when we see Jane enter the lab we want to enable Jane’s MAC address (added to the whitelist), and disable her MAC address when she leaves the lab. A dynamic whitelist would make it more difficult for a malicious user to abuse unused, authorized wireless MAC addresses for great lengths of time. Modification of the WLAN’s access control list in this way assumes the presence of a MAC whitelist; such an implementation is straightforward on OpenWRT enabled APs, using a CGI script to modify the device’s configuration. In addition, other physical sensors could be used in tandem with face detection as the trigger predicate in this expression; e.g., biometric sensors, magnetic card or RFID readers.
12
Virtualization and Programming Support for Video Sensor Networks
191
SN B ENCH as a Complete, Turn-Key Network Security Solution The Network Security provided by SN B ENCH need not be limited to Layer-2 alone. Integrating Layer-3 detection (e.g., Snort) as a sensor would enable the detection of misuse from IP contents that could be used to drive isolation or removal responses at Layer-2. Including port scanning or other fingerprinting tools as sensors could increase the accuracy of user identification thus further open the possibilities for more “severe” automated response. Ideally we could imagine migrating our own campus IT departments to use SN B ENCH as their Network Security and Intrusion suite, a transition that could be eased by the development of a declarative/rule-oriented domain-specific language (and corresponding STEP compiler) that is similar to existing network rule specification languages. Finally, our work on lightweight Sensor eXecution Environments for embedded devices could be used to run the SXE directly on OpenWRT enabled APs to provide SN B ENCH as a turn-key solution for Wireless Network Security services.
In Conclusion Network Security (specifically, wireless security) is not a problem that exists in a vacuum detached from the physical space in which the network is deployed. We promote an approach that unifies physical site surveillance and network security under the umbrella of SN B ENCH—our general purpose sensing infrastructure. In that regard, we have demonstrated how SN B ENCH enables the rapid development and deployment of cross-modal security services. We have shown that with SN B ENCH (1) detection of wireless anomalies can be correlated with other sensory inputs providing reciprocal benefit to merging security on the physical and cyber planes, (2) detection and response services may be easily composed and modified without technical knowledge of the specific protocols or implementations of the underlying sensory tools, and (3) adding new intrusion detection tools as input or other devices for response is straightforward given SN B ENCH’s modular architecture. The illustrative example programs provided include the status quo (simple logging and email alerts) and hint at where we may go from here, in an attempt to spark the reader’s imagination to consider what sensors, actuators and new hybrid services may be enabled by the SN B ENCH platform.
References 1. Adelstein, F., Alla, P., Joyce, R., Richard, G.G. III: Physically locating wireless intruders. In: ITCC ’04: Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04), Washington, DC, USA, vol. 2, p. 482. IEEE Computer Society, Los Alamitos (2004) 2. AirDefense, Inc.: AirDefense Enterprise Product Homepage, http://www.airdefense.net/ products/enterprise.php 3. Bahl, P., Padmanabhan, V.N.: RADAR: An in-building RF-based user location and tracking system. In: INFOCOM (2), pp. 775–784 (2000) 4. Bellardo, J., Savage, S.: 802.11 denial-of-service attacks: real vulnerabilities and practical solutions. In: SSYM’03: Proceedings of the 12th Conference on USENIX Security Symposium (Berkeley, CA, USA), p. 2. USENIX Association, Berkeley (2003)
192
A. Bestavros and M.J. Ocean
5. Bestavros, A., Bradley, A., Kfoury, A., Ocean, M.: SNBENCH: a development and run-time platform for rapid deployment of sensor network applications. In: IEEE International Workshop on Broadband Advanced Sensor Networks (Basenets), October (2005) 6. Devine, C.: Aircrack-ng homepage, http://www.aircrack-ng.org/ 7. Farshchi, J.: Wireless intrusion detection systems, http://www.securityfocus.com/infocus/ 1742, 2003-11-05 8. Kershaw, M.: Kismet (version 2007-01-r1b), http://www.kismetwireless.net/documentation. shtml 9. Ocean, M.J., Bestavros, A., Kfoury, A.J.: SNBENCH: programming and virtualization framework for distributed multitasking sensor networks. In: VEE ’06: Proceedings of the 2nd International Conference on Virtual Execution Environments (New York, NY, USA), pp. 89–99. ACM Press, New York (2006) 10. OpenWRT Project Homepage, http://openwrt.org/ 11. Roesch, M.: Snort—lightweight intrusion detection for networks. In: LISA ’99: Proceedings of the 13th USENIX Conference on System Administration (Berkeley, CA, USA), pp. 229– 238. USENIX Association, Berkeley (1999) 12. True, N.: Wi-viz: Wireless network environment visualization, http://devices.natetrue. com/wiviz/ 13. Vigna, G., Valeur, F., Kemmerer, R.A.: Designing and implementing a family of intrusion detection systems. SIGSOFT Softw. Eng. Notes 28(5), 88–97 (2003) 14. Vandoorselaere, Y., et al.: Prelude Hybrid IDS, http://www.prelude-ids.org/ 15. Youssef, M., Agrawala, A., Shankar, U.: WLAN Location Determination via Clustering and Probability Distributions (March 2003)
Chapter 13
Simulating Human Activities for Synthetic Inputs to Sensor Systems Jan M. Allbeck and Norman I. Badler
Abstract We are developing human activity simulations that could be used to test distributed video sensor networks. Our ultimate goals are to build statistical models of pedestrian density and flows at a number of urban locations and to correlate those flows with population movement and density models represented in a spatiotemporal modeling system. In order to create known populace flows, we have built a virtual populace simulation system, called CAROSA, which permits the authoring of functional crowds of people going about role-, context-, and schedule-dependent activities. The capabilities and authoring tools for these functional crowd simulations are described with the intention of readily creating ground truth data for distributed sensor system design and evaluation.
Keywords Crowd simulation · Virtual agents · Human activity · Computer animation
1 Overview Current urban populace modeling lacks appropriate computational methods to realistically represent complex dynamic interactions between the built environment, cultural artifacts, and population behavior. Our objective is to develop new methods to rapidly create a model of pedestrian flows correlated with appropriate spatiotemporal environment models. Such models admit both virtual simulation and J.M. Allbeck () George Mason University, 4400 University Dr., MSN 4A5, Fairfax, VA 22030, USA e-mail:
[email protected] N.I. Badler University of Pennsylvania, 3300 Walnut St., Philadelphia, PA 19104-6389, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_13, © Springer-Verlag London Limited 2011
193
194
J.M. Allbeck and N.I. Badler
understanding of a real world urban setting wherein individuals engage in culturally realistic and plausible behaviors and movements. These models also admit anomaly detection where pedestrian flows (or lack thereof) would be perceived as unusual or otherwise significant occurrences, perhaps alerting and directing the attention of human operators or other automated observers. Our project entails the creation of a computer software system for modeling, authoring, and controlling an urban populace. Our approach is synthetic in that we are developing computational simulation tools, but it is also experimental because we can use the simulation to model known conditions and validate its predictions. It is also general in that the developed methodology and system can be applied to a variety of urban terrain environments and human terrain conditions. This chapter is structured as follows. In the next section we will discuss some related crowd simulation work. We will then give an overview of the CAROSA (Crowds with Aleatoric, Reactive, Opportunistic, and Scheduled Actions) populace simulation framework, followed by some key components of the framework and how they enable easily authored situated, contextual populations. We conclude with future directions for this work in the context of distributed sensor systems.
2 The CAROSA System Distributed video sensing of human activities in an urban setting requires a fundamental understanding of what human behaviors are likely, normal, or anomalous in such an environment. Since it is difficult and costly to run actual actor-based situations (e.g., as portrayed in the movie The Truman Show), computer simulations can often provide the only realistic route for synthetic urban populace stimuli. Large-scale simulations are frequently done macroscopically, with limited models and graphics of the urban inhabitants and the architectural context, and thus fail to give a personal view of urban life, which is crucial for appropriate sensing of and responses to observed behaviors. While many computer graphics research groups are producing animated crowds of hundreds and even thousands of people, often the agents wander (apparently aimlessly) over the traversable areas. Urban environments are complex dynamic places, populated by individuals with roles, goals, schedules, and diverse but not entirely random behaviors that are driven by many forces: social, spatial, personal, and immediate needs. Urban movements are semi-structured and rather mundane, and lack the time-critical focus of panic and evacuation. People work, play, visit, shop and loiter. We have developed the CAROSA system [20] to cover a broad range of relevant issues related to this basic axiom, concentrating on both low level individual motions and high level selection and control of individual behaviors predicated on scheduled, reactive, opportunistic and stochastic actions. Actions are tied to semantic data associated with environments (e.g., buildings or rooms), thus providing flexibility for agents to function without fixing them to spatial locations or animation scripts [27]. Human schedules interact with spaces, work and other cycles [19]; computer graphics simulations mostly lack these socio-cultural and functional aspects of human behavior.
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
195
The creation of these heterogeneous populations with contextual behaviors needs to be feasible and, even better, based on empirical evidence. We do not want to demand that scenario authors be expert programmers. Likewise, defining these populations should not be an arduous, never-ending process requiring levels of detail beyond what is immediately important to the scenario author. Especially in a distributed sensor simulation application, spatiotemporally structured randomness may be a distinguishing characteristic required of a crowd synthesizer.
3 Related Work Many movies and games use software such as Massive™ to build background characters [15]. Massive™ provides mechanisms for creating and executing rules that govern the character behaviors that are replicated for large groups. While creating and refining these rules still takes time and skill, the software makes construction of relatively homogeneous crowds (with some statistical variations) much easier. While this is well suited to battle scenes where interactions are limited, scenes that require functional, contextual characters are not feasible. In many games and simulations, populations of non-player characters (NPCs) are programmed to follow a path and perhaps perform a few behaviors. They might also react to a certain set of stimuli, but they do not generally interact with objects in the environment. They lack context, often only existing when in the player’s field of view [3]. Crowd simulations based on Helbing’s empirical Social Forces model [9] use repulsion and tangential forces to simulate interactions between people and obstacles, pushing behaviors, and flow rates. More realistic human movement in low- and medium-density crowds can be obtained using rule-based models [24]. Cellularautomata models [6] are fast and simple to implement, but disallow contact between agents by explicitly prohibiting it. McDonnell et al. study the perception of character appearance and motion variations to enable the portrayal of more visibly interesting, reasonable populations [16]. Lerner et al. use real world data to fit behaviors into pedestrian simulations [13]. While these approaches have achieved visually interesting populations, the characters still lack larger purpose and context. Other researchers have created variations in behavior contextually by defining regions in the environment where certain behaviors would be displayed [7, 12, 26]. For example, a theater is labeled with sit and watch behaviors. However, within these regions behaviors are uniform. Composite agents, on the other hand, use agent proxies to differentiate behaviors and reactions based on factors such as character priority or authority [31]. Others have also explored using social or psychological factors to vary behaviors; e.g., Thalmann et al. included two states for the characters, SOCIAL and HUNGRY [28]. Pelechano and Badler’s work included leaders and followers as well as trained and untrained agents, but behavioral variants were limited to way-finding [22]. Both reactive and deliberative actions have been implemented [25] and decision networks have been used for action selection [32], though they require crafting the prior probabilities of each action in context.
196
J.M. Allbeck and N.I. Badler
Fig. 1 The CAROSA framework
Our CAROSA framework includes many components common to simulators (see Fig. 1). Graphics models are displayed using OpenGL, Cal3D [5], or Ogre [18]. We are also using HiDAC+MACES as an underlying crowd simulator [21]. HiDAC+MACES navigates characters from one location to another while avoiding collisions and provides messages when objects and other agents are perceived. Calls to playback animation clips also filter through HiDAC+MACES to the graphics figures. CAROSA’s unique combination of components include an action and object repository called the Actionary, Agent Processes, a Resource Manager, and a Scheduler. Scenario authors can use the Scheduler to schedule actions found in the Actionary and link them to locations or specific objects. These actions are also associated with the agents that are to perform them. Agent Processes receive and process these actions, using the Resource Manager to allocate or bind any object parameters needed by the actions. The next few sections will describe these components and how they facilitate the creation of functional, heterogeneous populations.
4 Parameterized Representations CAROSA uses the PAR (Parameterized Action Representation) system [2] for both parameterized actions and objects stored in a persistent hierarchic database: the Actionary. The Actionary currently contains more than 60 actions and nearly 100 object types. PARs use preparatory specifications that provide simple backward chain-
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
197
ing, which greatly reduces the scenario-authoring burden. For example, most instantiated actions are tied to an object participant (e.g., sitting in a chair, using a computer, eating food, drinking coffee, etc.), and these actions have a preparatory specification that makes the agent walk to the associated object before the action can be performed. General action definitions reference object class types as object participant parameters; e.g., a general definition of Sit includes Person-Supporting-Furniture. Similar to Smart Objects [11], PAR objects also include parameters that further specify how agents interact with them: e.g., where a character should stand to operate the object or how to grasp it. Functional characters can thus behave appropriately in context. CAROSA extends the PAR representations to include four action types: scheduled, reactive, opportunistic and aleatoric. Scheduled activities arise from specified roles for individuals or groups and give agents purpose and a storyline. For example, the scenario author may schedule a meeting between two characters to show that they have an association. Purely scheduled actions alone would result in simulations that are too structured or robotic. Reactive actions are triggered by contextual events or environmental constraints. Many of these behaviors arise from the underlying crowd simulator: e.g., an agent altering its heading or slowing to avoid collisions. Reactive actions, such as acknowledging someone as they pass by, are not handled by the crowd simulator. These reactions are specified and recognized in a rule-based Attention Mechanism. Reactive actions help to add life to simulations. Perceiving and interacting with a dynamic environment indicates that the characters are not focused solely on achieving a specific goal; they are also aware of the world around them. Opportunistic actions arise from explicit goals and priorities. These needfulfilling behaviors are akin to the hill-climbing behavior choices of characters in the popular game The Sims [29]. While our opportunistic actions are similar, the implementation is different. In The Sims current proximity to resources is heavily weighted when choosing an action, and time is ignored. We use both time and future proximities. For example, a character may be working in his office and have a nonemergent energy need and a meeting to attend in a few minutes. The character could then attempt to address the need by stopping by the lunch room for a cup of coffee on the way to the meeting. This requires them to leave a couple of minutes early and to know that the lunch room has proximity to the path to the meeting room. Finding an object resource is essentially done through a depth-first search where the depth limit is based on the need level. As the need level increases, so does the distance the agent is willing to go out of their way to fulfill the need. Like scheduled and reactive actions, opportunistic actions lead to the perception of functional characters. Aleatoric actions are random but structured by choices, distributions, or parametric variations. The aleatoric quality arises from frequency of occurrence of subactions. We could use, e.g., the Bureau of Labor Statistics (BLS) American Time Use Survey, which encodes time distributions of many home, work, and leisure activities. Aleatoric behaviors are designed to provide reasonable variations in behavior without overt scheduling. Working in an office might include using a computer,
198
J.M. Allbeck and N.I. Badler
speaking on a telephone, and filing papers. For many scenarios the exact timing of each sub-action is not important. The overarching WorkInOffice action should just look plausible.
5 Resource Management Creating functional populations involves associating actions with related objects. Making these associations by hand could overwhelm a simulation author. We implemented a Resource Manager that automatically allocates objects. Resources are initialized at the onset of the simulation where the building1 is defined. Object locations are set as the room that they are placed in and likewise the objects are listed as contents in those room objects. These relationships are then used to initiate resource management. There are three different methods for allocating resources from a resource group. One method just allocates any available resource from the resource group to the agent. Another method specifies an object to be allocated. The last method specifies a preference function to determine the best object to allocate to the agent. Objects are not allocated to agents until both are in the same location or room. This leads to natural behaviors including failures: an agent must enter a room to know that there are insufficient chairs.
6 Roles and Groups People’s functions or purposes through the course of a day are highly correlated to their roles: students attend classes, professors work in their offices and lecture, housekeepers clean, researchers research, etc. Creating roles for simulated characters provides them focus and purpose and also creates simulations of heterogeneous populations. Defining groups of characters with common roles means reduced work for simulation authors. When defining a role one can specify objects that the role should possess: e.g., a professor should have an office. An author does not need to specify an instance of an object, though they are permitted; it is sufficient to specify a class like office. Possessions are transitive: if you have an office, you also have the objects in the office. When a role has a possession specified, every character with that role is allocated (by the Resource Manager) a possession of that type during the initialization of the simulation. Whenever a character initiates the performance of her default behavior, she first checks to see if an object of the type needed is located in her possessions. If it is, she uses that object. If it is not, the next method is tried. 1 Work
is in progress to extend this scheme to large exterior environments; the modeling principles remain the same.
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
199
This method also uses the association of object types with default actions. For example, researchers in our scenario need laboratory desks in order to do their research. As we did with professors, we could say that all researchers should possess a lab desk, but in a university setting where resources are limited it might be more likely for researchers to take whatever desk is available. We do this by indicating that the action research requires an object participant of the type LabDesk. When a researcher initiates this default action, an object of type LabDesk is allocated to her. If no object can be allocated, a failure is produced and the action will not be performed. This might lead to the character performing a different action or just standing still (or idling as a default action) until an appropriate object can be allocated. A key aspect of CAROSA is facilitating the creation of heterogeneous populations. To achieve this, we need to provide a scenario author a way to define groups as well as individuals. An author can name a group and provide the number of members in it, or an author can create individual characters and assign them to groups. In both cases, the information is stored in the Actionary. As needed, agents are created to fulfill the indicated membership numbers. If the name of a group happens to correspond to the name of a role, then all of the members of the group are automatically assigned that role and inherit all of the default behaviors and specifications for it. There can be more than one group per role and there may be groups that do not indicate a role. An author may want to do this for a couple of reasons. First, groups can be assigned actions, so there may be a group of students that have class at the same time and can be scheduled as a group. There might also be meetings that group members of various roles attend, such as project meetings. When a group action is specified, it is simply copied and placed on the action queue of each member in the group. Normally, a location for the action is given as well as the type of object participant. The Resource Manager then allocates the needed object resources as each participant arrives at the location. Similar to real world email lists, creating groups that correspond to meaningful collections of characters enables group events to be quickly and easily scheduled. Naturally, scheduling conflicts can and do occur just as in the real world. The CAROSA framework resolves these conflicts seamlessly based on the actions’ priorities as specified in the PARs. If the actions have the same priority, the first one on the queue will be performed. Another reason for creating more than one group per role and groups that do not correspond to roles is plausible group reactions. Reactive actions can both be assigned to groups and groups can be the stimulus of reactive actions. We are not the first to consider roles definitions for virtual characters [10, 23]. Our efforts, however, focus less on the social interactions of the characters. For our work, roles are vital to facilitating the creation of functional, heterogeneous populations. Likewise, groups have been a part of virtual character research [4, 12, 17, 28]. For the most part, these previous efforts have focused on collision detection and dynamic group behaviors. In the CAROSA framework, groups generally have common purposes or functions. They may or may not gather during a simulation. For example, a simulation may contain a group of housekeepers or maintenance
200
J.M. Allbeck and N.I. Badler
workers to clean and maintain the building, but these characters may never gather into a visual group.
7 Scenario Authoring Beyond creating functional, heterogeneous animated crowds, our goal for the CAROSA framework was to facilitate scenario authoring by non-programmers. Certainly to simulate a virtual environment, one will still require models and animations, but once a repository of these items has been created and a populated Actionary obtained and linked to them, a Subject Matter Expert (SME) should be able to use these building blocks to construct a simulation. When scheduling actions, a SME only needs to specify the PAR action that should be performed, which character or group of characters should perform it, what objects or locations might participate in the action, and when it should be performed. In fact, the actions can either be simple PARs or complex lists of PARs composed together. A specific object participant can be specified, such as sit in Chair_4 or a location can be given, such as sit in ClassRoom_1. If a location is specified, a chair instance will be allocated by the Resource Manager when the character arrives in the specified location. If the required resource cannot be allocated, a failure is reported to the Agent Process and the action is removed from the queue. In the future, a more powerful planner could be used to attempt to acquire the resource needed. We are currently using the calendars of Microsoft Outlook® as a scheduling interface. Calendars can be created for groups of characters or individuals. Activities, locations, and times are then simply entered on the calendars. The GeniusConnect [8] software package then links these calendars to tables in the Actionary MySQL database. Activities are read from the database and assigned to the appropriate characters by placing them on the appropriate Action Queues of the Agent Processes (See Fig. 1). We are also using the interfaces of Microsoft Outlook® to enable SME’s to create groups and define roles. These definitions are created through Microsoft Outlook® ’s Contacts and Tasks panels. We have defined a Quantity field within the Contacts panel that can be used to specify the number of members of each group. Required possessions can be entered in the Office field. Default behaviors are associated with roles through the Tasks panel, simply by entering the action name in the Subject field and the role in the Contacts field. We could also create macros and other interface elements to streamline the process. We have constructed simple custom Graphical User Interfaces (GUI’s) for authoring reactive, opportunistic, and aleatoric actions. These GUI’s are directly connected to the Actionary. The drop-down lists are populated from table entries in the database and submitting new actions of these types writes directly to database tables. These newly created actions can then also be referenced from Microsoft Outlook® , e.g., to schedule aleatoric actions. The PAR representation of objects and actions provide semantic foundations that are referenced when authoring scenarios through these interfaces. The Resource
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
201
Manager provides a means of filling in information and tying the simulation to the environment. This allows a SME to concentrate on aspects of the simulation that are directly relevant to a scenario including the heterogeneity and functionality of the population.
8 Example Simulation As an initial test-bed for the CAROSA framework, we simulated a university building environment. The environment is based on one floor of an engineering building at a university. It includes laboratories, classrooms, hallways, offices, a restroom, and a lounge (See Fig. 2). The occupants of the building include professors, students, researchers, administrators, housekeepers, and maintenance personnel. Actions in this environment include working in an office, lecturing, attending class, lounging, cleaning, inspecting, researching, waving, eating, drinking, going to the restroom, picking up objects, as well as others. There is also collision-free locomotion. Characters in the simulation adhere to their schedules as created by the scenario author through the Scheduler, but they also greet each other as they pass by and attend to their needs through opportunistic actions. If a portion of their day is unscheduled, the characters revert to their default behaviors, which in many cases are aleatoric actions that provide ongoing reasonable behavior. We ran many simulations of this environment and noted several emergent behaviors. For example, students tended to gather in the same areas because of the resources available and therefore tended to walk to classes together. Furthermore when groups were instructed to react to other group members by waving, students would greet each other before a class, which seems like reasonable behavior. Students also tended to go directly for food and coffee after classes. Because need levels
Fig. 2 Sample scenario based on activity in a university environment
202
J.M. Allbeck and N.I. Badler
are currently all checked together, needs tend to be fulfilled at the same time. For example, a character might go to the restroom and then to get coffee. Again this emergent behavior seems quite plausible.
9 CAROSA Summary The CAROSA framework facilitates the creation and simulation of functional, heterogeneous populations. Characters in our simulations do more than navigate from one location to another without colliding with each other or obstacles: characters have roles that provide them purpose. Their behaviors are performed in proper context with the object participants. In fact, because actions reference PAR objects and not coordinates directly, the environmental setting of the simulation can be completely rearranged without requiring additional work from the simulation author. Objects can even be added or removed and the simulation will still run, although some actions may not be performed if there is a shortage of resources. Through the use of the PAR representations, a resource manager, and definitions of roles and groups, we have created a framework in which actions can be scheduled in forms analogous to the scheduling of real people. To demonstrate this, we have connected the CAROSA framework to Microsoft Outlook® . Through this widely used interface for scheduling real human and group activities, a subject matter expert can author simulations. To add additional richness and reasonable behaviors to simulations, we have also created simple custom GUI’s for authoring reactive, opportunistic, and aleatoric actions. Additional development is still needed for the CAROSA framework. In particular, we would like to increase the scale of the populations. Our university simulation can run 30 characters in real-time on a standard PC. The most costly algorithm in the CAROSA framework is the calculation of opportunistic actions. Scheduling the fulfillment of a need can require searching through time for a gap in the character’s schedule as well as through the environment to find a resource near a path the character will be traveling. We are considering caching these paths or locations or perhaps using stored waypoints that act as road signs for resources. We would also like to explore running each character of a simulation on a different processor with one additional processor used by the underlying HiDAC+MACES crowd simulator or another engine to handle collision detection. Fortunately in a simulated sensor input setting, real-time input is not as important a requirement as reasonable populace activities. Further, we are currently working on tying the CAROSA framework to other underlying engines such as the game engines Ogre and Dark GDK. Such engines would provide us with better computer graphics and action animations. Ideally, we would like the underlying animation engine to include inverse kinematics for the characters, allowing the PAR representation to drive more detailed human/object interactions. This would require additional semantic labeling of the objects in the environment (i.e., creation of sites), but, as objects can be reused in many simulations, the cost would grow linearly or slower with the size of the urban area being modeled.
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
203
10 Input to Distributed Sensor Networks Ultimately the input media for this project should be multiple real-time video data streams from a distributed suite of surveillance cameras, of both fixed and movable types. If fixed, their locations are known in advance and can be readily geo-located in a digital urban map. If movable, their present position could be sensed and reported via GPS and compass readings. CAROSA’s use as a simulator for distributed sensor system input has the advantage that the environment may be readily compartmentalized (e.g., into rooms) or into separated buildings. By positioning fixed or dynamic synthetic cameras in various desired locales within the 3D space, each camera will perceive only the activities within its local view. Meanwhile CAROSA simulates the entire environment (globally) with structure and consistency in character activities across the various spaces. By synthesizing populace motions and behaviors, we can both control the behaviors in the overall model, and have known ground truth against which to compare and evaluate sensor output and analysis results. Since we are still in the early stages of our work CAROSA allows us to bypass the important but difficult question of actually doing real-time video pedestrian tracking in a video stream in favor of the more global scale problem of building the requisite computational semantic structures in which such videos can be interpreted. In addition, obtaining unrehearsed, public, and urban surveillance video is a non-trivial undertaking. Using CAROSA, we can simulate a populace and base input features to be measured and descriptions produced (e.g., [1]) on flows rather than tracking individuals in crowds. The latter is a hard problem (e.g., [14, 30]), but we feel that this level of detail is not necessary for the accomplishment of our goals. In addition, an ability to track one person in a video sequence provides no explicit connection or correlation to any other people that may appear in other video streams; i.e., our interest is not in fusing tracks of individuals but rather of observing whether global and distributed pedestrian motions are consistent with spatiotemporal models of expected pedestrian flows.
11 Summary The principal challenge in creating a CAROSA simulation and obtaining the corresponding spatiotemporal features is data collection and organization for people, places, and events. The data needed to seed this project should be public and local to our environment, so that we may both measure and predict the cost of scaling data collection to a different and larger urban area. We have access to, and can observe, e.g., typical student schedules, room locations, and work patterns at the University of Pennsylvania, so these could be a starting point for our databases. This process protects individual privacy while providing us with an external verifiable measure of simulation viability. We have described a framework for building a baseline system that creates simulation-generated sensor data from a dynamic spatiotemporal model. We believe this baseline can be extended to use actual computer vision sensor data from
204
J.M. Allbeck and N.I. Badler
real cameras. This framework could be a major step toward an ultimate goal of a real-time predictive model of urban human behaviors that can supply both intelligence and geospatial reasoning to military and civil needs. Acknowledgements Partial support for this effort is gratefully acknowledged from the U.S. Army “SUBTLE” MURI, a Lockheed-Martin Corporation Strategic Technology Thread Grant, and George Mason University. We also appreciate donations from Autodesk and nVidia. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
References 1. Allbeck, J., Kipper, K., Adams, C., Schuler, W., Zoubanova, E., Badler, N., Palmer, M., Joshi, A.: ACUMEN: Amplifying control and understanding of multiple entities. In: Autonomous Agents and Multi-Agent Systems, pp. 191–198 (2002) 2. Bindiganavale, R.: Building parameterized action representations from observation. Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA (2000) 3. Brockington, M.: Level-of-detail AI for a large role-playing game. In: Rabin, S. (ed.) AI Game Programming Wisdom, pp. 419–425. Charles River Media, Inc., Hingham (2002) 4. Brogan, D., Hodgins, J.: Group behaviors for systems with significant dynamics. Auton. Robots 4, 137–153 5. CAL3D. 3D Character Animation Library: http://home.gna.org/cal3d/, Last visited May 2009 6. Chenney, S.: Flow tiles. In: ACM SIGGRAPH/ Eurographics Proceedings of Symposium on Computer Animation, pp. 233–242 (2004) 7. DePaiva, D.C., Vieira, R., Musse, S.R.: Ontology-based crowd simulation for normal life situations. In: Proceedings of Computer Graphics International, Stony Brook, NY, 2005, pp. 221– 226. IEEE, New York (2005) 8. GeniusConnect: GeniusConnect. http://www.geniusconnect.com/articles/Products/2/3/, Last visited May 2009 9. Helbing, D., Farkas, I., Vicsek, T.: Simulating dynamical features of escape panic. Nature 407, 487–490 10. Isbister, K., Hayes-Roth, B.: Social implications of using synthetic characters: an examination of a role-specific intelligent agent. Knowledge Systems Laboratory, Stanford University, Stanford, CA (1998) 11. Kallmann, M., Thalmann, D.: Direct 3D Interaction with smart objects. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, London, United Kingdom, 1999, pp. 124–130. ACM, New York (1999) 12. Lee, K.H., Choi, M.G., Hong, Q., Lee, J.: Group behavior from video: A data-driven approach to crowd simulation. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, San Diego, 2007, pp. 109–118 (2007) 13. Lerner, A., Fitusi, E., Chrysanthou, Y., Cohen-Or, D.: Fitting behaviors to pedestrian simulations. In: Symposium on Computer Animation, New Orleans, LA, 2009, pp. 199–208. ACM, New York (2009) 14. Ma, Y., Cisar, P., Kembhavi, A.: Motion segmentation and activity representation in crowds. Int. J. Imaging Syst. Technol. 19(2), 80–90 15. MASSIVE_SOFTWARE_Inc.: 3D animation system for crowd-related visual effects, http://www.massivesoftware.com, Last visited Oct. 2009 16. McDonnell, R., Micheal, L., Hernandez, B., Rudomin, I., O’Sullivan, C.: Eye-catching crowds: saliency based selective variation. In: ACM SIGGRAPH, New Orleans, Louisiana, 2009, pp. 1–10. ACM, New York (2009)
13
Simulating Human Activities for Synthetic Inputs to Sensor Systems
205
17. O’Sullivan, C., Cassell, J., Vilhjalmsson, H., Dobbyn, S., Peters, C., Leeson, W., Giang, T., Dingliana, J.: Crowd and group simulation with levels of detail for geometry, motion and behavior. In: Third Irish Workshop on Computer Graphics (2002) 18. Orge: http://www.ogre3d.org/, Last visited January 2010 19. Parkes, D., Thrift, N.: Times, Space, and Places: A Chronogeographic Perspective. Wiley, Chichester (1980) 20. Pelechano, N., Allbeck, J., Badler, N.: Virtual Crowds: Methods, Simulation, and Control. Morgan and Claypool Publishers, San Rafael (2008) 21. Pelechano, N., Allbeck, J.M., Badler, N.I.: Controlling individual agents in high-density crowd simulation. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA, San Diego, CA, 2007. ACM Press, New York (2007) 22. Pelechano, N., Badler, N.: Modeling crowd and trained leader behavior during building evacuation. IEEE Comput. Graph. Appl. 26(6), 80–86 23. Prendinger, H., Ishizuka, M.: Social role awareness in animated agents. In: 5th International Conference on Autonomous Agents, pp. 270–277. ACM Press, New York (2001) 24. Reynolds, C.: Flocks, herds, and schools: A distributed behavior model. In: Proceedings of ACM SIGGRAPH, pp. 25–34 (1987) 25. Shao, W., Terzopoulos, D.: Autonomous pedestrians. In: Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (Los Angeles, California, 2005), pp. 19–28. ACM Press, New York (2005) 26. Sung, M., Gleicher, M., Chenney, S.: Scalable behaviors for crowd simulation. Comput. Graph. Forum 23(3), 519–528 27. Thalmann, D., Musse, S.R.: Crowd Simulation. Springer, Berlin (2007) 28. Thalmann, D., Musse, S.R., Kallmann, M.: Virtual humans’ behavior: individuals, groups, and crowds. In: Proceedings of Digital Media Futures, pp. 13–15 (1999) 29. Wright, W.: The Sims. Electronic Arts, 1st edn. (2000) 30. Ya-Dong, W., Jian-Kang, W., Ashraf, A.K., Wei-Min, H.: Tracking a variable number of human groups in video using probability hypothesis density. In: Proceedings of the 18th International Conference on Pattern Recognition, vol. 03. IEEE Computer Society, Los Alamitos (2006) 31. Yeh, H., Curtis, S., Patil, S., van den Berg, J., Manocha, D., Lin, M.: Composite agents. In: Proceedings of Symposium on Computer Animation, Dublin, Ireland, 2008. Eurographics Association, Zurich (2008) 32. Yu, Q., Terzopoulos, D.: A decision network framework for the behavioral animation of virtual humans. In: Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation, San Diego, California, 2007, pp. 119–128. Eurographics Association, Zurich (2007)
Chapter 14
Cognitive Sensor Networks A Domain Theoretic Cognitive Architecture Thomas C. Henderson
Abstract Our overall goal is to develop a cognitive architecture which will allow autonomous and robust operation of sensor–actuator networks. To achieve this, the perception, concept formation, action cycle will be informed by domain theories of signal analysis, physical phenomena, and behavior. Example scenarios include cognitive vehicles and buildings in which the system understands itself and the activities in and around it by means of distributed video and other sensors. This includes discovery of the cognitive system’s own sensing and actuation capabilities.
Keywords Cognition · Domain theories
1 Introduction Cognitive sensor networks are able to perceive, learn, reason and act by means of a distributed, sensor–actuator, computation and communication system. In animals, cognitive capabilities do not arise from a tabula rasa, but they are due in large part to the intrinsic architecture (genetics) of the animal, which has evolved over a long period of time and depends on a combination of constraints: e.g., ingest nutrients, avoid death, etc. We have previously shown how organism morphology arises from genetic algorithms responding to such constraints [5]. Kuipers [11] proposed SODA (Self-Organizing Distinctive-State Abstraction) and mechanisms which allow an agent to learn spatial knowledge from unsupervised sensorimotor experience, but he argues that innate theories should not be used in this process; there is a statement that previous work [12] used certain background knowledge about sensors and control that should be excluded; also note that the knowledge was domain independent. T.C. Henderson () School of Computing, University of Utah, Salt Lake City, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_14, © Springer-Verlag London Limited 2011
207
208
T.C. Henderson
Recently, it has been suggested that abstract theories relevant to specific cognitive domains are likewise genetically coded in humans (e.g., language, physics of motion, logic, etc.); thus, these theories and models are abstracted from experience over time. We call this the Domain Theory Hypothesis. Some advantages of such embedded theories are that they (1) make perception and learning more efficient, (2) allow generalization across models, and (3) allow determination of true statements about the world beyond those available from direct experience. We have shown in previous work how theories of symmetry can dramatically improve representational efficiency and aid reinforcement learning on various problems [8–10]. However, it remains to be shown how sensory data can be organized into appropriate elements so as to produce a model of a given theory. We address this here by studying how a selected set of domain theories can be used to improve perception, concept formation, and action performance, as well as to enable more effective learning. We will also explore the role this plays in a cognitive system’s ability to discover knowledge about its own structure as well as about the surrounding physical world. Our view is that cognitive sensor networks which can learn these things will not need to be pre-programmed in detail.
2 Cognition The development of effective mental abilities for cognitive systems is a longstanding goal of the AI and intelligent systems communities. The major approaches are the cognitivist (physical symbol systems) and emergent (dynamical systems) paradigms. For a detailed review of the relevant characteristics of cognitive systems and how these two approaches differ, see [19]. Basically, cognitivists maintain that patterns of symbol tokens are manipulated syntactically, and through perceptsymbol associations perception is achieved as abstract symbol representations and actions are causal consequences of symbol manipulation. In contrast, emergent systems are concurrent, self-organizing networks with a global system state representation which is semantically grounded through skill construction where perception is a response to system perturbation and action is a perturbation of the environment by the system. The emergent approach searches the space of closed-loop controllers to build higher-level behavior sequences out of lower ones so as to allow a broader set of affordances in terms of the sensorimotor data stream. We propose to combine these approaches in order to exploit abstraction and specific cognitive domain theories to overcome sensor data analysis complexity. Our specific hypothesis is the following. The Domain Theory Hypothesis Semantic cognitive content may be effectively discovered by restricting sensor–actuator solutions to be models of specific domain theories intrinsic to the cognitive architecture. Sloman [17, 18] argues for this from an artificial intelligence point of view, while Chomsky [1] and Pinker [13] have explored universal structures for human natural
14
Cognitive Sensor Networks
209
Fig. 1 Domain theoretic cognitive architecture
language, and Rosenberg [15] explores the genetic evidence for cooperative behavior among humans. We proposed a framework for intrinsic sensor–actuator behaviors [3], and recently have studied the hypothesis in the context of some standard AI and robotics problems [8]. In particular, we considered there the role that a theory of symmetry can play in various learning scenarios. When symmetry can be exploited in reinforcement learning, the time to learn the solution to the task should be proportional to the size of the set of asymmetric states (note that this may be characterized in terms of the quotient space of the associated group where it exists). Figure 1 shows the Domain Theoretic Cognitive Architecture. Pinker previously proposed a schema for innate computational modules in humans; he also outlined the following tests for possible computational modules (what we call theories) in humans and gives some examples ([13], pp. 436–438): (1) Does the theory help solve a problem that our ancestors faced in their environment (biological anthropology)? (2) When children solve problems for which mental modules exist, they should know things they have not been taught. (3) Neuroscience should discover that the brain tissue computing the problem has some kind of physiological cohesiveness (tissue or subsystem). Pinker also lists some possible modules, including: (1) intuitive mechanisms: knowledge of the motions, forces and deformations that objects undergo, (2) intuitive biology: understanding how plants and animals work, (3) number, and (4) mental maps for large territories. Here we explore theories of time, space, and motion in cognitive sensor networks.
2.1 Sensor Networks In previous work, we have developed many aspects of sensor networks: an objectoriented methodology called logical sensors for sensor networks [6], knowledge based multisensor systems, instrumented logical sensors [2], leadership protocols, simulation experiments, gradient calculation, reaction-diffusion patterns, and computational sensor networks [4, 7, 16]. Our most recent work on computational sensor networks emphasizes the exploitation of PDE models of the sensed phenomena (e.g., the heat equation), and showed that a solution to the mote localization problem, as well as estimates of sensor bias, etc., can be posed as inverse problems with
210
T.C. Henderson
the model equations. Thus, models are very important for sensor networks, and the current work expands by demonstrating how models can be discovered. A sensor network involves hardware, software, sensors, radios, and physical phenomena. In order to handle this complexity, models are exploited to allow abstract descriptions and a quantitative analysis of performance. For example, a sensor model usually describes the resolution, quantization, accuracy, and noise associated with the sensor data. R/F models describe a broadcast range, energy consumption, and probability of packet reception. Models of the physical phenomena range from PDE’s to finite-state automata. With such models, it is possible to determine the performance capabilities of a sensor network, as well as to adapt the sensing, computation and communication to achieve more efficient operation. The major drawbacks of this approach include: (1) models must be constructed and exploited by human effort, and (2) models are usually static and may not be valid during execution. Thus, one of our goals is to develop methods to address these issues allowing the sensor network to become more cognitive by allowing it to: (1) learn models from its own observations, and (2) validate models during operation by comparing model predictions to sensed data. To achieve these goals, that is, to achieve a cognitive sensor network, we propose to provide the sensor network with: (1) representations of domain theories and inference rules for the theories, (2) a perceptual mechanism to determine elements of a set and operators on those elements, (3) a mechanism to determine that the set and its operators are a model of an domain theory, and (4) mechanisms to allow the exploitation of the model in learning and belief construction. Henderson et al. [8] demonstrated the last point by assuming that a theory of symmetry was available, and showed that reinforcement learning was made much more efficient on the Towers of Hanoi and the Cart-Pole Balancing problems. Here we address the first three points by examining how domain theories can be used to perceive objects of interest to the sensor network.
2.2 Domain Theory-Based Perception Domain theories provide encapsulated knowledge about a specific topic. This includes an abstract description of the relevant relations between conceptual elements of the domain. This knowledge is used at two levels: (1) in the direct analysis of sensor data to extract perceptual elements (PELs), and (2) to organize sets of PELs to form a model of a theory. Domain theories can be used to identify PELs since conceptual element formation is concerned with the identification of equivalence classes of elements congruent in some sense. The most basic relationship is the selfsimilarity of an object over time which provides its coherence and identity. In order to be the same object at two different times, its essential properties must be the same at those times. This is the most fundamental form of symmetry: temporal invariance. Congruence of elements is routinely assessed by means of sensor data. Given a set of sensors, S, elements to be sensed, E, and environmental processes, P , the
14
Cognitive Sensor Networks
211
(real-valued) sensor data, d, is a function of S, E, and P : d : S × E × P → . In general, identification of an element of interest in the data requires solving an inverse problem in terms of S and P . Theories of S and P may allow for such a solution and motivates the domain theory approach to cognition. In designing artificial perception systems, it is useful to consider biological organization and development. Perception provides: (1) awareness of the elements of the environment through physical sensation, (2) physical sensation interpreted in the light of experience, and (3) direct or intuitive cognition. The first implies not just a steady stream of sensor data, but the extraction of relevant elements from the environment through active control (of attention) and sampling. The second signifies that experience should be cross-referenced temporally as well as across modalities. Finally, the third is in line with our view that cognition appeals to pre-existing categories and their relations; one hypothesis is that these have arisen over the course of evolution of the species.
3 Symmetry Theory in Signal Processing One essential example domain theory is symmetry theory. E.g., Popplestone and Grupen [14] develop a formal description of general transfer functions (GTF’s) and their symmetries. The basic idea is that a transfer function characterizes the input– output relationship of a system. This means that they are functionals which map from a specification of how the input to a system evolves over time to a specification of how its output evolves over time. These can be used by a cognitive sensor network to build models of its own structure. The feasibility of exploiting symmetry in sensor networks is grounded in: (1) hardware similarity among sensors, (2) inherent symmetries in 4D space-time, and (3) intrinsic symmetry in problem domains of sensor net applications. In a sensor net, many sensors have similar physical structure. This ensures sensors generate similar responses to similar inputs, hence making sensor-wise symmetry possible. We live in a world where 3D space is generally uniform. Physical laws applicable in one place are also applicable at other places. This ensures symmetry can be found in the temporal and spatial dimensions. Sensor nets observe and study meaningful phenomena. Patterns and regularities emerge from observed image data and make motion and stereo analysis possible. Let us now consider how this framework can be applied so that a sensor network can learn models of sensed phenomena as well as its own structure. Suppose that we have a set of sensor elements (SEL’s), and that each SEL is equipped with an intensity sensor and a microphone. Moreover, the sensor at each SEL produces sequences of data which can be collected and analyzed. Each sensor, S, is viewed as a map from the integers to the reals: S : I → (i.e., we assume that time steps are equal and just use the ordinals). Thus, operators on can be extended to act on sensor functions. Thanks to GTF theory we are allowed to treat the symmetry operators on sensor data as a group. We can restrict our attention to translations on the integers due to the time indexes.
212
T.C. Henderson
Assume we have theories for certain subgroups of SR , the symmetric group on (i.e., the set of all one-to-one mappings from to ). Let I () be the group of isometries on ; that is, I () is the subgroup of elements of SR which preserve distance: σ ∈ I () ⇔ (σ ∈ SR ) ∧ ∀a, b ∈ , d(a, b) = d(aσ, bσ ). If σ ∈ I (), then rσ = r + 0σ where = ±1. More specifically, rσ = r + a moves the real line a units to the right, while rσ = −r + a inverts about the origin and then translates a units to the right. In order to compare signals from different sensors, it is first necessary to synchronize them. This can be achieved by actively having the network emit a signal which arrives at essentially the same time at all sensors. This event can be used to set all “clocks” to zero. Alternatively signals can be exchanged as pairs of sequences: ({ti }, {xi }), i = 1, . . . , n. Assume the latter case. In order to synchronize a SEL Si with a SEL Sj , we define an operator T (X(i) , X (j ) ) which attempts to find and a such that X(j ) = X (i) + a. A simple solution is to try = 1 and = −1 separately, and for each case, compute dm = X (j ) − X(i) . If the differences dm are small enough in value, then dm is a good estimate of a. A final check must be made, however, to (j ) (j ) (i) (i) assure that (tm+1 − tm ) ≈ (tm+1 − tm ); that is, that distance (in time) is preserved. Such an analysis can be extended to 2D and 3D signal analysis as well.
4 Conclusion A framework for domain theoretic cognitive architecture has been presented which offers many opportunities to improve the ability of a DVSN to perceive important elements in its surrounding environment as well as its own structure.
References 1. Chomsky, N.: Biolinguistic explorations: design, development, evolution. Int. J. Philos. Stud. 15(1), 1–21 (2007) 2. Dekhil, M., Henderson, T.C.: Instrumented sensor system architecture. Int. J. Robot. Res. 17(4), 402–417 (1998) 3. Grupen, R., Henderson, T.C.: Autochthonous behaviors: Mapping perception to action. In: Henderson, T.C. (ed.) NATO ASI on Traditional and Non-Traditional Robotic Sensors, pp. 285–312. Springer, Heidelberg (1990) 4. Henderson, T.C.: Computational Sensor Networks. Springer, New York (2009) 5. Henderson, T.C., Efros, A.A.: Evolutionary teleomorphology. J. Robot. Auton. Syst. 19(1), 23–32 (1996) 6. Henderson, T.C., Shilcrat, E.: Logical sensor systems. J. Robot. Syst. 1(2), 169–193 (1984) 7. Henderson, T.C., Sikorski, K., Grant, E., Luthy, K.: Computational sensor networks. In: IEEE Intl. Conf. on Intelligent Robots and Systems, San Diego, CA. Springer, Berlin (2007) 8. Henderson, T.C., Fan, X., Alford, A., Grant, E., Cohen, E.: Innate theories as a basis for autonomous mental development. In: Workshop on Autonomous Mental Development for Intelligent Systems, IROS 2009, St. Louis, MO. IEEE, New York (2009)
14
Cognitive Sensor Networks
213
9. Henderson, T.C., Fan, Y., Alford, A., Grant, E., Cohen, E.: Innate theories as a basis for autonomous mental development. Technical Report UUCS-09-004, The University of Utah (2009) 10. Henderson, T.C., Fan, Y., Devnani, S., Kumar, S., Cohen, E., Grant, E.: Symmetry as an organizational principle in cognitive sensor networks. Technical Report UUCS-09-005, The University of Utah (2009) 11. Kuipers, B.: Drinking from the firehose of experience. Artif. Intell. Med. 44, 155–170 (2008) 12. Kuipers, B., Beeson, P., Modayil, J., Provost, J.: Bootstrap learning of foundational representations. Connect. Sci. 18(2), 145–158 (2006) 13. Pinker, S.: The Language Instinct. Harper Collins, New York (1994) 14. Popplestone, R., Grupen, R.: Symmetries in world geometry and adaptive behaviour. In: Proceedings of the Workshop on Algebraic Frames for the Perception Action Cycle, Kiel, Germany, pp. 269–283 (2000) 15. Rosenberg, A.: Will genomics do more for metaphysics than Locke? In: Achinstein, P. (ed.) Scientific Evidence, pp. 186–205. The Johns Hopkins University Press, Baltimore (2005) 16. Sawo, F., Henderson, T.C., Sikorski, K., Hanebeck, U.: Sensor node localization methods based on local observations of distributed natural phenomena. In: IEEE Intl. Conf. on Multisensor Fusion and Integration, Seoul, South Korea, pp. 301–308 (2008) 17. Sloman, A.: Architectural and representational requirements for seeing processes and affordances. In: Heinke, D., Mavritsaki, E. (eds.) Computational Modelling in Behavioural Neuroscience: Closing the Gap between NeuroPhysiology and Behaviour Psychology Press, London (2008) 18. Sloman, A.: The well designed young mathematician. Artif. Intell. 172(18), 2015–2034 (2008) 19. Vernon, D., Metta, G., Sandini, G.: A survey of artificial cognitive systems: implications for the autonomous development of mental capabilities in computational agents. IEEE Trans. Evol. Comput. 11(2), 151–180 (2008). Special Issue on Autonomous Mental Development
Chapter 15
Ubiquitous Displays: A Distributed Network of Active Displays Aditi Majumder
Abstract In this chapter we present our work-in-progress on developing a new display paradigm where displays are not mere carriers of information, but active members of the workspace interacting with data, user, environment and other displays. The goal is to integrate such active displays seamlessly with the environment making them ubiquitous to multiple users and data. Such ubiquitous display can be a critical component of the future collaborative workspace. We have developed an active display unit—a projector augmented with sensors, and an embedded computation and communication unit. We are exploring for the first time, the challenges and capabilities resulting from instrumenting a workspace with a distributed network of such active displays to achieve ubiquitous displays. Our main objective is to develop novel distributed methodologies (a) to cover existing surfaces (e.g. walls, floors)—that can deviate considerably from planar, white and Lambertian—with multiple active displays; (b) provide scalability and reconfigurability (in terms of scale, resolution and form factor) of displays; (c) provide a framework for shared viewing and interaction modalities for multiple users.
Keywords Multi-projector displays · Smart displays · Scalable displays
1 Introduction Imagine a workspace covered by a large number of displays (Fig. 1) that are not only purveyors of information, but active entities that can interact with the other components of the workspace (e.g. user, data, other devices). Imagine the user moving around such a workspace engaged in a remote collaborative task. Instead of the A. Majumder () Computer Science Department, University of California, Irvine, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_15, © Springer-Verlag London Limited 2011
215
216
A. Majumder
Fig. 1 An artistic illustration of ubiquitous displays. A fresco is a spatially contiguous imagery from a set of active displays
user consciously restricting his movement to be near the display, a display of the desired resolution and scale factor moves around with him. Imagine a group of users designing annotations on a large map. The users interact with the map in a shared manner, zooming in and out at different locations, without worrying about synchronizing their interactions not to conflict with each other while the map appropriately changed to cater to these multiple, maybe conflicting, cues. Imagine the user working with a display in one area of the workspace upgrades to a display of larger scale and resolution simply by using gestures to move displays from other areas of the workspace to the desired location. Imagine a wall in the workspace that is textured (for e.g. wall-paper) and the displays modify the images to adapt to these in a way that is perceptually faithful to the original image. We call such intelligent displays that can be seamlessly integrated with the environment so that both the user and the data find them at their service anywhere within the workspace, as ubiquitous displays. Achieving ubiquitous displays requires a disruptive change in the display metaphor. The traditional display in a workspace is defined as a single passive output device for presentation of information that is transmitted by a computer. Moving away from this, we envision a display that is an active member of the workspace—an interacting medium between the user, data, environment, and other displays. In other words, these are active agents that are autonomous and smart entities with the capability to sense changes in the environment, data, or user and react
15
Ubiquitous Displays: A Distributed Network of Active Displays
217
to those changes. We are instrumenting a workspace with a pool of such active displays which can collectively and automatically (a) scale to any size and resolution; (b) reconfigure to any form factor; (c) migrate from one location to another; (d) be flexible in using almost any kind of surface; and finally (e) pool resources together to interact with other components of the workspace.
1.1 State of the Art: Centralized Displays Passive Multi-Displays Figure 2 shows the innovations in display architectures in chronological order. Traditionally, a single display was used with a computer. More recently, multiple displays—projectors or LCD panels—are tiled to create large high-resolution displays (e.g. 55 LCD panel HiperWall in UCI, 40-projector wall in NCSC at UIUC). Still, all these displays are passive, each driven by a slave PC and controlled by a central master computer which has the responsibility of pixel management across the displays. These systems have addressed the issue of seamless geometric and photometric integration with the underlying display surface, for the special case of planar Lambertian screens. The geometric mismatches (problem of an image not being aligned across multiple displays) and color non-uniformity (within and across projectors), as illustrated in Fig. 3, were solved using expensive hardware (6 DOF projector mounts) and optical systems (Fresnel lens) by the VR (CAVE™) and entertainment (IMAX™theaters) communities. More recently, several camera-based automated registration techniques have been proposed, within the domain of the centralized architecture, that are cost-effective. Here a single camera measures geometric and color mismatches off-line and uses this information to correct images interactively using the GPUs (Fig. 3) [4, 9]. Recently these techniques have been extended to
Fig. 2 Top: (Left) Passive single display; (Right) Passive multi-display. Bottom: (Left) Single active display; (Right) Proposed distributed network of active displays architecture
218
A. Majumder
Fig. 3 Left: A rectangular array of nine projectors on a planar surface, driven by a 9PC cluster, creating a 8 ft × 10 ft display of resolution 3000 × 2500 in our Large Area Display Laboratory (LADL) at UCI. Center: The uncalibrated display with brightness variation and misalignments (zoom in to see tables and umbrella); Right: The seamless display created by centralized camera-based geometric and brightness registration
special non-planar surfaces, like cylindrical/spherical surfaces [5, 15], using physical fiducials. Single Active Displays In parallel, displays made of a single projector camera that can actively interact with the environment and user have been explored. In the context of computational photography, such systems have been used to project on planar non-white surfaces [11], extract depth of a scene using projector defocus [22], and separate direct and indirect illumination of a scene [12]. In see-through augmented reality, they have been used to effectively react to dynamic changes in the environment [3] and in HCI, they have been used to interact with user gestures and objects of known shape and size [13, 21]. Bottleneck of Centralized Systems Raskar et al. [14] and Johanson et al. [6] have explored a ‘sea of cameras/projectors’ metaphor in a different context than ours, namely for 3D reconstruction of users or environments for 3D telepresence and novel I/O modalities. However, a centralized architecture in the presence of heterogeneous devices resulted in a severe bottleneck in terms of scalability of estimation and registration of device and surface properties; network and communication bandwidth; and handling dynamic addition, removal and movement of devices or data or user. This has been a serious hindrance to scalability and reproducibility of such environments, providing no scope to study other factors like interactivity, flexibility and reconfigurability.
1.2 Disruptive Change in Display Metaphor Our work breaks away from the centralized architecture and builds a novel distributed architecture for instrumenting a workspace with multiple active displays with the goal of exploring for the first time, the challenges and capabilities resulting from a pool of multiple autonomous active displays. We are developing distributed methodologies by which each display acts as an autonomous agent managing its own pixels but work together with the other displays (or agents) for common global
15
Ubiquitous Displays: A Distributed Network of Active Displays
219
goals of interacting with the environment, user and data in a scalable and efficient manner. The distributed approach helps us build on existing work on single displays and bridge the disconnect in research initiatives between passive multi-displays and active single displays. The primary components of an ubiquitous display infrastructure and distributed algorithms thereof are as follows.
Active Displays First we show that a projector augmented by an optimal number of sensors, and an embedded computing and communication unit serves as the most appropriate manifestation of an active display. Then, we propose a network of self-calibrating projector-camera systems for ubiquitous displays and explore their various capabilities when pooled together to instrument a large workspace. We use commodity devices to build the active displays for easy deployment and cost-effectiveness.
Interaction with Environment Environment constitutes the display surfaces and devices. We consider rigid display surfaces that deviate considerably from planar, white and Lambertian, and imperfect off-the-shelf devices. We are developing distributed methodologies to (a) estimate the device and surface properties and (b) register the input imagery seamlessly across the different active displays for viewing by multiple users. The guiding principal for these methodologies is achieving global goals via multiple local goals. We are also exploring novel representations that are compact for easy storage and communication, and amenable to quick updates with the change in the environment. We plan to exploit human perception limitations to achieve the best visual quality possible with the limited system resources.
Interaction with User and Data We are also developing shared interaction modalities for multiple users to interact with the same data across multiple active displays. The types of interaction most natural for displays of such scale are either gesture-based or using simple objects, like a laser pointer or simple shapes common in an office environment like rulers, papers or sometimes even the user himself (for example, moving the display with the user). Our focus is not on gesture/object recognition. Instead, we use existing methods for gesture/object recognition as building blocks and develop distributed methodologies for (a) tracking the gestures/objects/user across multiple displays; and an appropriate (b) reaction across the relevant displays. For interaction of the active displays with data, we plan to develop a middleware that provides an application-independent framework for any general application to use a the ubiquitous displays without considering the underlying arrangements of the active displays. This opens up the possibility of multiple users working together—being remote or co-located—on a large shared desktop.
220
A. Majumder
2 Initial Progress We have made preliminary progress in various directions toward making ubiquitous displays a reality. These are summarized as follows and described in detail in the subsequent sections. • We have proposed the first distributed infrastructure for multi-projector displays for the special case of planar, white, Lambertian screen in [2]. This has all the components of the proposed distributed infrastructure and methodologies to interact with the environment within this limited scope. • For interaction with user and data, we have started exploring a novel distributed interaction paradigm that can scale easily to multiple displays, users, applications and interaction modalities (e.g. laser, gestures) [8, 16]. • As building blocks to easy integration of the active displays with the environment, we have explored projector-camera self-calibration techniques and registering geometrically and photometrically imperfect projectors (as is common for commodity devices) on planar displays [1, 7]. • Estimating the display shape, especially with multiple uncalibrated devices, is challenging. However, our workspace is usually not totally unconstrained. For example, our office walls can be textured but are seldom highly specular; our poster designed for group viewing is non-white but usually does not deviate much from Lambertian; office walls can have gentle smooth curves and corners (due to load-bearing structure like pillars) but seldom have intricate embossings. We are exploring the space of display surface priors that are mathematically well defined but do not compromise our goal of allowing a practical workspace set-up. For example, a ruled surface along a piecewise parametric curve may provide a general description of the shape of the walls of a common office. We have explored calibrating such constrained non-planar displays using a single uncalibrated camera [17–19] instead of calibrated stereo camera pairs. • Finally, we have developed novel color registration techniques that can handle imperfect commodity projectors and are amenable to parallelization [10, 20].
2.1 Distributed Self-Calibration of Planar Display Walls We presented an asynchronous distributed calibration methodology via a display unit called the plug-and-play projector (PPP). The PPP consists of a projector, camera, computation and communication unit, thus creating a self-sufficient module that enables an asynchronous distributed architecture for multi-projector displays. This enables a critical ability of visual communication via the cameras. Each PPP can sense changes in its neighbors via its own camera through the overlapping field-ofview. This establishes a camera-based visual communication mode between adjacent units via analysis of the captured image. This additional communication channel enables critical capabilities like discovering the local topology of the display
15
Ubiquitous Displays: A Distributed Network of Active Displays
221
units and detecting addition/removal of neighboring units, that are not possible using network-based communications. Next, we present a single-program-multipledata (SPMD)calibration algorithm that runs on each PPP and achieves truly scalable and reconfigurable displays without any input from the user and instruments, novel capabilities like adding/removing PPPs from the display dynamically, detecting faults, and reshaping the display to a reasonable rectangular shape to react to the addition/removal/faults. The distributed registration method has three steps. (a) First, in a neighbor detection step a pattern made of four disjoint grids of 5 × 5 blobs (Fig. 4) is used to detect the neighbors of each PPP via their cameras. (b) Next, in the configuration identification step binary coded information is encoded in these grids of blobs and propagated in multiple rounds via communication using the cameras to decipher the total number of PPPs in the display, their configuration (total number of rows and columns) and the coordinates of the PPP in this display. (c) Finally, in the registration step, the pattern in the neighbor discovery step is used to register the display using a distributed homography tree technique. To see the self-calibration in action along with interactive addition and removal of projectors, please check out the video in http://www.ics.uci.edu/~majumder/docs/vizvideo.mov. However, this method has several shortcomings. First, the configuration identification step requires O(ln(mn)) rounds. Multiple rounds of such camera-based communication need considerable image processing and hence impacts performance. This also limits the scalability of the method across a larger number of PPPs. Further, since colored patterns are used, the image processing is not robust when using commodity cameras with lower color fidelity. Second, the homography tree technique is inherently a centralized technique and when adapting it to a distributed setting, a critical global optimization step has to be omitted. This results in significant misregistration errors. In our more recent work [16], we introduce the following innovations to address the above shortcomings. First, we use a single pattern, where a standard QR code is augmented with some Gaussian blobs (Fig. 5), to simultaneously achieve neighbor detection, configuration identification and registration. The projected patterns are illustrated in Fig. 6. This allows us to eliminate the O(ln(mn)) rounds of camerabased communication in the configuration identification round in [2] and achieve this in O(1) time. Also, this significantly reduces visual communication iterations and image processing time thus improving performance. This faster convergence is
Fig. 4 We show the patterns used in Bhasker et al. [2]. Left: The pattern projected by each
PPP. Middle: The image seen by a PPP when all neighbors and itself are projecting their patterns. Bottom: The image of a 3 × 3 array of nine projectors projecting their patterns
222
A. Majumder
Fig. 5 A standard version 1 QR code augmented by embedding 24 Gaussian blobs in its ‘quiet region’ (the surrounding white region)
Fig. 6 We show the QR-based patterns used to achieve the neighbor detection, configuration identification and registration simultaneously. Left: The pattern projected by each PPP. Middle: The image seen by a PPP when all neighbors and itself are projecting their patterns. Bottom: The image of a 3 × 3 array of nine projectors projecting their patterns
Fig. 7 Here we compare our radially cascading method (left) with the distributed homography tree method (right)
possible by supplementing the single camera-based communication with networkbased communications. However, the network overhead is still at most the same as [2]. Finally, since we use monochrome patterns instead of color patterns as in [2], our image processing is much more robust and allows inexpensive cameras with much lower color fidelity. Second, we propose a new radially cascading registration method that is amenable to a distributed implementation and achieves much superior registration (Fig. 7).
15
Ubiquitous Displays: A Distributed Network of Active Displays
223
2.2 Color Registration Amenable to Parallelization Our distributed self-calibration method achieves geometric registration accurately and robustly. However, for color registration, we treat each PPP independently and blend its overlaps with other adjacent projectors using a linear or cosine blending function (similar to edge blending applied in panoramic image generation). This provides acceptable results when using high end projectors that do not show much spatial variation of color or brightness within a single projector [9]. But when dealing with commodity projectors, which age in a non-linear manner, edge blending does not yield seamlessness, breaking the illusion of a single display. Our first work shows that achieving uniform brightness response across the projectors shows severe contrast compression [9] due to projector vignetting. Taking advantage of the fact that humans cannot detect smooth variations in brightness, but only sharp ones, we next design a perception-based constrained brightness smoothing method to maximize display contrast/brightness while minimizing the perceptible sharp seams [10]. This is solved by a dynamic programming method and is amenable to parallelization for a distributed implementation in the context of ubiquitous displays. The above works address brightness, but they do not address chrominance variation. Further, the smoothness in brightness variation achieved by this method is not derivative continuous and still shows visible color variations. In a more recent work [20], we show that chrominance variation across the overlap regions needs to be controlled carefully to achieve a truly seamless display. We show that this controlled chrominance blending can be achieved precisely by controlling the brightness proportion from the multiple overlapping projectors in the overlap region. Consequently, we presented a new Bezier-based brightness smoothing method that can retain the chrominance blending and can also achieve derivative continuity. All these steps together achieve a 3D gamut morphing from one pixel to another in the display and assures true seamlessness (Fig. 8). Since the color registration method is independent of geometry, we can also apply it when registering non-planar displays (Fig. 13). More importantly, this method is also amenable to parallelization, one of the directions we plan to pursue in the near future.
Fig. 8 Comparison of our method with existing work on the most difficult case of white on the planar display. Note that the white is the most difficult test case showing both chrominance and brightness variation. From left to right: Before any correction; After simple RGB blending; After applying constrained brightness smoothing; After gamut morphing algorithm
224
A. Majumder
2.3 Distributed Interaction with 2D Applications on Planar Display Walls We have explored, for the first time, the design of a scalable interaction paradigm for rear-projected tiled displays that can scale with multiple projectors, users, applications and even interaction modalities [8, 16]. We design a SPMD distributed interaction algorithm that runs on each PPP following the distributed registration to allow multiple users to interact with the display using any kind of interaction modality. Our method has several advantages. (a) Since we design an SPMD algorithm, it can easily scale to multiple projectors. Hence, adding and removing PPPs to reconfigure the display does not necessitate any change in the interaction algorithm. (b) Most modules of our algorithm are application independent. Hence, to adapt to different 2D applications, only a few application specific modules need to be modified. This allows our algorithm to scale to many 2D applications. (c) Similarly, changing the interaction modality requires modifying a small number of interface-dependent modules. This allows our algorithm to scale to different interaction modalities as well (e.g. laser pointers, gesture-based interface). (d) Unlike a centralized system where all the interaction from multiple users is handled by a single centralized server, a distributed algorithm distributes the load of handling multiple users to multiple PPPs. Hence, our algorithm can easily scale to multiple users. We consider interaction to be a set of two operations that occur consecutively: (a) a 2D gesture made by the user; and (b) a consequent reaction provided by the display. We assume that a gesture is a sequence of samples, also called actions, detected by the system. These samples can be generated through a multitude of input systems including touch (e.g. by placing the palm on the screen), or laser pointers. The meanings of isolated or temporal sequences of actions are predefined by applications for consistent interpretation. Note that since a gesture occurs over an extended period of time, it can span across multiple PPPs moving between the non-overlapping and overlapping areas of the PPPs. Further, it is important that the reaction does not wait for the gesture to complete. E.g. if the user is moving his/her hands from left to right, he/she is expecting the underlying image to move from left to right even before he/she completes the gesture. Hence, the goal is to identify the gesture even when it is not complete and start reacting as soon as possible. Our distributed interaction paradigm consists of two main components: a distributed gesture management and a distributed reaction management module. These are run as two threads in a producer-consumer fashion in each PPP (Fig. 9). The distributed gesture management module produces a queue of actions which are then processed (or consumed) by the distributed reaction manager in an FCFS manner. Note that though the user’s intentions are interpreted per gesture (which is a series of actions), the processing of these gestures is done per action. This difference in the granularity of interpretation and processing allows the system to respond to a gesture as soon as it commences without waiting for its end. Finally, the distributed
15
Ubiquitous Displays: A Distributed Network of Active Displays
225
Fig. 9 The distributed interaction paradigm
Fig. 10 The Distributed Gesture Management Protocol: The gesture management does not depend in any way on the 2D application and is hence an application-independent module
gesture management is application independent. The application specific modules occur only during reaction management. The distributed gesture management involves (a) a shared action management mechanism to decide which PPP handles which part of the gesture via their reaction managers, and (b) shared gesture tracking to follow the path of the gesture as it moves across multiple PPPs and is facilitated via an anticipatory action communication mechanism. The entire gesture management module is summarized in the flowchart in Fig. 10. Each PPP starts with detecting an action and deciding to pick up its management using the shared action management protocol. If the gesture continues within itself, the PPP tracks it. If the gesture moves close to an adjacent PPP, it communicates it to the relevant neighbor via the anticipatory action message. And if it receives an anticipatory action message, it picks up the gesture tracking and handling from an adjacent PPP. To provide scalability across multiple interaction modality, only the cyan box in Fig. 10 needs to be changed. The distributed reaction mechanism involves processing (consuming) the actions in the queue generated by the distributed gesture manager by reacting with a particular event. Note that the set of PPPs that need to react to a gesture may be larger than the set of PPPs across which the gesture spans. For example, if one wants to move an imagery by a gesture spanning just a couple of PPPs, the imagery across all PPPs must move in response. Further, the event may be associated with creation, movement, or deletion of data across PPPs. Hence, the reaction manager is also responsible for taking steps to assure data consistency across the PPPs. Finally, the job of the event manager also involves informing the PPPs that will be affected by the event so that reaction managers of the PPPs that did not see any gestures can
226
A. Majumder
Fig. 11 The Distributed Reaction Management Protocol. The cyan boxes represent the application specific modules
Fig. 12 This figure shows some of our applications in action. From left to right: Our collaborative map visualization application with two users visualizing different parts of the map at the same time on our 3 × 3 array of nine projectors; Our collaborative emergency management application with two users trying to draw a path to hazardous location and dispatching teams of first responders on our 3 × 3 array of nine projectors; Digital graffiti drawn using our collaborative graffiti application on a 2 × 3 array of six projectors. We deliberately did not edge blend the projectors to show the six projectors clearly; Four children working together on our digital graffiti application on a 3 × 3 array of nine projectors
receive events they need to perform from other PPPs. The function of the reaction manager is summarized as in Fig. 11. It dequeues an action from the queue of actions produced by the action manager and creates and processes the corresponding event. Following that, it checks if it has received any events from other PPPs and processes them. The event processing can be found in detail in [16] and has a few application specific modules that can be changed easily to provide scalability across multiple applications. We demonstrate the scalability of our method to multiple displays, users, applications and interaction modalities by showing a working prototype of multiple 2D applications (namely collaborative graffiti, map visualization, emergency management and virtual bulletin board) using both gesture- and laser-based interaction modality (Fig. 12). We believe that our paradigm can extend to 3D applications, however we would like to explore the different issues in detail in the future.
2.4 Projector Camera Self Calibration Techniques We have designed a new automatic photometric self-calibration method to estimate the intrinsic photometric properties (transfer function and vignetting effect) of a projector-camera pair simultaneously by projecting known patterns on planar diffused surfaces [1]. Using the observation that vignetting effects change smoothly across camera setting like aperture and focal length, we proposed a higher dimension Bezier patch for a compact predictive representation that can scale easily (by increasing the dimension of the patch) with more device settings parameters [7]. We
15
Ubiquitous Displays: A Distributed Network of Active Displays
227
can accurately predict the vignetting effect at 330 different devices settings (33 aperture × 10 focal length) by estimating it at only 40 non-uniformly sampled settings (5 aperture × 4 focal length). Similarly, for projectors, a higher dimension Bezier adequately represents its vignetting effect across different zoom settings.
2.5 Registering Constrained Non-Planar Displays Using a Single Uncalibrated Camera Registering multiple projectors on a non-planar shape usually entails recovery of the 3D shape of the display surface via calibrated stereo camera pairs. However, stereo camera pairs, especially when requiring calibration can make systems complicated. But, to our advantage, our regular office environments are often not completely arbitrary. On the contrary they are often simpler, for example the walls of an office can be thought of as an extruded surface. They are often close to white—maybe lightly checkered—and more often than not are Lambertian. We considered the space of specialized non-planar surfaces and explored if adding reasonable practical priors can make the problems of geometric registration on such constrained non-planar surfaces simpler. We show that if we know the aspect ratio of a smooth vertically extruded surface, we can register multiple projectors on it using a single uncalibrated camera instead of a calibrated stereo pair [17]. Using the priors of a vertically extruded shape and the known aspect ratio, we use a single image of the display surface from an uncalibrated camera to recover both the camera parameters and the 3D shape of the surface. To recover the camera parameters, we use a two-phase non-linear optimization method. In the first phase, the reprojection error of the four corner points of the display are minimized to provide a rough estimate of the seven camera parameters (six extrinsic parameters and focal length). Next, we use this rough estimate to initialize our second optimization which seeks to minimize the difference in reprojected shapes of the top and bottom curves of the display. This converges to a robust estimate of the camera calibration parameters. Consequently, we use these recovered camera parameters to reconstruct the 3D display shape. Then we capture a few images of patterns from the projectors to relate the projector coordinates with the display surface points, and represent this relationship using a rational Bezier patch. This relationship is then used to segment the appropriate parts of the image for each projector to register them and create a seamless image on the display screen (Fig. 13). Since we use a rational Bezier patch, we can even handle projector imperfections or non-linearities, often present due to mounting of short throw lenses to provide a compact set-up. The above method can handle smooth vertically extruded surfaces, but cannot handle sharp corners. This is related to the fact that the projectors are not completely calibrated. Instead the projector coordinates are related to the display coordinates via a rational Bezier patch that inherently assumes smoothness and hence the accuracy during interpolation of unsampled projector coordinates. However, office like environments and even common displays like CAVEs form a class of vertically extruded
228
A. Majumder
Fig. 13 Our geometric registration on a cylindrical display using a 2 × 4 array of eight projectors (top) and on a more general extruded surface using a 2 × 3 array of six non-linear projectors (bottom). Left: Before registration. Right: After registration. These displays have been color registered using our 3D gamut morphing method [20]
Fig. 14 Top: A picture of our CAVE set-up with the projectors overlapping on the edges. Bottom: Geometric registration on our CAVE set-up. The calibrating camera is visible in the top-left corner of the picture
surfaces that has sharp corners. Hence, in a subsequent work [18] we handle such a surface by performing an auto-calibration of projectors. We recover the camera parameters and the 3D display shape as before, but following this, we use two line correspondences between each projector and the camera to recover its complete projector view-frustum. We calibrate each projector independently without using any direct point correspondences, but by studying the distortion the line from the projector undergoes due to the display as observed by the camera. This allows us to achieve registration of CAVEs or room like displays (Fig. 14). More recently, we have also shown that more complex adaptations of the same technique can be used to register multiple projectors on swept surfaces (e.g. truncated half-dome) using an uncalibrated single camera [19] (Fig. 15). A swept surface
15
Ubiquitous Displays: A Distributed Network of Active Displays
229
Fig. 15 A truncated half-dome of 30 radius, 26 high, and 160 degrees angle subtended horizontally with six projectors is registered using multiple views from a single uncalibrated camera. The projectors are deliberately not color registered to show their boundaries
is a surface formed by sweeping a profile curve across a path curve. These are more complex than vertically extruded surfaces, but provide a greater sense of immersion. Finally, when the display is too large to be viewed by from the field-of-view (FOV) of a single camera, we have proposed a method to use multiple panned and tilted views from the same camera to achieve registration. In future, we will explore variations of the above methods to recover the display shape using multiple PPPs in a distributed manner in the context of ubiquitous displays.
3 Conclusion We are making steady progress toward the realization of ubiquitous displays. A large amount of work still needs to be done which can have significant impact on the designing of distributed vision algorithms. If successful, ubiquitous displays have the potential to be a critical and integral component of any collaborative workspace. Even in the near future, it can impact the society at large in multiple ways: (a) provide next generation visualization, training and simulation environments that can host displays of resolution that are an order of magnitude higher than the current ones and can be maintained and deployed much more easily; (b) facilitate commodity digital domes for edutainment purposes; (c) precise digital lighting techniques for cultural heritage explorations and presentation of real large architectural edifices; (d) gesture-based multi-user interaction modalities for mobilizing life-size image-like data in multi-user collaborative environments; and (e) novel body-based interaction tools for creating life-size artistic expressions.
References 1. Bhasker, E., Juang, R., Majumder, A.: Registration techniques for using imperfect and partially calibrated devices in planar multi-projector displays. IEEE Trans. Vis. Comput. Graph. 13(6), 1368–1375 (2007) 2. Bhasker, E., Sinha, P., Majumder, A.: Asynchronous distributed calibration for scalable reconfigurable multi-projector displays. IEEE Trans. Vis. Comput. Graph. 12(5), 1101–1108 (2006) 3. Bimber, O., Emmerling, A., Klemmer, T.: Embedded entertainment with smart projectors. IEEE Comput., 56–63 (2005)
230
A. Majumder
4. Grundhöfer, A., Bimber, O.: Real-time adaptive radiometric compensation. IEEE Trans. Vis. Comput. Graph. 14(1), 97–108 (2008) 5. Harville, M., Culbertson, B., Sobel, I., Gelb, D., Fitzhugh, A., Tanguay, D.: Practical methods for geometric and photometric correction of tiled projector displays on curved surfaces. In: IEEE PROCAMS (2006) 6. Johanson, B., Fox, A., Winograd, T.: The interactive workspaces project: Experiences with ubiquitous computing rooms. IEEE Pervasive Comput. Mag. 1(2) (2002) 7. Lazarov, M., Majumder, A.: Efficient device-independent representation of photometric properties of a projector-camera pair. Technical Report 08-11, Department of Computer Science, University of California, Irvine (2008) 8. Lazarov, M., Pirsiavash, H., Sajadi, B., Mukherjee, U., Majumder, A.: Data handling displays. In: IEEE/ACM Workshop on Projector-Camera Systems (2009) 9. Majumder, A., Stevens, R.: Color nonuniformity in projection-based displays: Analysis and solutions. IEEE Trans. Vis. Comput. Graph. 10(2) (2003) 10. Majumder, A., Stevens, R.: Perceptual photometric seamlessness in tiled projection-based displays. ACM Trans. Graph. 24(1), 118–139 (2005) 11. Nayar, S.K., Peri, H., Grossberg, M.D., Belhumeur, P.N.: A projection system with radiometric compensation for screen imperfections. In: Proceedings of IEEE International Workshop on Projector-Camera Systems (2003) 12. Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R.: Fast separation of direct and global components of a scene using high frequency illumination. In: ACM Transactions on Graphics (SIGGRAPH) (2006) 13. Pingali, G., Pinhanez, C., Levas, A., Kjeldsen, R., Podlaseck, M., Chen, H., Sukaviriya, N.: Steerable interfaces for pervasive computing spaces. In: IEEE International Conference on Pervasive Computing and Communications (2003) 14. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., Fuchs, H.: The office of the future: A unified approach to image based modeling and spatially immersive display. In: Proceedings of ACM Siggraph, pp. 168–176 (1998) 15. Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., Forlines, C.: ilamps: Geometrically aware and self-configuring projectors. ACM Trans. Graph. 22(3) (2003) 16. Roman, P., Lazarov, M., Majumder, A.: A scalable distributed paradigm for multi-user interaction with tiled rear projection display walls. Technical Report 10-02, Department of Computer Science, University of California, Irvine (2010) 17. Sajadi, B., Majumder, A.: Markerless view-independent registration of multiple distorted projectors on vertically extruded surfaces using single uncalibrated camera. IEEE Trans. Vis. Comput. Graph. (6), 1307–1316 (2009) 18. Sajadi, B., Majumder, A.: Auto-calbration of cylindrical multi-projector displays. In: IEEE Virtual Reality (2010) 19. Sajadi, B., Majumder, A.: Geometric registration of multiple projectors on swept surfaces using a single uncalibrated camera. Technical Report 10-03, Department of Computer Science, University of California, Irvine (2010) 20. Sajadi, B., Lazarov, M., Majumder, A., Gopi, M.: Color seamlessness in multi-projector displays using constrained gamut morphing. IEEE Trans. Vis. Comput. Graph. (6), 1317–1326 (2009) 21. Underkoffler, J., Ullmer, B., Ishii, H.: Emancipated pixels: Real-world graphics in the luminous room. In: Proceedings of ACM Siggraph (1999) 22. Zhang, L., Nayar, S.K.: Projection defocus analysis for scene capture and image display. In: ACM Transactions on Graphics (SIGGRAPH) (2006)
Part IV
Wireless Video Sensor Networks, Communications and Control
Chapter 16
Research Challenges for Wireless Multimedia Sensor Networks Tommaso Melodia and Ian F. Akyildiz
Abstract This chapter discusses the state of the art and the major research challenges in architectures, algorithms, and protocols, for wireless multimedia sensor networks (WMSNs). These are networks of wirelessly interconnected smart devices designed and deployed to retrieve video and audio streams, still images, and scalar sensor data. First, applications and key factors influencing the design of WMSNs are discussed. Then, the existing solutions at the application, transport, network, link, and physical layers of the communication protocol stack are investigated. Finally, fundamental open research issues are discussed and future research trends in this area are outlined.
Keywords Wireless sensor networks · Multimedia communications · Distributed smart cameras · Video sensor networks · Energy-aware protocol design · Cross-layer protocol design · Quality of service
1 Introduction The integration of low-power wireless networking technologies with inexpensive CMOS cameras and microphones is enabling the development of distributed networked systems referred to as wireless multimedia sensor networks (WMSNs), i.e., T. Melodia () Wireless Networks and Embedded Systems Laboratory, Department of Electrical Engineering, University at Buffalo, The State University of New York, Buffalo, USA e-mail:
[email protected] I.F. Akyildiz Broadband Wireless Networking Laboratory, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_16, © Springer-Verlag London Limited 2011
233
234
T. Melodia and I.F. Akyildiz
networks of wirelessly interconnected smart devices that allow retrieving video and audio streams, still images, and scalar sensor data [1]. WMSNs will allow retrieving multimedia streams, and store, process in real time, correlate and fuse multimedia content captured by heterogeneous sources. We envision that the users will be able to gather information about the physical environment by issuing simple textual queries, thus accessing multiple remote WMSNs connected to the Internet through application-level gateways. The characteristics of WMSNs diverge considerably from traditional network paradigms such as the Internet and even from “scalar” sensor networks. Most potential applications of WMSNs require the sensor network paradigm to be re-thought in view of the need for mechanisms to deliver multimedia content with a predetermined level of quality of service (QoS). While minimizing the energy consumption has been the main objective in sensor network research, mechanisms to efficiently deliver application-level QoS (e.g., target video distortion), and to map these requirements to network-layer metrics such as latency and jitter, have not been primary concerns. Delivery of multimedia content in sensor networks presents new specific system design challenges, which are the objectives of this chapter. We discuss the state of the art and the main research challenges for the development of wireless multimedia sensor networks. We lead off the discussion by describing the main applications enabled by WMSNs in Sect. 2, and by introducing a reference architecture for WMSNs in Sect. 3. Then, we point out the major factors influencing the design of WMSNs in Sect. 4. Next, in Sects. 5, 6, 7, 8, and 9, we discuss existing solutions and open research issues at the application, transport, network, link, and physical layers of the communication stack, respectively. Finally, in Sect. 10 we conclude the chapter.
2 Applications of Wireless Multimedia Sensor Networks Wireless multimedia sensor networks have the potential to enable numerous new applications. These can be classified as follows: • Multimedia Surveillance Networks. Surveillance sensor networks will be used to enhance and complement existing surveillance systems against crime and terrorist attacks. Multimedia content such as video streams and still images, along with signal processing and computer vision techniques, can be used to locate missing persons, identify criminals or terrorists, or infer and record other potentially relevant activities (thefts, car accidents, traffic violations). • Sensor Networks for Energy Efficiency in Buildings. A combination of video, audio, and scalar sensors will be used in smart buildings to infer human activities and consequently regulate heating, illumination, and air conditioning, among others, to reduce the overall carbon footprint of the building. • Advanced Health Care Delivery. Telemedicine sensor networks can be integrated with 3G/4G cellular networks to provide ubiquitous health care services. Patients will carry medical sensors to monitor parameters such as body temperature, blood
16
Research Challenges for Wireless Multimedia Sensor Networks
235
pressure, pulse oximetry, ECG, and breathing activity. Remote medical centers will monitor the condition of their patients to infer emergency situations. • Environmental and Structural Monitoring. Arrays of video sensors are already used by oceanographers to determine the evolution of sandbars via image processing techniques. Video and imaging sensors are also used to monitor the structural health of bridges or other civilian structures. • Industrial Process Control. Multimedia content such as imaging, temperature, or pressure, may be used for time-critical industrial process control. In automated manufacturing processes, the integration of machine vision systems with WMSNs can simplify and add flexibility to systems for visual inspections and automated actions.
3 Network Architecture We introduce a reference architecture for WMSNs in Fig. 1, where users connect through the Internet and issue queries to a deployed sensor network. The functionalities of the various network components are summarized in a bottom-up manner, as shown below: • Standard Video and Audio Sensors: These sensors capture sound, still or moving images of the sensed event and are typically of low resolution (in terms of
Fig. 1 Reference architecture of a wireless multimedia sensor network
236
•
•
•
• •
•
T. Melodia and I.F. Akyildiz
pixel/inch for the video sensors and in dB for the audio sensors). They can be arranged in a single-tier network, as shown in the first cloud, or in a hierarchical manner, as shown in the third cloud. Scalar Sensors: These sensors sense scalar data and physical attributes like temperature, pressure, humidity and report measured values to their clusterhead. They are typically resource-constrained devices in terms of energy supply, storage capacity, and processing capability. Multimedia Processing Hubs: These devices have comparatively large computational resources and are suitable for aggregating multimedia streams from the individual sensor nodes. They are integral in reducing both the dimensionality and the volume of data conveyed to the sink and storage devices. Storage Hubs: Depending upon the application, the multimedia stream may be desired in real time or after further processing. These storage hubs allow datamining and feature extraction algorithms to identify the important characteristics of the event, even before the data are sent to the end user. Sink: The sink is responsible for packaging high level user queries to network specific directives and return filtered portions of the multimedia stream back to the user. Multiple sinks may be needed in a large or heterogeneous network. Gateway: This serves as the last mile connectivity by bridging the sink to the Internet and is also the only IP-addressable component of the WMSN. It maintains a geographical estimate of the area covered under its sensing framework to allocate tasks to the appropriate sinks that forward sensed data through it. Users: Users are the highest end of the hierarchy and issue monitoring tasks to the WMSN based on geographical regions of interest. They are typically identified through their IP addresses and run application-level software that assigns queries and displays results obtained from the WMSN.
4 Factors Influencing the Design of Multimedia Sensor Networks A multimedia sensor device consists of several basic components, as shown in Fig. 2: a sensing unit, a processing unit (CPU), a communication subsystem, a coordination subsystem, a storage unit (memory), and an optional mobility/actuation unit. Sensing units are usually composed of two subunits: sensors (cameras, microphones, and/or scalar sensors) and analog to digital converters (ADCs). The analog signals produced by the sensors based on the observed phenomenon are converted to digital signals by the ADC, and then fed into the processing unit. The processing unit executes the system software in charge of coordinating sensing and communication tasks, and is interfaced with a storage unit. A communication subsystem interfaces the device to the network, and is composed of a transceiver unit, and of communication software. The latter includes a communication protocol stack, and system software such as middleware, operating systems, virtual machines, etc. A coordination subsystem is in charge of coordinating the operation of different network devices, by performing operations such as network synchronization and location
16
Research Challenges for Wireless Multimedia Sensor Networks
237
Fig. 2 Internal organization of a multimedia sensor
management. An optional mobility/actuation unit can enable movement or manipulation of objects. Finally, the whole system is powered by a power unit that may be supported by an energy scavenging unit such as solar cells. There are several factors that influence the design of a WMSN: • Application-specific QoS Requirements. In addition to data delivery modes typical of scalar sensor networks, multimedia data include snapshot and streaming multimedia content. Snapshot-type multimedia data contain event triggered observations obtained in a short time period (e.g., a still image). Streaming multimedia content is generated over longer time periods and requires sustained information delivery. • Resource Constraints. Embedded sensors are constrained in terms of battery, memory, processing capability, and achievable data rate. • Variable Channel Capacity. In multi-hop wireless networks the capacity of each wireless link depends on the interference level perceived at the receiver. This, in turn, depends on the interaction of several functionalities that are distributively handled by all network devices such as power control, routing, and rate policies. Hence, capacity and delay attainable on each link are location dependent, vary continuously, and may be bursty in nature, thus making QoS provisioning a challenging task. • Cross-layer Coupling of Functionalities. Because of the shared nature of the wireless communication channel, in multi-hop wireless networks there is a strict interdependence among functionalities handled at all layers of the communication
238
T. Melodia and I.F. Akyildiz
stack. This interdependence has to be explicitly considered when designing communication protocols aimed at QoS provisioning. • High Bandwidth Demand. Multimedia contents, especially video streams, require transmission bandwidth that is orders of magnitude higher than that supported by currently off-the-shelf sensors. For example, the nominal transmission rate of state-of-the-art IEEE 802.15.4 compliant devices such as Crossbow’s MICAz or iMOTE2 motes is 250 Kbit/s. Data rates at least one order of magnitude higher may be required for high-end multimedia sensors, with comparable power consumption. Hence, high data rate and low-power consumption transmission techniques need to be leveraged. In this respect, the Ultra Wide Band (UWB) transmission technology seems particularly promising for WMSNs, as discussed in Sect. 9. • Multimedia Source Coding Techniques. State-of-the-art video encoders rely on intra-frame compression techniques to reduce redundancy within one frame, and on inter-frame compression (also predictive encoding or motion estimation) to exploit redundancy among subsequent frames to reduce the amount of data to be transmitted and stored. Since state-of-the-art predictive encoding requires complex encoders, powerful processing algorithms, and entails high energy consumption, it may not be suited for low-cost multimedia sensors. • Multimedia In-network Processing. Processing and delivery of multimedia content are not independent and their interaction has a major impact on the achievable QoS. The QoS required by the application will be provided by means of a combination of cross-layer optimization of the communication process, and innetwork processing of raw data streams that describe the phenomenon of interest from multiple views, with different media, and on multiple resolutions. Hence, it is necessary to develop application-independent and self-organizing architectures to flexibly perform in-network processing of multimedia contents.
5 Application Layer In this section, we overview challenges and functionalities at the application layer in typical WMSN application.
5.1 Multimedia Encoding Techniques The main design objectives of a coder for WMSNs are: • High Compression Efficiency. It is necessary to achieve a high ratio of compression to effectively limit bandwidth and energy consumption. • Low-complexity. Multimedia encoders are embedded in sensor devices. Hence, they need to be low-complexity to reduce cost and form factors, and low-power to prolong the lifetime of sensor nodes.
16
Research Challenges for Wireless Multimedia Sensor Networks
239
• Error-resiliency. The source coder should provide robust and error-resilient coding of source data. The traditional broadcasting paradigm, where video is compressed once at the encoder and decoded several times, has been dominated by predictive encoding techniques. These, used in the widely spread ISO MPEG schemes, or the ITU-T recommendations H.263 and H.264 (also known as AVC or MPEG-4 part 10), reduce the bit rate generated by the source encoder by exploiting source statistics. Since the computational complexity is dominated by the motion estimation functionality, these techniques require complex encoders, powerful processing algorithms, and entail high energy consumption, while decoders are simpler and loaded with lower processing burden. For typical implementations of state-of-the-art video compression standards, such as MPEG or H.263 and H.264, the encoder is five to ten times more complex than the decoder [6]. Conversely, to realize low-cost, low-energyconsumption multimedia sensors it is necessary to develop simpler encoders, and still retain the advantages of high compression efficiency. Two potential alternatives to classical predictive encoding, i.e., distributed video coding and video encoding based on compressed sensing, are described in what follows. Distributed Video Coding It is known from information-theoretic bounds established by Slepian and Wolf for lossless coding and by Wyner and Ziv for lossy coding with decoder side information, that efficient compression can be achieved by leveraging knowledge of the source statistics at the decoder only. This way, the traditional balance of complex encoder and simple decoder can be reversed [6]. Techniques that build upon these results are usually referred to as distributed source coding. Distributed source coding refers to the compression of multiple correlated sensor outputs that do not communicate with each other [14]. Joint decoding is performed by a central entity that receives data independently compressed by different sensors. However, practical solutions have not been developed until recently. Clearly, such techniques are very promising for WMSNs and especially for networks of video sensors. The encoder can be simple and low-power, while the decoder at the sink will be complex and loaded with most of the processing and energy burden. The reader is referred to [14] and [6] for excellent surveys on the state of the art of distributed source coding in sensor networks and in distributed video coding, respectively. Video Encoding Based on Compressed Sensing As an alternative, video encoders based on the recently proposed compressive sensing (CS) paradigm [2, 4] can offer a viable solution to the problems of encoder complexity and limited resiliency to channel errors that characterizes predictive encoders. Compressed sensing (aka “compressive sampling”) is a new paradigm that allows the recovery of signals from far fewer measurements than methods based on Nyquist sampling. In particular, the main result of CS is that a N -dimensional signal can be reconstructed from M noise-like incoherent measurements as if one had observed the M/ log(N )
240
T. Melodia and I.F. Akyildiz
most important coefficients in a suitable base [12]. Hence, CS can offer an alternative to traditional video encoders by enabling imaging systems that sense and compress data simultaneously at very low computational complexity for the encoder. Image coding based on CS has been recently explored [12]. So-called single-pixel cameras that can operate efficiently across a much broader spectral range (including infrared) than conventional silicon-based cameras have also been discussed [12]. In [7, 11], the performance of CS encoded video transmission in WMSN was evaluated. In CS, the transmitted samples constitute a random, incoherent combination of the original image pixels. This means that, unlike traditional wireless imaging systems, in CS no individual sample is more important for image reconstruction than any other sample. Instead, the number of correctly received samples is the only main factor in determining the quality of the received image. Hence, a peculiar characteristic of CS video is its inherent and fine-grained spatial scalability. The video quality can be regulated at a much finer granularity than traditional video encoders, by simply varying the number of samples per frame. Also, as shown in [7, 11] a small amount of random channel errors does not affect the perceptual quality of the received image at all, since, for moderate BERs, the greater sparsity of the “correct” image will offset the error caused by the incorrect bit. CS image representation is completely unstructured: this fact makes CS video more resilient than existing video coding schemes to random channel errors. This simple fact has obvious, deep, consequences on protocol design for end-to-end wireless transport of CS video, which are to be addressed in future research.
5.2 Collaborative In-network Processing Given a source of data (e.g., a video stream), different applications may require diverse information (e.g., raw video stream vs. simple scalar or binary information inferred by processing the video stream). This is referred to as application-specific querying and processing. Hence, it is necessary to develop expressive and efficient querying languages, and to develop distributed filtering and in-network processing architectures, to allow real-time retrieval of useful information. Similarly, it is necessary to develop architectures to perform data fusion or other complex processing operations in-network. Algorithms for both inter-media and intra-media data aggregation and fusion need to be developed, as simple distributed processing schemes developed for existing scalar sensors are not designed for multimedia contents. In a densely deployed WMSN, there exists correlation among the visual information observed by cameras with overlapped fields of view (FoVs). In [3], a novel spatial correlation model for visual information in WMSNs is proposed. By studying the sensing model and deployments of cameras, a spatial correlation function is derived to describe the correlation characteristics of visual information observed by cameras with overlapped FoVs. In addition, an entropy-based analytical framework
16
Research Challenges for Wireless Multimedia Sensor Networks
241
is developed to measure the amount of information from multiple correlated cameras. Using this spatial correlation model, the correlation characteristics of visual information can be obtained at low communication and computation costs. In [13], an information-theoretic data compression framework is proposed with the objective to maximize the overall compression of the visual information retrieved by a WMSN. This framework consists of two components: (i) compression efficiency prediction, and (ii) coding hierarchy construction. Both components are independent of the specific coding algorithms and images types. The compression efficiency prediction components aims to estimate the compression gain from joint encoding of multiple cameras before the actual images are captured. To achieve this, an entropy-based divergence measure (EDM) scheme is proposed to estimate the joint entropy of multiple cameras, based on the spatial correlation model in [3]. Based on the predicted results from EDM, a distributed multi-cluster coding protocol (DMCP) is proposed to construct a compression-oriented coding hierarchy. The DMCP aims to partition the entire network into a set of coding clusters such that the global coding gain is maximized. Moreover, in order to enhance the decoding reliability at the sink, the DMCP also guarantees that each sensor camera is covered by at least two different coding clusters. To validate the effectiveness of the proposed data compression framework, experiments on H.264 standards are performed and demonstrate that the compression framework can reduce 10–23% total coding rate compared with the individual coding scheme, i.e., each camera sensor compresses its own image independently.
6 Transport Layer Protocols Classical transport layer functionalities, like providing end-to-end congestion control, become especially important in real-time delay-bounded applications, such as streaming media. We study these based on their underlying dependence on (i) UDP, (ii) TCP and TCP compatible. Furthermore, we discuss the recently proposed distortion-minimizing rate control (DMRC) protocol, who is specifically designed for video sensing applications.
6.1 UDP Based Protocols The User Datagram Protocol (UDP) is usually preferred over TCP in typical multimedia applications as timeliness is of greater concern than reliability. Selected features of existing standards for the Internet like RTP (Real-time Transport Protocol) may be adopted in the context of WMSNs. RTP uses a separate control stream called Real-time Transport Control Protocol (RTCP) that allows dynamic adaptation to the network conditions. RTP runs over UDP, but it provides support for a host of functionalities like bandwidth scaling and integration of different images into a single composite. In addition, the Application Level Framing (ALF) allows
242
T. Melodia and I.F. Akyildiz
manipulation of the header to suit application-specific needs. Through ALF, specific instructions can be encoded in the header that are typical to the WMSN application, while ensuring compatibility with the external IP-based data storage and monitoring network.
6.2 TCP and TCP Friendly Schemes for WMSNs Typically, packets sent to the sink are highly compressed at the source with only a subset of the nodes transmitting that have non-redundant data. Compression standards such as the JPEG2000 and the MPEG introduce features like the Region of Interest (ROI) and the I-frame respectively. These special packets carry original content that cannot be retrieved through interpolation. Hence, dropping packets indiscriminately, as in the case of UDP, may cause discernible disruptions in the multimedia content. We thus argue that some form of selective reliability, such as that provided by TCP, must be introduced for these packets in a WMSN. Two key factors that limit multimedia transport based on TCP-like rate control schemes, are the jitter introduced by the congestion control mechanism and the control message overhead. Existing solutions for transporting MPEG video in a TCPfriendly manner overcome this problem of jitter by assuming playout buffers at the sink. Distributed approaches, especially addressing the concerns of sensor networks, often cache TCP segments within the network and through local re-transmissions they reduce the message passing overhead. The use of TCP can also be argued from the point of bandwidth utilization. Multiple streams can be opened between source and sink, each of which may follow a different path. Thus by splitting the multimedia traffic into several smaller data rate paths, and by dynamically changing the TCP window size for each connection, a fine-grained control on sensor traffic is possible that may extend network lifetime and enhance performance. Being unable to distinguish between bad channel conditions and network congestion is a major problem in TCP.
6.3 Distortion-Minimizing Rate Control A different approach is pursued in [10]. A multi-hop wireless network of video sensors deployed for surveillance applications is considered, and the focus is on reliable and real-time transport of video traffic. The objective is to design algorithms to efficiently and fairly share the common network infrastructure among the video streams generated by different video sensors, to deliver high-quality video on resource-constrained devices. To achieve this objective, the Distortion-Minimizing Rate Control (DMRC) algorithm is proposed, a decentralized cross-layer control algorithm that jointly regulates the end-to-end data rate, the video quality, and the strength of the channel coding at the physical layer to minimize the distortion of the
16
Research Challenges for Wireless Multimedia Sensor Networks
243
received video. The end-to-end data rate is chosen to avoid congestion while maintaining fairness in the domain of video quality (rather than data rate as in traditional rate control algorithms). Once the end-to-end data rate has been determined, the sender calculates the optimal proportion of video encoder rate and channel encoder rate based on the overall rate available and on the current quality of the wireless channel on the source-destination path, with the objective of minimizing the video distortion at the receiver. Since DMRC is not designed to be fair to competing TCP flows (which are unlikely to be concurrently transmitted in a WMSN), DMRC regulates the rate more aggressively than TFRC, thus leading to lower video distortion.
7 Network Layer We focus our discussion here on the primary network-layer functionality of multimedia routing. We classify this further based on (i) architectural and spatial attributes, and (ii) real-time support. From Fig. 1, we observe that hierarchical WMSNs may be deployed that have different types of sensors with varying capabilities. Thus, there may be two sets of routes with, for example, low granularity image sensors forming one set and the overlaying high resolution video sensors constituting the other. Both these routing schemes may need close interaction, as they carry packets describing the same event. Also, the feed from varying sensor types may need to be periodically fused, thus needing common nodes along both paths. Hence, we believe that there must be a cooperative approach amongst routing algorithms operating at different layers of a hierarchical environment. Meeting strict time deadlines, as required for streaming applications, and maintaining reliability constraints are often contradictory goals. Reducing the delay at each link and routing based on local channel conditions may help alleviate this problem. The Multi-Path and Multi-SPEED Routing Protocol (MMSPEED) is one such approach that attempts to balance between these two goals and spans over the network and MAC layers [5]. It differentiates between flows with different delay and reliability requirements in order to channelize resources to flows. While each node selects its next hop based on link layer delay measurements, a feedback mechanism along the path helps correct local estimation inaccuracies. However, this research direction is still a best-effort practice and does not propose energy saving techniques or give firm guarantees in bursty multimedia traffic.
8 MAC Layer There are several approaches for regulating the channel access based on contention, and we advocate the use of contention-free protocols for WMSNs. Based on the nature of channel access, some MAC protocols are geared to provide high link-level throughput, or reduce delays, or guarantee QoS for a given packet type. The main categories of these protocols are listed next and their key features that may be useful for WMSNs are discussed.
244
T. Melodia and I.F. Akyildiz
Contention-Based Protocols Existing schemes are mostly based on variants of the Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) Medium Access Control (MAC) protocol. When a device is receiving data, transmissions from all the devices in its transmission range are impeded. However, this is achieved by the use of random timers and a carrier sense mechanism, which in turn, results in uncontrolled delay and idle energy consumption. Moreover, WMSNs may have different traffic classes, and approaches similar to the Hybrid Coordination Function Controlled Channel Access (HCCA) component present in IEEE 802.11e need to be used. Channel contention can be significantly reduced by using multiple channels in a spatially overlapped manner. This is commonly accomplished by using two radios, in which, one is delegated the task of channel monitoring. This is often a low-power transceiver and is responsible for waking up the main radio for data communication on a need basis. However, such multi-channel schemes introduce the problem of distinct channel assignment and add hardware complexity. Contention-free Single Channel Protocols Time Division Multiple Access (TDMA) is a representative protocol of this class. Usually, the frame is organized with a small reservation period (RP) that is generally contention-based, followed by a contention-free period that spans the rest of the frame. This RP could occur in each frame or at pre-decided intervals in order to assign slots to active nodes taking into consideration the QoS requirement of their data streams. Variable length of the TDMA frames (V-TDMA) and the frequency of the RP interval are some of the design parameters that can be exploited while designing a multimedia system. However, TDMA-based sensor networks usually present limited scalability and complex network-wide scheduling, apart from the problems of clock drift and synchronization issues.
9 Physical Layer The Ultra Wide Band (UWB) technology [15] has the potential to enable low power consumption, high data rate communications within tens of meters. TimeHopping Impulse Radio UWB (TH-IR-UWB) is particularly appealing for WMSNs for several reasons. First, TH-IR-UWB enables high data rate, very low-power carrierless communications, on simple-design, low-cost radios. Moreover, it provides large processing gain in presence of interference, and it is flexible, since data rate can be traded for power spectral density and multipath performance. Importantly, the impulse radio technology naturally allows for integrated MAC/PHY solutions, since interference mitigation techniques allow realizing MAC protocols that do not require mutual temporal exclusion between different transmitters [9]. Hence, simultaneous communications of neighboring devices are feasible without complex receivers. Furthermore, the large instantaneous bandwidth enables fine time resolution for accurate position estimation and for network synchronization. Finally, UWB signals have extremely low-power spectral density, with low probability of
16
Research Challenges for Wireless Multimedia Sensor Networks
245
intercept/detection (LPI/D), which is particularly appealing for military covert operations. While the UWB transmission technology is advancing rapidly, many challenges need to be solved to enable multi-hop networks of UWB devices. Recent work has addressed MAC layer design [9] for UWB devices. In [8], we have developed a cross-layer protocol suite that leverage UWB signaling to provide differentiated quality of service support to heterogeneous applications in WMSN. The proposed cross-layer controller jointly assigns resources at the physical (channel coding), medium access control, and network layers of the protocol stack to guarantee application requirements in terms of bandwidth, delay, and packet error rate.
10 Conclusions We discussed the state of the art of research on Wireless Multimedia Sensor Networks (WMSNs), and outlined the main research challenges. We discussed existing solutions and open research issues at the physical, link, network, transport, and application layers of the communication stack. In particular, we believe that recent work undertaken in Wyner-Ziv coding at the application layer, the leveraging of spatio-temporal aspects of multimedia sensing in designing routing and transport layer solutions, MAC protocols that provide link latency bounds, and UWB technology, amongst others, seem to be the most promising research directions in developing practical WMSNs.
References 1. Akyildiz, I.F., Melodia, T., Chowdury, K.R.: A survey on wireless multimedia sensor networks. Comput. Netw. 51(4), 921–960 (2007) 2. Candes, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006) 3. Dai, R., Akyildiz, I.F.: A spatial correlation model for visual information in wireless multimedia sensor networks. IEEE Trans. Multimed. 11(6), 1148–1159 (2009) 4. Donoho, D.: Compressed Sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006) 5. Felemban, E., Lee, C.-G., Ekici, E.: MMSPEED: Multipath multi-SPEED protocol for QoS guarantee of reliability and timeliness in wireless sensor networks. IEEE Trans. Mob. Comput. 5(6), 738–754 (2006) 6. Girod, B., Aaron, A., Rane, S., Rebollo-Monedero, D.: Distributed video coding. Proc. IEEE 93(1), 71–83 (2005) 7. Melodia, T., Pudlewski, S.: A case for compressive video streaming in wireless multimedia sensor networks. IEEE COMSOC MMTC E-Lett. 4(9) (2009) 8. Melodia, T., Akyildiz, I.F.: Cross-layer QoS-aware communication for ultra wide band wireless multimedia sensor networks. IEEE J. Sel. Areas Commun. (2010) 9. Merz, R., Widmer, J., Boudec, J.-Y.L., Radunovic, B.: A joint PHY/MAC architecture for low-radiated power TH-UWB wireless ad-hoc networks. Wirel. Commun. Mob. Comput. J. 5(5), 567–580 (2005)
246
T. Melodia and I.F. Akyildiz
10. Pudlewski, S., Melodia, T.: DMRC: Distortion-minimizing rate control for wireless multimedia sensor networks. In: Proc. of IEEE International Conference on Mobile Ad-hoc and Sensor Systems (MASS), Hong Kong S.A.R., P.R. China, October (2009) 11. Pudlewski, S., Melodia, T.: On the performance of compressive video streaming for wireless multimedia sensor networks. In: Proc. of IEEE Int. Conf. on Communications (ICC), Cape Town, South Africa, May (2010) 12. Romberg, J.: Imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 14–20 (2008) 13. Wang, P., Dai, R., Akyildiz, I.F.: Collaborative data compression using clustered source coding for wireless multimedia sensor networks. In: Proc. of IEEE Intl. Conf. on Computer Communications (INFOCOM), San Diego, USA, March (2010) 14. Xiong, Z., Liveris, A.D., Cheng, S.: Distributed source coding for sensor networks. IEEE Signal Process. Mag. 21, 80–94 (2004) 15. Yang, L., Giannakis, G.B.: Ultra-WideBand communications: An idea whose time has come. IEEE Signal Process. Mag. 3(6), 26–54 (2004)
Chapter 17
Camera Control and Geo-Registration for Video Sensor Networks James W. Davis
Abstract With the use of large video networks, there is a need to coordinate and interpret the video imagery for decision support systems with the goal of reducing the cognitive and perceptual overload of human operators. We present computer vision strategies that enable efficient control and management of cameras to effectively monitor wide-coverage areas, and examine the framework within an actual multi-camera outdoor urban video surveillance network. First, we construct a robust and precise camera control model for commercial pan-tilt-zoom (PTZ) video cameras. In addition to providing a complete functional control mapping for PTZ repositioning, the model can be used to generate wide-view spherical panoramic viewspaces for the cameras. Using the individual camera control models, we next individually map the spherical panoramic viewspace of each camera to a large aerial orthophotograph of the scene. The result provides a unified geo-referenced map representation to permit automatic (and manual) video control and exploitation of cameras in a coordinated manner. The combined framework provides new capabilities for video sensor networks that are of significance and benefit to the broad surveillance/security community.
Keywords Video networks · Surveillance · Camera control · Registration
1 Introduction In today’s society with the prolific use of video surveillance cameras, the predominate means of monitoring is still the use of relatively few personnel controlling cameras with a joystick and watching large banks of video monitors. In this scenario the J.W. Davis () Dept. of Computer Science & Engineering, Ohio State University, Columbus, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_17, © Springer-Verlag London Limited 2011
247
248
J.W. Davis
video feeds are typically stored for long periods of time, only to be recalled after an event has occurred (“video forensics”). What is needed are novel techniques for advanced video monitoring to access, control, coordinate, and interpret the video imagery to help reduce the perceptual and cognitive overload of human operators. We present a two-part approach for managing video surveillance networks that begins with novel and efficient algorithms for modeling and controlling individual camera coverage, then progresses to registration of the individual camera viewspaces and control methods to a geo-referenced base image to provide a single and intuitive strategy for camera access, control, and coordination. We first describe an efficient method for automatically creating a functional camera control model for commercial pan-tilt-zoom (PTZ) surveillance cameras. The goal is the ability to map any pixel location in the live video image to its corresponding camera PTZ world orientation, and vice versa. This capability is important for re-centering the camera to orient to a selected image location (or world coordinate) without using a joystick, and is fundamental to aid active/moving camera applications and multi-camera coordination. The approach is able to automatically construct the model and execute real-time control functions. The second component addresses a geo-registration technique to map the PTZ camera viewspaces to a wide-area aerial orthophotograph (e.g., Google Map image) to provide a single geo-referenced map representation for camera coordination and exploitation (e.g., useful for camera hand-off and cueing). We base our approach on the concept of using the automatically constructed spherical panoramas of the PTZ cameras (from the camera control model) as a means to map the entire viewspace possible of a PTZ camera onto the aerial image. The result can be used to re-orient a camera to a position selected on the orthophoto/map, and the reverse mapping can be used to show where the camera is currently directed. The overall framework provides a novel computational and interactive strategy for access, control, and exploitation of camera networks that together provide new capabilities that directly benefit current operational systems. We demonstrate the framework using our multi-camera outdoor urban video network on the campus of Ohio State University (OSU). The system currently includes seven PTZ outdoor color video cameras (Pelco Spectra III/IV SE) that are commonly used in surveillance. Results are provided showing spherical panoramas constructed from the learned camera control models, an active PTZ tracking application, and georeferenced trajectory mappings enabled by the panorama-orthophoto registration.
2 Related Work In relation to the PTZ camera modeling component, many existing camera models incorrectly assume idealized mechanics (where the optical and geometric center of the camera are collocated and the rotation axes are aligned) [1, 2, 5, 7, 21], while others provide complicated or computationally expensive approaches. Though [6] accounts for rotations around an arbitrary axis, a controlled environment is required and an expensive iterative minimization technique is employed. In [9], a complex
17
Camera Control and Geo-Registration for Video Sensor Networks
249
model is proposed where a highly non-linear relationship between image coordinates and world coordinates is iteratively solved for a solution. Similarly [10] employs a search mechanism to minimize a match score function. In [3], a non-precise correction matrix is used to continuously update the camera parameters. The main advantage of our model is that it is independent of the above alignment problems and the underlying geometry provides a simple and fast method for reliable and precise camera control. Regarding the panorama-orthophoto registration component, there is a rich history in registration but related work has typically dealt with finding homographies, which are based on pairs of planar views and not entire PTZ fields-of-coverage. In [13], tracking data between pairs of cameras and centroids of moving objects are used to solve for correspondences. Other ways to obtain these correspondences include feature matching techniques [12] and geometric methods [4, 11]. In [8], a catadioptric omnicamera is used to establish a lookup table mapping from a limitedresolution omnidirectional view to the ground plane (the inverse mapping cannot be expressed analytically). Our registration approach provides an analytical mapping (forward and reverse) between our camera model and the ground (orthophoto) and does not need to know the interrelationships of the cameras in the network.
3 PTZ Camera Viewspace Control Model Pan-tilt-zoom cameras are extensively used for wide-area surveillance. Fundamental to multiple applications, for either manual or automatic monitoring tasks, is the ability to efficiently redirect/re-center PTZ cameras to aim at desired locations in the scene. Moving beyond the use of a joystick, construction of PTZ camera control algorithms for automatic/manual camera redirection requires an understanding of the relationship between pixel coordinates (in the live image) and their corresponding PTZ camera world orientations. Complicated camera models have been previously suggested to address this mapping. However, we propose a simple, yet reliable, geometric PTZ model which given the current pan-tilt orientation of the camera, can compute in constant real-time, the pan-tilt camera orientation of any arbitrary pixel location in the given video image (used to re-center the camera on that pixel location). The principal idea behind our approach is that the locus of a point on a given image plane (infinitely extended), as the camera rotates, forms an ellipse from which the pan and tilt can be computed. The main advantage to this approach is that the problematic issues that typically arise from misaligned pan and tilt motor axes and their lack of co-location with the camera optical center are removed. Consider the principal point R at the center location in image I . As the camera pans, the projections of successive principal foci R1 , R2 , R3 , . . . onto the original extended image plane I form an ellipse. This is evident from the fact that the intersection of a plane with a cone is an ellipse. Therefore, the locus of projections of the Ri onto I is the ellipse e1 . Similarly, the locus of an arbitrary non-principal point P on I as the camera pans is also an ellipse (e2 ). Given the camera’s current pan-tilt
250
J.W. Davis
orientation (θ, φ), our goal is to map P ’s image coordinates (x, y) to a change in pan (δθ ) and a change in tilt (δφ) that would center the point P in the new image (after panning and tilting), thus making P = R in the new image. We provide our fundamental model equations here, with further details and derivations described in [18]. First we compute the change in pan δθ . From multiple geometric relationships and substitutions (see [18]), we can obtain the following equation for the change in pan x −1 . (1) δθ = tan y · sin φ + f · cos φ Here we need only the target point (x, y) (i.e., point P ) in the image, the current tilt φ of the camera, and the focal length f (in pixels) to compute the desired change in pan δθ . Next we compute the change in tilt δφ. The length of the major and minor axes, a and b respectively, of the ellipse e1 (described above) for the point (x, y) can be given [18] as a = tanf φ and b = sina φ . The resulting change in tilt can then be formulated [18] as y+a a −1 δφ = tan . (2) − x f f · cos tan−1 ab · y+a Again, we need only the target location (x, y) in the image, the current tilt φ, and the focal length f to compute the required change in tilt δφ. The focal length f (in pixels) can be calculated by pointing the camera to preconfigured locations which have overlapping coverage, matching corresponding point locations in the views (e.g., using SIFT [14]), and employing (1) with the preconfigured pan-tilt orientations to formulate a solution to f . To accommodate zoom into the model, we compute the value for f , as described above, independently at various zoom factors. This is used to create an f -to-zoom mapping function (by interpolating between various learned f -zoom pairs) which can then be exploited to adjust the pan and tilt at any zoom factor of the camera. Together, the resulting δθ and δφ can be applied to the camera motor to recenter the camera on the location projected through the selected image point (x, y). Note, as shown by the equations, that a simple linear mapping of (x, y) to (θ, φ) based only on the proportional change from the center in the sensor field-of-view is insufficient. We tested the feasibility of this model by capturing camera images across its entire viewspace (at various overlapping pan-tilt locations), mapping the individual pixel locations within each image to their corresponding location in pan-tilt using the proposed model, and generating a panoramic image with the transformed data (pixel-to-pan-tilt). The resulting transformed RGB data were plotted on a polar coordinate system where the radius varies linearly with tilt and the sweep angle represents the pan (i.e., linear fisheye view [15]). The resulting panoramas constructed for two different camera types are shown in Fig. 1, which displays the entire pan-tilt coverage of each camera in a single static image. Each panorama was constructed
17
Camera Control and Geo-Registration for Video Sensor Networks
251
Fig. 1 Spherical panoramas constructed for two different cameras (Pelco, Sony)
from ∼80 overlapping sub-images (stitched together). To overcome the problem of image brightness differences across the scene due to auto-gain camera settings (producing brightness “seams” at adjacent images in the panorama), we used a technique of deriving an intrinsic image [20] from multiple panoramas created with different starting offsets. A single reflectance intrinsic image is computed from grayscaleconverted panoramas and is combined with an averaged chrominance panorama. The result is a seamless spherical panorama (as shown in the figure). This PTZ model is not simply a panoramic view generator (which itself can be useful for visualization), but instead provides a complete pixel-to-pan-tilt control mapping. For example, given any live video image from a PTZ camera, we have an analytical mapping of any pixel in the image to the pan-tilt camera orientation needed to move and center the camera on that location. This is an especially powerful tool for active tracking. To demonstrate this idea, we developed an active camera tracker using our control model that moves the camera such that the target being tracked remains centered in the image (requires pan-tilt re-centering of the target by the camera in each successive video image). We employed an appearance-based covariance matching algorithm [17] with position, intensity, and gradient features to localize the target in the images (note that other trackers could also be used). The tracker also checks to determine if an update in zoom is necessary. To do so, it checks the match distance of a slightly larger and smaller patch size at the default match location. If one of the patch sizes produces a better match score, the zoom is adjusted (based on a new f computed from the selected patch size and the default constant patch size) so as to keep the target of a constant size during tracking. Frames from two active tracking experiments are shown in Fig. 2. The ground truth locations of the targets were obtained by manually marking the bounding box of the target’s torso and calculating its center in each frame. By comparing these with the center of the tracking box (of size 25 × 50, within a 320 × 240 image), the tracking error statistics (mean, standard deviation)
252
J.W. Davis
Fig. 2 Active tracking results for two targets (zoom listed for each frame)
in pixels were obtained as (2.77, 1.50) and (3.23, 1.63) for the two sequences. Of additional capability with the model is the use of the panorama as a user interface to instantly drive the camera to a location selected in the panorama (instead of using a joystick).
4 Scene-Based Camera Geo-Registration and Mapping Typical outdoor urban surveillance systems consist of multiple PTZ cameras overlooking different areas, with some cameras capable of observing the same area (from different viewpoints). To utilize distributed cameras in a coherent way, it is important to establish a common reference frame to which each of these cameras can be mapped (registered), controlled, and exploited. A natural frame of reference in this domain is an aerial orthophotograph, as it is an actual wide-area top-down image which provides visual context, image features for the registration process, and geo-referenced information (an orthophoto contains metadata of the physical world position for each pixel—e.g., in lat-lon). Most people are familiar with such ortho imagery through online map sources, such as Google Maps. Figure 3(a) shows a cropped portion from a large orthophoto taken over OSU. Current ortho imagery for the State of Ohio is publicly available (through OGRIP [16]) down to 6 inch pixel resolution. Here we present a method to register PTZ camera viewspaces to an orthophoto of the same scene. We seek a precise mapping between pixel locations in the orthophoto and the PTZ camera control model. The goal is a method that can determine what pan-tilt camera orientation (θ, φ) is needed to point the camera at a selected ground location (x, y) in the orthophoto (and vice versa). With multiple cameras registered to the same orthophoto scene, the pan-tilt orientation of each camera required to view the same orthophoto ground location can be computed (if the location is visible to each camera). The main advantage of the approach is that all cameras can be registered to a global frame, which then can be used to individually direct each PTZ camera or to coordinate/combine information across multiple
17
Camera Control and Geo-Registration for Video Sensor Networks
253
Fig. 3 Geo-registration. (a) Cropped orthophoto of a portion of the OSU campus. (b) Example coverage zones of multiple PTZ cameras
cameras (e.g., to automatically select the closest camera, to provide hand-off information for tracking across camera views, or to enable fusion). We map/register the PTZ cameras to a large aerial orthophotograph using the PTZ spherical panoramas (constructed using the previously described camera control model), as this registration process becomes an image-to-image (pano-to-ortho) correspondence task that will also allow for easy transfer of the PTZ camera control models to the orthophoto. Given our panoramic image (with a linear fisheye lens view), it is “defished” by projecting rays of the pan-tilt orientations onto the rectilinear orthophoto plane. The method essentially “unwraps” the camera’s angular pan-tilt spherical panorama and aligns it within the rectilinear orthophoto coordinate frame (our orthophoto is in state-plane coordinates). The proposed registration model does not need to know the PTZ camera height, only that the orthophoto contains the viewspace of the camera (panorama coverage) and that the ground lies on a plane. Corresponded feature points between the images (pano, ortho) and the associated pan-tilt camera orientations (given by their polar coordinates in the spherical panorama) are used to formulate a transformation which captures the pano-ortho registration degrees-of-freedom in our problem (defish, scale, rotation, translation). The relationship between selected orthophoto ground locations and the corresponding pan-tilt camera orientations can be reduced to a single transformation (details and derivations are provided in [19]): ⎤ ⎡ ⎤⎡ ⎤ ⎡ tan φ · cos θ a1 a2 tx xortho ⎣ yortho ⎦ = ⎣ a3 a4 ty ⎦ ⎣ tan φ · sin θ ⎦ (3) 1 0 0 1 1 with the pan (θ ) and tilt (φ) of a point location in the panorama, the corresponding orthophoto ground location (xortho , yortho ), a1−4 defish/scale/rotation parame-
254
J.W. Davis
ters, and translation tx,y . Using this formulation and multiple corresponding panoortho locations, the registration parameters can be efficiently determined (e.g., using a least squares formulation). The resulting registration matrix provides a direct mapping from the camera’s pan-tilt space to the orthophoto plane. Similarly, the inverse of this transformation matrix provides the “reverse” mapping of ground locations in the orthophoto back to the pan-tilt orientations needed to point the camera at those locations. We tested our method (using manually selected point correspondences) to register each camera’s viewspace panorama to an orthophoto of the same area. In Fig. 3(b), we show an illustration of the resulting registered coverage zones on the orthophoto for multiple cameras. Each highlighted “spotlight” zone shows the ground coverage extent for a particular camera located at the center of the zone. The range of each zone extent shown in the figure was fixed to maximum interpolation spread of the panorama mapping (at a particular zoom/resolution) to the orthophoto (i.e., distance between panorama points mapped to the orthophoto were limited to be within a certain distance). We also calculated the registration error of the panoortho mapping with three different cameras (using 10–12 feature points per camera) and a 1 foot pixel resolution orthophoto. For each of the registered camera panoramas, we selected approximately 20 ground locations in the panorama and compared their registered pixel locations with the corresponding ground truth locations of the points (determined manually) in the orthophoto. The results gave a mean error of less than 3 feet (3 pixels) on the ground with a standard deviation less than 1 foot. These error values could potentially be reduced with higher resolution orthophotos. With the registration results and our PTZ control model, any pixel location in a live image from the camera (in which we can compute its pan-tilt orientation) can be mapped instantly to its ground location in the orthophoto (e.g., useful for georeferencing a target). In Fig. 4 we show results with the proposed method to geo-
Fig. 4 Geo-referenced pedestrian trajectories
17
Camera Control and Geo-Registration for Video Sensor Networks
255
Fig. 5 Operational interface
reference trajectories of multiple people tracked with our active tracker (previously described). Alternatively, selection of any ground location in the orthophoto within the camera zone(s) can be used with the reverse/inverse transformation matrix to instantly drive a nearby camera (or all nearby cameras) to point at that location.
5 Operational Interface We implemented the control and registration components within an interactive operational interface to demonstrate the research and examine its applicability to realworld surveillance. A large computer console screen is employed to display an orthophoto of the OSU campus, and camera icons are presented at locations of the physical camera network. The user has the ability to open/view a live camera video feed by simply clicking on a camera icon. In addition to use of a joystick, the user has the ability to directly control camera repositioning with our algorithms by (1) clicking anywhere within the live video image, or (2) selecting a ground location in the orthophoto. The system is implemented within the open source NASA World Wind Java platform. A screen-shot of the system showing an orthophoto, camera icons, a live video feed, and other geo-located content (e.g., building footprints and context information) is shown in Fig. 5. We plan to fully examine this interface for various surveillance tasks in conjunction with campus security.
6 Summary To efficiently control and manage large video sensor networks, algorithms are needed to provide intuitive interaction and visualization capabilities. We presented
256
J.W. Davis
novel techniques for real-time PTZ camera control and for geo-registration of camera viewspaces to a single reference map. The camera model was demonstrated by generating panoramic viewspaces and by active tracking of pedestrians. The registration technique was used for geo-referencing pedestrian trajectories to the base map. Our methods are applicable to single and multiple camera configurations and work with standard commercial video cameras typically deployed for surveillance. The framework could conceivably be scaled to much larger, real-world surveillance centers (from street corner to city block to city-wide monitoring). As there is a growing popularity and use of geo-located visual content (e.g., Google Maps, Bing), the approach could easily integrate with other related endeavors. We expect the outcomes of this research to provide new capabilities of actual importance to the surveillance community. Acknowledgements The author wishes to thank graduate students Karthik Sankaranarayanan and Matt Nedrich for their contribution to the work presented, and for partial support provided by the National Science Foundation under grant No. 0236653.
References 1. Barreto, J., Araujo, H.: A general framework for the selection of world coordinate systems in perspective and catadioptric imaging applications. Int. J. Comput. Vis. 57, 23–47 (2004) 2. Basu, A., Ravi, K.: Active camera calibration using pan, tilt and roll. IEEE Trans. Syst. Man Cybern. 27(3), 559–566 (1997) 3. Bernardin, K., van de Camp, F., Stiefelhagen, R.: Automatic person detection and tracking using fuzzy controlled active cameras. In: Proc. Comp. Vis. and Pattern Rec. (2007) 4. Black, J., Ellis, T.: Multi camera image tracking. Image Vis. Comput. 24(11), 1256–1267 (2006) 5. Collins, R., Tsin, Y.: Calibration of an outdoor active camera system. In: Proc. Comp. Vis. and Pattern Rec., pp. 528–534 (1999) 6. Davis, J., Chen, X.: Calibrating pan-tilt cameras in wide-area surveillance networks. In: Proc. Int. Conf. Comp. Vis. (2003) 7. Fry, S., Bichsel, M., Muller, P., Robert, D.: Tracking of flying insects using pan-tilt cameras. J. Neurosci. Methods 101, 59–67 (2000) 8. Gaspar, J., Santos-Victor, J.: Visual path following with a catadioptric panoramic camera. In: International Symposium on Intelligent Robotic Systems—SIRS, pp. 139–147 (1999) 9. Jain, A., Kopell, D., Kakligian, K., Wang, Y.: Using stationary-dynamic camera assemblies for wide-area video surveillance and selective attention. In: Proc. Comp. Vis. and Pattern Rec. (2006) 10. Jethwa, M., Zisserman, A., Fitzgibbon, A.: Real-time panoramic mosaics and augmented reality. In: Brit. Mach. Vis. Conf. (1998) 11. Khan, S., Shah, M.: Consistent labeling of tracked objects in multiple cameras with overlapping fields of view. IEEE Trans. Pattern. Anal. Mach. Intell. 25(10), 1355–1360 (2003) 12. Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., Shafer, S.: Multi-camera multiperson tracking for EasyLiving. In: IEEE International Workshop on Visual Surveillance, pp. 3–10 (2000) 13. Lee, L., Romano, R., Stein, G.: Monitoring activities from multiple video streams: Establishing a common coordinate frame. IEEE Trans. Pattern. Anal. Mach. Intell. 22(8), 758–767 (2000) 14. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)
17
Camera Control and Geo-Registration for Video Sensor Networks
257
15. Miyamoto, K.: Fisheye lens. J. Opt. Soc. Am., 1060–1061 (1964) 16. OGRIP: Ohio Geographically Referenced Information Program (2009). http://ogrip.oit.ohio. gov/ 17. Porikli, F., Tuzel, O., Meer, P.: Covariance tracking using model update based on means on Riemannian manifolds. In: Proc. Comp. Vis. and Pattern Rec. (2006) 18. Sankaranarayanan, K., Davis, J.: An efficient active camera model for video surveillance. In: Proc. Wkshp. Applications of Comp. Vis. (2008) 19. Sankaranarayanan, K., Davis, J.: A fast linear registration framework for multi-camera GIS coordination. In: Proc. Advanced Video and Signal Based Surveillance (2008) 20. Weiss, Y.: Deriving intrinsic images from image sequences. In: Proc. Int. Conf. Comp. Vis. (2001) 21. Woo, D., Capson, D.: 3D visual tracking using a network of low-cost pan/tilt cameras. In: Canadian Conference on Electrical and Computer Engineering, vol. 2, pp. 884–889 (2000)
Chapter 18
Persistent Observation of Dynamic Scenes in an Active Camera Network Bi Song, Chong Ding, Amit Roy-Chowdhury, and Jay Farrell
Abstract This chapter deals with the problem of persistent observation of a wide area scene through decentralized, cooperative control of an active camera network. We focus on applications where events unfold over a large geographic area and need to be analyzed by multiple cameras. There is no central unit accumulating and analyzing all the data. The overall goal is to observe all objects (i.e., targets) in the region of deployment of the cameras, while selectively focusing at a high resolution on some particular target features based on application requirements. Efficient usage of resources in such a scenario requires that the cameras be active. However, this control cannot be based on separate analysis of the sensed video in each camera. They must act collaboratively to be able to acquire multiple targets at different resolutions. Our research focuses on developing accurate and efficient target acquisition and camera control algorithms in such scenarios using game theory. We show real-life experimental results of the approach.
Keywords Camera network · Decentralized processing · Persistent observation · Game theory · Active vision
The authors gratefully acknowledge support from ONR grant N00014-09-1-0666, NSF grant ECS-0622176 and ARO grant W911NF-07-1-0485. B. Song () · C. Ding · A. Roy-Chowdhury · J. Farrell University of California, Riverside, CA, USA e-mail:
[email protected] C. Ding e-mail:
[email protected] A. Roy-Chowdhury e-mail:
[email protected] J. Farrell e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_18, © Springer-Verlag London Limited 2011
259
260
B. Song et al.
1 Introduction Small and large networks of video cameras are being installed in many applications. It is natural to expect that these camera networks would be used to acquire targets at multiple resolutions, e.g., multiple people, a single person, a face. For efficiency and maximum resource utilization, it is desirable to be able to control the cameras based on the requirements of the scene being analyzed. It is also desirable that the control mechanism be decentralized for a number of reasons. In some of these applications, there may be constraints of bandwidth, secure transmission facilities, and difficulty in analyzing a huge amount of data centrally. In such situations, the cameras would have to act as autonomous agents and decisions would have to be taken in a decentralized manner. Also, to achieve an optimal result, the cameras should be working cooperatively with each other trying to achieve the same overall goal. In this chapter, we address the problem of persistently observing a wide area with multiple targets. The targets need to be imaged at different resolutions, as specified by a user. The targets are moving and the resolutions at which to image them may also be changing. Thus, the cameras need to be controlled dynamically so as to meet the imaging requirements. The cooperative and decentralized nature of this problem leads us to explore a game-theoretic solution. Specifically, we employ a framework in which the optimization of local sensor utility functions leads to an optimal value for a global utility. This achieves the goal of observing all targets at an acceptable resolution and some at high resolution. The optimal camera parameters in the sensor network are determined dynamically according to these utility functions and negotiation mechanisms between cameras. To employ suitable negotiation mechanisms between the different sensors is of great importance since the cameras have to take strategic decisions according to the perceived or anticipated actions of the other cameras. This entire framework leads to a decentralized approach. The remainder of this chapter is organized as follows: Sect. 2 presents a rationale for the need of a decentralized collaborative camera network. Section 3 states the problem with its solution in game-theoretic terms. Our experimental results are presented in Sect. 4. We summarize our work in Sect. 5 with some possible directions for future research.
2 Technical Rationale 2.1 Necessity of Collaboration in an Active Camera Network We start by motivating the necessity of a cooperative strategy in an intelligent camera network. For purposes of explanation, we will refer to this problem as Cooperative Target Acquisition (CTA). Two questions we need to address are the following: (i) why do we need active cameras (as opposed to having a network of cameras with a fixed set of parameters) and (ii) why does the control strategy need to be cooperative?
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
261
The main reason for having a dynamically self-configurable network is that it would be prohibitively expensive to have a static setup that would cater to all possible situations. For example, suppose we needed to focus on one person (possibly non-cooperative) or specific features (e.g., face) of the person as he walks around an airport terminal and obtain a high resolution image of him while also observing other activities going on in the terminal. To achieve this, we will either need to dynamically change the parameters of the cameras in which this person is visible or have a setup whereby it would be possible to capture high resolution imagery irrespective of where the person is in the terminal. The second option would be very expensive and a huge waste of resources, both technical and economical. Therefore we need a way to control the cameras based on the sensed data. Currently, similar applications try to cover the entire area or the most important parts of it with a set of passive cameras, and have difficulty in acquiring high resolution shots selectively. The control strategy must necessarily be cooperative because each camera’s parameter settings entail certain constraints on other cameras. For example, if a camera zooms in to focus on the face of one particular person, thus narrowing its field of view (FOV), it risks losing much of the person and the surroundings. Another camera can compensate for this by adjusting its parameters to observe the person with a lower zoom setting. This requires analysis of the video data in a network-centric manner, leading to a cost-effective method to obtain high resolution images for features at dynamically changing locations.
2.2 Necessity of a Decentralized Strategy As the problem complexity increases, it may be difficult to analyze all the data in a centralized manner and come up with an effective strategy for persistently observing the dynamic scene. There may not be enough bandwidth and transmission power available to send all the data to a central station. Furthermore, security of the transmission and interception by a hostile opponent may be a factor in some applications. Finally, even if it is possible to securely transmit all the data to a central unit, it may not be the best strategy as it is often intractable to come up with a centralized optimal policy, given the complex nature of these systems and the environments where they are deployed. In the proposed framework, each camera must make its own decisions based on analysis of its own sensed data and negotiation mechanisms with other sensors. We propose a game-theoretic solution to this problem, the motivation for which is given in Sect. 3.
2.3 Relation to Previous Work On the one hand, the research presented in this chapter is related to two classical problems in robotics and computer vision—sensor planning [5, 11, 15, 16, 20] and
262
B. Song et al.
active vision [1, 3, 4]. However, active vision in a camera network, involving cooperation and coordination between many cameras, is relatively unexplored. On the other hand, the work is related to camera networks, which has recently generated a lot of interest. Some recent work has dealt with networks of vision sensors, namely computing the statistical dependence between cameras, computing the camera network topology, tracking over “blind” areas of the network, and camera handoff [6, 9, 12, 18, 19, 21, 24]. However, these methods do not deal with the issue of actively reconfiguring the network based on feedback from the sensed data. The work that is perhaps most closely related to this chapter is [17]. Here, a virtual camera network environment was used to demonstrate a camera control scheme that is a mixture between a distributed and a centralized scheme using both static and PTZ cameras. Their work focused on how to group cameras which are relevant to the same task in a centralized manner while maintaining the individual groups decentralized. In [7], a solution to the problem of optimal camera placement given some coverage constraints was presented and can be used to come up with an initial camera configuration. Our proposed method is related the vehicle-target assignment problem using game theory as was presented in [2]; however, in that work the targets were not dynamic and it did not consider the constraints imposed by video cameras. Moreover, they had each target assigned to one vehicle, while we can have multiple targets observed in each camera and multiple cameras observing a target (many-to-many mapping). A camera handoff approach using game theory was presented in [10]. This method, however, considers only a set of static cameras and does not deal with the problem of persistently observing targets with varying resolutions over a large geographic area using a dynamic camera network with overlapping and nonoverlapping fields of view.
3 Cooperative Target Acquisition Using Game Theory Our goal is to develop a decentralized strategy for coordinated control that relies on local decision making at the camera nodes, while being aligned with the suitable global criterion of persistently observing multiple targets at multiple resolutions. For this purpose, we propose to use novel game-theoretic ideas that rely on multi-player learning and negotiation mechanisms [8]. The result is a decision-making process that aims to optimize a certain global criterion based on individual decisions by each component (sensor) and the decisions of other interconnected components. In the next section, we provide an intuitive argument to justify the presented solution strategy. Then we describe the criteria that each camera should use to evaluate its own sensed data and the design of the negotiation mechanisms. Finally, we describe the game-theoretic approach for distributed control of the sensor network.
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
263
3.1 Motivation for Game-Theoretic Formulation We assume that each sensor is a rational decision maker, optimizing its own utility function which indirectly translates to the optimization of a global utility function. The utility of a sensor increases when it can image a target at the desired resolution, provided that requirement has not been met by some other sensor. However, there is risk associated with each high resolution image attempt—acquisition of each target at high resolution risks losing the global picture of the relative positions of all the targets and subsequent loss of observation of a target if it leaves the field of view (FOV) of its imaging device. When the number of targets is larger than the number of cameras, the necessity of cooperation between the cameras is obvious. Even when the number of cameras is larger than the number of targets, the benefit of cooperation and information sharing is clear. A camera observing Ti at relatively low zoom allows other cameras to observe other targets at higher zoom levels, provided the state of the camera observing Ti can be communicated to them. In addition, cameras at intermediate zoom levels can allow observation of multiple targets at a sufficient level of accuracy to enable other cameras to attempt higher risk, but higher reward, images at high zoom levels. This shows that a decentralized strategy for camera control is feasible through local decision making and negotiations with neighboring cameras. The objective of the team of cameras is to optimize the global utility function by having each camera optimize its own local utility function, again subject to risk, but now also subject to predictions of the actions of the other cameras. The first step is to find suitable local utility functions such that the objectives of each camera are localized to that camera, yet aligned with a global utility function. The second step is to propose an appropriate negotiation mechanism between cameras to ensure convergence of the distributed solution toward the global solution. The actual computation of these functions depends upon the analysis of the sensed video.
3.2 Precise Problem Statement and Notation Let us consider Nt targets in the entire area of deployment and Nc sensors that need to be assigned to these targets. In the camera network setup, each target will be represented by a location vector and a resolution parameter. Let the cameras be denoted as C = {C1 , . . . , CNc } and the targets as T = {T1 , . . . , TNt }. Each target has an associated resolution parameter ri , i = 1, . . . , Nt . Camera Ci ∈ C will select its own set of targets ai (= {Ti }) ∈ Ai , where Ai is the set of targets that can be assigned to Ci , by optimizing its own utility function UCi (ai ). This is known as target assignment in the game-theory literature. Our problem is to design these utility functions and appropriate negotiation procedures that lead to a mutually agreeable assignment of targets resulting in meeting the global criterion.
264
B. Song et al.
3.3 Game-Theory Fundamentals A well-known concept in game theory is the notion of Nash equilibrium. In the context of our image network problem, it will be defined as a choice of targets ∗ ) such that no sensor could improve its utility further by deviating a ∗ = (a1∗ , . . . , aN c ∗ from a . Obviously, this is a function of time since the targets are dynamic and the sensors could also be mobile or capable of panning, tilting and zooming. For our problem, a Nash equilibrium will be reached at a particular instant of time when all the cameras are observing all the targets in the deployment region at an acceptable resolution and there is no advantage for a particular camera to choose some other target to observe. Mathematically, if a−i denotes the collection of targets for all cameras except camera Ci , then a ∗ is a pure Nash equilibrium if ∗ ∗ UCi (ai∗ , a−i ) = max UCi (ai , a−i ), ai ∈Ai
∀Ci ∈ C.
(1)
3.4 Choice of Utility Functions Target Utility If the number of cameras viewing Ti at an acceptable resolution r0 r is ni 0 , then the utility of covering Ti using a particular assignment profile a is r 1 if ni 0 > 0, UTi (a) = (2) r 0 if ni 0 = 0. Global Utility From the target utility function, we can now define the global utility function as the sum of the utilities generated by observing all the targets, i.e., UTi (a). (3) Ug (a) = Ti
Camera Utility We now come to the all-important question of defining camera utility in a suitable manner so that it is aligned to the global utility function. A target utility, UTi , represents the overall value of observing a target Ti at an acceptable resolution. The camera utility, UCi , represents a particular sensor’s share of this value. We will say that the camera utility is aligned with the global utility when the sensor can take an action improving its own utility if and only if it also improves the global utility [22]. This will require prediction of the actions of team-mates, which will be achieved through the negotiation mechanisms described in the next section. Under this definition, we can use a number of utility functions that have been proposed in the game-theory literature [14, 22]. In our application, we propose to use what is known as Wonderful Life Utility (WLU) [13]. In WLU, the utility of a sensor observing a particular target is the marginal contribution to the global utility as a result of this action—i.e., the sensor utility is the change in the global utility as
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
265
a result of that sensor observing that particular target as opposed to not observing it. The exact expression is UCi (ai , a−i ) = Ug (ai , a−i ) − Ug (a−i ).
(4)
This definition of camera utility works well if the teammate predictions are accurate and stable. As shown in [14], this camera utility leads to a potential game with the global utility function as the potential function, and hence they are aligned with the global utility. This ensures that the resulting set of targets that are chosen will be included within the set of pure Nash equilibria. In particular, the Wonderful Life Utility has been widely used in economics [13].
3.5 Negotiation Mechanisms Observing objects of interest in a dynamic setting requires negotiation mechanisms between the different sensors, allowing them to come up with the strategic decisions described above. Each sensor negotiates with other sensors to (i) accurately estimate the state of the targets, (ii) accurately predict their team-mates’ parameters, and (iii) decide its own action. This makes each sensor truly autonomous thus providing robustness in uncertain and adversarial environments, where there could be a lack of adequate communication resources, and incomplete knowledge of the sensed data and environmental conditions of the other sensors. Hence the cooperation between the sensors is limited to exchanging information about the states of other sensors and targets, not the actual video data. The overall idea of the proposed negotiation strategy is to use learning algorithms for multi-player games [14]. A particularly appealing strategy for this problem is Spatial Adaptive Play (SAP) [23]. This is because it can be implemented with a low computational burden on each camera and leads to an optimal assignment of targets with arbitrarily high probabilities for the WLU described above. Application of SAP Negotiation Mechanism In our method for camera control in a sensor network, we adopt the Spatial Adaptive Play (SAP) strategy [23]. Let us consider a camera Ci that is viewing the area under surveillance at an acceptable resolution. At any step of SAP negotiations, Ci is randomly chosen from the pool of cameras in the network according to a uniform distribution, and only this camera is given the chance to update its proposed parameter settings. At negotiation step k, Ci proposes a parameter setting according to the following probability distribution based on other cameras’ parameters in the previous step: ⎛ ⎡ ⎤⎞ UCi (A1i , a−i (k − 1)) .. ⎜1 ⎢ ⎥⎟ (5) pi (k) = σ ⎝ ⎣ ⎦⎠ . τ |Ai | UCi (Ai , a−i (k − 1)) for some τ > 0, where a(k − 1) denotes the profile of proposed parameter settings |A | at step k − 1, Ai = {A1i , . . . , Ai i } is the enumeration of all possible parameter
266
B. Song et al.
settings of camera Ci , and |Ai | is the number of elements in Ai . Function σ (·) is the logit or soft-max function, and its ith element is defined as σ (x) i =
e xi . ex1 + · · · + exn
In our application, the actual targets are moving. The SAP negotiation mechanism described above might take a relatively long time (>200 steps) to converge to an optimal assignment. Thus, in our implementation, Ci proposes a parameter setting to maximize its own utility based on other cameras’ parameters at the previous step in order to come to a stable assignment (probably a sub-optimal one) sooner. After Ci updates its settings, it broadcasts its parameters (i.e., pan, tilt and zoom) to the entire network. Based on calibration data, the other cameras can then calculate the area covered by Ci and use that information to update its parameters after being chosen at any negotiation step. After the negotiation converges, the entire area of interest is viewed at an acceptable resolution. An implicit assumption is that the amount of time it takes for the assignments to be finalized is less than the time for the targets to move from one camera to another. This will be true if each camera has sufficient processing power and if cameras are assigned to targets only when the targets will be in a cameras’ FOV for a reasonable amount of time. Let us now consider the case where a specific application requires the observation of a target at a high resolution. Initially, if a camera Cj determines that it can view the target at the highest resolution based on the parameters of the other cameras in the network, it starts observing the target. Once Cj takes on the task of observing the target at a high resolution, it will predict the state (i.e., position and velocity) of the target at each time step t + , where is the prediction time. If Cj predicts that it will not be able to view the target after a time tout , it will asynchronously broadcast tout and the target’s predicted position at tout to the entire network indicating that it needs to handoff the observation to another camera. When a camera Ci (i = j ) is now chosen to update its parameters at any negotiation step, it will first determine if it can adjust its parameter to view the target at the desired high resolution based on the predicted target parameters broadcast by Cj . If it can do so, it will set its parameters accordingly and take over the observation of the target. However, if Ci determines it will not be able to view the target at a high resolution, it will simply continue with the negotiation cycle maximizing its own utility as presented above. Once Cj is no longer able to observe the target, it returns to the negotiation cycle to maximize its own utility. This entire negotiation approach based on game theory can be seen in more detail in Algorithms 1–4 (variables are described in Table 1). In these algorithms, t = {t0 , t1 , t2 , . . .} need not be uniformly spaced time instants and will depend upon the time it takes for each target assignment to be complete. We will assume that the computation time to come up with a set of camera parameters, given a target configuration, is much smaller than the time it takes for the target configuration to change. Since all possible targets will always be tracked at a low resolution, this implies that we are concerned only with the time interval for which high resolution targets are chosen.
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
267
Algorithm 1 Maximize utility and negotiate Input: Camera Ci , Video from Ci Ci calculates parameters needed to maximize its utility at r0 based on area covered by other cameras obtained from negotiation mechanism if Ci needs to change its parameters to maximize its utility then Ci changes its parameters Ci broadcasts its parameters (pan/tilt/zoom) to all the cameras in the network end if
Algorithm 2 Initialize cameras to cover entire area at acceptable resolution Initial State: Cameras are not covering entire area at r0 while entire area is not being covered at r0 do Choose a camera Ci randomly according to a uniform distribution Call Algorithm1(Ci ) end while End State: Cameras are covering entire area at r0 . Each target is represented by (Tk , r0 ).
Algorithm 3 Track all targets at an acceptable resolution Initial State: Cameras are covering entire area at r0 for t = t0 , t1 , t2 , . . . do All Ci s observe all (Tk , r0 )s in their FOVs All Ci s predict pˆ k (t + ) for all (Tk , r0 )s in their FOVs if a Ci determines that a (Tkout , r0 ) will move out of its FOV then Ci broadcasts tˆkout and pˆ kout to all the cameras in the network end if if a Ci determines that it can observe (Tkout , r0 ) based on tˆkout and pˆ kout then Ci takes over observing of (Tkout , r0 ) end if end for End State: All cameras are observing all targets at an acceptable resolution
4 Experimental Results To evaluate the proposed game-theoretic method for camera control, we tested our approach on a real-life network of cameras. The area under surveillance is an outdoor region of approximately 10,000 sq. feet. The experiments were done in UCR’s Videoweb Lab which consists of dozens of pan-tilt-zoom network video cameras and a scalable architecture and display that incorporate 34 Windows-based servers for real time processing of the video streams. This laboratory is unique in its deployment of a large number of controllable (pan, tilt, zoom) network cameras in an indoor/outdoor setting, and each having its own IP address. We used a subset of these cameras installed outdoors for our experiments. The cameras used in our experiment are AXIS 215 PTZ network cameras with ±170° pan range, 180° tilt range and 12× optical, 4× digital zoom. The sensor network consists of nine PTZ cameras with a resolution of 320 × 240 pixels. We use this setting to show the performance of our approach with the
268
B. Song et al.
Algorithm 4 Track some targets at high resolution, others at acceptable resolution Initial State: All cameras are observing all targets at r0 handoffHighResTargetFlag = 0 for t = t0 , t1 , t2 , . . . do if a (Tk , rh ) has been selected (manually or automatically) then newHighResTargetFlag = 1 end if Choose a camera Ci randomly according to a uniform distribution if newHighResTargetFlag == 1 AND Ci can observe (Tk , rh ) based on the range of its FOV then Ci starts observing (Tk , rh ) and becomes Ch newHighResTargetFlag == 0 else if handoffHighResTargetFlag == 1 AND Ci can observe (Tkout , rh ) based on tˆkout , pˆ kout being within the range of its FOV then Ci takes over observing of (Tkout , rh ) and becomes Ch Previous Ch becomes Ci handoffHighResTargetFlag == 0 else Call Algorithm1(Ci ) end if Ch predicts pˆ k (t + ) for (Tk , rh ) if Ch determines that (Tk , rh ) will move out of its FOV thus becoming (Tkout , rh ) then Broadcast handoffHighResTargetFlag = 1, tˆkout and pˆ kout of (Tk , rh ) to all the cameras in the network end if end for
Table 1 Variables used in algorithms t
Discrete time step
c {Cl }N l=1
Set of all cameras
t {Tk }N k=1
Set of all targets
(Tk , r0 )
Target being tracked at acceptable resolution r0
(Tk , rh )
Target being tracked at specified high resolution rh
Ci
Camera observing (Tk , r0 )
Ch
Camera observing (Tk , rh )
Prediction time step
pˆ k (t + ) tˆkout
Predicted position of Tk at time t +
pˆ kout
Predicted time in which Tk is expected to leave Cl ’s FOV Predicted position of Tk at tˆkout
Tkout
Target about to leave Cl ’s FOV
increasing number of zooming cameras. It is illustrated in Fig. 1 that when multiple cameras zoom in, how other cameras automatically adjust themselves to cover as much area as they can. Figure 1(a) shows the initialization result to cover the entire area at an acceptable resolution. The coverage is shown on the bottom-right (blue areas are covered,
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
269
Fig. 1 Results of camera control with the number of cameras that zoom in increasing. The coverage of the entire space is showed on the right-bottom of each image, where blue areas are covered, while white areas are not. The darker the color of a block, the more overlapping number of cameras are assigned there. The results with multiple cameras zooming in are shown in (b)–(e). The number of cameras that zoom in is increasing from one to four. From (b) to (e), the view of new zooming in camera is bounded by red lines. Note that as the number of zoomed in cameras increases, more areas are left uncovered. (f) shows the re-initialization results when we reset the cameras with none of them zooming in
white areas are not, the darker the color of a block, the more overlapping cameras are assigned there). Figure 1(b) shows one camera zooming in (bounded with red lines) and other cameras automatically adjusting their settings to keep as much area covered as they can. Figure 1(b)–(e) show the effect of increasing the number of zooming in cameras. With an increasing number of zooming in cameras, the network of cameras is not able to keep the entire area covered (Fig. 1(d) and (e)). Figure 1(f) shows the re-initialization result when we reset the cameras with none of them zooming in—the network of cameras can again keep the entire area covered. By comparing 1(f) with (a), it can be noticed that the parameter settings in (f) are different from those in (a), although both of them could satisfy the coverage requirements. This illustrates that the Nash equilibrium is not unique.
270
B. Song et al.
5 Conclusion We presented in this article a novel approach for camera network control using a game-theoretic framework. While the sensor network is completely decentralized, the cameras work cooperatively toward the same goal of covering the entire area under surveillance while observing all the targets in that area at an acceptable resolution and some at a high resolution. This is achieved through the appropriate choice of utility functions and negotiation mechanisms. Real-life results show that this method can be effectively utilized for decentralized camera network control. In future work, we will study the issues of how tracking error and application requirements will affect the performance of the control algorithm.
References 1. Aloimonos, Y., Weiss, I., Bandyopadhyay, A.: Active vision. Int. J. Comput. Vis. 1(4), 333– 356 (1988) 2. Arslan, G., Marden, J., Shamma, J.: Autonomous vehicle-target assignment: A gametheoretical formulation. ASME J. Dyn. Syst. Meas. Control 129(5), 584–596 (2007) 3. Bajcsy, R.: Active perception. Proc. IEEE 76(8), 996–1005 (1988) 4. Ballard, D.H.: Animate vision. Artif. Intell. 48(1), 57–86 (1991) 5. Blake, A., Yuille, A.: Active Vision. MIT Press, Cambridge (1992) 6. Cai, Q., Aggarwal, J.K.: Automatic tracking of human motion in indoor scenes across multiple synchronized video streams. In: IEEE Intl. Conf. on Computer Vision, pp. 356–362 (1998) 7. Erdem, U.M., Sclaroff, S.: Automated camera layout to satisfy task-specific and floor planspecific coverage requirements. Comput. Vis. Image Underst. 103(3), 156–169 (2006) 8. Fudenberg, D., Levine, D.K.: The Theory of Learning in Games. Series on Economic Learning and Social Evolution. MIT Press, Cambridge (1998) 9. Khan, S., Javed, O., Rasheed, Z., Shah, M.: Camera handoff: Tracking in multiple uncalibrated stationary cameras. In: IEEE Workshop on Human Motion, pp. 113–118 10. Li, Y., Bhanu, B.: Utility based dynamic camera assignment and hand-off in a video network. In: IEEE/ACM International Conference on Distributed Smart Cameras, pp. 1–8 (2008) 11. Marchand, E., Chaumette, F.: Active vision for complete scene reconstruction and exploration. IEEE Trans. Pattern Anal. Mach. Intell. 21(1), 65–72 (1999) 12. Markis, D., Ellis, T., Black, J.: Bridging the gap between cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 205–210 (2004) 13. Mas-Colell, A., Whinston, M.D., Green, J.R.: Microeconomic Theory. Oxford University Press, Oxford (1996) 14. Monderer, D., Shapley, L.S.: Potential games. Games Econ. Behav. 14(1), 124–143 (1996) 15. Olague, G., Mohr, R.: Optimal camera placement for accurate reconstruction. Pattern Recognit. 35, 927–944 (2002) 16. Pito, R.: A solution to the next best view problem for automated surface acquisition. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 1016–1030 (1999) 17. Qureshi, F.Z., Terzopoulos, D.: Surveillance in virtual reality: System design and multi-camera control. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 18. Song, B., Roy-Chowdhury, A.: Stochastic adaptive tracking in a camera network. In: IEEE Intl. Conf. on Computer Vision, pp. 1–8 (2007) 19. Stancil, B., Zhang, C., Chen, T.: Active multicamera networks: From rendering to surveillance. IEEE J. Sel. Top. Signal Process. 2(4), 597–605 (2008). Special Issue on Distributed Processing in Vision Networks
18
Persistent Observation of Dynamic Scenes in an Active Camera Network
271
20. Tarabanis, K., Allen, P., Tsai, R.: A survey of sensor planning in computer vision. IEEE Trans. Robot. Autom. 11, 86–104 (1995) 21. Tieu, K., Dalley, G., Grimson, W.E.L.: Inference of non-overlapping camera network topology by measuring statistical dependence. In: IEEE Intl. Conf. on Computer Vision, vol. 2, pp. 1842–1849 (2005) 22. Wolpert, D., Tumor, K.: An overview of collective intelligence. In: Bradshaw, J.M. (ed.) Handbook of Agent Technology. AAAI Press/MIT Press, Cambridge (1999) 23. Young, H.P.: Individual Strategy and Social Structure: An Evolutionary Theory of Institutions. Princeton University Press, Princeton (1998) 24. Zhao, J., Cheung, S.C., Nguyen, T.: Optimal camera network configurations for visual tagging. IEEE J. Sel. Top. Signal Process. 2(4), 464–479 (2008). Special Issue on Distributed Processing in Vision Networks
Chapter 19
Proactive PTZ Camera Control A Cognitive Sensor Network That Plans Ahead Faisal Z. Qureshi and Demetri Terzopoulos
Abstract We present a visual sensor network—comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras—capable of automatically capturing closeup video of selected pedestrians in a designated area. The passive cameras can track multiple pedestrians simultaneously and any PTZ camera can observe a single pedestrian at a time. We propose a strategy for proactive PTZ camera control where cameras plan ahead to select optimal camera assignment and handoff with respect to predefined observational goals. The passive cameras supply tracking information that is used to control the PTZ cameras.
Keywords Smart cameras · Camera networks · Computer vision · PTZ cameras · Visual surveillance · Persistent human observation
1 Introduction Automated human surveillance systems comprising fixed CCTV cameras can detect and track multiple people, but they perform poorly on tasks that require higherresolution images, such as acquiring closeup facial images for biometric identification. On the other hand, active pan/tilt/zoom (PTZ) cameras can be used to capture high-quality video of relevant activities in the scene. This has led to surveillance systems that combine passive wide field-of-view (FOV) cameras and active PTZ F.Z. Qureshi () Faculty of Science, University of Ontario Institute of Technology, Oshawa, Canada e-mail:
[email protected] D. Terzopoulos Computer Science Department, University of California, Los Angeles, Los Angeles, USA e-mail:
[email protected] B. Bhanu et al. (eds.), Distributed Video Sensor Networks, DOI 10.1007/978-0-85729-127-1_19, © Springer-Verlag London Limited 2011
273
274
F.Z. Qureshi and D. Terzopoulos
cameras. Typically, the PTZ camera control routines rely on the tracking information supplied by the passive cameras. Manual control of PTZ cameras is clearly infeasible for large networks, especially as the number of persons and activities in the scene exceeds the number of available PTZ cameras. Consequently, it is desirable to develop control strategies that enable the PTZ cameras to carry out observation tasks autonomously or with minimal human intervention. The dynamic nature of the observation task greatly complicates the problem of assigning cameras to observe different pedestrians. We tackle the challenging problem of controlling active PTZ cameras in order to capture seamless closeup video of pedestrians present in a designated area. In general, no single camera is able to achieve this goal, as the pedestrians enter and exit the observational ranges of different cameras (Fig. 1(a)). Furthermore the camera network must be able to resolve conflicts that might arise when tasking multiple cameras to observe different pedestrians simultaneously. We treat the control of active PTZ cameras as a planning problem whose solution achieves optimal camera utilization with respect to predefined observational goals. A successful camera assignment and handoff strategy should consider both the short-term and long-term consequences of camera assignments when deciding how to carry out an observational task. Consider, for example, the scenario shown in Fig. 1(b). A control strategy that does not reason about the long-term consequences of camera assignment might task Cam3 to observe the red pedestrian (indicated by the circle) and task Cam2 to observe the green pedestrian (indicated by the square). While this assignment may satisfy the immediate goals, it creates a problem as the green pedestrian continues moving to the right while the red pedestrian continues moving to the left. It is impossible to capture seamless closeup video of the red pedestrian as he moves from Cam3 to Cam1 since there is no overlap between the two cameras. On the other hand, a camera control strategy that reasons about the long-term consequences of
Fig. 1 (a) A camera network for video surveillance consists of camera nodes that can communicate with other nearby nodes. Collaborative, persistent surveillance requires that cameras organize themselves to perform camera handover when the observed subject moves out of the sensing range of one camera and into that of another. (b) The need for planning in camera assignment and handoff: A control strategy that does not reason about the long-term consequences of camera assignments might prefer Assignment 1 over Assignment 2, which would eventually lead to an observation failure
19
Proactive PTZ Camera Control
275
camera assignment should assign Cam2 to the red pedestrian and Cam3 to the green pedestrian, assuming that both pedestrians will continue moving in their current directions. This allows for a seamless handoff between Cam2 and Cam3. The type of research that we report here would be very difficult to carry out in the real world given the expense of deploying and experimenting with an appropriately complex smart camera network in a large public space such as an airport or a train station. Moreover, privacy laws generally restrict the monitoring of people in public spaces for experimental purposes. To bypass the legal and cost impediments, we espouse Virtual Vision, a unique synthesis of computer graphics, artificial life, and computer vision technologies [15]. Virtual Vision is an advanced simulation framework for working with machine vision systems, including smart camera networks, that also offers wonderful rapid prototyping opportunities. Exploiting visually and behaviorally realistic environments, called reality emulators, virtual vision offers significantly greater flexibility and repeatability during the camera network design and evaluation cycle, thus expediting the scientific method and system engineering process. Our companion chapter in this volume provides a more detailed review of the Virtual Vision paradigm.
1.1 Related Work Several authors (e.g., [1, 3, 6, 11]) have studied multicamera issues related to lowlevel sensing, distributed inference, and tracking. Recently, however, the research community has been paying increasing attention to the problem of controlling or scheduling active cameras in order to capture high-resolution imagery of interesting events. High-resolution imagery not only allows for subsequent biometric analysis, it also helps increase the situational awareness of the surveillance system operators. In a typical setup, information gathered by stationary wide-FOV cameras is used to control one or more active cameras [5, 9, 14]. Generally speaking, the cameras are assumed to be calibrated and the total coverage of the cameras is restricted to the FOV of the stationary camera. Nearly all PTZ scheduling schemes rely on site-wide multitarget, multicamera tracking. Numerous researchers have proposed camera network calibration to achieve robust object identification and classification from multiple viewpoints, and automatic camera network calibration strategies have been proposed for both stationary and actively controlled camera nodes [2, 4, 13]. The problems of camera assignment and handoff have mainly been studied in the context of smart camera networks. To perform camera handoffs, [12] construct a distributed lookup table, which encodes the suitability of a camera to observe a specific location. For continuous tracking across multiple cameras, [7] propose the use of a handoff function, which is defined as the ratio of co-occurrence to occurrence for point pairs in two views. Their approach does not require calibration or 3D scene information. [10] develop a game-theoretic approach to achieve camera handoffs. When a target is visible in multiple cameras, the best camera is selected based on its expected utility. They also propose a number of criteria to construct the utility
276
F.Z. Qureshi and D. Terzopoulos
function, such as the number of pixels occupied by the selected target in an image. Their approach eschews spatial and geometric information. [8] develop a probabilistic framework for selecting the “dominant” camera for observing a pedestrian, defined as the camera with the highest proximity probability, which is computed as the ratio of the foreground blocks occupied by the selected pedestrian and the angular distance between the camera and that pedestrian. [18] present a game-theoretic strategy for cooperative control of a set of decentralized cameras. The cameras work together to track every target in the area at acceptable image resolutions. The camera network can also be tasked to record higher-resolution imagery of a selected target. Our work on proactive camera control differs from prior work in an important way. With the notable exception of [9], existing schemes for camera assignment do not reason about the long-term consequences of camera assignments and handoffs. In essence, existing schemes are purely reactive. By contrast, the strategy introduced in this paper is proactive and deliberative. When searching for the best current camera assignment, it considers the future consequences of possible camera assignments. The ability to reason about the future enables our system to avoid camera assignments that might appear optimal at present, but will eventually lead to observation and tracking failures. For an overview of