Innovations in Defence Support Systems - 3: Intelligent Paradigms in Security

Paolo Remagnino, Dorothy N. Monekosso, and Lakhmi C. Jain (Eds.) Innovations in Defence Support Systems – 3

Studies in Computational Intelligence, Volume 336 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 314. Lorenzo Magnani, Walter Carnielli, and Claudio Pizzi (Eds.) Model-Based Reasoning in Science and Technology, 2010 ISBN 978-3-642-15222-1 Vol. 315. Mohammad Essaaidi, Michele Malgeri, and Costin Badica (Eds.) Intelligent Distributed Computing IV, 2010 ISBN 978-3-642-15210-8 Vol. 316. Philipp Wolfrum Information Routing, Correspondence Finding, and Object Recognition in the Brain, 2010 ISBN 978-3-642-15253-5 Vol. 317. Roger Lee (Ed.) Computer and Information Science 2010 ISBN 978-3-642-15404-1 Vol. 318. Oscar Castillo, Janusz Kacprzyk, and Witold Pedrycz (Eds.) Soft Computing for Intelligent Control and Mobile Robotics, 2010 ISBN 978-3-642-15533-8 Vol. 319. Takayuki Ito, Minjie Zhang, Valentin Robu, Shaheen Fatima, Tokuro Matsuo, and Hirofumi Yamaki (Eds.) Innovations in Agent-Based Complex Automated Negotiations, 2010 ISBN 978-3-642-15611-3 Vol. 320. xxx Vol. 321. Dimitri Plemenos and Georgios Miaoulis (Eds.) Intelligent Computer Graphics 2010 ISBN 978-3-642-15689-2 Vol. 322. Bruno Baruque and Emilio Corchado (Eds.) Fusion Methods for Unsupervised Learning Ensembles, 2010 ISBN 978-3-642-16204-6 Vol. 323. Yingxu Wang, Du Zhang, and Witold Kinsner (Eds.) Advances in Cognitive Informatics, 2010 ISBN 978-3-642-16082-0 Vol. 324. Alessandro Soro, Vargiu Eloisa, Giuliano Armano, and Gavino Paddeu (Eds.) Information Retrieval and Mining in Distributed Environments, 2010 ISBN 978-3-642-16088-2

Vol. 325. Quan Bai and Naoki Fukuta (Eds.) Advances in Practical Multi-Agent Systems, 2010 ISBN 978-3-642-16097-4 Vol. 326. Sheryl Brahnam and Lakhmi C. Jain (Eds.) Advanced Computational Intelligence Paradigms in Healthcare 5, 2010 ISBN 978-3-642-16094-3 Vol. 327. Slawomir Wiak and Ewa Napieralska-Juszczak (Eds.) Computational Methods for the Innovative Design of Electrical Devices, 2010 ISBN 978-3-642-16224-4 Vol. 328. Raoul Huys and Viktor K. Jirsa (Eds.) Nonlinear Dynamics in Human Behavior, 2010 ISBN 978-3-642-16261-9 Vol. 329. Santi Caball´e, Fatos Xhafa, and Ajith Abraham (Eds.) Intelligent Networking, Collaborative Systems and Applications, 2010 ISBN 978-3-642-16792-8 Vol. 330. Steffen Rendle Context-Aware Ranking with Factorization Models, 2010 ISBN 978-3-642-16897-0 Vol. 331. Athena Vakali and Lakhmi C. Jain (Eds.) New Directions in Web Data Management 1, 2011 ISBN 978-3-642-17550-3 Vol. 332. Jianguo Zhang, Ling Shao, Lei Zhang, and Graeme A. Jones (Eds.) Intelligent Video Event Analysis and Understanding, 2011 ISBN 978-3-642-17553-4 Vol. 333. Fedja Hadzic, Henry Tan, and Tharam S. Dillon Mining of Data with Complex Structures, 2011 ISBN 978-3-642-17556-5 Vol. 334. Álvaro Herrero and Emilio Corchado (Eds.) Mobile Hybrid Intrusion Detection, 2011 ISBN 978-3-642-18298-3 Vol. 335. Radomir S. Stankovic and Jaakko Astola From Boolean Logic to Switching Circuits and Automata, 2011 ISBN 978-3-642-11681-0 Vol. 336. Paolo Remagnino, Dorothy N. Monekosso, and Lakhmi C. Jain (Eds.) Innovations in Defence Support Systems – 3, 2011 ISBN 978-3-642-18277-8

Paolo Remagnino, Dorothy N. Monekosso, and Lakhmi C. Jain (Eds.)

Innovations in Defence Support Systems – 3 Intelligent Paradigms in Security

123

Dr. Paolo Remagnino

Prof. Lakhmi C. Jain

Kingston University Faculty of Computing, Information Systems and Mathematics Penrhyn Road Campus Kingston upon Thames Surrey KT1 2EE United Kingdom E-mail: [email protected]

SCT-Building University of South Australia Adelaide Mawson Lakes Campus South Australia Australia E-mail: [email protected]

Dr. Dorothy N. Monekosso University of Ulster at Jordanstown Faculty of Computing and Engineering School of Computing and Mathematics Shore Road BT37 0QB Newtownabbey United Kingdom E-mail: [email protected]

ISBN 978-3-642-18277-8

e-ISBN 978-3-642-18278-5

DOI 10.1007/978-3-642-18278-5 Studies in Computational Intelligence

ISSN 1860-949X

c 2011 Springer-Verlag Berlin Heidelberg  This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed on acid-free paper 987654321 springer.com

This book collection is dedicated to all researchers in the field of intelligent environments.

Preface

Intelligent Paradigms in Security is a collection of articles introducing the latest advances in the field of intelligent monitoring. The book is intended for readers from a computer science or an engineering background. It describes techniques for the interpretation of sensor data in the context of automatic understanding of complex scenes captured in large public areas. The aim is to guide the reader through a number of research topics for which the existing video surveillance literature describes partial and incomplete solutions, and introduces the next challenges in intelligent video surveillance. Each chapter in the book presents a solution to one aspect of the problem of monitoring a public space. The types of environment of interest are often characterized by clutter and complex interactions between people and between people and objects. Each chapter proposes a sophisticated solution to a specific problem. Public environments, such as an airport concourse, a shopping mall, a train station and similar public spaces, are large and require numerous sensors to monitor the environment. The deployment of a large number of sensors produces a large quantity of video data (petabytes or larger) that must be processed; in addition different scenes may require processing at a different level of granularity. Service robots might soon inhabit public areas, collecting global information but also approaching regions / areas of interest in scene to collect more detailed information, such as an abandoned luggage. The processing of visual data or data from any sensor modality must occur at a speed consistent with the objective / goal. This might be real-time, although some data and information can be processed off-line, for instance to generate new knowledge about the scene for the purpose of enhancing the system. The performance of systems must be evaluated according to criteria that include time latency, accuracy in the detection of an event or the global dynamics, response to queries about the event monitored or the expected dynamics in the environment. To achieve this, the concept of normality is used. A model of what constitutes normality permits deviations and/or large anomalies in the behavior of people or position of objects in the monitored environment to be detected. We are confident that our collection will be of great use to practitioners of the covered fields of research, but could also be formative for doctoral students and researchers in

VIII

Preface

intelligent environments. We wish to express our gratitude to the authors and reviewers for their time and vision as well as to the Springer for the assistance during the publication phase of the book. London, Belfast, Adelaide September 2010

Paolo Remagnino Dorothy N. Monekosso Lakhmi Jain

Acknowledgements

The editors wish to thank all the contributors of the collection for their endeavors and patience.

Contents

1

2

Data Fusion in Modern Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lauro Snidaro, Ingrid Visentini, Gian Luca Foresti 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Terminology in Data Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Motivation to Sensor Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.3 The JDL Fusion Process Model . . . . . . . . . . . . . . . . . . . . . . . 1.2 Data Fusion and Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 JDL Model Contextualized to Surveillance: An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 A Closer Look to Fusion in Level 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Data Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Feature Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 Classifier Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.4 Combiner Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.5 An Example: Target Tracking via Classification . . . . . . . . . . 1.4 Sensor Management: A New Paradigm for Automatic Video Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distributed Intelligent Surveillance Systems Modeling for Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefano Maludrottu, Alessio Dore, Carlo S. Regazzoni 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Surveillance Systems as Data Fusion Architectures . . . . . . . 2.1.2 Surveillance Tasks Decomposition . . . . . . . . . . . . . . . . . . . . . 2.2 Intelligent Surveillance System Modeling . . . . . . . . . . . . . . . . . . . . . 2.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Smart Sensor Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 3 4 5 6 7 8 9 10 10 11 12 13 16 17 23 24 25 27 28 28 29

XII

3

4

5

Contents

2.2.3 Fusion Node Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Control Center Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 Performance Evaluation of Multi-sensor Architectures . . . . 2.3 A Case Study: Multi-sensor Architectures for Tracking . . . . . . . . . . 2.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Multisensor Tracking System Simulator . . . . . . . . . . . . . . . . 2.3.3 Performance Evaluation of Tracking Architectures . . . . . . . 2.3.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30 31 32 33 33 34 38 39 43 43

Incremental Learning on Trajectory Clustering . . . . . . . . . . . . . . . . . . Luis Patino, Franc¸ois Bremond, Monique Thonnat 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 General Structure of the Proposed Approach . . . . . . . . . . . . . . . . . . . 3.3 On-Line Processing: Real-Time Object Detection . . . . . . . . . . . . . . . 3.4 Trajectory Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Object Representation: Feature Analysis . . . . . . . . . . . . . . . . 3.4.2 Incremental Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Trajectory Analysis Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

47

Highly Accurate Estimation of Pedestrian Speed Profiles from Video Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Panagiotis Sourtzinos, Dimitrios Makris, Paolo Remagnino 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Motion Detection and Tracking . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Static Foot Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Speed Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System-Wide Tracking of Individuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christopher Madden, Massimo Piccardi 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Shape Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

47 50 51 52 53 53 54 59 64 65 66 68 69 71 71 72 73 73 73 77 78 80 80 83 83 87 88

Contents

XIII

5.2.2 Appearance Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.2.3 Mitigating Illumination Effects . . . . . . . . . . . . . . . . . . . . . . . . 94 5.3 Feature Fusion Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6

7

A Scalable Approach Based on Normality Components for Intelligent Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Javier Albusac, Jos´e J. Castro-Schez, David Vallejo, Luis Jim´enez-Linares, Carlos Glez-Morcillo 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Formal Model to Build Scalable and Flexible Surveillance Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 General Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Normality in Monitored Environments . . . . . . . . . . . . . . . . . 6.3.3 Global Normality Analysis by Aggregating Independent Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Model Application: Trajectory Analysis . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Normal Trajectory Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Set of Variables for the Trajectory Definition . . . . . . . . . . . . 6.4.3 Preprocessing Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.4 Constraint Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distributed Camera Overlap Estimation – Enabling Large Scale Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anton van den Hengel, Anthony Dick, Henry Detmold, Alex Cichowski, Christopher Madden, Rhys Hill 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Activity Topology and Camera Overlap . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Estimating Camera Overlap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 Joint Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Measurement and Discrimination . . . . . . . . . . . . . . . . . . . . . . 7.4.3 The Original Exclusion Estimator . . . . . . . . . . . . . . . . . . . . . 7.4.4 Accuracy of Pairwise Occupancy Overlap Estimators . . . . . 7.5 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 The Exclusion Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Partitioning of the Joint Sampling Matrices . . . . . . . . . . . . . 7.5.3 Analysis of Distributed Exclusion . . . . . . . . . . . . . . . . . . . . .

105

106 107 109 109 112 119 122 123 124 126 128 131 141 143 147

147 149 150 151 152 154 156 159 159 161 163 164 170

XIV

8

Contents

7.5.4 Evaluation of Distributed Exclusion . . . . . . . . . . . . . . . . . . . . 7.6 Enabling Network Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

172 176 180 181

Multi-robot Teams for Environmental Monitoring . . . . . . . . . . . . . . . . Maria Valera Espina, Raphael Grech, Deon De Jager, Paolo Remagnino, Luca Iocchi, Luca Marchetti, Daniele Nardi, Dorothy Monekosso, Mircea Nicolescu, Christopher King 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Overview of the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Video-Surveillance with Static Cameras . . . . . . . . . . . . . . . . 8.2.2 Multi-robot Monitoring of the Environment . . . . . . . . . . . . . 8.2.3 Experimental Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Multi-robot Patrolling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.2 Multi-robot Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.3 Task Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 Automatic Video Surveillance . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Representation Formalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Event-Driven Distributed Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.1 Layered Map for Event Detection . . . . . . . . . . . . . . . . . . . . . 8.5.2 From Events to Tasks for Threat Response . . . . . . . . . . . . . . 8.5.3 Strategy for Event-Driven Distributed Monitoring . . . . . . . . 8.6 Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 A Real-Time Multi-tracking Object System for a Stereo Camera – Scenario 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.2 Maximally Stable Segmentation and Tracking for Real-Time Automated Surveillance – Scenario 2 . . . . . . . . . 8.6.3 Multi-robot Environmental Monitoring . . . . . . . . . . . . . . . . . 8.6.4 System Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

183

183 185 186 186 186 187 187 188 189 189 191 191 193 193 195 196 197 197 198 202 204 206 207

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

The Editors

Dr Paolo Remagnino (PhD 1993, University of Surrey) is a Reader in the Faculty of Computing, Information Systems and Mathematics at Kingston University. His research interests include image and video understanding, pattern recognition, machine learning, robotics and artificial intelligence.

Dr Dorothy N. Monekosso (MSc. 1992 and PhD 1999, from University of Surrey) is a Senior Lecturer in the School of Computing and Mathematics at the University of Ulster at Jordanstown, Northern Ireland. Her research interests include machine learning, intelligent systems and intelligent environments applied to assisted living and robotics.

XVI

The Editors

Professor Lakhmi C. Jain is a Director/Founder of the Knowledge-Based Intelligent Engineering Systems (KES) Centre, located in the University of South Australia. He is a fellow of the Institution of Engineers Australia. His interests focus on the artificial intelligence paradigms and their applications in complex systems, art-science fusion, virtual systems, e-education, e-healthcare, unmanned air vehicles and intelligent agents.

List of Contributors

Chapter 1 L.Snidaro · I.Visentini · G.L.Foresti Department of Mathematics and Computer Science, University of Udine, 33100 Udine, Italy, [email protected] Chapter 2 S. Maludrottu · C.S. Regazzoni Department of Biophysical and Electronic Engineering (DIBE) - University of Genoa, Genoa, ITALY A.Dore Institute of Biomedical Engineering, Imperial College London, London, SW7 2AZ, UK, [email protected] Chapter 3 Luis Patino · Franc¸ois Bremond · Monique Thonnat INRIA Sophia Antipolis - M´editerran´ee, 2004 route des Lucioles - BP 93 - 06902 Sophia Antipolis, Jose-Luis.Patino [email protected] Chapter 4 Panagiotis Sourtzinos Dimitrios Makris · Paolo Remagnino Digital Imaging Research Centre, Kingston University, UK, [email protected] Chapter 5 Christopher Madden University of Adelaide, Australian Centre for Visual Technologies, Adelaide, SA 5007, Australia

XVIII

List of Contributors

Massimo Piccardi University of Technology, Sydney, Department of Computer Science, Ultimo, NSW 2007, Australia, [email protected] Chapter 6 J.Albusac · J.J.Castro-Schez · D.Vallejo · L.Jim´enez-Linares · C.Glez-Morcillo Escuela Superior de Inform´atica, Universidad de Castilla-La Mancha, Paseo de la Universidad, 4, 13071 Ciudad Real, Spain, [email protected] Chapter 7 Anton van den Hengel · Anthony Dick · Henry Detmold · Alex Cichowski · Christopher Madden · Rhys Hill University of Adelaide, Australian Centre for Visual Technologies, Adelaide, SA 5007, Australia, [email protected] Chapter 8 Maria Valera Espina · Raphael Grech · Deon De Jager · Paolo Remagnino Digital Imaging Research Centre, Kingston University, London, UK Luca Iocchi · Luca Marchetti · Daniele Nardi Department of Computer and System Sciences University of Rome “La Sapienza”, Italy Dorothy Monekosso Computer Science Research Institute, University of Ulster, UK Mircea Nicolescu · Christopher King Department of Computer Science and Engineering,University of Nevada, Reno [email protected]

Chapter 1

Data Fusion in Modern Surveillance Lauro Snidaro, Ingrid Visentini, and Gian Luca Foresti

Abstract. The performances of the systems that fuse multiple data coming from different sources are deemed to benefit from the heterogeneity and the diversity of the information involved. The rationale behind this theory is the capability of one source to compensate the error of another, offering advantages such as increased accuracy and failure resilience. While in the past ambient security systems were focused on the extensive usage of arrays of single-type sensors, modern scalable automatic systems can be extended to combine multiple information coming from mixed-type sources. All this data and information can be exploited and fused to enhance situational awareness in modern surveillance systems. From biometrics to ambient security, from robotics to military applications, the blooming of multi-sensor and heterogeneous-based approaches confirms the increasing interest in the data fusion field. In this chapter we want to highlight the advantages of the fusion of information coming from multiple sources for video surveillance purposes. We are thus presenting a survey of existing methods to outline how the combination of heterogeneous data can lead to better situation awareness in a surveillance scenario. We also discuss a new paradigm that could be taken into consideration for the design of next generation surveillance systems.

1.1 Introduction Interest in automatic surveillance systems has gained significant attention in the past few years. This is due to an increasing need for assisting and extending the capabilities of human operators in remotely monitoring large and complex spaces such as public areas, airports, railway stations, parking lots, bridges, tunnels, etc. The last generation of surveillance systems was designed to cover larger and larger areas dealing with multiple streams from multiple sensors [61, 12, 20]. Their Lauro Snidaro · Ingrid Visentini · Gian Luca Foresti Department of Mathematics and Computer Science, University of Udine 33100 Udine, Italy P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 1–21. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

2

L. Snidaro, I. Visentini, and G.L. Foresti

are meant to automatically assess the ongoing activities in the monitored environment flagging and presenting to the operator suspicious events as they happen in order to prevent dangerous situations. A key step that can help in carrying out this task is analysing the trajectories of the objects in the scene and comparing them against known patterns. The system can in fact be trained by the operator with models of normal and suspicious trajectories in the domain at hand. As recent research shows, this process can even be carried out semi-automatically [66]. Real-time detection, tracking, recognition and activity understanding of moving objects from multiple sensors represent fundamental issues to be solved in order to develop surveillance systems that are able to autonomously monitor wide and complex environments. Research can be conducted at different levels, from the preprocessing steps, involving mainly image processing algorithms, to the analysis and extraction of salient features to represent the objects moving in the scene, to their classification, to the tracking of their movements, and to the analysis of their behaviour. The algorithms that are needed span therefore from image processing to event detection and behaviour understanding, and each of them requires dedicated study and research. In this context, data fusion plays a pivotal role in managing the information and improving system performance. Data fusion is a relatively old concept: from the 70s, when it bloomed in the United States, through the 90s until nowadays this research field has been quite popular due to its polymorphism, and to the effective benefits it provides. The manifest example of human and animal senses has been a critical motivation for integrating information from multiple sources for a reliable and feature-rich perception. In fact, even in case of sensor deprivation, biological systems are able to compensate for lacking information by reusing data obtained from sensors with an overlapping scope. Based on this domain model, the individual interacts with the environment and makes decisions about present and future actions [26]. We can say that two main motivations brought data fusion to its actual point of evolution. First of all, the amount of collected information was growing very fast coherently with the expansion of commodity hardware. Storing the consequent increasing volume of available data was considered often too expensive in terms of resources like space and time. Thus, the idea to “condensate” all this data into a single decisions set, or into a subset of information with reduced dimensionality (but rich in semantic), has rapidly taken place [28]. As a natural evolution, new forms of sensors have evolved in the last two decades opening the door to a wide range of possibilities and potential. On the other hand, the concepts of redundancy, uncertainty and sensor feasibility promoted the good marriage between sensors and fusion algorithms. The resemblance with human and animal senses encouraged to integrate information from heterogeneous sources to improve the knowledge of the observed environment. The natural ability to fuse multi-sensory signals has evolved to a high degree in many animal species and it is in use for millions of years. Today the application of fusion concepts in technical areas has constituted a new discipline, that spans over many

1

Data Fusion in Modern Surveillance

3

fields of science, from economics [4, 36] to military applications [28], from commercial software [27] to videosurveillance [64] and multimodal applications [62, 33] to cite some.

1.1.1 Terminology in Data Fusion At this point, it is necessary to formalize the concepts regarding data fusion. Unfortunately, there is not much agreement on the terminology for fusion systems. The terms sensor fusion, data fusion, information fusion, multi-sensor data fusion, and multi-sensor integration have been widely used in the technical literature to refer to a variety of techniques, technologies, systems, and applications that use data derived from multiple information sources. Fusion applications range from real-time sensor fusion for the navigation of mobile robots to the on-line fusion of human or technical strategic intelligence data [63]. Several attempts have been made to define and categorize fusion terms and techniques. In [73], Wald proposes the term “data fusion” to be used as the overall term for fusion. However, while the concept of data fusion is easy to understand, its exact meaning varies from one scientist to another. Wald uses “data fusion” for “a formal framework in which are expressed the means and tools for the alliance of data originating from different sources. It aims at obtaining information of greater quality; the exact definition of greater quality will depend upon the application”. This is also the meaning given by the Geoscience and Remote Sensing Society, by the U. S. Department of Defense, and in many papers regarding motion tracking, remote sensing, and mobile robots. Unfortunately, the term has not always been used in the same meaning during the last years. In some models, data fusion is used to denote fusion of raw data [15]. There are classic books on fusion like “Multisensor Data Fusion” [74] by Waltz and Llinas and Hall’s “Mathematical Techniques in Multisensor Data Fusion” [29] that propose an extended term, multisensor data fusion. According to Hall, it is defined as the “technology concerned with the combination of data from multiple (and possible diverse) sensors in order to make inferences about a physical event, activity, or situation”. To avoid confusion on the meaning, Dasarathy decided to use the term information fusion as the overall term for fusion of any kind of data [16]. As a matter of fact, information fusion is an all-encompassing term covering all aspects of the fusion field. A literal definition of information fusion can be found in [16]: “Information Fusion. encompasses theory, techniques and tools conceived and employed for exploiting the synergy in the information acquired from multiple sources (sensor, databases, information gathered by human, etc.) such that the resulting decision or action is in some sense better (qualitatively or quantitatively, in terms of accuracy, robustness, etc.) than would be possible if any of these sources were used individually without such synergy exploitation”. By defining a subset of information fusion, the term sensor fusion is introduced as:

4

L. Snidaro, I. Visentini, and G.L. Foresti

“Sensor Fusion. is the combining of sensory data or data derived from sensory data such that the resulting information is in some sense better than would be possible when these sources were used individually”.

1.1.2 Motivation to Sensor Arrays Systems that employ sensor fusion methods expect a number of benefits over single sensor systems. A physical sensor measurement generally suffers from the following problems: • Sensor failure: The deprivation of the sensor means the total loss of the input. • Limited spatial coverage: Depending on the environment, a single sensor could simply be not enough. An obvious example, a fixed camera cannot monitor areas wider than its field of view. • Limited temporal coverage: Some sensors could not be able to provide data at all times. Usually, color cameras are useless at night without proper illumination, being substituted by infrared ones. • Imprecision: The accuracy of the data is the accuracy of the sensor. • Uncertainty: Uncertainty, in contrast to imprecision, depends on the object being observed rather than the observing device. Uncertainty arises when features are missing (e. g., occlusions), when the sensor cannot measure all relevant attributes of the percept, or when the observation is ambiguous [51]. The video surveillance application example is self-explanatory. A single camera cannot provide alone the needed spatial and temporal coverage. Data could be extremely inaccurate since precision is seriously affected by the camera-target distance: the closer the target is to the limit of the field of view, the more imprecise is the information on the target’s location the camera can provide. Noise and occlusions are the main sources for detection uncertainty in this case. Not to mention the total blindness given by sensor failures. On the contrary, broad areas can be monitored 24/7 by a network of heterogeneous sensors. Robust behaviour against sensor deprivation can be achieved by using cameras with overlapping fields of view. This also greatly reduces uncertainty since multiple detections of the same target from different views can be available. This can help discriminating true positives (target’s presence) from false positives (noise), and false negatives (occlusions) from true negatives (target’s absence). The following advantages can be expected from the fusion of sensor data from a set of heterogeneous or homogeneous sensors [8]: • Robustness and reliability: Multiple sensor suites have an inherent redundancy which enables the system to provide information even in case of partial failure. • Extended spatial and temporal coverage: One sensor can look where others cannot respectively can perform a measurement while others cannot. • Increased confidence: A measurement of one sensor is confirmed by measurements of other sensors covering the same domain.

1

Data Fusion in Modern Surveillance

5

• Reduced ambiguity and uncertainty: Joint information reduces the set of ambiguous interpretations of the measured value. • Robustness against interference: By increasing the dimensionality of the measurement space (e. g., measuring the desired quantity with optical sensors and ultrasonic sensors) the system becomes less vulnerable against interference. • Improved resolution: When multiple independent measurements of the same property are fused, the resolution of the resulting value is better than a single sensor’s measurement. In [59], the performance of sensor measurements obtained from an appropriate fusing process is compared to the measurements of the single sensor. According to this work, an optimal fusing process can be designed, if the distribution function describing measurement errors of one particular sensor is precisely known. Intuitively, if all this information is available, this optimal fusing process performs at least as well as the best single sensor. A further advantage of sensor fusion is the possibility of reducing system complexity. In a traditionally designed system the sensor measurements are fed into the application, which has to cope with a big number of imprecise, ambiguous and incomplete data streams. This is also the case for multisensor integration as will be described later. In a system where sensor data is preprocessed by fusion methods, the input to the controlling application can be standardized independently of the employed sensor types, thus facilitating application implementation and providing the capability of adapting the number and type of employed sensors without changing the software [19].

1.1.3 The JDL Fusion Process Model Several fusion process models have been developed over the years. The first and most known originates from the US Joint Directors of Laboratories (JDL) in 1985 under the guidance of the Department of Defense (DoD). The JDL model [28] comprises five levels of data processing and a database, which are all interconnected by a bus. The five levels are not meant to be processed in a strict order and can also be executed concurrently. Steinberg and Bowman proposed revisions and expansions of the JDL model involving broadening the functional model, relating the taxonomy to fields beyond the original military focus, and integrating a data fusion tree architecture model for system description, design, and development [67]. This updated model, sketched in Figure 1.1, is composed by the following levels: • Level 0 - Sub-Object Data Assessment: estimation and prediction of signal/object observable states on the basis of pixel/signal level data association and characterization; • Level 1 - Object Assessment: estimation and prediction of entity states on the basis of observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID);

6

L. Snidaro, I. Visentini, and G.L. Foresti

Fig. 1.1 The JDL data fusion process model.

• Level 2 - Situation Assessment: estimation and prediction of relations among entities, to include force structure and cross force relations, communications and perceptual influences, physical context, etc.; • Level 3 - Impact Assessment: estimation and prediction of effects on situations of planned or estimated/predicted actions by the participants; to include interactions between action plans of multiple players (e.g. assessing susceptibilities and vulnerabilities to estimated/predicted threat actions given ones own planned actions); • Level 4 - Process Refinement: adaptive data acquisition and processing to support mission objectives. The model is deliberately very abstract which sometimes makes it difficult to properly interpret its parts and to appropriately apply it to specific problems. However, as already mentioned, it was originally conceived more as a basis for common understanding and discussion between scientists rather than a real guide for developers in identifying the methods that should be used [28]. A recent paper by Llinas et al. [37] suggests revisions and extensions of the model in order to cope with issues and functions of nowadays applications. In partcular, further extensions of the JDL Model-version are proposed with an emphasis in four areas: (1) remarks on issues related to quality control, reliability, and consistency in data fusion (DF) processing, (2) assertions about the need for co-processing of abductive/inductive and deductive inferencing processes, (3) remarks about the need for and exploitation of an ontologically-based approach to DF process design, and (4) discussion on the role for Distributed Data Fusion (DDF).

1.2 Data Fusion and Surveillance While in the past ambient security systems were focused on the extensive usage of arrays of single-type sensors [34, 71, 38, 50], modern surveillance systems aim to combine information coming from different types of sources. Multi-modal systems [62, 33], even more often used in biometrics, or multi-sensor multi-cue approaches

1

Data Fusion in Modern Surveillance

7

[45, 21] fuse heterogeneous data in order to provide a more robust response and enhance situational awareness. The JDL model presented in section 1.1 can be contextualized and fitted to a surveillance context. In particular, we can imagine a typical surveillance scenario where multiple cameras monitor a wide area. A concrete example on how the levels of the JDL scheme can be reinterpreted can be found in Figure 1.2. In the proposed example, the levels correspond to specific video-surveillance tasks or patterns as follows: • Level 0 - Sub-Object Data Assessment: the raw data streams coming from the cameras can be individually pre-processed. For example, they can be filtered to reduced noise, processed to increase contrast, scaled down to reduce the processing time of subsequent elaborations. • Level 1 - Object Assessment: multiple objects in the scene (e.g. typically pedestrians, vehicles, etc.) can be detected, tracked, classified and recognized. The objects are the entities of the process, but no relationships are involved yet at this point. Additional data as, for instance, the map or sensible areas are a priori contextual information. • Level 2 - Situation Assessment: spatial or temporal relationships between entities are here drawn: a target moving, for instance, from a sensible Zone1 to

Fig. 1.2 Example of contextualization of the the JDL scheme of Figure 1.1 to a surveillance scenario.

8

L. Snidaro, I. Visentini, and G.L. Foresti

Fig. 1.3 Several fusion levels [41] in JDL Level 1.

Zone2 can constitute an event. Simple atomic events are built considering brief stand-alone actions, while more complex events are obtained joining several simple events. Possible alarms to the operator are given at this point. • Level 3 - Impact Assessment: a prediction of an event can be an example of what, in practice, may happen at this step. An estimation of a trajectory of a potential target, or a prediction of the behaviour of an entity can be a focus of this level. For instance, knowing that an object crossed Zone1 heading to Zone2 , we can presume it will cross even Zone3 according to the current trajectory. • Level 4 - Process Refinement: after the prediction given by Level 3, several optimization can be taken in this phase regarding all the previous levels. For instance, the sensors can be relocated to better monitor Zone3 , new thresholds can be imposed in Level 0 procedures, or different algorithms can be employed in Level 1.

1.3 A Closer Look to Fusion in Level 1 Within Level 1 of the JDL model, several fusion approaches can be considered. For instance, we can outline a hierarchy of different fusion steps, as shown in Figure 1.3, that combine data, features, and classifiers with various criteria [41]. The data level aims to fuse different data sets, to provide, for instance, different views of the same object. The feature-based level merges the output of different features, like colour histograms rather than shape features or the response of a heterogeneous feature set. The classifier level focuses on the use of several heterogeneous base classifiers, while the combination level is dedicated to the study of different combiners that follow various ensemble fusion paradigms. Under these assumptions, each level can be considered as a black box, in which different functions and methods can be alternated transparently.

1

Data Fusion in Modern Surveillance

9

More in general, this taxonomy reflects a bottom-up processing, from a low-level fusion that involves raw data coming straight from sensors to high-level combination of abstract and refined information. The data flow through the levels can be considered as a fan-in-tree, as each step provides a response to higher levels that receive input from the previous ones. A critical point in designing a data fusion application is deciding where the fusion should actually take place.

1.3.1 Data Level The data sources for a fusion process are not specified to originate from identical sensors. McKee distinguishes direct fusion, indirect fusion and fusion of the outputs of the former two [48]. Direct fusion means the fusion of sensor data from a set of heterogeneous or homogeneous sensors, and history values of sensor data, while indirect fusion uses information sources like a priori knowledge about the environment and human input. Therefore, sensor fusion describes direct fusion systems, while information fusion also includes indirect fusion processes. The sensor fusion definition of Section 1.1.1 does not require that inputs are produced by multiple sensors, it only says that sensor data or data derived from sensor data have to be combined. For example, the definition also comprises sensor fusion systems with a single sensor that take multiple measurements subsequently at different instants which are then combined. Another frequently used term is multisensor integration. Multisensor integration means the synergistic use of sensor data for the accomplishment of a task by a system. Sensor fusion is different to multisensor integration in the sense that it includes the actual combination of sensory information into one representational format [46]. The difference is outlined in Figure 1.4: while sensor fusion combines data (possibly applying a conversion) before handing it to the application, an application based on multisensor integration directly process the data from the sensors. A classical videosurveillance sensor fusion application is remote sensing where imagery data acquired with heterogeneous sensors (e.g. colour and infrared cameras) is fused in order to enhance sensing capabilities before further processing. Sensors don’t have to produce commensurate data either. As a matter of fact, sensors could be very diverse and producing different quantities. Sensor fusion reduces the incoming data into a format the application can further process.

Fig. 1.4 Conceptual comparison between (left) Sensor Fusion and (right) Multisensor Integration

10

L. Snidaro, I. Visentini, and G.L. Foresti

1.3.2 Feature Level Feature fusion (or selection) is a process used in machine learning and pattern recognition to obtain a subset of features capable to discriminate different input classes. Object detection and tracking in video sequences are known to benefit from the employment of multiple (heterogeneous) features (e.g. color, orientation histograms, etc.) as shown in [13, 23, 75]. The combination of different features within a higher level framework, as for instance the boosting meta-algorithm, can be a winning strategy in terms of robustness and accuracy [81]. In a surveillance context, a feature selection mechanism such as the one presented in [13] can be used to estimate the performance of the sensor in detecting a given target. The approach selects the most discriminative colour features to separate the target from the background by applying a two-class variance ratio to log likelihood distributions computed from samples of object and background pixels. The algorithm generates a likelihood map for each feature, ranks the map according to its variance ratio value, and then proceeds with a mean-shift tracking system that adaptively selects the top-ranked discriminative features for tracking. The idea was further developed in [78] where additional heterogeneous features were considered and fused via likelihood maps. Another example of successful feature selection for target tracking can be found in [54], where the fusion focus is on a high-level integration of responses: the confidence maps coming from a feature extraction step are merged in order to find the target position. Heterogeneous weighted data is combined without involving classifiers and without maintaining a model of the object, but performing a frame-to-frame tracking with salient feature extracted at each epoch. In [23] the heterogeneity of the features improves recognition performance, even when the target is occluded.

1.3.3 Classifier Level Classification is a crucial step in surveillance systems [79, 35]. Usually, a classifier is considered as a stand-alone entity, that formulates a decision over a problem providing a real-valued or a binary output, or simply a label. A classifier aims to translate the feature outputs into meaningful information, producing higher-level decisions. One classifier can work on raw data or be bound to one or more features; these can be independent or conditionally independent because referring to the same object. Some synonyms of classifier are hypothesis, learner, expert [41]. A classifier can be trained, that means that exploits past knowledge to formulate a decision (i.e., Neural Networks, Bayesian classifiers), or not (i.e., Nearest Neighbours, clustering algorithms); in the latter case, usually a classifier considers a neighbourhood of samples to formulate its response. A wide survey on base classifiers is presented in [7, 41]. In the scheme presented in Figure 1.3, the classifier level refers to the usage of different heterogeneous classifiers to create a fusion scheme that aims to improve the performances with respect to the single-type classifier algorithm. Heterogeneity

1

Data Fusion in Modern Surveillance

11

has, in fact, been linked to diversity [30] and its contribution to performance improvement has been empirically demonstrated in several cases [70, 30, 5].

1.3.4 Combiner Level The problem of fusing different classifiers to provide a more robust and accurate detection has been studied since the early 90s, when the first approaches to the problem were detailed [31, 60, 55, 32]. Classifier fusion is proved to benefit from the diverse decision capabilities of the combined experts, thus improving classification performance with particular respect to accuracy and efficiency [39]. Two advisable conditions are the accuracy (low error rates) and the diversity (committing different errors) of the base learners. A strong motivation to classifier fusion is the idea that the information coming from several classifiers is combined to obtain the final classification response, considering each one of their individual opinion, and mutually compensating their errors [42]. The literature on classifier fusion techniques, that is the design of different combiners, is really vast: the MCS workshops series has a noticeable importance in this sense, but classifier ensembles are often used in a broad number of application fields, from medical imaging [57] to network security [22], from surveillance [72] to handwritten digit recognition [43, 80] including a large range of real-world domains [53]. Trying to formalize the benefits of the aggregation of multiple classifiers, Dietterich [18] and Kuncheva [41] gave a few motivations why multiple classifiers systems may be better than a single classifier. The first one is statistical: the ensemble may be not better1 than the best individual classifier, but the risk of picking an “inadequate single classifier” is reduced. The second reason why we should prefer an ensemble rather than a single classifier is computational: a set of decision makers can provide a solution working each one on a subsection of the problem in less time than a single base learner. The last motivation refers to the possibility of the classifiers’ space not containing the optimum. However, the ability of a mosaic of classifiers to approximate a decision boundary has to be considered; in this respect, it may be difficult or even impossible to adapt the parameters of an individual classifier to fit the problem, but a set of tuned classifier can approach the solution with a good approximation. Under these considerations, the focus is on the fusion criterion that, even suboptimal, achieves better performances than a single trained classifier. Typically, classifiers can be combined by several fusion rules, from simple fixed ones (i.e., sum, mean, vote by majority, etc.) to more complex fusion schemes (i.e., belief functions, Dempster-Shafer evidence theories, etc.). Some meta-learners, as the Boosting technique, can be employed as well, preceded by an off-line training phase. An exhaustive survey of combination methods can be found in [41]. 1

The group’s average performance is not guaranteed to improve on the single best classifier [41].

12

L. Snidaro, I. Visentini, and G.L. Foresti

1.3.5 An Example: Target Tracking via Classification Modern surveillance systems operating in complex environments, like an urban setting, have to cope with many objects moving in the scene at the same time and with events of always increasing difficulty. In order to single out and understand the behaviour of every actor in the scene, the system should be able to track each target as it moves and performs its activities. Position, velocity, and trajectory followed constitute basic pieces of information from which simple events can be promptly inferred. For this reason, a robust and accurate tracking process is of paramount importance in surveillance systems. Tracking is not an easy task since sensor noise and occlusions are typical issues that have to be dealt with to keep each target associated with its ID. In a system with multiple cameras this task (multi-sensor multi-target tracking) is even more daunting, since measurements from different sensors but generated because of the observation of the same target have to be correctly associated. Significant progress in the object tracking problem has been made during the last few years (see [77] for a recent survey). However, no definitive solution has been proposed, considering challenging situations as illumination variation conditions, appearance changes events or unconstrained video sources. Tracking of an object can be performed at different levels. At sensor level, on the image plane of each sensor, the system executes an association algorithm to match the current detections with those extracted in the previous frame. In this case, available techniques range from template matching, to feature matching [11], to more recent and robust algorithms [14]. Another step forward considers to mix different features to enhance the robustness of the tracker; as seen in Section 1.3.2, heterogeneous features brought a big improvement in target localization and tracking. Recently the tracking via classification concept has received a new boost [3, 9, 25, 56, 69, 52]; a single classifier or a classifier ensemble track an object separating the target from the background. In general, using each one of these approaches to tackle the tracking problem we can potentially consider several fusion levels, as shown in Figure 1.5. We can combine more than one source of information, as, for instance, audio and video, as

Fig. 1.5 Example of application

1

Data Fusion in Modern Surveillance

13

they has been repeatedly considered as a potential improvement to the limitation imposed from single-type sensors [10, 6, 62]. Different techniques can be applied to the data to extract heterogeneous relevant features; the joint usage of different information can improve significantly the tracking result quality [77]. Eventually, tracking via classification is considered more robust to occlusions and illumination changes [24] and greater system robustness and performance is achievable with an ensemble of classifiers through information fusion techniques [9, 56].

1.4 Sensor Management: A New Paradigm for Automatic Video Surveillance Video surveillance systems have always been based on multiple sensors since their first generation (CCTV systems) [61]. Video streams from analog cameras were multiplexed on video terminals in control rooms to help human operators monitor entire buildings or wide open areas. The last generation makes use of digital equipment to capture and transmit images that can be viewed virtually everywhere by using Internet. Initially, multi-sensor systems were employed to extend surveillance coverage over wide areas. The recent advances in sensor and communication technology, in addition to lower costs, allow to use multiple sensors for the monitoring of the same area [64, 2]. This has opened new possibilities in the field of surveillance as multiple and possibly heterogeneous sensors observing the same scene provide redundant data that can be exploited to improve detections accuracy and robustness, enlarging monitoring coverage and reducing uncertainty [64]. While the advantages of using multiple sources of information are well know to the data fusion community [44], the full potential of multi-sensor surveillance is yet to be discovered. In particular, the enrichment of available sensor assets has allowed to take advantage of data fusion techniques for solving specific tasks like for example target localization and tracking [65], or person identification [62]. This can be formalized as the application of JDL Level 1 and 2 fusion techniques [44] to surveillance strictly following a processing stream that exploits multi-sensor data to achieve better system perception performance and in the end improved situational awareness . A brief exemplification of the techniques that can be employed in Level 1 and 2 have been presented in Section 1.2. While many technical problems remain to be solved for integrating heterogeneous suites of sensors for wide area surveillance, a principled top-down approach is probably still left unexplored. Given the acknowledged increased complexity of architectures that can be developed nowadays, a full exploitation of this potential is probably beyond the possibilities of a human operator. Think for example to all the possible combinations of configurations that are made available by modern sensors: Pan-Tilt-Zoom (PTZ) cameras can be controlled to cover different areas, day/night sensors offer different sensing modalities, radars can operate at different frequencies, etc. The larger the system the more likely it will be called to address many different surveillance needs. A topdown approach would be needed in order to develop surveillance systems that are

14

L. Snidaro, I. Visentini, and G.L. Foresti

able to automatically manage large arrays of sensors in order to enforce surveillance directives provided by the operator that in turn translate the security policies of the owning organization. Therefore, a new paradigm is needed to guide the design of architectures and algorithms in order to build the next generation of surveillance systems that are able to organize themselves to collect data relevant to the objectives specified by the operator. This new paradigm would probably need to take inspiration by the principles behind the Sensor Management policies foreseen by JDL Level 4 [37]. The JDL model Level 4 is also called Process Refinement step as it implies adaptive data acquisition and processing to support mission objectives. Conceptually, this refinement step should be able to manage the system in its entirety: from controlling hardware resources (e.g. sensors, processors, storage, etc.) to adjusting the processing flow in order to optimize the behaviour of the system to best achieve mission goals. It is therefore apparent that the Process Refinement step encompasses a broad spectrum of techniques and algorithms that operate at very different logical levels. In this regard, an implemented full-fledged Process Refinement would provide the system a form of awareness of its own capabilities and how they relate and interact with the observed environment. The Process Refinement part dedicated to sensors and data sources is often called Sensor Management and it can be defined as “a process that seeks to manage, or coordinate, the use of a set of sensors in a dynamic, uncertain environment, to improve the performance of the system” [76]. In other words, a Sensor Management process should be able to, given the current state of affairs of the observed environment, translate mission plans or human directives into sensing actions directed to acquire needed additional or missing information in order to improve situational awareness and fulfil the objectives. A fivelayered procedure has been proposed in [76] and is reproduced in Figure 1.6. The chart schematizes a general sensor management process that can be used to guide the design of a real sensor management module. In the following, the different levels will be contextualized in the case of a surveillance system.

Mission Planning This level takes as input the current situation and the requests from the human operator and performs a first breakdown of the objectives by trying to match them with the available services and functionalities of the system. In a surveillance system the requests from the operator can be events of interest to be detected (e.g. a vehicle being stationary outside a parking slot) and alarm conditions (e.g. a person trespassing a forbidden area). Each of the events should be given a priority by the operator. The Mission Planning module is in charge of selecting the functions to be used in order to detect the required events (e.g. target tracking, classification, plate reading, face recognition, trajectory analysis, etc.). Actually this module should work in a way similar to a compiler, starting from the description of the events of interest expressed in a high level language, parsing the description and determining

1

Data Fusion in Modern Surveillance

15

Fig. 1.6 Five-layered sensor managing process [76].

the relevant services to be employed. The module will also identify the areas to be monitored, the targets to look for, the frequency of measurements and the accuracy level.

Resource Deployment This level identifies the sensors to be used among the available ones. If mobile and/or active sensors are available their repositioning may be needed [49]. In particular, this level would take into consideration aspects such as coverage and sensing modality. For example, depending on the time of the day a certain event is to be detected a sensor may be preferred to another.

Resource Planning This level is in charge of tasking the individual sensors (e.g. movement planning for active sensors [17, 49]) and coordinating them (e.g. sensor hand-overs) in order to carry out a certain task (e.g. tracking). The level also deals with sensor selection

16

L. Snidaro, I. Visentini, and G.L. Foresti

techniques that can choose for every instant and every target the optimal subset of sensors for tracking or classifying it. Several approaches to sensor selection have been proposed in the literature such as, for example, information gain based [40] and detection quality based [65].

Sensor Scheduling Depending on the planning and requests coming from Resource Planning, this level is in charge of determining a detailed schedule of commands for each sensor. This is particularly appropriate for active (i.e. PTZ cameras), mobile (e.g. robots) and multimode (day/night cameras, multi-frequency radar) sensors. The problem of sensor scheduling has been addressed in [47], and a recent contribution on the scheduling of visual sensors can be found in [58].

Sensor Control This is the lowest level and possibly also the simplest. The purpose of this level is to optimize sensor parameters given the current commands imposed by Level 1 and 2. For video sensors this may involve regulating iris and focus to optimize image quality. Although this is performed automatically by sensor hardware in most of the cases, it could be beneficial to manage sensor parameters directly according to some figure of merit which is dependent on the content of the image. For example, contrast and focus may be adjusted specifically for a given target. An early treatment of the subject may be found in [68], while a recent survey may be found in [1].

1.5 Conclusions Borrowed from biological systems where multiple sensors with an overlapping scope are able to compensate the lack of information from other sources, the data fusion field has gained momentum since its first appearance in the early 70s. Its rapid spread from biometrics to ambient security field, from robotics to everyday applications matches the rise of multi-sensor and heterogeneous-based approaches. In this regard, modern surveillance systems are moving toward techniques that, from the past single-type sensors, rely on mixed-type sensors and exploit multiple cues to solve real-time tasks. Data fusion is a necessary tool to combine heterogeneous information, to provide flexibility to manage unpredictable events and to enhance situational awareness in modern surveillance systems. In this chapter, we discussed the impact of the fusion of information coming from multiple sources, presenting a survey of existing methods and proposing some examples to outline how the combination of heterogeneous data can lead to better situation awareness in a surveillance scenario. We have also presented a possible new paradigm regarding sensor management that could be taken into account in the design of next generation surveillance systems.

1

Data Fusion in Modern Surveillance

17

References [1] Abidi, B.R., Aragam, N.R., Yao, Y., Abidi, M.A.: Survey and analysis of multimodal sensor planning and integration for wide area surveillance. ACM Computing Surveys 41(1), 1–36 (2008), DOI http://doi.acm.org/10.1145/1456650.1456657 [2] Aghajan, H., Cavallaro, A. (eds.): Multi-Camera Networks. Elsevier, Amsterdam (2009) [3] Avidan, S.: Ensemble tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(2), 261–271 (2007), http://doi.ieeecomputersociety.org/10.1109/TPAMI.2007.35 [4] Baker, K., Harris, P., O’Brien, J.: Data fusion: An appraisal and experimental evaluation. Journal of the Market Research Society 31(2), 152–212 (1989) [5] Bian, S., Wang, W.: On diversity and accuracy of homogeneous and heterogeneous ensembles. International Journal of Hybrid Intelligent Systems 4(2), 103–128 (2007) [6] Bigun, J., Chollet, G.: Special issue on audio-based and video-based person authentication. Pattern Recognition Letters 18(9), 823–825 (1997) [7] Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics), 1st edn. Springer, Heidelberg (2007) [8] Boss, E., Roy, J., Grenier, D.: Data fusion concepts applied to a suite of dissimilar sensors. In: Proceedings of the Canadian Conference on Electrical and Computer Engineering, vol. 2, pp. 692–695 (1996) [9] Chateau, T., Gay-Belille, V., Chausse, F., Laprest, J.T.: Real-time tracking with classifiers. In: European Conference on Computer Vision (2006) [10] Choudhury, T., Clarkson, B., Jebara, T., Pentl, A.: Multimodal person recognition using unconstrained audio and video. In: International Conference on Audio- and VideoBased Person Authentication, pp. 176–181 (1998) [11] Collins, R.T., Lipton, A.J., Fujiyoshi, H., Kanade, T.: Algorithms for cooperative multisensor surveillance. Proceedings of the IEEE 89(10), 1456–1477 (2001) [12] Collins, R.T., Lipton, A.J., Kanade, T.: Introduction to the special section on video surveillance. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 745–746 (2000) [13] Collins, R.T., Liu, Y., Leordeanu, M.: Online selection of discriminative tracking features. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(10), 1631– 1643 (2005) [14] Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE Transactions on Pattern Analysis Machine Intelligence 25(5), 564–575 (2003) [15] Dasarathy, B.V.: Sensor fusion potential exploitation-innovative architectures and illustrative applications. Proceedings of the IEEE 85, 24–38 (1997) [16] Dasarathy, B.V.: Information fusion - what, where, why, when, and how? Information Fusion 2(2), 75–76 (2001) [17] Denzler, J., Brown, C.: Information theoretic sensor data selection for active object recognition and state estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(2), 145–157 (2002) [18] Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000) [19] Elmenreich, W., Pitzek, S.: Using sensor fusion in a time-triggered network. In: Proceedings of the 27th Annual Conference of the IEEE Industrial Electronics Society, Denver, CO, USA, vol. 1, pp. 369–374 (2001)

18

L. Snidaro, I. Visentini, and G.L. Foresti

[20] Foresti, G.L., Regazzoni, C.S., Varshney, P.K.: Multisensor Surveillance Systems: The Fusion Perspective. Kluwer Academic Publisher, Dordrecht (2003) [21] Gavrila, D.M., Munder, S.: Multi-cue pedestrian detection and tracking from a moving vehicle. Int. J. Comput. Vision 73(1), 41–59 (2007), DOI http://dx.doi.org/10.1007/s11263-006-9038-7 [22] Giacinto, G., Perdisci, R., Rio, M.D., Roli, F.: Intrusion detection in computer networks by a modular ensemble of one-class classifiers. Information Fusion 9(1), 69–82 (2008); Special Issue on Applications of Ensemble Methods, doi:10.1016/j.inffus.2006.10.002 [23] Gouet-Brunet, V., Lameyre, B.: Object recognition and segmentation in videos by connecting heterogeneous visual features. Computer Vision and Image Understanding 111(1), 86–109 (2008) [24] Grabner, H., Grabner, M., Bischof, H.: Real-time tracking via on-line boosting. In: Proceedings of the British Machine Vision Conference (BMVC), vol. 1, p. 4756 (2006) [25] Grabner, H., Sochman, J., Bischof, H., Matas, J.: Training sequential on-line boosting classifier for visual tracking. In: International Conference on Pattern Recognition (2008) [26] Grossmann, P.: Multisensor data fusion. The GEC Journal of Technology 15, 27–37 (1998) [27] Hall, D.L., Linn, R.J.: Survey of commercial software for multisensor data fusion. In: Aggarwal, J.K., Nandhakumar, N. (eds.) Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, vol. 1956, pp. 98–109 (1993) [28] Hall, D.L., Llinas, J.: An introduction to multisensor data fusion. Proceedings of the IEEE 85(1), 6–23 (1997) [29] Hall, D.L., McMullen, S.A.: Mathematical Techniques in Multisensor Data Fusion. Artech House, Boston (2004) [30] Hsu, K.W., Srivastava, J.: Diversity in combinations of heterogeneous classifiers. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, T.-B. (eds.) PAKDD 2009. LNCS, vol. 5476, pp. 923–932. Springer, Heidelberg (2009) [31] Hu, W., Hu, W., Maybank, S.: Adaboost-based algorithm for network intrusion detection. IEEE Transactions on Systems, Man, and Cybernetics, Part B 38(2), 577–583 (2008) [32] Islam, M., Yao, X., Nirjon, S., Islam, M., Murase, K.: Bagging and boosting negatively correlated neural networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B 38(3), 771–784 (2008), doi:10.1109/TSMCB.2008.922055 [33] Jain, A., Hong, L., Kulkarni, Y.: A multimodal biometric system using fingerprints, face and speech. In: 2nd International Conference on Audio- and Video- based Biometric Person Authentication, pp. 182–187 (1999) [34] Javed, O., Rasheed, Z., Shafique, K., Shah, M.: Tracking across multiple cameras with disjoint views. In: Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 952–957 (2003), doi:10.1109/ICCV.2003.1238451 [35] Javed, O., Shah, M.: Tracking and object classification for automated surveillance. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 343–357. Springer, Heidelberg (2002) [36] Jephcott, J., Bock, T.: The application and validation of data fusion. Journal of the Market Research Society 40(3), 185–205 (1998) [37] Llinas, J., Bowman, C.L., Rogova, G.L., Steinberg, A.N., Waltz, E.L., White, F.E.: Revisiting the JDL data fusion model II. In: Svensson, P., Schubert, J. (eds.) Proceedings of the Seventh International Conference on Information Fusion, vol. II, pp. 1218–1230. International Society of Information Fusion, Stockholm (2004), http://www.fusion2004.foi.se/papers/IF04-1218.pdf

1

Data Fusion in Modern Surveillance

19

[38] Kang, J., Cohen, I., Medioni, G.: Multi-views tracking within and across uncalibrated camera streams. In: IWVS 2003: First ACM SIGMM International Workshop on Video Surveillance, pp. 21–33. ACM, New York (2003), DOI http://doi.acm.org/10.1145/982452.982456 [39] Kittler, J., Hatef, M., Duin, R.P., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998) [40] Kreucher, C., Kastella, K., Hero III, A.O.: Sensor management using an active sensing approach. Signal Processing 85(3), 607–624 (2005), http://www.sciencedirect.com/science/article/ B6V18-4F017HY-1/2/a019fa31bf4135dfdcc38dd5dc6fc6c8, doi:10.1016/j.sigpro.2004.11.004 [41] Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley Interscience, Hoboken (2004) [42] Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles. Machine Learning 51, 181–207 (2003) [43] Lee, S.W., Kim, S.Y.: Integrated segmentation and recognition of handwritten numerals with cascade neural network. IEEE Transactions on Systems, Man, and Cybernetics, Part C 29(2), 285–290 (1999) [44] Liggins, M.E., Hall, D.L., Llinas, J.: Multisensor data fusion: theory and practice, 2nd edn. The Electrical Engineering & Applied Signal Processing Series. CRC Press, Boca Raton (2008) [45] Liu, H., Yu, Z., Zha, H., Zou, Y., Zhang, L.: Robust human tracking based on multi-cue integration and mean-shift. Pattern Recognition Letters 30(9), 827–837 (2009) [46] Luo, R.C., Kay, M.: Multisensor integration and fusion in intelligent systems. IEEE Transactions on Systems, Man, and Cybernetics 19(5), 901–930 (1989) [47] McIntyre, G., Hintz, K.: Sensor measurement scheduling: an enhanced dynamic, preemptive algorithm. Optical Engineering 37, 517 (1998) [48] McKee, G.T.: What can be fused? In: Multisensor Fusion for Computer Vision. Nato Advanced Studies Institute Series F, vol. 99, pp. 71–84 (1993) [49] Mittal, A., Davis, L.: A general method for sensor planning in multi-sensor systems: Extension to random occlusion. International Journal of Computer Vision 76(1), 31–52 (2008) [50] Monekosso, D., Remagnino, P.: Monitoring behavior with an array of sensors. Computational Intelligence 23(4), 420–438 (2007) [51] Murphy, R.R.: Biological and cognitive foundations of intelligent sensor fusion. IEEE Transactions on Systems, Man and Cybernetics 26(1), 42–51 (1996) [52] Nguyen, H.T., Smeulders, A.W.: Robust tracking using foreground-background texture discrimination. International Journal of Computer Vision 69(3), 277–293 (2006) [53] Oza, N.C., Tumer, K.: Classifier ensembles: Select real-world applications. Information Fusion 9(1), 4–20 (2008); Special Issue on Applications of Ensemble Methods, doi:10.1016/j.inffus.2007.07.002 [54] Parag, T., Porikli, F., Elgammal, A.: Boosting adaptive linear weak classifiers for online learning and tracking. In: International Conference on Computer Vision and Pattern Recognition (2008) [55] Parikh, D., Polikar, R.: An ensemble-based incremental learning approach to data fusion. IEEE Transactions on Systems, Man, and Cybernetics, Part B 37(2), 437–450 (2007), doi:10.1109/TSMCB.2006.883873

20

L. Snidaro, I. Visentini, and G.L. Foresti

[56] Petrovi´c, N., Jovanov, L., Piˇzurica, A., Philips, W.: Object tracking using naive bayesian classifiers. In: Blanc-Talon, J., Bourennane, S., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2008. LNCS, vol. 5259, pp. 775–784. Springer, Heidelberg (2008) [57] Polikar, R., Topalis, A., Parikh, D., Green, D., Frymiare, J., Kounios, J., Clark, C.M.: An ensemble based data fusion approach for early diagnosis of alzheimer’s disease. Information Fusion 9(1), 83–95 (2008); Special Issue on Applications of Ensemble Methods, doi:10.1016/j.inffus.2006.09.003 [58] Qureshi, F., Terzopoulos, D.: Surveillance camera scheduling: A virtual vision approach. Multimedia Systems 12(3), 269–283 (2006) [59] Rao, N.S.V.: A fusion method that performs better than best sensor. In: Proceedings of the First International Conference on Multisource-Multisensor Information Fusion, pp. 19–26 (1998) [60] Ratsch, G., Mika, S., Scholkopf, B., Muller, K.: Constructing boosting algorithms from svms: An application to one-class classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(9), 1184–1199 (2002) [61] Regazzoni, C.S., Visvanathan, R., Foresti, G.L.: Scanning the issue / technology - Special Issue on Video Communications, processing and understanding for third generation surveillance systems. Proceedings of the IEEE 89(10), 1355–1367 (2001) [62] Ross, A., Jain, A.: Multimodal biometrics: An overview. In: Proc. XII European Signal Processing Conf., pp. 1221–1224 (2004) [63] Rothman, P.L., Denton, R.V.: Fusion or confusion: Knowledge or nonsense? In: SPIE Data Structures and Target Classification, vol. 1470, pp. 2–12 (1991) [64] Snidaro, L., Niu, R., Foresti, G., Varshney, P.: Quality-Based Fusion of Multiple Video Sensors for Video Surveillance. IEEE Transactions on Systems, Man, and Cybernetics 37(4), 1044–1051 (2007) [65] Snidaro, L., Visentini, I., Foresti, G.: Quality Based Multi-Sensor Fusion for Object Detection in Video-Surveillance. In: Intelligent Video Surveillance: Systems and Technology, pp. 363–388. CRC Press, Boca Raton (2009) [66] Stauffer, C., Grimson, W.E.L.: Learning patterns of activity using real-time tracking. IEEE Pattern Analysis and Machine Intelligence 22(8), 747–757 (2000) [67] Steinberg, A.N., Bowman, C.: Revisions to the JDL data fusion process model. In: Proceedings of the 1999 National Symposium on Sensor Data Fusion (1999) [68] Tarabanis, K., Allen, P., Tsai, R.: A survey of sensor planning in computer vision. IEEE Transactions on Robotics and Automation 11(1), 86–104 (1995) [69] Tomasi, C., Petrov, S., Sastry, A.: 3d tracking = classification + interpolation. In: ICCV 2003: Proceedings of the Ninth IEEE International Conference on Computer Vision, p. 1441 (2003) [70] Tsoumakas, G., Katakis, I., Vlahavas, I.: Effective voting of heterogeneous classifiers. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 465–476. Springer, Heidelberg (2004) [71] Valin, J.M., Michaud, F., Rouat, J.: Robust localization and tracking of simultaneous moving sound sources using beamforming and particle filtering. Robot. Auton. Syst. 55(3), 216–228 (2007), http://dx.doi.org/10.1016/j.robot.2006.08.004 [72] Visentini, I., Snidaro, L., Foresti, G.: On-line boosted cascade for object detection. In: Proceedings of the 19th International Conference on Pattern Recognition (ICPR), Tampa, Florida, USA (2008) [73] Wald, L.: A european proposal for terms of reference in data fusion. International Archives of Photogrammetry and Remote Sensing 7, 651–654 (1998)

1

Data Fusion in Modern Surveillance

21

[74] Waltz, E., Llinas, J.: Multisensor Data Fusion. Artech House, Norwood (1990) [75] Wu, B., Nevatia, R.: Optimizing discrimination-efficiency tradeoff in integrating heterogeneous local features for object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008) [76] Xiong, N., Svensson, P.: Multi-sensor management for information fusion: issues and approaches. Information fusion 3(2), 163–186 (2002) [77] Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM Comput. Surv. 38(4), 13 (2006), DOI http://doi.acm.org/10.1145/1177352.1177355 [78] Yin, Z., Porikli, F., Collins, R.: Likelihood map fusion for visual object tracking. In: IEEE Workshop on Applications of Computer Vision, pp. 1–7 (2008) [79] Zhang, L., Li, S.Z., Yuan, X., Xiang, S.: Real-time object classification in video surveillance based on appearance learning. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) [80] Zhang, P., Bui, T.D., Suen, C.Y.: A novel cascade ensemble classifier system with a high recognition performance on handwritten digits. Pattern Recognition 40(12), 3415–3429 (2007) [81] Zhang, W., Yu, B., Zelinsky, G.J., Samaras, D.: Object class recognition using multiple layer boosting with heterogeneous features. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 323–330. IEEE Computer Society, Washington (2005)

Chapter 2

Distributed Intelligent Surveillance Systems Modeling for Performance Evaluation Stefano Maludrottu, Alessio Dore, and Carlo S. Regazzoni

Abstract. In the last few years, the decrease of hardware costs and the contemporary increase of processing capabilities make possible the development of more and more complex surveillance systems with hundreds of sensors deployed in large areas. New functionalities are now available such as scene understanding, contextdependent processing and real-time user-driven functionality selection. The efficient use of these tools in architecturally complex systems for advanced scene analysis needs the development of specific data fusion algorithms able to merge multi-source information provided by a large number of homogeneous or heterogeneous sensors. In this context, the possibility of distributing the intelligence is one of the more innovative and interesting research fields for such systems. Therefore, several studies are focused on how the logical tasks can be partitioned between smart sensors, intermediate processing nodes and control centers. Typical tasks of surveillance systems (context analysis, object detection, tracking...) are organized into hierarchical chains, where lower levels in the architectures provide input data for higher level tasks. Each element of the architecture is capable of autonomous data processing. The complexity of such systems remarks the importance of good design choices for both logical and physical architecture. The main objective of this book chapter is to present possible solutions to evaluate the overall performance and technical feasibility as well as the interactions of the subparts of distributed multi-sensor surveillance systems. Performance evaluation of a multi-level hierarchical architecture does not pertain only to the accuracy of involved algorithms but several other aspects must be considered as the data communication between smart sensors and higher level nodes, the computational complexity and the memory used. Then, in order to define Stefano Maludrottu · Carlo S. Regazzoni Department of Biophysical and Electronic Engineering (DIBE) - University of Genoa, Genoa, Italy Alessio Dore Institute of Biomedical Engineering, Imperial College London, London, SW7 2AZ, UK

P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 23–45. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

24

S. Maludrottu, A. Dore, and C.S. Regazzoni

a procedure to assess the performances of this kind of systems a general model of smart sensors, intermediate processing nodes and control centers has been studied taking into account the elements involved in a multi-sensor distributed surveillance tasks. The advantage of the proposed architecture analysis method are: 1) to allow quantitative comparison of different surveillance structures through the evaluation of performance metrics and 2) to validate the algorithm choice with respect to the physical structure (communication links, computational load...) available. Finally example structures are compared in order to find the best solution to some benchmark problems.

2.1 Introduction In recent years, due to the increasing robustness and accuracy of computer vision algorithms, to the decrease of sensors and processing hardware prices, and to an increasing demand for reliable security systems in different areas (such as transports, industrial plants, public places or crowded events), surveillance systems have become more and more sophisticated. Hundreds, even thousands of sensors can be deployed in very large areas to perform different security tasks typical of those systems: people counting, tracking, abandoned object detection, movement detection and so on. Accuracy can be achieved through sensor redundancy or by combining different sensor typologies in order to operate under a number of difficult situations such as crowded scenes or poorly illuminated areas. The large amount of multisensorial data produced by heterogeneous and spatially distributed sensors is fused to automatically understand the events in the monitored environment. This data fusion process is instantiated at different levels of the architecture in a distributed and sequential way in order to provide to remote control centers (and to the attention of operators) aggregated data in the form of alarms or collections of salient video sequences about events of interest. A typical description of surveillance systems is a tree-like hierarchical structure composed of smart sensors, intermediate processing nodes and control centers connected by heterogeneous communication links. In this structure, each sensor contributes to the global surveillance task monitoring a small physical area and gathering local data about the environment. Since in advanced surveillance systems data can be locally analyzed using sensors with processing capabilities (and therefore defined as smart sensors) environment. Only salient information can be transmitted to fusion nodes devoted to specific data association and data fusion tasks. Since modern surveillance systems can monitor very large areas, the data fusion process itself can be subdivided into a hierarchical task performed by multiple processing units. More than one layer of intermediate fusion nodes can be dedicated to data fusion. Control centers are devoted to gather the data pertaining the whole monitored area and to present them with appropriate interfaces to the human operators.

2

Performance Evaluation for Intelligent Surveillance

25

This decomposition does not imply a centralized monitoring/control approach, but it reflects a wide range of current systems and it is intended as a descriptive model of the general structure of surveillance systems [10] [5]. Moreover, modern surveillance systems can be defined as “intelligent”, following a definition in [25], if they are capable of integrating the ambient intelligence paradigm [21] into safety or security applications. Many works have been devoted in the last decade to link traditional computer vision tasks to high-level context aware functionalities such as scene understanding, behavior analysis, interaction classification or recognition of possible threats or dangerous situations ([26] [16] [3] [27]). Another notable improvement of modern state-of-the-art surveillance systems (third generation surveillance systems) is the capability of internal logical task decomposition and distributed data processing. The so-called distribution of intelligence in such systems is made possible by the autonomous processing capabilities of the elements of the architecture. It can be described as a dynamic process of mapping the logical architecture (decomposed into a set of sequential sub-tasks) into the physical structure of the surveillance system itself. In such a way processing and communication resources can be optimized within the system. Modern surveillance systems can be used to provide different services to a number of different end-users; all the communication and processing resources of the system can be shared between different users according to specific strategies. Therefore, a modern third generation intelligent surveillance system whose characteristics are compliant to the above definitions can be a large, expensive and structurally complex system. The design of such a system is an articulated task: hardware selection, communication links design, sensors placement are some of the possible constraints that have to be considered. Moreover, existing architectures can be reused for new security applications, therefore is important to assess in advance how new functionalities can perform on existing structures. Different aspects of surveillance architectures should be taken into account to assess their performances for what concerns the algorithms accuracy. Also communication or computational complexity issues that can arise in complex multi-sensor systems should be considered. An efficient performance evaluation should not be an a posteriori analysis of such systems: modifications or refinements in such a large scale can be difficult, time consuming and economically disadvantageous tasks. A better solution is to link performance evaluation to the design phase, in order to produce useful feedbacks regarding specific design choices. Therefore, a correct modeling of intelligent surveillance systems can be defined in order to take into account all the designcritical aspects of surveillance systems.

2.1.1 Surveillance Systems as Data Fusion Architectures Since the main goal of an intelligent surveillance system is to provide to human operators a complete, reliable and context-aware description of the environment based

26

S. Maludrottu, A. Dore, and C.S. Regazzoni

Fig. 2.1 Scheme of the data fusion process of a surveillance system derived from the JDL model.

upon raw data obtained from the deployed sensors. A data fusion process is distributed between different elements of the architecture during which information extraction and refinement is performed at different abstraction levels. The data fusion process can be modeled, according to the JDL model [11], as a hierarchical process composed by four steps (See Fig. 2.1): 0-preprocessing: Raw sensor data are filtered, conditioned and processed before any fusion with data from other sensors; significant information is obtained from raw data using feature extraction algorithms. The preprocessing techniques adopted depend on the specific nature of the sensors. 1-processing: Spatial and temporal alignment of data from different sensors is performed. A common description of objects present in the environment is provided for instance, in terms of position, of velocity, of attributes and of characteristics. Specific recognition or classification algorithms are instantiated in order to label objects perceived in the scene. 2-situation analysis: Automated reasoning or ambient intelligence methods are adopted to provide a complete description of events in the environment. Interactions between objects are compared to predefined models and classified. The meaning of events within a scene is determined. 3-threat analysis: The evolution of the current situation in the environment is evaluated to determine potential threats using predictive models or probabilistic frameworks. Alarms are raised if potentially dangerous situations are recognized. Aggregated data and alarms resulting from the data fusion process are provided to human operators through an appropriate human-computer interface (HCI). HCIs may contain maps of the environment, icons or message boards in order to facilitate an immediate understanding of potential threats or suspicious situations. Aside from the data fusion process, database management can be considered as a key task to effectively handle the large amount of data involved in the data fusion process. Since the database (DB) is a critical component in a surveillance systems, many works have been devoted to DB modeling in complex systems. To avoid risks

2

Performance Evaluation for Intelligent Surveillance

27

of data loss or DB failure, a common approach is a DB (virtual) replication in every node of the architecture [14]. Therefore modern third-generation surveillance systems can be defined and identified by the mapping of specific data analysis and data fusion tasks into an architecture of intelligent hardware devices. This mapping can be an a-priori design decision or can be dynamically modified according to resources available in real-time and specific events in the environment.

2.1.2 Surveillance Tasks Decomposition Distribution of intelligence is a critical feature of surveillance systems in order to concurrently optimize bandwidth usage and processing speed. For instance, data analysis performed at lower levels of the architecture can lead to a more efficient bandwidth and central processing usage, offering in the same time robustness through redundancy, since a distributed processing can better handle components failures. Typical surveillance tasks (e.g. tracking) can be decomposed into a chain of logical modules organized in a hierarchical structure: lower modules provide as output the data needed as input by higher level modules. In this way, starting from raw sensor data, at each level an higher inference can be reached. This decomposition, originally proposed in [13] defines the paradigm of intelligence distribution in terms of module allocation to different physical devices with autonomous processing capabilities. Three kinds of modules have been defined at different abstraction levels: representation modules (as information-processing tasks whose output is higher-symbolic representation of the input data), recognition modules (as algorithms that compare input data with a predefined set of event descriptors) and communication modules (that produce a coded representation of input data suitable for transmission). In [2] a mobile agent system (MAS) is defined as a dynamic task assignment framework to smart cameras in surveillance applications: high-level tasks are allocated to sensor clusters. The system itself decomposes them into subtasks and maps this set of subtasks to single sensors according to the available processing resources of cameras and the current state of the system. Moreover, software agents can dynamically migrate between different sensors according to changes in the observed environment or in the available resources. In [19] is defined an optimal task partitioning framework for distributed systems that is capable of a dynamic adaptation to system or environment changes by taking into consideration Quality of Service (QoS) constraints. In [22] an agents-based architecture is presented, where agents, defined as software modules, are capable of autonomous migration between nodes of the architecture. Within this framework, camera agents are responsible for detection and tracking tasks while object agents provide updated descriptions of temporal events.

28

S. Maludrottu, A. Dore, and C.S. Regazzoni

More in general, the mapping of tasks on the architecture of an intelligent surveillance system can be defined as an association function A: A:T →N

(2.1)

where T is the complete set of processing tasks ti that the system is required to perform and N is the set of intelligent devices (or nodes) ni in the system itself. For every node ni it must hold that ∑ j pi j ≤ p j where pi j is the cost value in terms of processing resources (memory or CPU) of task ti allocated to node n j and p j is the total processing capability of node n j .

2.2 Intelligent Surveillance System Modeling 2.2.1 Introduction In the following of this Section, a model is proposed to provide a general framework for the performance evaluation of complex multi-sensor intelligent surveillance systems. To this aim three main components of the system are identified, i.e., smart sensor, fusion node and control center. In this work we will consider a structure where sensors and control centers can exchange data only with the fusion nodes, albeit it would be possible to add other communication links (e.g. direct communication between sensors) to model specific architectures. This structure has been chosen since this work focuses on the hierarchical architecture where the fusion node overlooks the sensors and, eventually, it modifies their processing tasks or their internal parameters or it passes information gathered from other sensors. This model can be constituted by one or more fusion nodes according to the extension of the monitored area and the number of sensors. The fusion nodes send the processed data to a common remote control center. If the considered environment is very large or structurally complex, or if a huge quantity of sensors have to be considered, more than one level of intermediate nodes can be used to perform the data fusion process. A hierarchical multi-level partitioning of the environment can be adopted in order to manage the data flow between multiple layers of intermediate fusion nodes. Each zone at level i-th (“macro-zone”) is divided into a set of smaller zones (“micro-zones”) at (i + 1)-th level according to a specific partitioning strategy such as the Adaptive Recursive Tessellation [24]. Higher-level fusion nodes take as input metadata produced by lower-level fusion nodes. These data (related to contiguous micro-zones) are fused in order to get a global representation of a macro-zone of the environment. Control centers are directly connected to all the fusion nodes (or to highest-level fusion nodes in a multi-level fusion structure as defined before) of the architecture. In this way, the metadata descriptive of the entire environment are collected in order to provide a complete representation of the environment to the final users. Usually a single control center is present, although in multi-user surveillance systems, more than one control center can be realized.

2

Performance Evaluation for Intelligent Surveillance

29

Without loss of generality, we will consider the overall fields of view of the fusion nodes contiguous non-overlapped. A global metric used to evaluate the global performances of a multi-sensor system will be described in Sect. 2.2.5. The global metric will take into account not only specific data processing errors but also problems or failures of structural elements of the architecture. In Section 2.4 it will be shown how, keeping unaltered the processing algorithms involved, different design choices (in terms of sensor selection and positioning) can significantly affect the overall system performances.

2.2.2 Smart Sensor Model Each smart sensor Si generates a local representation of specific physical quantities (such as light, motion, sound...). Input data are locally processed by the sensor and appropriate metadata are sent to higher levels of the architecture. The processing function Ti (t) represents a data analysis task that produces a certain output data D f i (t + Δ ti ) given an input data D i (t) referring to the scene observed in its field of view VSi (t). The quantity Δ ti (defined as time response of sensor Si ) accounts for the computational time needed by the smart sensor to process input data. The value Δ ti depends on several factors, such as: the specific processing task considered, the computational complexity and the available processing resources. The input noise is modeled with E i (t) and the noise on the processed data sent to the fusion node is E f i (t + Δ ti ). The variables D i (t), D f i (t + Δ ti ), E i (t) and E f i (t + Δ ti ) are vectors of dimension proportional to the complexity of the scene observed by Si at time t (e.g. in a tracking application those vectors have dimension equal to the number of targets Ni (t)). The computational complexity C function of D i (t) is associated to the processing function. The error vector in this application coincides with the accuracy error that has to be evaluated through a suitable performance metric. To further improve modularity, different performance metrics can be adopted (including the ones mentioned in section 2.3.3). Since a third generation intelligent surveillance system is a context-aware and (possibly) multi-user multi-functional system, the specific processing task Ti (t) associated to each sensor Si can be modified according to specific user requests and salient changes in the environment. The database Di represents the memory M of the smart sensor. It has been decided to address it as a database since many surveillance algorithms require to compare actual data with known models or previous data. The algorithm T exchanges information with the database that can also send additional information to the fusion node. The communication unit Ci is responsible for transmitting data to the fusion node. The communication unit has to satisfy the requirements of error free data transmission considering possible bandwidth B limitations of the link toward the fusion node. In order to comply with real-time constraints, a maximum-delay value Δ tmax,s can

30

S. Maludrottu, A. Dore, and C.S. Regazzoni

be defined. Thus, if Δ ti > Δ tmax,s , the output of the processing task can be considered to contain obsolete data and it is discarded. Each smart sensor is considered to be placed in a certain known position of the environment; its field of view (FOV) depends on its angle of view and its orientation. If a smart sensor has a fixed orientation its FOV VSi (t) becomes a constant VSi . Therefore a smart sensor Si can be modeled as composed by four fundamental elements: • • • •

a processing function Ti (t), a database Di , a communication unit Ci , a field of view VSi (t).

The schematic diagram of a smart sensor model can be seen in Fig.2.2.

Fig. 2.2 Representation of the smart sensor model

2.2.3 Fusion Node Model In this stage the data coming from each sensor are integrated to realize an accurate single model of each object perceived in the environment. The structure of the fusion node N is similar to the smart sensor model but some differences are present in the processing function and communication unit. The fusion node processing function TF (t) takes M data D f i (t) (together with the respective error measure E f i (t), i = 1, ..., M) as input, where M is the number of smart sensors connected to the fusion node Fk . Due to failures of specific

2

Performance Evaluation for Intelligent Surveillance

31

sensors or communication links, or due to data loss for real-time constraints or bandwidth limitations, a subset of input data can be received by the fusion node. An “information loss” parameter IF (t) is defined to account for the percentage of the input data not correctly received by the fusion node. Thus, the transfer function D f 1 (t), ..., D f M (t), E f 1 (t), . . . , E f M (t), IF (t)). The cardinality of DF (t) and is: TF (D D f 1 (t) ∪ D f 2 (t) ∪ ...D D f M (t))). EF (t) vectors is NF (t) = TF (card(D The output data D ci (t + Δ tF ) of the processing function TF (t) is sent to the control centers. The time response of fusion node N (Δ tF ) accounts for the computational time needed by the fusion node to process input data. An efficient data fusion algoE f 1 (t), . . . , E f M (t)), an error vector E F (t + Δ tF ) in rithm produces, as output of T (E which each value is lower than the respective error value in E i (t) detected from the i-th sensor. The robustness of data fusion algorithms can be defined as a function E f i , IF ) → E F that records how the error of the output data is affected by the RF : (E errors of the input data and by information loss. Worth of note is that the computational complexity C of the fusion node processing is related to the dimension of the overlapped fields of view and the intricacy of the events analyzed in this region. Similarly to the smart sensor model, the processing function TF (t) depends on the specific task decomposition at time t. The database DF performs the same tasks described for the smart sensor model. The communication unit CF analyzes the transmission load for each communication link connecting the fusion node N to the smart sensors Si . Moreover, it manages the data transmission to control centers through a link of bandwidth BF . It can also send messages to the smart sensor communication unit in order to perform an adaptive optimization procedure. For example, if one link is overused the fusion node can ask to all the smart sensor connected to that link to reduce their transmission rate by a higher compression or a dynamic reallocation of processing tasks can be performed. Taking into account real-time constraints, a maximum delay value for the fusion node Δ tmax,F can be defined. Output data are sent to control centers only if the time response value Δ tF ≤ Δ tmax,F . A fusion node N is modeled by three fundamental elements: • a processing function TF (t), • a database DF , • a communication unit CF .

2.2.4 Control Center Model The control center C receives data produced by fusion nodes. Collected data can be further processed in order to produce a global representation of the environment. Aggregated data are shown to operators by means of an appropriate HCI. At this stage changes of allocation of data processing tasks within the system are issued if the user requests about specific functionalities are received or as a result of a context analysis (e.g. if threats or suspicious situation are detected by the system). The control center processing function TC (t) can be decomposed as TP (t) + THCI where TP (t) is the input data processing task and THCI is the processing related to

32

S. Maludrottu, A. Dore, and C.S. Regazzoni

the user interface (supposed constant over time). If more than one fusion node Nk is connected to the control center C, TP (t) is an high-level data fusion task: TP (t) takes   M data D ci (t) (together with the respective error measure E Fi (t), i = 1, ..., M ) as  input, where M is the number of intermediate fusion nodes connected to the control center Ck . An ’information loss’ parameter IC (t) can be defined for the control center as the percentage of input data not correctly received at time t. Thus, the transfer D c1 (t), ..., D cM (t), E F1 (t), . . . , E FM (t), IC (t)). function is: TF (D The Output data DC (t + Δ tC ) affected by an error E C (t + Δ tC ) is presented to operators through an appropriate interface. The value Δ tC is the time response value of the control center. The output error is affected by the information loss parameter similarly to the fusion node model. Similarly to the fusion node model, a robustness E Fi , IC ) → E F . function can be defined as RC : (E The database DD performs the same tasks described for the smart sensor model. The communication unit CC analyzes the transmission load for each communication link that connects the control center C to the intermediate fusion nodes Ni . It sends messages to fusion nodes communication units to optimize processing and communication resources or to manage/start/terminate specific surveillance tasks. The control center C is modeled considering: • a processing function TC (t), • a database DC , • a communication unit CC .

2.2.5 Performance Evaluation of Multi-sensor Architectures The Performance evaluation of data fusion systems, such as multi-sensors surveillance systems is usually done considering only the information processing and refinement algorithms. Much work has been dedicated in the recent years to evaluate the performances of surveillance algorithms usually by a comparison of the output of those algorithms with a manually obtained ground truth. In this way a quantitative evaluation and an effective comparison of different approaches can be usually assessed. In visual surveillance, for instance, since typical tasks are tracking or detection of moving objects, the corresponding ground truth can be obtained by manually labeling or drawing bounding boxes around objects. However, the evaluation of those systems does not pertain only on algorithms accuracy but several other aspects must be considered as well: the data communication between sensors, fusion nodes and control centers, the computational complexity and the memory used. A complete model must take into account all the significant elements in a multi-sensor system. Therefore, in order to define a more global procedure to assess the performances of intelligent third-generation surveillance systems a general model of smart sensors, fusion nodes and control centers has been studied. A global evaluation function P(F(T, R), DF , I) can be defined in order to assess the overall performances of surveillance architectures. The global result of this function will implicitly take into account both the robustness and the reliability of processing algorithms and the specific structure of the surveillance system itself.

2

Performance Evaluation for Intelligent Surveillance

33

P depends on several parameters. The performance metric F(T, R) is related to the specific surveillance task (tracking, detection, classification,...) that takes into account the processing functions T = {Ti , TF , TC } of robustness R = {Ri , RF , RC } assigned to the nodes of the architecture. I and DF are, respectively, the information loss (accounts for bandwidth/CPU limitation problems, real-time constraints...) and the database failure rate (accounts for storage/retrieval errors) of every element of the structure. In this way, different intelligent surveillance systems can be evaluated and compared on the basis of different design choices. The impact of specific data processing functions assigned to smart sensors Ti , intermediate fusion nodes TF and control centers TC on the overall performances can be assessed. Sensors selection and positioning can be validated for specific surveillance applications. Finally, the cost of data loss due to different causes (bandwidth limitations, real-time requirements,...) can be successfully evaluated.

2.3 A Case Study: Multi-sensor Architectures for Tracking 2.3.1 Introduction In order to test the proposed model for a real-world surveillance system, a distributed tracking architecture has been chosen as a benchmark problem as in [1]. The structure consists of a set of sensors Si that transmit over a common transmission channel C, the local results of tracking/localization tasks to a single fusion node N. In the following case study, video sensors have been considered since in real-world surveillance systems, video cameras are by far the most common sensors. However, this does not have to be considered as a limitation of the proposed model, since heterogeneous sensors (audio, radio, GPS,...) can be used as well in tracking applications to provide reliable information on moving objects in the environment. In the fusion node data association as well as data fusion are performed. The local object identifiers (IDs) are replaced by common identifiers and point-to-track association is done. A correct modeling solution of this problem can be obtained through a hierarchical decomposition of the architecture of the tracking system into its physical and logical subparts according to the model defined in section 2.2.1. In the literature many similar approaches have been proposed for distributed fusion architecture modeling: in [15], a hierarchical architecture for video surveillance and tracking is proposed; the camera network sends the data gathered to the upper level nodes in which tracking data are fused in a centralized manner. In some related works specific metrics are introduced in order to measure the reliability of each subpart involved in the data fusion. Typical problems arising in video data flow are due to the superimposition of targets and elements of the scene occupying different 3D in the image plane. This issue, called occlusion, causes a momentary lack of observations with consequent difficulties in the position estimation procedure. Moreover, background elements

34

S. Maludrottu, A. Dore, and C.S. Regazzoni

can produce misleading observations (or clutter) that are to be excluded from the measurements-target association procedures. A possible and widely exploited approach to overcome these issues and to enhance the tracking robustness consists in using more sensors monitoring with complete or partial fields of view overlapping the same area. In this way it is possible to obtain multiple observations of the same entity whose joint processing likely improves tracking performance. The fusion module can be considered as a function that associates in a proper way the data produced by the sensors with overlapped fields of views and estimates an unique scene representation. An example of the above mentioned architecture is shown in Figure 2.3. Sensors are considered as smart sensors , i.e. with an embedded processing unit and with the capability of performing autonomous localization and tracking tasks. Fusion nodes gather data from two or more sensors and process them jointly to generate an unique scene representation.

Fig. 2.3 Example of a data fusion system for target tracking

2.3.2 Multisensor Tracking System Simulator On the basis of the architectural model described in sect. 2.2.1 a tracking simulator has been realized to demonstrate the possibility to evaluate the system performances. A context generator has been designed in order to produce a set of realistic target trajectories simulating moving objects with stochastic dynamics. The simulated smart sensor Si acquires data from the context generator only for what concerns his field of view VSi (t) and processes them using a Kalman Filter. Multiple sensors

2

Performance Evaluation for Intelligent Surveillance

35

with overlapped fields of view send their processed data to one fusion node Fk that processes the data using a modified Kalman Filter in addition to a Nearest Neighbor data association technique [28]. 2.3.2.1

Context Generator

The context generator produces trajectories with linear autoregressive first order model with Gaussian noise. Trajectories lay in a two-dimensional space representing the map of a monitored environment. A Poissonian distribution is used to describe the probability of new target born and death. Occlusions are also simulated when two targets are aligned with respect to the camera line of sight and they are sufficiently close each other. Clutter noise is also introduced at each scan as a Poisson process. The context generator (see Fig. 2.4) allows us to have a realistic but controllable environment by which simulating different situations automatically generating the correspondent ground truth. Yet, the proposed context generator has to be considered as a simplified model of the environment, since some of the typical issues of visual-based surveillance applications (such as sudden or continuous lighting variations, adverse weather conditions, camera vibrations...) are not taken into consideration. In order to assess how these problems affect the overall performances of a given multi-sensor architecture, either a more detailed context generator can be implemented (as in [23]) or the simulator can be modified to use real data as input (as in [4]). However, in this work a simple context generator module has been chosen since this case study intends to show the possibility of using the proposed model for performance evaluation. A more realistic simulator can be implemented according to the specific application of the proposed general model. 2.3.2.2

Smart Sensor

When a target trajectory enters in the field of view VSi (t) of the i-th sensor a singleview tracking task is initialized to detect the position and to follow the movements of targets. For each target, in order to estimate the position of tracked objects a Kalman filter is used, whose equations are: x(k + 1) = Aw x(k) + ww (k) z (k) = H w x (k) + v w (k)

(2.2) (2.3)

w l and v l are, respectively, the transition model and the measurement model noise and they are independent, white Gaussian and zero mean. A l and H l respectively are the state transition matrix and the measurement matrix. In this work the computational complexity is a linear function of the number of targets present in the FOV. When this function exceeds a threshold that is defined to represent the unit processing capability, a random lack of data to be sent to the fusion node is simulated.

36

S. Maludrottu, A. Dore, and C.S. Regazzoni

Fig. 2.4 Context generator for multi-camera tracking: white dots are the moving targets and green dots are the sensors placed in the environment

In this example the communication unit of the sensor is responsible of sending data regarding the occupation of the link to the fusion node. A function has been implemented to simulate the optimization of data transmission. In this simulator the database is not modeled since the performed tasks do not need a data storage. 2.3.2.3

Fusion Node

The communication unit handles the data stream coming from the smart sensors. A maximum bandwidth value Bmax has been defined according to the physical constraints of the architecture. If ∑i Bi (k) > Bmax (where Bi is the bandwidth occupation of the sensor i at the time k ) a sensor subset S∗ (k) of the set S of all the sensors connected to the fusion node is chosen in order to optimize the bandwidth usage and such that ∑i B∗i (k) <= Bmax . Various strategies can be implemented to select the sensor subset S∗ (k); in this example a priority value pi is assigned to each sensor Si and we define the subset of selected sensors equal to an empty set S∗ (k) = 0; / then we will add the sensors one by one, choosing at each step the sensor between the remaining ones with the highest priority value. After the sensor selection a feasibility check is performed; if the bandwidth constraint is satisfied the sensor is added to the selected sensor subset (S∗ (k) = S∗ (k) + S j ).

2

Performance Evaluation for Intelligent Surveillance

37

In the processing function initially the association of measurements with targets is performed. Subsequently that these targets are updated and a target initialization procedure is applied for unmatched measurements. Each target xt is represented by using xt (k), state covariance Pt (k) and a category estimate et (k) for each frame k. The state vector is:  T x = x y x˙ y˙ (2.4) where (x, y) are the ground plane coordinates and (x, ˙ y) ˙ is the velocity. In the measurement mt we include the ground plane position zt (k) whose covari T ance is Rt (k), where: z = x y . Therefore we measure the state on the ground plane using a Kalman filter. The association between tracks is evaluated by using the Mahalanobis distance and it is computed as: H Tw + R (i) S jt (k) = H w Pt (k/k − 1)H j (k)

(2.5)

  (i) Thus, defining I(k) = z j − H w xˆ (k/k − 1) we can write: d 2jt = I(k)T S jt (k)−1 I(k)

(2.6)

where xˆ (k/k − 1) is the target prediction. To determine the association this distance is compared against a threshold and the Nearest Neighbor Algorithm is applied in order to establish the association. We can make the hypothesis that for each target, (i) in one view, there is at most one measurement, i.e. ∑ j β jt ≤ 1. Obviously if the (i)

measurement misses then ∑ j β jt = 0. The overall measurement for each target is the result of single camera measure(i) ments weighted with their covariance R j as follows:  Rt =

∑∑ i

(i) β jt



−1 (2.7)

j

 zt = R t

 (i) −1 Rj

∑∑ i

(i) β jt



 (i) −1 (i) Rj zt

(2.8)

j

These equations express the accuracy improvement obtained thanks to the fusion of all the views; as a matter of fact Equation 2.7 implies that the overall uncertainty is lower than the one provided by a single camera. Moreover Equation 2.8 ensures that the fused measurement is biased to the most accurate measurement from one camera. The target updating is, then, obtained in the following way: xˆ t (k/k) = xˆ t (k/k − 1) + K t (k) [zzt (k) − H w xˆ t (k/k − 1)]

(2.9)

 −1 H Tw H w Pt (k/k − 1)H H Tw + Rt (k) K t (k) = Pt (k/k − 1)H

(2.10)

H w ] Pt (k/k − 1) with 0 < η < 1. its covariance is: Pt (k/k) = [II − K t (k)H

38

S. Maludrottu, A. Dore, and C.S. Regazzoni

For those measurements that do not match to any existing targets, a target initialization procedure has to be applied. Hence, every unmatched measurement from a camera must be checked against the other unmatched measurements from other cameras to find new targets. (i∗ ) We define z j∗ as the unmatched measurement from the i∗ th camera and xn the new target associated with it. All the association matrices β (i) are extended by one (i) column, β jn , for the new target xn ; its elements represent the association between the jth measurement from ith camera and the new target. For the i∗ th camera the elements of the columns nth, referring to the new target, are:

1 if j = j∗ (i∗ ) β jn = (2.11) 0 otherwise (i∗ )

(i)

Then the Mahalanobis distance between z j∗ and each unmatched measurement z j from ith camera is calculated as:       (i) (i∗ ) T (i) (i∗ ) −1 (i) (i∗ ) di2∗ j = z j (k) − z j∗ R j − R j∗ z j (k) − z j∗

(2.12)

(i∗ )

(i)

where R j and R j∗ are the measurement covariances. Then, the new target is initialised as follows:  T xˆ n (k/k) = z n (k)T 0 0 (2.13)  R n (k) O 2 P n (k/k) = O 2 σv2 I 2

(2.14)

where O 2 and I 2 are 2 × 2 zero and identity matrices, respectively. The processing capability of the fusion node is limited due to the hardware specific processing resources (CPU, memory...). Similarly to the communication unit we define a maximum CPU load Cmax ; each of the Si ⊂ S∗ (k) sensors (output of the communication unit at a given time k) produces a CPU usage Ci (k). If ∑i Ci (k) >= Cmax we define a priority-based selection of the data streams in order    to obtain a feasible subset S (k) such that Ci (Si (k)) <= Cmax . This selection will cause a loss of input data in order to preserve the computational feasibility; thus the data fusion will be performed using only a subset of the whole input data to the fusion node. The information loss parameter for the fusion node is considered as: 

(k)| IF (k) = card|S card|S| . In this simulator the database is not modeled since the performed tasks do not need a data storage.

2.3.3 Performance Evaluation of Tracking Architectures The importance of evaluation of multi-sensor tracking architectures is demonstrated by the relevant body of research focused on the evaluation of reliability and accuracy

2

Performance Evaluation for Intelligent Surveillance

39

of automatic tracking applications. The general considerations about surveillance systems evaluation (see Sect. 2.2.5) are still valid. The vast majority of the works found in literature defines metrics related to specific parts of the architecture missing a more generic analysis. In order to evaluate the performances of such applications, different metrics have been defined: common performance metrics can be computed from true positives (TP), false positives (FP), true negatives (TN) or false negatives (FN) values. Other metrics can be used as well: in [17] an evaluation approach based on receiver operating curves (ROC) and an effectiveness F-measure metric is proposed. In [20] a multi-objective metric is adopted to evaluate and guide an evolutionary optimization of image processing algorithms. This function takes into account different accuracy and continuity values specific for a tracking application. The work described in [6] focuses on the definition of the main requirements for effective performance analysis for tracking methods. In [18], a number of metrics is defined for trajectory comparison, in order to determine how precisely a target is tracked by a computer vision system. In [29], an evaluation benchmark is presented to compare different visual surveillance algorithms developed for PETS (Performance Evaluation of Tracking and Surveillance) workshops. PETS metrics are specifically developed to provide an automatic mechanism to quantitatively compare similar purpose algorithms operating on the same data.

2.4 Experimental Results Three scenarios have been investigated in order to demonstrate the possibility of a complete evaluation of surveillance systems using the proposed method. In this experimental setup, no real-time constraints and no limitations induced by bandwidth/processing inadequacy are considered. Therefore, the information loss parameter Ii is considered equal to zero for each node i of the architecture. Moreover, since the DB is not modeled, we assume a DF equal to zero. In this case, the global performance metric P(F(T, R), DF , I) → P(F(T, R)) and is defined as P=

∑ i | f i − g i | + w · mi T OTi

(2.15)

where | fi − gi | is the Euclidean distance between the target position fi and its corresponding ground truth gi at timestep i. The term w is a penalty value that is set equal to 10, mi is the number of missing associations between ground truth data and fused tracks at timestep i, and T OTi is the total number of timesteps of the generated context (see Figure 2.5). The proposed metric combines two evaluation functions defined in [7]: a tracking matching error measure (TME) and a track detection failure measure (TDF). The former accounts for the positional error of tracking considered as the average distance between system tracks and ground truth tracks while the latter is the ratio of frames in which a moving object is not detected. A reference physical architecture has been defined as follows: five smart cameras c1...5 are placed in a parking lot area as shown in Fig.2.4. Their fields of view

40

S. Maludrottu, A. Dore, and C.S. Regazzoni

Fig. 2.5 Representation of the ground truth (’o’) and fused target tracking (’*’)

FOVs VSi (θi , ai ) are determined by two parameters: ai is the angle of view and θi is an orientation angle that accounts for the pose of the camera, defined as the angle between the central line of the FOV and an horizontal reference line (both parameters are measured in degrees). A common fusion node N and wireless communication channel W . In the following scenarios ai and θi have the following values: a1 = a2 = a3 = a4 = a5 = 45, θ1 = 90, θ2 = 200, θ3 = 90, θ4 = 45, θ5 = 330. First scenario. Sensor placement is an important design issue that has a critical impact on the overall performance of a multi-sensor surveillance system; many works address the sensors optimization problem and many comparative studies have been devoted to a careful comparison of different approaches (e.g. [8] and [9]). In this scenario a pose optimization problem is studied. The position of the cameras Ci is fixed but the orientation θi can be changed in real-time in order to obtain better tracking performances. An alternative optimized tracking architecture is defined to be evaluated and compared, using the proposed approach, to the default fixed cameras architecture. In this way it can be evaluated how the implementation of a pose optimization algorithm affects the overall performances of an already existing surveillance system. The overall structure (5 smart sensors, 1 fusion node and wireless communications) is maintained equal to the reference architecture. Only the orientation of sensors is changed over time to find a better configuration according to a genetic-based optimization technique. Genetic algorithms are population based search algorithms [12] used as optimization tools in a lot of different applications. In this particular case, a GA-based

2

Performance Evaluation for Intelligent Surveillance

41

optimization algorithm has been implemented to find the best orientations of a group of smart cameras in a distributed multi-sensor architecture. The input of GA is the current position of moving objects in the environment as perceived by the fusion node and the output is the orientation angle of each camera. The fitness function to be optimized for each sensor separately is formulated as ∑i wi di , where di is the perpendicular distance from an object position to the central line of FOV of sensor i and w is the number of objects in a given position. The camera is considered to be best oriented by minimizing the sum of distances. Parameters of GA are initialized as follows: population size of 20, number of generations equal to 100 and crossover and mutation rate respectively 0.8 and 0.04. Ten different scenes of various length with a fixed number of trajectories (10) have been provided by the context generator as input for the model. Many parameters influence the context generator algorithm behavior: the total number of trajectories, the generation rate, the Gaussian noise (i.e. the measurement error in smart sensors), etc. Those parameters have been randomly modified for each context generation in order to provide a ground truth as diversified as possible. Figure 2.6 presents the different results of the sample architectures and shows the connection between different camera placement strategies and performances: the dynamic camera orientation approach outperforms a fixed orientation solution up to 20%. Second scenario. Another critical issue in the design phase is the analysis of possible bottlenecks (defined as parts of the system that limit the overall performance) in

Fig. 2.6 Performance comparison of reference architecture (fixed camera orientation - in blue) and sensor-optimized architecture (in purple)

42

S. Maludrottu, A. Dore, and C.S. Regazzoni

Table 2.1 (a) Performance comparison of subparts of Architecture 1 (fixed cameras); (b) Performance comparison of subparts of Architecture 2 (optimized camera orientations). Cont refers to the generated context scene number used as input. Avg is the average performance over ten experiments. Cont Node 1 2 3 4 5 6 7 8 9 10

0.0188 0.0214 0.0182 0.0218 0.0362 0.0356 0.0439 0.0306 0.0348 0.035

S1

S2

S3

S4

S5

0.16 0.11 0.15 0.14 0.17 0.17 0.14 0.13 0.18 0.12

0.11 0.19 0.12 0.22 0.2 0.28 0.26 0.15 0.19 0.17

0.61 0.65 0.60 0.69 0.98 1.04 1.17 0.75 0.68 0.68

0.02 0.02 0.03 0.03 0.1 0.08 0.09 0.04 0.04 0.04

0.03 0.03 0.03 0.02 0.05 0.06 0.08 0.07 0.11 0.09

Avg 0.0296 0.15 0.19 0.78 0.05 0.06

Cont Node 1 2 3 4 5 6 7 8 9 10

0.0162 0.0189 0.0153 0.0184 0.03 0.0288 0.0347 0.0275 0.0296 0.033

S1

S2

S3

S4

S5

0.12 0.11 0.11 0.11 0.13 0.19 0.14 0.11 0.13 0.11

0.10 0.17 0.10 0.18 0.14 0.21 0.23 0.15 0.15 0.21

0.55 0.58 0.49 0.58 0.67 0.81 0.8 0.65 0.52 0.55

0.03 0.02 0.02 0.03 0.08 0.07 0.06 0.03 0.04 0.04

0.03 0.04 0.03 0.04 0.05 0.06 0.05 0.05 0.10 0.11

Avg 0.0252 0.13 0.16 0.62 0.04 0.06

Table 2.2 Performance comparison of different fusion architectures Context 1 2 3 4 5 6 7 8 9 10

Arch. A Arch. B 0.0254 0.0198 0.0238 0.0176 0.0164 0.0176 0.0258 0.022 0.0178 0.0312

0.0303 0.023 0.0316 0.0201 0.0271 0.0.275 0.0295 0.0278 0.025 0.0404

Average Perf. 0.0217 0.0282

the architecture. Detection of bottlenecks can avoid possible data losses or a better hardware selection in the design phase. Focusing on the architectures previously defined, the overall result in terms of tracking performances has been compared with the tracks produced by each smart sensor in the architecture in order to point out the contribution of each one to the overall performances. As shown in Table 2.1 sensor 3 has the worst performance since it acts as a bottleneck for the entire structure; a better placement or perhaps a more efficient sensor could improve the performance of the entire structure. Third scenario. The reference architecture (A) is compared to another possible surveillance architecture to perform multi-sensor tracking in the same environment.

2

Performance Evaluation for Intelligent Surveillance

43

The new architecture (B) is constituted by only three sensors (C1,C2 and C5 in Fig.    2.4) with fixed and wider fields of view (a1 = a2 = a3 = 60), more accurate target detection and tracking algorithms and better fusion node hardware. Ten different scenes with a fixed number of trajectories (10) have been provided as input for the model. In Table 2.2 it is shown that architecture A, even with worse hardware obtains a better overall result thanks to the higher number of sensor placed in the area.

2.5 Conclusions In this work a model of a system architecture for multi-sensor third generation surveillance systems has been proposed. These systems are defined as tree-like hierarchical structures, whose nodes (smart sensors, intermediate processing nodes and control centers) are capable of autonomous data processing functions. Surveillance tasks can be decomposed into a chain of logical modules according to a hierarchical data fusion structure. Each node has been modeled taking into account specific descriptive elements such as processing functions and communication units. Distribution of intelligence is realized as the mapping process of subtasks to single elements of the architecture. The aim of this model is to represent basic tasks performed by such a system in order to address the problem of performance evaluation. This description can be used to validate algorithms as well as system design with respect to the monitored scenario. A simulator of a multi-sensor hierarchical tracking system has been presented to show the appropriateness of the proposed model as a tool to assess system architectures and algorithms. Different design issues have been investigated, such as sensor selection and placement and bottleneck analysis.

References [1] Areta, J., Bar-Shalom, Y., Levedahl, M., Pattipati, K.R.: Hierarchical track association and fusion for a networked surveillance system. Journal of Advanced in Information Fusion 1(2), 140–157 (2006) [2] Bramberger, M., Doblander, A., Maier, A., Rinner, B., Schwabach, H.: Distributed embedded smart cameras for surveillance applications. Computer, 68–75 (2006) [3] Brdiczka, O., Chen, P., Zaidenberg, S., Reignier, P., Crowley, J.: Automatic acquisition of context models and its application to video surveillance. In: Proceedings of the International Conference in Pattern Recognition, vol. 1, pp. 1175–1178 (2006) [4] Chen, T., Haussecker, H., Bovyrin, A., Belenov, R., Rodyushkin, K., Kuranov, A., Eruhimov, V.: Computer vision workload analysis: Case study of video surveillance systems. INTEL Technology Journal 9, 109–118 (2005) [5] Dore, A., Pinasco, M., Regazzoni, C.: Multi-modal data fusion techniques and applications. In: Multi-camera Networks, pp. 213–237 (2009) [6] Ellis, T.: Performance metrics and methods for tracking in surveillance. In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (2002)

44

S. Maludrottu, A. Dore, and C.S. Regazzoni

[7] Eom, K.Y., Ahn, T.K., Kim, G.J., Jang, G.J., Jung, J.Y., Kim, M.H.: Hierarchically categorized performance evaluation criteria for intelligent surveillance system. In: International Symposium on Web Information Systems and Applications, pp. 218–221 (2009) [8] Ercan, A.O., Yang, D.B., El Gamal, A., Guibas, L.: Optimal placement and selection of camera network nodes for target localization. In: Gibbons, P.B., Abdelzaher, T., Aspnes, J., Rao, R. (eds.) DCOSS 2006. LNCS, vol. 4026, pp. 389–404. Springer, Heidelberg (2006) [9] Erdem, U., Sclaroff, S.: Automated camera layout to satisfy task-specific and floor-plan specific coverage requirements. Computer Vision and Image Understanding 103, 156– 169 (2006) [10] Foresti, G., Snidaro, L.: A Distributed Sensor Network for Video Surveillance of Outdoors, ch. 1, pp. 7–27 (2003) [11] Hall, D., McMullen, S.: Mathematical Techniques in Multisensor Data Fusion, ch. 2, pp. 37–72. Artech House, Boston (2004) [12] Holland, J.H.: Adaptation in natural and artificial systems. MIT Press, Cambridge (1992) [13] Marcenaro, L., Oberti, F., Foresti, G., Regazzoni, C.: Distributed architectures and logical-task decomposition in multimedia surveillance systems. Proceedings of the IEEE (10), 1419–1440 (2001) [14] Mathiason, G., Andler, S., Son, S., Selano, L.: Virtual full replication for wireless sensor networks, Proceedings of the 19th ECRTS (WiP) (2007) [15] Micheloni, C., Foresti, G., Snidaro, L.: A network of cooperative cameras for visualsurveillance. IEE Visual, Image and Signal Processing 152(2), 205–212 (2005) [16] Moncrieff, S., Venkatesh, S., West, G.: Context aware privacy in visual surveillance. In: International Conference in Pattern Recognition (ICPR 2008), pp. 1–4 (2008) [17] Nazarevic-McManus, N., Renno, J., Jones, G.: Performence evaluation in visual surveillance using the f-measure. In: ACM International Workshop on Video Surveillance and Sensor Networks, pp. 45–52 (2006) [18] Needham, C.J., Boyle, R.D.: Performance evaluation metrics and statistics for positional tracker evaluation. In: Crowley, J.L., Piater, J.H., Vincze, M., Paletta, L. (eds.) ICVS 2003. LNCS, vol. 2626, pp. 278–289. Springer, Heidelberg (2003) [19] Nogueira, L., Pinho, L.: Iterative refinement approach for qos-aware service configuration. In: Kleinjohann, B., Kleinjohann, L., Machado, R., Pereira, C., Thiagarajan, P. (eds.) From Model-Driven Design to Resource Management for Distributed Embedded Systems, IFIP International Federation for Information Processing. Springer, Boston ´ Garc´ıa, J., Berlanga, A., Molina, J.M.: Adjustment of surveillance video sys[20] P´erez, O., ´ tems by a performance evaluation function. In: Mira, J., Alvarez, J.R. (eds.) IWINAC 2005. LNCS, vol. 3562, pp. 499–508. Springer, Heidelberg (2005) [21] Remagnino, P., Foresti, G.: Ambient intelligence: A new multidisciplinary paradigm. IEEE Transactions on Systems, Man and Cybernetics - Part A 35(1) (2005) [22] Remagnino, P., Shihab, A., Jones, G.: Distributed intelligence for multi-camera visual surveillance. Pattern Recognition (37), 675–689 (2004) [23] Taylor, G.R., Chosak, A.J., Brewer, P.C.: Ovvv: Using virtual worlds to design and evaluate surveillance systems, IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007. pp. 1–8 (2007) [24] Tsui, P., Brimicombe, A.J.: The hierarchical tassellation model and its use in spatial analysis. Transactions in GIS 2(3), 267–279 (1997)

2

Performance Evaluation for Intelligent Surveillance

45

[25] Valera, M., Velastin, S.: Intelligent distributed surveillance systems: a review. IEE Proceedings - Vision Image and Signal Processing 152, 192–204 (2005) [26] Velastin, S., Khoudour, L., Lo, B., Sun, J., Vicencio-Silva, M.: Prismatica: A multisensor surveillance system for public transport network. In: Proceedings of 12th IEE Road Transport Information and Control Conference (RTIC 2004) (2004) [27] Xiaoling, X., Li, L.: Real time analysis of situation events for intelligent surveillance. In: Proceedings of International Symposium on Computational Intelligence and Design (ICID 2008), pp. 122–125 (2008) [28] Xu, M., Orwell, J., Lowey, L., Thirde, D.: Architecture and algorithms for tracking football players with multiple cameras. IEE Proceedings - Vision, Image and Signal Processing 152(2), 232–241 (2005) [29] Young, D., Ferryman, J.: Pets metrics: On-line performance evaluation service. In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (2005)

Chapter 3

Incremental Learning on Trajectory Clustering Luis Patino, Franc¸ois Bremond, and Monique Thonnat

Abstract. Scene understanding corresponds to the real time process of perceiving, analysing and elaborating an interpretation of a 3D dynamic scene observed through a network of cameras. The whole challenge consists in managing this huge amount of information and in structuring all the knowledge. On-line Clustering is an efficient manner to process such huge amounts of data. On-line processing is indeed an important capability required to perform monitoring and behaviour analysis on a long-term basis. In this paper we show how a simple clustering algorithm can be tuned to perform on-line. The system works by finding the main trajectory patterns of people in the video. We present results obtained on real videos corresponding to the monitoring of the Toulouse airport in France.

3.1 Introduction Scene understanding corresponds to the real time process of perceiving, analysing and elaborating an interpretation of a 3D dynamic scene observed through a network of sensors (including cameras and microphones). This process consists mainly in matching signal information coming from sensors observing the scene with a large variety of models which humans are using to understand the scene. This scene can contain a number of physical objects of various types (e.g. people, vehicle) interacting with each other or with their environment (e.g. equipment) more or less structured. The scene can last a few instants (e.g. the fall of a person) or a few months (e.g. the depression of a person), can be limited to a laboratory slide observed through a microscope or go beyond the size of a city. Sensors include usually cameras (e.g. omnidirectional, infrared), but also may include microphones and other sensors (e.g. optical cells, contact sensors, physiological sensors, smoke detectors, GNSS). Luis Patino · Franc¸ois Bremond · Monique Thonnat INRIA Sophia Antipolis - M´editerran´ee, 2004 route des Lucioles - BP 93 - 06902 Sophia Antipolis E-mail: [email protected] {Francois.Bremond,Monique.Thonnat}@inria.fr P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 47–70. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

48

L. Patino, F. Bremond, and M. Thonnat

Despite few success stories, such as traffic monitoring (e.g. Citilog), swimming pool monitoring (e.g. Poseidon) and intrusion detection (e.g. ObjectVideo, Keeneo), scene understanding systems remain erratic and can function only under restrictive conditions (e.g. during day rather than night, diffuse lighting conditions, no shadows). Having poor performance over time, they are hardly modifiable, containing little a priori knowledge on their environment. Moreover, these systems are very specific and need to be redeveloped from scratch for new applications. To answer these issues, most researchers have tried to develop original vision algorithms with focused functionalities, robust enough to handle real life conditions. Up to now no vision algorithms were able to address the large varieties of conditions characterising real world scenes, in terms of sensor conditions, hardware requirements, lighting conditions, physical object varieties, application objectives... Here we state that the scene understanding process relies on the maintenance of the coherency of the representation of the global 3D scene throughout time. This approach which can be called 4D semantic interpretation is driven by models and invariants characterising the scene and its dynamics. The invariants (called also regularities) are general rules characterising the scene dynamics. For instance, the intensity of a pixel can change significantly mostly in two cases: change of lighting conditions (e.g. shadow) or change due to a physical object (e.g. occlusion). Another rule verifies that physical objects cannot disappear in the middle of the scene. There is still an open issue which consists in determining whether these models and invariants are given a priori or are learned. The whole challenge consists in managing this huge amount of information and in structuring all this knowledge in order to capitalise experiences, to share them with other computer vision systems and to update them along experimentations. To face this challenge several knowledge engineering tools are needed: Tools for scene perception. A first category of tools contains vision algorithms to handle all the varieties of real world conditions. The goal of all these algorithms is to detect and classify the physical objects which are defined as interesting by the users. A first set of algorithms consists of robust segmentation algorithms for detecting the physical objects of interest. These segmentation algorithms are based on the hypothesis that the objects of interest are related to what is moving in the video, which can be inferred by detecting signal changes. Specific algorithms to separate physical objects from different categories of noise (e.g. due to light change, ghost, moving contextual object), and algorithms to extract meaningful features (e.g. 3D HOG, wavelet based descriptors, colour histograms) characterising the objects of interest belong to this category. Tools for verification of the 3D coherency throughout time (physical world) A second category of tools are the ones combining all the features coming from the detection of the physical objects observed by different sensors and in tracking these objects throughout time. Algorithms for tracking multiple objects in 2D or 3D with one camera or a network of cameras belong here; for instance, algorithms that take advantage of contextual information and of a graph of tracked moving regions where an object trajectory can be seen as the most probable path in the graph. This

3

Incremental Learning on Trajectory Clustering

49

property enables to process long video sequences and to ensure the trajectory coherence. Moreover, these tracking algorithms compute the uncertainty of the tracking process by estimating the matching probability of two objects at successive instants. A second example is to fuse the information coming from several sensors at different levels depending on the environment configuration. Information fusion at the signal level can provide more precise information, but information fusion at higher levels is more reliable and easier to accomplish. In particular, we are using three types of fusion algorithms: (1) multiple cameras with overlapping field of view, (2) a video camera with pressure sensors and sensors to measure the consumption of water and electrical appliances, and (3) video cameras coupled with other sensors (contact sensors and optical cells). Tools for Event recognition (semantic world). At the event level, the computation of relationships between physical objects constitutes a third category of tools. Here, the real challenge is to explore efficiently all the possible spatio-temporal relationships of these objects that may correspond to events (called also actions, situations, activities, behaviours, scenarios, scripts and chronicles). The varieties of these events are huge and depend on their spatial and temporal granularities, on the number of the physical objects involved in the events, and on the event complexity (number of components constituting the event and the type of temporal relationship). Different types of formalism can be used: HMM and Bayesian networks, temporal scenarios [28]. Tools for knowledge management. To be able to improve scene understanding systems, we need at one point to evaluate their performance. Therefore we have proposed a complete framework for performance evaluation which consists of a video data set associated with ground-truth, a set of metrics for all the tasks of the understanding process, an automatic evaluation software and a graphical tool to visualise the algorithm performance results (i.e. to highlight algorithm limitations and to perform comparative studies). Scene understanding systems can be optimised using machine learning techniques in order to find the best set of program parameters and to obtain an efficient and effective real-time process. It is also possible to improve system performance by adding a higher reasoning stage and a feedback process towards lower processing layers. Scene understanding is essentially a bottom-up approach consisting in abstracting information coming from signal (i.e. approach guided by data). However, in some cases, a top-down approach (i.e. approach guided by models) can improve lower process performance by providing a more global knowledge of the observed scene or by optimising available resources. In particular, the global coherency of the 4D world can help to decide whether some moving regions correspond to noise or to physical objects of interest. Tools for Communication, Visualisation Knowledge Acquisition and learning Even when the correct interpretation of the scene has been performed, a scene understanding system still has to communicate its understanding to the users or to adapt its processing to user needs. A specific tool can be designed for acquiring a priori knowledge and the scenarios to be recognised through end-user interactions.

50

L. Patino, F. Bremond, and M. Thonnat

3D animations can help end-users to define and to visualize these scenarios. Thus, these tools aim at learning the scenarios of interest for users. These scenarios can be often seen as the complex frequent events or as frequent combinations of primitive events called also event patterns. The present work belongs to the latter category. We aim at designing an unsupervised system for the extraction of structured knowledge from large video recordings. By employing clustering techniques, we define the invariants (as mentioned above) characterising the scene dynamics. However, not all clustering techniques are well adapted to perform on-line. On-line learning is indeed an important capability required to perform scene analysis on long-term basis. In this work we show how meaningful scene activity characterisation can be achieved through trajectory analysis. To handle the difficulty of processing large amounts of video, we employ a clustering algorithm that has been tuned to perform on-line. The approach has been validated on video data from the Toulouse airport in France (European project COFRIEND [1]). The reminder of the paper is organised as follows. We review some relevant work for scene interpretation from trajectory analysis in the following subsection (section 3.1.1 Related Work). We present the general structure of our approach in section 3.2 (General structure of the proposed approach). A brief description of the object detection and tracking employed in our system is given in section 3.3 (On-line processing: Real-time Object detection). The detailed description of the trajectory analysis undertaken in this work is given in section 4 (Trajectory analysis), including the algorithm to tune the clustering parameters. The evaluation of the trajectory analysis work is given in the following section. The results obtained after processing the video data from the Toulouse airport are presented in section 6. Our general remarks and conclusions are given at the end of the paper.

3.1.1 Related Work Extraction of the activities contained in the video by applying data-mining techniques represents a field that has only started to be addressed. Recently it has been shown that the analysis of motion from mobile objects detected in videos can give meaningful activity information. Trajectory analysis has become a popular approach due to its effectiveness in detecting normal/abnormal behaviours. For example, Piciarelli et al. [22] employ a splitting algorithm applied on very structured scenes (such as roads) represented as a zone hierarchy. Foresti et al. [11] employ an adaptive neural tree to classify an event occurring on a parking lot (again a highly structured scene) as normal/suspicious/dangerous. Anjum et al. [2] employ PCA to seek for trajectory outliers. In these cases the drawback of the approach is that the analysis is only adapted to highly structured scenes. Similarly, Naftel et al. [20] first reduce the dimensionality of the trajectory data employing Discrete Fourier Transform (DFT) coefficients and then apply a self-organizing map (SOM) clustering algorithm to find normal behaviour. Antonini et al. [3] transform the trajectory data employing Independent Component Analysis (ICA), while the final clusters

3

Incremental Learning on Trajectory Clustering

51

are found employing an agglomerative hierarchical algorithm. In these approaches it is however delicate to select the number of coefficients that will represent the data after dimensionality reduction. Data mining of trajectories has also been applied with statistical methods. Gaffney et al. [13] have employed mixtures of regression models to cluster hand movements, although the trajectories were constrained to have the same length. Hidden Markov Models (HMM) have also been employed [5, 21, 24]. In addition to activity clustering, so as to enable dynamic adaptation to unexpected event processing or newly observed data, we need a system able to learn the activity clusters in an on-line way. On-line learning is indeed an important capability required to perform behaviour analysis on long-term basis and to anticipate the human interaction evolutions. An on-line learning algorithm gives a system the ability to incrementally learn new information from datasets that consecutively become available, even if the new data introduce additional classes that were not formerly seen. This kind of algorithm does not require access to previously used datasets, yet it is capable of largely retaining the previously acquired knowledge and has no problem of accommodating any new classes that are introduced in the new data [23]. Some various restrictions, such as whether the learner has partial or no access to previous data [26, 17, 15], or whether new classes or new features are introduced with additional data [30], have also been proposed [19]. Many popular classifiers, however, are not structurally suitable for incremental learning; either because they are stable [such as the multilayer perceptron (MLP), radial basis function (RBF) networks, or support vector machines (SVM)], or because they have high plasticity and cannot retain previously acquired knowledge, without having access to old data (such as -nearest neighbor) [19]. Specific algorithms have been developed to perform on-line incremental learning, such as Leader [14], Adaptive Resonance Theory modules (ARTMAP) [8, 7], Evolved Incremental Learning for Neural Networks [25], leaders-subleaders [29], and BIRCH [18]. Among them, the Hartigan algorithm [14], also known as Leader algorithm, is probably the most employed in the literature. The Leader algorithm, computes the distance between new data and already built clusters to decide to associate these new data with the clusters or to generate new ones better characterising the data. However, all these approaches rely on a manually-selected threshold to decide whether the data is too far away from the clusters. To improve this approach we propose to control the learning rate with coefficients indicating how flexible the cluster can be updated with new data.

3.2 General Structure of the Proposed Approach The monitoring system is mainly composed of two different processing components (shown in Figure 3.1). The first one is a video analysis subsystem for the detection and tracking of objects. This is a processing that goes on a frame-by-frame basis. The second subsystem achieves the extraction of trajectory patterns from the video. This subsystem is composed of two modules: The trajectory analysis module and the statistical analysis module. In the first module we perform the analysis of trajectories

52

L. Patino, F. Bremond, and M. Thonnat

by clustering and obtain behavioural patterns of interaction. In the second module we compute meaningful descriptive measures on the scene dynamics. For the storage of video streams and the trajectories obtained from the video processing module, a relational database has been setup. The trajectory analysis modules read the trajectories from the database and return the identified trajectory types; the discovered activities on the video; and resulting statistics calculated from the activities. Streams of video are acquired at a speed of 10 frames per second. The video analysis subsystem takes its input directly from the data acquisition component; the video is stored into the DB parallel to the analysis process. The whole system helps the manager or designer who wants to get global and long-term information from the monitored site. The user can specify a period of time where he/she wishes to retrieve and analyse stored information. In particular the user can access the whole database to visualize specific events, streams of video and off-line information.

Fig. 3.1 General architecture of the system.

3.3 On-Line Processing: Real-Time Object Detection Tracking objects in video is not the main contribution of this paper and therefore only a general description is made here. Detecting objects in an image is a difficult and challenging task. One solution widely employed consists in performing a thresholding operation between the pixel intensity of each frame with the pixel intensity of a background reference image. The latter can be a captured image of the same scene having no foreground objects, or no moving objects in front of the camera. The result of the thresholding operation is a binary mask of foreground pixels. The neighbouring foreground pixels are grouped together to form regions often referred

3

Incremental Learning on Trajectory Clustering

53

to as blobs which correspond to the moving regions in the image. If the moving object projection in the image plane does not overlap with each other, i.e. no dynamic occlusion, then each detected moving blob corresponds to a single moving object. The detailed description of the background subtraction algorithm, which also estimates when the background reference image needs to be updated, can be found in [12]. Having 3D information about the scene under view enables the calibration of the camera. Point correspondences between selected 3D points in the scene and their corresponding point in the 2D image plane allow us to generate the 3D location of any points belonging to moving objects. Thus, the 3D (i.e. width and height) of each detected moving blob can be measured as well as their 3D location on the ground plane in the scene with respect to a chosen coordinate system. The 3D object information is then compared against several 3D models provided by the user. From this comparison, a detected object is linked to a semantic class. Detected and classified 3D objects in a scene can be tracked within the scope of the camera using the 3D information of their location on the ground as well as their 3D dimensions. Tracking a few objects in a scene can be easy as far as they do not interact heavily in front of the camera: i.e. occlusions are rare and short. However, the complexity of tracking several mobile objects becomes a non-trivial and very difficult task to achieve when several object projected images overlap with each other on the image plane. Occluded objects have missing or wrong 3D locations, which can create incoherency in the temporal evolution of their 3D location. Our tracking algorithm [4] builds a temporal graph of connected objects over time to cope with the problems encountered during tracking. The detected objects are connected between each pair of successive frames by a frame to frame (F2F) tracker. Links between objects are associated with a weight (i.e. a matching likelihood) computed from three criteria: the comparison between their semantic class, 3D dimensions, and their 3D distance on the ground plane. The graph of linked objects provided by the F2F tracker is then analysed by the tracking algorithm, also referred to as the Long Term tracker, which builds paths of mobiles according to the link weights. The best path is then taken out as the trajectory of the related mobiles. The proposed tracking approach has the advantage of being simple to implement and able to run at a ‘high’ frame rate. However, it is sensitive to noise and this could prevent tracking correctly long trajectories.

3.4 Trajectory Analysis The second layer of analysis in our approach is related to the knowledge discovery of higher semantic information from analysis of activities recorded over a period of time that can span, for instance, from minutes to a whole day (or several days of recording). Patterns of activity are extracted from the analysis of trajectories.

3.4.1 Object Representation: Feature Analysis For the trajectory pattern characterisation of the object, we have selected a comprehensive, compact, and flexible representation. It is suitable also for further analysis

54

L. Patino, F. Bremond, and M. Thonnat

as opposed to many video systems, which actually store the sequence of object locations for each frame of the video building thus a cumbersome representation with little semantic information. If the dataset is made up of N objects, the trajectory for object O j in this dataset is defined as the set of points [x j (t), y j (t)] corresponding to their position points; x and y are time series vectors whose length is not equal for all objects as the time they spend in the scene is variable. Two key points defining these time series are the beginning and the end, [x j (1), y j (1)] and [x j (end), y j (end)] as they define where the object is coming from and where it is going to. We build a feature vector from these two points. Additionally, we also include the directional information given as [cos(θ ), sin(θ )], where θ is the angle which defines the vector joining [x j (1), y j (1)] and [x j (end), y j (end)]. A mobile object seen in the scene is thus represented by the feature vector: v j = [x j (1), y j (1), x j (end), y j (end), cos(θ ), sin(θ )]

(3.1)

This feature vector constitutes a set of simple descriptors that have proven experimentally to be enough to describe activities in a large variety of domains (such as traffic monitoring, subway control, monitoring smart environments), mainly because they are the most salient, but also they are appropriate for real world videos depicting unstructured scenes where trajectories of different types have strong overlap and they are usually the ones used by end-users of different domains.

3.4.2 Incremental Learning We need a system able to learn the activity clusters in an on-line way. On-line learning is indeed an important capability required to perform behaviour analysis on a long-term basis. A first approach proposed in the state-of-the-art for on-line clustering is the Leader algorithm [14]. In this method, it is assumed that a rule for computing the distance D between any pair of objects, and a threshold T is given. The algorithm constructs a partition of the input space (defining a set of clusters) and a leading representative for each cluster, so that every object in a cluster is within a distance T of the leading object. The threshold T is thus a measure of the diameter of each cluster. The clusters CLi , are numbered CL1 , CL2 , CL3 , . . . , CLk . The leading object representative associated with cluster CL j is denoted by L j . The algorithm makes one pass through the dataset, assigning each object to the cluster whose leader is the closest and making a new cluster, and a new leader, for objects that are not close enough to any existing leaders. The process is repeated until all objects are assigned to a cluster. Leader-subleader [29], ARTMAP [8] and BIRCH [18] algorithms are of this type. The strongest advantages of the Leader algorithm are that it requires a single scan of the database, and only cluster representatives need to be accessed during processing. However, the algorithm is extremely sensitive to threshold parameter defining the minimum activation of a cluster CL. A new input object defined by its feature vector v will be allocated to cluster CL j if v falls into its input receptive field (hyper-sphere whose radio is given by r j = T ). Defining T is application dependent. It can be supplied by an expert with a deep knowledge

3

Incremental Learning on Trajectory Clustering

55

of the data or employing heuristics. In this work we propose to learn this parameter employing a training set and a machine learning process. Let each cluster CLi be defined by a radial basis function (RBF) centered at the position given by its leader Li : CLi (v) = φ (Li , v, T ) = exp(−v − Li2 T 2 )

(3.2)

The RBF function has a maximum of 1 when its input is v = Li and thus acts as a similarity detector with decreasing values outputted whenever v strides away from Li . We can make the choice that an object element will be included into a cluster if CLi (v) ≥ 0.5, which is a natural choice. The cluster receptive field (hyper-sphere) is controlled by the parameter T. Now, consider C = {CL1 , . . . ,CLk } is a clustering structure of a dataset X = {v1 , v2 , . . . , vN }; {L1 , L2 , . . . , Lk } are the leaders in this clustering structure and P = {P1, . . . , Ps } is the true partition of the data (Ground-truth) and {M1 , . . . , Ms } are the main representatives (or Leaders) in the true partition. We can define an error function given by E=

1 N 

N

∑ Ej

(3.3)

j=1

⎧    ⎨ 0 i f L (v j ) , v j ∈ CLi ; L (v j ) , v j ∈ Pi E j = −1 i f v j ∈ CLi ; |CLi | = 1 and L (v j ) = M (v j ) ⎩ 1 otherwise

(3.4)

and L(v j ) is the Leader associated to v j in the clustering structure Ci and |Ci | is the cluster cardinality. M(v j ) is the Leader associated to v j in the true partition P. In the above equation, the first case represents a good clustering when the cluster prototype and the cluster elements match the ground truth partition P. The error is zero and the cluster size is correct. The second case corresponds to a cluster made of a singleton element. This element prototype does not correspond to any expected cluster prototype in the ‘true’ partition P. In this case the cluster size has to grow in order to enclose the singleton element. The remaining case is where an element is wrongly included into a cluster; The cluster size has then to decrease to exclude unwilling border elements. Minimising this error is equivalent to refining the clustering structure C or is equivalent to adjusting the parameter T that controls the cluster receptive field. A straightforward way to adjust T and minimise the error is employing an iterative gradient-descent method: T (t + 1) = T (t) − η

∂ E(t) ∂T

(3.5)

where the error gradient at time t is:

∂ E(t) 1 ∂ Φˆ = ∑ E j (t) ∂T N j ∂T

(3.6)

56

L. Patino, F. Bremond, and M. Thonnat

and the cluster activation gradient is:  2  ∂ Φˆ ∂  = exp − v j − L(v j ) T 2 ∂T ∂T

(3.7)

  2 2 ∂ Φˆ = − v j − L(v j ) (2T ) exp(− v j − L(v j ) T 2 ) ∂T The threshold update can thus be written as:

(3.8)

 2  1 E j (t) −2T v j − L(v j ) Φˆ ∑ N j

(3.9)

T (t + 1) = T (t) − η

The final value is typically set when the error is sufficiently small or the process reaches a given number of iterations. Convergence to an optimum value for both, T and E is only guaranteed if data in the ‘true’ partition P is well structured (having high intra-class homogeneity and high inter-class separation; see unsupervised/supervised evaluation below). With the purpose of tuning parameter T, and for this application, we have defined a Training data set (with associated Ground-truth) containing sixty nine synthetic trajectories. The ground-truth trajectories were manually drawn on a top view scene image. Figure 3.2 shows the empty scene of the Toulouse airport with some drawn trajectories. Semantic descriptions such as From Taxi parking area to Tow-tractor waiting point were manually given. There are twenty three of such annotated semantic descriptions, which are called in the following trajectory types. Each trajectory type is associated with a main trajectory that best matches that description. Besides, two complementary trajectories define the confidence limits within which we can still associate that semantic description. In figure 3.2 the main trajectory of each trajectory type is represented by a red continuous line while blue broken lines represent the complementary trajectories of the trajectory type. Thus, each ground-truth trajectory is associated to a semantic descriptor or trajectory type. Each trajectory type contains a triplet of trajectories. The proposed gradient-descent methodology was applied to the ground-truth dataset. The threshold T, in the leader algorithm, is initially set to a large value (which causes a merge of most trajectory types). Figure 3.3 shows how this threshold value evolves as the gradient algorithm iterates. The graph for the corresponding error is shown in figure 3.4. Remark that for this application we have not encountered local minima problems. However, as gradient-descent algorithms are clearly exposed to this problem, it could be envisaged to verify whether the minima found is indeed the global optima. A multiresolution analysis would be of help for this. It is also possible to evaluate, in an unsupervised or supervised manner, the quality of the resulting clustering structure: Unsupervised evaluation: typical clustering validity indexes evaluating the intracluster homogeneity and inter-cluster separation such as Silhouette [6, 16], Dunn [10] and Davies-Bouldin [9] indexes (given in Annex 1) can be employed.

3

Incremental Learning on Trajectory Clustering

57

Fig. 3.2 Ground-truth for different semantic clusters.

Fig. 3.3 Evolution of the threshold T controlling the cluster receptive field.

Figure 3.5 shows the evolution of these three indexes on the clustering of trajectories as the gradient-descent algorithm evolves. Supervised evaluation: supervised validity indexes, which in this case compare the clustering results to the true data partition, such as the Jaccard index [27] (given in

58

L. Patino, F. Bremond, and M. Thonnat

Fig. 3.4 Evolution of the gradient-descent error with the number of iterations. The error gives an indication of how many elements of different trajectory types are merged together in a single cluster.

Fig. 3.5 Validity indexes such as Silhouette (higher values are better), Dunn (higher values are better) and Davies-Bouldin (lower values are better) at each iteration step of the gradientdescent algorithm.

Annex 2) can also be employed. Figure 3.6 shows the evolution of this index on the clustering of trajectories as the gradient-descent algorithm evolves. For large values of the threshold T (above 1.5) it is possible to see that a large number of trajectories are badly clustered (about 1/3 of the dataset). The unsupervised indexes are also unstable (presenting some oscillatory changes over the different iterations) and indicative of a bad clustering structure (meaning low inter-cluster distance and high intra-cluster distance). The mapping with the true partition is also poor (indicated by low values of the Jaccard index). For values of the threshold T below 1.4 there is an almost monotonically improvement of the unsupervised and

3

Incremental Learning on Trajectory Clustering

59

Fig. 3.6 Supervised evaluation at each iteration step of the gradient-descent algorithm. The Jaccard index compares the resulting clustering with the partition given by the trajectory ground-truth.

supervised clustering indexes. The Jaccard index reaches its maximum value (meaning a perfect matching with the true partition) for a threshold T=0.79, which is then selected for our analysis. The unsupervised indexes are also indicative of a good clustering structure. The Leaders defined from this process are selected as the initial cluster centres that will guide the partition of new incoming data.

3.5 Trajectory Analysis Evaluation In order to test the efficiency of the trajectory clustering algorithm we have analysed a new set of synthetic trajectories, which we denote by ‘experimental set’. This new set was composed of 230 trajectories, which have the same structure as the training dataset; that is, each trajectory is associated with a semantic meaning. Moreover, each trajectory in the test dataset was generated from the Ground-truth dataset in the following form. Trajectories are generated by randomly selecting among points uniformly distributed on each side between the main trajectory and the two adjacent trajectories. These points lie on segments linking the main trajectory to the two adjacent trajectories. Each segment starts on a sample point of the main trajectory and goes to the nearest sample point in the adjacent trajectory. Adjacent trajectories are up-sampled for better point distribution. Ten points lie on each segment linking the main trajectory and an adjacent trajectory. For this reason, the trajectories generated from the principal trajectory will convey the same semantic. Figure 3.7 below shows a couple of examples. The clustering algorithm is then run again without any knowledge of the semantic description for each trajectory on the experimental data set. After the clustering process is achieved, the resulting partition can be assessed by comparing with the one initially defined by the Ground-truth; what the Jaccard index does. In this case, the

60

L. Patino, F. Bremond, and M. Thonnat

Fig. 3.7 Four different sets of synthetic trajectories. Each set contains twenty trajectories different from the main and adjacent trajectories previously defined in the Ground-truth dataset (Figure 3.2).

Jaccard index takes a value of 0.9119 (our baseline). Moreover, typical metrics related to the ROC space (Receiver operating characteristics) can be computed which evaluate both the mapping between the clustering algorithm output and the Groundtruth partition. These measures are the true positive rate (TPR) and false positive rate (FPR), which in this case take the following values: TPR=0.9565, FPR=0.002. In order to assess the robustness of the trajectory clustering algorithm, we have evaluated our approach on more different sets of synthetic objects. Each set has the particularity of containing groups of very similar trajectories (even with an overlap in the most difficult cases), yet associated with different semantics. The evaluation consists thus in assessing how much the clustering algorithm will be affected by the different levels of complexity/noise introduced. To characterise the different datasets, we have computed the unsupervised clustering indexes Silhouette, Dunn and Davies-Bouldin. The table below (Table 3.1) summarises the results. Each experimental set mentioned in the table above contains an increasing complexity. For instance, some groups of trajectories defined in the ‘experimental set 2’, which contain some overlap between them, are also present in the next experimental sets (experimental set 3, 4 , 5., . . . ). For each experimental set, some new groups of trajectories are added, which in turn induce more overlapping situations and will also be present in the following experimental sets. The figure below (Figure 3.8) presents some examples of such trajectories in the different experimental sets. The structuring indexes ‘Silhouette’, ‘Dunn’ and ‘Davies-Bouldin’ reflect the less distinct separation induced between trajectories with different semantic meanings (Silhouette and Dunn indexes decrease, while the Davies-Bouldin index increases). The different experimental datasets cover situations presenting a very strong separation between groups of trajectories with different semantic meanings (structuring indexes Silhouette=0.85 Dunn=0.79 Davies-Bouldin=0.22), partial confusion

3

Incremental Learning on Trajectory Clustering

61

Table 3.1 Clustering results on different synthetic datasets. Input

Output

Name

Nb. Trj.

Nb. Structure GT’s characteristics Sil Dunn DB

Nb. Structure Clusters characteristics Sil Dunn DB

Jaccard ROC measures Index TPR FPR

ExpSet ExpSet2 ExpSet3 ExpSet4 ExpSet5 ExpSet6 ExpSet7 ExpSet8 ExpSet9 ExpSet10 ExpSet11 ExpSet12 ExpSet13 ExpSet14

230 280 340 440 520 590 650 710 750 840 890 920 990 1070

23 28 34 44 52 59 65 71 75 84 89 92 99 107

22 27 32 42 51 55 59 65 62 71 72 77 86 77

0.91 0.92 0.87 0.85 0.91 0.86 0.79 0.82 0.61 0.66 0.60 0.62 0.62 0.45

0.85 0.82 0.80 0.76 0.75 0.73 0.72 0.70 0.70 0.67 0.66 0.65 0.64 0.60

0.79 0.59 0.58 0.47 0.46 0.47 0.39 0.37 0.39 0.25 0.13 0.21 0.22 0.04

0.22 0.24 0.26 0.33 0.33 0.37 0.39 0.40 0.41 0.45 0.48 0.50 0.52 0.61

0.82 0.80 0.76 0.73 0.74 0.70 0.67 0.67 0.57 0.58 0.55 0.54 0.53 0.45

0.79 0.80 0.03 0.06 0.30 0.14 0.08 0.05 0.03 0.03 0.04 0.02 0.04 0.02

0.30 0.33 0.41 0.47 0.42 0.49 0.57 0.54 0.62 0.65 0.66 0.68 0.66 0.74

0.95 0.96 0.94 0.92 0.95 0.93 0.88 0.91 0.80 0.82 0.77 0.79 0.77 0.66

0.0020 0.0013 0.0018 0.0016 0.00075 0.0012 0.0017 0.0012 0.0025 0.0020 0.0024 0.0022 0.0019 0.0030

(Silhouette=0.7588 Dunn=0.4690 Davies-Bouldin=0.3331), high confusion (Silhouette=0.6517 Dunn=0.2131 Davies-Bouldin=0.5031) and very high confusion (Silhouette=0.6098 Dunn=0.0463 Davies-Bouldin=0.6157). The trajectory clustering algorithm performs accordingly, having more difficulty to retrieve all initial semantic groups when the confusion increases (thus the internal structure of the input data decreases); at the same time, the mapping between the trajectory clustering results and the semantic groups (GT) also worsens as exposed by the Jaccard Index. However, the overall behaviour shown by the true positive rate (TPR) and false positive rate (FPR) remains globally correct with TPR values near or above 0.77 for all studied cases except for the worst case experimental set 14 where the TPR is below 0.7. In order to assess the generalisation capability of the trajectory clustering algorithm, we have carried out new experiments employing the CAVIAR dataset (http://www-prima.inrialpes.fr/PETS04/caviar data.html). The dataset contains people observed at the lobby entrance of a building. The annotated groundtruth includes for each person its bounding box (id, centre coordinates, width, height) with a description of his/her movement type (inactive, active, walking, running) for a given situation (moving, inactive, browsing) and most importantly gives contextual information for the acted scenarios (browsing, immobile, left object, walking, drop down). In Figure 3.9 below, some examples of the acted scenarios in the CAVIAR dataset and the involved contextual objects are shown. From the CAVIAR dataset we have kept only the representative trajectory of the acted scenario (which we further call principal trajectory). Other non-related trajectories like supplementary movements or non-actor trajectories, which are thus not related to the acted scenario, are filtered out. In total, forty different activities can

62

L. Patino, F. Bremond, and M. Thonnat

Fig. 3.8 Different examples of trajectories added to a given experimental data set under study. Different colours in an experimental data set correspond to different semantics attributed to the trajectories (each trajectory is associated with only one semantic meaning). Although the semantics between trajectories may be different, their spatial similarity can be very close.

Fig. 3.9 Two different people trajectory types while going to look for information (Browsing) at two different places.

be distinguished. They include Browsing at different places of the scene, Walking (going through the hall) from different locations, and leaving or dropping an object at different locations of the hall. We have created the new evaluation set applying the following formulae: xi = N (α rx , x1 ) + N (β rx , xi )

(3.10)

yi = N (α ry , y1 ) + N (β ry , yi )

(3.11)

Where N (σu , u) is a random number from a normal distribution with mean u and standard deviation σu ; rx = |max (xi ) − min (xi )| is the range function on x; and α , β are two different constants to control the spread of the random functions. For each principal trajectory, we have generated 30 new synthetic trajectories by adding random noise as explained above. In total, the synthetic CAVIAR dataset contained

3

Incremental Learning on Trajectory Clustering

63

Fig. 3.10 Synthetic trajectories in the CAVIAR dataset generated from the activities shown in the previous figure. The trajectories are plotted employing their 3D coordinates on the ground.

1200 trajectories. Figure 3.10 shows the synthetic trajectories generated from the principal trajectories of the activities shown before. The CAVIAR synthetic evaluation data set was further divided into a Learning set (containing 1/3 of the trajectories in the synthetic evaluation data set, plus all of the principal trajectories) and Test set (containing the remaining 2/3 from the trajectories in the synthetic evaluation data set). The Learning set was employed to tune the ‘T’ threshold, which is critical to the clustering algorithm as indicated before. In this case, the tuning algorithm has fetched a ‘T’ value of T=1.05 and for which, the best clustering partition matches the ‘true’ data partition. When evaluating the algorithm in the Test set, the supervised Jaccard index used to compare the resulting partition with that ‘true’ expected partition gives a value of: Jaccard index=0.99. One supplementary evaluation set was created by adding more trajectories but which contain some spatial overlap to those already defined, yet they convey a different semantic (same procedure carried out in the first synthetic dataset). The structuring indexes ‘Silhouette’, ‘Dunn’ and ‘Davies-Bouldin’ reflect again the less distinct separation induced between trajectories with different semantic meanings. Table 3.2 summarises the results on both CAVIAR experimental datasets. Again, the same trend as for the synthetic COFRIEND dataset appears. When the confusion between semantics increases (thus the internal structure of the input data decreases), retrieving all initial semantic groups is more difficult and the mapping between the trajectory clustering results and the semantic groups also worsens (Jaccard index decreases). Table 3.2 Clustering results on CAVIAR synthetic datasets. Input Name

Output Nb. Trj.

ExpSet1 440 ExpSet2 740

Nb. GT’s 22 37

Sil

Nb. Jaccard Structure ROC measures Clusters Index Sil TPR FPR

0.79 0.65

22 26

Structure

0.79 0.60

0.99 0.56

0.99 0.69

0.0001 0.008

64

L. Patino, F. Bremond, and M. Thonnat

3.6 Results We have processed in total five video datasets corresponding to different monitoring instances of an aircraft in the airport docking area (in the following, these video datasets are to be named: cof1, cof2, cof3, cof4 and cof8). These correspond to about five hours of video which corresponds to about 8000 trajectories. The system was first tuned and initialised as previously described (i.e. employing a learning dataset with 230 trajectories distributed into 23 trajectory types). Figure 3.11 shows the online system learning as the different video sequences (datasets) are processed.

Fig. 3.11 Number of processed trajectories (blue curve) and number of trajectory clusters created by the online system as the different datasets are sequentially processed. Remark that the number of trajectory clusters does not increase much in relation to the number of trajectories analysed after the processing of the‘cof2’ dataset.

The structure of the resulting clustering after the processing of a given dataset can be measured again with unsupervised evaluation indexes (i.e. the intra-cluster homogeneity and inter-cluster separation) as in section 3.4 ‘Trajectory analysis’. We calculated the Silhouette index for the clustering partition induced on each analysed dataset. Table 3.3 below gives such results. When comparing these Silhouette values with those obtained in section ‘Trajectory analysis’ for the evaluation of the clustering algorithm, we can deduce that the analysed datasets contain still high levels of complexity/noise. Table 3.3 Clustering structure evaluated by the Silhouette index on the processed datasets. dataset Silhouette Index cof1 cof2 cof3 cof4 cof8

0.45 0.24 0.43 0.39 0.39

3

Incremental Learning on Trajectory Clustering

65

We employed the trajectory clusters to measure the similarity between the different datasets. For this purpose a histogram was built for each dataset where each bin of the histogram represents the number of mobile objects being associated with that particular trajectory cluster. The similarity between datasets comes down to measuring the similarity between the established histograms. For this purpose we employ the Kullback-Leibler divergence measure given next for any two different histograms h1 and h2: KL (h1, h2) = ∑ ph1 (r) log r

ph1 (r) ph2 (r) + ph2 (r) log ph2 (r) ∑ ph1 (r) r

(3.12)

and r is a given bin on the histogram of trajectories. Because the Kullback-Leibler divergence measure is a non-bounded measure, which equals to zero when h1=h2, we actually calculate the correlation (corr) between the different datasets by adding a normalisation factor and a unit offset: corr (h1, h2) = 1 +

KL (h1, h2) ∑r ph1 (r) log (ph1 (r)) + ∑r ph2 (r) log (ph2 (r))

(3.13)

The correlation between the different datasets is then given in the table below. Table 3.4 Trajectory-based correlation between the different analysed datasets.

cof1 cof2 cof3 cof4 cof8

cof1 1 0.77 0.75 0.79 0.76

cof2 0.77 1 0.78 0.80 0.78

cof3 0.75 0.78 1 0.76 0.71

cof4 0.79 0.80 0.76 1 0.81

cof8 0.76 0.78 0.71 0.81 1

From the trajectory-based correlation table it can then be observed that sequences cof2, cof4 and cof8 are the most similar, although in general all five sequences do contain a large number of common trajectories as their minimum correlation value is above 0.75.

3.7 Conclusions Activity clustering is one of the new trends in video understanding. Here we have presented an on-line learning approach for trajectory and activity learning. Previous state of the art has mainly focused on the recognition of activities mostly with the aim to label them as normal/suspicious/dangerous. However, the adaptation/update of the activity model with the analysis of long term periods has only been partially adressed. Moreover, most state of the art on activity analysis has been designed for the case of structured motions such as those observed in traffic monitoring (vehicle

66

L. Patino, F. Bremond, and M. Thonnat

going straight on the road, vehicle turning on a round-about, ...) or specific isolated body motions like walking, running, jumping. In this paper, we have adressed the problem of incremental learning of unstructured spatial motion patterns. The proposed algorithm allows monitoring and processing large periods of time (large amounts of data), and thus perform analysis on a long-term basis. The proposed approach employs a simple, yet advantageous incremental algorithm: The Leader algorithm. The strongest advantage is that it requires only a single scan of the data, and only cluster representatives need to be stored in the main memory. Generally, incremental approaches rely on a manually-selected threshold to decide whether the data is too far away from the clusters. To improve this approach we propose to control the learning rate with coefficients indicating when the cluster can be updated with new data. We solve the difficulty of tuning the system by employing a training set and machine learning. The system respects the main principles of incremental learning: The system learns new information from datasets that consecutively become available. The algorithm does not require access to previously used datasets, yet it is capable of largely retaining the previously acquired knowledge and has no problem of accommodating any new classes that are introduced in the new data. In terms of the studied application, the system has thus the capacity to create new clusters for new trajectories whose type had not been previously observed. Exhaustive evaluation is made on synthetic and real datasets employing unsupervised and supervised evaluation indexes. Results show the ability of trajectory clusters to characterise the scene activities. In this work we have adressed only the recognition of single mobiles appearing in the scene. In a future work, we will adress group-related activity such as ‘Meeting’ (trajectory merging) and ‘Splitting’. Our future work will also include more exhaustive analysis with temporal information, extracted from trajectories, to achieve more precise behaviour characterisation and to distinguish between mobiles moving at ‘walking’ speed or higher speed. We will also include normal/abnormal behaviour analysis from trajectory clustering.

Acknowledgements This work was partially funded by the EU FP7 project CO-FRIEND with grant no. 214975.

References [1] Cofriend, http://easaier.silogic.fr/co-friend/, http://easaier.silogic.fr/co-friend/ [2] Anjum, N., Cavallaro, A.: Single camera calibration for trajectory-based behavior analysi. In: IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 147–152. IEEE, Los Alamitos (2007) [3] Antonini, G., Thiran, J.: Counting pedestrians in video sequences using trajectory clustering. IEEE Transactions on Circuits and Systems for Video Technology 16, 1008– 1020 (2006)

3

Incremental Learning on Trajectory Clustering

67

[4] Avanzi, A., Bremond, F., Tornieri, C., Thonnat, M.: Design and assessment of an intelligent activity monitoring platform. EURASIP Journal on Advances in Signal Processing 2005, 2359–2374 (2005) [5] Bashir, F., Khokhar, A., Schonfeld, D.: Object trajectory-based activity classification and recognition using hidden markov models. IEEE Transactions on Image Processing 16, 1912–1919 (2007) [6] Campello, R., Hruschka, E.: A fuzzy extension of the silhouette width criterion for cluster analysis. Fuzzy Sets and Systems 157, 2858–2875 (2006) [7] Carpenter, G.A., Grossberg, S.: A self-organizing neural network for supervised learning, recognition, and prediction. IEEE Communications Magazine 30, 38–49 (1992) [8] Carpenter, G.A., Grossberg, S., Reynolds, J.: Artmap: supervised real-time learning and classification of nonstationary data by a self-organizing neural network. Neural Networks 4, 565–588 (1991) [9] Davies, D., Bouldin, D.: A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence 1, 224–227 (1979) [10] Dunn, J.: Well-separated clusters and optimal fuzzy partitions. Cybernetics and Systems 4, 95–104 (1974) [11] Foresti, G., Micheloni, C., Snidaro, L.: Event classification for automatic visual-based surveillance of parking lots. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 3, pp. 314–317. IEEE, Los Alamitos (2004) [12] Fusier, F., Valentin, V., Bremond, F., Thonnat, M., Borg, M., Thirde, D., Ferryman, J.: Video understanding for complex activity recognition. Machine Vision and Applications 18, 167–188 (2007) [13] Gaffney, S., Smyth, P.: Trajectory clustering with mixtures of regression models. In: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, California, United States (1999) [14] Hartigan, J.A.: Clustering algorithms. John Wiley & Sons, Inc., New York (1975) [15] Jantke, K.: Types of incremental learning. In: AAAI Symposium on Training Issues in Incremental Learning, pp. 23–25 (1993) [16] Kaufman, L., Rousseeuw, P.: Finding groups in data. An introduction to cluster analysis, New York (1990) [17] Lange, S., Grieser, G.: On the power of incremental learning. Theoretical Computer Science 288, 277–307 (2002) [18] Livny, M., Zhang, T., Ramakrishnan, R.: Birch: an efficient data clustering method for very large databases. In: ACM SIGMOD International Conference on Management of Data, Montreal, vol. 1, pp. 103–114 (1996) [19] Muhlbaier, M.D., Topalis, A., Polikar, R.: Learn++.nc: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes. IEEE Transactions on Neural Networks 20, 152–168 (2009) [20] Naftel, A., Khalid, S.: Classifying spatiotemporal object trajectories using unsupervised learning in the coefficient feature space. Multimedia Systems 12, 227–238 (2006) [21] Oliver, N., Rosario, B., Pentland, A.: A bayesian computer vision system for modeling human interactions. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 831–843 (2000) [22] Piciarelli, C., Foresti, G., Snidaro, L.: Trajectory clustering and its applications for video surveillance. In: Proceedings of IEEE Conference on Advanced Video and Signal Based Surveillance, 2005, pp. 40–45. IEEE, Los Alamitos (2005)

68

L. Patino, F. Bremond, and M. Thonnat

[23] Polikar, R., Upda, L., Upda, S., Honavar, V.: Learn++: an incremental learning algorithm for supervised neural networks. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) 31, 497–508 (2001) [24] Porikli, F.: Learning object trajectory patterns by spectral clustering. In: 2004 IEEE International Conference on Multimedia and Expo (ICME), vol. 2, pp. 1171–1174. IEEE, Los Alamitos (2004) [25] Seipone, T., Bullinaria, J.: Evolving neural networks that suffer minimal catastrophic forgetting. In: Modeling Language, Cognition and Action - Proceedings of the Ninth Neural Computation and Psychology Workshop, pp. 385–390. World Scientific Publishing Co. Pte. Ltd., Singapore (2005) [26] Sharma, A.: A note on batch and incremental learnability. Journal of Computer and System Sciences 56, 272–276 (1998) [27] Tan, P.N., Steinbach, M., Kumar, V.: Introduction to Data Mining. Addison-Wesley, Reading (2005) [28] Vu, V.-T., Br´emond, F., Thonnat, M.: Automatic video interpretation: A recognition algorithm for temporal scenarios based on pre-compiled scenario models. In: Crowley, J.L., Piater, J.H., Vincze, M., Paletta, L. (eds.) ICVS 2003. LNCS, vol. 2626, pp. 523– 533. Springer, Heidelberg (2003) [29] Vijaya, P.: Leaders subleaders: An efficient hierarchical clustering algorithm for large data sets. Pattern Recognition Letters 25, 505–513 (2004) [30] Zhou, Z., Chen, Z.: Hybrid decision tree. Knowledge-based systems 15, 515–528 (2002)

Appendix 1 Silhouette Index The Silhouette index is defined as follows: Consider a data object v j , j ∈ {1, 2, · · · , N}, belonging to cluster cli , i ∈ {1, 2, · · · , c}. This means that object v j is closer to the prototype of cluster cli than to any other prototype. Let the average distance of this object to all objects belonging to cluster cli be denoted by ai j . Also, let the average distance of this object to all objects belonging to another cluster i i = i be called di j . Finally let bi j be the minimum di j computed over i = 1, · · · , c which represents the dissimilarity of object j to its closest neighbouring cluster. The Silhouette index is then S=

1 N

bi j − ai j

N

∑ s j and s j = max (ai j , bi j )

j=1

Larger values of S correspond to a good clustering partition.

Dunn Index The Dunn index is defined as follows: Let cli and cli be two different clusters of the input dataset. Then, the diameter Δ of cli is defined as

Δ (cli ) = max

v j ,v j ∈cli



 d v j , v j

3

Incremental Learning on Trajectory Clustering

69

Let δ be the distance between cli and cli Then δ is defined as

δ (cli , cli ) =

max

v j ∈cli ,v j ∈cli



 d v j , v j

and, d (x, y) indicates the distance between points x and y. For any partition, the Dunn index is ⎧ ⎧ ⎫⎫ ⎨ ⎨ δ (cl , cl  ) ⎬⎬ i i D = min min and i, i ∈ {1, · · · , N} , i = i i ⎩ i ⎩ max (Δ (cli )) ⎭⎭ i

Larger values of D correspond to a good clustering partition.

Davies-Bouldin Index The Davies-Bouldin index is defined as follows: This index is a function of the ratio of the sum of within-cluster scatter to between-cluster separation. The scatter within cluster, cli , is computed as Si =

  1 v j − mi ∑ |cli | v j ∈cli

mi is the prototype for cluster cli . The distance δ between clusters cli and cli is defined as

δ (cli , cli ) = mi − mi  The Davies-Bouldin (DB) index is then defined as

DB =

1 N Si + Si  Rii ; i, i ∈ {1, · · · , N} , i = i and Rii = ∑ Ri with Ri = max N i=1 δ (cli , cli ) i,i

Low values of the DB index are associated with a proper clustering.

Appendix 2 Jaccard Index Consider C = CL1 , · · · ,CLm is a clustering structure of a data set X = {v1 , v2 , · · · , vn }; and P = {P1 , · · · , Ps } is a defined partition of the data . We refer to a pair of points (vi , v j ) from the data set using the following terms: • SS: if both points belong to the same cluster of the clustering structure C and to the same group of partition P. • SD: if points belong to the same cluster of C and to different groups of P.

70

L. Patino, F. Bremond, and M. Thonnat

• DS: if points belong to different clusters of C and to the same group of P. • DD: if both points belong to different clusters of C and to different groups of P. Assuming now that a, b, c and d are the number of SS, SD, DS and DD pairs respectively, then a + b + c + d = M which is the maximum number of all pairs in the data set (meaning, M = N (N − 1)/2 where N is the total number of points in the data set). Now we can define the Jaccard index (J) measuring the degree of similarity between C and P: J=

a a+b+c

Chapter 4

Highly Accurate Estimation of Pedestrian Speed Profiles from Video Sequences Panagiotis Sourtzinos, Dimitrios Makris, and Paolo Remagnino

Abstract. This paper presents a system that estimates accurately the speed of individual pedestrians with constant walking speed from monocular image sequences with people captured from aside view/camera. Such accurate estimations are needed to tune speed models for pedestrian simulation software. The system uses a combination of image segmentation and motion tracking to localize foot locations of pedestrians and convert them to ground plane speeds using camera calibration model.

4.1 Introduction Pedestrian simulation is increasingly being used in the design and optimization of public spaces such as transport terminals; sport, entertainment and leisure venues; shopping centers; commercial and public buildings; and venues for major international events such as the Olympics. To accurately simulate pedestrian behavior and crowd dynamics in such environments simulation tools must be calibrated and validated using precise real world data [1]. A key determinant of crowd behavior is the preferred walking speed of individuals within the crowd; this has been shown to vary by context and region [2] thus appropriate speed profiles are required for each study. While attempts have been made to automate the collection of pedestrian data for preferred walking speed [3] [4], the developers of pedestrian simulation software have found the precision to be insufficient and have adopted manual methods to extract pedestrian speeds from video footage. The most common approach is for a human operator to examine video sequences and manually mark position of pedestrians in each frame of the video. The paths of the tracked pedestrians are analyzed and the preferred speeds extracted. These manual techniques are time consuming, resource intensive and error prone; therefore there is demand for automated video analysis which can deliver a high degree of accuracy. In this work we present an automated Panagiotis Sourtzinos · Dimitrios Makris · Paolo Remagnino Digital Imaging Research Centre, Kingston University, UK

P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 71–81. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

72

P. Sourtzinos, D. Makris, and P. Remagnino

system for the accurate estimation of the preferred walking speed of pedestrians walking alone and unimpeded. Such a constraint comes from the requirement of a microscopic pedestrian simulation where each entity is modeled from an estimated speed profile. Thus the speed profile should be modeled by examining the speed of pedestrians walking without obstructions. The proposed system uses a combination of image segmentation and motion tracking methods to identify the foot locations of a pedestrian and convert them to a ground plane speed using camera de-projection and calibration models.

4.2 Background To automatically estimate the speed of a pedestrian, the object must be tracked in every frame. A trajectory is therefore extracted in terms of the sequence of their locations over time. Pedestrian tracking methods may be classified as motion segmentation tracking, pedestrian appearance detection tracking and feature based tracking. Motion segmentation aims to detect pedestrians as moving regions in image sequences. Motion segmentation techniques include temporal differencing, optical flow and background subtraction. In temporal differencing [5] moving regions are detected by pixel wise differences between two or three consecutive frames while optical flow techniques [6] use the flow of vectors of moving objects in order to perform motion segmentation. In background subtraction techniques motion is detected as the difference between the current image and the reference background as in [7], where an adaptive mixture of Gaussians is used to model the background. Alternatively, pedestrians may be detected using pedestrian appearance models, such as boosted edgelets body part detectors [8] and Histograms of Oriented Gradients HOG [9] that model the appearance of a person using statistical models. Pedestrian detections (blobs) extracted by either motion segmentation or appearance model methods are then temporally grouped into trajectories using methods such as the Kalman filter [10] or the particle filter [11] that associate detections across consecutive frames. In feature-based tracking a set of interest points of sufficient texture is selected and matched in successive frames and then they are clustered into pedestrian trajectories. For instance, Kanade-Lucas-Tomasi KLT features are used in [12] while in [13] Harris corners as features are employed. Although most of the above algorithms are sufficient for general pedestrian tracking they are not accurate enough for speed estimation. Accurate speed estimation requires pinpoint accuracy of the pedestrian location on the ground plane in real world coordinates. Even if camera calibration is used to project the tracking information on the ground plane accuracy will be affected by the accuracy of tracking and the uncertainty of the pedestrian height. Ismail, et al., [14] dealt with the same problem as in our work and they presented a system for automated pedestrian speed estimation. Camera calibration is performed by collecting linear field observations of entities appearing in the video images and the track of pedestrians calculated using the KLT feature tracker. However, their approach makes use of top-view scenarios, where the pedestrians are small in size. As a consequence small errors in image-based position estimation

4

Highly Accurate Estimation of Pedestrian Speed Profiles

73

may have a great impact on the speed estimation and cause significant errors. Thus in different scenarios such as looking at pedestrians sideways from a closer distance their approach will not be able to provide accurate results.

4.3 Methodology Accurate speed estimation of walking pedestrians, viewed from aside view/camera, is achieved by estimating the sequence of locations when a pedestrians foot touches the ground. Our system performs this detection by first applying motion detection through foreground-background separation on a video sequence, and then by tracking a foreground blob we roughly estimate the pedestrian location. Then, we locate the static foot of a pedestrian throughout the video frames. Finally, we calculate their ground plane speed, by projecting the spatiotemporal information of the pedestrian feet, using a camera calibration model. The following sections describe in more detail our methodology. An overview of our system is presented in Figure 4.1.

Fig. 4.1 Proposed Methodology.

4.3.1 Motion Detection and Tracking We perform background separation using the approach of KawTraKulPong and Bowden [7] (Figure 4.3 (b)). We track all blobs whose size are above a threshold by using a connected component tracking algorithm and mean shift particle filter as resolver for collisions between them [15][16]. Both motion detection and multi-target tracking methods are implemented in OpenCV [17].

4.3.2 Static Foot Localization We assume that the only moving objects in the video sequences are pedestrians who walk parallel to the image plane I, so every blob Bi (t) detected at frame t is

74

P. Sourtzinos, D. Makris, and P. Remagnino

considered to correspond to a pedestrian i. In order to measure the speed of a moving person we need to identify, localise and track a specific body part of this person throughout a video sequence. We choose to locate the static feet of a pedestrian, because we can directly derive ground plane positions that allow accurate speed estimations. Based on the approach of Bouchrika and Nixon [18], we locate the static foot of a pedestrian during his walking gait. When a pedestrian is viewed from the side they appear in an upright position. We assume that their foot will be located in the bottom part Li (t) (as it is defined in Figure 4.2) of the foreground blob that is associated with this pedestrian (Figure 4.3(c)). During walking one foot is static while the other foot steps forward, thus the accumulation of image corners located in the area of a static foot must be higher, while corners corresponding to the moving foot or other body parts are spread in larger areas due to their motion. Using the bottom part of the foreground blob as a mask (Figure 4.3(d)) for every

Fig. 4.2 The black rectangle is the blob boudning box, while the red rectangle defines the Li (t).

frame we select the corresponding area of the image plane from the original image and we apply a corner detection algorithm [19] (Figure 4.3(e)) for as long as the blob is visible. We create a 2D histogram map C i , which has the same size as the original image I, for every tracked human i. accumulates the corners cix,y (t), found at every frame t, within the lower part Li (t) of the blob Bi (t).

4

Highly Accurate Estimation of Pedestrian Speed Profiles i Cx,y =

 cix,y (t)=

75



i t cx,y (t)

i 1 if corner inIx,y at framet 0 otherwise

(4.1)

i Also for each pixel we create a set Tx,y , which stores the temporal information of the presence of the corners. That is: i i = t : if corner inIx,y at framet Tx,y

(4.2)

Fig. 4.3 (a) Part of the original frame, (b) foreground pixels, (c) tracked blob, (d) lower part of tracked blob that is used for foot localisation, (e) detected corners at the lower part of tracked blob.

However, it is difficult to identify local maxima in the histogram map due to the non smooth continuity of the brightness of the points in the image plane. Therefore, we calculate a proximity image P i for each pedestrian i by smoothing the histogram map using a combination of mean average and Gaussian filters of size N × N . The result of this process is illustrated in Figure 4.4(a). Then, we use the filter of equation 4.3 to force the peak location at the lower part of the foot. The size of N varies based on the video resolution and the expected size of the pedestrians. Finally we locate the local maxima (see Figure 4.4(b)) by scanning a window of size 4N × 4N .  1 if r < N/2 (4.3) Fr,c = −2 if r ≥ N/2 The set of peaks (local maxima points) contains estimates of the feet positions along with potential outliers. Assuming that a person is walking straight, the peaks which correspond to the feet must be located on a line. Therefore, we fit a line based on the peaks and discard any outlier peaks that their distance from the line is above

76

P. Sourtzinos, D. Makris, and P. Remagnino

a threshold ϑα (see Figure 4.4(c)). The set of the remaining peaks may contain multiple estimates of the same foot (e.g. because of high concentration of corners in the front and the back of the foot). This is because their distance (foot size) may be bigger than the local maximum scanning window size. We use the temporal i information of corners as recorded in Tx,y , in order to calculate the average frame at which a corner was detected. Peaks with a temporal difference of average frames of appearance lower than a threshold ϑβ are considered to belong to the same static foot and thus we group them together in a set Sf , where f is the number of foot hypothesis identified (see Figure 4.4(d)). Since each step of a pedestrian should be described by only one peak, we derive all the possible combinations of peaks Qn , where: n = |S1 | · |S2 | · . . . · |Sf | (4.4) To identify which combination describes best the walk of the pedestrian under examination, we exploit the assumption of constant speed, or equivalently that each step has similar distance from the next and the previous one (Figure 4.4(e)). That is which combination has the smallest variance in distance between steps:

k BQ n

k ), where argmin var(BQ n k = d(Qkn , Qk+1 n ), k = 1, 2, . . . , f − 1

(4.5)

d(a, b) is the Euclidean distance between a and b.

Fig. 4.4 (a) Proximity image, (b) local maxima, (c) discarding any outliers (lare circle) that do not fit the assumption of straight walking, (d) local maxima that seem to belong to the same foot are linked together, (e) final feet location estimations by minimizing the variance on step length.

4

Highly Accurate Estimation of Pedestrian Speed Profiles

77

4.3.3 Speed Estimation In order to identify the real world position of the peaks, we perform camera calibration using Tsai’s coplanar calibration method [20]. The dimensions of the floor tiles are known and we construct an artificial board (see Figure 4.5) based on the tiles position. Therefore, the corner locations of the checkered board are known both in the 2D image coordinates and in 3D real world coordinates and therefore the camera model is estimated using the Tsai method. The extracted camera model allows

Fig. 4.5 An artificially made checkered board on top of tiles is used to perform camera calibration.

direct convertion of the image-based coordinates to real world coordinates, as long as they are constrained on the ground plane. After identifying the real world coordinate position R for a static foot location, knowing the average frame ϕ of appearance of that location and by knowing the frame rate r of our video sequence we are able to estimate the speed Vik for each step of a pedestrian by using equation 4.6. Vik =

r · d(Rnk , Rnk+1 ) |ϕRkn − ϕRk+1 | n

(4.6)

Similarly we estimate the average speed of a pedestrian by considering the first and the last static feet locations detected. We are interested in pedestrians who move with constant speed. To identify those pedestrians we calculate the mean step speed V¯i for each one of them and we consider them as valid if they satisfy equation 4.7.  k ¯  |Vi − Vi | max < 0.15 (4.7) k V¯i

78

P. Sourtzinos, D. Makris, and P. Remagnino

4.4 Results Our dataset consists of 6 video sequences (720 by 576 pixels resolution) of pedestrians recorded in an underground station in Hong Kong (a frame is displayed in Figure 4.6). In order to produce our ground truth data we manually marked the position of the heel strike for each step of each pedestrian under consideration at the frame when the foot becomes static and inside the measurement area. In total we marked 502 pedestrians which moved on a straight line and we calculated their speed for each step. We used our approach to measure the speed of the pedestrians

Fig. 4.6 Sample frame from video sequence. The red line encloses the area within which measurements were estimated.

which were marked manually and we discarded those with 2 or less footsteps detected, since this implies that the tracking was not reliable enough to extract sufficient information. In total we estimated the feet locations of 398 people. From the 398 pedestrians using the ground truth data we discarded 10 with no constant speed. In order to evaluate our method we compare it with the speed estimate calculated from the localisation of the bounding box tracking. For fair comparison, we convert the results of bounding-box tracking to a sequence of steps. We achieve that by sampling uniformly the trajectory of the mid-lower point of the bounding box between the frames of the first and last step as identified by our approach. Using the constant speed constraint (eq. 4.7) we discarded 203 data using our method for calculating the static foot locations while for the bounding box tracking we discarded 154. The

4

Highly Accurate Estimation of Pedestrian Speed Profiles

79

high rate of not approved pedestrians is due to the noise generated when people walk close to one another, since our system will generate corners that do not belong to the object under observation. However our method managed to discard all 10 false positives while the bounding box tracking discarded 5 false positives. Since our objective is to estimate accurate speed profiles, discarding false negatives tracks is not an issue. What it really matters is to have very few (ideally zero) false positives and very accurate speed estimation of true positives. In Figure 4.7 we display the error

Fig. 4.7 Error rates.

Fig. 4.8 Speed profiles.

80

P. Sourtzinos, D. Makris, and P. Remagnino

rate of the two approaches. Using our approach 69% percent of all measured pedestrians have less than 5% speed error, from the ground truth measurements, while 97% of all pedestrians have less than 10% error. On the other hand using the bounding box tracker, at the same error rates we get 50% and 93% of the pedestrians. In Figure 4.8 we can see the estimated speed profiles. Our approach produces almost identical results as those in the ground truth. The Bhattacharyya distances between the speed profile distribution of the ground truth and our approach is 0.0262 while the distance of the ground truth from the bounding box tracker is 0.1568.

4.5 Conclusions In this paper we have presented an algorithm for accurate estimation of pedestrian speed. Our results have shown that the estimated speed profile is highly accurate. Our method may fail to track successfully some individuals, because of noise around the heel strike positions. Fortunately, the tracks are filtered out by the constant speed assumption. We will also investigate methods to calculate the speed profile of pedestrians who move perpendicular to the horizontal axis of the image plane.

References 1. Berrou, J.L., Beecham, J., Quaglia, P., Kagarlis, M.A., Gerodimos, A.: A calibration and validation of the legion simulation model using empirical data. In: Pedestrian and Evacuation Dynamics, pp. 167–181. Springer, Heidelberg (2005) 2. Lam, W., Cheung, C.: Pedestrian speed/flow relationships for walking facilities in hong kong. Journal of Transportation Engineering 126(4), 343–349 (2000) 3. Hoogendoorn, S., Daamen, W.: Pedestrian behavior at bottlenecks. Transportation Science 39(2), 147–159 (2005) 4. Willis, A., Kukla, R., Kerridge, J., Hine, J.: Laying the Foundations: The Use of Video Footage to Explore Pedestrian Dynamics in PEDFLOW, pp. 181–186 (2002) 5. Patil, R.S., Fujiyoshi, H., Lipton, A.J.: Moving target classification and tracking from real time video. In: Proceedings of the Image Understanding Workshop, pp. 129–136 (1998) 6. Meyer, D., Denzler, J.: Model based extraction of articulated objects in image sequences for gait analysis. In: Proceedings of the International Conference on Image Processing, pp. 78–81 (1997) 7. KaewTraKullPong, P., Bowden, R.: An improved adaptive background mixture model for realtime tracking with shadow detection. In: Proceedings of the European Workshop on Advanced Video Bases Surveillance Systems (2001) 8. Wu, B., Nevatia, R.: Detection and tracking of multiple, partially occluded humans by bayesian combination of edgelet based part detectors. International Journal of Computer Vision 75(2), 247–266 (2007) 9. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005) 10. Kalman, R.E.: A new approach to linear filtering and prediction problems. Transactions of the ASME - Journal of Basic Engineering 82(1), 35–45 (1960)

4

Highly Accurate Estimation of Pedestrian Speed Profiles

81

11. Isard, M., Blake, A.: CONDENSATION - conditional density propagation for visual tracking. International Journal of Computer Vision 29(1), 5–28 (1998) 12. Polana, R., Nelson, R.: Low level recognition of human motion. In: Proceedings of the IEEE Workshop Motion of Non-Rigid and Articulated Objects, pp. 77–82 (1994) 13. Perbet, F., Maki, A., Stenger, B.: Correlated probabilistic trajectories for pedestrian motion detection. In: Proceedngs of the IEEE International Conference on Computer Vision, pp. 1647–1654 (2009) 14. Ismail, K., Sayed, T., Saunier, N.: Automated collection of pedestrian data using computer vision techniques. In: Transportation Research Board Annual Meeting Compendium of Papers, Washington D.C (January 2009) Reference 09-1122 15. Comaniciu, D., Ramesh, V., Meer, P.: Real-time tracking of non-rigid objects using mean shift. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 142–149 (2000) 16. Nummiaro, K., Koller-Meier, E., Gool, L.V.: A color-based particle filter. In: Proceedings of the International Workshop on Generative-Model-Based Vision, pp. 53–60 (2002) 17. Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly Media, Sebastopol (2008) 18. Bouchrika, I., Nixon, M.S.: People detection and recognition using gait for automated visual surveillance. In: IET Conference on Crime and Security, pp. 576–581 (2006) 19. Shi, J., Tomasi, C.: Good features to track. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 593–600 (1994) 20. Tsai, R.Y.: A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation 3(4), 323–344 (1987)

Chapter 5

System-Wide Tracking of Individuals Christopher Madden and Massimo Piccardi

Abstract. Tracking the movements of people within large video surveillance systems is becoming increasingly important in the current security conscious environment. Such system-wide tracking is based on algorithms for tracking a person within a single camera, which typically operate by extracting features that describe the shape, appearance and motion of that person as they are observed in each video frame. These features can be extracted then matched across different cameras to obtain global tracks that span multiple cameras within the surveillance area. In this chapter, we combine a number of such features within a statistical framework to determine the probability of any two tracks being made by the same individual. Techniques are presented to improve the accuracy of the features. These include the application of spatial or temporal smoothing, the identification and removal of significant feature errors, as well as the mitigation of other potential error sources, such as illumination. The results of tracking using individual features and the combined system-wide tracks are presented based upon an analysis of people observed in real surveillance footage. These show that software operating on current camera technology can provide significant assistance to security operators in the system-wide tracking of individual people.

5.1 Introduction This chapter investigates the automated tracking of individual people within a set of cameras that defines a surveillance system. Such systems aim to locate the position Christopher Madden University of Adelaide, Australian Centre for Visual Technologies, Adelaide SA 5007, Australia e-mail: [email protected] http://www.acvt.com.au/research/surveillance/ Massimo Piccardi University of Technology, Sydney, Department of Computer Science, Sydney NSW, Australia e-mail: [email protected]

P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 83–103. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

84

C. Madden and M. Piccardi

of an individual person as they are observed over time, called a track, based upon a set of assumptions, which reflect the limitations of current surveillance technology. Whilst technologies such as RFID tags, security codes, or fingerprint scanners on every door may facilitate tracking, they are not widely available and could become an annoyance to constantly use. This work focuses upon CCTV surveillance cameras as they are the dominant technology that currently exists in the typical surveillance system to provide broad coverage. We specifically address building or campus surveillance as it allows for the modelling of an individual’s features, because there is a limited level of crowding compared to other institutions, such as train stations. Tracking individuals in heavily crowded scenes is such a difficult task that little research has been effective in this area [26]. Computer vision-based object tracking from surveillance cameras is based upon shape, motion, and appearance features of the tracked target [11]. These features are used to evaluate a set of observations to determine the targets location over time, or its ’track’. Motion features, such as location and velocity, are the primary feature used in tracking as they are less reliant upon camera quality, whilst low camera resolution reduces the ability of surveillance systems to exploit shape or appearance features effectively. Surveillance systems are created to assist human operators to view key locations around the surveillance area, at a minimum cost. They therefore consist of a relatively small number of low quality cameras sparsely located to cover key regions of the area under surveillance. Coverage of such key security locations is improved through the installation of additional cameras or other security devices, where they are considered cost effective. Minimisation of video footage size for storage requirements through image compression is problematic for automatic analysis, as are the changes in illumination conditions and colour responses for each camera in the system. Such low camera coverage, image compression and illumination conditions can be difficult for human operators to effectively manage, but are more problematic for automating aspects of the tracking process. Advances in camera technology is improving the affordability of cameras with higher resolutions, providing improved image quality where such cameras are installed. Increasing camera resolution provides improved information about the object being observed, including more reliable observations of shape and appearance features. Upgrading every camera in a system can be very costly, so most surveillance is still conducted on low resolution cameras. Motion features are extremely useful in single camera views, as well as groups of overlapping or near overlapping camera view, where there are strict restrictions on transitions between cameras [12, 27, 28]; however they are not reliable for tracking, or matching the tracks of individuals across large unobserved regions. Such regions occur within large gaps in camera coverage, where the movements of humans are not observed, and may not reflect average motion. Indeed the most important tracks may occur where an individual differs significantly from the average pattern as they attempt to avoid detection by performing illicit activities in blind spots. Shape and appearance features provide an ability to enhance feature models to improve the accuracy of matching individuals observed across multiple cameras. Through the combination of such matched tracks, a system-wide track of individuals is obtained.

5

System-Wide Tracking

85

Previous work on matching or combining the tracks of an individual across disjoint cameras has focused upon colour matching [3, 8, 13, 16] as a key feature for matching people, though some other biometrics such as gait and height estimates were also been explored [2, 3, 17]. Statistics on the transitions of objects between cameras has also been recently proposed [4, 27]. In this work we use the word ’track’ to refer to the information obtained from the uninterrupted location of a single individual as they are observed in a single camera over time. A track includes the indexes of the first and last frame of the track, the individuals segmented region and appearance in each frame, and other features that can be derived from this information, such as gait and shape. Individuals can be recognised based on their colour appearance, where individuals are wearing differing coloured clothes; however pose and illumination changes can be a major problem to the invariance of appearance features. Javed et al. [13] proposed to compensate for illumination variations by training their system to recognise sets of frequent illumination conditions in order to transform colours to a normalised colour set. This approach can not compensate for different illumination in different regions of the images, or for changes over time. Gandhi and Trivedi [8] present a cylindrical representation of the individual to obtain spatial colour information using multiple overlapping cameras. This is sensitive to the alignment of the cylindrical representation and articulated motion. Darrel et al. [3] propose the fusion of facial patterns with height and colour features, although typical surveillance cameras do not offer sufficient resolutoin for accurately measuring facial patterns. BenAbdekader et al. [2] present height estimation based upon converting the height of an individual object’s bounding box into a measurement using camera properties without a full camera calibration. Stride length and periodicity are also determined; however, they require a stable frame rate higher than twice the gait frequency, which may not always occur in surveillance systems. Model-based gait features such as those developed by Zhou et al. [29] may be used as a feature within the framework presented; however gait has yet been proven across multiple cameras [22]. Works by Zadjel et al. [27] and Monari et al. [4] explore the use of transitions between cameras as an additional feature to provide information about matching tracks, though they are based upon average statistics with small gaps between cameras, which does not reflect many existing surveillance systems. They also do not discuss the impact of individuals who might vary from the normal statistics in order to avoid detection. This chapter presents research into a framework that overcomes many of the existing limitations by fusing multiple features into a robust model extracted from the track of an individual target. This model can be used in a multi camera surveillance system to compare the tracks obtained from individuals and build up a set of locations that an individual target has been observed, a system-wide track of that individual. We define a surveillance session as ‘a portion of one day where people enter the surveillance area from a known set of entry points to perform their activities before leaving through known exit points’, where the entry and exit points are observed by a camera. This definition leads to the following simplifying assumptions about the surveillance area, and the people viewed within that area:

86

C. Madden and M. Piccardi

1. All entry and exit points of the surveillance area are in view of a surveillance camera, or multiple cameras. 2. Individuals are unlikely to change their clothing or footwear; hence, intrinsic shape and appearance features will remain relatively constant for the surveillance session. 3. Individuals are tracked accurately whilst they remain within the view of any of the system’s cameras. 4. Individuals are segmented from the background into a single region, or singly labelled group, but not necessarily accurately. 5. Individuals are often observed at a distance from a single surveillance camera without any other equipment available. Thus some biometric features, such as faces or fingerprints, are not generally available. 6. Where cameras are significantly distant, motion features may vary unpredictably between those cameras as individuals can move freely in such unobserved regions. 7. Illumination may vary between observable areas, though this will be limited by the nature of the area under surveillance. Whilst the above assumptions limit some difficulties inherent in the problem, they aim to reflect realistic surveillance environment. These assumptions hold for current tracking technology on video where traffic is sparse, though situations where they can be relaxed are discussed throughout the chapter. The more problematic assumptions relate to the reliability of object segmentation and object tracking. Some fragmentation of segmented and tracked objects does occur, especially when traffic becomes dense; however such errors tend to generate extra objects or broken tracks, which may be merged through matching within the framework. The assumptions also suggest motion features may be unreliable for regions where camera views do not overlap, so transitions between camera views, other than those with no other exit, are not included within the feature fusion framework presented in this chapter. Due to the articulated motion of people, few shape features other than height or gait are likely to remain stable during walking. Most appearance features are likely to remain stable within the extent of a surveillance session, although they may also be affected by articulation, motion direction and illumination changes. The framework for tracking via matching presented is therefore based upon the expandable Bayesian fusion of multiple features, based upon their reliability. Whilst exceptional cases may be easily constructed for any feature set to fail, the features proposed are designed to provide sufficient discrimination (at the ground truth level) for a majority of real cases. Where people may be difficult to discriminate between accurately by selecting the best matches, it can reduce the amount of manual footage revision that would be required for human operators to perform the task alone. This chapter first presents an exploration of object features that are utilised within a surveillance system. We include both shape and appearance features, as well as methods to mitigate the effect of illumination, and other general noise or errors in feature extraction. The framework for fusing these features to perform tracking via matching is presented in the context of the features that have been explored.

5

System-Wide Tracking

87

Results are presented for the individual features, and the fusion of multiple features to explore the impact of the selected features. The chapter concludes by examining the impact of the results and possible extensions to expand the framework.

5.2 Features Exploring features that can be used to distinguish between individuals is an important component of this research. Such features need to measure some stable differences between individuals tracked in surveillance cameras so they can be incorporated into a feature model. Such feature models can be combined and compared within a feature fusion framework to identify tracks that were created by the same individual, and hence build a system-wide track of that individual. It is important to consider the stability of features across multiple cameras spread across a large area, as this can create significant measurement noise. The assumptions in the previous section provide a guide to the conditions which are typical within surveillance, allowing for an initial investigation of features. They suggest that features which might provide a reliable identification of an individual, such as facial features [10, 23] or fingerprints [14], are not likely to be available with enough resolution in most locations, so other features to match individuals must be used. Non-biometric features are a trade-off for use within a multiple feature model as they are less discriminative between individuals, but tend to remain more invariant throughout the system. These include a range of features that are related to either shape, appearance, and location or movement. Of these features, location and movement have been the most widely used features in tracking [11]. This is because objects can be accurately observed at a location at a given point in time and followed whilst in view of that camera, or other overlapping cameras , until they exit the observed scene. Where cameras are near to overlapping, the movement of the individual can predict when they will enter the adjacent view [12, 27]; however where camera views are separated by large distances, the apparent movement of people becomes more erratic. Where there are direct transitions between cameras, such as a hallway observed at both ends, location can be used [27]. This is because a person can not move out of the corridor without being observed by the system, thus providing some location information when they are not observed. Care needs to be taken to handle cases where errors occur, as individuals may not be accurately observed or tracked on some occasions in some cameras. Where an individual may move outside an area without being observed, then knowing that they have entered that area is less useful. Indeed transition times between cameras become much less reliable as they grow larger because the uncertainty of individuals maintaining either their perceived motion, or some average motion dramatically diminishes. In such situations the reliable information is primarily about how far it is possible for them to move, as some transitions between cameras will be impossible given the time available. Due to the uncertainty of this features, we do not investigate such indirect transitions further.

88

C. Madden and M. Piccardi

Shape and appearance features are explored in detail in the subsequent subsections, along with methods to increase their robustness for errors. Examples are provided to demonstrate each of the features that are proposed with a focus upon individual features, as the feature fusion framework that combines the features to improve overall results is presented in Section 5.3.

5.2.1 Shape Features A variety of features can be extracted based upon the shape of an individual object observed in surveillance cameras , such as height or image moments of the object; however few of these are biometric in nature due to the articulation of human motion. As the resolution of camera views increases, shape feature accuracy may also increase, although the cost of camera upgrades will delay their widespread deployment. A person’s height [2, 17] and gait features [22, 29] have been proposed to provide a stable estimate from surveillance footage, which can be used to discriminate between individuals. The accuracy of gait features has been low from current systems, such as those elicited by the gait challenge of Sarkar et al. [22]. Factors such as different flooring, and even minor segmentation errors can lead to errors, especially when extracted from low resolution surveillance cameras, so this section will focus upon height features, and how they can be extracted from an existing surveillance system. The most common method of producing height estimates are from overlapping camera views, such as in [2]. This method produces a reliable estimate based upon the 3D position of the top of the head; however it is reliant upon multiple overlapping cameras, which are rare within surveillance systems. In [17] we demonstrated that height estimation can be achieved using the single camera views that dominate surveillance systems , although it requires camera calibration and reasonable segmentation of the individual to obtain useful measurements. Rather than using the intersection of the top of the head on two camera planes, this method uses a ground plane homography from the camera calibration to determine the real-world location of the feet. When an individual is walking, which is the usual case, the head is directly above the ground-plane line joining the feet, so the real-world location of the head can be determined from a combination of the head location in the image, and the real world location of the feet. The results of this method are shown per frame from 5 tracks of 2 individuals in Figure 5.1, and summarised in Table 5.1. In [20] we extended this idea by automatically extracting the key positions at the top of the head and a reasonably accurate estimation of the ground plane position. This provides a method to determine a height estimate for each frame of an individuals track; however gait effects will make this height estimate vary in a cyclical manner, which is sampled depending upon the frame rate. The automatic estimation of height from a single camera begins with a ground plane homography calculated by camera calibration [17]. This is used to convert the position of the feet from image coordinates, u,v, into real-world coordinates, x,y,z.

5

System-Wide Tracking

89

Fig. 5.1 Per frame height estimates calculated from manually identified points of 2 individuals observed in 5 tracks

Table 5.1 Statistical analysis of track heights from Figure 5.1 Track Average Height (cm) Standard Deviation Matching Tracks 1 170.64 1.63 2, 3 2 171.85 1.34 1, 3 3 171.23 1.73 1, 2 4 166.15 1.04 5 5 166.64 0.92 4

[2] uses the centre of the bottom line of the bounding box, but this can be improved by estimating the actual position of the feet using a k-curvature technique [7]. This method analyses the boundary of the object to find the high curvature regions of the object. As described in [20], high curvature at the bottom of the object can determine the feet location and thus estimate a ’bottom’ point b(u, v) in the image as shown in Figure 5.2. High curvature at the top of the object is likely to be the top of the head h(u, v). Using this method the two key image points can be used with the ground plane homography to estimate the height from a single image. Unlike the multiple camera height estimation method, using the monocular height estimate is more reliant upon segmentation, as effects like shadows or splitting the object can lead to significant errors in the height estimate from a frame. A robust method of combining the height estimate of each frame is needed to provide a reliable height estimate. In [17] a robust median was found to be the closest estimate over a track of the true height of that individual, which could then be compared with estimates from other tracks. It was suggested that the errors are likely to decrease as more observations are made at higher resolutions, either from

90

C. Madden and M. Piccardi

Fig. 5.2 Finding b(u, v) using two feet

improved cameras, or the object being closer to the camera. This is because each pixel error in location will lead to a smaller overall error. Given any two tracks to compare, their robust height estimates can then be tested for similarity. In [20], rather than comparing the robust height estimates of the two tracks, we proposed to directly compute a robust estimate of their differences. To this aim, we used the pairwise height differences, Hd, between each of the frames of two tracks under comparison. The height similarity sH is then given as: sH =

σ(Hd) μ(Hd)

(5.1)

where μ(Hd) and σ(Hd) are the average and standard deviation estimate of Hd respectively. Where there is an intrinsic difference in height of the two individuals, μ(Hd) will greatly exceed σ(Hd), which will be reflected as a low similarity. This statistical comparison will better reflect the measurement error that may occur in the height estimate. The following steps outline the height difference estimation process: 1. Determine the height estimate of the objects in each frame i of the track: a. The silhouette of the segemented object is analysed using a k-curvature technique [7]. b. Areas near the bottom of the object with high curvature k are then used to determine where the feet are positioned, and thus extract a midpoint at the bottom of the object b(u,v). c. This point is converted into world ground plane coordinates b(x,y,z) to determine location. d. This location is then used with the image plane position of the top of the head h(u,v) to estimate the height of the person in this frame, Hfi . 2. The pairwise differences, Hd, between the height in each frame, Hfi , and every other frame are computed between the two tracks under comparison. 3. The estimated height differences between the object tracks are statistically analysed to determine sH in (5.1).

5

System-Wide Tracking

91

5.2.2 Appearance Features The appearance of an object can be studied using a wide range of features, of which the most commonly used are facial appearance, and colour. Analysis of facial appearance, often called face recognition, is a large and growing field that aims to identify individuals based upon differences in their facial structures in images. Whilst there are many advances in the field to account for variations in aspects like scene illumination or face rotation, the most reliable results tend to come from systems with multiple cameras and images where the face occupies most of the image [24], though often results are inaccurate where large numbers of people are observed. [10] proposes a method to use PTZ cameras to obtain high resolution facial images of people observed in parts of their surveillance system, such cameras are not often installed for this purpose. PTZ cameras are typically used on a predefined guard tour to observe a large outdoor area, and are not available to obtain facial information. Facial features are therefore likely only to be available at specific locations, and will not provide a generally useful feature from surveillance footage. A variety of colour features can be used to describe the appearance of individual people [9]. The fundamental issues are the colour representation to use and the spatial domain over which to compute it. At one extreme, one could represent the appearance of an individual simply as the colour map of its segmented region, or blob; however, due to the continuous deformation of the persons shape, the comparison of two such representations would be problematic as they are not invariant to pose. At the other extreme, one could disregard completely the spatial information about colours within the blob and just collect overall colour statistics. While this representation would be highly invariant to deformation, it cannot discriminate, for instance, between a person with a red blouse and black skirt and one with a black shirt and red pants. A practical trade-off is therefore needed between invariance and discrimination, such as in [25]. Additional issues for consideration are invariance to changes in illumination, computational efficiency, and memory efficiency. Appearance features need to trade-off these issues whilst preserving enough information to discriminate between individual people [16]. Many people have proposed colour models for analysing an individual’s colour appearance [3, 8, 13, 15, 4, 27], though many of these provided simplistic contractions of the colour space, limiting the individual colour variation available for discriminating between individual people. In [16] we proposed a sparse histogram model based upon [15], which stores colours that are clustered in a normalised 3D RGB colour space. This requires no colour space changes and limits the memory requirements by only storing clusters of colours that occur in the object. Using the normalised colour space changes reduces the impact of high illumination conditions, whilst still capturing some intrinsic changes when the light is at lower levels. An additional step of ’controlled equalisation’, as outlined in the following section, is also used to mitigate the impact of colour effects such as illumination changes and camera colour response [16], which is detailed further in the following section. The advantage of using histograms over lower order statistics, such as Gaussians or Gaussian mixtures, is that they are non-parametric and thus adaptable to a wide

92

C. Madden and M. Piccardi

variety of clothing, as shown in Figure 5.3. The obvious disadvantages lie in the increased storage requirements and computational costs.

Fig. 5.3 Examples of red histograms from different people segmented with minor errors

To cater for spatial discrimination of colour appearance, in [20] we proposed two extra colour features relating to the upper and lower clothing colours of an individual, similar to those used in [4, 27], in addition to the global colours used in [3, 5, 16]. These features are chosen to represent the often different colours of the clothing on the upper torso, and those on the legs. The narrow spatial aspect of these features also allows for an analysis of the spatial positioning of an individuals colour appearance, ensuring that changes in the position of the colours can be detected and used for discrimination. The narrowness and positioning of both of the upper and lower sparse histograms regions also allow for them to remain constant under minor segmentation errors that will only have a minimal impact upon a person’s features, but are sensitive to large segmentation errors. Using the difference between

Fig. 5.4 Examples of upper and lower regions of segmented individuals

5

System-Wide Tracking

93

these features allows for the identification of single frames with large segmentation errors due to their difference from the rest of the frames within an individual targets track [19]. Changes in the features over extended periods of time are indicative of errors in the tracking process, such as where the wrong target has been included. Figure 5.5 shows the upper MCR feature region (enclosed between the two top lines) and the lower MCR feature region (between the two bottom lines). Positioning of such regions in the first and third frames in figure is regarded as acceptable. In the second frame, instead, the lower MCR feature region is significantly displaced; in this case, the sudden change in the lower MCR feature clearly indicates a very poorly segmented frame. In [19] we explored this use of change in features, as well as the change in size of the bounding box to identify frames with large segmentation errors, with the results are summarised in Table 5.2. The accuracy of the major segmentation error detection was as high as 84 percent with only 3 percent false alarms when a combination of features was used, indicating that whilst most of the erroneous frames were identified and discarded, the vast majority of reliable frames remained available. The low false alarm for segmentation errors is important as some of the tracks consisted of as few as 12 frames before the segmentation error removal process was applied.

Fig. 5.5 Example of upper and lower regions from three segmentations of an individual

Table 5.2 PD and PFA values of Bounding Box and MCR features for detecting segmentation errors Feature PD as % PFA as % Vertical Bounding Box changes 72 9 Global MCR 66 31 Upper MCR 53 11 Lower MCR 66 5 Fused MCR 84 3

Extracting the MCR for each of these three colour features utilises the same process for each feature, but analyses different spatial component of the appearance of the segmented object. The process for extracting the MCR’s is described in detail in [16], but summarises as: • A controlled equalisation step performs a data-dependent intensity transformation that spreads the histogram to mitigate illumination changes that may occur within surveillance environments.

94

C. Madden and M. Piccardi

• Online K-means clustering of pixels of similar colour within a normalised colour distance generates the MCR of each spatial region. • Once segmentation errors are removed, robust MCR features can be obtained over a small window of frames to improve robustness to articulated motion. The three colour features are: • The global MCR feature represents the colours of the whole segmented object without any spatial information. • The upper MCR feature represents the colour of the top portion of clothing. This corresponds to the region from 30 − 40 percent of the person from the top of the objects bounding box as shown in Figure 5.5. This narrow band was chosen to ensure that it avoids the inclusion of the head and hair of the object, as well as low necklines, but does not go so low to overlap with the leg area. • The lower MCR feature is aimed to represent the colour of the lower portion of clothing. This corresponds to the region from 65 − 80 percent of the object from the top of the objects bounding box as shown in Figure 5.5. This narrow band avoids the very bottom of the object which can be prone to shadows, or artifacts where the feet touch the ground. It also tries to avoid overlapping with the belt or upper torso area of the person.

5.2.3 Mitigating Illumination Effects The colour of an object in a camera view is not the intrinsic colour of the object itself, but rather a view-dependent measurement of the light reflected from the object, and the camera sensitivity to that light [6]. By recording the camera response to different wavelengths of light, the colour sensitivity can be estimated and exploited for model-based or empirical camera characterisation [1]; however illumination provides a more difficult challenge. Compensating for illumination changes are broadly classified by Finlayson et al. [6] into colour invariants, which seeks transformation of the colours that are illumination independent; or colour constancy, which seeks to estimate the illumination of the scene to extract the intrinsic colours of objects. Whilst accurate models of the illumination of the scene could extract the intrinsic colours of objects, the implementation of this technique is very difficult given that the illumination can change over time. In [18] we explored a number of transformation methods to mitigate the effect of illumination upon an objects appearance. These adaptations were local to the object, as the illumination changes on the 3D object surface may not form a large enough component of the image to be captured by global methods. Of the methods explored in [18] the controlled equalisation provided the lowest probability of missing correct matches, whilst full equalisation provided a slight improvement of overall errors. These methods were shown to reduce the overall errors of the appearance component of the track matching framework by over 30 percent. Histogram equalisation, also denoted here as full equalisation, aims to spread a given histogram across the entire bandwidth in order to equalise as far as possible the histogram values in the frequency domain. This operation is data-dependent and

5

System-Wide Tracking

95

inherently non-linear as shown in Figure 5.6; however it retains the rank order of the colours within the histogram. ’Controlled equalisation’ is based upon equalising a combination of the object pixels and an amount of pre-equalised pixels that is a proportion k of the object size. These pre-equalised pixels effectively ’control’ the amount of equalisation, such that the pixels are spread to a limited degree within the spectrum instead of being spread fully. Thus although an object becomes more matchable under a range of illumination conditions , it retains a higher degree of discrimination from objects of differing intrinsic colour. This technique is demonstrated on the 256 bin red colour channel at varying levels of k in Figure 5.6 below, where the vertical axis represents the percentage of pixels occurring in any bin, and k = 0 denoting full equalisation.

Fig. 5.6 Controlled Equalisation of the individuals pixels with varying k values

This equalisation can be formally described by designating the set of N pixels in a generic object as A, and calling B a second set of kN pixels which are perfectly equalised in their R, G, and B components. Note that the parameter k designates the proportionality of the amount of equalised pixels to the amount of pixels in A. From their union A ∪ B, the cumulative histograms of the R, G, and B components, pr (i) , pg (i), and pr (i) for i = 0 . . . 255 are computed. A histogram equalisation of the individual colour channels is then derived as shown in Equations 5.2-5.4: Tr (i) =

i  255 pr (j) (1 + k) N j=0

(5.2)

96

C. Madden and M. Piccardi

Tg (i) =

i  255 pg (j) (1 + k) N j=0

(5.3)

Tb (i) =

i  255 pb (j) (1 + k) N j=0

(5.4)

These intensity transforms can then be applied to re-map the R, G, and B components in the object’s pixels providing the ’controlled equalisation’. The parameter k controls the amount of pre-equalised pixels used, which in turn controls the spread of the object histogram. In [18] a number of illumination mitigation techniques were compared, with the results for the unmitigated data, and the controlled equalisation technique summarised in Table 5.3. These results are based upon the similarity values obtained from 50 matching and 70 non-matching track pairs, and compare their estimated probablity of false alarm, P F A, probability of missed detection, P M D, and their total error rate. Table 5.3 Results of similarity measurements for matching and non-matching tracks Method Parameter None Equal Equal Equal Equal

Full 0.5 1 2

Matching mean std 0.8498 0.1013 0.9095 0.0452 0.9080 0.0423 0.9116 0.0348 0.9135 0.0378

Non-Matching mean std 0.2230 0.3467 0.2522 0.3791 0.2637 0.3939 0.2680 0.3996 0.2726 0.4060

Theoretical Error % PMD PFA total 2.32 11.01 13.33 0.60 7.57 8.17 0.59 8.60 9.20 0.46 8.32 8.78 0.69 9.37 10.07

The results in Table 5.3 show that for all values of k investigated, there is a reduction in appearance matching errors within the track matching framework. More importantly the variation of similarity measurements for matching tracks is reduced, even as the mean is increased, but the mean and variation of the non-matching similarity measurements are not affected. This leads to the demonstrated reduction in both the probability of missed detection, and false alarms.

5.3 Feature Fusion Framework A framework is required to allow for the comparison of models generated from tracks within the surveillance system. In the previous sections details were provided upon how to compare the similarity of shape and appearance features, but it is important to note that such similarities are not always directly comparable. In [20] we proposed the use of Bayes theorem for classification to obtain the desired posteriors from the trained likelihoods to form a framework for fusing features. Bayes theorem utilises posterior probabilities of features being matched or non matched which are derived from trained likelihoods. These probabilities are derived by using a small but

5

System-Wide Tracking

97

representative sample set of known tracks from individual people that are known to be either matching H0 or non-matching H1, with the probability distributions being comparable between features. Fusing together the track similarities of each feature occurs as shown in (5.5) , (5.6), where sH is the height similarity, sUC relates to the upper clothing colour, sLC relates to the lower clothing colour and sGC relates to the global colour. A bias term B allows for the results to be adjusted to favour either matching H0 or non-matching H1 depending upon the cost of missed detections or false alarms within the application. This method also allows for the extension of the feature framework by adding extra terms to (5.5) , (5.6) relating to the H0 and H1 in a similar manner. P (H0|sH , sUC , sLC , sGC ) = B (P (sH |H0) P (sUC |H0) P (sLC |H0) P (sGC |H0))

(5.5)

P (H1|sH , sUC , sLC , sGC ) = P (sH |H1) P (sGC |H1) P (sLC |H1) P (sGC |H1)

(5.6)

The classification of the track pair is provided by the maximum probability between hypotheses H0 or H1. The fusion scheme holds for additional features where they are available, provided they can be treated as mutually independent. In addition, an indication of the information of a given feature can be determined by calculating equations (5.5) , (5.6) with and without the given feature. Where a feature compliments the other features, an improvement should be noted in the framework accuracy. Using this method, the global appearance features sGC were found to provide little additional information than that provided by the upper appearance sUC and lower appearance sLC features. This Bayesian feature fusion framework is based upon track level model similarities, but some features are available for comparison at other levels, such as the frame level. Features such as appearance vary little from frame to frame, and therefore could be compared on a frame by frame basis, whilst other cyclic features, such as height estimate, are more accurately estimated over the whole track duration. Such frame level comparisons could be combined together to form a track based similarity measure using a simple averaging process, but the application of robust statistical analysis for outlier removal, such as proposed in [21], should improve their accuracy.

5.4 Results The results presented here report track matching accuracy based on each feature, and for the fused case based upon a comparison of 26 tracks from four people across two cameras, giving over 300 possible comparison combinations. Of these, 60 comparison combinations are used as training data for the fusion framework, with the remaining used for testing. An indication of the clothing colour and good

98

C. Madden and M. Piccardi

Fig. 5.7 Four people of interest and good automatically segmented masks

segmentation examples for the four individuals is given in Figure 5.7, where it is easy to see that the individuals are wearing clothing of approximately 50 percent or more differing colours. Ground-truth height differences between the individuals ranged from approximately 5 centimeters to 30 centimeters. These results are from individuals observed in video surveillance cameras with a resolution of 293 x 214 pixels at approximately 6 frames per second. The results are given in Figure 5.8 as an ROC curve of the independent and fused variables determined by varying the bias term B.

Fig. 5.8 ROC curves of the height, colour and fused feature results to be revised

Figure 5.8 demonstrates that the fusion of the chosen features can provide a probability of detection of 91 percent with only 5 percent false alarms at a chosen operating point. A detailed analysis of the results demonstrated that some cameras lead

5

System-Wide Tracking

99

to more accurate segmentation and feature measurements due to increased contrast between the individual and the background and more stable lighting conditions, indicating that the feature probability distributions could be better defined based upon the camera pairs within which the tracks occur; however this was not utilised for these experiments. Compensation of the effects of variable illumination on the object’s appearance was performed as outlined in section 5.2.3; however the application of colour calibration [1] may further improve the discriminative power of the colour features.

Fig. 5.9 ROC curves of the height, colour and fused feature results to be revised

Figure 5.9 provides a more detailed analysis of the complementary nature of the features on a subset of the track data available. It specifically looks at the information gain of the system when the global appearance feature sGC is applied. The ROC curve shows that the performance when height and global colours are fused does not improve significant, and even performs worse at some operating points. The fusion of height with all three colour features produces very similar performance to the system when just the upper and lower colour features are fused with height. This suggests that whilst the global colour feature will include some colour components from regions such as skin and hair colour, these additional components do not provide additional ability to distinguish between the people in this dataset. The maximum-a-posteriori classification of (5) and (6) based upon the knowledge that only 1 in 4 tracks are matching provides an indication of the bias term B, which minimises the total Bayesian error. We suggest that it would be preferable to work at a different operating point along the ROC curve in order to achieve a higher probability of detection, because a human operator can quickly and easily identify a falsely matched pair by doing a manual review of the potentially matched tracks

100

C. Madden and M. Piccardi

Fig. 5.10 Pictorial storyboard summary of potentially matching tracks

as the difference in appearance is likely to be visually obvious to the operator. Manually correcting a missed detection is more arduous as an operator would then need to manually compare the current track to all other possible tracks to determine the best match. Hence, we have opted to adjust B in (5) by a factor of three, achieving the results reported. Whilst this method relies upon good segmentation, the overall detection rates show that this method works well with automated detection, and removal of frames with segmentation errors, using existing surveillance cameras. The results of this fusion framework indicate which track pairs are likely to be matching, and can therefore be combined together to provide the system-wide track of an individual person. This similarity is very dependent upon the assumption that their clothing and footwear do not change during a surveillance session; however where such changes occur, it is likely that a human operator may be capable of rectifying. For such an operator to compare each track to all other observed tracks within a surveillance system would involve a time consuming search, which can be minimised by automatically suggesting the most likely matches, as shown in Figure 5.10. The number of comparison can be reduced by using the assumption that all people are observed entering and leaving the surveillance system, as this would provide a limit on when they were within the system. Errors on observations in these key locations will reduce the effectiveness of this measure. Such a visual storyboard verification process is likely to limit both the time required for investigations and generating system-wide tracks of individual people, as well as reducing the errors within such tracks; however this requires significant further work in implementation to evaluating the true effectiveness of such a system.

5.5 Conclusions System-wide tracking of individual people is seen as an increasingly important security task. Currently this task is performed by a set of security operators, who

5

System-Wide Tracking

101

monitor anywhere from tens to thousands of cameras to attempt to identify suspicious individuals or activities, or to analyse events after they have happened. A key component of operator activity is the difficult task of tracking the movements of individual people of interest as they move through the system either in real-time, or for post-event analysis. This chapter has explored features that can be extracted and utilised within a Bayesian framework to assist operators to combine the observations of an individual across many cameras, generating a system-wide track. This has explored a variety of motion, shape and appearance features, as well as methods to make their measurements more robust, in order to determine those features which might provide useful complementary object information. The results presented are based upon an implementation of framework using the robust feature estimates after error correction has been applied. This error correction occurs primarily though the identification and removal of frames with large segmentation errors, which distort estimated object features, though the identification of changes in those features. The results of analysing such feature changes demonstrated that the majority of frames with large segmentation errors were removed, with few false alarms. The results of the track matching based upon robust features showed that features can be chosen carefully for the information they provide. For instance fused spatial appearance is not improved when global colour is included; however height provides complementary information which improves the overall result. The surveillance dataset examined was captured across a small set of real surveillance cameras, with a correct matching rate for the 4 analysed individuals as high as 91%, with only 5% false matches. Subsequent experiments on the appearance of a larger dataset of 6 individuals observed across 4 cameras supports the appearance components of these results, though more experimentation is required to accurately evaluate its use in large surveillence systems. The results indicate that where the features chosen are used within the proposed framework, they can provide significant information to assist security operators. This assistance, through the automatic analysis of the surveillance footage, can provide useful visual suggestions to an operator based upon the tracks which are most likely to have occurred from a single individual. This can improve the efficiency of the task, as operators will no longer have to search the entire set of tracks to determine matches, but can quickly verify those accurately identified matching tracks to build up the set of tracks which define the system-wide track of an individual.

References [1] Barnard, K., Funt, B.: Camera characterization for color research. Color Research and Application 27(3), 153–164 (2002) [2] BenAbdekader, C., Cultler, R., Davis, L.: Person identification using automatic height and stride estimation. In: Proceedings of International Conference on Image Processing (2002) [3] Darrell, T., Gordon, G., Harveille, M., Woodfill, J.: Integrated person tracking using stereo, colour, and pattern detection. International Journal of Computer Vision 37(2), 175–185 (2000)

102

C. Madden and M. Piccardi

[4] Monari, E., Maerker, J., Kroschel, K.: A robust and efficient approach for human tracking in multi-camera systems. In: Proceedings of Advanced Video and Signal-based Surveillance (2009) [5] Erdem, C.E., Ernst, F., Redert, A., Hendriks, E.: Temporal stabilization of video object segmentation for 3d-tv applications. In: Proceedings of International Conference on Image Processing (2004) [6] Finlayson, G., Hordley, S., Schaefer, G., Tian, G.Y.: Illuminant and device invariant colour using histogram equalisation. Pattern Recognition 38(2), 179–190 (2005) [7] Freeman, H., Davis, L.: A corner-finding algorithm for chain-coded curves. IEEE Transactions on Computing 26, 297–303 (1997) [8] Gandhi, T., Trivedhi, M.: Panoramic appearance map (pam) for multi-camera based person re-identification. In: Advanced Video and Signal Based Surveillance (2006) [9] Gonzales, R., Woods, R.: Digital Image Processing. Prentice-Hall, Englewood Cliffs (2002) [10] Hampapur, A., Brown, L., Connell, J., Ekin, A., Haas, N., Lu, M., Merkl, H., Pankanti, S.: Smart video surveillance: Exploring the concept of multiscale spatiotemporal tracking. IEEE Signal Processing Magazine 22(2), 38–51 (2005) [11] Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man and Cybernetics 34, 334– 352 (2004) [12] Javed, O., Rasheed, Z., Shafique, K., Shah, M.: Tracking across multiple cameras with disjoint views. In: International Conference on Computer Vision (2003) [13] Javed, O., Shafique, K., Shah, M.: Appearance modeling for tracking in multiple nonoverlapping cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (2005) [14] Lee, H., Gaensslen, R.: Advances in Fingerprint Technology. CRC Press, Boca Raton (2001) [15] Li, L., Huang, W., Gu, I., Tian, K., Tian, Q.: Principal color representation for tracking persons. In: International Conference on Systems, Man, and Cybernetics, vol. 1, pp. 1007–1012 (2003) [16] Madden, C., Cheng, E., Piccardi, M.: Tracking people across disjoint camera views by an illumination-tolerant appearance representation. Machine Vision Applications 18, 233–247 (2007) [17] Madden, C., Piccardi, M.: Height measurement as a session-based biometric for people matching across disjoint camera views. In: Proceedings of Image and Vision Computing, New Zealand (2005) [18] Madden, C., Piccardi, M.: Comparison of techniques for mitigating illumination changes on human objects in video surveillance. In: International Symposium on Visual Computing (2007) [19] Madden, C., Piccardi, M.: Detecting major segmentation errors for a tracked person using colour feature analysis. In: Proceedings of International Conference on Image Analysis and Processing (2007) [20] Madden, C., Piccardi, M.: A framework for track matching across disjoint cameras using robust shape and appearance features. In: Advanced Video and Signal based Surveillance Conference (2007) [21] Mosteller, C.F., Tukey, J.W.: Data Analysis and Regression: A Second Course in Statistics. Addison-Wesley, Reading (1977)

5

System-Wide Tracking

103

[22] Sarkar, S., Phillips, P.J., Liu, Z., Vega, I.R., Grother, P., Bowyer, K.W.: The humanid gait challenge problem: Data sets, performance, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 162–177 (2005) [23] del Solar, J.R., Navarrete, P.: Eigenspace-based face recognition: a comparative study of different approaches. IEEE Transactions on Systems, Man and Cybernetics, Part C 35(3), 315–325 (2005) [24] Wechsler, H.: Reliable Face Recognition Methods System Design, Implementation and Evaluation. Springer, Heidelberg (2007) [25] Yang, Y., Harwood, D., Yoon, K., Davis, L.: Human appearance modeling for matching across video sequences. Machine Vision and Applications 18(3), 139–149 (2007) [26] Zhang, Z., Gunes, H., Piccardi, M.: Tracking people in crowds by a part matching approach. In: Proceedings of Advanced Video and Signal-based Surveillance (2008) [27] Zajdel, W., Krose, B.: A sequential algorithm for surveillance with non-overlapping cameras. International Journal of Pattern Recognition and Artifcial Intelligence 19(9), 977–996 (2005) [28] Zhao, T., Nevatia, R.: Tracking multiple humans in complex situations. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(9), 1208–1221 (2004) [29] Zhou, Z., Prugel-Bennet, A., Damper, D.R.I.: A bayesian framework for extracting human gait using strong prior knowledge. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(11), 1738–1752 (2006)

Chapter 6

A Scalable Approach Based on Normality Components for Intelligent Surveillance Javier Albusac, Jos´e J. Castro-Schez, David Vallejo, Luis Jim´enez-Linares, and Carlos Glez-Morcillo

Abstract. Since their first developments, traditional video surveillance systems have been designed to monitor environments. However, these systems have several limitations to automatically understand events and behaviours without human collaboration. In order to overcome this problem, intelligent surveillance systems arise as a possible solution. This kind of systems are not affected by negative factors such as fatigue or tiredness and they can be more effective than people when recognising certain kinds of events, such as the detection of suspicious or unattended objects. Intelligent surveillance refers to using Artificial Intelligence and Computer Vision techniques in order to improve traditional surveillance and process semantic information, obtained from low-level security devices. Normally these systems consist of a set of independent analysis modules that deal with particular problems, such as the trajectory analysis of pedestrian in parking lots, speed estimation of vehicles, gait or facial recognition, etc. However, most of them present a common problem: lack of flexibility and scalability to include new kinds of analysis and combine all of them in order to obtain a global interpretation. In this work, a formal model to define normal events and behaviours in monitored environments and to build scalable surveillance systems is presented. This model is based on the use of normality components, which are independent and reusable for environments with different characteristics and different kinds of objects. Each component specifies how an object should ideally behave according to a surveillance aspect, such as trajectory or velocity. The model also includes the fusion mechanisms required for combining the particular analysis made by each component. Finally, when a new component is designed making use of the proposed model, the system increases its abilities to detect new kind of abnormal events, and the normality of an object depends on a higher number of factors. Javier Albusac · Jos´e J. Castro-Schez · David Vallejo · Luis Jim´enez-Linares · Carlos Glez-Morcillo University of Castilla-La Mancha, Paseo de la Universidad 4, Ciudad Real (Spain) ZIP 13170 Fax: +34 926 295 354 E-mail: [email protected] P. Remagnino et al. (Eds.): Innovations in Defence Support Systems – 3, SCI 336, pp. 105–145. c Springer-Verlag Berlin Heidelberg 2011 springerlink.com 

106

J. Albusac et al.

6.1 Introduction Currently, the security of citizens and infrastructures in public and private environments is one of the main topics of interest for most governments and institutions [26]. One representative example is London, where more than four million cameras have been deployed and connected to security control centres [13], where human operators supervise the video streams received in order to detect anomalous situations that might represent a danger for people. The main disadvantage of this kind of surveillance is the huge dependence on human supervisors, which usually involves an efficiency decrease after observing monitors for long periods of time, as stated in previous research [29]. Furthermore, there exist surveillance activities, such as the detection of unattended objects, where the automatic systems can be more effective than human surveillance [13]. Therefore, traditional video surveillance has several limitations to meet the security demands of monitored environments. In other words, the security systems should be driven by intelligence and proactiveness. In this way, intelligent surveillance systems need to be developed in order to improve the robustness of traditional systems and reduce the workload of security staff [7], calling their attention only at particular moments. In the last two decades, the research community has proposed different algorithms, techniques and models to develop the named second and third generation of surveillance systems [31]. Second generation systems combine CCTV (closed-circuit television) technology and Computer Vision and Artificial Intelligence methods, while third generation surveillance systems provide solutions for highly distributed and complex environments. There exist different proposals of intelligent surveillance systems such as the analysis of trajectories of vehicles and pedestrians in parking lots, the access control in buildings, the interaction among human beings in underground railway stations, the activities carried out in indoor environments, etc. Most of these systems have normally been designed to solve particular problems in specific scenarios. They can be understood as a set of pieces that act separately to detect specific events in particular scenarios, without establishing relationships between them to get a global analysis. Moreover, these proposals are not usually reused in scenarios different from the one they were initially considered. Within this context, a proposal of a scalable and flexible general model for the development of surveillance systems is needed, which allows to deploy instances and is adaptable to most of monitored environments. This model should also facilitate the analysis of different events of interest depending on the surveillance requirements, increasing the capability of the system when new aspects need to be monitored. In short, an advance surveillance system should be able to include new analysis modules that can be customised for the scenario where they will be deployed, determining the normality or abnormality of such a scenario at every moment. The discussed limitations motivate the present work, where a formal model for the design and normality analysis of surveillance environments is proposed. This model is inspired by the component-based software development, the multi-layered design and the division of complex problems into subproblems. According to this model, an intelligent surveillance system consists of a set of normality components

6

Normality Components for Intelligent Surveillance

107

and a set of methods to combine their output. A normality component defines how an object should ideally behave according to the monitored events of interest, considering a non-normal situation as suspicious or anomalous. These components can be easily integrated into the surveillance system and customised depending on the requirements and security level desired. In addition, knowledge acquisition tools and machine learning techniques are used to configure and deploy the components. The output of the components designed by means of the proposed model are highly descriptive so that the security expert responsible for interacting with the surveillance system can easily check whether the system works correctly and identifies anomalous or suspicious situations. This feedback, close to natural language, facilitates the system maintenance and the reuse of knowledge in environments with similar features. On the other hand, the security systems should provide responses in a short time as possible to reduce the hypothetical damage caused by the occurrence of anomalous situations (e.g. crowd detection). This is the reason why the systems developed through the proposed model must provide a good balance between response time and robustness. The rest of the chapter is organised as follows: the next section discusses previous work and their shortcomings. Section 3 describes a formal model to build scalable and flexible surveillance systems. Section 4, the model previously described is applied to design a new normality component, which deals with the trajectory analysis problem. The experimental results are presented and discussed in Section 5. The final section presents conclusions and future works.

6.2 Previous Work In the last decade several research projects have been developed in order to achieve advances in the area of intelligent surveillance. Some of the problems addressed in these projects are moving object detection, tracking , object classification (usually between pedestrians and vehicles), automatic learning of activity models to understand events and behaviours, or support for decision making. One relevant research project was CROMATICA (Crowd Monitoring with Telematic and Communication Assistance) [18], conceived with the goal of improving the surveillance of passengers in public transport. This project combined video analysis-based technologies and wireless data transfer [31]. Such a project was later extended with PRISMATICA (Pro-active Integrated Systems for Security Management by Technological Institutional and Communication Assistance) [33], where the social, ethic, organisational and technical aspects of surveillance in public transport were especially taken into account [31]. Regarding the technical advances, the system had a distributed nature and allowed the communication between heterogeneous devices by means of the middleware CORBA. Some of the anomalous events detected by the system were people staying at forbidden areas, intrusions (access to restricted areas), anomalous length of queues, or people walking in the wrong direction. In these systems, the analysis and interpretation of events and behaviours is carried out in indoor environments and mainly focused on people.

108

J. Albusac et al.

Another relevant research project was VSAM (Video Surveillance and Monitoring) [12], funded by U.S. Defence Department, concretely by DARPA (Defence Advanced Research Projects Agency). The system used the video streams gathered by the security cameras to classify moving objects as people, groups of people or vehicles. The layer responsible for identifying events was designed to analyse vehicle trajectories and people movements. Thanks to the study of the people walk and their poses, the system was able to determine if a person was walking, running or performing other activities. W 4 [16] was a system focused on analysing people behaviour from their poses in outdoor environments, ignoring the behaviour of elements such as vehicles. The system detected and determined the people figure by using the images obtained by security cameras and the information captured by infrared sensors. Once obtained the figure, a second module identified relevant parts of the human body (head, trunk, arms, and legs). The position of these parts was used to identify events of interest such as people running, walking with a bag, carrying objects, putting their hands up (which might involve an external thread), etc. ADVISOR (Annotated Digital Video for Intelligent Surveillance and Optimised Retrieval) [28] was another European project developed with the goal of analysing people behaviour in subway stations. Like PRISMATICA, this system is composed of multiple cameras distributed around a station, connected to a central server that performs the image analysis. This module stores in a database the sequence of images that represent an anomalous situation. Each one of these sequences are described by means of textual information. Precisely, this information is later used by human operators to retrieve the previously identify abnormal events. AVITRACK [5, 8, 2], Co-Friend [30] and ARGOS [6] are more recent research projects. The first two are related to CROMATICA and PRISMATICA while CoFriend evolves from AVITRACK. Such projects aim to analyse people and vehicles behaviour at airports, especially in maintenance tasks such as load and unload of aeroplanes, refuel, single repairs, etc. On the other hand, ARGOS was conceived to deal with the control of sea traffic. Most of previously cited works are composed of multiple analysis modules designed in an independent way without following a common scheme. These systems can be understood as a set of pieces that act separately to detect specific events in particular scenarios, without establishing relationships between them to get a global analysis. Nevertheless, the tendency when designing intelligent surveillance systems should be towards the development of scalable systems that provide mechanisms to include new analysis modules. Furthermore, this integration of components should not affect the already deployed ones, making easy their combination to increase the ability for the system to perform a global analysis of the environment. Another existing limitation to overcome is the low flexibility when configuring the behaviour analysis applied to each class of object. A flexible surveillance system should give the possibility of easily specifying what analysis modules monitor each class of object, according to the characteristics of the environment and the surveillance demands. Thus, if a system is composed of two modules to analyse trajectories and speed, to choose which module is applied to monitor people and

6

Normality Components for Intelligent Surveillance

109

vehicles should be allowed, fact that may vary for different environment. For these reasons, and as stated in the previous section, the main motivation of this work is to define a model that allows to build scalable and flexible surveillance systems, which will be discussed in depth in the next sections.

6.3 Formal Model to Build Scalable and Flexible Surveillance Systems 6.3.1 General Overview The complexity of surveillance systems can be reduced by adopting a multi-layered approach in the design and development phases, providing the system with high flexibility and modularity and making easy the maintenance and integration of new functionalities. The low-level layers process the information gathered by the environment sensors and generate spatio-temporal data associated to each one of the recognised objects of interest. The intermediate layers analyse the identified events and behaviours. Finally, the high-level layers monitor the results obtained by the previous layers and give support to the decision making process when a critical situations was detected. In order to simplify the design and development of surveillance systems, the problem of monitoring an environment can be structured into a set of subproblems. This is possible since surveillance generally involves monitoring a scenario from different points of view, the elements that exist in such a scenario (e.g. moving or static objects), the aspects or events of interest to monitor (e.g. trajectories, objects speed, normality in a pedestrian crossing, etc) or even the physical scenarios that compose the whole environment (e.g. different rooms in an indoor environment). Another relevant aspect when reducing the complexity of the design of these systems is the time. For instance, the period of time in which the surveillance is performed determines the functionality of the system and should be taken into account when carrying out the design of the system. On the other hand, to reuse existing and well-tested designs and developments may facilitate the task of devising new surveillance systems. This idea leads us to the develop intelligent surveillance systems based on components that can be reused as customised modules that deal with different surveillance tasks. These components may be previously designed and developed by third parties and available in public or private repositories in order to use them when designing a new surveillance system. The adoption of this approach implies that each component has a set of tools that allow its instantiation for particular problems or environments. From a general point of view and according to the previously introduced ideas, the development of surveillance systems implies the analysis of the surveillance tasks carried out to address these problems. This analysis is made by means of the following questions:

110

J. Albusac et al.

• What objects or elements of the environments will be analysed? • What aspects or events of interest will be monitored in relation to the previous objects or elements? • What kind of sensors are available in the monitored environment? What new kind of sensors need to be deployed on the scenario to gather useful information to perform the surveillance? • When these events of interest may take place? Once these questions are answered, a study of the existing solutions that can be used or adapted must be carried out, taking into account the following issues: • What processing mechanisms are there to process the data gathered by the deployed sensors? The main goal of these processing elements is to transform and adapt these data to generate the information used by the components that perform the different surveillance tasks. • What existing components are there to carry out such tasks? If a component was already designed and developed to accomplish a surveillance task, it has to be customised for the environment where it will be deployed. On the contrary, the component must be designed and developed. In this way, a surveillance component is associated to a scenario element, to an aspect or event of interest to monitor, to a sub-environment, and to a particular time interval. Generally, these facts imply that the perceptual or low-level layer gets the information from the environment (e.g. objects of the scenario, their location, physical features such as size or orientation, time interval of the monitoring, etc). The intermediate layer processes this information and performs an analysis about the environment state to determine whether the studied situations are normal or abnormal. To do that, this layer makes use of a model that defines in a general way the normality of objects according to a particular aspect, generating the named normality components which are independent of any environment. Within this context, the most undesired situation that can take place under this approach is an anomalous situation that was not previously defined and considered by a surveillance system. However, the system is aware of something suspicious or abnormal is happening at that time. These normality components defined in a general way must be instantiated for particular environments. To instantiate a component under the approach discussed in this work implies customising it by means of knowledge acquisition tools and/or machine learning algorithms. The use of this last kind of algorithms is ad