Handbook of Ambient Intelligence and Smart Environments
Hideyuki Nakashima · Hamid Aghajan · Juan Carlos Augusto Editors
Handbook of Ambient Intelligence and Smart Environments
123
Editors Hideyuki Nakashima Future University Hakodate Kameda-Nakano 116-2 Hakodate, Hokkaido 041-8655 Japan
[email protected]
Hamid Aghajan Department of Electrical Engineering Stanford University 350 Serra Mall Stanford, CA 94305-9515 USA
[email protected]
Juan Carlos Augusto School of Computing & Mathematics University of Ulster at Jordanstown Shore Road, Newtownabbey, Co. Antrim UK BT37 0QB
[email protected]
ISBN 978-0-387-93807-3 e-ISBN 978-0-387-93808-0 DOI 10.1007/978-0-387-93808-0 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2009935679 c Springer Science+Business Media, LLC 2010 c Chapter 11 2009 Frank Stajano. Used with permission All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Contents
Part I Introduction Ambient Intelligence and Smart Environments: A State of the Art . . . . . Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Sensors, Vision, and Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Mobile and Pervasive Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Human-centered Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Artificial Intelligence and Robotics . . . . . . . . . . . . . . . . . . . . . . . . . 6 Multi-Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Societal Implications and Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Selected Research Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Perspectives of the Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 7 10 14 17 20 22 24 26 29 30 31
Part II Sensor, Vision and Networks A Survey of Distributed Computer Vision Algorithms . . . . . . . . . . . . . . . . Richard J. Radke 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Distributed Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Topology Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Non-overlapping Topology Estimation . . . . . . . . . . . . . . . 3.2 Overlapping Topology Estimation . . . . . . . . . . . . . . . . . . . 4 Camera Network Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Non-overlapping Camera Calibration . . . . . . . . . . . . . . . .
35 35 37 38 40 41 42 43
v
vi
Contents
4.2 Overlapping Camera Calibration . . . . . . . . . . . . . . . . . . . . 4.3 Improving Calibration Consistency . . . . . . . . . . . . . . . . . . 5 Tracking and Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44 45 47 49 50
Video-Based People Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcus A. Brubaker, Leonid Sigal and David J. Fleet 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Tracking as Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Generative Model for Human Pose . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Kinematic Parameterization . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Body Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Image Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Image Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 2D Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Background Subtraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Appearance Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Edges and Gradient Based Features . . . . . . . . . . . . . . . . . 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Motion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Joint Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Smoothness and Linear Dynamical Models . . . . . . . . . . . 4.3 Activity Specific Models . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Physics-based Motion Models . . . . . . . . . . . . . . . . . . . . . . 5 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Annealed Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Markov Chain Monte Carlo Filtering . . . . . . . . . . . . . . . . 6 Initialization and Failure Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction to Discriminative Methods for Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Discriminative Methods as Proposals for Inference . . . . 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
Locomotion Activities in Smart Environments . . . . . . . . . . . . . . . . . . . . . . Björn Gottfried 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Why Locomotion Matters . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Related Work on Motion Analysis and Smart Environments . . . . . 2.1 Motion Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Viewpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Wayfinding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57 58 60 60 61 62 62 63 64 65 66 68 68 69 69 70 72 73 74 75 78 79 80 82 82 83 89 89 89 90 90 91 91 92 92
Contents
2.5 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Locomotion Activities in Smart Environments . . . . . . . . . . . . . . . . 3.1 Smart Hospitals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Scenarios in a Smart Hospital . . . . . . . . . . . . . . . . . . . . . . 3.3 Characterising Locomotion Activities . . . . . . . . . . . . . . . . 4 The Representation of Locomotion Activities . . . . . . . . . . . . . . . . . 4.1 Spatiotemporal Information . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Functional Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Allocentric View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Locomotion Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Planning Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Wayfinding Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Searching Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Monitoring Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manish Kushwaha, Songhwai Oh, Isaac Amundson, Xenofon Koutsoukos, Akos Ledeczi 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Challenges and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Audio Beamforming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Video Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Time Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Synchronization Methodology . . . . . . . . . . . . . . . . . . . . . . 6.2 Evaluation of HSN Time Synchronization . . . . . . . . . . . . 6.3 Synchronization Service . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Multimodal Target Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Sequential Bayesian Estimation . . . . . . . . . . . . . . . . . . . . . 7.2 Sensor Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Multiple-Target Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Sequential Bayesian Estimation . . . . . . . . . . . . . . . . . . . . . 8.2 MCMCDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vii
93 94 94 94 96 98 101 101 102 104 104 105 106 106 108 109 110 111 112 117
117 119 121 123 125 128 128 129 130 131 132 134 136 137 137 141 144 145
viii
Contents
Multi-Camera Vision for Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Noriko Takemura and Hiroshi Ishiguro 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Camera Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Fixed Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Active Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Mixed System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Multi-modal System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Surveillance Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Location and Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Occlusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Real-time cooperative multi-target tracking by dense communication among Active Vision Agents . . . . . . . . . 3.2 Tracking Multiple Occluding People by Localizing on Multiple Scene Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Applying a Form of Memory-based Attention Control to Activity Recognition at a Subway Station . . . . . . . . . . 3.4 DMCtrac : Distributed Multi Camera Tracking . . . . . . . . 3.5 Abnormal behavior-detection using sequential syntactical classification in a network of clustered cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Industry Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
149 149 150 150 152 152 154 154 156 156 157 157 158 160 161
163 163 164 165 165
Part III Mobile and Pervasive Computing Collaboration Support for Mobile Users in Ubiquitous Environments . . . Babak A. Farshchian and Monica Divitini 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Characteristics of Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Collaboration and Shared Context . . . . . . . . . . . . . . . . . . . 2.2 Embodied Interactions and Artifacts as Resources . . . . . 2.3 Mobility of People and Resources . . . . . . . . . . . . . . . . . . . 2.4 Physical Distribution of People . . . . . . . . . . . . . . . . . . . . . 2.5 Flexibility and the Need for Tailoring . . . . . . . . . . . . . . . . 3 Existing Technology in Support of Ubiquitous Collaboration . . . . 4 Human Grid as a Unifying Concept for AmI and CSCW . . . . . . . . 4.1 The Example of UbiBuddy . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Shared Context in Human Grid . . . . . . . . . . . . . . . . . . . . . 4.3 Embodiment in Human Grid . . . . . . . . . . . . . . . . . . . . . . . 4.4 Mobility in Human Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Support for Physical Distribution in Human Grid . . . . . .
173 173 174 174 177 178 178 179 180 181 183 186 186 187 187
Contents
ix
4.6 Flexibility and Tailoring in a Human Grid . . . . . . . . . . . . Implementation of Human Grid: The UbiCollab Platform . . . . . . . 5.1 UbiNode Overall Architecture . . . . . . . . . . . . . . . . . . . . . . 5.2 Collaboration Instance Manager . . . . . . . . . . . . . . . . . . . . 5.3 Session Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Collaboration Space Manager . . . . . . . . . . . . . . . . . . . . . . 5.5 Service Domain Manager . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Resource Discovery Manager . . . . . . . . . . . . . . . . . . . . . . 5.7 Identity Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
187 188 189 190 192 192 193 193 194 195 196 196
Pervasive Computing Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gregor Schiele, Marcus Handte and Christian Becker 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Organizational Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Provided Level of Abstraction . . . . . . . . . . . . . . . . . . . . . . 2.3 Supported Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Spontaneous Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Ubiquitous Communication and Interaction . . . . . . . . . . . 3.2 Integration of Heterogeneous Devices . . . . . . . . . . . . . . . 3.3 Dynamic Mediation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Context Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Acquisition and Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Modeling and Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Provisioning and Access . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Application Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Inter-Application Adaptation . . . . . . . . . . . . . . . . . . . . . . . 5.2 Intra-Application Adaptation . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
201
5
Case Study of Middleware Infrastructure for Ambient Intelligence Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tatsuo Nakajima 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 High Level Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Middleware Infrastructure for Distributed Audio and Video Appliances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Middleware Infrastructure to Hide Complex Details in Underlying Infrastructures . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Middleware Infrastructure to Support Spontaneous Smart Object Integration . . . . . . . . . . . . . . . . . . . . . . . . . .
201 202 202 204 205 206 207 210 211 214 216 217 218 218 219 222 224 225 229 229 232 233 233 237 241
x
Contents
3.4 Middleware Infrastructure for Context Analysis . . . . . . . Middleware Design Issues for Ambient Intelligence . . . . . . . . . . . 4.1 High Level Abstraction and Middleware Design . . . . . . . 4.2 Interface v.s. Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Non Functional Properties . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Human Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
245 249 249 249 250 251 251 253 253
Collaborative Context Recognition for Mobile Devices . . . . . . . . . . . . . . . Pertti Huuskonen, Jani Mäntyjärvi and Ville Könönen 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Humans: Context-aware Animals . . . . . . . . . . . . . . . . . . . 2 Context From Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Mobile Context Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Context Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Application Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Context Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Distributed Context Awareness . . . . . . . . . . . . . . . . . . . . . 4 Making Rational Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Distributed Decision Making . . . . . . . . . . . . . . . . . . . . . . . 4.2 Voting Protocols as Distributed Decision Making Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Voting as CCR Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Case Study: Physical Activity Recognition . . . . . . . . . . . . . . . . . . . 5.1 Data Set and Test Settings . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Empirical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Context Recognition Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 The Way Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Energy Savings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Knowledge-based CCR? . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
257
Security Issues in Ubiquitous Computing . . . . . . . . . . . . . . . . . . . . . . . . . . Frank Stajano 1 Fundamental Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Ubiquitous (Pervasive, Sentient, Ambient. . . ) Computing 1.2 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Security Issues in Ubiquitous Computing . . . . . . . . . . . . . 2 Application Scenarios and Technical Security Contributions . . . . 2.1 Wearable Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Location Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 RFID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Authentication and Device Pairing . . . . . . . . . . . . . . . . . .
281
4
257 258 259 260 260 262 263 264 264 265 265 266 270 270 272 273 274 274 275 276 277
281 282 284 286 287 288 289 292 298
Contents
xi
2.5 Beyond Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Going Up a Level: the Rest of the Story . . . . . . . . . . . . . . . . . . . . . . 3.1 Security and Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Understanding People . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Motivations and Incentives . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
301 304 304 305 307 309
Pervasive Systems in Health Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Achilles Kameas, Ioannis Calemis 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Overview of Pervasive Healthcare Systems . . . . . . . . . . . . . . . . . . . 2.1 @HOME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 HEARTFAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 ALARM-NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 CAALYX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 TeleCARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 CHRONIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 MyHeart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 OLDES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 SAPHIRE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Reference Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Input-Output System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Local System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Remote Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 The HEARTS System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Local Devices/Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 HEARTS-OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 HEARTS Local Subsystem Manager (LSM) . . . . . . . . . . 4.4 Remote Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
315
3
315 316 316 317 318 319 319 319 320 321 322 322 323 326 330 333 334 335 340 341 343 344 344
Part IV Human-centered Interfaces Human-centered Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicu Sebe 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 What is Human-centered Computing? . . . . . . . . . . . . . . . . . . . . . . . 2.1 Gateways and Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Scope of HCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 HCC, HCI, CSCW, User-centered Design, and Human Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Areas of Human-centered Computing . . . . . . . . . . . . . . . . . . . . . . . 3.1 Multimedia Production . . . . . . . . . . . . . . . . . . . . . . . . . . . .
349 349 350 350 351 353 355 356 357
xii
Contents
3.2 Multimedia Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Multimedia Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Human Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Ubiquitous Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Users with Disabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Public and Private Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Integrating HCC into a Real (Human) World . . . . . . . . . . . . . . . . . 5.1 Integrating Modalities and Media . . . . . . . . . . . . . . . . . . . 5.2 Integrating Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Integrating Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Research Agenda for HCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
358 359 361 361 362 362 362 363 363 363 363 364 365 365 366 367 368
End-user Customisation of Intelligent Environments . . . . . . . . . . . . . . . . . Jeannette Chin, Victor Callaghan and Graham Clarke 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The Home of the Future – A Vision . . . . . . . . . . . . . . . . . . . . . . . . . 3 Contemporary Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Pre-programmed Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Agent-programmed Rules . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 User-programmed Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Supporting Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Pervasive interactive Programming (PiP) – An Example of End-User Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 PiP Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 PiP System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 How the System Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 The dComp Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 dComp Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 The dComp Ontology – An Overview . . . . . . . . . . . . . . . 6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 PiP Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 dComp Performance Evaluation . . . . . . . . . . . . . . . . . . . . 6.3 The PiP Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Concluding Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
371 371 372 373 374 375 376 377 377 378 379 381 383 387 387 388 394 394 394 396 398 401 403
Contents
Intelligent Interfaces to Empower People with Disabilities . . . . . . . . . . . . Margrit Betke 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The Camera Mouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Intelligent Devices for People with Motion Impairments . . . . . . . . 4 Camera-Based Music Therapy for People with Motion Impairments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Enabling Users to Create Their Own Smart Environments . . . . . . 6 Assistive Software for Users with Severe Motion Impairments . . . 7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Successful Design of Camera-based Interfaces for People with Disabilities Requires a User-oriented Approach to Computer Vision Research . . . . . . . . . . . . . . 7.2 Designing Interfaces for People with Disabilities – Research Efforts in Human-Computer Interaction . . . . . . 7.3 Progress in Assistive Software Development . . . . . . . . . . 7.4 The Importance of Computational Social Science to Intelligent Interface Design . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visual Attention, Speaking Activity, and Group Conversational Analysis in Multi-Sensor Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Gatica-Perez and Jean-Marc Odobez 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 On-line Regulation of Group Interaction . . . . . . . . . . . . . 2.2 Assistance of Remote Group Interaction . . . . . . . . . . . . . 3 Perceptual Components: Speaking Turns and Visual Attention . . . 3.1 Estimating Speaking Turns . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Estimating Visual Focus of Attention . . . . . . . . . . . . . . . . 4 From Speaking Activity and Visual Attention to Social Inference: Dominance Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Dominance in Social Psychology . . . . . . . . . . . . . . . . . . . 4.2 Dominance in Computing . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Dominance Estimation from Joint Visual Attention and Speaking Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using Multi-modal Sensing for Human Activity Modeling in the Real World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Beverly L. Harrison, Sunny Consolvo, and Tanzeem Choudhury 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Technologies for Tracking Human Actvities . . . . . . . . . . . . . . . . . . 2.1 Methods for Logging Physical Activity . . . . . . . . . . . . . . 2.2 The Mobile Sensing Platform (MSP) and UbiFit Garden Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xiii
409 409 411 411 414 415 415 420
420 421 422 423 424 433 433 434 434 436 438 438 440 450 451 452 452 455 457 463 463 464 464 465
xiv
Contents
2.3 User Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Usability, Adaptability, and Credibility . . . . . . . . . . . . . . . . . . . . . . 3.1 Usability of Mobile Inference Technology . . . . . . . . . . . . 3.2 Adaptability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Credibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
468 469 469 472 475 476 477
Recognizing Facial Expressions Automatically from Video . . . . . . . . . . . . Caifeng Shan and Ralph Braspenning 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Problem Space and State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Level of Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Static Versus Dynamic Expression . . . . . . . . . . . . . . . . . . 2.3 Facial Feature Extraction and Representation . . . . . . . . . 2.4 Expression Subspace Learning . . . . . . . . . . . . . . . . . . . . . 2.5 Facial Expression Classification . . . . . . . . . . . . . . . . . . . . 2.6 Spontaneous Versus Posed Expression . . . . . . . . . . . . . . . 2.7 Expression Intensity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Controlled Versus Uncontrolled Data Acquisition . . . . . . 2.9 Correlation with Bodily Expression . . . . . . . . . . . . . . . . . 2.10 Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Facial Expression Recognition with Discriminative Local Features 3.1 Local Binary Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Learning Discriminative LBP-Histogram Bins . . . . . . . . 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
479
3
Sharing Content and Experiences in Smart Environments . . . . . . . . . . . . Johan Plomp, Juhani Heinilä, Veikko Ikonen, Eija Kaasinen and Pasi Välkkynen 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 What is Experience, Can You Share It? . . . . . . . . . . . . . . . . . . . . . . 2.1 Defining Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Experience Design Modeling . . . . . . . . . . . . . . . . . . . . . . . 2.3 On Sharing Experiences . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Changing Role of the User in Content Provision (Prosumers) 4 Smart Environments and Experience Sharing Features . . . . . . . . . 4.1 From Ubiquitous Computing to Digital Ecologies . . . . . 4.2 Features for Sharing Content and Experiences . . . . . . . . . 5 Examples of Research in Sharing Content and Experiences . . . . . 5.1 Video Content Management in Candela . . . . . . . . . . . . . . 5.2 Context Aware Delivery and Interaction in Nomadic Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Kontti - Context Aware Services for Mobile Users . . . . .
479 480 480 482 483 487 487 489 490 490 492 494 495 495 496 497 502 503 509
509 510 510 510 512 513 514 514 515 518 518 520 521
Contents
5.4 m:Ciudad - A Metropolis of Ubiquitous Services . . . . . . 5.5 Sharing with My Peer - ExpeShare . . . . . . . . . . . . . . . . . . 5.6 Ubimedia Based on Memory Tags . . . . . . . . . . . . . . . . . . 6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . User Interfaces and HCI for Ambient Intelligence and Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas Butz 1 Input/Output Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Interactive Display Surfaces . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Tangible/Physical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Adapting Traditional Input Devices . . . . . . . . . . . . . . . . . 1.5 Multi-Device User Interfaces . . . . . . . . . . . . . . . . . . . . . . . 2 Humans Interacting with the Smart Environment . . . . . . . . . . . . . . 2.1 Metaphors and Conceptual Models . . . . . . . . . . . . . . . . . . 2.2 Physicality as a Particularly Strong Conceptual Model . . 2.3 Another Example: the Peephole Metaphor . . . . . . . . . . . . 2.4 Human-centric UIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Designing User Interfaces for Ambient Intelligence and Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 The User-centered Process . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Novel Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Evaluating User Interfaces for Ambient Intelligence and Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Adaptation of Existing Evaluation Techniques . . . . . . . . 4.2 Novel Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Applications in the ParcTab project . . . . . . . . . . . . . . . . . . 5.2 The MIT Intelligent Room . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Implicit interaction: the MediaCup . . . . . . . . . . . . . . . . . . 5.4 Tangible Interaction: The MediaBlocks . . . . . . . . . . . . . . 5.5 Multi Device interaction: Sony CSL Interactive Workspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Hybrid Interfaces: The PhotoHelix . . . . . . . . . . . . . . . . . . 5.7 Summary and Perspective . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multimodal Dialogue for Ambient Intelligence and Smart Environments Ramón López-Cózar and Zoraida Callejas 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Context Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Context Detection and Processing . . . . . . . . . . . . . . . . . . . 2.2 User Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xv
524 524 526 529 529 529 533 533 533 535 537 538 540 540 540 541 543 544 544 545 545 546 546 546 547 547 548 549 550 550 552 553 553 557 557 558 559 561
xvi
Contents
3
Handling of Input Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Abstracting the Environment . . . . . . . . . . . . . . . . . . . . . . . 3.2 Processing Multimodal Input . . . . . . . . . . . . . . . . . . . . . . . 4 Dialogue Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Interaction Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Confirmation strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Response Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Contextual Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Evaluation from the Perspective of Human-computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
562 562 563 564 566 567 567 569 569 570 571 572
Part V Artificial Intelligence and Robotics Smart Monitoring for Physical Infrastructures . . . . . . . . . . . . . . . . . . . . . . Florian Fuchs, Michael Berger, and Claudia Linnhoff-Popien 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Infrastructure Monitoring in the Context of Ambient Intelligence 2.1 State-of-the-Art in Infrastructure Monitoring . . . . . . . . . . 2.2 Context-Awareness in Ambient Intelligence . . . . . . . . . . 2.3 Knowledge Representation and Reasoning in the Semantic Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Pattern-based Modeling of Context Information and Infrastructure Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Key Terms and Concepts of Condition Monitoring . . . . . 3.2 Ontology Design Pattern for Infrastructure Conditions . . 4 Smart Monitoring Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Reasoning over Distributed Context Information . . . . . . . . . . . . . . 5.1 Answering Conjunctive Queries . . . . . . . . . . . . . . . . . . . . 5.2 Implementation as a Framework . . . . . . . . . . . . . . . . . . . . 6 European Railway Monitoring as a Case Study . . . . . . . . . . . . . . . . 7 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Spatio-Temporal Reasoning and Context Awareness . . . . . . . . . . . . . . . . . Hans W. Guesgen and Stephen Marsland 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Sensory Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Spatio-Temporal Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Symbolic Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Machine Learning Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Frequent Pattern Data Mining . . . . . . . . . . . . . . . . . . . . . . 6.2 Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
581 581 582 584 584 586 586 587 588 591 594 595 599 600 602 603 605 605 606 609 609 614 617 617 620
Contents
xvii
6.3 Novelty Detection and Habituation . . . . . . . . . . . . . . . . . . 6.4 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
622 625 626 627
From Autonomous Robots to Artificial Ecosystems . . . . . . . . . . . . . . . . . . Fulvio Mastrogiovanni, Antonio Sgorbissa, and Renato Zaccaria 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 State-of-the-art: Integrated Frameworks for Robots in Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Ambience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 PEIS Ecology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Ubibot: Sobots, Embots and Mobots . . . . . . . . . . . . . . . . . 2.4 The Artificial Ecosystem . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Application Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Cooperation and Competition for Transportation and Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Ambience, Ubibot and PEIS Ecology at Work . . . . . . . . . 4 Case Study: the Artificial Ecosystem Approach . . . . . . . . . . . . . . . 4.1 Basic Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Cooperation and Competition in the Reference Scenario 4.3 Distributed knowledge representation and reasoning . . . 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
631
Behavior Modeling for Detection, Identification, Prediction, and Reaction (DIPR) in AI Systems Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . Rachel E. Goshorn, Deborah E. Goshorn, Joshua L. Goshorn, and Lawrence A. Goshorn 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 AI and Behavior Modeling for DIPR Subsystems . . . . . . . . . . . . . . 2.1 DIPR “Detection” Subsystem . . . . . . . . . . . . . . . . . . . . . . 2.2 DIPR “Identification” Subsystem . . . . . . . . . . . . . . . . . . . 2.3 DIPR “Predict” Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 DIPR “Reaction” Subsystem . . . . . . . . . . . . . . . . . . . . . . . 3 Infrastructure for AI Systems Solutions . . . . . . . . . . . . . . . . . . . . . . 3.1 Physical Infrastructure for AI Systems Solutions . . . . . . 3.2 Overlay Architecture for AI Systems Solutions . . . . . . . . 4 Application Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Application Model (A): AI Decision Making . . . . . . . . . 4.2 Application Model (B): Abnormal Behavior Detection . 4.3 Application Model (C): Enabling “Smart Environments” 4.4 Application Model (D): Environment or “Ambient Intelligence” (Learning) . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Application Model (E): Turbocharging Detection Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
631 633 633 634 635 636 638 640 642 645 645 651 657 661 663 665
665 668 668 670 672 676 676 677 678 680 681 682 686 689 690
xviii
Contents
5
Advanced DIPR Systems Engineering . . . . . . . . . . . . . . . . . . . . . . . 5.1 Advanced Fusion Methods Using the DIPR System . . . . 5.2 Adaptive Learning in the DIPR System . . . . . . . . . . . . . . 6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
692 692 692 694 695
Part VI Multi-Agents Multi-Agent Social Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Itsuki Noda, Peter Stone, Tomohisa Yamashita and Koichi Kurumatani 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Issues on Multiagent Social Simulation . . . . . . . . . . . . . . . . . . . . . . 3 Autonomous Vehicles at Intersections . . . . . . . . . . . . . . . . . . . . . . . 3.1 The Intersection Management Protocol . . . . . . . . . . . . . . 3.2 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Evaluation of Dial-a-Ride System by Simulation . . . . . . . . . . . . . . 4.1 Problem Domain and Formalization . . . . . . . . . . . . . . . . . 4.2 Simulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 User Model Constructed Based on Sensor Network . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Integrated Exhibition Support System . . . . . . . . . . . . . . . 5.3 Proofing Experiments in Expo 2005 Aichi . . . . . . . . . . . . 5.4 Tour Trend Analysis and Its Application . . . . . . . . . . . . . 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Concluding Remark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multi-Agent Strategic Modeling in a Specific Environment . . . . . . . . . . . . Matjaz Gams and Andraz Bezek 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Overview of the Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Multi-Agent Strategy Discovering Algorithm . . . . . . . . . . . . . 3.1 Domain-Dependent Knowledge . . . . . . . . . . . . . . . . . . . . 3.2 Graphical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Symbolic Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Human Analysis of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 RoboCup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 3vs2 Keepaway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Advantages of Strategic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
699 699 700 701 703 706 708 710 711 711 712 713 715 715 715 716 720 721 724 724 724 727 727 728 731 732 733 736 738 740 740 742 744 745
Contents
xix
Learning Activity Models for Multiple Agents in a Smart Space . . . . . . . Aaron Crandall and Diane J. Cook 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Data Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Time Delta Enhanced Classification . . . . . . . . . . . . . . . . . 5.2 Markov Model Classification Accuracy . . . . . . . . . . . . . . 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
747
Mobile Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ichiro Satoh 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Mobility and Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Mobile Agent Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Remote Procedure Call . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Mobile Agent Languages . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Agent Execution Management . . . . . . . . . . . . . . . . . . . . . . 2.4 Inter-agent Communication . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Locating Mobile Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Mobile Agent Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Remote Information Retrieval . . . . . . . . . . . . . . . . . . . . . . 3.2 Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Mobile Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Active Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Active Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Ambient Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Mobile Agent-based Middleware for Ambient Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Context-aware Mobile Agent . . . . . . . . . . . . . . . . . . . . . . . 4.3 Personal-Assistant Agents . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
767
747 749 751 753 754 757 760 762 763
767 769 770 771 773 774 775 775 775 777 777 777 778 779 779 779 780 781 781 782 782 783 785 785
Part VII Applications Ambient Human-to-Human Communication . . . . . . . . . . . . . . . . . . . . . . . Aki Härmä 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The Long Call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Spatial Attributes Of Presence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
789 789 790 792
xx
Contents
4
Ambient Speech Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Ambient Telephone Architecture . . . . . . . . . . . . . . . . . . . . 4.2 Speech Capture And Enhancement . . . . . . . . . . . . . . . . . . 4.3 Speech Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Spatial Speech Reproduction . . . . . . . . . . . . . . . . . . . . . . . 5 Tracking And Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Calibration And Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Ambient Visual Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
795 796 797 802 802 807 807 809 810 811 811
Smart Environments for Occupancy Sensing and Services . . . . . . . . . . . . Susanna Pirttikangas, Yoshito Tobe, and Niwat Thepvilojanapong 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Detecting Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Infrared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Ultrasonic Sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 FM Radio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 WiFi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Ultra-Wideband . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Pressure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Estimation Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Location Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 WiFi Locationing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 GSM cells and GPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Bayes Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Particle Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 From Measurements to Meaning and Services . . . . . . . . . . . . . . . . 4.1 Location-Aware Reminders . . . . . . . . . . . . . . . . . . . . . . . . 4.2 A Scenario on Routine Learning . . . . . . . . . . . . . . . . . . . . 5 Platforms of smart environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 EasyLiving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Aware Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 PlaceLab House_n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Robotic Room . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
819
Smart Offices and Intelligent Decision Rooms . . . . . . . . . . . . . . . . . . . . . . . Carlos Ramos, Goreti Marreiros, Ricardo Santos, Carlos Filipe Freitas 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Smart Offices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Active Badge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Monica SmartOffice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
845
819 819 820 820 823 823 824 824 825 826 826 827 827 829 829 830 830 830 834 834 835 836 837 838 839
845 846 849 849
Contents
2.3 Stanford’s Interactive Worksapaces . . . . . . . . . . . . . . . . . . 2.4 Intelligent Environment Laboratory of IGD Rostock . . . 2.5 Sens-R-Us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Smart Doorplate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Ambient Agoras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Intelligent Meeting Rooms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 The IDIAP Smart Meeting Room . . . . . . . . . . . . . . . . . . . 3.2 SMaRT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 M4 and AMI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 NIST Smart Space Project . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 MIT Intelligent Room Project . . . . . . . . . . . . . . . . . . . . . . 4 LAID - Laboratory of Ambient Intelligence for Decision Support 4.1 Ubiquitous Group Decision Making . . . . . . . . . . . . . . . . . 4.2 Ubiquitous System Architecture . . . . . . . . . . . . . . . . . . . . 4.3 ABS4GD and WebMeeting Plus . . . . . . . . . . . . . . . . . . . . 4.4 Idea Generation Techniques . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Smart Classroom: Bringing Pervasive Computing into Distance Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuanchun Shi, Weijun Qin, Yue Suo, Xin Xiao 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Smart Classroom: Sensor-Rich Smart Environment . . . . . . . . . . . . 2.1 The Layout of Smart Classroom . . . . . . . . . . . . . . . . . . . . 2.2 A Typical User Experience Scenario in Smart Classroom 3 Key Multimodal Interface and Context-Awareness Technologies . 3.1 Location Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Direct Manipulation based on Laser Pointer Tracking . . 3.3 Speaker Tracking based on Microphone Array and Location Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Context-awareness in Smart Classroom . . . . . . . . . . . . . . 4 Large-Scale Real-Time Collaborative Distance Learning Supporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Totally Ordered Reliable Multicast . . . . . . . . . . . . . . . . . . 4.2 Adaptive Multimedia Transport Model . . . . . . . . . . . . . . . 4.3 SameView: Real-Time Interactive Distance Learning Application in Smart Classroom . . . . . . . . . . . . . . . . . . . . 5 Smart Platform: Multiagent-Based Infrastructure . . . . . . . . . . . . . . 5.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 The Features and Design Objectives . . . . . . . . . . . . . . . . . 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xxi
850 850 851 851 851 852 853 853 853 854 855 855 857 858 860 865 865 869 870 870 875 875 877 877 878 879 880 883 887 890 896 896 897 898 900 901 902 903 903
xxii
Contents
Ambient Intelligence in the City . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marc Böhlen and Hans Frei 1 Overview and Motivation and Limitations . . . . . . . . . . . . . . . . . . . . 2 Ambient intelligence and Urbanism – a Reality Check . . . . . . . . . 3 Ambient Intelligence and Philosophies of Technology . . . . . . . . . . 4 Out of Balance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Hydra’s New Heads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Information Cities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Ambient Agoras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Combat Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Amateur Ambient Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Slow Space Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Unusual Encounters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Mobile Devices Everywhere . . . . . . . . . . . . . . . . . . . . . . . 7 Ambient Intelligence for The People . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Knowledge from Many Minds . . . . . . . . . . . . . . . . . . . . . . 8 Remaking Public Space, with and in Spite of Ambient Intelligence 8.1 Paths of Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Staging City Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Paths of Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Personal Pollution Control . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Full Body Ambient Intelligence . . . . . . . . . . . . . . . . . . . . 9 AmI in the City, Revitalized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Advancement of World Digital Cities . . . . . . . . . . . . . . . . . . . . . . . . . . Mika Yasuoka, Toru Ishida and Alessandro Aurigi 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 American Community Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 European Digital Cities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Asian City Informatization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Advances in Information and Communication Technologies . . . . . 6 Technologies in Digital City Kyoto . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 The Information Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 The Interface Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 The Interaction Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
905 905 906 908 909 910 911 912 914 915 916 917 917 918 919 919 920 921 921 922 922 925 926 927 933 933 935 937 939 941 942 943 945 948 949 950 951
Contents
xxiii
Part VIII Societal Implications and Impact Human Factors Consideration for the Design of Collaborative Machine Assistants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sung Park, Arthur D. Fisk, and Wendy A. Rogers 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Collaborative Machine Assistants (CMAs) . . . . . . . . . . . . . . . . . . . 3 User Acceptance of CMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 What is Acceptance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 User Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Technology Characteristics . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Relational Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Design Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
955 955 956 957 957 958 961 963 969 971 972
Privacy Sensitive Surveillance for Assisted Living – A Smart Camera Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 979 Sven Fleck and Wolfgang Straßer 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 979 2 Classification of AAL & Video Surveillance Systems . . . . . . . . . . 981 3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 982 3.1 Related Work – Surveillance . . . . . . . . . . . . . . . . . . . . . . . 982 3.2 Related Work – Smart Cameras . . . . . . . . . . . . . . . . . . . . . 984 4 Identified Problems of Current Approaches and Resulting Impact 985 4.1 Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 985 5 The Proposed SmartSurv Approach for AAL . . . . . . . . . . . . . . . . . 987 5.1 The Smart Camera Paradigm . . . . . . . . . . . . . . . . . . . . . . . 987 5.2 SmartSurv 3D Surveillance Architecture . . . . . . . . . . . . . 987 5.3 Privacy Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 988 5.4 Smart Camera Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . 989 5.5 SafeSenior - The Installation at An Assisted Living Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 989 6 Smart Camera Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 991 6.1 Background Modeling and AutoInit . . . . . . . . . . . . . . . . . 991 6.2 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 992 6.3 2D → 3D Conversion Unit . . . . . . . . . . . . . . . . . . . . . . . . . 993 6.4 Handover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 993 6.5 Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 993 6.6 The Classification & Activity Recognition Flow . . . . . . . 995 7 Visualization Fully Decoupled from Sensor Layer . . . . . . . . . . . . . 997 7.1 3D Visualization Node - XGRT . . . . . . . . . . . . . . . . . . . . . 997 7.2 Google Earth as Visualization Node . . . . . . . . . . . . . . . . . 998 7.3 Web Application for Visualization . . . . . . . . . . . . . . . . . . 1000 7.4 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1001
xxiv
Contents
8
Conclusion and Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1003 8.1 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005 Data Mining for User Modeling and Personalization in Ubiquitous Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009 Alejandro Jaimes 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009 2 User Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1010 3 Data Mining & Knowledge Discovery (KDD) . . . . . . . . . . . . . . . . 1013 4 Context & Mobile User Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 1015 5 Data Mining in User Modeling & Applications . . . . . . . . . . . . . . . 1020 6 Human Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1021 7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1024 8 Conclusions & Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1027 Experience Research: a Methodology for Developing Human-centered Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1033 Boris de Ruyter and Emile Aarts 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1033 1.1 From Entertaining to Caring . . . . . . . . . . . . . . . . . . . . . . . 1034 1.2 From System Intelligence to Social Intelligence . . . . . . . 1035 2 Experience Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1037 2.1 Context Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1038 2.2 Lab Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1040 2.3 Field Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1041 3 ExperienceLab Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1042 3.1 HomeLab: the Home Environment . . . . . . . . . . . . . . . . . . 1042 3.2 ShopLab: the Retail Environment . . . . . . . . . . . . . . . . . . . 1043 3.3 CareLab: the Assisted Living Environment . . . . . . . . . . . 1044 4 The ExperienceLab as an Assessment Instrument . . . . . . . . . . . . . . 1045 4.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046 4.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046 4.3 Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047 4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1048 4.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1052 5 Case Study of the Experience Research Approach . . . . . . . . . . . . . 1053 5.1 Context-Mapping Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 1053 5.2 Laboratory Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1055 5.3 Field Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1057 6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1058 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1058
Contents
xxv
Part IX Projects Computers in the Human Interaction Loop . . . . . . . . . . . . . . . . . . . . . . . . . 1065 A. Waibel, R. Stiefelhagen, R. Carlson, J. Casas, J. Kleindienst, L. Lamel, O. Lanz, D. Mostefa, M. Omologo, F. Pianesi, L. Polymenakos, G. Potamianos, J. Soldatos, G. Sutschet, J. Terken 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1065 2 Audio-Visual Perceptual Technologies . . . . . . . . . . . . . . . . . . . . . . . 1067 2.1 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068 2.2 Person Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1070 2.3 Person Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1073 2.4 Interaction Cues: Gestures, Body Pose, Head Pose, Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074 2.5 Activity Analysis, Situation Modeling . . . . . . . . . . . . . . . 1076 2.6 Audio-Visual Output Technologies . . . . . . . . . . . . . . . . . . 1078 2.7 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078 2.8 Natural Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079 3 Technology Evaluations and Data Collection . . . . . . . . . . . . . . . . . 1080 3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1081 3.2 Corpus Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1084 4 Software Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1085 4.1 SitCom - The Situation Composer . . . . . . . . . . . . . . . . . . . 1087 4.2 Agent Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088 5 CHIL Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1090 5.1 The Memory Jog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1091 5.2 The Collaborative Workspace . . . . . . . . . . . . . . . . . . . . . . 1093 5.3 Managing the Social Interaction: The Relational Cockpit and the Relational Report . . . . . . . . . . . . . . . . . . . 1096 5.4 The Connector: A Virtual Secretary in Smart Offices . . . 1100 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1101 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1102 Eye-based Direct Interaction for Environmental Control in Heterogeneous Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1111 Fulvio Corno, Alastair Gale, Päivi Majaranta, and Kari-Jouko Räihä 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1111 2 The COGAIN network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1113 3 Domotic Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1114 3.1 Overview of Domotic Technologies . . . . . . . . . . . . . . . . . 1115 3.2 Architecture for Intelligent Domotic Environment . . . . . 1119 3.3 The DOG Domotic OSGi Gateway . . . . . . . . . . . . . . . . . . 1122 4 Eye-based Environmental Control . . . . . . . . . . . . . . . . . . . . . . . . . . 1125 4.1 Overview of Eye Based Environmental Control . . . . . . . 1125 4.2 The ART system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1127 4.3 Eye Based Environmental Control and Communication Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1129
xxvi
Contents
5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1130 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1130 Middleware Architecture for Ambient Intelligence in the Networked Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1133 Nikolaos Georgantas, Valerie Issarny, Sonia Ben Mokhtar, Yerom-David Bromberg, Sebastien Bianco, Graham Thomson, Pierre-Guillaume Raverdy, Aitor Urbieta and Roberto Speicys Cardoso 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1133 2 Achieving Interoperability in AmI Environments . . . . . . . . . . . . . . 1135 2.1 The Amigo Service Architecture for Interoperability . . . 1136 2.2 Service Discovery Interoperability . . . . . . . . . . . . . . . . . . 1137 2.3 Service Communication Interoperability . . . . . . . . . . . . . 1139 3 AmIi Interoperable Service Discovery . . . . . . . . . . . . . . . . . . . . . . . 1141 3.1 ASDM: A Model for Semantic and Syntactic Service Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1142 3.2 ASDL: A Language for Semantic and Syntactic Service Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1144 3.3 Interoperable Matching of Service Capabilities . . . . . . . . 1147 3.4 Ranking Heterogeneous Matching Results . . . . . . . . . . . . 1149 4 AmIi Interoperable Service Communication . . . . . . . . . . . . . . . . . . 1150 4.1 Interoperable Service Discovery and Communication . . 1150 4.2 RPC Communication Stack . . . . . . . . . . . . . . . . . . . . . . . . 1153 4.3 Event-based Interoperability . . . . . . . . . . . . . . . . . . . . . . . 1155 4.4 AmIi-COM Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1157 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1158 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1160 The PERSONA Service Platform for AAL Spaces . . . . . . . . . . . . . . . . . . . 1165 Mohammad-Reza Tazari and Francesco Furfari and Juan-Pablo Lázaro Ramos and Erina Ferro 1 An Introduction to the PERSONA Project . . . . . . . . . . . . . . . . . . . . 1165 2 Requirements on a Service Platform for AAL Spaces . . . . . . . . . . 1167 2.1 User Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1167 2.2 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 1169 3 Evaluation of Existing Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1170 4 The PERSONA Architectural Design . . . . . . . . . . . . . . . . . . . . . . . . 1173 4.1 The Abstract Physical Architecture . . . . . . . . . . . . . . . . . . 1173 4.2 The Interoperability Framework . . . . . . . . . . . . . . . . . . . . 1174 4.3 The PERSONA Platform Services . . . . . . . . . . . . . . . . . . 1177 4.4 The Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1181 4.5 The Ontological Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 1184 5 Sensing the environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185 5.1 Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . 1186 5.2 ZigBee Networks Integration . . . . . . . . . . . . . . . . . . . . . . . 1188 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1191 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1191
Contents
xxvii
ALADIN - a Magic Lamp for the Elderly? . . . . . . . . . . . . . . . . . . . . . . . . . 1195 Edith Maier and Guido Kempter 1 ALADIN - Magic Lighting for the Elderly? . . . . . . . . . . . . . . . . . . 1195 1.1 The Impact of Lighting on People’s Health and Wellbeing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196 1.2 Wellbeing - a Multifaceted Concept . . . . . . . . . . . . . . . . . 1197 2 ALADIN’s Journey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1198 2.1 Theoretical and Methodological Underpinnings . . . . . . . 1198 2.2 Analyzing User Requirements . . . . . . . . . . . . . . . . . . . . . . 1199 2.3 Tests in a Laboratory Setting . . . . . . . . . . . . . . . . . . . . . . . 1200 2.4 User Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1202 3 The Modern Version of the Enchanted Lamp . . . . . . . . . . . . . . . . . 1204 3.1 Technical Architecture - Overview . . . . . . . . . . . . . . . . . . 1205 3.2 Hardware Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1206 3.3 Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1209 4 Discussion of Field Test Results and Outlook . . . . . . . . . . . . . . . . . 1216 4.1 Findings from Field Trials . . . . . . . . . . . . . . . . . . . . . . . . . 1216 4.2 Outlook On ALADIN’s Future . . . . . . . . . . . . . . . . . . . . . 1217 4.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1219 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1220 Japanese Ubiquotous Network Project: Ubila . . . . . . . . . . . . . . . . . . . . . . . 1223 Masayoshi Ohashi 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1223 2 Ubila’s Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225 3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225 4 Various Ambient Service Designs and Prototypes . . . . . . . . . . . . . 1226 4.1 Basic Component Technology . . . . . . . . . . . . . . . . . . . . . . 1226 4.2 Pilot Applications and Services . . . . . . . . . . . . . . . . . . . . . 1227 4.3 Collection and Dissemination of User Profiles . . . . . . . . . 1230 5 Field Trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1234 5.1 Live Commerce Akiba . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1234 5.2 Temperature Monitoring with Airy Notes . . . . . . . . . . . . 1235 5.3 Lifelog Trial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235 6 Creation of Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1237 6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1237 6.2 First Video: Small Stories in 2008 . . . . . . . . . . . . . . . . . . . 1237 6.3 Second Video: Aura . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1238 7 Building and Operating Smart Spaces . . . . . . . . . . . . . . . . . . . . . . . 1239 7.1 Akihabara Smart Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1239 7.2 Yurakucho Smart Space (uPlatea) . . . . . . . . . . . . . . . . . . . 1239 7.3 Kitakyushu Smart Space . . . . . . . . . . . . . . . . . . . . . . . . . . . 1240 8 Network or Protocol Oriented R&D . . . . . . . . . . . . . . . . . . . . . . . . . 1240 8.1 Protocol Enhancement: OSNAP and CASTANET . . . . . 1241 8.2 Adaptive Network Control Based on Network Context . 1242 8.3 Automatic Network Configuration . . . . . . . . . . . . . . . . . . 1243
xxviii
Contents
9
Defining Context Aware System Architecture . . . . . . . . . . . . . . . . . 1243 9.1 Ubiquitous Network Architecture and Context . . . . . . . . 1243 9.2 Contributions to ITU-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1244 10 Symposium and Contributions to Academia and Fora . . . . . . . . . . 1244 11 Related Activities and Future Trends in Japan . . . . . . . . . . . . . . . . . 1246 11.1 National R&D Projects in Parallel to Ubila . . . . . . . . . . . 1246 11.2 Other Ubiquitous Computing Related R&D Projects in Japan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1246 11.3 Next Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1247 12 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1247 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1248
Ubiquitous Korea Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1251 Minkoo Kim, We Duke Cho, Jaeho Lee, Rae Woong Park, Hamid Mukhtar, and Ki-Hyung Kim 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1251 2 Ubiquitous Computing Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1252 2.1 Project Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1252 2.2 Vision: Ubiquitous Smart Space(USS) . . . . . . . . . . . . . . . 1252 2.3 Reference System Architecture and Deployment of USS 1252 2.4 R&D Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1252 2.5 Actual Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1253 2.6 Sub-projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1254 2.7 Test Bed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1256 3 Intelligent Service Robots in Ubiquitous Environment . . . . . . . . . 1256 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1256 3.2 Task and Knowledge Management Framework . . . . . . . . 1257 3.3 Task Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1259 3.4 Knowledge Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1259 3.5 Service Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1261 3.6 Results and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 1262 4 U-Health Projects in Korea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1263 4.1 Concepts of u-Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1263 4.2 Why u-Health? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1264 4.3 U-Health Projects in Korea . . . . . . . . . . . . . . . . . . . . . . . . 1266 4.4 Today’s Reality to Overcome . . . . . . . . . . . . . . . . . . . . . . . 1269 5 USN Technology for Nation-wide Monitoring . . . . . . . . . . . . . . . . 1269 5.1 IP-USN as an Integrating Technology . . . . . . . . . . . . . . . . 1270 5.2 Design Goals of WSN Management . . . . . . . . . . . . . . . . . 1273 5.3 LNMP Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1274 5.4 Nationwide Deployment in Korea . . . . . . . . . . . . . . . . . . . 1275 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1276 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1279
Part I
Introduction
Ambient Intelligence and Smart Environments: A State of the Art Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
1 Introduction Advances in the miniaturization of electronics is allowing computing devices with various capabilities and interfaces to become part of our daily life. Sensors, actuators, and processing units can now be purchased at very affordable prices. This technology can be networked and used with the coordination of highly intelligent software to understand the events and relevant context of a specific environment and to take sensible decisions in real-time or a posteriori. AmI is aligned with the concept of the “disappearing computer” [23, 24, 22]: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
It is particularly interesting to contrast this time in history when accomplishing Weiser’s aims sounds plausible with the situation five decades ago when computers had the size of a room and the idea of a computer being camouflaged with any environment was an unimaginable notion. Today different appliances have successfully become integrated to our daily life surroundings to such an extent that we use them without consciously thinking about them. Computing devices have transitioned in this past half a century from big mainframes to small chips that can be embedded in a variety of places. This has allowed various industries to silently distribute computing devices all around us, often without us even noticing, both in public spaces and in our more private surroundings. Now washing machines, heating systems and even toys come equipped with some capability for autonomous decision making. AutoJuan Carlos Augusto University of Ulster, UK Hideyuki Nakashima Future University Hakodate, Japan Hamid Aghajan Stanford University, USA H. Nakashima et al. (eds.), Handbook of Ambient Intelligence and Smart Environments, DOI 10.1007/978-0-387-93808-0_1, © Springer Science+Business Media, LLC 2010
3
4
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
motive industry has become increasingly interested in embedding sensing mechanisms that allow the car to make decisions for a safer and less expensive journey, and as a result cars now have dozens of sensor and actuator systems that make decisions on behalf of or to assist the driver. Public spaces have become increasingly occupied with tracking devices which enable a range of applications such as sensing shoplifted object and crowd behavior monitoring in a shopping mall. Whether it is our home anticipating when the lights should be turned on and off, when to wake us up or order our favorite food from the supermarket, a transport station facilitating commuting, or a hospital room helping to care for a patient, there are strong reasons to believe that our lives are going to be transformed in the next decades by the introduction of a wide range of devices which will equip many diverse environments with computing power. These computing devices will have to be coordinated by intelligent systems that integrate the resources available to provide an “intelligent environment”. This confluence of topics has led to the introduction of the area of “Ambient Intelligence” (AmI): “a digital environment that proactively, but sensibly, supports people in their daily lives.” [7]
Ambient Intelligence started to be used as a term to describe this type of developments about a decade ago (see for example [25, 1, 16, 3]) and it has now been adopted as a term to refer to a multidisciplinary area which embraces a variety of pre-existing fields of computer science as well as engineering, see Figure 1. Given the diversity of potential applications this relationship naturally extends to other areas of science like education, health and social care, entertainment, sports, transportation, etc. Whilst AmI nourishes from all those areas, it should not be confused with any of those in particular. Networks, sensors, human-computer interfaces, pervasive computing and Artificial Intelligence (here we also include robotics and multi-agent systems) are all relevant but none of them conceptually fully covers AmI. It is AmI which brings together all these resources as well as many other areas to provide flexible and intelligent services to users acting in their environments. By Ambient Intelligence we refer to the mechanisms that rule the behavior of the environment: The definition of Ambient Intelligence given above includes the need for a “sensible” system, and this means a system with intelligence. The definition reflects an analogy with how a trained assistant, e.g. a nurse, typically behaves. The assistant will help when needed but will restrain to intervene except when necessary. Being sensible demands recognizing the user, learning or knowing her/his preferences, and the capability to exhibit empathy with or react to the user’s mood and the prevailing situation, i.e., it implicitly requires for the system to be sensitive. We reserve here the term “Smart Environments” (see for example [12]) to emphasize the physical infrastructure (sensors, actuators and networks) that supports the system. An important aspect of AmI has to do with interactivity. On the one hand there is a motivation to reduce explicit human-computer interaction (HCI) as the sys-
Ambient Intelligence and Smart Environments: A State of the Art
5
Fig. 1 Relationship between AmI and several scientific areas.
tem is supposed to use its intelligence to infer the situations and user needs from the observed activities, as if a passive human assistant were observing the activities unfold with the expectation to help when (and only if) required. Systems are expected to provide situation-aware information (I-want-here-now) through natural interfaces[18]. On the other hand, a diversity of users may need or voluntarily seek direct interaction with the system to indicate preferences, needs, etc. HCI has been an important area of computer science since the inception of computing as an area of study and development. Today, with so many gadgets incorporating computing power of some sort, HCI continues to thrive as an important AmI topic. It is also worthwhile to note that the concept of AmI is closely related to the “service science”[21] in the sense that the objective is to offer proper services to users. It may not be of interest to a user what kind of sensors are embedded in the environment or what type of middleware architecture is deployed to connect them. Only the services given to the user matter to them. Therefore, the main thrust of research in AmI should be integration of existing technologies rather than development of each elemental device. Within the notion of the service science, it is important to consider the implications of user evaluation in a “service-evaluation-research” loop (Figure 2). As part of their investigation, AmI researchers should assess efforts to setting up working intelligent environments in which users conduct their normal activities. As the system is distributed in the environment, users only notice services they receive. The
6
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
researchers then observe the interaction and (re-)evaluate the system. The result of the evaluation may lead to identifying new services, new viewpoints or even new evaluation criteria. Then all of the findings can be fed back to new research and development. This Handbook is organized in several thematic parts. Each of these parts provides a state of the art in research and development related to the specific sub-areas contributing to the development of the broader subject of this volume. Section 2 focuses on the infrastructure and how sensors (including vision as a high-level way of capturing information from an environment) are networked and utilized in a few application settings. Section 3 considers the technology that can be built over a networked sensing infrastructure to make resources widely available in an unobtrusive way. Section 4 gathers contributions on the interaction between humans and artificial systems. Section 5 focuses on developments that aim to make artificial system more rational. This is complemented with section 6 focused on Multi-Agent Systems (MAS). While an area with its own development architecture, these system contribute to the resources an artificial system can use to understand different situations and to decide intelligently. The volume provides in section 7 a wide range of applications as well as consideration of the impact this technology can have in our lives (safety, privacy, etc.). Section 9 provides insight into some of the recent major projects developed around the world.
Fig. 2 The loop of R&D, service, and evaluation.
Ambient Intelligence and Smart Environments: A State of the Art
7
2 Sensors, Vision, and Networks The focus of this thematic part is on the role vision and sensor networks play in acquiring information from the environment and events for responsive and reactive applications. As illustrated in Figure 3, these networks can offer a variety of inferences from the event and human activities of interest [5]. The information is then transferred to high-level reasoning modules for knowledge accumulation in applications involving behavior monitoring, or for reacting to the situation in applications based on ambient intelligence and smart environments. Another type of interaction between vision and higher-level application context may involve visualization of events or human actions which can take the form of video distribution or avatarbased visual communication [4]. In both kinds of interfaces, two-way communication between the vision and data processing modules on the one hand, and high-level reasoning and visualization modules on the other hand can enable effective information acquisition by guiding vision to the features of interest, and validating the vision output based on prior knowledge, context, and accumulated behavior models.
high-level reasoning
visualization
behavior
context
video / select view
voxel reconstruction
events
adaptation
avatar mapping
meta data
interface
knowledge accumulation
detection
observation validation
pose / activity
user preferences and availability
user interface
3D model
processing
vision and sensor network
Fig. 3 The role of vision and sensor networks in interaction with high-level reasoning and visualization.
In this section the contributing authors offer an overview of the latest trends in algorithm and application development based on visual and sensory inputs.
8
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
1. A Survey of Distributed Computer Vision Algorithms Recent years have seen great advances in computer vision research dealing with large numbers of cameras. However, many multi-camera computer vision algorithms assume that the information from all cameras is communicated in a lossless fashion to a central processor that solves the problem. This assumption is unrealistic for emerging wireless camera networks, which may contain processors with limited capability, antennas with limited power, and batteries with limited life. In this chapter, an overview of algorithms that solve computer vision problems in a distributed manner is presented, and their implementation on a visual sensor network is discussed. Methods for topology estimation, camera calibration, and visual tracking are introduced as case studies. 2. Vision-based People Tracking Human pose tracking is a key enabling technology for myriad applications, such as the analysis of human activities for perceptive environments and novel manmachine interfaces. Nevertheless, despite years of research, 3D human pose tracking remains a challenging problem. Pose estimation and tracking is currently possible in constrained environments with multiple cameras and under limited occlusion and scene clutter. Monocular 3D pose tracking in unconstrained natural scenes is much more difficult, but recent results are promising. This chapter provides an introduction to the principal elements of an approach to people tracking in terms of Bayesian filtering. It discusses likelihood functions which characterize the consistency between a hypothetical 3D pose and image measurements, including image appearance and image gradients. To help constrain pose estimation solutions, common prior models over plausible 3D poses and motions are considered. Toward this end, due to the high dimensionality of human pose, dimensionality reduction techniques are often employed as means of constraining the inferred poses to lie on a manifold spanned by training examples. Since the underlying problems are non-linear and result in non-Gaussian posterior distributions, online inference is achieved with particle filters or direct optimization. The chapter also discusses discriminative methods for static pose estimation, which are found helpful for initialization and transient failure recovery. 3. Locomotion Activities in Smart Environments One sub-area in the context of ambient intelligence concerns the support of moving objects, i.e. to monitor the course of events while an object crosses a smart environment and to intervene if the environment should provide assistance. For this purpose, the smart environment has to employ methods of knowledge representation and spatio-temporal reasoning. This enables the support of such diverse tasks as way-finding, spatial search, and collaborative spatial work. A number of case studies are introduced and examined in this chapter to demonstrate how smart environments can effectively make use of the information about the movement of people. A specific focus lies in the research
Ambient Intelligence and Smart Environments: A State of the Art
9
question as to which minimal information suffices the need of the smart environment to deal with approximate motion information in a robust and efficient way, and under realistic conditions. The study indicates that a thorough analysis of the task at hand can enable the smart environment to achieve results using a minimum of sensory information. 4. Tracking in Urban Environments using Sensor Networks based on Audio-Video Fusion Recent advances in sensor networks, micro-scale, and embedded vision technologies are enabling the vision of ambient intelligence and smart environments. Environments will react in an attentive, adaptive and active way to the presence and activities of humans and other objects in order to provide intelligent and smart services. Heterogeneous sensor networks with embedded vision, acoustic, and other sensor modalities offer a pivotal technology solution for realizing such systems. Networks with multiple sensing modalities are gaining popularity in different fields because they can support multiple applications that may require diverse resources. This chapter discusses a heterogeneous network consisting of audio and video sensors for multimodal tracking in an urban environment. If a moving target emits sound, then both audio and video sensors can be utilized for tracking. Audio and video can complement each other, and improve the performance of the other modality through data fusion. Multimodal tracking can also provide cues for the other available modalities for actuation. An approach for multimodal target tracking in urban environments is demonstrated, utilizing a network of mote class devices equipped with microphone arrays as audio sensors and embedded PCs equipped with web cameras as video sensors. 5. Multi-Camera Vision for Surveillance In this chapter, a variety of visual surveillance systems are surveyed. Visual surveillance methods are classified according to the criteria which distinguish between systems intended to acquire detailed information on a small number of subjects, systems designed to track as many subjects as possible, and systems designed to acquire trajectories of subjects and to recognize scenes in specific environments. These systems are also classified based on camera types and situations.
3 Mobile and Pervasive Computing This thematic part focuses on physical aspects of mobility and pervasiveness of devices, as well as systems that support mobility of them. Logical mobility such as mobile agents is treated in section 6 and contextual issues are treated in section 4.
10
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
Middleware architecture plays an important role here since the connection between devices are not fixed. We need a flexible and light-weight operating system not only for connecting sensors and actuators, but for providing seamless connection from devices to users (Figure 4). For context-aware computation, some of the context information such as location, time, temperature, etc. must be handed over to application software modules in some predefined format. To accomplish this, data conversion and even data fusion from many different devices must be handled by the middleware. Handling security also turns into a different issue due to mobility, since traditional methods that assume fixed configuration may not be applicable.
Fig. 4 AmI architecture with the middleware.
This thematic part consists of the following chapters: 1. Collaboration Support for Mobile Users in Ubiquitous Environments Collaboration among people happens ubiquitously. User mobility and seamless access to artifacts and devices are natural parts of collaboration. This chapter will first discuss some core characteristics of collaboration, addressing issues connected with the physical embodiment of resources, informal collaboration, and changes in context, considering that the context of any collaboration includes physical as well as social elements. Discussing these characteristics it focuses on the intersection between the research fields of Computer Supported Cooperative Work (CSCW) and Ambient Intelligence (AmI) and ubiquitous computing. A brief overview of technology supporting ubiquitous collaboration is provided. The last part of the chapter focuses on current research activities at
Ambient Intelligence and Smart Environments: A State of the Art
11
Ubicollab, including development of a platform for design of ubiquitous applications to support cooperation among distributed users. 2. Pervasive Computing Middleware Pervasive computing envisions seamless application support for user tasks. In most scenarios, these applications are executed on sets of networked computers that are integrated invisibly into everyday objects. Due to user mobility the resulting systems are dynamic. As a consequence, applications need to cope with ever-changing execution environments and vastly different user context. Without further support, the development of such applications is a complex and error prone task. Middleware systems can help to reduce the resulting complexity by providing a set of supportive services. This chapter discusses such middleware systems and their services. To do that, the chapter first presents a number of design considerations that have a significant impact on the middleware and service design. Thereafter, it discusses the three main functionalities of pervasive computing middleware. These functionalities provide support for (1) spontaneous remote interaction, (2) context management, and (3) application adaptation. To enable the analysis of a broad range of existing systems, the chapter introduces subcategories, compares their advantages and disadvantages, and classifies approaches accordingly. 3. Case Study of Middleware Infrastructure for Ambient Intelligence Environments A variety of application services in ambient intelligence environments can be realized at reduced cost by encapsulating complex issues in middleware infrastructures that are shared by the applications. Since these attractive services are typically quite complicated, their development is not easy for average programmers. High level abstractions offered by middleware infrastructures make it possible to hide the complexities in ambient intelligence environments. This chapter first discusses several potential future services in ambient intelligence environments, and how high level abstractions offered by middleware infrastructures facilitate their development. Then, it illustrates the effectiveness of the high level abstractions through several case studies of middleware designs. Finally, the chapter describes a number of important design issues for the future development of middleware infrastructures. 4. Collaborative Context Recognition for Mobile Devices This chapter investigates collaborative methods for context awareness with mobile devices. Interpretation of context often has to rely on unreliable sensor information that is noisy, conflicting, or simply missing. To limit the search space of possible interpretations it is useful to have reasonable default conditions. The chapter proposes to obtain the defaults from other people (or their devices). In a way, mobile devices would then mimic humans: observe other people and see how they behave in unfamiliar situations. For instance, if everyone else keeps
12
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
quiet in a concert, perhaps we should keep quiet too. If everyone has switched their phone to “silent” mode, maybe ours should be set to the same – automatically. The chapter discusses experiments suggesting that such collaboration not only provides useful first guesses for context analysis, but also improves the recognition accuracy over single-node algorithms. Through short-range links, mobile devices can query each other’s context information and then jointly (and individually) arrive at a shared interpretation of the situation. The chapter describes possible methods for using the shared context data and outlines some potential application areas as well as the potential risks of the approach. 5. Security Issues in Ubiquitous Computing With ubiquitous computing, everyday objects become networked computing devices. What new security risks does this revolution introduce? Small devices such as RFID tags severely constrain the gate count and the energy budget: how can we build secure protocols on such a foundation? At the next level up, devices with greater resources such as cell phones and A/V components still need to be paired up securely so as to be able to give orders to each other: how can this be done, especially over radio links where you can never be sure of the origin of a message? Going up another level, how can you defend your privacy if wearing networked devices makes you trackable everywhere you go? And finally, how can we reconcile security with usability? This last point is probably one of the most pressing challenges facing designers of ubiquitous systems today. 6. Pervasive Systems in Health Care As the computer disappears in the environments surrounding our activities, the objects therein are transformed to artifacts, that is, they are augmented with Information and Communication Technology (ICT) components (i.e. sensors, actuators, processor, memory, wireless communication modules) and can receive, store, process and transmit information. Artifacts may be totally new objects or improved versions of existing objects, which by using the ambient technology, allow people to carry out novel or traditional tasks in unobtrusive and effective ways. This chapter surveys the application of pervasive systems in supporting activities with health significance. Furthermore an abstract architecture of pervasive health care systems is presented. Then, as an example of a model architecture, the chapter describes and analyzes the HEARTS system, a pervasive system architecture that integrates a number of technologies to enable and support patients with congestive heart failure.
Ambient Intelligence and Smart Environments: A State of the Art
13
4 Human-centered Interfaces The area of smart environment design is the confluence of a multitude of disciplines involving sensor fusion, human-computer interfaces, networking and pervasive computing, and actuation and response services. Developed upon the premise of a user-centric data extraction and decision making paradigm, many smart environment applications support interactivity based on interpretation of the gesture, region of interest, and interactions of the user with the surrounding environment components [6]. Thanks to the proliferation of inexpensive sensors such as cameras, as well as embedded processors, unprecedented potentials exist for realizing novel real-time applications such as immersive human-computer interfaces for virtual reality, gaming, teleconferencing, smart presentations, and gesture-based control, as well as other human-centric applications such as patient monitoring and assistive technologies. Integration of multiple sensors in a distributed human-centric interface embodies new research and development opportunities in algorithm design methods based on collaborative sensing and processing, data fusion, event interpretation, context extraction, and generation of behavior models. while the notion of human centricity finds many interpretations, its true meaning goes beyond any of the individual aspects that have been discussed in the literature. It truly refers to a new paradigm in developing and utilizing technology to serve its user in whatever form and flavor that best offers the intended experience of the application of interest by the user. In the new paradigm, privacy management, ease of use, unobtrusive design, and customized system can each be part of the definition of a human-centric interface. However, these factors are not regarded as rigid requirements when the essence of human centricity is considered, meaning different levels of each factor and relative priorities between them need to be derived from the context and objective of the application. In other words, for example the same vision sensing mechanism that may be perceived as intrusive when used in surveillance applications can be employed in a human-centric paradigm to improve the quality of life of patients and the elderly by timely detection of abnormal events or accidents. Chapters in this thematic part offer an overview of the studies and methodologies of human-centric interface design for applications in smart environments. 1. Human-centered Computing Human-centered Computing (HCC) is a set of methodologies that applies to any field that uses computers, in any form, in applications in which humans directly interact with devices or systems that use computer technologies. This chapter offers an overview of HCC from different perspectives. It describes the three main areas of HCC: content production, analysis, and interaction. In addition, the chapter identifies the core characteristics of HCC, provides example applications and investigates the deployment of the existing systems into real world conditions. Finally, a research agenda for HCC is proposed for future development.
14
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
2. End-User Customization of Intelligent Environments This chapter describes a methodology that enables non-technical occupants of intelligent environments to translate conceptual ideas for the functionality of networked environments and appliances into concrete designs. It builds on a technique called “programming by example” which was first proposed by David Canfield Smith (Stanford University) in the mid-seventies and, later, continued by Henry Lieberman (MIT) in the nineties. The chapter shows how this approach, which has hitherto been applied only to single computing platforms, was extended to allow non-technical people to program distributed computing platforms of the type that make up intelligent environments. In order to achieve this vision the chapter introduces a number of novel concepts and methodologies, in particular a new networked service aggregation model (the deconstructed appliance model), a representation for virtual appliances (metaappliances/applications), an ontology to support decomposition (dComp) and a tool for programming coordinated behavior in rule-based network artifacts (Pervasive Interactive Programming). Finally, the chapter reports on a user evaluation of this methodology, which demonstrates that users find the methods both simple and enjoyable to use. 3. Intelligent Interfaces to Empower People with Disabilities Smart environments are essential for the well-being of people with severe paralysis. Intelligent interfaces can immensely improve their daily lives by allowing them to communicate and participate in the information society, for example, by browsing the web, posting messages, or emailing friends. People with motion impairments often prefer camera-based interfaces, because these are customizable, comfortable, and do not require user-borne accessories that could draw attention to their disability. This chapter presents smart environments that enable users to communicate with the computer via inexpensive video cameras. The video input is processed in real time so that users can work effectively with assistive software for text entry, web browsing, image editing, and animation. Because motion abilities differ from person to person and may change due to a progressive disease, the solution offers a number of environments from which the user may choose. Each environment addresses a different special need. Users may also create their own smart environments that understand their gestures. 4. Visual Attention, Speaking Activity, and Group Conversational Analysis in Multi-Sensor Environments One of the most challenging domains in multi-sensor environments is the automatic analysis of multi-party conversations from sensor data. While spoken language constitutes a strong communicative channel, nonverbal communication is also fundamental, has been substantially documented in social psychol-
Ambient Intelligence and Smart Environments: A State of the Art
15
ogy and cognition, and is opening new computational problems in smart spaces equipped with microphones and cameras. This chapter discusses the role of two nonverbal cues for analysis of group conversations, namely visual attention and speaking activity. The chapter reviews techniques for the extraction of such cues from real conversations - summarizing novel approaches based on contextual and audio-visual integration - and discusses their use in emerging applications on face-to-face and remote conversation enhancement and in the recognition of social behavior. 5. Using Multimodal Sensing for Human Activity Modeling in the Real World This chapter describes experiences from a five-year investigation on the design and deployment of a wearable system for automatically sensing, inferring, and logging a variety of human physical activity. The chapter highlights some of the key findings resulting from the deployment of a working system in three-week and three-month real world field trials, which are framed in terms of system usability, adaptability, and credibility. 6. Recognizing Facial Expressions Automatically from Video Facial expressions are the face changes in response to a person’s internal emotional states, intentions, or social communications. Computer recognition of facial expressions has many important applications such as intelligent humancomputer interaction, computer animation, and awareness systems. This chapter introduces recent advances in computer recognition of facial expressions. It first describes the problem space, which includes multiple dimensions: level of description, static versus dynamic expression, facial feature extraction and representation, expression subspace learning, facial expression recognition, posed versus spontaneous expression, expression intensity, controlled versus uncontrolled data acquisition, correlation with bodily expression, and databases. Meanwhile, the state of the art of facial expression recognition is also surveyed from different aspects. In the second part, the chapter proposes an approach to recognizing facial expressions using discriminative local statistical features. Particularly, Local Binary Patterns are investigated for facial representation. Finally, the current status, open problems, and research directions are discussed. 7. Sharing Content and Experiences in Smart Environments This chapter discusses the possibilities and challenges arising from the growing amount of content that can be accessed on-line anywhere, including being carried around in mobile devices or even being embedded in our environment. The chapter focuses on consumer contents such as music, pictures, video clips, streaming content and games, which people may create, carry around and share with others. The content may be enjoyed alone or together with others via mobile devices or appliances in the environment. Mobility allows novel content delivery mechanisms based on the identified context. Current and future application potentials, the issues relating to user expectations, the concept of expe-
16
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
rience as opposed to simple content, and the options for experience sharing are discussed. The chapter will review interaction solutions that exploit the environment to provide a synergetic interface for content management and rendering. User-driven design is emphasized: The user acts not only as content consumer but also as content producer and co-designer when setting up shared content environments. Technological and other challenges arising from these issues will be presented. 8. User Interfaces and HCI for Ambient Intelligence and Smart Environments This chapter aims to present a systematic structure of the field of User Interfaces, as it applies to Ambient Intelligence and Smart Environments. It starts with the machine side (i.e., the hardware in use) and the human side (i.e., conceptual models, human issues in interaction), then proceeds to sections about designing and evaluating UIs, and ends with some case studies. 9. Multimodal Dialogue for Ambient Intelligence and Smart Environments One goal of Ambient Intelligence and Smart Environments is to make it possible for users to interact naturally with their environment, for example, using speech and/or gestures to switch on the TV or turn off a lamp. This goal can be addressed by setting up a specialized dialogue system that interacts with the user and his environment. On the one hand, this system must process the data provided by a diversity of sensors placed in the environment (e.g., microphones, cameras, and presence detectors), which capture information from the user. The dialogue system may need to employ several dialogue management strategies, user models and contextual information to interact with the user and the environment. The interaction may be initiated by the user or by the system if the latter is designed to be proactive. The dialogue system must then decide on the proper actions to be performed in response to the data captured from the user and the status of the environment. This chapter discusses several issues regarding the implementation of this kind of systems, focusing on context awareness, handling of multimodal input, dialogue management, response generation and evaluation.
5 Artificial Intelligence and Robotics Artificial Intelligence and Robotics are well established areas of research which have witnessed good progress in the past few decades. Yet it is recognized they have not reached their full potential and much work needs to be done in these areas to develop methods for practical AmI and SmE applications. From expert systems to multi-agents there has been significant work and lessons learned. The arrival of the topic of AmI which calls for new computing methods to deliver services in the expected way (see [9] and [20]) and the deployment of AmI
Ambient Intelligence and Smart Environments: A State of the Art
17
concepts over Smart Environments which are characterized by a network of interconnected devices each with limited capability presents both a new opportunity and a new challenge. The opportunity lies in that now AI has another opportunity to get closer to people, to improve the way they live. Now the focus has shifted from justifying the need of AI/robots to support humankind in missions where it is not possible/desirable to involve humans (e.g., going to an inhospitable planet like Mars, entering a building in flames or in danger of collapse, or deactivating a bomb) to more humble and close to people services. Now systems can measure their intelligence in simpler (still enormously challenging) tasks like helping drivers to find their way round a city, helping tourists to make the most of a visit to the museum, making vulnerable people feel safer by notifying carers when their health or safety seems compromised. All this is very healthy for AI because it allows to demonstrate its usefulness for practical applications and to enable it to reach out to outside the lab and deal with the uncertainties of everyday life in the real world. The challenge lies in the complexity that is part of any element of real applications. Systems developed under this concept should be intelligent to detect the situations that matter, and to react to them appropriately and reliably. Relevant to the development of AI systems that can make an environment truly intelligent is the need for such an environment to be able to: (a) learn habits, preferences and needs of the occupants, (b) correctly diagnose situations, (c) be aware of where and when the events of interest occur, (d) integrate mobile elements like robots, and (e) provide a structured way to analyze, decide and react over that environment. The chapters in this thematic part provide a step forward in addressing these issues. 1. Learning Situation Models in Smart Spaces Smart Spaces enhance user capabilities and comfort by providing new services and automated service execution based on sensed user activity. Though, when becoming really “smart”, such spaces should not only provide some service automation, but further learn and adapt their behavior during use. This chapter motivates and investigates learning of situation models in a smart space, covering knowledge acquisition from observation as well as evolving situation models during use. An integral framework for acquiring and evolving different layers of a situation model is detailed. Different learning methods are presented as part of this framework: role detection per entity, unsupervised extraction of situations from multimodal data, supervised learning of situation representations, and the evolution of a predetermined situation model with feedback. The situation model serves as frame and support for the different methods, permitting to stay in an intuitive declarative framework. An implementation of the whole framework for a smart home environment is described, and the results of several evaluations are depicted. 2. Smart Monitoring for Physical Infrastructures Condition monitoring of physical infrastructures such as power and railway networks is an important prerequisite for optimizing operations and maintenance. Today, various sensor systems are in place but generally lack integration for as-
18
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
sessing conditions more precisely. This is a typical challenge faced by contextaware ambient intelligence systems, which tackle it using formal knowledge representation and reasoning. This chapter adopts the knowledge-based approach for infrastructure monitoring: Description Logic-based ontologies are used for developing declarative, extensible and machine-workable models of symptoms, faults, and conditions. A novel approach for reasoning over distributed knowledge bases of context information is introduced. Finally, a case study on European railway monitoring demonstrates the feasibility and potential of the proposed approach. 3. Spatio-Temporal Reasoning and Context Awareness An important aspect of ambient intelligence in smart environments is the ability to classify complex human behavior and to react to it in a context-specific way. For example, a heating system can detect if a human is present in the room, but for it to react appropriately and adjust the temperature to a comfortable level, it needs to consider a wider context: someone watching TV, for example, might require a warmer room than somebody engaged in exercise, whereas someone entering to simply look for a book may be leaving the room shortly. The classification of complex human behavior requires an advanced form of spatio-temporal reasoning, since such classification is often dependent on where or when the behavior occurs. For example, someone walking around a triangle repeatedly, stopping at various points along the route, would look odd unless it occurred in a kitchen, with the triangle being the space between the fridge, sink, and stove. Even in this space, a recurring pattern of behavior has to be interpreted in different ways, depending on the time interval it occurs in – cooking a large meal in the middle of the night would still be unusual. This chapter looks at some of the major conceptual and methodological challenges around spatio-temporal reasoning and context awareness and methods of approaching them. 4. From Autonomous Robots to Artificial Ecosystems In recent years, the new paradigm of ubiquitous robotics attracted many researchers worldwide for its tremendous implications: mobile robots are moving entities within smart environments, where they coexist and cooperate with a plethora of distributed and smart nodes that can be assigned with a certain amount of computational resources and a certain degree of autonomy. This chapter is organized in three main parts: the first part surveys the state-of-theart, pointing out analogies and correspondences between different approaches; the second part describes a real-world scenario: the setup of a robot team for autonomous transportation and surveillance within an intelligent building; and the third part introduces the artificial ecosystem framework as a case study to investigate how ubiquitous robotics can be exploited for the deployment of robotic applications in complex real world scenarios.
Ambient Intelligence and Smart Environments: A State of the Art
19
5. Behavior Modeling for Detection, Identification, Prediction, and Reaction in AI Systems Solutions Behavior modeling automation is a requirement in today’s application development in AI, and the need to fuse data from a large number of sensors demands such automation in the operations to observe, fuse and interpret the data. Applications in abnormal behavior detection, ambient intelligence, and intelligent decision making are a few of the unlimited applications of human intelligence automation. A generalized software approach to implement these solutions is presented in this chapter to break the artificial intelligence behavior modeling problem into four distinct intelligence functions to describe and implement using the distinct subsystems of: Detection, Identification, Prediction, and Reaction (DIPR). These subsystems also allow complete fusion of all sensor features into the intelligent use as a result. Behaviors can be modeled as sequences of classified features, fused with intelligence rules, and events into time and space. Multiple features are extracted with low-level classifiers in time and space in the “Detection” subsystem. The temporal (feature,space) matrix is used by the “Identification” subsystem to combine with low-level simple rules to form intelligent states (symbols). The “Prediction” subsystem concatenates symbols over time/space to form spatial-temporal sequences. These sequences are classified into behaviors and then passed through an inferred predicted behavior outcome module. The final stage is the “Reaction” subsystem for an action producing intelligent solution. The DIPR system is shown to be overlaid onto a number of different distributed networks and a number of pre-engineered application models are presented to demonstrate the applications of the system.
6 Multi-Agents Studies of intelligent agents are roughly divided into three categories: (1) those trying to design a human-like intelligent agents, (2) those trying to implement (semi-) intelligent autonomous software modules, and (3) those emphasizing on the coordination or collaboration among many agents (multi-agents). Agents in the latter two types are extremely useful in AmI. To provide useful functionality in the environments, we have to implement some autonomously functioning software into devices embedded in the environment. Agent technologies are useful in creating such functions. Moreover, to make those devices collaborate with each other and to make them function in unity, we need multi-agent technologies. 1. Multi-Agent Social Simulation The difficulty in the introduction of a new IT system is in the uncertainty of its effect in the society. The benefit of any IT systems appears when it is implemented deeply into the society and impacts social activities in a large scope. Social simulation, especially, multi-agent simulation will provide an important
20
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
tool to evaluate how the IT system may change the behavior of the society. This chapter discusses the capabilities of multi-agent systems in social simulations, and provides some examples of multi-agent simulations that evaluate new IT methodologies. 2. Multi-Agent Strategic Modeling in a Specific Environment One of the tasks in ambient intelligence is to deduce the behavior of agents in the environment. Here we present an algorithm for multi-agent strategic modeling applied in a robotic soccer domain. The algorithm transforms a multi-agent action sequence into a set of strategic action descriptions in a graphical and symbolic form. Moreover, graphic and symbolic strategic action descriptions are enriched with corresponding symbolic rules at different levels of abstraction. The domain independent method was evaluated on the RoboCup Soccer Server Internet League data, using hierarchically ordered domain knowledge. 3. Learning Activity Models for Multiple Agents in a Smart Space Intelligent environment research has resulted in many useful tools such as activity recognition, prediction, and automation. However, most of these techniques have been applied in the context of a single resident. A current looming issue for intelligent environment systems is performing these same techniques when multiple residents are present in the environment. This chapter investigates the problem of attributing sensor events to individuals in a multi-resident intelligent environment. Specifically, a naive Bayesian classifier and a hidden Markov model are used to identify the resident responsible for a sensor event. Results of experimental validation in a real intelligent workplace testbed are presented and the unique issues that arise in addressing this challenging problem are discussed. 4. Mobile Agents Mobile agent technology has been promoted as an emerging technology that makes it much easier to design, implement, and maintain distributed systems, including cloud computing and sensor networks. It provides an infrastructure not only for executing autonomous agents but also for migrating them between computers. This chapter discusses the potential uses of mobile agents in distributed systems, lists their potential advantages and disadvantages, and describes technologies for executing, migrating, and implementing mobile agents. The chapter presents several practical and potential applications of mobile agents in distributed systems and smart environments.
Ambient Intelligence and Smart Environments: A State of the Art
21
7 Applications This thematic part overviews the application areas of Ambient Intelligence and Smart Environments (AISE). One of the important aspects of many concepts developed in AISE is their closeness to the society. While other areas of computer science impact everyday life in our society, AISE provides a framework in which information technologies are applied to the real world situations in a systemic and often multi-disciplinary fashion to enhance our lives. It is worth noting that even subjects in computer science not covered in this handbook, for example, the Internet, language processing and search engines are nevertheless related to AISE because they are part of the intelligence or smartness of the environments we live or work in. We do not cover them here because if we did, this handbook would have been a handbook of the whole computer science! The list of topics covered in this part is not an exhaustive enumeration of applications based on AISE, but a selected set of representative topics from various application fields. The subjects are roughly ordered from applications for supporting an individual to providing services to a large number of people. 1. Ambient Human-to-Human Communication The traditional telephone and video conference systems are examples of a terminal-centered interaction scenario where the user is either holding the terminal in the hand or seated in front of it. The terminal centricity guides the interaction to take the form of an on/off session which starts by making a call and ends by terminating the call. A terminal-centered communication session has the same social characteristics as a formal face-to-face meeting. When people are physically together there is a larger variety of interaction modes available which may be characterized by different ranges of social or interpersonal distances beyond the face-to-face meeting mode. The model for the ambient communication technology is the natural interaction between people who are physically present. In this chapter we give a general overview of various topics related to ambient communication. A detailed description of the architecture and essential components of a voice-based ambient telephone system is provided as an example. 2. Smart Environments for Occupancy Sensing and Services This chapter discusses recent developments in occupancy sensing and applications related to sensing the location of users and physical objects and occupation in smart environments, mainly houses, offices, and urban spaces. The sensory equipment widely used for sensing are presented, as well as different tools and algorithms used to process, present, and apply the information. The focus is on different detecting devices, data mining, and statistical techniques that have been utilized in pervasive computing applications. Examples that utilize occupancy information to enable services for ordinary or professional users are studied.
22
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
3. Smart Offices and Smart Decision Rooms Decision making is one of the noblest activities of the human being. Most of times, decisions are made involving several persons (group decision making) in specific spaces (e.g. meeting rooms). On the other hand many scenarios (e.g. increased business opportunities, intense competition in the business community) demand fast decision making to achieve success. Smart offices and decision rooms contribute to reducing the decision cycle, and offer connectivity wherever the users are, aggregating the knowledge and information sources. This chapter provides a survey of smart offices and intelligent meeting rooms, considering the hardware and software perspectives. It presents the Smart Decision Room, a project platform created to provide support to meeting room participants, assisting them in the argumentation and decision making processes, while combining the rational and emotional aspects. 4. Smart Classroom: Bringing Pervasive Computing into Distance Learning The Smart Classroom project aims to build a real-time interactive classroom with tele-education experience by bringing pervasive computing technologies into traditional distance learning. This chapter presents the current state of the project which includes four key technologies: 1) a Smart Classroom, which embeds multiple sensors and actuators (e.g. location sensor, RFID, microphone array) to enhance the conventional classroom-based learning environment; 2) key multimodal interface technologies and context-aware applications, which merge speech, handwriting, gestures, location tracking, direct manipulation, large projected touch-sensitive displays, and laser pointer tracking, providing enhanced interaction experience for teachers and students; 3) a dedicated software called “SameView”, which uses a hybrid application-layer multicast protocol and adaptive content delivery scheme, providing a rich set of functions for teachers, local students and large-scale remote students to carry out realtime, collaborative distance learning experience; 4) a software infrastructure based on multi-agent architecture called Smart Platform, which integrates, coordinates and manages all the distributed collaborative modules in the smart environment, supporting various data transmission modes in terms of different quality of service (QoS) levels. The smart classroom system has been demonstrated as a prototype system in Tsinghua University. 5. Ambient Intelligence in the City This chapters aims to address deficits in the way ambient intelligence conceives of and designs for urban environments. It begins with an overview of issues surrounding ambient intelligence in the city, followed by critical voices and examples of alternative ambient intelligence systems. The text argues for the inclusion of pervasive sensing issues into urban planning, in particular to negotiate between private and public interests.
Ambient Intelligence and Smart Environments: A State of the Art
23
6. The Advancement of World Digital Cities This chapter reviews the advances in several worldwide activities on regional information spaces empowered by new technologies in sensors, information terminals, broadband networks and wireless networks. In the US and Canada, a large number of community networks supported by grass-root activities appeared in the early 1990s. In Europe, more than one hundred digital cities have been tried, often supported by local or central governments and the EU in the name of local digitalization. Asian countries have actively adopted the latest information technologies as a part of national initiatives. In the past 15 years since the first stage of digital cities, the development of the original digital cities has leveled off or stabilized. In spite of that, by looking back at the trajectory of the development of digital cities, it is clear that digital environments in cities have often benefited from the previous activities on various regional information technology spaces.
8 Societal Implications and Impact It is well known within the community that the areas covered in this volume can potentially have profound impacts in our society. On the one hand they can provide unprecedented support for the society to enter a true modern digital era where computers everywhere silently assist us. On the other hand, this can also prove to be a technical minefield where many problems arise as a result of attempting to offer the intended services [13, 15, 2, 10]. User-oriented design, privacy preservation, personalization and social intelligence are some of the fundamental concepts the area has to approach efficiently in order to encourage acceptance from the masses. Failing to do well in those aspects will most certainly cause the users to feel overwhelmed and distrustful of the technology that is entering their daily lives in a disruptive way. The chapters in this thematic part address some of these important user-related issues, namely how to achieve truly unobtrusive systems, give privacy issues a well deserved relevance, personalize systems to a user, and consider social interaction as an essential element of design. 1. Human Factors Consideration for the Design of Collaborative Machine Assistants The basic idea of ambient intelligence is to connect and integrate technological devices into our environment such that the technology itself disappears from sight and only the user interface remains. Representative technologies used in ambient intelligent environments are collaborative machine assistants (CMAs), for example, in the form of a virtual agent or robot. Designing effective CMAs requires attention to the user interaction making it critical to consider user characteristics (e.g., age, knowledge), technology characteristics (e.g., perceived usefulness, ease of use), and relational characteristics (e.g., voluntariness, attraction) during the design. We propose a framework for understanding the user
24
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
variables that influence the acceptance of CMAs. Our goal is to provide designers with heuristic tools to be used during the design process to increase user acceptance of such technology. 2. Privacy Sensitive Surveillance for Assisted Living: A Smart Camera Approach Western societies are aging rapidly. An automated 24/7 surveillance to ensure safety of the elderly while respecting privacy becomes a major challenge. At the same time this is a representative of novel and emerging video surveillance applications discovered lately besides the classic surveillance protection applications in airports, government buildings and industrial plants. Three problems of current surveillance systems are identified. A distributed and automated smart camera based approach is proposed that addresses these problems. The proposed system’s goal set is to analyze the real world and reflect all relevant – and only relevant – information live in an integrated virtual counterpart for visualization. It covers geo referenced person tracking and activity recognition (falling person detection). A prototype system is installed in a home for assisted living running 24/7 for several months now and shows quite promising performance. 3. Data Mining for User Modeling and Personalization in Ubiquitous Spaces User modeling has traditionally been concerned with analyzing a user’s interaction with a system and developing cognitive models that aid in the design of user interfaces and interaction mechanisms. In recent years, the large amounts of data that can be collected from users of technology in multiple scenarios (web, mobile, etc.) has created additional opportunities for research, not only for traditional data mining applications, but also for creating new user models that lead to personalized interaction and services. The explosion of available user data presents many opportunities and challenges, requiring the development of new algorithms and approaches for data mining, modeling and personalization, particularly considering the importance of the socio-cultural context in which technologies are used. This chapter will present an overview of data mining of ubiquitous data, review user modeling and personalization methods, and discuss how these can be used in ubiquitous spaces not just for designing new interfaces, but also in determining how user models can inform the decisions concerning what services and functionalities are adequate for particular users. 4. Experience Research: a Methodology for Developing Human-centered Interfaces Ten years of Ambient Intelligence (AmI) research have led to new insights and the understanding that Ambient Intelligence should not be so much about system intelligence and computing but more about social intelligence and innovation. We discuss these novel insights and their resulting impact on the AmI research landscape. We argue that new ways of working are required applying the concept of Experience Research resulting in a true user-centered approach
Ambient Intelligence and Smart Environments: A State of the Art
25
to ambient intelligence. The Experience Research approach is illustrated with a small case study on applying this methodology in the area of Ambient Assisted Living.
9 Selected Research Projects Another sign of the interest these areas are attracting is the number of projects that have been started in the last decade around the world. The number of such projects have grown exponentially thanks to the realization of governments (predominantly in the north hemisphere) of the strategic importance this problem has for the future development of the most industrialized nations. Both governments and companies have started to invest heavily in addressing different bottlenecks for the future development of the area and in the development of the first generation of system which can increase safety, entertainment, healthy lifestyle habits, and many other services that people can enjoy during work, life at home, and leisure time. This thematic part provides a glance on some of those projects, given space restrictions we can only offer a small number of them but they illustrate a variety of topics (human-centric models of computing, non-traditional ways of interaction, robust middleware, ambient assisted living, efficient and automated light control, and large ubiquitous experiments in Japan and Korea) which have been effectively addressed by large consortia of academic and industrial organizations in a collaborative way. 1. Computers in the Human Interaction Loop Computers have become an essential part of modern life, supporting human activities in a multiplicity of ways. Access to the services that computers provide, however, comes at a price: Human attention is bound and directed toward a technical artifact rather than at the human activity or interaction that computing is to serve. In CHIL (Computers in the Human Interaction Loop) Computing we explore an alternative paradigm for computing in a human centered way. Rather than humans attending to computers, we are developing computing services and technologies that attend to humans while they go about their normal activities and attempt to provide implicit support. By analogy to a human butler hovering in the background, computers that attend to human needs implicitly would require a much richer understanding and quite a bit of careful observation of human interaction and activities. This presumes a wide set of perceptual capabilities, allowing to determine who was doing what to whom, where, why and how, based on a multimodal integration of many sensors. It also presumes the ability to derive rich descriptions of human activities and state, so that suitable inferences for the determination of implied need can be made. In this chapter, we present several prototypical CHIL services and their supporting perceptual technologies. We also discuss the evaluation framework under which technological advances are made, and the software architecture by which perceptual components are integrated. Finally, we examine the social impact of CHIL com-
26
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
puters; how increasingly social machines can better cooperate with humans and how CHIL computers enhance their effectiveness. 2. Eye-based Direct Interaction for Environmental Control in Heterogeneous Smart Environments Eye-based environmental control requires innovative solutions for supporting effective user interaction, for allowing home automation and control, and for making homes more “attentive” to user needs. This chapter describes the research efforts within the COGAIN project, where the gaze-based home automation problem is tackled as a whole, exploiting stateof-the-art technologies and trying to integrate interaction modalities that are currently supported and that may be supported in the near future. User-home interaction is sought through an innovative interaction pattern: direct interaction. Integration between the eye tracking interfaces, or other interaction modalities, and the wide variety of equipment that may be present in an intelligent house (appliances and devices) is achieved by the implementation of a Domotic House Gateway that adopts technologies derived from the semantic web to abstract devices and to enable interoperability of different automation subsystems. Innovative points can be identified in the wide flexibility of the approach which allows on one hand integration of virtually all home devices having a communication interface, and, on the other hand, direct user interaction paradigms, exploiting the immediacy of command and feedback. 3. Middleware Architecture for Ambient Intelligence in the Networked Home The European IST FP6 Amigo project aimed at developing a networked home system enabling the ambient intelligence / pervasive computing vision. Key feature of the Amigo system is effectively integrating and composing heterogeneous devices and services from the four application domains (i.e., personal computing, mobile computing, consumer electronics and home automation) that are met in todayÃÔ home. The design of the supporting interoperability middleware architecture is the focus of this chapter. The Amigo middleware architecture is based on the service-oriented architectural style, hence enabling a networked home system structured around autonomous, loosely coupled services that are able to communicate, compose and evolve in the open, dynamic and heterogeneous home environment. The distinguishing feature of the proposed middleware architecture is that it poses limited technology-specific restrictions: interoperability among heterogeneous services is supported through semanticbased interoperability mechanisms that are key elements of the Amigo architecture. 4. The PERSONA Service Platform for Ambient Assisted Living Spaces An European project started in January 2007 in the field of Ambient Assisted Living, PERSONA strives for the development of sustainable and affordable solutions for the independent living of senior citizens. AAL Services with focus on social integration, assistance in daily activities, safety and security, and
Ambient Intelligence and Smart Environments: A State of the Art
27
mobility should be developed based on an open scalable technological platform. This chapter summarizes the conceptual design of this service platform for AAL spaces that is being finalized in the end of the second year after project start. It introduces a physical and a logical architecture for AAL spaces at an abstract level along with a set of more concrete functional components that are identified as part of the platform. The abstract physical architecture treats an AAL space as a dynamic ensemble of networked nodes and the interoperability framework abstracts all functionality not leading to context, input, or output events (or the direct processing of such events) as service. Hence, the proposed architecture is service oriented. A last topic treated in this chapter concerns the PERSONA solution for binding ultra-thin devices through a Sensor Abstraction and Integration Layer. 5. ALADIN - a Magic Lamp for the Elderly? The overall aim of ALADIN, an EU-funded research project in the realm of Ambient Assisted Living (AAL), is to extend our knowledge about the impact of lighting on the well-being and comfort of older people and translate this into a cost-effective open solution. The development approach is underpinned by an ambient intelligence framework which focuses on user experience and emphasizes activity support. The adaptive lighting system developed in ALADIN consists of an intelligent open-loop control, which can adapt various light parameters in response to the psycho-physiological data it receives. For adaptive control, simulated annealing and genetic algorithms have been developed. In the lab tests that were carried out to determine the psycho-physiological target values for activation and relaxation, skin conductance response emerged as the most appropriate target value with heart rate as the second best candidate. Tests with different light parameters (color temperature, intensity, distribution) showed a clear impact on older adults both in terms of activation and relaxation. The technical architecture and hardware prerequisites (sensor device, lighting installation) as well as the different software applications are described. Finally, the preliminary results of the extensive field trials conducted in twelve private households are discussed. The user feedback provides the basis for future application scenarios and exploitation plans of the Consortium partners. 6. Japanese Ubiquitous Network Project: Ubila From 2003 to March 2008, Ubila , a part of the “u-Japan” project, was a pioneering national ubiquitous network R&D project in Japan. Its objective was to develop core technologies to realize ambient environments with the cooperation of networks, middleware and information processing technologies. The Ubila project has demonstrated various activities. Every year, its research outputs were demonstrated at exhibitions and also periodically demonstrations were also conducted at smart spaces for guests. the Ubila project has released two service prototype videos that reflect its R&D directions. the Ubila project marked a unique and important step toward the advent of the ambient world.
28
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
7. Ubiquitous Korea Project In 2006 Korean government planed to construct u-Korea until 2015. Although the plan has not been executed exactly, the government has established several huge national projects for constructing u-Korea. In this chapter we introduce four major projects related with ubiquitous computing technology which has been conducted by Korean government.
10 Perspectives of the Area When we look back to the research in intelligent systems, we can see the general direction moving from the individual to the group, and from closed systems to open systems. When the research area of Artificial Intelligence was first established, the dominating concept was “the physical symbol system hypothesis” which postulates that the essence of intelligence is captured by and only by symbol processing[19]. As a natural consequence, an intelligent system was modeled as an input-computeoutput, or recognize-compute-act system (Figure 5).
Fig. 5 The old view of intelligence.
As experience was gained in the area, the community began to recognize the importance of the environment. There even exists a viewpoint declaring that only the environment is the key to intelligent behavior: An agent (or any actor in the environment) only demonstrates behavior by picking up an affordance provided by the environment [14]. This view, although regarded by some as extreme, has some alignment with the concept of Ambient Intelligence in emphasizing the various influences and multi-faceted services the environment has to offer to its occupants.
Ambient Intelligence and Smart Environments: A State of the Art
29
Around the same period, the concept of Autopoiesis which denies a predefined distinction between the system and the environment was proposed [17]. These concepts also match research directions taken on in robotics where sensors and actuators play more important roles than symbolic planning. Brooks [11] even denied the necessity of internal symbolic representation. These new view of intelligence can be formalized as Figure 6 where recognition, computation and action take place in parallel. The essence of general research direction in Ambient Intelligence, placing more emphasis on the environment than the agents acting in the environment, can be captured along the same direction but in a wider scope. The area is not limited to a single agent, but rather aims to support a team of agents in one environment, or even the society as a whole. This reflects the trend from focusing on the individual towards serving a group.
Fig. 6 A new view of intelligence, which includes the environment within the system.
11 Conclusions This handbook offers a snapshot of the state-of-the-art in Ambient Intelligence and Smart Environments. The various core disciplines participating in the creation of the concepts and applications of AmI and SmE are represented through examples of research and development efforts by leading experts in each field. The domain of potentials is vast with applications in all aspects of life for an individual or a community. The extent of the area overlaps with such diverse groups of disciplines as sensor networks and sociology, and artificial intelligence and communication networks. While this offers ample opportunities for many areas of science and technology to impact the future of AmI and SmE by providing tools from within the
30
Juan Carlos Augusto, Hideyuki Nakashima, Hamid Aghajan
focus area of each discipline, truly ground breaking impact is possible through collaboration among researchers from several disciplines. This handbook aims to offer and encourage such opportunities by introducing a representative set of fundamental concepts from the different areas. It is hoped that the presented material will inspire researchers to steer their work towards making novel contributions to this promising area. The rest of the handbook is divided into 8 thematic parts, each with a collection of contributions from recent studies. We hope you will find the material as exciting and inspiring as we have.
Acknowledgments We wish to thank Jennifer Evans and Melissa Fearon of Springer for their encouragement and support of this project. We are also grateful to Valerie Schofield and Jennifer Maurer of Springer for their assistance with the production of this handbook. Many thanks go to the contributors of the chapters for the book. Their active and timely cooperation is highly appreciated.
References [1] Aarts E, Appelo L (1999) Ambient intelligence: thuisomgevingen van de toekomst. IT Monitor (9):7–11 [2] Aarts E, de Ruyter B (2009) New research perspectives on ambient intelligence. Journal on Ambient Intelligence and Smart Environments (JAISE) 1(1):5–14 [3] Aarts E, Harwig R, Schuurmans M (2002) Ambient intelligence. In: The invisible future: the seamless integration of technology into everyday life, McGrawHill, pp 235–250 [4] Aghajan Y, Lacroix j, Cui J, van Halteren A, Aghajan H (2009) Home Exercise in a Social Context: Real-Time Experience Sharing using Avatars, Intetain’09 [5] Aghajan H, Cavallaro A (2009) Multi-Camera Networks: Principles and Applications, Academic Press 2009 [6] Aghajan H, Augusto J, Delgado R (2009) Human-centric Interfaces for Ambient Intelligence, Academic Press 2009 [7] Augusto JC (2007) Ambient Intelligence: the Confluence of Ubiquitous/Pervasive Computing and Artificial Intelligence, Springer London, pp 213–234. Intelligent Computing Everywhere [8] Augusto JC, Nugent CD (eds) (2006) Designing Smart Homes. The Role of Artificial Intelligence. Springer-Verlag [9] Augusto JC, Nugent CD (2006) Smart homes can be smarter. In: [8], SpringerVerlag, pp 1–15
Ambient Intelligence and Smart Environments: A State of the Art
31
[10] Böhlen M (2009) Second order ambient intelligence. Journal on Ambient Intelligence and Smart Environments (JAISE) 1(1):63–67 [11] Brooks R (1990) Elephants Don’t Play Chess. In: Robotics and Autonomous Systems 6, pp. 3–15 [12] Cook D, Das S (2004) Smart Environments: Technology, Protocols and Applications (Wiley Series on Parallel and Distributed Computing). WileyInterscience [13] Dertouzos M (2001) Human-centered systems. In: The Invisible Future, McGraw-Hill, pp 181–192 [14] Gibson J (1979) The Ecological Approach to Visual Perception. Houghton Mifflin, Boston [15] Huuskonen P (2007) Run to the hills! ubiquitous computing meltdown. In: Augusto JC, Shapiro D (eds) Advances in Ambient Intelligence, no. 164 in Frontiers in Artificial Intelligence and Applications (FAIA), IOS Press [16] IST Advisory Group (2001) The European union report, scenarios for ambient intelligence in 2010. URL: ftp://ftp.cordis.lu/pub/ist/docs/ istagscenarios2010.pdf [17] Maturana H and Varela F (1980) Autopoiesis and Cognition: The Realization of the Living. D. Reidel Publishing Co., Dordrecht, Holland [18] Nakashima H (2007) Cyber assist project for ambient intelligence. In: Augusto JC, Shapiro D (eds) Advances in Ambient Intelligence, IOS Press Inc. [19] Newell A, Simon H (1963), GPS: A Program that Simulates Human Thought. In: Feigenbaum E, Feldman J (eds) Computers and Thought, McGraw-Hill [20] Ramos C, Dey A, Kameas A (2009) Thematic Issue on Contribution of Artificial Intelligence to AmI. Journal on Ambient Intelligence and Smart Environments (JAISE) 1(3) [21] Spohrer J, Maglio P, Bailey J, Gruhl D (2007) Steps Toward a Science of Service Systems, IEEE Computer 40(1):71–77 [22] Streitz NA, Kameas A, Mavrommati I (eds) (2007) The Disappearing Computer, Lecture Notes in Computer Science, vol 4500, Springer [23] Weiser M (1991) The computer for the twenty-first century. Scientific American 165:94–104 [24] Weiser M (1993) Hot topics: Ubiquitous computing. IEEE Computer 26(10):71–72 [25] Zelkha E (1998) The future of information appliances and consumer devices, Palo Alto Ventures, Palo Alto, California
Part II
Sensor, Vision and Networks
A Survey of Distributed Computer Vision Algorithms Richard J. Radke
1 Introduction Over the past twenty years, the computer vision community has made great strides in the automatic solution to such problems as camera localization and visual tracking. Many algorithms have been made tractable by the rapid increases in computational speed and memory size now available to a single computer. However, the world of visual sensor networks poses several challenges to the direct application of traditional computer vision algorithms. First, visual sensor networks are assumed to contain tens to hundreds of cameras- many more than are considered in many vision applications. Second, these cameras are likely to be spread over a wide geographical area- much larger than the typical computer lab. Third, the cameras are likely to have modest local processors with no ability to communicate beyond a short range. Fourth, and perhaps most importantly, no powerful central processor typically exists to collect the images from all cameras and solve computer vision problems relating them all. Even if a central processor is available, the sheer number of cameras now feasible in a visual sensor network may preclude the use of a centralized algorithm. For example, more than half a million CCTV cameras observe the streets of London, with thousands in the airports and subways alone. Near real-time processing of this amount of data at a central processor (either with automatic vision algorithms or human observation) is simply infeasible, arguing for so-called “smart cameras” that can perform image processing and computer vision tasks locally before transmission to the broader network. These trends all point to the need for distributed algorithms for computer vision problems in visual sensor networks. In addition to being well-suited to ad-hoc wireRichard J. Radke Department of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute Troy, NY, USA e-mail:
[email protected] H. Nakashima et al. (eds.), Handbook of Ambient Intelligence and Smart Environments, DOI 10.1007/978-0-387-93808-0_2, © Springer Science+Business Media, LLC 2010
35
36
Richard J. Radke
less networks, distributed algorithms have no single point of failure, generally make fairer use of underyling communication links and computational resources, and are more robust and scalable than centralized algorithms. In this chapter, we survey the recent literature on distributed computer vision algorithms designed for implementation in a visual sensor network. We note that many algorithms described as “distributed” either require data to be transmitted to a central processor after some on-board processing at the sensor, or require each node to collect data from all other nodes in the network, effectively making every node perform the same central computation. While such research is valuable, in this chapter we emphasize techniques that are truly distributed: that is, nodes in the network make local decisions based only on information from their immediate neighbors. The design goal for such algorithms should be that the distributed result approaches what could be obtained had all the information been available at a central processor. We also do not consider algorithms that are “distributed” in the sense of being efficiently implemented in parallel on a central multi-core processor. The algorithms we survey fall into a few main categories. Section 2 reviews general work on non-computer-vision related distributed algorithms, many of which have recently found new applications in wireless sensor networks. Section 3 discusses the problem of topology estimation in a visual sensor network- that is, the determination of which nodes share overlapping fields of view, or have corresponding entry/exit points. Section 4 discusses the distributed calibration of a camera network- that is, the estimation of the three-dimensional location and orientation of each camera with respect to a common coordinate system. Section 5 discusses a common application of visual sensor networks: the distributed tracking of single or multiple moving objects, which may also include identification or classification of the objects. Section 6 concludes the chapter with a discussion of future trends and opportunities. We note that our interest here is at the application layer of a visual sensor network- that is, the computer vision algorithms running at each node as opposed to the physical layer of the sensor nodes. The algorithms described here are generally applicable to the many “smart camera” nodes under development in academia and industry that consist of a visual sensor, a set of embedded network and digital signal processors, and an antenna. An excellent taxonomy and historical overview of smart cameras was recently presented by Rinner et al. [83, 81]. Several chapters in the rest of this book go into more detail on specific instances of the algorithms generally surveyed here. We also note the increased academic and industrial interest in the area of distributed computer vision. Much of the most innovative work has appeared in the 2006 Distributed Smart Camera Workshop 1 , and the 2007 and 2008 International Conference on Distributed Smart Cameras 2 . Recent special issues of the Proceedings of the IEEE [82], the IEEE Journal on Selected Topics in Signal Processing [1], and the EURASIP Journal on Advances
1 2
http://www.iti.tugraz.at/dsc06/ http://www.icdsc.org/
A Survey of Distributed Computer Vision Algorithms
37
in Signal Processing [44] dedicated to multi-camera networks further address the topics.
2 Distributed Algorithms The basic theory of distributed algorithms is well-described in the textbook by Lynch [52]. Distributed algorithms are categorized by whether they are synchronous, asynchronous, or partially synchronous, and whether communication is established using shared memory, broadcast messages, or point-to-point communication. Typical problems include electing a leader, conducting a breadth-first search, finding shortest paths, constructing a minimum spanning tree, and reaching consensus. In general, provably convergent algorithms are more difficult to derive in asynchronous networks than in synchronous ones. However, most of the work surveyed in this chapter implicitly assumes at least partially synchronous networks, which may not be realistic for real-world visual sensor networks. The book by Bertsekas and Tsitsiklis [9] discusses somewhat higher-level parallel and distributed algorithms in the context of numerical analysis and optimization. They consider linear and nonlinear asynchronous iterative methods for algebraic and differential equations, dynamic programming, unconstrained and constrained optimization, and variational inequalities. The book by Varshney [92] treats classical detection theory and hypothesis testing in a distributed framework, including Bayesian, Neyman-Pearson, minimax, sequential, and locally optimum detection. We note that many distributed estimation problems arising from visual sensor networks can be viewed as special types of consensus problems. That is, all nodes in the network should come to an agreement about global parameters describing the network, or objects moving in it, without communicating all sensor data to a centralized location. For example, all the cameras in the network might need to agree on the estimated position of a central landmark. The past several years have seen an explosion of interest in distributed algorithms specifically designed for wireless ad-hoc sensor networks, often taking into account networking, communication, and power considerations. A substantial body of work exists on the distributed localization of nodes in wireless sensor networks based on considerations such as the time-of-flight between nodes. For example, Iyengar and Sikdar [38] proposed a distributed, scalable algorithm for sensor node localization in the plane that used time-of-arrival measurements instead of GPS. Marinakis and Dudek [59, 60] presented several innovative probabilistic algorithms for sensor network localization based on noisy inter-sensor range data. Langendoen and Reijers [45] overviewed and compared several algorithms for the localization problem. We can think of the algorithms discussed in Sections 3 and 4 as extensions of the localization problem in 3D that use visual information as the basis for estimation. Similarly, these algorithms, as well as the tracking algorithms considered in Section 5, can be considered as instances of distributed parameter estimation or opti-
38
Richard J. Radke
mization problems, another topic of interest in the sensor networking community. For example, Boyd et al. [11] overviewed gossip algorithms for information exchange in sensor networks. They showed how such algorithms could be applied to the distributed estimation of an average value in a wireless sensor network. Nowak [69] showed how the parameters of a mixture of Gaussians for which each node of a sensor network had different mixing coefficients could be estimated using a distributed version of the well-known expectation-maximization (EM) algorithm. This message-passing algorithm involves the transmission of sufficient statistics between neighboring nodes in a specific order, and was experimentally shown to converge to the same results as centralized EM. Kowalczyk and Vlassis [42] proposed a related gossip-based distributed algorithm called Newscast EM for estimating the parameters of a Gaussian mixture. Random pairs of nodes repeatedly exchange their parameter estimates and combine them by weighted averaging. Rabbat and Nowak [76] investigated a general class of distributed optimization algorithms for sensor networks. They showed that many natural optimizations could be solved incrementally, by each node updating the estimated parameters it receives to reduce its local cost and passing the updated parameters on. Using the theory of incremental subgradient optimization, they showed that the distributed algorithm converges to within a small amount of the globally optimal value after only a few cycles through the network. They applied the approach to robust least-squares estimation, energy-based source localization, and decision boundary problems. Finally, we note research on activating or orienting directional sensors to maximize spatial coverage. Ai and Abouzeid [2] described a distributed algorithm for maximizing spatial coverage in an ad-hoc network of directional sensors. They proposed a distributed greedy algorithm that compared favorably to a central greedy algorithm, as well as a dynamic sensor scheduling algorithm that favorably distributed its energy requirements. Hoffmann and Hähner [35] proposed a simple local algorithm to maximize visual coverage in a network of smart cameras. They simulated their results on networks of up to 80 cameras, and analyzed the underlying networking issues using the NS2 network simulator. Bash and Desnoyers [8] proposed an algorithm for the distributed exact computation of Voronoi cells in sensor networks, which could be useful for addressing coverage and tracking problems.
3 Topology Estimation We consider two main types of topology estimation problems in visual sensor networks. The first is what we refer to as the overlapping problem, where it is assumed that the cameras observe parts of the same environment from different perspectives. The relationships between the cameras can be modeled as an undirected graph in which an edge appears between two cameras if they observe some of the same scene points from different perspectives. In our research, we called this the vision graph [17]. We might naturally expect cameras’ fields of view to overlap in wireless ad-
A Survey of Distributed Computer Vision Algorithms
39
hoc camera networks, in which a large number of cameras are densely distributed around a large, open environment. The second topology estimation problem is what we refer to as the nonoverlapping problem, where it is assumed that no two cameras observe the same part of the environment. Instead, relationships between the cameras are induced by the likelihood that an object in one camera appears in another after some amount of time. The relationships between the cameras can be again be modeled as a graph, but in this case a directed graph with expected transition probabilities is a more natural model (for example, it may be much more likely for objects to travel from Camera A to B than vice versa). This situation naturally arises in indoor surveillance scenarios, such as cameras mounted in corridors. For this reason, most research on non-overlapping networks assumes a planar environment. In either case, we should keep in mind that the topology graph estimation problem in a wireless visual sensor network is accomplished by sending messages along underlying one-hop communication links. These links can be abstracted as a communication graph [29], which is mostly determined by the locations of the nodes, the powers of their antennas, and the topography of the environment. The camera topology estimation problem can be viewed as an overlay graph on this network. We note that the presence of an edge in the communication graph does not imply the presence of the same edge in the vision graph, since nearby cameras may be pointed in different directions, and conversely, physically distant cameras can image part of the same scene. Figure 1 illustrates the communication and vision graphs for a simulated example in which 30 cameras are scattered around several buildings. We can see that the communication and vision graphs are not highly correlated.
8
7
6
5
9
21
10
8
7
6
5
23
9
21
20
20
22 12
4
11
12
13
13 26
3
11
14
19
28
26
3 28
27
2 15
1
27
2 24
29
(a)
23
22
4
14
19
10
25
25
15
30
16
(b)
17
18
1
24
29
30
16
17
18
(c)
Fig. 1 (a) A simulated camera network (the focal lengths of the cameras have been exaggerated). (b) The corresponding communication graph, assuming each camera has the same fixed antenna range. (c) The corresponding vision graph.
As Figure 1 illustrates, there is generally no way to rule out the possibility of overlap between any two visual sensors without comparing their images (or extracted features from them). Consequently, correctly estimating the camera network topology inherently involves collecting the information from all sensors in one place, either at a central node, or at all nodes in parallel. We review several
40
Richard J. Radke
such methods here, emphasizing those designed with the constraints of a visual sensor network in mind. Even though such algorithms may be better characterized as decentralized than distributed, they are an important first step for the higher-level vision problems we discuss subsequently. We review approaches to the topology estimation problem for both cases below. Virtually all research in the area seems to cast itself in one domain or the other- that is, either the cameras are viewed as entirely non-overlapping, or non-overlapping cameras are not considered in the topology. The paper by Wang et al. [94] is an exception that allows for no, little, or large overlaps between fields of view, but the focus is on activity analysis of multi-camera trajectories rather than topology estimation.
3.1 Non-overlapping Topology Estimation The non-overlapping topology estimation problem is well-studied. Most researchers have approached the problem using the spatio-temporal paths of tracked objects that enter and exit the field of view of each camera. Marinakis and Dudek [58] described how to estimate a Markov model characterizing transitions between cameras, using only non-discriminative detected motions as input. They reconstructed object trajectories that best explain the cameras’ observations, and used these to estimate the transition graph. The number of objects moving through the camera network was assumed known, and their probabilistic behavior was assumed to be homogeneous. Makris et al. [55] proposed a similar approach that temporally correlates incoming and outgoing targets in detected entry and exit zones for each camera. Tieu et al. [89] used mutual information as a measure of statistical dependence to estimate object correspondences for topology estimation. Special cases of objects used for non-overlapping topology estimation include identifiable people [96] and vehicles [68]. Mandel et al. [56] described a simple algorithm for estimating camera network topology based on broadcast messages regarding time-stamped detected motions in different spatial regions. They applied a sequential probability ratio test (SPRT) to accept or reject the possibility that two cameras observe the same scene based on the correspondence (or lack thereof) of the aggregated detections. The use of the SPRT is a noteworthy idea for decision problems in sensor networks. Van den Hengel et al. [34] took a slightly different approach they called exclusion, in which the vision graph was initially assumed to be fully connected, and edges were removed that are contradicted by observed evidence over time. They tested their algorithm on a network of 100 real cameras. Javed et al. [39] incorporated the appearance of objects as well as their spatiotemporal motion into the topology estimation problem. While their emphasis was on tracking, they learned the topology of a set of cameras during a training phase that used Parzen windows to model the joint density of entry/exit times, locations, and velocities, and a Gaussian model for the Bhattacharyya distances between his-
A Survey of Distributed Computer Vision Algorithms
41
tograms of corresponding objects. They later extended their approach [40] to more accurately model the brightness transfer function relating pairs of cameras. It would be interesting to see how this approach scales to large networks of cameras. Farrell and Davis [26] proposed a different decentralized algorithm for topology discovery based on an information-theoretic appearance matching criterion on tracked objects moving through the network. More distinctive objects contribute proportionally more to the estimated transition model between cameras. This idea seems interesting, but it was not tested with real images. Medeiros et al. [61] took a slightly different approach to estimating the relationships between cameras based on what they termed event-driven clustering. Instead of explicitly constructing a graph representing the camera network topology, they used estimated detections of the same object to construct clusters of the vision graph. The proposed protocol addresses cluster formation, merging, splitting, and interaction in a distributed camera network. An alternate method for distributed feature matching (which implicitly defines a vision graph) was described by Avidan et al. [4], who used a probabilistic argument based on random graphs to analyze the propagation of wide-baseline stereo matching results obtained for a small number of image pairs to the remaining cameras. Farrell et al. [27] described a Bayesian framework for learning higher-order transition models in camera networks. That is, in addition to estimating the adjacency of cameras, the method can help answer questions such as “what fraction of objects passing through cameras X and Y will at some later time reach camera Z?”. They claimed that the higher-order models can improve tracking and inference performance. It would be interesting to see the implementation of this idea in a real network of cameras.
3.2 Overlapping Topology Estimation When the images from the cameras overlap, the topology estimation problem is often combined with the calibration problem discussed in the next section. Here we mention several techniques that “stop” at the point of topology estimation. Kulkarni et al. [43] proposed a method using the time-stamped appearances of a moving reference object to provide an approximate initialization for the cameras’ regions of overlap, which is indirectly related to their topology. Similar approaches applied to the camera calibration problem are discussed in the next section. However, for large-scale visual sensor networks, we believe it is important to develop techniques in which neither the cameras nor the scene are altered from their natural state, with no assumptions about camera synchronization or interactions between the user and the environment. Cheng et al. [17] described an algorithm for estimating the vision graph based on a fixed-length “feature digest” of automatically detected distinctive keypoints that each camera broadcasts to the rest of the network. These keypoints arise from the popular and successful Scale-Invariant Feature Transform (SIFT) detec-
42
Richard J. Radke
tor/descriptor proposed by Lowe [51]. Each sending camera makes a tradeoff between the number of keypoints it can send and how much their descriptors are compressed, assuming a fixed bit budget for their message. Each receiver camera decompresses the feature digest to recover the approximate feature descriptors, which are matched with its own features to generate putative matches. If enough matches are found, a vision graph edge is established. They analyzed the tradeoffs between the size of the feature digest, the number of transmitted features, the level of compression, and the overall performance of the edge generation. This method is appealing since it requires no user interaction with the environment, using static scene features as the basis for edge generation.
4 Camera Network Calibration Formally, camera calibration is the estimation of the parameters that describe how a perspective camera projects 3-D points in a fixed world coordinate system to 2-D points on an image plane. These calibration parameters can be divided into four or more internal parameters related to a camera’s optics, such as its principal point, aspect ratio, lens distortion, and focal length, and six external parameters, namely the rotation angles and translation vector that relate the camera coordinate frame to the world coordinate frame. Since the fixed internal parameters of a given camera can usually be estimated individually prior to deployment [19, 30, 85], most research is mainly concerned with estimating the external parameters. The classical problem of externally calibrating a pair of cameras is well-understood [91]; the parameter estimation usually requires a set of feature point correspondences in both images. When no points with known 3-D locations in the world coordinate frame are available, the cameras can be calibrated up to a similarity transformation [31]. That is, the cameras’ positions and orientations can be accurately estimated relative to each other, but not in an absolute sense. Without metric information about the scene, an unknown scale parameter also remains; for example, the same set of images would be produced by cameras twice as far away from a scene that is twice as large. Multi-camera calibration can be accomplished by minimizing a nonlinear cost function of the calibration parameters and a collection of unknown 3-D scene points projecting to matched image correspondences; this problem is also known as Structure from Motion (SFM). The optimization process for estimating the parameters is called bundle adjustment [90]. Good results are achievable when the images and correspondences are all accessible to a powerful, central processor. It is beyond the scope of this chapter to survey the centralized SFM problem here, but the book by Hartley and Zisserman [31] is an excellent reference. Also, Ma et al. [53] outlined a complete step-by-step recipe for the SFM problem, assuming the information from all cameras is collected in one place. We note that SFM is closely related to a problem in the robotics community called Simultaneous Localization and Mapping
A Survey of Distributed Computer Vision Algorithms
43
(SLAM) [49, 88], in which mobile robots must estimate their locations from sensor data as they move through a scene.
4.1 Non-overlapping Camera Calibration The algorithms discussed in this section generally assume a camera network with overlapping fields of view, as discussed in Section 3.2. In the non-overlapping camera case, estimating the actual 3D positions of the cameras based only on their visual information is generally a poorly-posed problem. That is, multiple cameras must actually observe many of the same 3D points as the basis for accurate estimation of localization parameters. However, several researchers have proposed solutions that make assumptions about the dynamics of objects transiting the environment to get rough estimates of camera positions and orientations. In this area, the work by Rahimi et al. [77] is the most notable. They encoded prior knowledge about the state (location and velocity) of targets as a linear Gaussian Markov model, and then combined this prior with measurements from the camera network to produce maximum a posteriori estimates of global trajectories and hence camera calibration parameters. The approach was illustrated with networks of four to ten downward-facing cameras observing a single target moving on a ground plane. It would be interesting to see how this method generalizes to large numbers of cameras in a fully 3-D environment. However, it is not reasonable to expect that highly accurate calibration parameters could be obtained with this type of approach using naturally-moving objects in a real-world, well-separated camera network due to difficult-to-model dynamics and unmodeled sources and sinks. Rekleitis et al. [80, 79] proposed the use of a mobile robot holding a calibration pattern (a checkboard or an array of unique fiducial markers) as the basis for nonoverlapping camera calibration. The robot’s pose on the ground plane, obtained from odometry, is used to provide each camera with the 3D positions of points on the calibration target.
4.2 Overlapping Camera Calibration Several methods, while still centralized, have used visual sensor networks as the motivation for the design of calibration algorithms. These algorithms are usually based on different cameras’ observations of the same object or objects in their environment. For this purpose, several researchers have suggested modulated light sources moved throughout a darkened room [5, 16, 7] or placed on the cameras themselves [86]. Maas [54] used a moving reference bar of known length, while Baker and Aloimonos [6] used grids of parallel lines and square boxes. Liu et al. [50] proposed an algorithm in which each camera independently calibrates itself based on images
44
Richard J. Radke
of a placed reference device with known world coordinates. As mentioned previously, algorithms that require no user interaction with the environment are generally preferable for large-scale ad-hoc deployments. Similar to the non-overlapping topology estimation problem, several researchers have proposed to obtain corresponding points for calibration by matching trajectories of objects that naturally move through the cameras’ environment. Lee et al. [47] described one such approach in which pedestrians and vehicles moving on a ground plane were used to calibrate two or three overlapping surveillance cameras. Other approaches along this line were proposed by Jaynes [41] and Meingast et al. [63]. Several researchers have addressed distributed versions of this type of camera network calibration. For example, Lee and Aghajan [46] used the joint observations of a moving target to solve a set of nonlinear equations for its position (and hence, the camera positions) using a distributed version of the Gauss-Newton method. Another excellent algorithm of this type called SLAT (Simultaneous Location and Tracking) was proposed by Funiak et al. [28]. The key innovations were a relative overparameterization of the cameras that allows Gaussian densities to be used, and a linearization procedure to address the uncertainty in camera angle for proper use of a Kalman filter. It would be very interesting to see how such an approach would scale to full calibration of large outdoor camera networks with many moving objects. Many distributed algorithms for full 3D camera network calibration rely on using image correspondences to estimate the epipolar geometry between image pairs. When the intrinsic camera parameters are known, this information can be used to extract estimates of the rotation and translation between each camera pair. For example, Barton-Sweeney et al. [7] used an Extended Kalman Filtering framework on the estimated epipolar geometry to estimate the rotation and translations between a camera pair. Mantzel et al. [57] proposed an algorithm called DALT (Distributed Alternating Localization and Triangulation) for camera network calibration. As the the title suggests, each camera calibrates itself by alternating between 1) triangulating 2D image projections into 3D space, assuming the camera matrices are known, and 2) estimating the camera matrices based on putative image-to-world correspondences. While their optimization approach is simpler than bundle adjustment, this work includes some interesting analysis of the computational and power requirements that would be incurred by the algorithm on a realistic platform. These algorithms used a user-inserted calibration object as the basis for establishing point correspondences. Devarajan et al. [21, 23] proposed a calibration algorithm in which each camera only communicates with the cameras connected to it by an edge in the vision graph, exchanging image projections of common 3D points. After performing a local projective-to-metric factorization and bundle adjustment, each camera has an estimate of 1) its own location, orientation, and focal length, 2) the corresponding parameters for each of its neighbors in the vision graph, and 3) the 3D positions of the image feature points it has in common with its neighbors. They observed that the distributed calibration algorithm not only resulted in good calibration accuracy compared to centralized bundle adjustment, but also that the messages per
A Survey of Distributed Computer Vision Algorithms
45
communication link required to calibrate the network were more fairly distributed, implying a longer node lifetime. Medeiros et al. [62] proposed a distributed cluster-based calibration algorithm for which they observed a similar result: the distributed algorithm required significantly fewer transmit and receive bits than for a centralized algorithm, while having comparable accuracy. However, their approach makes a restrictive assumption that distinctive target objects with fixed features at a known scale are used for calibration. We note that Brand et al. [13] presented a novel approach to large-scale camera network calibration using a graph embedding formalism. They first estimated nodeto-node direction vectors in 3D using corresponding features and vanishing points (the signs and lengths of the vectors were unknown). They then showed how the solution to a simple eigenvalue problem resulted in the embedding of 3D camera locations that was maximally consistent with the measured directions. The algorithm is remarkably fast and accurate, albeit centralized. It would be very interesting to investigate whether the eigenvalue problems can be solved in a distributed manner.
4.3 Improving Calibration Consistency Many researchers have proposed methods for dealing with inconsistent parameter estimates in wireless sensor networks. These are typically based on verifying whether estimates of the same parameter are sufficiently close [84, 14, 95] and/or consistent with prior knowledge [25]. Most such approaches are well-suited for sensors that measure a scalar quantity such as temperature or pressure. In contrast, camera networks contain inconsistencies in a continuous, high-dimensional parameter space, and require principled statistical methods for resolving them. Devarajan and Radke [22] proposed a calibration refinement method based on belief propagation [67], a method for probabilistic inference in networks that several researchers have recently applied to discrete or low-dimensional sensor networking and robotics problems (e.g., [93, 18, 87, 36]). The belief propagation algorithm involved passing messages about common camera parameters along the edges of the vision graph that iteratively brought inconsistent estimates of camera parameters closer together. They discussed solutions to several challenges unique to the camera calibration problem. They showed that the inconsistency in camera localization was reduced by factors of 2 to 6 after applying their belief propagation algorithm, while still maintaining high accuracy in the parameter estimates. Several researchers have recently applied more sophisticated and robust distributed inference algorithms than belief propagation to sensor fusion problems; the work of Paskin, Guestrin, and McFadden [71, 72] and Dellaert et al. [20] is notable. Future distributed camera calibration techniques could benefit from these new algorithms. Some researchers have suggested improvements in calibration by augmenting the input from visual sensors with non-visual sensor measurements. For example, Mein-
46
Richard J. Radke
gast et al. [64] fused image data with radio interferometry to estimate visual sensors’ position and orientation. However, this method relied on centralized processing.
5 Tracking and Classification A visual sensor network is usually calibrated for an application-level purpose. Commonly, the camera network is tasked to track and/or classify objects as they move through the environment. A massive literature on single-camera or centralized multicamera tracking exists; here we focus on distributed algorithms for the tracking problem designed with sensor networking considerations in mind. Rao and Durrant-Whyte [78] presented early work that recognized the importance of a decentralized algorithm for target tracking. They proposed a Bayesian approach, but required a fully connected network topology that is unrealistic for modern sensor networks. Hoffmann et al. [35] described a distributed algorithm for multi-camera tracking, modeled as a finite state machine. The proposed algorithm was validated in a simulated network in which it was assumed that object detection results were entirely accurate. Dockstader and Tekalp [24] presented an algorithm for distributed real-time multi-object tracking, based on a Bayesian belief network formalism for fusing observations with various confidence estimates. The basic strategy is for each camera to track each object independently in 2D, apply a distributed fusion/triangulation algorithm to obtain a 3D location, and then independently apply a 3D Kalman filter at each camera. Results were demonstrated for a three-camera network. Qu et al. [73] proposed a Bayesian algorithm for distributed multi-target tracking using multiple collaborative cameras. Their approach was based on a dynamic graphical model that updates estimates of objects’ 2D positions instead of tracking in 3D. The algorithm is well-suited for challenging environments such as crowds of similar pedestrians. Heath and Guibas [33] described an algorithm for multi-person tracking in a network of stereo (i.e., two closely-separated) cameras. The use of stereo cameras enables each node to estimate the trajectories of 3D points on each moving object. Each node communicates these feature positions to its neighbors and performs independent 3D particle filtering based on its received messages. They demonstrated results on small networks of real cameras as well as a larger simulated camera network observing a crowded scene. They analyzed the tradeoffs relating the communication budget for each camera, the number of tracked people in the environment, and the tracking precision. Heath and Guibas [32] also described an algorithm for tracking people and acquiring canonical (i.e., frontal-facing) face images in a camera network. They proposed a tracking algorithm based on 2D visual hulls synthesized at clusterhead nodes, and used the direction of pedestrian motion as a simple estimate for head orientation. The idea seems promising and a more fully distributed system would be interesting.
A Survey of Distributed Computer Vision Algorithms
47
Mensink et al. [65] described a multi-camera tracking algorithm in which the observation at each camera was modeled as a mixture of Gaussians, with each Gaussian representing the appearance features of a single person. The authors extended the gossip-based Newscast EM algorithm [42] discussed in Section 2 to incorporate multiple observations into the iterative, distributed computation of means, covariances, and mixing weights in the Gaussian mixture model. The preliminary results showed that the distributed estimation algorithm converged as quickly and accurately as a centralized EM algorithm. This is an excellent example of a distributed computer vision algorithm. Leistner et al. [48] described an interesting technique for automatic person detection that applied an online learning method called co-training to improve the local detection results. Cameras viewing the same scene exchange image patches that they are confident to contain positive and negative examples of an object, comprising a few hundred bytes per frame. They implemented the algorithm on smart cameras and demonstrated that the approach resulted in increasingly better detection results. The problem of matching objects between images from different cameras in a visual sensor network is sometimes called reacquisition or reidentification, and often results in a handoff of the responsibility of object tracking from one camera to another. Quaritsch et al. [74] described a distributed object tracking algorithm that illustrates the main ideas of decentralized tracking handoff. The basic idea is that the responsibility for tracking each object lies with a particular master camera. When the object nears the periphery of this camera’s field of view, neighboring cameras are put into slave mode, searching for the object until it appears. When a slave discovers the object in its field of view, it becomes the new master. Arth et al. [3] proposed an algorithm for reacquisition based on signatures, or compact representations of local object features. The signatures are much less expensive to transmit through the network than the original images. The idea has some resemblance to the feature digest of Cheng et al. [17] discussed in Section 3. The algorithm was demonstrated on a simulated traffic camera network and its performance and communication properties with respect to noise analyzed in some detail. Park et al. [70] proposed an algorithm for the distributed construction of look-up tables stored locally at each node that are used to select the cameras that are most likely to observe a specific location. These lookup tables could be used to improve camera handoff for multiple moving targets. Qureshi and Terzopoulos [75] described distributed algorithms for segmentation, visual tracking, camera assignment, and handoff in the context of a large-scale virtual camera network in a train station environment. Objects are detected and matched between cameras that are not assumed to have uniform imaging characteristics using a color indexing method based on histogram backprojection. Iwaki et al. [37] pointed out the importance of developing tracking algorithms for large-scale camera networks that did not assume synchronized image capture. They presented an algorithm based on evidence accumulation that operates in a hierarchical manner. Cameras individually detect human heads in the scene and send the results to clusterheads for outlier detection and localization refinement.
48
Richard J. Radke
Some researchers have suggested improvements in tracking by augmenting the input from visual sensors with non-visual sensor measurements. For example, Miyaki et al. [66] fused image tracking information with the signal strength from pedestrians’ Wi-Fi mobile devices to better estimate their locations.
6 Conclusions The past five years have seen an encouraging upturn in the development of distributed solutions to classical computer vision problems. We believe that there are many viable distributed alternatives for accurately estimating transition models and visual overlap, 3D calibration parameters, and 2D and 3D object tracks. The highest-level algorithms surveyed here emphasize simple tracking rather than the estimation of more subtle object/environment characteristics. We view this as a natural next step in the development of distributed computer vision algorithms. As an early example, Chang and Aghajan [15] described fusion algorithms based on linear quadratic regression and Kalman filtering to estimate the orientation of faces imaged by a camera network. Most algorithms also assume that the camera network is entirely engaged in performing the same vision task at every node, and that all tasks have the same priority. However, for distributed wireless camera networks deployed in the real world, it will be important to consider calibration and other computer vision tasks in the context of the many other tasks the node processors and network need to undertake. This will entail a tight coupling between the vision algorithms and the MAC, network, and link-layer protocols, organization, and channel conditions of the network, as well as the power supply, transmitter/receiver and kernel scheduler of each node. This tight integration would be critical for making the algorithms described here viable for embedded platforms, such as networks of cell-phone cameras [10]. Bramberger et al. [12] recently described an algorithm for allocating general tasks in a camera network, posing a distributed constraint satisfaction problem. When a smart camera poses a new task, the request is cycled through the cameras in the network, incrementally merging partial task allocations until the request arrives back at the originating camera. Finally, we note that prototyping vision algorithms in a real network of cameras is a time-, labor-, and money-consuming proposition. Visually accurate, large-scale camera network simulation systems such as those developed by Qureshi and Terzopoulos [75] or Heath and Guibas [33] will enable rapid prototyping of distributed computer vision algorithms without requiring the resources for an actual deployment.
A Survey of Distributed Computer Vision Algorithms
49
References [1] Aghajan H, Kleihorst R, Rinner B, Wolf W (eds) (2008) IEEE Journal of Selected Topics in Signal Processing Special Issue on Distributed Processing in Vision Networks, vol 2. IEEE [2] Ai J, Abouzeid A (2006) Coverage by directional sensors in randomly deployed wireless sensor networks. Journal of Combinatorial Optimization 11(1):21 – 41 [3] Arth C, Leistner C, Bischof H (2007) Object reacquisition and tracking in large-scale smart camera networks. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 156–163 [4] Avidan S, Moses Y, Moses Y (2007) Centralized and distributed multi-view correspondence. International Journal of Computer Vision 71(1):49–69 [5] Baker P, Aloimonos Y (2000) Complete calibration of a multi-camera network. In: Proceedings of IEEE Workshop on Omnidirectional Vision 2000 [6] Baker P, Aloimonos Y (2003) Calibration of a multicamera network. In: Proceedings of the IEEE Workshop on Omnidirectional Vision [7] Barton-Sweeney A, Lymberopoulos D, Savvides A (2006) Sensor localization and camera calibration in distributed camera sensor networks. In: Proceedings of the 3rd International Conference on Broadband Communications, Networks and Systems (BROADNETS), San Jose, CA, USA, pp 1–10 [8] Bash BA, Desnoyers PJ (2007) Exact distributed Voronoi cell computation in sensor networks. In: International Conference on Information Processing in Sensor Networks [9] Bertsekas DP, Tsitsiklis JN (1997) Parallel and Distributed Computation: Numerical Methods. Athena Scientific [10] Bolliger P, Köhler M, Römer K (2007) Facet: Towards a smart camera network of mobile phones. In: Proceedings of Autonomics 2007 (ACM First International Conference on Autonomic Computing and Communication Systems), Rome, Italy [11] Boyd S, Ghosh A, Prabhakar B, Shah D (2005) Gossip algorithms: design, analysis and applications. In: IEEE INFOCOM, vol 3, pp 1653–1664 [12] Bramberger M, Doblander A, Maier A, Rinner B, Schwabach H (2006) Distributed embedded smart cameras for surveillance applications. Computer 39(2):68–75 [13] Brand M, Antone M, Teller S (2004) Spectral solution of large-scale extrinsic camera calibration as a graph embedding problem. In: Proceedings of the European Conference on Computer Vision [14] Capkun S, Hubaux JP (2005) Secure positioning of wireless devices with application to sensor networks. In: Proceedings of IEEE INFOCOM [15] Chang CC, Aghajan H (2006) Collaborative face orientation detection in wireless image sensor networks. In: Workshop on Distributed Smart Cameras [16] Chen X, Davis J, Slusallek P (2000) Wide area camera calibration using virtual calibration objects. In: IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition
50
Richard J. Radke
[17] Cheng Z, Devarajan D, Radke R (2007) Determining vision graphs for distributed camera networks using feature digests. EURASIP Journal of Applied Signal Processing, Special Issue on Visual Sensor Networks Article ID 57034 [18] Christopher C, Avi P (2003) Loopy belief propagation as a basis for communication in sensor networks. In: Proceedings of the 19th Annual Conference on Uncertainty in Artificial Intelligence (UAI-03), Morgan Kaufmann Publishers, San Francisco, CA, pp 159–166 [19] Collins RT, Tsin Y (1999) Calibration of an outdoor active camera system. In: IEEE Computer Vision and Pattern Recognition 1999 [20] Dellaert F, Kipp A, Krauthausen P (2005) A multifrontal QR factorization approach to distributed inference applied to multirobot localization and mapping. In: Proceedings of National Conference on Artificial Intelligence (AAAI 05), pp 1261–1266 [21] Devarajan D, Radke R (2004) Distributed metric calibration for large-scale camera networks. In: Proceedings of the First Workshop on Broadband Advanced Sensor Networks (BASENETS) 2004 (in conjunction with BroadNets 2004), San Jose, CA [22] Devarajan D, Radke R (2007) Calibrating distributed camera networks using belief propagation. EURASIP Journal of Applied Signal Processing, Special Issue on Visual Sensor Networks Article ID 60696 [23] Devarajan D, Radke R, Chung H (2006) Distributed metric calibration of adhoc camera networks. ACM Transactions on Sensor Networks 2(3):380–403 [24] Dockstader S, Tekalp A (2001) Multiple camera tracking of interacting and occluded human motion. Proceedings of the IEEE 89(10):1441–1455 [25] Du W, Fang L, Peng N (2006) LAD: localization anomaly detection for wireless sensor networks. J Parallel Distrib Comput 66(7):874–886 [26] Farrell R, Davis L (2008) Decentralized discovery of camera network topology. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [27] Farrell R, Doermann D, Davis L (2007) Learning higher-order transition models in medium-scale camera networks. In: IEEE 11th International Conference on Computer Vision [28] Funiak S, Guestrin C, Paskin M, Sukthankar R (2006) Distributed localization of networked cameras. In: Proceedings of the Fifth International Conference on Information Processing in Sensor Networks [29] Haas Z, et al (2002) Wireless ad hoc networks. In: Proakis J (ed) Encyclopedia of Telecommunications, John Wiley [30] Hartley R (1994) Self-calibration from multiple views with a rotating camera. In: Proc. ECCV ’94, vol 1, pp 471–478 [31] Hartley R, Zisserman A (2000) Multiple View Geometry in Computer Vision. Cambridge University Press [32] Heath K, Guibas L (2007) Facenet: Tracking people and acquiring canonical face images in a wireless camera sensor network. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 117–124
A Survey of Distributed Computer Vision Algorithms
51
[33] Heath K, Guibas L (2008) Multi-person tracking from sparse 3D trajectories in a camera sensor network. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [34] van den Hengel A, Dick A, Detmold H, Cichowski A, Hill R (2007) Finding camera overlap in large surveillance networks. In: Proceedings of the 8th Asian Conference on Computer Vision [35] Hoffmann M, Hähner J (2007) Rocas: A robust online algorithm for spatial partitioning in distributed smart camera systems. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 267–274 [36] Ihler A, Fisher J, Moses R, Willsky A (2004) Nonparametric belief propagation for self-calibration in sensor networks. In: Proceedings of the 3rd International Symposium on Information Processing in Sensor Networks, Berkeley, CA [37] Iwaki H, Srivastava G, Kosaka A, Park J, Kak A (2008) A novel evidence accumulation framework for robust multi-camera person detection. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [38] Iyengar R, Sikdar B (2003) Scalable and distributed GPS free positioning for sensor networks. In: IEEE International Conference on Communications, vol 1, pp 338–342 [39] Javed O, Rasheed Z, Shafique K, Shah M (2003) Tracking across multiple cameras with disjoint views. In: Proceedings of the 9th International Conference on Computer Vision, Nice, France [40] Javed O, Shafique K, Shah M (2005) Appearance modeling for tracking in multiple non-overlapping cameras. In: The IEEE Computer Society Conference on Computer Vision and Pattern Recognition [41] Jaynes C (1999) Multi-view calibration from planar motion for video surveillance. In: Second IEEE Workshop on Visual Surveillance (VS’99), pp 59–66 [42] Kowalczyk W, Vlassis N (2004) Newscast EM. In: Neural Information Processing Systems Conference [43] Kulkarni P, Shenoy P, Ganesan D (2007) Approximate initialization of camera sensor networks. In: Proceedings of the 4th European Conference on Wireless Sensor Networks [44] Kundur D, Lin CY, Lu CS (eds) (2007) EURASIP Journal on Advances in Signal Processing Special Issue on Visual Sensor Networks. Hindawi Publishing Corporation [45] Langendoen K, Reijers N (2003) Distributed localization in wireless sensor networks: A quantitative comparison. Computer Networks 43(4):499–518 [46] Lee H, Aghajan H (2006) Collaborative node localization in surveillance networks using opportunistic target observations. In: Proceedings of the 4th ACM International Workshop on Video Surveillance and Sensor Networks, pp 9–18 [47] Lee L, Romano R, Stein G (2000) Monitoring activities from multiple video streams: Establishing a common coordinate frame. IEEE Transactions on Pattern Recognition and Machine Intelligence 22(8)
52
Richard J. Radke
[48] Leistner C, Roth P, Grabner H, Bischof H, Starzacher A, Rinner B (2008) Visual on-line learning in distributed camera networks. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [49] Leonard J, Durrant-Whyte H (1991) Simultaneous map building and localization for an autonomous mobile robot. In: Proceedings of the IEEE/RSJ International Workshop on Intelligent Robots and Systems, pp 1442–1447 [50] Liu X, Kulkarni P, Shenoy P, Ganesan D (2006) Snapshot: A self-calibration protocol for camera sensor networks. In: Proceedings of IEEE/CreateNet BASENETS 2006 [51] Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2):91–110 [52] Lynch N (1996) Distributed Algorithms. Morgan Kaufmann [53] Ma Y, Soatto S, Košecká J, Sastry S (2004) An Invitation to 3-D Vision: From Images to Geometric Models, Springer, chap 11 [54] Maas HG (1999) Image sequence based automatic multi-camera system calibration techniques. ISPRS Journal of Photogrammetry and Remote Sensing 54(5-6):352–359 [55] Makris D, Ellis T, Black J (2004) Bridging the gaps between cameras. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition [56] Mandel Z, Shimshoni I, Keren D (2007) Multi-camera topology recovery from coherent motion. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 243–250 [57] Mantzel W, Choi H, Baraniuk R (2004) Distributed camera network localization. In: Proceedings of the 38th Asilomar Conference on Signals, Systems and Computers [58] Marinakis D, Dudek G (2005) Topology inference for a vision-based sensor network. In: In Proc. of Canadian Conference on Computer and Robot Vision [59] Marinakis D, Dudek G (2006) Probabilistic self-localization for sensor networks. In: AAAI National Conference on Artificial Intelligence [60] Marinakis D, Dudek G (2008) Occam’s razor applied to network topology inference. IEEE Transactions on Robotics 24(2):293 – 306 [61] Medeiros H, Park J, Kak A (2007) A light-weight event-driven protocol for sensor clustering in wireless camera networks. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 203–210 [62] Medeiros H, Iwaki H, Park J (2008) Online distributed calibration of a large network of wireless cameras using dynamic clustering. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [63] Meingast M, Oh S, Sastry S (2007) Automatic camera network localization using object image tracks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshop on Visual Representations and Modeling of Large-scale Environments
A Survey of Distributed Computer Vision Algorithms
53
[64] Meingast M, Kushwaha M, Oh S, Koutsoukos X, Ledeczi A, Sastry S (2008) Heterogeneous camera network localization using data fusion. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [65] Mensink T, Zajdel W, Krose B (2007) Distributed EM learning for appearance based multi-camera tracking. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 178–185 [66] Miyaki T, Yamasaki T, Aizawa K (2007) Multi-sensor fusion tracking using visual information and wi-fl location estimation. In: First ACM/IEEE International Conference on Distributed Smart Cameras, pp 275–282 [67] Murphy KP, Weiss Y, Jordan MI (1999) Loopy belief propagation for approximate inference: An empirical study. In: Proceedings of Uncertainty in Artificial Intelligence (UAI ’99), pp 467–475 [68] Niu C, Grimson E (2006) Recovering non-overlapping network topology using far-field vehicle tracking data. In: Proceedings of the 18th International Conference on Pattern Recognition [69] Nowak R (2003) Distributed EM algorithms for density estimation and clustering in sensor networks. IEEE Transactions on Signal Processing 51(8):2245– 2253 [70] Park J, Bhat PC, Kak AC (2006) A look-up table based approach for solving the camera selection problem in large camera networks. In: Workshop on Distributed Smart Cameras [71] Paskin MA, Guestrin CE (2004) Robust probabilistic inference in distributed systems. In: Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence (UAI ’04), pp 436–445 [72] Paskin MA, Guestrin CE, McFadden J (2005) A robust architecture for inference in sensor networks. In: 4th International Symposium on Information Processing in Sensor Networks (IPSN ’05) [73] Qu W, Schonfeld D, Mohamed M (2007) Distributed Bayesian multipletarget tracking in crowded environments using multiple collaborative cameras. EURASIP Journal on Advances in Signal Processing 2007(38373) [74] Quaritsch M, Kreuzthaler M, Rinner B, Bischof H, Strobl B (2007) Autonomous multicamera tracking on embedded smart cameras. EURASIP Journal on Advances in Signal Processing 2007(92827) [75] Qureshi F, Terzopoulos D (2008) Smart camera networks in virtual reality. Proceedings of the IEEE 96(10):1640–1656 [76] Rabbat M, Nowak R (2004) Distributed optimization in sensor networks. In: International Conference on Information Processing in Sensor Networks [77] Rahimi A, Dunagan B, Darrell T (2004) Simultaneous calibration and tracking with a network of non-overlapping sensors. In: The IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 1, pp 187–194 [78] Rao B, Durrant-Whyte H (1993) A decentralized Bayesian algorithm for identification of tracked targets. IEEE Transactions on Systems, Man and Cybernetics 23(6):1683–1698
54
Richard J. Radke
[79] Rekleitis I, Meger D, Dudek G (2006) Simultaneous planning, localization, and mapping in a camera sensor network. Robotics and Autonomous Systems 54(11):921–932 [80] Rekleitis IM, Dudek G (2005) Automated calibration of a camera sensor network. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 401–406 [81] Rinner B, Wolf W (2008) An introduction to distributed smart cameras. Proceedings of the IEEE 96(10):1565–1575 [82] Rinner B, Wolf W (eds) (2008) Proceedings of the IEEE Special Issue on Distributed Smart Cameras, vol 96. IEEE [83] Rinner B, Winkler T, Schriebl W, Quaritsch M, Wolf W (2008) The evolution from single to pervasive smart cameras. In: Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford University, Palo Alto, CA, USA [84] Sastry N, Shankar U, Wagner D (2003) Secure verification of location claims. In: Proceedings of WiSe ’03, the 2nd ACM workshop on Wireless Security, pp 1–10 [85] Stein GP (1995) Accurate internal camera calibration using rotation, with analysis of sources of error. In: ICCV, pp 230–236 [86] Taylor CJ, Shirmohammadi B (2006) Self localizing smart camera networks and their applications to 3D modeling. In: Proceedings of the International Workshop on Distributed Smart Cameras [87] Thorpe J, McEliece R (2002) Data Fusion Algorithms for Collaborative Robotic Exploration. Interplanetary Network Progress Report 149:1–14 [88] Thrun S, Burgard W, Fox D (2005) Probabilistic Robotics: Intelligent Robotics and Autonomous Agents. MIT Press [89] Tieu K, Dalley G, Grimson W (2005) Inference of non-overlapping camera network topology by measuring statistical dependence. In: Tenth IEEE International Conference on Computer Vision, vol 2, pp 1842–1849 [90] Triggs B, McLauchlan P, Hartley R, Fitzgibbon A (2000) Bundle adjustment – A modern synthesis. In: Triggs W, Zisserman A, Szeliski R (eds) Vision Algorithms: Theory and Practice, LNCS, Springer Verlag, pp 298–375 [91] Tsai R (1992) A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. In: Wolff L, Shafer S, Healey G (eds) Radiometry – (Physics-Based Vision), Jones and Bartlett [92] Varshney PK (1997) Distributed Detection and Data Fusion. Springer [93] Venkatesh S, Alanyali M, Savas O (2006) Distributed detection in sensor networks with packet losses and finite capacity links. IEEE Transactions on Signal Processing 54(11):4118–4132 [94] Wang B, Sung K, Ng T (2002) The localized consistency principle for image matching under non-uniform illumination variation and affine distortion. In: European Conference on Computer Vision, vol LNCS 2350, pp 205–219, copenhagen, Denmark
A Survey of Distributed Computer Vision Algorithms
55
[95] Wei Y, Yu Z, Guan Y (2007) Location verification algorithms for wireless sensor networks. In: Proceedings of the 27th International Conference on Distributed Computing Systems, p 70 [96] Zou X, Bhanu B, Song B, Roy-Chowdhury A (2007) Determining topology in a distributed camera network. In: Proceedings of the IEEE International Conference on Image Processing
Video-Based People Tracking Marcus A. Brubaker, Leonid Sigal and David J. Fleet
1 Introduction Vision-based human pose tracking promises to be a key enabling technology for myriad applications, including the analysis of human activities for perceptive environments and novel man-machine interfaces. While progress toward that goal has been exciting, and limited applications have been demonstrated, the recovery of human pose from video in unconstrained settings remains challenging. One of the key challenges stems from the complexity of the human kinematic structure itself. The sheer number and variety of joints in the human body (the nature of which is an active area of biomechanics research) entails the estimation of many parameters. The estimation problem is also challenging because muscles and other body tissues obscure the skeletal structure, making it impossible to directly observe the pose of the skeleton. Clothing further obscures the skeleton, and greatly increases the variability of individual appearance, which further exacerbates the problem. Finally, the imaging process itself produces a number of ambiguities, either because of occlusion, limited image resolution, or the inability to easily discriminate the parts of a person from one another or from the background. Some of these issues are inherent, yielding ambiguities that can only be resolved with prior knowledge; others lead to computational burdens that require clever engineering solutions. The estimation of 3D human pose is currently possible in constrained situations, for example with multiple cameras, with little occlusion or confounding background clutter, or with restricted types of movement. Nevertheless, despite a decade of acMarcus A. Brubaker Department of Computer Science, University of Toronto, e-mail:
[email protected]. edu Leonid Sigal Department of Computer Science, University of Toronto, e-mail:
[email protected] David J. Fleet Department of Computer Science, University of Toronto, e-mail:
[email protected] H. Nakashima et al. (eds.), Handbook of Ambient Intelligence and Smart Environments, DOI 10.1007/978-0-387-93808-0_3, © Springer Science+Business Media, LLC 2010
57
58
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
(a)
(b)
(c)
(d)
(e)
Fig. 1 Challenges in human pose estimation. Variation in body size and shape (a), occlusions of body parts (b), inability to observe the skeletal motion due to clothing (c), difficulty segmenting the person from the background (d), and complex interactions between people in the environment (e), are challenges that plague the recovery of human pose in unconstrained scenes.
tive research, monocular 3D pose tracking remains largely unsolved. From a single view it is hard to escape ambiguities in depth and scale, reflection ambiguities where different 3D poses produce similar images, and missing observations of certain parts of the body because of self-occlusions. This chapter introduces the basic elements of modern approaches to pose tracking. We focus primarily on monocular pose tracking with a probabilistic formulation. While multiview tracking in constrained settings, e.g., with minimal occlusion, may be relatively straightforward [29, 10] the problems faced in monocular tracking often arise in the general multiview case as well. This chapter is not intended to be a thorough review of human tracking but rather a tutorial introduction for practitioners interested in applying vision-based human tracking systems. For a more exhaustive review of the literature we refer readers to [17, 38].
1.1 Tracking as Inference Because of the inescapable uncertainty that arises due to ambiguity, and the prevalence of noisy or missing observations of body parts, it has become common to formulate human pose tracking in probabilistic terms. As such, the goal is to determine the posterior probability distribution over human poses or motions, conditioned on the image measurements (or observations). Formally, let st denote the state of the body at time t. It represents the unknown parameters of the model we wish to estimate. In our case it typically comprises the joint angles of the body along with the position and orientation of the body in world coordinates. We also have observations at each time, denoted zt . This might simply be the image at time t or it might be a set of image measurements (e.g., edge locations or optical flow). Tracking can then be formulated as the problem of inferring the probability distribution over state sequences, s1:t = (s1 , . . . , st ), conditioned on the observation history, z1:t = (z1 , . . . , zt ); that is, p(s1:t |z1:t ) . Using Bayes’ rule, it is common to express the posterior distribution as
Video-Based People Tracking
59
p(s1:t |z1:t ) =
p(z1:t |s1:t )p(s1:t ) . p(z1:t )
(1)
Here, p(z1:t |s1:t ) is called the likelihood. It is the probability of observing the image measurements given a state sequence. In effect the likelihood provides a measure of the consistency between a hypothetical motion and the given image measurements. The other major factor in (1) is the prior probability of the state sequence, p(s1:t ). This prior distribution captures whether a given motion is plausible or not. During pose tracking we aim to find motions that are both plausible and consistent with the image measurements. Finally, the denominator in (1), p(z1:t ), often called the partition function, does not depend on the state sequence, and is therefore considered to be constant for the purposes of this chapter. To simplify the task of approximating the posterior distribution over human motion (1), or of finding the most probable motion (i.e., the MAP estimate), it is common to assume that the likelihood and prior models can be factored further. For example, it is common to assume that the observations at each time are independent given the states. This allows the likelihood to be rewritten as a product of simpler likelihoods, one at each time: p(z1:t |s1:t ) =
t
∏ p(zi |si ) .
(2)
i=1
This assumption and resulting factorization allows for more efficient inference and easier specification of the likelihood. Common measurement models and likelihood functions are in Section 3. The prior distribution over human motion also plays a key role. In particular, ambiguities and noisy measurements often necessitate a prior model to resolve uncertainty. The prior model typically involves a specification of which poses are plausible or implausible, and which sequences of poses are plausible. Often this involves learning dynamical models from training data. This is discussed in Section 4. The last two elements in a probabilistic approach to pose tracking are inference and initialization. Inference refers to the process of finding good computational approximations to the posterior distribution, or to motions that are most probable. This is discussed in Section 5. Furthermore, tracking most often requires a good initial guess for the pose at the first frame, to initialize the inference. Section 6 discusses methods for automatic initialization of tracking and for recovery from tracking failures.
2 Generative Model for Human Pose To begin to formulate pose tracking in more detail, we require a parameterization of human pose. While special parameterizations might be required for certain tasks, most approaches to pose tracking assume an articulated skeleton, comprising connected, rigid parts. We also need to specify the relation between this skeleton and
60
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
pr
οi
skeleton
skin & soft tissue
clothing
image
Fig. 2 Simple image formation model for a human pose. The skeleton of the human body is overlaid with soft tissue and clothing. Furthermore, the formation of an image of the person (on the right) also depends on lighting and camera parameters.
the image observations. This is complex since we do not observe the skeleton directly. Rather, as illustrated in Figure 2, the skeleton is overlaid with soft tissue, which in turn is often covered by clothes. The image of the resulting surface also then depends on the viewpoint of the camera, the perspective projection onto the image plane, the scene illumination and several other factors.
2.1 Kinematic Parameterization An articulated skeleton, comprising rigid parts connected by joints, can be represented as a tree. One part, such as the upper torso, is defined to be the root node and all remaining parts are either a child of the root or of another part. In this way, the entire pose can be described by the position and orientation of the root node in a global coordinate frame, and the position and orientation of each part in the coordinate frame of its parent. The state s then comprises these positions and orientations. If parts are rigidly attached at joints then the number of degrees of freedom (DOFs) required will be less than the full 6 DOFs necessary to represent pose in a 3D space. The precise number of degrees of freedom varies based on the type of joint. For instance, a hinge joint is commonly used to represent the knee and has one rotational DOF while a ball-and-socket joint, often used to represent the hip, has three rotational DOFs. While real joints in the body are significantly more complex, such simple models greatly reduce the number of parameters to estimate. One critical issue when designing the state space is the parameterization of rotations. Formally, rotations in R3 are 3×3 matrices with determinant 1, the set of which is denoted SO(3). Unfortunately, 3×3 matrices have significantly more parameters than necessary to specify the rotation, and it is extremely difficult to keep a matrix in SO(3) as it changes over time. Lower dimensional parameterizations of rotations are therefore preferred. Most common are Euler angles, which represent a rotation as a sequence of 3 elementary rotations about fixed axes. Unfortunately, Euler angles suffer from several problems including ambiguities, and singularities
Video-Based People Tracking
61
known as Gimbal lock. The most commonly used alternatives include exponential maps [21] and quaternions [34].
2.2 Body Geometry The skeleton is overlaid with soft tissue and clothing. Indeed we do not observe the skeleton but rather the surface properties of the resulting 3D volume. Both the geometry and the appearance of the body (and clothing) are therefore critical factors in the estimation of human pose and motion. Body geometry has been modeled in many ways and remains a largely unexplored issue in tracking and pose estimation. A commonly used model treats the segments of the body as rigid parts whose shapes can be approximated using simple primitives such as cylinders or ellipsoids. These geometric primitives have the advantage of being simple to design and efficient to work with under perspective projection [64, 73]. Other, more complex shape models have been used such as deformable super-quadrics [37], and implicit functions comprising mixtures of Gaussian densities to model 3D occupancy [47]. The greater expressiveness allows one to more accurately model the body, which can improve pose estimation, but it increases the number of parameters to estimate, and the projection of the body onto the image plane becomes more computationally expensive. Recent efforts have been made to build detailed models of shape in terms of deformable triangulated meshes that are anchored to a skeleton. A well-known example of which is the SCAPE model [2]. By using dimensionality reduction, the triangulated mesh is parameterized using a small number of variables, avoiding the potential explosion in the number of parameters. Using multiple cameras one can accurately recover both the shape and pose [4]. However, the computational cost of such models is high, and may only be practical with offline processing. Good results on 3D monocular hand tracking have also been reported, based on a mesh-based surface model with approximately 1000 triangular facets [11]. However, even deformable mesh body models cannot account for loose fitting clothing. Dresses and robes are extreme examples, but even loose fitting shirts and pants can be difficult to handle, since the relationship between the surface geometry observed in the image and the underlying skeleton is very complex. In most current tracking algorithms, clothing is assumed to be tight fitting so that the observed geometry is similar to the underlying body. To handle the resulting errors due to these assumptions, the observation models (and the likelihood functions) must be robust to the kinds of appearance variations caused by clothing. Some have attempted to explicitly model the effects of clothing and its interaction with the body to account for this, but these models are complex and computationally costly [3, 54]. This remains a challenging research direction.
62
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
2.3 Image Formation Given the pose and geometry of the body, the formation of an image of the person depends on several other factors. These include properties of the camera (e.g., the lens, aperture and shutter speed), the rest of the scene geometry (and perhaps other people), surface reflectance properties of clothing and background objects, the illumination of the scene, etc. In practice much of this information is unavailable or tedious to acquire. The exception to this is the geometric calibration of the camera. Standard methods exist (e.g., Forsyth and Ponce [16]) which can estimate camera parameters based on images of calibration targets.1 With fixed cameras this need only be done once. If the camera moves then certain camera parameters can be included in the state, and estimated during tracking. In either case, the camera parameters define a perspective projection, P(X), which maps a 3D point X ∈ R3 to a point on the 2D image plane.
3 Image Measurements Given the skeleton, body geometry and image formation model, it remains to formulate the likelihood distribution p(z|s) in (1).2 Conceptually, the observations are the image pixels, and the likelihood function is derived from ones generative model that maps the human pose to the observed image. As suggested in Figure 2, this involves modeling the surface shape and reflectance properties, the sources of illumination in the scene, and a photo-realistic rendering process for each pixel. While this can be done for some complex objects such as the human hand [11], this is extremely difficult for clothed people and natural scenes in general. Many of the necessary parameters about scene structure, clothing, reflectance and lighting are unknown, difficult to measure and not of direct interest. Instead, approximations are used that explain the available data while being (to varying degrees) independent of many of these unknown parameters. Toward that end it is common to extract a collection of image measurements, such as edge locations, which are then treated as the observations. This section briefly introduces the most common measurements and likelihood functions that are often used in practice.
3.1 2D Points One of the simplest ways to constrain 3D pose is with a set image locations that are projections of known points on the body. These 3D points might be joint centers or 1
Standard calibration code and tools are available as part of OpenCV (The Open Computer Vision Library), available from http://sourceforge.net/projects/opencvlibrary/. 2 In this section we drop the time subscript for clarity.
Video-Based People Tracking
63
points on the surface of the body geometry. For instance, it is easy to show that one can recover 3D pose up to reflection ambiguities from the 2D image positions to which the joint centers project [66]. If one can identify such points (e.g., by manual initialization or ensuring that subjects wear textured clothing that produce distinct features), then the observation z comprises a set of 2D image locations, {mi }M i=1 , where measurement mi corresponds to location i on part j(i). If we assume that the 2D image observations are corrupted by additive noise then the likelihood function can be written as M p({mi }M i=1 | s) = ∏ pi mi − P(K j(i) (i | s))
(3)
i=1
where P(X) is the 2D camera projection of the 3D point X, and K j (|s) is the 3D position in the global coordinate frame of the point on part j given the current state s. The function pi (d) is the probability density function of mean-zero additive noise on point i. This is often chosen to be Gaussian with a standard deviation of σi , i.e., 1 d2 exp − pi (d) = √ . (4) 2σi2 2π σi However, if it is believed that some of the points may be unreliable, for instance if they are not tracked reliably from the image sequence, then it is necessary to use a likelihood density with heavy tails, such as a Student’s t-distribution. The greater probability density in the tails reflects the belief that measurement outliers exist, and reduces the influence of such outliers in the likelihood function. One way to find the image locations to which the joint centers project is to detect and track a 2D articulated model [15, 51, 58]; unfortunately this problem is almost as challenging as the 3D pose estimation problem itself. Another approach is to find image patches that are projections of points on the body (possibly joint centers), and can be reliably tracked over time, e.g., by the KLT tracker [67] or the WSL tracker [28]. Such a likelihood is easy to implement and has been used effectively [69, 70]. Nevertheless, acquiring 2D point tracks frequently requires hand initialization and tuning of the tracking algorithm. Further, patch trackers often fail when parts are occluded or move quickly, requiring reinitialization or other modifications to maintain a reliable set of tracks.
3.2 Background Subtraction If the camera is in a fixed location and the scene is relatively static, then it is reasonable to assume that a background image B(x, y) of the scene can be acquired (see Figure 3 (b)). This can then be subtracted from an observed image I(x, y) and thresholded to determine a mask that indicates which pixels correspond to the foreground person [23, 49]. That is, M(x, y) = 1 if I(x, y)− B(x, y) > ε and M(x, y) = 0
64
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
(a) Image I(x, y)
(b) Background (c) Log Probability of BG B(x, y) log pB (I(x, y))
(d) Mask M(x, y)
(e) Clean Mask
Fig. 3 Background subtraction. Original image is illustrated in (a); the corresponding background image of the scene, B(x, y), in (b); (c) shows the log probability of each pixel I(x, y) belonging to the background (with light color corresponding to high probability); (d) illustrates the foreground mask (silhouette image) obtained by thresholding the probabilities in (c); in (e) a cleaned up version of the foreground mask in (d) obtained by simple morphological operations.
otherwise (e.g. Figure 3 (d)). The mask can be used to formulate a likelihood by peˆ y|s) nalizing discrepancies between the observed mask M(x, y) and a mask M(x, predicted from the image projection of the body geometry. For instance Deutscher and Reid [13] used ˆ y|s)| 1 |M(x, y) − M(x, exp − (5) p(M | s) = ∏ √ 2σ 2 (x,y) 2π σ where σ controls how strongly disagreements are penalized. Such a likelihood is attractive for its simplicity but there will be significant difficulty in setting the threshold ε to an appropriate value; there may be no universally satisfactory value. One can also consider a probabilistic version of background subtraction which avoids the need for a threshold (see Figure 3 (c)). Instead, it is assumed that background pixels are corrupted with mean-zero, additive Gaussian noise. This yields the likelihood function p(I | s) =
∏ pB(I(x, y))1−M(x,y|s) ˆ
(6)
(x,y)
where pB (I(x, y)) is the probability that pixel I(x, y) is consistent with the background. For instance, a hand specified Gaussian model can be used or more complex models such as mixtures of Gaussians can be learned in advance or during tracking [63]. Such a likelihood will be more effective than one based on thresholding. Nevertheless, background models will have difficulty coping with body parts that appear similar to the background; in such regions, like the lower part of the torso in Figure 3, the model will be penalized incorrectly. Problems also arise when limbs occlude the torso or other parts of the body, since then one cannot resolve them from the silhouette. Finally, background models often fail when the illumination changes (unless an adaptive model is used), when cameras move, or when scenes contain moving objects in the background.
Video-Based People Tracking
65
0.045
0.02
0.04
0.018 0.016
0.035
0.014
Probability
Probability
0.03 0.025 0.02 0.015
0.01 0.008 0.006
0.01
0.004
0.005 0 500
0.012
0.002 600
700
800
900
1000
1100
0 −300
1200
−200
−100
0
100
200
300
Optical Axis Translation
Z−Axis Translation
(a)
(b)
(c)
(d)
Fig. 4 Background likelihood. The behavior of the background subtraction likelihood described by Equation (5) is illustrated. A true pose, consistent with the pose of the subject illustrated in Figure 3, is taken and the probability of that pose as a function of varying a single degree of freedom in the state are illustrated in (a) and (c); in (a) the entire body is shifted up and down (along the Z-axis), in (d) along the optical axis of the camera. In (b) and (d) poses corresponding to the strongest peak in the likelihood of (a) and (c) respectively are illustrated. While ideally one would prefer the likelihood to have a single global maxima at the true value (designated by the vertical line in (a) and (c)), in practice, the likelihoods tend to be noisy, multi-modal and may not have a peak in the desired location. In particular in (c), due to the insensitivity of monocular likelihoods to depth, noise in the obtained foreground mask and inaccuracies in the geometric model of the body lead to severe problems. Also note that, in both figures, the noise in the likelihood indicates that simple search methods are likely to get stuck in local optima.
3.3 Appearance Models In order to properly handle uncertainty, e.g., when some region of the foreground appears similar to the background, it is useful to explicitly model the foreground appearance. Accordingly, the likelihood becomes p(I | s) =
∏ pB(I(x, y))1−M(x,y | s) pF (I(x, y) | s)M(x,y | s) ˆ
ˆ
(7)
(x,y)
where pF (I(x, y) | s) is the probability of pixel I(x, y) belonging to the foreground. Notice that if a uniform foreground model is assumed, i.e., pF (·) ∝ 1, then (7) simply becomes the probabilistic background subtraction model of (6). An accurate foreground model pF (I(x, y)|s) is often much harder to develop than a background model, because appearance varies depending on surface orientation with respect to the light sources and the camera, and due to complex non-rigid deformation of the body and clothing over time. It therefore requires offline learning based on a reasonable training ensemble of images [27, 50] or it can be updated online [76]. Simple foreground models are often learned from the image pixels to which the body projects to in one or more frames. For example one could learn the mean RGB color and the its covariance for the body, or for each part of the body if they differ in appearance. One can also model the statistics of simple filter outputs (e.g., gradient filters). One important consideration about likelihoods is computational expense, as evaluating every pixel in the image can be burdensome. Fortunately, this can usually be avoided as a likelihood function typically needs only be specified up to a multiplicative constant. By dividing the likelihood by the background model for each
66
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
··· Head
··· Torso
Head
Calf
Thigh
··· Thigh
Torso
···
(a) Image
(b) Log Probability of BG
(c) Log Probability of FG
I(x, y)
log pB (I(x, y))
log pF (I(x, y)|s)
Calf
(d) Log FG/BG Ratio log
pF (I(x,y)|s) pB (I(x,y))
Fig. 5 Modeling appearance. The appearance likelihood described by Equation (8) is illustrated. The observed image is shown in (a); the log probability of a pixel belonging to the background in (b); the probability of the pixel belonging to a foreground model (modeled by a mixture of Gaussians) for a given body part in (c); the final log ratio of the foreground to background probability is illustrated in (d). Notice that unlike the background likelihood, the appearance likelihood is able to attribute parts of the image to individual segments of the body. 0.012
0.04 0.035
0.01
0.03
Probability
Probability
0.008 0.025 0.02 0.015
0.006
0.004 0.01
0.002 0.005 0 500
600
700
800
900
1000
1100
0 −300
1200
(a)
−200
−100
0
100
200
300
Optical Axis Translation
Z−Axis Translation
(b)
(c)
(d)
Fig. 6 Appearance likelihood. The behavior of the appearance likelihood described by Equation (8) is illustrated. Similarly to Figure 4 a true pose, consistent with the pose of the subject illustrated in Figure 3, is taken and the probability of that pose as a function of varying a single degree of freedom in the state are illustrated in (a) and (c); as before in (a) the entire body is shifted up and down (along the Z-axis), in (d) along the optical axis of the camera. In (b) and (d) poses corresponding to the strongest peak in the likelihood of (a) and (c) respectively are illustrated. Notice that due to the strong separation between foreground and background in this image sequence, appearance likelihood performs similarly to the background likelihood model (illustrated in Figure 4); in sequences where foreground and background contain similar colors appearance likelihoods tend to produce superior performance.
pixel terms cancel out leaving p(I | s) ∝
pF (I(x, y) | s) pB (I(x, y)) ˆ (x,y) s.t. M(x,y | s)=1
∏
(8)
where the product is only over the foreground pixels, allowing a significant savings in computation. This technique can be more generally used to speed up other types of likelihood functions.
Video-Based People Tracking
67
3.4 Edges and Gradient Based Features Unfortunately foreground and background appearance models have several problems. In general they have difficulty handling large changes in appearance such as those caused by varying illumination and clothing. Additionally, near boundaries they can become inaccurate since most foreground models do not capture the shading variations that occur near edges, and the pixels near the boundary are a mixture of foreground and background colors due to limited camera resolution. For this reason, and to be relatively invariant to lighting and small errors in surface geometry, it has been common to use edge-based likelihoods [73]. These models assume that the projected edges of the person should correspond to some local structure in image intensity. Perhaps the simplest approach to the use of edge information is the Chamfer distance [5]. or the Hausdorff distance [25]. Edges are first extracted from the observed image using standard edge detection methods [16] and a distance map is computed where d(x) is the squared Euclidean distance from pixel x to the nearest edge pixel. The outline of the subject in the image is computed and the boundary is sampled at a set of points {bi }M i=1 . In the case of Chamfer matching the likelihood function is 1 M p(d | s) = exp − ∑ d(bi ) . (9) M i=1 Chamfer matching is fast, as the distance map need only be computed once and is evaluated only at edge points. Additionally, it is robust to changes in illumination and other appearance changes of the subject. However it can be difficult to obtain a clean set of edges as texture and clutter in the scene can produce spurious edges. Gavrila and Davis [19] successfully used a variant of Chamfer matching for pose tracking. To minimize the impact of spurious edges they performed an outlier rejection step on the points bi . Chamfer matching is also robust to inaccuracies in the geometry of the subject. If the edges of the subject can be predicted with a high degree of accuracy, then predictive models of edge structure can be used. Kollnig and Nagel [32] built hand specified models which predicted large gradient magnitudes near outer edges of the target. Later, this work was extended to predict gradient orientations and applied to human pose tracking by Wachter and Nagel [73]. Similarly, Nestares and Fleet [43] learned a probabilistic model of local edge structure which was used by Poon and Fleet [48] to track people. Such models can be effective however sufficiently accurate shape models can be difficult to build.
68
Marcus A. Brubaker, Leonid Sigal and David J. Fleet 0.02
0.08
0.14
0.018
0.07
0.12
0.016
0.06
0.1
0.012 0.01 0.008
0.05
Probability
Probability
Probability
0.014
0.04 0.03
0.006
0.08
0.06
0.04
0.02 0.004
0 −300
0.02
0.01
0.002 −200
−100
0
100
Optical Axis Translation
(1 view)
200
300
0 −300
−200
−100
0
100
200
300
0 −300
Optical Axis Translation
(2 views)
−200
−100
0
100
200
300
Optical Axis Translation
(3 views)
Fig. 7 Number of views. Effect of combining measurements from a number of image views on the (background) likelihood. With a single view the likelihood exhibits a wide noisy mode relatively far from the true value of the translation considered (denoted by the vertical line); with more views contributing image measurements the ambiguity can be resolved, producing a stronger peak closer to the desired value.
3.5 Discussion There is no consensus as to which form of likelihood is best. However, some cues are clearly more powerful than others. For instance, if 2D points are practical in a given application then they should certainly be used as they are an extremely strong cue. Similarly, some form of background model is invaluable and should be used whenever it is available. Another effective technique is to use multiple measurements. To correctly combine measurements, the joint probability of the two observations p(z(1) , z(2) | s) needs to be specified. This is often done by assuming the conditional independence of the observations p(z(1) , z(2) | s) = p(z(1) | s)p(z(2) | s) .
(10)
This assumption, often referred to as naïve Bayes, is unlikely to hold as errors in one observation source are often correlated with errors in others. However, it is reasonable when, for instance, the two observations are from different cameras or when one set of observations is explaining edges and the other is explaining pixels not at the boundary. The behavior of the background likelihood (previously illustrated in Figure 4) as a function of image measurements combined from multiple views is illustrated in Figure 7.
4 Motion Models Prior information about human pose and motion is essential for resolving ambiguity, for combining noisy measurements, and for coping with missing observations. A prior model biases pose estimation toward plausible poses, when pose might otherwise be under-constrained. In principle one would like to have priors that are weak enough to admit all (or most) allowable motions of the human body,
Video-Based People Tracking
69
but strong enough to constrain ambiguities and alleviate challenges imposed by the high-dimensional inference. The balance between these two competing goals is often elusive. This section discusses common forms of motion models and introduces some emerging research directions.
4.1 Joint Limits The kinematic structure of the human body permits a limited range of motion in each joint. For example, knees cannot hyperextend and the torso cannot tilt or twist arbitrarily. A central role of prior models is to ensure that recovered poses satisfy such biomechanical limits. While joint limits can be encoded by thresholds imposed on each rotational DOF, the true nature of joint limits in the human body is more complex. In particular, the joint limits are dynamic and dependant on other joints [22]. Unfortunately, joint limits by themselves do not encode enough prior knowledge to facilitate tractable and robust inference.
4.2 Smoothness and Linear Dynamical Models Perhaps the simplest commonly used prior model is a low-order Markov model, based on an assumption that human motion is smooth [73, 56, 48]. A typical firstorder model specifies that the pose at one time is equal to the previous pose up to additive noise: (11) st+1 = st + η where the process noise η is usually taken to be Gaussian η ∼ N (0, Σ ). The resulting prior is then easily shown to be p(st+1 | st ) = G(st+1 ; st , Σ )
(12)
where G(x; m,C) is the Gaussian density function with mean m and covariance C, evaluated at x. Second-order models express st+1 in terms of of st and st−1 , allowing one to use velocity in the motion model. For example, a common, damped secondorder model is (13) st+1 = st + κ (st − st−1 ) + η where κ is a damping constant which is typically between zero and one. Equations (11) and (13) are instances of linear models, the general form of which is st+1 = ∑Nn=1 An st−n+1 + η , i.e., an N-th order linear dynamical model. In many cases, as in (11) and (13), it is common to set the parameters of the transition model by hand, e.g., setting An , assuming a fixed diagonal covariance matrix Σ , or letting the diagonal elements of the covariance matrix in (12) be proportional to ||st −st−1 ||2 [13]. One can also learn dynamical models from motion capture data [45]. This
70
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
would allow one, for example, to capture the coupling between different joints. Nevertheless, learning good parameters is challenging due to the high-dimensionality of the state space, for which the transition matrices, An ∈ RN×N , can easily suffer from over-fitting. Smoothness priors are relatively weak, and as such allow a diversity of motions. While useful, this is detrimental when the model is too weak to adequately constrain tracking in monocular videos. In constrained settings, where observations from 3 or more cameras are available and occlusions are few, such models have been shown to achieve satisfactory performance [13]. It is also clear that human motion is not always smooth, thereby violating smoothness assumptions. Motion at ground contact, for example, is usually discontinuous. One way to accommodate this is to assume a heavy-tailed model of process noise that allows occasional, large deviations from the smooth model. One might also consider the use of switching linear dynamical models, which produce piece-wise linear motions [46].
4.3 Activity Specific Models Assuming that one knows or can infer the type of motion being tracked, or the identity of the person performing the motion, one can apply stronger prior models that are specific to the activity or subject [35]. The most common approach is to learn models off-line (prior to tracking) from motion capture data. Typically one is looking for some low-dimensional parameterization of the pose and motions. To introduce the idea, consider a dataset Ψ = {ψ (i) } consisting of K kinematic poses ψ (i) , i ∈ (1, ..., K) obtained, for example, using a motion capture system. Since humans often exhibit characteristic patterns of motion, these poses will often lie on or near a low-dimensional manifold in the original high-dimensional pose space. Using such data for training, methods like Principle Component Analysis (PCA) can be used to approximate poses by the linear combination of a mean pose μΨ = K1 ∑Ki=1 ψ (i) and a set of learned principal directions of variation. These principle directions are computed using the singular value decomposition (SVD) of a matrix S whose i-th row is ψ (i) − μΨ . Using SVD, matrix S is decomposed into two orthonormal matrices U and V (U = [u1 , u2 , ..., um ] consisting of the eigenvectors, (a.k.a., eigen-poses) and a diagonal matrix Λ containing ordered eigenvalues such that S = U Λ V T . Given this learned model, a pose can be approximated by q
ψ ≈ μΨ + ∑ ui ci
(14)
i=1
where ci is the set of scalar coefficients and q m controls the amount of variance accounted for by the model. As such, the inference over the pose can be replaced by the inference over the coefficients s = [c1 , c2 , ..., cq ]. Since q is typically
Video-Based People Tracking
71
Fig. 8 Illustration of the latent space motion prior model. Results of learning a Gaussian Process Dynamical Model that encodes both the non-linear low-dimensional latent pose space and the dynamics in that space. On the left a few walking motions are shown embedded in the 3D latent space. Each point on a trajectory is an individual pose. For six of the points the corresponding mean pose in the full pose space is shown. On the right the distribution over plausible poses in the latent space is shown. This figure is re-printed from [74].
small (e.g. 2 − 5) with respect to the dimensionality of the pose space this transformation facilitates faster pose estimation. However, the new low-dimensional state space representation also requires a new model of dynamics that has to operate on the coefficients. The models of dynamics in the linear latent-space such as the one obtained using the eigen-decomposition are typically more complex then those in the original pose space and are often nonlinear. One alternative to simplifying the motion models is to learn the eigen-decomposition for entire trajectories of motion rather then the individual poses [56, 71]. Regardless, linear models such us the one described here are typically insufficient to capture intricacies of real human poses or motion. More recent methods have shown that non-linear embeddings are more effective [60]. Gaussian Processes Latent Variable Models (GPLVMs) have became a popular choice since they have been shown to generalize from small amounts of training data [69]. Furthermore, one can learn a low-dimensional embedding that not only models the manifold for a given class of motions, but also captures the dynamics in that learned manifold [36, 70]. This allows the inference to proceed entirely in the low-dimensional space alleviating complexities imposed by the high-dimensional pose space all together. An example of a 3D latent space for walking motions is illustrated in Figure 8 and results of tracking with that model is shown in Figure 9. Alternatively, methods that use motion capture directly to implicitly specify stronger priors have also been proposed. These types of priors make the assumption that the observed motion should be akin to the motion exhibited in the database of exemplar motions. Simply said, given a pose at time t such approaches find an exemplar motion from the database that contains a closely resembling pose, and uses that motion to look up the next pose in the sequence. Priors of this form can also be formulated probabilistically [57].
72
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
Fig. 9 Tracking with the GPDM. 56 frames of a walking motion that ends with almost total occlusion (just the head is visible) in a cluttered and moving background. Note how the prior encourages realistic motion as occlusion becomes a problem. This figure is re-printed from [70].
All of these methods have proven effective for monocular pose inference in specific scenarios for relatively simple motions. However, due to their action specific nature, learning models that successfully generalize and represent multiple motions and transitions between those motions has been limited.
4.4 Physics-based Motion Models Recently, there has been preliminary success in using physics-based motion models as priors. Physics-based models have the potential to be as generic as simple smoothness priors but more informative. Further, they may be able to recover subtleties of more realistic motion which would be difficult, if not impossible, with existing motion models. The use of physics-based prior models in human tracking dates back to the early 1990’s with pioneering work by Metaxas and Terzopoulos [37] and Wren and Pentland [75]. However, it is only recently that success with monocular imagery has been shown. The fundamental motivation for physics-based motion models is the possibility that motions are best described by the forces which generated them, rather than a sequence of kinematic poses. These forces include not only the internal (e.g., muscle generated) forces used to propel limbs, but also external forces such as gravity, ground reaction forces, friction and so on. Many of these forces can be derived from first principles and provide important constraints on motion. Modeling the remaining forces, either deterministically or stochastically is then the central difficulty of physics-based motion models. This class of models remains a promising but relatively unexplored area for future research. The primary difficulty with physics-based models is the instability of complex dynamical systems. Sensitivity to initial conditions, discontinuities of motion and other non-linearities have made robust, realistic control of humanoid robots an elusive goal of robotics research. To address this in the context of tracking Brubaker, Fleet, and Hertzmann [8] used a simplified, physical model that is stable and easy to control. While this model was restricted to simple walking motions, the work was extended by Brubaker and Fleet [7] to a more complex physical model, capable
Video-Based People Tracking
73
of a wider range of motions. An alternative strategy employed by Vondrak, Sigal, and Jenkins [72] used a motion capture database to guide the dynamics. Using inverse dynamics, they solved for the forces necessary to mimic motions found in the database.
5 Inference In a probablistic framework our goal is to compute some approximation to the distribution p(s1:t | z1:t ). Often this is formulated as online inference, where the distribution is computed one frame at a time as the observations arrive, exploiting the well-known recursive form of the posterior (assuming conditional independence of the observations): p(s1:t | z1:t ) ∝ p(zt | st ) p(st | s1:t−1 ) p(s1:t−1 | z1:t−1 ) .
(15)
The motion model, p(st |s1:t−1 ), is often a first order Markov model which simplifies to p(st |st−1 ). While this is not strictly necessary for the inference methods presented here, it is important because the motion model then depends only on the last state as opposed to the entire trajectory. The classic, and perhaps simplest, approach to this problem is the Kalman filter [73]. However, the Kalman Filter is not suitable for human pose tracking where the dynamics are non-linear and the likelihood functions are non-Gaussian. As a consequence, Sequential Monte Carlo techniques are amongst the most commonly used to perform this inference. Sequential Monte Carlo methods were first applied to visual tracking with the CONDENSATION algorithm of Isard and Blake [26] but were applied earlier for time series analysis by Gordon, Salmond, and Smith [20] and Kong, Liu, and Wong [33]. For a more detailed discussion of Sequential Monte Carlo methods, we refer the reader to the review article by Doucet, Godsill, and Andrieu [14]. In this section, we present a very simple algorithm, particle filtering, in which stochastic simulation of the motion model is combined with weighting by the likelihood to produce weighted samples which approximate the posterior. We also present two variants which attempt to work around the deficiencies of the basic particle filter.
5.1 Particle Filter A particle filter represents a distribution with a weighted set of sample states, de(i) (i) noted {(s1:t , w1:t )|i = 1, . . . , N}. When the samples are fairly weighted, then sample statistics approximate expectation under the target distribution, i.e.,
74
Marcus A. Brubaker, Leonid Sigal and David J. Fleet N
∑ wˆ 1:t f (s1:t ) (i)
(i)
≈ E[ f (s1:t )]
(16)
i=1
(i) ( j) −1 (i) w1:t is the normalized weight. The sample statistics where wˆ 1:t = ∑Nj=1 w1:t approach that of the target distribution as the number of samples, N, increases. In a simple particle filter, given a fairly weighted sample set from time t, the samples at time t + 1 are obtained with importance sampling. First samples are (i) (i) drawn from a proposal distribution q(st+1 | s1:t , z1:t+1 ). Then the weights are updated (i)
(i)
w1:t+1 = w1:t
(i)
(i)
(i)
p(zt+1 | st+1 )p(st+1 | s1:t ) (i)
(i)
q(st+1 | s1:t , z1:t+1 )
(17)
to be fairly weighted samples at time t + 1. The proposal distribution must be nonzero everywhere the posterior is non-zero, but it is otherwise largely unconstrained. The simplest and most common proposal distribution is motion model, p(st+1 | s1:t ), (i) (i) (i) which simplifies the weight update to be w1:t+1 = w1:t p(zt+1 |st+1 ). This simple procedure, while theoretically correct, is known to be degenerate. As t increases, the normalized weight of one particle approaches 1 while the others approach 0. Weights near zero require a significant amount of computation but contribute very little to the posterior approximation, effectively reducing the posterior approximation to a point estimate. To mitigate this problem, a resampling step is introduced where particles with small weights are discarded. Before the propagation stage a new set of samples (i) ( j) {˜s1:t |i = 1, . . . , N} is created by drawing an index j such that p( j) = wˆ 1:t , and then (i) ( j) (i) setting s˜1:t = s1:t . The weights for this new set of particles are then w˜ 1:t = 1/N for all i. This resampling procedure can be done at every frame, at a fixed frequency, or only when heuristically necessary. While it may seem good to do this at every frame, as done by Isard and Blake [26], it can cause problems. Specifically, resampling introduces bias in finite sample sets, as the samples are no longer independent and can even exacerbate particle depletion over time. Resampling when necessary balances the need to avoid degeneracy without introducing undue bias. One of the most commonly used heuristics is an estimate of the effective sample size Ne f f =
N
∑
(i) (wˆ 1:t )2
−1 (18)
i=1
which takes values from 1 to N. Intuitively, this can be thought of as the average number of independent samples that would survive a resampling step. Notice that after resampling, Ne f f is equal to N. With this heuristic, resampling is then performed when Ne f f < Nthresh , otherwise it is skipped. The particle filtering algorithm, with this heuristic resampling strategy, is outlined in Algorithm 1.
Video-Based People Tracking
75 (i)
(i)
Initialize the particle set {(s1 , w1 )|i = 1, . . . , N} for t = 1, 2, . . . do (i) Compute the normalized weights wˆ 1:t and the effective number of samples Ne f f as in (18) if Ne f f < Nthresh then for i = 1, . . . , N do ( j) Randomly choose an index j ∈ (1, ..., N) with probability p( j) = wˆ 1:t (i) ( j) (i) Set s˜1:t = s1:t and w˜ 1:t = 1 end for (i) (i) Replace the old particle set {(s1:t , w1:t )|i = 1, . . . , N} with (i) (i) {(˜s1:t , w˜ 1:t )|i = 1, . . . , N} end if for i = 1, . . . , N do (i) (i) Sample st+1 from the proposal distribution q(st+1 |s1:t , z1:t+1 ) (i)
(i)
(i)
Construct the new state trajectory s1:t+1 = (s1:t , st+1 ) (i)
Update the weights w1:t+1 according to equation (17) end for end for Algorithm 1: Particle Filtering.
5.2 Annealed Particle Filter Unfortunately, resampling does not solve all the problems of the basic particle filter described above. Specifically, entire modes of the posterior can still be missed, particularly if they are far from the modes of the proposal distribution or if modes are extremely peaked. One solution to this problem is to increase the number of particles, N. While this will solve the problem in theory, the number of samples theoretically needed is generally computationaly untenable. Further, many samples will end up representing uninteresting parts of the space. While these issues remain challenges, several techniques have been proposed in an attempt to improve the efficiency of particles filters. One approach, inspired by simulated annealling and continuation methods, is Annealed Particle Filtering (APF) [13]. The APF algorithm is outlined in Algorithm 2. At each time t, the APF goes through L levels of annealing. For each particle i, annealing level , begins by ( j) choosing a sample from the previous annealing level, s1:t+1,−1 , with probability (i)
p( j) = wˆ 1:t+1,−1 . The state at time t + 1 of sample j is then diffused to create a new (i)
hypothesis st+1, according to T (st+1, |s1:t+1,−1 ) = N (st+1, |st+1,−1 , α Σ ) , and weights the new hypothesis by
(19)
76
Marcus A. Brubaker, Leonid Sigal and David J. Fleet (i)
(i)
Initialize the weighted particle set {(s1,L , w1,L )|i = 1, . . . , N}. for t = 1, 2, . . . do for i = 1, . . . , N do (i) (i) Sample st+1 from the proposal distribution q(st+1 |s1:t,L , z1:t+1 ) (i)
(i)
(i)
Construct the new state trajectory s1:t+1,0 = (s1:t,L , st+1 ) (i)
Assign the weights w1:t+1,0 =
(i)
W0 (s1:t+1,0 |z1:t+1 ) (i)
(i)
q(st+1,0 |s1:t,L ,z1:t+1 )
end for for = 1, . . . , L do for i = 1, . . . , N do Compute the normalized weights −1 (i) ( j) (i) w1:t+1,−1 wˆ 1:t+1,−1 = ∑Nj=1 w1:t+1,−1 Randomly choose an index j ∈ (1, ..., N) with probability ( j) p( j) = w1:t+1,−1 (i)
( j)
Sample st+1, from the diffusion distribution T (st+1, |s1:t+1,−1 ) (i)
(i)
(i)
Construct the new state trajectory s1:t+1, = (s1:t , st+1, ) (i)
(i)
Compute the annealed weights w1:t+1, = W (s1:t+1, |z1:t+1 ) end for end for end for Algorithm 2: Annealed Particle Filtering. β W (s1:t+1, |z1:t+1 ) = p(zt+1 | st+1, )p(st+1, |st, ) .
(20)
In the above α ∈ (0, 1) is called the annealing rate and is used to control the scale of covariance, Σ , in the diffusion process. The β is the temperature parameter, derived based on the annealing rate, α , and the survival diagnostics of the particle set (for details see Deutscher and Reid [13]) to ensure that a fixed fraction of samples survive from one stage of annealing to the next. The sequence β0 , . . . , βL is a gradually increasing sequence between zero and 1, ending with βL = 1. When β is small, the difference in height between peaks and troughs of the posterior are attenuated. As a result it is less likely that one mode, by chance, will dominate and attract all the particles, thereby neglecting other, potentially important modes. In this way the APF allows particles to broadly explore the posterior in the early stages. This means that the particles are better able to find different potential peaks, which then attract the particles more strongly as the likelihood becomes more strongly peaked (as β increases). It is worth noting that, with L = 1, the APF reduces to the standard particle filter discussed in the previous section. While often effective in finding significant modes of the posterior, the APF does not produce fairly weighted samples from the posterior. As such it does not
Video-Based People Tracking
77
Fig. 10 Annealed Particle Filter (APF) tracking from multi-view observations. This figure is re-printed from [13]. (i)
(i)
Initialize {(s1 , w1 )|i = 1, . . . , N} for t = 1, 2, . . . do for i = 1, . . . , N do (i) (i) Sample s˜t+1 from the proposal distribution q(˜st+1 |s1:t , z1:t+1 ) (i)
(i)
Construct the new state trajectory s˜1:t+1 = (s1:t , s˜t+1 ) (i)
(i)
(i)
Update the weights w1:t+1 according to equation (17) with st+1 = s˜t+1 end for −1 (i) ( j) (i) w1:t+1 Compute the normalized weights wˆ 1:t+1 = ∑Nj=1 w1:t+1 for i = 1, . . . , N do ( j) Randomly choose an index j ∈ (1, ..., N) with probability p( j) = wˆ 1:t+1 . ( j)
Set the target distribution to be P(q) ∝ p(zt+1 |q)p(q|˜s1:t ) ( j) Set the initial state of the Markov Chain to q0 = s˜t+1 for r = 1, . . . , R do Sample qr from the MCMC transition density T (qr |qr−1 ), e.g., using Hybrid Monte Carlo as described in Algorithm 4 end for (i) (i) (i) Set s1:t+1 = (˜s1:t , qR ) and w1:t+1 = 1 end for end for Algorithm 3: Markov Chain Monte Carlo Filtering.
accurately represent the posterior and the sample statistics of (16) are not representative of expectations under the posterior. Recent research has shown that by restricting the form of the diffusion and properly weighting the samples, one can obtain fairly weighted samples [18, 42].
78
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
5.3 Markov Chain Monte Carlo Filtering Another way to improve the efficiency of particle filters is with the help of Markov Chain Monte Carlo (MCMC) methods to explore the posterior. A Markov Chain3 is a sequence of random variables q0 , q1 , q2 , . . . with the property that for all i, p(qi |q0 , . . . , qi−1 ) = p(qi |qi−1 ). In MCMC, the goal is to construct a Markov Chain chain such that, as i increases, p(qi ) approaches the desired target distribution P(q). In the context of particle filtering at time t, the random variables, q, are hypothetical states st and the target distribution, P(q), is the posterior p(s1:t |z1:t ). The key to MCMC is the definition of a suitable transition density p(qi |qi−1 ) = T (qi |qi−1 ). To this end there are several properties that must be satisfied, one of which is P(q) =
ˆ P(q)d ˆ qˆ . T (q|q)
(21)
This means that the chain has the target distribution P(q) as its stationary distribution. For a good review of the various types of Markov transition densities used, and a more thorough introduction to MCMC in general, see [41]. A general MCMC-filtering algorithm is given in Algorithm 3. It begins by propogating samples through time and updating their weights according to a conventional particle filter. These particles are then are chosen with probability proportional to their weights, as the initial states in N independant Markov chains. The target distribution for each chain is the posterior p(st |, z1:t ). The final states of each chain are then taken to be fair samples from the posterior. Choo and Fleet [9] used an MCMC method known as Hybrid Monte Carlo. The Hybrid Monte Carlo algorithm (Algorithm 4) is an MCMC technique, based on ideas developed for molecular dynamics, which uses the gradient of the posterior to efficiently find high probability states. A single step begins by sampling a vector p0 ∈ Rm from a Gaussian distribution with mean zero and unit variance. Here m is the dimension of the state vector q0 . The randomly drawn vector, known as the momentem, is then used to perform a simulation of the system of differen∂q tial equations ∂∂ pτ = − ∂ E(q) ∂ q and ∂ τ = p where τ is an artificial time variable and E(q) = − log P(q). The simulation begins at (q0 , p0 ) and proceeds using a leapfrog step which is explicitly given in Algorithm 4. There, L is the number of steps to simulate for and Δ is a diagonal matrix whos entries specify the size of step to take in each dimension of the state vector q. At the end of the simulation, the ending state of the physical simulation qL is accepted with probability a = min(1, e−c ) where 1 1 c = (E(qL ) + p0 2 ) − (E(q0 ) + pL 2 ) . 2 2
(22)
The specific form of the simulation procedure and the acceptance test at the end are designed such that P(q) is the stationary distribution of the transition distribu3
A full review of MCMC methods is well beyond the scope of this chapter and only a brief introduction will be presented here.
Video-Based People Tracking
79
Given a starting state q0 ∈ Rm and a target distribution P(q), define E(q) = − log P(q). Draw a momentum vector p0 ∈ Rm from a Gaussian distribution with mean 0 and unit variance. for = 1, . . . , L do ∂ E(q ) p−0.5 = p−1 − 12 Δ ∂ q−1 q = q−1 + Δ p−0.5 ) p = p−0.5 − 12 Δ ∂ E(q ∂q end for Compute the acceptance probability a = min(1, e−c ) where c is computed according to (22) Set u to be a uniformly sampled random number between zero and one if u < a then return qL else return q0 end if Algorithm 4: Hybrid Monte Carlo Sampling.
tion. The parameters of the algorithm, L and the diagonals of Δ are set by hand. As a rule of thumb Δ and L should be set so that roughly 75% of transitions are accepted. An important caveat is that the values of L and Δ cannot be set based on q and must remain constant throughout a simulation. For more information on Hybrid Monte Carlo see [41].
6 Initialization and Failure Recovery The final issue we must address concerns initialization and the recovery from tracking failures. Because of the large number of unknown state variables one cannot assume that the filter can effectively search the entire state space without a good prior model (or initial guess). Fortunately, over the last few years, progress on the development of discriminative methods for detecting people and pose inference has been encouraging.
6.1 Introduction to Discriminative Methods for Pose Estimation Discriminative approaches aim to recover pose directly from a set of measurements, usually through some form of regression applied to a set of measurements from a single frame. Discriminative techniques are typically learned from a set of
80
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
Fig. 11 Illustration of the simple discriminative model. The model introduced by [1] is illustrated. From left to right the figure shows (1) silhouette image, (2) contour of the silhouette image, (3) shape context feature descriptor for a point on a contour, (4) a set of shape context descriptors in the high (60-dimensional) space, (5) a 100-dimensional vector quantized histogram of shape descriptors that is used to obtain (6) the pose of the person through linear regression. This figure is re-printed from [1].
training exemplars, D = {(s(i) , z(i) ) ∼ p(s, z)|i = 1...N}), which are assumed to be fair samples from the joint distribution over states and measurements. The goal is to learn to predict an output for a given input. The inputs, z ∈ RM , are generic image measurements, 4 and outputs s ∈ RN , as above, represent the 3D poses of the body. The simplest discriminative method is Nearest-Neighbor (NN) lookup [24, 39], where, given a set of features observed in an image, the exemplar from the training database with the closest features is found, i.e., k∗ = arg mink d(˜z, z(k) ). The pose ∗ s(k ) for that exemplar is returned. The main challenge is to define a useful similarity measure d(·, ·), and a fast indexing scheme. One such approach was proposed by Shakhnarovich, Viola, and Darrell [55]. Unfortunately, this simple approach has three drawbacks: (1) large training sets are required, (2) all the training data must be stored and used for inference, and (3) it produces unimodal predications and hence ambiguities (or multi-modality) in image-to-pose mappings cannot be accounted for (e.g., see Sminchisescu, Kanaujia, Li, and Metaxas [61]). To address (1) and (2) a variety of global [1] and local [53] parametric models have been proposed. These models learn a functional mapping from image features to 3D pose. While these methods have been demonstrated successfully on restricted domains, and with moderately large training sets, they do not provide one-to-many mappings, and therefore do not cope with multi-modality. Multi-modal mappings have been formulated in a probabilistic setting, where one explicitly models multi-modal conditional distributions, p(s|z, Θ ), where Θ are parameters of the mapping, learned by maximizing the likelihood of the training data D. One example is the conditional Mixture of Experts (cMoE) model introduced by Sminchisescu, Kanaujia, Li, and Metaxas [61], which takes the form p(s | z, Θ ) =
K
∑ gk (z | Θ )ek (s | z, Θ ),
(23)
k=0
4 For instance, histograms-of-oriented-gradients, vector quantized shape contexts, HMAX, spatial pyramids, vocabulary trees and so on. See Kanaujia, Sminchisescu, and Metaxas [30] for details.
Video-Based People Tracking
81
Fig. 12 Discriminative Output. Pose estimation results obtained using the discriminative method introduced by Kanaujia, Sminchisescu, and Metaxas. This figure is re-printed from [30].
where K is the number of experts, gk are positive gating functions which depend on the input features, and ek are experts that predict the pose (e.g., kernel regressors). This model under various incarnations has been shown to work effectively with large datasets [6] and with partially labeled data5 [30]. The MoE model, however, still requires moderate to large amounts of training data to learn parameters of the gates and experts. Recently, methods that utilize an intermediate low dimensional embedding have been shown to be particularly effective in dealing with little training data in this setting [40, 31]. Alternatively, nonparametric approaches for handling large amounts of training data efficiently that can deal with multi-modal probabilistic predictions have also been recently proposed by Urtasun and Darrell [68]. Similar in spirit to the simple NN method above, their model uses the local neighborhood of the query to approximate a mixture of Gaussian Process (GP) regressors.
6.2 Discriminative Methods as Proposals for Inference While discriminative methods are promising alternatives to generative inference, it is not clear that they will be capable of solving the pose estimation problem in a general sense. The inability to generalize to novel motions, deal with significant occlusions and a variety of other realistic phenomena seem to suggest that some generative component is required. Fortunately, discriminative models can be incorporated within the generative setting in an elegant way. For example, multimodal conditional distributions that are the basis of most recent discriminative methods [6, 40, 61, 68] can serve directly as proposal distributions (i.e., q(st+1 |s1:t , z1:t+1 )) to improve the sampling efficiency of the Sequential Monte Carlo methods discussed above. Some preliminary work on Since joint samples span a very high dimensional space, RN+M , obtaining a dense sampling of the joint space for the purposes of training is impractical. Hence, incorporating samples from marginal distributions p(s) and p(z) is of great practical benefit.
5
82
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
combining discriminative and generative methods in this and other ways has shown promise. It has been shown that discriminative components provide for effective initialization and the recovery from transient failures, and that generative components provide effective means to generalize and better fit image observations [59, 61, 62].
7 Conclusions This chapter introduced the basic elements of modern approaches to pose tracking. Using the probabilistic formulation introduced in this chapter one should be able to build a state-of-the-art framework for tracking relatively simple motions of single isolated subjects in a compliant (possibly instrumented) environment. The more general problem of tracking arbitrary motion in monocular image sequences of unconstrained environments remains a challenging and active area of research. While many advances have been made, and the progress is promising, no system to date can robustly deal with all the complexities of recovering the human pose and motion in an entirely general setting. While the need to track human motion from images is motivated by a variety of applications, currently there have been relatively few systems that utilize the imagebased recovery of the articulated body pose for higher-level tasks or consumer applications. This to a large extent can be attributed to the complexity of obtaining an articulated pose in the first place. Nevertheless, a few very promising applications in biomechanics [10] and human computer interfaces [12, 52, 65] have been developed. The articulated pose has also proved useful as a front end for action recognition applications [44]. We believe that as the technologies for image-based recovery of articulated pose grows over the next years, so will the applications that utilize that technology.
References [1] Agarwal A, Triggs B (2006) Recovering 3d human pose from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(1):44–58 [2] Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J (2005) SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics 24(3):408–416 [3] Balan A, Black MJ (2008) The naked truth: Estimating body shape under clothing. In: IEEE European Conference on Computer Vision [4] Balan AO, Sigal L, Black MJ, Davis JE, Haussecker HW (2007) Detailed human shape and pose from images. In: IEEE Conference on Computer Vision and Pattern Recognition
Video-Based People Tracking
83
[5] Barrow HG, Tenenbaum JM, Bolles RC, Wolf HC (1977) Parametric correspondence and chamfer matching: Two new techniques for image matching. In: International Joint Conference on Artificial Intelligence, pp 659–663 [6] Bo L, Sminchisescu C, Kanaujia A, Metaxas D (2008) Fast algorithms for large scale conditional 3d prediction. In: IEEE Conference on Computer Vision and Pattern Recognition [7] Brubaker M, Fleet DJ (2008) The kneed walker for human pose tracking. In: IEEE Conference on Computer Vision and Pattern Recognition [8] Brubaker M, Fleet DJ, Hertzmann A (2007) Physics-based person tracking using simplified lower-body dynamics. In: IEEE Conference on Computer Vision and Pattern Recognition [9] Choo K, Fleet DJ (2001) People tracking using hybrid Monte Carlo filtering. In: IEEE International Conference on Computer Vision, vol II, pp 321–328 [10] Corazza S, Muendermann L, Chaudhari A, Demattio T, Cobelli C, Andriacchi T (2006) A markerless motion capture system to study musculoskeletal biomechanics: visual hull and simulated annealing approach. Annals of Biomedical Engineering 34(6):1019–1029 [11] de la Gorce M, Paragos N, Fleet DJ (2008) Model-based hand tracking with texture, shading and self-occlusions. In: IEEE Conference on Computer Vision and Pattern Recognition [12] Demirdjian D, Ko T, Darrell T (2005) Untethered gesture acquisition and recognition for virtual world manipulation. Virtual Reality 8(4):222–230 [13] Deutscher J, Reid I (2005) Articulated body motion capture by stochastic search. International Journal of Computer Vision 61(2):185–205 [14] Doucet A, Godsill S, Andrieu C (2000) On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing 10(3):197–208 [15] Felzenszwalb P, Huttenlocher DP (2005) Pictorial structures for object recognition. International Journal of Computer Vision 61(1):55–79 [16] Forsyth DA, Ponce J (2003) Computer Vision: A Modern Approach. Prentice Hall [17] Forsyth DA, Arikan O, Ikemoto L, O’Brien J, Ramanan D (2006) Computational studies of human motion: Part 1, tracking and motion synthesis. Foundations and Trends in Computer Graphics and Vision 1(2&3):1–255 [18] Gall J, Potthoff J, Schnorr C, Rosenhahn B, Seidel HP (2007) Interacting and annealing particle filters: Mathematics and a recipe for applications. Journal of Mathematical Imaging and Vision 28:1–18 [19] Gavrila DM, Davis LS (1996) 3-D model-based tracking of humans in action: a multi-view approach. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 73–80 [20] Gordon N, Salmond DJ, Smith AFM (1993) Novel approach to nonlinear/nonGaussian Bayesian state estimation. IEE Proceedings Part F Radar and signal processing 140:107–113 [21] Grassia FS (1998) Practical parameterization of rotations using the exponential map. Journal of Graphics Tools 3(3):29–48
84
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
[22] Herda L, Urtasun R, Fua P (2005) Hierarchical implicit surface joint limits for human body tracking. Computer Vision and Image Understanding 99(2):189– 209 [23] Horprasert T, Harwood D, Davis L (1999) A statistical approach for realtime robust background subtraction and shadow detection. In: FRAME-RATE: Frame-rate applications, methods and experiences with regularly available technology and equipment [24] Howe N (2007) Silhouette lookup for monocular 3d pose tracking. Image and Vision Computing 25:331–341 [25] Huttenlocher DP, Klanderman GA, Rucklidge WJ (1993) Comparing images using the Hausdorf distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(9):850–863 [26] Isard M, Blake A (1998) CONDENSATION - conditional density propagation for visual tracking. International Journal of Computer Vision 29(1):5–28 [27] Isard M, MacCormick J (2001) BraMBLe: a bayesian multiple-blob tracker. In: IEEE International Conference on Computer Vision, vol 2, pp 34–41 [28] Jepson AD, Fleet DJ, El-Maraghi TF (2003) Robust Online Appearance Models for Visual Tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(25):1296–1311 [29] Kakadiaris L, Metaxas D (2000) Model-based estimation of 3D human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1453–1459 [30] Kanaujia A, Sminchisescu C, Metaxas D (2007) Semi-supervised hierarchical models for 3d human pose reconstruction. In: IEEE Conference on Computer Vision and Pattern Recognition [31] Kanaujia A, Sminchisescu C, Metaxas D (2007) Spectral latent variable models for perceptual inference. In: IEEE International Conference on Computer Vision [32] Kollnig H, Nagel HH (1997) 3d pose estimation by directly matching polyhedral models to gray value gradients. International Journal of Computer Vision 23(3):283–302 [33] Kong A, Liu JS, Wong WH (1994) Sequential imputations and bayesian missing data problems. Journal of the American Statistical Association 89(425):278–288 [34] Kuipers JB (2002) Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace, and Virtual Reality. Princeton University Press [35] Lee CS, Elgammal A (2007) Modeling view and posture manifolds for tracking. In: IEEE International Conference on Computer Vision [36] Li R, Tian TP, Sclaroff S (2007) Simultaneous learning of non-linear manifold and dynamical models for high-dimensional time series. In: IEEE International Conference on Computer Vision [37] Metaxas D, Terzopoulos D (1993) Shape and nonrigid motion estimation through physics-based synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(6):580–591
Video-Based People Tracking
85
[38] Moeslund TB, Hilton A, Krüger V (2006) A survey of advances in visionbased human motion capture and analysis. Computer Vision and Image Understanding 104(2-3):90–126 [39] Mori G, Malik J (2002) Estimating human body configurations using shape context matching. In: IEEE European Conference on Computer Vision, pp 666–680 [40] Navaratnam R, Fitzgibbon A, Cipolla R (2007) The joint manifold model for semi-supervised multi-valued regression. In: IEEE International Conference on Computer Vision [41] Neal RM (1993) Probabilistic inference using markov chain monte carlo methods. Tech. Rep. CRG-TR-93-1, Department of Computer Science, University of Toronto [42] Neal RM (2001) Annealed importance sampling. Statistics and Computing 11:125–139 [43] Nestares O, Fleet DJ (2001) Probabilistic tracking of motion boundaries with spatiotemporal predictions. In: IEEE Conference on Computer Vision and Pattern Recognition, vol II, pp 358–365 [44] Ning H, Xu W, Gong Y, Huang TS (2008) Latent pose estimator for continuous action recognition. In: IEEE European Conference on Computer Vision [45] North B, Blake A (1997) Using expectation-maximisation to learn dynamical models from visual data. In: British Machine Vision Conference [46] Pavolvic V, Rehg J, Cham TJ, Murphy K (1999) A dynamic bayesian network approach to figure tracking using learned dynamic models. In: IEEE International Conference on Computer Vision, pp 94–101 [47] Plankers R, Fua P (2001) Articulated soft objects for video-based body modeling. In: IEEE International Conference on Computer Vision, vol 1, pp 394–401 [48] Poon E, Fleet DJ (2002) Hybrid Monte Carlo filtering: edge-based people tracking. In: Workshop on Motion and Video Computing, pp 151–158 [49] Prati A, Mikic I, Trivedi MM, Cucchiara R (2003) Detecting moving shadows: Algorithms and evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(7):918–923 [50] Ramanan D, Forsyth DA, Zisserman A (2007) Tracking people by learning their appearance. IEEE Transactions on Pattern Analysis and Machine Intelligence 29:65–81 [51] Rehg J, Kanade T (1995) Model-based tracking of self-occluding articulated objects. In: IEEE International Conference on Computer Vision, pp 612–617 [52] Ren L, Shakhnarovich G, Hodgins J, Pfister H, Viola P (2005) Learning silhouette features for control of human motion. ACM Transactions on Graphics 24(4):1303–1331 [53] Rosales R, Sclaroff S (2002) Learning body pose via specialized maps. In: Advances in Neural Information Processing Systems [54] Rosenhahn B, Kersting U, Powel K, Seidel HP (2006) Cloth X-Ray: MoCap of people wearing textiles. In: Pattern Recognition, DAGM
86
Marcus A. Brubaker, Leonid Sigal and David J. Fleet
[55] Shakhnarovich G, Viola P, Darrell TJ (2003) Fast pose estimation with parameter-sensitive hashing. In: IEEE International Conference on Computer Vision, pp 750–757 [56] Sidenbladh H, Black M, Fleet D (2000) Stochastic tracking of 3d human figures using 2d image motion. In: IEEE European Conference on Computer Vision, vol 2, pp 702–718 [57] Sidenbladh H, Black MJ, Sigal L (2002) Implicit probabilistic models of human motion for synthesis and tracking. In: IEEE European Conference on Computer Vision, vol 1, pp 784–800 [58] Sigal L, Black MJ (2006) Measure locally, reason globally: Occlusionsensitive articulated pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 2, pp 2041–2048 [59] Sigal L, Balan A, Black MJ (2007) Combined discriminative and generative articulated pose and non-rigid shape estimation. In: Advances in Neural Information Processing Systems [60] Sminchisescu C, Jepson A (2004) Generative modeling for continuous nonlinearly embedded visual inference. In: International Conference on Machine Learning, pp 759–766 [61] Sminchisescu C, Kanaujia A, Li Z, Metaxas D (2005) Discriminative density propagation for 3d human motion estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 1, pp 390–397 [62] Sminchisescu C, Kanajujia A, Metaxas D (2006) Learning joint top-down and bottom-up processes for 3d visual inference. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 2, pp 1743–1752 [63] Stauffer C, Grimson W (1999) Adaptive background mixture models for realtime tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 2, pp 246–252 [64] Stenger BDR (2004) Model-based hand tracking using a hierarchical bayesian filter. PhD thesis, University of Cambridge [65] Sukel K, Catrambone R, Essa I, Brostow G (2003) Presenting movement in a computer-based dance tutor. International Journal of Human-Computer Interaction 15(3):433–452 [66] Taylor CJ (2000) Reconstruction of articulated objects from point correspondences in a single uncalibrated image. Computer Vision and Image Understanding 80(10):349–363 [67] Tomasi C, Kanade T (1991) Detection and tracking of point features. Tech. Rep. CMU-CS-91-132, Carnegie Mellon University [68] Urtasun R, Darrell T (2008) Local probabilistic regression for activityindependent human pose inference. In: IEEE Conference on Computer Vision and Pattern Recognition [69] Urtasun R, Fleet DJ, Hertzmann A, Fua P (2005) Priors for people tracking from small training sets. In: IEEE International Conference on Computer Vision, vol 1, pp 403–410
Video-Based People Tracking
87
[70] Urtasun R, Fleet DJ, Fua P (2006) 3D people tracking with gaussian process dynamical models. In: IEEE Conference on Computer Vision and Pattern Recognition, vol 1, pp 238–245 [71] Urtasun R, Fleet DJ, Fua P (2006) Motion models for 3D people tracking. Computer Vision and Image Understanding 104(2-3):157–177 [72] Vondrak M, Sigal L, Jenkins OC (2008) Physical simulation for probabilistic motion tracking. In: IEEE Conference on Computer Vision and Pattern Recognition [73] Wachter S, Nagel HH (1999) Tracking persons in monocular image sequences. Computer Vision and Image Understanding 74(3):174–192 [74] Wang JM, Fleet DJ, Hertzmann A (2006) Gaussian process dynamical models. In: Advances in Neural Information Processing Systems 18, pp 1441–1448 [75] Wren CR, Pentland A (1998) Dynamic models of human motion. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp 22– 27 [76] Wren CR, Azarbayejani A, Darrell T, Pentland AP (1997) Pfinder: Real-time tracking of the human body. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):780–785
Locomotion Activities in Smart Environments Björn Gottfried
1 Introduction One subarea in the context of ambient intelligence concerns the support of moving objects, i. e. to monitor the course of events while an object crosses a smart environment and to intervene if the environment could provide assistance. For this purpose, the smart environment has to employ methods of knowledge representation and spatiotemporal reasoning. This enables the support of such diverse tasks as wayfinding, spatial search, and collaborative spatial work. While this chapter provides methods for dealing with these issues, a specific focus lies on the research question as to which minimal information already suffices a smart environment to deal with approximate motion information in a robust and efficient way, that is under realistic conditions. It shows that the thorough analysis of the task at hand enables the smart environment to get by on a minimum of sensory information: the choice of an adequate representation and the use of knowledge about the smart environment forces a simple technical implementation. Additionally, this requires to process just a minimum of sensory information. A number of case studies will be introduced and carefully examined in order to demonstrate how smart environments can effectively make use of locomotion patterns.
1.1 Why Locomotion Matters The objective of smart environments is to support human beings who live and act in such environments. This requires the analysis of their behaviours in order to enable a smart environment to determine how it can help. A fundamental category of behaviours are locomotion behaviours, addressing the question as to how someone Björn Gottfried Centre for Computing Technologies, University of Bremen, e-mail:
[email protected] H. Nakashima et al. (eds.), Handbook of Ambient Intelligence and Smart Environments, DOI 10.1007/978-0-387-93808-0_4, © Springer Science+Business Media, LLC 2010
89
90
Björn Gottfried
changes his location over time. This chapter investigates locomotion behaviours. It looks at how the motion of someone who crosses a smart environment can be described. For this purpose a top down approach is proposed. It asks for the distinctions which are relevant regarding specific tasks. The goal of this chapter is to clarify in logical terms what the essential ingredients of smart environments are insofar such environments make use of how, and that at all, people move around. This makes it necessary to analyse locomotion behaviours in relation to specific tasks and problems. In doing so, only a specific aspect of the concept of motion, or rather locomotion, is of interest here: motion always results in a change of location; and what we shall consider in the context of a smart environment is where objects are within this environment: which locations an object has visited or should visit in order to cope with a specific task. The result will provide a specification which aids the implementation of smart environments that model object motion in terms of locations and changes of locations.
1.2 Overview The body of this chapter is structured as follows. The next section describes how the presented approach fits into the related work (Section 2). After that it is analysed which concepts are relevant in the context of a smart environment that monitors how people move around. For this purpose, a number of scenarios describe in which way a smart environment could support those people (Section 3). From these scenarios a formalisation can be derived that defines functional relationships among locations (Section 4). These will be used to define more specific algorithms that show how a smart environment will deal with the challenges described in the previous scenarios (Section 5). The chapter concludes with a discussion, outlook, and summary (Sections 6, 7 and 8).
2 Related Work on Motion Analysis and Smart Environments Any analysis of object motion has to make several commitments. We have identified in particular five objectives we should look at when dealing with motion information in smart environments. (1) Different motion parameters exist. Which are relevant to realise smart environments? (2) The precision to describe object motion can be arbitrarily fine. How fine should the precision be in order to solve the problems at hand? (3) Object motion can be described from different viewpoints. Which is the viewpoint a smart environment should adopt? (4) Human navigation and wayfinding plays an important role in smart environments. Which aspects should be taken into account? (5) Eventually, bottom-up and top-down approaches to motion analysis can be distinguished. Does one of these strategies take precedence over the other in the context of smart environments? Besides the consideration of issues concerning
Locomotion Activities in Smart Environments
91
motion analysis, a look at past investigations on smart environments shows that the topics of this chapter complement those investigations.
2.1 Motion Parameters The motion of a body is frequently described in terms of the relation between the distance it covers and the time it needs to cover this distance. The notions of velocity and acceleration can then be used in order to describe how fast an object moves and how it changes its speed. But the velocity of a body is not the only quantity of interest in motion analysis. In the context of smart environments we are primarily interested in where an object moves and where it intends to move to. This is why we shall analyse how to locate objects in smart environments. In recent years many techniques have been invented that enable indoor localisation of objects, for instance by means of RFID [52, 26], smart floors [49, 44], wireless LAN [32, 28], or infrared positioning techniques [29, 27]; especially, different techniques enable a different resolution of location information; overviews of techniques at the sensory level for locating objects and identifying them are provided by [36, 24]. A location model for smart environments that includes in particular the consideration of locations of computing devices and services and which focuses on location-aware and user-aware services is proposed by [48]. Regardless of the positioning technique used, algorithms are necessary which meaningfully make use of positional information in smart environments. Such algorithms should be independent of the sensors employed because sensor technology rapidly improves. That is, motion analysis algorithms should abstract from the sensory level. Assumption 1 (Sensor independence) Methods for motion analysis should be sensor independent.
2.2 Precision At first glance, it seems advantageous to deploy sensors in a way to obtain locations of objects as precise as possible. However, more important than precision is accuracy which concerns the correctness of information. We argue that coarse but correct information about the location of someone in a smart environment is better than precise information with a high probability of inaccuracy. Spatial and temporal information which fall into this class of coarse concepts are investigated in the field of qualitative representations [6]; and with regard to built environments see in particular [5]. Even more important is that coarse information is frequently adequate. It is this latter statement that we shall prove to be valid for a couple of meaningful tasks in smart environments.
92
Björn Gottfried
Assumption 2 (Information coarseness) Coarse sensory information is sufficient for a number of important problems arising in the context of motion analysis.
2.3 Viewpoint For a robust and efficient solution to the problem of motion analysis qualitative spatial reasoning methods are employed [43]. These methods originally derive from combinatorial, computational, and discrete geometry [9, 15] and have been brought into the field of qualitative navigation by [35]. While such techniques are employed for simultaneous localisation and mapping problems, for short SLAM [34], the design of a smart environment differs significantly from the design of a mobile agent. A smart environment is distinguished by knowing its infrastructure that supports agents moving in it, whereas the task of mobile agents frequently consists in finding their way in an unknown environment. In both cases qualitative spatial reasoning techniques are useful. What these cases distinguish are their viewpoints: a smart environment has an allocentric view on the entire environment, while a mobile agent has just an egocentric, and hence partial view on a larger area it is contained in. As opposed to the egocentric perspective, an allocentric view is considered to be independent of the actual position of the perceiver since it is anchored to the external space and defines spatial relations with respect to elements of the environment [50]. It is the aim of the present chapter to analyse how to solve motion analysis problems by means of the allocentric view of a smart environment. Assumption 3 (Allocentric view) A smart environment should adhere to an allocentric view.
2.4 Wayfinding Investigating how a smart environment could support locomotion activities of humans relates to a large body of work on human navigation and wayfinding. Problems are taken into account from the point of view of architectural design, spatial orientation and wayfinding [12]; means to generate adaptive route directions are elaborated on [47]; and, from the psychological perspective effects of familiarity, visual access, and orientation aids are considered [13]. As opposed to such external cues that help in navigation and wayfinding, [25] investigate how cognitive maps determine wayfinding tasks from the point of view of Cognitive Sciences. The choice of salient features which are used in wayfinding tasks is treated by [53] and means for analysing behaviour patterns of pedestrians are investigated in [41]. Additionally, in the Artificial Intelligence community much work exists on navigation in physical environments [11, 33, 38, 30].
Locomotion Activities in Smart Environments has
Smart Environment
93 Allocentric View
needs
Location Maintenance
is based on
Information Coarseness
allows
Sensor Independence
defines
Sensory Equipment
Fig. 1 Assumptions to be satisfied by smart environments.
We shall consider human navigation and wayfinding problems. For this purpose a smart environment will be assumed that has a map of its infrastructure (in accordance with the allocentric view Assumption) and that maintains the whereabouts of each mobile object using this infrastructure. On this basis algorithms can be defined which solve wayfinding problems. Assumption 4 (Location maintenance) A smart environment should maintain the whereabouts of each mobile object.
2.5 Modelling One line of research employs data mining and learning techniques in order to search for regularities in the data. In [8] for example anomalies in event data of a health care smart home are sought by data mining techniques. Such approaches argue that smart environments are richly equipped with sensors that accumulate huge amounts of sensory data. This makes it necessary to investigate how to make sense of these data, similar to sensor networks for which challenges arise from the volumes of data collected by the huge number of mobile and stationary sensors [4]. Methods in the field of visual analytics are yet another way to search for regularities in large data sets [3]. We shall follow another line of research that instead analyses a given problem domain in order to determine which sensory information is relevant for specific tasks. Such a top-down analysis enables one to determine the sensors which are actually necessary, and hence, leads to data reduction and avoids the need for data mining. Assumption 5 (Sensory equipment) An application driven top-down analysis determines the necessary sensors.
94
Björn Gottfried
2.6 Summary A number of assumptions have been identified which should be realised by smart environments that make use of locomotion behaviours. Later on methods will be introduced that satisfy these assumptions. They are summarised in Fig. 1.
3 Locomotion Activities in Smart Environments This section analyses locomotion behaviours in smart environments. For this purpose a specific scenario is chosen – a smart hospital (Section 3.1). The complexity of such an environment should enable the thorough investigation of most of the relevant issues concerned with the analysis of motion activities in smart environments. Four scenarios are discussed that differ with regard to significant aspects (Section 3.2). Their analysis aids in identifying fundamental characteristics of smart environments (Section 3.3).
3.1 Smart Hospitals The idea of a smart hospital is not new. Until now however smart hospitals concentrate on increasing the efficiency of clinical processes and knowledge management [23, 39]. Moreover, they support medication and primarily aim at paperless documentation. For example the Mountain View’s El Camino Hospital in the USA is using technologies to improve communication among staff members and to make processes more efficient: nurses and other staff members use wireless, hands-free devices to communicate with each other; devices are employed that allow authorised staff members to open cabinet doors to medications and other supplies with thumbprint readings; tablet PCs and handhelds are used by doctors who will eventually do away with the clipboard; supply-replenishing systems are used that automatically alert the suppliers of the hospital when anything from needles to dressing kits runs low; lab samples move through various processing stations on an automated production line [7]. Apart from the lab samples, will locomotion based technologies also have a future in a smart hospital? Motion behaviours come into existence in several different ways. Here we shall focus on those motion activities in which a human being changes her location over time, that is we are concerned with locomotion behaviours. The consideration of what is going on at a hospital site allows a broad spectrum of different locomotion behaviours to be taken into account that might arise in a smart environment. The environment of a hospital has a complex structure. It consists of a building or even of a set of buildings on a larger area that define a complex spatial infrastructure. This infrastructure serves several purposes which require people to move around. In this context, a smart hospital should support people finding their way or coping with
Locomotion Activities in Smart Environments
95
Fig. 2 A taxonomy of motion processes at a hospital site.
other tasks within the hospital site. This support strongly depends on several factors including the urgency of a task, the roles of the people involved, as well as temporal and spatial constraints. Fig. 2 shows a taxonomy of some typical locomotion processes at a hospital site. It distinguishes single moving objects as opposed to groups of moving objects. Then, it looks at different spatial scales in order to find specific scenarios that involve specific locomotion behaviours. We shall investigate some of them in Section 3.2. Their analysis aids in identifying fundamental characteristics which are relevant for realising these scenarios within a smart environment (Section 3.3). The results of this analysis will be used for a functional specification which relates the different characteristics (Section 4). Eventually, algorithms can be defined on this specification (Section 5). Fig. 3 summarises this design process: the arrows indicate the information flow. It shows that the concepts can be derived from the introduced scenarios, that the specification is based on the found concepts, and that the algorithms are defined by the scenarios and by the specification. While scenarios and algorithms are specific in that they describe individual cases, the concepts and specification are defined in a general way.
96
Björn Gottfried
Specific
General
Scenarios (3.2)
Concepts (3.3)
Algorithms (5)
Specification (4)
Fig. 3 Design process used: numbers indicate corresponding sections, arrows information flow. Scenarios and algorithms are specific, while concepts and specification are generally defined.
3.2 Scenarios in a Smart Hospital For each of the following scenarios a problem or challenge is identified. Afterwards it is described how the smart environment should cope with this challenge. This concerns the first design stage (see Fig. 4). Scenario 1 (Planning) A nurse has to take care of several patients. To decide whom to visit next, urgency, space constraints, and other tasks are to be taken into account. In order to realise this scenario the smart environment should provide means for tracking individuals. It is however sufficient to determine positions at the level of single rooms. The smart environment can then tell the nurse where she has to go next. More formally, the whereabouts of a nurse as well as the rooms of the patients she has to visit are represented by regions. A nurse is contained in a region, and as far as regions represent disjoint places (different rooms and hallways), she is contained in exactly one region. The smart environment further represents the arrangement of regions at the hospital site. In this way, it can provide the nurse an efficient plan about where to go next when she is finished with a patient. Describing the path the nurse has to take amounts to consider an ordered list of regions she has to cross. Scenario 2 (Wayfinding) A visitor is going to visit a patient and he wants to talk to the responsible doctor. The visitor is in this hospital for the first time and does not know where to go. An additional challenge is that the responsible doctor will himself move around instead of being in his office all the time. The smart environment has to track an individual. In general the visitor will not be familiar with the environment, meaning for the smart environment to provide him information about where to go. For this purpose a map could be displayed at the wall which the visitor will most likely perceive depending on from where he enters a specific room or hallway. In particular, a personalised map will be confined to what is the most relevant content for the given situation; for customised maps see [51].
Locomotion Activities in Smart Environments
97
Specific
General
Scenarios (3.2)
Concepts (3.3)
Algorithms (5)
Specification (4)
Fig. 4 Design process used: numbers indicate corresponding sections, arrows information flow. Scenarios and algorithms are specific, while concepts and specification are generally defined.
This is in fact a challenge for the smart environment as soon as a number of visitors cross the same hallway. In short, the smart environment will lead the visitor to his destination after he has told the system at the entrance who he wants to see. More formally, the visitor is contained in a specific region which is the region of the entrance hall. There he lets the system know who he wants to visit, patient or doctor. Since the system knows everything about the whereabouts of everybody it can compute the order of regions the visitor has to pass. Even if the patient or doctor changes his location in the meantime the smart environment can react promptly by adapting the order of regions the visitor has to cross to reach his goal. Scenario 3 (Searching) A patient needs help, requiring as fast as possible a doctor to be called and also a nurse to provide assistance. The smart environment has to look for which doctor and which nurse is closest to the patient; it might also consider whether doctor and nurse are available, and if not, to look for the next available doctor and nurse. As soon as the smart hospital has identified doctor and nurse it has to show them effectively where to go. Furthermore, it can provide them with as much meaningful information about the state of the patient as available. More formally, both doctor and nurse are contained in either the same region or in different regions. As in the wayfinding scenario the smart environment can tell doctor and nurse the order of regions they have to cross in order to get as fast as possible to the patient. Although they are familiar with the environment, the smart environment will help in getting to the target region as fast as possible. Additionally, further information about the patient and his state can be shown on smart displays at appropriate places on their way or by a smart audio system. Scenario 4 (Monitoring) It is not only of interest for the doctor to monitor the vital signs of a patient but also how they correlate with his spatial activities. This is for the purpose of analysing his progress. The system has to track an individual who is a patient and who will therefore be within the smart environment for a longer period of time, for example a week. The
98
Björn Gottfried
Table 1 Locomotion scenarios summarised in terms of participating continuants and activities
Scenario
Continuant
Activities
planning wayfinding searching monitoring
nurse visitor doctor patient
perform tasks at different locations search for locations follow path to locations show modes of movements at locations
system will monitor his recovery by observing how often he leaves his bed, room, or even the building and by correlating his locomotion with his vital signs. The smart environment will tell the responsible doctor whether the patient makes advances. This might take place during the ward round, that is a display will show the doctor how the vital signs of the patient correlate with his activities. More formally, as opposed to the previous scenarios finer locomotion behaviours might be of interest. For instance, how fast the patient moves through his room as between bed and window, or between bed and washing room. Those parts of the room can also be represented by regions, but instead of measuring only whether the patient is contained in a region the speed with which he reaches another region is also of interest.
3.3 Characterising Locomotion Activities In the following a set of four different categories are identified that derive from the previous scenarios which are summarised in Table 1. These categories will aid in designing smart environments by showing which aspects are to be taken into account (see Fig. 5).
3.3.1 Continuants The first category, continuants, is about the people who are to be supported by the smart environment, and hence, which are to be monitored: a patient, a visitor, a nurse, or a doctor. In most cases the smart environment has to provide support for a single person. But as the searching scenario shows sometimes the collaboration of a number of people is required and needs to be supported by the smart environment. When a group of people, such as a team of doctors and nurses who are preparing an operation, have to work together the smart environment could suggest a room, could even reserve it, as well as remind and lead the continuants to it when it is time to get together — the smart environment is the common memory of the continuants who have to work together. Many possibilities exist on how a collection of people need to act together. A classification scheme for collectives can be found in [54]
Locomotion Activities in Smart Environments
99
Specific
General
Scenarios (3.2)
Concepts (3.3)
Algorithms (5)
Specification (4)
Fig. 5 Design process used: numbers indicate corresponding sections, arrows information flow. Scenarios and algorithms are specific, while concepts and specification are generally defined.
who analyse typical characteristics of different kinds of groups and their movement patterns.
3.3.2 Events Each scenario can be conceived of as some more or less specific event that corresponds to a specific problem or task the continuants are confronted with: an event might be defined by a couple of tasks which are to be performed at different locations (planning scenario); an event might consist in finding the way between locations in an unknown environment (wayfinding scenario); an event might be defined as a search problem, for example specific people with a specific competence are to be sought as fast as possible (searching scenario); not so much an event but rather an ongoing process is the observation of the spatial activities of the patient; but then, an event is defined by a specific time interval for which measurements are taken (monitoring scenario). This summary makes clear how the scenarios generalise to other domains in which also planning, wayfinding, searching, and monitoring problems in space arise. In general, such events can be separated in domain independent events: {wayfinding, searching, hiding, provide security} and domain dependent events: {operation, round, serving meals, visiting, cleaning}
3.3.3 Time The time is a fundamental dimension in that the speed of supporting someone might be crucial, as in the searching scenario. The speed that is needed that someone gets from one location to another location is relevant in this scenario, as is the speed with which someone is crossing a region like in the monitoring scenario. The speed is normally captured by the concept of velocity that relates a temporal interval to
100
Björn Gottfried
a distance someone has travelled during this time interval. Instead of measuring a precise velocity value it might be sufficient to relate a temporal interval to the number of regions someone has passed or has to pass while walking between specific locations; additionally the smart environment knows the distances covered for every pair of regions. For a qualitative concept of velocity see [42]. Also, the temporal order of how something is performed might be relevant (planning scenario), or rather sufficient, as well as how events temporally relate [2].
3.3.4 Space Each event takes place in space and the smart environment itself has a specific spatial structure that determines how it has to provide support. Among others, spatial aspects are about considering space at different scales. The largest scale concerns the whole hospital site (of interest for all scenarios and in particular for the wayfinding scenario); intermediate scales concern connected portions of it (primarily of interest for the planning scenario); the smallest scale is that one of a single human (monitoring scenario). It is the idea to approximate locations at specific scales which are relevant for the given tasks. We will do this later at the level of rooms. In other words a function called position will be introduced that returns regions corresponding to rooms and hallways. In addition, the distance between locations might be of interest and should be defined in accordance to the spatial scale used. Moreover, the neighbourhood of a region refers to all other regions which are directly connected to that region. This concept is of interest when the smart environment provides the user with information about which region he should enter next in order to get to a specific destination, as in the wayfinding scenario. Other spatial aspects such as a closer relation might be of interest. Then, both the neighbourhood and the closer relation can be employed for realising the planning scenario.
3.3.5 Distinguishing Events Each of these four categories determines locomotion behaviours and can therefore be used for linking observable criteria, primarily concerning temporal and spatial constraints, to knowledge about specific events, thus provide the smart environment with what it needs to know in order to support people. Then, the nurse who starts to work notifies the smart environment that it knows that it has to provide her assistance during her round; this relieves her from planning this round. By contrast, the visitor registers himself at the entrance and triggers a wayfinding process, while the urgency case of a patient triggers a search process. Finally, a monitoring process is launched when a new patient checks in. There might be further temporal and spatial aspects of interest in other domains. An overview on spatial and temporal reasoning techniques for smart homes can be found in [19] and further issues about spatiotemporal reasoning and context awareness can be found in [21].
Locomotion Activities in Smart Environments
101
4 The Representation of Locomotion Activities The allocentric viewpoint of a smart environment enables it to determine where a moving agent is. In this way it can track any mobile object that moves through it. A functional specification that realises this is presented in this section. At first, some general issues concerning the treatment of spatiotemporal information are discussed in 4.1. Then a representation describing the functional relationships for realising the previous scenarios are presented in Section 4.2, that is a representation is proposed which is entirely based on regions and allows to distinguish both the locations where objects are and relations among those locations. Accordingly, motion patterns are described by means of locations of objects and the change of locations. Section 4.3 describes the allocentric component in this context and how the allocentric viewpoint is part of every process that solves a problem within the smart environment. Section 5 will afterwards show what the algorithmic solution of the previously described scenarios look like on this basis.
4.1 Spatiotemporal Information Spatiotemporal information has been thoroughly dealt with in the past. Many approaches are related to the Triad framework of [45] who distinguishes the three dimensions referred to as What, Where and When, similar to [40] who distinguish location, time, and theme perspectives. Here the What or theme component divides into continuants and occurrences (events), continuants enduring through time and occurrences being extended in time [20]. As the examples of the previous section indicate, a formalisation based on regions is sufficient. A region based formalism is the 9-intersection calculus [10] and the region connection calculus [46] which set into relation pairs of regions in a topological way. Here we are not interested in comparing how regions relate topologically but in describing how a person crosses a smart environment that is divided into regions. This is similar to [16] who proposes to consider the tessellation of an environment as a basis relative to which locomotion patterns are described. Another related approach is that one of [31] who describe movement patterns of single agents represented by directed lines in relation to regions.
4.2 Functional Specification We shall now turn our attention to specifying relations on the concepts previously introduced (see Fig. 6). According to what has been derived in Section 3.3, we assume a primitive ontology of continuants c ∈ C, events e ∈ E, times t ∈ T , and regions r ∈ R. In the case of a hospital the regions represent places such as entrance hall, hallways, and rooms. The four domains are related by means of a couple of
102
Björn Gottfried
Specific
General
Scenarios (3.2)
Concepts (3.3)
Algorithms (5)
Specification (4)
Fig. 6 Design process used: numbers indicate corresponding sections, arrows information flow. Scenarios and algorithms are specific, while concepts and specification are generally defined.
functions that define how the smart environment makes sense of these domains. The first one determines the position of a continuant at a specific time: position : C × T → R
(1)
It is this function that solely defines the sensory equipment necessary for the smart environment. This is because for each other function nothing more is required than to determine in which region a specific continuant is. The sophistication of these functions is due to the smart environment knowing its spatial structure, so that for instance it can derive what the shortest path between two regions is. The smart environment defines a kind of metric that returns for two regions their minimal distance: distance : R × R → R (2) Based on this a function can be defined that returns which of two regions is closer to a third one: closer : R × R × R → R (3) Then this function can itself be used to determine the region one should enter when being in a specific region ri and when aiming at reaching another region: visitOnRoute : R × R → R
(4)
A somewhat more general function is needed when one is interested in determining for a set of regions which to visit next, e. g. the nearest one, given that one is contained in a specific region ri : visitNearest : R × 2R → R
(5)
Another distinction between (4) and (5) is that the former is restricted to search in the neighbourhood of the current region ri , while (5) searches in a set of regions that are not necessarily in the neighbourhood of ri . A number of continuants might be contained in the same region. The following function is needed in order to determine for a region who is contained in that region:
Locomotion Activities in Smart Environments
containedIn : R → 2C
103
(6)
and another function is needed that returns the role of a continuant: getRole : C → {nurse, doctor, patient, visitor, other}
(7)
All rooms which are in the direct neighbourhood of a given room, more precisely which are connected by a door, are returned by the following function: neighbourhood : R → 2R
(8)
The function position tells the smart environment the position of the person under consideration and a couple of functions can be defined on the basis of the distance function. Since the smart environment has a static spatial structure, much can be computed already when defining the smart environment. This includes distances between regions as well as all those functions that build upon the distance between regions: closer, visitOnRoute, and visitNearest. In principle, for a given smart environment these functions can be defined in advance by means of tables that enable efficient access. The defined functions can be conceived of as an abstract data type with which a number of more specific functions can be defined for the smart environment. Besides the position the role has to be determined automatically, for instance by RFID and a table that maps identifiers to roles. For determining the speed with which someone is crossing a specific region, in the simplest implementation it is just the time which is to be measured during which this person is contained in that region; this time and the size of the region are to be set into relation.
4.3 Allocentric View Each process solving a given problem of the smart environment has an allocentric view on this environment. This means that each process has access to a data structure that represents the spatial structure of the whole hospital site, enabling it to process the given functions of the previous paragraph. The distinct places of the environment can be represented by a graph that captures the topological structure of the environment. That is, in particular the neighbourhood of a place which is represented by a node can be directly read off this graph by following the edges which emanate from it. Each node or region or room has itself the complete structure of the smart environment annotated with distances, shortest path information and everything else which can be computed in advance. Also, the whereabouts of each continuant is maintained in such a graph structure. The latter is the only part which changes over time and a change is triggered whenever a mobile crosses the boundary between two distinct places. This is the only challenge the sensory equipment of the smart environment has to manage. Independent of the sensory equipment, this structure is used by the reasoning mechanisms
104
Björn Gottfried
Specific
General
Scenarios (3.2)
Concepts (3.3)
Algorithms (5)
Specification (4)
Fig. 7 Design process used: numbers indicate corresponding sections, arrows information flow. Scenarios and algorithms are specific, while concepts and specification are generally defined.
of the smart environment. Conversely, each sensory equipment which will maintain this graph structure can be deployed. In this way this structure forms the level of abstraction between the sensors at the basis and the reasoning mechanisms at the top end. Since each process has access to the current whereabouts of all persons within the smart environment, algorithms are straightforward that rely on the current position of somebody, that is the position function can always be used instead of documenting in a data structure where someone currently is or should be – which is in fact quite susceptible to errors. For short, the allocentric paradigm enables each process to determine at any time the whereabouts of everybody and this is the fundamental basis on which the following algorithms will build on. A role and an event type is assigned to each continuant. This enables each part of the smart environment to determine how it has to support the concerned individual. Additional parameters might also be necessary, such as the identifier of another continuant who the former continuant is looking for or a destination location.
5 Locomotion Based Algorithms In Section 4 a representation has been defined that includes both a number of domains and a number of functions that show the interrelationships of these domains. Thereby, regions and continuants are the two most important domains which manifest how people move around in space. We shall now investigate how we can make use of these domains and functions in order to define algorithms (see Fig. 7); these algorithms show that our assumptions are valid: the scenarios described earlier can be solved by the smart environment on the basis of a simple sensory equipment. This sensory equipment determines the whereabouts of continuants solely at the level of rooms and hallways, as well as their roles.
Locomotion Activities in Smart Environments
105
5.1 Planning Scenario There is a nurse who is to be supported by the smart environment that tells her which patient out of a set, called patients, she has to visit next. Here, the patient who is nearest to the nurse is selected. Other constraints are conceivable such as the degree of urgency. rooms = 0/ for i = 1 to |patients| do rooms = rooms ∪ {position(patients(i), now)} end for if rooms = 0/ then return visitNearest(position(nurse, now), rooms) else return report that no one is left end if Algorithm 1: Visit Next (nurse ∈ C, patients ⊂ C) This algorithm returns the room the nurse has to visit next, if there is any patient left. For this purpose the rooms are determined that contain the patients. Afterwards the room which is nearest to the position of the nurse is computed. The latter computation is to be replaced by another one as soon as other constraints are to be taken into account which identify the patient who is to be visited next.
5.2 Wayfinding Scenario A visitor is going to visit a patient. The visitor identifies himself at the entrance and the smart environment will lead him to the room of the patient. source = position(visitor, now) destination = position(patient, now) while source = destination do source = visitOnRoute(source, destination) print Now enter source source = position(visitor, now) destination = position(patient, now) end while Algorithm 2: Wayfinding (visitor ∈ C, patient ∈ C)
106
Björn Gottfried
Assuming that the patient will never leave his room, the last row in the algorithm could be omitted (destination = position(patient, now)). Similarly, the last but one row could also be omitted when the visitor would always take the correct path (source = position(visitor, now)); but this algorithm takes into account that the visitor might take the wrong path sometimes and therefore determines his position now and then. While this algorithm just outlines the main parts for supporting a wayfinding task, several extensions are conceivable, such as that the smart environment orders a lift in advance when the visitor has to change the floor or a lift could even be reserved for a doctor who has to get as fast as possible to a specific place; visitor or doctor even do not have to chose the correct button inside the lift since the lift already knows the destination. Similar to the case of the lift, doors could be automatically opened that lead to other parts of the hallway or into other rooms, so that the user is afforded to take the according door or the according lift – this being an example of how the concept of affordance by Gibson [14] might be an integral part of a smart environment. Or, the smart environment can determine how crowded a hallway is which is part of a route, and consequently can suggest alternative routes, especially if the doctor is in a hurry. Eventually, one might choose between the fastest route, the most comfortable route for those who cannot walk very well, or routes which support wheelchairs.
5.3 Searching Scenario The next algorithm searches for a doctor. This search starts at the room of the patient and successively takes into account rooms in the neighbourhood of the room of the patient, and then in the neighbourhood of the rooms in the neighbourhood, etc., until it has found a doctor. In this way, the doctor who is closest to the patient can be identified. A simple extension would check whether the doctor has currently time to visit the patient. Note how the variable checkedRooms takes care of terminating the search. Without subtracting the rooms already checked from the new neighbourhood, those rooms would always be included in the next iteration phase and the algorithm would endlessly check where the next doctor is. One might argue that this would not pose a problem. Either there is a doctor in the hospital, then the algorithm would find him. Or there is currently no doctor in the hospital. In the latter case it would make sense to loop through this algorithm again and again until there is a doctor available. One can easily imagine how this function generalises to finding a specific person having a particular role, and hence how the searching scenario is realised by the smart environment: This algorithm can easily be adapted for the purpose of calling someone with a particular competence to a specific place.
Locomotion Activities in Smart Environments
107
d := null N := neighbourhood(position(patient, now)) checkedRooms := {position(patient, now)} while d = null do people := containedIn(N) for i := 1 to |people| do if getRole(people(i)) = doctor then d := people(i) end if end for checkedRooms := checkedRooms ∪ N N = neighbourhood(N) − checkedRooms end while Algorithm 3: Find Doctor for Patient (patient ∈ C) d := Find Person for Patient (doctor, patient) n := Find Person for Patient (nurse, patient) Wayfinding (d, patient) Wayfinding (n, patient) Algorithm 4: Lead Doctor and Nurse to Patient (patient ∈ C)
5.4 Monitoring Scenario The monitoring scenario is an example of how the smart environment can make use of the available motion data which is acquired over a longer period of time. It memorises the whereabouts of every patient. This enables it to determine in particular how long one needs to reach the washing room or to pace along a hallway or to get to the canteen. The speed information with which a patient crosses different regions can be set into relation with his vital signs and this provides the doctor additional information concerning the activities of the patient and how well he deals with these activities. This example illustrates how monitoring scenarios split up into two different parts, the monitoring and the interpretation part. The monitoring part captures the positional data, and therewith the locomotion behaviour of a patient; it repeatedly saves positional information by taking into account a defined time-lag: The function put is a database function that saves the position of a given continuant at a given time and the function delay stops the tracking of a continuant for an amount of time which is given by break. Several interpretation methods for positional data are conceivable, depending on what a doctor will evaluate. The following example determines how long the patient stayed on average in a region, which approximately indicates the mean velocity with which the patient crossed this specific region, for example a hallway:
108
Björn Gottfried
t = now put positionalDB for patient to position(patient, t) at t delay(break) Monitor Patient Position (patient, break) Algorithm 5: Monitor Patient Position (patient ∈ C, break ∈ R) let durations = select p.tend − p.tstart from positionalDB as p where p.id = patient and p.location = r averageDuration = 0 numO fVisits = 0 for i := 1 to |durations| do averageDuration := averageDuration + durations(i) numO fVisits = numO fVisits + 1 end for return averageDuration / numO fVisits Algorithm 6: Observe Patient in Region (patient ∈ C, r ∈ R)
This example shows how the employment of spatiotemporal databases become of interest for evaluating what has been monitored about the locomotion of people in a smart environment. Note that the let statement is conceived of as a database function that retrieves in this case the temporal interval (tend − tstart ) during which the patient stayed in the given region. For a survey on spatiotemporal databases see [1]. A number of specific methods on moving object databases is provided by [22].
6 Discussion Whereas the present work exemplifies important issues on smart environments by the spatial infrastructure of a hospital, in [18] spatial aspects of diseases are investigated. There it has been analysed how diseases develop in space and time. A result of this analysis is that diseases themselves define the environment to be considered in order to track a disease. This is complementary to the current work that presumes a specific spatial infrastructure of a smart environment and analyses how events are realised within this infrastructure. Technical details about how to localise people in this infrastructure have been deliberately omitted. As has been argued in Section 2 there exist many sensors which would provide the smart environment with the required information. One could for instance easily imagine to equip everybody entering the entrance of the hospital with an RFID chip and each door frame with RFID reading devices. Specific identifiers
Locomotion Activities in Smart Environments
109
can then be mapped to specific roles of people. However, the goal of this work is to show how the confinement to a simple sensory equipment already enables a smart environment to provide people with sophisticated support. An adequate representation makes this possible. Coming back to the assumptions stated in Section 2, it can be shown how the chosen representation fulfils them. The sensor independence could easily be realised by the introduction of the position and getRole functions which are at the interface between sensory level and reasoning level. The information coarseness assumption is satisfied as we need not to distinguish finer locations than rooms and hallways in the scenarios. The position function simultaneously enables an allocentric view and takes care of the location maintenance. The sensory equipment derives from how position and getRole functions are implemented in a specific case. Taken these assumptions together, it has been shown how an abstraction level clearly defines the interface between the sensory level and the reasoning level: a graph structure representing the topology of the environment is used for maintaining the locations of mobile objects. The sensory level can be interchanged whenever new technologies are available. Changes at the sensory level do not influence the reasoning level. Moreover, this strategy argues in favour of a sensory equipment which is defined by what is actually needed from the point of view of the reasoning components, and thus from the tasks at hand. Such a top-down view on designing smart environments is not only meaningful for dealing with locomotion activities but also for monitoring any other stationary activities or changes. Confining the sensory technologies to what is actually relevant for a specific environment, data mining and learning techniques become obsolete. Those techniques might in fact be useful in order to learn something about the behaviours of people; this can help in inventing new smart devices and smart environments. But the final solution for a new technology should work properly after installation, from the very beginning. Otherwise the system behaviour would not be deterministic from the point of view of the user, i. e. regarding his perception; he would think that the system is defective as long as it behaves irregular.
7 Outlook Several extensions are conceivable in particular when deploying further positional sensors, allowing finer distinctions such as how often nurses have to get to specific cupboards in order to fetch medicine or specific devices or something else; knowledge about this might be of interest to optimise the spatial organisation of the hospital. Another scenario also based alone on the proposed positional technology is the generation of evacuation plans which enable in the case of fire the rescue service to determine where everybody is, who is attached to specific devices, who is not able to walk on his own, etc.; such a plan can be printed at any time; as soon as
110
Björn Gottfried
the whole technology breaks down, at least the last plan will be available at some service-point outside the burning hospital building. Or, a doctor or nurse might look for a colleague; since the smart environment knows everything about the whereabouts of the staff, including who is currently not within the hospital, it can tell everybody where to find another person; it can additionally check whether this person has enough time in order to meet, and if this is the case, the smart environment can suggest both continuants a place where they could meet, for example a close by conference room where they can talk in private. While the present chapter has focused on the concept of locomotion that is restricted to the consideration of rough locations and the change of locations, finer descriptions of locomotion behaviours might be of interest. Such small scale locomotion behaviours are for instance relevant in the case of behaviours of people suffering from cognitive impairments. In this context [37] distinguish different modes of locomotion the elderly show in a nursing home: random, i. e. changing frequently the direction of movement and hesitating now and then; lapping, i. e. taking awkward paths and going a long way round; and pacing, i. e. running to and fro between specific locations. A general calculus for characterising locomotion patterns among groups of people is given by [17]. Such a calculus becomes of interest when modelling collaborative spatial work, for example when a team of doctors and nurses perform an operation. Then the intersection of movement paths should be avoided, the spatial interplay among all people should be supported, and other means for optimising the spatial activities in a group of people should be planned and monitored. Therefore, small scale motion patterns of either single people [37] or groups of people [17] should be further elaborated on in order to make them available as further tools for monitoring complex behaviours in smart environments. This chapter has been basically framed as a vision piece that argues for the deployment of locomotion based support within smart environments. In doing so a first formalisation in terms of a functional specification has been suggested which will enable such technologies to be realised in the future. The next step will consist in further elaborating on the formal aspects of this specification and to put it into a broader context that shows how other application areas will profit from these technologies too.
8 Summary A class of tasks have been presented which can be supported by smart environments. These tasks have in common that they include a motion component. That is during such tasks people move around in space: they have to follow an unfamiliar path, they want to meet others or are looking for others, or they have to perform tasks successively at different locations. For these purposes, the smart environment can show them the way (wayfinding tasks), it can tell them everything about the whereabouts of others and themselves (search tasks), it can provide help in deciding where to go next in order to stick to constraints which optimise spatial conditions
Locomotion Activities in Smart Environments
111
(spatial planning tasks), and it can evaluate the past whereabouts of patients in order to analyse their behaviours (monitoring tasks). For the purpose of dealing with these tasks it has been shown that the coarse determination of the locations of people is sufficient. That is to say that the sensory equipment of a smart environment can be restricted in the sense that it just has to sense the whereabouts of people at the level of rooms and hallways as opposed to tracking people precisely. Such an approach is likely to be more robust in that the sensory output needs no particular precision. As a consequence, being at different locations at different times is the only way movements of people are expressed in this work. This suffices to cope with a number of tasks which require the locomotion of people in smart environments. A yet more general result is achieved. It could be shown that a top-down approach for designing smart environments has many advantages: it defines a clear interface between sensory level and reasoning level. Thereby it becomes possible to interchange the sensory level with more advanced technologies once available and without users becoming aware of this. Moreover, it is possible to decide which sensory information is to be processed at all in order to realise a smart environment with a specific functionality. This allows a least commitment strategy to be employed that looks for which sensory information will already be sufficient to realise the intended functionality. In the sense of the assumption of information coarseness (Assumption 2), this improves the robustness of the whole system that needs not to rely on precise measurements.
References [1] Abraham T, Roddick JF (1999) Survey of spatio-temporal databases. GeoInformatica 3(1):61–99 [2] Allen JF (1983) Maintaining knowledge about temporal intervals. Communications of the ACM 26(11):832–843 [3] Andrienko N, Andrienko G (2007) Extracting patterns of individual movement behaviour from a massive collection of tracked positions. In: Gottfried B (ed) Proceedings of the 1st Workshop on Behaviour Monitoring and Interpretation (BMI’07), CEURS Proceedings, vol 296, pp 1–16 [4] Balazinska M, Deshpande A, Franklin MJ, Gibbons PB, Gray J, Handsen M, Liebhold M, Nath S, Szalay A, Tao V (2007) Data Management in the Worldwide Sensor Web. Pervasive Computing 6(2):30–40 [5] Bittner T (2001) The qualitative structure of built environments. Fundamenta Informaticae 46(1-2):97–128 [6] Cohn AG, Hazarika SM (2001) Qualitative spatial representation and reasoning: An overview. Fundamenta Informaticae 43:2–32 [7] Colliver V (2004) Hospital of the future: High-tech system helps improve quality of care. San Francisco Chronicle p E 1
112
Björn Gottfried
[8] Cook DJ (2006) Health Monitoring and Assistance to Support Aging in Place. Journal of Universal Computer Science 12(1):15–29 [9] Edelsbrunner H (1987) Algorithms in combinatorial geometry. Springer, Berlin [10] Egenhofer M, Franzosa R (1991) Point-Set Topological Spatial Relations. International Journal of Geographical Information Systems 5(2):161–174 [11] Epstein SL (1997) Spatial representation for pragmatic navigation. In: Hirtle SC, Frank AU (eds) Spatial Information Theory COSIT ’97, LNCS, Springer, pp 373–388 [12] Gaerling T, Lindberg E, Maentylae T (1983) Orientation in buildings: Effects of familiarity, visual access, and orientation aids. Applied Psychology 68:177– 186 [13] Gaerling T, Book A, Lindberg E (1986) Spatial orientation and wayfinding in the designed environment: A conceptual analysis and some suggestions for postoccupancy evaluation. Journal of Architectural Planning Resources 3:55– 64 [14] Gibson J (1979) The ecological approach to visual perception. Hughton Mifflin Company, Boston [15] Goodman J, Pollack R (1993) Allowable sequences and order types in discrete and computational geometry. In: Pach J (ed) New trends in discrete and computational geometry, Springer, Berlin, pp 103–134 [16] Gottfried B (2006) Spatial health systems. In: Bardram JE, Chachques JC, Varshney U (eds) 1st International Conference on Pervasive Computing Technologies for Healthcare (PCTH 2006), November 29 – December 1, Innsbruck, Austria, IEEE Press, p 7 [17] Gottfried B (2008) Representing short-term observations of moving objects by a simple visual language. Journal of Visual Languages and Computing 19:321–342 [18] Gottfried B (2009) Modelling spatiotemporal developments in spatial health systems. In: Olla P, Tan J (eds) Mobile Health Solutions for Biomedical Applications, IGI Global (Idea Group Publishing), pp 270–284 [19] Gottfried B, Guesgen HW, Hübner S (2006) Spatiotemporal reasoning for smart homes. In: Augusto JC, Nugent CD (eds) Designing Smart Homes, The Role of Artificial Intelligence, LNCS, vol 4008, Springer, Heidelberg, pp 16– 34 [20] Grenon P, Smith B (2004) SNAP and SPAN: Towards dynamic spatial ontology. Spatial Cognition and Computation 4:69–104 [21] Guesgen HW, Marsland S (2009) Spatio-temporal reasoning and context awareness. In: Nakashima H, Augusto JC, Aghajan H (eds) Handbook of Ambient Intelligence and Smart Environments, LNCS (this volume), Springer, Heidelberg [22] Güting RH, Schneider M (2005) Moving Object Databases. Morgan Kaufmann Publishers, San Fransisco, CA, USA [23] Heine C, Kirn S (2004) Adapt at agent.hospital - agent based support of clinical processes. In: Proceedings of the 13th European Conference on Informa-
Locomotion Activities in Smart Environments
[24]
[25]
[26] [27]
[28]
[29]
[30] [31]
[32]
[33] [34]
[35] [36]
113
tion Systems, The European IS Profession in the Global Networking Environment, ECIS 2004, Turku, Finland, June 14–16, p 14 Hightower J, Borriello G (2001) A Survey and Taxonomy of Location Systems for Ubiquitous Computing. Technical Report UW-CSE 01-08-03, University of Washington, Computer Science and Engineering, Box 352350, Seattle, WA 98195 Hirtle S, Heidron B (1993) The structure of cognitive maps: Representations and processes. In: Gaerling T, Golledge R (eds) Behaviour and Environment: Psychological and Geographical Aspects, Elsevier Science Publishers, pp 170–192 Hopkinson A (2007) State of the Art in RFID Technology. INFOTHECA – Journal of Informatics and Librarianship 8(1–2):179–187 Kaufmann R, Bollhalder H, Gysi M (2003) Infrared positioning systems to identify the location of animals. In: Werner A, Jarfe A (eds) Joint conference of ECPA-ECPLF, ATB Agrartechnik Bornim / Wageningen Academic Publishers, p 721 Kitasuka T, Hisazumi K, Nakanishi T, Fukuda A (2005) Positioning Technique of Wireless LAN Terminals Using RSSI between Terminals. In: Yang LT, Ma J, Takizawa M, Shih TK (eds) Proceedings of the 2005 International Conference on Pervasive Systems and Computing, PSC 2005, Las Vegas, Nevada, June 27–30, 2005, CSREA Press, pp 47–53 Krohn A, Beigl M, Hazas M, Gellersen HW (2005) Using fine-grained infrared positioning to support the surface-based activities of mobile users. In: 25th International Conference on Distributed Computing Systems Workshops (ICDCS 2005 Workshops), 6–10 June 2005, Columbus, OH, USA, IEEE Computer Society, pp 463–468 Kuipers B (1978) Modeling spatial knowledge. Cognitive Science 2:129–154 Kurata Y, Egenhofer M (2007) The 9+ -Intersection for Topological Relations between a Directed Line Segment and a Region. In: Gottfried B (ed) Proceedings of the 1st Workshop on Behaviour Monitoring and Interpretation (BMI’07), CEURS Proceedings, vol 296, pp 62–76 Kushki A, Plataniotis KN, Venetsanopoulos AN (2008) Indoor Positioning with Wireless Local Area Networks (WLAN). In: Shekhar S, Xiong H (eds) Encyclopedia of GIS, Springer, pp 566–571 Leiser D, Zilbershatz A (1989) The traveler – a computational model for spatial network learning. Environment and Behaviour 21(3):435–463 Leonard JJ, Durrant-Whyte HF (1991) Simultaneous map building and localisation for an autonomous mobile robot. In: Proceedings of IEEE/RSJ International Workshop on Intelligent Robots and Systems, pp 1442–1447 Levitt T, Lawton D (1990) Qualitative navigation for mobile robots. Artificial Intelligence 44(6):305–360 Liu H, Darabi H, Banerjee P, Liu J (2007) Survey of wireless indoor positioning techniques and systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 37(6):1067–1080
114
Björn Gottfried
[37] Martino-Saltzman D, Blasch B, Morris R, McNeal L (1991) Travel behaviour of nursing home residents perceived as wanderers and nonwanderers. Gerontologist 11:666–672 [38] McDermott D, Davis E (1984) Planning routes through uncertain territory. Artificial Intelligence 22(1):551–560 [39] Mea VD, Pittaro M, Roberto V (2004) Knowledge management and modelling in health care organizations: The standard operating procedures. In: Wimmer M (ed) Knowledge Management in Electronic Government, 5th IFIP International Working Conference, KMGov 2004, Krems, Austria, May 17–19, 2004, Proceedings, Springer, LNCS, vol 3035, pp 136–146 [40] Mennis JL, Peuquet DJ, Qian L (2000) A conceptual framework for incorporating cognitive principles into geographical database recognition. International Journal of Geographical Information Science 14:501–520 [41] Millonig A, Gartner G (2007) Monitoring pedestrian spatio-temporal behaviour. In: Gottfried B (ed) Proceedings of the 1st Workshop on Behaviour Monitoring and Interpretation (BMI’07), CEURS Proceedings, vol 296, pp 29–42 [42] Monferrer EMT, Lobo FT (2002) Qualitative Velocity. In: Escrig MT, Toledo F, Golobardes E (eds) CCIA 2002, Springer Berlin Heidelberg, LNAI, vol 2504, pp 29–39 [43] Muller P (1998) A qualitative theory of motion based on spatio-temporal primitives. In: Cohn A, Schubert L, Shapiro S (eds) Proceedings of the 6th International Conference on Principles of Knowledge Representation and Reasoning, Morgan Kaufmann, pp 131–141 [44] Orr RJ, Abowd GD (2000) The Smart Floor: A Mechanism for Natural User Identification and Tracking. In: Proceedings of the 2000 Conference on Human Factors in Computing Systems (CHI 2000), pp 1–6 [45] Peuquet DJ (1994) It’s about time: A conceptual framework for the representation of temporal dynamics in geographic information systems. Annals of the Association of American Geographers 84:441–461 [46] Randell DA, Cui Z, Cohn AG (1992) A spatial logic based on regions and connection. In: Proc 3rd Int. Conf. on Knowledge Representation and Reasoning: Cambridge, Massachusetts, USA, October 25–29, 1992, Morgan Kaufman, San Mateo, pp 165–176 [47] Richter KF, Tomko M, Winter S (2008) A Dialog-Driven Process of Generating Route Directions. Computers, Environment and Urban Systems 32(3):26 [48] Satoh I (2007) A location model for smart environments. Pervasive and Mobile Computing 3:158–179 [49] Steinhage A, Lauterbach C (2008) Monitoring movement behaviour by means of a large area proximity sensor array in the floor. In: Gottfried B, Aghajan H (eds) 2nd Workshop on Behaviour Monitoring and Interpretation (BMI’08), vol 396, CEURS Proceedings, pp 15–27 [50] Volcic R, Kappers AML (2008) Allocentric and egocentric reference frames in the processing of three-dimensional haptic space. Experimental Brain Research 188:199–213
Locomotion Activities in Smart Environments
115
[51] Weakliam J, Bertolotto M, Wilson DC (2005) Implicit interaction profiling for recommending spatial content. In: Shahabi C, Boucelma O (eds) 13th ACM International Workshop on Geographic Information Systems, ACM-GIS 2005, November 4-5, 2005, Bremen, Germany, Proceedings, ACM, pp 285–294 [52] Weinstein R (2005) RFID: a technical overview and its application to the enterprise. IT Professional 7(3):27–33 [53] Winter S (2003) Route Adaptive Selection of Salient Features. In: Kuhn W, Worboys M, Timpf S (eds) Spatial Information Theory: Foundations of Geographic Information Science, Int. Conference COSIT 2003, LNCS, SpringerVerlag, Ittingen, Switzerland, pp 101–117 [54] Wood Z, Galton A (2008) Collectives and how they move: A tale of two classifications. In: Gottfried B, Aghajan H (eds) 2nd Workshop on Behaviour Monitoring and Interpretation (BMI’08), CEURS Proceedings, vol 396, pp 57–71
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion Manish Kushwaha, Songhwai Oh, Isaac Amundson, Xenofon Koutsoukos, Akos Ledeczi
1 Introduction Heterogeneous sensor networks (HSNs) are gaining popularity in diverse fields, such as military surveillance, equipment monitoring, and target tracking [41]. They are natural steps in the evolution of wireless sensor networks (WSNs) driven by several factors. Increasingly, WSNs will need to support multiple, although not necessarily concurrent, applications. Different applications may require different resources. Some applications can make use of nodes with different capabilities. As the technology matures, new types of nodes will become available and existing deployments will be refreshed. Diverse nodes will need to coexist and support old and new applications. Furthermore, as WSNs are deployed for applications that observe more complex phenomena, multiple sensing modalities become necessary. Different sensors usually have different resource requirements in terms of processing power, memory capacity, or communication bandwidth. Instead of using a network of homogeneous devices supporting resource intensive sensors, an HSN can have different nodes for different sensing tasks [22, 34, 10]. For example, at one end of the spectrum low data-rate sensors measuring slowly changing physical parameters such as temperature, humidity or light, require minimal resources; while on the other end even low resolution video sensors require orders of magnitude more resources. Manish Kushwaha Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA, e-mail:
[email protected] Songhwai Oh Electrical Engineering and Computer Science, University of California at Merced, Merced, CA, USA e-mail:
[email protected] Isaac Amundson Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA, e-mail:
[email protected] H. Nakashima et al. (eds.), Handbook of Ambient Intelligence and Smart Environments, DOI 10.1007/978-0-387-93808-0_5, © Springer Science+Business Media, LLC 2010
117
118
Manish Kushwaha et al.
Tracking is one such application that can benefit from multiple sensing modalities [9]. If the moving target emits sound signal then both audio and video sensors can be utilized. These modalities can complement each other in the presence of high background noise that impairs the audio, or in the presence of visual clutter that handicap the video. Additionally, tracking based on the fusion of audio-video data can improve the performance of audio-only or video-only approaches. Audio-video tracking can also provide cues for the other modality for actuation. For example, visual steering information from a video sensor may be used to steer the audio sensor (microphone array) toward a moving target. Similarly, information from the audio sensors can be used to steer a pan-tilt-zoom camera toward a speaker. Although, audio-visual tracking has been applied to smart videoconferencing applications [42], it does not use a wide-area distributed platform . The main challenges in target tracking is to find tracks from noisy observation. This requires solutions to both data association and state estimation problems. Our system employs a Markov Chain Monte Carlo Data Association (MCMCDA) algorithm for tracking. The MCMCDA algorithm is a data-oriented, combinatorial optimization approach to the data association problem that avoids the enumeration of tracks. The MCMCDA algorithm enables us to track an unknown number of targets in noisy urban environment citeoh:2004. In this chapter, we describe our ongoing research in multimodal target tracking in urban environments utilizing an HSN of mote class devices equipped with microphone arrays as audio sensors and embedded PCs equipped with web cameras as video sensors. The targets to be tracked are moving vehicles emitting engine noise. Our system has many components including audio processing, video processing, WSN middleware services, multimodal sensor fusion, and target tracking based on sequential Bayesian estimation and MCMCDA. While none of these is necessarily novel, their composition and implementation on an actual HSN requires addressing a number of significant challenges. Specifically, we have implemented audio beamforming on audio sensors utilizing an FPGA-based sensor board and evaluated its performance as well its energy consumption. While we are using a standard motion detection algorithm on video sensors, we have implemented post-processing filters that represent the video data in a similar format as the audio data, which enables seamless audio-video data fusion. Furthermore, we have extended time synchronization techniques for HSNs consisting of mote and PC networks. Finally, the main challenge we address is system integration including making the system work on an actual platform in a realistic deployment. The paper provides results gathered in an uncontrolled urban environment and presents a thorough evaluation including a comparison of different fusion and tracking approaches. The rest of the paper is organized as follows. Section 2 presents related work. In Section 3, we describe the overall system architecture. It is followed by the description of the audio processing in Section 4 and then the video processing in Section 5. In Section 6, we present our time synchronization approach for HSNs. Multimodal target tracking is described in Section 7. Section 8 presents the experimental deployment and its evaluation. Finally, we conclude in Section 9.
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
119
2 Challenges and Related Work In this section, we present the challenges for multimodal sensor fusion and tracking. We also present various existing approach for sensor fusion and tracking. In multimodal multisensor fusion, the data may be fused at a variety of levels including the raw data level, where the raw signals are fused, a feature-level, where representative characteristics of the data are extracted and fused, and a decision-level fusion wherein target estimates from each sensor are fused [17]. At successive levels more information may be lost, but in collaborative applications, such as WSN applications, the communication requirement of transmitting large amounts of data is reduced. Another significant challenge is sensor conflict, when different sensors report conflicting data. When sensor conflict is very high, fusion algorithms produce false or meaningless results [18]. Reasons for sensor conflict may be sensor locality, different sensor modalities, or sensor faults. If a sensor node is far from a target of interest then the data from that sensor will not be useful, and will have higher variance. Different sensor modalities observe different physical phenomena. For example, audio and video sensors observe sound sources and moving objects, respectively. If a sound source is stationary or a moving target is silent, the two modalities might produce conflicting data. Also, different modalities are affected by different types of background noise. Finally, poor calibration and sudden change in local conditions can also cause conflicting sensor data. Another classical problem in multitarget tracking is to find a track of each target from the noisy data. If the association of sequence of data-points with each target is known, multitarget tracking reduces to a set of state estimation problems. The data association problem is to find which data-points are generated by which targets, or in other words, associate each data-point with either a target or noise. Rest of the section presents existing work in acoustic localization, video tracking, time synchronization and multimodal fusion.
Acoustic Localization An overview of the theoretical aspects of Time Difference of Arrival (TDOA) based acoustic source localization and beamforming is presented in [7]. Beamforming methods have successfully been applied to detect single or even multiple sources in noisy and reverberant environments. A maximum likelihood (ML) method for single target and an approximate maximum likelihood (AML) method for direction of arrival (DOA) based localization in reverberant environments are proposed in [7, 4]. In [1], an empirical study of collaborative acoustic source localization based on an implementation of the AML algorithm is shown. An alternative technique called accumulated correlation combines the speed of time-delay estimation with the accuracy of beamforming [5]. Time of Arrival (TOA) and TDOA based methods are proposed for collaborative source localization for wireless sensor network
120
Manish Kushwaha et al.
applications [24]. A particle filtering based approach for acoustic source localization and tracking using beamforming is described in [39].
Video Tracking A simple approach to motion detection from video data is via frame differencing, which requires a robust background model. There exist a number of challenges for the estimation of robust background models [38], including gradual and sudden illumination changes, vacillating backgrounds, shadows, visual clutter, occlusion, etc. In practice, most of the simple motion detection algorithms have poor performance when faced with these challenges. Many adaptive background-modeling methods have been proposed to deal with these challenges. The work in [13] models each pixel in a camera frame by an adaptive parametric mixture model of three Gaussian distributions. A Kalman filter to track the changes in background illumination for every pixel is used in [21]. An adaptive nonparametric Gaussian mixture model to address background modeling challenges is done in [36]. A kernel estimator for each pixel is proposed in [11] with kernel exemplars from a moving window. Other techniques using high-level processing to assist the background modeling have been proposed; for instance, the Wallflower tracker [38] which circumvents some of these problems using high-level processing rather than tackling the inadequacies of the background model. The algorithm in [19] proposes an adaptive background modeling method based on the framework in [36]. The main differences lie in the update equations, initialization method and the introduction of a shadow detection algorithm.
Time Synchronization time synchronization in sensor networks has been studied extensively in the literature and several protocols have been proposed. Reference Broadcast Synchronization (RBS) [12] is a protocol for synchronizing a set of receivers to the arrival of a reference beacon. The Timing-sync Protocol for Sensor Networks (TPSN) is a sender-receiver protocol that uses the round-trip latency of a message to estimate the time a message takes along the communication pathway from sender to receiver. This time is then added to the sender timestamp and transmitted to the receiver to determine the clock offset between the two nodes. Elapsed Time on Arrival (ETA) [23] provides a set of application programming interfaces for an abstract time synchronization service. RITS [35] is an extension of ETA over multiple hops. It incorporates a set of routing services to efficiently pass sensor data to a network sink node for data fusion. In [15], mote-PC synchronization was achieved by connecting the GPIO ports of a mote and IPAQ PDA using the mote-NIC model. Although using this technique can achieve microsecond-accurate synchronization, it was implemented as a hardware modification rather than in software.
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
121
Multimodal Tracking A thorough introduction to multisensor data fusion, with focus on data fusion applications, process models, and architectures is provided in [17]. The paper surveys a number of related techniques, and reviews standardized terminology, mathematical techniques, and design principles for multisensor data fusion systems. A modular tracking architecture that combines several simple tracking algorithms is presented in [31]. Several simple and rapid algorithms run on different CPUs, and the tracking results from different modules are integrated using a Kalman filter. In [37], two different fusion architectures based on decentralized recursive (extended) Kalman filters are described for fusing outputs from independent audio and video trackers. Multimodal sensor fusion and tracking have been applied to smart videoconferencing and indoor tracking applications [44, 42, 3, 6]. In [44], data is obtained using multiple cameras and microphone arrays. The video data consist of pairs of image coordinates of features on each tracked object, and the audio data consist of TDOA for each microphone pair in the array. A particle filter is implemented for tracking. In [3], a graphical model based approach is taken using audiovisual data. The graphical model is designed for single target tracking using a single camera and a microphone pair. The audio data consist of the signals received at the microphones, and the video data consist of monochromatic video frames. Generative models are described for the audio-video data that explain the data in terms of the target state. An expectation-maximization (EM) algorithm is described for parameter learning and tracking, which also enables automatic calibration by learning the audio-video calibration parameters during each EM step. In [6], a probabilistic framework is described for multitarget tracking using audio and video data. The video data consist of plan-view images of the foreground. Plan-view images are projections of the images captured from different points-of-view to a 2D plane, usually the ground plane where the targets are moving [8]. Acoustic beamforms for a fixed length signal are taken as the audio data. A particle filter is implemented for tracking.
3 Architecture Figure 1 shows the system architecture. The audio sensors, consisting of MICAz motes with acoustic sensor boards equipped with a microphone array, form an IEEE 802.15.4 network. This network does not need to be connected; it can consist of multiple connected components as long as each of these component have a dedicated mote-PC gateway. The video sensors are based on Logitech QuickCam Pro 4000 cameras attached to OpenBrick-E Linux embedded PCs. These video sensors, along with the mote-PC gateways, the sensor fusion node and the reference broadcaster for time synchronization (see Section 6) are all PCs forming a peer-to-peer 802.11b wireless network. Figure 2 shows the conceptual layout of the sensor network. The audio sensors perform beamforming, and transmit the audio detections to the corresponding mote-PC gateway utilizing a multi-hop message routing service [2].
122
VS#1
τ, λ
daemon
VS#2
τ, λ
daemon
Time synchronization daemon
τ, λ
Network
AS#2
Base Station
τ, λ
daemon
AS#1
Manish Kushwaha et al. Scheduler (t) AS#1 τ1, λ1
Sensor Fusion Node λ1(t)
τn, λn
AS#2 τ1, λ1
λ2(t)
τn, λn
MSF and Tracking
X(t)
VS#1 τ1, λ1
λk-1(t)
τn, λn
VS#2 τ1, λ1
λk(t)
τn, λn
Fig. 1 Architecture for the multimodal target tracking system. Here, τ denotes measurement timestamps, λ denotes sensor measurements (also called detection functions that are described in later sections), and t denotes time. The blocks shown inside the sensor fusion node are circular buffers that store timestamped measurements.
Fig. 2 Conceptual layout of sensor network for multimodal target tracking. Circles represent the audio sensors, camera silhouettes represent the video sensors and rectangles represent PCs.
The routing service also performs time translation of the detection timestamps. The video sensors run a motion detection algorithm and compute timestamped video detection functions. Both audio and video detections are routed to the central sensor fusion node. Time translation is also carried out in the 802.11b network utilizing an RBS-based time synchronization service. This approach ensures that sensor data arrives at the sensor fusion node with timestamps in a common timescale. On the sensor fusion node, the sensor measurements are stored in appropriate sensor buffers, one for each sensor. A sensor fusion scheduler triggers periodically and generates a fusion timestamp. The trigger is used to retrieve the sensor measurements from the sensor buffers with timestamps closest to the generated fusion timestamp. The
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
123
retrieved sensor measurements are then used for multimodal fusion and target tracking. In this study, we developed and compared target tracking algorithms based on both Sequential Bayesian Estimation (SBE) and MCMCDA. Note that the triggering mechanism of the scheduler decouples the target tracking rate from the audio and video sensing rates, which allows us to control the rate of the tracking application independently of the sensing rate.
4 Audio Beamforming Beamforming is a signal processing algorithm for DOA estimation of a signal source. In a typical beamforming array, each of the spatially separated microphones receive a time-delayed source signal. The amount of time delay at each microphone in the array depends on the microphone arrangement and the location of the source. A typical delay-and-sum single source beamformer discretizes the sensing region into directions, or beams, and computes a beam energy for each of them. The beam energies are collectively called the beamform. The beam with maximum energy indicates the direction of the acoustic source.
Beamforming Algorithm The data-flow diagram of our beamformer is shown in Fig. 3. The amplified microphone signal is sampled at a high frequency (100 KHz) to provide high resolution for the time delay, which is required for the closely placed microphones. The raw signals are filtered to remove unwanted noise components. The signal is then fed to a tapped delay line (TDL), which has M different outputs to provide the required delays for each of the M beams. The delays are set by taking into consideration the exact relative positions of the microphones so that the resulting beams are steered to the beam angles, θi = i 360 M degrees, for i = 0, 1, ...M − 1. The signal is downsam-
Fig. 3 Data-flow diagram of the real-time beamforming sensor.
124
Manish Kushwaha et al.
pled and the M beams are formed by adding the four delayed signals together. Data blocks are formed from the data streams (with a typical block length of 5-20ms) and an FFT is computed for each block. The block power values, μ (θi ), are smoothed by exponential averaging into the beam energy, λ (θi )
λ t (θi ) = αλ t−1 (θi ) + (1 − α )μ (θi)
(1)
where α is an averaging factor.
Audio Hardware In our application, the audio sensor is a MICAz mote with an onboard Xilinx XC3S1000 FPGA chip that is used to implement the beamformer [40]. The onboard Flash (4MB) and PSRAM (8MB) modules allow storing raw samples of several acoustic events. The board supports four independent analog channels, featuring an electret microphone each, sampled at up to 1 MS/s (million samples per second). A small beamforming array of four microphones arranged in a 10cm × 6cm rectangle is placed on the sensor node, as shown in Fig. 4(a). Since the distances between the microphones are small compared to the possible distances of sources, the sensors perform far-field beamforming. The sources are assumed to be on the same two-dimensional plane as the microphone array, thus it is sufficient to perform planar beamforming by dissecting the angular space into M equal angles, providing a resolution of 360/M degrees. In the experiments, the sensor boards are configured to perform simple delay-and-sum-type beamforming in real time with M = 36, and an angular resolution of 10 degrees. Finer resolution increases the communication requirements.
Fig. 4 (a) Sensor Node Showing the Microphones, (b) Beamform of acoustic source at a distance of 50 feet and an angle of 120 degrees.
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
125
Evaluation We test the beamformer node outdoors using recorded speech as the acoustic source. Measurements are taken by placing the source at distances of 3, 5, 10, 25, 50, 75, 100, and 150 feet from the sensor, and at an angle from -180◦ to +180◦ in 5◦ increments. Figure 4(b) shows the beamforming result for a single audio sensor when the source was at a distance of 50 feet. Mean DOA measurement error for 1800 human speech experiments is shown in Figure 5. The smallest error was recorded when the source was at a distance of 25 feet from the sensor. The error increases as the source moves closer to the sensor. This is because the beamforming algorithm assumes a planar wavefront, which holds for far-field sources but breaks down when the source is close to the sensor. However, as the distance between the source and sensor grows, error begins accumulating again as the signal-to-noise ratio decreases.
Fig. 5 Human speech DOA error for distances of 3, 5, 10, 25, 50, 75 feet between acoustic source and beamformer node.
Messages containing the audio detection functions require 83 bytes, and include node ID, sequence number, timestamps, and 72 bytes for 36 beam energies. These data are transmitted through the network in a single message. The default TinyOS message size of 36 bytes was changed to 96 bytes to accommodate the entire audio message. The current implementation uses less than half of the total resources (logic cells, RAM blocks) of the selected mid-range FPGA device. The application runs at 20 MHz, which is relatively slow in this domain—the inherent parallel processing topology allows this slow speed. Nonetheless, the FPGA approach has a significant impact on the power budget, the sensor draws 130mA current (at 3.3 V) which is nearly a magnitude higher then typical wireless sensor node power currents. Flashbased FPGAs and smart duty cycling techniques are promising new directions in our research project for reducing the power requirements.
126
Manish Kushwaha et al.
5 Video Tracking Video tracking systems seek to automatically detect moving targets and track their movement in a complex environment. Due to the inherent richness of the visual medium, video based tracking typically requires a pre-processing step that focuses attention of the system on regions of interest in order to reduce the complexity of data processing. This step is similar to the visual attention mechanism in human observers. Since the region of interest is primarily characterized by regions containing moving targets (in the context of target tracking), robust motion detection is the first step in video tracking. A simple approach to motion detection from video data is via frame differencing. It compares each incoming frame with a background model and classifies the pixels of significant variation into the foreground. The foreground pixels are then processed for identification and tracking. The success of frame differencing depends on the robust extraction and maintenance of the background model. Performance of such techniques tends to degrade when there is significant camera motion, or when the scene has significant amount of change.
Algorithm The dataflow in Figure 6 shows the motion detection algorithm and its components used in our tracking application. The first component is background-foreground seg-
Fig. 6 Data-flow diagram of real-time motion detection algorithm
mentation of the currently captured frame (It ) from the camera. We use the algorithm described in [19]. This algorithm uses an adaptive background mixture model for real-time background and foreground estimation. The mixture method models each background pixel as a mixture of K Gaussian distributions. The algorithm provides multiple tunable parameters for desired performance. In order to reduce speckle noise and smooth the estimated foreground (Ft ), the foreground is passed through a median filter. In our experiments, we use a median filter of size 3 × 3. Since our sensor fusion algorithm (Section 7) utilizes only the angle of moving targets, it is desirable and sufficient to represent the foreground in a simpler detection function. Similar to the beam angle concept in audio beamforming (Section 4), the field-of-view of the camera is divided into M equally-spaced angles
θi = θmin + (i − 1)
θmax − θmin : i = 1, 2, ..., M M
(2)
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
127
where θmin and θmax are the minimum and maximum field-of-view angles for the camera. The detection function value for each angle is simply the number of foreground pixels in that direction. Formally, the detection function for the video sensors can be defined as
λ (θi ) =
W
H
∑ ∑ F( j, k) : i = 1, 2, ..., M
(3)
j∈θi k=1
where F is the binary foreground image, H and W are the vertical and horizontal resolutions in pixels, respectively and j ∈ θi indicates columns in the frame that fall within angle θi . Figure 7 shows a snapshot of motion detection.
Fig. 7 Video detection. The frame on the left shows the input image, the frame in the middle shows the foreground and the frame on the right shows the video detection function.
Video Post-processing In our experiments, we gathered video data of vehicles from multiple sensors from an urban street setting. The data contained a number of real-life artifacts such as vacillating backgrounds, shadows, sunlight reflections and glint. The algorithm described above is not able to filter out such artifacts from the detections. We implemented two post-processing filters to improve the detection performance. The first filter removes any undesirable persistent background. For this purpose we keep a background of moving averages which was removed from each detection. The background update and filter equations are bt (θ ) = α bt−1 (θ ) + (1 − α )λ t (θ )
(4)
λ (θ ) = λ (θ ) − b
(5)
t
t
t−1
(θ )
where bt (θ ) is the moving average background, and λ t (θ ) is the detection function at time t. The second filter removes any sharp spikes (typically caused by sunlight reflections and glint). For this we convolved the detection function with a small linear kernel to add a blurring effect. This essentially reduces the effect of any sharp spikes in detection function due to glints. The equation for this filter is
λ t (θ ) = λ t (θ ) ∗ k
(6)
128
Manish Kushwaha et al.
where k is a 7 × 1 vector of equal weights, and ∗ denotes convolution. We implemented the motion detection algorithm using OpenCV (open source computer vision) library. We use Linux PCs equipped with the QuickCam Pro 4000 as video sensors. The OpenBrick-E has 533 MHz CPU, 128 MB RAM, and a 802.11b wireless adapter. The QuickCam Pro supports up to 640 × 480 pixel resolution and up to 30 frames-per-second frame rate. Our motion detection algorithm implementation runs at 4 frames-per-second and 320 × 240 pixel resolution. The number of angles in Equation (2) is M = 160.
6 Time Synchronization In order to seamlessly fuse time-dependent audio and video sensor data for tracking moving objects, participating nodes must have a common notion of time. Although several microsecond-accurate synchronization protocols have emerged for wireless sensor networks (e.g. [12, 14, 27, 35]), achieving accurate synchronization in a heterogeneous sensor network is not a trivial task.
6.1 Synchronization Methodology Attempting to synchronize the entire network using a single protocol will introduce a large amount of error. For example, TPSN [14], FTSP [27], and RITS [35] were all designed to run on the Berkeley Motes, and assume the operating system is tightly integrated with the radio stack. Attempting to use such protocols on an 802.11 PC network will result in poor synchronization accuracy because the necessary low-level hardware control is difficult to achieve. Reference Broadcast Synchronization (RBS) [12], although flexible when it comes to operating system and computing platform, is accurate only when all nodes have access to a common network medium. A combined network of motes and PCs will be difficult to synchronize using RBS because each platform uses different communication protocols and wireless frequencies. Instead, we adopted a hybrid approach [2], which pairs a specific network with the synchronization protocol that provides the most accuracy with the least amount of overhead. To synchronize the entire network, it is necessary for gateway nodes (i.e., nodes connecting multiple networks,) to handle several synchronization mechanisms.
Mote Network We used Elapsed Time on Arrival (ETA) [23] to synchronize the mote network. ETA timestamps synchronization messages at transmit and receive time, thus removing the largest amount of nondeterministic message delay from the communi-
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
129
cation pathway. On the sender side, a timestamp is taken upon transmission of a designated synchronization byte, and placed at the end of the outgoing message. On the receiver side, a timestamp is taken upon arrival of the synchronization byte. The difference between these two timestamps is generally deterministic, and is based primarily on the bitrate of the transceivers and the size of the message. We selected ETA for the mote network due to its accuracy and low message overhead.
PC Network We used RBS to synchronize the PC network. RBS synchronizes a set of nodes to the arrival of a reference beacon. Participating nodes timestamp the arrival of a message broadcast over the network, and by exchanging these timestamps, neighboring nodes are able to maintain reference tables for timescale transformation. RBS minimizes synchronization error by taking advantage of the broadcast channel of wireless networks.
Mote-PC Network To synchronize a mote with a PC in software, we adopted the underlying methodology of ETA and applied it to serial communication. On the mote, a timestamp is taken upon transfer of a synchronization byte and inserted into the outgoing message. On the PC, a timestamp is taken immediately after the UART issues the interrupt, and the PC regards the difference between these two timestamps as the PCmote offset. Serial communication bit rate between the mote and PC is 57600 baud, which approximately amounts to a transfer time of 139 microseconds per byte. However, the UART will not issue an interrupt to the CPU until its 16-byte buffer nears capacity or a timeout occurs. Because the synchronization message is six bytes, reception time in this case will consist of the transfer time of the entire message in addition to the timeout time and the time it takes to transfer the date from the UART buffer into main memory by the CPU. This time is compensated for by the receiver, and the clock offset between the two devices is determined as the difference between the PC receive time and the mote transmit time.
6.2 Evaluation of HSN Time Synchronization Our HSN contains three communication pathways: mote-mote, mote-PC, and PCPC. We first examine the synchronization accuracy of each of these paths, then present the synchronization results for the entire network.
130
Manish Kushwaha et al.
Mote Network We evaluated synchronization accuracy in the mote network with the pairwise difference method. Two nodes simultaneously timestamp the arrival of an event beacon, then forward the timestamps to a sink node two hops away. At each hop, the timestamps are converted to the local timescale. The synchronization error is the difference between the timestamps at the sink node. For 100 synchronizations, the average error is 5.04 μ s, with a maximum of 9μ s.
PC Network We used a separate RBS transmitter to broadcast a reference beacon every ten seconds over 100 iterations. Synchronization error, determined using the pairwise difference method, was as low as 17.51 μ s on average, and 2050.16 μ s maximum. The worst-case error is significantly higher than reported in [12] because the OpenBrickE wireless network interface controllers in our experimental setup are connected via USB, which has a default polling frequency of 1 kHz. For our tracking application, this is acceptable because we use a sampling rate of 4 Hz. Note that the reason for selecting such a low sampling rate is due to bandwidth constraints and interference and not because of synchronization error.
Mote-PC Connection GPIO pins on the mote and PC were connected to an oscilloscope, and set high upon timestamping. The resulting output signals were captured and measured. The test was performed over 100 synchronizations, and the resulting error was 7.32μ s on average, and did not exceed 10μ s. The majority of the error is due to jitter, both in the UART and the CPU. A technique to compensate for such jitter on the motes is presented in [27], however, we did not attempt it on the PCs.
HSN We evaluated synchronization accuracy across the entire network using the pairwise difference method. Two motes timestamped the arrival of an event beacon, and forwarded the timestamp to the network sink, via one mote and two PCs. RBS beacons were broadcast at four-second intervals, and therefore clock skew compensation was unnecessary, because synchronization error due to clock skew would be insignificant compared with offset error. The average error over the 3-hop network was 101.52μ s, with a maximum of 1709μ s. The majority of this error is due to the polling delay from the USB wireless network controller. However, synchronization accuracy is still sufficient for audio and video sensor fusion at 4Hz.
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
131
6.3 Synchronization Service The implementation used in these experiments was bundled into a time synchronization and routing service for sensor fusion applications. Figure 8 illustrates the interaction of each component within the service. The service interacts with sensing applications that run on the local PC, as well as other service instances running on remote PCs. The service accepts timestamped event messages on a specific port, converts the embedded timestamp to the local timescale, and forwards the message toward the sensor-fusion node. The service uses reference broadcasts to maintain synchronization with the rest of the PC network. In addition, the service accepts mote-based event messages, and converts the embedded timestamps using the ETAbased serial timestamp synchronization method outlined above. Kernel modifications in the serial and wireless drivers were required in order to take accurate timestamps. Upon receipt of a designated synchronization byte, the time is recorded and passed up to the synchronization service in the user space. The modifications are unobtrusive to other applications using the drivers, and the modules are easy to load into the kernel. The mote implementation uses the TimeStamping interface, provided with the TinyOS distribution [25]. A modification was made to the UART interface to insert a transmission timestamp into the event message as it is being transmitted between the mote and PC. The timestamp is taken immediately before a synchronization byte is transmitted, then inserted at the end of the message.
Fig. 8 PC-based time synchronization service.
132
Manish Kushwaha et al.
7 Multimodal Target Tracking This section describes the tracking algorithm and the approach for fusing the audio and video data based on sequential Bayesian estimation. We use following notation: Superscript t denotes discrete time (t ∈ Z+ ), subscript k ∈ {1, ..., K} denotes the sensor index, where K is the total number of sensors in the network, the target state at time t is denoted as x(t) , and the sensor data at time t is denoted as z(t) .
7.1 Sequential Bayesian Estimation We use sequential Bayesian estimation to estimate the target state x(t) at time t, similar to the approach presented in [26]. In sequential Bayesian estimation, the target state is estimated by computing the posterior probability density p(x(t+1) |z(t+1) ) using a Bayesian filter described by p(x(t+1) |z(t+1) ) ∝ p(z(t+1) |x(t+1) )
p(x(t+1) |x(t) )p(x(t) |z(t) )dx(t)
(7)
where p(x(t) |z(t) ) is the prior density from the previous step, p(z(t+1) |x(t+1) ) is the likelihood given the target state, and p(x(t+1) |x(t) ) is the prediction for the target state x(t+1) given the current state x(t) according to a target motion model. Since we are tracking moving vehicles it is reasonable to use a directional motion model based on the target velocity. The directional motion model is described by x(t+1) = x(t) + v + U[−δ , +δ ]
(8)
where x(t) is the target state time t, x(t+1) is the predicted state, v is the target velocity, and U[−δ , +δ ] is a uniform random variable. Because the sensor models (described later in Subsection 7.2) are nonlinear, the probability densities cannot be represented in a closed form. It is, therefore, reasonable to use a nonparametric representation for the probability densities. The nonparametric densities are represented as discrete grids in 2D space, similar to [26]. For nonparametric representation, the integration term in Equation (7) becomes a convolution operation between the motion kernel and the prior distribution. The resolution of the grid representation is a trade-off between tracking resolution and computational capacity.
Centralized Bayesian Estimation Since we use resource constrained mote class sensors, centralized Bayesian estimation is a reasonable approach because of its modest computational requirements. The likelihood function in Equation (7) can be calculated either as a product or
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
133
weighted summation of the individual likelihood functions. We describe the two methods next.
Product of Likelihood functions Let pk (z(t) |x(t) ) denote the likelihood function from sensor k. If the sensor observations are mutually independent conditioned on the target state, the likelihood functions from multiple sensors are combined as p(z(t) |x(t) ) =
∏
pk (z(t) |x(t) )
(9)
k=1,...,K
Weighted-Sum of Likelihood functions An alternative approach to combine the likelihood functions is to compute their weighted-sum. This approach allows us to give different weights to different sensor data. These weights can be used to incorporate sensor reliability and quality of (t) sensor data. We define a quality index qk for each sensor k as (t)
(t)
qk = rk max (λk (θ )) θ
(t)
where rk is a measure of sensor reliability and λk (θ ) is the sensor detection function. The combined likelihood function is given by p(z(t) |x(t) ) =
(t)
∑k=1,...,K qk pk (z(t) |x(t) ) (t)
∑k=1,...,K qk
(10)
We experimented with both methods in our evaluation. The product method produces more accurate results with low uncertainty in target state. The weighted-sum method performs better in cases with high sensor conflict, though it suffers from high uncertainty. The results are presented in Section 8.
Hybrid Bayesian Estimation In sensor fusion a big challenge is to account for conflicting sensor data. When sensor conflict is very high, sensor fusion algorithms produce false or meaningless fusion results [18]. Reasons for sensor conflict are sensor locality, different sensor modalities, and sensor faults. Selecting and clustering the sensors in different groups based on locality or modality can mitigate poor performance due to sensor conflict. For example, clustering the sensors close to the target and fusing the data from only the sensors in the cluster would remove the conflict caused by distant sensors.
134
Manish Kushwaha et al.
The sensor network deployment in this paper is small and the sensing region is comparable to the sensing ranges of the audio and video sensors. For this reason, we do not use locality based clustering. However, we have multimodal sensors that can report conflicting data. Hence, we developed a hybrid Bayesian estimation framework [17] by clustering sensors based on modalities, and compare it with the centralized approach. Figure 9 illustrates the framework. The likelihood function
Fig. 9 Hybrid Bayesian estimation framework.
from each sensor in a cluster is fused together using Equation (9) or (10). The combined likelihood is then used in Equation (7) to calculate the posterior density for that cluster. The posterior densities from all the clusters are then combined together to estimate the target state. For hybrid Bayesian estimation with audio-video clustering, we compute the likelihood functions using Equation (9) or (10). The audio posterior density is calculated using paudio (x(t+1) |z(t+1) ) ∝ paudio (z(t+1) |x(t+1) )
p(x(t+1) |x(t) )p(x(t) |z(t) )dx(t)
while the video posterior density is calculated as pvideo (x(t+1) |z(t+1) ) ∝ pvideo (z(t+1) |x(t+1) )
p(x(t+1) |x(t) )p(x(t) |z(t) )dx(t)
The two posterior densities are combined either as (product fusion) p(x(t+1) |z(t+1) ) ∝ paudio (z(t+1) |x(t+1) )pvideo (z(t+1) |x(t+1) ) or (weighted-sum fusion) p(x(t+1) |z(t+1) ) ∝ α paudio (z(t+1) |x(t+1) ) + (1 − α )pvideo(z(t+1) |x(t+1) ) where α is a weighing factor.
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
135
7.2 Sensor Models We use a nonparametric model for the audio sensors, while a parametric mixture-ofGaussian model for the video sensors is used to mitigate the effect of sensor conflict in object detection.
Audio Sensor Model The nonparametric DOA sensor model for a single audio sensor is the piecewise linear interpolation of the audio detection function
λ (θ ) = wλ (θi−1 ) + (1 − w)λ (θi), if θ ∈ [θi−1 , θi ] where w = (θi − θ )/(θi − θi−1 ).
Video Sensor Model The video detection algorithm captures the angle of one or more moving objects. The detection function from Equation (3) can be parametrized as a mixture-ofGaussian n
λ (θ ) = ∑ ai fi (θ ) i=1
where n is the number of components, fi (θ ) is the probability density, and ai is the mixing proportion for component i. Each component is a Gaussian density given by fi (θ ) = N (θ |μi , σi2 ), where the component parameters μi , σi2 and ai are calculated from the detection function.
Likelihood Function The 2D search space is divided into N rectangular cells with center points at (xi , yi ), for i = 1, 2, ..., N as illustrated in Figure 10. The likelihood function value for kth sensor at ith cell is the average value of the detection function in that cell, given by pk (z|x) = pk (xi , yi ) =
1
∑
λk (θ ) (k,i) (k,i) (ϕB − ϕA ) ϕ (k,i) ≤θ ≤ϕ (k,i) B A
136
Manish Kushwaha et al.
Fig. 10 Computation of the likelihood value for kth sensor at ith cell, (xi , yi ). The cell is centered at P0 with vertices at P1 , P2 , P3 , and P4 . The angular interval subtended at the sensor due to the ith (k,i) (k,i) cell is θ ∈ [ϕA , ϕB ], or θ ∈ [0, 2π ] if the sensor is inside the cell.
7.3 Multiple-Target Tracking The essence of the multi-target tracking problem is to find a track of each object from noisy measurements. If the sequence of measurements associated with each object is known, multi-target tracking reduces to a set of state estimation problems for which many efficient algorithms are available. Unfortunately, the association between measurements and objects is unknown. The data association problem is to work out which measurements were generated by which objects; more precisely, we require a partition of measurements such that each element of a partition is a collection of measurements generated by a single object or noise. Due to this data association problem, the complexity of the posterior distribution of the states of objects grows exponentially as time progresses. It is well-known that the data association problem is NP-hard [32], so we do not expect to find efficient, exact algorithms for solving this problem. In order to handle highly nonlinear and non-Gaussian dynamics and observations, a number of methods based on particle filters has recently been developed to track multiple objects in video [30, 20]. Although particle filters are highly effective in single-target tracking, it is reported that they provide poor performance in multi-target tracking [20]. This is because a fixed number of particles is insufficient to represent the posterior distribution with the exponentially increasing complexity (due to the data association problem). As shown in [20, 43], an efficient alternative is to use Markov chain Monte Carlo (MCMC) to handle the data association problem in multi-target tracking. For our problem, there is an additional complexity. We do not assume the number of objects is known. A single-scan approach, which updates the posterior based only on the current scan of measurements, can be used to track an unknown number
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
137
of targets with the help of trans-dimensional MCMC [43, 20] or a detection algorithm [30]. But a single-scan approach cannot maintain tracks over long periods because it cannot revisit previous, possibly incorrect, association decisions in the light of new evidence. This issue can be addressed by using a multi-scan approach, which updates the posterior based on both current and past scans of measurements. The well-known multiple hypothesis tracking (MHT) [33] is a multi-scan tracker, however, it is not widely used due to its high computational complexity. A newly developed algorithm, called Markov chain Monte Carlo data association (MCMCDA), provides a computationally desirable alternative to MHT [29]. The simulation study in [29] showed that MCMCDA was computationally efficient compared to MHT with heuristics (i.e., pruning, gating, clustering, N-scan-back logic and k-best hypotheses). In this chapter, we use the online version of MCMCDA to track multiple objects in a 2-D plane. Due to the page limitation, we omit the description of the algorithm in this paper and refer interested readers to [29].
8 Evaluation In this section, we evaluate target tracking algorithms based on the sequential Bayesian estimation and MCMCDA.
8.1 Sequential Bayesian Estimation
Fig. 11 Experimental setup
The deployment of our multimodal target tracking system is shown in Figure 11. We employ 6 audio sensors and 3 video sensors deployed on either side of a road. The complex urban street environment presents many challenges including gradual change of illumination, sunlight reflections from windows, glints due to cars, high visual clutter due to swaying trees, high background acoustic noise due
138
Manish Kushwaha et al.
to construction and acoustic multipath effects due to tall buildings. The objective of the system is to detect and track vehicles using both audio and video under these conditions. Sensor localization and calibration for both audio and video sensors are required. In our experimental setup, the sensor nodes are manually placed at marked locations and orientations. The audio sensors are placed on 1 meter high tripods to minimize audio clutter near the ground. An accurate self-calibration technique, e.g. [16, 28], is desirable for a target tracking system. Our experimental setup consists of two wireless networks as described in Section 3. The mote network is operating on channel 26 (2.480 GHz) while the 802.11b network is operating on channel 6 (2.437 GHz). Both the channels are non-overlapping and different from the infrastructure wireless network, which operates on channel 11 (2.462 GHz). We choose these nonoverlapping channels to minimize interference and are able to achieve less than 2% packet loss. We gather audio and video detection data for a total duration of 43 minutes. Table 1 presents the parameter values that we use in our tracking system. We run our Number of beams in audio beamforming, Maudio 36 Number of angles in video detection Mvideo 160 Sensing region (meters) 35 × 20 Cell size (meters) 0.5 × 0.5 Table 1 Parameters used in experimental setup
sensor fusion and tracking system online using centralized sequential Bayesian estimation based on the product of likelihood functions. We also collect all the audio and video detection data for offline evaluation. This way we are able to experiment with different fusion approaches on the same data set. For offline evaluations, we shortlist 10 vehicle tracks where there is only a single target in the sensing region. The average duration of tracks is 4.25 sec with 3.0 sec minimum and 5.5 sec maximum. The tracked vehicles are part of an uncontrolled experiment. The vehicles are traveling on road at a speed of 20-30 mph speed. Sequential Bayesian estimation requires a prior density of the target state. We initialize the prior density using a simple detection algorithm based on audio data. If the maximum of the audio detection functions exceeds a threshold, we initialize the prior density based on the audio detection. In our simulations, we experiment with eight different approaches. We use audioonly, video-only and audio-video sensor data for sensor fusion. For each of these data sets, the likelihood is computed either as the weighted-sum or product of the likelihood function for the individual sensors. For the audio-video data, we use centralized and hybrid fusion. Following is the list of different target tracking approaches. 1. audio-only, weighted-sum (AS) 2. video-only, weighted-sum (VS)
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
3. 4. 5. 6. 7. 8.
139
audio-video, centralized, weighted-sum (AVCS) audio-video, hybrid, weighted-sum (AVHS) audio-only, likelihood product (AP) video-only, likelihood product (VP) audio-video, centralized, likelihood product (AVCP) audio-video, hybrid, likelihood product (AVHP)
The ground truth is estimated post-facto based on the video recording by a separate camera. The standalone ground truth camera is not part of any network, and have the sole responsibility of recording ground truth video. For evaluation of tracking accuracy, the center of mass of the vehicle is considered to be the true location. Figure 12 shows the tracking error for a representative vehicle track. The tracking
Fig. 12 Tracking error (a) weighted sum, (b) product
error when audio data is used is consistently lower than the case when the video data is used. When we use both audio and video data, the tracking error is lower than either of those considered alone. Figure 13 shows the determinant of the covariance of the target state for the same vehicle track. The covariance, which is an indicator of
Fig. 13 Tracking variance (a) weighted sum, (b) product
uncertainty in the target state is significantly lower for product fusion than weightedsum fusion. In general, covariance for audio-only tracking is higher than video-only tracking, while using both modalities lowers the uncertainty. Figure 14(a) shows the tracking error in the case of fusion based on weightedsum. The tracking error when using only video data shows a large value at time t = 1067 second. In this case, the video data has false peaks not corresponding to the target. The audio fusion works fine for this track. As expected, when we use both
140
Manish Kushwaha et al.
Fig. 14 Tracking error (a) poor video tracking, (b) poor audio tracking
audio and video data together, the tracking error is decreased. Further, Figure 14(a) shows the tracking error when using 6, 3 and 2 audio sensors for the centralized audio-video fusion. Using as few as two audio sensors can assist video tracking to disambiguate and remove false peaks. Similarly, when there is an ambiguity in the audio data, the video sensors can assist and improve tracking performance. Figure 14(b) shows the tracking error for a track where audio data is poor due to multiple sound sources. When fused with video data the tracking performance is improved. The figure shows tracking performance when using two and three video sensors for the centralized and hybrid fusion.
Fig. 15 Tracking errors (a) weighted sum, (b) product
Figure 15 shows average tracking errors for all ten vehicle tracks for all target tracking approaches mentioned above. Figure 16(a) averages tracking errors for all
Fig. 16 (a) Average tracking errors and (b) average of determinant of covariance for all tracks
the tracks to compare different tracking approaches. Audio and video modalities are able to track vehicles successfully, though they suffer from poor performance in the presence of high background noise and clutter. In general, audio sensors are
Tracking in Urban Environments Using Sensor Networks Based on Audio-Video Fusion
141
able to track vehicles with good accuracy, but they suffer from high uncertainty and poor sensing range. Video tracking is not very robust in the presence of multiple targets and noise. As expected, fusing the two modalities consistently gives better performance. There are some cases where audio tracking performance is better than fusion. This is due to poor performance of video tracking. Fusion based on the product of likelihood functions gives better performance but it is more vulnerable to sensor conflict and errors in sensor calibration. The weighted-sum approach is more robust to conflicts and sensor errors, but it suffers from high uncertainty. The centralized estimation framework consistently performs better than the hybrid framework. Figure 17 shows the determinant of the covariance for all tracks and all approaches. Figure 16(b) presents averages of covariance measure for all tracks to
Fig. 17 Determinant of covariance (a) weighted sum, (b) product
compare the performance of tracking approaches. Among modalities, video sensors have lower uncertainty than audio sensors. Among the fusion techniques, product fusion produces lower uncertainty, as expected. There was no definite comparison between the centralize